I'll add this one:
FromCharacterCode @ DeleteDuplicates @ ToCharacterCode @ "113233454766"
"1324576"
Comparison
First, @belisarius' and @IstvánZachar have the only solutions that do not convert the string to a list. So +1 for that. But they're somewhat to very slow. I'll use Beowulf as an example string:
text = ExampleData[{"Text", "BeowulfOldEnglish"}];
Timings:
István has one that is the same idea as mine, but it uses Characters
instead of ToCharacterCode
. The other one of his I include is the faster of the two that do not convert the string to a list.
SetAttributes[timeAvg, HoldFirst]
timeAvg[func_] := Do[If[# > 0.3, Return[#/5^i]] & @@ Timing@Do[func, {5^i}], {i, 0, 15}]
FromCharacterCode@DeleteDuplicates@ToCharacterCode@text // timeAvg (* Michael E2)
StringJoin@DeleteDuplicates@Characters@text // timeAvg (* Istvan's fastest *)
Block[{f}, f[x_] := (f[x] = ""; x); (* Istvan's String *)
StringReplace[text, x_ :> f@x]] // timeAvg
FixedPoint[ (* belisarius *)
StringReplace[#, a___ ~~ x_ ~~ b___ ~~ x_ ~~ c___ :> a ~~ x ~~ b ~~ c] &,
text] // timeAvg
0.00130796
0.0344223
0.090330
56.499878
Memory:
ByteCount@text
ByteCount@ToCharacterCode@text
ByteCount@Characters@text
124528
900312
5401088
One can see that ToCharacterCode
uses roughly eight times the memory. (It converts the string to packed array.) Characters
is very wasteful of memory, using roughly 45 times the amount as String
.
On the other hand, if I start from a fresh kernel, MaxMemoryUsed[]
returns about 56MB. We can compare how much memory is used by evaluating MaxMemoryUsed[]
after running a method. For this test, I joined 100 copies of Beowulf. Mine used 241MB and for István's used 629MB. For @belisarius' it was about 113MB when I aborted it as it would take too long to run to completion.
Test code (substitute a desired method for the third line):
text2 = StringJoin[Table[#, {i, 100}] &@ ExampleData[{"Text", "BeowulfOldEnglish"}]];
MaxMemoryUsed[]
FromCharacterCode@DeleteDuplicates@ToCharacterCode@text2;
MaxMemoryUsed[]
If eight times greater is too much, you might consider splitting the string, deleting duplicates in each pieces, combining the results, and deleting again.
From a comment by the OP, it seems there are lots of strings, and perhaps the longest string may not be overly large. If so, then the strings may be processed separately (and in parallel, as desired).
String
, it looks like splitting to character codes then deleting is faster. $\endgroup$