String replacement in a large text file

I have to replace a string, let's say "ABC" to "XYZ", in a huge plain text file (10 Gb and above). Obviously, the Import[] and StringReplace[] approach is unfeasible so I'm trying a char-by-char processing inside a While[] loop, but this coarse method is, as one would expect, extremely slow. Can you, please, suggest any smarter algorithm or method?

• Used sed instead of mathematica, that's what I do when I have large files. Just type in the terminal sed -i 's/ABC/XYZ/g' filename.txt > newfile.txt That's assuming you have a unix computer though. Jan 11 '16 at 18:22
• if you must use mathematica for this, process in blocks with something like: ReadList[file, "Character", blocksize] in a loop. Jan 11 '16 at 19:03
• Is there a specific reason why you need to use Mathematica for this? If not, what OS do you use? Jan 11 '16 at 20:34
• @george2079 It's not so easy as that. What happens when the search string happens to lie on the border of two blocks? Jan 11 '16 at 20:50
• Right, you need to look at two blocks at a time to catch strings on the border (I didn't say it was easy !) Jan 11 '16 at 21:15

Edit take 2: this turns out bit cleaner. As we read each block look for characters at the end that might be the beginning of a match and push those to the next block.

Export["test.txt", input = "manmanman"]
outfile = OpenWrite["out.txt"];
subs = "man" (* strings only, no string expressions *)
repl = "woman"
block = 5;  (*for practical use this should be as large as memory permits*)

potentialendmatch[blockdata_, substr_] := Module[{
npush = Min[{StringLength[substr] - 1, StringLength[blockdata]}],
(*this tmp replacement takes care of cases where substr occurs in
blockdata near the end and contains its own beginning
eg blockdata="xxxabab" substr="aba" : dont push the trailing "ab"*)
tmps = StringReplace[blockdata,
substr -> StringReplace[substr, _ -> FromCharacterCode[0]]]},
While[ StringTake[tmps, -npush] =!=
StringTake[substr, npush] && npush > 0 , --npush]; npush]

While[ StringLength@currentblock != 0 ,
npush = potentialendmatch[StringTake[currentblock,
-Min[StringLength@currentblock, (2 StringLength@subs - 1)]],
subs];
If[npush < StringLength@currentblock,
WriteString[outfile, StringReplace[
StringTake[currentblock, {1, -npush - 1}]  , subs -> repl]]];
currentblock = StringJoin[
StringTake[currentblock, -npush],
];
Close /@ {infile, outfile}
Import["out.txt"]


"womanwomanwoman"

• I don't think this works correctly. I used "manmanman" as input, block=5 and plen=4 and want to replace "man" with "woman". Output: womanmanw and not "womanwomanwoman" as expected. Jan 11 '16 at 22:10
• @SjoerdC.deVries good catch, as-is the replacement needs to be the same length. I'm afraid fixing that using this framework will get really cumbersome. Jan 11 '16 at 23:08
• Thanks for your erudite reply. I'm studying it in details, but I find it a bit complex. Please, can you produce any guidance about the "flow chart" ? Jan 12 '16 at 9:21
• hold tight, trying to think how to fix the string length issue. Jan 12 '16 at 12:18