I have to replace a string, let's say "ABC" to "XYZ", in a huge plain text file (10 Gb and above). Obviously, the Import[] and StringReplace[] approach is unfeasible so I'm trying a char-by-char processing inside a While[] loop, but this coarse method is, as one would expect, extremely slow. Can you, please, suggest any smarter algorithm or method?

  • 8
    $\begingroup$ Used sed instead of mathematica, that's what I do when I have large files. Just type in the terminal sed -i 's/ABC/XYZ/g' filename.txt > newfile.txt That's assuming you have a unix computer though. $\endgroup$ Jan 11 '16 at 18:22
  • $\begingroup$ if you must use mathematica for this, process in blocks with something like: ReadList[file, "Character", blocksize] in a loop. $\endgroup$
    – george2079
    Jan 11 '16 at 19:03
  • $\begingroup$ Is there a specific reason why you need to use Mathematica for this? If not, what OS do you use? $\endgroup$ Jan 11 '16 at 20:34
  • 2
    $\begingroup$ @george2079 It's not so easy as that. What happens when the search string happens to lie on the border of two blocks? $\endgroup$ Jan 11 '16 at 20:50
  • $\begingroup$ Right, you need to look at two blocks at a time to catch strings on the border (I didn't say it was easy !) $\endgroup$
    – george2079
    Jan 11 '16 at 21:15

Edit take 2: this turns out bit cleaner. As we read each block look for characters at the end that might be the beginning of a match and push those to the next block.

Export["test.txt", input = "manmanman"]
infile = OpenRead["test.txt"];
outfile = OpenWrite["out.txt"];
subs = "man" (* strings only, no string expressions *)
repl = "woman"
block = 5;  (*for practical use this should be as large as memory permits*)

potentialendmatch[blockdata_, substr_] := Module[{
   npush = Min[{StringLength[substr] - 1, StringLength[blockdata]}],
   (*this tmp replacement takes care of cases where substr occurs in
     blockdata near the end and contains its own beginning
     eg blockdata="xxxabab" substr="aba" : dont push the trailing "ab"*)
   tmps = StringReplace[blockdata,
     substr -> StringReplace[substr, _ -> FromCharacterCode[0]]]},
   While[ StringTake[tmps, -npush] =!=
     StringTake[substr, npush] && npush > 0 , --npush]; npush]

currentblock = StringJoin@ReadList[infile, "Character", block]
While[ StringLength@currentblock != 0 ,
  npush = potentialendmatch[StringTake[currentblock,
      -Min[StringLength@currentblock, (2 StringLength@subs - 1)]], 
  If[npush < StringLength@currentblock,
   WriteString[outfile, StringReplace[
     StringTake[currentblock, {1, -npush - 1}]  , subs -> repl]]];
  currentblock = StringJoin[
    StringTake[currentblock, -npush],
    StringJoin@ReadList[infile, "Character", block]];
Close /@ {infile, outfile}


  • $\begingroup$ I don't think this works correctly. I used "manmanman" as input, block=5 and plen=4 and want to replace "man" with "woman". Output: womanmanw and not "womanwomanwoman" as expected. $\endgroup$ Jan 11 '16 at 22:10
  • $\begingroup$ @SjoerdC.deVries good catch, as-is the replacement needs to be the same length. I'm afraid fixing that using this framework will get really cumbersome. $\endgroup$
    – george2079
    Jan 11 '16 at 23:08
  • $\begingroup$ Thanks for your erudite reply. I'm studying it in details, but I find it a bit complex. Please, can you produce any guidance about the "flow chart" ? $\endgroup$ Jan 12 '16 at 9:21
  • $\begingroup$ hold tight, trying to think how to fix the string length issue. $\endgroup$
    – george2079
    Jan 12 '16 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.