2
$\begingroup$

With perl we can run perl -p . That wraps a foreach loop around the code and executes the code on the file. What I'm looking for is something like

StringReplace[filename,"\t"->","]

and then I get a new file in which the string replace happened (the above would convert a .tsv to a .csv). Doesn't have to be StringReplace, can be anything that takes a line of stuff as input. Total[filename] would total all numbers in the rows of the file.

I do NOT want the whole file in kernel memory!

also like sed on Linux (stream editor -- works line by line)

That is like "scripting" the M command line-by-line (during MS-DOS we used to call that "batch mode" -- really means line-by-line).

$\endgroup$
  • $\begingroup$ So you want a global text search and replace within a file? I guess you could use e.g. WriteString["newfile", StringReplace[ReadString["file"], "\t" -> ","]]. However, I would guess that any dedicated command-line tool would be orders of magnitude faster than MMA at this. $\endgroup$ – MarcoB Dec 5 '19 at 20:57
  • $\begingroup$ yes. I guess I should have added: I do NOT want the whole file in kernel memory! $\endgroup$ – Andreas Lauschke Dec 5 '19 at 20:58
  • $\begingroup$ ... then same approach but with ReadLine + StringReplace + WriteLine? $\endgroup$ – MarcoB Dec 5 '19 at 21:52
3
$\begingroup$

I think you're looking for the -linewise option to wolframscript.

Here's my little script, tsv2csv

#!/usr/bin/env wolframscript -linewise

Print@StringReplace[$ScriptInputString,"\t"->","]

I tried it, via ./tsv2csv < in > out on the file in

a   b   c
d   e   f
g   h   i

and got

a,b,c
d,e,f
g,h,i

I believe the script is essentially called 3 times, because if I replace the StringReplace[...] with "Foo" the file out contains 3 lines, each says Foo.

$\endgroup$
  • $\begingroup$ I get: /usr/bin/env: ‘wolframscript -linewise’: No such file or directory /usr/bin/env: use -[v]S to pass options in shebang lines $\endgroup$ – Andreas Lauschke Dec 6 '19 at 14:17
  • $\begingroup$ What kind of system are you on? I’m not sure I can reproduce, you may need to man env to see how to ensure the -linewise is seen as a parameter to wolframscript, rather than the whole thing being seen as one command. The error message suggests -S, but my env works fine without any additional flags. If I had to wager I’d guess -S goes before wolframscript in the shebang line. $\endgroup$ – evanb Dec 6 '19 at 14:24
  • $\begingroup$ That was it. Thanks a billion. I'm on Linux, Fedora 30. As per policy I'm supposed to wait a few days before I mark an answer as the solution, but I think this is the solution, so I'll upvote and select it. In the end, cut from the command line is still faster, the tsv -> csv was just an example, but it's good to know about the -linewise option. $\endgroup$ – Andreas Lauschke Dec 6 '19 at 14:34
  • $\begingroup$ Sure thing! I recently have been battling wolframscript so all the options are in the front of my mind ;) $\endgroup$ – evanb Dec 6 '19 at 14:36
0
$\begingroup$

This is not robust but should be enough to get you started.

ClearAll[applyToFile]
applyToFile[filename_, typeSpec_, func_, batchSize_Integer?Positive] :=
 Module[{read = OpenRead[filename], write = OpenWrite[], readBach, 
   writeFileName},
  readBach = ReadList[read, typeSpec, batchSize];
  While[readBach =!= {},
   readBach = func /@ readBach;
   WriteLine[write, #] & /@ readBach;
   readBach = ReadList[read, typeSpec, batchSize]
   ];
  Close@read;
  Close@write
  ]

Then

SeedRandom[123]
fn = Export[FileNameJoin[{$TemporaryDirectory, "nums.txt"}], RandomInteger[20, {5, 5}], "CSV"];
FilePrint[fn]
14,18,0,4,7
6,7,10,11,6
10,12,9,17,0
9,11,16,15,6
12,5,15,12,20

and

newFileName = applyToFile[fn, Record, StringReplace["," -> "\t"], 3];
FilePrint[newFileName]
14    18  0   4   7
6   7   10  11  6
10  12  9   17  0
9   11  16  15  6
12  5   15  12  20

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.