4
$\begingroup$

I have about 1 million .txt text files in a directory which I would like to all concatenate in one single text file (separated by a newline character "\n"). Because of the number of files and their size I would rather not like to read everything in memory first. So I was wondering - what is the fastest way to do this in Mathematica, without having to resort to read everything in memory first?

cheers, Tom

Improve this question
$\endgroup$
  • 2
    $\begingroup$ If you have Unix tools available, just cat *.txt > hugefile.txt is sufficient to join the files. $\endgroup$ – cormullion Nov 24 '13 at 12:00
  • $\begingroup$ Yes that's true, good point - although in my application, it would be nice if I could do it from within Mathematica $\endgroup$ – Tom Wenseleers Nov 24 '13 at 12:23
  • 1
    $\begingroup$ There's OpenAppend. Basically I'm thinking you could append one file at a time. Or you might use Unix tools from within Mathematica. $\endgroup$ – Michael E2 Nov 24 '13 at 13:07
  • 1
    $\begingroup$ If you really want to do it from within Mathematica you can still use cormullions suggestion: Run["cat *.txt > hugefile.txt"]. $\endgroup$ – Mike Honeychurch Nov 24 '13 at 20:47
  • $\begingroup$ Yes good idea - but there is no command line option in cat by any chance that will allow me to insert a next line character between each file? $\endgroup$ – Tom Wenseleers Nov 25 '13 at 13:20
10
$\begingroup$

Here's a straightforward way:

Clear[copy, merge];
copy[file1_, out_OutputStream] := Module[{input},
   input = Check[Import[file1, "Text"], Return[$Failed]];
   BinaryWrite[out, input];
   BinaryWrite[out, "\n"];
   ];

merge[files_List, outFile_String] := Module[{out},
  Check[out = OpenWrite[outFile, BinaryFormat -> True], Return[$Failed]];
  Do[copy[in, out], {in, files}];
  Close[out];
  ]

Example use:

merge[
 FileNames[DigitCharacter ~~ ".txt", {"/tmp"}],
 "/tmp/foo.txt"
 ]

I'm not sure how to analyze its speed.

Improve this answer
$\endgroup$
  • $\begingroup$ Ha many thanks - that's great!! $\endgroup$ – Tom Wenseleers Nov 24 '13 at 15:58
8
$\begingroup$

To regularly run terminal stuff from within Mathematica use CellEvaluationFunction as described by WReach on this site.

Step 1 set up a cell style:

Cell[StyleData["Terminal"],
 CellFrame->2,
 ShowGroupOpener->False,
 CellMargins->{{66, 4}, {10, 8}},
 Evaluatable->True,
 StripStyleOnPaste->True,
 CellEvaluationFunction->Function[{$CellContext`x, $CellContext`y}, 
   Import[
    StringExpression["!", $CellContext`x], "Text"]],
 CellFrameColor->GrayLevel[0.5],
 Hyphenation->False,
 AutoQuoteCharacters->{},
 PasteAutoQuoteCharacters->{},
 LanguageCategory->"Formula",
 ScriptLevel->1,
 MenuSortingValue->1800,
 FontFamily->"Monaco",
 FontSize->13,
 FontWeight->"Plain",
 FontSlant->"Plain",
 FontColor->RGBColor[0, 1, 0],
 Background->GrayLevel[0]]

enter image description here

Step 2 run a terminal command e.g.

open -a "QuickTime Player"

enter image description here

or

cat *.txt > hugefile.txt

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.