4
$\begingroup$

I have about 1 million .txt text files in a directory which I would like to all concatenate in one single text file (separated by a newline character "\n"). Because of the number of files and their size I would rather not like to read everything in memory first. So I was wondering - what is the fastest way to do this in Mathematica, without having to resort to read everything in memory first?

cheers, Tom

$\endgroup$
5
  • 2
    $\begingroup$ If you have Unix tools available, just cat *.txt > hugefile.txt is sufficient to join the files. $\endgroup$
    – cormullion
    Commented Nov 24, 2013 at 12:00
  • $\begingroup$ Yes that's true, good point - although in my application, it would be nice if I could do it from within Mathematica $\endgroup$ Commented Nov 24, 2013 at 12:23
  • 1
    $\begingroup$ There's OpenAppend. Basically I'm thinking you could append one file at a time. Or you might use Unix tools from within Mathematica. $\endgroup$
    – Michael E2
    Commented Nov 24, 2013 at 13:07
  • 1
    $\begingroup$ If you really want to do it from within Mathematica you can still use cormullions suggestion: Run["cat *.txt > hugefile.txt"]. $\endgroup$ Commented Nov 24, 2013 at 20:47
  • $\begingroup$ Yes good idea - but there is no command line option in cat by any chance that will allow me to insert a next line character between each file? $\endgroup$ Commented Nov 25, 2013 at 13:20

2 Answers 2

10
$\begingroup$

Here's a straightforward way:

Clear[copy, merge];
copy[file1_, out_OutputStream] := Module[{input},
   input = Check[Import[file1, "Text"], Return[$Failed]];
   BinaryWrite[out, input];
   BinaryWrite[out, "\n"];
   ];

merge[files_List, outFile_String] := Module[{out},
  Check[out = OpenWrite[outFile, BinaryFormat -> True], Return[$Failed]];
  Do[copy[in, out], {in, files}];
  Close[out];
  ]

Example use:

merge[
 FileNames[DigitCharacter ~~ ".txt", {"/tmp"}],
 "/tmp/foo.txt"
 ]

I'm not sure how to analyze its speed.

$\endgroup$
1
  • $\begingroup$ Ha many thanks - that's great!! $\endgroup$ Commented Nov 24, 2013 at 15:58
9
$\begingroup$

To regularly run terminal stuff from within Mathematica use CellEvaluationFunction as described by WReach on this site.

Step 1 set up a cell style:

Cell[StyleData["Terminal"],
 CellFrame->2,
 ShowGroupOpener->False,
 CellMargins->{{66, 4}, {10, 8}},
 Evaluatable->True,
 StripStyleOnPaste->True,
 CellEvaluationFunction->Function[{$CellContext`x, $CellContext`y}, 
   Import[
    StringExpression["!", $CellContext`x], "Text"]],
 CellFrameColor->GrayLevel[0.5],
 Hyphenation->False,
 AutoQuoteCharacters->{},
 PasteAutoQuoteCharacters->{},
 LanguageCategory->"Formula",
 ScriptLevel->1,
 MenuSortingValue->1800,
 FontFamily->"Monaco",
 FontSize->13,
 FontWeight->"Plain",
 FontSlant->"Plain",
 FontColor->RGBColor[0, 1, 0],
 Background->GrayLevel[0]]

enter image description here

Step 2 run a terminal command e.g.

open -a "QuickTime Player"

enter image description here

or

cat *.txt > hugefile.txt

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.