4
$\begingroup$

I have about 1 million .txt text files in a directory which I would like to all concatenate in one single text file (separated by a newline character "\n"). Because of the number of files and their size I would rather not like to read everything in memory first. So I was wondering - what is the fastest way to do this in Mathematica, without having to resort to read everything in memory first?

cheers, Tom

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ If you have Unix tools available, just cat *.txt > hugefile.txt is sufficient to join the files. $\endgroup$
    – cormullion
    Nov 24, 2013 at 12:00
  • $\begingroup$ Yes that's true, good point - although in my application, it would be nice if I could do it from within Mathematica $\endgroup$
    – Tom Wenseleers
    Nov 24, 2013 at 12:23
  • 1
    $\begingroup$ There's OpenAppend. Basically I'm thinking you could append one file at a time. Or you might use Unix tools from within Mathematica. $\endgroup$
    – Michael E2
    Nov 24, 2013 at 13:07
  • 1
    $\begingroup$ If you really want to do it from within Mathematica you can still use cormullions suggestion: Run["cat *.txt > hugefile.txt"]. $\endgroup$
    – Mike Honeychurch
    Nov 24, 2013 at 20:47
  • $\begingroup$ Yes good idea - but there is no command line option in cat by any chance that will allow me to insert a next line character between each file? $\endgroup$
    – Tom Wenseleers
    Nov 25, 2013 at 13:20

2 Answers 2

Reset to default
10
$\begingroup$

Here's a straightforward way:

Clear[copy, merge];
copy[file1_, out_OutputStream] := Module[{input},
   input = Check[Import[file1, "Text"], Return[$Failed]];
   BinaryWrite[out, input];
   BinaryWrite[out, "\n"];
   ];

merge[files_List, outFile_String] := Module[{out},
  Check[out = OpenWrite[outFile, BinaryFormat -> True], Return[$Failed]];
  Do[copy[in, out], {in, files}];
  Close[out];
  ]

Example use:

merge[
 FileNames[DigitCharacter ~~ ".txt", {"/tmp"}],
 "/tmp/foo.txt"
 ]

I'm not sure how to analyze its speed.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Ha many thanks - that's great!! $\endgroup$
    – Tom Wenseleers
    Nov 24, 2013 at 15:58
9
$\begingroup$

To regularly run terminal stuff from within Mathematica use CellEvaluationFunction as described by WReach on this site.

Step 1 set up a cell style:

Cell[StyleData["Terminal"],
 CellFrame->2,
 ShowGroupOpener->False,
 CellMargins->{{66, 4}, {10, 8}},
 Evaluatable->True,
 StripStyleOnPaste->True,
 CellEvaluationFunction->Function[{$CellContext`x, $CellContext`y}, 
   Import[
    StringExpression["!", $CellContext`x], "Text"]],
 CellFrameColor->GrayLevel[0.5],
 Hyphenation->False,
 AutoQuoteCharacters->{},
 PasteAutoQuoteCharacters->{},
 LanguageCategory->"Formula",
 ScriptLevel->1,
 MenuSortingValue->1800,
 FontFamily->"Monaco",
 FontSize->13,
 FontWeight->"Plain",
 FontSlant->"Plain",
 FontColor->RGBColor[0, 1, 0],
 Background->GrayLevel[0]]

enter image description here

Step 2 run a terminal command e.g.

open -a "QuickTime Player"

enter image description here

or

cat *.txt > hugefile.txt

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.