19
$\begingroup$

I have looked and looked but I do not see a simple efficient way to get Mathematica to return the number of lines in a text file?

I thought about reading the file until it returned the EOF marker, but that could take a long time when working with very large files.

Any suggestions are appreciated!

$\endgroup$
  • 1
    $\begingroup$ I assume Length@Import[file.txt, "Lines"] is not good enough? Or Module[{str = OpenRead[fn], i = 0}, While[Skip[fn, "String"] === Null, ++i]; i ] $\endgroup$ – Rojo Apr 17 '13 at 20:14
  • $\begingroup$ Unfortunately not when working with a 1GB sized file. Anything more than a maybe 100MB seems to just wreck MMA even on a fast machine with ample RAM, particularly using the Import command. I will try the second suggestion. Thanks. $\endgroup$ – Sinistar Apr 17 '13 at 20:19
15
$\begingroup$

Lots of answers, but none of them leveraging this, so here is another.

null[_String] := Null

Length @ ReadList["data.txt", null @ String, NullRecords -> True]

On my system this is more than three times as fast as Rolf Mertig's CountLines, and a lot more concise as well.

If even one Null for every record is too much memory usage then read in blocks of e.g. 1000:

num[Longest[x__String], ___] := Length @ {x}

Tr @ ReadList["data.txt", num @@ Table[String, {1000}], NullRecords -> True]
$\endgroup$
  • 1
    $\begingroup$ +1 very nice. It is roughly 10 times faster than CountLines, but 10 times slower than using wc. Maybe you could add NullRecords -> True such that the results agree. $\endgroup$ – Rolf Mertig Apr 19 '13 at 7:37
  • $\begingroup$ @RolfMertig Indeed, that was a bad omission on my part. $\endgroup$ – Mr.Wizard Apr 19 '13 at 8:26
13
$\begingroup$

A simple Mathematica-only solution is:

    CountLines[file_String /; FileExistsQ[file]] := 
    Module[{counter = 0, str = OpenRead@file},
           While[ Read[str, Record, NullRecords -> True] =!= EndOfFile, 
                  counter++
           ]; 
           Close[str];
           counter];

which is quite slow of course, so 123 MB (1978142 lines) needs 20 seconds in Mathematica 9 and 13 seconds in Mathematica 8. However, it only used 45 MB RAM at most (MaxMemoryUsed[]), so I guess you can easily count GB files.

I could not get Leonid's code to work on my Windows machine quickly, but if you can, that is of course better. Somehow Link technologies (JLink, NETLink,MathLink) will nearly always beat pure Mathematica. Unfortunately.

Another fast method on Windows is to use Gnu coreutils and then

AbsoluteTiming[
 ReadList["!\"C:\\Program Files (x86)\\GnuWin32\\bin\\wc\" -l < " <> 
    "I:\\allfiles.txt", Number] // First]

returns

{0.154009, 1978142}
$\endgroup$
  • $\begingroup$ My GB file is a 'master data set' that I want to breakdown into derivative data, so my files end up being small. But prying these nuggets of information out of the big file becomes an issue. This could work for my derivative of the derivatives, if you will. $\endgroup$ – Sinistar Apr 17 '13 at 21:22
11
$\begingroup$

The simplest and most efficient way would probably be using the common wc external utility.

For example,

In[33]:= Import["!wc ~/test.m", "Table"]
Out[33]= {{6, 5, 56, "/Users/szhorvat/test.m"}}

You'll get wc by default on Linux/OSX, but you can install it on Windows too.

$\endgroup$
  • $\begingroup$ All great suggestions! Thanks! $\endgroup$ – Sinistar Apr 17 '13 at 22:11
  • $\begingroup$ Be careful of spaces in filename when using "Table", count values would be correct but length of returned sublist would vary. $\endgroup$ – Dave May 8 '17 at 21:20
  • $\begingroup$ Why not just use RunProcess? $\endgroup$ – SumNeuron Jun 25 '17 at 15:51
  • $\begingroup$ @SumNeuron I don't remember, but: was there RunProcess in April 2013? If you use RunProcess, you get a string as a result, which you still need to parse. Import does that in one step. $\endgroup$ – Szabolcs Jun 30 '17 at 4:59
4
$\begingroup$

Here is a Java-based solution, based on this answer. You will need the Java reloader, which was described here. Load it first, then evaluate the following:

JCompileLoad@
"
 import java.io.*;

 public class LineCounter{

   public static int count(String filename) throws IOException {
     InputStream is = new BufferedInputStream(new FileInputStream(filename));
     try {
       byte[] c = new byte[1024];
       int count = 0;
       int readChars = 0;
       boolean empty = true;
       while ((readChars = is.read(c)) != -1) {
         empty = false;
         for (int i = 0; i < readChars; ++i) {
             if (c[i] == '\\n') {
                 ++count;
             }
         }
       }
       return (count == 0 && !empty) ? 1 : count;
     } finally {
       is.close();
     }
  }
}"

I took a 80 Mb file I had, which contained a mostly complete list of all files on my computer:

LineCounter`count["C:\\Temp\\AllFiles.m"] // AbsoluteTiming

(* {0.1806640, 1367563}  *)

I think this can be fast enough for you.

$\endgroup$
  • $\begingroup$ I found a routine in C somewhere on the web that did this in I think about 10 lines of code. I haven't got far enough to figure out how to import it for execution in an MMA notebook though. I think the little in-line solution below should work. I appreciate the feedback though! $\endgroup$ – Sinistar Apr 17 '13 at 22:09
4
$\begingroup$

Here's a C++ version using LibraryLink:

Needs["CCompilerDriver`"]

src = "
   #include \"WolframLibrary.h\"
   #include <fstream>
   #include <algorithm>

   EXTERN_C DLLEXPORT int CountLines(WolframLibraryData libData,
      mint Argc, MArgument *Args, MArgument Res)
   {
      int count = -1;
      char *filename = MArgument_getUTF8String(Args[0]);
      std::ifstream inFile(filename);
      if(inFile.good())
        count = std::count(std::istreambuf_iterator<char>(inFile), 
                  std::istreambuf_iterator<char>(), '\\n');
      MArgument_setInteger(Res, count);
      return LIBRARY_NO_ERROR;
   }";

countLinesLibrary = CreateLibrary[src, "CountLines",
  "Language" -> "C++", "CompileOptions" -> {"/EHsc"}]

countLines = LibraryFunctionLoad[countLinesLibrary, "CountLines", {"UTF8String"}, Integer]

countLines["C:\\filelist.txt"] // AbsoluteTiming
(*  {1.591210, 2314105}  *)
$\endgroup$
  • 1
    $\begingroup$ Funnily, using the wc utility is still much faster than this on my system (even if I use sync_with_stdio(false)). Somehow I've never been satisfied with the performance naive C++ I/O code gives me, nor patient enough to get to the bottom of why it performs so bad. $\endgroup$ – Szabolcs Apr 19 '13 at 1:02
  • $\begingroup$ It should also depend heavily on implementation. I tried it on a Mac now (timings: 0.21 this one vs 0.07 wc), but I remember that when I experimented on WinXP, code similar to your compiled with MinGW gcc ran considerably faster than when compiled with MSVC. $\endgroup$ – Szabolcs Apr 19 '13 at 1:07
  • 3
    $\begingroup$ As written, the code is reading char by char. Who knows if the underlying implementation is reading in more efficient chunks? I agree that C++ I/O is a bit disappointing (and not the easiest to use either!). I might go buy a solid state drive to compensate. $\endgroup$ – wxffles Apr 19 '13 at 3:19
2
$\begingroup$

Hope this is helpful to somebody:

m = Length@Import["/path/to/file/datos.dat", "Lines"]     (*  # lines in file  *)
$\endgroup$
  • $\begingroup$ This is mentioned in Rojo's comment and the comment below it explains why the original author found it unacceptable. Nevertheless welcome to Mathematica Stack Exchange. $\endgroup$ – Mr.Wizard Dec 21 '14 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.