20
$\begingroup$

I have looked and looked but I do not see a simple efficient way to get Mathematica to return the number of lines in a text file?

I thought about reading the file until it returned the EOF marker, but that could take a long time when working with very large files.

Any suggestions are appreciated!

$\endgroup$
2
  • 1
    $\begingroup$ I assume Length@Import[file.txt, "Lines"] is not good enough? Or Module[{str = OpenRead[fn], i = 0}, While[Skip[fn, "String"] === Null, ++i]; i ] $\endgroup$
    – Rojo
    Apr 17, 2013 at 20:14
  • $\begingroup$ Unfortunately not when working with a 1GB sized file. Anything more than a maybe 100MB seems to just wreck MMA even on a fast machine with ample RAM, particularly using the Import command. I will try the second suggestion. Thanks. $\endgroup$
    – Sinistar
    Apr 17, 2013 at 20:19

6 Answers 6

15
$\begingroup$

Lots of answers, but none of them leveraging this, so here is another.

null[_String] := Null

Length @ ReadList["data.txt", null @ String, NullRecords -> True]

On my system this is more than three times as fast as Rolf Mertig's CountLines, and a lot more concise as well.

If even one Null for every record is too much memory usage then read in blocks of e.g. 1000:

num[Longest[x__String], ___] := Length @ {x}

Tr @ ReadList["data.txt", num @@ Table[String, {1000}], NullRecords -> True]
$\endgroup$
2
  • 1
    $\begingroup$ +1 very nice. It is roughly 10 times faster than CountLines, but 10 times slower than using wc. Maybe you could add NullRecords -> True such that the results agree. $\endgroup$ Apr 19, 2013 at 7:37
  • $\begingroup$ @RolfMertig Indeed, that was a bad omission on my part. $\endgroup$
    – Mr.Wizard
    Apr 19, 2013 at 8:26
13
$\begingroup$

A simple Mathematica-only solution is:

    CountLines[file_String /; FileExistsQ[file]] := 
    Module[{counter = 0, str = OpenRead@file},
           While[ Read[str, Record, NullRecords -> True] =!= EndOfFile, 
                  counter++
           ]; 
           Close[str];
           counter];

which is quite slow of course, so 123 MB (1978142 lines) needs 20 seconds in Mathematica 9 and 13 seconds in Mathematica 8. However, it only used 45 MB RAM at most (MaxMemoryUsed[]), so I guess you can easily count GB files.

I could not get Leonid's code to work on my Windows machine quickly, but if you can, that is of course better. Somehow Link technologies (JLink, NETLink,MathLink) will nearly always beat pure Mathematica. Unfortunately.

Another fast method on Windows is to use Gnu coreutils and then

AbsoluteTiming[
 ReadList["!\"C:\\Program Files (x86)\\GnuWin32\\bin\\wc\" -l < " <> 
    "I:\\allfiles.txt", Number] // First]

returns

{0.154009, 1978142}
$\endgroup$
1
  • $\begingroup$ My GB file is a 'master data set' that I want to breakdown into derivative data, so my files end up being small. But prying these nuggets of information out of the big file becomes an issue. This could work for my derivative of the derivatives, if you will. $\endgroup$
    – Sinistar
    Apr 17, 2013 at 21:22
12
$\begingroup$

The simplest and most efficient way would probably be using the common wc external utility.

For example,

In[33]:= Import["!wc ~/test.m", "Table"]
Out[33]= {{6, 5, 56, "/Users/szhorvat/test.m"}}

You'll get wc by default on Linux/OSX, but you can install it on Windows too.

$\endgroup$
4
  • $\begingroup$ All great suggestions! Thanks! $\endgroup$
    – Sinistar
    Apr 17, 2013 at 22:11
  • $\begingroup$ Be careful of spaces in filename when using "Table", count values would be correct but length of returned sublist would vary. $\endgroup$
    – Dave
    May 8, 2017 at 21:20
  • $\begingroup$ Why not just use RunProcess? $\endgroup$
    – SumNeuron
    Jun 25, 2017 at 15:51
  • $\begingroup$ @SumNeuron I don't remember, but: was there RunProcess in April 2013? If you use RunProcess, you get a string as a result, which you still need to parse. Import does that in one step. $\endgroup$
    – Szabolcs
    Jun 30, 2017 at 4:59
5
$\begingroup$

Here is a Java-based solution, based on this answer. You will need the Java reloader, which was described here. Load it first, then evaluate the following:

JCompileLoad@
"
 import java.io.*;

 public class LineCounter{

   public static int count(String filename) throws IOException {
     InputStream is = new BufferedInputStream(new FileInputStream(filename));
     try {
       byte[] c = new byte[1024];
       int count = 0;
       int readChars = 0;
       boolean empty = true;
       while ((readChars = is.read(c)) != -1) {
         empty = false;
         for (int i = 0; i < readChars; ++i) {
             if (c[i] == '\\n') {
                 ++count;
             }
         }
       }
       return (count == 0 && !empty) ? 1 : count;
     } finally {
       is.close();
     }
  }
}"

I took a 80 Mb file I had, which contained a mostly complete list of all files on my computer:

LineCounter`count["C:\\Temp\\AllFiles.m"] // AbsoluteTiming

(* {0.1806640, 1367563}  *)

I think this can be fast enough for you.

$\endgroup$
1
  • $\begingroup$ I found a routine in C somewhere on the web that did this in I think about 10 lines of code. I haven't got far enough to figure out how to import it for execution in an MMA notebook though. I think the little in-line solution below should work. I appreciate the feedback though! $\endgroup$
    – Sinistar
    Apr 17, 2013 at 22:09
5
$\begingroup$

Here's a C++ version using LibraryLink:

Needs["CCompilerDriver`"]

src = "
   #include \"WolframLibrary.h\"
   #include <fstream>
   #include <algorithm>

   EXTERN_C DLLEXPORT int CountLines(WolframLibraryData libData,
      mint Argc, MArgument *Args, MArgument Res)
   {
      int count = -1;
      char *filename = MArgument_getUTF8String(Args[0]);
      std::ifstream inFile(filename);
      if(inFile.good())
        count = std::count(std::istreambuf_iterator<char>(inFile), 
                  std::istreambuf_iterator<char>(), '\\n');
      MArgument_setInteger(Res, count);
      return LIBRARY_NO_ERROR;
   }";

countLinesLibrary = CreateLibrary[src, "CountLines",
  "Language" -> "C++", "CompileOptions" -> {"/EHsc"}]

countLines = LibraryFunctionLoad[countLinesLibrary, "CountLines", {"UTF8String"}, Integer]

countLines["C:\\filelist.txt"] // AbsoluteTiming
(*  {1.591210, 2314105}  *)
$\endgroup$
3
  • 1
    $\begingroup$ Funnily, using the wc utility is still much faster than this on my system (even if I use sync_with_stdio(false)). Somehow I've never been satisfied with the performance naive C++ I/O code gives me, nor patient enough to get to the bottom of why it performs so bad. $\endgroup$
    – Szabolcs
    Apr 19, 2013 at 1:02
  • $\begingroup$ It should also depend heavily on implementation. I tried it on a Mac now (timings: 0.21 this one vs 0.07 wc), but I remember that when I experimented on WinXP, code similar to your compiled with MinGW gcc ran considerably faster than when compiled with MSVC. $\endgroup$
    – Szabolcs
    Apr 19, 2013 at 1:07
  • 3
    $\begingroup$ As written, the code is reading char by char. Who knows if the underlying implementation is reading in more efficient chunks? I agree that C++ I/O is a bit disappointing (and not the easiest to use either!). I might go buy a solid state drive to compensate. $\endgroup$
    – wxffles
    Apr 19, 2013 at 3:19
2
$\begingroup$

Hope this is helpful to somebody:

m = Length@Import["/path/to/file/datos.dat", "Lines"]     (*  # lines in file  *)
$\endgroup$
1
  • $\begingroup$ This is mentioned in Rojo's comment and the comment below it explains why the original author found it unacceptable. Nevertheless welcome to Mathematica Stack Exchange. $\endgroup$
    – Mr.Wizard
    Dec 21, 2014 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.