Determine the Number of Lines in a Text File

Question

I have looked and looked but I do not see a simple efficient way to get Mathematica to return the number of lines in a text file?

I thought about reading the file until it returned the EOF marker, but that could take a long time when working with very large files.

Any suggestions are appreciated!

Answer 1

Lots of answers, but none of them leveraging this, so here is another.

null[_String] := Null

Length @ ReadList["data.txt", null @ String, NullRecords -> True]

On my system this is more than three times as fast as Rolf Mertig's CountLines, and a lot more concise as well.

If even one Null for every record is too much memory usage then read in blocks of e.g. 1000:

num[Longest[x__String], ___] := Length @ {x}

Tr @ ReadList["data.txt", num @@ Table[String, {1000}], NullRecords -> True]

Answer 2

A simple Mathematica-only solution is:

    CountLines[file_String /; FileExistsQ[file]] := 
    Module[{counter = 0, str = OpenRead@file},
           While[ Read[str, Record, NullRecords -> True] =!= EndOfFile, 
                  counter++
           ]; 
           Close[str];
           counter];

which is quite slow of course, so 123 MB (1978142 lines) needs 20 seconds in Mathematica 9 and 13 seconds in Mathematica 8. However, it only used 45 MB RAM at most (MaxMemoryUsed[]), so I guess you can easily count GB files.

I could not get Leonid's code to work on my Windows machine quickly, but if you can, that is of course better. Somehow Link technologies (JLink, NETLink,MathLink) will nearly always beat pure Mathematica. Unfortunately.

Another fast method on Windows is to use Gnu coreutils and then

AbsoluteTiming[
 ReadList["!\"C:\\Program Files (x86)\\GnuWin32\\bin\\wc\" -l < " <> 
    "I:\\allfiles.txt", Number] // First]

returns

{0.154009, 1978142}

Answer 3

The simplest and most efficient way would probably be using the common wc external utility.

For example,

In[33]:= Import["!wc ~/test.m", "Table"]
Out[33]= {{6, 5, 56, "/Users/szhorvat/test.m"}}

You'll get wc by default on Linux/OSX, but you can install it on Windows too.

Answer 4

Here is a Java-based solution, based on this answer. You will need the Java reloader, which was described here. Load it first, then evaluate the following:

JCompileLoad@
"
 import java.io.*;

 public class LineCounter{

   public static int count(String filename) throws IOException {
     InputStream is = new BufferedInputStream(new FileInputStream(filename));
     try {
       byte[] c = new byte[1024];
       int count = 0;
       int readChars = 0;
       boolean empty = true;
       while ((readChars = is.read(c)) != -1) {
         empty = false;
         for (int i = 0; i < readChars; ++i) {
             if (c[i] == '\\n') {
                 ++count;
             }
         }
       }
       return (count == 0 && !empty) ? 1 : count;
     } finally {
       is.close();
     }
  }
}"

I took a 80 Mb file I had, which contained a mostly complete list of all files on my computer:

LineCounter`count["C:\\Temp\\AllFiles.m"] // AbsoluteTiming

(* {0.1806640, 1367563}  *)

I think this can be fast enough for you.

Answer 5

Here's a C++ version using LibraryLink:

Needs["CCompilerDriver`"]

src = "
   #include \"WolframLibrary.h\"
   #include <fstream>
   #include <algorithm>

   EXTERN_C DLLEXPORT int CountLines(WolframLibraryData libData,
      mint Argc, MArgument *Args, MArgument Res)
   {
      int count = -1;
      char *filename = MArgument_getUTF8String(Args[0]);
      std::ifstream inFile(filename);
      if(inFile.good())
        count = std::count(std::istreambuf_iterator<char>(inFile), 
                  std::istreambuf_iterator<char>(), '\\n');
      MArgument_setInteger(Res, count);
      return LIBRARY_NO_ERROR;
   }";

countLinesLibrary = CreateLibrary[src, "CountLines",
  "Language" -> "C++", "CompileOptions" -> {"/EHsc"}]

countLines = LibraryFunctionLoad[countLinesLibrary, "CountLines", {"UTF8String"}, Integer]

countLines["C:\\filelist.txt"] // AbsoluteTiming
(*  {1.591210, 2314105}  *)

Answer 6

Hope this is helpful to somebody:

m = Length@Import["/path/to/file/datos.dat", "Lines"]     (*  # lines in file  *)