Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have looked and looked but I do not see a simple efficient way to get MMA to return the number of lines in a text file?

I thought about reading the file until it returned the EOF marker, but that could take a long time when working with very large files.

Any suggestions are appreciated!

share|improve this question
    
I assume Length@Import[file.txt, "Lines"] is not good enough? Or Module[{str = OpenRead[fn], i = 0}, While[Skip[fn, "String"] === Null, ++i]; i ] –  Rojo Apr 17 '13 at 20:14
    
Unfortunately not when working with a 1GB sized file. Anything more than a maybe 100MB seems to just wreck MMA even on a fast machine with ample RAM, particularly using the Import command. I will try the second suggestion. Thanks. –  Sinistar Apr 17 '13 at 20:19
add comment

5 Answers

A simple Mathematica-only solution is:

    CountLines[file_String /; FileExistsQ[file]] := 
    Module[{counter = 0, str = OpenRead@file},
           While[ Read[str, Record, NullRecords -> True] =!= EndOfFile, 
                  counter++
           ]; 
           Close[str];
           counter];

which is quite slow of course, so 123 MB (1978142 lines) needs 20 seconds in Mathematica 9 and 13 seconds in Mathematica 8. However, it only used 45 MB RAM at most (MaxMemoryUsed[]), so I guess you can easily count GB files.

I could not get Leonid's code to work on my Windows machine quickly, but if you can, that is of course better. Somehow Link technologies (JLink, NETLink,MathLink) will nearly always beat pure Mathematica. Unfortunately.

Another fast method on Windows is to use Gnu coreutils and then

AbsoluteTiming[
 ReadList["!\"C:\\Program Files (x86)\\GnuWin32\\bin\\wc\" -l < " <> 
    "I:\\allfiles.txt", Number] // First]

returns

{0.154009, 1978142}
share|improve this answer
    
My GB file is a 'master data set' that I want to breakdown into derivative data, so my files end up being small. But prying these nuggets of information out of the big file becomes an issue. This could work for my derivative of the derivatives, if you will. –  Sinistar Apr 17 '13 at 21:22
add comment

The simplest and most efficient way would probably be using the common wc external utility.

For example,

In[33]:= Import["!wc ~/test.m", "Table"]
Out[33]= {{6, 5, 56, "/Users/szhorvat/test.m"}}

You'll get wc by default on Linux/OSX, but you can install it on Windows too.

share|improve this answer
    
All great suggestions! Thanks! –  Sinistar Apr 17 '13 at 22:11
add comment

Lots of answers, but none of them leveraging this, so here is another.

null[_String] := Null

Length @ ReadList["data.txt", null @ String, NullRecords -> True]

On my system this is more than three times as fast as Rolf Mertig's CountLines, and a lot more concise as well.

If even one Null for every record is too much memory usage then read in blocks of e.g. 1000:

num[Longest[x__String], ___] := Length @ {x}

Tr @ ReadList["data.txt", num @@ Table[String, {1000}], NullRecords -> True]
share|improve this answer
    
+1 using ReadList and native Mathematica implementation. This feels more like Mathematica than other answers. –  RunnyKine Apr 19 '13 at 3:03
1  
+1 very nice. It is roughly 10 times faster than CountLines, but 10 times slower than using wc. Maybe you could add NullRecords -> True such that the results agree. –  Rolf Mertig Apr 19 '13 at 7:37
    
@RolfMertig Indeed, that was a bad omission on my part. –  Mr.Wizard Apr 19 '13 at 8:26
add comment

Here is a Java-based solution, based on this answer. You will need the Java reloader, which was described here. Load it first, then evaluate the following:

JCompileLoad@
"
 import java.io.*;

 public class LineCounter{

   public static int count(String filename) throws IOException {
     InputStream is = new BufferedInputStream(new FileInputStream(filename));
     try {
       byte[] c = new byte[1024];
       int count = 0;
       int readChars = 0;
       boolean empty = true;
       while ((readChars = is.read(c)) != -1) {
         empty = false;
         for (int i = 0; i < readChars; ++i) {
             if (c[i] == '\\n') {
                 ++count;
             }
         }
       }
       return (count == 0 && !empty) ? 1 : count;
     } finally {
       is.close();
     }
  }
}"

I took a 80 Mb file I had, which contained a mostly complete list of all files on my computer:

LineCounter`count["C:\\Temp\\AllFiles.m"] // AbsoluteTiming

(* {0.1806640, 1367563}  *)

I think this can be fast enough for you.

share|improve this answer
    
I found a routine in C somewhere on the web that did this in I think about 10 lines of code. I haven't got far enough to figure out how to import it for execution in an MMA notebook though. I think the little in-line solution below should work. I appreciate the feedback though! –  Sinistar Apr 17 '13 at 22:09
add comment

Here's a C++ version using LibraryLink:

Needs["CCompilerDriver`"]

src = "
   #include \"WolframLibrary.h\"
   #include <fstream>
   #include <algorithm>

   EXTERN_C DLLEXPORT int CountLines(WolframLibraryData libData,
      mint Argc, MArgument *Args, MArgument Res)
   {
      int count = -1;
      char *filename = MArgument_getUTF8String(Args[0]);
      std::ifstream inFile(filename);
      if(inFile.good())
        count = std::count(std::istreambuf_iterator<char>(inFile), 
                  std::istreambuf_iterator<char>(), '\\n');
      MArgument_setInteger(Res, count);
      return LIBRARY_NO_ERROR;
   }";

countLinesLibrary = CreateLibrary[src, "CountLines",
  "Language" -> "C++", "CompileOptions" -> {"/EHsc"}]

countLines = LibraryFunctionLoad[countLinesLibrary, "CountLines", {"UTF8String"}, Integer]

countLines["C:\\filelist.txt"] // AbsoluteTiming
(*  {1.591210, 2314105}  *)
share|improve this answer
1  
Funnily, using the wc utility is still much faster than this on my system (even if I use sync_with_stdio(false)). Somehow I've never been satisfied with the performance naive C++ I/O code gives me, nor patient enough to get to the bottom of why it performs so bad. –  Szabolcs Apr 19 '13 at 1:02
    
It should also depend heavily on implementation. I tried it on a Mac now (timings: 0.21 this one vs 0.07 wc), but I remember that when I experimented on WinXP, code similar to your compiled with MinGW gcc ran considerably faster than when compiled with MSVC. –  Szabolcs Apr 19 '13 at 1:07
2  
As written, the code is reading char by char. Who knows if the underlying implementation is reading in more efficient chunks? I agree that C++ I/O is a bit disappointing (and not the easiest to use either!). I might go buy a solid state drive to compensate. –  wxffles Apr 19 '13 at 3:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.