I'm curious, as I understand that answer this way:

If you are using a Linux derivative or something like cygwin on Windows, then this is an efficient solution for reading only the third field from your files ( assumes whitespace separated data, other delimiters can be specified ).

data= ReadList["!head --lines=-2 " <>#<> " | tail --lines=+39 | cut -f 3 "]& /@ FileNames[]

If one deals with data files in the 2 digit GB range, and often only some columns of the data are needed for further evaluation in experimental science, should one think about running mathematica on Linux, cygwin,... Is the efficiency boost for importing data small, big, huge? Do I have more options to select specific data elements in files?

  • $\begingroup$ I prefer macos, all the power of unix (including head, tail, and cut) with a really nice gui. $\endgroup$
    – rcollyer
    Apr 29, 2013 at 13:23
  • $\begingroup$ @rcollyer I was unaware of this, linux mathematica comes with some few other/additional commands and different gui? Is there a overview page summarizing the differences/nuances? $\endgroup$
    – James Last
    Apr 29, 2013 at 13:43
  • $\begingroup$ using such an approach is possible for every OS: all you need is powerful command line programs to extract the data. head, tail and cut are standard unix programs which are available in probably every Linux distro and MacOS but not by default on Windows. You can install such command line programs on Windows, cygwin being a very convenient way to install a whole bunch of those running in an environment emulating a complete unix system. You can instead only install those cmd-line tools you're actually using. For more possibilities see the documentation (==man pages) of those programs... $\endgroup$ Apr 29, 2013 at 13:55
  • $\begingroup$ I was referring to the command line programs that your example references, which are all standard unix commands. To my knowledge, Mathematica is substantially the same across all platforms, and they work hard to ensure consistency. Honestly, my comment above was stating a preference for a specific unix variant because of the OS's other features, not Mathematica's. $\endgroup$
    – rcollyer
    Apr 29, 2013 at 14:19
  • $\begingroup$ The efficiency here comes from the fact that you aren't asking Mathematica to import all the data and then remove throw away two thirds of it. You could achieve the same effect by simply using the linux commands given to save a file with only the column you are interested in. I think this is more about ensuring that you read only the data you want. Matheamtica isn't particularly frugal with memory as it is. $\endgroup$ Apr 30, 2013 at 0:08


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.