I'm curious, as I understand that answer this way:

If you are using a Linux derivative or something like cygwin on Windows, then this is an efficient solution for reading only the third field from your files ( assumes whitespace separated data, other delimiters can be specified ).

data= ReadList["!head --lines=-2 " <>#<> " | tail --lines=+39 | cut -f 3 "]& /@ FileNames[]

If one deals with data files in the 2 digit GB range, and often only some columns of the data are needed for further evaluation in experimental science, should one think about running mathematica on Linux, cygwin,... Is the efficiency boost for importing data small, big, huge? Do I have more options to select specific data elements in files?

  • $\begingroup$ I prefer macos, all the power of unix (including head, tail, and cut) with a really nice gui. $\endgroup$ – rcollyer Apr 29 '13 at 13:23
  • $\begingroup$ @rcollyer I was unaware of this, linux mathematica comes with some few other/additional commands and different gui? Is there a overview page summarizing the differences/nuances? $\endgroup$ – James Last Apr 29 '13 at 13:43
  • $\begingroup$ using such an approach is possible for every OS: all you need is powerful command line programs to extract the data. head, tail and cut are standard unix programs which are available in probably every Linux distro and MacOS but not by default on Windows. You can install such command line programs on Windows, cygwin being a very convenient way to install a whole bunch of those running in an environment emulating a complete unix system. You can instead only install those cmd-line tools you're actually using. For more possibilities see the documentation (==man pages) of those programs... $\endgroup$ – Albert Retey Apr 29 '13 at 13:55
  • $\begingroup$ I was referring to the command line programs that your example references, which are all standard unix commands. To my knowledge, Mathematica is substantially the same across all platforms, and they work hard to ensure consistency. Honestly, my comment above was stating a preference for a specific unix variant because of the OS's other features, not Mathematica's. $\endgroup$ – rcollyer Apr 29 '13 at 14:19
  • $\begingroup$ The efficiency here comes from the fact that you aren't asking Mathematica to import all the data and then remove throw away two thirds of it. You could achieve the same effect by simply using the linux commands given to save a file with only the column you are interested in. I think this is more about ensuring that you read only the data you want. Matheamtica isn't particularly frugal with memory as it is. $\endgroup$ – image_doctor Apr 30 '13 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.