11
$\begingroup$

I have a very huge file (1GB) from which I desire to import only the 5th column. It is a tab delimited text file with a .txt extension. Can you please help me with this?

I generally use Import["filename.txt","Table"] to import my files, but now, I don't want the whole file, which takes a long time to load.

$\endgroup$
1
  • $\begingroup$ Welcome to Mathematica.SE! I suggest the following: 1) As you receive help, try to give it too, by answering questions in your area of expertise. 2)Read the FAQs! 3) When you see good Q&A, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. ALSO, remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign` $\endgroup$
    – chris
    Nov 6, 2012 at 20:39

3 Answers 3

8
$\begingroup$

On Linux you could use this simple solution.

Linux Shell

From the shell:

$cat myfile.txt | cut -f 5 > col5.txt

Then import into Mathematica as usual.

data = Import["col5.txt","Table"]

ReadList

This can be condensed to one ReadList command.

data = ReadList["!cat myfile.txt | cut -f 5"];
$\endgroup$
6
  • 2
    $\begingroup$ Or awk '{print $5}' myfile.txt :) $\endgroup$
    – rm -rf
    Nov 6, 2012 at 18:08
  • $\begingroup$ @rm-rf Yes, there is a beauty in the simplicity of awk :) $\endgroup$ Nov 6, 2012 at 18:14
  • 1
    $\begingroup$ oh I didn't know we can incorporate linux commands in mathematica. I use cut command all the time. instead of cat and then piping it, we can directly use the cut command on the file as well. Thanks for your solution! $\endgroup$
    – preeti
    Nov 6, 2012 at 19:04
  • $\begingroup$ Works perfectly on windows as well using sourceforge.net/projects/unxutils or by installing msysgit :) $\endgroup$
    – Ajasja
    Nov 6, 2012 at 19:21
  • $\begingroup$ @image_doctor How can I use this solution if 'my file.txt' is given as ToFileName[...]? Also, how is the command for reading the first four columns only? $\endgroup$
    – Valerio
    Dec 14, 2013 at 14:32
11
$\begingroup$

You might try this:

Import["data.txt", {"Data", All, 5}]

If you want to load only the fifth column of the first two lines you can run this:

Import["data.txt", {"Data", {1, 2}, 5}]

You can find more about this here.

Update

Some testing on a relatively small data file (12MB) might prove that this is not really the solution you're after. It seems that it takes much more time to load just one column (~130 seconds) than to load the entire file with Import["data.txt","Table"] (~4 seconds).

$\endgroup$
2
  • $\begingroup$ yes, I tried this from mathematica documentation and had the time issue. That was the reason behind this post. I missed to mention that. $\endgroup$
    – preeti
    Nov 6, 2012 at 19:03
  • 1
    $\begingroup$ ReadList is almost always faster then Import. $\endgroup$ Nov 6, 2012 at 21:24
2
$\begingroup$

If you are on a Unix-like system or have installed cut or awk on other systems letting these do the extraction is most probably the most efficient way to solve your problem. If for some reason you are looking for a Mathematica-only way here is one:

readColumn[filename_, colnum_, numcols_, numlinesperchunk_Integer: 500] := 
  Module[{
    str = OpenRead[filename],
    res = {""}, data
  },
  data = Reap[
  While[Length[res] > 0, 
   Sow[res = 
     ReadList[str, Table[Real, {numcols}], numlinesperchunk][[All,colnum]]]
   ]
  ][[2]];
  Close[str];
  Flatten[data]
]

to extract the 5th out of 6 columns you would use:

readColumn[filename,5,6]

using the 4th argument you can play around with numlinesperchunk for further optimization. Note that there is a tradeoff between maximal speed (very large values for numlinesperchunk) and minimal memory usage (low values for numlinesperchunk). For largish files I would expect a good value might be some hundreds of lines per chunk, but your milage may vary...

This might look a bit complicated for such a simple tasks but it tries to be efficient not only concerning runtimes but especially concerning memory which usually is a larger problem when importing large data files into Mathematica. Note that the code here is very similar to that for my answer to this question, where you find other ways to do similar things even more efficient at the price of more involved programming in Java and also some more background information...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.