If you are on a Unix-like system or have installed cut
or awk
on other systems letting these do the extraction is most probably the most efficient way to solve your problem. If for some reason you are looking for a Mathematica-only way here is one:
readColumn[filename_, colnum_, numcols_, numlinesperchunk_Integer: 500] :=
Module[{
str = OpenRead[filename],
res = {""}, data
},
data = Reap[
While[Length[res] > 0,
Sow[res =
ReadList[str, Table[Real, {numcols}], numlinesperchunk][[All,colnum]]]
]
][[2]];
Close[str];
Flatten[data]
]
to extract the 5th out of 6 columns you would use:
readColumn[filename,5,6]
using the 4th argument you can play around with numlinesperchunk
for further optimization. Note that there is a tradeoff between maximal speed (very large values for numlinesperchunk
) and minimal memory usage (low values for numlinesperchunk
). For largish files I would expect a good value might be some hundreds of lines per chunk, but your milage may vary...
This might look a bit complicated for such a simple tasks but it tries to be efficient not only concerning runtimes but especially concerning memory which usually is a larger problem when importing large data files into Mathematica. Note that the code here is very similar to that for my answer to this question, where you find other ways to do similar things even more efficient at the price of more involved programming in Java and also some more background information...
Read the FAQs
! 3) When you see good Q&A, vote them up byclicking the gray triangles
, because the credibility of the system is based on the reputation gained by users sharing their knowledge. ALSO, remember to accept the answer, if any, that solves your problem,by clicking the checkmark sign
` $\endgroup$