3
$\begingroup$

I have to import columns 1,2,3, and 6 from a long (2 * 10^5 rows) file with this structure:

{-17763.1, -8182.33, 26000.9, -0.561626, 5.08542, -28.8257, -17763.1, -8182.33, 26000.9}
{-17762.7, -8181.8, 25999.8, -0.560809, 4.04628, 1.31158, -17763.1, -8182.33, 26000.9}
{-17763.1, -8182.33, 26000.9, -0.561326, 0.0121829, 17.303178201333805208, -17763.1, -8182.33, 26000.9}

I use Import as follows for column 6:

Import[file, {"Table", "Data", All, 6}, "FieldSeparators" -> {"{", ",", "}"}]

and analogously for the other. It works but it takes forever. How can I speed up the process?

$\endgroup$
3
  • 1
    $\begingroup$ have you seen this question and answers to it? It is probably not exactly the same but I think the answers should give you all the information you need... $\endgroup$ Feb 15, 2019 at 12:35
  • $\begingroup$ You could preprocess the file with awk before reading it into Mathematica. $\endgroup$
    – Roman
    Feb 16, 2019 at 7:06
  • $\begingroup$ Have you tried ReadList["filename", Expression][[All, {1, 2, 3, 6}]]? $\endgroup$
    – Roman
    Apr 12, 2019 at 8:11

2 Answers 2

3
$\begingroup$

For very efficient processing you can combine the awk command-line utility with ReadList: the first extracts the desired columns without doing any conversions, and the second performs conversions only on the extracted columns.

ReadList["!awk -F '[{},]' '{print $2 $3 $4 $7}' filename.txt", Number, RecordLists -> True]

{{-17763.1, -8182.33, 26000.9, -28.8257}, {-17762.7, -8181.8, 25999.8, 1.31158}, {-17763.1, -8182.33, 26000.9, 17.30317820133380521}}

Notice that we're extracting columns 2, 3, 4, 7 instead of 1, 2, 3, 6 because there is an offset coming from the initial curly brace {.

A general reading function would be

read[filename_String, columns_?(VectorQ[#, IntegerQ] &)] := 
  ReadList["!awk -F '[{},]' '{print $" <> 
           StringRiffle[ToString /@ (columns + 1), " $"] <> 
           "}' " <> filename,
    Number, RecordLists -> True]

read["filename.txt", {1, 2, 3, 6}]

{{-17763.1, -8182.33, 26000.9, -28.8257}, {-17762.7, -8181.8, 25999.8, 1.31158}, {-17763.1, -8182.33, 26000.9, 17.30317820133380521}}

$\endgroup$
0
$\begingroup$

How about reading the file one line at a time:

str = OpenRead["file"];
col6 = {};
While[True,
  l = Read[str];
  If[l == EndOfFile, Break[]];
  AppendTo[col6, l[[6]]];
  ];

col6
(* {-28.8257, 1.31158, 17.30317820133380521} *)
$\endgroup$
1
  • $\begingroup$ Read can be quite slow if memory serves. Try ReadList[str, String, 1][[1]] instead for a faster version. E.g.: read-list vs read/read line $\endgroup$
    – b3m2a1
    Apr 12, 2019 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.