Import columns from a long .txt file

Question

I have to import columns 1,2,3, and 6 from a long (2 * 10^5 rows) file with this structure:

{-17763.1, -8182.33, 26000.9, -0.561626, 5.08542, -28.8257, -17763.1, -8182.33, 26000.9}
{-17762.7, -8181.8, 25999.8, -0.560809, 4.04628, 1.31158, -17763.1, -8182.33, 26000.9}
{-17763.1, -8182.33, 26000.9, -0.561326, 0.0121829, 17.303178201333805208, -17763.1, -8182.33, 26000.9}

I use Import as follows for column 6:

Import[file, {"Table", "Data", All, 6}, "FieldSeparators" -> {"{", ",", "}"}]

and analogously for the other. It works but it takes forever. How can I speed up the process?

Answer 1

For very efficient processing you can combine the awk command-line utility with ReadList: the first extracts the desired columns without doing any conversions, and the second performs conversions only on the extracted columns.

ReadList["!awk -F '[{},]' '{print $2 $3 $4 $7}' filename.txt", Number, RecordLists -> True]

{{-17763.1, -8182.33, 26000.9, -28.8257}, {-17762.7, -8181.8, 25999.8, 1.31158}, {-17763.1, -8182.33, 26000.9, 17.30317820133380521}}

Notice that we're extracting columns 2, 3, 4, 7 instead of 1, 2, 3, 6 because there is an offset coming from the initial curly brace {.

A general reading function would be

read[filename_String, columns_?(VectorQ[#, IntegerQ] &)] := 
  ReadList["!awk -F '[{},]' '{print $" <> 
           StringRiffle[ToString /@ (columns + 1), " $"] <> 
           "}' " <> filename,
    Number, RecordLists -> True]

read["filename.txt", {1, 2, 3, 6}]

{{-17763.1, -8182.33, 26000.9, -28.8257}, {-17762.7, -8181.8, 25999.8, 1.31158}, {-17763.1, -8182.33, 26000.9, 17.30317820133380521}}

Answer 2

How about reading the file one line at a time:

str = OpenRead["file"];
col6 = {};
While[True,
  l = Read[str];
  If[l == EndOfFile, Break[]];
  AppendTo[col6, l[[6]]];
  ];

col6
(* {-28.8257, 1.31158, 17.30317820133380521} *)