I'm working with some data files that are just too large for Import
.
I tried using ReadList
, specifying either Number
or Real
for all the columns, but this fails, because a few of the values in the data are NaNs, and are thus not considered numerical by Mathematica1.
I can get ReadList
to work if I use Record
as the types for all the columns, and the performance is not terrible (at least with the relatively small input files I'm using to prototype this import code), but the subsequent call to ToExpression
is a performance killer.
I give a detailed example below, including timing data.
Any suggestions on how to get around this problem?
ADDENDUM
One promising possibility I discovered shortly after posting this question would be to define some input stream method skipLinesWithNaNs
that (somehow) filters out the lines that contain NaNs. Then I could just call
ReadList[OpenRead[pathToTSV, listOfNumericTypes, ...],
Method -> "skipLinesWithNaNs"];
...but I have not been able to figure out how to do this. I'm sure it's not difficult, but I'm too sleep-deprived at the moment to see how to do it. I guess my question may boil down to how to implement the skipLinesWithNaNs
input stream method. The difficulty, as I see it, is that input stream methods are byte-oriented (AFAICT), whereas the filtering I need done is best described, and easiest to implement, at the line level. It would be great if somewhere in the ocean of Mathematica there was an adapter to convert a byte-oriented stream into a line-oriented one, but I have not yet found such...
EDIT
In response to episanty's request:
makeRow[ncols_, withNaNs_] :=
If[withNaNs,
RandomSample[Prepend[makeRow[ncols - 1, False], "nan"]],
RandomReal[{0, 1}, ncols]];
makeSampleInput[path_, nrows_, ncols_, nnans_] :=
Export[path,
RandomSample[Table[makeRow[ncols, i <= nnans], {i, nrows}]], "TSV"];
Given these helper functions, one can use
makeSampleInput[pathToTSV, 40000, 400, 10];
to create a file at pathToTSV
that is comparable to the input file I used for the EXAMPLE below. Also, the following definition of the types
parameter (which the EXAMPLE refers to) is consistent with the sample input file.
types = Table["Real", {400}];
EXAMPLE
First some setup:
fmt = StringTemplate["`1` s\t`2`"];
printTime[expr_] := Module[{timing, value},
{timing, value} = AbsoluteTiming[expr];
Print[fmt[timing, ToString[HoldForm[expr]]]];
value
];
SetAttributes[printTime, HoldFirst];
tab = "\t";
nl = "\n";
Now, a data import sequence, as described at the beginning of this post:
(* read in rows of "Records" (in this case, simple strings) from TSV file *)
strings =
ReadList[pathToTSV, Table[Record, {types}],
RecordSeparators -> {tab, nl}] // printTime;
(* determine the indices of the "NaN-free" rows *)
keep = Flatten[
Position[strings,
Except[{___, "nan", ___}], 1, Heads -> False]] // printTime;
(* extract the "NaN-free" rows from table of strings *)
keptStrings = strings[[keep, ;;]] // printTime;
(* convert remaining string values to numbers *)
rows = ToExpression[keptStrings] // printTime;
The output produced by the calls to printTime
are shown below; the first column shows the timing results for the expression in the second column:
2.796349 s ReadList[pathToTSV, Table[Record, {types}], RecordSeparators -> {tab, nl}]
0.37655 s Flatten[Position[strings, Except[{___, nan, ___}], 1, Heads -> False]]
0.005952 s strings[[keep,1 ;; All]]
42.976491 s ToExpression[keptStrings]
As you can see, the call to ToExpression
just blows away the performance.
BTW, the size of the file used for this experiment is about one meg:
NumberForm[ByteCount[strings], DigitBlock -> 3]
==> 770,461,048
1Producing pre-processed versions of the input files to remove the NaNs would pose a whole host of complications with the rest of our pipeline, so it is out of the question.
ReadLine
be of help in this case? It returns lines of input as strings; you would then be left with checking for NaN, then discarding or processing the vakd strings (maybe withImportString
?) $\endgroup$ – MarcoB May 12 '15 at 13:44