How to efficiently read blank line separated data file into list of lists

For example, I have a test.dat file with contents as below:

1   2
3   4
5   6

11  22
33  44
55  66


(Note that there could be more than two numbers in a row.)

I want to read it as

{{{1,2},{3,4},{5,6}},{{11,22},{33,44},{55,66}}}


If we do

ReadList["test.dat",Number,RecordLists->True]


it is super fast, however then we loose the blank lines.

What I do now is

Clear[extractNumFromStr];
extractNumFromStr[string_]:=StringCases[string,x:((NumberString~~("e"|"E")~~NumberString)|NumberString):>InternalStringToDouble@x];
file="test.dat";
DeleteCases[SplitBy[extractNumFromStr /@ ReadList[file, String, NullRecords -> True], # == {} &], {{}}]]


It is so awkward and is only a little faster than Import[…, "Table"]. Is there any neat and efficient way?

Example file:

 f = OpenWrite["test00.txt"]
Do[
WriteString[f,
ExportString[RandomInteger[1000, {RandomInteger[{3, 15}], 3}],
"Table"] <> "\n\n"], {40}]
Close[f]


Import method:

(idata = DeleteCases[
SplitBy[Import["test00.txt", "Table"], Length], {{}}]) //
AbsoluteTiming // First


0.031

Note Import itself is the bottleneck. It doesn't matter how you do the splitting in terms of time.

low level read method. The trick here for efficiency is we need to also avoid using ToExpression . To do that we read the whole file twice.

 (data = Module[{f = OpenRead["test00.txt"], vals, breaks, data},
SetStreamPosition[f, 1];
breaks =
Join[{0},
""]]];
data =
vals[[#[[1]] + 1 ;; #[[2]]]] & /@
Partition[MapIndexed[# - (First@#2 - 1) &, breaks], 2, 1];
Close[f]; data]) // AbsoluteTiming // First


0.00297188

10x faster than Import !

data == idata


True

Notice we needed to know how many columns to read. This also assumes there is a single blank at the end. You could add a check for that of course.

• Oh, my god! You ReadList a file twice, two fast duplicate ReadList is still fast! this is very creative : ) +1 – matheorem Apr 21 '17 at 11:43
• I would expect ReadList[f, Number, RecordLists -> True, NullRecords -> True] to do the whole work for us. But NullRecords -> True is ignored for Number, although RecordLists -> True is respected. Looks like a bug, isn't it? – Alexey Popkov Apr 21 '17 at 11:59
• I've made separate question on this: mathematica.stackexchange.com/q/144149/280 – Alexey Popkov Apr 21 '17 at 12:28
ReadList["C:\\Users\\Ali Hashmi\\Desktop\\file.dat", String,
NullRecords -> True] // SequenceCases[#, {Except[""] ..}] & //
ToExpression@*StringSplit /@ # &

(* {{{1, 2}, {3, 4}, {5, 6}}, {{11, 22}, {33, 44}, {55, 66}}, {{22, 33, 44},
{55, 66}}, {{2, 3}}} *)


ReadList-based post-processing solution

Here is a solution which doesn't require reading the file twice (inspired by this answer by Mr.Wizard):

read[s_String] := # &[ReadList[#, Number, RecordLists -> True], Close[#]] &@
StringToStream[s];

AbsoluteTiming // First

0.276596


(the file "test00.txt" contains 71641 lines from which 7201 are single blank lines).

The above method ignores repeated blank lines in our input file with blank line separated datasets. This means that if adjacent datasets are separated by more than one blank line, we won't get an empty dataset {} in the resulting list. To the contrary, the method by george2079 will put empty lists {} between successive blank lines.

On my machine the above method is even more efficient than the method by george2079. The following is an improved and optimized version of his solution with no assumptions about how many numbers must be on each line and with proper handling of files without blank line at the end:

(data = Module[{f = OpenRead["test00.txt"], vals, breaks, data, i = 0},
vals = ReadList[f, Number, RecordLists -> True];
SetStreamPosition[f, 1];
breaks =
Flatten[{0, Position[#, ""], Length[#] + 1}] &@
If[breaks[[-1]] == breaks[[-2]] + 1, breaks = Most[breaks]];
data = Table[Take[vals, {i + 1, i += n}], {n, Differences[breaks] - 1}];
Close[f]; data]) // AbsoluteTiming // First

0.313066


(for efficient construction of the ragged array I used the method from this answer by Mr.Wizard).

Without repeated blank lines the results are identical:

data2 == data

True


If the input file contains repeated blank lines, the results will be identical after removing empty lists {}:

data2 == DeleteCases[data, {}]

True


String-manipulation pre-processing approach

(data3 = Get[#]; Close[#];) &@
StringToStream@
StringReplace[
Import["test00.txt", "Text"], {StartOfString -> "{{{", "\n\n" -> "}},{{",
"\n" -> "},{", (" " | "\t") .. -> ",", EndOfString -> "}}}"}] //
AbsoluteTiming // First

0.64618

data2 == data3 //. {} -> Nothing

True


you can do something like this:

data=Import@"C:\\ Users \\ Ali Hashmi \\ Desktop \\ file.dat";
(* {{}, {1, 2}, {3, 4}, {5, 6}, {}, {11, 22}, {33, 44}, {55,66},
{}, {22, 33, 44}, {55, 66}, {}, {}, {2, 3}} *)

SequenceCases[list, pat : {{__Integer} ..} :> pat]
(* {{{1, 2}, {3, 4}, {5, 6}}, {{11, 22}, {33, 44}, {55, 66}}, {{22, 33, 44},
{55, 66}}, {{2, 3}}} *)

• Hi, Ali Hashmi. Thank you for answing, But Import is no efficient – matheorem Apr 20 '17 at 12:46
• @matheorem i do not know why Import is not efficient. I always thought it is optimized for handling files. Btw, check the other answer that uses ReadList` – Ali Hashmi Apr 20 '17 at 14:24