How to efficiently read blank line separated data file into list of lists

Question

For example, I have a test.dat file with contents as below:

1   2
3   4
5   6

11  22
33  44
55  66

(Note that there could be more than two numbers in a row.)

I want to read it as

{{{1,2},{3,4},{5,6}},{{11,22},{33,44},{55,66}}}

If we do

ReadList["test.dat",Number,RecordLists->True]

it is super fast, however then we loose the blank lines.

What I do now is

Clear[extractNumFromStr];
extractNumFromStr[string_]:=StringCases[string,x:((NumberString~~("e"|"E")~~NumberString)|NumberString):>Internal`StringToDouble@x];
file="test.dat";
DeleteCases[SplitBy[extractNumFromStr /@ ReadList[file, String, NullRecords -> True], # == {} &], {{}}]]

It is so awkward and is only a little faster than Import[…, "Table"]. Is there any neat and efficient way?

Answer 1

Example file:

 f = OpenWrite["test00.txt"]
 Do[
  WriteString[f, 
    ExportString[RandomInteger[1000, {RandomInteger[{3, 15}], 3}], 
    "Table"] <> "\n\n"], {40}]
 Close[f]

Import method:

(idata = DeleteCases[
     SplitBy[Import["test00.txt", "Table"], Length], {{}}]) // 
  AbsoluteTiming // First

0.031

Note Import itself is the bottleneck. It doesn't matter how you do the splitting in terms of time.

low level read method. The trick here for efficiency is we need to also avoid using ToExpression . To do that we read the whole file twice.

 (data = Module[{f = OpenRead["test00.txt"], vals, breaks, data},
 vals = ReadList[f, ConstantArray[Number, 3]];
 SetStreamPosition[f, 1];
 breaks = 
  Join[{0}, 
   Flatten[Position[ReadList[f, Record, NullRecords -> True], 
     ""]]];
 data = 
  vals[[#[[1]] + 1 ;; #[[2]]]] & /@ 
   Partition[MapIndexed[# - (First@#2 - 1) &, breaks], 2, 1];
 Close[f]; data]) // AbsoluteTiming // First

0.00297188

10x faster than Import !

data == idata

True

Notice we needed to know how many columns to read. This also assumes there is a single blank at the end. You could add a check for that of course.

Answer 2

ReadList["C:\\Users\\Ali Hashmi\\Desktop\\file.dat", String, 
NullRecords -> True] // SequenceCases[#, {Except[""] ..}] & // 
ToExpression@*StringSplit /@ # &

(* {{{1, 2}, {3, 4}, {5, 6}}, {{11, 22}, {33, 44}, {55, 66}}, {{22, 33, 44},
{55, 66}}, {{2, 3}}} *)

Answer 3

ReadList-based post-processing solution

Here is a solution which doesn't require reading the file twice (inspired by this answer by Mr.Wizard):

read[s_String] := # &[ReadList[#, Number, RecordLists -> True], Close[#]] &@
   StringToStream[s];

data2 = ReadList["test00.txt", read[Record], RecordSeparators -> "\n\n"]; // 
  AbsoluteTiming // First

0.276596

(the file "test00.txt" contains 71641 lines from which 7201 are single blank lines).

The above method ignores repeated blank lines in our input file with blank line separated datasets. This means that if adjacent datasets are separated by more than one blank line, we won't get an empty dataset {} in the resulting list. To the contrary, the method by george2079 will put empty lists {} between successive blank lines.

On my machine the above method is even more efficient than the method by george2079. The following is an improved and optimized version of his solution with no assumptions about how many numbers must be on each line and with proper handling of files without blank line at the end:

(data = Module[{f = OpenRead["test00.txt"], vals, breaks, data, i = 0},
     vals = ReadList[f, Number, RecordLists -> True];
     SetStreamPosition[f, 1];
     breaks = 
      Flatten[{0, Position[#, ""], Length[#] + 1}] &@
       ReadList[f, Record, NullRecords -> True];
     If[breaks[[-1]] == breaks[[-2]] + 1, breaks = Most[breaks]];
     data = Table[Take[vals, {i + 1, i += n}], {n, Differences[breaks] - 1}];
     Close[f]; data]) // AbsoluteTiming // First

0.313066

(for efficient construction of the ragged array I used the method from this answer by Mr.Wizard).

Without repeated blank lines the results are identical:

data2 == data

True

If the input file contains repeated blank lines, the results will be identical after removing empty lists {}:

data2 == DeleteCases[data, {}]

True

---

String-manipulation pre-processing approach

(data3 = Get[#]; Close[#];) &@
   StringToStream@
    StringReplace[
     Import["test00.txt", "Text"], {StartOfString -> "{{{", "\n\n" -> "}},{{", 
      "\n" -> "},{", (" " | "\t") .. -> ",", EndOfString -> "}}}"}] // 
  AbsoluteTiming // First

0.64618

data2 == data3 //. {} -> Nothing

True

Answer 4

you can do something like this:

data=Import@"C:\\ Users \\ Ali Hashmi \\ Desktop \\ file.dat";
(* {{}, {1, 2}, {3, 4}, {5, 6}, {}, {11, 22}, {33, 44}, {55,66},
{}, {22, 33, 44}, {55, 66}, {}, {}, {2, 3}} *)

SequenceCases[list, pat : {{__Integer} ..} :> pat]
(* {{{1, 2}, {3, 4}, {5, 6}}, {{11, 22}, {33, 44}, {55, 66}}, {{22, 33, 44},
{55, 66}}, {{2, 3}}} *)