How to read a part of huge matrix saved in a TXT file as single expression?

Question

I saved a huge matrix of numbers with dimensions {400, 4001, 2} using Put and got a TXT file formatted in the following way:

{{{1.1, 1.2}, {1.8,
0.3}, {2.1, 0.7},
<...>
{1257.9, -3.6*^-8}, {1258.2, 
-3.4*^-9}}}

Actually I have hundreds of such files. Is there a smart way to read only a part of the matrix efficiently? For example, how to take the first 20 or 40 sub-arrays out of 400? Simple Import or Get on the whole file takes too much time.

Here is one of my files: link.

Any suggestion to better save data in the future is welcome.

Answer 1

Number counting approach

Since your file contains (with rare exceptions) only 3 numbers on each line and the data has dimensions {400,4001,2}, it is possible to calculate how many lines you should read in order to obtain the first 40 sets out of 400.

At first, some checks:

data = << "EigF-V1_1.0-V2_0.000.txt";
Dimensions[data]

{400, 4001, 2}

linesString = ReadList["D:/desktop/EigF-V1_1.0-V2_0.000.txt", "String"];

Length[linesString]

1066932

lengths = Table[
   Length[StringCases[
     line, (NumberString ~~ "*^" ~~ NumberString) | NumberString]], {line, 
    linesString}];

Tally[lengths]

{{3, 1066925}, {4, 6}, {1, 1}}

So we see that there are only 4 lines containing 4 numbers and only one (the last) line with single number.

You need 40 sets each containing 4001*2 = 8002 numbers. So you need 40*8002 = 320080 numbers which occupy approximately 320080/3 = 106693 lines. These lines can contain more than 320080 numbers because some lines contain 4 numbers.

Now you should proceed as follows with your files (assuming that they have the same structure):

linesString = 
  ReadList["D:/desktop/EigF-V1_1.0-V2_0.000.txt", "String", 106693];

numbers = StringCases[
   linesString, (NumberString ~~ "*^" ~~ NumberString) | NumberString];

Length[Flatten@numbers]

320081

We need only the first 320080 numbers:

numbers = Take[Flatten@numbers, 320080];

Now we just need to recreate the array:

dataArray = ArrayReshape[numbers, {40, 4001, 2}];

That's all.

P.S. In the above solution I assumed that your file doesn't contain numbers with precision marks like 1.23`, 1.23`20 or 0.``20. If your file does contain such numbers you should use more involved pattern for matching them than the pattern (NumberString ~~ "*^" ~~ NumberString) | NumberString I used above. You can take such patterns from this thread (I haven't tested myself).

---

Closing braces counting approach

This approach is much more efficient and robust than the previous and doesn't depend on the knowledge of the structure of your file (like how many numbers you have on one line). It also automatically supports all InputForm Mathematica numbers including numbers with precision marks. The idea is to read the file line-by-line and count closing braces until you get as many sets as you need. It is possible because every set contains 4001 + 1 = 4002 closing braces according to Dimensions of your data. After finishing we truncate the last line dropping everything after the last closing brace, then add one additional closing brace at the end in order to form complete and valid Mathematica expression.

The variable nSetsToTake determines how many sets you wish to take:

str = OpenRead["EigF-V1_1.0-V2_0.000.txt"];

nSetsToTake = 40;
nBraces = nSetsToTake*(4001 + 1);

count = 0;
lines = Reap[
    While[(count += StringCount[Sow[Read[str, "String"]], "}"]) < 
      nBraces]][[2, 1]];
Close[str];
pos = StringPosition[lines[[-1]], "}"];
lines[[-1]] = StringTake[lines[[-1]], First@pos[[nBraces - count - 1]]];
dataArray = ToExpression[StringJoin[lines] <> "}", InputForm];
Dimensions@dataArray

{40, 4001, 2}

---

How to increase accessibility of your data

Any suggestion to better save data in the future is welcome.

It looks like you have created you file with Mathematica. For the future I recommend instead of saving the whole data array as a single expression, to Write each of the sets as a separate expression ending with a new line, for example as follows:

str = OpenWrite[];
Do[Write[str, set], {set, dataArray}];
Close[str];

Now you can easily get as many sets as you need in a very efficient way, for example taking the first 4 sets:

dataSet = ReadList[%, "Expression", 4];

Dimensions[dataSet]

{4, 4001, 2}