0
$\begingroup$

I saved a huge matrix of numbers with dimensions {400, 4001, 2} using Put and got a TXT file formatted in the following way:

{{{1.1, 1.2}, {1.8,
0.3}, {2.1, 0.7},
<...>
{1257.9, -3.6*^-8}, {1258.2, 
-3.4*^-9}}}

Actually I have hundreds of such files. Is there a smart way to read only a part of the matrix efficiently? For example, how to take the first 20 or 40 sub-arrays out of 400? Simple Import or Get on the whole file takes too much time.

Here is one of my files: link.

Any suggestion to better save data in the future is welcome.

$\endgroup$
  • 1
    $\begingroup$ It would seem easier to read all of it in, the use Part on the imported result. The amount of data to read in seems modest. $\endgroup$ – MarcoB Jun 4 '16 at 2:50
  • $\begingroup$ Thank you but I have hundreds of files, the single Readlist takes about 12 sec, there should be a quicker way to do it but I can't see it now. $\endgroup$ – jset Jun 4 '16 at 10:47
  • $\begingroup$ Are all the numbers written to the same precision? For example, are they all 3 ASCII characters (2 numbers plus a decimal point)? If so, you might be able to jump to the appropriate spot in the file with a binary read. $\endgroup$ – Rashid Jun 4 '16 at 12:31
  • 1
    $\begingroup$ Would you be able to post a link to one of the files for us to test out different import strategies? $\endgroup$ – Rashid Jun 4 '16 at 12:32
  • $\begingroup$ They are not of the same lenght, as an example: {{{1.5707963267948966, 2.886750801321724*^-6}, {1.8849555921538759, 0.00003053776721007641}, {2.199114857512855, 0.00006931880488431765}, ...},...}. When I saved, I had a big matrix and I used Put to create a txt file. One file is 70MB, is it really useful to put it online? $\endgroup$ – jset Jun 4 '16 at 14:15
3
$\begingroup$

Number counting approach

Since your file contains (with rare exceptions) only 3 numbers on each line and the data has dimensions {400,4001,2}, it is possible to calculate how many lines you should read in order to obtain the first 40 sets out of 400.

At first, some checks:

data = << "EigF-V1_1.0-V2_0.000.txt";
Dimensions[data]
{400, 4001, 2}
linesString = ReadList["D:/desktop/EigF-V1_1.0-V2_0.000.txt", "String"];

Length[linesString]
1066932
lengths = Table[
   Length[StringCases[
     line, (NumberString ~~ "*^" ~~ NumberString) | NumberString]], {line, 
    linesString}];

Tally[lengths]
{{3, 1066925}, {4, 6}, {1, 1}}

So we see that there are only 4 lines containing 4 numbers and only one (the last) line with single number.

You need 40 sets each containing 4001*2 = 8002 numbers. So you need 40*8002 = 320080 numbers which occupy approximately 320080/3 = 106693 lines. These lines can contain more than 320080 numbers because some lines contain 4 numbers.

Now you should proceed as follows with your files (assuming that they have the same structure):

linesString = 
  ReadList["D:/desktop/EigF-V1_1.0-V2_0.000.txt", "String", 106693];

numbers = StringCases[
   linesString, (NumberString ~~ "*^" ~~ NumberString) | NumberString];

Length[Flatten@numbers]
320081

We need only the first 320080 numbers:

numbers = Take[Flatten@numbers, 320080];

Now we just need to recreate the array:

dataArray = ArrayReshape[numbers, {40, 4001, 2}];

That's all.

P.S. In the above solution I assumed that your file doesn't contain numbers with precision marks like 1.23`, 1.23`20 or 0.``20. If your file does contain such numbers you should use more involved pattern for matching them than the pattern (NumberString ~~ "*^" ~~ NumberString) | NumberString I used above. You can take such patterns from this thread (I haven't tested myself).


Closing braces counting approach

This approach is much more efficient and robust than the previous and doesn't depend on the knowledge of the structure of your file (like how many numbers you have on one line). It also automatically supports all InputForm Mathematica numbers including numbers with precision marks. The idea is to read the file line-by-line and count closing braces until you get as many sets as you need. It is possible because every set contains 4001 + 1 = 4002 closing braces according to Dimensions of your data. After finishing we truncate the last line dropping everything after the last closing brace, then add one additional closing brace at the end in order to form complete and valid Mathematica expression.

The variable nSetsToTake determines how many sets you wish to take:

str = OpenRead["EigF-V1_1.0-V2_0.000.txt"];

nSetsToTake = 40;
nBraces = nSetsToTake*(4001 + 1);

count = 0;
lines = Reap[
    While[(count += StringCount[Sow[Read[str, "String"]], "}"]) < 
      nBraces]][[2, 1]];
Close[str];
pos = StringPosition[lines[[-1]], "}"];
lines[[-1]] = StringTake[lines[[-1]], First@pos[[nBraces - count - 1]]];
dataArray = ToExpression[StringJoin[lines] <> "}", InputForm];
Dimensions@dataArray
{40, 4001, 2}

How to increase accessibility of your data

Any suggestion to better save data in the future is welcome.

It looks like you have created you file with Mathematica. For the future I recommend instead of saving the whole data array as a single expression, to Write each of the sets as a separate expression ending with a new line, for example as follows:

str = OpenWrite[];
Do[Write[str, set], {set, dataArray}];
Close[str];

Now you can easily get as many sets as you need in a very efficient way, for example taking the first 4 sets:

dataSet = ReadList[%, "Expression", 4];

Dimensions[dataSet]
{4, 4001, 2}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.