Number counting approach
Since your file contains (with rare exceptions) only 3 numbers on each line and the data has dimensions {400,4001,2}
, it is possible to calculate how many lines you should read in order to obtain the first 40 sets out of 400.
At first, some checks:
data = << "EigF-V1_1.0-V2_0.000.txt";
Dimensions[data]
{400, 4001, 2}
linesString = ReadList["D:/desktop/EigF-V1_1.0-V2_0.000.txt", "String"];
Length[linesString]
1066932
lengths = Table[
Length[StringCases[
line, (NumberString ~~ "*^" ~~ NumberString) | NumberString]], {line,
linesString}];
Tally[lengths]
{{3, 1066925}, {4, 6}, {1, 1}}
So we see that there are only 4 lines containing 4 numbers and only one (the last) line with single number.
You need 40 sets each containing 4001*2 = 8002
numbers. So you need 40*8002 = 320080
numbers which occupy approximately 320080/3 = 106693
lines. These lines can contain more than 320080 numbers because some lines contain 4 numbers.
Now you should proceed as follows with your files (assuming that they have the same structure):
linesString =
ReadList["D:/desktop/EigF-V1_1.0-V2_0.000.txt", "String", 106693];
numbers = StringCases[
linesString, (NumberString ~~ "*^" ~~ NumberString) | NumberString];
Length[Flatten@numbers]
320081
We need only the first 320080 numbers:
numbers = Take[Flatten@numbers, 320080];
Now we just need to recreate the array:
dataArray = ArrayReshape[numbers, {40, 4001, 2}];
That's all.
P.S. In the above solution I assumed that your file doesn't contain numbers with precision marks like 1.23`
, 1.23`20
or 0.``20
. If your file does contain such numbers you should use more involved pattern for matching them than the pattern (NumberString ~~ "*^" ~~ NumberString) | NumberString
I used above. You can take such patterns from this thread (I haven't tested myself).
Closing braces counting approach
This approach is much more efficient and robust than the previous and doesn't depend on the knowledge of the structure of your file (like how many numbers you have on one line). It also automatically supports all InputForm
Mathematica numbers including numbers with precision marks. The idea is to read the file line-by-line and count closing braces until you get as many sets as you need. It is possible because every set contains 4001 + 1 = 4002
closing braces according to Dimensions
of your data. After finishing we truncate the last line dropping everything after the last closing brace, then add one additional closing brace at the end in order to form complete and valid Mathematica expression.
The variable nSetsToTake
determines how many sets you wish to take:
str = OpenRead["EigF-V1_1.0-V2_0.000.txt"];
nSetsToTake = 40;
nBraces = nSetsToTake*(4001 + 1);
count = 0;
lines = Reap[
While[(count += StringCount[Sow[Read[str, "String"]], "}"]) <
nBraces]][[2, 1]];
Close[str];
pos = StringPosition[lines[[-1]], "}"];
lines[[-1]] = StringTake[lines[[-1]], First@pos[[nBraces - count - 1]]];
dataArray = ToExpression[StringJoin[lines] <> "}", InputForm];
Dimensions@dataArray
{40, 4001, 2}
How to increase accessibility of your data
Any suggestion to better save data in the future is welcome.
It looks like you have created you file with Mathematica. For the future I recommend instead of saving the whole data array as a single expression, to Write
each of the sets as a separate expression ending with a new line, for example as follows:
str = OpenWrite[];
Do[Write[str, set], {set, dataArray}];
Close[str];
Now you can easily get as many sets as you need in a very efficient way, for example taking the first 4 sets:
dataSet = ReadList[%, "Expression", 4];
Dimensions[dataSet]
{4, 4001, 2}
Part
on the imported result. The amount of data to read in seems modest. $\endgroup$