1
$\begingroup$

Consider a large numerical variable (almost 1GB) which was constructed like below:

output={scalar1,{Matrix1,Matrix2,Matrix3,scalar2}};

I used Export to save output variable as a .csv file. Now I want to import specific array of output variable to notebook, say row 1 of Matrix1, and do some post-processing operations. Clearly, Import command is not an effective and applicable choice due to high time & memory usage. Hence, one may use OpenRead and then ReadList to read desired arrays. The question is, how I can implement ReadList to import specific array of a multi-dimensional variable. Here is a simple example of output:

output = {0.049003,
{{{0, 0., -1.97949}, {0, 5., 9.01025}},
{{0, -1.97949, -6.53072}, {10, 9.01025, 5.13141}},
{{0, -7.91796, -10.287}, {0, -3.95898, -11.5564}},
0.048003}};

and the corresponding .csv file:

0.049003 "{{0, 0., -1.9794901687515782}, {0, 5., 9.01025491562421}}","{{0, -1.9794901687515782, -6.530723746803215}, {10, 9.01025491562421, 5.131408741303968}}","{{0, -7.917960675006313, -10.286973637200234}, {0, -3.9589803375031565, -11.556404359777813}}",0.048003

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Notice that the records in the CSV file are saved as strings due to the ragged structure of your array. This will make the recovery of your data slow, and possibly complicated. I would suggest saving to a different file format instead. If you don't need to share the datafiles across different computer platforms, you could also look at saving your data in a proprietary binary format using DumpSave. $\endgroup$
    – MarcoB
    Oct 14, 2015 at 20:47
  • $\begingroup$ @MarcoB for my purpose, DumpSave is not an option because of the fact that you've mentioned. it's not portable between different architectures. Moreover, I don't need to load the entire context. $\endgroup$
    – user3127040
    Oct 15, 2015 at 7:54
  • $\begingroup$ I don't understand why you absolutely want to export your output in that particular list of lists form if, anyway, you just want later to import only specific parts of it. Have you considered just to create separate files (with specific clear names) for each specific expression, then import them directly: i mean for example output[[2,1]]>>"matrix1.m" then importing it with <<"matrix1.m". $\endgroup$
    – SquareOne
    Oct 15, 2015 at 9:50
  • $\begingroup$ @SquareOne I cannot reproduce output. it is the result of a long time evaluation on a high-performance computing system. But your idea seems simple and interesting. I'll consider your suggestion in my next evaluations. However, Matrix1 itself may contain a large amount of data. $\endgroup$
    – user3127040
    Oct 15, 2015 at 10:23

1 Answer 1

Reset to default
1
$\begingroup$

After some digging, somehow I have found my solution. I developed a module for this purpose. But since I'm a beginner in Mathematica, the algorithm may seem elementary. Here it is:

GetArray[fName_, rowIndex_, matIndex_] := 
Module[{fileName = fName, matID = matIndex, row = rowIndex, stream, 
tempArray, mat, outputList},

stream = OpenRead[StringJoin[NotebookDirectory[], fileName]];
tempArray = 
ReadList[stream, Record, 3, RecordSeparators -> {{"{{"}, {"}}"}}];
Close[stream];
mat = ToString[tempArray[[matIndex]]];
Clear[tempArray];
mat = StringSplit[mat, "}, {"];
outputList = 
StringSplit[
StringReplace[
mat[[row]], {"}" -> "", "{" -> "", WhitespaceCharacter -> ""}], 
 ","];
Return[ToExpression[outputList]];
];

for a 511mb csv file, in order to extract fourth row of Matrix1:

GetArray["myFile.csv", 4, 1] // AbsoluteTiming

Utilizing this module, It took 23.460668 seconds to process the file on a 4GB Ram with Intel Core i5-2410M CPU, while it's almost impossible to load myFile.csv using Import on this machine.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.