0
$\begingroup$

Trying to import a small (15 MB) CSV file and the Import command is either not responding or responding very slowly. Watching the memory usage of the Mathematica kernel when initializing the command it hits some maximum and stalls without changing for several minutes.

I assumed the size of the file would be small enough for it to do this without any real issues. The data I'm trying to import is seven rows, each with 721 columns, where each entry is 148 points.

I've tried this technique

<< JLink`;  
InstallJava[];  
ReinstallJava[JVMArguments -> "-Xmx1024m"]

Where I change the last few numbers, but I didn't see any improvement. Maybe I'm misunderstanding what's bogging it down.

I can successfully use Import to grab image files and for other CSV files. The images and CSV files are 2 orders of magnitude smaller in file size, though.

$\endgroup$

closed as unclear what you're asking by MarcoB, happy fish, yohbs, Yves Klett, Feyre Nov 16 '16 at 10:22

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ To make sure the problem comes from the size, you could split the csv file into two or three parts and try importing them. Could you share the file, or a part of the file? $\endgroup$ – anderstood Nov 15 '16 at 18:06
  • $\begingroup$ @anderstood That's a good point. I'll try that after it finishes running this other thing that may take a couple hours. $\endgroup$ – Jordan Watkins Nov 15 '16 at 18:11
  • 1
    $\begingroup$ what do you mean "each entry is 148 points" ? $\endgroup$ – george2079 Nov 15 '16 at 19:17
  • 2
    $\begingroup$ It is not a proper csv if its nested 3 deep. Import is I suspect treating each of those 140 point sublists as a string. $\endgroup$ – george2079 Nov 15 '16 at 19:35
  • 1
    $\begingroup$ Then you need to "Flatten" it: you should just have lines of comma-separated numbers (and not list). You can always go back to the original 3 deep list using Partition afterwards. For example, if one line is {1,2,3},{4,5,6}, you could flatten it as 1,2,3,4,5,6, export it, import it back and partition if as {1,2,3},{4,5,6}. $\endgroup$ – anderstood Nov 15 '16 at 20:45
1
$\begingroup$

This should work. I dont know how fast it will be on a big file.

test file:

f = OpenWrite["test.csv"];
WriteLine[f, "{1,2,3},{4,5,6}"];
WriteLine[f, "{7,8,9},{10,11,12}"]
Close[f];

read lines as strings, wrap each in {} and extract data:

f=OpenRead["test.csv"];
data=ToExpression["{"<>#<>"}"&/@ReadList[f,String]]
Close[f];

{{{1, 2, 3}, {4, 5, 6}}, {{7, 8, 9}, {10, 11, 12}}}

another approach: (here you need to know how to partition it)

f = OpenRead["test.csv"];
rowlen = 6;
sublen = 3;
data = Partition[#, sublen] & /@ Partition[
      ToExpression@ReadList[f, Record, NullRecords -> False,
      RecordSeparators -> {",", "}", "{", "\n", "\r\n", "\r"}], 
      rowlen];
Close[f];

Really I'd avoid using that file format if you can.

Note as I guessed if you Import this you get strings:

 Import["test.csv"] // InputForm

{{"{1, 2, 3}", "{4, 5, 6}"}, {"{7, 8, 9}", "{10, 11, 12}"}}

so you could do ToExpression@Import["test.csv"] if Import succeeded.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.