How to import a (large) .CSV file maintaining its shape (defined by curly brackets)

Question

I'm working on large tables generated with Mathematica, evaluated on different computers. I plan on join the tables on my computer and proceed with the interpolation of the data there contained.

The structure of the table is {{x,y,z},\[CapitalGamma]}, where x, y and z are parameters for the function \[CapitalGamma]. I created a 3D mesh on which I'm sampling the function, and I'll need to create an InterpolatingFunction.

I split the original mesh (about 30 millions of points) in sixteen different files, computed on four computers. Each file is just shy of 100 MB in size. The table is then exported as a flattened list, and a C program reshapes it in the following way:

{{0.000000000000000000000000000000,-0.001175000000000000000000000000,0.545714285714285710000000000000},0.000000000000757534038528606580}
{{0.000000000000000000000000000000,-0.001145839061933960200000000000,0.545714285714285710000000000000},0.000000000000718990648693323850}
{{0.000000000000000000000000000000,-0.001116678123867920300000000000,0.545714285714285710000000000000},0.000000000000681423522535585060}
{{0.000000000000000000000000000000,-0.001087517185801880300000000000,0.545714285714285710000000000000},0.000000000000644832928422345040}
{{0.000000000000000000000000000000,-0.001058356247735840400000000000,0.545714285714285710000000000000},0.000000000000609219154540147720}
{{0.000000000000000000000000000000,-0.001029195309669800800000000000,0.545714285714285710000000000000},0.000000000000574582510216643140}

This is a sample of the table.

The problem arises at the moment of importing the tables back in Mathematica. The best I am able to do at the moment is:

TunnelingRates = Import["compacted_short.csv", "Table", 
   "FieldSeparators" -> {"{", "}", ","}];

which returns a table of the type:

{{0.,-0.00117500000000000000000000000,0.545714285714285710000000000000,7.57534038528606580*10^-13},
{0.,-0.00114583906193396020000000000,0.545714285714285710000000000000,7.1899064869332385*10^-13},
{0.,-0.00111667812386792030000000000,0.545714285714285710000000000000,6.81423522535585060*10^-13},
{0., -0.001087517185801880300000000000,0.545714285714285710000000000000,6.4483292842234504*10^-13}}

That is, shaped as {x,y,z,\[CapitalGamma]} instead of the desired {{x,y,z},\[CapitalGamma]}.

Others attempts result often in mixed tables of strings and real numbers.

Is it possible to import a table stored in a CSV file with a shape defined by curly brackets?

Answer 1

We can reshape the data using Replace. Here are two examples that do the same thing

r1 = Import["compacted_short.csv", "Table", 
    "FieldSeparators" -> {"{", "}", ","}] /. 
    {x_, y_, z_, γ_}?VectorQ :> {{x, y, z}, γ};

r2 = Replace[Import["compacted_short.csv", "Table", 
    "FieldSeparators" -> {"{", "}", ","}],
   {x_, y_, z_, γ_} :> {{x, y, z}, γ}, -1];

It may be advantageous to modify the C program so it writes the data as a Wolfram Language expression with the data already in the desired format. Then the data could be read using the Get command. For example, the Wolfram Language file could be formatted like this, but with as many decimal places as you like

{{{0.00,-0.001175,0.545714},0.00000000000075},
 {{0.00,-0.001145,0.545714},0.00000000000071},
 {{0.00,-0.001116,0.545714},0.00000000000068}}

and read with r3 = Get["data.wl"].

Answer 2

For demonstration purposes, I'll define the string form of your file like this:

str = "{{0.000000000000000000000000000000,-0.001175000000000000000000000000,0.545714285714285710000000000000},0.000000000000757534038528606580}
{{0.000000000000000000000000000000,-0.001145839061933960200000000000,0.545714285714285710000000000000},0.000000000000718990648693323850}
{{0.000000000000000000000000000000,-0.001116678123867920300000000000,0.545714285714285710000000000000},0.000000000000681423522535585060}
{{0.000000000000000000000000000000,-0.001087517185801880300000000000,0.545714285714285710000000000000},0.000000000000644832928422345040}
{{0.000000000000000000000000000000,-0.001058356247735840400000000000,0.545714285714285710000000000000},0.000000000000609219154540147720}
{{0.000000000000000000000000000000,-0.001029195309669800800000000000,0.545714285714285710000000000000},0.000000000000574582510216643140}"

You should get something similar when you do:

str = Import[file.csv, "String" (*or "Text"*)]

You can wrangle this string into a WL-friendly form and then use ImportString to turn it into a list:

data = ImportString[
  "{" <> StringTrim[
    StringReplace[
      StringTrim@str, EndOfLine -> ", "],
      ("," | WhitespaceCharacter) ..
  ] <> "}"
  ,
  "Package"
];

Note, however, that this list format does not support packed arrays, so it will be very memory intensive. If you only need machine-precision numbers I highly recommend importing this data as an n-by-4 matrix and forgetting about the ragged format you're proposing here.