Alternatives to Export to write long list as a table on a file

Question

I have a large list (approx 50 millions elements).

I could Export it as a table like this:

Export["path/to/file.dat",list,"Table"]

but this takes a large amount of memory. Are there any alternatives for writing my list on a file as a table, which use less RAM?

Answer 1

You can do a batched export using an output stream. For example:

batchedExport[file_, data_List, format_, n_Integer] := Module[{
   stream = OpenAppend[file]
   },
  WithCleanup[
   Scan[
    Function[
     WriteString[
      stream,
      ExportString[#, format] <> "\n"
      ]
     ],
    Partition[data, UpTo[n]]
    ];
   file
   ,
   Close[stream]
   ]
 ];

data = RandomChoice[{1, 2., "a"}, {100000, 4}];
batchedExport["test.dat", data, "Table", 100]

You can use n to change the amount of data that gets processed each step.