# Efficient way to storage large lists in a file(s)

Let us assume that I need to storage in a file, or files, data with the following form

alist =
Table[Table[RandomReal[{0, 1}, 2], {i, 1, 200}], {j, 1, 10000}];
blist =
Table[Table[{RandomReal[{0, 1}, 6], RandomReal[],
RandomReal[{0, 1}, 6], RandomReal[{0, 1}, 6]}, {i, 1, 200}], {j,
1, 10000}];
clist = RandomReal[{0, 1}, 10000];


In a few seconds, Mathematica is able to create these lists.

However, when I try to export this data to a .txt file, using

Export["C:\\pathname\\abclists.txt", {alist, blist, clist}];


My computer starts to freeze, and I have to abort the evalutation. When I check the result, Mathematica has created a 600MB .txt file.

Is there a more efficient way to storage data in a file or files? I have no preference for the file type, as long as I can easily rebuild the original data, preserving the list levels.

• Have you tried DumpSave? – Roman Jan 9 at 16:27

Using ByteCount will tell you to expect more than 1 GB:

ByteCount @ { alist, blist, clist }

(* 1 136 880 448 *)


Compress

I would thus try Compress:

cexpr = Compress @ { alist, blist, clist };
(* Export["pathname\\data.m", cexpr ]; *)
ByteCount @ cexpr

(* 437 359 280 *)


I am getting a 433 MB file (which roughly matches ByteCount). You can Uncompress the expression after loading.

BinarySerialize

Another possibility as of Version 11.1 or later is BinarySerialize:

bexpr = BinarySerialize @ { alist, blist, clist };
ByteCount @ bexpr

(* 378 170 128 *)


So we are down to about 378 MB (the file is 312 MB on my computer). You can use BinaryDeserialize to get the original expression again (see below for explicit instructions for writing/reading binary data).

If we give the option PerformanceGoal -> "Size"

bexpr = BinarySerialize[ {alist,blist,clist}, PerformanceGoal -> "Size" ];
ByteCount @ bexpr

(* 327 494 245 *)


we are down to about 327 MB.

The documentation tells you how to write/read binary data:

stream = OpenWrite[ "pathname\\data.mx", BinaryFormat -> True ];
BinaryWrite[ stream, bexpr ];
Close @ stream;


data = BinaryDeserialize @ ByteArray @ BinaryReadList[ "pathname\\data.mx", "Byte" ];

• Thanks for your answer. To load the data, would we use ReadList or Import? If it's with import, could you show me how? The documentation talks about 'elements', but I don't know what they are, and how they relate to our compressed data. I tried using ReadList, but I had to abort the evaluation... – An old man in the sea. Jan 9 at 14:36
• Considering that the structure contains $4.2\times10^7$ fully random elements with 8 bytes each, I don't think you can do much better, even with an approach that stores the structure of the data more efficiently – Lukas Lang Jan 9 at 14:37
• @LukasLang Yes, that should indeed be mentioned: The random data here are a "worst case" - but the problem still is a general one. – gwr Jan 9 at 14:38
• Thanks gwr. I've tried BinarySerialize, but Mathematica doesn't recognize it as a built-in function... Also, I've noticed that if we separate the file into 3, for each list, then the combined time for import, export, compress, and uncompress are much lower in total... – An old man in the sea. Jan 9 at 20:29
• @Anoldmaninthesea. One has to play around with it. I added the information, that BinarySerialize is only available for Version 11.1 or later. – gwr Jan 9 at 21:34

Old (OS and Mma version dependent!) "MX" format is also pretty fast and good for storing.

abcList = {alist, blist, clist};
ByteCount[abcList]


Out[2]= 1136880448

Export[
FileNameJoin[{NotebookDirectory[EvaluationNotebook[]],
"test.mx"}], abcList, "MX"];

FileByteCount[
FileNameJoin[{NotebookDirectory[EvaluationNotebook[]], "test.mx"}]]


Out[4]= 434116760

abcListRead =
Import[FileNameJoin[{NotebookDirectory[EvaluationNotebook[]],
"test.mx"}]];