11
$\begingroup$

Let us assume that I need to storage in a file, or files, data with the following form

alist = 
  Table[Table[RandomReal[{0, 1}, 2], {i, 1, 200}], {j, 1, 10000}];
blist = 
  Table[Table[{RandomReal[{0, 1}, 6], RandomReal[], 
     RandomReal[{0, 1}, 6], RandomReal[{0, 1}, 6]}, {i, 1, 200}], {j, 
    1, 10000}];
clist = RandomReal[{0, 1}, 10000];

In a few seconds, Mathematica is able to create these lists.

However, when I try to export this data to a .txt file, using 

Export["C:\\pathname\\abclists.txt", {alist, blist, clist}];

My computer starts to freeze, and I have to abort the evalutation. When I check the result, Mathematica has created a 600MB .txt file.

Is there a more efficient way to storage data in a file or files? I have no preference for the file type, as long as I can easily rebuild the original data, preserving the list levels.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Have you tried DumpSave? $\endgroup$
    – Roman
    Jan 9, 2019 at 16:27

2 Answers 2

Reset to default
11
$\begingroup$

Using ByteCount will tell you to expect more than 1 GB:

ByteCount @ { alist, blist, clist }

(* 1 136 880 448 *)

Compress

I would thus try Compress:

cexpr = Compress @ { alist, blist, clist };
(* Export["pathname\\data.m", cexpr ]; *)
ByteCount @ cexpr

(* 437 359 280 *)

I am getting a 433 MB file (which roughly matches ByteCount). You can Uncompress the expression after loading.

BinarySerialize

Another possibility as of Version 11.1 or later is BinarySerialize:

bexpr = BinarySerialize @ { alist, blist, clist };
ByteCount @ bexpr

(* 378 170 128 *)

So we are down to about 378 MB (the file is 312 MB on my computer). You can use BinaryDeserialize to get the original expression again (see below for explicit instructions for writing/reading binary data).

If we give the option PerformanceGoal -> "Size"

bexpr = BinarySerialize[ {alist,blist,clist}, PerformanceGoal -> "Size" ];
ByteCount @ bexpr

(* 327 494 245 *)

we are down to about 327 MB.

Writing and Reading Binary Data

The documentation tells you how to write/read binary data:

stream = OpenWrite[ "pathname\\data.mx", BinaryFormat -> True ];
BinaryWrite[ stream, bexpr ];
Close @ stream;

Reading the data:

data = BinaryDeserialize @ ByteArray @ BinaryReadList[ "pathname\\data.mx", "Byte" ];
Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your answer. To load the data, would we use ReadList or Import? If it's with import, could you show me how? The documentation talks about 'elements', but I don't know what they are, and how they relate to our compressed data. I tried using ReadList, but I had to abort the evaluation... $\endgroup$
    – An old man in the sea.
    Jan 9, 2019 at 14:36
  • 1
    $\begingroup$ Considering that the structure contains $4.2\times10^7$ fully random elements with 8 bytes each, I don't think you can do much better, even with an approach that stores the structure of the data more efficiently $\endgroup$
    – Lukas Lang
    Jan 9, 2019 at 14:37
  • $\begingroup$ @LukasLang Yes, that should indeed be mentioned: The random data here are a "worst case" - but the problem still is a general one. $\endgroup$
    – gwr
    Jan 9, 2019 at 14:38
  • 1
    $\begingroup$ Thanks gwr. I've tried BinarySerialize, but Mathematica doesn't recognize it as a built-in function... Also, I've noticed that if we separate the file into 3, for each list, then the combined time for import, export, compress, and uncompress are much lower in total... $\endgroup$
    – An old man in the sea.
    Jan 9, 2019 at 20:29
  • $\begingroup$ @Anoldmaninthesea. One has to play around with it. I added the information, that BinarySerialize is only available for Version 11.1 or later. $\endgroup$
    – gwr
    Jan 9, 2019 at 21:34
0
$\begingroup$

Old (OS and Mma version dependent!) "MX" format is also pretty fast and good for storing. 

abcList = {alist, blist, clist};
ByteCount[abcList]

Out[2]= 1136880448

Export[
 FileNameJoin[{NotebookDirectory[EvaluationNotebook[]], 
   "test.mx"}], abcList, "MX"];

FileByteCount[
 FileNameJoin[{NotebookDirectory[EvaluationNotebook[]], "test.mx"}]]

Out[4]= 434116760

abcListRead = 
  Import[FileNameJoin[{NotebookDirectory[EvaluationNotebook[]], 
     "test.mx"}]];

ByteCount[abcListRead]

Out[6]= 1136880448

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.