Compress uses too much memory and sometimes crashes the Kernel

Question

Using Compress is a good idea to save memory when working with large expressions. The problem is that Compressing large expressions also uses a large amount of memory to produce the compressed result (sometimes 10× the ByteCount of the original expression). This can lead to unexpected kernel crashes. I tried MemoryConstrained but it seems it does not catch Compress's memory consumption.

The following example shows in steps how memory is consumed.

data = Range[30000000];
ByteCount[data] (* -> 120000104 *)

{MemoryInUse[], MaxMemoryUsed[]} (* -> {140468260, 140520204} *)

data = Compress[data];
ByteCount[data] (* -> 55320440 *)

{MemoryInUse[], MaxMemoryUsed[]} (* -> {75106708, 528489092} *)

As you can see, to compress 120MB of data the Compress method uses additionally at least 380MB.

If you use larger data (for example use $10^8$ instead of $3\times10^7$), the kernel will probably ingloriously crash.

So:

1. I have seen that Options[Compress] == {Method->{}}. Is there a way to tweak the compression algorithm?

2. Is there a way to "break" the expression in chunks (regardless the levels of the expression). My expression to compress may be a tree with "unbalanced" branches. Note that I tried to transform the expression to string (with ToString), but it also takes a huge amount of memory to do that.

3. Using an external compression would be really nice, although this may lead to delays and cross-compatibility issues. Of course any ideas on that also would may prove usefull.

Any help will be really appreciate since I am puzzled with this the whole last week.

Answer 1

This is not a proper answer to the question. I had forgotten what the proper syntax was for the Method option of Compress, but when I tried to find it again, I noticed that it seems like this is not documented anywhere. So, I just wanted to record this for posterity, as well as note that there are three working methods (as of version 9.0.1):

test = Fold[{#2, #1} &, "this is a test", ConstantArray[1, {10}]]
(* -> {1, {1, {1, {1, {1, {1, {1, {1, {1, {1, "this is a test"}}}}}}}}}} *)
ByteCount[test] (* -> 768 *)

Note that this ByteCount seems to be correct, even though the 1's are most likely all the same object (ConstantArray produces arrays of pointers to its first argument).

MaxMemoryUsed[method1 = Compress[test, Method -> {Version -> 1}]] (* -> 283296 *)
InputForm[method1]
 (* -> "1:eJxTTMoPSmNiYGAoZgESPpnFJZmMQMYQEgrmAxIlGZnFCkCUqFCSWlwCACP0I60=" *)
ByteCount[method1] (* -> 152 *)
MaxMemoryUsed[result1 = Uncompress[method1]] (* -> 26240 *)
result1 === test (* -> True *)

MaxMemoryUsed[method2 = Compress[test, Method -> {Version -> 2}]] (* -> 283296 *)
InputForm[method2]
 (* -> "2:Gaq]+aq%*[email protected]?N'PAo&dmb'&g\\\\E``<nkpW)/X\\\\>QOmMoHUI" *)
ByteCount[method2] (* -> 96 *)
MaxMemoryUsed[result2 = Uncompress[method2]] (* -> 26232 *)
result2 === test (* -> True *)

MaxMemoryUsed[method3 = Compress[test, Method -> {Version -> 3}]] (* -> 273928 *)
InputForm[method3]
 (* -> it produces a string containing unprintable characters *)
ByteCount[method3] (* -> 88 *)
MaxMemoryUsed[result3 = Uncompress[method3]] (* -> 53704 *)
result3 === test (* -> True *)

Specifying version numbers higher than 3 will not yield a reasonable output, although no message is produced until Method -> {Version -> 6}, at which point:

Compress::comprver: Value of Method option "Version" -> 6 should be 1 or 2.

So, method 3 seems to be the one that produces the smallest compressed output, and takes the least memory to do so (at least for this, not particularly demanding, test input). But it is at the cost of twice as much memory used in decompression, and the result being a string that cannot be represented sensibly by the front end.

Answer 2

It's not clear exactly what you plan to do with the compressed data, but maybe this alternative approach will be useful. It uses ExportString and ImportString with a format that is especially suited for large data sets, HDF5.

The compression step is very fast and efficient. I'm using the original data in the question:

data = Range[30000000];

ByteCount[data]

(* ==> 240000144 *)

{MemoryInUse[], MaxMemoryUsed[]}

(* ==> {268604816, 269539448} *)

data2 = ExportString[data, "HDF5"];
ByteCount[data2]

(* ==> 163217248 *)

{MemoryInUse[], MaxMemoryUsed[]}

(* ==> {432851720, 553046640} *)

So far, so good. Of course, if your goal is to export the compressed data, you wouldn't even need to use ExportString - just Export. It's also very fast.

Now I test how to return the compressed data2 back into the original data:

data3 = ImportString[data2, {"HDF5", "Data"}];

{MemoryInUse[], MaxMemoryUsed[]}

(* ==> {672939272, 2861887368} *)

Here the memory usage shoots up quite a bit. However, the question was about memory footprint during compression, and that is very well-behaved.