# Compress uses too much memory and sometimes crashes the Kernel

Using Compress is a good idea to save memory when working with large expressions. The problem is that Compressing large expressions also uses a large amount of memory to produce the compressed result (sometimes 10× the ByteCount of the original expression). This can lead to unexpected kernel crashes. I tried MemoryConstrained but it seems it does not catch Compress's memory consumption.

The following example shows in steps how memory is consumed.

data = Range[30000000];
ByteCount[data] (* -> 120000104 *)

{MemoryInUse[], MaxMemoryUsed[]} (* -> {140468260, 140520204} *)

data = Compress[data];
ByteCount[data] (* -> 55320440 *)

{MemoryInUse[], MaxMemoryUsed[]} (* -> {75106708, 528489092} *)


As you can see, to compress 120MB of data the Compress method uses additionally at least 380MB.

If you use larger data (for example use $10^8$ instead of $3\times10^7$), the kernel will probably ingloriously crash.

So:

1. I have seen that Options[Compress] == {Method->{}}. Is there a way to tweak the compression algorithm?

2. Is there a way to "break" the expression in chunks (regardless the levels of the expression). My expression to compress may be a tree with "unbalanced" branches. Note that I tried to transform the expression to string (with ToString), but it also takes a huge amount of memory to do that.

3. Using an external compression would be really nice, although this may lead to delays and cross-compatibility issues. Of course any ideas on that also would may prove usefull.

Any help will be really appreciate since I am puzzled with this the whole last week.

This is not a proper answer to the question. I had forgotten what the proper syntax was for the Method option of Compress, but when I tried to find it again, I noticed that it seems like this is not documented anywhere. So, I just wanted to record this for posterity, as well as note that there are three working methods (as of version 9.0.1):

test = Fold[{#2, #1} &, "this is a test", ConstantArray[1, {10}]]
(* -> {1, {1, {1, {1, {1, {1, {1, {1, {1, {1, "this is a test"}}}}}}}}}} *)
ByteCount[test] (* -> 768 *)


Note that this ByteCount seems to be correct, even though the 1's are most likely all the same object (ConstantArray produces arrays of pointers to its first argument).

MaxMemoryUsed[method1 = Compress[test, Method -> {Version -> 1}]] (* -> 283296 *)
InputForm[method1]
(* -> "1:eJxTTMoPSmNiYGAoZgESPpnFJZmMQMYQEgrmAxIlGZnFCkCUqFCSWlwCACP0I60=" *)
ByteCount[method1] (* -> 152 *)
MaxMemoryUsed[result1 = Uncompress[method1]] (* -> 26240 *)
result1 === test (* -> True *)

MaxMemoryUsed[method2 = Compress[test, Method -> {Version -> 2}]] (* -> 283296 *)
InputForm[method2]
(* -> "2:Gaq]+aq%*S@UNRlAcWN.RDgJ?N'PAo&dmb'&g\\\\E<nkpW)/X\\\\>QOmMoHUI" *)
ByteCount[method2] (* -> 96 *)
MaxMemoryUsed[result2 = Uncompress[method2]] (* -> 26232 *)
result2 === test (* -> True *)

MaxMemoryUsed[method3 = Compress[test, Method -> {Version -> 3}]] (* -> 273928 *)
InputForm[method3]
(* -> it produces a string containing unprintable characters *)
ByteCount[method3] (* -> 88 *)
MaxMemoryUsed[result3 = Uncompress[method3]] (* -> 53704 *)
result3 === test (* -> True *)


Specifying version numbers higher than 3 will not yield a reasonable output, although no message is produced until Method -> {Version -> 6}, at which point:

Compress::comprver: Value of Method option "Version" -> 6 should be 1 or 2.

So, method 3 seems to be the one that produces the smallest compressed output, and takes the least memory to do so (at least for this, not particularly demanding, test input). But it is at the cost of twice as much memory used in decompression, and the result being a string that cannot be represented sensibly by the front end.

• Uncompress[Compress[Range[10000], Method -> {Version -> 3}]] fails with a message Uncompress::corrupt: Compressed data "3:x4ػE]]֟\.12[RawEscape] .Ȣ[7û4¡" is corrupt and does not represent an expression. >> – Vladimir Reshetnikov Mar 3 '16 at 3:19

It's not clear exactly what you plan to do with the compressed data, but maybe this alternative approach will be useful. It uses ExportString and ImportString with a format that is especially suited for large data sets, HDF5.

The compression step is very fast and efficient. I'm using the original data in the question:

data = Range[30000000];

ByteCount[data]

(* ==> 240000144 *)

{MemoryInUse[], MaxMemoryUsed[]}

(* ==> {268604816, 269539448} *)

data2 = ExportString[data, "HDF5"];
ByteCount[data2]

(* ==> 163217248 *)

{MemoryInUse[], MaxMemoryUsed[]}

(* ==> {432851720, 553046640} *)


So far, so good. Of course, if your goal is to export the compressed data, you wouldn't even need to use ExportString - just Export. It's also very fast.

Now I test how to return the compressed data2 back into the original data:

data3 = ImportString[data2, {"HDF5", "Data"}];

{MemoryInUse[], MaxMemoryUsed[]}

(* ==> {672939272, 2861887368} *)


Here the memory usage shoots up quite a bit. However, the question was about memory footprint during compression, and that is very well-behaved.

• I am not sure whether the HDF5 support provided by Mathematica extends much beyond numerical arrays (and in that sense, perhaps the test case in the question was not too well chosen), but I do agree that it seems to be good for that purpose, at least. GZIPed HDF5 could be even better from the ByteCount perspective. – Oleksandr R. May 31 '15 at 2:51
• @OleksandrR. Yes, it all depends what kind of "expressions" are of interest. But for the example, my method works much faster than Compress, and I can also crank the data size up to $10^8$ without encountering a crash either in compression or decompression. In that case, Compress/ Uncompress aren't able to compete with HDF5. – Jens May 31 '15 at 3:07