High memory use while exporting a BMP

Question

Something mysterious is going on when exporting images. I'm using Mathematica 9.0.1 on Windows 7 (64 bit), and find that the memory use is excessive when exporting to a bmp. But it doesn't happen with other image formats.

For example, running the following (with different values for k):

k = 1;  extensions = {"png", "tif", "jpg", "bmp"};  SeedRandom[42];
image = Image@RandomReal[{0, 1}, {1500, 1500, 3}];
base = MaxMemoryUsed[];
Export["C:\\mem." <> extensions[[k]], image];
{extensions[[k]], (MaxMemoryUsed[] - base)/1024.^2}
Quit[]

gives

{"png",  79.987}
{"tif",  28.617}
{"jpg",  60.453}
{"bmp", 927.602}

file sizes (added by s0rce):

png: 12.8 MB
tif: 12.8 MB
jpg: 1.29 MB
bmp: 6.43 MB

Why does exporting a bmp require over 10 times the memory of other image formats? And what can I do about it?

EDIT:

Here's the results of some more testing (code at the end).

format  texture  type    memory(MB)  time(s)  size(KB)

png     random   real        89        3.4     15 028
                 bit16        0        2.6     15 028
                 byte         0        1.5      7 515
        smooth   real        75        1.3        726
                 bit16        0        1.2        726
                 byte         0        0.5         85

tif     random   real        17        0.2     15 000
                 bit16        0        0.1     15 000
                 byte         0        0.1      7 500
        smooth   real        17        0.3     15 000
                 bit16        0        0.1     15 000
                 byte         0        0.2      7 500

jpg     random   real        68        0.4      1 500
                 bit16        0        0.8      1 503
                 byte         0        0.7      1 500
        smooth   real        68        0.3         50
                 bit16        0        0.3         50
                 byte         0        0.3         50

bmp     random   real      1069       25.3      7 500
                 bit16      967       22.8      7 500
                 byte        69    $Aborted         0
        smooth   real      1055      100.8      7 500
                 bit16      953       32.4      7 500
                 byte        71        1.4      7 500

Notes:

• By default Mathematica will treat the image type as Real.
• When exporting, a Real type image will need to be converted to an integer type. This requires extra memory and time.
  • tif and png support 16 bit pixels, so this will be used by default for Real images.
  • jpg and bmp only support up to 8 bits per pixel.
• My computer was thrashing the memory disk cache for the bmp smooth real, so it took extra time.
• The bmp random byte was using 100% for 5 minutes before I gave up.

Conclusions:

• Use tif for speed.
• Use png for saving disk space.
• Never use jpg because it is lossy.
• Never use bmp because it is broken.

Code:

k = 1; (* 1 .. 24 *)
{e, t, c} = Tuples[{Range[4], Range[3], Range[2]}][[k]];
extensions = {"png", "tif", "jpg", "bmp"};
types = {"real", "byte", "bit16"};
compress = {"random", "smooth"};
res = 1600;
SeedRandom[42];
data = Switch[compress[[c]],
  "random", RandomReal[{0, 1}, {res, res, 3}],
  "smooth", Table[{i, j, 1 - i j}, {i, 0., 1, 1/(res - 1)}, {j, 0., 1, 1/(res - 1)}]
];
image = Switch[types[[t]],
  "real", Image@data,
  "byte", Image[Round[(2^8 - 1) data], "Byte"],
  "bit16", Image[Round[(2^16 - 1) data], "Bit16"]
];
filename = "C:\\_" <> types[[t]] <> "_" <> compress[[c]] <> "." <> extensions[[e]];
base = MaxMemoryUsed[];
time = AbsoluteTiming[Export[filename, image]][[1]];
CellPrint@TextCell[ToString@{
  filename,(MaxMemoryUsed[] - base)/1024.^2, time, FileByteCount[filename]/1024.
}, "Text", CellAutoOverwrite -> False]
Quit[]

Answer 1

I believe this is a bug in Mathematica. It can be verified that the unreasonable memory usage is due to array unpacking by

On["Packing"]
ExportString[Image@RandomReal[1, {30, 30}], "BMP"];

So please report it to support@wolfram.com so this can be fixed for the benefit of all of us, and point them to this discussion.

Some further playing with Simon Woods's excellent spelunker and the (much underrated) built-in debugger revealed the culprit. It is an innocent looking line inside System`Convert`BitmapDump`ExportBMP24:

Module[{bitmap, ...},
  ...
  bitmap = Floor[(255 bitmap)/max];  (* <-- this unpacks *)
  ...
]

Now bitmap appears to be a packed array of integer values and max is an integer as well. Did you already guess why this unpacks? Hover below for the spoiler.

255/max evaluates to a Rational and of course multiplying integers by Rationals will typically give a Rational and unpack the integer array. Using Floor[(255. bitmap)/max] would fix the problem.

This shows how counterintuitive Mathematica can be at times. I would have fallen into the same trap.

---

To verify that what I'm saying here is correct, one can do the following experiment:

MaxMemoryUsed[ExportString[Image@RandomReal[1, {1500, 1500}], "BMP"];]

(* ==> 1007548312 , i.e. 1 GB *)

Now evaluate DownValues[System`Convert`BitmapDump`ExportBMP24], search for Floor, change 255 to 255. and set the resulting expression back to DownValues[System`Convert`BitmapDump`ExportBMP24] = .... Let's measure the memory usage again:

MaxMemoryUsed[ExportString[Image@RandomReal[1, {1500, 1500}], "BMP"];]

(* ==> 289673368 i.e. 290 MB *)

Now it uses much less memory and ExportString finishes much faster.

The reason why it still uses quite a bit of memory is other flaws in the code which cause partial unpacking further down the pipeline.