# High memory use while exporting a BMP

Something mysterious is going on when exporting images. I'm using Mathematica 9.0.1 on Windows 7 (64 bit), and find that the memory use is excessive when exporting to a bmp. But it doesn't happen with other image formats.

For example, running the following (with different values for k):

k = 1;  extensions = {"png", "tif", "jpg", "bmp"};  SeedRandom[42];
image = Image@RandomReal[{0, 1}, {1500, 1500, 3}];
base = MaxMemoryUsed[];
Export["C:\\mem." <> extensions[[k]], image];
{extensions[[k]], (MaxMemoryUsed[] - base)/1024.^2}
Quit[]


gives

{"png",  79.987}
{"tif",  28.617}
{"jpg",  60.453}
{"bmp", 927.602}


png: 12.8 MB
tif: 12.8 MB
jpg: 1.29 MB
bmp: 6.43 MB


Why does exporting a bmp require over 10 times the memory of other image formats? And what can I do about it?

## EDIT:

Here's the results of some more testing (code at the end).

format  texture  type    memory(MB)  time(s)  size(KB)

png     random   real        89        3.4     15 028
bit16        0        2.6     15 028
byte         0        1.5      7 515
smooth   real        75        1.3        726
bit16        0        1.2        726
byte         0        0.5         85

tif     random   real        17        0.2     15 000
bit16        0        0.1     15 000
byte         0        0.1      7 500
smooth   real        17        0.3     15 000
bit16        0        0.1     15 000
byte         0        0.2      7 500

jpg     random   real        68        0.4      1 500
bit16        0        0.8      1 503
byte         0        0.7      1 500
smooth   real        68        0.3         50
bit16        0        0.3         50
byte         0        0.3         50

bmp     random   real      1069       25.3      7 500
bit16      967       22.8      7 500
byte        69    \$Aborted         0
smooth   real      1055      100.8      7 500
bit16      953       32.4      7 500
byte        71        1.4      7 500


Notes:

• By default Mathematica will treat the image type as Real.
• When exporting, a Real type image will need to be converted to an integer type. This requires extra memory and time.
• tif and png support 16 bit pixels, so this will be used by default for Real images.
• jpg and bmp only support up to 8 bits per pixel.
• My computer was thrashing the memory disk cache for the bmp smooth real, so it took extra time.
• The bmp random byte was using 100% for 5 minutes before I gave up.

Conclusions:

• Use tif for speed.
• Use png for saving disk space.
• Never use jpg because it is lossy.
• Never use bmp because it is broken.

Code:

k = 1; (* 1 .. 24 *)
{e, t, c} = Tuples[{Range[4], Range[3], Range[2]}][[k]];
extensions = {"png", "tif", "jpg", "bmp"};
types = {"real", "byte", "bit16"};
compress = {"random", "smooth"};
res = 1600;
SeedRandom[42];
data = Switch[compress[[c]],
"random", RandomReal[{0, 1}, {res, res, 3}],
"smooth", Table[{i, j, 1 - i j}, {i, 0., 1, 1/(res - 1)}, {j, 0., 1, 1/(res - 1)}]
];
image = Switch[types[[t]],
"real", Image@data,
"byte", Image[Round[(2^8 - 1) data], "Byte"],
"bit16", Image[Round[(2^16 - 1) data], "Bit16"]
];
filename = "C:\\_" <> types[[t]] <> "_" <> compress[[c]] <> "." <> extensions[[e]];
base = MaxMemoryUsed[];
time = AbsoluteTiming[Export[filename, image]][[1]];
CellPrint@TextCell[ToString@{
filename,(MaxMemoryUsed[] - base)/1024.^2, time, FileByteCount[filename]/1024.
}, "Text", CellAutoOverwrite -> False]
Quit[]

-
Do you have to use bmp? Reminds me of the old joke: Patient -- Doctor, it hurts when I do this; Doctor -- Don't do that (see Wikipedia). –  m_goldberg Mar 4 '13 at 2:03
Just for the record, I get pretty much the same results with V9.0.1 on OS X. –  m_goldberg Mar 4 '13 at 2:06
For completeness sake, could you post the 4 corresponding file sizes? –  Sjoerd C. de Vries Mar 4 '13 at 13:25
This looks like something worth reporting to support@wolfram.com. Side comment: In v9 it's easier to just use the (undocumented) MaxMemoryUsed[Export["max.png", image]] to measure how much memory an operation uses. I'm not sure how accurate it is though. –  Szabolcs Mar 4 '13 at 21:35
On["Packing"] shows that exporting to BMP unpacks the whole array. No wonder it's so slow and takes so much memory. Probably it's best if you report this to support and mention the packing thing. Tagging as bug. –  Szabolcs Mar 4 '13 at 21:44

I believe this is a bug in Mathematica. It can be verified that the unreasonable memory usage is due to array unpacking by

On["Packing"]
ExportString[Image@RandomReal[1, {30, 30}], "BMP"];


So please report it to support@wolfram.com so this can be fixed for the benefit of all of us, and point them to this discussion.

Some further playing with Simon Woods's excellent spelunker and the (much underrated) built-in debugger revealed the culprit. It is an innocent looking line inside SystemConvertBitmapDumpExportBMP24:

Module[{bitmap, ...},
...
bitmap = Floor[(255 bitmap)/max];  (* <-- this unpacks *)
...
]


Now bitmap appears to be a packed array of integer values and max is an integer as well. Did you already guess why this unpacks? Hover below for the spoiler.

255/max evaluates to a Rational and of course multiplying integers by Rationals will typically give a Rational and unpack the integer array. Using Floor[(255. bitmap)/max] would fix the problem.

This shows how counterintuitive Mathematica can be at times. I would have fallen into the same trap.

To verify that what I'm saying here is correct, one can do the following experiment:

MaxMemoryUsed[ExportString[Image@RandomReal[1, {1500, 1500}], "BMP"];]

(* ==> 1007548312 , i.e. 1 GB *)


Now evaluate DownValues[SystemConvertBitmapDumpExportBMP24], search for Floor, change 255 to 255. and set the resulting expression back to DownValues[SystemConvertBitmapDumpExportBMP24] = .... Let's measure the memory usage again:

MaxMemoryUsed[ExportString[Image@RandomReal[1, {1500, 1500}], "BMP"];]

(* ==> 289673368 i.e. 290 MB *)


Now it uses much less memory and ExportString finishes much faster.

The reason why it still uses quite a bit of memory is other flaws in the code which cause partial unpacking further down the pipeline.

-
Big +1! Some further playing with Simon Woods's excellent spelunker and the (much underrated) built-in debugger revealed the culprit. It is an innocent looking line inside is just screaming blog post topic. Well at least to me -- I don't know how many people would be interested in spelunking. But on second thought: a step by step explanation if how you fond this bug would be useful in debugging in general... –  Ajasja Mar 5 '13 at 15:49
@Ajasja Yes, that's a good idea. I'll think about it. It would be a very short blog though, and I'm not an expert with the debugger. –  Szabolcs Mar 5 '13 at 15:50
Well, it would give you the saved our blog` badge... :) It doesn't matter if it's on the short side. Also, since the debugger is so underused, who is an expert with it anyway? –  Ajasja Mar 5 '13 at 16:17