# How to Flatten and stack Efficiently 50000 Images

I have 50000 grayscale images with 56*56 pixels each one. I need to flatten images and stakck it in the same array with dimensions 50000*3136 after that export the file as .CSV. I am doing this code which load all images in the memory which can cause a problem if we have not enough memory space.

imagesToArray[pathsrc_, pathdes_] := Module[{filesList, data}, (
(* pathsrc_ : list of images path *)
(* pathdes_ : CSV file path*)

filesList = FileNames["*.png", pathsrc];
Print["Number of images :", Length@filesList];
data =
Table[Flatten[ImageData[Import[filesList[[i]]]]], {i, 1,
Length@filesList}];
Export[pathdes, data];
data
)]


Is there any alternative to do it faster without memory problem?

• Why not data = Flatten [ ImageData /@ Import /@ fileList ] instead of Table[ ]? Am I missing something? – Dr. belisarius Mar 18 '16 at 16:15
• anyway, importing 50K images will take quite a long time – Dr. belisarius Mar 18 '16 at 16:16
• @Dr.belisarius, may be it will be faster but I think it load all data in memory – BetterEnglish Mar 18 '16 at 16:17
• Why would you store such a large amount of binary data in csv? In any case you can read each file and write on the fly so you don't load everything in memory at once, you need to forgo Export to achieve that though. – george2079 Mar 18 '16 at 16:19
• If you can forgo all the behind-the-scene checking that Import does, you can try using the built-in function that Import ultimately calls to read in a PNG file, which is ImageImportExportDumpImageReadPNG. It returns a list containing the image imported. You can use e.g. First@ImageImportExportDumpImageReadPNG[pathToImage] to get each image. It is roughly 10x faster than using Import. – MarcoB Mar 18 '16 at 17:54

You can combine @nikie's approach above with the use of a faster built-in import function.

If you can forgo all the behind-the-scene checking that Import does, you can try using the built-in function that Import ultimately calls to read in a PNG file, which is ImageImportExportDumpImageReadPNG. I found that out by using Trace@Import[somePNGfile] and wading through quite a lot of output. Incidentally, the counterpart to this function also exists, i.e. ImageWritePNG, but it requires multiple arguments and I haven't figured that one out yet; however, Export is already quite a bit zippier than Import at least for PNG files, so the need is less pronounced there.

It returns a list containing the image imported. To get each image, you can use e.g.

First@ImageImportExportDumpImageReadPNG[pathToImage]


It is at least 10x faster than using Import, and quite possibly more.

For instance, let's generate 200 small PNGs of the kind you are working with:

MapIndexed[
Export["images\\" <> ToString[First@#2] <> ".png", #1] &,
Image /@ RandomReal[{0, 1}, {200, 56, 56}]
];


Now let's compare timings:

First@ImageImportExportDumpImageReadPNG["images\\" <> ToString[#] <> ".png"] & /@
Range[1, 200]; // AbsoluteTiming

Import["images\\" <> ToString[#] <> ".png"] & /@ Range[1, 200]; // AbsoluteTiming

(* Out:
{0.221601, Null}
{15.5932, Null}
*)


That's a 70x speedup!

• Awesome!Does the same things can be happened in “.JPG”?Like something ImageReadJPG? – yode Mar 18 '16 at 18:41
• I did notnrun this code, I get this error {\$Failed, 2} and this function seems not defined because it colored in blue – BetterEnglish Mar 18 '16 at 18:51
• Thanks for your answer! How does we get the help about this kind of function. Is it really a built-in function? I have looked in the help but i did not get any infos! It seems magic ! – BetterEnglish Mar 18 '16 at 18:55
• @Developer2000 Try evaluating the following: ImageImportExportDumpImageReadPNG[]. Although the function name will not be recognized (i.e. it will stay blue), you should obtain the following output: image. If you do, then the code above should work for you as well. You may also want to check that all the finicky backticks, etc. were copied correctly to your notebook. – MarcoB Mar 18 '16 at 18:57
• Thanks a lot.I like this answer very much.And then do you mean ImageImportExportDump requires multiple arguments ?But ImageImportExportDumpImageWritePNG["1.png",Image@RandomReal[{0,1},{200,56,3}]] work well in my 10.4.0.It speed up many times than Export too. – yode Mar 18 '16 at 23:42

You can use stream functions, like this (untested):

f = OpenWrite[pathdes];
chunkSize = 100;
Monitor[Do[
(
lines=ParallelTable[Flatten[ImageData[Import[filesList[[j]]]]],
{j,i,Min[i+chunkSize-1,Length@filesList]}];
WriteString[f, ExportString[lines, "CSV"]];
),
{i, 1, Length@filesList, chunkSize}], i];
Close[f];


to avoid the memory cost. I'd guess Import would be the bottleneck in that code, and there's not much you can do to make it faster.

• Thanks for your answer but it does not gain more time than mine with 500 images: My solution gives 44s and yours gives 42s. – BetterEnglish Mar 18 '16 at 16:52
• Like I said, you can avoid the memory cost, but there's not much you can do to improve Import`'s speed without rewriting the functionality. – Niki Estner Mar 18 '16 at 17:03