Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have 50000 grayscale images with 56*56 pixels each one. I need to flatten images and stakck it in the same array with dimensions 50000*3136 after that export the file as .CSV. I am doing this code which load all images in the memory which can cause a problem if we have not enough memory space.

imagesToArray[pathsrc_, pathdes_] := Module[{filesList, data}, (
   (* pathsrc_ : list of images path *)
   (* pathdes_ : CSV file path*)

   filesList = FileNames["*.png", pathsrc];
   Print["Number of images :", Length@filesList];
   data = 
    Table[Flatten[ImageData[Import[filesList[[i]]]]], {i, 1, 
      Length@filesList}];
   Export[pathdes, data];
   data
   )]

Is there any alternative to do it faster without memory problem?

share|improve this question
    
Why not data = Flatten [ ImageData /@ Import /@ fileList ] instead of Table[ ]? Am I missing something? – Dr. belisarius Mar 18 at 16:15
    
anyway, importing 50K images will take quite a long time – Dr. belisarius Mar 18 at 16:16
    
@Dr.belisarius, may be it will be faster but I think it load all data in memory – Developer2000 Mar 18 at 16:17
    
Why would you store such a large amount of binary data in csv? In any case you can read each file and write on the fly so you don't load everything in memory at once, you need to forgo Export to achieve that though. – george2079 Mar 18 at 16:19
4  
If you can forgo all the behind-the-scene checking that Import does, you can try using the built-in function that Import ultimately calls to read in a PNG file, which is Image`ImportExportDump`ImageReadPNG. It returns a list containing the image imported. You can use e.g. First@Image`ImportExportDump`ImageReadPNG[pathToImage] to get each image. It is roughly 10x faster than using Import. – MarcoB Mar 18 at 17:54
up vote 7 down vote accepted

You can combine @nikie's approach above with the use of a faster built-in import function.

If you can forgo all the behind-the-scene checking that Import does, you can try using the built-in function that Import ultimately calls to read in a PNG file, which is Image`ImportExportDump`ImageReadPNG. I found that out by using Trace@Import[somePNGfile] and wading through quite a lot of output. Incidentally, the counterpart to this function also exists, i.e. ImageWritePNG, but it requires multiple arguments and I haven't figured that one out yet; however, Export is already quite a bit zippier than Import at least for PNG files, so the need is less pronounced there.

It returns a list containing the image imported. To get each image, you can use e.g.

First@Image`ImportExportDump`ImageReadPNG[pathToImage]

It is at least 10x faster than using Import, and quite possibly more.

For instance, let's generate 200 small PNGs of the kind you are working with:

MapIndexed[
  Export["images\\" <> ToString[First@#2] <> ".png", #1] &,
  Image /@ RandomReal[{0, 1}, {200, 56, 56}]
];

Now let's compare timings:

First@Image`ImportExportDump`ImageReadPNG["images\\" <> ToString[#] <> ".png"] & /@ 
   Range[1, 200]; // AbsoluteTiming

Import["images\\" <> ToString[#] <> ".png"] & /@ Range[1, 200]; // AbsoluteTiming

(* Out:
{0.221601, Null}
{15.5932, Null}
*)

That's a 70x speedup!

share|improve this answer
    
Awesome!Does the same things can be happened in “.JPG”?Like something ImageReadJPG? – yode Mar 18 at 18:41
    
I did notnrun this code, I get this error {$Failed, 2} and this function seems not defined because it colored in blue – Developer2000 Mar 18 at 18:51
    
Thanks for your answer! How does we get the help about this kind of function. Is it really a built-in function? I have looked in the help but i did not get any infos! It seems magic ! – Developer2000 Mar 18 at 18:55
    
@Developer2000 Try evaluating the following: Image`ImportExportDump`ImageReadPNG[]. Although the function name will not be recognized (i.e. it will stay blue), you should obtain the following output: image. If you do, then the code above should work for you as well. You may also want to check that all the finicky backticks, etc. were copied correctly to your notebook. – MarcoB Mar 18 at 18:57
1  
Thanks a lot.I like this answer very much.And then do you mean Image`ImportExportDump requires multiple arguments ?But Image`ImportExportDump`ImageWritePNG["1.png",Image@RandomReal[{0,1},{200,56,3}]‌​] work well in my 10.4.0.It speed up many times than Export too. – yode Mar 18 at 23:42

You can use stream functions, like this (untested):

f = OpenWrite[pathdes];
chunkSize = 100;
Monitor[Do[
  (
     lines=ParallelTable[Flatten[ImageData[Import[filesList[[j]]]]],
            {j,i,Min[i+chunkSize-1,Length@filesList]}];
     WriteString[f, ExportString[lines, "CSV"]];
  ), 
  {i, 1, Length@filesList, chunkSize}], i];
Close[f];

to avoid the memory cost. I'd guess Import would be the bottleneck in that code, and there's not much you can do to make it faster.

share|improve this answer
    
Thanks for your answer but it does not gain more time than mine with 500 images: My solution gives 44s and yours gives 42s. – Developer2000 Mar 18 at 16:52
    
Like I said, you can avoid the memory cost, but there's not much you can do to improve Import's speed without rewriting the functionality. – nikie Mar 18 at 17:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.