8
$\begingroup$

When using large set of images for training neural networks (>10000, size 250 x 250 each) the time to read the images is quite long. My first approach was to use the DataSet functionality of Mathematica but saving large DataSets seems to consume all system memory (16GB) not reaching completion.

Is it possible to use the REC format as used in MxNet? And if so, how to do it?

Improve this question
$\endgroup$
1
  • $\begingroup$ This isn't an answer to your question, but in my experience using flash memory disk caching can prevent stalling if you know before hand approximately how much memory you will consume. $\endgroup$ – Feyre Apr 14 '17 at 7:41
7
$\begingroup$

As of 11.1, NetEncoder["Image"] uses a fast multithreaded image loader that accepts File[...] objects. 11.0 also supported File[...] objects but wasn't parallelized.

It supports formats like JPEG and PNG (it will fall back to slower code for GIF and more obscure image formats). Therefore all you need is to attach the appropriate NetEncoder to your net, and in your training data use File[...] objects instead of in-memory Image objects, and you should have no trouble doing out-of-core training at full speed.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thanks for the information. Is this multithreaded image loader based on the MXNet data iterators? Could you describe a little bit more about the implementations of this image loader? $\endgroup$ – xslittlegrass Apr 14 '17 at 17:39
  • 2
    $\begingroup$ I should add one proviso: this is only fast enough on SSD drives. Random seek on a non-SSD might be too slow to feed GPU's (often its 100x slower, see mxnet.io/tutorials/computer_vision/imagenet_full.html). The REC format that MXNet uses is designed to give good speeds on both SSD's and non-SSD's, as it avoids random seek. $\endgroup$ – Sebastian Apr 15 '17 at 17:59
  • 1
    $\begingroup$ @xslittlegrass: the image loader is written in C++ and uses OpenCV to load and decode the JPG, and also do other post-processing (resizing, color space conforming, etc). It launches one C++11 thread per physical core, and each thread processes a batch of images. $\endgroup$ – Sebastian Apr 15 '17 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.