0
$\begingroup$

My memory is 32GB, windows 10, cpu 7700k, GTX1080

50k images[400MB] cost me 30 minutes. In a CAPTCHA example, a simple net and in tunning hyper-parameters step with small values of MaxTrainingRounds, just 15 minutes training time consuming.

If Mathematica Kernel crashes, then I should import images again, it's very often[long term] and annoying.

So, I can DumpSave, too many disk memory costs, acceptable.

Can I speed up the import this images for purpose of training network?

Any other methods to make the whole thing efficient? wdx, readlist as txt?

$\endgroup$
5
  • $\begingroup$ Did you try to store your images in one multipage file, like tif? I guess, reading of the one big file is faster for traditional hard drives. I've tried such tif's up to 2GB with 20k frames, the import time is ~ 1min. $\endgroup$
    – Rom38
    Jun 20, 2017 at 4:21
  • $\begingroup$ You can refer my question How to use Mathematica to train a network Using out of core classification? And how do to generate a .h5 file $\endgroup$
    – partida
    Jun 23, 2017 at 5:10
  • $\begingroup$ @partida Hi, There is a bug in partida ... see mathematica.stackexchange.com/questions/148714/… $\endgroup$ Jun 23, 2017 at 7:26
  • $\begingroup$ @Rom38 hi, that's fast, what's your images' dimension? 100*100 or 300*300? $\endgroup$ Jun 25, 2018 at 9:41
  • $\begingroup$ @HyperGroups, they are from 288*288 to 768*512. But of course, high-res is shorter.. $\endgroup$
    – Rom38
    Jun 25, 2018 at 11:09

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.