0
$\begingroup$

My memory is 32GB, windows 10, cpu 7700k, GTX1080

50k images[400MB] cost me 30 minutes. In a CAPTCHA example, a simple net and in tunning hyper-parameters step with small values of MaxTrainingRounds, just 15 minutes training time consuming.

If Mathematica Kernel crashes, then I should import images again, it's very often[long term] and annoying.

So, I can DumpSave, too many disk memory costs, acceptable.

Can I speed up the import this images for purpose of training network?

Any other methods to make the whole thing efficient? wdx, readlist as txt?

Improve this question
$\endgroup$
5
  • $\begingroup$ Did you try to store your images in one multipage file, like tif? I guess, reading of the one big file is faster for traditional hard drives. I've tried such tif's up to 2GB with 20k frames, the import time is ~ 1min. $\endgroup$
    – Rom38
    Commented Jun 20, 2017 at 4:21
  • $\begingroup$ You can refer my question How to use Mathematica to train a network Using out of core classification? And how do to generate a .h5 file $\endgroup$
    – partida
    Commented Jun 23, 2017 at 5:10
  • $\begingroup$ @partida Hi, There is a bug in partida ... see mathematica.stackexchange.com/questions/148714/… $\endgroup$
    – HyperGroups
    Commented Jun 23, 2017 at 7:26
  • $\begingroup$ @Rom38 hi, that's fast, what's your images' dimension? 100*100 or 300*300? $\endgroup$
    – HyperGroups
    Commented Jun 25, 2018 at 9:41
  • $\begingroup$ @HyperGroups, they are from 288*288 to 768*512. But of course, high-res is shorter.. $\endgroup$
    – Rom38
    Commented Jun 25, 2018 at 11:09

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.