I am training a deep net on large training dataset. I read the tutorial from the following link: Training on Large Datasets. I have put each predictor-preditand tuple into a separate mx file. My question is, since my input data is not image, how can I encode the file, so that NetTrain[net,{File[…]->…,…},…] can automatically does out-of-core learning on the dataset where File[…] represents my training data? Thanks!

  • $\begingroup$ I think that the easiest way to go, is to use a generator function as explained in the section "Training with a Generator Function" on that page. Just adjust the function to import the data from the files with Import or Get. I'm guessing that the File[...] -> out format is optimized only for images. $\endgroup$ – Sjoerd Smit Oct 21 at 8:26
  • $\begingroup$ Thanks Sjoered, "Input"->NetEncoder[{"Function",Import,dimension}]]] works for the NetEncoder. A following question is, can I have similar import function for the output, using NetDecoder? $\endgroup$ – Baoxiang Pan Oct 21 at 20:51
  • $\begingroup$ You can just specify the generator function in the 2nd argument of NetTrain such that each evaluation of the function yields a batch of training examples (both input and output). That's probably the easiest. I don't know if you can do it through NetDecoder. $\endgroup$ – Sjoerd Smit Oct 22 at 6:59
  • $\begingroup$ I tried NetTrain[net,Map[#[[1]]->Import[#[[2]]]&,data]], where data are in the format of {File[]->File[]}, it is working now. Thank you so much! $\endgroup$ – Baoxiang Pan Oct 23 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.