# Deep neural network training on large datasets

I am training a deep net on large training dataset. I read the tutorial from the following link: Training on Large Datasets. I have put each predictor-preditand tuple into a separate mx file. My question is, since my input data is not image, how can I encode the file, so that NetTrain[net,{File[…]->…,…},…] can automatically does out-of-core learning on the dataset where File[…] represents my training data? Thanks!

• I think that the easiest way to go, is to use a generator function as explained in the section "Training with a Generator Function" on that page. Just adjust the function to import the data from the files with Import or Get. I'm guessing that the File[...] -> out format is optimized only for images. – Sjoerd Smit Oct 21 at 8:26
• Thanks Sjoered, "Input"->NetEncoder[{"Function",Import,dimension}]]] works for the NetEncoder. A following question is, can I have similar import function for the output, using NetDecoder? – Baoxiang Pan Oct 21 at 20:51
• You can just specify the generator function in the 2nd argument of NetTrain such that each evaluation of the function yields a batch of training examples (both input and output). That's probably the easiest. I don't know if you can do it through NetDecoder. – Sjoerd Smit Oct 22 at 6:59
• I tried NetTrain[net,Map[#[[1]]->Import[#[[2]]]&,data]], where data are in the format of {File[]->File[]}, it is working now. Thank you so much! – Baoxiang Pan Oct 23 at 17:11