I've been trying to train an RNN on some data that I pre-processed externally and saved to disk. My dataset is too large to fit in memory and is additionally varying in shape.

Looking for a way to do out-of-core training with such requirements, the closest thing I've found was this question, however it's not quite what I need.

So, my question is whether or not I can create custom NetEncoder (i.e. something I can pass into my NetGraph). Or if this is a feature being worked on?

Otherwise, is my only option to load subsets of my data for training?


The most general way of doing out-of-core training is to use the 5th usage of NetTrain (NetTrain[net, f]). You can write a custom data loader/sampler f that reads from disk (make sure its fast enough though!).

I would also recommend this tutorial to anyone thinking of doing out-of-core training: http://reference.wolfram.com/language/tutorial/NeuralNetworksLargeDatasets.html


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.