2
$\begingroup$

I've been trying to train an RNN on some data that I pre-processed externally and saved to disk. My dataset is too large to fit in memory and is additionally varying in shape.

Looking for a way to do out-of-core training with such requirements, the closest thing I've found was this question, however it's not quite what I need.

So, my question is whether or not I can create custom NetEncoder (i.e. something I can pass into my NetGraph). Or if this is a feature being worked on?

Otherwise, is my only option to load subsets of my data for training?

$\endgroup$
1
$\begingroup$

The most general way of doing out-of-core training is to use the 5th usage of NetTrain (NetTrain[net, f]). You can write a custom data loader/sampler f that reads from disk (make sure its fast enough though!).

I would also recommend this tutorial to anyone thinking of doing out-of-core training: http://reference.wolfram.com/language/tutorial/NeuralNetworksLargeDatasets.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.