I have some nicely formatted training data in HDF5 files, i.e. the images and labels are serialized within the H5 file as groups and datasets.

AFAIK NetTrain only works in an out-of-core mode on lists of paths each wrapped in File.

Is there any way to get NetTrain to work off of the training data stored in an H5 file efficiently (with parallelized batch loading)?

Update from comment:

@AlexeyGolyshev suggested to use @TaliesinBeynon's undocumented answer which doesn't seem to work anymore in v11.3:

{input, output} = RandomReal[1, {2, 1000, 2}];
<< GeneralUtilities`;
fn = FileNameJoin[{CreateDirectory[], "test.h5"}]
ExportStructuredHDF5[fn, <|"Input" -> input, "Output" -> output|>];
NetTrain[LinearLayer["Input" -> 2, "Output" -> 2], File[fn]]

enter image description here


Example of how to pack images in HDF5 and train neural network

X = Table[RandomImage[1, {32, 32}, ColorSpace -> "RGB"], 10];
Y = RandomInteger[1, 10];

enc = NetEncoder[{"Image", {32, 32}, ColorSpace -> "RGB"}];

  "Datasets" ->
    "Input" -> Flatten /@ enc@X,
    "Output" -> N@Y
  "DataFormat" -> {Automatic, Automatic}

net = NetChain[{
   ReshapeLayer[{3, 32, 32}],
   ConvolutionLayer[32, {3, 3}],
  "Input" -> 3*32*32,
  "Output" -> NetDecoder[{"Class", {0, 1}}]

enter image description here

netT = NetTrain[net, File["data.h5"]]

enter image description here

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.