# How to use Mathematica to train a network Using out of core classification?

I see there is doc about how to train a network Using out of core image classification and this question.But the object is only image.

I want to use a binary file as data(Sequence to Sequence case),for example like this.

data = Flatten@Table[{x, y} -> x*y, {x, -1, 1, .05}, {y, -1, 1, .05}];
mydata = Flatten[data /. {(a_ -> b_) -> {a, b}}];
BinaryWrite[file, mydata, "Real32", ByteOrdering -> -1];
Close[file];


Length of data:1681

The data looks like this:

Usually,the size of data is very large,so it is only a example.

I use this code:

fileName = "C:\\Users\\xiaoz\\Downloads\\test_data_SE.dat";
file = OpenRead[fileName, BinaryFormat -> True];
net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
size = FileByteCount[fileName];

batchSize*3(*length of data in one batch*)*4(*float data*)> size,

batchSize = 128;
trainingData = #[[1 ;; 2]] -> #[[3]] & /@ Partition[data, 3];
net = NetTrain[net, trainingData, BatchSize -> batchSize,
MaxTrainingRounds -> 1,TrainingProgressReporting -> None], {200}]

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1},
ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]
Close[file]


You can see,it is slow and the result is not perfect with

So how to use Mathematica to train a network Using out of core classification?

Releated: I use TensorFlow can handle this Using Queue and Multi-thread: What's going on in tf.train.shuffle_batch and tf.train.batch?

And the wolfram blog says:

Another thing that’s being introduced as an experiment in Version 11.3 is the MongoLink package, which supports connection to external MongoDB databases. We use MongoLink ourselves to manage terabyte-and-beyond datasets for things like machine learning training. And in fact MongoLink is part of our large-scale development effort—whose results will be seen in future versions—to seamlessly support extremely large amounts of externally stored data.

Okay here's how you do out-of-core training with HDF5:

input = RandomReal[1, {1000, 2}];
output = RandomReal[1, {1000, 2}];

Get["GeneralUtilities"];
ExportStructuredHDF5["test.h5", <|"Input" -> input,
"Output" -> output|>]

NetTrain[LinearLayer["Input" -> 2, "Output" -> 2], File["test.h5"]]


The use of ExportStructuredHDF5 is just for convenience, you could also Export but it doesn't support associations directly. But again you'll need to make a dataset that consists of extendible columns if you want a real-world out-of-core example.

Also important to note is that you need to randomize the order of data yourself before writing it to the H5 file.

• "...you need to randomize the order of data yourself..." - RandomSample[] ought to do the trick. May 17, 2017 at 18:46
• @J.M. True. But if it was small enough that you could fit it in memory (and use RandomSample) then you probably wouldn't need to use an H5 file in the first place! H5 files can be useful for just holding training data, of course, but the main intended use is out-of-core learning. May 17, 2017 at 22:28
• @TaliesinBeynon Does it support ValidationSet -> File["data.h5"] yet? May 18, 2017 at 8:01
• @partida auto-shuffling would result in non-contiguous lookups which would be prohibitively slow. ValidationSet is not supported yet, sorry. May 18, 2017 at 17:05
• @TaliesinBeynon I think it would still be beneficial to support multiple large h5 files while training (maybe sample each batch by cycling through the files if that's efficient enough). This would be easier to organize and manage than one giant h5 file, for example when continuously generating and transferring datasets from different machines and when removing older data. May 18, 2017 at 22:44

There are two parts to your question: 1. How to use out-of-core classification and 2. Why is the result bad.

For the first part, you can use a generator to solve the problem. And for the second part, the reason for a bad result is because the data is not randomized.

fileName = "/Users/xslittlegrass/Downloads/test_data_SE.dat";

data = RandomSample@Flatten@Table[{x, y} -> x*y, {x, -1, 1, .05}, {y, -1, 1, .05}];
mydata = Flatten[data /. {(a_ -> b_) -> {a, b}}];
Close@BinaryWrite[fileName, mydata, "Real32", ByteOrdering -> -1];


Notice I use RandomSample to shuffle the data.

file = OpenRead[fileName, BinaryFormat -> True];
net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
size = FileByteCount[fileName];
If[StreamPosition[file] +
batchSize*3(*length of data in one batch*)*4(*float data*)> size,
SetStreamPosition[file, 0];

batchSize = 128;


We can define a generator that reads the data from the file

generator = Function[#[[1 ;; 2]] -> #[[3]] & /@ Partition[read[file, #BatchSize], 3]];

net = NetTrain[net, generator, BatchSize -> 128, MaxTrainingRounds -> 1000]
Close[file];


The result looks much better now

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1},
ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]


• Thank you very much!!! I want to add sth that If the size of file is large and want to shuffle data.The read function can be modified to read[file_, batchSize_] := Table[SetStreamPosition[file, RandomInteger[{0, size/(3*4) - 1}]*3*4]; BinaryReadList[file, "Real32", 3], batchSize]; Apr 9, 2017 at 2:43

@xslittlegrass's answer is perfect, but I want to give a heads up that we will ship a way to stream training data to NetTrain from an ".h5" file that can be arbitrarily big (e.g. hundreds of gigabytes). This will hopefully ship in 11.1.1 or 11.2. The ".h5" file must have a (very simple format): one dataset for each port, so in your example an "Input" dataset and an "Output" dataset.

Unfortunately it will remain undocumented for now for the reason that our existing HDF5 exporter cannot create extendible datasets using the documented functionality, so it's hard for you to use Mathematica to create the out-of-core dataset in the first place. You could obviously create it in something else, like Python. But for some power users it will be just the ticket, and much faster than using BinaryRead + your own generator.

• Amazing! What a big step forward in neutral networks of Mathematica. Apr 9, 2017 at 0:51
• An update: this made it into 11.1.1. I'll provide an example of how to use it once that ships. Apr 12, 2017 at 12:45
• 11.1.1 is release,where is the example? May 2, 2017 at 10:45
• @TaliesinBeynon tell us, oh tell us please! ;)
– M.R.
May 4, 2017 at 23:15
• Haha @partida sorry! See my answer below. May 17, 2017 at 18:39

Thank you @Taliesin Beynon.

And I write a python code to transform binary file to hdf5 file.

import os
import h5py
import numpy as np
import struct
import random
float_size=4

input_node=2
output_node=1
input_file='test_data_SE.dat'
out_file='test_data_SE.h5'

input_and_output_node=input_node+output_node
with open(input_file,'rb') as f:
f.seek(0,os.SEEK_END)
file_len=f.tell()/(float_size*input_and_output_node)

with h5py.File(out_file, "w") as f:
Input =  f.create_dataset('Input',  (1681,input_node ),dtype='float', chunks=True)
Output = f.create_dataset('Output', (1681,output_node),dtype='float', chunks=True)
fin=open(input_file,'rb')
index=range(file_len)
random.shuffle(index)
for i in index:
fin.close()


Then import data to Mathematica, you can view the data:

net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
net = NetTrain[net, File["test_data_SE.h5"],ValidationSet -> Scaled[0.2]]


ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1}, ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]
(*well done!*)


• I'm not familiar with h5py, but it could be because you are using chunks that are too granular. In fact isn't your chunk-size 1 here? You should use a bigger size, some multiple of the batch size, like 1024 or something. May 18, 2017 at 17:08