How to use Mathematica to train a network Using out of core classification?

Question

I see there is doc about how to train a network Using out of core image classification and this question.But the object is only image.

I want to use a binary file as data(Sequence to Sequence case),for example like this.

data = Flatten@Table[{x, y} -> x*y, {x, -1, 1, .05}, {y, -1, 1, .05}];
mydata = Flatten[data /. {(a_ -> b_) -> {a, b}}];
BinaryWrite[file, mydata, "Real32", ByteOrdering -> -1];
Close[file];

Length of data:1681

The data looks like this:

Usually,the size of data is very large,so it is only a example.

I use this code:

fileName = "C:\\Users\\xiaoz\\Downloads\\test_data_SE.dat"; 
file = OpenRead[fileName, BinaryFormat -> True]; 
net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
size = FileByteCount[fileName];

read[file_, batchSize_] := If[StreamPosition[file] + 
 batchSize*3(*length of data in one batch*)*4(*float data*)> size,
 SetStreamPosition[file, 0]; BinaryReadList[file, "Real32", batchSize*3], 
 BinaryReadList[file, "Real32", batchSize*3]];

batchSize = 128;
Do[data = read[file, batchSize];
 trainingData = #[[1 ;; 2]] -> #[[3]] & /@ Partition[data, 3];
 net = NetTrain[net, trainingData, BatchSize -> batchSize, 
 MaxTrainingRounds -> 1,TrainingProgressReporting -> None], {200}]

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1}, 
 ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]
Close[file]

You can see,it is slow and the result is not perfect with

So how to use Mathematica to train a network Using out of core classification?

Releated: I use TensorFlow can handle this Using Queue and Multi-thread: What's going on in tf.train.shuffle_batch and `tf.train.batch?

And the wolfram blog says:

Another thing that’s being introduced as an experiment in Version 11.3 is the MongoLink package, which supports connection to external MongoDB databases. We use MongoLink ourselves to manage terabyte-and-beyond datasets for things like machine learning training. And in fact MongoLink is part of our large-scale development effort—whose results will be seen in future versions—to seamlessly support extremely large amounts of externally stored data.

Answer 1

Okay here's how you do out-of-core training with HDF5:

input = RandomReal[1, {1000, 2}];
output = RandomReal[1, {1000, 2}];

Get["GeneralUtilities`"];
ExportStructuredHDF5["test.h5", <|"Input" -> input, 
  "Output" -> output|>]

NetTrain[LinearLayer["Input" -> 2, "Output" -> 2], File["test.h5"]]

The use of ExportStructuredHDF5 is just for convenience, you could also Export but it doesn't support associations directly. But again you'll need to make a dataset that consists of extendible columns if you want a real-world out-of-core example.

Also important to note is that you need to randomize the order of data yourself before writing it to the H5 file.

Answer 2

There are two parts to your question: 1. How to use out-of-core classification and 2. Why is the result bad.

For the first part, you can use a generator to solve the problem. And for the second part, the reason for a bad result is because the data is not randomized.

fileName = "/Users/xslittlegrass/Downloads/test_data_SE.dat";

data = RandomSample@Flatten@Table[{x, y} -> x*y, {x, -1, 1, .05}, {y, -1, 1, .05}];
mydata = Flatten[data /. {(a_ -> b_) -> {a, b}}];
Close@BinaryWrite[fileName, mydata, "Real32", ByteOrdering -> -1];

Notice I use RandomSample to shuffle the data.

file = OpenRead[fileName, BinaryFormat -> True];
net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
size = FileByteCount[fileName];
read[file_, batchSize_] := 
  If[StreamPosition[file] + 
     batchSize*3(*length of data in one batch*)*4(*float data*)> size,
    SetStreamPosition[file, 0]; 
   BinaryReadList[file, "Real32", batchSize*3], 
   BinaryReadList[file, "Real32", batchSize*3]];

batchSize = 128;

We can define a generator that reads the data from the file

generator = Function[#[[1 ;; 2]] -> #[[3]] & /@ Partition[read[file, #BatchSize], 3]];

net = NetTrain[net, generator, BatchSize -> 128, MaxTrainingRounds -> 1000]
Close[file];

The result looks much better now

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1}, 
 ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]

Answer 3

@xslittlegrass's answer is perfect, but I want to give a heads up that we will ship a way to stream training data to NetTrain from an ".h5" file that can be arbitrarily big (e.g. hundreds of gigabytes). This will hopefully ship in 11.1.1 or 11.2. The ".h5" file must have a (very simple format): one dataset for each port, so in your example an "Input" dataset and an "Output" dataset.

Unfortunately it will remain undocumented for now for the reason that our existing HDF5 exporter cannot create extendible datasets using the documented functionality, so it's hard for you to use Mathematica to create the out-of-core dataset in the first place. You could obviously create it in something else, like Python. But for some power users it will be just the ticket, and much faster than using BinaryRead + your own generator.

Answer 4

Thank you @Taliesin Beynon.

And I write a python code to transform binary file to hdf5 file.

import os
import h5py
import numpy as np
import struct
import random
float_size=4

input_node=2
output_node=1
input_file='test_data_SE.dat'
out_file='test_data_SE.h5'

input_and_output_node=input_node+output_node
with open(input_file,'rb') as f:
    f.seek(0,os.SEEK_END)
    file_len=f.tell()/(float_size*input_and_output_node)

with h5py.File(out_file, "w") as f:
    Input =  f.create_dataset('Input',  (1681,input_node ),dtype='float', chunks=True)
    Output = f.create_dataset('Output', (1681,output_node),dtype='float', chunks=True)
    fin=open(input_file,'rb')
    index=range(file_len)
    random.shuffle(index)
    for i in index:
        Input[i]  = np.array(struct.unpack('<'+str(input_node )+'f',fin.read(float_size*input_node)))
        Output[i] = np.array(struct.unpack('<'+str(output_node)+'f',fin.read(float_size*output_node)))
    fin.close()    

Then import data to Mathematica, you can view the data:

net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
net = NetTrain[net, File["test_data_SE.h5"],ValidationSet -> Scaled[0.2]]

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1}, ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]
(*well done!*)