18
$\begingroup$

I see there is doc about how to train a network Using out of core image classification and this question.But the object is only image.

I want to use a binary file as data(Sequence to Sequence case),for example like this.

data = Flatten@Table[{x, y} -> x*y, {x, -1, 1, .05}, {y, -1, 1, .05}];
mydata = Flatten[data /. {(a_ -> b_) -> {a, b}}];
BinaryWrite[file, mydata, "Real32", ByteOrdering -> -1];
Close[file];

Length of data:1681

The data looks like this:

enter image description here

Usually,the size of data is very large,so it is only a example.

I use this code:

fileName = "C:\\Users\\xiaoz\\Downloads\\test_data_SE.dat"; 
file = OpenRead[fileName, BinaryFormat -> True]; 
net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
size = FileByteCount[fileName];

read[file_, batchSize_] := If[StreamPosition[file] + 
 batchSize*3(*length of data in one batch*)*4(*float data*)> size,
 SetStreamPosition[file, 0]; BinaryReadList[file, "Real32", batchSize*3], 
 BinaryReadList[file, "Real32", batchSize*3]];

batchSize = 128;
Do[data = read[file, batchSize];
 trainingData = #[[1 ;; 2]] -> #[[3]] & /@ Partition[data, 3];
 net = NetTrain[net, trainingData, BatchSize -> batchSize, 
 MaxTrainingRounds -> 1,TrainingProgressReporting -> None], {200}]

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1}, 
 ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]
Close[file]

enter image description here

You can see,it is slow and the result is not perfect with enter image description here

So how to use Mathematica to train a network Using out of core classification?

Releated: I use TensorFlow can handle this Using Queue and Multi-thread: What's going on in tf.train.shuffle_batch and `tf.train.batch?

And the wolfram blog says:

Another thing that’s being introduced as an experiment in Version 11.3 is the MongoLink package, which supports connection to external MongoDB databases. We use MongoLink ourselves to manage terabyte-and-beyond datasets for things like machine learning training. And in fact MongoLink is part of our large-scale development effort—whose results will be seen in future versions—to seamlessly support extremely large amounts of externally stored data.

$\endgroup$
15
$\begingroup$

Okay here's how you do out-of-core training with HDF5:

input = RandomReal[1, {1000, 2}];
output = RandomReal[1, {1000, 2}];

Get["GeneralUtilities`"];
ExportStructuredHDF5["test.h5", <|"Input" -> input, 
  "Output" -> output|>]

NetTrain[LinearLayer["Input" -> 2, "Output" -> 2], File["test.h5"]]

The use of ExportStructuredHDF5 is just for convenience, you could also Export but it doesn't support associations directly. But again you'll need to make a dataset that consists of extendible columns if you want a real-world out-of-core example.

Also important to note is that you need to randomize the order of data yourself before writing it to the H5 file.

$\endgroup$
  • $\begingroup$ "...you need to randomize the order of data yourself..." - RandomSample[] ought to do the trick. $\endgroup$ – J. M. is away May 17 '17 at 18:46
  • 2
    $\begingroup$ @J.M. True. But if it was small enough that you could fit it in memory (and use RandomSample) then you probably wouldn't need to use an H5 file in the first place! H5 files can be useful for just holding training data, of course, but the main intended use is out-of-core learning. $\endgroup$ – Taliesin Beynon May 17 '17 at 22:28
  • 1
    $\begingroup$ @TaliesinBeynon Does it support ValidationSet -> File["data.h5"] yet? $\endgroup$ – partida May 18 '17 at 8:01
  • 1
    $\begingroup$ @partida auto-shuffling would result in non-contiguous lookups which would be prohibitively slow. ValidationSet is not supported yet, sorry. $\endgroup$ – Taliesin Beynon May 18 '17 at 17:05
  • 1
    $\begingroup$ @TaliesinBeynon I think it would still be beneficial to support multiple large h5 files while training (maybe sample each batch by cycling through the files if that's efficient enough). This would be easier to organize and manage than one giant h5 file, for example when continuously generating and transferring datasets from different machines and when removing older data. $\endgroup$ – dan7geo May 18 '17 at 22:44
16
$\begingroup$

There are two parts to your question: 1. How to use out-of-core classification and 2. Why is the result bad.

For the first part, you can use a generator to solve the problem. And for the second part, the reason for a bad result is because the data is not randomized.

fileName = "/Users/xslittlegrass/Downloads/test_data_SE.dat";

data = RandomSample@Flatten@Table[{x, y} -> x*y, {x, -1, 1, .05}, {y, -1, 1, .05}];
mydata = Flatten[data /. {(a_ -> b_) -> {a, b}}];
Close@BinaryWrite[fileName, mydata, "Real32", ByteOrdering -> -1];

Notice I use RandomSample to shuffle the data.

file = OpenRead[fileName, BinaryFormat -> True];
net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
size = FileByteCount[fileName];
read[file_, batchSize_] := 
  If[StreamPosition[file] + 
     batchSize*3(*length of data in one batch*)*4(*float data*)> size,
    SetStreamPosition[file, 0]; 
   BinaryReadList[file, "Real32", batchSize*3], 
   BinaryReadList[file, "Real32", batchSize*3]];

batchSize = 128;

We can define a generator that reads the data from the file

generator = Function[#[[1 ;; 2]] -> #[[3]] & /@ Partition[read[file, #BatchSize], 3]];

net = NetTrain[net, generator, BatchSize -> 128, MaxTrainingRounds -> 1000]
Close[file];

The result looks much better now

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1}, 
 ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]

enter image description here

$\endgroup$
  • $\begingroup$ Thank you very much!!! I want to add sth that If the size of file is large and want to shuffle data.The read function can be modified to read[file_, batchSize_] := Table[SetStreamPosition[file, RandomInteger[{0, size/(3*4) - 1}]*3*4]; BinaryReadList[file, "Real32", 3], batchSize]; $\endgroup$ – partida Apr 9 '17 at 2:43
14
$\begingroup$

@xslittlegrass's answer is perfect, but I want to give a heads up that we will ship a way to stream training data to NetTrain from an ".h5" file that can be arbitrarily big (e.g. hundreds of gigabytes). This will hopefully ship in 11.1.1 or 11.2. The ".h5" file must have a (very simple format): one dataset for each port, so in your example an "Input" dataset and an "Output" dataset.

Unfortunately it will remain undocumented for now for the reason that our existing HDF5 exporter cannot create extendible datasets using the documented functionality, so it's hard for you to use Mathematica to create the out-of-core dataset in the first place. You could obviously create it in something else, like Python. But for some power users it will be just the ticket, and much faster than using BinaryRead + your own generator.

$\endgroup$
  • 1
    $\begingroup$ Amazing! What a big step forward in neutral networks of Mathematica. $\endgroup$ – partida Apr 9 '17 at 0:51
  • 5
    $\begingroup$ An update: this made it into 11.1.1. I'll provide an example of how to use it once that ships. $\endgroup$ – Taliesin Beynon Apr 12 '17 at 12:45
  • $\begingroup$ 11.1.1 is release,where is the example? $\endgroup$ – partida May 2 '17 at 10:45
  • 2
    $\begingroup$ @TaliesinBeynon tell us, oh tell us please! ;) $\endgroup$ – M.R. May 4 '17 at 23:15
  • 1
    $\begingroup$ Haha @partida sorry! See my answer below. $\endgroup$ – Taliesin Beynon May 17 '17 at 18:39
4
$\begingroup$

Thank you @Taliesin Beynon.

And I write a python code to transform binary file to hdf5 file.

import os
import h5py
import numpy as np
import struct
import random
float_size=4

input_node=2
output_node=1
input_file='test_data_SE.dat'
out_file='test_data_SE.h5'

input_and_output_node=input_node+output_node
with open(input_file,'rb') as f:
    f.seek(0,os.SEEK_END)
    file_len=f.tell()/(float_size*input_and_output_node)

with h5py.File(out_file, "w") as f:
    Input =  f.create_dataset('Input',  (1681,input_node ),dtype='float', chunks=True)
    Output = f.create_dataset('Output', (1681,output_node),dtype='float', chunks=True)
    fin=open(input_file,'rb')
    index=range(file_len)
    random.shuffle(index)
    for i in index:
        Input[i]  = np.array(struct.unpack('<'+str(input_node )+'f',fin.read(float_size*input_node)))
        Output[i] = np.array(struct.unpack('<'+str(output_node)+'f',fin.read(float_size*output_node)))
    fin.close()    

Then import data to Mathematica, you can view the data:

enter image description here

net = NetChain[{32, Tanh, 1}, "Input" -> 2, "Output" -> "Scalar"];
net = NetTrain[net, File["test_data_SE.h5"],ValidationSet -> Scaled[0.2]]

enter image description here

ContourPlot[net[{x, y}], {x, -1, 1}, {y, -1, 1}, ColorFunction -> "RedGreenSplit", PlotLegends -> Automatic]
(*well done!*)

enter image description here

$\endgroup$
  • $\begingroup$ I'm not familiar with h5py, but it could be because you are using chunks that are too granular. In fact isn't your chunk-size 1 here? You should use a bigger size, some multiple of the batch size, like 1024 or something. $\endgroup$ – Taliesin Beynon May 18 '17 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.