5
$\begingroup$

After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this

{1, 17, 39, 53, 44, 23} -> {18, 53, 50, 38, 6, 31}

For an entire 66GB file. I have tried

ReadList[$fileStream, Expression,1000]

but this only reads the first 1000.

Any assistance would be appreciated.

Also, any good books on this subject?

$\endgroup$
6
$\begingroup$
f = OpenWrite["data.txt"];

SeedRandom[0];
Do[Write[f, RandomInteger[{0, 100}, 5] -> RandomInteger[{0, 100}, 5]], {10000}]

Close[f];
f = OpenRead["data.txt"];

generator = Function[
   Table[
      With[
       {r = Read[f, Record]},
       If[r === EndOfFile,
        SetStreamPosition[f, 0]; Read[f, Record],
        r
        ]
       ],
      {#BatchSize}
      ] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
   ];

net = NetChain[
  {
   LinearLayer[16],
   LinearLayer[5]
   },
  "Input" -> 5,
  "Output" -> 5
  ]

SetStreamPosition[f, 0];
netT = NetTrain[
  net,
  {generator, "RoundLength" -> 10},
  All,
  BatchSize -> 1000, MaxTrainingRounds -> 10
  ]

enter image description here

$\endgroup$
  • $\begingroup$ Note: Mathematica seems to ignore RoundLength for my generator. Play with MaxTrainingRounds. $\endgroup$ – Alexey Golyshev Mar 28 at 14:19
  • $\begingroup$ Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions? $\endgroup$ – Michel Mesedahl Mar 30 at 12:52
  • $\begingroup$ @MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f] inserted in generator). For your 66 GB file, you must change MaxTrainingRounds (to millions or billions). $\endgroup$ – Alexey Golyshev Mar 30 at 17:43
  • $\begingroup$ Ok, now I get the jist of it. Thank You $\endgroup$ – Michel Mesedahl Mar 30 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.