I have trained a classifier named c and want to apply it to testing set. However, due to the huge size of testing set, I have to make prediction in blocks/groups to save memory. Here is my code:

testFile = OpenRead["test.txt"];
resultFile = OpenWrite["prediction.txt"];
WriteLine[resultFile, "id,label"];

  Do[Block[{records, testX, testY},
     blockEnd = Min[blockBegin + blockSize - 1, totalLength - 1];
     actualBlockSize = blockEnd - blockBegin + 1;
     (*records is read from the file stream*)
     records = 
      Table[StringSplit[ReadLine[testFile], " "], actualBlockSize];
      testX = Table[
        SparseArray[#[[1]] -> #[[2]] & /@ 
          Map[Read[StringToStream[#], Number] &, 
           StringSplit[Drop[records[[i]], 1], ":"], {2}], maxcol, 
         0], {i, 1, actualBlockSize}], 
      ProgressIndicator[i, {1, actualBlockSize}]];
     testY = c[#, "Probabilities"][1] & /@ testX;

        ToString[#[[1]], CForm] <> "," <> ToString[#[[2]], CForm]] & /@
       Transpose@{Range[blockBegin, blockEnd], testY}
     ], {blockBegin, 0, totalLength - 1, blockSize}];
  , ProgressIndicator[blockBegin, {0, totalLength}]

I expect that when I execute the Do loop, memory cost should be stable. However, in reality, memory cost is continuously increasing, and the tendency is that my RAM will be run out. I expect that Block will clear memory occupied by variables records, testX and testY after execution every time, but it seems not. I still can't find any solution to this problem...

  • $\begingroup$ does the same thing happen with Module instead of Block? Any special reason you are using Block instead of Module btw? $\endgroup$ – Nasser May 8 '18 at 4:19
  • $\begingroup$ @Nasser I have tried Module[] but got the same performance checked in task manager in win10. No special reason indeed. I even tried Clear[] those temporary variables before the end of Block[] but it also cost too much memory. $\endgroup$ – MJ Z May 8 '18 at 5:59
  • $\begingroup$ May be you can try also $HistoryLength = 0 to see if it helps. $\endgroup$ – Nasser May 8 '18 at 6:02
  • $\begingroup$ I have tried at the very beginning of my code, just after setting my working directory. But unfortunately, it does not work, since the code shown above is as an independent program block. Should I set in this program block again? $\endgroup$ – MJ Z May 8 '18 at 6:12
  • $\begingroup$ I think $HistoryLength = 0 works on global. But you could always try. It is hard to suggest something, since one can't run your code as is. If you can make a MWE that does not require special files that only you have, may be someone can try better then. Otherwise may be someone else can spot what the issue is by looking at the code. I can't. $\endgroup$ – Nasser May 8 '18 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.