0
$\begingroup$

I have a strange issue where the following program eats all memory. By commenting/uncommentig I found that the problem is connected to the KolmogorovSmirnovTest command. The txt files are just files with numbers. i is the Length of list (dis) which we get from the file. Files with the same i have the same size. All files combined are ~30GB if you really need it I can try to upload it.

$HistoryLength = 0;
ic = 50;
i = Table[2500*j, {j, ic}];
n = {200};
r = 1000;
KSTL = Table[0, {j, ic}];
Do[
 Print["i=" <> ToString[i[[j]]] <> " r=" <> ToString[k] <> 
   " Memory= " <> ToString[MemoryInUse[]]];

 str = OpenRead[
   "/home/someone/somewhere/results_r1000/fi_\
i_" <> ToString[i[[j]]] <> "_n_" <> ToString[n[[1]]] <> "_r_" <> 
    ToString[k - 1] <> ".txt"];
 dis = ReadList[str, Number];
 re = If[KolmogorovSmirnovTest[dis,NormalDistribution[], "ShortTestConclusion"] == 
"Do not reject", 1, 0];
 KSTL[[j]] = KSTL[[j]] + re;
 Remove[dis];
 Close[str];
 , {j, ic}, {k, r}]

part of the output.

i=2500 r=1 Memory= 75359488

KolmogorovSmirnovTest::ties: Ties exist in the data and will be     ignored for the KolmogorovSmirnov test, which assumes unique values.

i=2500 r=2 Memory= 118608248

KolmogorovSmirnovTest::ties: Ties exist in the data and will be ignored for the KolmogorovSmirnov test, which assumes unique values.

Remove::remal: Symbol Removed[dis] already removed.

i=2500 r=3 Memory= 118716136

Remove::remal: Symbol Removed[dis] already removed.

i=2500 r=4 Memory= 118738672

KolmogorovSmirnovTest::ties: Ties exist in the data and will be ignored for the KolmogorovSmirnov test, which assumes unique values.

General::stop: Further output of KolmogorovSmirnovTest::ties will be suppressed during this calculation.

Remove::remal: Symbol Removed[dis] already removed.

General::stop: Further output of Remove::remal will be suppressed during this calculation.

i=2500 r=5 Memory= 118762120

i=2500 r=6 Memory= 118784760

i=2500 r=7 Memory= 118806984

i=2500 r=8 Memory= 118829208

...

i=2500 r=44 Memory= 119643952

i=2500 r=45 Memory= 119666488

i=2500 r=46 Memory= 119689024

i=2500 r=47 Memory= 119711560

Be cause I have 50k files even the little difference fills all memory after some time.

$\endgroup$
  • $\begingroup$ Those errors are suggesting that the data is not in a numerical format. You will likely need to focus on getting the data into the program correctly. Try FullForm[dis] to see what you have. $\endgroup$ – bill s Mar 27 '17 at 12:13
  • $\begingroup$ The output of FullForm[dis] is List[0.704683, 0.230878,...,-0.641165, -0.028328]. The errors mean that I have points that are equal in my list, and KolmogorovSmirnovTest will ignored these. $\endgroup$ – Most Perfect Username Mar 27 '17 at 12:16
  • $\begingroup$ Do you get an error in the first iteration? Try removing the Do loop (set ic=1) and see if the first iteration finishes without error. $\endgroup$ – bill s Mar 27 '17 at 12:24
  • $\begingroup$ @bills Yes, I get the same error. I don't think the memory issue is because of the errors. I can write DeleteDuplicates[dis] which solves the errors, but the memory problem is the same. $\endgroup$ – Most Perfect Username Mar 27 '17 at 12:28
  • $\begingroup$ What is the memory issue? It looks to me as if the KS test disallows duplicate values -- hence removing them fixes that. Other than that, what is wrong with the memory? And of course, it is true that you have already removed the variable dis. Why not leave it in? $\endgroup$ – bill s Mar 27 '17 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.