3
$\begingroup$

here is how you can generate my problem, I'm using Mathematica 11 on Ubuntu 14.04 ("11.0.1 for Linux x86 (64-bit) (September 21, 2016)").

$HistoryLength = 0;
ic = 10;
MemoryInUse[]
Do[dis = Flatten[
Import["/home/someone/tests/fi.txt", "Table"]];
KolmogorovSmirnovTest[dis], {k, ic}]
MemoryInUse[]
Remove["Global`*"]
MemoryInUse[]

In my case there is ~78 MB what should be ~27 MB, which will be the case if I use QuitKernel from Evaluation. Is such behavior normal? If yes then why?

fi.txt is just a file with numbers (pastebin ).

ps I deleted a part that was essential while simplifying the code, now it "works", if you run the code for larger ic the difference is getting bigger. The following is the output.

Out[3]= 27698584

During evaluation of In[1]:= KolmogorovSmirnovTest::ties: Ties exist in the data and will be ignored for the KolmogorovSmirnov test, which assumes unique values. 

During evaluation of In[1]:= KolmogorovSmirnovTest::ties: Ties exist in the data and will be ignored for the KolmogorovSmirnov test, which assumes unique values.

During evaluation of In[1]:= KolmogorovSmirnovTest::ties: Ties exist in the data and will be ignored for the KolmogorovSmirnov test, which assumes unique values.

During evaluation of In[1]:= General::stop: Further output of KolmogorovSmirnovTest::ties will be suppressed during this calculation.

Out[5]= 78117472

Out[7]= 78056240
$\endgroup$
  • $\begingroup$ I used ic = 1000; because I didn't want to wait but I don't see the memory growth you describe in v10.1 under Windows x64; it remains essentially the same after the first run. $\endgroup$ – Mr.Wizard Mar 22 '17 at 12:46
  • $\begingroup$ Please post the full output of $Version, and show the actual numbers you get after running this once, twice, three time, etc. in a newly started kernel. $\endgroup$ – Szabolcs Mar 22 '17 at 12:52
  • $\begingroup$ OK, but what happens if you run it several more times? Does the memory use increase? On my machine it doesn't. Thus I do not see the problem. $\endgroup$ – Szabolcs Mar 22 '17 at 15:02
6
$\begingroup$

The lingering memory usage that you are seeing might be due to packages being auto-loaded upon their first use. I can reproduce the reported behaviour, but only from a fresh kernel session. The memory "leak" does not occur on subsequent evaluations.

The exhbited code triggers the Units functionality to be auto-loaded. We can observe that this package and its dependencies consume 30+ MB of memory. From a fresh kernel session:

$HistoryLength = 0;

{ "unit symbols before"
, CalculateUnits`UnitCommonSymbols`UnitLookup // DownValues // ByteCount
, QuantityUnits`Private`PrimaryDestinationLookup // DownValues // ByteCount
}

{ "memory before", Share[]; MemoryInUse[] }

QuantityUnit; (* triggers the auto-load *)

{ "unit symbols after"
, CalculateUnits`UnitCommonSymbols`UnitLookup // DownValues // ByteCount
, QuantityUnits`Private`PrimaryDestinationLookup // DownValues // ByteCount
}

{ "memory after", Share[]; MemoryInUse[] }

The results on my machine (V11.0.1, Win7, x64):

{unit symbols before, 40, 40}
{memory before, 32603504}
{unit symbols after, 22254960, 10750424}
{memory after, 68659000}

The individual symbol byte counts are only approximate because they do not take shared structures into consideration. Note how the reported memory usage is very close to the figures reported in the question -- and this without loading a large data file.

If I change the example in the question to auto-load the units functionality (by evaluating QuantityUnit; immediately after $HistoryLength = 0;) then I no longer see a radical increase in memory usage after the global symbols are cleared. I still see an increase, but I suggest that it is due to other packages being auto-loaded in addition to the units package.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.