2
$\begingroup$

I recognised that for some reason when I use Compile and use a C-compiler the memory consumption of Mathematica's MathKernel is very large even after I reset all the stored data or variables with Clear["Global`*"].

I thought the aforementioned command would reset just everything that is stored inside the kernel. Why is Mathematica still occupying a huge amount of memory. Is the overhead of Compile so big and what can I do to improve it?

[Edit]

My problem seems to be related to the Parallelization option within Compile. If enter my arguments as follows

a=1;
b=1;
c=1;
cfun[a,b,c]

the huge overhead during the evaluation of cfun is removed after calling cfun and only the output of cfun occupies memory (no matter the value of the Parallelization option within compile).

However, if I call Parallelization->True together with RuntimeAttributes -> {Listable} and insert the arguments as follows

n=1;
a=Table[1,{n}];
b=Table[1,{n}];
c=1Table[1,{n}]
cfun[a,b,c]

then even for n=1 a huge overhead is produced that eventually occupies more memory than the actual output of cfun. The goal why I wanted to write the arguments as lists and use the listable attribute was to use effective parallelization when increasing n to higher integers like n=3.

So for my program, with Parallelization->False and the latter technique of entering the arguments, my program occupies eventually only 4.9% of my memory. Setting Parallelization->True with the same length of output it consumes 19.3% of my memory.

This is a remarkable huge overhead when calling the Parallelization option. Is there any way to improve it, or is this option simply like a black box?

[Edit2]

I think this thread might be slightly related to my problem:

Listable compiled function crashes the kernel with "no more memory available"

$\endgroup$

closed as off-topic by halirutan, xzczd, m_goldberg, anderstood, LCarvalho Feb 28 '18 at 1:07

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question cannot be answered without additional information. Questions on problems in code must describe the specific problem and include valid code to reproduce it. Any data used for programming examples should be embedded in the question or code to generate the (fake) data must be included." – halirutan, xzczd, m_goldberg, anderstood, LCarvalho
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ If you genuinely want to reset everything in the kernel I would use Quit. $\endgroup$ – user6014 Feb 8 '18 at 15:34
  • $\begingroup$ Quit quits the kernel yes. But I want to find out what is causing the huge memory consumption. Are there any commands to see what Mathematica stores and where it is stored? Is it necessary for Compile to store something somewhere which I am not aware of? Or should Compile only occupy as much memory as the output of the compiled function requires? $\endgroup$ – Display Name Feb 8 '18 at 15:42
  • $\begingroup$ Ok, so if I use $HistoryLength = 0 and then clear my compiled function cfun then all the memory seems to be accessible again. Within my Compile I use a Block environment with many local variables. Is there a way to just clear everything within my compiled function after using it, but keeping the function itself available? (if I use clear I need to re-input the function again if I want to use it again) $\endgroup$ – Display Name Feb 8 '18 at 16:04
  • $\begingroup$ See my Edit in the OP. I figured out that my problem is related to the Parallelization option within compiled. $\endgroup$ – Display Name Feb 9 '18 at 8:20
  • $\begingroup$ Without a simple example to test, it is hard to say what happens on your machine. $\endgroup$ – halirutan Feb 25 '18 at 5:52

Browse other questions tagged or ask your own question.