0
$\begingroup$

When I inspect the details of a long computation using Trace, is there a way to know when a certain intermediate result is stored in the memory and, more importantly, when the corresponding memory is cleared?

For example, I create a set of sparse matrices

Ngamma = 24;
length = 2^(Ngamma/2);
s1 = SparseArray[{{0, 1}, {1, 0}}];
s2 = SparseArray[{{0, -I*1}, {I*1, 0}}];
s3 = SparseArray[{{1, 0}, {0, -1}}];
id = SparseArray[{{1, 0}, {0, 1}}];
gamma[1] = {1};
gamma[2] = {1};
Do[gamma[h] = {1}, {h, 3, Ngamma}];
Do[(
  Do[gamma[d] = KroneckerProduct[id, gamma[d]], {h, 1, (d - 1)/2 }];
  Do[gamma[d + 1] = KroneckerProduct[id, gamma[d + 1]], {h, 1, (d - 1)/2 }];
  gamma[d] = KroneckerProduct[gamma[d], s1];
  gamma[d + 1] = KroneckerProduct[gamma[d + 1], s2];
  Do[gamma[d] = KroneckerProduct[gamma[d], s3], {h, (d + 3)/2, Ngamma/2 }];
  Do[gamma[d + 1] = KroneckerProduct[gamma[d + 1], s3], {h, (d + 3)/2, Ngamma/2 }];
  ), {d, 1, Ngamma - 1, 2}
];

and I start to do some manipulations on them, using this function that I define here

gammaProduct[list_] := Dot @@ (gamma /@ list)

Just to give a very simple example:

Trace[gammaProduct[{1, 4, 5, 7}] + gammaProduct[{2, 3, 5, 9}]]

by inspecting the Trace, is there a way to know at each step, how many copies of the various matrices are stored and when the memory is released?

UPDATE:

As suggested I put a more detailed example, which can be thought as a follow up of my question

huge difference in memory usage

Let us consider these two pieces of code

Sum[RandomReal[] *
    gammaProduct[{i, j, k, l}], {l, 4, Ngamma}, {k, 3, 4}, {j, 2, 
   2}, {i, 1, 1}];

and

subsetCouplings = Subsets[Range @ Ngamma, {4}, 41]

followed by

Sum[ RandomReal[] *
    gammaProduct[i], {i, subsetCouplings}];

which do exactly the same thing, but the memory used in the two computations is very different. I checked explicitly that indeed in the intermediate steps there are differences in the way the two computations are performed (using Trace) but I cannot understand how these differences affect the memory use.

Notice that the difference cannot be given by the fact that in the second method we store all the tuples {i, j ,k ,l}, over which we sum, since the memory use to store these tuples is negligible with respect to the use of the memory for the actual computations involving the sparse matrices.

$\endgroup$
  • $\begingroup$ No, not by inspecting the Trace output as is. You could sprinkle some Echo[“some label” ~~ ToString@MemoryInUse[]] throughout your code maybe. $\endgroup$ – MarcoB May 28 at 14:20
  • $\begingroup$ Thank you, @MarcoB. Actually the idea is good, but I am not sure this approach can work: look for example the very simple line I posted: the code is very short (but on the back the system does several computations) and I do not know how to put some Echo[] there. $\endgroup$ – Dario Rosa May 28 at 14:46
  • $\begingroup$ Perhaps some context might help here. Why do you need to know about the transient memory consumption? Do you run out of memory? What is the actual problem you are trying to solve? $\endgroup$ – MarcoB May 28 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.