# Out of memory in a Do loop

I have the following code:

out = {};
f[x_, y_, z_] := f[x, y, z] = x^3 + y^3 + z^3
Do[SeedRandom[];
x = RandomInteger[{10^2, 10^3}];
y = -RandomInteger[{10^2, 10^3}];
z = RandomChoice[{-1, 1}] RandomInteger[{10^2, 10^3}];
sol = f[x, y, z];
If[3 <= sol <= 1000,{AppendTo[out, {sol, {x,y,z}}],Export["new_k.dat", out, "Table"], Continue[]}, Continue[]],10^6];
out


which does a conceptually simple task: it randomly choses a triple {x,y,z} and computes sol as the sum of cubes. If sol is small enough, I want it to AppendTo to a predefined list out and Export that list to a file. The code is supposed to run a given number of iterations regardless of the number of sol's found (in fact, there will be very few instances that pass my criteria).

The problem is that even with the 10^6 iterations it uses a few Gb of memory. When I wanted to do 10^7, my computer crashed.

The procedure is rather straightforward: it draws a few numbers, do some algebra and comparison, and IF it fulfills the condition, it gets to be stored in out. So I expected it will use almost none memory, as I don't need it to remember all the instances when sol didn't meet the condition. It goes like numbers-condition-store or not-if not then don't remember anything-continue for a given number of iterations.

My intention in writing this in such a way is to run it for e.g. 10^10 iterations, go away and see in the file new_k.txt after a few hours if maybe some triple {x,y,z,} was found without interferring with the still running computation. Writing directly to a file is in case of power shortage or something.

(The range of {x,y,z} is intended to be bit larger, those above are just for the code to give an output sometimes)

• Why memoize f? (f[x_, y_, z_] := f[x, y, z] = ...). It's probably eating up your memory. (Note there are only about 6 billion combinations of x, y, z, so it makes more sense to run through them sequentially instead randomly, if you're contemplating 10^10 iteration.) – Michael E2 Aug 6 '16 at 2:00
• That's a habit I gained and I don't really know why... Nevertheless, that one change allowed to end up with using only hundreds of MB memory. So, in fact this is the best answer to "how to fix the memory issue with my code" as it does not affect all of my design. Although @Bill's answer below is slightly faster (by a few per cent). – corey979 Aug 6 '16 at 8:04

## 2 Answers

This

out = {};
f[x_, y_, z_] := x^3 + y^3 + z^3
Do[SeedRandom[];
x = RandomInteger[{10^2, 10^3}];
y = -RandomInteger[{10^2, 10^3}];
z = RandomChoice[{-1, 1}] RandomInteger[{10^2, 10^3}];
sol = f[x, y, z];
If[3 <= sol <= 1000,
AppendTo[out, {sol, {x, y, z}}];
Export["new_k.dat", out, "Table"]
];
If[Mod[i, 10^4] == 0,
LinkWrite[$ParentLink, SetNotebookStatusLine[FrontEndEvaluationNotebook[], ToString[{i, Length[out], MaxMemoryUsed[]}]]]]; , {i, 5*10^7}]; out  is mostly written in the style you prefer and just ran 5*10^7 iterations without using up gigabytes of memory. And it provides a little status display in the lower left corner of the window so you can keep track of progress without interrupting it. On your comment above about minds, what is in your mind, what is in Mathematica's mind and what you think is in Mathematica's mind are almost certainly completely different things. Even after you have more experience with Mathematica, it is almost certainly doing things internally that are very different from what you might assume. # Update If you need many more then generate smaller chunks and process in parallel. f[x_, y_, z_] := f[x, y, z] = x^3 + y^3 + z^3 find[n_] := Module[{vals = RandomInteger[{10^2, 10^3}, {n, 3}]}, vals[[All, 2]] *= -1; vals[[All, 3]] *= RandomChoice[{-1, 1}, n]; Select[{f[Sequence @@ #], #} & /@ vals, Between[{3, 1000}]@First@# &] ]  Then LaunchKernels and ParallelMap. LaunchKernels[]; out = DeleteDuplicates@*Join @@ ParallelMap[find, ConstantArray[10^5, 100]] CloseKernels[];  # OP There are quite a few issues with your code. I recommend that you read What are the most common pitfalls awaiting new users? In particular the bits on the proper use of semicolon ; and comma ,. In short you are not making very efficient use of memory and are using Do where it is not necessary to do so. Mathematica is a functional language so you need to program functionally to get the most out of it. Notice the difference in performance of the Do and the code below. f is defined as in your post. n = 10^6; vals = RandomInteger[{10^2, 10^3}, {n, 3}]; vals[[All, 2]] *= -1; vals[[All, 3]] *= RandomChoice[{-1, 1}, n]; Export["new_k.dat", Select[{f[Sequence @@ #], #} & /@ vals, Between[{3, 1000}]@First@# &], "Table"];  Also note that you don't need to put all your code in one cell in the notebook. Using multiple cells makes producing your results easier. Hope this helps. • Thanks for your effort but that does not really help. Your code is almost 10 times faster than mine for 10^7 iterations, but i) you store vals from the beginning what I wanted to avoid, and ii) it completely crashed when I took$5\cdot 10^7\$ iterations. My intent was to generate {x,y,z}, test if it passes my criteria, and if it does - write it to a file and go again; if it doesn't - forget about it and continue. If one does this procedure in his head, after an iteration his mind will be completely clear, with maybe some solution written on a sheet. I want MMA to work similarly. – corey979 Aug 6 '16 at 1:01
• "you store vals` from the beginning what I wanted to avoid" - @corey, I'm not sure why you object to this. Generating three random numbers in a single blow is more efficient than sequentially generating them one at a time. – J. M. will be back soon Aug 6 '16 at 4:03