Out of memory in a Do loop

Question

I have the following code:

out = {};
f[x_, y_, z_] := f[x, y, z] = x^3 + y^3 + z^3
Do[SeedRandom[];
x = RandomInteger[{10^2, 10^3}];
y = -RandomInteger[{10^2, 10^3}];
z = RandomChoice[{-1, 1}] RandomInteger[{10^2, 10^3}];
sol = f[x, y, z];
If[3 <= sol <= 1000,{AppendTo[out, {sol, {x,y,z}}],Export["new_k.dat", out, "Table"], Continue[]}, Continue[]],10^6];
out

which does a conceptually simple task: it randomly choses a triple {x,y,z} and computes sol as the sum of cubes. If sol is small enough, I want it to AppendTo to a predefined list out and Export that list to a file. The code is supposed to run a given number of iterations regardless of the number of sol's found (in fact, there will be very few instances that pass my criteria).

The problem is that even with the 10^6 iterations it uses a few Gb of memory. When I wanted to do 10^7, my computer crashed.

The procedure is rather straightforward: it draws a few numbers, do some algebra and comparison, and IF it fulfills the condition, it gets to be stored in out. So I expected it will use almost none memory, as I don't need it to remember all the instances when sol didn't meet the condition. It goes like numbers-condition-store or not-if not then don't remember anything-continue for a given number of iterations.

My intention in writing this in such a way is to run it for e.g. 10^10 iterations, go away and see in the file new_k.txt after a few hours if maybe some triple {x,y,z,} was found without interferring with the still running computation. Writing directly to a file is in case of power shortage or something.

(The range of {x,y,z} is intended to be bit larger, those above are just for the code to give an output sometimes)

Answer 1

This

out = {};
f[x_, y_, z_] := x^3 + y^3 + z^3
Do[SeedRandom[];
   x = RandomInteger[{10^2, 10^3}];
   y = -RandomInteger[{10^2, 10^3}]; 
   z = RandomChoice[{-1, 1}] RandomInteger[{10^2, 10^3}]; 
   sol = f[x, y, z];
   If[3 <= sol <= 1000,
      AppendTo[out, {sol, {x, y, z}}];
      Export["new_k.dat", out, "Table"]
   ];
   If[Mod[i, 10^4] == 0, 
   LinkWrite[$ParentLink, SetNotebookStatusLine[FrontEnd`EvaluationNotebook[], 
      ToString[{i, Length[out], MaxMemoryUsed[]}]]]];
  , {i, 5*10^7}];
out

is mostly written in the style you prefer and just ran 5*10^7 iterations without using up gigabytes of memory. And it provides a little status display in the lower left corner of the window so you can keep track of progress without interrupting it.

On your comment above about minds, what is in your mind, what is in Mathematica's mind and what you think is in Mathematica's mind are almost certainly completely different things. Even after you have more experience with Mathematica, it is almost certainly doing things internally that are very different from what you might assume.

Answer 2

Update

If you need many more then generate smaller chunks and process in parallel.

f[x_, y_, z_] := f[x, y, z] = x^3 + y^3 + z^3

find[n_] :=
 Module[{vals = RandomInteger[{10^2, 10^3}, {n, 3}]},
  vals[[All, 2]] *= -1;
  vals[[All, 3]] *= RandomChoice[{-1, 1}, n];
  Select[{f[Sequence @@ #], #} & /@ vals, Between[{3, 1000}]@First@# &]
  ]

Then LaunchKernels and ParallelMap.

LaunchKernels[];
out = DeleteDuplicates@*Join @@ ParallelMap[find, ConstantArray[10^5, 100]]
CloseKernels[];

OP

There are quite a few issues with your code. I recommend that you read What are the most common pitfalls awaiting new users? In particular the bits on the proper use of semicolon ; and comma ,.

In short you are not making very efficient use of memory and are using Do where it is not necessary to do so. Mathematica is a functional language so you need to program functionally to get the most out of it. Notice the difference in performance of the Do and the code below. f is defined as in your post.

n = 10^6;
vals = RandomInteger[{10^2, 10^3}, {n, 3}];
vals[[All, 2]] *= -1;
vals[[All, 3]] *= RandomChoice[{-1, 1}, n];
Export["new_k.dat",
  Select[{f[Sequence @@ #], #} & /@ vals, Between[{3, 1000}]@First@# &],
  "Table"];

Also note that you don't need to put all your code in one cell in the notebook. Using multiple cells makes producing your results easier.

Hope this helps.