# Memory Management for Large Datasets [closed]

I've written some Mathematica code to generate polynomial roots. The code take an argument n as the highest degree of polynomial to solve for and then exports a file containing a list of the roots:

ClearAll["Global*"];

fileloc = FileNameJoin[{$HomeDirectory, "out.txt"}]; n = 16; coefficients = {-1, 0, 1}; polyGen[list_] := Total[((#1*x^#2 & ) @@ #1 & ) /@ Transpose[{list, Range[0,n]}]]; tupList[1] = {{1}}; tupList[n_] := Join[Prepend[#, 1] & /@ Tuples[coefficients, n - 1], Prepend[#, 0] & /@ tupList[n - 1]]; computeRoots[] := x /. Apply[Union, Map[NSolve[polyGen[#] == 0, x] &, tupList[n + 1]]]; Export[fileloc, computeRoots[]];  The code works by creating a list of polynomials represented by n-length lists and then parallel mapping a solving function over it. I'm pretty happy with the performance of the script but running this on a 120 Gb AWS instance, I've found that it devours memory and fails for$n \geq 15$. I attempted to solve this by writing a new version that writes to file as it computes and simply maps each term in the polynomial to Null. To deal with file IO with parallel threads, I simply have each kernel write to a different file: ClearAll["Global*"]; n = 16; ParallelEvaluate[file = OpenWrite[FileNameJoin[{$HomeDirectory,ToString[\$KernelID] <> ".txt"}]]];
coefficients = {-1, 0, 1};

polyGen[list_] := Total[((#1*x^#2 & ) @@ #1 & ) /@ Transpose[{list, Range[0,n]}]];
tupList[1] = {{1}};
tupList[n_] := Join[Prepend[#, 1] & /@ Tuples[coefficients, n - 1], Prepend[#, 0] & /@ tupList[n - 1]];
exe[] := ParallelMap[Write[file, x /. NSolve[polyGen[#]]] &, tupList[n + 1]];

exe[];
ParallelEvaluate[Close@file];


In early tests, this seems to do a little bit better than the last script but I still cant get the program to run in reasonable amounts of RAM and I'm not really sure what to do at this point. It seems like Mathematica doesnn't really have a precise memory management controls and I'm finding it difficult to understand the precise behavior of the garbage collector. Does anyone have any advice? How do you deal with memory when generating really large datasets?

## closed as off-topic by MarcoB, Jens, ilian, RunnyKine, Dr. belisariusJul 27 '15 at 23:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question arises due to a simple mistake such as a trivial syntax error, incorrect capitalization, spelling mistake, or other typographical error and is unlikely to help any future visitors, or else it is easily found in the documentation." – MarcoB, Jens, ilian, RunnyKine, Dr. belisarius
If this question can be reworded to fit the rules in the help center, please edit the question.

• A back-of-the-envelope calculation shows that tupList[16] is expected to take at least 11 GB (tuples grow like crazy). I guess that mapping NSolve over that will multiply that several times. So what else would you expect? – Sjoerd C. de Vries Jul 26 '15 at 21:45