# Memory Management for Large Datasets [closed]

I've written some Mathematica code to generate polynomial roots. The code take an argument n as the highest degree of polynomial to solve for and then exports a file containing a list of the roots:

ClearAll["Global*"];

fileloc = FileNameJoin[{$HomeDirectory, "out.txt"}]; n = 16; coefficients = {-1, 0, 1}; polyGen[list_] := Total[((#1*x^#2 & ) @@ #1 & ) /@ Transpose[{list, Range[0,n]}]]; tupList = {{1}}; tupList[n_] := Join[Prepend[#, 1] & /@ Tuples[coefficients, n - 1], Prepend[#, 0] & /@ tupList[n - 1]]; computeRoots[] := x /. Apply[Union, Map[NSolve[polyGen[#] == 0, x] &, tupList[n + 1]]]; Export[fileloc, computeRoots[]];  The code works by creating a list of polynomials represented by n-length lists and then parallel mapping a solving function over it. I'm pretty happy with the performance of the script but running this on a 120 Gb AWS instance, I've found that it devours memory and fails for$n \geq 15$. I attempted to solve this by writing a new version that writes to file as it computes and simply maps each term in the polynomial to Null. To deal with file IO with parallel threads, I simply have each kernel write to a different file: ClearAll["Global*"]; n = 16; ParallelEvaluate[file = OpenWrite[FileNameJoin[{$HomeDirectory,ToString[\$KernelID] <> ".txt"}]]];
coefficients = {-1, 0, 1};

polyGen[list_] := Total[((#1*x^#2 & ) @@ #1 & ) /@ Transpose[{list, Range[0,n]}]];
tupList = {{1}};
tupList[n_] := Join[Prepend[#, 1] & /@ Tuples[coefficients, n - 1], Prepend[#, 0] & /@ tupList[n - 1]];
exe[] := ParallelMap[Write[file, x /. NSolve[polyGen[#]]] &, tupList[n + 1]];

exe[];
ParallelEvaluate[Close@file];


In early tests, this seems to do a little bit better than the last script but I still cant get the program to run in reasonable amounts of RAM and I'm not really sure what to do at this point. It seems like Mathematica doesnn't really have a precise memory management controls and I'm finding it difficult to understand the precise behavior of the garbage collector. Does anyone have any advice? How do you deal with memory when generating really large datasets?

• A back-of-the-envelope calculation shows that tupList is expected to take at least 11 GB (tuples grow like crazy). I guess that mapping NSolve over that will multiply that several times. So what else would you expect? – Sjoerd C. de Vries Jul 26 '15 at 21:45