Dealing with a huge dataset

Question

I have two ingredients here:

• a big dataset contained in a list, with ~ 20M values.
• a function that takes as each element of the list as input and yields True or False

I want to save somewhere the elements of the list that yielded True. Usually I would do something like that:

list = Range[10];
fun = PrimeQ;
Reap[
  If[fun[#]&,Sow[#]]& /@ list
]

This works perfectly, except the fact that we have to wait the end of the computation in order to be able to see all the results. When dealing with such huge lists sometimes waiting for the computation to finish is not an option.

This is what I am doing now to split the computation is more chunks,

SaveResult[list_, partitions_, fun_] := Table[
  Print["doing iteration ", i];
  If[
        #[[2]] =!= {},
        #[[2]] >>> NotebookDirectory[] <> "/Data/" <> ToString[i]
       ] &@
      Reap[(If[fun[#], Sow[#]]) & /@ Partition[list, partitions][[i]]], 
  {i, 1, 9}
  ];

My question is: is there a better way to saving partial results or dealing with huge datasets?

Answer 1

Rather than partition the whole list which will just gobble space (Edit: Turns out that's false in general for simple partitioning, but nonetheless the way you're doing it in your example can cause data to be missed unless the number of partitions is an exact divisor of the length of the list - you'd need Partition[list, psize, psize, {1, 1}, {}] otherwise), something like:

test = Range[20];

rasherFn[x_] := Pick[x, Positive[Mod[x, 3]]];

savedata[range_, data_] := {range, data};

doPartial = 
  Module[{data = #1, func = #3, saver = #4, partial, partrange},
     Map[(Print["Doing range: ", 
            partrange = #[[1]] + 1 ;; Min[Length@data, #[[2]]]];
            partial = func[data[[partrange]]];
            saver[partrange, partial]) &, 
          Partition[FindDivisions[{0, Length@data, 1}, #2], 2, 1]]] &;

doPartial[test, 5, rasherFn, savedata]


(*

Doing range: 1;;5
Doing range: 6;;10
Doing range: 11;;15
Doing range: 16;;20

{{1 ;; 5, {1, 2, 4, 5}}, {6 ;; 10, {7, 8, 10}}, {11 ;; 15, {11, 13, 14}}, {16 ;; 20, {16, 17, 19, 20}}}

*)

Where the arguments to doPartial are your list, the number of divisions to make (e.g. 10 will try to divide the work into 10 chunks), the function to apply to the segment of data, and the function to save the result, respectively.

I've used a simple example function (picking the Mod 3 values of the data where the result is positive), and a saving function that in this case just returns a list with the range of data and results of the function (here you'd want to export the result to a file).

Check the documentation for FindDivisions for more info on its operation, and note that it will try to get "close" to your desired # of divisions, but won't always match exactly. In any case, the end result will be division of the work with the union being the total work...

Answer 2

If the data set is truly large, I'd suggest saving some results to the disk if it is an option. This way, if anything goes wrong at any point, you don't have to re-compute everything all over again.

Of course, the disk operations will be slow, but I sometimes find persisting the state of the computation helpful.