Tracking progress of ParallelTable Computations

Question

I have a notebook where I work with large datasets, where the data is stored as an A x B array of signals, where each signal is often 1000 samples long. In the past I would track the progress of various filters by having a 'counter' variable incrementing and using ProgressIndicator to monitor the progress.

With larger datasets I'm able to make huge improvements with ParallelTable, but as everyone knows using SetSharedVariable kills the computation speed, so I'm losing the ability to monitor the progress (some routines take 30 minutes or more to complete). Is there some other way to track the progress of ParallelTable aside from my method?

Below is a simple Table based example of my code:

Make a 100 x 100 array of 1000 sample length time signals.

counter = 0;
data = Table[ Sin[10 2 Pi t^2], {100}, {100}, {t, 0, 1, 0.001}];
Dynamic@ProgressIndicator[ Dynamic@counter, {1, 100*100}]
t0=AbsoluteTime[];
dataF = Table[ counter++; 
BandpassFilter[ data[[i, j]], {10*2*Pi, 30*2*Pi}, SampleRate -> 1000], {i, 100}, {j, 100}];
AbsoluteTime[] - t0

The above code completes on my machine in about 8 seconds. Simply changing to ParallelTable reduces down to under 2 seconds. Utilizing the following code works, but it slows down considerably.

counter = 0;
SetSharedVariable@counter;
data = Table[ Sin[10 2 Pi t^2], {100}, {100}, {t, 0, 1, 0.001}];
Dynamic@ProgressIndicator[ Dynamic@counter, {1, 100*100}]
t0=AbsoluteTime[];
dataF = ParallelTable[ counter++; 
BandpassFilter[ data[[i, j]], {10*2*Pi, 30*2*Pi}, SampleRate -> 1000], {i, 100}, {j, 100}];
AbsoluteTime[] - t0

Answer 1

You can divide your workload into let's say 20 batches, evaluate your function over each batch in parallel and update the counter in between each batch. With decently sized batches, this will give you roughly the same speed as ParallelTable without any shared variables while also giving you decent enough feedback.

flat = Flatten[data, 1];
parts = Partition[flat, UpTo[500]];

eval[data_] := BandpassFilter[data, {10*2*Pi, 30*2*Pi}, SampleRate -> 1000]

Dynamic@ProgressIndicator[Dynamic@counter, {1, 100*100}]

t0 = AbsoluteTime[];
counter = 0;
groupedDataF = Module[{res}, Table[
    res = ParallelMap[eval, part];
    counter += Length[part];
    res
    ,
    {part, parts}
    ]];
dataF = ArrayReshape[Flatten[groupedDataF, 1], {100, 100, 1001}]
AbsoluteTime[] - t0

This takes 3.9 seconds on my computer, versus 3.1 with ParallelTable without shared variables. To be compared with 36 seconds when using a shared variable. With large batches it would be even faster, but the visual feedback would not be as detailed. ~0.4 seconds is spent on the array reshaping, so the difference is not entirely because of the parallelism.