Tracking progress of ParallelTable Computations

I have a notebook where I work with large datasets, where the data is stored as an A x B array of signals, where each signal is often 1000 samples long. In the past I would track the progress of various filters by having a 'counter' variable incrementing and using ProgressIndicator to monitor the progress.

With larger datasets I'm able to make huge improvements with ParallelTable, but as everyone knows using SetSharedVariable kills the computation speed, so I'm losing the ability to monitor the progress (some routines take 30 minutes or more to complete). Is there some other way to track the progress of ParallelTable aside from my method?

Below is a simple Table based example of my code:

Make a 100 x 100 array of 1000 sample length time signals.

counter = 0;
data = Table[ Sin[10 2 Pi t^2], {100}, {100}, {t, 0, 1, 0.001}];
Dynamic@ProgressIndicator[ Dynamic@counter, {1, 100*100}]
t0=AbsoluteTime[];
dataF = Table[ counter++;
BandpassFilter[ data[[i, j]], {10*2*Pi, 30*2*Pi}, SampleRate -> 1000], {i, 100}, {j, 100}];
AbsoluteTime[] - t0


The above code completes on my machine in about 8 seconds. Simply changing to ParallelTable reduces down to under 2 seconds. Utilizing the following code works, but it slows down considerably.

counter = 0;
SetSharedVariable@counter;
data = Table[ Sin[10 2 Pi t^2], {100}, {100}, {t, 0, 1, 0.001}];
Dynamic@ProgressIndicator[ Dynamic@counter, {1, 100*100}]
t0=AbsoluteTime[];
dataF = ParallelTable[ counter++;
BandpassFilter[ data[[i, j]], {10*2*Pi, 30*2*Pi}, SampleRate -> 1000], {i, 100}, {j, 100}];
AbsoluteTime[] - t0

• It's hacky, but maybe you could have one progress indicator per $KernelID, and $KernelCount many counters. That way there's no shared variable and each kernel is independent to work freely, but you can track the progress. Commented Jun 20, 2020 at 22:27

You can divide your workload into let's say 20 batches, evaluate your function over each batch in parallel and update the counter in between each batch. With decently sized batches, this will give you roughly the same speed as ParallelTable without any shared variables while also giving you decent enough feedback.

flat = Flatten[data, 1];
parts = Partition[flat, UpTo[500]];

eval[data_] := BandpassFilter[data, {10*2*Pi, 30*2*Pi}, SampleRate -> 1000]

Dynamic@ProgressIndicator[Dynamic@counter, {1, 100*100}]

t0 = AbsoluteTime[];
counter = 0;
groupedDataF = Module[{res}, Table[
res = ParallelMap[eval, part];
counter += Length[part];
res
,
{part, parts}
]];
dataF = ArrayReshape[Flatten[groupedDataF, 1], {100, 100, 1001}]
AbsoluteTime[] - t0


This takes 3.9 seconds on my computer, versus 3.1 with ParallelTable without shared variables. To be compared with 36 seconds when using a shared variable. With large batches it would be even faster, but the visual feedback would not be as detailed. ~0.4 seconds is spent on the array reshaping, so the difference is not entirely because of the parallelism.

• I like this approach however on larger data it slows down too much. For instance, on my workstation (3.33 GHz 6 core) the simple Table code completed in 269 seconds when run on a 155 x 144 x 650 data set. Using ParallelTable gets me down to 35 seconds. Implementing your code took 70 seconds. I played around with the parts length and it changed by maybe 2 seconds tops. Commented Jun 21, 2020 at 16:07
• This is obviously still much better that 269 seconds, but I try various parameters and rerun these filters so even that 35 seconds becomes valuable. I also tried just ParallelMap on Flatten[ data, 1] (forgoing the Partition step) and then reshaping. Surprisingly that completed in only 25 seconds. This was interesting because in other threads here Table was usually faster than Do, Map, etc. I'm going to hold out on accepting your answer to see if something else pops up, but it certainly was valuable and I'm looking at ParallelMap more now. Commented Jun 21, 2020 at 16:08