Strange behavior of ParallelTable

Question

I was trying to carry out the following calculations:

j = 0; SetSharedVariable[j];
resultlist1 = ParallelTable[j++; GaussianPotential[fieldlist[[ind2]], theta1, vz, v2, matrixsize, DisorderRangeRatio -> 0], {ind2, 1, num}];

At the same time, I am monitoring the progress by executing

Dynamic[Refresh[TableForm[{"ind1=", ind1, "ind2=", ind2, "j=", j}], UpdateInterval -> 1]]

There are two strange things:

(1) The behavior of j. The second command just quickly updates j from 1 to num, and stops there. I was expecting that the second command updates j each time it carries out a new calculation. How can I correct this?

(2) ParallelTable actually takes more time than Table. In contrast, when I execute the example in the official document,

Table[PrimeQ[x], {x, 10^1000, 10^1000 + 5000}]; // AbsoluteTiming
ParallelTable[PrimeQ[x], {x, 10^1000, 10^1000 + 5000}]; // AbsoluteTiming

The parallel version does save much time (5.87s vs 3.40s). What could be the reason for ParallelTable to be slower? Does it matter that the function GaussianPotential comes from a separate package I wrote?

Answer 1

Not a complete answer, but it may help to illustrate some of this behavior. If I run the simple example,

Dynamic[Refresh[TableForm[{"ind2=", ind2, "j=", j}], UpdateInterval -> 1]]

j = 0; num = 100; SetSharedVariable[j]; 
ParallelTable[ind2^2; Pause[2]; j++, {ind2, 1, num}]

I find that Dynamic does not display ind2. It does, however, display j, which increases in steps of 4 on my four-processor machine. The AbsoluteTiming is 50.2

On the other hand, with ParallelTable replaced by Table, both variables display, increasing in steps of 1. Here, AbsoluteTiming is 200.5. Thus, ParallelTable is four times as fast here, as expected.

Evidently, the OP finds ParallelTable slow, because it is moving large amount of data, as suggested by Sascha.

The actual ParallelTable output of this example is

{0, 4, 8, 14, 18, 22, 26, 30, 34, 1, 5, 9, 12, 16, 20, 24, 28, 32, 2, 
 6, 10, 13, 17, 21, 25, 29, 33, 3, 7, 11, 15, 19, 23, 27, 31, 35, 36, 
 40, 44, 48, 52, 56, 60, 65, 37, 41, 45, 49, 53, 57, 61, 64, 38, 42, 
 46, 50, 54, 58, 62, 66, 39, 43, 47, 51, 55, 59, 63, 67, 69, 73, 77, 
 81, 85, 89, 93, 97, 68, 72, 76, 80, 84, 88, 92, 96, 70, 74, 78, 82, 
 86, 90, 94, 98, 71, 75, 79, 83, 87, 91, 95, 99}

showing that ParallelTable assigns blocks of nine calculations at a time to each of the kernels in this case. Method can be used to specify how the computation of the table is distributed, and this sometimes affects timing.

Of course,

j = 0; num = 100; SetSharedVariable[j]; 
ParallelTable[Pause[2]; j++; ind2, {ind2, 1, num}]

simply returns a list of 1 to 100, as it should.