Recently I'm trying to process some experiment data, my PC with OS Win10 (mathematica v10.4) is not enough, so I connect to another PC with OS Ubuntu 16.04 (mathematica 11.0) by the way supported here.
I hope to speed up by this way since I can use another two kernels.
However, when I use ParallelTable, I find only one local kernel works well and another three local kernels are at leisure at first, then all of the kernels seems at leisure I guess all the computation has been transferred to remote kernels, I don't know why.
At first:
Later:
In the picture, the fist four kernels are local kernels and other two are remote kernels. Here is my code (the code used to configure remote kernel is as same as in the preceding link):
filelist = Table[FileNames["*.dat", ToString[i] <> "\\" <> ToString[j]],{i,1,3}, {j, 1, 21}];
specRow = Table[Import[filelist[[j, k, l]]], {j, 1, 3}, {k, 1, 21}, {l, 1, 401}];
od[specFileNameList_] := With[{pr = Flatten@specFileNameList[[401]]},Table[With[{spec =Flatten@specFileNameList[[k]]},Reverse[ArrayReshape[Riffle[frequency, -Log[N[(spec-bg)/(pr-bg)]]], {1024,2}]]], {k, 1, 400}]]
spec = ParallelTable[od[specRow[[i, j]]], {i, 1, 3}, {j, 1, 21}];
Here, specRow, pr and bg are lists.
Update I've change my code following Szabolcs's advice below, but situation doesn't change, here is my new code:
list = Tuples[{Range[1, 3], Range[1, 21]}];
spec = ParallelTable[Apply[od[Part[specRow, #1, #2]] &, list[[i]]], {i, 1, 63}];
ParallelTable[od[specRow[[i, j]]], {i, 1, 3}, {j, 1, 21}];
<- parallelization happens only according to the first iterator,{i, 1, 3}
, meaning that this particular computation won't use more than 3 kernels. Workaround: Create the{i,j}
pairs first, usingTuples
, thenParallelMap
an appropriate function over them. $\endgroup$