I have written a bunch of code to simulate some measurements. To speed things up, I intended to obtain the measurement values via a
ParallelTable. However, it turns out that there seem to be side-effects which cause the results from
ParallelTable to be occasionally wrong (the
Table equivalent produces reasonable results).
Since the code is pretty long with many functions that are temporarily needed, I would like to sketch the general scheme and ask for a way how to debug such issues. It appears to me that this has not been covered in a (somehow) general way, although I find it very relevant.
The system that is being measured is set up in a matrix
system[var1_,var2_,...]:=system[var1,var2,...]=... which depends on some global variables that are totally constant throughout all computation and really no longer touched after initialization. Basically, the time-dependence of
system is given by a scalar function
f[var1_,var2_,...]:=f[var1,var2,...]=... that takes the same arguments as
system. In a nutshell, I have a function that solves a system of coupled ODEs that depend on
system and measures some quantity (e.g. take the trace
Tr) of the particular solution after a certain time. The process of evolving the system and measuring is wrapped up in a function
measure[var1_,var2_,...]:=Module[...] that essentially also depends on the same variables as
gives moreorless the same results - with exceptions for some values of
var1,var2. Evaluating the
ParallelTable with same parameters a couple of times reveals that these exceptions "change position" (with respect to
var1,var2) so that I conclude it must be side-effects.
However, since I do not touch any global variables and all local variables are defined within
Block (it doesn't matter which one I pick), I do have no clue where the side-effects originate from.
I thought that memoization of
f might be an issue, but since they are never called twice with the same set of
var1,var2 during a
(Parallel)Table it appears to me that this should also not be an issue.
Because this is a problem that may often appear in general, I would like to know if
- someone can spot a possible origin of side-effects in my brief sketch of the code above
- someone can tell me how these issues can be "debugged" in general? I believe there must be a way to figure out which part of the code causes trouble
- it might be related to the fact that
measureinvolves functions like
MatrixExpthat automatically use multiple cores and it could lead to some conflicts with all local kernels since each kernel might want to distribute over the cores...
Update It appears to me that this is a problem specific to my Mathematica 10.0.2 on Linux 64bit at work (8cores). I wanted to replicate the behavior at home (Windows 8 64bit, Mathematica 10.0.1, 4 cores) - without success... So I will most likely update this question with code on Monday, when I am at work again.
I believe that I found out the issue, and I would like to know if you think that makes sense. Actually, as mentioned, none of my functions should cause side-effects. I just noticed, that my Mathematica kernels do not quit reliably. That means, a
Quit sometimes leaves a
WolframKernel and maybe followers open in
top. These left-overs also exist if I completly close Mathematica via the GUI, so that no Help/notebooks/whatsoever are opened. Obviously, this is not intended and I noticed that after closing all those left-overs there seem to be no side-effects. This somehow seems to solve my issue for this particular case but introduces even more problems because I do not want to investigate all running processes before I use Mathematica and clear manually - in fact, I would have to close everything, investigate
ps -A | grep "WolframKernel\|mathematica\|MathKernel" and
kill processes everytime I change some functions to be sure there is no unwanted crosstalk between the actual kernel and some "zombies". I know, probably there should be no connection to those left-over kernels, but apparently I observe no issues if they are killed.