# How to determine side-effects in parallel computation? Possibly associated to Kernel not closing reliably

I have written a bunch of code to simulate some measurements. To speed things up, I intended to obtain the measurement values via a ParallelTable. However, it turns out that there seem to be side-effects which cause the results from ParallelTable to be occasionally wrong (the Table equivalent produces reasonable results).

Since the code is pretty long with many functions that are temporarily needed, I would like to sketch the general scheme and ask for a way how to debug such issues. It appears to me that this has not been covered in a (somehow) general way, although I find it very relevant.

The system that is being measured is set up in a matrix system[var1_,var2_,...]:=system[var1,var2,...]=... which depends on some global variables that are totally constant throughout all computation and really no longer touched after initialization. Basically, the time-dependence of system is given by a scalar function f[var1_,var2_,...]:=f[var1,var2,...]=... that takes the same arguments as system. In a nutshell, I have a function that solves a system of coupled ODEs that depend on system and measures some quantity (e.g. take the trace Tr) of the particular solution after a certain time. The process of evolving the system and measuring is wrapped up in a function measure[var1_,var2_,...]:=Module[...] that essentially also depends on the same variables as system.

Now, evaluating

result1=ParallelTable[measure[var1,var2],{var1,v1min,v1max},{var2,v2min,v2max}];
result2=Table[measure[var1,var2],{var1,v1min,v1max},{var2,v2min,v2max}];


gives moreorless the same results - with exceptions for some values of var1,var2. Evaluating the ParallelTable with same parameters a couple of times reveals that these exceptions "change position" (with respect to var1,var2) so that I conclude it must be side-effects. However, since I do not touch any global variables and all local variables are defined within Module or Block (it doesn't matter which one I pick), I do have no clue where the side-effects originate from.

I thought that memoization of system and f might be an issue, but since they are never called twice with the same set of var1,var2 during a (Parallel)Table it appears to me that this should also not be an issue.

Because this is a problem that may often appear in general, I would like to know if

1. someone can spot a possible origin of side-effects in my brief sketch of the code above
2. someone can tell me how these issues can be "debugged" in general? I believe there must be a way to figure out which part of the code causes trouble
3. it might be related to the fact that measure involves functions like MatrixExp that automatically use multiple cores and it could lead to some conflicts with all local kernels since each kernel might want to distribute over the cores...

Update It appears to me that this is a problem specific to my Mathematica 10.0.2 on Linux 64bit at work (8cores). I wanted to replicate the behavior at home (Windows 8 64bit, Mathematica 10.0.1, 4 cores) - without success... So I will most likely update this question with code on Monday, when I am at work again.

Update 2 I believe that I found out the issue, and I would like to know if you think that makes sense. Actually, as mentioned, none of my functions should cause side-effects. I just noticed, that my Mathematica kernels do not quit reliably. That means, a Quit[] sometimes leaves a WolframKernel and maybe followers open in top. These left-overs also exist if I completly close Mathematica via the GUI, so that no Help/notebooks/whatsoever are opened. Obviously, this is not intended and I noticed that after closing all those left-overs there seem to be no side-effects. This somehow seems to solve my issue for this particular case but introduces even more problems because I do not want to investigate all running processes before I use Mathematica and clear manually - in fact, I would have to close everything, investigate ps -A | grep "WolframKernel\|mathematica\|MathKernel" and kill processes everytime I change some functions to be sure there is no unwanted crosstalk between the actual kernel and some "zombies". I know, probably there should be no connection to those left-over kernels, but apparently I observe no issues if they are killed.

• It is probably impossible to say much of use without having full code needed to replicate the issue. – Daniel Lichtblau Feb 19 '16 at 15:38
• @DanielLichtblau Thanks! I already thought so and will try to minimize the code as much as possible. After that I will of course post it here – Lukas Feb 19 '16 at 16:16
• @DanielLichtblau Please see my latest update. The issue seems to be related to kernels that do not close properly, rather than true side-effects in my code. – Lukas Feb 22 '16 at 10:58
• see also: mathematica.stackexchange.com/questions/104328/… and mathematica.stackexchange.com/questions/104310/…. I still do not have a solution ... – mrz Feb 22 '16 at 11:46
• Certainly this might be a problem of kernels not closing properly. Regardless, we really need full code to replicate in order to say much. – Daniel Lichtblau Feb 22 '16 at 16:52