Bug in parallel computing

Question

The following is a simplified example. We construct a library function f and another library function g, in which the library function f is called.

f = Compile[x, Integrate[1 + t^2, {t, 0, x}], CompilationTarget -> "C"];
g = With[{f = f}, Compile[x, x + f[x], CompilationTarget -> "C"]];

CompilePrint shows the following for the function g:

1 R1 = LibraryFunction[<>, compiledFunction0, {{Real, 0, Constant}}, Real][ R0]]
2 R2 = R0 + R1
3 Return

Now we test the functions f and g on the main kernel and on two subkernels:

LaunchKernels[];
DistributeDefinitions[f,g];
f[2.3]
g[1.2]
ParallelEvaluate[f[2.3], 1]
ParallelEvaluate[g[1.2], 1]
ParallelEvaluate[g[1.2], 2]

(* Out[7]= 6.35567  
  Out[8]= 2.976  
  Out[9]= 6.35567  
  Out[10]= 2.976  
  During evaluation of In[5]:= LinkObject::linkd: Unable to communicate with closed link LinkObject["C:\Program Files\Wolfram Research\Mathematica\10.0\MathKernel" -subkernel -noinit -mathlink -noicon,225,5]. >>  
  During evaluation of In[5]:= KernelObject::rdead: Subkernel connected through KernelObject[2,local] appears dead. >>  
  Out[11]= $Failed  *)

The evaluations on the main kernel and on subkernel 1 are fast. On subkernel 1 we first evaluated f and next g. On subkernel 2 we immediately called g, without having used f before. The subkernel crashes. I am running Mathematica 10.0.2 32 bits on Windows 7.

Answer 1

As demonstrated by the asker, Mathematica is failing to distribute the definitions of C-compiled library functions in a way that can be called by other C-compiled functions.

I agree this is a bug. I have submitted a bug report to Wolfram Research and hope to update the post to include the result of that.

1. One workaround is to compile to C only in the last function. You can use

CompilationOptions->{"InlineCompiledFunctions"->True}

to ensure it all goes to optimized C-code. (if you remove the CompilationTarget->"C" it compiles to the Wolfram Virtual Machine.)

2. Another option is to first call all of the child functions from the parallel Kernels, which appear to get loaded properly for use. For your example

LaunchKernels[];
f = Compile[x, Integrate[1 + t^2, {t, 0, x}], CompilationTarget -> "C"];
g = With[{f = f}, Compile[x, x + f[x], CompilationTarget -> "C"]];
ParallelEvaluate[f];
ParallelEvaluate[g[1.2], 2]

{2.976, 2.976}

Although a question wasn't asked, I hope one this solves the askers problems, as it solved mine.

Update 7/21/2016: I received an email containing the following.

I was able to reproduce the issue, and consequently, I filed a report with our development team raising the issues and also shared your contact information with them so you can be notified once the issue is resolved.

Update 8/26/2016: The situation appears the same in version 11.0.0.0.