# How to run mathlink external commands in parallel?

I have an external function called increment (for the sake of illustration) that I call from Mathematica using MathLink. I.e.,

Install[<path-to-mathlink-executable>]


which is defined as

?increment


It works fine when I call it like this:

result = Map[increment[#]&, N@Range[10]]

{2., 3., 4., 5., 6., 7., 8., 9., 10., 11.}


When I try to do this in parallel using a ParallelMap[],

ParallelMap[increment[#] &, N@Range[10]]


I get ten $Failed results, and messages like LinkObject::linkn: Argument LinkObject[<path-to-mathlink-executable>,10,10] in LinkWrite[LinkObject[<path-to-mathlink-executable>,10,10], CallPacket[0,{1.}]] has an invalid LinkObject number; the link may be closed.  But running the non-parallel version (Map[]) again works correctly, proving that the link is not closed. This is on MacOSX Lion, with two CPUs, running Mathematica 8. Is there a way to run an external command, set up by MathLink, in parallel? - Did you make any progress on this? Though I use LibraryLink now, which I find superior for simple applications, I am interested too. – Szabolcs Jan 24 '12 at 15:40 @Szabolcs Thanks for the question - Mma no longer complains about closed links (I had to run LaunchKernels[]), but the processing is seemingly not parallelized as it takes as long to do the calculations in parallel as it does on a single processor. I may edit the question to follow up on this, or ask a separate question. – JxB Jan 24 '12 at 20:02 Did you try the approach described in my answer? As I said I suspect it may have an issue, but in my quick test it worked fine and I managed to run several calculations in parallel. Actually I wanted to know if that issue I mentioned came up. Maybe you are connecting to the same executable from all parallel kernels (if that's possible at all..)? – Szabolcs Jan 24 '12 at 20:19 I'm wondering if it is connecting to the same executable from all kernels. Actually, my external executable is linked against an external library (gel) which itself might be shared amongst the kernels...Maybe I have to parallelize the external function? – JxB Jan 24 '12 at 20:33 @Szabolcs Had to run ParallelNeeds[] (in addition to Needs[]; see docs) to get this to work, since my code uses stuff from other packages. Now it is FAST! I added a note about ParallelNeeds[] in your answer, if you don't mind. But I did indeed follow all else from your answer, so will accept that one. – JxB Jan 24 '12 at 21:29 ## 1 Answer First I'd like to mention that I am currently using LibraryLink, and that LibraryFunctions can be shared with parallel kernels with DistributeDefinitions. When using Parallelize, this happens automatically. (I'm assuming that you have local parallel kernels. For remote ones, see here.) For MathLink programs, this will not work. Let's try it: First, let's launch the kernels: In[6]:= LaunchKernels[]  Then install the addtwo example into the main kernel: In[7]:= SetDirectory[$InstallationDirectory <>
"/SystemFiles/Links/MathLink/DeveloperKit/" <> $SystemID <> "/PrebuiltExamples/"]; In[8]:= link = Install["addtwo"] Out[8]= LinkObject["addtwo", 23, 11]  It will not work in parallel kernels even after distribution the definitions because the connection exists only between addtwo and the main kernel: In[9]:= DistributeDefinitions[AddTwo] Out[9]= AddTwo In[10]:= ParallelEvaluate[Print@AddTwo[2, 2]] (kernel 2) LinkObject::linkn: Argument LinkObject[addtwo,23,11] in LinkWrite[LinkObject[addtwo,23,11],CallPacket[0,{2,2}]] has an invalid LinkObject number; the link may be closed. (kernel 1) LinkObject::linkn: Argument LinkObject[addtwo,23,11] in LinkWrite[LinkObject[addtwo,23,11],CallPacket[0,{2,2}]] has an invalid LinkObject number; the link may be closed. (kernel 2)$Failed

(kernel 1) \$Failed

Out[10]= {Null, Null}


Now lets install it into each parallel kernel:

In[11]:= ParallelEvaluate[Install["addtwo"]]

(kernel 1) 4

(kernel 2) 4

Out[12]= {Null, Null}


As you can see, now it works in each of them, but the addtwo process will also exist in three copies (on a two-core machine). Note: you will have to run ParallelNeeds[] if you load any packages required by the sub kernels.

I must note that unlike with LibraryLink, I have no practical experience with this.

Caveat: This problem didn't come up in my tests, but I think it may be possible that the symbol AddTwo will get auto-synchronized between the kernels, so they might lose the connection to their own addtwo process. This could be avoided by placing the AddTwo symbol into a different context than Global ` (see here and here), but unfortunately you won't have control over this unless you are writing your MathLink programs yourself.

-
Oops, I must have missed that example, sorry. Removing my comment, and +1. –  Leonid Shifrin Jan 18 '12 at 0:54