Tell me more ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.
up vote 9 down vote favorite
1

I want to use a self-written package in a parallel computation. However while Needs["mypackage`"] works without problems, ParallelNeeds["mypackage`"] can't open my package.

The immediate reason is related to the fact that I added to $Path, and the package is in one of the directories I've added. ParallelTable[$Path,{1}] reveals that this addition is not propagated to the subkernels, despite the fact that it is changed in the user kernel init file ($UserBaseDirectory/Kernel/init.m), and thus long before any subkernels are launched in the Notebook. Explicitly using DistributeDefinitions[$Path] didn't help either.

So my question is: Why isn't the $Path propagated to the kernels, and more importantly, what can I do about it?

share|improve this question
1  
It seems symbols in the `System`` context aren't distributed at all – Rojo Oct 5 '12 at 11:07

2 Answers

up vote 4 down vote

I've now found the solution.

The reason that the path is not available is that it is set in an init file, and subkernels do not execute any init files. However, as it turns out, the reason why they do not execute any init files is that they are explicitly passed the option -noinit when started (you can easily see that by looking at the process listing on your OS; e.g. on Linux, ps x reveals the command line if the window is wide enough).

How to change the options on start does not seem to be documented, however I've found out that in the package SubKernels` there exists a variable SubKernels`Protected`stdargs which contains a string with the (non-mathlink) arguments passed to subkernels. By default it has the value " -subkernel -noinit". Therefore executing the following before launching the subkernels causes them to execute the standard initialization file:

Needs[SubKernels`]
SubKernels`Protected`stdargs = " -subkernel"

Obviously this can also be used to pass other options to the subkernels (an obvious candidate would be -initfile to have separate initialization for main and sub kernels, however I haven't tested that yet).

share|improve this answer
1  
I think the reasoning behind all that might be that the parallel kernels don't necessarily run on the same machine and don't necessarily see the same file system -- thus every kernel could in such a szenario read a different init file -- not a very reliable environment. For just getting your parallel job up and running I think what you do is perfectly good, but if you give away code or want to run on a cluster letting the kernels read an init file might not always be a good idea. The same is of course true for package files - in general you'd have to ensure there are available to every kernel – Albert Retey Oct 5 '12 at 20:30
@AlbertRetey: Actually where I'm working, all computers mount the very same file system at the very same position (with the exception of local scratch space which, being local, is of course different on all computers). And that is how it should be. Anyway, the whole ParallelNeeds concept already presumes that the same package is available on all computers. Also, if there are different file systems on the different computers, then there would also be different init files which then would also set up the different paths correctly for those different environments. – celtschk Oct 6 '12 at 8:10
sorry, my above comment was mixing two different things: What I actually had in mind is what the reasoning to not distribute $Path from the master to the slave kernels by default might be. I think that was your original question and from your comment I think we agree that automatically distributing $Path would in general have the potential to break such a setup... – Albert Retey Oct 6 '12 at 9:35
... The -noinit options is a slightly different story, but I can well understand why WRI decided to have it set for the slave kernels: I hardly have seen personal init files that would have worked for a heterogenous environment, so that I'd expect a lot of error messages on kernel startup for many users launching kernels in such an environment with initialization switched on. Other reasons might be performance considerations and to ensure a uniform minimal setup among all slave kernels. – Albert Retey Oct 6 '12 at 9:40
up vote 4 down vote

You can just add you package path to the path on each of the kernels

ParallelEvaluate[AppendTo[$Path, ToFileName[{$HomeDirectory, "MyPackages"}]]]

now running 

ParallelTable[$Path, {1}]

will show that the change has been propagated.

Besides using an init.m file, there is another way to run code on each kernel after start-up (full credit goes to this excellent answer by Oleksandr R.)

Needs["Parallel`Developer`"]; (* Load required package *)
$InitCode = Hold[AppendTo[$Path, ToFileName[{$HomeDirectory, "MyPackages"}]]];
share|improve this answer
Thank you, that solved my immediate problem (actually it's simple enough that I should have gotten that idea myself). However, I still wonder why this is necessary at all, and how to solve it in general (I'd prefer not to have to do that every time I do a parallel evaluation; OTOH I don't want to unnecessarily start parallel kernels from my init.m). – celtschk Oct 5 '12 at 10:58
1  
@celtschk Perhaps the subkernels have their own init.m file somewhere. or can be configured to read another file using -initfile xxx. But quickly browsing the docs, I have not found anything useful yet. – Ajasja Oct 5 '12 at 11:56
Thanks for this comment. This brought me on the right track. I don't know if this is documented somewhere, but I've now found out that after Needs["SubKernels`"] there is a variable SubKernels`Protected`stdargs with value " -subkernel -noinit". Setting it to just " -subkernel" (before launching the subkernels) causes the subkernels to load the init file. – celtschk Oct 5 '12 at 12:18
@celtschk that could be in a self-answer, I'd upvote it – Rojo Oct 5 '12 at 13:03
@Rojo: Thanks, done. – celtschk Oct 5 '12 at 13:31
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.