Package found with Needs, but not with ParallelNeeds

I want to use a self-written package in a parallel computation. However while Needs["mypackage"] works without problems, ParallelNeeds["mypackage"] can't open my package.

The immediate reason is related to the fact that I added to $Path, and the package is in one of the directories I've added. ParallelTable[$Path,{1}] reveals that this addition is not propagated to the subkernels, despite the fact that it is changed in the user kernel init file ($UserBaseDirectory/Kernel/init.m), and thus long before any subkernels are launched in the Notebook. Explicitly using DistributeDefinitions[$Path] didn't help either.

So my question is: Why isn't the $Path propagated to the kernels, and more importantly, what can I do about it? • It seems symbols in the System context aren't distributed at all – Rojo Oct 5 '12 at 11:07 • If case you used my AddPath solution, I updated the answer with an easier way to deal with parallel kernels and argument completion. – Szabolcs Nov 27 '16 at 21:31 3 Answers I've now found the solution. The reason that the path is not available is that it is set in an init file, and subkernels do not execute any init files. However, as it turns out, the reason why they do not execute any init files is that they are explicitly passed the option -noinit when started (you can easily see that by looking at the process listing on your OS; e.g. on Linux, ps x reveals the command line if the window is wide enough). How to change the options on start does not seem to be documented, however I've found out that in the package SubKernels there exists a variable SubKernelsProtectedstdargs which contains a string with the (non-mathlink) arguments passed to subkernels. By default it has the value " -subkernel -noinit". Therefore executing the following before launching the subkernels causes them to execute the standard initialization file: Needs[SubKernels] SubKernelsProtectedstdargs = " -subkernel"  Obviously this can also be used to pass other options to the subkernels (an obvious candidate would be -initfile to have separate initialization for main and sub kernels, however I haven't tested that yet). • I think the reasoning behind all that might be that the parallel kernels don't necessarily run on the same machine and don't necessarily see the same file system -- thus every kernel could in such a szenario read a different init file -- not a very reliable environment. For just getting your parallel job up and running I think what you do is perfectly good, but if you give away code or want to run on a cluster letting the kernels read an init file might not always be a good idea. The same is of course true for package files - in general you'd have to ensure there are available to every kernel – Albert Retey Oct 5 '12 at 20:30 • @AlbertRetey: Actually where I'm working, all computers mount the very same file system at the very same position (with the exception of local scratch space which, being local, is of course different on all computers). And that is how it should be. Anyway, the whole ParallelNeeds concept already presumes that the same package is available on all computers. Also, if there are different file systems on the different computers, then there would also be different init files which then would also set up the different paths correctly for those different environments. – celtschk Oct 6 '12 at 8:10 • sorry, my above comment was mixing two different things: What I actually had in mind is what the reasoning to not distribute $Path from the master to the slave kernels by default might be. I think that was your original question and from your comment I think we agree that automatically distributing $Path would in general have the potential to break such a setup... – Albert Retey Oct 6 '12 at 9:35 • ... The -noinit options is a slightly different story, but I can well understand why WRI decided to have it set for the slave kernels: I hardly have seen personal init files that would have worked for a heterogenous environment, so that I'd expect a lot of error messages on kernel startup for many users launching kernels in such an environment with initialization switched on. Other reasons might be performance considerations and to ensure a uniform minimal setup among all slave kernels. – Albert Retey Oct 6 '12 at 9:40 • If you use Autoload packages, they will load even with -noinit. mathematica.stackexchange.com/q/114957/12 – Szabolcs Nov 23 '16 at 9:21 You can just add you package path to the path on each of the kernels ParallelEvaluate[AppendTo[$Path, ToFileName[{$HomeDirectory, "MyPackages"}]]]  now running ParallelTable[$Path, {1}]


will show that the change has been propagated.

Besides using an init.m file, there is another way to run code on each kernel after start-up (full credit goes to this excellent answer by Oleksandr R.)

Needs["ParallelDeveloper"]; (* Load required package *)
$InitCode = Hold[AppendTo[$Path, ToFileName[{$HomeDirectory, "MyPackages"}]]];  • Thank you, that solved my immediate problem (actually it's simple enough that I should have gotten that idea myself). However, I still wonder why this is necessary at all, and how to solve it in general (I'd prefer not to have to do that every time I do a parallel evaluation; OTOH I don't want to unnecessarily start parallel kernels from my init.m). – celtschk Oct 5 '12 at 10:58 • @celtschk Perhaps the subkernels have their own init.m file somewhere. or can be configured to read another file using -initfile xxx. But quickly browsing the docs, I have not found anything useful yet. – Ajasja Oct 5 '12 at 11:56 • Thanks for this comment. This brought me on the right track. I don't know if this is documented somewhere, but I've now found out that after Needs["SubKernels"] there is a variable SubKernelsProtectedstdargs with value " -subkernel -noinit". Setting it to just " -subkernel" (before launching the subkernels) causes the subkernels to load the init file. – celtschk Oct 5 '12 at 12:18 • @celtschk that could be in a self-answer, I'd upvote it – Rojo Oct 5 '12 at 13:03 • @Rojo: Thanks, done. – celtschk Oct 5 '12 at 13:31 I use the following solution for this problem: • I have a package called AddPath which can add pre-defined directories to the $Path. This package is placed in $UserBaseDirectory/Autoload, so it gets loaded on parallel kernels as well. • I put another very basic package in Autoload which simply contains Needs["AddPath"] AddPath[{"Pack1","Pack2",...}]  Then I always have these packages available in the $Path both on the main kernel and subkernels.

I also use a ParallelAddPath function which registers packages to be added to the $Path on newly started parallel kernels. In this way it is similar to ParallelNeeds. Implementation details: Finally, I set up argument-completion for AddPath to make it easier to type. Implementation notes: I realize that this is overkill, and you could just create a file Autoload/MyInit/init.m which contains AppendTo[$Path, ...]. But I have been using this AddPath setup for quite a while and it works well for me, so I thought I would share it. I also think it is relevant for a question about changing $Path. Note: I only started using the Autoload part today though, before that I used the standard init.m to load AddPath. I originally came up with AddPath for version management (I had multiple versions of packages I was developing) and dependency management. I looks roughly like below. BeginPackage["AddPath"] AddPath::usage = "AddPath[\"package\"] will add \"package\" to the$Path."
RemovePath::usage = "RemovePath[\"package\"] will remove \"package\" to the $Path." ParallelAddPath::usage = "ParallelAddPath[\"package\"] will register \"package\" to be added to the$Path on parallel kernels.";

Begin["Private"]

Get["AddPathConfig"];

AppendTo[packageList, _ -> Null];
AppendTo[dependencies, _ -> {}];

AddPath::path = "The package  is already in the $Path."; AddPath::pack = " is not a known package."; SetAttributes[AddPath, Listable] AddPath[pack_String] := Module[{deps, path}, Quiet@AddPath@Replace[pack, dependencies]; path = Replace[pack, packageList]; If[path === Null, Message[AddPath::pack, pack]; Return[] ]; If[MemberQ[$Path, path],
Return[]
];
AppendTo[$Path, path]; ] RemovePath::path = "The package  is not in the$Path.";

(* RemovePath will not check for equivalent path names.
~/dir and /Users/someone/dir are considered different *)

SetAttributes[RemovePath, Listable]
RemovePath[pack_String] :=
Module[{path},
path = Replace[pack, packageList];
If[path === Null,
Message[RemovePath::pack, pack];
Return[]
];
If[Not@MemberQ[$Path, path], Message[RemovePath::path, pack]; Return[] ];$Path = DeleteCases[$Path, path]; ] ParallelAddPath[arg : (_String|{___String})] := (Parallelize; ParallelNeeds["AddPath"]; ParallelProtectedAddInitCode[ ParallelClientHoldCompound[AddPath[arg]] ];) (* Set up argument completion. *) addCompletion = FEEvaluate[FEPrivateAddSpecialArgCompletion[#]] &; addCompletion[# -> {Most[packageList][[All, 1]]}]& /@ {"AddPath", "RemovePath", "ParallelAddPath"}; End[] (* Private *) EndPackage[]  I manually edit packageList and dependencies in Config.m whenever I need to. These define the package locations (usually Git repos) and their dependencies. packageList = { "Spelunking" -> "~/Repos/Spelunking", "SciDraw" -> "~/Documents/Mathematica/SciDraw-0.0.7/packages", ... "MATLink10" -> "~/Documents/Mathematica/MATLink10", (* example of multiple versions of the same package *) "MATLink11" -> "~/Documents/Mathematica/MATLink11", "MATLink1Dev" -> "~/Repos/MATLink1", ... "LTemplate" -> "~/Repos/LTemplate", "IGraphM" -> "~/Repos/IGraphM", ... _ -> Null }; dependencies = { ... "IGraphM" -> "LTemplate", ... _ -> {} };  • I'd be tempted to elevate packagList and dependencies to package level variables, and then set them in FileNameJoin[{$UserBaseDirectory, "Kernel", "init.m"}] with _ -> Null appended internally. Additionally, I think I would merge the two and use "package" -> {path, dependency ..} or "package" -> {path, "Dependencies" -> {dependency ..}}, instead. But, that's just me. – rcollyer May 12 '16 at 13:41
• @rcollyer You're right, that makes complete sense (except for the part of setting them in init.m, which won't be read by parallel kernels). In fact I just moved those out into a separate file. The reason why this feels half-done is that it is half done. I used it for my own purposes, and it was good enough. I only changed it when something got annoying enough to get me to do something about it. Take it as an idea rather than a ready-to-use implementation. Overall it did work rather well. – Szabolcs May 12 '16 at 13:45
• I hadn't realized init.m was only picked up by the master kernel. Overall, I meant them as suggestions. Most of my ideas are half-baked, and only implemented as far as I need them at any given moment (or so over-engineered that changing them when they need updating is "interesting"), so I understand, it is a work in progress. – rcollyer May 12 '16 at 14:22
• @rcollyer I completely agree with your suggestions, and updated the post a bit. In fact I did something like this on my machine just minutes before I read your comment. – Szabolcs May 12 '16 at 14:44
• this is only strictly necessary for versions > 11.2, right? Before that, all my custom packages were perfectly working in parallel evaluations, by simply using ParallelNeeds["package"] and ParallelEvaluate[AppendTo[\$Path, path]]; – Santi Mar 29 '18 at 8:46