0
$\begingroup$

I have a package that redefines a system symbol, and I am struggling to find a clean solution that will ensure that those redefinitions will still work inside a parallelized environment.

The core structure of the package is of the following form:

BeginPackage["Utilities`"];
f::usage = "...";
Begin["`Private`"];
f[x_] := x;
End[];
EndPackage[];
DistributeDefinitions["Utilities`"];

(Shamelessly stolen from here, so hopefully I'm not the only one that does this.) Doing the DistributeDefinitions call in this way has several advantages:

  • it does not attempt to launch new subkernels when they are not necessary
  • it keeps the package definitions in some mysterious place where they are always available for any new subkernels that get launched subsequently, even if e.g. one of them crashes and gets re-launched
  • it does no harm if the package is called from inside a parallelized environment (i.e. it doesn't return errors of the form "you tried to parallelize from inside a subkernel, but this is not Inception and you are not Leonardo di Caprio")

Now, I want to expand this package to redefine the workings of some system symbol. For the sake of argument, suppose that I want to modify RandomComplex in the following form:

Unprotect[RandomComplex];
RandomComplex[{range1_List, moreRanges___}, number_] :=  Transpose[
                         RandomComplex[#, number] & /@ {range1, moreRanges}]
Protect[RandomComplex];

Unfortunately, even if I do this within the BeginPackage[] and EndPackage[] calls, the redefinition does not get sent to any subkernels that get launched afterwards. Thus, suppose I write this into the code above, load the package, launch some new kernels, and then run

ParallelEvaluate[RandomComplex[{{0, 1 + I}, {0, 1 + I}}, 3]]

I will then get a bunch of error messages, because each subkernel is using the unpatched version.

How can I make sure that this redefinition gets distributed downstream, ideally in a way that preserves the advantages I mentioned above, and with the least amount of pain possible.

(In particular, I can obviously run ParallelEvaluate on the redefinition, but then it is not stable against a subkernel crash-plus-relaunch, and it will kick and scream if I Get or Needs the package from inside a parallelized environment.)

$\endgroup$
  • $\begingroup$ In short: I think you shouldn't distribute System context symbols (or any package's context for that matter). Instead: load the package on each subkernel (ParallelNeeds) and let it redefine the symbol on each. I won't comment on redefining system symbols, I guess you know the dangers ... $\endgroup$ – Szabolcs Dec 8 '16 at 22:06
  • $\begingroup$ @Szabolcs Yes, I know the dangers. I'm unsure of how to use ParallelNeeds while indicating a specific file from which to take the package, though. $\endgroup$ – Emilio Pisanty Dec 8 '16 at 22:08
  • $\begingroup$ My feeling is still that it's better not to distribute, as that's a nasty hack. I would use another hack: use Get and register it to be evaluated on newly launched kernels. mathematica.stackexchange.com/q/131856/12 To be fair, distribution may not be that bad, but it's a complicated thing and I don't like complicated things because they tend to cause surprises. ParallelNeeds, as well as registering init code is simple and predictable. $\endgroup$ – Szabolcs Dec 8 '16 at 22:12
  • $\begingroup$ @Szabolcs I'm confused as to what you're actually proposing. Should I pack a Get call inside some funky init code? Or should I just ask for a ParallelNeeds? The problem with the latter is that it does not allow me to specify the source of the package, so I have no idea how to use it in that situation. $\endgroup$ – Emilio Pisanty Dec 8 '16 at 22:20
  • $\begingroup$ Either install the package properly, and then you can use ParallelNeeds and you don't need to refer to a file. Or use Get (which can refer to a file), and put it in some init code. Simply load the parallel tools (evaluate Parallelize) then set Parallel`Developer`$InitCode = Hold[Get["your/file.m"]] before launching any kernels. If you do not need to care about kernels which will be launched in the future in ways you cannot control, then just do ParallelEvaluate[Get["your/file.m"]]. $\endgroup$ – Szabolcs Dec 8 '16 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.