Can a package append its context to $DistributedContexts?

Question

I am writing a package (for reference, this one) which I use extensively in parallel computations, and to lighten the coding load on the 'run-time' notebook, I want my package to sit on the $DistributedContexts list of sessions in which it is called.

The way I'm doing it now is to simply call

$DistributedContexts := {$Context, "Test`"}

at the end of the package, and this mostly works. However, it has a definite disadvantage in that if the user has already put something in that variable, calling Needs["Test`"] will overwrite that previous value, without even telling the user, and that's not completely a good thing. (It's not terrible, either, but I'd rather it didn't happen.)

So far, what I've done is include the following message,

$DistributedContexts::overwrite = "Warning: overwriting previous value of
        $DistributedContexts. Reinstate your old definition, and include the 
        Test context to ensure proper parallelization of Test calculations.";
If[
  ValueQ[$DistributedContexts],
  Message[$DistributedContexts::overwrite]
]

which produces a warning when Needs["Test`"] is called and the variable already had something in there. This draws, already, from pretty tricky territory, as $DistributedContexts is not an easy variable to handle. For instance, on a clean kernel

Quit
ValueQ[$DistributedContexts]

will return False, and indeed calling

$DistributedContexts

will return unevaluated, even though it clearly has a "value" in the sense that it is the default value of the DistributedContexts option of Parallelize and (at least some) related functions, and it is by default equal to $Context.

---

What I want, to be more specific, is a way to append the context name "Test`" on to the value of $DistributedContexts if it already has one. So, for example, I want

$DistributedContexts:={$Context, "OtherContext`"}
Needs["Test`"]

to be equivalent to

$DistributedContexts:={$Context, "OtherContext`", "Test`"}

This turns out to be very tricky!

• Calling AppendTo[$DistributedContexts, "Test`"] doesn't work, and returns the error message

   AppendTo::rvalue: $DistributedContexts is not a variable with a value, so its
   value cannot be changed. 
  

• Calling $DistributedContexts := {$DistributedContexts, "Test`"} seems to work - it just swallows it up without complaint - but then calling, say, ParallelTable[Pause[0.1], {j, 100}], returns the error message

  $RecursionLimit::reclim2: Recursion depth of 1024 exceeded during evaluation 
   of {$DistributedContexts,Test`}. 
  

  because, of course, the initial call was a SetDelayed which did not evaluate, but the parallelized evaluation tried to ask for the value of $DistributedContexts which is of course recursively ill-defined.

• If $DistributedContexts already has a value, and particularly if it's been defined sensibly as something like $DistributedContexts:={$Context, "OtherContext`"}, it's very hard to actually get that value, because if you do that and then try to ask for the value, you will get different answers depending on where you are on the context tree,

  Begin["Context1`"];
  $DistributedContexts
  End[];
  
  Begin["Context2`"];
  $DistributedContexts
  End[];
  
  (*
  {"Context1`", "OtherContext`"}
  {"Context2`", "OtherContext`"}
  *)
  

  and it certainly won't return a plain $Context willingly on its own.

So: is there a way to produce the behaviour I want?

Answer 1

OP wrote:

I want to be able to simply call Needs["Test`"] and have functions such as Test`f available for use inside ParallelTable without any further calling of $DistributedContexts or DistributeDefinitions

We had a similar problem designing mathStatica functions to operate in a parallel environment. There can be varying degrees of complexity. For example, we wanted:

mathStaticaFunction1 to call say 6 parallel copies of mathStaticaFunction2

which means that all functions have to be available on all kernels (can get a bit messy when mathStaticaFunction2 also wants to calls things in parallel!).

Anyway, the way we resolved the problem was for the master package to launch itself in parallel.

One way to do this is:

 ParallelNeeds["mathStatica`mathStaticaCode`"]

But this did not work for us, mostly due to recursion issues with slave kernels calling the parent. But it might work just fine for you, just using the simple structure:

  ParallelNeeds["Test`"]

It might be as simple as that!

What we use is a slightly different:

We have a master file:

• mathStatica.m - which just defines the context (see below)

and

• mathStaticaCode.m - which contains the real code.

The master mathStatica.m file has a simple 3-line structure like:

  If[ MemberQ[$ContextPath, "mathStatica`"]  === False, 
      BeginPackage["mathStatica`"];
      Get["mathStatica`mathStaticaCode`"];
      EndPackage[]
    ]

and the line:

 If[$ProcessorCount > 1,  mathStaticaParallel[On]];  

The mathStaticaCode.m package contains all the actual code, including the mathStaticaParallel function:

 mathStaticaParallel[Q_] := Module[{qqq}, 
    Which[Q === On, 
           If[$ProcessorCount > 1,    Quiet[LaunchKernels[],  {LaunchKernels::nodef}]; 

            ParallelEvaluate[   If[ MemberQ[$ContextPath, "mathStatica`"]  =!= True, 
                                      BeginPackage["mathStatica`"]; 
                                      Get["mathStatica`mathStaticaCode`"]; 
                                      EndPackage[]  ]
                                ];  

     Q === Off, 
        blahBlah
    ] ]

The ParallelEvaluate manually creates the mathStatica CONTEXT on each kernel, and then loads the mathStaticaCODE inside it. You may not need the separate code file and master file, but that's the way we do it, for various reasons.

So, any time anyone starts it up, it automatically works on all available kernels ... the number of which are controlled under the Menu:

Evaluation -> Parallel Kernel Configuration ...

Hope this helps