6
$\begingroup$

I am writing a package (for reference, this one) which I use extensively in parallel computations, and to lighten the coding load on the 'run-time' notebook, I want my package to sit on the $DistributedContexts list of sessions in which it is called.

The way I'm doing it now is to simply call

$DistributedContexts := {$Context, "Test`"}

at the end of the package, and this mostly works. However, it has a definite disadvantage in that if the user has already put something in that variable, calling Needs["Test`"] will overwrite that previous value, without even telling the user, and that's not completely a good thing. (It's not terrible, either, but I'd rather it didn't happen.)

So far, what I've done is include the following message,

$DistributedContexts::overwrite = "Warning: overwriting previous value of
        $DistributedContexts. Reinstate your old definition, and include the 
        Test context to ensure proper parallelization of Test calculations.";
If[
  ValueQ[$DistributedContexts],
  Message[$DistributedContexts::overwrite]
]

which produces a warning when Needs["Test`"] is called and the variable already had something in there. This draws, already, from pretty tricky territory, as $DistributedContexts is not an easy variable to handle. For instance, on a clean kernel

Quit
ValueQ[$DistributedContexts]

will return False, and indeed calling

$DistributedContexts

will return unevaluated, even though it clearly has a "value" in the sense that it is the default value of the DistributedContexts option of Parallelize and (at least some) related functions, and it is by default equal to $Context.


What I want, to be more specific, is a way to append the context name "Test`" on to the value of $DistributedContexts if it already has one. So, for example, I want

$DistributedContexts:={$Context, "OtherContext`"}
Needs["Test`"]

to be equivalent to

$DistributedContexts:={$Context, "OtherContext`", "Test`"}

This turns out to be very tricky!

  • Calling AppendTo[$DistributedContexts, "Test`"] doesn't work, and returns the error message

     AppendTo::rvalue: $DistributedContexts is not a variable with a value, so its
     value cannot be changed. 
    
  • Calling $DistributedContexts := {$DistributedContexts, "Test`"} seems to work - it just swallows it up without complaint - but then calling, say, ParallelTable[Pause[0.1], {j, 100}], returns the error message

    $RecursionLimit::reclim2: Recursion depth of 1024 exceeded during evaluation 
     of {$DistributedContexts,Test`}. 
    

    because, of course, the initial call was a SetDelayed which did not evaluate, but the parallelized evaluation tried to ask for the value of $DistributedContexts which is of course recursively ill-defined.

  • If $DistributedContexts already has a value, and particularly if it's been defined sensibly as something like $DistributedContexts:={$Context, "OtherContext`"}, it's very hard to actually get that value, because if you do that and then try to ask for the value, you will get different answers depending on where you are on the context tree,

    Begin["Context1`"];
    $DistributedContexts
    End[];
    
    Begin["Context2`"];
    $DistributedContexts
    End[];
    
    (*
    {"Context1`", "OtherContext`"}
    {"Context2`", "OtherContext`"}
    *)
    

    and it certainly won't return a plain $Context willingly on its own.

So: is there a way to produce the behaviour I want?

$\endgroup$
  • 3
    $\begingroup$ I would also be interested in how the code in this question got scraped into here within ten minutes of posting. $\endgroup$ – Emilio Pisanty Apr 15 '16 at 18:52
  • $\begingroup$ Could you please explain what your real objective is? i.e. not the specifics of worrying about $DistributedContexts, but why you are trying to do this? i.e. Is your intention to allow a package function MyPackFunc1 (that may be operating in parallel) to call other package functions say MyPackFunc2, so that the other kernels are aware of the package functions? Or do you just want to update variables SetSharedVariable between the different kernels? $\endgroup$ – wolfies Apr 24 '16 at 17:42
  • $\begingroup$ @wolfies I want to be able to simply call Needs["Test`"] and have functions such as Test`f available for use inside ParallelTable without any further calling of $DistributedContexts or DistributeDefinitions. $\endgroup$ – Emilio Pisanty Apr 24 '16 at 17:55
  • $\begingroup$ I guess that that is also doable by calling DistributeDefinitions["Test`"], though I'm not sure I like the relation of that with launching extra kernels in an environment that may or may not end up using parallelized environments. More generally, though, this is surely something that's doable via the $DistributedContexts route, and I'm mostly interested in how you'd circumvent the difficulties I outlined. $\endgroup$ – Emilio Pisanty Apr 24 '16 at 18:03
3
+250
$\begingroup$

OP wrote:

I want to be able to simply call Needs["Test`"] and have functions such as Test`f available for use inside ParallelTable without any further calling of $DistributedContexts or DistributeDefinitions

We had a similar problem designing mathStatica functions to operate in a parallel environment. There can be varying degrees of complexity. For example, we wanted:

mathStaticaFunction1 to call say 6 parallel copies of mathStaticaFunction2

which means that all functions have to be available on all kernels (can get a bit messy when mathStaticaFunction2 also wants to calls things in parallel!).

Anyway, the way we resolved the problem was for the master package to launch itself in parallel.

One way to do this is:

 ParallelNeeds["mathStatica`mathStaticaCode`"]

But this did not work for us, mostly due to recursion issues with slave kernels calling the parent. But it might work just fine for you, just using the simple structure:

  ParallelNeeds["Test`"]

It might be as simple as that!

What we use is a slightly different:

We have a master file:

  • mathStatica.m - which just defines the context (see below)

and

  • mathStaticaCode.m - which contains the real code.

The master mathStatica.m file has a simple 3-line structure like:

  If[ MemberQ[$ContextPath, "mathStatica`"]  === False, 
      BeginPackage["mathStatica`"];
      Get["mathStatica`mathStaticaCode`"];
      EndPackage[]
    ]

and the line:

 If[$ProcessorCount > 1,  mathStaticaParallel[On]];  

The mathStaticaCode.m package contains all the actual code, including the mathStaticaParallel function:

 mathStaticaParallel[Q_] := Module[{qqq}, 
    Which[Q === On, 
           If[$ProcessorCount > 1,    Quiet[LaunchKernels[],  {LaunchKernels::nodef}]; 

            ParallelEvaluate[   If[ MemberQ[$ContextPath, "mathStatica`"]  =!= True, 
                                      BeginPackage["mathStatica`"]; 
                                      Get["mathStatica`mathStaticaCode`"]; 
                                      EndPackage[]  ]
                                ];  

     Q === Off, 
        blahBlah
    ] ]

The ParallelEvaluate manually creates the mathStatica CONTEXT on each kernel, and then loads the mathStaticaCODE inside it. You may not need the separate code file and master file, but that's the way we do it, for various reasons.

So, any time anyone starts it up, it automatically works on all available kernels ... the number of which are controlled under the Menu:

Evaluation -> Parallel Kernel Configuration ...

Hope this helps

$\endgroup$
  • $\begingroup$ Thanks for this, though it feels like overkill for my purposes. I'm wary of ParallelNeeds, though, particularly for situations where the user might normally call Needs["Test`", FileNameJoin[{ NotebookDirectory[], "Test.m"}]], which has no ParallelNeeds equivalent. (That can be circumvented with e.g. With[{directory=NotebookDirectory[]} and a ParallelEvaluate[ Needs[]] call, but then the effect is not persistent if the kernels are closed and respawned.) Hopefully a simpler alternative will come up. $\endgroup$ – Emilio Pisanty Apr 24 '16 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.