13
$\begingroup$

I've reduced my actual problem to:

ParallelEvaluate[
  Module[{slot=1},
    Slot[slot]
  ]
]

gives error messages that I think shouldn't be there. It seems I can work around them with 

ParallelEvaluate[
  Module[{slot=1},
    Slot@@{slot}
  ]
]

and

ParallelEvaluate[
  Module[{slot=1,s=Slot},
    s[slot]
  ]
]

but I still think this is a bug in the parallel framework. Am I missing something obvious? It seems the numeric check happens with the local Module placeholder values passed to the parallelism framework in a syntactic way, not in a semantic way. But that's wrong, I should be able to pass Slot[anything]

v8.0.4 and v9 alike, Windows and Linux.

| improve this question | |
$\endgroup$
  • $\begingroup$ It happens on OS X as well (8 & 9) and even if you replace Module with With $\endgroup$ – Szabolcs Jan 13 '13 at 19:24
  • $\begingroup$ These things happen when the expression is put in Function: Function@Module[{s = 1}, Slot[s]]. Perhaps constructing a function is part of the parallel distribution/retrieval process. $\endgroup$ – Szabolcs Jan 13 '13 at 19:40
13
$\begingroup$

This message is issued by Function itself. To see this, try

Function[Module[{slot = 1}, Slot[slot]]]

If Function has named formal parameters, the message goes away:

Function[x, Module[{slot = 1}, Slot[slot]]]

So to fix this first we need to find out where is the argument passed to ParallelEvaluate wrapped by Function. Fortunately the parallel tools are defined in a plain .m (not .mx) file so we can read the source code. The relevant part turns out to be in $InstallationDirectory/AddOns/Applications/Parallel/Parallel.m, in the definition of Send:

Send[kernels:{___kernel}, expr_] := Send[#, expr]& /@ kernels

The part causing the problem is Send[#, expr]&. Looks innocent enough, doesn't it? If we change this to Function[k, Send[k, expr]] then the message goes away.

Warning: I do not know if the form of Function with named formal parameters can cause any trouble (e.g. name collisions) in certain edge cases so be careful (or ask Leonid) ... Update: Here's an example of what may go wrong with named formal parameters.

Perhaps you can report this problem to WRI and let us know what they said.

Now that we know what the problem is caused by, we can manually construct simple problem cases. For example,

Mathematica graphics

Generally, passing any Slot not wrapped in a Function without formal arguments will cause problems (messages and unexpected results). I would consider this a bug, so I think it's a good idea to report it.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Excellent diagnosis. Certainly a bug. Care to tag with [bugs]? $\endgroup$ – Oleksandr R. Jan 13 '13 at 20:19
  • $\begingroup$ +1, but still waiting a few days for accept. I've already sent the programmer at WRI an email. $\endgroup$ – Andreas Lauschke Jan 13 '13 at 20:34
  • $\begingroup$ Nice work, earned a +1 $\endgroup$ – Rojo Jan 15 '13 at 3:52
10
$\begingroup$

I would not classify this as a bug. The behaviour of a Slot expression that is not directly contained within a Function expression is not defined by the documentation, and is unreliable in practice. Consider the following two functions:

f[x_] := x + 100
g[x_] := x + # &[100]

They appear to be essentially equivalent, but it just so happens that the implementation of the second function involves a pure function. However, they return different results when passed a "dangling" slot reference (i.e. a slot reference that is not directly contained within a pure function):

f[#]
(* 100 + #1 *)

g[#]
(* 200 *)

The second result might surprise us, especially if the documentation for g said "adds 100 to its argument". Consider what happens if we call g with the Module expression from the question:

g[Module[{slot = 1}, Slot[slot]]]
(* 200 *)

200 is perhaps not exactly what we expected, but at least it is consistent with g[#].

On the other hand, let's try it with a new function h which is the same as g but holds its argument:

SetAttributes[h, HoldAll]
h[x_] := x + # &[100]

h[Module[{slot = 1}, Slot[slot]]]
(*
Function::slot: Slot[slot] (in Module[{slot=1},Slot[slot]]+#1&) should contain a non-negative integer. >>
(Module[{slot=1},Slot[slot]]+#1&)[100]
*)

We get the same warning message as in the question. This occurs because this expression ultimately resolves to:

Module[{slot = 1}, Slot[slot]] &

... which is manifestly incorrect since arguments to Slot in a function body must be non-negative integers -- not symbols.

ParallelEvaluate happens to be implemented somewhat like h. It holds the argument, and its implementation happens to use a pure function.

The moral of this story is that the behaviour of a dangling slot reference is essentially undefined. Its behaviour depends upon the exact implementation of the functions to which it is passed. If we are unaware of those exact implementation details, then we should avoid the use of dangling slot references since we can never be sure if that slot expression might not accidentally find its way into an internal Function expression.

To work around this difficulty, we can use a temporary symbol in place of Slot and then substitute it out after the evaluations are complete:

Module[{t}
, ParallelEvaluate[Module[{slot = 1}, t[slot]]] /. t -> Slot
]

This behaviour is just another example of the kind of "gotchas" that occur due to Mathematica's simulation of functional programming constructs through pattern matching techniques.

| improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ I agree in the sense that this is not a bug in Function or Slot, but this behavior should not be manifested in the Parallel` package. Actually I would argue that your g and h have a precedence bug; g should be defined as e.g. g[x_] := x + (# &)[100], whereupon it gives the same result as f. The documentation implicitly countenances (in the Applications section) building expressions containing Slot without a containing Function--their example of Function[Evaluate[f @@ Array[Slot, 5]]] is logically identical to e.g. With[{slots = f @@ Array[Slot, 5]}, Function[slots]]. $\endgroup$ – Oleksandr R. Jan 13 '13 at 20:49
  • 1
    $\begingroup$ @OleksandrR It is not always a simple matter to bulletproof a pure function body against injected slot references. What about the Send function exhibited in @Szabolcs answer? Named pure function arguments don't help either -- they have problems of their own. Even if such bulletproofing were simple, it is not common practice. I'm not arguing that this behaviour is desirable, but I am saying that we shouldn't expect well-defined behaviour from an ill-defined language construct. $\endgroup$ – WReach Jan 13 '13 at 21:21
  • 2
    $\begingroup$ The problem is that there are no clear hints in the docs that there is anything wrong with using naked slots. As Oleksandr points out there is actually an example of programmatically constructing a pure function, which could be perceived as an encouragement to do such things. (I believe @Andreas must have been doing something similar: constructing code programmatically. Though it's true that combining this with parallel evaluation is a little unusual.) Also consider that ParallelEvaluate usually takes code as its argument, unlike most other functions. I wouldn't say that this problem was ... $\endgroup$ – Szabolcs Jan 14 '13 at 1:49
  • 2
    $\begingroup$ @Szabolcs. True the documentation has an example of generated slots, but the example conforms to my advice to only use it when the exact context is known (f has no definitions). I totally agree to report it -- I reported the issue in my blog. Sounds like a great possible issue for the docs. However, I anticipate the same response: avoid the situation. ... $\endgroup$ – WReach Jan 14 '13 at 2:28
  • 2
    $\begingroup$ @WReach +1, agree with all your points. By the way, I wrote a tiny framework to resolve such nasty variable collisions / leaks as in your example in the blog. It lives here. When I call runWithRenamings[ ClearAll[f, g]; f[x_] := g[Function[a, x]]; g[fn_] := Module[{h}, h[a_] := fn[a]; h[0]]; f[999]], I get the correct result. Of course this does not eliminate the problem, but it might be useful for debugging, and also one can run some code in a "safer mode", when wrapped in runWithRenamings. $\endgroup$ – Leonid Shifrin Jan 14 '13 at 10:14
8
$\begingroup$

This is easily fixed by:

  Send[kernels:{___kernel}, expr_] := Table[Send[k, expr], {k, kernels}]

instead of 

Send[kernels:{___kernel}, expr_] := Send[#, expr]& /@ kernels

While changing Parallel.m is one possibility, we can also
change the DownValues programmatically:

Parallel`Developer`Send;
Unprotect[Parallel`Developer`Send];
Parallel`Developer`Send[kernels : {___Parallel`Kernels`kernel}, expr_] := 
  Table[Parallel`Developer`Send[k, expr], {k, kernels}];
Protect[Parallel`Developer`Send];

This first idea is of giving Send Attribute Listable is wrong, thank's Oleksandr. Might be nice if there would be a ListableFirst attribute. But there is not.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The problem with this is that it needs to thread only over those lists appearing in the first argument. Personally, I'd rather do the threading explicitly than rely on the vagaries of Listable. $\endgroup$ – Oleksandr R. Jan 14 '13 at 13:22
  • $\begingroup$ Just like Power[{1, 2, 3}, 2]. So what is vague here? Listable behavior is not going to change I guess. $\endgroup$ – Rolf Mertig Jan 14 '13 at 14:21
  • 1
    $\begingroup$ You're probably right that it won't change. But what if someone wants to ParallelEvaluate a list that happens to be the same length as the list of kernels? $\endgroup$ – Oleksandr R. Jan 14 '13 at 14:29
  • $\begingroup$ It's a nice idea to rely on Listable for a solution. Hopefully this could be robust (unlike a Function with named arguments). $\endgroup$ – Szabolcs Jan 14 '13 at 15:32
  • $\begingroup$ The only answer offering a simple good solution in the first lines, deserves more votes, +1! $\endgroup$ – Rojo Jan 15 '13 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.