Module variable scoping in Scheduled Tasks?

I ran across a scoping puzzle while experimenting with ScheduledTasks, and I'd be grateful for an explanation from the sages here. I found a workaround by intuitive horse sense, but was unable to explain to myself adequately what was wrong with my original attempt.

First, a straightforward, tail-recursive, synchronous Module

Module[{state = 0, doNextIteration},
doNextIteration = Function[
state = state + 1;
If[state < 4,
(Print[state]; doNextIteration[])]];
doNextIteration[]]
(* 1 2 3 *)


Next, my first attempt to do the same with asynchronous tasks, which I need to tail-chain as one-shot ScheduledTasks. If CreateScheduledTask has a second List argument, then it creates a one-shot task delayed by the time in seconds in the list, just what I need for my bigger application. In this simplified sample, the ScheduledTask tail-chains a new one-shot ScheduledTask by a tail-recursive call -- just like the successful synchronous code -- once every tenth of a second until the condition on state produces False (ignore cleanup of the task objects for simplicity, here).

Module[{state = 0, runNextTask},
state = state + 1;
If[state < 4,
{0.10}]];


OK, the problem is this only Prints once, not the desired three times. I had a sickening hunch that the problem has something to do with runNextTask being local to the Module and somehow not being able to refer to itself recursively -- even though there is only one Module -- in the bizzaro-land of asynchrony, so I "fixed" the code as follows:

Module[{state = 0, runNextTask = Unique[]},
If[state < 3,
Module[{},
state = state + 1;
Print[state];
{0.10}]];


I made runNextTask refer to a Unique global symbol, and made my function into a rewrite rule attached to runNextTask[]. This works great, but I don't understand well why it does. I hate being as dumb as a horse, even if I can jump the fences.

-

You hit a rather subtle behavior, related to the garbage-collection and the Temporary attribute, and the semantics of Module regarding returning expressions. The thing is, to achieve your goal, you need the Module-generated variable (function)'s definition(s) to be exported outside Module (to be persistent). But, since you return not the symbol itself, but its r.h.s. (which is a chunk of code with delayed evaluation, either Function or ScheduledTaskObject in this case), the definitions of runNextTask are destroyed because it has a Temporary attribute, despite the fact that code containing this variable has been exported outside Module.

To illustrate my point, here are two work-arounds which both will lead to your desired effect:

1. Instead of assigning runNextTask to Unique[], insert a line ClearAll[runNextTask] right after the Module declarations. This will remove the Temporary attribute and the definition of runNextTask will persist.

2. Wrap your final call in Hold (so, return Hold[runNextTask[]]), and then use ReleaseHold outside Module.

This is your code for this option:

res =
state = state + 1;
If[state < 4,
],
{0.10}];

ReleaseHold[res]


In this case, the definition also persists.

Note that this problem is not seen in examples where all evaluation happens at the time when Module is left: the following code will execute promptly

Module[{f, n = 0},
f[] := (n++; If[n < 10, Print["*"]; f[]]);
f[]]


even though all definitions for f are destroyed at the end - because the execution here is immediate (the same evaluation process, so that f still has all definitions during this evaluation), while delayed and asynchronous code induces a separate evauation process.

-
Thanks for this clue! I have a follow-on that seems to be related coming in a new question. –  Reb.Cabin Apr 1 '12 at 22:10
@Reb.Cabin Glad I could help. I actually did not previously encounter this exact behavior, so I learned something too. Thanks for the accept. –  Leonid Shifrin Apr 1 '12 at 22:17
good explanation, thanks for that. I just wanted to mention that, if the function is just called for the side effect like here, one could just do this to make the definition persist: Module[{state = 0, runNextTask}, runNextTask = Function[StartScheduledTask@CreateScheduledTask[state = state + 1; If[state < 4, (Print[state]; runNextTask[])], {0.10}]]; runNextTask[]; Hold@runNextTask] –  Albert Retey Apr 2 '12 at 9:32
@Albert Indeed, a good option! I did not think about it, thanks. –  Leonid Shifrin Apr 2 '12 at 11:47
Well, actually shortly after my comment I decided that I would actually recommend to rather use Unique than Module to create the symbol: Not only because no additional "tricks" are needed to make things work. It also seems easier to guess what the code actually is supposed to do for a reader. As far as I can see the OP is using the version I'd recommend anyway... –  Albert Retey Apr 2 '12 at 14:12