37
$\begingroup$

Bug introduced in 9.0 or earlier and persisting through 12.0

In 11.3 it is even worse. An example from the accepted answer with a Slider is now broken too:

DynamicModule[{x}, { 
  Slider[Dynamic[First@List@x]]
, Button["press", x = 1; Pause[2]; x = RandomReal[],  Method ->"Queued"]
}]

Why plain Dynamic[x] is less sensitive and less likely to automatically be updated than e.g. Dynamic[{x}]?

The issue was raised here couple of times but I don't remember any general answer, just work arounds.

  • The problem can be reproduced with this example:

    DynamicModule[{x},
      {
        Dynamic[x],
        Button["press", x = 0; Pause[2]; x = 1, Method -> "Queued"]
      }
    ]
    

    The 0 step is not being shown. Source: Dynamic and Refresh

  • Or

    DynamicModule[{x = 1, z}
      ,
     {  Dynamic[x],  Dynamic[List@x]  }
     ,
     Initialization :> (
       SetOptions[EvaluationNotebook[],
        CellEventActions :> { "UpArrowKeyDown" -> ( x++;) }
        ]
       )
     ]
    
    {1,{9}}
    
  • another place I've faced it in different circumstances:

    Is it possible to update Dynamic objects even when they are off screen?


I'm tempted to call it a bug.

Both cases are working when x is e.g. Global`, not scoped to the DynamicModule.

$\endgroup$
  • 1
    $\begingroup$ An extension of your first example: DynamicModule[{x, z = {0}}, Panel[Column[{Dynamic[x], Dynamic[Identity[x]], Dynamic[Sequence[x]], Dynamic[# &@x], Dynamic[1*x], Dynamic[x^1], Dynamic[{x}], Dynamic[Sequence @@ {x}], Dynamic[z], Button["press", x = 0; z = {0}; Pause[2]; x = RandomInteger[{1, 9}]; z = {RandomInteger[{1, 9}]}, Method -> "Queued"]}]]] $\endgroup$ – Karsten 7. Nov 30 '15 at 16:22
  • 3
    $\begingroup$ I believe the culprit is that the default value for Dynamic is TrackedSymbols -> All and it's missing the update (no idea why). Any other reasonable setting gets it right DynamicModule[{x, z}, {Dynamic[x, TrackedSymbols -> Full], Dynamic[List@x]}, Initialization :> (x = 1; SetOptions[EvaluationNotebook[], CellEventActions :> {"UpArrowKeyDown" -> (x++;)}])] $\endgroup$ – Dr. belisarius Nov 30 '15 at 16:30
  • 2
    $\begingroup$ Please compare DynamicModule[{x = 1, z}, {Dynamic[x], Dynamic[List@x]}, Initialization :> (SetOptions[EvaluationCell[], CellEventActions :> {"UpArrowKeyDown" -> (x++;)}])] (which doesn't work), with DynamicModule[{x = 1, z}, EventHandler[{Dynamic[x], Dynamic[List@x]}, {"UpArrowKeyDown" :> (x++;)}]] (which works) Perhaps it can't be explained by the behavior of Dynamic[ ] alone, but it's its coupling with other constructs. $\endgroup$ – Dr. belisarius Nov 30 '15 at 18:34
  • 3
    $\begingroup$ actually I have premier service so I will ask for an official explanation and post as community wiki if i get one. $\endgroup$ – Mike Honeychurch Dec 1 '15 at 21:47
  • 5
    $\begingroup$ WRI tech support confirm that this is a bug $\endgroup$ – Mike Honeychurch Dec 15 '15 at 22:11
13
$\begingroup$

This is my third answer to this question. To summarize what follows: I do not longer think it is a bug. What we see is completely predictable and can be considered at most as undesirabable behaviour in a slightly pathological example.

Some terminology. I will call a displayed Dynamic expression a dynamic object. As everything shown in a notebook, it is under control of the frontend.

Similarly, I will call a displayed DynamicModule expression an interface. It is constructed by a kernel evaluation of DynamicModule[variables, body]. The local variables in the DynamicModule expression become frontend owned parameters in the interface; I will call these the states of the interface. The interface usually shows one or more controllers and one or more dynamic objects.

Every interface has a unique interface number and every dynamic object, whether displayed in an interface or not, has a unique dynamic object number.

A dynamic object updates continuously. By inspecting the LinkSnooper output, it turns out that the updating mechanism for a dynamic object in an interface is different from the updating mechanism for a dynamic object not in an interface.

Let us first consider a dynamic object not in an interface. Such an object is the displayed result of the evaluation of Dynamic[expression]. In a sense, this evaluation goes in two steps. In the first step, the kernel returns a DynamicBox to the frontend with the expression still unevaluated. Then the frontend is going to display the object. It assigns a dynamic object number to it and then sends the expression to the kernel together with the object number. The kernel evaluates the expression and during that evaluation, it tags all symbols that are met with the object number. Then it returns the result to the frontend and the frontend displays that result.

When later on the kernel changes the value of a tagged variable, it informs the frontend that the corresponding dynamic object(s) should be updated and it removes the tagging. When the frontend wants to update the dynamic object (that is not always; the object could have been removed or be not visible on the screen), it again sends the expression and the object number to the kernel, the kernel evaluates and tags, returns the result to the frontend and the frontend displays the new result.

Not surprising: updating of a dynamic object not in an interface is triggered by the kernel.

Now a dynamic object in an interface. Such an interface is the displayed result of the evaluation of DynamicModule[variables, ... Dynamic[f[variables], ...]. In the interface, the variables (states) belong to the frontend and the dynamic object has to show f[variables], or better f[states].

Two situations can occur. It may be that f[states] can be evaluated by the frontend itself. About the simplest situation is

DynamicModule[{x=0.5}, Column[{ Slider[Dynamic[x], Dynamic[x]}]]

Then the frontend does not need the kernel at all. When we use such an interface, LinkSnooper does not show any traffic between the frontend and the kernel. Observe that the dynamic object displays a frontend owned value, not the value of a kernel expression.

Quite often, the frontend is unable to compute f[states]. In that situation, LinkSnooper shows that during the construction of the interface, for every state of the interface an auxiliary kernel variable (with name FE`state$nn) is created and that these auxiliary kernel variables are tagged with the dynamic object number. When we change any of the states of the interface (e.g. by moving a slider), the frontend asks the kernel to assign the current states to the auxiliary variables, and to compute f[variables]. The kernel returns this value and the frontend updates the interface.

Here is a typical example, where we want the dynamic object to display h[x], where h is some or other privately defined or advanced kernel function. The frontend is not aware of this function, so it needs the kernel.

DynamicModule[{x = 0.5}, Panel[Column[{Slider[Dynamic[x]], Dynamic[h[x]]}]]]

It works fine. The state of this interface is the position of the slider. When we move the slider, the dynamic object updates and shows the value of h at the position of the slider.

The buggish example:

DynamicModule[{x=RandomReal[]}, Row[{Button["new value", x=0; Pause[1]; x=RandomReal[], Method->"Queued"], Spacer[10], Dynamic[x]}]]

When the interface is constructed, the kernel is not needed for the contents of the dynamic object. No auxiliary variables are created. When we press the button for the first time, the frontend finds that it cannot execute the button action. Only now the auxiliary variable for the state is created. This creation was triggered by the button, a button cannot be updated, so the auxiliary variable cannot be tagged. Then the button action is send to the kernel for evaluation. It assigns 0 to the auxiliary variable, pauses for a second and then assigns a random real to this variable. Finally it returns this value to the frontend and the frontend displays the result. Since the auxiliary variable is not tagged, the assignments do not result in an updating. Therefore the first number 0 does not turn up.

Now that we know why the 0 does not turn up, we also know the remedy: we have to take care that, when the interface is constructed, the frontend finds that it has to use the kernel for the computation of the dynamic object. Then automatically an auxiliary variable for the state is created and tagged with the dynamic object number.

DynamicModule[{x=RandomReal[]}, Row[{Button["new value", x=0; Pause[1]; x=RandomReal[], Method->"Queued"], Spacer[10], Dynamic[$[]; x]}]]

The contents of Dynamic is $[];x, which evaluates to x. The frontend cannot handle this $[];x (I will discuss some simpler arguments soon), so an auxiliary variable is created and tagged with the dynamic object number. When the button action is sent to the kernel, the kernel assigns 0 to this variable, which forces an updating of the dynamic object (not of the interface!), pauses a second and then assigns a random real, which also gives an updating of the dynamic object. Then the result is sent back to the frontend and the frontend processes the new state of the interface, but that does not change the display any more.

It is clear that we could have used any expression that evaluates to x as argument of Dynamic, as long as the frontend cannot evaluate that expression with x replaced with the state. Before version 12, we could use expressions such as x+0 or 1*x, because of the frontend could not do arithmetic. But in version 12 the frontend can evaluate these expressions. (It indeed looks like the frontend has its own small kernel, and that this frontend kernel is enlarged in version 12.)

When we use the option TrackedSymbols->Full (or Automatic or :>{x}), we also force the immediate creation of an auxiliary variable:

DynamicModule[{x=RandomReal[]},Row[{Button["new value",x=0; Pause[1]; x=RandomReal[],Method->"Queued"],Spacer[10],Dynamic[x, TrackedSymbols:>Automatic]}]]

However, that this works has as such nothing to do with tagged variables. As with the other solutions, it lets the frontend create a tagged auxiliary variable. Only the dynamic object is updated. In my concluding example the slider does not move to 0 when we press the button.

DynamicModule[{x=RandomReal[]},Column[{Button["new value",x=0; Pause[1]; x=RandomReal[],Method->"Queued"], Slider[Dynamic[x]],Dynamic[x, TrackedSymbols:>Automatic]}]]
$\endgroup$
  • $\begingroup$ Hi, nice to see you, thanks for showing that functions. Though I don't think Internal`SetValueNoTrack[x, False] is a fix, as shown in my answer, whatever placed in Dynamic makes the FrontEnd more sensitive. So Dynamic["whatever";x] works too, it is just a coincidence. And if you run SetValueNoTrack in Initialization the problem still remains. $\endgroup$ – Kuba Jan 31 '17 at 13:55
  • $\begingroup$ Hi @Kuba. Indeed nice that we have a discussion again. I added an update with respect to your interesting remark on Dynamic["whatever", x]. You will see that I do not agree with you that is just coincidence. I still have to investigate why it does not work in the initialization option. I have some ideas, but I am not sure that I can test these ideas. If so, I will add it later. $\endgroup$ – Fred Simons Jan 31 '17 at 19:25
  • 1
    $\begingroup$ Internal`GetTrackedSymbols is a very nice finding, it will be a great help when debugging more complicated cases, so +1 for that alone. As for the fixes: from a practical point of view it probably doesn't matter how exactly that is fixed, it should of course not be necessary at all. Internal`SetValueNoTrack -- if it works as indicated -- just gives us one more handle to try when things go wrong, so I think it is also a good finding. Unfortunately both are not documented, but I think it is worth a remark that the obviously were needed internally as well :-) $\endgroup$ – Albert Retey Jan 31 '17 at 20:29
  • $\begingroup$ Sorry for the late reply. Of course +1. Otoh I don't agree with not considering it a bug. While you provide a good explanation based on internal workings, it is not what is 'sold' to the user. From the perspective of documentation of Dynamic that should not happen. Either documentation explains the process in details or we are free to call Dynamic abstraction a leaky one. $\endgroup$ – Kuba Jun 19 at 7:58
  • $\begingroup$ Btw, you may be interested in: mathematica.stackexchange.com/a/200571/5478 and linked topics. $\endgroup$ – Kuba Jun 19 at 7:59
4
$\begingroup$

Actually this does not really give a new answer to the question but probably some more insight and another way of a workaround. As Fred Simons has shown we can use Internal`GetTrackedSymbols[] to get a list of the currently tracked symbols and if a symbol doesn't appear there, we can expect it not updating correctly in the given examples.

Unfortunately his suggestion that using Internal`SetValueNoTrack could be used to make a symbol appear in this list and make it work as intended is not true. In his example most function wrappers would have achieved the same, something that Kuba has shown in his answer. On the other hand, Kubas example Dynamic["whatever";x] given in a comment would actually not solve the problem. The deeper reason is that there seems to exist some exclusion list for symbols which will when used as wrappers not cause the symbol to be put on the tracked lists, among them e.g. Identity and obviously CompoundExpression. Many others, like Times do work and of course any self-defined function will work.

Knowing that, we can create another way to fix Kuba's example which is slightly shorter and will indicate what it actually is doing:

trackfix[x_]:=x;
trackfix /: Set[trackfix[x_], v_] := Set[x, v];

the second fix makes it work in the Slider case as well. It might be too simplistic for other cases, but for this and I think many other basic use cases it will work alright:

DynamicModule[{x}, Panel[Column[{
  Slider[Dynamic[trackfix@x]],
  Button["press", x = 1; Pause[2]; x = RandomReal[], Method -> "Queued"],
  Dynamic[trackfix@x]
}]]]

As for Internal`SetValueNoTrack I think it can well be used to switch between the tracked and not tracked states, but it unfortunately doesn't help in the case where the symbol doesn't appear in Internal`GetTrackedSymbols[] in the first place. This can be demonstrated with one of the working examples:

DynamicModule[{x}, {
  Dynamic[1*x], Dynamic[ToString[Unevaluated@x]],
  Button["press", x=0; Pause[1]; x=RandomInteger[{1,9}], Method->"Queued"]
}]

note that I have added another Dynamic which shows the exact frontend symbol name. Using that shown name we can now switch between the two behaviours by evaluating the following in an extra cell (of course you need to replace FE`x$$77 with what the above Dynamic shows for you):

Internal`SetValueNoTrack[FE`x$$77, True]

and

Internal`SetValueNoTrack[FE`x$$77, False]

Unfortunately in the cases where the corresponding variable does not appear in the list of tracked symbols, Internal`SetValueNoTrack does not add them, so it doesn't help to fix the issue. This can be demonstrated with e.g.:

DynamicModule[{x}, {
  Dynamic[x], Dynamic[ToString[Unevaluated@x]],
  Button["press", x=0; Pause[1]; x=RandomInteger[{1,9}], Method->"Queued"]
}]

and then evaluating the corresponding Internal`SetValueNoTrack expressions in an extra cell. There are some other Internal`*Track* functions, but it would need some spelunking (which I now have not time to do) to find how they need to be used and whether they can help to solve the issue in a better way...

$\endgroup$
  • $\begingroup$ Very nice answer, thanks. I just observed the puzzling fact that GetTrackedSymbols, called from inside a dynamic object, may give a different result from called outside that object. Evaluate x = 1; Dynamic[If[x == 10, lst1 = Internal`GetTrackedSymbols[]]; x++] and, while the loop is still running, lst2 = Internal`GetTrackedSymbols[]. Then lst1 contains the symbol x, lst2 does not. I am not sure what this means for SetValueNoTrack. It works at least in some circumstances; e.g. InternalSetValueNoTrack[x, True] stops the loop. $\endgroup$ – Fred Simons Feb 1 '17 at 10:47
  • $\begingroup$ @FredSimons: I think that is the basic conclusion: without any documentation we don't really know what both these function really do and how and when one would use them correctly. As there are obviously some things which are at least not well behaved it is also difficult if not impossible to find out about how to use them by experiment. I'm afraid that we will have to live with the known workarounds for some time and be happy that at least those exist... $\endgroup$ – Albert Retey Feb 1 '17 at 11:12
  • $\begingroup$ I completely agree, thanks. $\endgroup$ – Fred Simons Feb 1 '17 at 11:25
4
$\begingroup$

As you say in a comment below, the difference between Dynamic[x] and Dynamic[List@x] is that the former can be evaluated and typeset completely in the Front End, and the latter somehow needs the kernel for typesetting, as can be seen by using LinkSnooper (on a simpler example). This means that when {x} is typeset, it uses the value of x stored in the kernel. When Dynamic[x] is typeset, it uses the value of x stored in the Front End.

The value of x in the Front End and the one in the kernel are not always the same, and all your examples illustrate this.

In your snippet you set an option of the evaluation notebook to be an action that increases a variable that is supposed to be local to your dynamic module. This may not be a good idea. It seems that your "UpArrowKeyDown" :> (x++) only updates what the kernel thinks that this x is and it cannot access the local variable of the DynamicModule. This creates a persistent discrepancy between the front end value and the kernel value of x.

Note that in the following snippet Dynamic[x] and Dynamic[List@x] do work the same (there is no discrepancy). Be sure to place your cursor well inside the expression before pressing up.

DynamicModule[{x = 1, z},
 EventHandler[
  {Dynamic[x], Dynamic[List@x]}, {"UpArrowKeyDown" :> (x++)}
  ]
 ]

Uncommenting the first commented line in the following fixes the discrepancy. It sets the Front End value of x to the kernel value of x. Uncommenting the second line does not fix the discrepancy, because {x}[[1]] will only use the Front End, so that it will use the Front End value of x.

DynamicModule[{x = 1}, {
  Dynamic[
   (*FEPrivate`Set[x,First@List@x];*)
   (*FEPrivate`Set[x,{x}[[1]]];*)
      x],

  Dynamic[{x}]
  }, Initialization :> (SetOptions[EvaluationNotebook[], 
    CellEventActions :> {"UpArrowKeyDown" -> (x++;)}])]
$\endgroup$
  • $\begingroup$ {x} needs Kernel for typesetting (List), not evaluation (x). It is the same as here: Dynamic[CurrentValue[{"MousePosition"}]], this is sent to the Kernel but it doesn't have to, try: Dynamic[CurrentValue[{"MousePosition"}][[1]]]. CellEventActions are arguable but Button's evaluation is a clear case. There is a flaw in a synchronization scheme. Synchronization between kernel evaluation and updating the state of Dynamic Objects. $\endgroup$ – Kuba Oct 5 '16 at 8:35
  • $\begingroup$ @Kuba I have to think about that, but please see my awesome last example, which fixes the mismatch between the x in the kernel and the x in the Front End $\endgroup$ – Jacob Akkerboom Oct 5 '16 at 8:49
  • $\begingroup$ @Kuba you make a good point, I am once again confused :P $\endgroup$ – Jacob Akkerboom Oct 5 '16 at 9:00
  • $\begingroup$ @Kuba ok, so it seems the Front End can handle ToBoxes[1], but not ToBoxes[{1}] $\endgroup$ – Jacob Akkerboom Oct 5 '16 at 10:19
  • $\begingroup$ @Kuba I suppose this also means that the front end "backtracks" if it finds an expression it cannot call ToBoxes on itself, and even though it has evaluated the expression inside ToBoxes, it sends the entire unevaluated version of the expression with head ToBoxes over to the kernel. $\endgroup$ – Jacob Akkerboom Oct 5 '16 at 10:21
15
$\begingroup$

Thanks to Mike Honeychurch we know it's a bug:

WRI tech support confirm that this is a bug

– Mike Honeychurch 10 hours ago

What can we do?

  • In simple cases, just add a stupid vanishing wrapper:

    Dynamic[First @ List @ x]
    

    both examples are working with that.

  • But it has a limitation, e.g. when the Dynamic we are talking about is in some kind of controller, like Slider:

    DynamicModule[{x}, Panel[Column[{
      Slider[Dynamic[First @ List @ x]],
      Button["press", x = 1; Pause[2]; x = RandomReal[], 
         Method -> "Queued"]}]
    ]]
    

    So now Button action behaves correctly but we can't use slider due to

    Set::write: Tag First in First[{0.937591}] is Protected. >>
    

    We have to improve our work around :) this will work:

    Slider[Dynamic[First @ List @ x, (x = #) &]]
    
$\endgroup$
  • $\begingroup$ Does anyone know if this was fixed in 10.3.1? $\endgroup$ – John McGee Jan 5 '16 at 14:46
  • $\begingroup$ @JohnMcGee 10.3.1 here, wasn't fixed. $\endgroup$ – Kuba Jan 5 '16 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.