What is the difference between Dynamic[x] and Dynamic[ h[x] ] for DynamicModule variables?

Question

Bug introduced in 9.0 or earlier and persisting through 12.0

In 11.3 it is even worse. An example from the accepted answer with a Slider is now broken too:

DynamicModule[{x}, { 
  Slider[Dynamic[First@List@x]]
, Button["press", x = 1; Pause[2]; x = RandomReal[],  Method ->"Queued"]
}]

---

Why plain Dynamic[x] is less sensitive and less likely to automatically be updated than e.g. Dynamic[{x}]?

The issue was raised here couple of times but I don't remember any general answer, just work arounds.

• The problem can be reproduced with this example:

  DynamicModule[{x},
    {
      Dynamic[x],
      Button["press", x = 0; Pause[2]; x = 1, Method -> "Queued"]
    }
  ]
  

  The 0 step is not being shown. Source: Dynamic and Refresh

• Or

  DynamicModule[{x = 1, z}
    ,
   {  Dynamic[x],  Dynamic[List@x]  }
   ,
   Initialization :> (
     SetOptions[EvaluationNotebook[],
      CellEventActions :> { "UpArrowKeyDown" -> ( x++;) }
      ]
     )
   ]
  

  {1,{9}}
  

• another place I've faced it in different circumstances:

  Is it possible to update Dynamic objects even when they are off screen?

---

I'm tempted to call it a bug.

Both cases are working when x is e.g. Global`, not scoped to the DynamicModule.

Answer 1

Thanks to Mike Honeychurch we know it's a bug:

WRI tech support confirm that this is a bug

– Mike Honeychurch 10 hours ago

What can we do?

• In simple cases, just add a stupid vanishing wrapper:

  Dynamic[First @ List @ x]
  

  both examples are working with that.

• But it has a limitation, e.g. when the Dynamic we are talking about is in some kind of controller, like Slider:

  DynamicModule[{x}, Panel[Column[{
    Slider[Dynamic[First @ List @ x]],
    Button["press", x = 1; Pause[2]; x = RandomReal[], 
       Method -> "Queued"]}]
  ]]
  

  So now Button action behaves correctly but we can't use slider due to

  Set::write: Tag First in First[{0.937591}] is Protected. >>
  

  We have to improve our work around :) this will work:

  Slider[Dynamic[First @ List @ x, (x = #) &]]
  

Answer 2

This is my third answer to this question. To summarize what follows: I do not longer think it is a bug. What we see is completely predictable and can be considered at most as undesirabable behaviour in a slightly pathological example.

Some terminology. I will call a displayed Dynamic expression a dynamic object. As everything shown in a notebook, it is under control of the frontend.

Similarly, I will call a displayed DynamicModule expression an interface. It is constructed by a kernel evaluation of DynamicModule[variables, body]. The local variables in the DynamicModule expression become frontend owned parameters in the interface; I will call these the states of the interface. The interface usually shows one or more controllers and one or more dynamic objects.

Every interface has a unique interface number and every dynamic object, whether displayed in an interface or not, has a unique dynamic object number.

A dynamic object updates continuously. By inspecting the LinkSnooper output, it turns out that the updating mechanism for a dynamic object in an interface is different from the updating mechanism for a dynamic object not in an interface.

Let us first consider a dynamic object not in an interface. Such an object is the displayed result of the evaluation of Dynamic[expression]. In a sense, this evaluation goes in two steps. In the first step, the kernel returns a DynamicBox to the frontend with the expression still unevaluated. Then the frontend is going to display the object. It assigns a dynamic object number to it and then sends the expression to the kernel together with the object number. The kernel evaluates the expression and during that evaluation, it tags all symbols that are met with the object number. Then it returns the result to the frontend and the frontend displays that result.

When later on the kernel changes the value of a tagged variable, it informs the frontend that the corresponding dynamic object(s) should be updated and it removes the tagging. When the frontend wants to update the dynamic object (that is not always; the object could have been removed or be not visible on the screen), it again sends the expression and the object number to the kernel, the kernel evaluates and tags, returns the result to the frontend and the frontend displays the new result.

Not surprising: updating of a dynamic object not in an interface is triggered by the kernel.

Now a dynamic object in an interface. Such an interface is the displayed result of the evaluation of DynamicModule[variables, ... Dynamic[f[variables], ...]. In the interface, the variables (states) belong to the frontend and the dynamic object has to show f[variables], or better f[states].

Two situations can occur. It may be that f[states] can be evaluated by the frontend itself. About the simplest situation is

DynamicModule[{x=0.5}, Column[{ Slider[Dynamic[x]], Dynamic[x]}]]

Then the frontend does not need the kernel at all. When we use such an interface, LinkSnooper does not show any traffic between the frontend and the kernel. Observe that the dynamic object displays a frontend owned value, not the value of a kernel expression.

Quite often, the frontend is unable to compute f[states]. In that situation, LinkSnooper shows that during the construction of the interface, for every state of the interface an auxiliary kernel variable (with name FE`state$nn) is created and that these auxiliary kernel variables are tagged with the dynamic object number. When we change any of the states of the interface (e.g. by moving a slider), the frontend asks the kernel to assign the current states to the auxiliary variables, and to compute f[variables]. The kernel returns this value and the frontend updates the interface.

Here is a typical example, where we want the dynamic object to display h[x], where h is some or other privately defined or advanced kernel function. The frontend is not aware of this function, so it needs the kernel.

DynamicModule[{x = 0.5}, Panel[Column[{Slider[Dynamic[x]], Dynamic[h[x]]}]]]

It works fine. The state of this interface is the position of the slider. When we move the slider, the dynamic object updates and shows the value of h at the position of the slider.

The buggish example:

DynamicModule[{x=RandomReal[]}, Row[{Button["new value", x=0; Pause[1]; x=RandomReal[], Method->"Queued"], Spacer[10], Dynamic[x]}]]

When the interface is constructed, the kernel is not needed for the contents of the dynamic object. No auxiliary variables are created. When we press the button for the first time, the frontend finds that it cannot execute the button action. Only now the auxiliary variable for the state is created. This creation was triggered by the button, a button cannot be updated, so the auxiliary variable cannot be tagged. Then the button action is send to the kernel for evaluation. It assigns 0 to the auxiliary variable, pauses for a second and then assigns a random real to this variable. Finally it returns this value to the frontend and the frontend displays the result. Since the auxiliary variable is not tagged, the assignments do not result in an updating. Therefore the first number 0 does not turn up.

Now that we know why the 0 does not turn up, we also know the remedy: we have to take care that, when the interface is constructed, the frontend finds that it has to use the kernel for the computation of the dynamic object. Then automatically an auxiliary variable for the state is created and tagged with the dynamic object number.

DynamicModule[{x=RandomReal[]}, Row[{Button["new value", x=0; Pause[1]; x=RandomReal[], Method->"Queued"], Spacer[10], Dynamic[$[]; x]}]]

The contents of Dynamic is $[];x, which evaluates to x. The frontend cannot handle this $[];x (I will discuss some simpler arguments soon), so an auxiliary variable is created and tagged with the dynamic object number. When the button action is sent to the kernel, the kernel assigns 0 to this variable, which forces an updating of the dynamic object (not of the interface!), pauses a second and then assigns a random real, which also gives an updating of the dynamic object. Then the result is sent back to the frontend and the frontend processes the new state of the interface, but that does not change the display any more.

It is clear that we could have used any expression that evaluates to x as argument of Dynamic, as long as the frontend cannot evaluate that expression with x replaced with the state. Before version 12, we could use expressions such as x+0 or 1*x, because of the frontend could not do arithmetic. But in version 12 the frontend can evaluate these expressions. (It indeed looks like the frontend has its own small kernel, and that this frontend kernel is enlarged in version 12.)

When we use the option TrackedSymbols->Full (or Automatic or :>{x}), we also force the immediate creation of an auxiliary variable:

DynamicModule[{x=RandomReal[]},Row[{Button["new value",x=0; Pause[1]; x=RandomReal[],Method->"Queued"],Spacer[10],Dynamic[x, TrackedSymbols:>Automatic]}]]

However, that this works has as such nothing to do with tagged variables. As with the other solutions, it lets the frontend create a tagged auxiliary variable. Only the dynamic object is updated. In my concluding example the slider does not move to 0 when we press the button.

DynamicModule[{x=RandomReal[]},Column[{Button["new value",x=0; Pause[1]; x=RandomReal[],Method->"Queued"], Slider[Dynamic[x]],Dynamic[x, TrackedSymbols:>Automatic]}]]

Answer 3

Actually this does not really give a new answer to the question but probably some more insight and another way of a workaround. As Fred Simons has shown we can use Internal`GetTrackedSymbols[] to get a list of the currently tracked symbols and if a symbol doesn't appear there, we can expect it not updating correctly in the given examples.

Unfortunately his suggestion that using Internal`SetValueNoTrack could be used to make a symbol appear in this list and make it work as intended is not true. In his example most function wrappers would have achieved the same, something that Kuba has shown in his answer. On the other hand, Kubas example Dynamic["whatever";x] given in a comment would actually not solve the problem. The deeper reason is that there seems to exist some exclusion list for symbols which will when used as wrappers not cause the symbol to be put on the tracked lists, among them e.g. Identity and obviously CompoundExpression. Many others, like Times do work and of course any self-defined function will work.

Knowing that, we can create another way to fix Kuba's example which is slightly shorter and will indicate what it actually is doing:

trackfix[x_]:=x;
trackfix /: Set[trackfix[x_], v_] := Set[x, v];

the second fix makes it work in the Slider case as well. It might be too simplistic for other cases, but for this and I think many other basic use cases it will work alright:

DynamicModule[{x}, Panel[Column[{
  Slider[Dynamic[trackfix@x]],
  Button["press", x = 1; Pause[2]; x = RandomReal[], Method -> "Queued"],
  Dynamic[trackfix@x]
}]]]

As for Internal`SetValueNoTrack I think it can well be used to switch between the tracked and not tracked states, but it unfortunately doesn't help in the case where the symbol doesn't appear in Internal`GetTrackedSymbols[] in the first place. This can be demonstrated with one of the working examples:

DynamicModule[{x}, {
  Dynamic[1*x], Dynamic[ToString[Unevaluated@x]],
  Button["press", x=0; Pause[1]; x=RandomInteger[{1,9}], Method->"Queued"]
}]

note that I have added another Dynamic which shows the exact frontend symbol name. Using that shown name we can now switch between the two behaviours by evaluating the following in an extra cell (of course you need to replace FE`x$$77 with what the above Dynamic shows for you):

Internal`SetValueNoTrack[FE`x$$77, True]

and

Internal`SetValueNoTrack[FE`x$$77, False]

Unfortunately in the cases where the corresponding variable does not appear in the list of tracked symbols, Internal`SetValueNoTrack does not add them, so it doesn't help to fix the issue. This can be demonstrated with e.g.:

DynamicModule[{x}, {
  Dynamic[x], Dynamic[ToString[Unevaluated@x]],
  Button["press", x=0; Pause[1]; x=RandomInteger[{1,9}], Method->"Queued"]
}]

and then evaluating the corresponding Internal`SetValueNoTrack expressions in an extra cell. There are some other Internal`*Track* functions, but it would need some spelunking (which I now have not time to do) to find how they need to be used and whether they can help to solve the issue in a better way...

Answer 4

As you say in a comment below, the difference between Dynamic[x] and Dynamic[List@x] is that the former can be evaluated and typeset completely in the Front End, and the latter somehow needs the kernel for typesetting, as can be seen by using LinkSnooper (on a simpler example). This means that when {x} is typeset, it uses the value of x stored in the kernel. When Dynamic[x] is typeset, it uses the value of x stored in the Front End.

The value of x in the Front End and the one in the kernel are not always the same, and all your examples illustrate this.

In your snippet you set an option of the evaluation notebook to be an action that increases a variable that is supposed to be local to your dynamic module. This may not be a good idea. It seems that your "UpArrowKeyDown" :> (x++) only updates what the kernel thinks that this x is and it cannot access the local variable of the DynamicModule. This creates a persistent discrepancy between the front end value and the kernel value of x.

Note that in the following snippet Dynamic[x] and Dynamic[List@x] do work the same (there is no discrepancy). Be sure to place your cursor well inside the expression before pressing up.

DynamicModule[{x = 1, z},
 EventHandler[
  {Dynamic[x], Dynamic[List@x]}, {"UpArrowKeyDown" :> (x++)}
  ]
 ]

Uncommenting the first commented line in the following fixes the discrepancy. It sets the Front End value of x to the kernel value of x. Uncommenting the second line does not fix the discrepancy, because {x}[[1]] will only use the Front End, so that it will use the Front End value of x.

DynamicModule[{x = 1}, {
  Dynamic[
   (*FEPrivate`Set[x,First@List@x];*)
   (*FEPrivate`Set[x,{x}[[1]]];*)
      x],

  Dynamic[{x}]
  }, Initialization :> (SetOptions[EvaluationNotebook[], 
    CellEventActions :> {"UpArrowKeyDown" -> (x++;)}])]