# Docked cell showing keystrokes and mouse buttons actions

When I send bug reports to WRI, sometimes I attach a screen video record, like on the following example (that I just sent as an improvement suggestion). As you can see, I use OSD Hotkey, to make sure that no operation is hidden on the explanation. Is there a way of having a notebook docked cell showing exactly the same thing as OSD Hotkey is doing, so that I can keep it all under Mathematica (although, not the video recording, yet...):

• key strokes
• mouse buttons actions

(the video I sent is a little bigger, and more paused... I just compressed it a little to reduce the page loading time...)

• +1 That would also be great for any kind of presentation/lecture in Mathematica (especially the ability to maybe include Hotkey for various operating systems in the display) – Sascha Dec 30 '15 at 22:53

Sadly, this can't be done with 100% fidelity. For example, a significant issue is that on some platforms, there is no way to directly capture keystrokes which are mapped to menu commands. The operating system automatically translates the keystrokes to the appropriate menu commands long before any user-accessible code has any hope of seeing what transpired.

Also, there's no way to tell when most keys have been released. Ideally, EventHandler would support "KeyUp", but it's just not there right now. Nonetheless, you can still do a fair amount. Here's something I put together which is a good start.

CurrentValue[EvaluationNotebook[],DockedCells]=
{Cell[BoxData[RowBox[{
DynamicBox[RowBox[CurrentValue["ModifierKeys"]]],
DynamicBox[If[CurrentValue["ModifierKeys"]==={},"","+"]],
DynamicBox[CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"},None]]}]],Background->Yellow]};

CurrentValue[EvaluationNotebook[],NotebookEventActions]={
"ReturnKeyDown":>
(CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]="Return"),
"EscapeKeyDown":>
(CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]="Escape"),
"LeftArrowKeyDown":>
(CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]="LeftArrow"),
"RightArrowKeyDown":>
(CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]="RightArrow"),
"UpArrowKeyDown":>
(CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]="UpArrow"),
"DownArrowKeyDown":>
(CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]="DownArrow"),
"KeyDown":>Switch[CurrentValue["EventKey"],
None, 0 (* do nothing *),
"\"", CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]="\"\\\"\"",
_, CurrentValue[EvaluationNotebook[],{TaggingRules,"KeyStroke"}]=CurrentValue["EventKey"]],
PassEventsDown->True};


By way of explanation, this creates...

• A docked cell with a Dynamic that directly tracks a notebook-level tagging rule and the modifier keys list.
• That Dynamic has slightly obscure code because I was trying to avoid any kernel overhead. This should be completely resolved by the FE.
• A notebook-level event handler which sets the tagging rule accordingly for non-modifier keys. For a bunch of cases (but not all).
• And it passes events through so you can still use your notebook.
• But, unfortunately, it does go to the kernel. There's no way to stop event handling from going to the kernel. So, if your kernel is very busy, you could find that this slows down typing.
• The event handler has special handling of " to prevent pink-boxing.

Note that this docked cell immediately responds to modifier key changes, but it's "sticky" on the pressing of any other key. So, if you press a "d", it will continue to show "d" until you press something else.

• Thank you. The way it is, it's hard to follow it. The action should probably be added to a list, that would Drop itself on timestamp or amount of actions (I don't know if OSD has a little more intelligence than that, like on word/space). Also, it seems that the modifiers are added to "stickies" that are no longer being pressed, which gives a wrong reading of what happening. Do you see it possible for the limitations you pointed out to be reduced in the future? Do you think that using an external library would be more complete (like retrieving the info from an "OSD" library into Mathematica)? – P. Fonseca Dec 31 '15 at 10:13
• Some limitations could be dealt with by cleverer coding than I was willing to put the effort in for here. Perhaps somebody would like to follow up with an answer that builds on mine and improves it. But the limit of not having a "KeyUp" event handler is very problematic. And perhaps not too difficult to resolve in a future version of Mathematica. But the limitation of it not showing things like Ctrl+Z because the operating system snatches the event...that's more fundamental, and seems less likely to see a solution in the foreseeable future. – John Fultz Dec 31 '15 at 19:35
• When you say that the OS snatches the event, I guess you are referring to OSX, right? Because when I use OSD, I can see the Ctrl+Z action. – P. Fonseca Dec 31 '15 at 23:52
• @P.Fonseca I mean that, the way Mathematica implements its event handling (which is fairly standard), the operating system steals the keyboard event before Mathematica sees it and transforms it into a menu command event. It's not that it's impossible to capture the keyboard event before it's transformed...it's that there's no very compelling reason for Mathematica to do otherwise (and bear the extra complexity, risk of bugs, extra developer hours contributed to this, etc.). – John Fultz Jan 1 '16 at 4:49