Main Problem

I'm trying to create a widely applicable keypress simulation program, but I don't know how to use Java so I can only search for Q&As on the site, which is inefficient. So I want to open a new post and deal with these problems all in once:

I need the following functionality:

  1. Press arbitrary keys (using key codes shown here) for example, some code like press[65] should press A.

  2. Press arbitrary mouse keys (including mouse down, mouse up and mouse click)

  3. Move mouse to a specific location

  4. rotate the scrolling of the mouse

  5. Screen shot

How to do these small functionalities? Thanks!

Some work I've done

One can use these as a reference:

JLink part:


robotclass = JavaNew["java.awt.Robot"];

move[{x_, y_}] := robotclass@mouseMove[x, y];
mouseleftdown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
mouseleftup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
mouseleftclick[] := CompoundExpression[mouseleftdown[], mouseleftup[]];
mouseleftdoubleclick[] := Do[mouseleftclick[], 2];

Mathematica Part:



move cursor

left mouse clicks

key press

screen shots

  • $\begingroup$ related: search?q=robotClass and search?q=screenshot $\endgroup$ – Kuba Aug 25 '16 at 14:17
  • $\begingroup$ What, that can't be found above, is missing? $\endgroup$ – Kuba Aug 25 '16 at 14:18
  • $\begingroup$ What's missing? $\endgroup$ – Wjx Aug 25 '16 at 14:21
  • $\begingroup$ What answers have you failed to find? $\endgroup$ – Kuba Aug 25 '16 at 14:25
  • $\begingroup$ rotate scrolling and press arbitrary mouse keys (except left mouse button). but just willing to merge them together so that further readers can get a full image of how to manipulate such things with this post. $\endgroup$ – Wjx Aug 25 '16 at 14:27

A summary to all the stuffs:


robotclass = JavaNew["java.awt.Robot"];

mousemove[{x_, y_}] := robotclass@mouseMove[x, y];

leftmousedown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
leftmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
rightmousedown[] := robotclass@mousePress[InputEvent`BUTTON3UMASK];
rightmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON3UMASK];
midmousedown[] := robotclass@mousePress[InputEvent`BUTTON2UMASK];
midmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON2UMASK];
leftmouseclick[t_: 1] := 
  Do[CompoundExpression[leftmousedown[], leftmouseup[]], t];
rightmouseclick[] := 
  CompoundExpression[rightmousedown[], rightmouseup[]];
midmouseclick[] := CompoundExpression[midmousedown[], midmouseup[]];

scroll[x_] := robotclass@mouseWheel@x;

keypress = (Function[k, 
      robotclass[keyPress[Symbol["KeyEvent`VKU" <> k]]]] /@ {##};
      robotclass[keyRelease[Symbol["KeyEvent`VKU" <> k]]]] /@ {##}) &;

getpixelcolor[{x_, y_}] := 
  StringCases[robotclass@getPixelColor[x, y]@toString[], 
   "=" ~~ d : DigitCharacter .. :> ToExpression@d];

screeninfo = 
  "FullScreenArea" /. 
   Flatten@SystemInformation["Devices", "ScreenInformation"];
screenshot[range : {{x0_, x1_}, {y0_, y1_}} : screeninfo] := 

The name of those functions already explains everything, so no need for extra explanations~ This piece of code has already include all the key features in 按键精灵, a keypress or mouse move simulator used to automate repetitive works, e.g. play HearthStone. But with the ability of Mathematica, I think this piece of code can do much more! :)

  • $\begingroup$ how do I get keypress to emulate "f" $\endgroup$ – Conor Cosnett Jul 22 '17 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.