How to create keypress simulation program?

Question

Main Problem

I'm trying to create a widely applicable keypress simulation program, but I don't know how to use Java so I can only search for Q&As on the site, which is inefficient. So I want to open a new post and deal with these problems all in once:

I need the following functionality:

1. Press arbitrary keys (using key codes shown here) for example, some code like press[65] should press A.

2. Press arbitrary mouse keys (including mouse down, mouse up and mouse click)

3. Move mouse to a specific location

4. rotate the scrolling of the mouse

5. Screen shot

How to do these small functionalities? Thanks!

---

Some work I've done

One can use these as a reference:

JLink part:

Needs["JLink`"];
ReinstallJava[];

robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];

move[{x_, y_}] := robotclass@mouseMove[x, y];
mouseleftdown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
mouseleftup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
mouseleftclick[] := CompoundExpression[mouseleftdown[], mouseleftup[]];
mouseleftdoubleclick[] := Do[mouseleftclick[], 2];

Mathematica Part:

MousePosition[];
CurrentValue[{"MouseButtonTest",1}]

---

Related

move cursor

left mouse clicks

key press

screen shots

Answer 1

A summary to all the stuffs:

Needs["JLink`"];
ReinstallJava[];

robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];
LoadJavaClass["java.awt.event.KeyEvent"];
LoadJavaClass["java.awt.Event"];

(*Action*)
mousemove[{x_, y_}] := robotclass@mouseMove[x, y];

leftmousedown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
leftmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
rightmousedown[] := robotclass@mousePress[InputEvent`BUTTON3UMASK];
rightmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON3UMASK];
midmousedown[] := robotclass@mousePress[InputEvent`BUTTON2UMASK];
midmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON2UMASK];
leftmouseclick[t_: 1] := 
  Do[CompoundExpression[leftmousedown[], leftmouseup[]], t];
rightmouseclick[] := 
  CompoundExpression[rightmousedown[], rightmouseup[]];
midmouseclick[] := CompoundExpression[midmousedown[], midmouseup[]];

scroll[x_] := robotclass@mouseWheel@x;

keypress = (Function[k, 
      robotclass[keyPress[Symbol["KeyEvent`VKU" <> k]]]] /@ {##};
    Function[k, 
      robotclass[keyRelease[Symbol["KeyEvent`VKU" <> k]]]] /@ {##}) &;

(*Sense*)
getpixelcolor[{x_, y_}] := 
  StringCases[robotclass@getPixelColor[x, y]@toString[], 
   "=" ~~ d : DigitCharacter .. :> ToExpression@d];

screeninfo = 
  "FullScreenArea" /. 
   Flatten@SystemInformation["Devices", "ScreenInformation"];
screenshot[range : {{x0_, x1_}, {y0_, y1_}} : screeninfo] := 
  GUIScreenShot[range];

The name of those functions already explains everything, so no need for extra explanations~ This piece of code has already include all the key features in 按键精灵, a keypress or mouse move simulator used to automate repetitive works, e.g. play HearthStone. But with the ability of Mathematica, I think this piece of code can do much more! :)