7
$\begingroup$

Main Problem

I'm trying to create a widely applicable keypress simulation program, but I don't know how to use Java so I can only search for Q&As on the site, which is inefficient. So I want to open a new post and deal with these problems all in once:

I need the following functionality:

  1. Press arbitrary keys (using key codes shown here) for example, some code like press[65] should press A.

  2. Press arbitrary mouse keys (including mouse down, mouse up and mouse click)

  3. Move mouse to a specific location

  4. rotate the scrolling of the mouse

  5. Screen shot

How to do these small functionalities? Thanks!


Some work I've done

One can use these as a reference:

JLink part:

Needs["JLink`"];
ReinstallJava[];

robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];

move[{x_, y_}] := robotclass@mouseMove[x, y];
mouseleftdown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
mouseleftup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
mouseleftclick[] := CompoundExpression[mouseleftdown[], mouseleftup[]];
mouseleftdoubleclick[] := Do[mouseleftclick[], 2];

Mathematica Part:

MousePosition[];
CurrentValue[{"MouseButtonTest",1}]

Related

move cursor

left mouse clicks

key press

screen shots

$\endgroup$
6
  • $\begingroup$ related: search?q=robotClass and search?q=screenshot $\endgroup$
    – Kuba
    Aug 25, 2016 at 14:17
  • $\begingroup$ What, that can't be found above, is missing? $\endgroup$
    – Kuba
    Aug 25, 2016 at 14:18
  • $\begingroup$ What's missing? $\endgroup$
    – Wjx
    Aug 25, 2016 at 14:21
  • $\begingroup$ What answers have you failed to find? $\endgroup$
    – Kuba
    Aug 25, 2016 at 14:25
  • $\begingroup$ rotate scrolling and press arbitrary mouse keys (except left mouse button). but just willing to merge them together so that further readers can get a full image of how to manipulate such things with this post. $\endgroup$
    – Wjx
    Aug 25, 2016 at 14:27

1 Answer 1

5
$\begingroup$

A summary to all the stuffs:

Needs["JLink`"];
ReinstallJava[];

robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];
LoadJavaClass["java.awt.event.KeyEvent"];
LoadJavaClass["java.awt.Event"];

(*Action*)
mousemove[{x_, y_}] := robotclass@mouseMove[x, y];

leftmousedown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
leftmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
rightmousedown[] := robotclass@mousePress[InputEvent`BUTTON3UMASK];
rightmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON3UMASK];
midmousedown[] := robotclass@mousePress[InputEvent`BUTTON2UMASK];
midmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON2UMASK];
leftmouseclick[t_: 1] := 
  Do[CompoundExpression[leftmousedown[], leftmouseup[]], t];
rightmouseclick[] := 
  CompoundExpression[rightmousedown[], rightmouseup[]];
midmouseclick[] := CompoundExpression[midmousedown[], midmouseup[]];

scroll[x_] := robotclass@mouseWheel@x;

keypress = (Function[k, 
      robotclass[keyPress[Symbol["KeyEvent`VKU" <> k]]]] /@ {##};
    Function[k, 
      robotclass[keyRelease[Symbol["KeyEvent`VKU" <> k]]]] /@ {##}) &;

(*Sense*)
getpixelcolor[{x_, y_}] := 
  StringCases[robotclass@getPixelColor[x, y]@toString[], 
   "=" ~~ d : DigitCharacter .. :> ToExpression@d];

screeninfo = 
  "FullScreenArea" /. 
   Flatten@SystemInformation["Devices", "ScreenInformation"];
screenshot[range : {{x0_, x1_}, {y0_, y1_}} : screeninfo] := 
  GUIScreenShot[range];

The name of those functions already explains everything, so no need for extra explanations~ This piece of code has already include all the key features in 按键精灵, a keypress or mouse move simulator used to automate repetitive works, e.g. play HearthStone. But with the ability of Mathematica, I think this piece of code can do much more! :)

$\endgroup$
1
  • $\begingroup$ how do I get keypress to emulate "f" $\endgroup$ Jul 22, 2017 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.