44
$\begingroup$

I want to build a bot to automate web browsing, this mean something like:

  • filling forms
  • press "submit" buttons
  • find certain text inside pages
  • and so on...

How can I do this with Mathematica?

The Import function just make you download a single web page but it doesn't support cookies and similar things to build a complete automated bot, does Mathematica have some useful packet to do so?

Improve this question
$\endgroup$
20
  • $\begingroup$ If the form uses the GET method, submitting the form just means composing an URL (on which Import works). The question is very general in this form, and I'd be inclined to say Mathematica is not the right tool for this (you'll end up using JLink or .NETLink anyway). But if you can give a very specific example, we can think about how to implement it in Mathematica (or will be able to say with more confidence that it's not possible without external libraries) $\endgroup$
    – Szabolcs
    Feb 15, 2012 at 10:32
  • 2
    $\begingroup$ if you need support for cookies then use wget. You can include it in Mma code by using Run. About to go to bed but do a search cause posted something here in answer to another question a couple of weeks ago. ...here it is: mathematica.stackexchange.com/questions/1186/… $\endgroup$
    – Mike Honeychurch
    Feb 15, 2012 at 11:26
  • 3
    $\begingroup$ You can also try curl: stackoverflow.com/a/6977128/695132 $\endgroup$
    – Szabolcs
    Feb 15, 2012 at 12:08
  • 1
    $\begingroup$ Hi Rolf, can you put your comment in an answer with separated code? It's difficult to understand what you wrote in this way. Thanks a lot :) $\endgroup$
    – Francesco
    Feb 15, 2012 at 12:44
  • 1
    $\begingroup$ I've failed in this before, especially w/r/t cookies. When trying to find text, I just use Import[x,"Source"] where x is the site (all manually downloaded w/ wget) and then find content using StringCases[] i.e. trlist = StringCases[pagetext, Shortest["<tr>" ~~ ___ ~~ "</tr>"]]; (which would find all text within rows in a page arranged in that way, for example) $\endgroup$
    – canadian_scholar
    Feb 15, 2012 at 14:24

1 Answer 1

Reset to default
39
$\begingroup$

Here is a package which does what you want:

https://github.com/arnoudbuzing/webunit

Clone the repository from github, and place the WebUnit folder under $UserBaseDirectory/Applications

To use it:

  1. Needs["WebUnit`"]

  2. InstallWebUnit[] (* launches chromedriver.exe *)

  3. StartWebSession[] (* launches Chrome web browser, assuming you have that installed *)

  4. OpenWebPage["http://mathematica.stackexchange.com"] (* opens the web page *)

  5. ClickElement[Id["nav-users"]] (* clicks the web element 'nav-users' the users tab *)

And then TypeElement works similarly (assuming you have an input field with an id).

Edit: You can also use JavascriptExecute["alert('hi');"] to execute arbitrary javascript (in this example case it brings up the alert dialog).

Improve this answer
$\endgroup$
23
  • 2
    $\begingroup$ Great application! But could you provide a WebUnit documentation in a pdf file? On my computer the provided documentation does not open properly. I am not sure whether the directory structure in the provided bundle is OK. BTW: is it possible to set a value to a RadioButton? $\endgroup$
    – jano
    Dec 23, 2014 at 22:49
  • 1
    $\begingroup$ @Arnoud Buzing Can you provide a minimal example of how to use Execute[] to run some javascript? I can find this function in the WebUnit source code. There is no documentation so I am asking you here. $\endgroup$
    – PlatoManiac
    Jan 6, 2015 at 4:30
  • $\begingroup$ This works for me: Execute["alert('hi');"] $\endgroup$
    – Arnoud Buzing
    Jan 9, 2015 at 20:17
  • 2
    $\begingroup$ @GregH, I put a new version with a new link in the post above (not sure how this disappeared). Also, in this new version I renamed the function Execute to JavascriptExecute (more precise function name). $\endgroup$
    – Arnoud Buzing
    Aug 20, 2015 at 15:41
  • 1
    $\begingroup$ @ArnoudBuzing I have solved the unsupported command-line flag warning problem just by downloading the latest release of the ChromeDriver (currently it is 2.22) and placing it into the appropriate directory instead of the existing file chromedriver.exe (which comes as a part of the package)! $\endgroup$
    – Alexey Popkov
    Jul 2, 2016 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.