2
$\begingroup$

In an effort to explore the new WebExecute capabilities in Version 12 I have created a simple algorithm to download some data that are stored on an external server and displayed on a webpage. I am using the datatable from https://trumpgolfcount.com/displayoutings as a test, since the authors have no problem of making the data fully available to anyone either in .XLS or JSON format. Thus, allowing me to check if my algorithm is working as intended.

My WebExecute Script correctly loads the first page of this data and automatically generates a Mathematica Dataset from the data generated by the Javascript program displaying the data:

session = StartWebSession["Chrome"];
WebExecute[session, "OpenWebPage" ->  http://trumpgolfoutings/displayoutings"];
rowheaders = WebExecute[session, "LocateElements" -> "Tag" -> "th"];
rowrecords = WebExecute[session, "LocateElements" -> "Tag" -> "td"];
headers = WebExecute["ElementText" -> rowheaders];
numberOfHeaders = Length[headers];
golfrecords = WebExecute["ElementText" -> rowrecords];    
newgolfrecords = Partition[golfrecords, numberOfHeaders];
dataset = Dataset[Association /@ (MapThread[Rule, {headers, #}] & /@ newgolfrecords)];
DeleteObject[session];

All this works fine to upload part of the original table into dataset. However, the data are provided 100 records at a time, with buttons to move from one page of data to another also generated in Javascript and not as hyperlinks. Because subsequent data are also generated from the encoded Javascript the dataset is incomplete. Since there are now over 200 records, my algorithm only creates a dataset only for the first 100 records.

Is there a way using WebExecute (or an alternative route) to effect the pressing of these script generated buttons so that the above code could be repeated for each page containing additional records, all while under control of the WebExecute session?

Although the documentation for WebExectue discusses handling input to input boxes, how to perform an action such as a click doesn't seem to be available except in the contexts of executing hyperlinks via the hyperlink anchor element.

$\endgroup$
  • 3
    $\begingroup$ how about this? page[n_] := WebExecute[ "ClickElement" -> Evaluate[ WebExecute[session, "LocateElements" -> "CSSSelector" -> "#table_id_paginate>span>a:nth-child(" <> ToString@n <> ")"] // First]] session = StartWebSession["Chrome"]; WebExecute[session, "OpenWebPage" -> "https://trumpgolfcount.com/displayoutings"]; page[2]; DeletedObject[session]; $\endgroup$ – Xminer Sep 29 '19 at 10:40
  • 1
    $\begingroup$ Thanks your code does generate the data for pages 2 and 3 (of 3). A bit of cleanup is needed to get the full dataset built, so I'll try to compose your code into an answer tomorrow (getting late here in Kyoto). However, it brings up another question. How did you know how to determine/construct the CSSSelector value? The documentation is skimpy concerning this option. I am not at all familiar with Cascading Style Sheet technology behind this option. Any suggestions for further reading? $\endgroup$ – Stuart Poss Sep 29 '19 at 11:23
  • $\begingroup$ Selector's value can be found with developer mode(F12). the tech for finding CSS-Selector is the same in every language, so how about starting with this(youtube)? If looking into CSS itself, this would be helpful. $\endgroup$ – Xminer Oct 1 '19 at 11:30
  • $\begingroup$ Thank you very much for expanding my horizons. After your insightful post, I've spent time reading Meyer & Wyle, CSS: The definitive Guide. Selectors add many opportunities to more effectively scape the web. $\endgroup$ – Stuart Poss Oct 2 '19 at 5:01
3
$\begingroup$

I would probably do it like this.

  1. Start a new session:
browser = StartWebSession[]
  1. Open the page in question:
WebExecute[browser, "OpenPage" -> "https://trumpgolfcount.com/displayoutings"]
  1. Get the innerText of the table:
tableText = WebExecute[browser, "JavascriptExecute" -> 
    "return document.getElementById('table_id').innerText;"];
  1. Import that text with SemanticImportString:
dataset = SemanticImportString[tableText]

This imports the text as a Dataset (partially shown below):

enter image description here

  1. For navigation to the next page, use the CSS selector for the Next button:
WebExecute[browser, "ClickElement" -> {"CSSSelector", "#table_id_next"}]
$\endgroup$
  • $\begingroup$ Is it possible to check whether the "Next" button is active in order to know when to stop pressing it? Also I'm interested in a way to know whether the page has finished loading. $\endgroup$ – Alexey Popkov Oct 12 '19 at 8:31
  • $\begingroup$ Interesting approach. Not knowing JavaScript, I wouldn't have thought to do it this way. More learning to do. Thanks. $\endgroup$ – Stuart Poss Oct 12 '19 at 12:04
  • $\begingroup$ @AlexeyPopkov it is not generally possible to know if a web page has finished loading when it includes javascript, since javascript can run indefinitely. If you disable javascript on a page, this becomes easier to predict. $\endgroup$ – Arnoud Buzing Oct 14 '19 at 14:35
  • $\begingroup$ @StuartPoss The combination of an external controller (WL+notebooks) and a controllable client (browser) is super useful. Using javascript snippets to send back precise information is something I've done a lot and I am happy to help if you have more questions (just ask here and tag them with 'webtools') $\endgroup$ – Arnoud Buzing Oct 14 '19 at 14:42
  • $\begingroup$ +1 ".innerText" usage is v. useful for those not across Javascript - worth including as an example in the documentation IMO - might save numerous image processing attempts. $\endgroup$ – Ronald Monson Dec 6 '19 at 3:24
0
$\begingroup$

The code now modified to incorporate Xminer's answer is now as follows:

page[n_] := WebExecute["ClickElement" -> Evaluate[WebExecute[session, "LocateElements" -> "CSSSelector" -> "#table_id_paginate>span>a:nth-child(" <> ToString@n <> ")"] // First]]; 
session = StartWebSession["Chrome"]; WebExecute[session, 
 "OpenWebPage" -> "https://trumpgolfcount.com/displayoutings"];
page[1]
rowheaders = WebExecute[session, "LocateElements" -> "Tag" -> "th"];
rowrecords1 = WebExecute[session, "LocateElements" -> "Tag" -> "td"];
headers = WebExecute["ElementText" -> rowheaders];
numberOfHeaders = Length[headers];
golfrecords1 = WebExecute["ElementText" -> rowrecords1];
newgolfrecords1 = Partition[golfrecords1, numberOfHeaders];
page[2]
rowrecords2 = WebExecute[session, "LocateElements" -> "Tag" -> "td"];
golfrecords2 = WebExecute["ElementText" -> rowrecords2];
newgolfrecords2 = Partition[golfrecords2, numberOfHeaders];
page[3]
rowrecords3 = WebExecute[session, "LocateElements" -> "Tag" -> "td"];
golfrecords3 = WebExecute["ElementText" -> rowrecords3];
newgolfrecords3 = Partition[golfrecords3, numberOfHeaders];
newgolfrecords = Join[newgolfrecords1, newgolfrecords2, newgolfrecords3];
dataset = Dataset[Association /@ (MapThread[Rule, {headers, #}] & /@ 
       newgolfrecords)];

This works as intended, but I have not yet figured out how to structure a function that handles those lines that require repetition of scraping and then subsequently joining, particularly when one does not have a clear idea of exactly over how many rows (web pages) these will need to be repeated, noting that the row headers need to be calculated only once for the first webpage (set of row-records).

No doubt, there is a way to do this more efficiently than accomplished so far.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.