Scrape AJAX Website with Webdriver

Question

I hope the new Webdriver functionality can help as the normal Import and HTTPRequest don't work.

The sites I have found are a little out there in content (sports betting) but any solution will help on countless other sites.

If you open Chrome, open the Inspector tool and then go to 10bet. You may need to use a VPN depending on where you live.

You will see this call:

With the response being JSON in the browser.

The same call has these headers:

The cURL for that call is:

curl 'https://www .10bet.com/methods/sportscontent.ashx/GetLeagues?'
-H 'RequestTarget: AJAXService' -H 'Referer: https://www.10bet.com/sports/'
-H 'DNT: 1' -H 'X-JWToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJTZXNzaW9uSUQiOiJtdm5zeGd4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZXhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.3zQMCUdqkLfMTft-XZXd9GxuFT8NxOaAFTb1iAHtnNI'
-H 'ADRUM: isAjax:true' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_ 14_ 3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded' --compressed

If you transform that cURL into Mathematica it looks like this:

request = HTTPRequest["https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?",
        <|Method -> "GET", "ContentType" -> "application/x-www-form-urlencoded",
          "UserAgent" -> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
          "Headers" -> <|"X-JWToken" ->"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9eyJTZXNzaW9uSUQiOiJtdm5zeGd4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZXhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.3zQMCUdqkLfMTftXZXd9GxuFT8NxOaAFTb1iAHtnNI",
          "RequestTarget" -> "AJAXService", "DNT" -> "1","ADRUM" -> "isAjax:true"|>|>]

URLExecute[request]

The request is unsuccessful.

I then use Webdriver to try and navigate to the URL:

session = StartExternalSession["WebDriver-Chrome"]
ExternalEvaluate[session, "OpenWebPage" -> "https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?"]

The result is not JSON but unevaluated AJAX call. I am stuck and have no experience with AJAX. I think one needs to use Webdiver to run a JavaScript script but have no idea how to find the script and therefore use Webdiver to run it.

I would like to interact with the API/JavaScript/AJAX directly and not use Webunit to navigate to the page and scrape it that way, if possible.

var SportsContentPageMethods={Base:"/methods/sportscontent.ashx/GetLeagues", SBase:"/methods/sportscontent.ashx/GetLeagues", ROBase:"/methods/sportscontent_ros.ashx/GetLeagues", ROSBase:"/methods/sportscontent_ros.ashx/GetLeagues" ,
GetBranchesAndLeagues:function(__shouldGetLeagues,__sc, __ec, __cc) {var p={shouldGetLeagues:__shouldGetLeagues};AJAX.call(false,this.Base,"GetBranchesAndLeagues",p,__sc, __ec, __cc);},
GetLeagues:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetLeagues",p,__sc, __ec, __cc);},
GetAllLiveContent:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetAllLiveContent",p,__sc, __ec, __cc);},
GetPartialLiveContent:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetPartialLiveContent",p,__sc, __ec, __cc);},
GetAllUpcomingContent:function(__nextHoursToLoad,__sc, __ec, __cc) {var p={nextHoursToLoad:__nextHoursToLoad};AJAX.call(false,this.Base,"GetAllUpcomingContent",p,__sc, __ec, __cc);},
GetUpcomingContentPerDay:function(__branchID,__numberOfDaysToAdd,__currentTimeZone,__sc, __ec, __cc) {var p={branchID:__branchID,numberOfDaysToAdd:__numberOfDaysToAdd,currentTimeZone:__currentTimeZone};AJAX.call(false,this.Base,"GetUpcomingContentPerDay",p,__sc, __ec, __cc);},
GetUpcomingDaysWithContent:function(__branchID,__numberOfDaysToAdd,__currentTimeZone,__sc, __ec, __cc) {var p={branchID:__branchID,numberOfDaysToAdd:__numberOfDaysToAdd,currentTimeZone:__currentTimeZone};AJAX.call(false,this.Base,"GetUpcomingDaysWithContent",p,__sc, __ec, __cc);}};

Answer 1

For me, the request is not unsuccessful (it gives a 200 status code). However, the JSON it returns throws an error when you attempt to import it: "An error occurred near character ',', at line 1:97", and thus URLExecute fails, because it tries to automatically import the result correctly.

However, the response is essentially just a big list of rows for a table, so we can get around this limitation by importing it as a CSV format instead.

The following code prints out the same structure as the browser:

request = 
 HTTPRequest[
  "https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?", <|
   Method -> "GET", 
   "ContentType" -> "application/x-www-form-urlencoded", 
   "UserAgent" -> 
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) \
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 \
Safari/537.36", 
   "Headers" -> <|
     "X-JWToken" -> 
      "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9eyJTZXNzaW9uSUQiOiJtdm5zeGd\
4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZ\
XhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.\
3zQMCUdqkLfMTftXZXd9GxuFT8NxOaAFTb1iAHtnNI", 
     "RequestTarget" -> "AJAXService", "DNT" -> "1", 
     "ADRUM" -> "isAjax:true"|>|>]

body = URLRead[request]["Body"]

Now we can import the result as CSV, if we replace "[" with the empty string and "]" with a new line. (We use Most to avoid a final empty row).

data = Most@ImportString[StringReplace[body, {"[" -> "", "]" -> "\n"}], 
  "CSV"]

will import the data as a big list of rows.