8
$\begingroup$

I hope the new Webdriver functionality can help as the normal Import and HTTPRequest don't work.

The sites I have found are a little out there in content (sports betting) but any solution will help on countless other sites.

If you open Chrome, open the Inspector tool and then go to 10bet. You may need to use a VPN depending on where you live.

You will see this call: enter image description here

With the response being JSON in the browser.

The same call has these headers: enter image description here

The cURL for that call is:

curl 'https://www .10bet.com/methods/sportscontent.ashx/GetLeagues?'
-H 'RequestTarget: AJAXService' -H 'Referer: https://www.10bet.com/sports/'
-H 'DNT: 1' -H 'X-JWToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJTZXNzaW9uSUQiOiJtdm5zeGd4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZXhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.3zQMCUdqkLfMTft-XZXd9GxuFT8NxOaAFTb1iAHtnNI'
-H 'ADRUM: isAjax:true' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_ 14_ 3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded' --compressed

If you transform that cURL into Mathematica it looks like this:

request = HTTPRequest["https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?",
        <|Method -> "GET", "ContentType" -> "application/x-www-form-urlencoded",
          "UserAgent" -> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
          "Headers" -> <|"X-JWToken" ->"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9eyJTZXNzaW9uSUQiOiJtdm5zeGd4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZXhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.3zQMCUdqkLfMTftXZXd9GxuFT8NxOaAFTb1iAHtnNI",
          "RequestTarget" -> "AJAXService", "DNT" -> "1","ADRUM" -> "isAjax:true"|>|>]

URLExecute[request]

The request is unsuccessful.

I then use Webdriver to try and navigate to the URL:

session = StartExternalSession["WebDriver-Chrome"]
ExternalEvaluate[session, "OpenWebPage" -> "https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?"]

The result is not JSON but unevaluated AJAX call. I am stuck and have no experience with AJAX. I think one needs to use Webdiver to run a JavaScript script but have no idea how to find the script and therefore use Webdiver to run it.

I would like to interact with the API/JavaScript/AJAX directly and not use Webunit to navigate to the page and scrape it that way, if possible.

var SportsContentPageMethods={Base:"/methods/sportscontent.ashx/GetLeagues", SBase:"/methods/sportscontent.ashx/GetLeagues", ROBase:"/methods/sportscontent_ros.ashx/GetLeagues", ROSBase:"/methods/sportscontent_ros.ashx/GetLeagues" ,
GetBranchesAndLeagues:function(__shouldGetLeagues,__sc, __ec, __cc) {var p={shouldGetLeagues:__shouldGetLeagues};AJAX.call(false,this.Base,"GetBranchesAndLeagues",p,__sc, __ec, __cc);},
GetLeagues:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetLeagues",p,__sc, __ec, __cc);},
GetAllLiveContent:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetAllLiveContent",p,__sc, __ec, __cc);},
GetPartialLiveContent:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetPartialLiveContent",p,__sc, __ec, __cc);},
GetAllUpcomingContent:function(__nextHoursToLoad,__sc, __ec, __cc) {var p={nextHoursToLoad:__nextHoursToLoad};AJAX.call(false,this.Base,"GetAllUpcomingContent",p,__sc, __ec, __cc);},
GetUpcomingContentPerDay:function(__branchID,__numberOfDaysToAdd,__currentTimeZone,__sc, __ec, __cc) {var p={branchID:__branchID,numberOfDaysToAdd:__numberOfDaysToAdd,currentTimeZone:__currentTimeZone};AJAX.call(false,this.Base,"GetUpcomingContentPerDay",p,__sc, __ec, __cc);},
GetUpcomingDaysWithContent:function(__branchID,__numberOfDaysToAdd,__currentTimeZone,__sc, __ec, __cc) {var p={branchID:__branchID,numberOfDaysToAdd:__numberOfDaysToAdd,currentTimeZone:__currentTimeZone};AJAX.call(false,this.Base,"GetUpcomingDaysWithContent",p,__sc, __ec, __cc);}};
$\endgroup$
  • $\begingroup$ Does this help? $\endgroup$ – Rohit Namjoshi Mar 13 at 2:07
  • $\begingroup$ @RohitNamjoshi thank you but that post is about a Mathematica JSON bug (which I have also seen first hand and reported to Wolfram) and they used Python to get around the bug. That post was making a call and getting a response, my problem is I am not getting a response or don't know how to properly make the call due to the dynamic nature of AJAX. $\endgroup$ – Ray Troy Mar 13 at 11:27
  • $\begingroup$ @RohitNamjoshi, my apologies. You were correct that the problem was importing. I never checked the status code to see if the request was successful or not so I assumed the call was failing. $\endgroup$ – Ray Troy Mar 16 at 13:21
  • 1
    $\begingroup$ No worries. Looks like @CarlLange (yet again) came up with a workaround. $\endgroup$ – Rohit Namjoshi Mar 16 at 19:13
4
$\begingroup$

For me, the request is not unsuccessful (it gives a 200 status code). However, the JSON it returns throws an error when you attempt to import it: "An error occurred near character ',', at line 1:97", and thus URLExecute fails, because it tries to automatically import the result correctly.

However, the response is essentially just a big list of rows for a table, so we can get around this limitation by importing it as a CSV format instead.

The following code prints out the same structure as the browser:

request = 
 HTTPRequest[
  "https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?", <|
   Method -> "GET", 
   "ContentType" -> "application/x-www-form-urlencoded", 
   "UserAgent" -> 
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) \
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 \
Safari/537.36", 
   "Headers" -> <|
     "X-JWToken" -> 
      "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9eyJTZXNzaW9uSUQiOiJtdm5zeGd\
4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZ\
XhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.\
3zQMCUdqkLfMTftXZXd9GxuFT8NxOaAFTb1iAHtnNI", 
     "RequestTarget" -> "AJAXService", "DNT" -> "1", 
     "ADRUM" -> "isAjax:true"|>|>]

body = URLRead[request]["Body"]

Now we can import the result as CSV, if we replace "[" with the empty string and "]" with a new line. (We use Most to avoid a final empty row).

data = Most@ImportString[StringReplace[body, {"[" -> "", "]" -> "\n"}], 
  "CSV"]

will import the data as a big list of rows.

enter image description here

$\endgroup$
  • $\begingroup$ thank you for the help. I never thought to check the status code of the call. I hope Mathematica fixes that JSON bug! There are a lot more calls (to different routes) to the site I would like to make and if they fail, will report back. $\endgroup$ – Ray Troy Mar 16 at 13:26
  • $\begingroup$ It's not totally clear to me that the JSON issue is actually a bug, but if I'm honest I spent no time whatsoever debugging it :) JavaScriptExpression is also an option as an Import format that might be worth checking out (though it has the same issue in this case). $\endgroup$ – Carl Lange Mar 16 at 13:41
  • 3
    $\begingroup$ @RayTroy The JSON is invalid because it has consecutive commas in it, which isn't allowed. $\endgroup$ – C. E. Mar 16 at 16:07
  • $\begingroup$ @CarlLange do you think there is a way to preserve the structure and not have it be a flat lists of lists? I would like to keep the nesting of the lists. $\endgroup$ – Ray Troy Mar 18 at 11:55
  • 1
    $\begingroup$ @RayTroy Ah, I honestly didn't notice that the data wasn't just a flat structure. I'm afraid I don't have time to spend on it now - I would probably end up just playing around with mapping Partition over the data to add the structure back in. $\endgroup$ – Carl Lange Mar 18 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.