I hope the new Webdriver functionality can help as the normal Import
and HTTPRequest
don't work.
The sites I have found are a little out there in content (sports betting) but any solution will help on countless other sites.
If you open Chrome, open the Inspector tool and then go to 10bet. You may need to use a VPN depending on where you live.
With the response being JSON in the browser.
The same call has these headers:
The cURL for that call is:
curl 'https://www .10bet.com/methods/sportscontent.ashx/GetLeagues?'
-H 'RequestTarget: AJAXService' -H 'Referer: https://www.10bet.com/sports/'
-H 'DNT: 1' -H 'X-JWToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJTZXNzaW9uSUQiOiJtdm5zeGd4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZXhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.3zQMCUdqkLfMTft-XZXd9GxuFT8NxOaAFTb1iAHtnNI'
-H 'ADRUM: isAjax:true' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_ 14_ 3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded' --compressed
If you transform that cURL into Mathematica it looks like this:
request = HTTPRequest["https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?",
<|Method -> "GET", "ContentType" -> "application/x-www-form-urlencoded",
"UserAgent" -> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
"Headers" -> <|"X-JWToken" ->"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9eyJTZXNzaW9uSUQiOiJtdm5zeGd4ZXlicWJ3amZoYXZucHE0ZmEiLCJTaXRlSUQiOiI1NiIsIm5iZiI6MTU1MjQzODUxNCwiZXhwIjoxNTUzMDQzMzE0LCJpYXQiOjE1NTI0Mzg1MTR9.3zQMCUdqkLfMTftXZXd9GxuFT8NxOaAFTb1iAHtnNI",
"RequestTarget" -> "AJAXService", "DNT" -> "1","ADRUM" -> "isAjax:true"|>|>]
URLExecute[request]
The request is unsuccessful.
I then use Webdriver to try and navigate to the URL:
session = StartExternalSession["WebDriver-Chrome"]
ExternalEvaluate[session, "OpenWebPage" -> "https://www.10bet.com/methods/sportscontent.ashx/GetLeagues?"]
The result is not JSON but unevaluated AJAX call. I am stuck and have no experience with AJAX. I think one needs to use Webdiver to run a JavaScript script but have no idea how to find the script and therefore use Webdiver to run it.
I would like to interact with the API/JavaScript/AJAX directly and not use Webunit to navigate to the page and scrape it that way, if possible.
var SportsContentPageMethods={Base:"/methods/sportscontent.ashx/GetLeagues", SBase:"/methods/sportscontent.ashx/GetLeagues", ROBase:"/methods/sportscontent_ros.ashx/GetLeagues", ROSBase:"/methods/sportscontent_ros.ashx/GetLeagues" ,
GetBranchesAndLeagues:function(__shouldGetLeagues,__sc, __ec, __cc) {var p={shouldGetLeagues:__shouldGetLeagues};AJAX.call(false,this.Base,"GetBranchesAndLeagues",p,__sc, __ec, __cc);},
GetLeagues:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetLeagues",p,__sc, __ec, __cc);},
GetAllLiveContent:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetAllLiveContent",p,__sc, __ec, __cc);},
GetPartialLiveContent:function(__sc, __ec, __cc) {var p={};AJAX.call(false,this.Base,"GetPartialLiveContent",p,__sc, __ec, __cc);},
GetAllUpcomingContent:function(__nextHoursToLoad,__sc, __ec, __cc) {var p={nextHoursToLoad:__nextHoursToLoad};AJAX.call(false,this.Base,"GetAllUpcomingContent",p,__sc, __ec, __cc);},
GetUpcomingContentPerDay:function(__branchID,__numberOfDaysToAdd,__currentTimeZone,__sc, __ec, __cc) {var p={branchID:__branchID,numberOfDaysToAdd:__numberOfDaysToAdd,currentTimeZone:__currentTimeZone};AJAX.call(false,this.Base,"GetUpcomingContentPerDay",p,__sc, __ec, __cc);},
GetUpcomingDaysWithContent:function(__branchID,__numberOfDaysToAdd,__currentTimeZone,__sc, __ec, __cc) {var p={branchID:__branchID,numberOfDaysToAdd:__numberOfDaysToAdd,currentTimeZone:__currentTimeZone};AJAX.call(false,this.Base,"GetUpcomingDaysWithContent",p,__sc, __ec, __cc);}};