How to execute JavaScipt on a webpage and then import the result on OSX?

Question

I would like to import data from this page through Mathematica,

http://www.oddsportal.com/soccer/netherlands/eredivisie/ .

I tried importing through the default Import function

d = Import["http://www.oddsportal.com/soccer/netherlands/eredivisie/","Data"[[3, 1, 1, 2]];
Select[d, ! StringFreeQ[#[[1]], " - "] &] // Quiet // TableForm

I got a simular table like this where all the interessting data is replaced by a '-'.

Heracles - Cambuur      -   -   -
Willem II - AZ Alkmaar  -   -   -
Heerenveen - Excelsior  -   -   -
G.A. Eagles - Dordrecht -   -   -
Zwolle - Vitesse        -   -   -
Breda - Twente          -   -   -
Den Haag - Groningen    -   -   -
Feyenoord - Utrecht     -   -   -
Ajax - PSV              -   -   -

I've been told this is because the webpage updates the data through a JavaScript, that fetches the information form a database. So is there a way of letting the webpage load it's data before importing it?

Futhermore, I found a solution for Windows users, where the webpage is loaded with Internet Explorer by NETLink. But since I mostly use a OSX machine, this won't work for me.

Needs["NETLink`"];
InstallNET[];
ie = CreateCOMObject["InternetExplorer.Application"];
ie@Navigate[
"http://www.oddsportal.com/soccer/netherlands/eredivisie/"];
Pause[3];
doc = ie@Document;
html = doc@body@innerHTML;

This made able to construct a table with all the data, after combing through the html code.

AZ Alkmaar - Ajax       2.96    3.47    2.26
Excelsior - G.A. Eagles 2.38    3.37    2.86
Utrecht - Willem II     2.19    3.4     3.16
Groningen - Heracles    1.7     4.      4.41

So again, is there a way of executing the JavaScript on a page before importing the data? Maybe through a JLink equivalent of the code above?

Answer 1

In this case it would be far more reliable to request the information from the source directly. You can find the source using your web browser's developer tool, in Google Chrome you find this information under "Network." It should look something like:

Importing that data as JSON gives us a nested list of rules containing all the data:

rawData = Import["http://fb.oddsportal.com/ajax-sport-country-tournament/1/d0aaiJN1/X0/1/?_=1408357250275"];
data = ImportString[StringReplace[rawData, "globals.jsonpCallback('/ajax-sport-country-tournament/1/d0aaiJN1/X0/1/', " ~~ d__ ~~ ");" :> d], "JSON"]

Each game has its own ID that is used in the data source. You can find the ID of the game by looking at the URL of the game's page, for example we have

http://www.oddsportal.com/soccer/netherlands/eredivisie/heracles-cambuur-h2uJFB4h/

which tells us that the ID of the Heracles-Cambuur game is h2uJFB4h.