6
$\begingroup$

Here is the website from the official federation: http://results.ittf.link/index.php?option=com_fabrik&view=list&listid=36&Itemid=158 enter image description here

I want to export the Data (the table). There are 33704 rows from several tabs. Can someone explain to me how to do this?

Export function in Mathematica can't identify the tab. enter image description here

$\endgroup$
  • 1
    $\begingroup$ What code are you using to import the first page? $\endgroup$ – Edmund Sep 23 '18 at 11:13
  • 2
    $\begingroup$ It will be difficult because that information is not present in the original HTML that results.ittf.link/… points to. What happens is that it loads some JavaScript, and then the JavaScript, in turn, loads the data. The idea is to play around with the Chrome driver to be able to navigate between the pages and extract the information with that. That's the only way you can evaluate the JavaScript from inside Mathematica. $\endgroup$ – C. E. Sep 23 '18 at 11:18
  • 1
    $\begingroup$ Otherwise you have to do some reverse engineering on the website and figure out what the JavaScript does and how it gets the information, and then implement the same thing in Mathematica (usually it turns out that is it downloading the data from a JSON API, in this case it is straightforward once you know how the API works.) $\endgroup$ – C. E. Sep 23 '18 at 11:19
6
$\begingroup$

Here is a start towards a solution, using the Chrome drivers that were introduced in 11.3. At this point, the Chrome drivers are still labeled as "experimental" and it shows. Many of the things that I tried did not work and gave unreasonable errors. In the future, it will probably be very useful, however, if they keep working on it.

First, we start a session:

session = StartExternalSession["WebDriver-Chrome"]

Mathematica graphics

At this point, a blank page should open up in a new Chrome window. Next we load your website:

ExternalEvaluate[
 session,
 "OpenWebPage" -> 
  "http://results.ittf.link/index.php?option=com_fabrik&view=list&listid=36&Itemid=158"
 ];

At this point, you should see the website in your browser window. Go ahead and select the year 2018 so that the table of tournaments that you are interested in shows up.

The following is a code that find the "Next" button and then clicks on it, whenever we click our own "Next" button in our interface.

next := (
  nextButton = First@ExternalEvaluate[
     session,
     "LocateElements" -> <|"LinkText" -> "Next"|>
     ];
  ExternalEvaluate[
   session,
   "ClickElement" -> nextButton
   ];
  )

Button["Next page", next]

It should look and work like this:

Example

What this shows is that we can programmatically go to the next page. Now, we need to be able to read the table data on each page. This is where it gets tricky because of the experimental state of the Chrome driver. First of all, what I would really like to see is a function to get the HTML for the page. If that option existed, it would be great. I could parse it with my jSoupLink package or whatever. Instead, it seems we have to do this with the ElementText command. The next problem is that I can't get CSSSelector to work, which means I can't select the cells in the way that I want. The best I could find that didn't throw arcane error messages was this:

table = First@ExternalEvaluate[
   session,
   "LocateElements" -> <|"Id" -> "list_36_com_fabrik_36"|>
   ];
data = ExternalEvaluate[session, "ElementText" -> table];

Now all the content of the table is in that string, but it requires post-processing and it isn't easy to do since all the columns have been mashed together.

Here's an example of what we can do:

rows = StringSplit[data, "\n"][[27 ;;]];
rows = StringRiffle[#, ", "] & /@ Partition[rows, 2, 2];
rows[[1]]

"2018 - African Junior and Cadet Championships, Abidjan (CIV) SESAY \ Alieu (SLE) () ABRAHAMS Byrone (RSA) (), 0 3 ABRAHAMS Byrone"

So we've extracted the rows, but it isn't obvious how we can separate the columns since there are no separators in between them.

If it becomes possible (and maybe it is already working for others, or maybe there is a way that I didn't try) to select the cells themselves, e.g. using CSS selectors, then this could be a viable approach to loading your data.

$\endgroup$
  • 1
    $\begingroup$ I could never get any of WebDrivers to work. Are there any setup instructions anywhere in the docs (e.g. how to specify the location of the browser)? Maybe this is worth a separate question. $\endgroup$ – Szabolcs Sep 23 '18 at 15:50
  • 1
    $\begingroup$ @Szabolcs I'm afraid I don't know, for me it "just worked." $\endgroup$ – C. E. Sep 23 '18 at 16:07
  • $\begingroup$ Thank you @C.E.for your help. I have tried jSoupLink, but the federation just changed the layout with JS. :( It's too hard to export the function automatically. $\endgroup$ – Nam Nguyen Sep 24 '18 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.