10
$\begingroup$

I'm trying to import this table from a wikipedia page containing the list of lunar missions into a Mathematica as an Association. Here, I'm trying to use the new WikipediaData function, but it doesn't seem to work.

WikipediaData["PageID" -> "656253283", "ArticleWikicode"]
$\endgroup$
5
  • $\begingroup$ When I use your code, it doesn't work but WikipediaData["List of missions to the Moon", "ArticleWikicode"] works fine. $\endgroup$
    – Mahdi
    Commented May 6, 2015 at 17:15
  • $\begingroup$ Oh PageID is wrong, the correct one is 38293476. $\endgroup$
    – Mahdi
    Commented May 6, 2015 at 17:17
  • 1
    $\begingroup$ You can alwars just use Import : Import["http://en.wikipedia.org/wiki/List_of_missions_to_the_Moon", {"HTML", "Source"}] . ( WikipediaData doesn't work for me either by the way ) $\endgroup$
    – george2079
    Commented May 6, 2015 at 17:45
  • 1
    $\begingroup$ This package could be used. The third example in that post is actually a table on Wikipedia. $\endgroup$
    – C. E.
    Commented May 7, 2015 at 0:22
  • 1
    $\begingroup$ The function you are using is not going to work because that's not the PageID of the article you are trying to import. The correct one is 38293476, so, WikipediaData["PageID" -> "38293476", "ArticleWikicode"] works fine. But as Mahdi said, you can also use the actual title as in WikipediaData["List of missions to the Moon", "ArticleWikicode"] $\endgroup$
    – xtian777x
    Commented Aug 27, 2015 at 3:57

4 Answers 4

7
$\begingroup$

my stab at it..

 source = Import[
    "http://en.wikipedia.org/wiki/List_of_missions_to_the_Moon",
            {"HTML", "Source"}];
 tstart = StringPosition[source, 
        "<table class=\"wikitable sortable\"" ][[1, 1]];
 tend = Select[ 
          StringPosition[source, "</table>"] , #[[1]] > tstart & ][[1, 2]];
 tabledata = 
       ImportString[
          StringTake[ source , {tstart, tend} ], {"html", "Data"}];
 Grid@tabledata

If there are more elegant ways of parsing html I'd like to see..

By the way WikipediaData gives me these sorts of errors no matter what I try:

" ServiceObject::noget: The parameter Wikicode is not available for the service Wikipedia. >> "

$\endgroup$
6
$\begingroup$

As george2079 said, you can use Import:

text = Import["https://en.wikipedia.org/wiki/List_of_missions_to_the_Moon", "Data"];

and then naively because table has 7 columns:

table = Cases[text, {_, _, _,_, _, _, _}, ∞];

But you need to do some clean-up afterwards!

$\endgroup$
6
$\begingroup$
session = StartWebSession["Chrome", Visible->False];

WebExecute[session, "OpenWebPage" ->
 "https://en.wikipedia.org/wiki/List_of_missions_to_the_Moon"];

import = WebExecute[
  "LocateElements" -> "CSSSelector" -> "table:nth-of-type(5)>*>tr>*"];

data=Partition[WebExecute[session, "ElementText" -> import], 5];     

Grid[data, Dividers -> All]

table 1

Edit

A Dataset may be a better option:

Dataset[Association@MapThread[Rule, {data[[1]], #}] & /@ data[[2 ;;]],
      Alignment -> "Center", HeaderAlignment -> {Automatic, Center}]

datset of Wikipedia table

CSS Selector

The CSS Pseudo-class nth-child(5) selects the fifth table tag on the html page (and the child-combinator (>) and universal (*) CSS selectors are used to specifically import the contents of each th and td tag as a string)

The approach works well, provided that the table is 'well-structured'. Merged cells and the like make things more difficult. (In these cases it is possibly better to import each table row, and process the resulting string with Mathematica)

For example, the selector

"table:nth-of-type(4)>*>tr>*"

imports the the contents of each th and td tag from the 4th table on the html page and (remembering to change the final arg in Partition to 8!) a table is obtained as above.

Checks

(i) Imported Tags

WebExecute[session, "ElementTag" -> import] // Counts

(* <|"th" -> 5, "td" -> 50|> *)

(ii) CSS Class of Imported Table

import2 = 
  WebExecute[
    "LocateElements" -> "CSSSelector" -> "table:nth-of-type(5)"];

WebExecute[
  "ElementAttribute" -> {First@import2, "class"}]; 

StringReplace[%, " " -> "."] 

(* "wikitable.sortable.jquery-tablesorter" *) 

(iii) Content of table <th> tags

import3 = 
  WebExecute["LocateElements" -> 
    "CSSSelector" -> "table:nth-of-type(5)>*>tr>th"];

WebExecute[session, "ElementText" -> import3]

(* {"Country", "To reach surface", "To soft land", "To return sample", 
    "To land crew"} *) 

(iv) Content of Each Table Row as a String

Compare the above with the case where the CSS-selector is made less specific, so that the content of each table row is obtained as a string.

import4 = 
   WebExecute[
     "LocateElements" -> "CSSSelector" -> "table:nth-of-type(5) >*>tr"];

 tableText = WebExecute[session, "ElementText" -> import4]


(* {
     "Country To reach surface To soft land To return sample To land crew", 
     "Soviet Union 1st 1st 2nd", 
     "United States 2nd 2nd 1st 1st", 
     "Japan 3rd 5th", 
     "ESA 4th", 
     "India 5th 4th", 
     "China 6th 3rd 3rd", 
     "Israel 7th", 
     "Luxembourg 8th", 
     "UAE 9th", 
     "Russia 10th"
   } *)
   

In this case, string processing is required to get a matrix suitable for Grid.

tbody = StringSplit[#, 
    RegularExpression["(?<=[a-zA-Z]).(?=[1-9])"]] &@tableText[[2 ;;]]
    
(* {
    {"Soviet Union", "1st", "1st", "2nd"}, 
    {"United States", "2nd", "2nd", "1st", "1st"}, 
    {"Japan", "3rd", "5th"}, 
    {"ESA", "4th"}, 
    {"India", "5th", "4th"}, 
    {"China", "6th", "3rd",  "3rd"}, 
    {"Israel", "7th"},
    {"Luxembourg", "8th"}, 
    {"UAE", "9th"}, 
    {"Russia", "10th"}
   } *)

thead = StringSplit[#, RegularExpression[".(?=To)"]] &@tableText[[1]]

(*
  {"Country", 
   "To reach surface", 
   "To soft land", 
   "To return sample", 
   "To land crew"} 
 *)

(v) Content <thead> and <tbody> as Strings

As the table is very nicely marked up (see below), by once again making the CSS-selector less specific, the content of thead and tbody may be obtained as strings.

theadString = WebExecute[session, "ElementText" -> import5]
tbodyString = WebExecute[session, "ElementText" -> import6]

import5 = WebExecute[
  "LocateElements" -> "CSSSelector" -> "table:nth-of-type(5) >thead"];
import6 = WebExecute[
   "LocateElements" -> "CSSSelector" -> "table:nth-of-type(5) >tbody"];
   
(* {
    "Country To reach surface To soft land To return sample To land crew"
  } *)

  (* {"Soviet Union 1st 1st 2nd United States 2nd 2nd 1st 1st\
       Japan 3rd 5th ESA 4th India 5th 4th China 6th 3rd 3rd\
       Israel 7th Luxembourg 8th UAE 9th Russia 10th"
     } *)  
  

And, of course, to obtain the contents of the whole table as a string, the CSS-selector table:nth-of-type(5) may be used in the WebExecute command.

(v) Table Markup

Execution of the following command, followed by some cleanup (all span tags have been removed) gives the table markup, possibly making the action of the various CSS-selectors a bit clearer.

WebExecute[session, 
  "JavascriptExecute" -> 
    "return document.getElementsByClassName(
      'wikitable sortable jquery-tablesorter')[0].outerHTML"]

In particular, the first universal selector (*) in table:nth-of-type(5)>*>tr>* is used to select all thead, tbody and (empty) tfoot content.

<table class = "wikitable sortable jquery-tablesorter">
  <thead>
     <tr>
         <th class="headerSort">Content</th>
         <th class="headerSort">Content</th>
         <th class="headerSort">Content</th>
         <th class="headerSort">Content</th>
         <th class="headerSort">Content</th>
     </tr>
  </thead>
  <tbody>
     <tr>
         <td>Content</td>
         <td>Content</td>
         <td>Content</td>
         <td>Content</td>
         <td>Content</td>
     </tr>
    
     ...

     <tr>
         <td>Content</td>
         <td>Content</td>
         <td>Content</td>
         <td>Content</td>
         <td>Content</td>
     </tr>
  </tbody>

  <tfoot></tfoot>

</table>

The First 3 Columns of Table 4

A second example of table extraction.

session = StartWebSession["Chrome", Visible -> False];

WebExecute[session, 
  "OpenWebPage" -> 
 "https://en.wikipedia.org/wiki/List_of_missions_to_the_Moon"];

import = 
  WebExecute[
   "LocateElements" -> 
     "CSSSelector" -> "table:nth-of-type(4)>*>tr>*:nth-child(-n+3)"];

Partition[WebExecute[session, "ElementText" -> import], 3];

table = Grid[%, Dividers -> All]

table 4 with 3 columns

$\endgroup$
1
  • 1
    $\begingroup$ +1 - If there were several tables with 5 columns: How do I distinguish between them? Would your solution import all of them? $\endgroup$
    – eldo
    Commented Apr 20 at 10:43
4
$\begingroup$

I will introduce a wonderful package jSoupLink here. Then we can use it easely:

Needs["jsoupLink`"]
dom = Import[
   "https://en.wikipedia.org/wiki/List_of_missions_to_the_Moon#21st_century", "HTMLDOM"];
Grid[ImportString[
   First[dom["Select", ".wikitable.sortable"]]["OuterHTML"], {"HTML", 
    "FullData"}][[1, 1, 1]], Frame -> All]

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.