6
$\begingroup$

I am trying to import this table from Wikipedia into Mathematica as a Dataset. The fields I'm interested are Name, Release Date (Y-M-D) (parsed as a Wolfram date entity), and PPD.

I tried

Import["https://en.wikipedia.org/wiki/Comparison_of_virtual_reality_\
headsets#Tethered_VR_(PC-based_and_PlayStation_VR)", "Dataset"]

but get The Import element "Dataset" is not present when importing as HTML.

Per another post I also tried

Import["https://en.wikipedia.org/wiki/Comparison_of_virtual_reality_\
headsets#Tethered_VR_(PC-based_and_PlayStation_VR)", "Data"]

but seemingly get a bunch of stuff imported other than just the table.

$\endgroup$
  • 1
    $\begingroup$ In a URL the # just tells the browser to scroll to that location. It doesn't actually mean just the thing displayed when you do. That's your fundamental issue. Secondly you'll probably want to write some stuff to parse it with XMLElement and Cases. It's not particularly tough, but it can be annoying. Should be no more than a 10 minute exercise, though. 3 if you're comfortable with this kind of thing. $\endgroup$ – b3m2a1 Dec 10 '19 at 17:16
11
$\begingroup$

After some mental hemming and hawing, I decided it'd be good to have a proper answer about importing HTML tables using something other than the often buggy "Data" field.

So here's what I got. First something to pull all the "table" fields:

getHTMLTables[url_String?(StringStartsQ[URLParse[#]["Scheme"], "http"] &)] :=

    getHTMLTables[Import[url, {"HTML", "XMLObject"}]];
getHTMLTables[xml : _XMLElement | XMLObject[___][___]] :=

  Cases[xml, XMLElement["table", _, _], Infinity];

Then something to make a Dataset from a given table:

xmlTableToDataset[tab_] :=
 Module[{headerRow, headers, data, rows},
  headerRow =
   FirstCase[tab[[3]], XMLElement["tr", __], None, Infinity];
  headers =
   Cases[
    headerRow, 
    d : XMLElement["th", _, _] :>
     StringTrim@
      StringJoin@Flatten@System`Convert`HTMLImportDump`SymbolicXML2Text[d],
    Infinity
    ];
  rows =
   Cases[tab, XMLElement["tr", _, _], Infinity];
  data =
   Map[
    Cases[#, 
      d : XMLElement["td", _, _] :>
       StringTrim@
        StringJoin@Flatten@System`Convert`HTMLImportDump`SymbolicXML2Text[d],
      Infinity
      ] &,
    rows
    ];
  Dataset@Map[
    AssociationThread[headers, #] &,
    Select[data, Length@# == Length@headers &]
    ]
  ]

Finally this:

tables = getHTMLTables[
   "https://en.wikipedia.org/wiki/Comparison_of_virtual_reality_headsets"];

ds = xmlTableToDataset@tables[[3]]

ds // Keys // First // Normal

{"Name", "Release Date (Y-M-D)", "Positional tracking", "Display", \
"Resolution (per eye)", "PPD", "Aspect Ratio (per eye)", "Refresh rate", \
"Field of view", "Audio", "Connectivity", "Weight", "Price", "Platform", \
"Other"}

ds[[All, {"Name", "PPD"}]] // First // Normal

<|"Name" -> "Razer OSVR HDK 1.4[1]", "PPD" -> "9.6"|>

The data is pretty dirty, but that's to be expected for tabular data. Unfortunately there isn't likely to be a JSON dump of that, so there's nothing to be done.

$\endgroup$
  • $\begingroup$ This is cool! Thank you for contributing it. $\endgroup$ – MarcoB Dec 10 '19 at 17:51
  • $\begingroup$ @b3m2a1: This is a very useful code for web-based research. $\endgroup$ – Tugrul Temel Dec 10 '19 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.