2
$\begingroup$

I've been trying to learn how to do this and I tried getting the raw source and also as XMLObjects. However, I'm having a difficult time extract the data that I need from a section on the page. I feel that it would be easier if it was a table, but it's a number of lists (HTML "< ul>< li>...< /li >< /ul >") with a header on each (A to Z).

You don't have to give me the solution so much as just guidance. Although I would learn from your code as well.

Here is the page: https://en.wikipedia.org/wiki/List_of_plants_used_in_herbalism

I'm hoping to get each list item and the first hyperlink URL in each item.

For "Abscess root" it would return the text and the URL of the first link (https://en.wikipedia.org/wiki/Polemonium_reptans). I'm guessing it would be returned as a list [{"Abscess root...", "http://www....."}, {...,...}]

Any guidance is appreciated.

$\endgroup$
5
  • $\begingroup$ I think the data you want is in WikipediaData["List of plants used in herbalism", "LinksList"] and WikipediaData["List of plants used in herbalism", "ExternalLinksList"], but it isn't paired up in a neat form like you want. $\endgroup$
    – Jason B.
    Oct 4 '16 at 13:55
  • 1
    $\begingroup$ If you're feeling brave, you may try extracting the information with my package jSoupLink. $\endgroup$
    – C. E.
    Oct 4 '16 at 14:09
  • $\begingroup$ Thanks, @JasonB, I'll try it out! $\endgroup$
    – Sam B
    Oct 4 '16 at 17:26
  • $\begingroup$ @C.E. I'm trying to use jSoupLink but don't understand what the "element" should be under Properties. Example, element["TagName"]. How would that be used? herbs = Import[ "en.wikipedia.org/wiki/List_of_plants_used_in_herbalism", "HTMLDOM"] gave me an HTMLElement object, which I was able to view the DOMTree of. How do I get all children of <ul> tags that are in a <div> with a specific css class? $\endgroup$
    – Sam B
    Oct 5 '16 at 0:54
  • $\begingroup$ Try the selector ".className li" $\endgroup$
    – C. E.
    Oct 5 '16 at 2:23
1
$\begingroup$

You may use the String Manipulation functions on the "ArticleWikicode" property of article using WikipediaData.

awc = WikipediaData["List of plants used in herbalism", "ArticleWikicode"];

Notice that the section you are interested is preceded by {{Compact ToC}} and followed by ==Databases== in wiki code. Use StringPosition and StringTake to get this part of the wiki code.

tocDetails = StringTake[awc,
   {1, -1} + 
    MapThread[#1[#2] &, {{Last, First}, 
      StringPosition[awc, {"{{Compact ToC}}", "==Databases=="}]}]];

Each item is a bullet with the link immediately following it. In wiki code the link is the article name. Some links have alternatives delimited by a pipe (|). Use StringCases to get the article titles and StringSplit to get the first article name when there are alternatives.

titles = StringCases[ tocDetails, 
   "\n* [[" ~~ a : Shortest[__] ~~ "]]" :> First@StringSplit[a, "|"], 
   Overlaps -> False];

Next notice that Wikipedia article URLs follow a standard pattern with the article title in the URL. The spaces in an article title are replace by an underscore in the URL. Use StringTemplate and StringReplace to setup this URL pattern.

wikiURL = 
  StringTemplate["https://en.wikipedia.org/w/index.php?title=`1`", 
   InsertionFunction -> StringReplace[" " -> "_"]];

Finally, Map (/@) the template over the list of article titles for the list for the list of URLs. Hyperlink is used to create functioning hyperlinks.

{#, Hyperlink@wikiURL@#} & /@ titles

Hope this helps.

$\endgroup$
1
  • $\begingroup$ That's fantastic! Thanks for doing it all in Mathematica, I learned a lot from it. $\endgroup$
    – Sam B
    Oct 6 '16 at 12:59
0
$\begingroup$

In version 0.8 of jSoupLink, you could do it like this:

doc = Import[
   "https://en.wikipedia.org/wiki/List_of_plants_used_in_herbalism", 
   "HTMLDOM"
];
herbs = doc["Select", "#mw-content-text > ul li"];

extractInformation[herb_] := {
  herb["AllText"],
  Through[herb["Select", "a"]["Attribute", "href"]]
  }
data = extractInformation /@ herbs;

We now have for example

data[[9]]

{"Astragalus (Astragalus propinquus) has long been used in traditional Chinese medicine to strengthen the immune system, and is used in modern China to treat hepatitis and as an adjunctive therapy in cancer.[18]", {"/wiki/Astragalus", "/wiki/Astragalus_propinquus",
"/wiki/Traditional_Chinese_medicine", "/wiki/Immune_system",
"/wiki/Hepatitis", "/wiki/Cancer", "#cite_note-18"}}

$\endgroup$
1
  • $\begingroup$ Thank you, that worked! I'll add a filter to get rid of links to citations. $\endgroup$
    – Sam B
    Oct 5 '16 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.