4
$\begingroup$

I can get a list of links in a Wikipedia article using WikipediaData:

sortedLinks = WikipediaData["County", "LinksList"];
sortedLinks // Length
586

I need a list of links in order of their appearance in the article. I tried to write a function to extract the links from the article's code:

unsortedWikiLinks[title_String] := 
 DeleteDuplicates@Append[
   StringCases[WikipediaData[title, "ArticleWikicode"], 
    "[[" ~~ link : Except["]" | "|", _] .. :> 
     ToUpperCase[StringTake[link, 1]] <> StringDrop[link, 1]], 
  title]

However, many links are missed because "ArticleWikicode" does not parse the templates in the article.

unsortedLinks = unsortedWikiLinks["County"];
unsortedLinks // Length
244

The official Wikipedia API has an action to parse the templates, however I can't find this option in Mathematica's implementation.

How to get a list of links in order of their appearance with Mathematica?

$\endgroup$
  • 1
    $\begingroup$ I wouldn't expect to see this part of the API exposed in WikipediaData. If you want to use that part of the API, I would use it directly without WikipediaData using URLFetch. $\endgroup$ – Searke Jan 4 '16 at 15:37
4
$\begingroup$

Searke in the comments has a good idea. I found that one can get the article code with templates expanded by using query with revisions (there may be other ways). Here is an implementation of such an API query using URLExecute:

expandedWikitext[title_String] := Cases[
   URLExecute[
    "https://en.wikipedia.org/w/api.php",
    {"format" -> "json",
     "action" -> "query",
     "titles" -> title,
     "prop" -> "revisions",
     "rvprop" -> "content",
     "rvexpandtemplates" -> "True"}], 
   ("*" -> text_String) :> text, Infinity
  ] // First

The above will return the code with templates expanded. Therefore, to construct an unsorted link function, do something like:

unsortedWikilinks[title_String] := DeleteDuplicates@Join[
   {title},
   Select[
    StringCases[
     expandedWikitext[title],
     "[[" ~~ link : Except["]" | "|" | "#", _] .. :> Capitalize@link
    ],
    ! StringContainsQ[#, ":"] &
   ]
  ]

(It's still a little kludgy, but it works).

unsortedLinks = unsortedWikiLinks["County"];
unsortedLinks // Length

586

ContainsExactly[unsortedlinks, sortedlinks]

True

$\endgroup$
  • $\begingroup$ Thanks, I am not familiar with URLExecute, so this is very helpful. Also for the "#" gotcha in link parsing. $\endgroup$ – shrx Jan 4 '16 at 16:57
  • $\begingroup$ @shrx, happy to help! URLExecute is new, but really suitable for API queries (and similar to other URL post-type functions in other languages.) $\endgroup$ – Virgil Jan 4 '16 at 18:10
  • $\begingroup$ I mean "get-type". $\endgroup$ – Virgil Jan 4 '16 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.