10
$\begingroup$

I am trying to search people from Wikipedia(For example .*John.* in REGEX).

The following query is close, but it isn't generalized like .*John.* and only gives back historic figures that have passed(They only aren't Alphabetized).

Import["http://en.wikipedia.org/w/api.php?action=query&list=search&\srsearch=wikipedia&srprop=timestamp&eititle=Template:Persondata", "XML"]

link

The following query works although I can't seem to limit the results to people only.

Import["http://en.wikipedia.org/w/api.php?action=query&list=embeddedin&eititle=Template:Persondata&eilimit=100&format=xml&redirects", "XML"]

link2 | API sandbox

Although currently I'm running into Wikipedia API issues, I am looking for a way to get the results in Mathematica (and therefore I believe this question makes most sense on M.stackexchange).

$\endgroup$
10
$\begingroup$

This will download the titles of all articles that transclude the Persondata template, if that's what you're trying to do.

Flatten@NestWhileList[
   Import["http://en.wikipedia.org/w/api.php?action=query&list=\
embeddedin&eititle=Template:Persondata&format=json&eilimit=500" <> 
      If[Length@# > 1, "&eicontinue=" <> #[[2, 2, 1, 2, 1, 2]], ""], 
     "JSON"] &, {}, Length@# != 1 &][[2 ;;, 1, 2, 1, 2, All, 3, 2]]

You can page through them manually here: https://en.wikipedia.org/w/index.php?title=Special:WhatLinksHere/Template:Persondata&limit=20&hideredirs=1&hidelinks=1

The API only lets you request 500 at a time though (and there are more than 1.1 million results), so expect to wait about 15 minutes or so.

$\endgroup$
6
  • $\begingroup$ I believe that only includes historical figures correct? $\endgroup$
    – William
    Apr 5 '14 at 2:03
  • 1
    $\begingroup$ Wikipedia is only supposed to include notable people. en.wikipedia.org/wiki/Wikipedia:Notability_%28people%29 $\endgroup$ Apr 5 '14 at 2:06
  • $\begingroup$ +1 Although for example the list you show doesn't link to Aaron *? For example Aaron Swartz doesn't show on the list but he certainly has a wikipedia page. Somehow Wikipedia decides who to link to. In addition I was looking to search based on regex for example *name* $\endgroup$
    – William
    Apr 5 '14 at 2:12
  • 1
    $\begingroup$ The list is not alphabetical. I think it is in the order of the date a template was added to a page. Aaron Swartz is number 156433 in the list I got, and Select[list, StringMatchQ[#, "Aaron*"] &] // Length gives me 907. Some people add these templates to their user pages though, so you'll want to clear out ones that match "User:*", for example. I'm sure there are some other types of noise too. I use the API to work with the category system a lot, but people are split into many subcategories so you'd have to merge quite a few queries. $\endgroup$ Apr 5 '14 at 2:35
  • $\begingroup$ I don't understand where you want to search for *name* though. $\endgroup$ Apr 5 '14 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.