10
$\begingroup$

I am trying to search people from Wikipedia(For example .*John.* in REGEX).

The following query is close, but it isn't generalized like .*John.* and only gives back historic figures that have passed(They only aren't Alphabetized).

Import["http://en.wikipedia.org/w/api.php?action=query&list=search&\srsearch=wikipedia&srprop=timestamp&eititle=Template:Persondata", "XML"]

link

The following query works although I can't seem to limit the results to people only.

Import["http://en.wikipedia.org/w/api.php?action=query&list=embeddedin&eititle=Template:Persondata&eilimit=100&format=xml&redirects", "XML"]

link2 | API sandbox

Although currently I'm running into Wikipedia API issues, I am looking for a way to get the results in Mathematica (and therefore I believe this question makes most sense on M.stackexchange).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
10
$\begingroup$

This will download the titles of all articles that transclude the Persondata template, if that's what you're trying to do.

Flatten@NestWhileList[
   Import["http://en.wikipedia.org/w/api.php?action=query&list=\
embeddedin&eititle=Template:Persondata&format=json&eilimit=500" <> 
      If[Length@# > 1, "&eicontinue=" <> #[[2, 2, 1, 2, 1, 2]], ""], 
     "JSON"] &, {}, Length@# != 1 &][[2 ;;, 1, 2, 1, 2, All, 3, 2]]

You can page through them manually here: https://en.wikipedia.org/w/index.php?title=Special:WhatLinksHere/Template:Persondata&limit=20&hideredirs=1&hidelinks=1

The API only lets you request 500 at a time though (and there are more than 1.1 million results), so expect to wait about 15 minutes or so.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I believe that only includes historical figures correct? $\endgroup$
    – William
    Commented Apr 5, 2014 at 2:03
  • 1
    $\begingroup$ Wikipedia is only supposed to include notable people. en.wikipedia.org/wiki/Wikipedia:Notability_%28people%29 $\endgroup$
    – Michael Hale
    Commented Apr 5, 2014 at 2:06
  • $\begingroup$ +1 Although for example the list you show doesn't link to Aaron *? For example Aaron Swartz doesn't show on the list but he certainly has a wikipedia page. Somehow Wikipedia decides who to link to. In addition I was looking to search based on regex for example *name* $\endgroup$
    – William
    Commented Apr 5, 2014 at 2:12
  • 1
    $\begingroup$ The list is not alphabetical. I think it is in the order of the date a template was added to a page. Aaron Swartz is number 156433 in the list I got, and Select[list, StringMatchQ[#, "Aaron*"] &] // Length gives me 907. Some people add these templates to their user pages though, so you'll want to clear out ones that match "User:*", for example. I'm sure there are some other types of noise too. I use the API to work with the category system a lot, but people are split into many subcategories so you'd have to merge quite a few queries. $\endgroup$
    – Michael Hale
    Commented Apr 5, 2014 at 2:35
  • $\begingroup$ I don't understand where you want to search for *name* though. $\endgroup$
    – Michael Hale
    Commented Apr 5, 2014 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.