Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am trying to search people from Wikipedia(For example .*John.* in REGEX).

The following query is close, but it isn't generalized like .*John.* and only gives back historic figures that have passed(They only aren't Alphabetized).

Import["http://en.wikipedia.org/w/api.php?action=query&list=search&\srsearch=wikipedia&srprop=timestamp&eititle=Template:Persondata", "XML"]

link

The following query works although I can't seem to limit the results to people only.

Import["http://en.wikipedia.org/w/api.php?action=query&list=embeddedin&eititle=Template:Persondata&eilimit=100&format=xml&redirects", "XML"]

link2 | API sandbox

Although currently I'm running into Wikipedia API issues, I am looking for a way to get the results in Mathematica (and therefore I believe this question makes most sense on M.stackexchange).

share|improve this question

1 Answer 1

up vote 9 down vote accepted

This will download the titles of all articles that transclude the Persondata template, if that's what you're trying to do.

Flatten@NestWhileList[
   Import["http://en.wikipedia.org/w/api.php?action=query&list=\
embeddedin&eititle=Template:Persondata&format=json&eilimit=500" <> 
      If[Length@# > 1, "&eicontinue=" <> #[[2, 2, 1, 2, 1, 2]], ""], 
     "JSON"] &, {}, Length@# != 1 &][[2 ;;, 1, 2, 1, 2, All, 3, 2]]

You can page through them manually here: https://en.wikipedia.org/w/index.php?title=Special:WhatLinksHere/Template:Persondata&limit=20&hideredirs=1&hidelinks=1

The API only lets you request 500 at a time though (and there are more than 1.1 million results), so expect to wait about 15 minutes or so.

share|improve this answer
    
I believe that only includes historical figures correct? –  Liam William Apr 5 at 2:03
1  
Wikipedia is only supposed to include notable people. en.wikipedia.org/wiki/Wikipedia:Notability_%28people%29 –  Michael Hale Apr 5 at 2:06
    
+1 Although for example the list you show doesn't link to Aaron *? For example Aaron Swartz doesn't show on the list but he certainly has a wikipedia page. Somehow Wikipedia decides who to link to. In addition I was looking to search based on regex for example *name* –  Liam William Apr 5 at 2:12
1  
The list is not alphabetical. I think it is in the order of the date a template was added to a page. Aaron Swartz is number 156433 in the list I got, and Select[list, StringMatchQ[#, "Aaron*"] &] // Length gives me 907. Some people add these templates to their user pages though, so you'll want to clear out ones that match "User:*", for example. I'm sure there are some other types of noise too. I use the API to work with the category system a lot, but people are split into many subcategories so you'd have to merge quite a few queries. –  Michael Hale Apr 5 at 2:35
    
I don't understand where you want to search for *name* though. –  Michael Hale Apr 5 at 2:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.