Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Wikipedia's "Edit Watchlist" page (for logged in users) has a list of hyperlinks to the user's watchlist (here is an example from my account).

enter image description here

I've tried to use URLFetch and URLSave as follows

URLFetch["http://en.wikipedia.org/wiki/Special:EditWatchlist"]
URLSave["http://en.wikipedia.org/wiki/Special:EditWatchlist","test.html"]

but it seems that they cannot grab the links in the watchlist. I thought URLSave is the same as saving a webpage as a HTML file by hand in Internet Explorer... Am I wrong?

Even Import, with

Import["http://en.wikipedia.org/wiki/Special:EditWatchlist","Hyperlinks"]

(and also with "Source"), cannot get the hyperlinks.

How can I log into Wikipedia and get these links using Mathematica?

share|improve this question
    
Do you have to login to that site? –  Mike Honeychurch Aug 11 '13 at 4:32
    
@MikeHoneychurch Yes, logged user could favorite pages. –  HyperGroups Aug 11 '13 at 4:43
1  
Well if you need to login then Import is never going to work. You need to show us what code you have tried -- obviously replace actual user name and password with generic strings. –  Mike Honeychurch Aug 11 '13 at 5:20
3  
Here is how to retrieve a Wikipedia user's watchlist by using the MediaWiki API. My watchlist has over 10,000 articles and it gets the full list in a few seconds (the API paging gives you 500 results at a time). pastebin.com/0Q6LJM9r –  Michael Hale Aug 11 '13 at 10:34
1  
@SjoerdC.deVries One reason is useful now, and the other might be more useful in the future. For now, I occasionally like to go on vandalism cleaning sprees to relieve stress. It is fun to click undo when people are damaging something so valuable. I hope in the future to be able to use my watchlist change feed to keep up to date on very specific research topics (at a finer level of topic granularity than science news sites). I'll add my response as an answer. –  Michael Hale Aug 11 '13 at 17:32
show 9 more comments

1 Answer

up vote 7 down vote accepted

The MediaWiki API is good, fast, well-tested, and is usable within Mathematica for tasks that require authentication by using URLFetch. To login you need to send two POSTs. The second sends back a token that you receive from the first request. Then Mathematica automatically sets $HTTPCookies. This login function will be added to the utilities for the Wikicode project. Then you can request your watchlist once your Mathematica client has been authenticated. I use a loop to retrieve paged results if your watchlist is longer than 500 pages.

(* Returns True or False for success or failure *)
Login[username_String, password_String] := 
 Module[{url = "http://en.wikipedia.org/w/api.php", 
   parameters = {"action" -> "login", "lgname" -> username, 
     "lgpassword" -> password, "format" -> "json"}}, 
  MatchQ["result" /. ("login" /. 
      ImportString[
       If[MatchQ[#, "NeedToken"], 
          URLFetch[url, "Method" -> "POST", 
           "Parameters" -> Append[parameters, "lgtoken" -> #2]], 
          Return@False] & @@ ({"result", 
           "token"} /. ("login" /. 
            ImportString[
             URLFetch[url, "Method" -> "POST", 
              "Parameters" -> parameters], "JSON"])), "JSON"]), 
   "Success"]]

urlEncode[string_String] := 
 StringReplace[string, 
  c : Except@
     Flatten@{CharacterRange @@@ {{"A", "Z"}, {"a", "z"}, {"0", "9"}},
        Characters["-_.~"]} :> 
   StringJoin[
    "%" <> IntegerString[#, 16] & /@ ToCharacterCode[c, "UTF-8"]]]

(* retrieve watchlist and create hyperlinks *)
Hyperlink[#, "http://en.wikipedia.org/wiki/" <> urlEncode@#] & /@ 
 Replace["title", 
  Flatten[Replace["watchlistraw", 
    Rest@NestWhileList[
      ImportString[
        URLFetch[
         "http://en.wikipedia.org/w/api.php?action=query&list=\
watchlistraw&format=json&wrlimit=500" <> 
          If[Length@# > 1, "&wrcontinue=" <> #[[1, 2, 1, 2, 1, 2]], 
           ""]], "JSON"] &, {}, Length@# != 1 &]], 1]]
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.