5
$\begingroup$

Wikipedia's "Edit Watchlist" page (for logged in users) has a list of hyperlinks to the user's watchlist (here is an example from my account).

enter image description here

I've tried to use URLFetch and URLSave as follows

URLFetch["http://en.wikipedia.org/wiki/Special:EditWatchlist"]
URLSave["http://en.wikipedia.org/wiki/Special:EditWatchlist","test.html"]

but it seems that they cannot grab the links in the watchlist. I thought URLSave is the same as saving a webpage as a HTML file by hand in Internet Explorer... Am I wrong?

Even Import, with 

Import["http://en.wikipedia.org/wiki/Special:EditWatchlist","Hyperlinks"]

(and also with "Source"), cannot get the hyperlinks.

How can I log into Wikipedia and get these links using Mathematica?

Improve this question
$\endgroup$
14
  • $\begingroup$ Do you have to login to that site? $\endgroup$
    – Mike Honeychurch
    Aug 11, 2013 at 4:32
  • $\begingroup$ @MikeHoneychurch Yes, logged user could favorite pages. $\endgroup$
    – HyperGroups
    Aug 11, 2013 at 4:43
  • 1
    $\begingroup$ Well if you need to login then Import is never going to work. You need to show us what code you have tried -- obviously replace actual user name and password with generic strings. $\endgroup$
    – Mike Honeychurch
    Aug 11, 2013 at 5:20
  • 3
    $\begingroup$ Here is how to retrieve a Wikipedia user's watchlist by using the MediaWiki API. My watchlist has over 10,000 articles and it gets the full list in a few seconds (the API paging gives you 500 results at a time). pastebin.com/0Q6LJM9r $\endgroup$
    – Michael Hale
    Aug 11, 2013 at 10:34
  • 1
    $\begingroup$ @SjoerdC.deVries One reason is useful now, and the other might be more useful in the future. For now, I occasionally like to go on vandalism cleaning sprees to relieve stress. It is fun to click undo when people are damaging something so valuable. I hope in the future to be able to use my watchlist change feed to keep up to date on very specific research topics (at a finer level of topic granularity than science news sites). I'll add my response as an answer. $\endgroup$
    – Michael Hale
    Aug 11, 2013 at 17:32

1 Answer 1

Reset to default
10
$\begingroup$

The MediaWiki API is good, fast, well-tested, and is usable within Mathematica for tasks that require authentication by using URLFetch. To login you need to send two POSTs. The second sends back a token that you receive from the first request. Then Mathematica automatically sets $HTTPCookies. This login function will be added to the utilities for the Wikicode project. Then you can request your watchlist once your Mathematica client has been authenticated. I use a loop to retrieve paged results if your watchlist is longer than 500 pages.

(* Returns True or False for success or failure *)
Login[username_String, password_String] := 
 Module[{url = "http://en.wikipedia.org/w/api.php", 
   parameters = {"action" -> "login", "lgname" -> username, 
     "lgpassword" -> password, "format" -> "json"}}, 
  MatchQ["result" /. ("login" /. 
      ImportString[
       If[MatchQ[#, "NeedToken"], 
          URLFetch[url, "Method" -> "POST", 
           "Parameters" -> Append[parameters, "lgtoken" -> #2]], 
          Return@False] & @@ ({"result", 
           "token"} /. ("login" /. 
            ImportString[
             URLFetch[url, "Method" -> "POST", 
              "Parameters" -> parameters], "JSON"])), "JSON"]), 
   "Success"]]

urlEncode[string_String] := 
 StringReplace[string, 
  c : Except@
     Flatten@{CharacterRange @@@ {{"A", "Z"}, {"a", "z"}, {"0", "9"}},
        Characters["-_.~"]} :> 
   StringJoin[
    "%" <> IntegerString[#, 16] & /@ ToCharacterCode[c, "UTF-8"]]]

(* retrieve watchlist and create hyperlinks *)
Hyperlink[#, "http://en.wikipedia.org/wiki/" <> urlEncode@#] & /@ 
 Replace["title", 
  Flatten[Replace["watchlistraw", 
    Rest@NestWhileList[
      ImportString[
        URLFetch[
         "http://en.wikipedia.org/w/api.php?action=query&list=\
watchlistraw&format=json&wrlimit=500" <> 
          If[Length@# > 1, "&wrcontinue=" <> #[[1, 2, 1, 2, 1, 2]], 
           ""]], "JSON"] &, {}, Length@# != 1 &]], 1]]
Improve this answer
$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.