Getting Mathematica.SE reputation data

Question

I'm asking the following on behalf of Mr.Wizard.

On StackOverflow I used the old v1.0 SO API and Mathematica to obtain user reputation data. Now we are no longer a tag on StackOverflow and the v2.1 API is the current top dog, can we still use similar techniques to get the reputation data here?

Answer 1

Certainly. There have indeed been some changes to certain tags in the 2.1 API and also to the method to sort out how many pages to import. And, since we are now a full site under StackExchange, the server has changed and a site parameter was necessary.

I did a bit of clean-up, hardened the code a little to accommodate connection failures, made the basic JSON import support for Mathematica v7 transparent (so the same code can be used for v7 and v8 now) and also got a new API key.

Code is at the bottom of this post. The result is the following graph with tooltips to identify the various posters. The text annotation in this picture has been done manually. While this plot only shows the top-10 users the getRepChanges[userID] function can be used to get the reputation changes of any user. Note that this gets you the changes, so you have to Accumulate these to get curves similar to those in the figure.

[Credits go to rm -rf for spotting errors in the API output and working with me to find a solution]

If[$VersionNumber < 8,
 Unprotect[Import];
 Import[path_, "JSON"] := 
  ImportString[
   StringReplace[Import[path, "Text"], {"://" -> "://", 
     "\":" -> "\"->", "[" -> "{", "]" -> "}", "true" -> "True", 
     "false" -> "False"}], "NB"];
 Protect[Import];
 ]

getRepChanges[userID_Integer] :=
 Module[{pageRes = $Failed, finalRes = {}, hasMore = True, page = 1},
      While[hasMore,
       While[
        (pageRes = 
           Quiet@Import[
             "http://api.stackexchange.com/2.1/users/" <> 
              ToString[userID] <> 
              "/reputation-history?key=ESD8XZahB6LafRHhFH8cZQ((&pagesize=100&page=" <> ToString[page] <> "&site=mathematica", "JSON"]) == $Failed,
    PrintTemporary["Import Failure. Retrying in 10 secs..."];
    Pause[10];
    ];
   page++;
   hasMore = ToExpression["has_more" /. pageRes];
   AppendTo[
    finalRes, ({"creation_date", 
        "reputation_change"} /. ("items" /. pageRes)) /. {"creation_date" ->
        1350671864, "reputation_change" -> 0}];
   ];
  {DateList[#[[1]] + AbsoluteTime["January 1, 1970"]], #[[2]]} & /@ 
    Partition[Flatten[finalRes], 2] // Sort
  ]

topAnswerers = ({"display_name", "user_id", 
       "profile_image"} /. #) & /@ ("items" /. 
     Import["http://api.stackexchange.com/2.1/users?page=1&key=\
ESD8XZahB6LafRHhFH8cZQ((&pagesize=10&order=desc&sort=reputation&site=\
mathematica", "JSON"]);
topAnswerers = {#, #2, Import[#3]} & @@@ topAnswerers;

Monitor[
  repChangesTopUsers = (userID = #; getRepChanges[#]) & /@ 
    topAnswerers[[All, 2]],
  ProgressIndicator[
   Position[topAnswerers[[All, 2]], userID][[1, 1]]/
    Length[topAnswerers]]
  ];

accRepChange = {#[[All, 1]], Accumulate[#[[All, 2]]]}\[Transpose] & /@
    repChangesTopUsers;

pl = DateListPlot[
  Tooltip @@@ Transpose[{accRepChange, Column[#] & /@ topAnswerers}], 
  Joined -> True, Mesh -> None, ImageSize -> 1000, 
  BaseStyle -> {FontFamily -> "Arial-Bold", FontSize -> 16}, 
  DateTicksFormat -> {"MonthNameShort", " ", "Year"}, 
  GridLines -> {True, True}, 
  FrameLabel -> (Style[#, FontSize -> 18] & /@ {"Date", "Reputation", 
      "Top-10 answerers", ""}), 
  PlotRange -> {{{2012, 1, 1}, All}, Automatic}, 
  FrameTicks -> {{Automatic, 
     Automatic}, {{2012, #} & /@ Range[1, 12, 2], Automatic}}]