Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have been using Mathematica to analyse some data from the StackExchange API. It is conveniently available in JSON form, which Mathematica interprets as replacement rules. However, some of the rules are nested as the right-hand side of an outer rule, as you can see in the case of badge_counts here:

Import["http://api.stackexchange.com/2.0/users?page=1&order=desc&sort=reputation&site=mathematica",
   "JSON"][[2, 2, 10]]

(* Out[1]= 
{"accept_rate" -> 75, "account_id" -> 395497, "age" -> 41, 
 "badge_counts" -> {"bronze" -> 35, "gold" -> 0, "silver" -> 11}, 
 "creation_date" -> 1326833982, "display_name" -> "Verbeia", 
 "is_employee" -> False, "last_access_date" -> 1331949804, 
 "last_modified_date" -> 1330990001, 
 "link" -> "http://mathematica.stackexchange.com/users/8/verbeia", 
 "location" -> "Sydney, Australia", 
 "profile_image" -> "http://www.gravatar.com/avatar/3df2379fc0221bb0281c0d608542bd84?d=\identicon&r=PG", 
 "reputation" -> 3571, "reputation_change_day" -> 0, 
 "reputation_change_month" -> 857, 
 "reputation_change_quarter" -> 3475, "reputation_change_week" -> 605,
 "reputation_change_year" -> 3475, "user_id" -> 8, 
 "user_type" -> "registered", 
 "website_url" -> "http://www.verbeia.com/mathematica"}
*)

Users without any badges don't have this nested rule. To handle both cases I am doing something like this to get the badge counts in a flattened list, so each row of the final list corresponds to a user of the site:

getUserData[page_Integer] := Module[{ad},
  ad = Import[
     "http://api.stackexchange.com/2.0/users?page=" <> 
      IntegerString[page] <> 
      "&order=desc&sort=reputation&site=mathematica", "JSON"][[2,2]];
  If[Length[ad] > 0,
   Join[Most[#], {"bronze", "silver", "gold"} /. 
       Last[#]] & /@ (({"display_name", "creation_date", "reputation",
          "reputation_change_week", "is_employee", "last_access_date",
          "user_type", "badge_counts"} /. ad) /. 
      "badge_counts" -> {"bronze" -> 0, "silver" -> 0, 
        "gold" -> 0}), {}]
  ]

Which I can then build up into a list of all users and their stats like this:

Monitor[
 pageM = 0;
 resultM = 
  FixedPoint[Join[#, getUserData[++pageM]] &, {}];,
  StringForm["Loading page `` ...", pageM]
 ]

The final output is list of vectors that look like this:

resultM[[10]]
(*Out[2]= {"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 11, 0} *)

My question is: is there a more concise or elegant way to write the getUserData function, without need a Join and three separate ReplaceAll (/.) constructions?

share|improve this question

3 Answers 3

up vote 9 down vote accepted

It is possible to get rid of all but one ReplaceAll. First, you need to remember that ReplaceAll tests each Rule in order, so to deal with the missing "badge_counts" condition, you can simply add that to the list, as follows

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 "badge_counts"} /. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0} }]

where record is one user record. Since, "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0} is listed after the one in the user info, it will only be used if there is no "badge_counts" in the user info. To eliminate the "badge rules", I would remove them using ReplaceRepeated (//.) and one more replacement rule and a final Flatten:

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 "badge_counts"} //. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0},
   Rule[_, a_]:> a}] // Flatten

which returns for your info

{"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 0, 11}

Edit: Originally, I used a slightly different approach instead of Rule[_, a_]:> a. I am reproducing it here, for the curious:

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 "badge_counts"} /. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0}}] // 
Block[{Rule=#2&}, Flatten[#]]&

which relies on the property of Block to temporarily override the behavior of Rule.

Edit 2: Here's one more alternative which should reduce it to a single pass, unlike ReplaceRepeated. This method, also, eliminates the need for the final Flatten

Clear[killRules]
killRules[a : {_Rule ..}] := Sequence @@ a[[All, 2]]

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 killRules["badge_counts"]} /. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0}}]
share|improve this answer

Update: Athanassios correctly pointed out that my attempt to set defaults doesn't work.
The problem appears to be that the absence of the element from the first list (ad) causes the defaults list to never be checked. A condensed exhibit:

ad = {"accept_rate" -> 80, "badge_counts" -> {"bronze" -> 39, "silver" -> 9}};

defaults = {"age" -> Missing[], 
   "badge_counts" -> {"bronze" -> #, "gold" -> #, "silver" -> #} &@
    Missing["NotAvailable"]};

OptionValue[{ad, defaults},
  {"accept_rate", "age", "badge_counts" -> "bronze", "badge_counts" -> "gold"}]

OptionValue::optnf: "Option name "badge_counts" -> "gold" not found in defaults ..."

{80, Missing[], 39, "badge_counts" -> "gold"}

One can use the modified form OptionValue[{}, {ad, defaults}, . . .] to avoid the message but the result is still incorrect. I am sorry I missed this problem when I posted this answer.


It seems to me that this is the correct way to extract values from a nested list of rules:

Data

ad = {"accept_rate" -> 75, "account_id" -> 395497, "age" -> 41, 
   "badge_counts" -> {"bronze" -> 35, "gold" -> 0, "silver" -> 11}, 
   "creation_date" -> 1326833982, "display_name" -> "Verbeia", 
   "is_employee" -> False, "last_access_date" -> 1331949804, 
   "last_modified_date" -> 1330990001, 
   "link" -> "http://mathematica.stackexchange.com/users/8/verbeia", 
   "location" -> "Sydney, Australia", 
   "profile_image" -> 
    "http://www.gravatar.com/avatar/3df2379fc0221bb0281c0d608542bd84?\
d=\ identicon&r=PG", "reputation" -> 3571, 
   "reputation_change_day" -> 0, "reputation_change_month" -> 857, 
   "reputation_change_quarter" -> 3475, 
   "reputation_change_week" -> 605, "reputation_change_year" -> 3475, 
   "user_id" -> 8, "user_type" -> "registered", 
   "website_url" -> "http://www.verbeia.com/mathematica"};

Defaults

include default values for any rules that might be missing

defaults = {"badge_counts" ->
      {"bronze" -> #, "gold" -> #, "silver" -> #}& @ Missing["NotAvailable"]};

Extraction

OptionValue[{ad, defaults}, {
   "display_name",
   "creation_date",
   "reputation",
   "reputation_change_week",
   "is_employee", "last_access_date",
   "user_type",
   "badge_counts" -> "bronze",
   "badge_counts" -> "silver",
   "badge_counts" -> "gold"
}]

{"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 11, 0}

share|improve this answer
    
+1, I hadn't thought of OptionValue. –  rcollyer Mar 17 '12 at 13:13
    
@rcollyer is there any way to set a default value for any missing option? That would make this more elegant. Nevertheless I think this is the best way to deal with nested rules; getting a value for e.g. a -> b -> c -> d is pretty hairy otherwise. –  Mr.Wizard Mar 17 '12 at 13:19
    
The only way I'm a aware of is via SetOptions which isn't exactly what you want. –  rcollyer Mar 17 '12 at 17:02
    
@rcollyer how would you do it via SetOptions? –  Mr.Wizard Mar 17 '12 at 17:07
1  
@Mr.Wizard your solution is great but it seems to me default values work only for non-nested values. I have tested this by creating another set where "badge_counts"->"gold" value is missing and I am getting an error message Option name badge_counts ->gold not found in defaults for \ {{display_name ->Stones ,accept_rate ->80,account_id \ ->495497,badge_counts ->{bronze ->39,silver ->9}},{age \ ->Missing[],badge_counts ->{bronze ->Missing[NotAvailable ],gold \ ->Missing[NotAvailable ],silver ->Missing[NotAvailable ]}}}. >> –  Athanassios Jul 18 at 8:36

The introduction of Assocation brings a powerful new way to handle this problem.

  • Version 10.2 provides for import of JSON as nested associations without use of ToAssociations by using the format "RawJSON".

 

(* archive data *)

ad = {"accept_rate" -> 75, "account_id" -> 395497, "age" -> 41, 
   "badge_counts" -> {"bronze" -> 35, "gold" -> 0, "silver" -> 11}, 
   "creation_date" -> 1326833982, "display_name" -> "Verbeia", "is_employee" -> False, 
   "last_access_date" -> 1331949804, "last_modified_date" -> 1330990001, 
   "link" -> "http://mathematica.stackexchange.com/users/8/verbeia", 
   "location" -> "Sydney, Australia", 
   "profile_image" -> 
    "http://www.gravatar.com/avatar/3df2379fc0221bb0281c0d608542bd84?d=\ identicon&r=PG", 
   "reputation" -> 3571, "reputation_change_day" -> 0, "reputation_change_month" -> 857, 
   "reputation_change_quarter" -> 3475, "reputation_change_week" -> 605, 
   "reputation_change_year" -> 3475, "user_id" -> 8, "user_type" -> "registered", 
   "website_url" -> "http://www.verbeia.com/mathematica"};

Convert to nested Association (reference: Converting hierarchies of rules to associations)

Needs["GeneralUtilities`"]
asc = ToAssociations[ad];

Query a value:

asc["badge_counts", "silver"]
11

Missing values are automatically handled:

asc["badge_counts", "platinum"]
Missing["KeyAbsent", "platinum"]

Perform multiple lookups at once:

lookups = {{"display_name"}, {"creation_date"}, {"reputation"}, \
{"reputation_change_week"}, {"is_employee"}, {"last_access_date"}, {"user_type"}, \
{"badge_counts", "bronze"}, {"badge_counts", "silver"}, {"badge_counts", 
    "gold"}, {"badge_counts", "platinum"}};

Extract[asc, lookups]
{"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 11, 0, 
 Missing["KeyAbsent", "platinum"]}

More advanced queries are easy. With current data:

data = Import[
    "http://api.stackexchange.com/2.0/users?page=1&order=desc&sort=reputation&site=\
mathematica", "JSON"] // ToAssociations;

Find my current bronze badge count:

data // Query["items", Select @ MemberQ @ "Mr.Wizard", "badge_counts", "bronze"]
{632}
share|improve this answer
1  
+1 In 10.2, we can use Import[..., "RawJSON"] which automatically returns associations. –  WReach Jul 18 at 19:15
    
@WReach Yes, I noted that in (87879) -- I guess I should put it here too. –  Mr.Wizard Jul 19 at 2:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.