29
$\begingroup$

I have been using Mathematica to analyse some data from the StackExchange API. It is conveniently available in JSON form, which Mathematica interprets as replacement rules. However, some of the rules are nested as the right-hand side of an outer rule, as you can see in the case of badge_counts here:

Import["http://api.stackexchange.com/2.0/users?page=1&order=desc&sort=reputation&site=mathematica",
   "JSON"][[2, 2, 10]]

(* Out[1]= 
{"accept_rate" -> 75, "account_id" -> 395497, "age" -> 41, 
 "badge_counts" -> {"bronze" -> 35, "gold" -> 0, "silver" -> 11}, 
 "creation_date" -> 1326833982, "display_name" -> "Verbeia", 
 "is_employee" -> False, "last_access_date" -> 1331949804, 
 "last_modified_date" -> 1330990001, 
 "link" -> "http://mathematica.stackexchange.com/users/8/verbeia", 
 "location" -> "Sydney, Australia", 
 "profile_image" -> "http://www.gravatar.com/avatar/3df2379fc0221bb0281c0d608542bd84?d=\identicon&r=PG", 
 "reputation" -> 3571, "reputation_change_day" -> 0, 
 "reputation_change_month" -> 857, 
 "reputation_change_quarter" -> 3475, "reputation_change_week" -> 605,
 "reputation_change_year" -> 3475, "user_id" -> 8, 
 "user_type" -> "registered", 
 "website_url" -> "http://www.verbeia.com/mathematica"}
*)

Users without any badges don't have this nested rule. To handle both cases I am doing something like this to get the badge counts in a flattened list, so each row of the final list corresponds to a user of the site:

getUserData[page_Integer] := Module[{ad},
  ad = Import[
     "http://api.stackexchange.com/2.0/users?page=" <> 
      IntegerString[page] <> 
      "&order=desc&sort=reputation&site=mathematica", "JSON"][[2,2]];
  If[Length[ad] > 0,
   Join[Most[#], {"bronze", "silver", "gold"} /. 
       Last[#]] & /@ (({"display_name", "creation_date", "reputation",
          "reputation_change_week", "is_employee", "last_access_date",
          "user_type", "badge_counts"} /. ad) /. 
      "badge_counts" -> {"bronze" -> 0, "silver" -> 0, 
        "gold" -> 0}), {}]
  ]

Which I can then build up into a list of all users and their stats like this:

Monitor[
 pageM = 0;
 resultM = 
  FixedPoint[Join[#, getUserData[++pageM]] &, {}];,
  StringForm["Loading page `` ...", pageM]
 ]

The final output is list of vectors that look like this:

resultM[[10]]
(*Out[2]= {"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 11, 0} *)

My question is: is there a more concise or elegant way to write the getUserData function, without need a Join and three separate ReplaceAll (/.) constructions?

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
10
$\begingroup$

It is possible to get rid of all but one ReplaceAll. First, you need to remember that ReplaceAll tests each Rule in order, so to deal with the missing "badge_counts" condition, you can simply add that to the list, as follows

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 "badge_counts"} /. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0} }]

where record is one user record. Since, "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0} is listed after the one in the user info, it will only be used if there is no "badge_counts" in the user info. To eliminate the "badge rules", I would remove them using ReplaceRepeated (//.) and one more replacement rule and a final Flatten:

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 "badge_counts"} //. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0},
   Rule[_, a_]:> a}] // Flatten

which returns for your info

{"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 0, 11}

Edit: Originally, I used a slightly different approach instead of Rule[_, a_]:> a. I am reproducing it here, for the curious:

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 "badge_counts"} /. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0}}] // 
Block[{Rule=#2&}, Flatten[#]]&

which relies on the property of Block to temporarily override the behavior of Rule.

Edit 2: Here's one more alternative which should reduce it to a single pass, unlike ReplaceRepeated. This method, also, eliminates the need for the final Flatten

Clear[killRules]
killRules[a : {_Rule ..}] := Sequence @@ a[[All, 2]]

{"display_name", "creation_date", "reputation", "reputation_change_week",
 "is_employee", "last_access_date", "user_type", 
 killRules["badge_counts"]} /. Flatten[{ record, 
   "badge_counts" -> {"bronze" -> 0, "silver" -> 0, "gold" -> 0}}]
Improve this answer
$\endgroup$
19
$\begingroup$

Update: Athanassios correctly pointed out that my attempt to set defaults doesn't work.
The problem appears to be that the absence of the element from the first list (ad) causes the defaults list to never be checked. A condensed exhibit:

ad = {"accept_rate" -> 80, "badge_counts" -> {"bronze" -> 39, "silver" -> 9}};

defaults = {"age" -> Missing[], 
   "badge_counts" -> {"bronze" -> #, "gold" -> #, "silver" -> #} &@
    Missing["NotAvailable"]};

OptionValue[{ad, defaults},
  {"accept_rate", "age", "badge_counts" -> "bronze", "badge_counts" -> "gold"}]

OptionValue::optnf: "Option name "badge_counts" -> "gold" not found in defaults ..."

{80, Missing[], 39, "badge_counts" -> "gold"}

One can use the modified form OptionValue[{}, {ad, defaults}, . . .] to avoid the message but the result is still incorrect. I am sorry I missed this problem when I posted this answer.

It seems to me that this is the correct way to extract values from a nested list of rules:

Data

ad = {"accept_rate" -> 75, "account_id" -> 395497, "age" -> 41, 
   "badge_counts" -> {"bronze" -> 35, "gold" -> 0, "silver" -> 11}, 
   "creation_date" -> 1326833982, "display_name" -> "Verbeia", 
   "is_employee" -> False, "last_access_date" -> 1331949804, 
   "last_modified_date" -> 1330990001, 
   "link" -> "https://mathematica.stackexchange.com/users/8/verbeia", 
   "location" -> "Sydney, Australia", 
   "profile_image" -> 
    "http://www.gravatar.com/avatar/3df2379fc0221bb0281c0d608542bd84?\
d=\ identicon&r=PG", "reputation" -> 3571, 
   "reputation_change_day" -> 0, "reputation_change_month" -> 857, 
   "reputation_change_quarter" -> 3475, 
   "reputation_change_week" -> 605, "reputation_change_year" -> 3475, 
   "user_id" -> 8, "user_type" -> "registered", 
   "website_url" -> "http://www.verbeia.com/mathematica"};

Defaults

include default values for any rules that might be missing

defaults = {"badge_counts" ->
      {"bronze" -> #, "gold" -> #, "silver" -> #}& @ Missing["NotAvailable"]};

Extraction

OptionValue[{ad, defaults}, {
   "display_name",
   "creation_date",
   "reputation",
   "reputation_change_week",
   "is_employee", "last_access_date",
   "user_type",
   "badge_counts" -> "bronze",
   "badge_counts" -> "silver",
   "badge_counts" -> "gold"
}]

{"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 11, 0}

Improve this answer
$\endgroup$
9
  • $\begingroup$ +1, I hadn't thought of OptionValue. $\endgroup$
    – rcollyer
    Commented Mar 17, 2012 at 13:13
  • $\begingroup$ @rcollyer is there any way to set a default value for any missing option? That would make this more elegant. Nevertheless I think this is the best way to deal with nested rules; getting a value for e.g. a -> b -> c -> d is pretty hairy otherwise. $\endgroup$
    – Mr.Wizard
    Commented Mar 17, 2012 at 13:19
  • $\begingroup$ The only way I'm a aware of is via SetOptions which isn't exactly what you want. $\endgroup$
    – rcollyer
    Commented Mar 17, 2012 at 17:02
  • $\begingroup$ @rcollyer how would you do it via SetOptions? $\endgroup$
    – Mr.Wizard
    Commented Mar 17, 2012 at 17:07
  • 2
    $\begingroup$ @Mr.Wizard your solution is great but it seems to me default values work only for non-nested values. I have tested this by creating another set where "badge_counts"->"gold" value is missing and I am getting an error message Option name badge_counts ->gold not found in defaults for \ {{display_name ->Stones ,accept_rate ->80,account_id \ ->495497,badge_counts ->{bronze ->39,silver ->9}},{age \ ->Missing[],badge_counts ->{bronze ->Missing[NotAvailable ],gold \ ->Missing[NotAvailable ],silver ->Missing[NotAvailable ]}}}. >> $\endgroup$
    – Athanassios
    Commented Jul 18, 2015 at 8:36
6
$\begingroup$

The introduction of Assocation brings a powerful new way to handle this problem.

  • Version 10.2 provides for import of JSON as nested associations without use of ToAssociations by using the format "RawJSON".

 

(* archive data *)

ad = {"accept_rate" -> 75, "account_id" -> 395497, "age" -> 41, 
   "badge_counts" -> {"bronze" -> 35, "gold" -> 0, "silver" -> 11}, 
   "creation_date" -> 1326833982, "display_name" -> "Verbeia", "is_employee" -> False, 
   "last_access_date" -> 1331949804, "last_modified_date" -> 1330990001, 
   "link" -> "https://mathematica.stackexchange.com/users/8/verbeia", 
   "location" -> "Sydney, Australia", 
   "profile_image" -> 
    "http://www.gravatar.com/avatar/3df2379fc0221bb0281c0d608542bd84?d=\ identicon&r=PG", 
   "reputation" -> 3571, "reputation_change_day" -> 0, "reputation_change_month" -> 857, 
   "reputation_change_quarter" -> 3475, "reputation_change_week" -> 605, 
   "reputation_change_year" -> 3475, "user_id" -> 8, "user_type" -> "registered", 
   "website_url" -> "http://www.verbeia.com/mathematica"};

Convert to nested Association (reference: Converting hierarchies of rules to associations)

Needs["GeneralUtilities`"]
asc = ToAssociations[ad];

Query a value:

asc["badge_counts", "silver"]

11

Missing values are automatically handled:

asc["badge_counts", "platinum"]

Missing["KeyAbsent", "platinum"]

Perform multiple lookups at once:

lookups = {{"display_name"}, {"creation_date"}, {"reputation"}, \
{"reputation_change_week"}, {"is_employee"}, {"last_access_date"}, {"user_type"}, \
{"badge_counts", "bronze"}, {"badge_counts", "silver"}, {"badge_counts", 
    "gold"}, {"badge_counts", "platinum"}};

Extract[asc, lookups]

{"Verbeia", 1326833982, 3571, 605, False, 1331949804, "registered", 35, 11, 0, 
 Missing["KeyAbsent", "platinum"]}

More advanced queries are easy. With current data:

data = Import[
    "http://api.stackexchange.com/2.0/users?page=1&order=desc&sort=reputation&site=\
mathematica", "JSON"] // ToAssociations;

Find my current bronze badge count:

data // Query["items", Select @ MemberQ @ "Mr.Wizard", "badge_counts", "bronze"]

{632}
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ +1 In 10.2, we can use Import[..., "RawJSON"] which automatically returns associations. $\endgroup$
    – WReach
    Commented Jul 18, 2015 at 19:15
  • $\begingroup$ @WReach Yes, I noted that in (87879) -- I guess I should put it here too. $\endgroup$
    – Mr.Wizard
    Commented Jul 19, 2015 at 2:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.