8
$\begingroup$

I've got some code that has to extract a lot of information, including some statistics, from an association. There's a lot of repetition in my code and I'm hoping that there's some way to simplify it by using Map. The following is a simplified version of what I'm doing.

data1 = {{"a", "x", "k", 1, 3}, {"a", "x", "k", 2, 4}, {"a", "x", "k", 7, 8},
    {"b", "z", "m", 11, 33}, {"b", "z", "m", 10, 23},
    {"c", "w", "n", 3, 100},
    {"g", "y", "p", 2, 7}, {"g", "y", "p", 23, 31}, 
    {"g", "y", "p", 19, 13}, {"g", "y", "p", 201, 55}};
dataAssoc = 
  AssociationThread[{"zone", "type", "name", "weight", "cost"}, #] & /@ data1;
grouped = GroupBy[dataAssoc, Key["zone"]];
extract = {First[#[[All, Key["zone"]]]],
   First[#[[All, Key["name"]]]],
   Max[#[[All, Key["weight"]]]],
   Min[#[[All, Key["weight"]]]],
   Mean[#[[All, Key["weight"]]]],
   Max[#[[All, Key["cost"]]]],
   Min[#[[All, Key["cost"]]]],
   Mean[#[[All, Key["cost"]]]]} & /@ grouped;

extract

  <|"a" -> {"a", "k", 7, 1, 10/3, 8, 3, 5}, 
 "b" -> {"b", "m", 11, 10, 21/2, 33, 23, 28}, 
 "c" -> {"c", "n", 3, 3, 3, 100, 100, 100}, 
 "g" -> {"g", "p", 201, 2, 245/4, 55, 7, 53/2}|>

My real work has a lot more fields, and more may be added in the future. It seems to me that there should be some way to map the Min, Max, Mean over a list of fields rather than repeating it over for each numeric field I want this for. Also, when I'm pulling out specific elements using First, it also seems like there should be a way to iterate over a list of fields.

I can't figure out how to do a Map within a Map, which is what I think I'd need since this whole thing gets mapped over my Association.

$\endgroup$
9
$\begingroup$

This is a bit cleaner:

Query[
  GroupBy[{#zone, #name} &],
  Query[Transpose /* Query[All, {Min, Max, Mean}], {"weight", "cost"}]] @ dataAssoc

enter image description here

Map[Flatten, Thread@{Keys@%, Values@Values@%}]

enter image description here

$\endgroup$
  • $\begingroup$ Much, much cleaner! But could you explain in what order things are happening in your Query function? I'm trying to understand it, primarily by taking it apart and seeing how it works one piece at a time. For example in the last line of your first section, The Query applies to dataAssoc. But does Query[All, {Min, Max, Mean} etc., happen first? I'm having a lot of trouble parsing this. In my testing I have y1 = Query[All, {Min, Max, Mean}, "EPI"]@ y; using my actual data. This works. You've combined this in another query to get statistics for a bunch of keys at once. $\endgroup$ – Mitchell Kaplan Sep 23 '15 at 21:32
  • $\begingroup$ It is worth reading the the complete documentation page on Query -- there are a lot of nuanced options and it is quite powerful. A relevant part is: 'In Query[operator1,…][expr], the are applied at successively deeper levels in expr, but any given one may be applied either while "descending" into expr or while "ascending" out of it. In general, part specifications and filtering operators are "descending" operators. Aggregation operators, subquery operators, and arbitrary functions are "ascending" operators.' $\endgroup$ – mfvonh Sep 24 '15 at 0:57
  • $\begingroup$ So as constructed it does this: 1) GroupBy (desc), 2) skips Query[Transpose...] (asc) [i.e. treats it like All], 3) applies {"weight", "cost"} to every row within each GroupBy partition, 4) now steps up and hits the Query[Transpose...] function. So it 5) transposes the rows to <|weight -> {...}, cost -> {...}|> then 6) maps over All (= weight and cost), and applies Min, Max, and Mean to each group. So that is what we are left with as the value of each groupby partition. Then the next line just deconstructs it to the flat structure you want. $\endgroup$ – mfvonh Sep 24 '15 at 1:06
  • $\begingroup$ The use of Query several times was to force things to be ascending. My first pass was to use Transpose /* Map[{Min, Max, Mean}] but that was treated as descending, so I just reformed it as query because I was lazy. Likely it could be made more elegant. $\endgroup$ – mfvonh Sep 24 '15 at 1:07
  • $\begingroup$ It is nice to use the Query operator this way when you are just poking around, but @WReach's answer is the same idea but executed in a way that is much more readable. That way is far better if this is code you will have to maintain. $\endgroup$ – mfvonh Sep 24 '15 at 1:10
4
$\begingroup$

By using Query along with a couple of small helper functions that generate subqueries, we can get a pretty direct expression of the requirement:

first[key_] := Query[First, key]
maxMinMean[key_] := Sequence @@ Thread[Query[{Max, Min, Mean}, key]]

dataAssoc // Query[
  GroupBy["zone"]
, Join[first /@ {"zone", "name"}, maxMinMean /@ {"weight", "cost"}]
]

(* <|"a" -> {"a", "k", 7, 1, 10/3, 8, 3, 5}, 
     "b" -> {"b", "m", 11, 10, 21/2, 33, 23, 28}, 
     "c" -> {"c", "n", 3, 3, 3, 100, 100, 100}, 
     "g" -> {"g", "p", 201, 2, 245/4, 55, 7, 53/2}|> *)

first generates a subquery to extract the first value of a key:

first["zone"]
(* Query[First, "zone"] *)

maxMinMean generates a spliced triple of subqueries to extract the numeric aggregations:

{ maxMinMean["weight"] }
(* {Query[Max, "weight"], Query[Min, "weight"], Query[Mean, "weight"]} *)

The final full query is generated using these helpers:

Query[
  GroupBy["zone"]
, Join[first /@ {"zone", "name"}, maxMinMean /@ {"weight", "cost"}]
]

(* Query[
     GroupBy["zone"]
   , { Query[First, "zone"]
     , Query[First, "name"]
     , Query[Max, "weight"], Query[Min, "weight"], Query[Mean, "weight"]
     , Query[Max, "cost"], Query[Min, "cost"], Query[Mean, "cost"]
     }
   ] *)

When executed against the original association list, this generates the desired result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.