6
$\begingroup$

Consider an example Dataset:

icds = Dataset[
  Map[
   Association[
     Thread[Rule[{"brand", "flavor", "parameter", "response"}, #]]] &,
   {
    {"a", "vanilla", 1, RandomReal[NormalDistribution[], 10]},
    {"b", "vanilla", 1.5, RandomReal[NormalDistribution[], 10]},
    {"a", "chocolate", 2, RandomReal[NormalDistribution[], 10]},
    {"b", "chocolate", 2.5, RandomReal[NormalDistribution[], 10]}
    }
   ]
  ]

Now I'm examining "response", grouped according to some function applied to "parameter". Here I'm using Floor but in general the function is expensive enough that I can't calculate it more than once.

rds = icds[GroupBy[Floor[#"parameter"] &], All, "response"]

enter image description here

So far, so good. The expensive result of Floor is stored in the Keys.

Now I'd like to do an operation where the Key is used for something. For instance, I'll make a pair of plots where the key is used as the PlotLabel. But any further Query or Map applied to the dataset seems to apply only to the values and not the keys. I can achieve the right result by exiting Dataset land with a double application of Normal.

Row@Map[
 SmoothHistogram[#[[2]], PlotLabel -> #[[1]]] &,
 Normal[Normal[rds]]
]

enter image description here

That is the desired outcome, but Normal[Normal[]] seems like a workaround. How can the Keys of the Dataset be made available to Map (or similar) without transforming to Normal form?

$\endgroup$
4
$\begingroup$

Histograms can be generated within the dataset using AssociationMap:

icds[
  GroupBy[Floor[#"parameter"] &] /*
  AssociationMap[#[[1]] -> SmoothHistogram[#[[2]], PlotLabel -> #[[1]]] &]
, All
, "response"
]

dataset screenshot

If desired, the histograms can be extracted from the dataset by adding an additional ascending operator to convert the association's values into a Row:

icds[
  GroupBy[Floor[#"parameter"] &] /*
  AssociationMap[#[[1]] -> SmoothHistogram[#[[2]], PlotLabel -> #[[1]]] &] /*
  (Row @ Values @ # &)
, All
, "response"
]

row screenshot


Side Note

One might be tempted to express the row conversion as Values /* Row. This does not work because Values is a descending operator. When descending operators are composed after ascending operators (AssociationMap in this case), the query plan can give surprising results. Here is the plan generated when the conversion is an ascending operator:

Dataset`ShowPlan[
  GroupBy[Floor[#"parameter"] &] /*
  AssociationMap[#[[1]] -> SmoothHistogram[#[[2]], PlotLabel -> #[[1]]] &] /*
  (Row @ Values @ # &)
, All
, "response"
]

(*
  GroupBy[Floor[#parameter]&] /*
  Map[Map[GeneralUtilities`Slice[response]]] /*
  AssociationMap[#1[[1]]->SmoothHistogram[#1[[2]],PlotLabel->#1[[1]]]&] /*
  (Row[Values[#1]]&)
*)

Now observe how the plan is altered when the row conversion composition contains the descending operator Values:

Dataset`ShowPlan[
  GroupBy[Floor[#"parameter"] &] /*
  AssociationMap[#[[1]] -> SmoothHistogram[#[[2]], PlotLabel -> #[[1]]] &] /*
  Values /*
  Row
, All
, "response"
]

(*
  Map[Map[GeneralUtilities`Slice[response]]] /*
  GroupBy[Floor[#parameter]&] /*
  AssociationMap[#1[[1]]->SmoothHistogram[#1[[2]],PlotLabel->#1[[1]]]&] /*
  Values /*
  Row
*)

The latter plan does not generate the desired results (in fact, it generates an error). The v10.0.1 documentation is silent on the meaning of trailing descending operators in a composition, so perhaps it is best to avoid them for the time being.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.