Is there a way to operate on Dataset across levels to apply a function f to a key -> value pair in an Association as follows:

pivotApply[f_, key_ -> value_] :=  key -> f[key, value]

For example, take the Titanic dataset and pivot up "class":

titanicClass = titanic[GroupBy[Key@"class"], KeyDrop[Key@"class"]];

For this problem, assume only titanicClass. "class" key has been deleted and so can't referenced. The motivation is tree-structured data where it would be inefficient to insert the keys {"1st"...} at the deeper level.

Here foo represents a "client" function to be applied to deeper level data but depenent on class values.

EDIT // #age >= 18 &

foo["1st", data_] := Dataset[data][All, #age >= 18 &];
foo["2nd", data_] := Dataset[data][All, "sex"];
foo["3rd", data_] := Dataset[data][All, "survived"];

Is there a way to avoid inefficient downcasting to Normal and back to Dataset to use AssociationMap? Also note there are Datasets nested in a larger one, which should also be cast to Lists (since there is an outer Dataset).

AssociationMap[pivotApply[foo, #] &, titanicClass // Normal] // Dataset

enter image description here

Is there a way to query Datasets across levels like this? This seems extremely roundabout, but haven't figured an implementation using, say Keys and Values and related Dataset operators.


3 Answers 3


Note: the question has evolved, as has my understanding of it. My response has been changed completely, although I have kept my earlier responses below.

If we can drop the requirement to use AssociationMap, pivotApply and foo, then we can generate the desired output directly using subqueries:

titanicClass @
  { "1st" -> Query[All, #age > 18&]
  , "2nd" -> Query[All, "sex"]
  , "3rd" -> Query[All, "survived"]

dataset query screenshot

As requested, the inner lists are not wrapped in Dataset.

Original Responses

If we are willing to dispense with the requirement to use AssociationMap, here are a couple of options to "query across levels" in the described manner.

If we introduce bar as a slightly rewritten form of the function foo from the question...

bar[a_] := bar[a, a@"class"]
bar[a_, "1st"] := a@"age"
bar[a_, "2nd"] := a@"sex"
bar[a_, "3rd"] := a@"survived"

... then the function can be called directly within the dataset query after grouping:

titanic[GroupBy @ Key @ "class", All, bar]

dataset screenshot

Alternatively, we could dispense with the function altogether and extract the various components with an inline Switch statement.

  GroupBy @ Key @ "class"
, All
, Switch[#class, "1st", #age, "2nd", #sex, "3rd", #survived]&

dataset screenshot

Edit: Starting From titanicClass

My original response mistakenly assumed that titanic was the input data. The question has been revised to emphasize that titanicClass is actually the input data. Under that assumption, we can generate the described output as follows...

First, we supplement the definition of pivotApply to include a one-argument "lifting" form of the operator:

pivotApply[f_] := pivotApply[f, #]&

Next, we create the function baz which is yet another variation upon foo:

baz["1st", a_] := a[[All, "age"]]
baz["2nd", a_] := a[[All, "sex"]]
baz["3rd", a_] := a[[All, "survived"]]

Now, we can generate the required output using AssociationMap:

titanicClass[AssociationMap @ pivotApply @ baz]

dataset screenshot

Once again, we can dispense with the function baz and express the query inline if desired:

titanicClass @ AssociationMap @ pivotApply[
  #2[[All, # /. {"1st" -> "age", "2nd" -> "sex", "3rd" -> "survived"}]]&

dataset screenshot

  • $\begingroup$ I would accept an approach that avoids AssociationMap, however, I reworded question to highlight that "class" is not an available key. (Assume a tree structure where it's not available to begin with and it's too inefficient to insert it into large HRS data at the deeper level. $\endgroup$ Commented Jul 25, 2014 at 17:08
  • $\begingroup$ Please see the new section in my response, "starting from titanicClass". $\endgroup$
    – WReach
    Commented Jul 25, 2014 at 19:07
  • $\begingroup$ Your solution (with 'baz') references the data via Part, eg a[[All,"age"]] but it doesn't leverage Dataset's named slot semantics. How to pass a function eg: #age >= 18 &? (This would be a variation on foo). Also, can this be done with your inline version? $\endgroup$ Commented Jul 26, 2014 at 18:13
  • $\begingroup$ Okay, I think I finally see what you are after -- fingers crossed :). Please see my all-new response. $\endgroup$
    – WReach
    Commented Jul 26, 2014 at 19:25
  • $\begingroup$ This is a great solution WReach, thanks. I will accept if you factor it through something like pivotApply b/c in general the top-level keys {"1st"...} are not know a priori, and so the Query's should be threaded programmatically. $\endgroup$ Commented Jul 26, 2014 at 22:03

Using this helper function

keyStrip[Key[k_]|{Key[k_]}] := k;
keyStrip[expr_] := expr;

you can achieve what you asked for as

titanicClass[MapIndexed[foo[keyStrip[First@#2], #1] &]]

The general issue about being able to reference some elements of the parent structures inside child structures during the traversal (in the queries) is more complicated. As of now, Dataset query language does not, as far as I know, easily allow this, but such capabilities may be added in the future.


Here is a version which does not use the foo function, but uses Lookup and specifies the lookup explicitly via an association:

      <|"1st" -> "age", "2nd" -> "sex", "3rd" -> "survived"|>[keyStrip[#2]]
    ] &
  • $\begingroup$ This works but was hoping to avoid wrapping "data" in Dataset for each call. Is there a way to make this approach work inline? $\endgroup$ Commented Jul 25, 2014 at 17:42
  • $\begingroup$ @alancalvitti I was using your own function foo. Now you say you don't want to use it. Your question, precisely, was "Is there a way to avoid inefficient downcasting to Normal and back to Dataset to use AssociationMap?", asked about your last line of code, - and this was precisely what I was answering. I will revisit this later if I have the time. Try to formulate your questions better, at least if you care about me answering them - this is not the first time I found your formulations vague, while I usually do not mis-interpret questions, and certainly have other things to do as well. $\endgroup$ Commented Jul 26, 2014 at 8:36
  • $\begingroup$ I always appreciate your insight. I slightly reworded my Q. The function foo stands for a "client" function to be applied to the data, so it must be used in the answer. The question is how to apply it, while retaining Dataset named slot semantics, but as operators without having to create Dataset and subsequently Normal'izing. Let me know if this clears it up. $\endgroup$ Commented Jul 26, 2014 at 17:35
  • $\begingroup$ @alancalvitti See if my edit works for you. $\endgroup$ Commented Jul 26, 2014 at 18:12
  • $\begingroup$ As per my comment to WReach, foo is using named slot semantics - that's the reason for casting to Dataset. What if foo["1st", data_] := Dataset[data][All, {"age" -> (# >= 18 &)}] ? $\endgroup$ Commented Jul 26, 2014 at 18:26
titanic[GroupBy[Key@"class" -> (Switch[#class,
    "1st", #age,
    "2nd", #sex,
    "3rd", #survived] &)
]] // Normal // Dataset

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.