Using Select on aggregate properties in a hierarchical dataset (association of associations)

Question

I have a hierarchical dataset, called finalData, like this, with 31 entries per 'outer' association.

What I want to do is select the outer associations (let's call them 'chunks') based on calculated properties of the chunk. So, for example, I might want to keep all chunks in which the minimum #Interval value is >=900. If this was a nested list, and given that Interval is the 9th column in each subarray, I would do

Select[finalData,Min[#[[All,9]]]>=900&]

Simple! But can this be done within the Dataset paradigm, using a query based on keys? FYI I ca think of plenty of 'hybrid' methods in which one does the test calculation first using something like

finalData2[All, Min[#] >= 900 &, "Distance"]

and uses the Values[Normal[]] version of this to Pick from the Dataset, but this seems inelegant. Can it be done using a single query?

(This question is the closest but wants to filter the rows within the 'chunks', rather than the chunks themselves.)

Answer 1

Here are two approaches:

titanic = ExampleData[{"Dataset", "Titanic"}][GroupBy["class"]]

Since the minimal age is 1 for all classes, let's take those where the max is less than 80:

titanic[All, Max, "age"]

titanic[Select[Query[Max /* LessThan[80], "age"]]]

titanic[Select[Max@#[[All, "age"]] < 80 &]]
(* same output *)

The second example is quite straightforward, it's essentially equivalent to your example. The first one does essentially the same, but in a more dataset-like way (although one can argue whether it's actually any nicer). It works by using a subquery to select the chucks. To illustrate, consider the following partial examples:

titanic[All, Max, "age"]

titanic[All, Max /* LessThan[80], "age"]

titanic[All, Query[Max /* LessThan[80], "age"]]

titanic[Select[Query[Max /* LessThan[80], "age"]]]