# A simple way to extract information from this dataset with Query?

I have a dataset comparable to the following (you may wrap it in Dataset if you like). I am looking for a relatively simple way to query it and extract information.

### Sample dataset

ranStr[] := StringJoin@RandomChoice[CharacterRange["a", "z"], 5]

SeedRandom
asc =
CharacterRange["a", "e"],
Table[
Association@Table[
With[{n = RandomInteger[{10, 20}]},
ranStr[] -> <|
"x" -> RandomInteger[10, n],
"y" -> RandomInteger[10, n],
"c" -> RandomInteger,
"d" -> RandomWord[]
|>
],
{RandomInteger[3, 6]}
],
{5}
]
];


### Description of the dataset structure

Let me describe it. First, we have two nested levels of associations, but this is included just to mimic an obstacle in the real data I have. What is important is the 3rd, innermost level, which contains the actual data in the keys "x", "y", "c", "d".

At the innermost level, "x" and "y" are a pair of vectors of exactly the same length. The $$i$$th element of "x" goes with the $$i$$th element of "y", so it might have been better to store them like this:

asc // Query[1, 1, {"x", "y"} /* Transpose]

(* {<|"x" -> 10, "y" -> 10|>, <|"x" -> 5, "y" -> 9|>, <|"x" -> 6, "y" -> 7|>, ... *)


But that is not how they are stored in this dataset.

"c" is a number which represents a threshold value.

"d" is just some arbitrary additional data to mimic the actual dataset I have better. In there real data, there are several such keys, so repeating their names in the Query just to retain them is very inconvenient.

• Select those values of "x" where the corresponding value of "y" is greater than "c".
• Ideally, retain "c" and "d" in the data.
• Ideally, use Query or Dataset.

### Example output

Element 1,1 of the resulting dataset should look like the output of this:

xy = asc // Query[1, 1, {"x", "y"} /* Transpose];
c = asc // Query[1, 1, "c"];
d = asc // Query[1, 1, "d"];

<|"x" -> (xy // Query[Select[#y > c &], "x"]), "c" -> c, "d" -> d|>
(* <|"x" -> {10, 5, 6, 2, 7, 3, 2, 7, 10, 7}, "c" -> 5, "d" -> "bide"|> *)


Or like the output of this:

<|"xy" -> (xy // Query[Select[#y > c &]]), "c" -> c, "d" -> d|>


Or like the output of this:

<|"x" -> (xy // Query[Select[#y > c &], "x"]),
"y" -> (xy // Query[Select[#y > c &], "y"]),
"c" -> c, "d" -> d|>


What I am looking for is a relatively simple and intuitive Query or combination of Querys that achieves this.

The problems with doing something similar to how the example output is constructed above is:

• It involves splitting the dataset into parts, which would need to be Merged back together. The two outer nested levels of associations make that inconvenient.
• Repeating the names of keys like "d" to retain them is very inconvenient.

The following would be a working but rather ugly solution that I am not happy with:

asc //
Query[All, All, <|#, "xy" -> KeyTake[{"x", "y"}][#]|> & /* KeyDrop[{"x", "y"}]] (* bundle x and y together *) //
Query[All, All, {"xy" -> Transpose}] (* rearrange xy *) //
Query[All, All, <|#, With[{c = #c}, "xy" -> Select[#xy, #x > c &]]|> &] (* filter with a threshold of x *) //
Query[1, 1] (* remove this to see the whole output *)


Here is a way that produces output that conforms to the first option:

val = asc //
Query[
All
, All
, <| "x" -> Pick[#x, Positive[#y-#c]], KeyDrop[#, {"x", "y"}] |> &
];

val[[1, 1]]

(* <|"x" -> {10, 5, 6, 2, 7, 3, 2, 7, 10, 7}, "c" -> 5, "d" -> "bide"|> *)