Problem with nested mapping for Dataset structures

Question

Consider the following dataset:

data = Dataset[{ <|"a" -> <|"A" -> 1, "B" -> 2|>, "b" -> <|"X" -> 5, "Y" -> 7|>|>,
                 <|"a" -> <|"A" -> 4, "B" -> 2|>, "b" -> <|"X" -> 6, "Y" -> 8|>|>,
                 <|"a" -> <|"A" -> 2, "B" -> 7|>, "b" -> <|"X" -> 3, "Y" -> 1|>|>}]

It has nested fields for grouping, so, for example, we'd access what might be called data[2].b.X in another language as data[2, "b", "X"] (to get the value 6).

In my application, I'm looking to modify each of the values of a given (terminal) field by combining them in a certain way with the values of another (terminal) field. For example, I might want to adjust all the so-called data[].a.B's by replacing them with data[].a.B - data[].b.Y.

My first approach was to use the way Datasets allow you to map at a field name. For example, if I wanted to double all of the data[].a.B's, I could do that with data = data[{All -> {"a" -> {"B" -> (2*# &)}}}], which gives me back the original dataset with that 'column' of 1, 4, 2 pictured above having become 2, 8, 4.

But to include the field values under b, this has to become a function of each of the list indices; that is, it needs to become data = data[{All -> somefunction}]. This function, in turn, involves mapping a function at specific field names of a dataset: each of its arguments is the sub-dataset consisting of a single association (with keys "a", "b"). So to implement data[].a.B - data[].b.Y, we could try something like

data = data[{All -> Function[a, a[{"a" -> {"B" -> (# - a["b", "X"] &)}}] ] }]

However, this doesn't work! It seems as though you can't nest dataset queries/actions inside things acting on parts of datasets.

Why doesn't this work, and is there a convenient workaround?

Answer 1

The exhibited expression does not work because the function argument a is an association. Query syntax is only recognized when it appears within the arguments of a Dataset or Query expression. We can use Query to fix up the function:

data[{All -> Function[a, a // Query[{"a" -> {"B" -> (# - a["b", "X"] &)}}]]}]

As an aside... the following expression violates the spirit of the question but it reads fairly well:

data[All, Module[{a = #}, a["a", "B"] -= a["b", "X"]; a] &]

Answer 2

I realized while writing this what the problem was, and thought I'd write it up just to share. The problem is this line:

This function, in turn, involves mapping a function at specific field names of a dataset: each of its arguments is the sub-dataset consisting of a single association (with keys "a", "b").

That's not true! Each argument to somefunction is the un-Dataset-ified association. It's easy to think "all subparts of Datasets are Datasets", though; after all, querying the dataset to get the first association doesn't return an Association object—it returns a Dataset! That is, data[1] has head Dataset.

Anyway, to solve it, we have to revert back to using Association tools—in this case, MapAt. we want

data[{All -> Function[a, MapAt[# - a["b", "X"] &, a, {"a", "B"}]] }]

That does the trick. However, I'd be interested if anyone knows how to do this by employing only the Dataset structure and not reverting back to acting on its component pieces—especially the functionality of combining parts of the same dataset for each element in a dataset.

(Would this be worth a feature suggestion if there is clean Dataset-only way to do it, or not?)