I love datasets in Mathematica. Say I have one that looks like this:
ds = Dataset[{
<|"a" -> 3, "c" -> 5, "w" -> Pi/4|>,
<|"a" -> 6, "c" -> 1, "w" -> Pi/8|>,
<|"a" -> 1, "c" -> 12, "w" -> Pi/6|>}]
How can I add/append columns? That was already covered nicely in a previous post (see below). Example:
appendY = Append[#, <|"y" -> #a Cos[#w] + #c|>] &
ds[All, appendY]
where the function appendY takes in an association and spits out another one with the extra key y. The reason why appendY is so nice to read and use is the #a notation, which returns key a of the association in the argument of appendY. Equivalently, one can also write #["a"] or Slot["a"].
The topic of this question is how to write code that generates something like appendY. I figured it out, but I think my code could be more elegant. There are also a few snags I do not understand, all related to the use of the Slot function.
Say I want to add columns (u,v,p,q,e) to ds, and I have a list of rules defining these columns:
rules = {u -> a Sinh[2 w] - c, v -> a Tan[w/3] + 2 c,
p -> (u + v)/2, q -> Norm[{u, v}], e -> a^2 q + p}
Note that the symbols are undefined, so I can expand these symbols using ReplaceRepeated:
rulesRHS = rules[[All, 2]] //. rules
rulesExpanded = {ToString /@ rules[[All, 1]], rulesRHS} // MapThread[Rule, #] &
(I'm sure the above code could be rewritten more elegantly.) For example, the last element of rulesExpanded is:
"e" -> a^2 Sqrt[Abs[-c + a Sinh[2 w]]^2 + Abs[2 c + a Tan[w/3]]^2] + 1/2 (c + a Sinh[2 w] + a Tan[w/3])
The ugly way forward is to paste that result into my code, place # in front of occurrences of the knowns (a,c,w), and slap it in Append. It's also impractical for me: my actual project has 16 known columns, to which I append 21 derived columns. The rules behind the 21 new columns are readable and intuitive when written out like in rules, but the equivalent of rulesExpanded spans several screens.
The functional way forward involves the Slot function:
knowns = {a, c, w}
rulesWithSlots = rulesExpanded /. (# -> f[ToString[#]] & /@ knowns) /. f -> Slot
I had to include a dummy function f here. Otherwise I get many Function::slot errors with this code:
rulesExpanded /. (# -> Slot[#] & /@ knowns)
Similarly, Slot /@ {"a", "b", "c"} works fine but Slot[#] & /@ {"a", "b", "c"} gives the same errors. I do not know why.
Now that we have slots, we can construct a function:
In: makeList = Evaluate[rulesWithSlots] &;
In: ds[1] // Normal // makeList // N
Out: {"u" -> 1.9039, "v" -> 10.8038, "p" -> 6.35387, "q" -> 10.9703, "e" -> 105.087}
The function makeList takes in an association and spits out a list with the new columns. Note that Evaluate is crucial. Otherwise, garbage comes out. I don't know why. Furthermore, there seems to be a snag with evaluating expressions where slots are inside associations. The following code also produces garbage:
makeAssoc = Evaluate[rulesWithSlots // Association] &
ds[1] // Normal // makeAssoc // N
Or, for a simpler example:
In: slots = #a + #c;
In: ds[1] // Normal // Evaluate[slots + slots^2] &
In: ds[1] // Normal // Evaluate[{slots, {slots, slots}}] &
In: ds[1] // Normal // Evaluate[<|"sum" -> slots|>] &
Out: 72
Out: {8, {8, 8}}
Out: <|"sum" -> #a + #c|>
I expected the last input to yield <|"sum" -> 8|>. I don't know why garbage comes out.
This snag isn't too bad because Append is forgiving and can slip in a list of rules into an association:
In: appendColumns = Append[#, makeList[#]] &;
In: ds[1] // Normal // appendColumns // N
Out: <|"a" -> 3., "c" -> 5., "w" -> 0.785398, "u" -> 1.9039,
"v" -> 10.8038, "p" -> 6.35387, "q" -> 10.9703, "e" -> 105.087|>
and finally:
ds[All, appendColumns]
Relevant question:
