# Writing code which generates a function to append many columns to a dataset

I love datasets in Mathematica. Say I have one that looks like this:

ds = Dataset[{
<|"a" -> 3, "c" -> 5, "w" -> Pi/4|>,
<|"a" -> 6, "c" -> 1, "w" -> Pi/8|>,
<|"a" -> 1, "c" -> 12, "w" -> Pi/6|>}]


How can I add/append columns? That was already covered nicely in a previous post (see below). Example:

appendY = Append[#, <|"y" -> #a Cos[#w] + #c|>] &
ds[All, appendY]


where the function appendY takes in an association and spits out another one with the extra key y. The reason why appendY is so nice to read and use is the #a notation, which returns key a of the association in the argument of appendY. Equivalently, one can also write #["a"] or Slot["a"].

The topic of this question is how to write code that generates something like appendY. I figured it out, but I think my code could be more elegant. There are also a few snags I do not understand, all related to the use of the Slot function.

Say I want to add columns (u,v,p,q,e) to ds, and I have a list of rules defining these columns:

rules = {u -> a Sinh[2 w] - c, v -> a Tan[w/3] + 2 c,
p -> (u + v)/2, q -> Norm[{u, v}], e -> a^2 q + p}


Note that the symbols are undefined, so I can expand these symbols using ReplaceRepeated:

rulesRHS = rules[[All, 2]] //. rules
rulesExpanded = {ToString /@ rules[[All, 1]], rulesRHS} // MapThread[Rule, #] &


(I'm sure the above code could be rewritten more elegantly.) For example, the last element of rulesExpanded is:

"e" -> a^2 Sqrt[Abs[-c + a Sinh[2 w]]^2 + Abs[2 c + a Tan[w/3]]^2] + 1/2 (c + a Sinh[2 w] + a Tan[w/3])


The ugly way forward is to paste that result into my code, place # in front of occurrences of the knowns (a,c,w), and slap it in Append. It's also impractical for me: my actual project has 16 known columns, to which I append 21 derived columns. The rules behind the 21 new columns are readable and intuitive when written out like in rules, but the equivalent of rulesExpanded spans several screens.

The functional way forward involves the Slot function:

knowns = {a, c, w}
rulesWithSlots = rulesExpanded /. (# -> f[ToString[#]] & /@ knowns) /. f -> Slot


I had to include a dummy function f here. Otherwise I get many Function::slot errors with this code:

rulesExpanded /. (# -> Slot[#] & /@ knowns)


Similarly, Slot /@ {"a", "b", "c"} works fine but Slot[#] & /@ {"a", "b", "c"} gives the same errors. I do not know why.

Now that we have slots, we can construct a function:

In:  makeList = Evaluate[rulesWithSlots] &;
In:  ds[1] // Normal // makeList // N
Out: {"u" -> 1.9039, "v" -> 10.8038, "p" -> 6.35387, "q" -> 10.9703, "e" -> 105.087}


The function makeList takes in an association and spits out a list with the new columns. Note that Evaluate is crucial. Otherwise, garbage comes out. I don't know why. Furthermore, there seems to be a snag with evaluating expressions where slots are inside associations. The following code also produces garbage:

makeAssoc = Evaluate[rulesWithSlots // Association] &
ds[1] // Normal // makeAssoc // N


Or, for a simpler example:

In:  slots = #a + #c;
In:  ds[1] // Normal // Evaluate[slots + slots^2] &
In:  ds[1] // Normal // Evaluate[{slots, {slots, slots}}] &
In:  ds[1] // Normal // Evaluate[<|"sum" -> slots|>] &
Out: 72
Out: {8, {8, 8}}
Out: <|"sum" -> #a + #c|>


I expected the last input to yield <|"sum" -> 8|>. I don't know why garbage comes out.

This snag isn't too bad because Append is forgiving and can slip in a list of rules into an association:

In:  appendColumns = Append[#, makeList[#]] &;
In:  ds[1] // Normal // appendColumns // N
Out: <|"a" -> 3., "c" -> 5., "w" -> 0.785398, "u" -> 1.9039,
"v" -> 10.8038, "p" -> 6.35387, "q" -> 10.9703, "e" -> 105.087|>


and finally:

ds[All, appendColumns]


Relevant question:

How can I add a column into a existing Dataset?

• Greetings! To make the most of Mma.SE please take the tour now. Help us to help you, write an excellent question. Edit if improvable, show due diligence, give brief context, include minimum working examples of code and data in formatted form. As you receive give back, vote and answer questions, keep the site useful, be kind, correct mistakes and share what you have learned. Oct 21, 2015 at 14:39

I needed to do this well but needed functions for reuse. These functions only need the list of rules that define the new columns. They perform the rule substitution calculate the column values and build the new column rules.

Function to calculate named column from rule definition and named column dataset row (Association)

ClearAll[calcColumn]

calcColumn[colDef_Rule, row_?AssociationQ] :=
With[{colSlots = Rule @@@ Transpose@{Keys[row], Range@Length@row}},
Evaluate[
ToString@Keys[colDef] -> (Values[colDef] /. {col : _Symbol /;
ContainsAny[{ToString@col}, Keys[row]] :>
(Slot[ToString@col /. colSlots])})] &[Sequence @@ Values@row]
]


Function to perform subsitution on new column rules.

ClearAll[ruleSubstitution]

ruleSubstitution[r_List /; ArrayQ[r, 1, Head[#] === Rule &]] :=
Rule @@@ Transpose@{Keys[r], Values@r //. r}


Applied to ds.

ds[All,
<|#, With[{dsrow = #},
(calcColumn[#, dsrow]) & /@ (ruleSubstitution[rules])]
|> &
]


• Thanks! I learned a bunch going through your lovely haiku of code. Nov 18, 2015 at 14:51