# How to iterate over the columns of a Dataset?

I have a Dataset with a simple structure, like this:

Dataset[{
<|"X" -> 0.,  "N" ->  14, "S" -> 106.85, "M0" -> 8962.85, "M1" -> 129.71|>,
<|"X" -> 0.5, "N" ->  14, "S" -> 104.81, "M0" -> 8956.78, "M1" -> 135.78|>,
<|"X" -> 1.,  "N" -> 434, "S" -> 43.89,  "M0" ->  239.46, "M1" ->  53.84|>,
<|"X" -> 1.5, "N" -> 529, "S" -> 49.97,  "M0" ->  168.72, "M1" ->  71.90|>,
<|"X" -> 2.,  "N" -> 578, "S" -> 52.61,  "M0" ->  139.30, "M1" ->  80.93|>
}]


(The actual dataset has around 20 columns, and about 7K rows.)

Does Mathematica have a built-in way to iterate over the columns of such a dataset?

(I'm looking for something analogous to the Keys method for Associations.)

Ultimately what I want to do is this: for each column Y other than X, extract the list of pairs that one would get from transposing the pair of columns {X, Y}, so that I can pass it to ListPlot.

Here an approach that uses some Dataset related functionality. Doesn't look too readable to me, but in its core it uses the column names to access the data-set and things like Keys and Values. Everything can be found in the documentation of Dataset:

 d = Dataset[{<|"X" -> 0., "N" -> 14, "S" -> 106.85, "M0" -> 8962.85,
"M1" -> 129.71|>, <|"X" -> 0.5, "N" -> 14, "S" -> 104.81,
"M0" -> 8956.78, "M1" -> 135.78|>, <|"X" -> 1., "N" -> 434,
"S" -> 43.89, "M0" -> 239.46, "M1" -> 53.84|>, <|"X" -> 1.5,
"N" -> 529, "S" -> 49.97, "M0" -> 168.72, "M1" -> 71.90|>, <|
"X" -> 2., "N" -> 578, "S" -> 52.61, "M0" -> 139.30,
"M1" -> 80.93|>}];


First create the list of column pairs we want:

tup = Tuples[{{First[#]}, Rest[#]}] &[Normal[d[[1, Keys]]]]
(* {{"X", "N"}, {"X", "S"}, {"X", "M0"}, {"X", "M1"}} *)


Then you can plot it by extracting the values:

ListLinePlot[
Normal[d[[All, #]][Values]] & /@ tup
, PlotRange -> All] To get the data arranged for use in ListPlot, you'll have to use 'Normal - e.g. like this:

data = Transpose[Normal[Map[Values,
Dataset[{<|"X" -> 0., "N" -> 14, "S" -> 106.85, "M0" -> 8962.85,
"M1" -> 129.71|>, <|"X" -> 0.5, "N" -> 14, "S" -> 104.81,
"M0" -> 8956.78, "M1" -> 135.78|>, <|"X" -> 1., "N" -> 434,
"S" -> 43.89, "M0" -> 239.46, "M1" -> 53.84|>, <|"X" -> 1.5,
"N" -> 529, "S" -> 49.97, "M0" -> 168.72, "M1" -> 71.90|>, <|
"X" -> 2., "N" -> 578, "S" -> 52.61, "M0" -> 139.30,
"M1" -> 80.93|>}]
]]];

ListLinePlot[Map[Transpose[{data[], #}] &, Rest[data]],
PlotRange -> All] By using Transpose on the array of values, I get the "X" entries as the first row. This allows me to combine this row with all other rows, one at a time, in ListLinePlot. To get the plot, the Transpose has to be undone by another Transpose.

This approach works for the DataSet in your question because is has the structure of a full array that can be transposed, i.e., the keys in each Association are the same.

I'll present an answer that doesn't require us to first convert the whole Dataset via Normal and then applying ListLinePlot. Instead, we collect the Keys and use the Dataset query approach.

(* Here ds is the Dataset *)

keys = Normal@Keys[ds]; (* The Column names of the Dataset *)
{first, rest} = {First@keys, Rest@keys}; (* separating the columns of interest *)
cn = Evaluate@ToExpression[{"#" ~~ first, "#" ~~ #} & /@ rest] &;
ds[ListLinePlot[Transpose[##], PlotRange -> All] &, cn] This is another alternative using Column indexes rather than Keys.

Table[Values[Normal[data[All, {1, i}]]], {i, 2, Length[data]}] // ListLinePlot


I cant help thinking it should be easier than this though. ListLinePlot accepts lists of associations it should really accept lists of datasets too. It also accepts the Key of a dataset as the x value so it should be possible to form an approach based on that too, by making column X values the keys for Value N, S etc.

• In retrospect I seem to arrived independently at a close relative of @Halirutan 's answer. Happy to delete :) – Gordon Coale Jun 18 '15 at 8:07
• No, leave it. I don't think my answer looks especially beautiful and the more different ways (even if the differ only a bit) we have, the better. – halirutan Jun 18 '15 at 11:15
• Thanks! But note: ... {..., Length[data]}... is not right. With a more complete implementation of datasets, it would be Width[data], but here we may have to settle for Length[data], or some such. – kjo Jun 18 '15 at 13:39
• Good point. Fooled by a symmetrical dataset. Corrected. – Gordon Coale Jun 18 '15 at 14:33

Here's a way using a utility, associationOuter that takes lists of Associations as inputs (as opposed to Outer which works on sequences). User passes functions to apply to Keys and Values separately:

associationOuter[{f_, fOpts__}, {g_, gOpts__}][as_List] :=
Module[{keyout, valout},
keyout =   as // Map[Keys] /* (Outer[f, Sequence @@ #, fOpts] &) ;
valout = as // Map[Values] /* (Outer[g, Sequence @@ #, gOpts] &);
{keyout, valout} // Transpose //
Map[Transpose /* Map[Apply[Rule]]] // Flatten // Association
]


Then, given:

ds[All , {{"X"}, Rest} ] The plots can be obtained separately (shown below) or combined by moving ListPlot a level up in the query.

ds[All , {{"X"}, Rest} /* associationOuter[{List, 1}, {List, 1}]][
Transpose][All, ListPlot[#, Joined -> True] &] // Normal 