# What is the natural way of using Dataset to get a FittedModel?

The new Dataset in Mathematica 10 is a great addition to the language, potentially reducing the convenience gap with R. Given this, I would like to understand what the equivalent simple or natural workflow in Mathematica is for a model fit.

In R, one can set up a data frame by columns (c creates a vector):

dataset=data.frame(days=c(1,2,6,8),area=c(3,6,8,2),frequency=c(1,4,4,2),height=c(2,3,11,6))


which is displayed like this:

  days area frequency height
1    1    3         1      2
2    2    6         4      3
3    6    8         4     11
4    8    2         2      6


You can then fit a model using a model formula:

fit=lm(height~days+area,data=dataset)


You could view the coefficients, for example, with coef(fit)

(Intercept)        days        area
-2.8167116   0.9198113   0.9278976


In Mathematica one option that occurs to me is:

dataset = Dataset[MapThread[Association, Thread /@ {"days" -> {1, 2, 6, 8},
"area" -> {3, 6, 8, 2}, "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}]]
LinearModelFit[dataset[[All, {"days", "area", "height"}]] // Normal // Values,
{days, area}, {days, area}]


The specific questions are:

1. What is a more natural way to create the Dataset in Mathematica? I find it helpful to have the data specified by columns and for the column name to be near the data, but if that idiom is not natural in Mathematica, that would be good to know.
2. Is there a simpler way to do the fit, perhaps one as straightforward as the R approach? Specifying the column names is important for experimentation and model selection when the number of potential variables is large.

Given some comments below, here are some clarifications:

• In R the data frame is a core structure and it is manipulated in many ways e.g. by combining data frames, adding new columns, modifying existing columns etc. My underlying interest is to understand to what extent Dataset can play this role. I think the column view is pretty important for this.
• Functional wrappers can be written to suit an individual's way of doing things. However, I am interested to know if there is any fundamental or natural way of approaching this in Mathematica. There is clearly a natural way to do it in R.

Inspired by WReach's answer, I started playing with Query based approach and here's what I came up with:

data = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2},
"frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}


With the data in the above form, we just create a Dataset simply as follows:

dataset = Dataset[data];


Don't worry that it does not look how you want yet, it gets better.

We can transform that Dataset into what you want as follows (the cool part):

(* note the v10 operator syntax and RightComposition form *) Cool huh?

The fitting part remains the same as in my previous answer (see that for a different approach), so let's lump em all:

var = {days, area};

[LinearModelFit[#, var, var] &, {#days, #area, #height} &]


Actually, if your data will be in this form in most cases then you can store the transformation operation as a symbols OwnValues:

toDataset = Map[Thread] /* Transpose /* Map[Association];


Now, even cleaner:

dataset[toDataset][LinearModelFit[#, var, var] &, {#days, #area, #height} &] So fresh and so clean.

• This is indeed fresh and interesting, but as WReach said in introducing the idea, not particularly natural (nor elegant in my opinion). Still, it makes one think. – Simplex Jul 29 '14 at 9:29
• I think what I’m missing is a way to express the relationship between the independent and dependent variables symbolically and for LinearModelFit to make use of the Dataset appropriately and parsimoniously. Although I might get used to it, I also found it somewhat jarring to pass a function to the Dataset when the end result is not a Dataset. Nevertheless, I think enough time has passed to conclude that this is a natural approach for Mathematica at present, so I accept. – Simplex Sep 18 '14 at 8:06

I was playing around, trying to come up with a purely query-based solution. The result is by no means natural, but perhaps it holds some academic interest or will spark some insight into a better answer:

Dataset[
{ "days" -> {1, 2, 6, 8}
, "area" -> {3, 6, 8, 2}
, "frequency" -> {1, 4, 4, 2}
, "height" -> {2, 3, 11, 6}
}
][ All, Association
][ LinearModelFit[#, {days, area}, {days, area}]&, {#days, #area, #height}&
] • Definitely an interesting approach. By the way you don't need the quotations in the named pure functions since the keys are strings. +1 – RunnyKine Jul 29 '14 at 5:32
• @RunnyKine Indeed ;) Leftovers from one of the other 50 variations I tried, no doubt. Removed. – WReach Jul 30 '14 at 13:46
• I understand, I tried about as many :) – RunnyKine Jul 30 '14 at 13:52
• Thanks, that is surprisingly clean! Clearly LinearModelFit should get smarter about dealing with Dataset (as should Classify and Predict, etc). – Taliesin Beynon Jul 31 '14 at 21:49

For your first question assuming you have the data in the following arrangement:

data = {{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}
header = {"days", "area", "frequency", "height"}


You can create the Dataset as follows:

dataset = Dataset @ Map[AssociationThread[header, #] &] @ Transpose[data] While I think the above method is most natural, if you still insist on having the data arranged the R way, you can still do the pre-processing once and use the above approach:

rdata = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2},
"frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}


Pre-process:

{header, data} = Transpose[rdata /. Rule -> List]

 {{"days", "area", "frequency", "height"},
{{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}}


Then proceed as above. Now, to Query the data for the fit just do:

dataset[LinearModelFit[#, {days, area}, {days, area}] &, {#days, #area, #height} &]


Or if we set var = {days, area}, then:

dataset[LinearModelFit[#, var, var] &, {#days, #area, #height} &]


OR

Query[LinearModelFit[#, var, var] &, {#days, #area, #height} &]@dataset • Thank you. This creates the Dataset, but in the definition the column data are still separated from their corresponding names (e.g. height is not obviously linked to {2,3,11,6}, which is how I tend to see things in R. – Simplex Jul 29 '14 at 3:53
• Normal @ dataset[All, "height"] gives the list, where dataset is the Dataset in the above answer. – Yi Wang Jul 29 '14 at 4:02
• @YiWang my question is about the definition and not how to query the Dataset. – Simplex Jul 29 '14 at 4:12
• @RunnyKine using the rdata you defined, the approach given in my question (edited) would be: Dataset[MapThread[Association, Thread /@ rdata]] which seems simpler than what you propose. – Simplex Jul 29 '14 at 4:37
• @Simplex, it looks simpler because I've used the operator form of /@ here and because the name AssociationThread is long. But I assure you, it's faster. – RunnyKine Jul 29 '14 at 4:42

If your data is most conveniently expressed as in the form

{key_1 -> (val_11, val_12, ..., val_1n}, ..., key_m -> (val_m1, val_m2, ..., val_mn},}


then it would be a good idea to define a function to create datasets directly form such data, which can be done as follows:

rulesToDataset[data : {Rule[_, {__}] ..}] :=
Dataset[Association /@ Inner[Rule, Sequence @@ Transpose[List @@@ data], List]]


Example:

data =
{"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2},
"frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}};
rulesToDataset[data]
` • I can see the value of defining a function if there isn't an obvious straightforward way of doing it directly. I gather there might not be, which is what I was interested to know. – Simplex Jul 29 '14 at 8:32