# What is the natural way of using Dataset to get a FittedModel?

The new Dataset in Mathematica 10 is a great addition to the language, potentially reducing the convenience gap with R. Given this, I would like to understand what the equivalent simple or natural workflow in Mathematica is for a model fit.

In R, one can set up a data frame by columns (c creates a vector):

dataset=data.frame(days=c(1,2,6,8),area=c(3,6,8,2),frequency=c(1,4,4,2),height=c(2,3,11,6))


which is displayed like this:

  days area frequency height
1    1    3         1      2
2    2    6         4      3
3    6    8         4     11
4    8    2         2      6


You can then fit a model using a model formula:

fit=lm(height~days+area,data=dataset)


You could view the coefficients, for example, with coef(fit)

(Intercept)        days        area
-2.8167116   0.9198113   0.9278976


In Mathematica one option that occurs to me is:

dataset = Dataset[MapThread[Association, Thread /@ {"days" -> {1, 2, 6, 8},
"area" -> {3, 6, 8, 2}, "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}]]
LinearModelFit[dataset[[All, {"days", "area", "height"}]] // Normal // Values,
{days, area}, {days, area}]


The specific questions are:

1. What is a more natural way to create the Dataset in Mathematica? I find it helpful to have the data specified by columns and for the column name to be near the data, but if that idiom is not natural in Mathematica, that would be good to know.
2. Is there a simpler way to do the fit, perhaps one as straightforward as the R approach? Specifying the column names is important for experimentation and model selection when the number of potential variables is large.

Given some comments below, here are some clarifications:

• In R the data frame is a core structure and it is manipulated in many ways e.g. by combining data frames, adding new columns, modifying existing columns etc. My underlying interest is to understand to what extent Dataset can play this role. I think the column view is pretty important for this.
• Functional wrappers can be written to suit an individual's way of doing things. However, I am interested to know if there is any fundamental or natural way of approaching this in Mathematica. There is clearly a natural way to do it in R.

Inspired by WReach's answer, I started playing with Query based approach and here's what I came up with:

data = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2},
"frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}


With the data in the above form, we just create a Dataset simply as follows:

dataset = Dataset[data];


Don't worry that it does not look how you want yet, it gets better.

We can transform that Dataset into what you want as follows (the cool part):

(* note the v10 operator syntax and RightComposition form *)


Cool huh?

The fitting part remains the same as in my previous answer (see that for a different approach), so let's lump em all:

var = {days, area};

[LinearModelFit[#, var, var] &, {#days, #area, #height} &]


Actually, if your data will be in this form in most cases then you can store the transformation operation as a symbols OwnValues:

toDataset = Map[Thread] /* Transpose /* Map[Association];


Now, even cleaner:

dataset[toDataset][LinearModelFit[#, var, var] &, {#days, #area, #height} &]


So fresh and so clean.

• This is indeed fresh and interesting, but as WReach said in introducing the idea, not particularly natural (nor elegant in my opinion). Still, it makes one think. Jul 29, 2014 at 9:29
• I think what I’m missing is a way to express the relationship between the independent and dependent variables symbolically and for LinearModelFit to make use of the Dataset appropriately and parsimoniously. Although I might get used to it, I also found it somewhat jarring to pass a function to the Dataset when the end result is not a Dataset. Nevertheless, I think enough time has passed to conclude that this is a natural approach for Mathematica at present, so I accept. Sep 18, 2014 at 8:06

I was playing around, trying to come up with a purely query-based solution. The result is by no means natural, but perhaps it holds some academic interest or will spark some insight into a better answer:

Dataset[
{ "days" -> {1, 2, 6, 8}
, "area" -> {3, 6, 8, 2}
, "frequency" -> {1, 4, 4, 2}
, "height" -> {2, 3, 11, 6}
}
][ All, Association
][ LinearModelFit[#, {days, area}, {days, area}]&, {#days, #area, #height}&
]


• Definitely an interesting approach. By the way you don't need the quotations in the named pure functions since the keys are strings. +1 Jul 29, 2014 at 5:32
• @RunnyKine Indeed ;) Leftovers from one of the other 50 variations I tried, no doubt. Removed. Jul 30, 2014 at 13:46
• I understand, I tried about as many :) Jul 30, 2014 at 13:52
• Thanks, that is surprisingly clean! Clearly LinearModelFit should get smarter about dealing with Dataset (as should Classify and Predict, etc). Jul 31, 2014 at 21:49

For your first question assuming you have the data in the following arrangement:

data = {{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}
header = {"days", "area", "frequency", "height"}


You can create the Dataset as follows:

dataset = Dataset @ Map[AssociationThread[header, #] &] @ Transpose[data]


While I think the above method is most natural, if you still insist on having the data arranged the R way, you can still do the pre-processing once and use the above approach:

rdata = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2},
"frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}


Pre-process:

{header, data} = Transpose[rdata /. Rule -> List]

 {{"days", "area", "frequency", "height"},
{{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}}


Then proceed as above. Now, to Query the data for the fit just do:

dataset[LinearModelFit[#, {days, area}, {days, area}] &, {#days, #area, #height} &]


Or if we set var = {days, area}, then:

dataset[LinearModelFit[#, var, var] &, {#days, #area, #height} &]


OR

Query[LinearModelFit[#, var, var] &, {#days, #area, #height} &]@dataset


• Thank you. This creates the Dataset, but in the definition the column data are still separated from their corresponding names (e.g. height is not obviously linked to {2,3,11,6}, which is how I tend to see things in R. Jul 29, 2014 at 3:53
• Normal @ dataset[All, "height"] gives the list, where dataset is the Dataset in the above answer. Jul 29, 2014 at 4:02
• @YiWang my question is about the definition and not how to query the Dataset. Jul 29, 2014 at 4:12
• @RunnyKine using the rdata you defined, the approach given in my question (edited) would be: Dataset[MapThread[Association, Thread /@ rdata]] which seems simpler than what you propose. Jul 29, 2014 at 4:37
• @RunnyKine, performance is important for many, but for the amount of data I'm using (<100000 rows), the difference is negligible. Clarity of expression is far more important for my purposes. Nevertheless, I realise this is a matter of opinion, so thank you for sharing yours. Jul 29, 2014 at 7:11

If your data is most conveniently expressed as in the form

{key_1 -> (val_11, val_12, ..., val_1n}, ..., key_m -> (val_m1, val_m2, ..., val_mn},}


then it would be a good idea to define a function to create datasets directly form such data, which can be done as follows:

rulesToDataset[data : {Rule[_, {__}] ..}] :=
Dataset[Association /@ Inner[Rule, Sequence @@ Transpose[List @@@ data], List]]


Example:

data =
{"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2},
"frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}};
rulesToDataset[data]
`

• I can see the value of defining a function if there isn't an obvious straightforward way of doing it directly. I gather there might not be, which is what I was interested to know. Jul 29, 2014 at 8:32