What is the natural way of using Dataset to get a FittedModel?

Question

The new Dataset in Mathematica 10 is a great addition to the language, potentially reducing the convenience gap with R. Given this, I would like to understand what the equivalent simple or natural workflow in Mathematica is for a model fit.

In R, one can set up a data frame by columns (c creates a vector):

dataset=data.frame(days=c(1,2,6,8),area=c(3,6,8,2),frequency=c(1,4,4,2),height=c(2,3,11,6))

which is displayed like this:

  days area frequency height
1    1    3         1      2
2    2    6         4      3
3    6    8         4     11
4    8    2         2      6

You can then fit a model using a model formula:

fit=lm(height~days+area,data=dataset)

You could view the coefficients, for example, with coef(fit)

(Intercept)        days        area 
 -2.8167116   0.9198113   0.9278976

In Mathematica one option that occurs to me is:

dataset = Dataset[MapThread[Association, Thread /@ {"days" -> {1, 2, 6, 8},
 "area" -> {3, 6, 8, 2}, "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}]]
LinearModelFit[dataset[[All, {"days", "area", "height"}]] // Normal // Values,
 {days, area}, {days, area}]

The specific questions are:

1. What is a more natural way to create the Dataset in Mathematica? I find it helpful to have the data specified by columns and for the column name to be near the data, but if that idiom is not natural in Mathematica, that would be good to know.
2. Is there a simpler way to do the fit, perhaps one as straightforward as the R approach? Specifying the column names is important for experimentation and model selection when the number of potential variables is large.

Given some comments below, here are some clarifications:

• In R the data frame is a core structure and it is manipulated in many ways e.g. by combining data frames, adding new columns, modifying existing columns etc. My underlying interest is to understand to what extent Dataset can play this role. I think the column view is pretty important for this.
• Functional wrappers can be written to suit an individual's way of doing things. However, I am interested to know if there is any fundamental or natural way of approaching this in Mathematica. There is clearly a natural way to do it in R.

Answer 1

Inspired by WReach's answer, I started playing with Query based approach and here's what I came up with:

data = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, 
         "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}

With the data in the above form, we just create a Dataset simply as follows:

dataset = Dataset[data];

Don't worry that it does not look how you want yet, it gets better.

We can transform that Dataset into what you want as follows (the cool part):

(* note the v10 operator syntax and RightComposition form *)
dataset[Map[Thread] /* Transpose /* Map[Association]] 

Cool huh?

The fitting part remains the same as in my previous answer (see that for a different approach), so let's lump `em all:

var = {days, area};

dataset[Map[Thread] /* Transpose /* Map[Association]]
                         [LinearModelFit[#, var, var] &, {#days, #area, #height} &]

Actually, if your data will be in this form in most cases then you can store the transformation operation as a symbols OwnValues:

toDataset = Map[Thread] /* Transpose /* Map[Association];

Now, even cleaner:

dataset[toDataset][LinearModelFit[#, var, var] &, {#days, #area, #height} &]

So fresh and so clean.

Answer 2

I was playing around, trying to come up with a purely query-based solution. The result is by no means natural, but perhaps it holds some academic interest or will spark some insight into a better answer:

Dataset[
  { "days" -> {1, 2, 6, 8}
  , "area" -> {3, 6, 8, 2}
  , "frequency" -> {1, 4, 4, 2}
  , "height" -> {2, 3, 11, 6}
  }
][ Transpose, Thread
][ All, Association
][ LinearModelFit[#, {days, area}, {days, area}]&, {#days, #area, #height}&
]

Answer 3

For your first question assuming you have the data in the following arrangement:

data = {{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}
header = {"days", "area", "frequency", "height"}

You can create the Dataset as follows:

dataset = Dataset @ Map[AssociationThread[header, #] &] @ Transpose[data]

While I think the above method is most natural, if you still insist on having the data arranged the R way, you can still do the pre-processing once and use the above approach:

rdata = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, 
         "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}

Pre-process:

{header, data} = Transpose[rdata /. Rule -> List]

 {{"days", "area", "frequency", "height"},
 {{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}}

Then proceed as above. Now, to Query the data for the fit just do:

dataset[LinearModelFit[#, {days, area}, {days, area}] &, {#days, #area, #height} &]

Or if we set var = {days, area}, then:

dataset[LinearModelFit[#, var, var] &, {#days, #area, #height} &]

OR

Query[LinearModelFit[#, var, var] &, {#days, #area, #height} &]@dataset

Answer 4

If your data is most conveniently expressed as in the form

{key_1 -> (val_11, val_12, ..., val_1n}, ..., key_m -> (val_m1, val_m2, ..., val_mn},}

then it would be a good idea to define a function to create datasets directly form such data, which can be done as follows:

rulesToDataset[data : {Rule[_, {__}] ..}] :=
  Dataset[Association /@ Inner[Rule, Sequence @@ Transpose[List @@@ data], List]]

Example:

data = 
  {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, 
   "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}};
rulesToDataset[data]