Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

The new Dataset in Mathematica 10 is a great addition to the language, potentially reducing the convenience gap with R. Given this, I would like to understand what the equivalent simple or natural workflow in Mathematica is for a model fit.

In R, one can set up a data frame by columns (c creates a vector):

dataset=data.frame(days=c(1,2,6,8),area=c(3,6,8,2),frequency=c(1,4,4,2),height=c(2,3,11,6))

which is displayed like this:

  days area frequency height
1    1    3         1      2
2    2    6         4      3
3    6    8         4     11
4    8    2         2      6

You can then fit a model using a model formula:

fit=lm(height~days+area,data=dataset)

You could view the coefficients, for example, with coef(fit)

(Intercept)        days        area 
 -2.8167116   0.9198113   0.9278976

In Mathematica one option that occurs to me is:

dataset = Dataset[MapThread[Association, Thread /@ {"days" -> {1, 2, 6, 8},
 "area" -> {3, 6, 8, 2}, "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}]]
LinearModelFit[dataset[[All, {"days", "area", "height"}]] // Normal // Values,
 {days, area}, {days, area}]

The specific questions are:

  1. What is a more natural way to create the Dataset in Mathematica? I find it helpful to have the data specified by columns and for the column name to be near the data, but if that idiom is not natural in Mathematica, that would be good to know.
  2. Is there a simpler way to do the fit, perhaps one as straightforward as the R approach? Specifying the column names is important for experimentation and model selection when the number of potential variables is large.

Given some comments below, here are some clarifications:

  • In R the data frame is a core structure and it is manipulated in many ways e.g. by combining data frames, adding new columns, modifying existing columns etc. My underlying interest is to understand to what extent Dataset can play this role. I think the column view is pretty important for this.
  • Functional wrappers can be written to suit an individual's way of doing things. However, I am interested to know if there is any fundamental or natural way of approaching this in Mathematica. There is clearly a natural way to do it in R.
share|improve this question

4 Answers 4

up vote 21 down vote accepted

Inspired by WReach's answer, I started playing with Query based approach and here's what I came up with:

data = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, 
         "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}

With the data in the above form, we just create a Dataset simply as follows:

dataset = Dataset[data];

Don't worry that it does not look how you want yet, it gets better.

We can transform that Dataset into what you want as follows (the cool part):

(* note the v10 operator syntax and RightComposition form *)
dataset[Map[Thread] /* Transpose /* Map[Association]] 

Mathematica graphics

Cool huh?

The fitting part remains the same as in my previous answer (see that for a different approach), so let's lump `em all:

var = {days, area};

dataset[Map[Thread] /* Transpose /* Map[Association]]
                         [LinearModelFit[#, var, var] &, {#days, #area, #height} &]

Actually, if your data will be in this form in most cases then you can store the transformation operation as a symbols OwnValues:

toDataset = Map[Thread] /* Transpose /* Map[Association];

Now, even cleaner:

dataset[toDataset][LinearModelFit[#, var, var] &, {#days, #area, #height} &]

Mathematica graphics

So fresh and so clean.

share|improve this answer
    
This is indeed fresh and interesting, but as WReach said in introducing the idea, not particularly natural (nor elegant in my opinion). Still, it makes one think. –  Simplex Jul 29 at 9:29
    
I think what I’m missing is a way to express the relationship between the independent and dependent variables symbolically and for LinearModelFit to make use of the Dataset appropriately and parsimoniously. Although I might get used to it, I also found it somewhat jarring to pass a function to the Dataset when the end result is not a Dataset. Nevertheless, I think enough time has passed to conclude that this is a natural approach for Mathematica at present, so I accept. –  Simplex Sep 18 at 8:06

I was playing around, trying to come up with a purely query-based solution. The result is by no means natural, but perhaps it holds some academic interest or will spark some insight into a better answer:

Dataset[
  { "days" -> {1, 2, 6, 8}
  , "area" -> {3, 6, 8, 2}
  , "frequency" -> {1, 4, 4, 2}
  , "height" -> {2, 3, 11, 6}
  }
][ Transpose, Thread
][ All, Association
][ LinearModelFit[#, {days, area}, {days, area}]&, {#days, #area, #height}&
]

fitted model screenshot

share|improve this answer
2  
Definitely an interesting approach. By the way you don't need the quotations in the named pure functions since the keys are strings. +1 –  RunnyKine Jul 29 at 5:32
    
@RunnyKine Indeed ;) Leftovers from one of the other 50 variations I tried, no doubt. Removed. –  WReach Jul 30 at 13:46
    
I understand, I tried about as many :) –  RunnyKine Jul 30 at 13:52
4  
Thanks, that is surprisingly clean! Clearly LinearModelFit should get smarter about dealing with Dataset (as should Classify and Predict, etc). –  Taliesin Beynon Jul 31 at 21:49

For your first question assuming you have the data in the following arrangement:

data = {{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}
header = {"days", "area", "frequency", "height"}

You can create the Dataset as follows:

dataset = Dataset @ Map[AssociationThread[header, #] &] @ Transpose[data]

Mathematica graphics

While I think the above method is most natural, if you still insist on having the data arranged the R way, you can still do the pre-processing once and use the above approach:

rdata = {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, 
         "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}}

Pre-process:

{header, data} = Transpose[rdata /. Rule -> List]
 {{"days", "area", "frequency", "height"},
 {{1, 2, 6, 8}, {3, 6, 8, 2}, {1, 4, 4, 2}, {2, 3, 11, 6}}}

Then proceed as above. Now, to Query the data for the fit just do:

dataset[LinearModelFit[#, {days, area}, {days, area}] &, {#days, #area, #height} &]

Or if we set var = {days, area}, then:

dataset[LinearModelFit[#, var, var] &, {#days, #area, #height} &]

OR

Query[LinearModelFit[#, var, var] &, {#days, #area, #height} &]@dataset

Mathematica graphics

share|improve this answer
    
Thank you. This creates the Dataset, but in the definition the column data are still separated from their corresponding names (e.g. height is not obviously linked to {2,3,11,6}, which is how I tend to see things in R. –  Simplex Jul 29 at 3:53
    
Normal @ dataset[All, "height"] gives the list, where dataset is the Dataset in the above answer. –  Yi Wang Jul 29 at 4:02
    
@YiWang my question is about the definition and not how to query the Dataset. –  Simplex Jul 29 at 4:12
    
@RunnyKine using the rdata you defined, the approach given in my question (edited) would be: Dataset[MapThread[Association, Thread /@ rdata]] which seems simpler than what you propose. –  Simplex Jul 29 at 4:37
    
@Simplex, it looks simpler because I've used the operator form of /@ here and because the name AssociationThread is long. But I assure you, it's faster. –  RunnyKine Jul 29 at 4:42

If your data is most conveniently expressed as in the form

{key_1 -> (val_11, val_12, ..., val_1n}, ..., key_m -> (val_m1, val_m2, ..., val_mn},}

then it would be a good idea to define a function to create datasets directly form such data, which can be done as follows:

rulesToDataset[data : {Rule[_, {__}] ..}] :=
  Dataset[Association /@ Inner[Rule, Sequence @@ Transpose[List @@@ data], List]]

Example:

data = 
  {"days" -> {1, 2, 6, 8}, "area" -> {3, 6, 8, 2}, 
   "frequency" -> {1, 4, 4, 2}, "height" -> {2, 3, 11, 6}};
rulesToDataset[data]

dataset

share|improve this answer
    
I can see the value of defining a function if there isn't an obvious straightforward way of doing it directly. I gather there might not be, which is what I was interested to know. –  Simplex Jul 29 at 8:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.