5
$\begingroup$

I have a rectangular dataset, initially created with SemanticImport of a CSV file and then manipulated. Rather surprisingly, LinearModelFit does not seem able to work directly with a dataset. Am I overlooking something? Do I need to produce the associated data matrix? If so, what is the most efficent approach to that? I'm currently using Values /@ Normal[dataset].

$\endgroup$
7
$\begingroup$

LinearModelFit does not presently (v11.0.1) operate directly upon datasets:

$ds = SemanticImportString["1,2\n4,5\n10,20\n", {"Integer", "Integer"}]

dataset screenshot

LinearModelFit[$ds, x, x]

error message

We can, however, invoke LinearModelFit as a query operator:

$ds[LinearModelFit[#, x, x] &]

fitted model

It is probably not worth the trouble, but if we have are hearts set upon direct application of LinearModelFit then we can supply the definition ourselves:

Dataset;
Unprotect[Dataset];

Dataset /: LinearModelFit[ds_Dataset?Dataset`ValidDatasetQ, rest___] :=
  LinearModelFit[Normal@ds, rest]

Protect[Dataset];

So then:

LinearModelFit[$ds, x, x]

fitted model

Update

If the original CSV data includes headers, then the imported structure will be a list of associations:

$ds2 = SemanticImportString["x,y\n1,2\n4,5\n10,20\n", {"Integer", "Integer"}]

dataset with headers

Since LinearModelFit cannot work with associations, with have little option but to convert those associations to lists. For example, using a query:

$ds[Values /* (LinearModelFit[#, x, x] &)]

fitted model

... or direct application (with the extra definition in place):

LinearModelFit[$ds2[Values], x, x]

fitted model

Both of these options offer only a meager improvement over the notation proposed in the question (i.e. the omission of Normal). To dispense with the use of Values would require that LinearModelFit accept lists of associations. This time the return is even smaller in comparison to the effort involved, but we can arrange it if we wish:

LinearModelFit;
Unprotect[LinearModelFit];

PrependTo[DownValues[LinearModelFit]
, HoldPattern[LinearModelFit[data:{_Association..}, rest__]] :>
  LinearModelFit[Values[data], rest]
];

Protect[LinearModelFit];

So then:

$ds2[LinearModelFit[#, x, x] &]

fitted model

LinearModelFit[$ds2, x, x]

fitted model

$\endgroup$
  • $\begingroup$ Note that this definition will not quite work if the dataset has column headers (so that its Normal form is a list of associations, not a list of lists). Is there a simpler way than the one I mentioned in the question to extract the numerical matrix in this case? $\endgroup$ – Alan Nov 19 '16 at 19:02
  • $\begingroup$ Yes, you are right: the alternate formulations offer only a marginal notational convenience when the original data has headers. While we can dispense with Normal, we are still stuck with Values. It is possible to do more (see my update), but it feels like diminishing returns are setting in. Perhaps WRI will apply these changes in some future release. $\endgroup$ – WReach Nov 19 '16 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.