# LinearModelFit with Dataset

I have a rectangular dataset, initially created with SemanticImport of a CSV file and then manipulated. Rather surprisingly, LinearModelFit does not seem able to work directly with a dataset. Am I overlooking something? Do I need to produce the associated data matrix? If so, what is the most efficent approach to that? I'm currently using Values /@ Normal[dataset].

LinearModelFit does not presently (v11.0.1) operate directly upon datasets:

$ds = SemanticImportString["1,2\n4,5\n10,20\n", {"Integer", "Integer"}]  LinearModelFit[$ds, x, x]


We can, however, invoke LinearModelFit as a query operator:

$ds[LinearModelFit[#, x, x] &]  It is probably not worth the trouble, but if we have are hearts set upon direct application of LinearModelFit then we can supply the definition ourselves: Dataset; Unprotect[Dataset]; Dataset /: LinearModelFit[ds_Dataset?DatasetValidDatasetQ, rest___] := LinearModelFit[Normal@ds, rest] Protect[Dataset];  So then: LinearModelFit[$ds, x, x]


Update

If the original CSV data includes headers, then the imported structure will be a list of associations:

$ds2 = SemanticImportString["x,y\n1,2\n4,5\n10,20\n", {"Integer", "Integer"}]  Since LinearModelFit cannot work with associations, with have little option but to convert those associations to lists. For example, using a query: $ds[Values /* (LinearModelFit[#, x, x] &)]


... or direct application (with the extra definition in place):

LinearModelFit[$ds2[Values], x, x]  Both of these options offer only a meager improvement over the notation proposed in the question (i.e. the omission of Normal). To dispense with the use of Values would require that LinearModelFit accept lists of associations. This time the return is even smaller in comparison to the effort involved, but we can arrange it if we wish: LinearModelFit; Unprotect[LinearModelFit]; PrependTo[DownValues[LinearModelFit] , HoldPattern[LinearModelFit[data:{_Association..}, rest__]] :> LinearModelFit[Values[data], rest] ]; Protect[LinearModelFit];  So then: $ds2[LinearModelFit[#, x, x] &]


LinearModelFit[\$ds2, x, x]


• Note that this definition will not quite work if the dataset has column headers (so that its Normal form is a list of associations, not a list of lists). Is there a simpler way than the one I mentioned in the question to extract the numerical matrix in this case? – Alan Nov 19 '16 at 19:02
• Yes, you are right: the alternate formulations offer only a marginal notational convenience when the original data has headers. While we can dispense with Normal, we are still stuck with Values`. It is possible to do more (see my update), but it feels like diminishing returns are setting in. Perhaps WRI will apply these changes in some future release. – WReach Nov 19 '16 at 21:09