4
$\begingroup$

Bug introduced in 10.4.1 or earlier and fixed in 11.1.1


I recently installed MMA v11.1 and encountered an issue with the memory usage of the LinearModelFit[] command. It appears than when mixing numeric and nominal variables, the LinearModelFit[] command uses a very large block of memory. I first noticed with issue on a large Linux server, where the the MMA kernel kept crashing after consuming all 256 GB of memory for a relatively small problem.

I created a simple example in an attempt to illustrate the problem:

In[44]:= $Version

Out[44]= "11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)"

First create a simple data matrix of size n:

regDat[n_] := Transpose[{
    RandomChoice[{"Yes", "No"}, n],  (* nominal variable *)
    RandomReal[{1, 10}, n],  (* numeric #1 *)
    RandomReal[{1, 10}, n],  (* numeric #2 *)
    RandomReal[{1, 10}, n]  (* dependent variable *)
    }
   ];

Now run a regression with n = 25,000, using only the numeric variables:

mem1 = MaxMemoryUsed[
   LinearModelFit[regDat[25000], {v2, v3}, {v1Nom, v2, v3}, 
    NominalVariables -> {v1Nom}]] // AbsoluteTiming

{0.10582, 8813008}

This uses about 8.8 MB of memory. Now run the same regression, except add the nominal variable (which has values {"Yes","No"}):

mem2 = MaxMemoryUsed[
   LinearModelFit[regDat[25000], {v1Nom, v2, v3}, {v1Nom, v2, v3}, 
    NominalVariables -> {v1Nom}]] // AbsoluteTiming

{6.92887, 4120980912}

This new regression takes 65x longer and uses 4.12 GB of memory.

I've confirmed this behavior on v11.1 on Windows and Linux. My original problem had n=257,000 observations, with 6 numeric variables and 4 nominal variables, and was unable to run due to excessive memory usage. But the same code ran without issue on v10 and v11.

(Note: The only time I've encountered memory issues using the LinearModelFit[] command is when I've inadvertently treated a numeric value as a nominal. I'm speculating that perhaps v 11.1 is treating all variables as nominal when a regression has a both types).

Can anyone else confirm this behavior?

Thanks,

Mark

$\endgroup$
  • $\begingroup$ I get the same results on Windows 7 with Mathematica versions 10.4.1 and 11.0.1 (with a deviation in the code in that that I think you must have not used NominalVariables -> {v1Nom} in mem1.) The solution is for large datasets to avoid NominalVariables and create your own dummy variables. $\endgroup$ – JimB Mar 31 '17 at 18:52
2
$\begingroup$

I had similar issues using your code with Mathematica versions 10.4.1 and 11.0.1 (but removing NominalVariables -> {v1Nom} on mem1):

regDat[n_] := 
  Transpose[{RandomChoice[{"Yes", "No"}, n],(*nominal variable*)
    RandomReal[{1, 10}, n],(*numeric #1*)
    RandomReal[{1, 10}, n],(*numeric #2*)
    RandomReal[{1, 10}, n]  (*dependent variable*)}];

data = regDat[25000];

mem1 = MaxMemoryUsed[LinearModelFit[data, {v2, v3}, {v1Nom, v2, v3}]] // AbsoluteTiming
(* {0.1282489100989303`, 6409488} *)

mem2 = MaxMemoryUsed[
   LinearModelFit[data, {v1Nom, v2, v3}, {v1Nom, v2, v3}, 
    NominalVariables -> {v1Nom}]] // AbsoluteTiming
(* {6.661942728611961`, 4120505936} *)

A workaround for large datasets is to create your own dummy variables:

data = data /. {"Yes" -> 1, "No" -> 0};
mem3 = MaxMemoryUsed[LinearModelFit[data, {v1Nom, v2, v3}, {v1Nom, v2, v3}]] // AbsoluteTiming
(* {0.19569940752999232`,11740464} *)
$\endgroup$
2
$\begingroup$

Along with posting this question, I also filed a ticket with WRI technical support. They confirmed this behavior under MMA v11.1.0.

With the release of MMA v11.1.1 on April 18, 2017, this bug has been resolved:

In[11]:= $Version

Out[11]= "11.1.1 for Microsoft Windows (64-bit) (April 18, 2017)"

regDat[n_] := Transpose[{
    RandomChoice[{"Yes", "No"}, n],  (* nominal variable *)
    RandomReal[{1, 10}, n],  (* numeric #1 *)
    RandomReal[{1, 10}, n],  (* numeric #2 *)
    RandomReal[{1, 10}, n]  (* dependent variable *)}
];

Using only numeric variables gives:

In[13]:= MaxMemoryUsed[
  LinearModelFit[
   regDat[25000], {v2, v3}, {v1Nom, v2, v3}]] // AbsoluteTiming

Out[13]= {0.110184, 6613256}

So the LinearModelFit command uses about 6.6 Mb of memory. Now add a nominal variable

In[14]:= MaxMemoryUsed[
  LinearModelFit[regDat[25000], {v1Nom, v2, v3}, {v1Nom, v2, v3}, 
   NominalVariables -> {v1Nom}]] // AbsoluteTiming

Out[14]= {0.309891, 21218712}

The regression uses 21.2Mb of memory, a much more reasonable amount.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.