Help: Classify and Predict Fail for large datasets

Question

<< JLink`;
InstallJava[];
ReinstallJava[JVMArguments -> "-Xmx8192m"]

data = Import["E:\\data2.xlsx"][[1]];

fieldNames = data[[{1}, {1, 2, 4, 5, 6, 7, 8}]];
training = data[[2 ;; 220206, {1, 2, 4, 5, 6, 7, 8}]];
test = data[[220207 ;;, {1, 2, 4, 5, 6, 7, 8}]];

model = Predict[Cases[training, x_ :> (x[[;; 6]] -> x[[7]])]]

The last line produces the following error

No more memory available.
Mathematica kernel has shut down.
Try quitting other applications and then retry.

How should I get mathematica to produce a classifier/predictor function which i can use. Is mathematica capable of working with large datasets like sql server can using disk reads rather than memory.

Please help i have to produce results and really need a workable solution.

Answer 1

I've run into this issue several times over the past couple of years when testing and building classifier. On my current 16GB Windows laptop, I'm confident I can handle problems up to about 1Gb of data. Beyond that, things get a little dicier. Occasionally the specific nature of the problem supports some simple ways to reduce the size of the dataset, say by some form of random sampling, or by breaking the problem into smaller chunks. But otherwise I've fallen back on the tried and true way of solving this problem -- toss more horsepower at it. At my current company, we installed MMA on a mid-sized server (8 cores, 256 Gb of Ram). I've been able to handle sizable datasets (up to 70Gb) on this machine without any difficulties, using standard MMA commands. Some of these models have successfully run for over 100 CPU hours. At my previous company, we did something similar using a virtual server in AWS running Ubuntu. This works exceedingly well.The bottom line is that the machine you are using puts a constraint on the size of the problem you can handle in MMA.

Now as a couple of other commenters have noted, one of the challenges of using MMA is that it does not easily integrate with the production workflows that many company routinely use. But that's a different issue than the OP raised.