4
$\begingroup$ 
<< JLink`;
InstallJava[];
ReinstallJava[JVMArguments -> "-Xmx8192m"]

data = Import["E:\\data2.xlsx"][[1]];

fieldNames = data[[{1}, {1, 2, 4, 5, 6, 7, 8}]];
training = data[[2 ;; 220206, {1, 2, 4, 5, 6, 7, 8}]];
test = data[[220207 ;;, {1, 2, 4, 5, 6, 7, 8}]];

model = Predict[Cases[training, x_ :> (x[[;; 6]] -> x[[7]])]]

The last line produces the following error

No more memory available.
Mathematica kernel has shut down.
Try quitting other applications and then retry.

How should I get mathematica to produce a classifier/predictor function which i can use. Is mathematica capable of working with large datasets like sql server can using disk reads rather than memory.

Please help i have to produce results and really need a workable solution.

Improve this question
$\endgroup$
14
  • $\begingroup$ I beleive you need to use File. See this answer. $\endgroup$ – Edmund Oct 7 '16 at 23:37
  • $\begingroup$ mma isn't built to run on large scales, just small toy prototypes $\endgroup$ – M.R. Oct 8 '16 at 0:27
  • $\begingroup$ @M.R. Evidence? - For example a 2013 WTC presentation shows a quantum chemistry simulation running on Mathematica for 5.5 hours vs 5 hours on C++. What is toy vs scale? $\endgroup$ – alancalvitti Oct 8 '16 at 2:37
  • $\begingroup$ I don't know how to use File function since Predict and Classify input format is like {{x11,x12,x13,...}->y1,{x21,x22,x23,...}->y2,...}. But is there a way to feed data in chunks to Predict or Classify or even run these function on the chunks and then "merge" them? $\endgroup$ – user13892 Oct 8 '16 at 9:33
  • $\begingroup$ @alancalvitt anecdotal evidence: no one I know uses mathematica in any sort of production env - also take any ml vision example and multiply the input size by 10000 and mma will crash $\endgroup$ – M.R. Oct 8 '16 at 19:45
2
$\begingroup$

I've run into this issue several times over the past couple of years when testing and building classifier. On my current 16GB Windows laptop, I'm confident I can handle problems up to about 1Gb of data. Beyond that, things get a little dicier. Occasionally the specific nature of the problem supports some simple ways to reduce the size of the dataset, say by some form of random sampling, or by breaking the problem into smaller chunks. But otherwise I've fallen back on the tried and true way of solving this problem -- toss more horsepower at it. At my current company, we installed MMA on a mid-sized server (8 cores, 256 Gb of Ram). I've been able to handle sizable datasets (up to 70Gb) on this machine without any difficulties, using standard MMA commands. Some of these models have successfully run for over 100 CPU hours. At my previous company, we did something similar using a virtual server in AWS running Ubuntu. This works exceedingly well.The bottom line is that the machine you are using puts a constraint on the size of the problem you can handle in MMA.

Now as a couple of other commenters have noted, one of the challenges of using MMA is that it does not easily integrate with the production workflows that many company routinely use. But that's a different issue than the OP raised.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.