Classifier training on a large data set

I am interested in binary classification and I have a huge set of training examples (around 4 millions). Each example is a $$10 \times 2$$ matrix. When I run the classification after two hour run the kernel failed. (I used Windows 10 and WL 12.)

Any suggestion on how to run it?

Remark: This question is about on-boarding a large data set into WL's classifier training. It is not (that much) about getting good classification results. Something similar to the approach presented in the tutorial "Training on Large Datasets".

Model training code:

rfModel =
Classify[traningSet,
Method -> {"RandomForest", "LeafSize" -> 6, "TreeNumber" -> 850},
TrainingProgressReporting -> "Print"];


An example of training data:

{{{0.0249777, 250., 0.012, 0.117647, 20., -0.0735972, 4., 1235., 0., 1.},
{0.0178571, 41., 0.0243902, 0., 5., 0.926403, 5., 456., 0., 1.}} -> 1.,
{{-1., -1., -1., -1., -1., -0.0735972, 4., 1075., 0., 1.},
{0.0255319, 469., 0.0234542, 0.037037, 57., -2.0736, 2., 79., 0., 0.}} -> 1.,
{{-1., -1., -1., -1., -1., -1.0736, 3., 1706., 0., 0.},
{-1., -1., -1., -1., -1., -0.0735972, 4., 2402., 0., 1.}} -> 1.}

• Please provide a larger sample of your data. Also, at first you probably do not need to train with all 4 million records. See do you get any good results with a smaller (representative) sample. – Anton Antonov May 31 at 19:18
• @AntonAntonov it is a link to the data dropbox.com/s/rdolcupy6e9swzh/traningSet.mx?dl=0. Long story short, this classification task is part of my research. I try to compare between few approaches to create a synthetic training data ( I don't get a good result with small data set this is a resson to try to incease the dataset). – Kiril Danilchenko May 31 at 19:25
• Thanks for providing the data! I downloaded it, I will investigate later today. – Anton Antonov May 31 at 19:39
• Is it a memory limitation? You could use Leonid Shifrin's very clever filesystem-backed list: mathematica.stackexchange.com/questions/36/…. Or use ActiveClassification to which you can supply your own data reader function. You could also cook your own classifier with NetTrain/NetEncoder as in the tutorial. Mathematica tends to choke on large in-memory datasets, so aim for out-of-core. – Robert Jacobson Jun 6 at 23:13

After posting this answer OP clarified the question with:

To be more specific, I am not looking for the solution which will improve my result but, for the one which will allow me to load a big trining set.

This answer is about getting good classification results over a large dataset, not about on-boarding large datasets into Classify.

Investigation steps

A list of the steps I took follows.

2. Basic data analysis.
• There are duplicates in the data.
• Removed them with Union.
3. Flatten the 2D arrays.
4. Further data analysis.
For example, applying SVD produced 3-5 significant vectors.
That though was not that useful for the classification.
5. Take a sample of 10^5 or 3*10^5 records and do classification experiments.
Relatively good results were obtained.
6. Use large samples, e.g. 1.5*10^6 records.
7. Turned out that using a training set sample of ~5*10^4 and doing classification over the rest produced fairly good results.

For the experiments I used the package "MonaticContextualClassification.m". See the article "A monad for classification workflows".

The important class label is the label "0" (or "No" below) because only ~13% of the records have it. Using the ROC curves and selecting the threshold 0.175 with the obtained classifier produced the following classifier performance measures:

(* <|FalsePositiveRate-><|No->0.138243,Yes->0.124086|>,
Recall-><|No->0.875914,Yes->0.861757|>,
Specificity-><|No->0.861757,Yes->0.875914|>,
MatthewsCorrelationCoefficient-><|No->0.737745,Yes->0.737745|>,
Accuracy->0.863442|> *)


I assume the overall accuracy and class label recall shown above are satisfactory. If not further experiments can be done using classifier ensembles, removing outliers, etc.

Code

AbsoluteTiming[
]
Length[trainingSet]

(* {3.36351, Null}

3467240 *)

(* Remove duplicates. *)
AbsoluteTiming[
trainingSet2 = Union[trainingSet];
]
Length[trainingSet2]

(* {16.4582, Null}
3312223 *)

(* Flatten 2D arrays. *)
trainingSet2[[All, 1]] = Flatten /@ trainingSet2[[All, 1]];
Dimensions[trainingSet2[[All, 1]]]

(* {3312223, 20} *)

(* See the class labels representation. *)
Tally[trainingSet2[[All, 2]]]

(* {{1., 2918036}, {0., 394187}} *)

(* Using categorical labels. *)
trainingSet2[[All, 2]] = trainingSet2[[All, 2]] /. {1. -> "Yes", 0. -> "No"};
Tally[trainingSet2[[All, 2]]]

(* {{"Yes", 2918036}, {"No", 394187}} *)

(* Taking relatively smaller sample. *)
SeedRandom[3439]
trainingSet2Small = RandomSample[trainingSet2, 3*10^5];

(* Load the ClCon package. *)
Import["https://raw.githubusercontent.com/antononcube/\

Importing from GitHub: ClassifierEnsembles.m

Importing from GitHub: ROCFunctions.m

Importing from GitHub: VariableImportanceByClassifiers.m *)

(* Application of the software monad ClCon. *)
AbsoluteTiming[
clObj =
ClConUnit[trainingSet2Small]⟹
ClConSplitData[0.2, 0.2]⟹
ClConClassifierMeasurements⟹
ClConEchoValue⟹
ClConROCPlot["FPR", "TPR", ImageSize -> 350]⟹
ClConROCListLinePlot[{"TPR", "FPR", "SPC", "MCC", "Accuracy"}, ImageSize -> 350];
]


 (* {70.5563, Null} *)

(* This gets the classifier: *)
(clObj⟹ClConTakeClassifier)[[1]]

(* This verifies that the obtained classifier produces good results \
with the selected threshold, 0.175 . *)
AbsoluteTiming[
ClConUnit[]⟹
ClConSetTestData[trainingSet2]⟹
ClConSetClassifier[clObj⟹ClConTakeClassifier]⟹
ClConClassifierMeasurementsByThreshold[{ "FalsePositiveRate", "Recall", "Specificity", "MatthewsCorrelationCoefficient", "Accuracy"}, "No" -> 0.175]⟹
ClConEchoValue;
]

(* value: <|FalsePositiveRate-><|No->0.138243,Yes->0.124086|>,
Recall-><|No->0.875914,Yes->0.861757|>,
Specificity-><|No->0.861757,Yes->0.875914|>,
MatthewsCorrelationCoefficient-><|No->0.737745,Yes->0.737745|>,
Accuracy->0.863442|> *)

(* {229.404, Null} *)

• thank you. I do not familiar with this package, and it looks very helpful. Unfortunatlly, I am looking for a solution which use all the training set, my current problem is that the WL kernel is failed when I use all data in Classify. To be more specific, I am not looking for the solution which will improve my result but, for the one which will allow me to load a big trining set. Something similar to approach presented in the following tutorial :reference.wolfram.com/language/tutorial/… – Kiril Danilchenko Jun 1 at 9:27
• I should have read through this answer more carefully. It's better than mine. In any case, @KirilDanilchenko, Anton's answer will likely work for your full dataset. I just did the simplest thing and it worked. – Robert Jacobson Jun 7 at 5:19
• @RobertJacobson Well, and if I clarified with OP what exactly he wants, I would not have spent time researching and posting my answer. I am almost embarrassed I did... (I should have payed attention to these sentences from OP's comments: "I try to compare between few approaches to create a synthetic training data. I don't get a good result with small data set this is a reason to try to increase the dataset.") – Anton Antonov Jun 7 at 13:07
• @KirilDanilchenko Your technical problem was solved by fixing your dataset. I get no errors, even with millions of data elements, after fixing the dataset. – Robert Jacobson Jun 11 at 19:59

The Problem

We have a training set of ~2.7 million elements with which we wish to train a random forest classifier. However, Mathematica errors out before the classifier is finished training.

What Went Wrong

Mathematica can have trouble with large datasets, but that isn't the problem here. Let's investigate the data:

 AbsoluteTiming[trainingSet = Union[Import["~/Downloads/trainingSet.mx"]];
{Length[trainingSet],
Length[trainingSet] - Length[DeleteDuplicates[trainingSet]]}

{16.9372,Null}
{2748500, 14522}


One problem is that the training set has duplicate elements. Since we are training a binary classifier, we want the categories to be equally represented. Let's check the distribution.

Count[Union[trainingSet], _ -> 1.0]

2160225


So of the 2733978 unique elements, ~80% of them are in category 1. This is not a good training set, because flipping a coin will give an accuracy of 80%.

We split the dataset up into a training set and a test set, each of which are equal parts category 0 and category 1 data. Note that I am using 100% of the 0 category data to do this.

trainingSet = DeleteDuplicates[Import["~/Downloads/trainingSet.mx"]];
SeedRandom[42];
trainingSet =
RandomSample[
Join[RandomSample[Select[trainingSet, Last[#] == 1.0 &], 573753],
Select[trainingSet, Last[#] == 0.0 &]]];
{trainingSet, testSet} = {Take[trainingSet, 573753],
Take[trainingSet, -573753]};


Train and save the result to disk just in case. I'm using the notebook frontend, so I use TrainingProgressReporting -> "Panel".

rfModel =
Classify[trainingSet,
Method -> {"RandomForest", "LeafSize" -> 6, "TreeNumber" -> 850},
TrainingProgressReporting -> "Panel"];


Evaluate the performance

It's always best to create a ClassifierMeasurementsObject so you don't have to run the test again.

cmo = ClassifierMeasurements[rfModel, testSet];
cmo["Report"]


Transpose[{
{"MatthewsCorrelationCoefficient", "CohenKappa", "EvaluationTime"},
cmo /@ {"MatthewsCorrelationCoefficient", "CohenKappa", "EvaluationTime"}
}] // TableForm

MatthewsCorrelationCoefficient  <|0.->0.917584,1.->0.917584|>
CohenKappa                      0.916081
EvaluationTime                  0.0371


If you set it loose on your entire 2,733,978, be sure to use something like the Matthews Correlation Coefficient to account for the fact that you have wildly different numbers of elements in each category.

• In the last update of my answer I added Mathews Correlation Coefficient (MCC) plots. – Anton Antonov Jun 8 at 14:14