After posting this answer OP clarified the question with:
To be more specific, I am not looking for the solution which will improve my result but, for the one which will allow me to load a big trining set.
This answer is about getting good classification results over a large dataset, not about on-boarding large datasets into Classify
.
Investigation steps
A list of the steps I took follows.
- Download the data and import it.
- Basic data analysis.
- There are duplicates in the data.
- Removed them with
Union
.
- Flatten the 2D arrays.
- Further data analysis.
For example, applying SVD produced 3-5 significant vectors.
That though was not that useful for the classification.
- Take a sample of 10^5 or 3*10^5 records and do classification experiments.
Relatively good results were obtained.
- Use large samples, e.g. 1.5*10^6 records.
- Turned out that using a training set sample of ~5*10^4 and doing classification over the rest produced fairly good results.
For the experiments I used the package "MonaticContextualClassification.m". See the article "A monad for classification workflows".
The important class label is the label "0" (or "No" below) because only ~13% of the records have it. Using the ROC curves and selecting the threshold 0.175 with the obtained classifier produced the following classifier performance measures:
(* <|FalsePositiveRate-><|No->0.138243,Yes->0.124086|>,
Recall-><|No->0.875914,Yes->0.861757|>,
Specificity-><|No->0.861757,Yes->0.875914|>,
MatthewsCorrelationCoefficient-><|No->0.737745,Yes->0.737745|>,
Accuracy->0.863442|> *)
I assume the overall accuracy and class label recall shown above are satisfactory. If not further experiments can be done using classifier ensembles, removing outliers, etc.
Code
AbsoluteTiming[
trainingSet = Import["~/Downloads/traningSet.mx"];
]
Length[trainingSet]
(* {3.36351, Null}
3467240 *)
(* Remove duplicates. *)
AbsoluteTiming[
trainingSet2 = Union[trainingSet];
]
Length[trainingSet2]
(* {16.4582, Null}
3312223 *)
(* Flatten 2D arrays. *)
trainingSet2[[All, 1]] = Flatten /@ trainingSet2[[All, 1]];
Dimensions[trainingSet2[[All, 1]]]
(* {3312223, 20} *)
(* See the class labels representation. *)
Tally[trainingSet2[[All, 2]]]
(* {{1., 2918036}, {0., 394187}} *)
(* Using categorical labels. *)
trainingSet2[[All, 2]] = trainingSet2[[All, 2]] /. {1. -> "Yes", 0. -> "No"};
Tally[trainingSet2[[All, 2]]]
(* {{"Yes", 2918036}, {"No", 394187}} *)
(* Taking relatively smaller sample. *)
SeedRandom[3439]
trainingSet2Small = RandomSample[trainingSet2, 3*10^5];
(* Load the ClCon package. *)
Import["https://raw.githubusercontent.com/antononcube/\
MathematicaForPrediction/master/MonadicProgramming/\
MonadicContextualClassification.m"]
(* Importing from GitHub: StateMonadCodeGenerator.m
Importing from GitHub: ClassifierEnsembles.m
Importing from GitHub: ROCFunctions.m
Importing from GitHub: VariableImportanceByClassifiers.m *)
(* Application of the software monad ClCon. *)
AbsoluteTiming[
clObj =
ClConUnit[trainingSet2Small]⟹
ClConSplitData[0.2, 0.2]⟹
ClConTrainClassifier[{"GradientBoostedTrees"}]⟹
ClConClassifierMeasurements⟹
ClConEchoValue⟹
ClConROCPlot["FPR", "TPR", ImageSize -> 350]⟹
ClConROCListLinePlot[{"TPR", "FPR", "SPC", "MCC", "Accuracy"}, ImageSize -> 350];
]
![enter image description here](https://i.sstatic.net/frb7I.png)
(* {70.5563, Null} *)
(* This gets the classifier: *)
(clObj⟹ClConTakeClassifier)[[1]]
(* This verifies that the obtained classifier produces good results \
with the selected threshold, 0.175 . *)
AbsoluteTiming[
ClConUnit[]⟹
ClConSetTestData[trainingSet2]⟹
ClConSetClassifier[clObj⟹ClConTakeClassifier]⟹
ClConClassifierMeasurementsByThreshold[{ "FalsePositiveRate", "Recall", "Specificity", "MatthewsCorrelationCoefficient", "Accuracy"}, "No" -> 0.175]⟹
ClConEchoValue;
]
(* value: <|FalsePositiveRate-><|No->0.138243,Yes->0.124086|>,
Recall-><|No->0.875914,Yes->0.861757|>,
Specificity-><|No->0.861757,Yes->0.875914|>,
MatthewsCorrelationCoefficient-><|No->0.737745,Yes->0.737745|>,
Accuracy->0.863442|> *)
(* {229.404, Null} *)