I am using LogitModelFit to implement a classification system for handwritten digits, based on the MNIST ExampleData set. The problem is that the full learning set - A 60,000 x 784 FP matrix - crashes LogitModelFit. A smaller set works, but I was wondering how to avoid the crash as I (right or wrong) assumed it should be capable handling these size learning sets.
I know this can be done with Classify, but it's a bit too black box for me and I am not sure if does cope with similar size trainingsets.
This is what I do:
Import the data
{X, Y} = (ExampleData[{"MachineLearning", "MNIST"}, "Data"] /. Rule -> List)\[Transpose];
Extract imagedata, vectorize the raster and define vector size
θsize = AbsoluteTiming[Dimensions[matrixX =Flatten[Map[ImageData, X], {3, 2}]\[Transpose]]][[2,2]];
θ= Table[Symbol["θ" <> ToString[i]], {i,θsize}];
Separate training and test data (X) and predicates (Y)
trainingX = matrixX[[;; 60000]];
trainingY = Y[[;; 60000]];
testX = matrixX[[60001 ;;]];
testY = Y[[60001 ;;]];
Clear[X, Y, matrixX]
Identify ranges for digits 0..9
individualTrainingSets = MovingMap[List[#[[1]] + 1, #[[2]]] &, Prepend[Accumulate[Length /@ Split[trainingY]], 0], 2]
(* {{1, 5923}, {5924, 12665}, {12666, 18623}, {18624, 24754}, {24755, 30596}, {30597, 36017}, {36018, 41935}, {41936, 48200}, {48201, 54051}, {54052, 60000}}
Yranges = Table[Symbol["trainingY" <> ToString[i]], {i, 0, 9}];
Y = SparseArray[i_ /; #1 <= i <= #2 -> 1, {60000}] & @@@individualTrainingSets;
Construct trainingset for "0"
data = Transpose[trainingX\[Transpose]~Join~{Y[[1]]}];
This work for smaller subsets, EDIT
reducedData = Take[data, {1, 60000, 100}];
rlm = LogitModelFit[reducedData,θ,θ]; // AbsoluteTiming
END EDIT
crashes for the full set (without error message)
rlm = LogitModelFit[data,θ,θ]; // AbsoluteTiming
I would like to understand why this function cannot cope with the size of the training set and - possibly - how to resolve this.