3
$\begingroup$

I want to reproduce a Kaggle's python version's result of MNIST.

PCA+SVM with accuracy=0.98 on test set.

By default in Classify, with SVM method, I get a result accuracy 0.9675 and now I get 0.975

And I want to know what's the best result of PerfanceGoal->"Quality" in the whole dataset.

The problem is it's too too too slow, 100 times time consumption of python's version, my cpu is intel i7-7700k-desktop-version. In running Classify with MNIST full dataset, the CPU usage and Memory usage is low about 30%.

Yes, the dataset is big, size of training data is 60000. So, the key problem is the calculating speed. Though, I know we can use a subset to do experiment, but I want to know is there some magic way to speed up the performance of training-speed also keep quality of SVM or NeuralNetwork in Classify?

I know in NetTrain we can use GPU, it's cool. I want to know even set all parameters, options, is it possible to speed up the performance of training like Compile,Parallel or TargetDevice->GPU, or something else

Question1: is there a global method to speed up Classify.

Question2: is there an algorithm-related method with Classify-SupportVectorMachine. If the answer of question 1 is No, do you know what's the most time-comsuming options or parameters in SVM and without importance of improving accuracy.

my notebook

$\endgroup$
  • $\begingroup$ One thing you can do is to run cl = Classify[data, opts] with a smaller dataset and then generate 2 classifiers: 1 without PerformanceGoal -> "Quality" and one with. After than you examine cl[[1]] to see what classify actually did and how the PerformanceGoal influenced the creation of the classifier function. Once you know that, you can forget about specifying the PerformanceGoal option completely and just put in the Method you need and taylor it. $\endgroup$ – Sjoerd Smit Sep 1 '17 at 13:03
  • $\begingroup$ @SjoerdSmit yes, that's a workflow in tune parameters, I want to know even set all parameters, options, is it possible to speed up the performance of training like Compile or TargetDevice->GPU, or something else. $\endgroup$ – HyperGroups Sep 1 '17 at 13:45
  • $\begingroup$ It might help to post the code you use and the timings you are seeing. $\endgroup$ – Daniel Lichtblau Sep 2 '17 at 17:29
5
$\begingroup$

I would suggest using a singular values decomposition explicitly as a PCA-type dimension reduction, then letting the SVM classifier loose on the result. The code below is adapted from a related method I've used elsewhere. The SVD part subtracts the mean of each vector to zero-center, and also adds a "normalizing" component to each vector obtained from the SVD step.

First step is to obtain all the images.

trainingBytes = Import[
   "http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz"
   , "Byte"];
trainingImages = Map[Image[Partition[#, 28]] &,
   Partition[Drop[trainingBytes, 16], 28^2]];
testBytes = Import[
   "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz"
   , "Byte"];
testImages = Map[Image[Partition[#, 28]] &,
   Partition[Drop[testBytes, 16], 28^2]];
trainingLabels = Drop[Import[
    "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz"
    , "Byte"], 8];
testLabels = Drop[Import[
    "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz"
    , "Byte"], 8];

The code below will preprocess using SVD, build the classifier using SVM, and also process query images using the right-side orthogonal matrix given by the SVD step. I leave the PerformanceGoal option to default setting. Changing it to "Quality" will make this quite slow.

nearestImages[ilist_, vals_, keep_] :=
 Module[
  {idata, images = ilist,
   topvecs, uu, ww, vv, udotw, norms},
  idata = Map[ImageData, images];
  topvecs = Map[Flatten, idata];
  topvecs = Map[# - Mean[#] &, topvecs];
  {uu, ww, vv} =
   SingularValueDecomposition[topvecs, keep];
  udotw = uu.ww;
  norms = Map[Sqrt[#.#] &, udotw];
  udotw = udotw/norms;
  udotw = Join[udotw, Transpose[{Log[norms]}], 2];
  {Classify[udotw -> vals, Method -> "SupportVectorMachine"], vv}]

processInput[ilist_, vv_] :=
 Module[
  {idata, images = ilist,
   topvecs, tdotv, norms},
  idata = Map[ImageData, images];
  topvecs = Map[Flatten, idata];
  topvecs = Map[# - Mean[#] &, topvecs];
  tdotv = topvecs.vv;
  norms = Map[Sqrt[#.#] &, tdotv];
  tdotv = tdotv/norms;
  tdotv = Join[tdotv, Transpose[{Log[norms]}], 2];
  tdotv]

Now we run it, with the setting to retain the largest 40 singular values.

keep = 40;
AbsoluteTiming[{nf, vv} =
   nearestImages[trainingImages,
    trainingLabels, keep];]
AbsoluteTiming[testvecs =
   processInput[testImages, vv];]

(* Out[10]= {285.201, Null}

Out[11]= {0.624939, Null} *)

We can speed up the lookup step using coarse-grained parallelization.

guesses[nf_, tvecs_] := 
 ParallelMap[nf, tvecs, Method -> "CoarsestGrained"]
correct[guess_, actual_] /;
  Length[guess] == Length[actual] :=
 Count[guess - actual, 0]

AbsoluteTiming[guessed = guesses[nf, testvecs];]
correct[guessed, testLabels]

(* Out[20]= {316.824, Null}

Out[21]= 9834 *)

We somewhat exceed 98% recognition in this way.

$\endgroup$
  • $\begingroup$ Glad to know the trick of the use of Log, also ParallelMap which I haven't try. See my edit, I add an notebook link in my GitHub, in that notebook I use DimensionReduce by LatentSemanticAnalysis which is based on SVD I think. $\endgroup$ – HyperGroups Sep 4 '17 at 5:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.