4
$\begingroup$

I only have classifications (a list of inputs and outputs), but not a ClassifierFunction, how can I still use ClassificationMeasurements to compute statistics on my data?

With PrintDefinitions[] I was thinking we could do something like this to create the ClassifierMeasurementsObject

predictions = {"A", "A", "B", "A"}
truth = {2.2 -> "A", 2.7 -> "A", 4.5 -> "B", 1.4 -> "B"};
cm = Quiet@MachineLearning`file115ClassifierPredictor`PackagePrivate`routerC[predictions, truth, {}]

But this didn't quite fit the bill.

$\endgroup$
3
+125
$\begingroup$

You can use the package ROCFunctions.m.

ROCFunctions.m implements all the functions defined in the Wikipedia page "Receiver operating characteristic" (ROC). More details are given in the blog post "ROC for classifier ensembles, bootstrapping, damaging, and interpolation".

Code

Data (from the question):

predictions = {"A", "A", "B", "A"};
truth = {2.2 -> "A", 2.7 -> "A", 4.5 -> "B", 1.4 -> "B"};

Get the ROCFunctions.m package:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/ROCFunctions.m"]

See the definition of ToROCAssociation from that package:

?ToROCAssociation

ToROCAssociation[ {trueLabel, falseLabel}, actualLabels, predictedLabels] converts two labels lists (actual and predicted) into an Association that can be used as an argument for the ROC functions. See ROCFunctions .

Make a ROC association:

ra = ToROCAssociation[{"A", "B"}, truth[[All, 2]], predictions]

(* <|"TruePositive" -> 2, "FalsePositive" -> 1, 
     "TrueNegative" -> 1, "FalseNegative" -> 0|> *)

Compute measures:

msrFuncs = {"Accuracy", "Precision", "Recall", "FalsePositiveRate", "SPC"};
AssociationThread[msrFuncs, N@Through[ROCFunctions[msrFuncs][ra]]]

(* <|"Accuracy" -> 0.75, "Precision" -> 0.666667, 
      "Recall" -> 1., "FalsePositiveRate" -> 0.5, "SPC" -> 0.5|> *)

Related question and answer

See this MSE answer of the related (very similar) MSE question "Visualization of experimental results: making a plot of the success rate vs. a parameter".

$\endgroup$
  • $\begingroup$ This is helpful but I was hoping for an answer that enabled me to use the built-in symbol ClassifierMeasurements. $\endgroup$ – M.R. May 18 '18 at 3:43
  • $\begingroup$ Couple of points. 1. I made the package ROCFunctions.m before ClassifierMeasurements started providing ROC functions. 2. I have tried to utilize ClassifierFunction and ClassifierMeasurements in ways that would make it convenient to use ensemble classifiers or custom classifiers. I did not have any success. This monad implementation diminishes those needs for ensembles. $\endgroup$ – Anton Antonov May 18 '18 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.