I only have classifications (a list of inputs and outputs), but not a ClassifierFunction, how can I still use ClassificationMeasurements to compute statistics on my data?

With PrintDefinitions[] I was thinking we could do something like this to create the ClassifierMeasurementsObject

predictions = {"A", "A", "B", "A"}
truth = {2.2 -> "A", 2.7 -> "A", 4.5 -> "B", 1.4 -> "B"};
cm = Quiet@MachineLearning`file115ClassifierPredictor`PackagePrivate`routerC[predictions, truth, {}]

But this didn't quite fit the bill.


1 Answer 1


You can use the package ROCFunctions.m.

ROCFunctions.m implements all the functions defined in the Wikipedia page "Receiver operating characteristic" (ROC). More details are given in the blog post "ROC for classifier ensembles, bootstrapping, damaging, and interpolation".


Data (from the question):

predictions = {"A", "A", "B", "A"};
truth = {2.2 -> "A", 2.7 -> "A", 4.5 -> "B", 1.4 -> "B"};

Get the ROCFunctions.m package:


See the definition of ToROCAssociation from that package:


ToROCAssociation[ {trueLabel, falseLabel}, actualLabels, predictedLabels] converts two labels lists (actual and predicted) into an Association that can be used as an argument for the ROC functions. See ROCFunctions .

Make a ROC association:

ra = ToROCAssociation[{"A", "B"}, truth[[All, 2]], predictions]

(* <|"TruePositive" -> 2, "FalsePositive" -> 1, 
     "TrueNegative" -> 1, "FalseNegative" -> 0|> *)

Compute measures:

msrFuncs = {"Accuracy", "Precision", "Recall", "FalsePositiveRate", "SPC"};
AssociationThread[msrFuncs, N@Through[ROCFunctions[msrFuncs][ra]]]

(* <|"Accuracy" -> 0.75, "Precision" -> 0.666667, 
      "Recall" -> 1., "FalsePositiveRate" -> 0.5, "SPC" -> 0.5|> *)

Related question and answer

See this MSE answer of the related (very similar) MSE question "Visualization of experimental results: making a plot of the success rate vs. a parameter".

  • $\begingroup$ This is helpful but I was hoping for an answer that enabled me to use the built-in symbol ClassifierMeasurements. $\endgroup$
    – M.R.
    Commented May 18, 2018 at 3:43
  • $\begingroup$ Couple of points. 1. I made the package ROCFunctions.m before ClassifierMeasurements started providing ROC functions. 2. I have tried to utilize ClassifierFunction and ClassifierMeasurements in ways that would make it convenient to use ensemble classifiers or custom classifiers. I did not have any success. This monad implementation diminishes those needs for ensembles. $\endgroup$ Commented May 18, 2018 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.