ClassificationMeasurements without the Classifier

I only have classifications (a list of inputs and outputs), but not a ClassifierFunction, how can I still use ClassificationMeasurements to compute statistics on my data?

With PrintDefinitions[] I was thinking we could do something like this to create the ClassifierMeasurementsObject

predictions = {"A", "A", "B", "A"}
truth = {2.2 -> "A", 2.7 -> "A", 4.5 -> "B", 1.4 -> "B"};
cm = Quiet@MachineLearningfile115ClassifierPredictorPackagePrivaterouterC[predictions, truth, {}]


But this didn't quite fit the bill.

You can use the package ROCFunctions.m.

ROCFunctions.m implements all the functions defined in the Wikipedia page "Receiver operating characteristic" (ROC). More details are given in the blog post "ROC for classifier ensembles, bootstrapping, damaging, and interpolation".

Code

Data (from the question):

predictions = {"A", "A", "B", "A"};
truth = {2.2 -> "A", 2.7 -> "A", 4.5 -> "B", 1.4 -> "B"};


Get the ROCFunctions.m package:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/ROCFunctions.m"]


See the definition of ToROCAssociation from that package:

?ToROCAssociation


ToROCAssociation[ {trueLabel, falseLabel}, actualLabels, predictedLabels] converts two labels lists (actual and predicted) into an Association that can be used as an argument for the ROC functions. See ROCFunctions .

Make a ROC association:

ra = ToROCAssociation[{"A", "B"}, truth[[All, 2]], predictions]

(* <|"TruePositive" -> 2, "FalsePositive" -> 1,
"TrueNegative" -> 1, "FalseNegative" -> 0|> *)


Compute measures:

msrFuncs = {"Accuracy", "Precision", "Recall", "FalsePositiveRate", "SPC"};

(* <|"Accuracy" -> 0.75, "Precision" -> 0.666667,
"Recall" -> 1., "FalsePositiveRate" -> 0.5, "SPC" -> 0.5|> *)


See this MSE answer of the related (very similar) MSE question "Visualization of experimental results: making a plot of the success rate vs. a parameter".

• This is helpful but I was hoping for an answer that enabled me to use the built-in symbol ClassifierMeasurements.
– M.R.
Commented May 18, 2018 at 3:43
• Couple of points. 1. I made the package ROCFunctions.m before ClassifierMeasurements started providing ROC functions. 2. I have tried to utilize ClassifierFunction and ClassifierMeasurements` in ways that would make it convenient to use ensemble classifiers or custom classifiers. I did not have any success. This monad implementation diminishes those needs for ensembles. Commented May 18, 2018 at 14:35