2
$\begingroup$

When doing a classification one can determine the ClassifierMeasurements to get Information about the classifier. I now wanted to get a histogram of the computed probabilities for a classifier. Maybe I misunderstand something, but I can not figure out how to use this to get the histograms (for each class). Example (from the documentation)

    c = Classify[
  ExampleData[{"MachineLearning", "FisherIris"}, "TrainingData"]];

and

    cm = ClassifierMeasurements[c, 
  ExampleData[{"MachineLearning", "FisherIris"}, "TestData"]]

then I get (here we have three classes)

cm["ProbabilityHistogram"]

enter image description here

I can interpret this as to probabilities for the three classes

But doing the same for the Titanic dataset from WRI (with two classes) one gets enter image description here

Can someone give ma a hint how to get the histograms per class. My attempt (according to the documentation) to use

cm["ProbabilityHistogram" -> "survived"]

gives no output (My workaround is to calculate the Histograms "by hand" from the probabilities ;-) )

$\endgroup$
  • $\begingroup$ What version of Mathematica and operating system are you using? On Mathematica 12.0.0.0 and Windows 10 I get a histogram with 8 bins having a positive count for the Fisher Iris data. Even if the difference in the results is a version issue, of course, that doesn't get at your question as to how to get the separate histograms. $\endgroup$ – JimB Jan 25 at 17:00
  • $\begingroup$ @JimB Interesting. I'm using 12.0.0.0 on MacOS 10.15.2. I know how to get the separate Histograms (via cm["Probabilities"]but what delivers cm[ProbabilityHistogram]. I have no idea what the output is (o.k. a histogram, but what does it show?) $\endgroup$ – mgamer Jan 27 at 7:43
  • $\begingroup$ cm[ProbabilityHistogram] gives you a histogram of the predicted probabilities for the classes of the observations. If you have 45 observations, then the histogram consists of 45 probabilities. If an individual is "setosa", then it is the probability of being "setosa" that is used. Because that histogram is of probabilities of a mixture of classes, I'm not seeing a great use for it. (Still don't know why you see only 3 bars with the Fisher iris data.) $\endgroup$ – JimB Jan 27 at 14:03
  • $\begingroup$ @JimB: Than you for the explanation and, I agree, I also see only little use for it ;-) $\endgroup$ – mgamer Jan 27 at 14:51
2
$\begingroup$

Update - Address comment.

They are actually identical, just different binning / range

All classes

cm["Probabilities"] // Histogram[#, {0.1}, PlotRange -> {{0., 1.}, {All, All}}] &

enter image description here

cm["ProbabilityHistogram"]

enter image description here

What is different is your result for cm["ProbabilityHistogram"]. Perhaps you specified a Method->?


Here is one way to do it.

train = ResourceData["Sample Data: Titanic Survival", "TrainingData"];
test = ResourceData["Sample Data: Titanic Survival", "TestData"];
classifier = Classify[train]
cm = ClassifierMeasurements[classifier, test]

{cm["Examples"] // Map[Last], cm["Probabilities"]} //
   Thread //
   GroupBy[First -> Last] //
   Map[Histogram[#, ImageSize -> Medium] &]

enter image description here

|improve this answer|||||
$\endgroup$
  • $\begingroup$ This is close to my workaround as I wrote "(My workaround is to calculate the Histograms "by hand" from the probabilities ;-) )". These your (and my) results form the probabilities deliver the histograms, but the output of cm[ProbabilityHistogram] differs dramatically from our computed output. So what ist this what we get here? $\endgroup$ – mgamer Jan 24 at 17:04
  • $\begingroup$ @mgamer They are identical for me. See update. $\endgroup$ – Rohit Namjoshi Jan 24 at 23:04
  • 1
    $\begingroup$ Sure. But this is not the point of my question. It is regarding cm["ProbabilityHistogram] (and not ´cm["probabilities"]` and this output is deftly different from ´cm["probabilities"]` . The question is why and what is shown there? $\endgroup$ – mgamer Jan 25 at 9:35
  • $\begingroup$ Not sure what you mean. They are different because one evaluates to a Histogram and the other evaluates to a List of probabilities. The update shows the histogram generated by cm["ProbabilityHistogram"] and compares it with the histogram of probability values returned by cm["Probabilities"]. They are identical. BTW "probabilities" is not a valid property name, should be "Probabilities". $\endgroup$ – Rohit Namjoshi Jan 26 at 23:35
  • $\begingroup$ If you are asking why the probability histogram for the Fisher Iris data has just 3 values and why the histogram for the Titanic data has more than 2 values then the answer is that for the Fisher data the Method automatically chosen is "DecisionTree" so of course there are only going to be 3 values. $\endgroup$ – Rohit Namjoshi Jan 26 at 23:41
1
$\begingroup$

It depends on if you want the predicted probabilities for each "true" class or each "predicted" class.

cm["Probabilities"]

gives you the predicted probabilities associated with the "true" class (but, of course, by itself doesn't tell you what that true class is).

cm[[1,2,2]]

gets you the true status.

cm[[1,3]]

gets you the predicted status.

Exp[cm[[1,4]]]

gets you the predicted probabilities for each class and observation.

Here is a list of the first 20 observations for the Titanic data:

(* Get log of prediction probabilities and conver to probabilities *)
p = Exp[cm[[1, 4]]];
(* Combine all of the necessary data *)
probabilities = Transpose[{cm["Probabilities"], p[[All, 1]], p[[All, 2]], cm[[1, 3]], cm[[1, 2, 2]]}];
(* Show in Table Form just the first 20 *)
TableForm[probabilities[[1 ;; 20, All]], 
 TableHeadings -> {None, {"cm[\"Probabilities\"]", "Predicted\nPr(Died)", "Predicted\nPr(Survived)", 
  "Predicted\nStatus", "True\nStatus"}}]

Table of predictions of first 20 observations

To obtain histograms for the predicted probabilities for each true class...

predictions = Transpose[{cm["Probabilities"], cm[[1, 2, 2]]}];
Histogram[Select[predictions, #[[2]] == "died" &][[All, 1]], 20,
 "PDF", PlotRange -> {{0, 1}, {0, 7}}, Frame -> True, PlotLabel -> "Died",
 FrameLabel -> {"Predicted probability", "Probability density"}]
Histogram[Select[predictions, #[[2]] == "survived" &][[All, 1]], 20,
 "PDF", PlotRange -> {{0, 1}, {0, 7}}, Frame -> True, PlotLabel -> "Survived",
 FrameLabel -> {"Predicted probability", "Probability density"}]

Histogram of predicted probabilities for class = "died"

Histogram of predicted probabilities for class = "survived"

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.