3
$\begingroup$

I'm looking for clarity on Classify's ClassPriors option, and I've listed some specific questions below. According to the documentation:

ClassPriors is an option for Classify and related functions that specifies explicit prior probabilities to assume for output classes, independent of anything deduced from the training set.

The canonical example/usecase here is for imbalanced data:

data = {1 -> True, 2 -> True, 3 -> True, 4 -> True, 5 -> False, 6 -> True};
c = Classify[data, Method -> "LogisticRegression"]
{c[5, "Probabilities"],
 c[5, "Probabilities", ClassPriors -> <|False -> 0.5, True -> 0.5|>]}
(* this yields *)
 {<|False -> 0.452497, True -> 0.547503|>, 
 <|False -> 0.712596, True -> 0.287404|>} (* ok, this makes sense! *)

This option is supported in many symbols: Classify, ClassifierMeasurements, ClassifierFunction, etc. So here are some specific questions:

  1. If I use it in Classify, do I still need to use it in calling ClassifierMeasurements? Does one use it in the same way during inference? What is the interplay between usages?

  2. What exactly does ClassPriors change mathematically during model training for different Method options? For example, it seems to have no consistent effect, so what are the guidelines if any?

    c = Classify[data, Method -> "NeuralNetwork"]
    {c[5, "Probabilities", ClassPriors -> <|False -> 0.5, True -> 0.5|>], 
     c[5, "Probabilities"]}
    {<|False -> 0.999551, True -> 0.000448898|>, 
     <|False -> 0.998655, True -> 0.00134548|>}
    
  3. Is the following usage correct: I have 1000 negative samples and 6000 positive samples in a binary classification problem, assuming the true distribution has 90% negative and 10% positive?

    Classify[trainingset, ClassPriors-><|"Positive"->0.1, "Negative"->0.9|>]
    

    or

    Classify[trainingset, ClassPriors-><|"Positive"->6/7, "Negative"->1/7|>]
    
  4. For multi-class classification, what if I know some of the priors but not all? For multi-label classification, how/can I specify know priors for specific (sub)-labels? Is this even supported?

Any additional examples showing how the ClassPriors can help with unbalanced data (if it can) would be very useful.

$\endgroup$
  • $\begingroup$ As described this seems much more like a statistical methods question rather than a Mathematica question. Have you tried stats.stackexchange.com ? I'll note that having "unbalanced data" is not a sin nor is it an impediment to performing an analysis. Setting priors depends on your objective. $\endgroup$ – JimB Mar 3 at 1:16
  • 1
    $\begingroup$ @JimB no need to worry, this is a Mathematica question, I simply need clarity on the functionalities of a specific option $\endgroup$ – M.R. Mar 3 at 1:21
  • $\begingroup$ At least to me your question still reads as needing statistical help. Class priors don't "help" with unbalanced data. You are fitting a particular model and priors don't need to match to the sample size. Hence, I think you need statistical help rather than Mathematica help. $\endgroup$ – JimB Mar 3 at 1:31
  • 6
    $\begingroup$ I'm voting to close this question as off-topic because the question is written as if the OP does not know the "purpose" of supplying priors as opposed to "how" to supply priors in Mathematica. Priors do not "help" nor do they "mess with the confusion matrix" (as stated by the OP). This question should be asked at CrossValidated. $\endgroup$ – JimB Mar 3 at 14:26
  • $\begingroup$ @JimB please don't, I am only looking for more information/use cases of a particular option $\endgroup$ – M.R. Mar 3 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.