UtilityFunction for actual and predicted values (machine learning)

Question

In machine learning, the utility function for a classifier assigns the utility value to an actual value v(act) and a predicted value v(pred), typically e.g. in form of

<|0 -> <|0 -> 1, 1 -> 0,Indeterminate -> 0.05|>, 
 1 -> <|0 -> -1, 1 -> 1,Indeterminate -> 0.9|>|>

binary class {0,1}.

My question:

"How are the Indeterminate thresholds are related when it comes to a decision (e.g. for class 0 or class1?"

More accurate: Let p=(p1,p2) the probability vector (e.g. p=(0.1,0.9), according to the utility equations, the corresponding utility values are u1 = 0.1 for actual class = 0 and u2 = 0.8.

Does this imply that we decide in this case for class 0 because u1=0.1>0.05 and u2=0.8<0.9?

Although it looks simply, I cannot find a reference or description how this exactly works.

Any suggestion would be great.

Answer 1

There are two parts about the UtilityFunction, One is threshold. Two is the utility function or utility matrix.

Question:

"How are the Indeterminate thresholds are related when it comes to a decision (e.g. for class 0 or class1?"

the Indeterminate thresholds is easy to understand, which we can simply do it without using the UtilityFunction

pred = model[#, "Probability"] &
classes = {0, 1}
classThresh[data_, thresh0_: 0.5, thresh1_: 0.9] := 
Block[{probs}, probs = pred@data;
Which[probs@0 > thresh0, 0, probs@1 > thresh1, 1, True, 
Indeterminate]
]

The tricky is the utility matrix and utility value. In machine learning, the utility function for a classifier assigns the utility value to an actual value v(act) and a predicted value v(pred), typically e.g. in form of

Question2:

What is an actual value, what is a predicted value? actual value is the predict value without using utility function.

Unlike the utility matrix which we can manually set values, in an example of Predict, the utility function seems cannot simply be used like this.

utility[va=predictor[data]] or utility[va=predictor[data],data]

Does wolfram mathematica calculate something like Maximize[utility[va,vp]]? Or there is a constraint condition like f[va]+f[vp]=1 by default?

In the example of HelpPage of Predict, utility=Function[-(#2-#1)^2] gives the mean value of the distribution.