Calibration
Calibration: The degree to which predicted confidence matches the frequency with which predictions are correct. Among cases with 0.8 confidence, a calibrated model should be wrong about 20% of the time. Reliability diagrams, Expected Calibration Error and temperature scaling measure and improve this property.
In italiano: CalibrationLegge II · Probabilità, softmax e modelli generativiAmong cases with 0.8 confidence, a calibrated model should be wrong about 20% of the time. Reliability diagrams, Expected Calibration Error and temperature scaling measure and improve this property.
Formula
ECE = Σ_b (n_b/n) · |acc(b) − conf(b)|
Examples
- Temperature scaling on a validation set
- Ten-bin reliability diagram
- Probabilities used in medical decisions