description: Compute the Expected Calibration Error (ECE).
View source on GitHub |
Compute the Expected Calibration Error (ECE).
tfp.substrates.numpy.stats.expected_calibration_error(
num_bins, logits=None, labels_true=None, labels_predicted=None, name=None
)
This method implements equation (3) in [1]. In this equation the probability of the decided label being correct is used to estimate the calibration property of the predictor.
Note: a trade-off exist between using a small number of num_bins
and the
estimation reliability of the ECE. In particular, this method may produce
unreliable ECE estimates in case there are few samples available in some bins.
As an alternative to this method, consider also using
bayesian_expected_calibration_error
.
[1]: Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning (ICML 2017). arXiv:1706.04599 https://arxiv.org/pdf/1706.04599.pdf
Args | |
---|---|
num_bins
|
int, number of probability bins, e.g. 10. |
logits
|
Tensor, (n,nlabels), with logits for n instances and nlabels. |
labels_true
|
Tensor, (n,), with tf.int32 or tf.int64 elements containing ground truth class labels in the range [0,nlabels]. |
labels_predicted
|
Tensor, (n,), with tf.int32 or tf.int64 elements
containing decisions of the predictive system. If None , we will use
the argmax decision using the logits .
|
name
|
Python str name prefixed to Ops created by this function.
|
Returns | |
---|---|
ece
|
Tensor, scalar, tf.float32. |