Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration

Ao, Shuang; Rüger, Stefan and Siddharthan, Advaith (2023). Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration. In: UAI '23: Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence Vol. 216, Proceedings of Machine Learning Research (PMLR), Pittsburgh, pp. 77–87.

DOI: https://doi.org/10.5555/3625834.3625842

Abstract

Proper confidence calibration of deep neural networks is essential for reliable predictions in safety-critical tasks. Miscalibration can lead to model over-confidence and/or under-confidence; i.e., the model’s confidence in its prediction can be greater or less than the model’s accuracy. Recent studies have highlighted the over-confidence issue by introducing calibration techniques and demonstrated success on various tasks. However, miscalibration through under-confidence has not yet to receive much attention. In this paper, we address the necessity of paying attention to the under-confidence issue. We first introduce a novel metric, a miscalibration score, to identify the overall and class-wise calibration status, including being over or under-confident. Our proposed metric reveals the pitfalls of existing calibration techniques, where they often overly calibrate the model and worsen under-confident predictions. Then we utilize the class-wise miscalibration score as a proxy to design a calibration technique that can tackle both over and under-confidence. We report extensive experiments that show our proposed methods substantially outperforming existing calibration techniques. We also validate our proposed calibration technique on an automatic failure detection task with a risk-coverage curve, reporting that our methods improve failure detection as well as trustworthiness of the model. The code are available at https://github.com/AoShuang92/miscalibration_TS

Viewing alternatives

Download history

Metrics

Public Attention

Altmetrics from Altmetric

Number of Citations

Citations from Dimensions

Item Actions

Export

About