Confidence Calibration in Contrastive Vision-Language Models

Contrastive Vision-Language Models (VLMs) have emerged as powerful tools, excelling in various open-vocabulary tasks such as image recognition, retrieval-augmented task adaptation, and visual chatbots. To better adapt to downstream tasks, various parameter-efficient fine-tuning approaches have been developed by the community, e.g., prompt learning. However, an important issue has received little attention: the confidence calibration problem in zero-shot or fine-tuned VLMs, which can significantly undermine the reliability of these models in downstream applications. This chapter addresses this issue by systematically studying the confidence calibration problem in the context of prompt learning for CLIP. The analysis reveals that existing calibration techniques are inadequate, particularly in open-vocabulary scenarios. This chapter then discusses a simple yet effective approach called Distance-Aware Calibration (DAC), which automatically adjusts the temperature scaling parameter based on the distance between predicted text labels and base classes. The effectiveness of the approach is validated on 7 prompt learning methods across 11 downstream tasks.