Differentiable Prompt Learning for Vision-Language Models

Large pre-trained vision-language models like CLIP have demonstrated significant potential for learning representations that can be applied across a variety of downstream tasks. Unlike traditional representation learning, which primarily relies on labeled data, vision-language pre-training aligns images and text within a shared feature space. This alignment allows for zero-shot transfer to downstream tasks through prompting, where classification weights are generated from natural language descriptions of the target classes. However, a major hurdle in deploying these models is prompt engineering, which is time-consuming and requires substantial domain expertise. Small changes in wording can significantly affect performance, making the process labor-intensive. In this chapter, we discuss a simple yet effective approach called Context Optimization (CoOp) for adapting CLIP-like vision-language models for image recognition tasks. CoOp uses learnable vectors to model the context words of prompts while keeping the pre-trained model parameters fixed. On 11 benchmark datasets, CoOp outperforms hand-crafted prompts with as few as one or two examples. Despite being a learning-based approach, CoOp also exhibits excellent domain generalization, surpassing zero-shot models that use hand-crafted prompts.