Efficient Tuning of Vision Foundation Models with Neural Prompt Search

The size of vision models has grown exponentially in recent years, particularly with the rise of Vision Transformers. This rapid growth has driven the development of parameter-efficient tuning methods, such as learning adapter layers or low-rank adaptation layers, which enable fine-tuning of a small subset of model parameters while keeping the vast majority of pretrained parameters frozen. However, designing an effective tuning method is not straightforward: it often involves exploring numerous design choices, and each downstream dataset may require custom-tailored solutions. In this chapter, we introduce Neural prOmpt seArcH (NOAH), a novel approach that leverages a neural architecture search algorithm to automatically learn the optimal design of prompt modules for large vision models, tailored specifically for each downstream dataset.