Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models

Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, Lei Zhang*

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

2 Citations (Scopus)

Abstract

With the emergence of pre-trained vision-language models like CLIP, how to adapt them to various downstream classification tasks has garnered significant attention in re-cent research. The adaptation strategies can be typically categorized into three paradigms: zero-shot adaptation, few-shot adaptation, and the recently-proposed training-free few-shot adaptation. Most existing approaches are tai-lored for a specific setting and can only cater to one or two of these paradigms. In this paper, we introduce a versa-tile adaptation approach that can effectively work under all three settings. Specifically, we propose the dual memory networks that comprise dynamic and static memory components. The static memory caches training data knowledge, enabling training-free few-shot adaptation, while the dynamic memory preserves historical test features online during the testing process, allowing for the exploration of additional data insights beyond the training set. This novel capability enhances model performance in the few-shot setting and enables model usability in the absence of training data. The two memory networks employ the same flexible memory interactive strategy, which can operate in a training-free mode and can be further enhanced by in-corporating learnable projection layers. Our approach is tested across 11 datasets under the three task settings. Re-markably, in the zero-shot scenario, it outperforms existing methods by over 3% and even shows superior results against methods utilizing external training data. Addition-ally, our method exhibits robust performance against nat-ural distribution shifts. Codes are available at https://github.com/YBZh/DMN.

Original languageEnglish
Title of host publicationProceedings - 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024
PublisherIEEE Computer Society
Pages28718-28728
Number of pages11
ISBN (Electronic)9798350353006
ISBN (Print)9798350353013
DOIs
Publication statusPublished - 21 Jun 2024
Event2024 37th IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 - Seattle Convention Center, Seattle, United States
Duration: 17 Jun 202421 Jun 2024
https://cvpr.thecvf.com/Conferences/2024 (Conference website)
https://cvpr.thecvf.com/virtual/2024 (Conference website)
https://cvpr.thecvf.com/virtual/2024/calendar (conference schedule)
https://media.eventhosts.cc/Conferences/CVPR2024/CVPR_main_conf_2024.pdf (Conference program)
https://openaccess.thecvf.com/CVPR2024 (Conference proceedings)
https://ieeexplore.ieee.org/xpl/conhome/10654794/proceeding (Conference proceedings)

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (Print)1063-6919
ISSN (Electronic)2575-7075

Conference

Conference2024 37th IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024
Abbreviated titleCVPR 2024
Country/TerritoryUnited States
CitySeattle
Period17/06/2421/06/24
Internet address

Scopus Subject Areas

  • Software
  • Computer Vision and Pattern Recognition

User-Defined Keywords

  • dual memory networks
  • versatile adaptation
  • Vision-language models

Fingerprint

Dive into the research topics of 'Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models'. Together they form a unique fingerprint.

Cite this