I-Tuning: Tuning Frozen Language Models with Image for Lightweight Image Captioning

Ziyang Luo, Zhipeng Hu, Yadong Xi, Rongsheng Zhang, Jing Ma*

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceeding

3 Citations (Scopus)

Abstract

Image Captioning is a traditional vision-and-language task that aims to generate the language description of an image. Recent studies focus on scaling up the model size and the number of training data, which significantly increase the cost of model training. Different to these heavy-cost models, we introduce a lightweight image captioning framework (I-Tuning), which contains a small number of trainable parameters. We design a novel I-Tuning cross-attention module to connect the non-trainable pre-trained language decoder GPT2 and vision encoder CLIP-ViT. Since most parameters are not required to be updated during training, our framework is lightweight and fast. Experimental results conducted on three image captioning benchmarks reveal that our frame-work achieves comparable or better performance than the large-scale baseline systems. But our models contain up to 10 times fewer trainable parameters and require much fewer data for training compared with state-of-the-art baselines.
Original languageEnglish
Title of host publicationICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing
PublisherIEEE
Pages1-5
Number of pages5
ISBN (Electronic)9781728163277
ISBN (Print)9781728163284
DOIs
Publication statusPublished - Jun 2023
Event2023 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023 - Rhodes Island, Greece
Duration: 4 Jun 202310 Jun 2023
https://ieeexplore.ieee.org/xpl/conhome/10094559/proceeding

Publication series

NameICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
PublisherIEEE
ISSN (Print)1520-6149

Competition

Competition2023 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023
Country/TerritoryGreece
CityRhodes Island
Period4/06/2310/06/23
Internet address

Scopus Subject Areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering

User-Defined Keywords

  • Cross-Modal
  • Language models
  • Lightweight image captioning
  • Transformer

Fingerprint

Dive into the research topics of 'I-Tuning: Tuning Frozen Language Models with Image for Lightweight Image Captioning'. Together they form a unique fingerprint.

Cite this