Prompting Vision Foundation Models for Pathology Image Analysis

Chong Yin, Siqi Liu, Kaiyang Zhou, Vincent Wai Sun Wong, Pong Chi Yuen

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

Abstract

The rapid increase in cases of non-alcoholic fatty liver disease (NAFLD) in recent years has raised significant public concern. Accurately identifying tissue alteration regions is crucial for the diagnosis of NAFLD but this task presents challenges in pathology image analysis particularly with small-scale datasets. Recently the paradigm shift from full fine-tuning to prompting in adapting vision foundation models has offered a new perspective for small-scale data analysis. However existing prompting methods based on task-agnostic prompts are mainly developed for generic image recognition which fall short in providing instructive cues for complex pathology images. In this paper we propose Q uantitative A ttribute-based P rompting (QAP) a novel prompting method specifically for liver pathology image analysis. QAP is based on two quantitative attributes namely K-function-based spatial attributes and histogram-based morphological attributes which are aimed for quantitative assessment of tissue states. Moreover a conditional prompt generator is designed to turn these instance-specific attributes into visual prompts. Extensive experiments on three diverse tasks demonstrate that our task-specific prompting method achieves better diagnostic performance as well as better interpretability. Code is available at \href https://github.com/7LFB/QAP https://github.com/7LFB/QAP .
Original languageEnglish
Title of host publicationProceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024
PublisherIEEE
Pages11292-11301
Number of pages10
Publication statusPublished - Jun 2024
EventThe IEEE / CVF Computer Vision and Pattern Recognition Conference, CVPR 2024 - Seattle Convention Center, Seattle, United States
Duration: 17 Jun 202421 Jun 2024
https://cvpr.thecvf.com/ (conference website)
https://cvpr2023.thecvf.com/virtual/2023/calendar (Link to conference schedule)
https://media.eventhosts.cc/Conferences/CVPR2024/CVPR_main_conf_2024.pdf (Link to conference booklet)
https://openaccess.thecvf.com/CVPR2024 (Conference proceedings)

Publication series

NameProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Conference

ConferenceThe IEEE / CVF Computer Vision and Pattern Recognition Conference, CVPR 2024
Abbreviated titleCVPR 2024
Country/TerritoryUnited States
CitySeattle
Period17/06/2421/06/24
Internet address

Fingerprint

Dive into the research topics of 'Prompting Vision Foundation Models for Pathology Image Analysis'. Together they form a unique fingerprint.

Cite this