Negative Label Guided OOD Detection With Pretrained Vision-Language Models

Xue Jiang; Feng Liu; Zhen Fang; Hong Chen; Tongliang Liu; Feng Zheng; Bo Han

Negative Label Guided OOD Detection With Pretrained Vision-Language Models

Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng^*, Bo Han

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

2 Citations (Scopus)

Abstract

Out-of-distribution (OOD) detection aims at identifying samples from unknown classes, playing a crucial role in trustworthy models against errors on unexpected inputs. Extensive research has been dedicated to exploring OOD detection in the vision modality. Vision-language models (VLMs) can leverage both textual and visual information for various multi-modal applications, whereas few OOD detection methods take into account information from the text modality. In this paper, we propose a novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases. We design a novel scheme for the OOD score collaborated with negative labels. Theoretical analysis helps to understand the mechanism of negative labels. Extensive experiments demonstrate that our method NegLabel achieves state-of-the-art performance on various OOD detection benchmarks and generalizes well on multiple VLM architectures. Furthermore, our method NegLabel exhibits remarkable robustness against diverse domain shifts. The codes are available at https://github.com/tmlr-group/NegLabel.

Original language	English
Title of host publication	Proceedings of the Twelfth International Conference on Learning Representations, ICLR 2024
Publisher	International Conference on Learning Representations
Pages	1-29
Number of pages	29
Publication status	Published - May 2024
Event	12th International Conference on Learning Representations, ICLR 2024 - Messe Wien Exhibition and Congress Center, Vienna, Austria Duration: 7 May 2024 → 11 May 2024 https://iclr.cc/Conferences/2024 (Conference website) https://iclr.cc/virtual/2024/calendar (Conference schedule ) https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-accept-oral (Conference proceedings)

Publication series

Name	Proceedings of the International Conference on Learning Representations, ICLR

Conference

Conference	12th International Conference on Learning Representations, ICLR 2024
Country/Territory	Austria
City	Vienna
Period	7/05/24 → 11/05/24
Internet address	https://iclr.cc/Conferences/2024 (Conference website) https://iclr.cc/virtual/2024/calendar (Conference schedule ) https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-accept-oral (Conference proceedings)

Scopus Subject Areas

Language and Linguistics
Computer Science Applications
Education
Linguistics and Language

Access to Document

https://openreview.net/pdf?id=xUO1HXz4an

Cite this

Jiang, X., Liu, F., Fang, Z., Chen, H., Liu, T., Zheng, F., & Han, B. (2024). Negative Label Guided OOD Detection With Pretrained Vision-Language Models. In Proceedings of the Twelfth International Conference on Learning Representations, ICLR 2024 (pp. 1-29). (Proceedings of the International Conference on Learning Representations, ICLR). International Conference on Learning Representations. https://openreview.net/pdf?id=xUO1HXz4an

@inproceedings{22318665dc9b45ee9c0f43d78aafc58c,

title = "Negative Label Guided OOD Detection With Pretrained Vision-Language Models",

abstract = "Out-of-distribution (OOD) detection aims at identifying samples from unknown classes, playing a crucial role in trustworthy models against errors on unexpected inputs. Extensive research has been dedicated to exploring OOD detection in the vision modality. Vision-language models (VLMs) can leverage both textual and visual information for various multi-modal applications, whereas few OOD detection methods take into account information from the text modality. In this paper, we propose a novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases. We design a novel scheme for the OOD score collaborated with negative labels. Theoretical analysis helps to understand the mechanism of negative labels. Extensive experiments demonstrate that our method NegLabel achieves state-of-the-art performance on various OOD detection benchmarks and generalizes well on multiple VLM architectures. Furthermore, our method NegLabel exhibits remarkable robustness against diverse domain shifts. The codes are available at https://github.com/tmlr-group/NegLabel.",

author = "Xue Jiang and Feng Liu and Zhen Fang and Hong Chen and Tongliang Liu and Feng Zheng and Bo Han",

note = "This work was supported by the National Natural Science Foundation of China (Grant NO. 62122035, and 62376104). XJ and BH were supported by the NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, HKBU Faculty Niche Research Areas No. RC-FNRA-IG/22-23/SCI/04, and HKBU CSD Departmental Incentive Scheme. FL was supported by the Australian Research Council with grant numbers DP230101540 and DE240101089, and the NSF & CSIRO Responsible AI program with grant number 2303037. TLL was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. Publisher Copyright: {\textcopyright} 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved.; 12th International Conference on Learning Representations, ICLR 2024 ; Conference date: 07-05-2024 Through 11-05-2024",

year = "2024",

month = may,

language = "English",

series = "Proceedings of the International Conference on Learning Representations, ICLR",

publisher = "International Conference on Learning Representations",

pages = "1--29",

booktitle = "Proceedings of the Twelfth International Conference on Learning Representations, ICLR 2024",

url = "https://iclr.cc/Conferences/2024, https://iclr.cc/virtual/2024/calendar, https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-accept-oral",

}

Jiang, X, Liu, F, Fang, Z, Chen, H, Liu, T, Zheng, F & Han, B 2024, Negative Label Guided OOD Detection With Pretrained Vision-Language Models. in Proceedings of the Twelfth International Conference on Learning Representations, ICLR 2024. Proceedings of the International Conference on Learning Representations, ICLR, International Conference on Learning Representations, pp. 1-29, 12th International Conference on Learning Representations, ICLR 2024, Vienna, Austria, 7/05/24. <https://openreview.net/pdf?id=xUO1HXz4an>

Negative Label Guided OOD Detection With Pretrained Vision-Language Models. / Jiang, Xue; Liu, Feng; Fang, Zhen et al.
Proceedings of the Twelfth International Conference on Learning Representations, ICLR 2024. International Conference on Learning Representations, 2024. p. 1-29 (Proceedings of the International Conference on Learning Representations, ICLR).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Negative Label Guided OOD Detection With Pretrained Vision-Language Models

AU - Jiang, Xue

AU - Liu, Feng

AU - Fang, Zhen

AU - Chen, Hong

AU - Liu, Tongliang

AU - Zheng, Feng

AU - Han, Bo

N1 - This work was supported by the National Natural Science Foundation of China (Grant NO. 62122035, and 62376104). XJ and BH were supported by the NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, HKBU Faculty Niche Research Areas No. RC-FNRA-IG/22-23/SCI/04, and HKBU CSD Departmental Incentive Scheme. FL was supported by the Australian Research Council with grant numbers DP230101540 and DE240101089, and the NSF & CSIRO Responsible AI program with grant number 2303037. TLL was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. Publisher Copyright: © 2024 12th International Conference on Learning Representations, ICLR 2024. All rights reserved.

PY - 2024/5

Y1 - 2024/5

N2 - Out-of-distribution (OOD) detection aims at identifying samples from unknown classes, playing a crucial role in trustworthy models against errors on unexpected inputs. Extensive research has been dedicated to exploring OOD detection in the vision modality. Vision-language models (VLMs) can leverage both textual and visual information for various multi-modal applications, whereas few OOD detection methods take into account information from the text modality. In this paper, we propose a novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases. We design a novel scheme for the OOD score collaborated with negative labels. Theoretical analysis helps to understand the mechanism of negative labels. Extensive experiments demonstrate that our method NegLabel achieves state-of-the-art performance on various OOD detection benchmarks and generalizes well on multiple VLM architectures. Furthermore, our method NegLabel exhibits remarkable robustness against diverse domain shifts. The codes are available at https://github.com/tmlr-group/NegLabel.

AB - Out-of-distribution (OOD) detection aims at identifying samples from unknown classes, playing a crucial role in trustworthy models against errors on unexpected inputs. Extensive research has been dedicated to exploring OOD detection in the vision modality. Vision-language models (VLMs) can leverage both textual and visual information for various multi-modal applications, whereas few OOD detection methods take into account information from the text modality. In this paper, we propose a novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases. We design a novel scheme for the OOD score collaborated with negative labels. Theoretical analysis helps to understand the mechanism of negative labels. Extensive experiments demonstrate that our method NegLabel achieves state-of-the-art performance on various OOD detection benchmarks and generalizes well on multiple VLM architectures. Furthermore, our method NegLabel exhibits remarkable robustness against diverse domain shifts. The codes are available at https://github.com/tmlr-group/NegLabel.

UR - https://openreview.net/forum?id=xUO1HXz4an

UR - http://www.scopus.com/inward/record.url?scp=85196966928&partnerID=8YFLogxK

M3 - Conference proceeding

AN - SCOPUS:85196966928

T3 - Proceedings of the International Conference on Learning Representations, ICLR

SP - 1

EP - 29

BT - Proceedings of the Twelfth International Conference on Learning Representations, ICLR 2024

PB - International Conference on Learning Representations

T2 - 12th International Conference on Learning Representations, ICLR 2024

Y2 - 7 May 2024 through 11 May 2024

ER -

Jiang X, Liu F, Fang Z, Chen H, Liu T, Zheng F et al. Negative Label Guided OOD Detection With Pretrained Vision-Language Models. In Proceedings of the Twelfth International Conference on Learning Representations, ICLR 2024. International Conference on Learning Representations. 2024. p. 1-29. (Proceedings of the International Conference on Learning Representations, ICLR).

Negative Label Guided OOD Detection With Pretrained Vision-Language Models

Abstract

Publication series

Conference

Scopus Subject Areas

Access to Document

Other files and links

Fingerprint

Cite this