GradToken: Decoupling tokens with class-aware gradient for visual explanation of Transformer network

Lin Cheng, Yanjie Liang, Yang Lu*, Yiu-ming Cheung

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

Abstract

Transformer networks have been widely used in the fields of computer vision, natural language processing, graph-structured data analysis, etc. Subsequently, explanations of Transformer play a key role in helping humans understand and analyze its decision-making and working mechanism, thereby improving the trustworthiness in its real-world applications. However, it is difficult to apply the existing explanation methods for convolutional neural networks to Transformer networks, due to the significant differences between their structures. How to design a specific and effective explanation method for Transformer poses a challenge in the explanation area. To address this challenge, we first analyze the semantic coupling problem of attention weight matrices in Transformer, which puts obstacles in providing distinctive explanations for different categories of targets. Then, we propose a gradient-decoupling-based token relevance method (i.e., GradToken) for the visual explanation of Transformer’s predictions. GradToken exploits the class-aware gradient to decouple the tangled semantics in the class token to the semantics corresponding to each category. GradToken further leverages the relations between the class token and spatial tokens to generate relevance maps. As a result, the visual explanation results generated by GradToken can effectively focus on the regions of selected targets. Extensive quantitative and qualitative experiments are conducted to verify the validity and reliability of the proposed method.
Original languageEnglish
Article number106837
Number of pages13
JournalNeural Networks
Volume181
DOIs
Publication statusPublished - Jan 2025

Scopus Subject Areas

  • Artificial Intelligence
  • Cognitive Neuroscience

User-Defined Keywords

  • Explanation
  • Interpretability
  • Transformer
  • Visualization

Fingerprint

Dive into the research topics of 'GradToken: Decoupling tokens with class-aware gradient for visual explanation of Transformer network'. Together they form a unique fingerprint.

Cite this