Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation

Sixing Yan*, William K. Cheung, Keith Chiu, Terence M. Tong, Ka Chun Cheung, Simon See

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review


Despite the recent success of deep learning models for text generation, generating clinically accurate reports remains challenging. More precisely modeling the relationships of the abnormalities revealed in an X-ray image has been found promising to enhance the clinical accuracy. In this paper, we first introduce a novel knowledge graph structure called an attributed abnormality graph (ATAG). It consists of interconnected abnormality nodes and attribute nodes for better capturing more fine-grained abnormality details. In contrast to the existing methods where the abnormality graph are constructed manually, we propose a methodology to automatically construct the fine-grained graph structure based on annotated X-ray reports and the RadLex radiology lexicon. We then learn the ATAG embeddings as part of a deep model with an encoder-decoder architecture for the report generation. In particular, graph attention networks are explored to encode the relationships among the abnormalities and their attributes. A hierarchical attention attention and a gating mechanism are specifically designed to further enhance the generation quality. We carry out extensive experiments based on the benchmark datasets, and show that the proposed ATAG-based deep model outperforms the SOTA methods by a large margin in ensuring the clinical accuracy of the generated reports.
Original languageEnglish
Pages (from-to)2211-2222
Number of pages12
JournalIEEE Transactions on Medical Imaging
Issue number8
Early online date15 Feb 2023
Publication statusPublished - Aug 2023

Scopus Subject Areas

  • Software
  • Radiological and Ultrasound Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

User-Defined Keywords

  • Visualization
  • Radiology
  • Lung
  • X-ray imaging
  • Biomedical imaging
  • Annotations
  • Feature extraction
  • Medical report generation
  • deep learning models
  • attributed abnormality graphs
  • radiology lexicon
  • clinical accuracy


Dive into the research topics of 'Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation'. Together they form a unique fingerprint.

Cite this