Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation

Sixing Yan*, William K. Cheung, Keith Chiu, Terence M. Tong, Ka Chun Cheung, Simon See

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

12 Citations (Scopus)

Abstract

Despite the recent success of deep learning models for text generation, generating clinically accurate reports remains challenging. More precisely modeling the relationships of the abnormalities revealed in an X-ray image has been found promising to enhance the clinical accuracy. In this paper, we first introduce a novel knowledge graph structure called an attributed abnormality graph (ATAG). It consists of interconnected abnormality nodes and attribute nodes for better capturing more fine-grained abnormality details. In contrast to the existing methods where the abnormality graph are constructed manually, we propose a methodology to automatically construct the fine-grained graph structure based on annotated X-ray reports and the RadLex radiology lexicon. We then learn the ATAG embeddings as part of a deep model with an encoder-decoder architecture for the report generation. In particular, graph attention networks are explored to encode the relationships among the abnormalities and their attributes. A hierarchical attention attention and a gating mechanism are specifically designed to further enhance the generation quality. We carry out extensive experiments based on the benchmark datasets, and show that the proposed ATAG-based deep model outperforms the SOTA methods by a large margin in ensuring the clinical accuracy of the generated reports.
Original languageEnglish
Pages (from-to)2211-2222
Number of pages12
JournalIEEE Transactions on Medical Imaging
Volume42
Issue number8
Early online date15 Feb 2023
DOIs
Publication statusPublished - Aug 2023

Scopus Subject Areas

  • Software
  • Radiological and Ultrasound Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

User-Defined Keywords

  • Visualization
  • Radiology
  • Lung
  • X-ray imaging
  • Biomedical imaging
  • Annotations
  • Feature extraction
  • Medical report generation
  • deep learning models
  • attributed abnormality graphs
  • radiology lexicon
  • clinical accuracy

Fingerprint

Dive into the research topics of 'Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation'. Together they form a unique fingerprint.

Cite this