Towards Clinically Accurate Medical Image Report Generation Using Deep Generative Models

Project: Research project

Project Details


Medical images are complex. We typically need radiologists' expertise to examine them, recognize normal and abnormal findings, and prepare the medical image report. The process of medical image examination and reporting sometimes can be error-prone, especially when the workload is excessive, e.g., due to staff shortage. Computer-aided reporting systems therefore will be helpful, especially if the reports can be generated automatically to be verified by the medical doctors. As compared to other text generation application, the key challenge of medical image report generation is the report’s clinical accuracy. Despite the recent attempts to tackle the challenge, their performance in terms of clinical accuracy is still far from satisfactory.

In this project, we propose to develop computational algorithms to address the research challenges which hinder the clinical accuracy of medical report generation, and integrate them to be a complete system. In particular, we will develop a robust deep generative model with an attributed knowledge graph of abnormality so that more detailed aspects of the abnormality can be captured to make the generation of more clinically accurate reports possible. Also, we will develop a computational methodology to automatically infer the structure of the attributed knowledge graph of abnormality based on annotated medical image reports. To leverage the relevant sentences to be retrieved from the existing medical reports to assist the report generation, sentence semantics is the key. We propose to adopt a hierarchical contrastive self- learning approach for better capturing the semantics of sentences in medical reports. All the proposed methods will be evaluated based on a number of benchmark datasets, and their clinical correctness will also be evaluated by clinical experts. We will also develop a related software package to be shared via GitHub. An interdisciplinary research team has been formed with expertise on health data analytics, natural language processing, and radiology which are essential to be able to identify the key issues and evaluate the effectiveness of the system.
Effective start/end date1/01/2231/12/24

UN Sustainable Development Goals

In 2015, UN member states agreed to 17 global Sustainable Development Goals (SDGs) to end poverty, protect the planet and ensure prosperity for all. This project contributes towards the following SDG(s):

  • SDG 3 - Good Health and Well-being


Explore the research topics touched on by this project. These labels are generated based on the underlying awards/grants. Together they form a unique fingerprint.