Towards Clinically Accurate Medical Image Report Generation Using Deep Generative Models

Medical images are complex. We typically need radiologists' expertise to examine them, recognize normal and abnormal findings, and prepare the medical image report. The process of medical image examination and reporting sometimes can be error-prone, especially when the workload is excessive, e.g., due to staff shortage. Computer-aided reporting systems therefore will be helpful, especially if the reports can be generated automatically to be verified by the medical doctors. As compared to other text generation application, the key challenge of medical image report generation is the report’s clinical accuracy. Despite the recent attempts to tackle the challenge, their performance in terms of clinical accuracy is still far from satisfactory.

In this project, we propose to develop computational algorithms to address the research challenges which hinder the clinical accuracy of medical report generation, and integrate them to be a complete system. In particular, we will develop a robust deep generative model with an attributed knowledge graph of abnormality so that more detailed aspects of the abnormality can be captured to make the generation of more clinically accurate reports possible. Also, we will develop a computational methodology to automatically infer the structure of the attributed knowledge graph of abnormality based on annotated medical image reports. To leverage the relevant sentences to be retrieved from the existing medical reports to assist the report generation, sentence semantics is the key. We propose to adopt a hierarchical contrastive self- learning approach for better capturing the semantics of sentences in medical reports. All the proposed methods will be evaluated based on a number of benchmark datasets, and their clinical correctness will also be evaluated by clinical experts. We will also develop a related software package to be shared via GitHub. An interdisciplinary research team has been formed with expertise on health data analytics, natural language processing, and radiology which are essential to be able to identify the key issues and evaluate the effectiveness of the system.