DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

Wenfang Yao, Kejing Yin, William K. Cheung, Jia Liu, Jing Qin

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

2 Citations (Scopus)

Abstract

The combination of electronic health records (EHR) and medical images is crucial for clinicians in making diagnoses and forecasting prognoses. Strategically fusing these two data modalities has great potential to improve the accuracy of machine learning models in clinical prediction tasks. However, the asynchronous and complementary nature of EHR and medical images presents unique challenges. Missing modalities due to clinical and administrative factors are inevitable in practice, and the significance of each data modality varies depending on the patient and the prediction target, resulting in inconsistent predictions and suboptimal model performance. To address these challenges, we propose DrFuse to achieve effective clinical multi-modal fusion. It tackles the missing modality issue by disentangling the features shared across modalities and those unique within each modality. Furthermore, we address the modal inconsistency issue via a disease-wise attention layer that produces the patient- and disease-wise weighting for each modality to make the final prediction. We validate the proposed method using real-world large-scale datasets, MIMIC-IV and MIMIC-CXR. Experimental results show that the proposed method significantly outperforms the state-of-the-art models.

Original languageEnglish
Title of host publicationProceedings of the 38th AAAI Conference on Artificial Intelligence
PublisherAAAI press
Pages16416-16424
Number of pages9
ISBN (Print)1577358872 , 9781577358879
DOIs
Publication statusPublished - 25 Mar 2024
Event38th AAAI Conference on Artificial Intelligence, AAAI 2024 - Vancouver, Canada
Duration: 20 Feb 202427 Feb 2024
https://ojs.aaai.org/index.php/AAAI/issue/archive (Conference proceeding)

Publication series

NameProceedings of the AAAI Conference on Artificial Intelligence
PublisherAAAI press
Number15
Volume38
ISSN (Print)2159-5399
ISSN (Electronic)2374-3468

Conference

Conference38th AAAI Conference on Artificial Intelligence, AAAI 2024
Country/TerritoryCanada
CityVancouver
Period20/02/2427/02/24
Internet address

Scopus Subject Areas

  • Artificial Intelligence

User-Defined Keywords

  • ML: Applications
  • APP: Other Applications
  • CV: Multi-modal Vision
  • ML: Representation Learning

Fingerprint

Dive into the research topics of 'DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency'. Together they form a unique fingerprint.

Cite this