Cohesive Multi-modality Feature Learning and Fusion for COVID-19 Patient Severity Prediction

Jinzhao Zhou, Xingming Zhang, Ziwei Zhu, Xiangyuan Lan*, Lunkai Fu, Haoxiang Wang, Hanchun Wen

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

14 Citations (Scopus)


The outbreak of coronavirus disease (COVID-19) has been a nightmare to citizens, hospitals, healthcare practitioners, and the economy in 2020. The overwhelming number of confirmed cases and suspected cases put forward an unprecedented challenge to the hospital's capacity of management and medical resource distribution. To reduce the possibility of cross-infection and attend a patient according to his severity level, expertly diagnosis and sophisticated medical examinations are often required but hard to fulfil during a pandemic. To facilitate the assessment of a patient's severity, this paper proposes a multi-modality feature learning and fusion model for end-To-end covid patient severity prediction using the blood test supported electronic medical record (EMR) and chest computerized tomography (CT) scan images. To evaluate a patient's severity by the co-occurrence of salient clinical features, the High-order Factorization Network (HoFN) is proposed to learn the impact of a set of clinical features without tedious feature engineering. On the other hand, an attention-based deep convolutional neural network (CNN) using pre-Trained parameters are used to process the lung CT images. Finally, to achieve cohesion of cross-modality representation, we design a loss function to shift deep features of both-modality into the same feature space which improves the model's performance and robustness when one modality is absent. Experimental results demonstrate that the proposed multi-modality feature learning and fusion model achieves high performance in an authentic scenario.

Original languageEnglish
Pages (from-to)2535-2549
Number of pages15
JournalIEEE Transactions on Circuits and Systems for Video Technology
Issue number5
Early online date4 Mar 2021
Publication statusPublished - May 2022

Scopus Subject Areas

  • Media Technology
  • Electrical and Electronic Engineering

User-Defined Keywords

  • Multimodality
  • COVID-19 severity prediction
  • factorization methods
  • attention mechanism
  • convolutional neural network


Dive into the research topics of 'Cohesive Multi-modality Feature Learning and Fusion for COVID-19 Patient Severity Prediction'. Together they form a unique fingerprint.

Cite this