Local Style Transfer via Latent Space Manipulation for Cross-Disease Lesion Segmentation

Fei Lyu, Mang Ye, Terry Cheuk Fung Yip, Grace Lai Hung Wong, Pong C. Yuen*

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review


Automatic lesion segmentation is important for assisting doctors in the diagnostic process. Recent deep learning approaches heavily rely on large-scale datasets, which are difficult to obtain in many clinical applications. Leveraging external labelled datasets is an effective solution to tackle the problem of insufficient training data. In this paper, we propose a new framework, namely LatenTrans, to utilize existing datasets for boosting the performance of lesion segmentation in extremely low data regimes. LatenTrans translates non-target lesions into target-like lesions and expands the training dataset with target-like data for better performance. Images are first projected to the latent space via aligned style-based generative models, and rich lesion semantics are encoded using the latent codes. A novel consistency-aware latent code manipulation module is proposed to enable high-quality local style transfer from non-target lesions to target-like lesions while preserving other parts. Moreover, we propose a new metric, Normalized Latent Distance, to solve the question of how to select an adequate one from various existing datasets for knowledge transfer. Extensive experiments are conducted on segmenting lung and brain lesions, and the experimental results demonstrate that our proposed LatenTrans is superior to existing methods for cross-disease lesion segmentation.

Original languageEnglish
Pages (from-to)273-284
Number of pages12
JournalIEEE Journal of Biomedical and Health Informatics
Issue number1
Early online date26 Oct 2023
Publication statusPublished - Jan 2024

Scopus Subject Areas

  • Computer Science Applications
  • Health Informatics
  • Electrical and Electronic Engineering
  • Health Information Management

User-Defined Keywords

  • Cross-disease lesion segmentation
  • GAN inversion
  • knowledge transfer
  • latent space manipulation


Dive into the research topics of 'Local Style Transfer via Latent Space Manipulation for Cross-Disease Lesion Segmentation'. Together they form a unique fingerprint.

Cite this