Cooperative Colorization: Exploring Latent Cross-Domain Priors for NIR Image Spectrum Translation

Xingxing Yang, Jie Chen*, Zaifeng Yang

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review


Near-infrared (NIR) image spectrum translation is a challenging problem with many promising applications. Existing methods struggle with the mapping ambiguity between the NIR and the RGB domains, and generalize poorly due to the limitations of models' learning capabilities and the unavailability of sufficient NIR-RGB image pairs for training. To address these challenges, we propose a cooperative learning paradigm that colorizes NIR images in parallel with another proxy grayscale colorization task by exploring latent cross-domain priors (i.e., latent spectrum context priors and task domain priors), dubbed CoColor. The complementary statistical and semantic spectrum information from these two task domains -- in the forms of pre-trained colorization networks -- are brought in as task domain priors. A bilateral domain translation module is subsequently designed, in which intermittent NIR images are generated from grayscale and colorized in parallel with authentic NIR images; and vice versa for the grayscale images. These intermittent transformations act as latent spectrum context priors for efficient domain knowledge exchange. We progressively fine-tune and fuse these modules with a series of pixel-level and feature-level consistency constraints. Experiments show that our proposed cooperative learning framework produces satisfactory spectrum translation outputs with diverse colors and rich textures, and outperforms state-of-the-art counterparts by 3.95dB and 4.66dB in terms of PNSR for the NIR and grayscale colorization tasks, respectively.
Original languageEnglish
Title of host publicationMM 2023 - Proceedings of the 31st ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery (ACM)
Number of pages8
ISBN (Print)9798400701085
Publication statusPublished - 27 Oct 2023
Event31st ACM International Conference on Multimedia, MM 2023 - Ottawa, Canada
Duration: 29 Oct 20233 Nov 2023 (Conference proceedings) (Conference website)

Publication series

NameProceedings of the ACM International Conference on Multimedia


Conference31st ACM International Conference on Multimedia, MM 2023
Internet address

User-Defined Keywords

  • Near-Infrared image colorization
  • grayscale image colorization
  • generative prior


Dive into the research topics of 'Cooperative Colorization: Exploring Latent Cross-Domain Priors for NIR Image Spectrum Translation'. Together they form a unique fingerprint.

Cite this