Spatial-temporal Regularized Multi-modality Correlation Filters for Tracking with Re-detection

Xiangyuan Lan, Zifei Yang, Wei Zhang, Pong Chi Yuen*

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

7 Citations (Scopus)
67 Downloads (Pure)


The development of multi-spectrum image sensing technology has brought great interest in exploiting the information of multiple modalities (e.g., RGB and infrared modalities) for solving computer vision problems. In this article, we investigate how to exploit information from RGB and infrared modalities to address two important issues in visual tracking: robustness and object re-detection. Although various algorithms that attempt to exploit multi-modality information in appearance modeling have been developed, they still face challenges that mainly come from the following aspects: (1) the lack of robustness to deal with large appearance changes and dynamic background, (2) failure in re-capturing the object when tracking loss happens, and (3) difficulty in determining the reliability of different modalities. To address these issues and perform effective integration of multiple modalities, we propose a new tracking-by-detection algorithm called Adaptive Spatial-temporal Regulated Multi-Modality Correlation Filter. Particularly, an adaptive spatial-temporal regularization is imposed into the correlation filter framework in which the spatial regularization can help to suppress effect from the cluttered background while the temporal regularization enables the adaptive incorporation of historical appearance cues to deal with appearance changes. In addition, a dynamic modality weight learning algorithm is integrated into the correlation filter training, which ensures that more reliable modalities gain more importance in target tracking. Experimental results demonstrate the effectiveness of the proposed method.

Original languageEnglish
Article number57
Pages (from-to)1-16
Number of pages16
JournalACM Transactions on Multimedia Computing, Communications and Applications
Issue number2
Publication statusPublished - May 2021

Scopus Subject Areas

  • Hardware and Architecture
  • Computer Networks and Communications

User-Defined Keywords

  • behavior understanding
  • Multi-modality fusion
  • tracking


Dive into the research topics of 'Spatial-temporal Regularized Multi-modality Correlation Filters for Tracking with Re-detection'. Together they form a unique fingerprint.

Cite this