Triplet fusion network hashing for unpaired cross-modal retrieval

Zhikai Hu, Xin Liu*, Xingzhi Wang, Yiu Ming CHEUNG, Nannan Wang, Yewang Chen

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference contributionpeer-review

2 Citations (Scopus)

Abstract

With the dramatic increase of multi-media data on the Internet, cross-modal retrieval has become an important and valuable task in searching systems. The key challenge of this task is how to build the correlation between multi-modal data. Most existing approaches only focus on dealing with paired data. They use pairwise relationship of multi-modal data for exploring the correlation between them. However, in practice, unpaired data are more common on the Internet but few methods pay attention to them. To utilize both paired and unpaired data, we propose a one-stream framework triplet fusion network hashing (TFNH), which mainly consists of two parts. The first part is a triplet network which is used to handle both kinds of data, with the help of zero padding operation. The second part consists of two data classifiers, which are used to bridge the gap between paired and unpaired data. In addition, we embed manifold learning into the framework for preserving both inter and intra modal similarity, exploring the relationship between unpaired and paired data and bridging the gap between them in learning process. Extensive experiments show that the proposed approach outperforms several state-of-the-art methods on two datasets in paired scenario. We further evaluate its ability of handling unpaired scenario and robustness in regard to pairwise constraint. The results show that even we discard 50% data under the setting in [19], the performance of TFNH is still better than that of other unpaired approaches and that only 70% pairwise relationships are preserved, TFNH can still outperform almost all paired approaches.

Original languageEnglish
Title of host publicationICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages141-149
Number of pages9
ISBN (Electronic)9781450367653
DOIs
Publication statusPublished - 5 Jun 2019
Event2019 ACM International Conference on Multimedia Retrieval, ICMR 2019 - Ottawa, Canada
Duration: 10 Jun 201913 Jun 2019

Publication series

NameICMR 2019 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval

Conference

Conference2019 ACM International Conference on Multimedia Retrieval, ICMR 2019
Country/TerritoryCanada
CityOttawa
Period10/06/1913/06/19

Scopus Subject Areas

  • Software
  • Computer Graphics and Computer-Aided Design
  • Computer Vision and Pattern Recognition

User-Defined Keywords

  • Cross-modal retrieval
  • Hashing technique
  • Triplet network
  • Unpaired data

Fingerprint

Dive into the research topics of 'Triplet fusion network hashing for unpaired cross-modal retrieval'. Together they form a unique fingerprint.

Cite this