Semantic-driven representation disentanglement for robust open-set retrieval of cross-modal remote sensing images

Cross-modal retrieval seeks to retrieve content-relevant data from databases of heterogeneous modalities. Existing methods typically assess content relevance of pre-defined categories through similarity measurement within a common space, which is constructed by learning modality-invariant representations. However, in real-world scenarios, the database may contain instances from unseen categories, whose content relevance cannot be effectively characterized by the mapping learned under the closed-set assumption. In addition, the presence of external noise is inevitable when collecting remote sensing data under diverse conditions. Consequently, data representations tend to erroneously match unrelated instances or deviate from the distribution of corresponding contents, resulting in unreliable retrieval outputs. This challenge is further compounded by the heterogeneity of different remote sensing platforms, which complicates the extraction of unified representations across modalities. To solve this problem, we propose semantic-driven representation disentanglement (SDRD) for robust open-set cross-modal retrieval of remote sensing images. Driven by the semantic information, input from each modality is disentangled into apparent and intrinsic representations. With the removal of modality-specific and noise factors, the intrinsic representations exhibit inherently strong content discriminability and noise robustness. Then, an interactive reconstruction strategy is employed to facilitate the capture of semantic information, thereby strengthening the correlation between intrinsic representations and underlying content. Moreover, the distribution gap between modalities in the common space is further mitigated by adopting a heterogeneous triplet loss. Extensive experiments on two large-scale cross-modal remote sensing datasets under different scenarios demonstrate that the proposed method consistently achieves state-of-the-art performance.