Privileged information assisted learning from noisy correspondence

Noisy correspondence which refers to the mismatch in the collected paired data, can inevitably degrade the performance and generalization of cross-modal models. To address this, previous works typically focus on internal signals, such as model certainty or softening hard labels, to mitigate the influence of noise. Distinctly, we explore a novel external structure by leveraging privileged information with a core intuition: both modalities of a matched pair should be closely correlated with their shared privileged information, while for a mismatched pair, at least one modality will likely fail to align. Specifically, we propose a novel Privileged Information Assisted Learning method, which uses privileged information to explain away noisy correspondence by deriving an adaptive weighting mechanism. PIAL first disentangles the problem by estimating preliminary indicators for both cross-modal and the privileged information correspondence, then introduces a confidence-oriented fusion function to arrive at the final weighting term. Extensive experiments demonstrate the rationality, compatibility and consistent effectiveness of PIAL over the current state-of-the-art methods.