SENA: Leveraging set-level consistency adversarial learning for robust pre-trained language model adaptation

Using pre-trained language models (PLM) to generate embeddings for downstream tasks has achieved great success in recent years. The pre-trained embeddings can be adapted to downstream tasks by encouraging the embedding similarity among samples within the same class through auxiliary tasks with contrastive learning (CL) objectives. However, existing methods face two issues: (i) class imbalance and over-representation caused by instance sampling bias in CL, and (ii) gradient conflicts between auxiliary and downstream tasks. To deal with these issues, we propose a novel approach called set-level consistency adversarial learning (SENA). Specifically, SENA leverages on two techniques, i.e., instance-to-set function and consistency adversarial learning, to yeild task-specific embeddings. To mitigate the issue of instance sampling bias in CL, SENA incorporates set-level discriminative features into individual instance embeddings by employing an instance-to-set function, which are then employed as prototypes for each category in contrastive learning. Additionally, to tackle gradient conflicts between CL and downstream tasks, SENA first identifies the most inconsistent cases and then eliminates the inconsistency in an adversarial learning manner. SENA is validated on GLUE benchmark and three intent classification datasets. Comprehensive experiments demonstrate the effectiveness of SENA on various tasks.