DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks

Ziyang Luo*, Yadong Xi, Jing Ma, Zhiwei Yang, Xiaoxi Mao, Changjie Fan, Rongsheng Zhang

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference contribution

Abstract

Since 2017, the Transformer-based models play critical roles in various downstream Natural Language Processing tasks. However, a common limitation of the attention mechanism utilized in Transformer Encoder is that it cannot automatically capture the information of word order, so explicit position embeddings are generally required to be fed into the target model. In contrast, Transformer Decoder with the causal attention masks is naturally sensitive to the word order. In this work, we focus on improving the position encoding ability of BERT with the causal attention masks. Furthermore, we propose a new pre-trained language model DecBERT and evaluate it on the GLUE benchmark. Experimental results show that (1) the causal attention mask is effective for BERT on the language understanding tasks; (2) our DecBERT model without position embeddings achieve comparable performance on the GLUE benchmark; and (3) our modification accelerates the pre-training process and DecBERT w/ PE achieves better overall performance than the baseline systems when pre-training with the same amount of computational resources.
Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics: NAACL 2022
PublisherAssociation for Computational Linguistics (ACL)
Pages1185–1197
Number of pages13
ISBN (Print)9781955917766
Publication statusPublished - 10 Jul 2022
EventFindings of the Association for Computational Linguistics: NAACL 2022 - Seattle, United States
Duration: 10 Jul 202215 Jul 2022

Conference

ConferenceFindings of the Association for Computational Linguistics: NAACL 2022
Country/TerritoryUnited States
CitySeattle
Period10/07/2215/07/22

Fingerprint

Dive into the research topics of 'DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks'. Together they form a unique fingerprint.

Cite this