Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model

Gongzheng Li; Yadong Xi; Jingzhen Ding; Duan Wang; Ziyang Luo; Rongsheng Zhang; Bai Liu; Changjie Fan; Xiaoxi Mao; Zeng Zhao

Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model

Gongzheng Li, Yadong Xi, Jingzhen Ding, Duan Wang, Ziyang Luo, Rongsheng Zhang, Bai Liu, Changjie Fan, Xiaoxi Mao, Zeng Zhao^*

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

1 Citation (Scopus)

Abstract

Recently, large-scale transformer-based models have been proven to be effective over various tasks across many domains. Nevertheless, applying them in industrial production requires tedious and heavy works to reduce inference costs. To fill such a gap, we introduce a scalable inference solution: Easy and Efficient Transformer (EET), including a series of transformer inference optimization at the algorithm and implementation levels. First, we design highly optimized kernels for long inputs and large hidden sizes. Second, we propose a flexible CUDA memory manager to reduce the memory footprint when deploying a large model. Compared with the state-of-the-art transformer inference library (Faster Transformer v4.0), EET can achieve an average of 1.40-4.20x speedup on the transformer decoder layer with an A100 GPU.

Original language	English
Title of host publication	Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track
Editors	Anastassia Loukina, Rashmi Gangadharaiah, Bonan Min
Publisher	Association for Computational Linguistics (ACL)
Pages	62-68
Number of pages	7
ISBN (Electronic)	9781955917728
Publication status	Published - 10 Jul 2022
Event	2022 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2022 - Virtual, Seattle, United States Duration: 10 Jul 2022 → 15 Jul 2022 https://2022.naacl.org/ (Conference website) https://2022.naacl.org/downloads/handbook-final-v2.pdf (Conference handbook) https://aclanthology.org/events/naacl-2022/ (Conference proceedings)

Publication series

Name	Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers

Conference

Conference	2022 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2022
Country/Territory	United States
City	Seattle
Period	10/07/22 → 15/07/22
Internet address	https://2022.naacl.org/ (Conference website) https://2022.naacl.org/downloads/handbook-final-v2.pdf (Conference handbook) https://aclanthology.org/events/naacl-2022/ (Conference proceedings)

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

https://aclanthology.org/2022.naacl-industry.8/

Cite this

Li, G., Xi, Y., Ding, J., Wang, D., Luo, Z., Zhang, R., Liu, B., Fan, C., Mao, X., & Zhao, Z. (2022). Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model. In A. Loukina, R. Gangadharaiah, & B. Min (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track (pp. 62-68). (Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers). Association for Computational Linguistics (ACL). https://aclanthology.org/2022.naacl-industry.8/

Li, Gongzheng ; Xi, Yadong ; Ding, Jingzhen et al. / Easy and Efficient Transformer : Scalable Inference Solution For Large NLP Model. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track. editor / Anastassia Loukina ; Rashmi Gangadharaiah ; Bonan Min. Association for Computational Linguistics (ACL), 2022. pp. 62-68 (Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers).

@inproceedings{b86d323617694ac0bb604e5990d703d1,

title = "Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model",

abstract = "Recently, large-scale transformer-based models have been proven to be effective over various tasks across many domains. Nevertheless, applying them in industrial production requires tedious and heavy works to reduce inference costs. To fill such a gap, we introduce a scalable inference solution: Easy and Efficient Transformer (EET), including a series of transformer inference optimization at the algorithm and implementation levels. First, we design highly optimized kernels for long inputs and large hidden sizes. Second, we propose a flexible CUDA memory manager to reduce the memory footprint when deploying a large model. Compared with the state-of-the-art transformer inference library (Faster Transformer v4.0), EET can achieve an average of 1.40-4.20x speedup on the transformer decoder layer with an A100 GPU.",

author = "Gongzheng Li and Yadong Xi and Jingzhen Ding and Duan Wang and Ziyang Luo and Rongsheng Zhang and Bai Liu and Changjie Fan and Xiaoxi Mao and Zeng Zhao",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2022 ; Conference date: 10-07-2022 Through 15-07-2022",

year = "2022",

month = jul,

day = "10",

language = "English",

series = "Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers",

publisher = "Association for Computational Linguistics (ACL)",

pages = "62--68",

editor = "Anastassia Loukina and Rashmi Gangadharaiah and Bonan Min",

booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track",

address = "Australia",

url = "https://2022.naacl.org/, https://2022.naacl.org/downloads/handbook-final-v2.pdf, https://aclanthology.org/events/naacl-2022/",

}

Li, G, Xi, Y, Ding, J, Wang, D, Luo, Z, Zhang, R, Liu, B, Fan, C, Mao, X & Zhao, Z 2022, Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model. in A Loukina, R Gangadharaiah & B Min (eds), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers, Association for Computational Linguistics (ACL), pp. 62-68, 2022 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2022, Seattle, Washington, United States, 10/07/22. <https://aclanthology.org/2022.naacl-industry.8/>

Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model. / Li, Gongzheng; Xi, Yadong; Ding, Jingzhen et al.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track. ed. / Anastassia Loukina; Rashmi Gangadharaiah; Bonan Min. Association for Computational Linguistics (ACL), 2022. p. 62-68 (Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers).

Research output: Chapter in book/report/conference proceeding › Conference proceeding › peer-review

TY - GEN

T1 - Easy and Efficient Transformer

T2 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2022

AU - Li, Gongzheng

AU - Xi, Yadong

AU - Ding, Jingzhen

AU - Wang, Duan

AU - Luo, Ziyang

AU - Zhang, Rongsheng

AU - Liu, Bai

AU - Fan, Changjie

AU - Mao, Xiaoxi

AU - Zhao, Zeng

PY - 2022/7/10

Y1 - 2022/7/10

N2 - Recently, large-scale transformer-based models have been proven to be effective over various tasks across many domains. Nevertheless, applying them in industrial production requires tedious and heavy works to reduce inference costs. To fill such a gap, we introduce a scalable inference solution: Easy and Efficient Transformer (EET), including a series of transformer inference optimization at the algorithm and implementation levels. First, we design highly optimized kernels for long inputs and large hidden sizes. Second, we propose a flexible CUDA memory manager to reduce the memory footprint when deploying a large model. Compared with the state-of-the-art transformer inference library (Faster Transformer v4.0), EET can achieve an average of 1.40-4.20x speedup on the transformer decoder layer with an A100 GPU.

AB - Recently, large-scale transformer-based models have been proven to be effective over various tasks across many domains. Nevertheless, applying them in industrial production requires tedious and heavy works to reduce inference costs. To fill such a gap, we introduce a scalable inference solution: Easy and Efficient Transformer (EET), including a series of transformer inference optimization at the algorithm and implementation levels. First, we design highly optimized kernels for long inputs and large hidden sizes. Second, we propose a flexible CUDA memory manager to reduce the memory footprint when deploying a large model. Compared with the state-of-the-art transformer inference library (Faster Transformer v4.0), EET can achieve an average of 1.40-4.20x speedup on the transformer decoder layer with an A100 GPU.

UR - http://www.scopus.com/inward/record.url?scp=85137689786&partnerID=8YFLogxK

M3 - Conference proceeding

AN - SCOPUS:85137689786

T3 - Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers

SP - 62

EP - 68

BT - Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track

A2 - Loukina, Anastassia

A2 - Gangadharaiah, Rashmi

A2 - Min, Bonan

PB - Association for Computational Linguistics (ACL)

Y2 - 10 July 2022 through 15 July 2022

ER -

Li G, Xi Y, Ding J, Wang D, Luo Z, Zhang R et al. Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model. In Loukina A, Gangadharaiah R, Min B, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track. Association for Computational Linguistics (ACL). 2022. p. 62-68. (Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers).

Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model

Abstract

Publication series

Conference

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this