DARKER: Efficient Transformer with Data-driven Attention Mechanism for Time Series

Rundong Zuo, Guozhong Li*, Rui Cao, Byron Choi*, Jianliang Xu, Sourav S. Bhowmick

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

Abstract

Transformer-based model shave facilitated numerous applications with superior performance. A key challenge in transformers is the quadratic dependency of it straining time complexity on the length of the input sequence. A recent popular solution is using random feature attention (RFA) to approximate the costly vanilla attention mechanism. However, RFA relies on only a single, fixed projection for approximation, which does not capture the input distribution and can lead to low efficiency and accuracy, especially on time series data. In this paper, we propose DARKER, an efficient transformer with an ovel DAta-dRiven KERnel-based attention mechanism. To precisely present the technical details, this paper discusses them with a fundamental time series task, namely, time series classification(tsc). First, the main novelty of DARKER lies in approximating the soft maxkernel by learning multiple machine learning models with train able weights as multiple projections offline, moving beyond the limitation of a fixed projection. Second, we propose a projection index (calledpIndex) to efficiently search the most suitable projection for the input for training transformer. As a result, the over all time complexity of DARKER is linear with the input length. Third, we propose an indexing technique for efficiently computing the inputs required for transformer training. Finally, we evaluate our method on 14 real-world and 2 synthetic time series data sets. The experiments show that DARKER is 3×4×faster than vanilla transformer and 1.5×-3×faster than other SOTAs for long sequences. In addition, the accuracy of DARKER is comparable to or higher than that of all compared transformers.

Original languageEnglish
Pages (from-to)3229-3242
Number of pages14
JournalProceedings of the VLDB Endowment
Volume17
Issue number11
DOIs
Publication statusPublished - Jul 2024
Event50th International Conference on Very Large Data Bases, VLDB 2024 - Guangzhou, China
Duration: 26 Aug 202430 Aug 2024
https://vldb.org/2024/ (Conference website)
https://dl.acm.org/toc/pvldb/2024/17/11 (Conference proceeding)

Scopus Subject Areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Cite this