DARKER: Efficient Transformer with Data-driven Attention Mechanism for Time Series

Transformer-based model shave facilitated numerous applications with superior performance. A key challenge in transformers is the quadratic dependency of it straining time complexity on the length of the input sequence. A recent popular solution is using random feature attention (RFA) to approximate the costly vanilla attention mechanism. However, RFA relies on only a single, fixed projection for approximation, which does not capture the input distribution and can lead to low efficiency and accuracy, especially on time series data. In this paper, we propose DARKER, an efficient transformer with an ovel DAta-dRiven KERnel-based attention mechanism. To precisely present the technical details, this paper discusses them with a fundamental time series task, namely, time series classification(tsc). First, the main novelty of DARKER lies in approximating the soft maxkernel by learning multiple machine learning models with train able weights as multiple projections offline, moving beyond the limitation of a fixed projection. Second, we propose a projection index (calledpIndex) to efficiently search the most suitable projection for the input for training transformer. As a result, the over all time complexity of DARKER is linear with the input length. Third, we propose an indexing technique for efficiently computing the inputs required for transformer training. Finally, we evaluate our method on 14 real-world and 2 synthetic time series data sets. The experiments show that DARKER is 3×4×faster than vanilla transformer and 1.5×-3×faster than other SOTAs for long sequences. In addition, the accuracy of DARKER is comparable to or higher than that of all compared transformers.