Towards Understanding Evolving Patterns in Sequential Data

In many machine learning tasks, data is inherently sequential. Most existing algorithms learn from sequential data in an auto-regressive manner, which predicts the next unseen data point based on the observed sequence, implicitly assuming the presence of an evolving pattern embedded in the data that can be leveraged. However, identifying and assessing evolving patterns in learning tasks heavily relies on human expertise, and lacks a standardized quantitative measure. In this paper, we show that such a measure enables us to determine the suitability of employing sequential models, measure the temporal order of time series data, and conduct feature/data selections, which can be beneficial to a variety of learning tasks: time-series forecastings, classification tasks with temporal distribution shift, video predictions, etc. Specifically, we introduce the EVOLVING RATE (EVORATE), which quantifies the evolving patterns in the data by approximating mutual information between the next data point and the observed sequence. To address cases where the correspondence between data points at different timestamps is absent, we develop EVORATE_W, a simple and efficient implementation that leverages optimal transport to construct the correspondence and estimate the first-order EVORATE. Experiments on synthetic and real-world datasets including images and tabular data validate the efficacy of our EVORATE method.