DeBLAS: Accelerate LLM Pretraining by Length-based Sequence Scheduling

Pretraining large language models (LLMs) is computationally intensive, typically requiring massive datasets and training iterations. Although recent advances in data selection have shown improvement in training efficiency, their gains often diminish under scaling laws. In this work, we dive into the impact of sequence length on language model pretraining and propose a length-based online data scheduling method for acceleration. Specifically, we design a dense-balanced sequence scheduling framework for LLM pretraining: 1) at the first stage, the model is exposed to uniform-length dense token batches to encourage the formation of global language representations; 2) the second stage incorporates variable- length sequences, which reinforces learned abstractions while significantly reducing the total number of training iterations. Our idealized theoretical analysis suggests that our approach can accelerate language model pretraining through the gradient variance reduction achieved by the combination of both stages. Empirical results show that our approach achieves comparable perplexity to standard pretraining with substantially fewer optimization steps, pinpointing a promising way to reduce the computational burden of LLM pretraining. The source code is publicly available at: https://github.com/warriors-30/DeBLAS.