TransKV: A Data-Driven Pruning Method for Large Foundation Models

Large Foundation Models (LFMs) have achieved remarkable success across various domains, yet their enormous parameter counts severely hinder deployment on resource-constrained devices. Recent pruning methods leverage low-rank structures in attention mechanisms, with SOTA techniques like CLOVER and Olica targeting redundancies in weight pairs (e.g., W^ i _Q (W^ i _K)^\top, i-th attention head) to enable more aggressive compression than single-matrix pruning. However, these static approaches overlook data-driven interactions, as singular value distributions of static weight pairs differ significantly from data-driven counterparts (e.g., X W^ i _Q (W^ i _K)^\top X^\top), often yielding suboptimal performance. To address this, we introduce TransKV, a data-driven pruning framework that derives a projection matrix from keys (X W^ i _K) and values (X W^ i _V), seamlessly integrating it into weight-pair computations for precise low-rank truncation. TransKV natively supports core LFM components, including Multi-Head Attention (MHA), Grouped Query Attention (GQA), RMSNorm, and Rotary Position Embeddings (RoPE), ensuring compatibility with contemporary LFMs. Extensive experiments on diverse benchmarks across multiple domains validate TransKV's superiority, outperforming SOTA methods on various LFMs (including DeepSeek-OCR) at equivalent compression ratios.