TY - JOUR
T1 - Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture
AU - Kong, Chenqi
AU - Luo, Anwei
AU - Bao, Peijun
AU - Li, Haoliang
AU - Wan, Renjie
AU - Zheng, Zengwei
AU - Rocha, Anderson
AU - Kot, Alex C.
N1 - This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62502187; in part by the Natural Science Foundation of Jiangxi Province of China under Grant 20252BAC240015; in part by Sichuan Science and Technology Fund under Grant 2025ZNS-FSC0511; in part by the Research Grant Council (RGC) of Hong Kong, SAR, under a General Research Fund (GRF) Grant 12203124 and an Early Career Scheme (ECS) Grant 22201125; in part by São Paulo Research Foundation (Fapesp) Horus Project under Grant 2023/12865-8; in part by Brazilian National Council for Scientific and Technological Development (CNPq), Aletheia Project under Grant 442229/2024-0; in part by the Rapid-Rich Object Search (ROSE) Laboratory, School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore; and in part by the Agency for Science, Technology and Research (A*STAR) under its Online Trust and Safety (OTS) Research Program under Award S24T2TS006.
Publisher Copyright:
© Copyright 2026 IEEE
PY - 2026/7
Y1 - 2026/7
N2 - Open-set face forgery detection poses significant security threats and presents substantial challenges for existing detection models. These detectors primarily have two limitations: they cannot generalize across unknown forgery domains or inefficiently adapt to new data. To address these issues, we introduce an approach that is both general and parameter-efficient for face forgery detection. Our method builds on the assumption that different forgery source domains exhibit distinct style statistics. Specifically, we design a forgery-style-mixture formulation that augments the diversity of forgery source domains, enhancing the model’s generalizability across unseen domains. In addition, previous methods typically require fully fine-tuning pretrained networks, consuming substantial time and computational resources. Drawing on recent advancements in vision transformers (ViT) for face forgery detection, we develop a parameter-efficient ViT-based detection model that includes lightweight forgery feature extraction modules and enables the model to extract global and local forgery clues simultaneously. We only optimize the inserted lightweight modules during training, maintaining the original ViT structure with its pre-trained weights. This training strategy effectively preserves the informative pre-trained knowledge while flexibly adapting the model to the task of Deepfake detection. Extensive experimental results demonstrate that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters, representing an important step toward open-set Deepfake detection in the wild.
AB - Open-set face forgery detection poses significant security threats and presents substantial challenges for existing detection models. These detectors primarily have two limitations: they cannot generalize across unknown forgery domains or inefficiently adapt to new data. To address these issues, we introduce an approach that is both general and parameter-efficient for face forgery detection. Our method builds on the assumption that different forgery source domains exhibit distinct style statistics. Specifically, we design a forgery-style-mixture formulation that augments the diversity of forgery source domains, enhancing the model’s generalizability across unseen domains. In addition, previous methods typically require fully fine-tuning pretrained networks, consuming substantial time and computational resources. Drawing on recent advancements in vision transformers (ViT) for face forgery detection, we develop a parameter-efficient ViT-based detection model that includes lightweight forgery feature extraction modules and enables the model to extract global and local forgery clues simultaneously. We only optimize the inserted lightweight modules during training, maintaining the original ViT structure with its pre-trained weights. This training strategy effectively preserves the informative pre-trained knowledge while flexibly adapting the model to the task of Deepfake detection. Extensive experimental results demonstrate that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters, representing an important step toward open-set Deepfake detection in the wild.
KW - Deepfakes
KW - face forgery detection
KW - generalization
KW - open-set
KW - parameter-efficient learning
KW - robustness
KW - style mixture
UR - https://www.scopus.com/pages/publications/105032257127
U2 - 10.1109/TCSVT.2026.3669190
DO - 10.1109/TCSVT.2026.3669190
M3 - Journal article
SN - 1051-8215
VL - 36
SP - 10480
EP - 10496
JO - IEEE Transactions on Circuits and Systems for Video Technology
JF - IEEE Transactions on Circuits and Systems for Video Technology
IS - 7
ER -