Accurate Forgetting for Heterogeneous Federated Continual Learning

Abudukelimu Wuerkaixi, Sen Cui, Jingfeng Zhang, Kunda Yan, Bo Han, Gang Niu, Lei Fang, Changshui Zhang*, Masashi Sugiyama*

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference proceedingpeer-review

1 Citation (Scopus)

Abstract

Recent years have witnessed a burgeoning interest in federated learning (FL). However, the contexts in which clients engage in sequential learning remain under- explored. Bridging FL and continual learning (CL) gives rise to a challenging practical problem: federated continual learning (FCL). Existing research in FCL primarily focuses on mitigating the catastrophic forgetting issue of continual learning while collaborating with other clients. We argue that forgetting phenomena are not invariably detrimental. In this paper, we consider a more practical and challenging FCL setting characterized by potentially unrelated or even antagonistic data/tasks across different clients. In the FL scenario, statistical heterogeneity and data noise among clients may exhibit spurious correlations which result in biased feature learning. While existing CL strategies focus on the complete utilization of previous knowledge, we found that forgetting biased information was beneficial in our study. Therefore, we propose a new concept accurate forgetting (AF) and develop a novel generative-replay method AF-FCL that selectively utilizes previous knowledge in federated networks. We employ a probabilistic framework based on a normalizing flow model to quantify the credibility of previous knowledge. Comprehensive experiments affirm the superiority of our method over baselines.
Original languageEnglish
Title of host publicationProceedings of the Twelfth International Conference on Learning Representations, ICLR 2024
PublisherInternational Conference on Learning Representations
Pages1-19
Number of pages19
Publication statusPublished - May 2024
Event12th International Conference on Learning Representations, ICLR 2024 - Messe Wien Exhibition and Congress Center, Vienna, Austria
Duration: 7 May 202411 May 2024
https://iclr.cc/Conferences/2024 (Conference website)
https://iclr.cc/virtual/2024/calendar (Conference schedule )
https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-accept-oral (Conference proceedings)

Publication series

NameProceedings of the International Conference on Learning Representations, ICLR

Conference

Conference12th International Conference on Learning Representations, ICLR 2024
Country/TerritoryAustria
CityVienna
Period7/05/2411/05/24
Internet address

Scopus Subject Areas

  • Language and Linguistics
  • Computer Science Applications
  • Education
  • Linguistics and Language

Fingerprint

Dive into the research topics of 'Accurate Forgetting for Heterogeneous Federated Continual Learning'. Together they form a unique fingerprint.

Cite this