Server-Client Collaborative Distillation for Federated Reinforcement Learning

Weiming Mai, Jiangchao Yao*, Chen Gong, Ya Zhang, Yiu Ming Cheung, Bo Han*

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

1 Citation (Scopus)

Abstract

Federated Learning (FL) learns a global model in a distributional manner, which does not require local clients to share private data. Such merit has drawn lots of attention in the interaction scenarios, where Federated Reinforcement Learning (FRL) emerges as a cross-field research direction focusing on the robust training of agents. Different from FL, the heterogeneity problem in FRL is more challenging because the data depends on the policy of agents and the environment dynamics. FRL learns to interact under the non-stationary environment feedback, while the typical FL methods aim at handling the constant data heterogeneity. In this article, we are among the first attempts to analyze the heterogeneity problem in FRL and propose an off-policy FRL framework. Specifically, a student-teacher-student model learning and fusion method, termed as Server-Client Collaborative Distillation (SCCD), is introduced. Unlike the traditional FL, we distill all local models on the server side for model fusion. To reduce the variance of the training, a local distillation is also conducted every time the agent receives the global model. Experimentally, we compare SCCD with a range of straightforward combinations between FL methods and RL. The results demonstrate that SCCD has a superior performance in four classical continuous control tasks with non-IID environments.

Original languageEnglish
Article number9
Pages (from-to)1–22
Number of pages22
JournalACM Transactions on Knowledge Discovery from Data
Volume18
Issue number1
Early online date19 Jun 2023
DOIs
Publication statusPublished - Jan 2024

Scopus Subject Areas

  • Computer Science(all)

User-Defined Keywords

  • Federated learning
  • collaborative learning
  • heterogeneous environment
  • PhrasesFederated learning

Fingerprint

Dive into the research topics of 'Server-Client Collaborative Distillation for Federated Reinforcement Learning'. Together they form a unique fingerprint.

Cite this