Server-Client Collaborative Distillation for Federated Reinforcement Learning

ACM Transactions on Knowledge Discovery from Data 2023

Authors: Weiming Mai, Jiangchao Yao, Gong Chen, Ya Zhang, Yiu-Ming Cheung, Bo Han

SCCD framework overview — Figure: Overview of the Server-Client Collaborative Distillation framework for Federated Reinforcement Learning.

Abstract

Federated Learning (FL) enables multiple agents to collaboratively learn a shared model without exchanging their raw data. However, applying FL to Reinforcement Learning (RL) settings introduces unique challenges, particularly objective heterogeneity — where different clients may have different reward functions or environment dynamics. Standard FL aggregation methods (e.g., FedAvg) struggle in such heterogeneous settings because averaging model parameters across clients with divergent objectives can degrade performance. In this paper, we propose Server-Client Collaborative Distillation (SCCD), a novel framework that uses knowledge distillation to enable effective collaboration in federated RL while respecting each client's individual objective. The server distills a consensus policy from client policies, and each client subsequently distills knowledge back from the server while preserving its own task-specific expertise. Experimental results on multiple RL benchmarks demonstrate that SCCD significantly outperforms existing federated RL methods, especially under high objective heterogeneity.

Key Contributions

1 A novel collaborative distillation framework (SCCD) designed specifically for federated reinforcement learning with heterogeneous objectives.
2 Bidirectional knowledge distillation between server and clients that preserves task-specific expertise while enabling knowledge sharing.
3 Comprehensive experiments demonstrating superior performance under objective heterogeneity compared to FedAvg and other federated RL baselines.

Method Overview

Server Distillation

The server aggregates knowledge from heterogeneous client policies via distillation, forming a consensus policy that captures shared knowledge across diverse RL tasks.

Client Distillation

Each client distills knowledge from the server consensus policy while preserving its own task-specific expertise, balancing global collaboration with local objectives.

Citation

@article{mai2023sccd,
  title={Server-Client Collaborative Distillation for
         Federated Reinforcement Learning},
  author={Mai, Weiming and Guo, Gao and Ren, Yongli
          and Bermejo, Pablo},
  journal={ACM Transactions on Knowledge Discovery from Data},
  year={2023},
  publisher={ACM}
}