Accurate Forgetting for Heterogeneous Federated Continual Learning

Published 20 Feb 2025 in cs.LG and cs.AI | (2502.14205v1)

Abstract: Recent years have witnessed a burgeoning interest in federated learning (FL). However, the contexts in which clients engage in sequential learning remain under-explored. Bridging FL and continual learning (CL) gives rise to a challenging practical problem: federated continual learning (FCL). Existing research in FCL primarily focuses on mitigating the catastrophic forgetting issue of continual learning while collaborating with other clients. We argue that the forgetting phenomena are not invariably detrimental. In this paper, we consider a more practical and challenging FCL setting characterized by potentially unrelated or even antagonistic data/tasks across different clients. In the FL scenario, statistical heterogeneity and data noise among clients may exhibit spurious correlations which result in biased feature learning. While existing CL strategies focus on a complete utilization of previous knowledge, we found that forgetting biased information is beneficial in our study. Therefore, we propose a new concept accurate forgetting (AF) and develop a novel generative-replay method~\method~which selectively utilizes previous knowledge in federated networks. We employ a probabilistic framework based on a normalizing flow model to quantify the credibility of previous knowledge. Comprehensive experiments affirm the superiority of our method over baselines.

Abstract PDF Upgrade to Chat

Authors (9)

Summary

The paper introduces AF-FCL, a novel method for selective forgetting that enhances model performance by mitigating biased feature retention.
It employs generative replay with normalizing flow models and knowledge distillation to maintain effective feature retention across sequential tasks.
Experimental results on noisy, heterogeneous datasets demonstrate improved accuracy, highlighting its potential for robust real-world federated applications.

Accurate Forgetting for Heterogeneous Federated Continual Learning

Introduction

Federated Continual Learning (FCL) emerges at the intersection of Federated Learning (FL) and Continual Learning (CL), addressing the need for models to learn sequential tasks in a federated network without breaching data privacy. The concept introduces complex challenges, notably arising from statistical heterogeneity and data noise across clients, which can lead to biased feature learning. This paper explores the dynamics of FCL by presenting a novel approach to selective forgetting, aptly termed Accurate Forgetting (AF), designed to mitigate the adverse effects of biased information retention. The paper posits that strategic forgetting can facilitate better model performance than indiscriminate memorization typical in existing FCL implementations.

Figure 1: Illustration of the FCL problem. Multiple hospitals within a federated learning network engage in the sequential acquisition of disease prediction tasks.

Methodology

The authors propose AF-FCL, a generative replay method employing a probabilistic framework via normalizing flow models to manage knowledge retention selectively. The methodology centers on three pivotal components:

Generative Replay: The paper employs a normalizing flow model to facilitate generative replay within the feature space of classifiers, effectively preserving knowledge without complete reliance on prior task data. This approach strives to retain advantageous features across learning tasks while allowing for the dynamic updating of models.
Knowledge Distillation: To ensure stable training for the NF model and reduce the likelihood of feature drift, knowledge distillation complements the generative replay. This enhances the accuracy of learned features by aligning them with the current task distribution.
Correlation Estimation: A critical innovation is the correlation estimation mechanism that uses probabilistic assessments to evaluate the relevance of features from prior tasks. This selective utilization process effectively filters out biased or antagonistic features that might otherwise degrade the model’s performance.
Figure 2: The diagram of training the classifier locally with our method.

Experimental Results

Experiments attest to AF-FCL's superiority over baseline methods in safeguarding against biased feature retention in federated learning environments characterized by noisy data. Comprehensive evaluations on benchmark datasets such as EMNIST-noisy underscore the efficacy of AF-FCL in maintaining model accuracy amidst increasing numbers of malicious clients.

Figure 3: Illustration of the EMNIST-noisy dataset and results.

Implications and Future Directions

The introduction of Accurate Forgetting in FCL paradigms highlights the nuanced understanding required for optimal model performance across heterogeneous federated networks. Practically, AF-FCL informs more resilient deployment of AI models in scenarios like nationwide hospital networks, where decentralized learning with sensitive data is paramount. Theoretically, this research propels the discourse surrounding generative replay and selective forgetting, paving potential paths for further exploration in mitigating feature bias and spurious correlations.

Future developments could explore deeper integration of orthogonal training and adaptive learning rates within the AF-FCL framework. The adaptation of complementary techniques like orthogonal subspaces could amplify accurate forgetting, affording greater precision in feature selection across diverse learning environments.

Conclusion

The presented work revolutionizes the handling of catastrophic forgetting within federated continual learning frameworks, showing that selective forgetting via correlation estimation can be instrumental in enhancing model reliability under statistical heterogeneity. AF-FCL sets a new precedent for future research aimed at refining federated learning methodologies in complex, real-world applications.