A Layer-Wise Personalization Approach for Transformer-Based Federated Anomaly Detection

Personalized Federated Learning (PFL) tools have been recently applied in Anomaly Detection (AD) setups to accurately monitor complex industrial systems under data heterogeneity while complying with strict privacy regulations. PFL techniques integrating transformer models in AD setups are still overlooked even though they provide outstanding performances that are hardly matched by other Neural Network (NN) architectures. This paper thus focuses on developing transformer-based PFL techniques in AD contexts to improve AD accuracy under data heterogeneity. Specifically, we propose decoupling the FL optimization process in a layer-wise manner by carefully selecting which model fragments are learned collaboratively and which are personalized (i.e., trained without cooperation). We refer to our proposed methodology as Layer-Wise Personalized FL (LPFL). The developed approach is evaluated with four design choices for selecting the model layers tailored according to the peculiar architecture of transformer NNs (e.g., the self-attention mechanism). Experimental results on four widely-adopted AD datasets highlight that the self-attention mechanism should always be learned collaboratively while all other trainable parameters should be personalized. Adopting such a choice boosts AD accuracy and reduces the communication overhead by up to 16% and 52%, respectively, compared to other personalization choices, standard FL policies, and individual training strategies.