面向数据分布不平衡问题的联邦学习算法研究

近年来，联邦学习作为一种隐私保护的深度学习范式，受到了广泛的关注。联邦学习利用一个服务器和多个客户端来训练一个适用于所有客户端的深度模型。在训练模型时，联邦学习不会传输原始的训练数据，而是传输模型的参数。作为一种分布式深度学习方法，客户端上的数据分布会对联邦学习算法的性能产生很大影响。具体来说，当参与联邦学习的客户端上的数据分布不平衡时，联邦学习算法的性能会出现较为明显的下降。本文研究了联邦学习中的两类数据分布不平衡问题，分别是：数据标签分布不平衡和数据特征分布不平衡。针对数据标签分布不平衡问题，本文提出了一种基于模型蒸馏的联邦学习算法。算法的核心思想是利用深度互学习来控制本地模型与全局模型之间的距离，既保留了客户端上个性化模型的知识，又能从全局模型中学习到其他客户端上的知识。针对数据特征分布不平衡问题，本文提出了一种基于本地注意力机制的联邦学习算法。该算法包含两种策略，单一类型的注意力机制和多种类型的注意力机制。
  该算法可以与多种已有的联邦学习算法结合使用，不仅可以提升模型的训练效果，并且不会带来额外的通信负载。多个数据集上的实验结果表明，本文提出的联邦学习有助于减轻联邦学习中的数据分布不平衡问题的影响。
  

[1] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition. 2016.

[2] MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]//Artificial intelligence and statistics. PMLR, 2017: 12731282.

[3] LI X, HUANG K, YANG W, et al. On the convergence of fedavg on non-iid data[C]// International Conference on Learning Representations. 2020.

[4] KAIROUZ P, MCMAHAN H B, AVENT B, et al. Advances and open problems in federated learning[J]. Foundations and Trends® in Machine Learning.

[5] LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[C]// Proceedings of Machine Learning and Systems. 2020.

[6] WANG J, LIU Q, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization[C]//Advances in Neural Information Processing Systems. 2020.

[7] KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for federated learning[C]//International Conference on Machine Learning: volume 119. PMLR, 2020: 5132-5143.

[8] ZHUANG W, GAN X, WEN Y, et al. Collaborative unsupervised visual representation learning from decentralized data[C]//IEEE International Conference on Computer Vision. IEEE, 2021: 4892-4901.

[9] LI X, QU Z, TANG B, et al. Fedlga: Towards system-heterogeneity of federated learning via local gradient approximation[J]. CoRR, 2021, abs/2112.11989.

[10] DIAO E, DING J, TAROKH V. Heterofl: Computation and communication efficient federated learning for heterogeneous clients[C]//International Conference on Learning Representations. OpenReview.net, 2021.

[11] KONEČNÝ J, MCMAHAN H B, YU F X, et al. Federated learning: Strategies for improving communication efficiency[J]. CoRR, 2016, abs/1610.05492.

[12] BOUACIDA N, HOU J, ZANG H, et al. Adaptive federated dropout: Improving communication efficiency and generalization for federated learning[J]. IEEE Conference on Computer Communications Workshops, 2021: 1-6.

[13] ASAD M, MOUSTAFA A, ITO T, et al. Evaluating the communication efficiency in federated learning algorithms[J]. International Conference on Computer Supported Cooperative Work in Design, 2021: 552-557.

[14] ZHU L, LIU Z, HAN S. Deep leakage from gradients[C]//Advances in Neural Information Processing Systems. 2019.

[15] ZHAO B, MOPURI K R, BILEN H. idlg: Improved deep leakage from gradients[J]. CoRR, 2020, abs/2001.02610.

[16] GEIPING J, BAUERMEISTER H, DRÖGE H, et al. Inverting gradients - how easy is it to break privacy in federated learning?[C]//Advances in Neural Information Processing Systems. 2020.

[17] YIN H, MALLYA A, VAHDAT A, et al. See through gradients: Image batch recovery via gradinversion[J]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021: 16332-16341.

[18] AWAN S M, LUO B, LI F. Contra: Defending against poisoning attacks in federated learning[C]//European Symposium on Research in Computer Security. 2021.

[19] COSTA G, PINELLI F, SODERI S, et al. Covert channel attack to federated learning systems[J]. CoRR, 2021, abs/2104.10561.

[20] TOLPEGIN V, TRUEX S, GURSOY M E, et al. Data poisoning attacks against federated learning systems[C]//European Symposium on Research in Computer Security. 2020.

[21] LI T, SANJABI M, BEIRAMI A, et al. Fair resource allocation in federated learning[C]// International Conference on Learning Representations. 2020.

[22] LI T, HU S, BEIRAMI A, et al. Ditto: Fair and robust federated learning through personalization[C]//International Conference on Machine Learning. 2021.

[23] HORVÁTH S, LASKARIDIS S, ALMEIDA M, et al. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout[J]. CoRR, 2021, abs/2102.13451.

[24] HAO W, EL-KHAMY M, LEE J, et al. Towards fair federated learning with zero-shot data augmentation[J]. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2021: 3305-3314.

[25] GUO S, ZENG D. Pedagogical data federation toward education 4.0[C]//International Conference on Frontiers of Educational Technologies. 2020: 51-55.

[26] WU J, HUANG Z, LIU Q, et al. Federated deep knowledge tracing[C]//ACM International Conference on Web Search and Data Mining. 2021: 662-670.

[27] DONG N, VOICULESCU I. Federated contrastive learning for decentralized unlabeled medical images[C]//International Conference on Medical Image Computing and Computer Assisted Intervention: volume 12903. Springer, 2021: 378-387.

[28] CHEN Z, ZHU M, YANG C, et al. Personalized retrogress-resilient framework for real-world medical federated learning[C]//International Conference on Medical Image Computing and Computer Assisted Intervention. Springer, 2021: 347-356.

[29] LIU Q, YANG H, DOU Q, et al. Federated semi-supervised medical image classification via inter-client relation matching[C]//International Conference on Medical Image Computing and Computer Assisted Intervention. 2021.

[30] LIU Q, CHEN C, QIN J, et al. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space[J]. IEEE Conference on Computer Vision and Pattern Recognition, 2021: 1013-1023.

[31] ROTH H R, YANG D, LI W, et al. Federated whole prostate segmentation in MRI with personalized neural architectures[C]//International Conference on Medical Image Computing and Computer Assisted Intervention: volume 12903. Springer, 2021: 357-366.

[32] MILLS J, HU J, MIN G. Communication-efficient federated learning for wireless edge intelligence in iot[J]. IEEE Internet of Things Journal, 2020, 7: 5986-5994.

[33] NGUYEN V D, SHARMA S K, VU T X, et al. Efficient federated learning algorithm for resource allocation in wireless iot networks[J]. IEEE Internet of Things Journal, 2021, 8: 33943409.

[34] ZHAO Y, ZHAO J, JIANG L, et al. Privacy-preserving blockchain-based federated learning for iot devices[J]. IEEE Internet of Things Journal, 2021, 8: 1817-1829.

[35] PANG J, HUANG Y, XIE Z, et al. Realizing the heterogeneity: A self-organized federated learning framework for iot[J]. IEEE Internet of Things Journal, 2021, 8: 3088-3098.

[36] ZHANG W, LU Q, YU Q, et al. Blockchain-based federated learning for device failure detection in industrial iot[J]. IEEE Internet of Things Journal, 2021, 8: 5926-5937.

[37] QI Y, HOSSAIN M S, NIE J, et al. Privacy-preserving blockchain-based federated learning for traffic flow prediction[J]. Future Generation Computer Systems, 2021, 117: 328-337.

[38] LIU Y, JAMES J, KANG J, et al. Privacy-preserving traffic flow prediction: A federated learning approach[J]. IEEE Internet of Things Journal, 2020, 7(8): 7751-7763.

[39] UPRETY A, RAWAT D B, LI J. Privacy preserving misbehavior detection in iov using federated machine learning[C]//Annual Consumer Communications & Networking Conference. IEEE, 2021: 1-6.

[40] LIANG X, LIU Y, CHEN T, et al. Federated transfer reinforcement learning for autonomous driving[J]. CoRR, 2019, abs/1910.06001.

[41] WEI K, LI J, MA C, et al. Vertical federated learning: Challenges, methodologies and experiments[A]. 2022.

[42] LUO X, WU Y, XIAO X, et al. Feature inference attack on model predictions in vertical federated learning[J]. International Conference on Data Engineering, 2021: 181-192.

[43] WU Y, CAI S, XIAO X, et al. Privacy preserving vertical federated learning for tree-based models[J]. Proceedings of the Very Large Data Bases Conference Endowment, 2020, 13: 2090 - 2103.

[44] ZHANG Q, GU B, DENG C, et al. Secure bilevel asynchronous vertical federated learning with backward updating[C]//AAAI Conference on Artificial Intelligence. 2021.

[45] YANG S, REN B, ZHOU X, et al. Parallel distributed logistic regression for vertical federated learning without third-party coordinator: abs/1911.09824[A]. 2019.

[46] LIU Y, KANG Y, XING C, et al. A secure federated transfer learning framework[J]. IEEE Intelligent Systems, 2020, 35: 70-82.

[47] GAO D, LIU Y, HUANG A, et al. Privacy-preserving heterogeneous federated transfer learning[J]. IEEE International Conference on Big Data, 2019: 2552-2559.

[48] CHEN Y, WANG J, YU C, et al. Fedhealth: A federated transfer learning framework for wearable healthcare[J]. IEEE Intelligent Systems, 2020, 35: 83-93.

[49] JU C, GAO D, MANE R, et al. Federated transfer learning for eeg signal classification[J]. International Conference of the IEEE Engineering in Medicine & Biology Society, 2020: 30403045.

[50] CHENG Y, LU J, NIYATO D T, et al. Federated transfer learning with client selection for intrusion detection in mobile edge computing[J]. IEEE Communications Letters, 2022, 26: 552-556.

[51] OTOUM Y, WAN Y, NAYAK A. Federated transfer learning-based IDS for the internet of medical things (iomt)[C]//IEEE Global Communications Conference Workshops. IEEE, 2021: 1-6.

[52] AHMED K M, IMTEAJ A, AMINI M H. Federated deep learning for heterogeneous edge computing[J]. IEEE International Conference on Machine Learning and Applications, 2021: 1146-1152.

[53] KOPPARAPU K, LIN E. Tinyfedtl: Federated transfer learning on tiny devices[J]. CoRR, 2021, abs/2110.01107.

[54] ZHU H, XU J, LIU S, et al. Federated learning on non-iid data: A survey[J]. Neurocomputing, 2021, 465: 371-390.

[55] YUROCHKIN M, AGARWAL M, GHOSH S, et al. Bayesian nonparametric federated learning of neural networks[C]//International Conference on Machine Learning: volume 97. PMLR, 2019: 7252-7261.

[56] DINH C T, TRAN N H, NGUYEN T D. Personalized federated learning with moreau envelopes[C]//Advances in Neural Information Processing Systems. 2020.

[57] HAN S, MAO H, DALLY W J. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding[C]//International Conference on Learning Representations. 2016.

[58] HINTON G E, VINYALS O, DEAN J. Distilling the knowledge in a neural network[J]. CoRR, 2015, abs/1503.02531.

[59] ZHANG Y, XIANG T, HOSPEDALES T M, et al. Deep mutual learning[J]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 4320-4328.

[60] REDDI S J, CHARLES Z, ZAHEER M, et al. Adaptive federated optimization[C]//International Conference on Learning Representations. 2021.

[61] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.

[62] KRIZHEVSKY A, HINTON G, et al. Learning multiple layers of features from tiny images[M]. Citeseer, 2009.

[63] LECUN Y, BOSER B E, DENKER J S, et al. Backpropagation applied to handwritten zip code recognition[J]. Neural computation, 1989, 1(4): 541-551.

[64] LI X, JIANG M, ZHANG X, et al. Fedbn: Federated learning on non-iid features via local batch normalization[C]//International Conference on Learning Representations. 2021.

[65] SUN B, HUO H, YANG Y, et al. Partialfed: Cross-domain personalized federated learning via partial initialization[J]. Advances in Neural Information Processing Systems, 2021, 34.

[66] BILEN H, VEDALDI A. Universal representations: The missing link between faces, text, planktons, and cat breeds[J]. CoRR, 2017, abs/1701.07275.

[67] LI Y, WANG N, SHI J, et al. Adaptive batch normalization for practical domain adaptation[J]. Pattern Recognition, 2018, 80: 109-117.

[68] WANG Y, ZHANG Z, HAO W, et al. Attention guided multiple source and target domain adaptation[J]. IEEE Transactions on Image Processing, 2021, 30: 892-906.

[69] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 7132-7141.

[70] WOO S, PARK J, LEE J, et al. CBAM: convolutional block attention module[C]//European Conference on Computer Vision: volume 11211. Springer, 2018: 3-19.

[71] WANG Q, WU B, ZHU P, et al. Eca-net: Efficient channel attention for deep convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition. 2020.

[72] HOU Q, ZHOU D, FENG J. Coordinate attention for efficient mobile network design[C]//IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2021: 13713-13722.

[73] NETZER Y, WANG T, COATES A, et al. Reading gigits in natural images with unsupervised feature learning[C]//Neural Information Processing Systems Workshop on Deep Learning and Unsupervised Feature Learning. 2011.

[74] HULL J J. A database for handwritten text recognition research[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1994.

[75] GANIN Y, LEMPITSKY V. Unsupervised domain adaptation by backpropagation[C]// International Conference on Machine Learning. 2015.

[76] ARBELAEZ P, MAIRE M, FOWLKES C C, et al. Contour detection and hierarchical image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33 (5): 898-916.

[77] SAENKO K, KULIS B, FRITZ M, et al. Adapting visual category models to new domains[C]// European Conference on Computer Vision. 2010.

[78] VENKATESWARA H, EUSEBIO J, CHAKRABORTY S, et al. Deep hashing network for unsupervised domain adaptation[C]//IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[79] PENG X, BAI Q, XIA X, et al. Moment matching for multi-source domain adaptation[C]//IEEE International Conference on Computer Vision. 2019.