Developing normalization schemes for data isolated distributed deep learning

Zhou，Yujue1,2; He，Ligang1; Yang，Shuang Hua2

2021

10.1049/cps2.12004

IET Cyber-Physical Systems: Theory and Applications   影响因子和分区

2398-3396

Distributed deep learning is an important and indispensable direction in the field of deep learning research. Earlier research has proposed many algorithms or techniques on accelerating distributed neural network training. This study discusses a new distributed training scenario, namely data isolated distributed deep learning. Specifically, each node has its own local data and cannot be shared for some reasons. However, in order to ensure the generalization of the model, the goal is to train a global model that required learning all the data, not just based on data from a local node. At this time, distributed training with data isolation is needed. An obvious challenge for distributed deep learning in this scenario is that the distribution of training data used by each node could be highly imbalanced because of data isolation. This brings difficulty to the normalization process in neural network training, because the traditional batch normalization (BN) method will fail under this kind of data imbalanced scenario. At this time, distributed training with data isolation is needed. Aiming at such data isolation scenarios, this study proposes a comprehensive data isolation deep learning scheme. Specifically, synchronous stochastic gradient descent algorithm is used for data exchange during training, and provides several normalization approaches to the problem of BN failure caused by data imbalance. Experimental results show the efficiency and accuracy of the proposed data isolated distributed deep learning scheme.

EI

英语

其他

20211410167577

Electronic data interchange ; Gradient methods ; Neural networks ; Stochastic systems

Data Processing and Image Processing:723.2 ; Numerical Methods:921.6 ; Systems Science:961

2-s2.0-85103405649

Scopus

1.Department of Computer Science,University of Warwick,Coventry,United Kingdom
2.Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,China

Zhou，Yujue,He，Ligang,Yang，Shuang Hua. Developing normalization schemes for data isolated distributed deep learning[J]. IET Cyber-Physical Systems: Theory and Applications,2021,6:105-115.

Zhou，Yujue,He，Ligang,&Yang，Shuang Hua.(2021).Developing normalization schemes for data isolated distributed deep learning.IET Cyber-Physical Systems: Theory and Applications,6,105-115.

Zhou，Yujue,et al."Developing normalization schemes for data isolated distributed deep learning".IET Cyber-Physical Systems: Theory and Applications 6(2021):105-115.