SSDC: A Scalable Sparse Differential Checkpoint for Large-scale Deep Recommendation Models

Lingrui Xiang1,2; Xiaofen Lu1; Rui Zhang2; Zheng Hu2

10.1109/ISCAS58744.2024.10557880

2024-05-22

0271-4302

979-8-3503-3100-4

2024 IEEE International Symposium on Circuits and Systems (ISCAS)

19-22 May 2024

Singapore, Singapore

Today deep recommendation models have become increasingly large, with parameter sizes reaching hundreds of GB or even TB scale. As a result, it requires large-scale cluster computing resources to train such models. However, large-scale computing clusters tend to experience frequent failures during runtime, so fault-tolerance mechanisms such as checkpointing and restart are widely used in model training. Traditional checkpointing techniques periodically save all parameters of model, resulting in significant overhead. To address this issue, we propose an improved partial checkpointing mechanism for recommendation models named SSDC. SSDC uses an adaptive threshold strategy to reduce expensive operations when saving checkpoints, thereby having good scalability. Furthermore, SSDC saves the differential value of the model parameters, making it feasible to sparsify the otherwise dense embedding tables, thus reducing the bandwidth and time overhead to reconstruct checkpoints. Our evaluations show that compared to state-of-the-art methods, SSDC greatly reduces the time overhead of saving and reconstructing checkpoints, while achieving comparable training accuracy.

第一

EI

1.Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
2.RAMS Lab, Huawei, Shenzhen, China

Lingrui Xiang,Xiaofen Lu,Rui Zhang,et al. SSDC: A Scalable Sparse Differential Checkpoint for Large-scale Deep Recommendation Models[C],2024.