CMN: a co-designed neural architecture search for efficient computing-in-memory-based mixture-of-experts

Han, Shihao1; Liu, Sishuo1; Du, Shucheng1,2; Li, Mingzi1,2; Ye, Zijian1; Xu, Xiaoxin3,4,5; Li, Yi1,2,3,4; Wang, Zhongrui1,2,6; Shang, Dashan3,4,5

2024-10-01

10.1007/s11432-024-4144-y

SCIENCE CHINA-INFORMATION SCIENCES   影响因子和分区

1674-733X

1869-1919

Artificial intelligence (AI) has experienced substantial advancements recently, notably with the advent of large-scale language models (LLMs) employing mixture-of-experts (MoE) techniques, exhibiting human-like cognitive skills. As a promising hardware solution for edge MoE implementations, the computing-in-memory (CIM) architecture collocates memory and computing within a single device, significantly reducing the data movement and the associated energy consumption. However, due to diverse edge application scenarios and constraints, determining the optimal network structures for MoE, such as the expert's location, quantity, and dimension on CIM systems remains elusive. To this end, we introduce a software-hardware co-designed neural architecture search (NAS) framework, CIM-based MoE NAS (CMN), focusing on identifying a high-performing MoE structure under specific hardware constraints. The results of the NYUD-v2 dataset segmentation on the RRAM (SRAM) CIM system reveal that CMN can discover optimized MoE configurations under energy, latency, and performance constraints, achieving 29.67x (43.10x) energy savings, 175.44x(109.89x) speedup, and 12.24x smaller model size compared to the baseline MoE-enabled Visual Transformer, respectively. This co-design opens up an avenue toward high-performance MoE deployments in edge CIM systems.

mixture-of-experts computing-in-memory neural architecture search resistive random-access memory static random-access memory

SCI

英语

通讯

National Key R&D Program of China[2020AAA0109005] ; Beijing Natural Science Foundation[Z210006] ; National Naturel Science Foundation of China["62122004","62374181"] ; Strategic Priority Research Program of the Chinese Academy of Sciences[XDB44000000] ; Hong Kong Research Grant Council["27206321","17205922","17212923"] ; Hong Kong Innovation and Technology Fund[ITP/048/22AP]

Computer Science ; Engineering

Computer Science, Information Systems ; Engineering, Electrical & Electronic

WOS:001321914100001

SCIENCE PRESS

Web of Science

1.Univ Hong Kong, Dept Elect & Elect Engn, Hong Kong 999077, Peoples R China
2.ACCESS AI Chip Ctr Emerging Smart Syst, InnoHK Ctr, Hong Kong Sci Pk, Hong Kong 999077, Peoples R China
3.Chinese Acad Sci, Key Lab Fabricat Technol Integrated Circuits, Beijing 100049, Peoples R China
4.Chinese Acad Sci, Lab Microelect Devices & Integrated Technol, Inst Microelect, Beijing 100029, Peoples R China
5.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
6.Southern Univ Sci & Technol, Sch Microelect, Shenzhen 518055, Peoples R China

Han, Shihao,Liu, Sishuo,Du, Shucheng,et al. CMN: a co-designed neural architecture search for efficient computing-in-memory-based mixture-of-experts[J]. SCIENCE CHINA-INFORMATION SCIENCES,2024,67(10).

Han, Shihao.,Liu, Sishuo.,Du, Shucheng.,Li, Mingzi.,Ye, Zijian.,...&Shang, Dashan.(2024).CMN: a co-designed neural architecture search for efficient computing-in-memory-based mixture-of-experts.SCIENCE CHINA-INFORMATION SCIENCES,67(10).

Han, Shihao,et al."CMN: a co-designed neural architecture search for efficient computing-in-memory-based mixture-of-experts".SCIENCE CHINA-INFORMATION SCIENCES 67.10(2024).