FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy

Zuhao Yang1; Yingfang Yuan2; Yang Xu3; Shuo Zhan1; Huajun Bai4; Kefan Chen2

2023

37th Conference on Neural Information Processing Systems (NeurIPS)

1049-5258

ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 36 (NEURIPS 2023)

DEC 10-16, 2023

null,New Orleans,LA

10010 NORTH TORREY PINES RD, LA JOLLA, CALIFORNIA 92037 USA

NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)

Measuring the distance between machine-produced and human language is a critical open problem. Inspired by empirical findings from psycholinguistics on the periodicity of entropy in language, we propose FACE, a set of metrics based on Fourier Analysis of the estimated Cross-Entropy of language, for measuring the similarity between model-generated and human-written languages. Based on an open-ended generation task and the experimental data from previous studies, we find that FACE can effectively identify the human-model gap, scales with model size, reflects the outcomes of different sampling methods for decoding, correlates well with other evaluation metrics and with human judgment scores.

第一

英语

CPCI-S

National Science Foundation[2105192]

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Information Systems

WOS:001220818807016

Web of Science

1.Department of Computer Science, Southern University of Science and Technology
2.School of Computer Science and Engineering, Nanyang Technological University
3.School of Mathematical and Computer Sciences, Heriot-Watt University
4.Genify, Inc.

Zuhao Yang,Yingfang Yuan,Yang Xu,et al. FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy[C]. 10010 NORTH TORREY PINES RD, LA JOLLA, CALIFORNIA 92037 USA:NEURAL INFORMATION PROCESSING SYSTEMS (NIPS),2023.