Revisiting Self-Training for Few-Shot Learning of Language Model

Chen，Yiming1,2; Zhang，Yan1; Zhang，Chen1; Lee，Grandee1; Cheng，Ran2; Li，Haizhou1,3,4

2021

Conference on Empirical Methods in Natural Language Processing (EMNLP)

EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings

9125-9135

NOV 07-11, 2021

null,Punta Cana,DOMINICAN REP

209 N EIGHTH STREET, STROUDSBURG, PA 18360 USA

ASSOC COMPUTATIONAL LINGUISTICS-ACL

As unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. The question is how to effectively make use of such data. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM. Given two views of a text sample via weak and strong augmentation techniques, SFLM generates a pseudo label on the weakly augmented version. Then, the model predicts the same pseudo label when fine-tuned with the strongly augmented version. This simple approach is shown to outperform other state-of-the-art supervised and semi-supervised counterparts on six sentence classification and six sentence-pair classification benchmarking tasks. In addition, SFLM only relies on a few in-domain unlabeled data. We conduct a comprehensive analysis to demonstrate the robustness of our proposed approach under various settings, including augmentation techniques, model scale, and few-shot knowledge transfer across tasks.

通讯

英语

EI ; CPCI-S ; CPCI-SH

Norsk Revmatikerforbund[A18A2b0046];

Computer Science ; Linguistics

Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Linguistics

WOS:000860727003020

20221411909963

Computational linguistics ; Learning systems

Computer Theory, Includes Formal Logic, Automata Theory, Switching Theory, Programming Theory:721.1 ; Computer Applications:723.5 ; Information Retrieval and Use:903.3

2-s2.0-85127402421

Scopus

1.National University of Singapore,Singapore
2.Southern University of Science and Technology,China
3.The Chinese University of Hong Kong,Shenzhen,Hong Kong
4.Kriston AI Lab,China

Chen，Yiming,Zhang，Yan,Zhang，Chen,et al. Revisiting Self-Training for Few-Shot Learning of Language Model[C]. 209 N EIGHTH STREET, STROUDSBURG, PA 18360 USA:ASSOC COMPUTATIONAL LINGUISTICS-ACL,2021:9125-9135.