![]() To retrieve useful documents for unseen target LMs, we propose augmentation-adapted retriever (AAR), which learns LM’s preferences obtained from a known source LM. ![]() In this paper, we explore the scheme of generic retrieval plug-in: the retriever is to assist target LMs that may not be known beforehand or are unable to be fine-tuned together. Prior works on retrieval augmentation usually jointly fine-tune the retriever and the LM, making them closely coupled. Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Experimental results show that learnable methods significantly reduce PLMs’ confidence in wrong predictions.Īugmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In Besides unlearnable calibration methods (e.g., label smoothing), we adapt and extend two recently proposed learnable methods that directly collect data to train models to have reasonable confidence estimations. Next, we study the effectiveness of existing calibration methods in mitigating the overconfidence issue. ![]() We highlight that our finding somewhat contradicts two established conclusions: (a) Larger PLMs are more calibrated (b) Pretraining improves model calibration. ![]() We find that PLMs don’t learn to become calibrated in training, evidenced by the continual increase in confidence, no matter whether the predictions are correct or not. We observe a consistent change in calibration performance across six factors. We consider six factors as control variables, including dataset difficulty, available training samples, training steps, the number of tunable parameters, model scale, and pretraining. We take a close look into this problem, aiming to answer two questions: (1) Do PLMs learn to become calibrated in the training process? (2) How effective are existing calibration methods? For the first question, we conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) We release our code and models at to facilitate future work.Ī Close Look into the Calibration of Pre-trained Language Models At the same time, models trained with SubChar tokenizers perform competitively on downstream tasks. 2) Pronunciation-based SubChar tokenizers can encode Chinese homophones into the same transliteration sequences and produce the same tokenization output, hence being robust to homophone typos. Experimental results show that SubChar tokenizers have two main advantages over existing tokenizers: 1) They can tokenize inputs into much shorter sequences, thus improving the computational efficiency. Specifically, we first encode the input text by converting each Chinese character into a short sequence based on its glyph or pronunciation, and then construct the vocabulary based on the encoded text with sub-word segmentation. To utilize such information, we propose sub-character (SubChar for short) tokenization. However, they ignore the unique feature of the Chinese writing system where additional linguistic information exists below the character level, i.e., at the sub-character level. Existing tokenization methods for Chinese PLMs typically treat each character as an indivisible token. ![]() Tokenization is fundamental to pretrained language models (PLMs). ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |