Ercong Nie [ɚ'tsʰʊŋ, niɛ]

Selected Publications

1. EACL 2024

   ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks

   Bolei Ma*, Ercong Nie*, Shuzhou Yuan, Helmut Schmid, Färber Michael, and 2 more authors

   In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics Mar 2024

   Abs URL

   Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based method for token-level sequence labeling tasks. The ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer, especially for languages that are typologically different from the source language English. Our method also attains state-of-the-art performance when employed with the mT5 model. Besides, our exploratory study in multilingual large language models shows that ToPro performs much better than the current in-context learning method. Overall, the performance improvements show that ToPro could potentially serve as a novel and simple benchmarking method for sequence labeling tasks.

2. EMNLP 2023

   Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration

   Ercong Nie, Helmut Schmid, and Hinrich Schütze

   In Findings of the Association for Computational Linguistics: EMNLP 2023 Dec 2023

   Abs URL

   Pretraiend multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is accomplished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model parameters. However, the performance of this pattern is limited by the model’s bias toward predicting label words which frequently occurred during the pretraining. These words typically receive high probabilities. To address this issue, we combine the models with various calibration techniques which modify the probabilities of label words predicted by the models. We evaluate the effectiveness of these calibration methods on monolingual encoders as well as multilingual encoders. Across a diverse range of tasks, we achieve substantial performance gains through calibration. Furthermore, with only very few training samples, the trained calibration parameters are able to yield additional enhancements.

3. ACL 2023

   Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages

   Ercong Nie*, Sheng Liang*, Helmut Schmid, and Hinrich Schütze

   In Findings of the Association for Computational Linguistics: ACL 2023 Jul 2023

   Abs URL

   Multilingual Pretrained Language Models (MPLMs) perform strongly in cross-lingual transfer. We propose Prompts Augmented by Retrieval Crosslingually (PARC) to improve zero-shot performance on low-resource languages (LRLs) by augmenting the context with prompts consisting of semantically similar sentences retrieved from a high-resource language (HRL). PARC improves zero-shot performance on three downstream tasks (sentiment classification, topic categorization, natural language inference) with multilingual parallel test sets across 10 LRLs covering 6 language families in unlabeled (+5.1%) and labeled settings (+16.3%). PARC also outperforms finetuning by 3.7%. We find a significant positive correlation between cross-lingual transfer performance on one side, and the similarity between high- and low-resource languages as well as the amount of low-resource pretraining data on the other side. A robustness analysis suggests that PARC has the potential to achieve even stronger performance with more powerful MPLMs.