Ercong Nie [ɚ'tsʰʊŋ, niɛ]

Selected Publications

1. ECML-PKDD 2024

   A Unified Data Augmentation Framework for Low-Resource Multi-Domain Dialogue Generation

   Yongkang Liu*, Ercong Nie*, Shi Feng, Zheng Hua, Zifeng Ding, and 3 more authors

   In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases Sep 2024

   Abs URL

   Current state-of-the-art dialogue systems heavily rely on extensive training datasets. However, challenges arise in domains where domain-specific training datasets are insufficient or entirely absent. To tackle this challenge, we propose a novel data Augmentation framework for Multi-Domain Dialogue Generation, referred to as AMD2G. The AMD2G framework consists of a data augmentation process and a two-stage training approach: domain-agnostic training and domain adaptation training. We posit that domain corpora are a blend of domain-agnostic and domain-specific features, with certain representation patterns shared among diverse domains. Domain-agnostic training aims to enable models to learn these common expressive patterns. To construct domain-agnostic dialogue corpora, we employ a de-domaining data processing technique used to remove domain-specific features. By mitigating the impact of domain-specific features, the model trained on the de-domained corpora can effectively learn shared expressive patterns across various domains. Subsequently, we adapt the learned domain-agnostic features to the target domain through domain adaptation training. We conduct experiments on Chinese dialogue datasets from five different domains, demonstrating that AMD2G achieves superior performance compared to both direct training on the target domain corpus and training on all five domain corpora collectively. Our work underscores AMD2G as a viable alternative solution for low-resource multi-domain dialogue generation.

2. ACL 2024

   GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network

   Shuzhou Yuan, Ercong Nie, Michael Färber, Helmut Schmid, and Hinrich Schütze

   In Findings of the Association for Computational Linguistics: ACL 2024 Aug 2024

   Abs URL

   Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are applied to them. However, fine-tuning still remains crucial to further enhance their adaptability. Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality. We address this issue by introducing a prompt-based parameter-efficient fine-tuning (PEFT) approach. GNNavi leverages insights into ICL’s information flow dynamics, which indicates that label words act in prompts as anchors for information propagation. GNNavi employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN. Our experiments on text classification tasks with GPT-2 and Llama2 shows GNNavi surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. We compare GNNavi with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency. Our analysis reveals that GNNavi enhances information flow and ensures a clear aggregation process.

3. LREC-COLING 2024

   Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairs

   Linyang He, Peilin Chen, Ercong Nie, Yuanning Li, and Jonathan R. Brennan

   In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation May 2024

   Abs URL

   Inspired by cognitive neuroscience studies, we introduce a novel ‘decoding probing’ method that uses minimal pairs benchmark (BLiMP) to probe internal linguistic characteristics in neural language models layer by layer. By treating the language model as the brain and its representations as ’neural activations’, we decode grammaticality labels of minimal pairs from the intermediate layers’ representations. This approach reveals: 1) Self-supervised language models capture abstract linguistic structures in intermediate layers that GloVe and RNN language models cannot learn. 2) Information about syntactic grammaticality is robustly captured through the first third layers of GPT-2 and also distributed in later layers. As sentence complexity increases, more layers are required for learning grammatical capabilities. 3) Morphological and semantics/syntax interface-related features are harder to capture than syntax, which also aligns with cognitive neuroscience studies. 4) For Transformer-based models, both embeddings and attentions capture grammatical features but show distinct patterns. Different attention heads exhibit similar tendencies toward various linguistic phenomena, but with varied contributions. Notably, specific heads consistently offer the most contributions.

4. EACL 2024

   ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks

   Bolei Ma*, Ercong Nie*, Shuzhou Yuan, Helmut Schmid, Färber Michael, and 2 more authors

   In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics Mar 2024

   Abs URL

   Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based method for token-level sequence labeling tasks. The ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer, especially for languages that are typologically different from the source language English. Our method also attains state-of-the-art performance when employed with the mT5 model. Besides, our exploratory study in multilingual large language models shows that ToPro performs much better than the current in-context learning method. Overall, the performance improvements show that ToPro could potentially serve as a novel and simple benchmarking method for sequence labeling tasks.

5. EMNLP 2023

   Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration

   Ercong Nie, Helmut Schmid, and Hinrich Schütze

   In Findings of the Association for Computational Linguistics: EMNLP 2023 Dec 2023

   Abs URL

   Pretraiend multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is accomplished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model parameters. However, the performance of this pattern is limited by the model’s bias toward predicting label words which frequently occurred during the pretraining. These words typically receive high probabilities. To address this issue, we combine the models with various calibration techniques which modify the probabilities of label words predicted by the models. We evaluate the effectiveness of these calibration methods on monolingual encoders as well as multilingual encoders. Across a diverse range of tasks, we achieve substantial performance gains through calibration. Furthermore, with only very few training samples, the trained calibration parameters are able to yield additional enhancements.

6. ACL 2023

   Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages

   Ercong Nie*, Sheng Liang*, Helmut Schmid, and Hinrich Schütze

   In Findings of the Association for Computational Linguistics: ACL 2023 Jul 2023

   Abs URL

   Multilingual Pretrained Language Models (MPLMs) perform strongly in cross-lingual transfer. We propose Prompts Augmented by Retrieval Crosslingually (PARC) to improve zero-shot performance on low-resource languages (LRLs) by augmenting the context with prompts consisting of semantically similar sentences retrieved from a high-resource language (HRL). PARC improves zero-shot performance on three downstream tasks (sentiment classification, topic categorization, natural language inference) with multilingual parallel test sets across 10 LRLs covering 6 language families in unlabeled (+5.1%) and labeled settings (+16.3%). PARC also outperforms finetuning by 3.7%. We find a significant positive correlation between cross-lingual transfer performance on one side, and the similarity between high- and low-resource languages as well as the amount of low-resource pretraining data on the other side. A robustness analysis suggests that PARC has the potential to achieve even stronger performance with more powerful MPLMs.