An Investigation on Group Query Hallucination Attacks

With the widespread use of large language models (LLMs), understanding their potential failure modes during user interactions is essential. In practice, users often pose multiple questions in a single conversation with LLMs. Therefore, in this study, we propose Group Query Attack, a technique that simulates this scenario by presenting groups of queries to LLMs simultaneously. We investigate how the accumulated context from consecutive prompts influences the outputs of LLMs. Specifically, we observe that Group Query Attack significantly degrades the performance of models fine-tuned on specific tasks. Moreover, we demonstrate that Group Query Attack induces a risk of triggering potential backdoors of LLMs. Besides, Group Query Attack is also effective in tasks involving reasoning, such as mathematical reasoning and code generation for pre-trained and aligned models.

Key Contributions

Proposes Group Query Attack (GQA), a black-box prompt-level attack that presents multiple queries simultaneously to exploit accumulated context and significantly degrade fine-tuned LLM performance
Demonstrates that GQA can trigger potential pre-implanted backdoors in LLMs, revealing a new activation vector for backdoor exploitation
Shows GQA is effective on reasoning tasks (mathematical reasoning, code generation) for pre-trained and aligned models, though less so on multiple-choice and translation tasks

🛡️ Threat Analysis

Model Poisoning

Paper explicitly investigates whether Group Query Attack can trigger potential backdoors embedded in fine-tuned LLMs — models are fine-tuned with backdoors injected, then tested to see if multi-query context activates those triggers, revealing a new backdoor activation vector.