ML Security PapersSECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations
Buyun Liang , Liangzu Peng , Jinqi Luo , Darshan Thaker , Kwan Ho Ryan Chan , René Vidal
Published on arXiv
2510.04398
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
SECA achieves higher hallucination elicitation success rates than prior methods while incurring almost no semantic equivalence or coherence violations on both open-source and gradient-inaccessible commercial LLMs.
SECA (Semantically Equivalent and Coherent Attacks)
Novel technique introduced
Large Language Models (LLMs) are increasingly deployed in high-risk domains. However, state-of-the-art LLMs often exhibit hallucinations, raising serious concerns about their reliability. Prior work has explored adversarial attacks to elicit hallucinations in LLMs, but these methods often rely on unrealistic prompts, either by inserting nonsensical tokens or by altering the original semantic intent. Consequently, such approaches provide limited insight into how hallucinations arise in real-world settings. In contrast, adversarial attacks in computer vision typically involve realistic modifications to input images. However, the problem of identifying realistic adversarial prompts for eliciting LLM hallucinations remains largely underexplored. To address this gap, we propose Semantically Equivalent and Coherent Attacks (SECA), which elicit hallucinations via realistic modifications to the prompt that preserve its meaning while maintaining semantic coherence. Our contributions are threefold: (i) we formulate finding realistic attacks for hallucination elicitation as a constrained optimization problem over the input prompt space under semantic equivalence and coherence constraints; (ii) we introduce a constraint-preserving zeroth-order method to effectively search for adversarial yet feasible prompts; and (iii) we demonstrate through experiments on open-ended multiple-choice question answering tasks that SECA achieves higher attack success rates while incurring almost no semantic equivalence or semantic coherence errors compared to existing methods. SECA highlights the sensitivity of both open-source and commercial gradient-inaccessible LLMs to realistic and plausible prompt variations. Code is available at https://github.com/Buyun-Liang/SECA.
Key Contributions
- Formulates hallucination elicitation as a constrained optimization problem over prompt space under semantic equivalence and coherence constraints
- Introduces a constraint-preserving zeroth-order search method for finding adversarial yet natural-sounding prompts without gradient access
- Demonstrates higher attack success rates on open-source and commercial LLMs with near-zero semantic errors compared to existing hallucination-eliciting methods