ML Security PapersThe Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search
Rongzhe Wei 1, Peizhi Niu 2, Xinjie Shen 1, Tony Tu 1, Yifan Li 3, Ruihan Wu 4, Eli Chien 5, Pin-Yu Chen 6, Olgica Milenkovic 2, Pan Li 1
1 Georgia Institute of Technology
2 University of Illinois Urbana-Champaign
Published on arXiv
2512.01353
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
CKA-Agent achieves over 95% jailbreak success rate against Gemini2.5-Flash/Pro, GPT-oss-120B, and Claude-Haiku-4.5 by decomposing harmful objectives into individually benign sub-queries.
CKA-Agent (Correlated Knowledge Attack Agent)
Novel technique introduced
Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs. Existing approaches overwhelmingly operate within the prompt-optimization paradigm: whether through traditional algorithmic search or recent agent-based workflows, the resulting prompts typically retain malicious semantic signals that modern guardrails are primed to detect. In contrast, we identify a deeper, largely overlooked vulnerability stemming from the highly interconnected nature of an LLM's internal knowledge. This structure allows harmful objectives to be realized by weaving together sequences of benign sub-queries, each of which individually evades detection. To exploit this loophole, we introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base. The CKA-Agent issues locally innocuous queries, uses model responses to guide exploration across multiple paths, and ultimately assembles the aggregated information to achieve the original harmful objective. Evaluated across state-of-the-art commercial LLMs (Gemini2.5-Flash/Pro, GPT-oss-120B, Claude-Haiku-4.5), CKA-Agent consistently achieves over 95% success rates even against strong guardrails, underscoring the severity of this vulnerability and the urgent need for defenses against such knowledge-decomposition attacks. Our codes are available at https://github.com/Graph-COM/CKA-Agent.
Key Contributions
- Identifies a structural vulnerability in LLMs: harmful knowledge can be assembled from sequences of individually innocuous sub-queries that each evade guardrails.
- Proposes CKA-Agent, an adaptive tree-structured exploration framework that dynamically decomposes harmful objectives into benign sub-queries and aggregates responses to achieve the original malicious goal.
- Demonstrates >95% attack success rate against state-of-the-art commercial LLMs (Gemini2.5-Flash/Pro, GPT-oss-120B, Claude-Haiku-4.5) with strong safety guardrails.