benchmark 2025

Prompt-Induced Over-Generation as Denial-of-Service: A Black-Box Attack-Side Benchmark

Manu , Yi Guo , Jo Plested ¹, Tim Lynar , Kanchana Thilakarathna ², Nirhoshan Sivaroopan ², Jack Yang ³, Wangli Yang ³

¹ The University of Sydney

² University of New South Wales

³ University of Wollongong

1 citations · 1 influential · 24 references · arXiv

Published on arXiv

2512.23779

Model Denial of Service

OWASP LLM Top 10 — LLM04

Key Finding

RL-GOAL achieves OGF = 2.70 ± 1.43 with 64.3% success rate and drives budget-hit non-termination in 46% of trials on Phi-3, nearly doubling severity over the evolutionary baseline

EOGen / RL-GOAL

Novel technique introduced

Large Language Models (LLMs) can be driven into over-generation, emitting thousands of tokens before producing an end-of-sequence (EOS) token. This degrades answer quality, inflates latency and cost, and can be weaponized as a denial-of-service (DoS) attack. Recent work has begun to study DoS-style prompt attacks, but typically focuses on a single attack algorithm or assumes white-box access, without an attack-side benchmark that compares prompt-based attackers in a black-box, query-only regime with a known tokenizer. We introduce such a benchmark and study two prompt-only attackers. The first is an Evolutionary Over-Generation Prompt Search (EOGen) that searches the token space for prefixes that suppress EOS and induce long continuations. The second is a goal-conditioned reinforcement learning attacker (RL-GOAL) that trains a network to generate prefixes conditioned on a target length. To characterize behavior, we introduce Over-Generation Factor (OGF): the ratio of produced tokens to a model's context window, along with stall and latency summaries. EOGen discovers short-prefix attacks that raise Phi-3 to OGF = 1.39 +/- 1.14 (Success@>=2: 25.2%); RL-GOAL nearly doubles severity to OGF = 2.70 +/- 1.43 (Success@>=2: 64.3%) and drives budget-hit non-termination in 46% of trials.

Key Contributions

Black-box, query-only benchmark for comparing prompt-induced over-generation (DoS) attackers under a consistent interface and decoding regime
Over-Generation Factor (OGF) metric — ratio of produced tokens to context window — plus stall and latency summaries for model-agnostic DoS characterization
Two novel attackers: EOGen (gradient-free evolutionary token-space search) and RL-GOAL (goal-conditioned RL prefix generator), with RL-GOAL achieving OGF = 2.70 and 64.3% success rate on Phi-3

🛡️ Threat Analysis

Details

Domains

nlp

Model Types

llmrl

Threat Tags

black_boxinference_time

Datasets

Phi-3

Applications

llm inference apisllm serving infrastructure

Read PDF arXiv DOI

Prompt-Induced Over-Generation as Denial-of-Service: A Black-Box Attack-Side Benchmark

Key Contributions

🛡️ Threat Analysis

Details

Similar Papers

Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model

SHIELD: An Auto-Healing Agentic Defense Framework for LLM Resource Exhaustion Attacks

POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization

ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking

Sponge Tool Attack: Stealthy Denial-of-Efficiency against Tool-Augmented Agentic Reasoning

RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stress

LoopLLM: Transferable Energy-Latency Attacks in LLMs via Repetitive Generation

ReasoningBomb: A Stealthy Denial-of-Service Attack by Inducing Pathologically Long Reasoning in Large Reasoning Models