Gaming the Answer Matcher: Examining the Impact of Text Manipulation on Automated Judgment

Automated answer matching, which leverages LLMs to evaluate free-text responses by comparing them to a reference answer, shows substantial promise as a scalable and aligned alternative to human evaluation. However, its reliability requires robustness against strategic attacks such as guesswork or verbosity that may artificially inflate scores without improving actual correctness. In this work, we systematically investigate whether such tactics deceive answer matching models by prompting examinee models to: (1) generate verbose responses, (2) provide multiple answers when unconfident, and (3) embed conflicting answers with the correct answer near the start of their response. Our results show that these manipulations do not increase scores and often reduce them. Additionally, binary scoring (which requires a matcher to answer with a definitive "correct" or "incorrect") is more robust to attacks than continuous scoring (which requires a matcher to determine partial correctness). These findings show that answer matching is generally robust to inexpensive text manipulation and is a viable alternative to traditional LLM-as-a-judge or human evaluation when reference answers are available.

Key Contributions

• Systematic evaluation of three text manipulation attacks (verbosity inflation, forward conflicting answers, strategic hedging) against LLM-based answer matchers
• Finding that none of these manipulations increase scores and often reduce them, demonstrating answer matcher robustness
• Comparison showing binary scoring is more robust to gaming than continuous/partial-credit scoring

🛡️ Threat Analysis

Details

Similar Papers