Quantifying Document Impact in RAG-LLMs

Retrieval Augmented Generation (RAG) enhances Large Language Models (LLMs) by connecting them to external knowledge, improving accuracy and reducing outdated information. However, this introduces challenges such as factual inconsistencies, source conflicts, bias propagation, and security vulnerabilities, which undermine the trustworthiness of RAG systems. A key gap in current RAG evaluation is the lack of a metric to quantify the contribution of individual retrieved documents to the final output. To address this, we introduce the Influence Score (IS), a novel metric based on Partial Information Decomposition that measures the impact of each retrieved document on the generated response. We validate IS through two experiments. First, a poison attack simulation across three datasets demonstrates that IS correctly identifies the malicious document as the most influential in $86\%$ of cases. Second, an ablation study shows that a response generated using only the top-ranked documents by IS is consistently judged more similar to the original response than one generated from the remaining documents. These results confirm the efficacy of IS in isolating and quantifying document influence, offering a valuable tool for improving the transparency and reliability of RAG systems.

Key Contributions

Introduces the Influence Score (IS), a novel metric grounded in Partial Information Decomposition that quantifies each retrieved document's individual contribution to a RAG-generated response.
Validates IS through a poison attack simulation showing it correctly identifies the malicious document as most influential in 86% of cases across three datasets.
Demonstrates via ablation that IS-selected top documents reproduce the original RAG response more faithfully than the remaining documents, confirming its utility for both security and transparency.