Measuring citation faithfulness: notes towards a benchmark

This is a working note rather than a result. It sets out the design of grounding-bench, a benchmark we are building for citation faithfulness, and the reasoning behind some choices we expect to be argued with.

The gap

"Grounded" has quietly come to mean "displays citations". These are different properties, and the difference is measurable.

The reference point is Liu, Zhang and Liang (2023), who audited four production generative search engines and found that on average only 51.5% of generated sentences were fully supported by their citations, while only 74.5% of citations supported the sentence they were attached to. The paper also noted an uncomfortable inverse relationship: responses that human raters found more fluent and helpful tended to contain more unsupported content.

That last finding is the one that matters for enterprise deployment. It means perceived quality is not just an unreliable proxy for faithfulness, under some conditions it is an inverted one. Which in turn means faithfulness cannot be left to reviewer intuition. It has to be a number in a test suite.

Why existing metrics are not enough for our purposes

There are good tools in this space already. Frameworks such as RAGAS and TruLens provide faithfulness and groundedness scores, and they are a reasonable default for teams starting out. Our reservations are specific rather than dismissive:

Sentence-level granularity is too coarse. A single sentence in a professional document frequently carries several independent claims, an obligation, a condition, a deadline. A binary supported/unsupported judgement on the sentence loses which part failed, and that is the part a reviewer needs.

Generic entailment models mis-handle domain semantics. In a contract, "shall" and "may" are not paraphrases. In a clinical note, negation and uncertainty markers carry the entire meaning. General-purpose NLI models trained on newswire and web text are unreliable exactly where the stakes are highest.

Partial support has no representation. The common case in practice is not a fabrication. It is a claim that is mostly supported, with an added qualification, a rounded number, or a generalisation the source does not license. Collapsing that to a binary discards the most common real failure.

Almost nothing is measured on non-English or code-switched text. For our market that is not an edge case.

Design

The benchmark decomposes into three levels, which we think is the minimum that is diagnostically useful.

Level 1, claim decomposition. A generated response is split into atomic, independently verifiable claims. Each claim retains a pointer to the citation it was emitted with. Decomposition is itself a model-driven step, so it needs its own validation against human decomposition, we are treating that as part of the benchmark rather than an implementation detail.

Level 2, support classification. Each claim is classified against its cited span on a four-way scale rather than a binary:

Label	Meaning
supported	The span entails the claim.
partial	The span supports the claim's core but not a qualifier, quantity or scope.
unsupported	The span is relevant but does not entail the claim.
contradicted	The span asserts something incompatible with the claim.

The partial and contradicted distinction is the point of the exercise. Systems fail very differently in those two modes and a single score hides it.

Level 3, citation sufficiency. Given the retrieved set, could a correct citation have been produced? This separates a retrieval failure from an attribution failure, which are fixed by entirely different work. Without this split, a faithfulness number tells you a system is broken but not which team owns it.

Open questions we do not have answers to

Who arbitrates the gold labels? Domain experts disagree about entailment more than one would like, particularly on partial. We are planning to publish inter-annotator agreement per domain alongside the scores, and to treat low agreement as a finding about the domain rather than noise to be smoothed away.

How do we avoid grading with the system under test? If the judge and the generator share a family, self-preference bias is a live concern, the LLM-as-a-judge literature documents position, verbosity and self-enhancement biases clearly (Zheng et al., 2023). Our current plan is a validated ensemble of judges from different families plus a human-labelled calibration set that ships with the benchmark, so anyone can check the grader before trusting the grade.

How do we keep it from being gamed? Any published benchmark eventually leaks into training data. We intend to hold out a rotating private split and publish the delta between public and held-out performance, which at least makes contamination visible.

Is partial one category or three? We suspect it decomposes into scope errors, quantity errors and modality errors, and that these have different downstream costs. We do not yet have enough labelled data to say.

What we want

Hard cases, from real domains, in any language. Specifically: a source passage, a generated claim citing it, and a human judgement with a one-line reason. The cases we most want are the ones where two competent reviewers would disagree.

grounding-bench will be released under CC BY-SA 4.0 with the annotation protocol, the calibration set and the disagreement data included, not just the final labels. A benchmark that hides its uncertainty is not much better than the systems it is grading.

---

Contributions and objections both welcome: research@rudvanth.com.