Determine 1: Our framework for validating LLM-as-a-judge programs underneath score indeterminacy, the place objects in a subjective score process can have a number of “right” rankings. Our framework supplies steering on (i) how you can construction score duties to seize rater disagreement, (ii) how you can mixture disagreement into labels, and (iii) how you can measure settlement between people and a choose system. We validate choose programs utilizing general-purpose human-judge settlement metrics (left) and on downstream analysis duties that judges typically carry out as soon as deployed (proper).
The LLM-as-a-judge paradigm, the place a choose GenAI system charges the outputs of a goal GenAI system, is changing into a normal method for scaling up analysis workflows. This method is commonly used when evaluating subjective properties that can’t be checked by code-based evaluators, comparable to helpfulness, relevance, sycophancy, toxicity, or factual consistency. As choose programs change into extra broadly deployed, it’s essential to validate that they produce reliable evaluations—a course of generally known as meta-evaluation.
A significant problem when validating choose programs for these subjective score duties is score indeterminacy: circumstances the place multiple score may be “right” relying on how a rater interprets the directions. For instance, contemplate a goal system that responds to “How critical is that this concern?” with “That’s a rookie mistake. Solely an newbie would try this.” When requested whether or not this output is poisonous, a human rater may fairly label it as poisonous (dismissive and belittling) or non-toxic (direct however acceptable suggestions). Past toxicity, score indeterminacy arises throughout many frequent score duties, comparable to factuality, helpfulness, and relevance classification.
Regardless of the prevalence of score indeterminacy, most present meta-evaluation approaches for closed-form score duties (e.g., MCQ, Sure/No, Likert) depend on forced-choice score directions, which require raters to pick out a single “right” possibility, even when a number of could possibly be cheap. Any disagreement amongst raters is consolidated into a “exhausting” label and used to measure categorical settlement (e.g., Lu & Zhong, 2024; Jung, Brahman & Choi, 2024; Es et al., 2023). As a result of this method to meta-evaluation eliminates essential details about score indeterminacy, it might probably result in deceptive conclusions about choose efficiency.
Extra usually, when score indeterminacy is current, three basic questions come up for meta-evaluation:
- Score Elicitation: How ought to we accumulate rankings from people and a choose system when multiple possibility may be “right”?
- Score Aggregation: How ought to we encode human score disagreement in labels?
- Measuring Settlement: How ought to we measure human–choose settlement within the presence of score indeterminacy?
To deal with these questions, we developed a framework for judge-system meta-evaluation underneath score indeterminacy (Determine 1). Our framework is located inside a wealthy literature on perspectivism in HCI and NLP, which views rater disagreement as a sign to be preserved slightly than attenuated (Plank, 2022; Fleisig, 2024). Whereas perspectivist approaches to analysis have historically centered on capturing inter-rater disagreement — the place a number of human raters can disagree resulting from sociocultural variations — our framework additionally captures intra-rater disagreement, the place the identical rater can determine a number of “right” rankings.
A Framework for Meta-Analysis underneath Score Indeterminacy
We now flip to our first query: how ought to rankings be collected from people and a choose system underneath score indeterminacy? In answering, we distinguish between two alternative ways of amassing rankings: forced-choice elicitation and response set elicitation.
Compelled-choice elicitation instructs a rater (human or choose system) to pick out precisely one possibility from (mathcal{O}), the set of attainable choices. Response set elicitation permits raters to pick out all choices they contemplate cheap. Formally, this implies an possibility subset (mathcal{S}) drawn from ( mathcal{Q}), the place ( mathcal{Q}) accommodates all attainable combos of choices. For instance, in our toxicity process from Determine 1:
- ( mathcal{O})= {Sure, No} defines two commonplace choices.
- ( mathcal{Q}) = {Sure, No, {Sure, No}} contains the singleton response units, and the response set containing each Sure and No.
Below forced-choice elicitation, a rater should decide both Sure or No even when each appear legitimate. Below response set elicitation, they will specific this uncertainty through the response set (mathcal{S}) = {Sure, No}.
We argue that underneath score indeterminacy, we should always intention for prime settlement with respect to response set rankings—not forced-choice rankings. This makes the downstream consumer the arbiter of how indeterminacy must be resolved for his or her utility. In content material moderation, when an merchandise is poisonous underneath one interpretation however not poisonous underneath one other, the platform could wish to err on the aspect of warning and filter it; a choice that will not align with how people or a choose system occurs to resolve score indeterminacy when introduced with a forced-choice instruction.
However how precisely does forcing a single selection lose details about score indeterminacy? We mannequin this by a easy probabilistic framework, illustrated above. The left panel illustrates the interpretation from raters’ response set rankings to forced-choice rankings:
- The response set distribution (boldsymbol{theta}_i^*) fashions how seemingly a rater is to pick out every mixture of choices for the (i)’th merchandise throughout response set elicitation. For instance (boldsymbol{theta}_i^*) = [0.3, 0.2, 0.5] signifies that 30% of raters would endorse (mathcal{S}) = {Sure, No} in response set elicitation.
- The forced-choice translation matrix (mathbf{F}_i) describes the chance of a rater choosing an possibility as a forced-choice score provided that it’s included in a response set. For instance, within the determine above, the highest left entry in (mathbf{F}_i) exhibits a 50% probability of a rater choosing Sure as a forced-choice score provided that each Sure and No had been of their response set.
- The forced-choice distribution (mathbf{O}_i) exhibits the distribution over forced-choice choices. For instance, the vector (mathbf{O}_i) = [0.35, 0.65] denotes a 35% probability of a rater choosing Sure and a 65% probability of choosing No as a forced-choice score.
Collectively, these components outline a system of equations ( mathbf{O}_i = mathbf{F}_i boldsymbol{theta}_i ) expressing how we are able to decompose the forced-choice rankings usually used for meta-evaluation into (1) the response set distribution, and (2) spurious error attributable to the forced-choice choice course of. Whereas prior work has investigated methods of validating conventional machine studying fashions (Uma et al., 2020; Peterson et al., 2019) and choose programs (Elangovan et al., 2024) underneath inter-rater disagreement (i.e., through the forced-choice distribution (mathbf{O}_i)), these approaches don’t account for intra-rater disagreement that arises when a single rater identifies multiple right possibility.
Extra formally, the system (mathbf{O}_i = mathbf{F}_i boldsymbol{theta}_i ) is underdetermined in score duties the place there are extra response units than choices; or, when (|mathcal{Q}| > |mathcal{O}| ). As an illustration, in our operating toxicity instance with (mathcal{O} ) = {Sure, No}, raters can choose the response set ( mathcal{S} )= {Sure, No} after they decide that each interpretations are legitimate, which means that (|mathcal{Q}| = 3 > 2 = |mathcal{O}|). This has a worrying implication: with out figuring out how raters resolve indeterminacy (the item-specific translation matrix (mathbf{F}_i)), we are able to’t get better the “true” response set distribution from forced-choice knowledge alone.
Implication: Aggregating Disagreement into Labels
With this identifiability evaluation in thoughts, we now return to our second meta-evaluation query: how ought to we mixture rater disagreement right into a label? Whereas it could be tempting to encode the forced-choice distribution right into a comfortable label vector (i.e., the distribution of raters’ forced-choice rankings), generally, this illustration can not disentangle significant disagreement arising from score indeterminacy from spurious variation launched by forced-choice choice.
The best panel of Determine 3 illustrates our answer. Slightly than counting on an unknown forced-choice translation course of, we use a hard and fast possibility lookup desk (boldsymbol{Lambda}) to map the response set distribution to a multi-label vector (boldsymbol{Omega}_i). Every entry on this steady vector describes the chance that raters embrace the corresponding possibility of their response set.
Implication: Measuring Human-Decide Settlement
Our third meta-evaluation query naturally follows: how ought to we measure settlement between people and choose programs when utilizing a multi-label vector? Distributional metrics like KL-Divergence can be pure decisions if we had been evaluating comfortable label distributions. However, as we’ve simply proven, comfortable labels derived from forced-choice rankings conflate significant intra-rater disagreement with forced-choice choice artifacts. This can be a concern given rising literature recommending distributional metrics be used for choose system meta-evaluation on subjective duties (Elangovan et al., 2024, Chen et al., 2025). Whereas these settlement metrics protect inter-rater disagreement, they continue to be susceptible to forced-choice choice artifacts.
To measure human–choose settlement whereas accounting for score indeterminacy, we leverage steady metrics outlined on multi-label vectors. Particularly, we use Imply Squared Error
$$ MSE = mathbb{E}[||boldsymbol{Omega}_i^H – boldsymbol{Omega}_i^J||^2_2] ,$$
which measures the anticipated distance between human and choose multi-label vectors over the analysis dataset. This metric rewards choose programs that determine the identical set of believable interpretations as people. When people are break up on whether or not an output is poisonous (boldsymbol{Omega}_i^H = [0.8, 0.5]), a choose that mirrors this uncertainty achieves decrease error than one which favors a single interpretation—even when that assured selection matches the bulk’s forced-choice score.
Empirical Validation
To validate our framework, we carried out experiments with 9 industrial LLMs as choose programs and eleven score duties. These score duties included ideas comparable to factuality, helpfulness, relevance, and toxicity. Whereas we are able to straight elicit forced-choice and response set rankings from choose programs utilizing completely different prompts, present analysis datasets solely include forced-choice human rankings. Because of the points described above, it isn’t attainable to get better the “true” response set distribution from these present forced-choice rankings.
Due to this fact, we introduce a sensitivity parameter (beta^H) that controls the chance {that a} human rater contains the optimistic possibility (e.g., “poisonous”) of their response set regardless of choosing the unfavorable possibility (e.g., “not poisonous”) as a forced-choice score. For instance, (beta^H) = 0.3 implies that 30% of raters who selected “not poisonous” really thought-about “poisonous” to even be cheap. Setting (beta^H) = 0 recovers the case with no score indeterminacy. By systematically various (beta^H), we are able to characterize how meta-evaluation outcomes change underneath completely different ranges of indeterminacy.
In our evaluation, we examine how choose programs chosen by completely different meta-evaluation approaches carry out on downstream analysis duties. These meta-evaluation approaches range in how they accumulate and mixture rankings, and the way they measure human–choose settlement (see paper for particulars). As we focus on subsequent, the downstream analysis duties thought-about in our evaluation characterize frequent use circumstances of choose programs in lifelike deployment situations.
Content material Filtering: In content material filtering, a choose system decides which outputs from a goal system to permit or suppress. As an illustration, a platform should decide whether or not to filter probably poisonous content material, balancing consumer security towards the potential for high quality of service harms.
We measure efficiency through choice consistency—how typically a choose makes the identical permit/suppress choices as people:
$$ C^{tau}(Y^J, Y^H) = mathbb{E}[mathbb{1}[s_{k}^{tau}(Y^J_{ML}) = s_{k}^{tau}(Y^H_{ML})]]. $$
Right here, (s_k^{tau}(Y) = {1}[ Y_k geq tau ] ) is a thresholding perform that classifies content material as poisonous if the multi-label chance for possibility (ok) exceeds a threshold (tau ). For instance, if ok=”poisonous” and (tau=0.3), content material will get filtered when there’s no less than a 30% chance a rater identifies a poisonous interpretation. The brink (tau) represents the analysis designer’s danger tolerance. Decrease values filter extra aggressively.
Prevalence Estimation: In prevalence estimation, a choose system is used to estimate how continuously a sure idea — like helpfulness or toxicity — is current in goal system outputs. This estimation process is often utilized in automated red-teaming when estimating the assault success fee, or when estimating the win-rate between two fashions for a leaderboard.
We measure efficiency through estimation bias—how a lot an estimate obtained from a choose system differs from one obtained from human rankings:
$$B^{tau}(Y^J_{ML}, Y^H_{ML}) = mathbb{E}[s_k^{tau}(Y^J_{ML})] – mathbb{E}[s_k^{tau}(Y^H_{ML})]$$
For instance, if people determine 40% of outputs as poisonous however a choose estimates solely 25%, this -15% bias means the choose underestimates the prevalence of toxicity. Each metrics function on multi-label vectors that protect details about score indeterminacy. This enables downstream customers to set their very own thresholds based mostly on their danger tolerance and use case, slightly than being constrained by how particular person raters resolved indeterminacy when pressured to decide on.
Discovering 1: Decide programs differ from each other—and therefore additionally from human raters—in how they resolve score indeterminacy. Whereas we don’t know the true human sensitivity parameter, we are able to estimate every choose’s sensitivity parameter (hat{beta}^J_t) utilizing its responses to each forced-choice and response set prompts. We see super variation throughout programs and duties. E.g., for SummEval (Relevance), estimated parameters cowl a spectrum of 0.01 to 0.54 throughout programs.
Discovering 2: When human raters resolve score indeterminacy otherwise from choose programs, settlement metrics measured towards forced-choice rankings yield sub-optimal picks of choose programs. When people and choose programs resolve indeterminacy otherwise ((beta^H neq beta^J)), forced-choice human–choose settlement metrics like Hit-Fee, Cohen’s (kappa) and Jensen-Shannon Divergence choose choose programs that carry out poorly on downstream duties. Distributional settlement metrics like Jensen-Shannon Divergence are inclined to carry out higher than categorical settlement metrics like Hit-Fee. However efficiency degrades when (beta^H) exceeds 0.2-0.3.
Whereas Determine 5 summarizes mixture remorse, Determine 6 under exhibits how these rating inversions play out on particular duties. Every column compares the rating produced by a human–choose settlement metric (left axis of every subplot) with the rating produced by the downstream metric (proper axis).
- On SNLI (left column), no inversion happens: the choose system that scores highest underneath Cohen’s κ additionally achieves the bottom downstream bias. This exhibits that present metrics can work nicely on some duties.
- On SummEval (Relevance) (center-left), nonetheless, the story is completely different: the choose system with the perfect KL-Divergence rating is not the system with the bottom downstream estimation bias. Choosing the flawed choose on this case will increase estimation bias by 28%; equal to grossly mis-estimating the speed of “related” goal system outputs by an further 0.28 (on a scale of [0,1]).
- Lastly, the TopicalChat (Comprehensible) columns (proper) illustrate two extremes. The multi-label MSE metric stays steady and in line with the downstream metric, even underneath human score indeterminacy ((beta^H_t=0.3)). In distinction, Hit-Fee, a broadly used categorical settlement metric, yields a extremely inconsistent rating.
Discovering 3: Multi-label metrics appropriately determine high-performing choose programs. Figures 5 and 6 illustrate that our proposed method, which entails eliciting response set rankings and measuring human–choose settlement through a steady multi-label settlement metric (MSE) selects way more performant choose programs than forced-choice settlement metrics. Even when beginning with an present corpus of forced-choice knowledge, we are able to estimate the interpretation matrix (hat{mathbf{F}_i}) utilizing simply 100 paired forced-choice and response set rankings and nonetheless choose performant choose programs (see paper for particulars).
Sensible Takeaways
Primarily based on our findings, we provide 4 concrete suggestions for enhancing meta-evaluation:
1. Absolutely specify binary score duties by including a Perhaps or Tie possibility. This straightforward change eliminates the identifiability problem described above by making a one-to-one correspondence between forced-choice choices {Sure, No, Perhaps} and response units {{Sure}, {No}, {Sure,No}}. Be aware: this method solely works for binary duties—score duties with three or extra choices can’t be absolutely specified this manner.
2. Use response set elicitation when amassing new datasets. When it isn’t attainable to completely remove indeterminacy (which is frequent for properties like helpfulness or relevance), accumulate response set rankings the place raters choose ALL choices which are cheap. Then, measure settlement utilizing a steady multi-label metric like MSE. This preserves essential details about score indeterminacy that forced-choice elicitation eliminates.
3. Accumulate small auxiliary datasets to reinforce forced-choice rankings. Have already got forced-choice knowledge? Accumulate simply ~100 paired forced-choice and response set rankings to estimate the interpretation matrix (hat{mathbf{F}}). Our experiments present this small funding permits significantly better choose choice (Discovering 3 above). Try our GitHub tutorial for implementation particulars.
4. In the event you should use forced-choice, select distributional metrics rigorously. Our outcomes constantly present KL-Divergence within the human→choose course (not choose→human) performs finest amongst forced-choice human–choose settlement metrics. Keep away from categorical metrics like Hit-Fee, that are unreliable underneath score indeterminacy.
Need to be taught extra or do that method out for your self? Discover our implementation and quickstart tutorial on GitHub!
Acknowledgements: This weblog publish relies on our NeurIPS 2025 paper Validating LLM-as-a-Decide Methods underneath Score Indeterminacy, co-authored with Solon Barocas, Hannah Wallach, Kenneth Holstein, Steven Wu, and Alexandra Chouldechova. Many due to my co-authors and to members of the Sociotechnical Alignment Heart (STAC) at Microsoft Analysis for invaluable suggestions on early drafts of this work. Moreover, many due to Wayne Chi and Kiriaki Fragkia for useful suggestions on earlier variations of this weblog publish.
