Repeating an eval does not make it more reliable. Human labels do.

Many people run evals to measure how well their AI works. A common practice to trust the eval is to repeat it a few times. You judge each trace several times and take the majority answer. This motivation is quite reasonable. Through repetitions and majority voting, the correct votes can outnumber the wrong ones.

Unfortunately, this reasoning does not always hold. It turns out LLM judges often fail on the same traces. In other words, their errors are correlated.

The implication of error correlation is that running a judge multiple times, or running different models and taking their consensus, does not give you much new information. Your evals do not become more accurate.

In this article, we will explore what error correlation means in the context of evals, run simulations on how it affects the accuracy you get back from repeated evals, and see how we can build more trust in our evals.

TL;DR

• Correlated errors in evals reduce the value of repetitions.
• Eval agreement ≠ correctness.
• Use ground truth human labels to trust your evals.

What is error correlation in the context of evals?

For now, let's assume we have a simple, pass/fail eval setup. Say we have 200 traces and we are using an LLM judge to score each one. For this eval, let's say we know the ground truth answers and the judge's accuracy is 80%. Now, let's rerun this eval multiple times and use the majority answer for each trace as the final verdict.^{1We score the judge with one accuracy number here to keep things simple. Real evals are usually lopsided, with far more correct traces than wrong ones, and a simple accuracy score can be misleading. In practice, you want to compute the true positive rate (the share of correct traces the judge catches) and the true negative rate (the share of wrong traces it catches) and report them together. The findings below hold if you use true positive and true negative rates as well.}

Figure 1 puts two cases side by side. Every run in both panels is 80% accurate, meaning it gets two of the ten traces wrong.

The difference between the left panel and the right panel is the level of error correlation. In plain terms, error correlation is how much the runs get the same traces wrong. Near zero, each run misses its own traces and the misses move around from run to run. Near one, every run misses exactly the same traces.

On the left, the errors land on different traces each run, so every wrong judgment is outvoted four to one and the consensus is right for all traces. On the right the same two traces are wrong every run, so the majority repeats both mistakes and the consensus lands on eight of ten. The extra runs cost more and tell us nothing new.

Reruns of the same model can also be highly correlated, because changing the sample does not necessarily change the mistakes the model tends to make. An interesting finding is that different judge models have a high degree of error correlation as well. In one study, nine judges from seven model families had a mean error correlation of 0.39ref1Nine Judges, Two Effective Votes (arXiv 2605.29800). In another experiment in the same paper, the error correlation was 0.44. This means that a council of judges from different model families, say GPT and Claude, is less independent than many people think.

Simulating error correlation to interpret accuracy of repeated evals

Figure 2 narrows the consensus simulation down to two inputs. The first is how accurate a single judgment is, and the second is the error correlation between the repeated judgments.

In this simulation, the majority verdict of nine judges, each at 80% accuracy, reaches 98% accuracy when their errors are independent. With 39% error correlation, it reaches 84.5%. In effect, we multiplied the compute by nine for a 4.5 point gain in accuracy. If you push the error correlation higher, the curve collapses toward 80%, even at 25 repetitions.

In short, the agreement between judges is not a good reason by itself to trust your evalsref2Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness (arXiv 2603.06612). The judges can agree on the same wrong answer when their errors are correlated, as is the case even with models from different labs.

Use human labels to trust your eval's accuracy

What should we do then? To trust any measured accuracy, we need evidence from outside the voting system. That evidence is human labels.^{2Human labels are not automatically perfect. In the examples here, we assume the domain experts apply the rubric reliably and provide trustworthy reference labels.}

Grounding works because a person is not another copy of the judge. A human reading the trace applies the same criterion without the model's quirks. In effect, the human answer acts as genuinely new information rather than a repeat of the same mistake. This helps us ground the accuracy of individual judges, or even a council of judges, in realityref3Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685).

A simple way to use human labels for grounding:

• Take a random sample of traces.
• Have your domain expert label each one without seeing what the judge said.
• You now hold a small set of traces where you know both answers.
• Compare the two and see how often the eval agrees. Say you label 50 traces and the judge agreed with the person on 44 of them. That is 44 out of 50, or 88%, and that number is your estimate of the judge's accuracy on the whole eval.
• The more human labels you have, the more confidence you can build that your judge is working as intended, and the sooner you can fix it if it is not.^{3Since domain expert labels are valuable, it is often good practice to keep most of them as a held-out set rather than spending them as few-shot examples in your judge prompt. We will not get into the details in this article.}

The catch here is that it can be hard to pick the traces for labeling. Sometimes you get a lucky or an unlucky draw, which pushes the estimated accuracy away from the true one. Figure 3 is a simulation that illustrates this. The main idea is that the more you label, the tighter your estimate gets around the judge's true accuracy.

In this example, a judge with 80% true accuracy can measure anywhere from about 60% to 100% when we label only 10 traces. Labeling 100 traces puts 9 out of 10 measurements within about 5% of the judge's true accuracy. Other evals may need more or fewer labels depending on their size, composition, and the precision you need.

The same logic holds for consensus evals as well. When we score the majority answer against the same human labels, its measured accuracy behaves exactly like the chart above, with the uncertainty shrinking as labels accumulate. Ground truth labels let you measure whether your eval repetition strategy actually improves accuracy.

Conclusion

• Repeating an eval does not make it more trustworthy on its own. Repeated runs tend to have correlated errors, meaning they get the same traces wrong.
• The best way to trust evals is through ground truth labels from your domain experts.
• About 100 randomly selected traces is often a useful starting point. How many you need depends on the size and mix of your eval, and how precise you need the result to be. Make sure the human reviewers are blind to the judge's answers so they can make independent assessments.
• Compare eval strategies against the labels. For example, if repeated judgments improve accuracy against human labels and justify their cost, use them. If not, spend your budget on a better rubric, a better judge, or more trusted labels.