Give each LLM judge a single job

You need to run many evals to gain confidence in your AI application. For example, if you build an agentic support bot, you might want to build evals for escalation to a human, documentation retrieval accuracy, or for tool calls that automatically apply promotions.

One mistake is to put all these evals into a single LLM judge.

When you lump all your evals into a single LLM judge, it becomes hard to debug and improve your evals when new edge cases appear. It also becomes hard to remove evals that you no longer need.

In this article, we will cover the advantages of atomic judges. We will also show how much time and money this approach can save your team with simple assumptions.

TL;DR

• Make each LLM judge atomic (one specific failure mode per judge).
• Make each verdict pass or fail.
• Atomic judges are easier to tune, replace, and run in parallel.
• Atomic judges cost less when your evals are used as guardrails that gate outputs.

What do we mean by atomic and binary judges?

It is good practice to make your evals atomic and binary. Atomic means that an eval covers one failure mode. Escalation to a human, documentation retrieval accuracy, and tool calls that automatically apply promotions are three atomic checks.

Binary means pass or fail. Each check must have clear instructions for what passes and what fails.

Advantages of atomic and binary LLM judges

Compared with a single judge, a collection of atomic and binary judges lets you:

• Make modifications to one judge without affecting the others. For example, you can tune the prompt and model for the tool call check without affecting the human escalation check.
• If an eval is saturating and is no longer needed, you can remove or replace it without affecting the others.
• When you discover a new failure mode, you can add it as a separate eval without affecting the others.
• Run all your checks at the same time in parallel and save wall-clock time.
• If you use evals as a guardrail and the output needs to pass, you can save time and money. When one check fails, you can regenerate the output without waiting for the rest. With a combined judge, you have to wait for the full eval.

This last point is something many people overlook. Here is a simulation that helps illustrate it.

Say we are using an eval as a quality gate that an agent needs to pass. Let's assume we have different criteria. With atomic checks, each check is a separate LLM judge. With a combined judge, all checks are part of a single LLM judge. Let's also assume each check has the same failure chance, which you can control in the simulation.

failure chance per criterion10%

atomic judgescombined judge

The takeaway is that the higher the failure chance for each check, the faster the expected number of rechecks increases. The effect becomes more pronounced as the number of criteria increases: with 10 criteria at a 30% failure rate, the combined judge needs 9,718 rechecks, while the atomic checks need 3,000.

In summary

Make each LLM judge atomic, with one criterion for each failure mode you want to catch. This lets you reduce costs, iterate on and tune your evals more easily, and reduce wall-clock time.