Run a rubric check on AI output from the CLI

You have an AI agent producing work that no test can grade: a pull request description, a support reply, a summary of a customer call. You know what "good enough to ship" means, and you can say it in a sentence. What you want is to type one command, hand it the output, and get back a straight pass or fail against that sentence, with a reason. Then you want your whole team typing the same command and getting the same answer. This guide walks through exactly that: author a rubric, deploy it, run it from the terminal, and bind it to a skill so the agent runs it on itself.

Run the check in three commands

Goodeye is the hosted service the rubric lives in: it stores your checks and your agent skills, and gives everyone on your team one version of each. You reach it with the goodeye CLI, an MCP server, or a REST API. The CLI needs an account and a one-time sign-in, both covered in getting started.

A rubric in Goodeye is a semantic verifier: one criterion, judged by an LLM, returning pass or fail with reasoning. You define it as a single JSON object. Here is a rubric for pull request descriptions:

{
  "name": "pr-description-review-ready",
  "description": "A PR description a reviewer can act on without asking questions.",
  "criterion": "Return passed=true only when the description states what changed, why it changed, and how the author verified it, and calls out every behavior change a reviewer must check against the diff. Return passed=false if any of the four is missing, or if a claim in the description is not supported by the diff.",
  "input_contract": "text",
  "input_fields": ["description", "diff"],
  "few_shot_examples": [
    {
      "inputs": {
        "description": "Switch the session cookie to SameSite=Lax.\n\nWhy: the old None value let a third-party iframe replay a logged-in session.\n\nVerified: signed in on staging in Chrome and Safari, checked the Set-Cookie header, and confirmed the embedded docs widget still loads.\n\nBehavior change: the widget now needs its own token, so any integration leaning on the parent session breaks.",
        "diff": "- SESSION_COOKIE_SAMESITE = \"None\"\n+ SESSION_COOKIE_SAMESITE = \"Lax\""
      },
      "passed": true,
      "reasoning": "States the change, the reason, how it was verified, and names the integration a reviewer has to check."
    },
    {
      "inputs": {
        "description": "Cleans up the cache layer.",
        "diff": "- CACHE_TTL_SECONDS = 300\n+ CACHE_TTL_SECONDS = 30\n- def warm_cache(keys):\n-     for key in keys:\n-         fetch(key)"
      },
      "passed": false,
      "reasoning": "No reason and no verification, and \"cleans up\" hides a tenfold TTL cut plus a removed warm_cache function the reviewer must weigh."
    }
  ],
  "model_settings": {"model": "openai/gpt-5.4-mini", "reasoning_effort": "medium"}
}

Deploy it. The command takes a file path, or - to read the JSON from stdin, which is the path to prefer when an agent generated the config:

goodeye verifiers deploy ./pr-description-review-ready.json

Deployed pr-description-review-ready v1 (verifier_id=8f2c…, version_token=9c1f8a3e-…)

Save that version_token. You need it to deploy the next version of this same rubric. Now run the check against a real output:

goodeye verifiers run pr-description-review-ready \
  --inputs-json '{"description": "Bumps the retry ceiling to 5.", "diff": "- MAX_RETRIES = 3\n+ MAX_RETRIES = 5\n- retry_backoff_ms = 100\n+ retry_backoff_ms = 250"}'

FAIL verifier_run_id=3ad1…
The description states what changed but gives no reason and no verification. The diff also
changes the `retry_backoff_ms` default, which a reviewer must check and which is not mentioned.

Fill the gaps the judge named and run it again, and you get the other verdict shape:

PASS verifier_run_id=7b04…
The description states the change, the reason, and the staging replay used to verify it, and
it calls out the `retry_backoff_ms` default the diff changes.

That is the whole loop. run accepts a verifier UUID or your own verifier name, and the --inputs-json keys must match the deployed input_fields exactly, no extras and nothing missing.

One thing to know before you script this: a FAIL is a finished judgment, not an error, so the command exits 0 on it and saves a non-zero exit for the case where no verdict came back at all. Gate on the verdict, not the exit code. Add --json and you get the whole run object (status, passed, reasoning, verifier_run_id, version); read passed, which is true, false, or null when status is error:

goodeye verifiers run pr-description-review-ready --inputs-json "$INPUTS" --json \
  | jq -e '.passed == true' > /dev/null || exit 1

The full flag list lives in the CLI reference, and the same operations are available as the deploy_verifier and run_verifier MCP tools and over REST, so an agent can do all of this without a shell.

Write a rubric, not a score

The reason this is useful and a "quality: 3/5" number is not comes down to three parts of the config, and they are worth getting right.

The criterion is where you draw the line. Write it as a direct instruction to the judge in the form "return passed=true when…, otherwise passed=false," and make it about one property of the output. If you cannot say what a pass and a fail look like in a single sentence, the check is not specific enough yet, and the honest fix is to split it into two verifiers rather than to soften the wording. A rubric that asks "is this good?" gives you a number you cannot act on; a rubric that asks "does every claim in the description trace to the diff?" gives you a verdict an agent can respond to.

Calibration examples keep the judge's taste aligned with yours. Attach 3 to 10 labeled examples, each marked passed true or false, each with a short reasoning. The judge sees them as demonstrations. This is the difference between an LLM's generic idea of a good PR description and your team's idea of one, and it is the single highest-leverage thing you can tune. When a verdict disagrees with you, do not rewrite the criterion first: add the disputed case as a labeled example and redeploy.

The input contract decides what the judge gets to look at. Use text for text output; text_image and image exist too. The contract matters more than it looks, because a judge can only check a claim against evidence you hand it. Giving the rubric above both description and diff is what turns "does this read well?" into "is this supported?"

You can also pin the judge model and a reasoning-effort hint in model_settings. See semantic verifier concepts for the full contract table, and run goodeye verifiers deploy --help for the judge model values the deploy accepts.

Deploy once, and everyone runs that exact one

This is the part that makes a check a team artifact rather than a snippet in someone's dotfiles. A deployed verifier is a hosted, versioned object with an id. Nothing is copied into a repo, so there is no second copy to drift.

Versions are immutable once written. To change a criterion or its calibration you deploy again under the same name, which appends a new version and leaves the old one running for anyone who pinned it. Each redeploy is guarded by the version_token from the previous one, passed as expected_version_token in the config JSON.

A redeploy sends the whole config again, not a patch. Nothing carries forward from the previous version, so start from the file you deployed last time, edit what you want to change, and add the token:

{
  "name": "pr-description-review-ready",
  "expected_version_token": "9c1f8a3e-4b7d-4a21-9f60-2ec8a5d31b04",
  "description": "A PR description a reviewer can act on without asking questions.",
  "criterion": "Return passed=true only when ... (your sharpened wording)",
  "input_contract": "text",
  "input_fields": ["description", "diff"],
  "few_shot_examples": [
    {"inputs": {"description": "...", "diff": "..."}, "passed": true, "reasoning": "..."}
  ],
  "model_settings": {"model": "openai/gpt-5.4-mini", "reasoning_effort": "medium"}
}

name, description, criterion, and input_contract are always required, and a text contract needs its input_fields too, so a config that drops one of those is rejected outright. few_shot_examples is the dangerous one: it is optional, so a config that leaves it out is accepted and ships a version with no calibration at all, which is the single worst thing you can do to a rubric that was working.

If the token does not match the current one, the deploy is rejected as a conflict, so two people sharpening the same rubric on the same afternoon cannot silently clobber each other. Lost the token you saved at deploy time? Look up the current version and token for anything you own at any time:

goodeye verifiers list --json

Ask for --json specifically. The --table view lists each verifier's current version but not its version_token, so JSON is the mode to read the token back from.

To read a rubric back, including its criterion and its calibration examples, use goodeye verifiers show pr-description-review-ready --json, adding --version N to inspect an older one. It also prints a config_hash, which is a fast way to tell whether two versions actually differ. A verifier is not a black box: anyone who can reach it can read exactly what it grades on, which is what lets a teammate argue with the rubric instead of guessing at it.

When you want a pass to mean precisely one thing for the life of a release, pin the version rather than tracking the current one. On a one-off run that is --version N; where a skill references a verifier it is verifier_id@version. Both are covered in versioning and the version token.

Grade a skill against a rubric

Running the check by hand proves it works. The point is to stop running it by hand.

A skill is the runbook your agent follows, and it names the verifiers it depends on. Bind yours when you save the skill, with a repeatable --verifier flag that maps a logical name to the deployed verifier:

goodeye skills publish ./SKILL.md \
  --verifier review-ready=<verifier_id>@1

The @1 pins the version. Leave it off and the binding stores no version at all: it follows the verifier, so every run resolves to whatever the current version happens to be at that moment, and each redeploy quietly changes what the check means. That is the reason to pin.

That command is a first save. Updating a skill that is already hosted needs one more flag, --expected-version-token, read from goodeye skills get <name> --json; the save is rejected without it rather than overwriting a version someone else just wrote. Note that this is the skill's own token, a different object from the verifier version_token above: each has its own. On a later save, omitting --verifier preserves the bindings you already had, and --clear-verifiers removes them all; see verifier references for the exact update semantics.

Binding the verifier records the dependency. It does not make the agent call it. What makes the check run is naming it in the skill body: write a step that says to run the review-ready check on the draft and to revise on a fail, naming the surface your agent has: the run_verifier MCP tool, or goodeye verifiers run from a shell. The body is the one part that travels on every path, so that one paragraph is what closes the loop. Do this even though you already passed --verifier.

How much else travels with the skill depends on which of the two paths your agent uses.

When your agent fetches the whole skill record, via the get_skill MCP tool or goodeye skills get --json, the response carries the skill body, its verifier bindings, and a standing instruction on how to report those checks: quote every verdict from the real verifier response, never claim a pass a verifier did not return, and close by naming each verifier and its actual outcome, including any check that did not run. That reporting instruction keeps a missed call from sliding by unmentioned; it does not invoke the bindings on the agent's behalf. Plain goodeye skills get prints the runbook markdown, which is the body plus that instruction and no bindings, so reach for --json when the agent needs to see what the skill is bound to.

When your agent reads skills from disk, via goodeye skills sync into ~/.claude/skills or an equivalent directory, only the skill body is written to the file. The bindings are tracked in the local sync index, not inside SKILL.md, so an agent that opens the file sees no check at all unless the body names it.

Structural and functional checks (schema validity, a test suite, a regex, a numeric bound) stay inline in the skill body either way, because there is no reason to pay for a judge on something you can decide with code. Only the interpretive judgment goes to a semantic verifier. For how to write the runbook half of this well, see writing an AI agent skill your team can rely on.

Sharing keeps the pairing intact. When you grant the skill to a teammate or a whole team, the verifiers it references cascade with the grant automatically: a view grant lets them read and run the judge, edit or admin lets them deploy new versions of it. With a pinned binding, their agent runs the same rubric at the same version yours does, so a pass means the same thing on their machine as on yours. That property is documented under verifier grants cascade.

Let the agent run the check on itself

Once a verifier is bound to a skill and named in its body, the interesting move is not reading the verdict yourself. It is handing the verdict back to the agent. The agent drafts, runs the check, and on a fail gets the reasoning and revises before it finishes, so the first version you see is one that already passed. That is why the verdict is a binary with an explanation rather than a score: "3/5" gives an agent nothing to do, while "the removed config key is not mentioned" is an instruction.

The full case for checking inside the loop instead of measuring after the fact is made in verify AI agent output before it ships, including the difference between the three kinds of check. This page is the mechanics; that one is the argument.

Where to start

Pick the check you currently perform by eye most often. Write it as one criterion with a clean pass/fail line, add three or four examples from real outputs you have already accepted and rejected, and deploy it. Run it from your terminal a few times against work the agent has produced, and add every disagreement as a new calibration example until its verdicts match yours. Then bind it to the skill, pin the version, and grant it to the people who need it.

Install the CLI and sign in with getting started, then keep verifiers open while you write your first criterion.