Skip to content
goodeye

Migrate off OpenAI Evals before it shuts down

By Randy Olson, Co-Founder & CTO, Goodeye

OpenAI Evals goes read-only on October 31, 2026 and shuts down on November 30, and OpenAI points users at Promptfoo. If you want a like-for-like batch runner over a dataset, take that path. If you wanted a standard the agent itself has to satisfy on the work in front of it, write it as a Goodeye verifier, deploy it once, and have the skill call it inside the agent's loop.

OpenAI Evals goes read-only on October 31, 2026 and shuts down entirely on November 30, 2026. The read-only date is the one that should be on your calendar: after it, you can look at what you built but not change or export your way out of it. The same announcement retired Agent Builder and Reusable Prompts on the same November 30 date.

OpenAI's own recommendation is Promptfoo, and there is an official migration guide for it. That is the right move for some of what you were running and the wrong move for the rest, so sort before you port.

First, decide which job you are replacing

Look at what your evals were actually for.

You were scoring a dataset: a fixed corpus of inputs, expected outputs, a score you look at after a change to catch regressions. That is batch evaluation, it is useful, and the closest replacement is a batch runner. Take the Promptfoo path and the official migration guide. Check the export before you plan the work: if it hands you result files rather than a runnable config, you are rebuilding the harness, not importing it.

Or you were trying to catch bad output while it could still be fixed. You wrote an eval because an agent kept producing work that was subtly wrong, and you wanted it caught. A batch runner catches regressions across a corpus on a schedule, which is a different question from whether the specific draft the agent just produced is good enough to send. That second job needs a check that runs on the actual output, inside the agent's loop, where a failure means the agent does the work again.

What a verifier is

A verifier is one check that judges an agent's output against a standard you set, and returns pass or fail with reasoning. Goodeye hosts it: you deploy it once, it gets an id and an immutable version, and anyone you share it with runs that exact one.

Three kinds coexist, and the split matters when you are porting evals across:

Structural checks cover format, schema, required fields, and presence. Functional checks cover tests, numeric bounds, regex, and hashes. Both kinds are deterministic and free, so they live inline in the skill body. Semantic checks are the interpretive ones, such as tone, factual support, or policy adherence, and they need an LLM judge calibrated with your labeled pass and fail examples, so they are deployed as a private record and referenced from the skill by id.

If your eval was asserting a JSON shape, you do not need a judge for it. If it was asserting "the summary does not claim anything the transcript does not support," you do.

Port one check, not the whole suite

Start with the check you would most hate to lose, and take the criterion rather than the harness.

Read what the eval asserted and write it as one sentence with an explicit pass line. Then build the config:

{
  "name": "summary-supported-by-transcript",
  "description": "A call summary that claims nothing the transcript does not support.",
  "criterion": "Return passed=true only when every factual claim in the summary is supported by the transcript, including numbers, dates, commitments, and names. Return passed=false if any claim is unsupported, if a hedged statement in the transcript is reported as settled, or if a commitment is attributed to the wrong party.",
  "input_contract": "text",
  "input_fields": ["summary", "transcript"],
  "few_shot_examples": [
    {
      "inputs": {
        "summary": "Dana agreed to send the revised SOW by Friday and confirmed the budget is approved.",
        "transcript": "Dana: I can get you the revised SOW by Friday. On budget, I think we're approved but let me confirm with finance."
      },
      "passed": false,
      "reasoning": "The transcript hedges the budget, the summary reports it as confirmed."
    },
    {
      "inputs": {
        "summary": "Dana will send the revised SOW by Friday. Budget approval is still pending confirmation from finance.",
        "transcript": "Dana: I can get you the revised SOW by Friday. On budget, I think we're approved but let me confirm with finance."
      },
      "passed": true,
      "reasoning": "Both claims match the transcript, and the hedge is preserved."
    },
    {
      "inputs": {
        "summary": "Priya committed to a security review before launch.",
        "transcript": "Marcus: We should get a security review before launch. Priya: Agreed, I'll ask the platform team to scope it this week."
      },
      "passed": false,
      "reasoning": "Priya committed to asking another team to scope the review, not to the review itself, so the commitment is attributed too strongly."
    }
  ]
}

Deploy it:

goodeye verifiers deploy ./summary-supported-by-transcript.json

That returns a verifier_id, a version, and a version_token. Run it against any output:

goodeye verifiers run summary-supported-by-transcript \
  --inputs-json '{"summary": "...", "transcript": "..."}'

You get PASS or FAIL plus the judge's reasoning. Add --json and read the passed field when you want to gate a script on it.

Pull your calibration examples from cases you already have. Evals you are migrating are a good source, because the rows you argued about are exactly the boundary cases a judge needs to see.

One eval usually becomes several verifiers. If a single eval asserted that a summary was accurate, appropriately brief, and correctly formatted, split it into three. Each verifier should target one failure mode, so a failure tells the agent what to fix rather than handing it a number.

Put the check where the work happens

Deploying a verifier gives you a check you can call. The reason to bother is binding it to the skill so the agent runs it on its own output:

VERIFIER_ID=$(goodeye verifiers list --json \
  | jq -r '.items[] | select(.name=="summary-supported-by-transcript") | .verifier_id')
SKILL_VERSION_TOKEN=$(goodeye skills get my-skill --json | jq -r .version_token)

goodeye skills publish ./SKILL.md \
  --verifier "summary-supported=$VERIFIER_ID@1" \
  --expected-version-token "$SKILL_VERSION_TOKEN"

The token is required when the skill already exists and must be left off a first save. Pin the version with @1 when you want a pass to mean exactly one thing for the life of a release. An unpinned binding follows the verifier's current version, so every redeploy quietly changes the bar.

Then write the call into the skill body, so the agent invokes the check on its draft and gates its next step on the result instead of treating the check as someone else's review step. Run a rubric check on AI output from the CLI covers that binding in detail, and write an AI agent skill your team can rely on covers the skill body itself.

What survives the next deprecation

The reason to move the standard rather than the harness is that the standard is the part worth keeping. A verifier stores your criterion and your calibration examples. It does not store a dependency on one vendor's evaluation product, and it is reachable from the CLI, an MCP server, and the REST API, so changing which agent or model does the work does not invalidate the check on that work.

That is the useful thing to take from a forced migration. The harness was replaceable. What you actually built was a written-down definition of good enough, and that should outlive whichever tool happened to be running it.

Where to start this week

  1. List the checks you would miss on December 1.
  2. Sort them: batch regression scoring goes to a runner, output gating becomes a verifier.
  3. Port one verifier end to end, calibration examples included, and bind it to the skill that produces the output.
  4. Do the rest once you have seen one work.

Anything that needs exporting or editing on the OpenAI side has to happen before October 31, when the platform goes read-only. The sorting step is what decides whether you spend the migration rebuilding what you had, or replacing it with something that catches the problem while the agent can still fix it.

Frequently asked questions

What should I use instead of OpenAI Evals now that it is shutting down?

It depends on what you were using it for, and there are two different jobs here. If you were running batch scores over a fixed dataset in CI, OpenAI recommends Promptfoo and that is the closest replacement; take it. If what you wanted was a standard the agent has to satisfy on the actual work it just produced, a batch runner is the wrong shape: it scores a corpus after the fact instead of gating the draft in front of it. That job is a Goodeye verifier: one criterion you author, deployed once, called by the skill inside the agent's loop, returning pass or fail with reasoning so the agent revises and tries again.

How do I migrate an OpenAI eval to a Goodeye verifier?

Take the grading criterion, not the harness. Read what your eval was actually asserting and write it as one sentence with a clear pass line, then put it in a JSON config with a criterion, an input_contract, the input_fields the judge reads, and 3 to 10 labeled pass and fail examples drawn from cases you already have. Deploy it with goodeye verifiers deploy ./my-check.json, which returns a verifier_id and a version. Run it with goodeye verifiers run my-check --inputs-json '{"field": "..."}'. One eval that asserted five different things becomes five verifiers, one per failure mode, which is the point: a check that fails should tell you what broke.

What's your process for automated testing for AI agents?

Split it by what kind of wrong you are catching. Structural and functional checks (required fields, schema, numeric bounds, regex, tests) are deterministic and free, so write them inline in the skill body and let the agent run them on itself. Interpretive judgment (is this tone right, is this claim supported, does this follow the policy) cannot be asserted that way, so deploy it as a semantic verifier: one criterion, judged by an LLM calibrated with your own labeled examples, returning pass or fail with reasoning. The agent calls the checks, reads the verdict, and revises until it passes, so failures get fixed inside the loop rather than in a review pass afterward.

Will my checks break when an OpenAI model is deprecated?

Not the way an eval suite pinned to a retiring model does. A Goodeye verifier stores the criterion and its calibration examples, so the standard you wrote survives a model retirement even though the judge behind it can be repointed at a current model in model_settings. The same verifier is reachable from the CLI, an MCP server, and the REST API, so it also survives changing which agent or harness runs the work. Pin a verifier version with --version N when you want a pass to keep meaning exactly one thing across a release.

How is a verifier different from an eval?

Where it runs and what happens when it fails. An eval typically runs as a separate suite over a dataset and produces a score you read afterward, which makes it a reporting tool. A verifier runs inside the agent's own loop on the actual output the agent just produced, and returns pass or fail with reasoning that the agent acts on, so a fail means redo the work rather than a number to interpret later. Both are useful and they are not substitutes: keep a batch runner for regression testing across a corpus, and use verifiers for holding a specific piece of work to a standard while the agent can still fix it.