Insights
- Article
Repeating an eval does not make it more reliable. Human labels do.
LLM judges often fail on the same traces, so their errors are correlated. TL;DR: use ground truth human labels to trust your evals.
- Article
Evaluation Is the Foundation of Agent Harness Design
OpenAI and Anthropic independently arrived at similar patterns for designing effective agent harnesses. In both cases, evaluation is the foundation.
- Article
The Tufte Test: Teaching an AI Agent to Make Better Data Visualizations
AI agents make generic charts. I held one to Tufte's data visualization principles and had it keep improving until the chart carried the story.
- Experiment
The November 2025 AI Coding Surprise, Model by Model
In November 2025, AI coding tools went from halting and clumsy to surprisingly capable. We gave 22 models the same prompt and ran five replicates each.
- Article
The Return of the Data Scientists
Generating code is easy now, but more code has never meant better products. As production costs fall toward zero, the bottleneck shifts to judgment and taste.
- Article
The AI Mirror Effect: Why Your AI Evaluations Need Domain Experts
The Anthropic Economic Index shows that the quality of what you put into AI almost perfectly predicts the quality of what you get out.
- TalkPortland AI Engineers
Beyond the Demo: Building Reliable AI with LLM Evaluations
Why traditional testing breaks with stochastic systems, how generic LLM-as-Judge approaches miss domain nuance, and how to implement contextual evaluations.
- Article
2025 Year in Review for LLM Evaluation: When the Scorecard Broke
In 2025 we discovered we had been measuring memorization, not intelligence. This year exposed the crisis in AI evaluation, and taught us what to build instead.