Evals & benchmarking

How to benchmark your skills in 2026

8 min read · Updated July 2026

Leaderboards tell you which model wins on someone else's tasks. They can't tell you which agent will do yourwork best — the research, the triage, the drafting you actually ship. Here's how to run a benchmark that answers the only question that matters: which harness and model, on my work, right now.

1.Define what good means for your work

A benchmark is only as honest as its tasks. Skip the synthetic suites and write down five to ten jobs you did this month that you'd happily hand to an agent: “summarize this 40-page RFC into a decision memo,” “triage these support threads and draft replies,” “pull this week's competitor launches and tag them.” For each one, name the outcome you actually care about — accuracy, tone, how few follow-ups it takes — so scoring later isn't a vibe.

2.Build the matrix: harness × model

Two variables decide an agent's skill: the harness (Claude Code, Codex, Hermes, OpenClaw — how it plans, calls tools, and keeps state) and the model (the brain inside it). They interact, so test them as a grid, not one at a time. Pick two or three harnesses and two or three models and you have a six-to-nine-cell matrix. On Asteroids, each cell is one click: launch an agent per combination, no infra and no API wiring to stand up first.

3.Hold every other condition identical

If two contenders get different prompts, you're benchmarking your prompt-writing, not the agents. Give every cell the same system prompt, the same capabilities (search, scraping, files), and the same channels. The point of the matrix is that the only things changing are the harness and the model — everything else is a constant you fix once and reuse across the grid.

4.Run on real tasks in real channels

Send your five-to-ten jobs to every agent the way you'd really send them — over Slack, Telegram, or web chat, not a benchmark script. Real channels surface things synthetic runs never do: how an agent handles a vague follow-up, whether it asks a clarifying question, how it recovers when a tool call fails. Keep the transcripts; they are your evidence.

5.Swap models mid-run without resetting

The sharpest test is continuity: give an agent a long, multi-step task, then change its model halfway through and watch how it picks up the thread. Because agents on Asteroids keep their full history when you swap the model, you can A/B how different brains continue the samework instead of restarting from a cold prompt — the closest thing to a controlled experiment you'll get.

6.Score, decide, and park the losers

Lay the transcripts side by side and score each against the outcomes you named in step one. A rubric of three or four numbers beats a gut call and makes the decision defensible to your team. Then keep the winner working and park the rest — parked agents cost nothing, so there's no reason to tear down the matrix until you're sure. Re-run it next quarter when the frontier moves; the setup is already there.