Skip to main content

Claude Opus 5: Coding Agent Harnesses Need Recalibration

· 9 min read
Isaac Zhao
AI Coding Club Creator

The longer I use coding agents, the less I trust a model ranking on its own.

Rankings are useful, but they only tell me so much. I run Claude Code and Codex on real projects, and over time one thing has become hard to ignore: the same model can produce very different results under different harnesses. How context enters the run, which tools are available, and where verification happens all change the final output. Those choices can matter as much as the model itself, and sometimes more.

That is why my first reaction to a new model release is no longer to scan the leaderboard. I want to know what Agent, what harness, and what effort setting produced the new number.

Claude Opus 5 was released on July 24. This time, the public material gives us enough detail to ask that question more clearly.

Every Benchmark Row Has Conditions​

On Frontier-Bench's public leaderboard on July 26, Opus 5 paired with mini-SWE-agent at max effort reached a 43.5% resolution rate (±1.7%) and ranked first. Artificial Analysis gave Opus 5 an Intelligence Index score of 61 at max effort, also first when I checked. Vals AI listed Opus 5 at 74.82%, second overall, while reporting that it ranked first on 14 of the 27 leaderboards evaluated so far.

Three independent evaluation organizations point in the same direction: Opus 5 belongs in the top tier for difficult agentic work. That is the most stable conclusion I can draw from the public evidence today.

The conditions behind each row matter. Frontier-Bench used mini-SWE-agent. Anthropic's internal Frontier-Bench run used a GKE backend, averaged five attempts per task, and configured Opus 4.8 as a fallback when the safety classifier refused a request. Artificial Analysis ran GDPval-AA v2 with the Stirrup Agent Loop, shell access, and web access. Vals used max effort for most runs, high effort for Terminal-Bench 2.1, and the same Opus 4.8 server-side fallback. Vals also reports that if every fallback-assisted result is counted as a failure, the overall score moves from 74.82% to 74.47%.

A benchmark row is the result of a model, an Agent, tools, effort, fallback behavior, and a verification environment working together. You cannot strip away the Agent and harness differences across rows and call what remains a pure model comparison.

Benchmark results combine the model, Agent, tools, effort, fallback, and judging environment

GDPval-AA v2 does give us one useful within-model comparison. Opus 5 scored an Elo of 1861 at max effort and 1632 at medium effort, while Opus 4.8 scored 1593 at max effort. The gap shows that effort materially affects this task distribution. It does not show that medium is generally optimal, or that max is always worth paying for. The right effort setting depends on the task.

What Anthropic Is Actually Recommending​

Anthropic's release material draws a useful boundary around the improvement. Opus 5 shows larger gains on difficult coding, multi-file features, large refactors, and end-to-end work. The difference is smaller on easy, single-turn edits.

That distinction matters to me because it says the value is not distributed evenly. Complex work with a larger execution budget is more likely to capture the gain.

The more interesting part is the migration guidance. Anthropic says Opus 5 verifies its work and self-corrects more actively. It also warns that prompts which still demand a final verification pass, a double-check, or subagent review can trigger over-verification and waste tokens. Independent verification scaffolding that was added to compensate for weaker models may now need to be reevaluated.

I read that as a concrete engineering change. Older scaffolding was compensatory by design: you expected the model to skip a verification step, so you wrapped another one around it. If the model now performs that step on its own, the outer layer may stop being an extra safety net and start becoming a repeated trigger. The result can be another loop and more token spend. That is a specific recalibration problem, not a generic piece of good news.

Legacy verification loops compared with a recalibrated Coding Agent harness

Anthropic makes two related recommendations. Narrow tasks need explicit scope boundaries so the model does not expand the assignment on its own. Subagents are better reserved for genuinely independent, substantial work; small tasks need limits on delegation count and cost.

I have not independently validated those migration recommendations. They are Anthropic's guidance. But they all point to the same operational issue: when model behavior changes, the harness needs to change with it.

The Friction in Early Use​

Claire Vo ran a blind evaluation across seven models and six tasks. Opus 5 scored well, but her report also records several points of friction: verbose output, initiative that expanded the task, problems in a merge-conflict scenario, and instruction conflicts.

Zvi Mowshowitz reads the public benchmarks as strong while keeping several concerns open: unproductive self-verification, overengineering, overconfidence, and self-correction loops that do not stop when they should.

The Hacker News launch discussion is mixed in the same way. Positive reports sit next to complaints about overengineering, token burn, fabricated references, and weak visual verification.

These reports are not incidence estimates, and they do not establish that Opus 5 regressed from 4.8. They show that early use is mixed. There is still a gap between ranking highly and feeling smooth in daily work. That gap appears with every major model release.

One Bounded Replay​

On July 16, I deployed a CTA fix and completed production verification. Opus 5 was released on July 24. On July 26, I reconstructed the evidence that had existed before deployment, hid the final outcome, and gave it to Opus 5 as a review task.

The environment was Claude Code 2.1.220 with claude-opus-5, standard mode, Fast disabled, and medium effort. Tools, web access, MCP, and subagents were all disabled. It was a single call.

The API run took 58.666 seconds, cost about $0.10, and produced 3,570 output tokens. The verdict was approve_with_followup, with no blocking finding. Its most useful follow-up was to add an end-to-end or live-environment smoke test. That recommendation matched the type of production verification I had already completed on July 16.

Two things are true at the same time.

First, the model was sensitive to a gap in the evidence. The prompt already included the bug description, target behavior, runtime facts, diff, and existing tests. The model could not investigate the repository and had no browser, but it still identified the absence of live-environment verification and made that its most important follow-up. That was a useful judgment.

Second, 3,570 output tokens was long for such a narrow task. With no tools available, the model had limited room to investigate, yet it produced more text than the decision required. That lines up with the verbosity Claire Vo recorded and with the direction of some early community feedback.

This was one case with no control group. It describes the behavior of one specific configuration. It is not a comparison against an earlier Claude model, Codex, another tool, or a human reviewer. Opus 5 did not participate in, approve, or change the July 16 deployment. The timeline matters.

The Harness Problem Did Not Disappear​

The public evaluations point consistently toward strong Opus 5 performance on difficult agentic tasks. I think that direction is credible.

Capability still does not mean "plug it in and you are done." A model that verifies proactively can become more expensive when paired with legacy verification scaffolding. A narrow task without a scope boundary can expand. An uncalibrated effort setting can under-invest in the work or overrun the budget.

Those are harness-level problems. A better model does not solve them automatically. Anthropic's migration guidance is aimed at this layer.

My current judgment is that Opus 5 can take on harder work. I believe that is real. The responsibility for deterministic tests, live-environment verification, and local evals has not moved, though. Those mechanisms still close the loop, and the model's own verification cannot replace them.

Model capability and the harness are two variables. A new model release updates only one of them.

Sources

If you want the underlying framework before choosing a model, start with What Is a Coding Agent Harness?. The related Eval-Driven Development for Coding Agents article explains why local evidence still matters after a model upgrade.