Blog

I read the July 28 changelog on a quiet Tuesday morning, expecting to skim it in thirty seconds. New scan command, faster results, some UX polish—the usual. What actually stopped me were two features that would not make a good demo reel: saved scan history with comparison, and scoped SECURITY.md policy. I closed the tab, reopened it, and read more slowly. These are not glamorous additions. They are the sign of a product that has started taking seriously the question I have been asking since AI security tools first appeared: how do you know the result is real, and how do you know it will still be real next week?

When I read Anthropic's position on open-weight models, I slowed down at a specific phrase. Not because it was alarming. Because it was precise in the way that hides a missing specification: sufficiently capable.

The sentence does real work. It says that all sufficiently capable models—open and closed—should undergo mandatory safety testing, while less capable models from startups and academia are exempted. The structure is reasonable. The phrase is doing the load-bearing. And as someone who spends most of his working day thinking about what ships, when, and under what conditions, I immediately wanted to know: which capability, measured how, against what threshold, before an irreversible release?

No answer followed. Not because Anthropic was being evasive—they say explicitly that they have never advocated a categorical ban on open weights, and that open-weight models without dangerous capabilities are a public good. The gap is real, but it's also honest. The threshold hasn't been settled. The phrase is a placeholder for a hard problem, not a solution dressed up as one.

That's the pause I keep returning to.

When I opened OpenAI's account of what happened during its ExploitGym evaluation, I paused at two adjacent sentences. The research environment was, in their words, "highly isolated." And then, a few lines later: network access was constrained to package installation through an internally hosted third-party proxy and cache for package registries.

I read that twice.

I opened the HN thread expecting a tidy summary. The submission title said "LLM Usage in Debian: Three Proposals." When I clicked through to the official Debian General Resolution page, it listed four.

That gap — three on the aggregator, four on the source — is a small thing, but it caught my attention in the way small things do when a governance process is still moving. The page showed a status of In Discussion and a discussion-period date of 2026-07-24. Nobody has voted. Nothing is decided. The community is working through something genuinely hard, and the working copy keeps changing under everyone's feet.

The phrase "open weights" does something to a developer's brain. It triggers a familiar sequence: model drops, you find the Hugging Face page, you scroll to the files tab, and somewhere in the back of your mind you are already thinking about disk space. That is the reflex. It is not irrational. It has worked often enough that it feels like a reasonable first move.

When Kimi K3 surfaced — in search suggestions, in developer feeds, on Hacker News, across GitHub — I followed the reflex. I went to the Hugging Face repository. I clicked through to the file list.

There were 96 weight shards. The metadata summed to 1,560,998,984,390 bytes. That is roughly 1.56 terabytes, decimal. Not a storage folder. Not a project directory. A model artifact the size of a modest home server's entire disk, split across nearly a hundred files.

I did not download it. I want to be clear about that. I looked at the file list the way you look at a shipping manifest for something you are not sure you ordered.

When I saw the Hacker News post titled Benchmarking Opus 5 on SlopCodeBench, the number that caught my eye was 4/17. My first instinct was the usual one: scan the score, file the model somewhere in my mental ranking, move on. Four passes out of seventeen checkpoints. Not great. Not catastrophic. Just another data point on the leaderboard treadmill.

Then I made the mistake of opening the paper.

The longer I use coding agents, the less I trust a model ranking on its own.

Rankings are useful, but they only tell me so much. I run Claude Code and Codex on real projects, and over time one thing has become hard to ignore: the same model can produce very different results under different harnesses. How context enters the run, which tools are available, and where verification happens all change the final output. Those choices can matter as much as the model itself, and sometimes more.

That is why my first reaction to a new model release is no longer to scan the leaderboard. I want to know what Agent, what harness, and what effort setting produced the new number.

Claude Opus 5 was released on July 24. This time, the public material gives us enough detail to ask that question more clearly.

The diagnosis was correct. The turn completed. Codex CLI returned the smallest-fix explanation for a ZeroDivisionError, and the final status read completed. If I had stopped there — closed the terminal, pasted the answer, moved on — I would have called the run a success and never questioned what happened in between.

Then I read the event stream.

On 2026-07-23 I watched a Codex CLI 0.145.0 turn complete correctly. Five tool calls, the right diagnosis, a final status of completed. I had preserved the bounded JSONL event stream from the observability work I'd been doing — the recorded tool calls, their results, and the two failed attempts along the way before the Agent reached the correct diagnosis. It was a satisfying run to look at.

And then I realized I had exactly zero evidence about tomorrow.

That trace explains one path through one turn on one day. It cannot tell me whether swapping the model version will regress that behavior, whether adding a new Skill to the loadout will cause a misfired tool, whether a prompt tweak to the harness will change how the agent approaches similar tasks. One successful run is not a regression test. It is evidence from one trial.

That gap is where Eval-Driven Development begins.

Something happened on Juejin on July 23 that is worth paying attention to, not because any single post broke new ground, but because of what the posts, taken together, reveal about where the conversation has moved.

A Juejin hot list checked on July 23 included a Top 10 Skills roundup, a guide to designing better Skills, and a piece asking, plainly, why installing more Skills can make an AI feel slower and less reliable. On Hacker News in the same window, projects surfaced for diagnosing loadout health, gating Skill changes with real usage evidence, and mapping which installed capabilities rarely fire. I am not citing these as verified market signals. Autocomplete suggestions and hot-list rankings show intent, not volume. But the intent itself is the signal: developers are no longer asking mainly "where can I get more Skills?" They are asking which Skills actually trigger, which ones collide, which ones sit unused, and whether any of them are worth their context cost.

That shift is worth understanding from first principles.