Give the same model a repository that can check its work, and “done” starts meaning done. In the audit beside this, the one number holding the other six back was 14 out of 100, and that is where the fix starts.
$7,500, first repository. Fixed scope, published price.
Agents don’t ask. They guess. A new engineer who can’t tell whether a change is right will ask someone. Your agent has one way to find out whether it succeeded: whatever your repository can prove.
One audited repository, a TypeScript and SQL trading library of about 5,000 lines, gave its agent plenty to read, 75 out of 100, and almost nothing to check against: 14 out of 100. Agent context files, a README, a lockfile, a clean secrets scan, and very few tests beside the source. Every change the agent made there was marked done on its own say-so, because nothing in the repo could tell it otherwise. Until that second number moves, a better prompt and a better model land on the same guess.
A lock you can’t open isn’t the same as a lock you can’t inspect.
Which is the useful part. The fix is ordinary engineering with a schedule attached: tests on the paths that matter, checks wired into CI, the conventions your team already follows written down once. Waiting for the next model release won’t help. It has never seen your codebase.
Verification Surface 14 / 100 · the bottleneck
Every score points at something you can open: a file, a line, and the measured value behind it. Nothing here is scored by opinion, and nothing by a model. The numbers below come from a real audited repository.
| Dimension | What it asks | Score |
|---|---|---|
| Context Qualitycontext-architecture | Can the agent find out how your codebase works before it touches anything? A low score feels like an assistant asking what your README already answers. | 75/ 100 |
| Verification Surfaceeval-harness + security-hardening | Can the agent tell whether its own work is correct? A low score feels like “done” arriving in review as “not done”. This is usually the bottleneck, and until it moves, nothing else you fix will show up. | 14/ 100 |
| Convention Entropycontext-architecture | Does your codebase do the same thing the same way twice? A low score feels like every pull request arguing about style instead of behaviour. | 55/ 100 |
| Security Posturesecurity-hardening | What can the agent reach, and what stops it? Checks written for your repository, never a generic vulnerability scan wearing a new label. | 90/ 100 |
| Token Economymodel-routing + context-architecture | What does every request pay before it even starts? A context file that grew without a budget is a tax collected on every call, forever. | 80/ 100 |
| Dependency & Env Riskdrift-sentinel | Will the same run give the same result tomorrow? Without that, “works on my machine” turns into “worked on that run”. | 85/ 100 |
| Knowledge Capturecontext-architecture | Does a decision survive the person who made it? A low score feels like the agent repeating a mistake your team already fixed once, a year ago, in a thread. | 50/ 100 |
Which starts with the number we refuse to quote. Every vendor in this space has a percentage: “our suite makes the agent X% better”. X is nearly always a raw average across every model, task and attempt, and once you run that many comparisons, some of the wins are real and some are luck. Averaging them hides which is which. Here’s the method instead, in five steps.
Each task is a problem your team already solved: the repository is rewound to just before the fix, and the agent gets the problem as a prompt. Your own tests decide whether it succeeded, never the diff your engineer happened to write. Some of those fixes are public and may already sit in a model’s training data, so each task is flagged for that risk, and a held-out set is rewritten with different names.
Every task runs twice, across three models and at least five attempts each: once against your repository exactly as it arrived, agent context files and all, and once with the suite installed. Either the suite beats your real status quo or the report says it didn’t. Comparing against a deliberately weak starting point would make the number bigger and worthless.
A run passes when the failing tests pass and nothing that was already passing breaks. How closely its code resembles what your engineer wrote never counts, because a different correct answer is still correct.
They all share that one task’s difficulty, so a hundred runs are worth rather less than a hundred. Counting them as independent is how a method manufactures confidence in itself. So the method measures how much they overlap, discounts the sample size to match, and caps how far any one result can be pulled toward the average of the others.
Test enough combinations and a few will look good by pure chance. So a standard false-discovery correction runs across every claim the report might make, and a result is printed only if it survives that and its range of uncertainty stays clear of zero. The same run always produces the same figures to the last digit: there’s no randomness anywhere a claim gets made.
All of this makes our headline number smaller than one from a competitor willing to quote the raw average. That’s the point. A number that holds up under scrutiny is worth more than a bigger one that doesn’t.
There are six ways this method can still be wrong. Every one is published in full in our methodology paper, and printed here at the same size as everything else on the page. A limitation you have to go looking for is one somebody wanted you to miss.
The method combines two standard techniques that weren’t designed to be combined, and the guarantee that comes with one of them doesn’t strictly carry over. In technical terms: Benjamini–Hochberg applied to posterior tail probabilities is not the textbook setting, and under a shrinkage prior the procedure can under-control the false-discovery rate in exactly the regime we usually operate in. So the real rate is measured by running known null data through the actual pipeline, and printed as found rather than assumed.
Borrowing strength across results is what keeps a small sample from producing wild claims, but it also pulls each one toward the middle, so a strong result reads slightly weaker and a weak one slightly stronger. The raw pass rates sit next to the adjusted ones, so you can see both and judge for yourself.
There’s a cap on how far that nudging can go, and we picked the number, in the cautious direction: a cap that binds means wider uncertainty and fewer claims, never more. If it starts binding in practice, the fix is a properly re-derived model, and never a retuned constant.
How much repeated attempts at a task overlap is itself estimated from the data, and that estimate is then used as though it were a known fact. Its own error doesn’t flow through into the final intervals. At the sample sizes we typically have, this is a small effect. It’s still an approximation, and it’s ours, so we say so.
The same tasks run on both sides of the comparison, and we could exploit that to tighten our intervals. We don’t. Ignoring it makes every claim harder to make and none easier, and we’d rather a method fail toward silence than toward a headline.
A result has to pass the correction and have an uncertainty range clear of zero, which sounds like two independent checks. In practice the two nearly always agree, so the second adds little. The real protection is the calibration in the first disclosure. A belt-and-braces claim that turns out to be one belt should be called one belt.
We withdrew the previous version of this method ourselves. It marked all fifteen results of a test run “significant”: the exact signature of a method fooling itself, which this whole discipline exists to catch. Run through the current method, that same set is correctly flagged as too small to conclude anything, and nothing survives.
The test set is sealed before the check is written
The published numbers include the bad one. A check either catches a bad change or it doesn’t. Ours run against sets of deliberately broken code, written and sealed before the checks existed and never looked at while building them. Each rate carries the number of cases behind it, because “14.3%” out of seven is a number you should read with the seven in view.
28 of 34 held-out violations
2 false positives
4 of 9 held-out violations
1 false positive
1 of 7 held-out violations
0 false positives
The 14.3% stays on the page for the same reason the 82.4% does. A vendor who shows you only their best result has told you nothing about the one you need.
These cover only the kinds of mistake a script can recognise by pattern. After closing the gaps that round exposed, a fresh sealed set scores 16/16, 11/11 and 10/10 with no false alarms across the same three checks, and everything listed as out of scope is still missed, by design. Both sets of numbers are true. Neither proves your code is safe. The older figures stay above, and the better ones sit here, labelled.
Every delivered file is plain Markdown. You can read it, keep it, and hand it to a competitor. That’s a term of the licence. What doesn’t travel in a copied file is the part that decides anything: the scripts that run the checks, the test sets sealed before those checks were written, and the tamper-evident record each delivered file carries. Instructions are easy to copy. Proof isn’t.
The seven scores with the evidence behind each one, plus a fix list ordered by what moves the most.
Builds a benchmark out of your own git history, and re-runs it whenever you want to check.
Rewrites what the agent reads before it starts, so it reads less and gets it right more often.
Sends cheap work to cheap models, and escalates instead of shipping when a cheap model trips a check.
Security rules written for your repository, enforced by scripts, with a published catch rate.
Tells you when a new model release changes how any of this behaves, and what needs doing about it.
For trading, ledger, pricing and payments code: money held as whole units, no peeking at future data, and wrappers that stop rather than guess.
A tamper-evident record, signed by a named reviewer, on every change to a rule or check under a governed path.
Maps what’s installed to ISO 42001 or SOC 2, and proves every claimed control is a check that runs.
Keeps personal data out of logs and test fixtures, and flags data flows that shouldn’t exist.
You get a set of machine-readable files and one written assessment. Every number in the assessment traces back to a file you can open, and every file can be regenerated from scratch. Run it twice and you get identical output, byte for byte. If you ever don’t, that’s a defect, and it’s ours.
The scorecard, with each of the seven dimensions carrying the file, line and measured value behind its number.
The before-and-after result for each kind of task, with its range of uncertainty, the correction applied, and what survived it.
What passed and what failed on every single run, with tokens and time, in a standard format your own review team can read without a translator.
The written assessment as Markdown, Word and PDF, with a scope of work for the fixes, ordered by what moves your score the most.
A change record for each skill where every entry is fingerprinted against the one before it, so an edited history fails loudly.
Exactly what was delivered, at which version, on which date, with the licence grant and a fingerprint tying the files to you.
On retainer: within five business days of a new model shipping, a verdict on each installed skill and what, if anything, needs your attention.
Good documentation, almost no tests. The agent had plenty to read and almost nothing to check its own work against: 14 out of 100. So the fix list put that one dimension ahead of everything else, because until it moves, nothing else you improve shows up in the result.
Enough commit history to build seven real tasks from past fixes, features and migrations rather than invented ones. One flat documentation file covering the whole service scored 40 out of 100 and led the fix list. A simulated model release followed: the impact report ran unattended across all ten installed skills and found nothing needing action.
Neither case quotes an improvement figure, on purpose. Both ran offline, against a simulated agent. The task-finding and the structure around it are real, but our own method correctly flags about seven tasks as too small to conclude anything from. An earlier draft of Case B did report a +0.068 improvement as “significant”. That figure came from the method we later withdrew, and we deleted it rather than leave standing a number we could no longer defend. A real improvement figure comes from a live run against your repository.
Fixed scope, per repository. You know what it costs before you get on the call, so the call can be about your codebase instead of the price.
Your first repository: the seven scores with the evidence behind them, a before-and-after baseline, the written report, and a fix list ordered by what moves the most.
Send us a repository, or describe one if it can’t leave your network. You’ll get the scope, the price, and exactly what the report will contain, before anything is signed.
The catch rates above cover mistakes a script can recognise by pattern. Cross-function dataflow, semantic intent and anything needing a human reading are out of scope and missed on purpose. They’re listed by name so no headline can imply coverage we don’t have.
The instruction files and rulesets work across Claude Code, Codex CLI, Cursor and Gemini CLI. The enforcement (the checks wired into CI, the routing, the benchmark harness) is built for Claude Code first. Support for the others is on the roadmap and isn’t in the box yet, and this page won’t imply otherwise.
The checks are plain scripts. When a case is unclear they fail it rather than pass it. Models write code; scripts decide whether it’s right. This is the one rule with no exception, because the moment it has one, every number above becomes an opinion.