Usually the model isn’t the problem. Your repository is: it gives your agent nothing to check its own work against, so “done” is a guess.
$7,500, first repository. Fixed scope, published price.
Agents don’t ask. They guess. A new engineer who can’t tell whether a change is right will ask someone. Your agent won’t. It reads your documentation literally, works at machine speed, holds your credentials, and has exactly one way to find out whether it succeeded: whatever your repository can prove.
So most of what gets reported as “the model keeps getting this wrong” is neither a model problem nor a prompting problem. The agent simply has no way to check itself. In one repository we audited, there was plenty for the agent to read — 75 out of 100 — and almost nothing to check against: 14 out of 100. Better prompts do not move that. Neither does a better model. The ceiling is set by what your repo can verify, not by what it can explain.
A lock you can’t open isn’t the same as a lock you can’t inspect.
Which is the useful part. The fix is ordinary engineering with a schedule attached: tests on the paths that matter, checks wired into CI, the conventions your team already follows written down once. Waiting for the next model release will not help. It has never seen your codebase.
Verification Surface 14 / 100 — the bottleneck
Every score points at something you can open: a file, a line, and the measured value behind it. Nothing here is scored by opinion, and nothing is scored by a model. These are the real numbers from a repository we audited, not an illustration.
| Dimension | What it asks | Score |
|---|---|---|
| Context Qualitycontext-architecture | Can the agent find out how your codebase works before it touches anything? A low score feels like an assistant asking what your README already answers. | 75/ 100 |
| Verification Surfaceeval-harness + security-hardening | Can the agent tell whether its own work is correct? A low score feels like “done” arriving in review as “not done”. This is usually the bottleneck — and until it moves, nothing else you fix will show up. | 14/ 100 |
| Convention Entropycontext-architecture | Does your codebase do the same thing the same way twice? A low score feels like every pull request arguing about style instead of behaviour. | 55/ 100 |
| Security Posturesecurity-hardening | What can the agent reach, and what stops it? Checks written for your repository — not a generic vulnerability scan wearing a new label. | 90/ 100 |
| Token Economymodel-routing + context-architecture | What does every request pay before it even starts? A context file that grew without a budget is a tax collected on every call, forever. | 80/ 100 |
| Dependency & Env Riskdrift-sentinel | Will the same run give the same result tomorrow? Without that, “works on my machine” quietly becomes “worked on that run”. | 85/ 100 |
| Knowledge Capturecontext-architecture | Does a decision survive the person who made it? A low score feels like the agent repeating a mistake your team already fixed once, a year ago, in a thread. | 50/ 100 |
Which starts with the number we refuse to quote. Every vendor in this space has a percentage: “our suite makes the agent X% better”. X is nearly always a raw average across every model, task and attempt — and once you run that many comparisons, some of the wins are real and some are luck. Averaging them together hides which is which. Here is what we do instead, in five steps.
We find problems your team already solved, rewind the repository to just before each fix, and hand the agent the problem as a prompt. Your own tests decide whether it succeeded — we never grade it against the diff your engineer happened to write. Some of those fixes are public and may already sit in a model’s training data, so each task is flagged for that risk, and a held-out set is rewritten with different names.
Every task runs twice — once against your repository exactly as we received it, agent context files and all, and once with our suite installed — across three models and at least five attempts each. We either beat your real status quo or we report that we did not. Comparing against a deliberately weak starting point would make the number bigger and worthless.
A run passes when the failing tests pass and nothing that was already passing breaks. We never score the agent on how closely its code resembles what your engineer wrote, because a different correct answer is still correct.
They all share that one task’s difficulty, so a hundred runs are worth rather less than a hundred. Counting them as independent is how a method manufactures confidence in itself. We measure how much they overlap, discount the sample size to match, and cap how far any one result can be pulled toward the average of the others.
Test enough combinations and a few will look good by pure chance. So we apply a standard false-discovery correction across every claim we might make, and a result is reported only if it survives that and its range of uncertainty stays clear of zero. The same run always produces the same figures to the last digit — there is no randomness anywhere a claim gets made.
All of this makes our headline number smaller than one from a competitor willing to quote the raw average. That is the point. A number that holds up under scrutiny is worth more than a bigger one that does not.
There are six ways this method can still be wrong. Every one is published in full in our methodology paper, and printed here at the same size as everything else on the page. A limitation you have to go looking for is a limitation somebody wanted you to miss.
We combine two standard techniques that were not designed to be combined, and the guarantee that comes with one of them does not strictly carry over. In technical terms: Benjamini–Hochberg applied to posterior tail probabilities is not the textbook setting, and under a shrinkage prior the procedure can under-control the false-discovery rate in exactly the regime we usually operate in. So we measure the real rate by running known null data through the actual pipeline, and publish what we find rather than assume.
Borrowing strength across results is what keeps a small sample from producing wild claims, but it also pulls each one toward the middle — so a genuinely strong result reads slightly weaker, and a weak one slightly stronger. We print the raw pass rates next to the adjusted ones so you can see both and judge for yourself.
We cap how far that nudging can go. The number we picked is a judgement call, made in the cautious direction — a cap that binds means wider uncertainty and fewer claims, never more. If it starts binding in practice the fix is a properly re-derived model, not a quietly retuned constant.
How much repeated attempts at a task overlap is itself estimated from the data, and we then use that estimate as though it were a known fact. Its own error does not flow through into the final intervals. At the sample sizes we typically have this is a small effect — but it is an approximation, and it is ours, so we say so.
The same tasks run on both sides of the comparison, and we could exploit that to tighten our intervals. We do not. Ignoring it makes every claim harder to make and none easier, and we would rather a method fail toward silence than toward a headline.
We require a result to pass the correction and to have an uncertainty range clear of zero, which sounds like two independent checks. In practice the two nearly always agree, so the second adds little. The real protection is the calibration in the first disclosure. A belt-and-braces claim that is really one belt should be described as one belt.
We withdrew the previous version of this method ourselves. It marked all fifteen results of a test run “significant” — the exact signature of a method fooling itself that this whole discipline exists to catch. Run through the current method, that same set is correctly flagged as too small to conclude anything, and nothing survives.
The test set is sealed before the check is written
The numbers we publish include the bad one. A check either catches a bad change or it does not. We measure ours against sets of deliberately broken code, written and sealed before the checks existed and never looked at while building them. We print how many cases each rate came from, because “14.3%” out of seven is a number you should read with the seven in view.
28 of 34 held-out violations
2 false positives
4 of 9 held-out violations
1 false positive
1 of 7 held-out violations
0 false positives
We publish the 14.3% for the same reason we publish the 82.4%. A vendor who shows you only their best result has told you nothing about the one you actually need.
These cover only the kinds of mistake a script can recognise by pattern. After closing the gaps that round exposed, a fresh sealed set scores 16/16, 11/11 and 10/10 with no false alarms across the same three checks — and everything we listed as out of scope is still missed, by design. Both sets of numbers are true. Neither proves your code is safe. We keep the older figures above rather than quietly swapping in the better ones.
Every file we deliver is plain Markdown. You can read it, keep it, and hand it to a competitor — that is a term of the licence, not an oversight. What does not travel in a copied file is the part that decides anything: the scripts that run the checks, the test sets sealed before those checks were written, and the tamper-evident record each delivered file carries. Instructions are easy to copy. Proof is not.
The seven scores with the evidence behind each one, plus a fix list ordered by what moves the most.
Builds a benchmark out of your own git history, and re-runs it whenever you want to check.
Rewrites what the agent reads before it starts, so it reads less and gets it right more often.
Sends cheap work to cheap models — and escalates instead of shipping when a cheap model trips a check.
Security rules written for your repository, enforced by scripts, with a published catch rate.
Tells you when a new model release changes how any of this behaves, and what needs doing about it.
Money held as whole units, no peeking at future data, and wrappers that stop rather than guess — for trading, ledger, pricing and payments code.
A tamper-evident record, signed by a named reviewer, on every change to a rule or check under a governed path.
Maps what is installed to ISO 42001 or SOC 2, and proves every claimed control is a running check, not a policy document.
Keeps personal data out of logs and test fixtures, and flags data flows that should not exist.
You get a set of machine-readable files and one written assessment. Every number in the assessment traces back to a file you can open, and every file can be regenerated from scratch. Run it twice and you get identical output, byte for byte. If you ever do not, that is a defect, and it is ours.
The scorecard, with each of the seven dimensions carrying the file, line and measured value behind its number.
The before-and-after result for each kind of task, with its range of uncertainty, the correction applied, and what survived it.
What passed and what failed on every single run, with tokens and time, in a standard format your own review team can read without a translator.
The written assessment as Markdown, Word and PDF, with a scope of work for the fixes, ordered by what moves your score the most.
A change record for each skill where every entry is fingerprinted against the one before it, so an edited history fails loudly instead of quietly.
Exactly what was delivered, at which version, on which date — with the licence grant and a fingerprint tying the files to you.
On retainer: within five business days of a new model shipping, a verdict on each thing we installed and what, if anything, needs your attention.
Good documentation, almost no tests. The agent had plenty to read and almost nothing to check its own work against — 14 out of 100. So the fix list put that one dimension ahead of everything else, because until it moves, nothing else you improve will show up in the result.
Enough commit history to build seven real tasks out of actual fixes, features and migrations rather than invented ones. One flat documentation file covering the whole service scored 40 out of 100 and led the fix list. We then simulated a model release: the impact report ran unattended across all ten installed skills and found nothing needing action.
What neither case study claims. Neither of these quotes an improvement figure, and that is deliberate. Both ran offline, against a simulated agent. The task-finding and the structure around it are real, but about seven tasks is correctly flagged by our own method as too small to conclude anything from. An earlier draft of Case B did report a +0.068 improvement as “significant”. That figure came from the method we later withdrew, and we deleted it rather than leave a number standing we could no longer defend. A real improvement figure comes from a live run against your repository — not from an illustration on ours.
Fixed scope, per repository. You know what it costs before you get on the call, so means the call can be about your codebase instead of about the price.
Your first repository: the seven scores with the evidence behind them, a before-and-after baseline, the written report, and a fix list ordered by what moves the most.
Send us a repository — or just describe one, if it can’t leave your network. You’ll get the scope, the price, and exactly what the report will contain, before anything is signed.
The catch rates above cover mistakes a script can recognise by pattern. Cross-function dataflow, semantic intent and anything needing a human reading are out of scope and missed on purpose — we list them by name rather than let a headline imply coverage we do not have.
The instruction files and rulesets work across Claude Code, Codex CLI, Cursor and Gemini CLI. The enforcement — the checks wired into CI, the routing, the benchmark harness — is built for Claude Code first. Support for the others is on the roadmap, not in the box, and we are not going to imply otherwise.
The checks are plain scripts. When a case is unclear they fail it rather than pass it. Models write code; scripts decide whether it is right. This is the one rule with no exception, because the moment it has one, every number above becomes an opinion.