Reproducibility

docs.stripe.com · audited 2026-07-27 · AgentFit v3.4.0 · rubric v4 · Claude Opus 5

A score you cannot repeat is not a measurement. So here is the test we owe you: we audited the same site six times in one day — three times by asking a language model to apply a documentation rubric, three times by running AgentFit — and we published what we recorded, including the parts that are inconvenient for us.

Read the columns downwards, not across. The left rubric has 26 criteria; AgentFit has 28, and the two weight them differently — a machine-readable specification is worth 25 points on the left and 17 here. 57 is not "the right answer" and 78 is not "wrong". The claim on this page is narrower, and it is the only one the data supports: one of these two procedures returns the same number every time, and the other does not.

Where the runs disagreed

All three model runs did the work properly: they fetched live URLs, worked through the criteria one by one, and reported what they found. By our own count from the transcripts — which we did not publish — each run made between 36 and 47 tool calls, and none of them invented a fact. Take both of those as our reading of the runs, not as something the files below let you check. They still landed 15 points apart, and eleven of those points sit in a single category.

Two kinds of disagreement are mixed together there. The small ones — A5, B3, B5, B6, D1, D3, D6, a point here or there, in both directions — are what we would expect from three attentive readers of the same page: one found the SDK tabs in the DOM, the other two concluded they exist only inside a JavaScript payload. Whether that is ordinary measurement noise, three runs cannot tell you; that part is a guess. Only the D1 disagreement is visible in the published summaries — the rest comes from our own comparison of the three transcripts. The large disagreement is a different animal, and it is why this page exists.

The remaining criteria in category C scored exactly 10 points in each of the three runs. The entire eleven-point gap is two criteria.

C1a: the same facts, three different scores

Criterion C1a — from the model's 26-criterion rubric; AgentFit folds discovery and validity into a single C1 — asks whether the API has a machine-readable specification an agent can reach. All three runs established precisely the same two facts about docs.stripe.com:

• the conventional locations — /openapi.json and its siblings — return 404;
• a valid, complete OpenAPI specification is published in Stripe's GitHub repository.

Nobody was wrong. Nobody hallucinated. They scored it 0 of 8, 3 of 8 and 4 of 8.

The disagreement is not about the site. It is about a question the rubric never asked: does a specification count as published if it lives on somebody else's domain and takes one more hop to reach? That is a genuine fork with no obvious answer, and each run walked up to it alone, in the middle of a long task, with no rule to consult. Three runs, three policies.

From there it cascades. C1b scores the quality of the specification you found. The run that decided GitHub does not count had nothing to validate and scored 0 of 7. The two that fetched it scored 7 of 7 — and then reported a different number of paths, 414 and 136, because Stripe's repository holds more than one specification file and they picked different ones (spec3.json and spec3.yaml). Two runs agreed on the score for opposite reasons.

E5 — "are the terms of use stated?" — came out 3/3, 2/3 and 1/3. One run found a footer link to stripe.com/legal/ssa. One rejected that link and credited the Content-Signal directive in robots.txt instead. One looked for /terms, /legal and /privacy on the documentation host, got three 404s, and scored what it saw. Three defensible readings of one line of rubric.

The self-check varied too

The prompt ships with a calibration anchor: a table of previously measured sites on which Stripe sits at 56, measured 13 May 2026. It is there precisely to catch drift. That table is also why the site here is docs.stripe.com — Stripe was already in somebody else's calibration anchor before we ran anything, so the site was picked for us, and picked before the experiment.

Every run used it. Every run noticed its own deviation — +22, +24, +9. Every run then re-examined its criteria, reasoned about what might have changed since May, and concluded that the anchor was stale and its own number was the better one. Three times, independently, in the same direction, with three different numbers.

This is the part we find most instructive. It is not carelessness — it is what conscientious work looks like when the procedure still has a free parameter in it. A self-check whose verdict changes between runs is not a self-check.

The anchor happens to sit one point from AgentFit's 57. Read nothing into that: it is a hand measurement, on a different rubric, from a different month, and it is not evidence that 57 is the true score.

Why the code answers the same

Nothing clever. Every criterion is a fetch plus a rule, and both are written down before the audit starts.

D2 is the machine-learning one. Two small classifiers run inside the audit: one scores how realistic a code example is, the other judges endpoint-page completeness. They are ONNX graphs with frozen weights compiled into the binary. The same input gives the same tensor and the same float, every time. Determinism does not require avoiding models; it requires avoiding models whose output you cannot pin down.

C1 is the very fork the three runs walked differently, and AgentFit scores 0 of 8 on it. Our rule is not "GitHub does not count" — it is a fixed list of places to look: seventeen conventional paths on the host being audited, an RFC 9727 api-catalog link, and any specification link found on the homepage we fetched. On docs.stripe.com none of those paths returned a specification and that page carried no such link, so nothing was found. A rule that followed one more hop into a vendor's repository would not be less correct. Ours is simply fixed before the audit starts, applied identically to every site in the corpus, and its scoring published on the rubric page. That is the whole difference — not a better answer, a fixed one. How C1 is scored.

Check it against a live run

The repository is private and we publish no binaries, so we are not going to tell you to build it — you cannot. What is checkable without us is narrower, and we would rather state it exactly. The files above were written by a command-line build on our machine on 27 July 2026. The public service is a different build, running elsewhere, and it will hand you its own report for the same site. If the procedure is what we say it is, the two agree on every score. Here is how to make that comparison, and what it is worth.

Without running anything (no quota)

The public dataset carries the run id of the most recent valid audit of every host. Take Stripe's, read that run back from the API, reduce both documents to the scored fields, and diff them:

$ RUN=$(curl -s https://agentfit.dev/dataset.csv \
        | awk -F, '$1=="docs.stripe.com" && $3=="true" {print $7; exit}')

$ curl -s https://agentfit.dev/static/reproducibility/agentfit-run1.json \
  | jq -S '{total_score, max_score, categories,
            criteria: [.criteria[] | {id, status, score, max}]}' > ours.json

$ curl -s https://agentfit.dev/api/public/audit/$RUN \
  | jq -S '.report | {total_score, max_score, categories,
            criteria: [.criteria[] | {id, status, score, max}]}' > theirs.json

$ diff ours.json theirs.json

On 27 July 2026 that diff printed nothing: 200 lines, no output. Two different builds, two different machines, the same 28 verdicts.

Running it yourself (one per site per day)

Paste https://docs.stripe.com into the form on the home page. When it finishes, the address reads /r/<run id>. Put that id in place of $RUN and run the same two commands. Note what this does and does not add: the audit still leaves our servers, so what is yours is the request and the moment, not the vantage point.

One audit of the same site per network per 24 hours. A second attempt returns 429 with no run id — deliberately, so the service cannot be pointed at somebody else's documentation as a hammer. If you hit it, the path above needs no submission at all. The MCP server shares that quota: asking an agent to audit the same site again the same day meets the same wall. And there is a second wall in front of that one which is not yours at all: two audits of the same host inside ten minutes, from anybody, use up a shared burst, and the next caller gets 429 with code rate_limited rather than site_quota. That one clears by itself in ten minutes.

Without curl or jq

Open agentfit-run1.json in one browser tab and the run's report page, /r/<run id>, in another. Seven numbers decide it — the six category scores and the total. They are printed here so you can do it from this page alone.

Three things will differ, and should

• auditor — the build tag of whatever wrote the report. Ours says agentfit/v3.4.0 because that was the local build on the day; a fresh report says whatever is deployed now. It names the binary, not the measurement, which is precisely why the comparison above drops it.
• evidence_snippet — the words the site served to the machine that fetched them. Our three runs left an address Stripe answered in German; the service's run got English. On 27 July that was the entire difference: B1, B3 and E1 carried different text under an identical URL, an identical status code and an identical score. The snippet records the input, not the verdict.
• The order of the keys. Our files are what the tool wrote; the API returns the same object after a round trip through Postgres, which stores keys shortest-first. Byte-comparing the two documents is hopeless and would tell you nothing about the scores — jq -S puts both in one order, which is what makes the diff mean something.

A match means one thing: the same procedure, given the same site, returned the same verdict from a different build on a different machine. It does not mean 57 is correct — nothing on this page argues that. And it is conditional on the site: if docs.stripe.com has changed since 27 July 2026, the numbers move, and that is the instrument working rather than failing. Check the audited_on date in the dataset row before you conclude anything from a mismatch.

These files are frozen and we do not regenerate them at each release. Two reasons: the checksum on this page is taken over these exact bytes, and every number in the prose above was read out of them — a self-refreshing exhibit could carry neither. The tripwire is on the live side, not inside the frozen file: column 9 of the dataset row you read above is rubric_version, and for as long as this page stands it must read 4. The exhibit predates that column and does not carry it, so the number to check it against is the one printed in this sentence — we are stating it rather than letting you find it, which is a weaker guarantee and is why it is written out. If that column ever reads anything else, the ruler changed after this page was written, the field comparison is void, and what you are looking at is a historical record. We would rather the page say that out loud than quietly re-run itself into agreement.

Two of the 28 criteria, D2 and C3, are decided by small classifiers rather than by a rule. They are compiled into the binary and versioned with it — there is no model server, and nobody's retraining moves a score that has already been published.

What we are not claiming

• We are not claiming 57 is right. We do not publish a true score for docs.stripe.com, because nobody has one. We publish a procedure and its output. And reproducibility is a necessary property, not a sufficient one: echo 57 is perfectly reproducible too — what it is not is sensitive to the site. Whether ours is, /browse is the place to judge: the same procedure spreads real sites across the whole range.
• We are not comparing two numbers. Different rubrics: different criteria counts, different weights. The comparison is meaningful within each column, not between them.
• This is one site, one day, one model, three runs. It is a demonstration, not a study, and we are not promising to re-run it on a schedule.
• The model is Claude Opus 5 — the same model that helped build this page and a good part of this service. We are showing a variance we reproduced in the tool we use ourselves, every day; read it with that in mind. We are not saying language models are bad at this. We are saying we could not get one to return the same number twice.
• Determinism is a property of the procedure, not a promise about the internet. Our three runs left one machine on one day, and the site served German-language bodies to that address — visible in the B1 and E1 evidence, and one model run recorded the same thing. Change the vantage point and you change the input. Same input, same output is the entire guarantee.

What a model is genuinely good at here

None of this makes the model runs worthless — it makes them a different instrument. The diagnosis in each of the three runs is good on its own terms: why a missing specification hurts an agent, what to publish instead, which of Stripe's pages would survive being read without a browser. That is explanation, and a model explains beautifully.

A language model is excellent at explaining what to fix. It is a poor instrument for measuring whether you fixed it. We are about the second one. AgentFit is the instrument; take its number to a model and ask what to do about it. That is the right division of labour, and it is the one we use.

The data

All six runs are served from this site. The three AgentFit reports are the files the tool wrote, byte for byte — that is what the checksum is taken over. The three model files are ours:

The raw transcripts are not published. What you get for the model column are structured summaries we wrote afterwards from those transcripts: the final score, the category totals, and the criterion-level notes quoted above. The scores and totals are each run's own output; the wording of the notes is ours.

• three AgentFit reports and the checksum — run1.json · run2.json · run3.json · sha256.txt
• three model runs, as summaries we wrote from the transcripts — runA.json · runB.json · runC.json
• the prompt used for the model column, pinned at the commit we ran — ai-ready-audit

Nothing was filtered out — no run, and no site. Three model audits and three tool audits, on the one site the prompt's calibration table already named; we ran no others and discarded none. Had a fourth run come back at 57, we would have published that too, and this page would be shorter.

Audit your documentation →

Free, no sign-up, about 30 seconds. 28 criteria, HTTP evidence under every one, a shareable report link — and the same number tomorrow if nothing has changed.