What independent review actually catches

What did we measure?

How many pull requests on this website’s own repository received a real blocker from an independent review before they were allowed to merge, and how many did not.

Why does this number matter?

Almost nobody publishes it. Teams say review catches real problems, and reviewers say it about themselves, but the actual rate is rarely written down anywhere a reader can check. Every change here is read by someone other than its author before it merges, and no author merges their own work. That leaves a record with nothing about a client in it, so we counted it.

Amended 2026-08-18. The sentence above has been corrected. It used to read that we run every change through an independent pass before it merges, which folded two separate things into one. One of them is that every change is read by someone other than its author before it merges and no author merges their own work, and that has held without exception. The other is a separate publication review, which runs before a change reaches this site and does not run on every one. That reviewer has never been calibrated, so nobody has measured what it catches and what it misses, and its verdicts inform a merge decision rather than clear a change on their own. There is now a written test for when that review may be skipped, and which way it came out is recorded against each change it is applied to, but it was only written down on 2026-08-18. By then three changes had already been let through without that review, on judgement alone, and two of those three would not pass the test as it now stands. We would rather say so than date the test earlier than it deserves, on a page whose whole argument is that a claim should be checkable rather than taken on trust.

Why can you not check the record yourselves?

Because it sits in a private repository, and we are keeping it private. The repository that holds this website’s code also holds our planning documents and our working notes, and the line between what we publish and what we keep is one we hold on purpose. Opening the record to readers would mean opening all of it.

So the sources at the foot of this page point into that private repository, and they will not open for you. We have kept them anyway, labelled for what they are, because they say exactly where each claim comes from even though you cannot follow them there.

That puts this entry in an awkward spot, and naming it is better than hiding it. A page arguing that claims should be checked rather than trusted is asking you to take a described record on trust. What we can do is describe each case concretely enough that you can judge it on what this page says, and that is what the rest of it tries to do.

What did we find?

Five closed pull requests. Two came back with a real blocker, something that had to be fixed before the work could merge at all. Three merged clean, and each of those three says so in its own words: each author’s pull request explicitly left the work open rather than merging on their own say-so, because merging your own work here is not allowed.

PR #39 is one of the two. It added the layer that describes this site to search and answer engines: the structured data and the machine-readable summary of each page. Review found two content defects in that layer’s wording, sentences that claimed a little more than the site could actually stand behind. Both were fixed by rewording, the pull request records each changed line before and after, and the change merged.

PR #63 is the other, and it built the machinery behind this very page. Review found two places where the page promised a reader more than it could deliver at the time. The page said every finding carries the date it was measured and the source it came from, but nothing yet failed the build when one did not. And the site’s navigation pointed at this section while it had nothing in it to read. Both were fixed before the merge: the promise became a check that breaks the build when it is broken, and the link came down until there was something here.

PR #32, which rebuilt the site’s design system and its Projects page, PR #48, which added a check that the mirrored copies of our working playbooks match their master versions, and PR #49, which fixed a version check inside the guard that keeps working documents out of the published site, are the three that merged clean, with no blocker recorded against any of them.

We are not going to turn that into a percentage. Five pull requests is not a rate, it is the start of one. We will keep counting and update this line as the number grows.

Amended 2026-08-18. The count above is as at 2026-08-17, the date it was measured, and it has not been re-run since. More pull requests have merged on this repository in the time between, enough that the population has more than doubled. We are not recounting it in this change: a recount means judging every new merge for whether an independent pass returned a real blocker, and some of them deliberately had no separate publication review at all, which changes what the number is counting rather than simply adding to it. Doing that carelessly on a page whose whole argument is honest measurement would be worse than saying what is true today, which is that the recount is owed and has not been done.

Does review only ever stop a merge?

No. Issue #50 is a case where it did not. Review of the two pull requests above, #48 and #49, surfaced two real gaps in a different control on this repository: an entry in a newly added check that could never actually fail, so it would show green whether or not it had checked anything, and a way of running both guards that nothing had ever tested. Neither was serious enough to hold up the work already in front of it, so neither blocked those merges. Both were filed as their own follow-up instead. PR #53 is that follow-up, and it is still open as we publish this, unreviewed itself.

Amended 2026-08-18. PR #53 was closed on 2026-08-18 without merging. Its blockers and every smaller finding raised against it were written into its closing comments first, so the analysis outlives the branch that carried it. Why it closed is the more useful part: the gap that mattered was never the one #53 set out to harden. A check running inside this repository can prove that a mirrored copy of one of our working playbooks matches the version we recorded for it, but proving that copy is still current means reaching the master it was copied from, and nothing running inside this repository can reach that.

Review does not only produce a stop sign on the change in front of it. Sometimes it produces a second, smaller piece of work that would otherwise never have been written.

What does this cost?

It is not free. Fixing the blockers on #39 and #63 meant new commits, a second run through every check, and both pieces of work sitting unmerged for longer than either author expected going in. A verdict that says “not yet” on something you believed was finished costs real time, and it costs a bit of your own confidence too, the second or third time it happens.

It costs you something too, and on this page more than most. Because the repository is private, you cannot open a single pull request we have described, so you are weighing our count against our own account of it. We have tried to make each case complete enough to judge from this page alone. That is still less than letting you read the record yourself, and we know it.

We think the cost is worth paying. We would think that regardless of what the count above says, so do not take our word for it. Weigh the cost against the count, not against our opinion of it.

What would change this?

More pull requests. If the next twenty all merge clean, that is real information, and we will publish it. If the current rate holds, that is real information too. Either way, the number is what updates. The reason for publishing it does not.