What a Gate Must Declare

A gate reports a verdict. Someone who did not run it has to decide whether to act on it, and a verdict nobody can audit is a rumour with an exit code.

Four things make it auditable. Each is here because leaving it out cost something real. Three further rules follow them, on reading a verdict rather than writing one.

Its denominator

Every verdict carries the number of things it examined.

Two runs of the same suite at the same commit reported 1,433 and 1,429 executed specs. The second was green. Four specs had never reported at all, and nothing in a green result distinguishes “everything passed” from “everything that ran passed, and four did not run”.

So a count that moves needs its explanation attached, not requested afterwards: a dropped count with no diff explaining it is not green. When the expected total legitimately changes — a removed test, a renamed class — the declaration cites the change: the path, and the git diff --stat that shows it. An unverified delta fails in both directions. It can hide a lost spec, and it can raise an alarm about a deletion someone did on purpose.

Its scope

A run that examined a subset says so, in the verdict.

A filtered run and a run that lost most of its cases produce the same shape of output: a smaller number, and green. The reader cannot tell them apart, and the person who applied the filter is usually not the person reading the result three days later.

A scoped run declares its own expected: n of the n targeted by this filter. A run that declares no scope is compared against the full suite, because that is the only safe default — the failure of a silent subset is that it looks like a success of the whole.

Two traps make this sharper than it sounds.

A filter can report rather than execute. Some runners apply a name filter to what they report and not to what they run, so a filtered green can sit on top of executed failures that the filter removed from the accounting. The filter changed the denominator without changing the work.

A filter that matches nothing passes. A test class named in a filter must be the class the file actually declares, not the file’s name. A file called ComponentStartupTest.kt containing a class named ComponentStartsTest matches nothing under a filter for the former — and a run of nothing is a run with no failures. Verify that the name in the filter is the name in the declaration; the file name is not evidence.

That it can fail

A gate whose only recorded output is a pass is indistinguishable from a gate that cannot fail.

Before a new gate, check, watch or conformance case is trusted, its first recorded run is against a case that must fail. Break the thing it guards, watch it go red, put the thing back. Where staging a real failure is genuinely expensive, the run states instead what would make it fail — a specific condition, not a reassurance.

This applies with particular force to watches, which fail quietly by nature. A watch on a review queue died because a || echo 0 fallback swallowed an interpreter error, and from outside, a broken watch and a quiet queue looked identical for twenty-five minutes. Three rules follow. A watch never swallows an error on its signal path — no || true, no 2>/dev/null on the line that carries the signal. A watch that has produced nothing for twice its interval reports itself as suspect rather than staying silent. And a new watch pattern is proved able to fail before it is trusted, exactly as a gate is.

Its subject

A verdict is about an artifact, and the verdict names which one.

A test result belongs to a deployed image digest, not to the branch someone believes was deployed. A readiness claim belongs to the commit the remote actually holds, not the local tip that was asserted from memory — two claims of readiness were once made against remote CI while the accepted tip was still unpushed, so the CI that passed had never seen the code being vouched for.

Both are the same rule as the one about refs elsewhere in these pages: a property measured at one artifact and reported as a property of a branch, a stack, or “the code” is a claim about something that has no state. Name the digest, name the SHA, and verify it where the reader would — for a push, on the remote.

A local test environment adds a second edge. Where one deployment target is shared, a run can be measuring somebody else’s deployment. Serialize the jobs, and have each verdict cite the digest it found deployed against the SHA it declared.

The shape underneath all four

These are four instruments of one rule, and the team that collected them said it best while fixing the fourth defect of the same kind in a single day: a check whose surface is narrower than the question answers a different question, quietly.

A count narrower than the suite. A filter narrower than the run. A history containing only passes. A subject that is a branch name rather than an artifact. In each case the check is real, it runs, it returns, and what it returns is not what anybody reads it as.

Reading a verdict, once it declares those four

The four above are what a verdict must say. The three that follow are what a reader must check before believing one — they are how a declaration that looks complete still misleads.

Its own detail, checked against its own total

A raw count and a per-item enumeration disagreeing inside a single output is the cheapest available signal that one of them is wrong.

The denominator rule above says a count needs its explanation attached when it moves between runs. This says a count needs checking against its own detail within one, and it costs nothing, because both numbers are usually already on the screen.

A process count read 4 while its own listing showed 3. The discrepancy went unnoticed and the right answer came out by luck of having enumerated at all. The entries dismissed as noise were the measuring pipeline’s own members, caught in the listing they were producing — the measurement stating its fault, in the operator’s terminal, free.

The rule then reproduced inside the experiment run to demonstrate it: enumeration 3, count 2, two invocations with different pipeline membership. That forecloses the reading that the first discrepancy was one unlucky command. It is a property of the method.

Self-capture has four sources, and the familiar bracket idiom removes one. A pattern existing nowhere on the machine except inside the searching pipeline matched three times: the shell wrapper whose command line is the whole text typed, a second subshell for the pipeline, and the matcher itself. A later stage whose program text contains the pattern is the fourth — an awk matching its own regex and printing it as data. The idiom addresses only the matcher, which is what it was built for, and it still removes a false match, so the output looks better while the dominant terms stand. Use a tool that excludes itself, and verify that it does.

Dated 2026-09: true of this harness and its aliases. grep here resolves to ugrep, so an idiom written for another engine is executed by this one — the same root as a lookahead silently rejected elsewhere, where a link check reported no missing links because it had matched nothing.

An aggregate cannot be checked, by anybody

An advisory check posted size warnings on three files, and the sentence a report reaches for is “all pre-existing”. Measured against the base, the three made three different claims: +19, +1, and −22 — a file the branch leaves shorter than the base.

A reader who accepts the blanket sentence and then opens the third finds the report was wrong, and wrong in the direction that flatters its author — the expensive direction, because it costs the report its credibility on everything else.

A per-item figure can be checked by anyone. An aggregate cannot be checked by anybody, including its author, which is why “all pre-existing” survives self-review: there is nothing in it to check, so checking it feels like it has been done.

A summary that cannot be disagreed with is not a measurement, and “unchanged”, “no regressions” and “all advisory” are the same shape. Give the per-item figures, or say plainly that you did not measure them.

The connection worth carrying: an aggregate over branches is the same error as a fix over branches, and the member that differs is what both miss. The prose version writes “all pre-existing” over three files, one of which shrank. The code version applies a fix to one member of a pair, or carries a disposition right about a shared collection and wrong about two callers keeping their own store. One error at two sizes.

A negative control for an instrument you suspect of measuring itself

A process search returning a count of two followed by an enumeration of one looks exactly like the tool matching its own command. The operator was one step from reporting the rule as broken.

The control was one command: search for a pattern that exists only inside the running command. It matched zero, so the tool excludes itself, and the discrepancy was real change between two samples.

Real change over time and instrument error are indistinguishable from a discrepancy alone.

This is the sharpest witness on this page for proving an instrument can be wrong before trusting it, because it caught a false alarm rather than a false pass. Every other instance here is an instrument failing to fail; this one is an instrument wrongly suspected, cleared in a second, where the suspicion would have cost a working rule.

A green suite says nothing about what it has stopped asking

The denominator rule covers a count that moves. This is the case where the count moves correctly and something is still wrong.

A library change made an illegal call unrepresentable rather than refused. Three test cases went away, one arrived, and the suite stayed green throughout.

Two of the three removals were the change doing its job: they asserted a runtime refusal of a path that no longer compiles, so they had no subject left, and the added case asserts the positive half. The third was a backward-compatibility assertion dropped rather than adapted when a type parameter changed a serializer’s arity. The behaviour it guarded still existed and was now unguarded. Nobody reported removing it; every gate was green.

Netting those to “minus two cases” would have been arithmetically true and would have hidden the regression completely. Name the cases, or the honest number does the concealing.

Two precisions the instance supplies. Executions and cases are different quantities — one case can produce several executions — so a delta computed across the two is not a delta at all. And the finding came from diffing case names between two refs, which is the check that answers this question; a suite’s own total cannot, because the total is what stayed green.

Sources

From the process record of the Orders v2 Phase-III work, notebooks/domain-ontology/streams/process-changelog.md, where each rule is one dated entry carrying the failure that produced it, who proposed it, and who adopted it. The denominator and scope rules are also written into that project’s acceptance contract.