What makes a release defensible.
Published 7 October 2026
A large suite is not a confident release
Ask a team how good their testing is and the answer is often a number: how many tests, what percentage of coverage. These numbers are easy to collect and easy to improve, and they say surprisingly little about whether a given release is safe.
A suite can be large and still miss the failures that matter, because tests cluster where testing is easy, not where the risk is. It can be green and misleading, because some tests assert nothing meaningful. It can be red for reasons unrelated to the change, which teaches everyone to ignore red.
Confidence is a property of the whole system of checks: how they are placed, what they block and how people respond when they fail.
Start from risk, then choose the layers
We begin by asking what must not fail for the business: a payment, a data export, an integration with a partner, a regulatory report. Those behaviours get the strongest checks. Everything else gets proportionate checks.
For the layers, a familiar shape still works. Many fast unit and component checks give quick feedback on logic. A smaller number of integration and contract tests verify that parts agree with each other, which is where many production failures live. A handful of end-to-end journeys confirm that the most important paths work the way a user would see them.
The shape is a guide, not a rule. Some systems need more contract tests, others more exploratory testing, and the right balance follows the architecture and the risk.
Gates need a purpose and an owner
A quality gate is a rule that decides whether a change moves forward. Useful gates are few and specific: this check blocks a merge, that check blocks a release, this one only reports. Each has an owner who can explain why it exists and change it.
Gates without purpose accumulate. Every incident adds a check, nobody removes one, and the pipeline slows until people look for ways around it. A periodic review of the gates, asking which failure each one has actually prevented, keeps the system honest. A gate that has never failed for a good reason is a candidate for removal or redesign.
Flaky tests are defects
A test that fails intermittently without a code change is not a nuisance. It is a defect in the test or in the system, and it has a cost. Each rerun hides a signal and trains people to doubt the suite.
The practical response is to make flakiness visible and finite. Quarantine the unstable test so it stops blocking everyone, assign an owner, set a date by which it is fixed or deleted, and track how many are in quarantine. The goal is not zero. The goal is that a red result is believed.
Measure what changes decisions
Useful measures are the ones that change what the team does: how long feedback takes, how stable the pipeline is, how many defects are found after release and of what kind. We agree on a small set at the start of an engagement and review them honestly, including when they get worse.
Quality engineering is, in the end, the work of making a release decision defensible. If your team cannot explain why it trusts a green build, that is the place to begin.
