Test automation as a service, and when it stops paying

A suite we inherited took the better part of an hour to run and failed roughly one run in four, each time for a different reason. The team had a shared understanding that red meant nothing in particular, so they shipped anyway, which was the rational response. The suite had been costing money for about a year while providing an assurance nobody believed.

Automation earns its keep where a defect is expensive and the behaviour is stable. On an e-commerce platform we built a framework for, that meant registration, search, cart and payment, wired into the delivery pipeline so a break appeared during development rather than after release. It did not mean covering every screen. A first framework should target the areas where the cost of a defect is highest, and expand only where the failure history says it should.

That figure is real, and it is the easy half of the story. A suite is code. It rots like code, and it carries an operating cost that rarely appears in the original business case.

The suite has a running cost

Every test carries a maintenance obligation against something that changes: a selector, a fixture, a schema, an upstream service that is now rate limited. When a test fails for a reason unrelated to the defect it was written to catch, the failure costs an engineer an hour and returns nothing. Enough of those and the team stops reading failures at all. That is the point where automation goes negative, because the suite now costs money and supplies false assurance at the same time.

The rule we apply is quarantine with a deadline. A flaky test comes out of the blocking run immediately and gets a fixed window to be repaired. If nobody repairs it inside that window, it is deleted. A test that has sat in quarantine for two months is a deleted test that nobody had the nerve to delete.

Tests worth deleting

Deletion is a maintenance activity and it deserves the same care as writing. The candidates are reasonably consistent across projects.

• Tests that assert implementation detail. They fail on every refactor and have never caught a defect a user would have seen.
• Duplicate coverage. The same rule verified at unit, service and interface level, so one behaviour change breaks three tests and costs three fixes.
• Tests for features nobody uses. Check the usage logs before assuming a screen matters.
• Tests whose failure history contains nothing but environment problems. That history is sitting in the CI database and is worth reading before an argument about coverage.

What survives is the set worth paying for indefinitely: the money paths, the authentication paths, the calculations somebody would be sanctioned for getting wrong, and the small number of end-to-end journeys that prove the system has been assembled correctly. That set is smaller than most suites, and because it is smaller it runs fast enough that people wait for the result.

Test automation as a service has a natural end, and we say so at the start. The framework should finish in the hands of the client's own engineering and QA team, which is how we ran it for a high-growth startup whose manual QA gate had become the release bottleneck. A suite owned by a supplier while the code is owned by the client is a second codebase with a different owner, and every change to the first one costs a round trip through a contract.

We judge a suite by what the team does when it goes red at five o'clock on a Friday. If they open the failure, it is paying for itself. If they press re-run, it stopped paying some time ago and nobody wrote that down.