Why we still write integration tests for ledgers

The first production ledger we shipped went live in 2016. Within a week, most of what the team knew about fixing mistakes had stopped applying. On an ordinary system a bad write is a support ticket: find the rows, correct them, note it in the change log, move on. On an append-only ledger the bad write is history. The only remedy available is another entry recording that the previous one was wrong.

That property is the reason anyone chooses a ledger. It also changes the economics of a defect. A correcting entry does not remove the original, so every future reader of the history encounters both, and every report that aggregates it has to know which one to believe. A bug written into an immutable log becomes a permanent feature of the dataset and of the explanation given to anyone who asks about it.

Unit tests catch the failures that are easiest to catch

Contract-level unit tests are cheap and worth having. They find arithmetic errors, access control mistakes and boundary conditions. What they cannot see is the behaviour of the whole path: a client signing a transaction, a node accepting it, an indexer building state from the events, a service reading that state, a report summing it. Nearly every serious ledger defect we have dealt with lived between two components rather than inside one.

The recurring shapes are familiar enough to list.

• A retry producing a second entry because the client could not tell whether the first one landed.
• A chain reorganisation reverting a transaction the application had already treated as confirmed.
• A value carried at eighteen decimal places on-chain and two in the reporting layer, rounded in the wrong direction.
• An event the indexer never sees, because it was emitted inside a sub-call that later reverted.

Every component in those examples behaves correctly on its own. That is precisely why the unit tests pass.

The tests that earn their keep

Integration tests run against a real node rather than a mock, and they exercise a full lifecycle. Submit. Wait for the confirmation depth the business actually requires. Read the state back through the same query path production uses. Then reconcile a derived report against the chain. The reconciliation step is the one teams drop when the schedule tightens, and it is the one that catches the class of defect producing a wrong number instead of an error. That class is the expensive one, because nothing alerts.

Failure paths get the same treatment. Submit a transaction that will be rejected. Run out of gas halfway through a batch. Send the same idempotency key twice. Restart the indexer from a block behind and confirm it converges on the same state. Each of these is tedious to write and tedious to maintain, and each corresponds to an incident that would otherwise be discovered by a customer.

The economics here are not peculiar to ledgers. Wiring an automation suite into the release pipeline for an e-commerce platform we worked with had the same effect on the same category of defect.

An audit answers a different question

We had the CRBT token contract audited as part of the Cruisebit build, and audits are worth their cost. An audit reads code for known vulnerability classes and logic errors. It does not run the client, the indexer or the reporting stack, and it holds no view on what the business intends the numbers to mean. A programme that funds the audit and skips the integration suite tends to find the gap in production, where the correcting entries are permanent.

All of this is heavy discipline for a system whose main appeal is that it only ever appends. Append-only is the reason it is warranted. A conventional database lets a team quietly fix yesterday. A ledger asks the team to be right the first time, and then keeps the receipt.