Practice

The rule: a change that breaks one direction becomes three deploys, and the middle one is the one people skip.

Step 1 — deploy code that tolerates both shapes.

Add currency as an optional field with a default. New code writes it; new code reading an old record without it falls back to the default. Old instances still running ignore the field entirely, because it is additive.

At this point both directions hold, so the rolling deploy is safe. Nothing in the database has changed yet.

Step 2 — backfill the 40 million rows.

In batches, with a rate limit, resumable from where it stopped. Not one transaction — a 40-million-row update holds locks and generates enough WAL to cause its own incident.

Two questions have to be answered before this starts, and they are product questions rather than engineering ones: what currency do historical orders get, and how do you know? If the answer is “GBP, because we only operated in the UK until last month” that is fine, but it must be written down, because in a year someone will ask why every pre-2026 order says GBP.

Verify by counting: SELECT count(*) WHERE currency IS NULL should reach zero and stay there. The “stay there” part matters — if it stops falling, something is still writing rows without it, which means step 1 was incomplete.

Step 3 — make it required.

Only once the count is zero and has been zero for long enough that any lagging instance is gone. Now add the NOT NULL constraint and remove the default handling from the code, if you want to.

What makes this go wrong:

• Skipping step 2’s verification and going straight to the constraint. The migration fails on the first null row, usually with a lock held.
• Backfilling before step 1 is fully rolled out. Instances still running the old code keep writing rows without the field, so the backfill never converges.
• Deploying steps 1 and 3 together because they are both “the currency change” in the ticket. This is the single most common version of this incident.

Worth adding: if the field can be derived rather than backfilled — from the account’s country, say — consider computing it on read for old rows instead of rewriting 40 million of them. The cheapest migration is the one you do not run.

No — three different answers, because the constraints barely overlap.

Public API: JSON.

The consumers are other companies, using languages you do not control, debugging with curl. Readability is worth far more than bytes here, and every client already has a parser. The volume is low enough that size is irrelevant.

The cost you accept: no schema enforcement, so compatibility is by convention — add optional fields, ignore unknown ones, never reinterpret an existing field. Publish an OpenAPI spec so at least the contract is written down, and remember that a strict validator rejecting unknown fields quietly destroys forward compatibility.

Internal RPC: Protobuf, over gRPC.

Both ends are yours, so nobody needs to read the bytes. Volume is high enough that a threefold size reduction is real money in bandwidth and latency. Schemas are stable and hand-maintained, which suits tag numbers — a human assigns them and a human keeps them reserved.

The advantage over convention is that compatibility is enforced by the encoding rather than remembered by a person. Renames are free; unknown fields are skipped; the generated code is available in every language you use.

Event archive: Avro.

This is the one people get wrong by defaulting to Protobuf for both internal cases.

Avro writes no tags at all, only values, so it produces the smallest encoding of the three — which over seven years of events is a meaningful storage bill. The writer’s schema goes in the file header once and is amortised over millions of records, so the usual objection does not apply to large files.

More importantly, schema resolution is by name against a writer’s schema travelling with the data. That means the archive is self-describing: code written in 2033 can read a file from 2026 by being handed the schema that wrote it, without anyone having preserved a tag registry for seven years. With Protobuf you would be relying on reserved declarations surviving a decade of refactors — a bet on organisational memory rather than on a property of the format.

The pattern: text where humans and unknown clients are; tagged binary where both ends are yours and schemas are hand-maintained; writer’s-schema binary where data outlives the code and must stay self-describing.