13 min read

Break the cycle without breaking the system: Zero-Downtime Deployments

Written by

Daniela Duh

Published on

November 3, 2026

https://andamp-1.webflow.io/blog/zero-downtime-deployments

Every team has its version of the Thursday-night deploy: five services that have to ship together, in the right order, after hours, everyone on a call, hoping nothing breaks. If you’re unlucky, the safest bet is to put up the maintenance page and stop the world. We call it the release train, a lock-step deployment where everything leaves the station together or not at all. And the thing is, nobody designed it. It assembled itself out of individually reasonable decisions, until it became the only way that felt safe.

The argument of this whole post is this: lock-step deployments aren’t caused by your tools, your database, or your deployment pipeline. They’re caused by coupling: changes that have to go out together, or in a strict order, because one service can’t cope with another’s old or new version. Remove that coupling and the lock-step quietly dissolves, whatever stack you happen to use. So the real question isn’t how to make deployments smoother. It’s this: how do we make a change to one service without forcing anyone else to ship at the same time?

TL;DR

Zero-downtime deployments aren’t a single strategy — they are the natural conclusion of shipping without breaking changes.

Ask “in which order?”, never “together?” Every change to an event can be shipped as a sequence of independent deploys: consumers first, producers first, or both in turns.
Go up a level when a field isn’t enough. If a change can’t be made inside one schema, run a whole new event type or topic next to the old one. A v2 topic is just expand and contract at a bigger scale.
Don’t let the train derail over time, enforce these paradigms with schema checks and always document the clean up task with the new feature

Events outlive the code that wrote them

Why do we have this problem at all? Because we chose microservices, and for good reasons. For us the main one is ownership: every team owns its services, its data, and its release schedule. Sometimes regulations even require certain data to stay inside one service’s boundary. Either way, the promise is independent deployability: a team ships when it’s ready. A lock-step deployment is the exact opposite.

Our services talk asynchronously over Kafka. They publish events, and whoever cares consumes them in their own time. Kafka only guarantees ordering per key, and events stick around after they’ve been read, for a retention period or indefinitely.

That means an event has to be understood long after it was written. During a rolling update, old and new instances read the same topic. A consumer added next year may start reading from the very beginning. And many services don’t just react to events, they use them to keep their own copy of data another service owns, instead of calling that service at runtime. For them, the events on that topic aren’t history. They’re the current state of the world, and they stay relevant for as long as the data does.

That’s the first root of the problem: events outlive the code that wrote them. Consumers read events from producer versions that have already been retired, so every event format becomes a contract between teams that deploy on their own schedules.

Two kinds of “zero-downtime”

Search “zero-downtime deployment” and you’ll get rolling updates, canary, blue-green. Every one of them answers the same narrow question: how do I swap running instances without dropping traffic?

The question they leave open is our second root: can the old and new version coexist while both are live? Every one of these strategies, by design, creates a moment where they have to. During a rolling update, both builds serve requests and read and write the same database and event streams. Orchestration doesn’t solve the hard problem. It creates it.

Press enter or click to view image in full size

Picture a flawless rolling deploy: every health check green, no request dropped. But the new version publishes a parcel’s weight in kilograms instead of grams. Same field, same type, so every schema check waves it through. For a few minutes the topic holds a mix: 2500 from old instances, 2.5 from new. Nothing crashes or even warns. Downstream, shipping reads 2.5 as grams and charges almost nothing for a 2.5-kilo parcel. Nobody notices until the invoices arrive.

If a change touches nothing shared, orchestration is all you need. This post is about everything else.

How coupling turns into downtime

Coupling comes in two flavors: services that have to ship together, and services that have to ship in order, A strictly before B. Either way, someone has to coordinate. And once you’re coordinating anyway, batching starts to feel safer. Why not release everything as a set?

But a bigger batch is a bigger blast radius. More changes go out at once, more can go wrong, and when something does, it’s harder to tell which change caused it. That breeds fear, and fear justifies more batching. Deployments get rarer, bigger, and scarier.

Then, sooner or later, one of them goes sideways in a way nobody saw coming. For a few minutes old and new versions ran side by side, and in that overlap something broke. The post-mortem draws what seems like the obvious conclusion: the overlap was the problem, so get rid of it. Put up the maintenance page, scale everything down, deploy, bring it all back up.

And it works. That’s the trap. Stopping the world doesn’t remove the coupling, it hides it. The services depend on each other exactly as much as before, but now every deployment is planned downtime. A coordination problem has become an availability problem. Zero downtime isn’t lost to bad tooling. It’s the price of coupling nobody removed. So how do we break the cycle without breaking the system?

Expand and contract, and the habits that make it work

Expand and contract, also known as parallel change, is an idea that goes back at least to Scott J. Ambler and Pramod J. Sadalage’s Refactoring Databases (2006), where old and new schema run side by side during a “transition period”. Never make a breaking change in one step. Make it in three:

Expand: add the new thing next to the old one. A new field, a new event, a new topic. Everybody who doesn’t know about it yet keeps working, as long as they’re tolerant readers (more on that below).
Migrate: move everyone over. That’s mostly code: producers start filling the new thing, consumers start reading it. Sometimes it’s data too: if existing records also need the new thing, they have to be backfilled.
Contract: remove the old thing once nobody writes or reads it anymore.

Each step is safe on its own, ships on its own, and can be rolled back on its own. The breaking change never actually happens; you walk around it. Expand and migrate can often go out in the same release. Contract almost never can, because it has to wait until the old thing is really gone.

Tolerant readers. Make your consumers ignore fields they don’t recognize instead of blowing up on them. Then adding a new field can’t break anyone who hasn’t upgraded yet; they simply skip what they don’t know. The catch: it only rescues added fields. Removing or renaming something is still a real break.

Services that expect mess. Rollouts are messy, so build for it. Consuming a message twice must have the same effect as consuming it once, because Kafka delivers at least once and a restart mid-batch means duplicates. Don’t lose messages when an instance restarts mid-publish: the outbox pattern and committing offsets only after processing take care of that. And send a genuinely unreadable message to a dead letter topic instead of getting stuck on it.

The discipline

How do you stop someone from shipping a breaking change by accident? Start by asking one question for every change to an event: who has to deploy first?

Hold on, wasn’t the whole point that nobody has to? Almost. The goal isn’t that order never matters. It’s that no two deploys ever have to happen at the same time. “Consumers first” still means each service deploys alone, on its own schedule. The producer just waits until the consumers are done. That’s a dependency in the backlog, not during deployment.

What about services that are both? Say a service consumes topic A and produces topic B, and a feature changes both. Treat every topic as its own contract: the service can be consumer-first on A and producer-first on B, in two independent steps.

A schema registry turns that question into a gate. Every topic gets a compatibility mode, and the modes are simply the answers with names: backward means consumers deploy first, forward means producers deploy first, full means any order. Every new schema version is checked against the registered ones, and with that check in your pipeline, an incompatible change never deploys. Just know its limits: it compares schemas with schemas. It doesn’t know who consumes a topic or whether anyone still reads a field, and it can’t see meaning. The kilograms-for-grams change sails right through.

Backward is the usual default, because it guarantees consumers can always read everything already on the topic; forward suits a producer with many consumers that can’t all upgrade first.

A registry is also one more service to run and maintain. For a small, technologically uniform system, you can build the same gate without one: keep a schema of every event in the repository, compare it against the last released version in CI, and keep real sample events from older versions as test fixtures that every consumer must still parse. If you share event classes through a library, the library version is your schema version, and it follows exactly the same rules.

The cheat sheet

Most changes you’ll ever make fall into a short list. None of them needs a lock-step deployment. Each one is a sequence of deploys, and every deploy in the sequence is safe on its own, as long as it happens in the right order.

Any order

Add an optional field (with a default). Old consumers skip it, new ones fall back to the default.

Consumers → Producers

Add a new topic or event type on a shared topic. Consumers learn to handle or skip the new type before it shows up.
Make a field optional. Consumers handle a missing value before producers stop sending it.
Remove a field. Consumers stop reading it, then producers stop writing it. Nothing can tell you automatically whether a consumer still uses it: search the code, ask the teams, check your contract tests.

Producers → wait → Consumers

Add a required field, or make an optional one required. Producers always send it. Once no message without it is left, consumers start relying on it.

Producers → Consumers → Producers

Rename a field. Producers send both the old and the new field. Consumers switch to the new one. Producers drop the old one.
Change what a field means without changing its shape. Treat it as a rename: a new field with the new meaning.
Split or merge event types. Producers publish the new types next to the old one. Consumers switch. Producers stop publishing the old one.
Change the key or the number of partitions. Producers publish to a new topic with the new key, next to the old one. Consumers switch. Producers stop publishing to the old topic, and it’s retired.

Same change, different data

The cheat sheet makes breaking changes look almost easy. And often it really is. But the same change can play out very differently depending on what the event is used for.

Some events trigger logic: something happened, and consumers react to it. An adoption was completed, so send a care packet. Once handled, they scroll off into the past. Other events replicate data: “this is what pet 42 looks like now.” Consumers use them to keep their own copy of someone else’s data. Martin Fowler calls the first kind event notification and the second event-carried state transfer. Same Kafka, same serializer, very different rules.

To see why, let’s visit the good old pet store. We have three services:

Inventory owns the list of pets and publishes every change to a pet topic.
Checkout publishes an AdoptionCompleted event whenever someone takes a pet home.
Care sends every new owner a care packet. It keeps its own copy of all pets, built from Inventory’s topic, and looks up the adopted pet whenever an adoption comes in.

Now let’s take one change, add a required field, and apply it to both topics.

Events that trigger logic

Say the care packet now needs to know whether the adoption happened in-store or online, so Care can mail a welcome kit to online adopters. Call the field adoptionChannel. It’s genuinely required, since the new logic can’t run without it, and there’s no way to work it out from anything already in the event.

You can’t just add it as required: during the rollout, old and new Checkout instances are live side by side, and the old ones don’t send it. So you pass through a soft middle step:

Checkout starts sending the field. Care instances that haven’t upgraded ignore it; nothing breaks.
Care upgrades to read it, keeping a fallback for the occasional older event that predates it.
Once every old event has been consumed, the field is effectively required, and Care can drop the fallback.

The key thing: nobody re-reads old adoption events. So, “Required” arrives on its own, just by waiting. Two deploys, no coordination, no window.

Events that replicate data

Now say the care packet should also remind new owners when the pet’s next vaccination is due. Call the field nextVaccinationDue. Unlike the adoption channel, this isn’t a fact about a single moment. It belongs to the pet, it changes every time the vet sees them, and only Inventory knows it. So it lives on the pet topic, the one Care builds its copy from.

Same change, same play. Except step 3 never comes. A topic that replicates data has to hold the current record of every pet, or a new consumer would never learn about the ones that haven’t changed in a while. That’s why it’s compacted: Kafka keeps the latest record per pet, indefinitely. So a pet that has sat in the back room for a year still carries its year-old record without nextVaccinationDue. Whether Care keeps its copy in a database or rebuilds it from the topic makes no difference: the untouched pets have the same gap either way, and nothing Care does can fill it. You can’t wait for the old records to drain. They never drain, because they aren’t history. They’re the current state.

Sending it again

So Inventory has to send every pet again, this time with nextVaccinationDue. There are two ways to do that:

Republish into the same topic. Each pet’s old record gets overwritten by a complete one. Care tolerates the missing field until the last pet is through, then drops the fallback.
Publish into a new v2 topic where the field is required from day one. Inventory fills v2 with every pet and keeps it up to date next to v1. Care builds a fresh copy from v2, switches over once it has caught up, and v1 is retired once nobody reads it anymore. More work, but no consumer ever sees a half-migrated topic.

Either way, watch out for two things.

First, the republish races against live updates: if a vet visit lands after the job has read a pet but before its record hits the topic, the stale record wins. Send the republish through the same path as live changes.

Second, and this is where the distinction above pays off: a republish sends an update for every single pet. Any consumer that reacts to pet updates with business logic, say by emailing the owner whenever something changes, will fire for all of them at once. Events that replicate data should do exactly that and nothing more. Keep the logic on the events that trigger it.

The v2 topic is expand and contract one level up: instead of a field, a whole topic runs in parallel. The same works for event types, PetUpdatedV2 next to PetUpdated, and in the literature all of this goes by the umbrella term event versioning.

The hard part isn’t technical

By now the technical side is mostly solved. The cheat sheet covers the everyday changes, a new event type or topic covers the rest. If zero-downtime deployments were only an engineering problem, the post would end here.

But parallel change asks something of a team: a change that could ship in one deploy now takes three. On a two-week release cycle, a single required field can stretch across a month. Somebody has to explain that to a stakeholder who asked for a small feature, and “we’re in the migrate step” won’t satisfy someone with a deadline. The shortcut is right there: ship it all at once, ten minutes of maintenance page tonight. Just this once.

Then there’s the contract step, which delivers nothing anyone asked for. It slides from sprint to sprint while teams change, until someone finds a topic called pets-v1 that nobody reads but nobody dares to delete. Every unfinished parallel change is a piece of coupling waiting for someone to forget why it exists.

So here are a few things that help foster cultural change:

Make the rules non-negotiable, not heroic. A registry check in CI doesn’t argue with deadlines, so nobody has to win that discussion under pressure.
Plan the contract on the day you expand. Create the follow-up ticket right away, so it survives sprint planning and team changes.
Talk about risk, not steps. Stakeholders don’t care about producers and consumers. They do care that a feature can ship on a Tuesday afternoon without an outage or a rollback plan.
Write the conventions down. Put them in a team guideline, an ADR, or a link in the PR template. Habits only survive turnover if they live outside people’s heads.

The maintenance page rarely comes back because a change truly needed it. It comes back because, on one particular Thursday, defending the slower path felt harder than stopping the world.

Decouple the deploys, not just the services

Lock-step deployments don’t come from bad tooling or careless teams. They come from coupling, and in an event-driven system it hides in events that outlive the code that wrote them. The way out is one idea at different scales: expand and contract a field, an event type, or a topic, with a schema registry making sure nobody skips a step.

What’s left is the part no tool can enforce: the patience to take three deploys instead of one, the discipline to finish the contract step, and the willingness to defend both when a deadline is breathing down your neck.

It’s worth it. The reward is being able to deploy any service, in the middle of an ordinary working day, without a call, without a checklist, and without that knot in your stomach. No more Thursday nights.