Ilya Silvestrov's blog - From 8 Hours to 3 Minutes:<br>Zero‑Downtime Smart‑Contract Migration on Canton

Posted on August 2, 2026

In my first year at Obsidian Systems, the most impactful project I delivered was a zero-downtime migration on the Canton Network: I reduced smart-contract migration downtime for our client from 8 hours to 3 minutes.

What made it possible wasn’t blockchain expertise—it was that zero-downtime data migration patterns are storage-agnostic. I had applied them to relational databases before; the challenge was adapting them to a ledger where records are immutable and every change needs explicit authorization from the relevant stakeholders.

Why migrations on a blockchain are slow

Canton Network is a privacy-enabled blockchain for regulated financial institutions. Smart contracts are written in Daml, a flavor of Haskell. Like any other blockchain, it maintains a ledger of immutable transactions hosted and validated by multiple nodes. Unlike most blockchains, each node only keeps a subset of the ledger it has business knowing about. If you build a real estate escrow service on Canton, only the nodes operated by the bank, the seller’s real-estate agency, and the buyer’s real-estate agency will know about those transactions. Competing agencies never see each other’s transactions, by design.

Blockchains don’t permit changes to smart contracts without explicit authorization by the involved parties. Where a database system would only require an alteration of the schema, a blockchain may require copying data from an old contract version to a new one. Consider a PromissoryNote:

PromissoryNote
{ noteId       = "FA585E9C-...FA894"
, payer        = bob
, payee        = alice
, amount       = USD 100
, maturityDate = Date 2027 February 25
}

Bob promised to pay Alice one hundred dollars on February 25, 2027. Both Alice and Bob are signatories on this contract, meaning authorization from both of them was required to create the record. You cannot just go and make an arbitrary change to this record. The contract may well allow Alice to transfer the note to another party—but there is no operation that lets Alice or anybody else change, say, the amount.

Now suppose we want to add a maturityTime field to existing PromissoryNote contracts. In a database we could just add a column with the default value 23:59:59.999—if I promised to pay my debt on February 25th, it’s reasonable that I have the entire day to keep my promise. On a blockchain, we need authorization from the stakeholders. So we create a special “upgrade” smart contract with an upgradePromissoryNote operation that

sets maturityTime to 23:59:59.999 while copying the old note into the new one, and
is authorized by all parties to be invoked by an upgrader tool.

The upgrader tool then finds every PromissoryNote and feeds each through the upgradePromissoryNote operation. These operations on contracts are called “choices” in Daml: stakeholders are presented with a list of actions they may choose to take.

The process is slow because every call to the authorized upgrade choice is treated as a regular invocation: it undergoes the same authentication and synchronization across the Canton Network and produces a fresh copy of the old contract. I observed this kind of migration running at about 100–200 contracts per second, so a migration of several million contracts can take 8 hours or more. For comparison, I’d estimate that a straightforward batched copy from one table to another through an SQL driver in Postgres (not via a COPY command) runs at roughly \(10^4\) records per second, about 100 times faster.

Since version 2.10, Canton’s Smart Contract Upgrade (SCU) feature may silently perform an upgrade-on-read whenever the new contract version is compatible with the previous one. The example above, however, is deliberately crafted to be backwards incompatible under SCU rules. It would be compatible if maturityTime were declared as Optional TimeOfDay rather than just TimeOfDay: adding an optional field permits automatic upgrade-on-read under SCU, whereas adding a non-optional field does not. Backwards-incompatible changes still require a real migration, and that’s the case this post is about.

The naïve approach was to run the upgrader tool during downtime. Reducing this 8+ hour downtime was the problem I was tasked with solving.

The options I weighed

Every zero-downtime approach to a backwards-incompatible migration boils down to one basic idea: replication from the old data schema to the new one has to happen in the background, while the system keeps serving traffic. The solutions fall into two broad strategies:

Shrink the downtime until it becomes irrelevant—small enough to fit inside your error budget or downtime window, or to hide behind write buffering.
Make the new backend version backwards-compatible with the old data schema, so the system can keep operating while old and new versions run side by side.

Within those two strategies, here are the concrete techniques for relational databases that I adapted to the blockchain.

Upgrade on read / Downgrade on write

This is an instance of the second strategy: old and new backend versions run side by side, so the migration never stops the system, even for an instant.

When handling a read request, the new backend loads both contract versions and, if the new-schema record is stale, upgrades the old-schema record on the fly.
When handling an update request, it writes to both contract versions at once, so the old backend keeps working exactly as before—every change is reflected in the old schema too.

In a relational database, deciding whether the new-schema copy is stale means building some kind of logical timestamp yourself—for example, a version column that you increment on each update. On a blockchain you get one for free: the transaction offset (or block number). If the new-schema record sits at a lower offset than the old-schema record, then the new copy is stale.

This works seamlessly when downgrades are lossless. Renaming a field is the friendly case: the downgrade maps the new name back to the old one, and no information is discarded. Adding a required field is not: the old schema has nowhere to put it, so the downgrade drops it. Consider the PromissoryNote example in Why migrations on a blockchain are slow section. Suppose Alice sets maturityTime to 10:00 through a new backend, and an old backend later handles any update to the same note. The old-schema copy is now the freshest one, so the next upgrade-on-read rebuilds the new-schema copy from it—and maturityTime silently reverts to the default. Whatever a downgrade loses is lost for good the moment an old backend touches the record.

The trick is to split the upgrade in two, so that each step has a lossless downgrade.

Instead of adding a required field to PromissoryNote, introduce a separate contract, PromissoryNoteExtension { noteId, maturityTime }, with the same signatories as the note. The expanded backend reads both contracts, treating a missing extension as the default 23:59:59.999, and writes maturityTime to the extension only. Old backends don’t know extensions exist and—this is the point—cannot touch them.

Notice that this first step is not an upgrade-on-read/downgrade-on-write migration at all yet. PromissoryNote itself is unchanged, so old and expanded backends read and write the very same contracts. There is nothing to keep in sync, a plain rolling deployment does it.

Once no old backends remain, move maturityTime into PromissoryNote itself. This is the real upgrade-on-read/downgrade-on-write migration, from the pair schema to the merged one—but now the downgrade is lossless: downgrading a merged note means splitting it back into a (PromissoryNote, PromissoryNoteExtension) pair, and the pair can represent everything the merged note can. An expanded backend updating either half of the pair merely makes the merged copy stale; the next upgrade-on-read rebuilds it from the pair with nothing lost.

So the one upgrade with a lossy downgrade becomes two upgrades with lossless ones: the first because old backends can’t reach the new data at all, the second because a split discards nothing. The price is three deployments instead of one:

Initial state: Old backend works with v1.PromissoryNote.
Deployment 1: Deploy the expanded backend alongside the old one, rolling it out at whatever pace canarying and cross-organizational coordination demand. It operates on the pair
```
( v1.PromissoryNote
  { noteId : UUID, ..}
, PromissoryNoteExtension
  { noteId : UUID
  , maturityTime : TimeOfDay
  }
)
```
Stop the last old-version backend. From now on, every write lands in the pair schema.

Deployment 2: Deploy the dual-schema backend alongside the expanded one, with upgrade-on-read and the now-lossless downgrade-on-write between

{ oldSchema =
    ( v1.PromissoryNote
      { noteId : UUID, ..}
    , PromissoryNoteExtension
      { noteId : UUID
      , maturityTime : TimeOfDay
      }
    )
, newSchema =
    v2.PromissoryNote 
    { ..
    , maturityTime : TimeOfDay
    }
}

The dual-schema backend reads both the old and the schema concurrently and upgrades from the old schema into the new schema if the old schema record is stale.

Keep the mixed-version deployment running for as long as necessary, then stop the last expanded backend.
Migrate: Once no updates are landing in the pair schema, replicate every remaining stale or not-yet-upgraded pair into merged contracts in the background, using the same staleness check and upgrade procedure described above.
Deployment 3: Remove the now-unneeded dual-schema code and deploy the final backend, which operates on
```
v2.PromissoryNote
{ ..
, maturityTime : TimeOfDay
}
```

This method requires three deployments instead of one, and the first two carry extra logic—reading through the extension in Deployment 1, juggling both schemas in Deployment 2—embedded at every call site that reads or writes the affected smart-contract types.

Because each newer backend stays backwards-compatible with the contract types of the previous one, adjacent versions coexist during every rollover without interrupting service. The rollover can last as long as needed, leaving room for canarying, testing, rollback, and slow cross-organizational coordination.

Change data capture

The previous option runs old and new backends together and does background replication after the last old instance is gone. You may invert that: replicate up front and keep the two schemas in sync, so the eventual flip from the old backend to the new one is fast enough that the downtime is irrelevant.

Change Data Capture (CDC) is the most natural way to achieve this when the data lives on a blockchain. The original CDC pattern for relational databases rests on building a second view of the data by traversing the database’s underlying append‑only log. But a blockchain is already an append‑only log. There is no need to install extra database modules or plugins to capture update events—every transaction is already there on the chain.

The release process goes as follows:

Perform an initial migration at some initial offset while the old backend instances are running. This replicates old smart contracts into new ones in the background, but no one is using the new smart contracts yet.
Traverse the log from the initial offset and migrate the result of any data modification encountered along the way.
Keep tailing the log and migrating new data, so the new schema stays in sync with the old one, net some replication lag.
Shut down the old backend instances when ready.
Wait an instant for the replication gap to close.
Start the new backend instances.

Mis-coordinating the shutdown/start steps (4–6) can cause problems. For example, starting the new backend too early can leave a contract with the same key written to both schemas by concurrent requests dispatched to old and new nodes. If the migration is built to overwrite duplicate data in the new schema, the data submitted through the new backend would be overwritten and lost.

Which option I chose and why

I considered and discarded a few other options, but these two were the hardest to choose between. Here is the main tension:

Upgrade on read / Downgrade on write requires
- maintaining custom code that works with multiple contract versions at every call site where the affected contract types are read from or written to the ledger, and
- managing that code across several deployments
Change data capture requires a coordinated shutdown/start across all running backend nodes that use the same smart contracts. Those nodes can be operated by independent organizations, which turns the cutover into a synchronized cross-organizational release.

I presented both options to the client and collected feedback. Every release of their system was expected to carry a non-SCU-compatible upgrade, and they were explicitly worried about the code-maintenance burden and the extra deployment cycles that the “Upgrade on read / Downgrade on write” option would impose. The coordinated release process was judged to be the lesser evil.

What I built

I built a Change Data Capture implementation on Canton that

traverses the transaction stream via the Ledger API for multiple users on different participants concurrently,
tracks the migration state of every contract in its own status database, so each contract is migrated exactly once,
relies on Canton to de-duplicate contract migrations left in a “replication attempted” state after an abrupt interruption—for instance, when the network connection drops and the command-submission result is never received, we don’t know whether the migration succeeded,
supports batching and pruning of transaction events for efficient processing.

In addition, I built a payload reconciliation tool that validates, for most migrations, that

each old-type smart contract before a migration has a matching new-type contract after it, and
the payloads of those contracts match.

Canton’s privacy model means a participant stores a transaction (or a transaction projection) only if it hosts a party authorized to see what’s in it. Every user on every participant gets its own view of the ledger—a stream of create and archive events for smart contracts, scoped to the parties associated with that user.

For my CDC implementation, the participant’s administrator creates an “upgrade user” that “can read as” any party hosted by that participant. The core of my design was an algorithm that traverses the transaction streams of “upgrade users” across all participants concurrently—batching and pruning—while preserving event order. Put formally: for any two events e1 and e2 in a single user’s transaction stream, if e1 comes before e2, the algorithm guarantees the migration for e1 runs before the migration for e2.

This ordering is essential. Replaying the log in the wrong order either corrupts the result or fails outright.

If the same contract is updated several times, those updates generate a sequence of events, and processing every one would be wasteful. When the algorithm detects multiple events for the same contract while traversing the stream, it ignores all but the last—it prunes them.

Likewise, submitting “upgrade” smart-contract calls one by one would be extremely inefficient, so the upgrade operations are invoked in batches.

Designing an algorithm that concurrently processes dozens of user transaction streams from different participants—batching and pruning, all while the system is online and accepting new updates—is a hard problem. I was fairly certain my implementation may have concurrency bugs that would show up infrequently, which is why I

built the reconciliation tool, and
added runtime consistency assertions to the migration tool.

The reconciliation tool acts like an integration test for the migration you just ran: it checks that every contract was migrated, and migrated correctly. If it fails, whether from a concurrency bug or anything else, the release engineer reverts the migration and starts over.

Results

Migration-related downtime dropped from 8+ hours to 3 minutes—from “schedule a maintenance weekend” to “fits comfortably in a regular Friday-night maintenance window.” Those 3 minutes could be cut further to a few seconds, and even that brief downtime could be limited to write operations only. But this wasn’t a priority: going from 8+ hours to 3 minutes was already good enough.

The downtime reduction is important. But what’s equally important is a change of process that gives peace of mind to the release engineer by moving the bulk of the migration outside of the downtime window. The release process before was as follows:

Stop the service,
Perform the migration,
Start the service.

If something goes wrong with the migration, you need to investigate and roll back during the downtime. This decision-making while the clock is ticking is a major source of stress. The migration was scheduled over the weekend because it took hours, and to leave some breathing room to make decisions in case things went sour. Somebody was tied to the computer during the weekend, pinging others during non-working hours if something went wrong.

The new process looks as follows:

Perform an initial migration at some initial offset one week in advance.
Take all the time you need to resolve problems. Roll back and re-migrate as many times as necessary—it doesn’t matter how long it takes, because the service keeps running while migration and rollback happen in the background.
Leave the migration tool running to keep the new schema in sync.
Perform the shutdown, wait for the replication gap to close, and start the new version in just 3 minutes on Friday night.
Enjoy your weekend with the family.

The reconciliation tool proved useful, but in a different way than I anticipated. It didn’t find bugs in the migration algorithm itself, but it did catch several bugs in per-release migration logic across a few releases.

Remember the upgradePromissoryNote choice that we discussed before? It’s custom code that an engineer has to write for a release. Imagine a bug that makes upgradePromissoryNote set the PromissoryNote { amount } field to 0: it would be a nasty surprise to discover that every PromissoryNote has a zero amount after the upgrade.

In one migration, an engineer set a smart-contract field taxClassificationOverride to None in the migration function, and it slipped through review. The reconciliation tool complained that a contract had different data after migration, and I confirmed this was indeed a bug in the per-release migration logic (not a migration tool bug). The tool also fires false positives—for example, a migration may legitimately delete smart contracts that are no longer used, leaving unmatched old contracts behind. The tool has no way of knowing that was intentional, so a human still has to adjudicate.

But the reconciliation tool never found a concurrency bug after migrating more than 100 million contracts total across multiple releases—the part of the implementation I was most concerned about turned out to hold.

Stepping back, the headline number is the 8 hours to 3 minutes. But the result I’m proudest of is the one that doesn’t show up in a benchmark: the migration moved from a tense, time-boxed event that someone babysat over a weekend to a routine, reversible task with a one-week runway. The downtime shrank because the hard work—replication and validation—now happens in the background while the system is online, where there’s time to get it right.

And none of the core design ideas are blockchain-native. The solution is a data-migration pattern I’d used on relational databases before, adapted to the Canton blockchain, where every change needs explicit stakeholder authorization and where, instead of a single append-only log, many transaction streams must be traversed in the correct order. The blockchain made the constraints sharper, but the shape of the answer is the same: do the migration in the background to shrink the downtime.

From 8 Hours to 3 Minutes:Zero‑Downtime Smart‑Contract Migration on Canton