Skip to main content

Disaster recovery

You now have two tools. A snapshot of ORDERS sits off-site under ./backups/orders/, and a live mirror, ORDERS_DR, runs at site2. Each protects against a different failure, and reaching for the wrong one during an outage costs you either data or hours.

This page is the runbook: an ordered procedure that names the failure, picks the right tool, and carries out the recovery against the real ORDERS deployment. It also covers the one operation the earlier pages set up but never performed: promotion, turning the read-only ORDERS_DR into a writable primary.

Match the failure to the tool​

A runbook starts before the outage. The decision you don't want to make under pressure during an outage is which tool. Decide it ahead of time, per failure class.

What happenedReach forWhy
The whole east cluster is gonepromote ORDERS_DRThe mirror already holds the data; promotion is minutes, a restore is hours.
Someone deleted or purged ORDERS by mistakerecover from ORDERS_DR, or restore the snapshotThe mirror survives an upstream delete: it keeps every message it had copied and just stops updating, so it's usually the freshest intact copy. The snapshot is the older fallback.
Messages on ORDERS are corrupt or wrong (a bad publisher)restore a known-good snapshot, or purge the bad rangeThe bad data replicated to the mirror as well. The snapshot predates the corruption.
A consumer lost its position (shipping redelivering from zero)restore a --consumers snapshotOnly the snapshot captured the consumer's saved delivery position.
Enough east servers are permanently gone that the meta group has no leaderrescue the meta group, then remove the dead peersNothing can be created, moved, or restored while the metadata layer has no quorum.

One principle runs through the whole table: a snapshot is the only copy that predates a mistake. A mirror follows the upstream's live writes, so a corrupt write replicates into it; it survives an upstream delete but only as a copy frozen at the break, never an earlier point you chose. That's why the chapter built both, and why R3 replication is on neither row: replicating a bad write three times doesn't undo it.

Failover: promote the mirror​

When the east cluster is gone, the goal is to make ORDERS_DR start accepting writes so order-svc and the consumers can carry on at site2. That's promotion, and it's a short, ordered sequence. The animation below shows it end to end.

Step 1 — verify the lag is zero​

A mirror trails its upstream by some lag: the count of messages it hasn't copied yet. Promote it while lag is non-zero and you start publishing on top of a stream that's still missing its tail. So the first runbook step is always the same: read the lag.

#!/bin/bash

# Before you promote ORDERS_DR, read how far it trails the upstream.
# A mirror is eventually consistent: it follows its upstream over the
# network, so at any instant it can be a few messages behind. The Lag
# field is that gap, in messages. Promote a mirror with non-zero lag
# and you publish on top of a stream that is still missing tail
# messages.

# Point at site2, where the ORDERS_DR mirror lives.
nats --server nats://127.0.0.1:5222 stream info ORDERS_DR

# Look at the Mirror block in the output:
#
#   Mirror Information:
#
#            Stream Name: ORDERS
#                    Lag: 0
#              Last Seen: 1.2s
#
# Lag: 0 means the mirror has caught up to every message the upstream
# had at last contact — it is safe to promote. A non-zero Lag means
# messages are still in flight; wait and re-run until it reaches 0.
#
# Last Seen is how long ago the mirror last heard from the upstream. If
# the upstream site is already gone, Last Seen climbs and Lag freezes at
# whatever it was when contact dropped — that frozen Lag is your data
# loss, your actual RPO for this failover.

If the east site is fully unreachable, the mirror can't reach its upstream and the lag stops at whatever it was when contact dropped. That stalled number is your real recovery point: the messages written to ORDERS after the last successful copy are lost. Note it, then proceed. Waiting for a lag that will never move to zero only extends the outage.

Step 2 — drop the mirror config​

A mirror is read-only by design: it rejects direct publishes because its only job is to follow its upstream. To make ORDERS_DR writable, you remove the mirror relationship from its configuration.

# At site2: edit ORDERS_DR so it is no longer a mirror.
# Removing the mirror source makes the stream a standalone primary.
nats --server nats://site2:4222 stream edit ORDERS_DR --no-mirror

Once the mirror config is gone, the stream stops following east. It still holds every message it had copied. Promotion doesn't touch the data, only the relationship.

Step 3 — clear the lost stream's assignment​

The subject bind in the next step fails if the old ORDERS assignment is still in the JetStream metadata. The server checks the subjects you're adding against every stream in the account, and losing the site doesn't remove the dead ORDERS on its own, so the edit comes back with subjects overlap with an existing stream (10065). Clear the stale entry first:

# Remove the assignment for the lost ORDERS so its subjects are free.
nats --server nats://site2:4222 stream rm ORDERS --force

This is the official promotion order: free the mirrored stream's subjects, drop the mirror config (step 2), then bind those subjects to the promoted stream (step 4).

Step 4 — add the subjects so it accepts writes​

A mirror has no subjects of its own; it receives messages through the mirror mechanism, not by listening on orders.>. A writable primary needs to bind those subjects so publishers can reach it. Add them:

# Give the promoted stream the subjects ORDERS used to own.
nats --server nats://site2:4222 stream edit ORDERS_DR --subjects "orders.>"

ORDERS_DR now captures orders.created, orders.shipped, orders.canceled — the same subjects the lost primary held. It now functions as a full primary.

Step 5 — redirect publishers and consumers​

The last step redirects traffic. Point order-svc and the consumers at site2 and they resume against the promoted stream:

# Publishers and consumers now connect to site2.
nats --server nats://site2:4222 pub orders.created \
  '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'

The order JSON is byte-for-byte what order-svc always sent; only the server address changed. Failover is complete: the platform writes and reads at site2, and the data loss is exactly the stalled lag you noted in step 1.

Two deployment preconditions sit under this whole sequence. First, every stream rm and stream edit here goes through the JetStream metadata group, so that group has to keep quorum after the site is lost — if the failed site held the meta majority, no edit succeeds until the cluster recovers. Second, the promoted stream must live where that quorum survives. That's why a DR mirror is normally placed in its own JetStream domain (a leaf node) or an independent cluster, rather than sharing one meta group that spans both sites.

How the mirror replicated those messages in the first place (the config, the filters, the start position) is the JetStream chapter's job, covered in Mirrors and sources. The runbook only reads the lag and changes the relationship.

Recovery from a mistake: restore the snapshot​

The other rows of the table roll back rather than fail over. When corrupt messages were written to ORDERS, the mirror is no help, because it copied the bad writes; the snapshot is the intact copy, taken before the bad event. An accidental delete or purge is the exception: the mirror keeps everything it had already replicated, so it's often the freshest surviving copy — recover from it when it holds more than the last snapshot, and fall back to the snapshot otherwise.

#!/bin/bash

# Recovery for a hard site loss or an accidental delete: rebuild ORDERS
# from the last off-site snapshot. Restore recreates the stream byte
# for byte — same messages, same sequence numbers, same config, and
# (because this snapshot was taken with --consumers) the same shipping
# and analytics consumers at their saved delivery position.
#
# Do NOT recreate an empty ORDERS first. Restore creates the stream
# itself; an existing stream of the same name makes the restore fail.
# After an accidental delete, restore straight from the snapshot.

nats --server nats://127.0.0.1:4222 stream restore ./backups/orders/2026-06-04

# Expected tail of the output:
#
#   Starting restore of Stream "ORDERS" from ./backups/orders/2026-06-04
#   ...
#   Restored stream "ORDERS" in 0.38s
#
#   Information for Stream ORDERS
#   ...
#            Messages: 1,000
#       First Sequence: 1
#        Last Sequence: 1,000
#     Active Consumers: 2
#
# Then verify: the message count and last sequence must match what the
# snapshot held. Everything published after the snapshot was taken is
# gone — that gap is your recovery point. A snapshot you never test is
# a guess, so rehearse this restore on a schedule, not during the
# outage.

For an accidental delete, restore straight from the snapshot. Don't recreate an empty ORDERS first, because restore creates the stream itself and an existing stream of the same name makes it fail. For logical corruption, stop the publishers first so no new bad data races in, then either purge the corrupt sequence range or restore a known-good snapshot that predates it. For a lost consumer position, a snapshot taken with --consumers brings the durable consumer config and its delivery position back together.

The mechanics of the snapshot itself (chunking, the backup.json, the restore name rule) live one page back on Stream backup and restore. Here it's one step in a larger procedure.

Last resort: rescue the meta group​

Every recovery above assumes the metadata layer still works. Streams are created, restored, and moved by the meta group, and it needs a majority of its peers to agree. Lose enough servers permanently — three of five gone with their disks — and the survivors can't form that majority. There's no leader, so peer-remove can't run, because removing a peer is itself a change the meta group has to agree on.

From NATS 2.15 there's a supported way out. Rescue temporarily lowers the number of peers the meta group needs, so the survivors can agree again and you can remove the dead ones:

nats server cluster rescue

The command asks every server for its view first and prints what it found, so you can see how many are actually reachable before changing anything:

Current Cluster State

   Server   Cluster  Size  Peers               Quorum Requires  Rescuing
   n1-east  east     5     2 online / 5 peers  3                false
   n2-east  east     5     2 online / 5 peers  3                false

Two servers online where quorum needs three is the situation rescue exists for. It then asks how many peers to require, defaulting to the number still online, broadcasts the change, and reports back per server:

Server "n1-east" now requires 2 quorum size (was 3)
Server "n2-east" now requires 2 quorum size (was 3)

The meta group elects a leader again, and you finish the job by removing the servers that aren't coming back:

nats server cluster peer-remove n3-east

Only ever peer-remove a server that is stopped. In this example, these servers are already gone, which is why you're rescuing at all, but the rule holds everywhere: a running server that gets peer-removed disables JetStream on itself. If one of the "dead" servers turns out to be reachable, shut it down before removing its peer.

Once only live servers remain in the meta group, normal operation resumes and the recoveries earlier on this page become possible again.

Where more than one JetStream domain is present, you must specify the domain in the rescue command. Each domain runs its own meta group, with its own peers and its own quorum. A rescue lowers the quorum of exactly one of them, so on a topology with several domains — a hub and its leaf domains, say — the command has to be told which meta group it is rescuing. Left to guess, it stops with multiple domains found, pick one from ... and lists what answered. Name the domain:

nats server cluster rescue --js-domain hub

This command only works when the system is unable to elect a leader, printing Cluster is healthy with a leader, no rescue needed. In this case, revisit the earlier sections of this guide to restore stream or consumer operations. It also skips servers older than 2.15.0, naming each one, so a half-upgraded cluster tells you which nodes it ignored.

Pitfalls​

The runbook fails most often on the order of the steps and the assumptions around them rather than on the commands themselves. Five common mistakes happen mid-outage.

Never promote a mirror before lag reaches zero. Promotion makes the mirror writable. If you do it while messages are still in flight from the upstream, those tail messages are lost and new writes land on top of the gap. Always run the lag check first, and only proceed at Lag: 0 or with a stalled lag you've consciously accepted as your recovery point. Read the lag and don't skip step 1.

You can make that check a gate. Run the lag check from step 1 above, decide on the number, then promote, never the reverse.

R3 replication will not save you from a mistake. A three-replica stream survives a node loss, but an accidental delete or a bad publish replicates to all three copies at once. R3 is availability, not a backup. Don't put it on the mistake rows of the table; that's what snapshots are for.

Stop publishers before purging corrupted messages. Purging a bad sequence range while order-svc is still writing lets new corrupt data arrive behind you, so you keep purging against a tail that keeps growing. Stop the publishers, purge or restore, then resume.

Rescue is not peer management. Lowering the quorum makes a smaller set of servers authoritative over the cluster's metadata which, if not used carefully, can cause a split-brain condition. Therefore, it is important that the servers are genuinely never going to return. Retiring a node in the normal course is nats server cluster evacuate and nats server cluster peer-remove while the meta group still has a leader — never rescue.

An untested snapshot is unverified. A healthy nats stream info on the live stream tells you the live stream is healthy; it proves nothing about the archive in ./backups/orders/. Rehearse the restore on a schedule (quarterly is a reasonable minimum) into a throwaway stream or server, so the first time you run it isn't during the outage.

Where you are​

You can now name a NATS failure and reach for the right tool quickly. A lost site means promote ORDERS_DR: verify lag, drop the mirror config, add the subjects, redirect traffic. A mistake (delete, corruption, or a lost consumer position) means restore the snapshot, the copy that predates the mistake — a mirror copies a corrupt write and only freezes at an upstream delete. R3 is on neither path; it's availability, not recovery.

The data plane is now fully covered: a snapshot you can restore to, and a site where you can promote the mirror.

What's next​

One layer is still unprotected. If a laptop full of keys is lost, or the servers are rebuilt clean, the data is recoverable but no one can prove who they are: the operator JWT, the account JWTs, the nkeys, and the creds are gone. The next page backs up the identity plane.

Continue to Config and JWT backup.

See also​