# NATS Documentation > Connective Technology for Adaptive Edge & Distributed Systems — official NATS messaging documentation. - [NATS Documentation](/index.md) ## concepts ### ecosystem NATS is more than the nats-server binary. Around it sits a set of officially maintained clients, extension libraries, command-line and Kubernetes tooling, identity tools, and integrations. - [The NATS Ecosystem](/concepts/ecosystem.md): NATS is more than the nats-server binary. Around it sits a set of officially maintained clients, extension libraries, command-line and Kubernetes tooling, identity tools, and integrations. ### getting-started Get up and running with NATS in minutes. This guide will walk you through installation, basic setup, and your first NATS application. - [Getting Started with NATS](/concepts/getting-started.md): Get up and running with NATS in minutes. This guide will walk you through installation, basic setup, and your first NATS application. ### intro Welcome to the official documentation for NATS - a simple, secure and high-performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures. - [Welcome to NATS Documentation](/concepts/intro.md): Welcome to the official documentation for NATS - a simple, secure and high-performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures. ### jetstream NATS's built-in persistence and streaming layer - [JetStream](/concepts/jetstream.md): NATS's built-in persistence and streaming layer ### pub-sub-basics The foundational messaging pattern in NATS — everything else builds on this - [Publish-Subscribe](/concepts/pub-sub-basics.md): The foundational messaging pattern in NATS — everything else builds on this ### queue-groups Built-in load balancing for NATS subscribers - [Queue Groups](/concepts/queue-groups.md): Built-in load balancing for NATS subscribers ### request-reply Synchronous communication pattern in NATS - [Request-Reply](/concepts/request-reply.md): Synchronous communication pattern in NATS ### security Authentication, authorization, and encryption in NATS - [Security](/concepts/security.md): Authentication, authorization, and encryption in NATS ### subjects Understanding NATS subject-based messaging and wildcards - [Subjects](/concepts/subjects.md): Understanding NATS subject-based messaging and wildcards ### topologies How NATS servers can be arranged — from a single process to globally distributed deployments - [Topologies](/concepts/topologies.md): How NATS servers can be arranged — from a single process to globally distributed deployments ### what-is-nats NATS is an open source messaging system. Applications connect to a NATS server and exchange messages by subject, without knowing each other's network addresses. - [What is NATS?](/concepts/what-is-nats.md): NATS is an open source messaging system. Applications connect to a NATS server and exchange messages by subject, without knowing each other's network addresses. ## learn Long-form deep dives into NATS subsystems - [Learn](/learn.md): Long-form deep dives into NATS subsystems ### backup-recovery Keep a built NATS platform alive across a node loss, a fat-fingered delete, or a lost laptop full of keys - [Backup & Recovery Deep Dive](/learn/backup-recovery.md): Keep a built NATS platform alive across a node loss, a fat-fingered delete, or a lost laptop full of keys #### config-and-jwt-backup Back up and restore the operator, accounts, keys, and server config off-site so the identity plane survives a clean-room rebuild - [Config and JWT backup](/learn/backup-recovery/config-and-jwt-backup.md): Back up and restore the operator, accounts, keys, and server config off-site so the identity plane survives a clean-room rebuild #### disaster-recovery A runbook that picks restore-from-snapshot or promote-the-mirror per failure class, then promotes ORDERS_DR cleanly - [Disaster recovery](/learn/backup-recovery/disaster-recovery.md): A runbook that picks restore-from-snapshot or promote-the-mirror per failure class, then promotes ORDERS_DR cleanly #### mirrors-and-sources Stand up a cross-site mirror of ORDERS for disaster recovery, read its lag, and learn why a mirror is not a backup - [Mirrors as a DR tool](/learn/backup-recovery/mirrors-and-sources.md): Stand up a cross-site mirror of ORDERS for disaster recovery, read its lag, and learn why a mirror is not a backup #### stream-backup-restore Take a point-in-time snapshot of the ORDERS stream, restore it byte-identical, and verify the counts match - [Stream backup and restore](/learn/backup-recovery/stream-backup-restore.md): Take a point-in-time snapshot of the ORDERS stream, restore it byte-identical, and verify the counts match #### where-next Recap the backup-and-recovery triad and point to what comes after this chapter - [Where to go next](/learn/backup-recovery/where-next.md): Recap the backup-and-recovery triad and point to what comes after this chapter ### clustering How a NATS cluster agrees and replicates — routes, RAFT, quorum commits, placement, and peers - [Clustering & Replication Deep Dive](/learn/clustering.md): How a NATS cluster agrees and replicates — routes, RAFT, quorum commits, placement, and peers #### forming-a-cluster How servers find each other — explicit seed routes and the gossip that grows one seed into a full mesh - [Forming a cluster](/learn/clustering/forming-a-cluster.md): How servers find each other — explicit seed routes and the gossip that grows one seed into a full mesh #### placement Constrain where a stream's replicas live using cluster and server tags, and ask a chosen server to take leadership - [Placement](/learn/clustering/placement.md): Constrain where a stream's replicas live using cluster and server tags, and ask a chosen server to take leadership #### raft-and-leaders How a NATS cluster reaches agreement — RAFT groups, leaders and followers, and the election that picks a leader on a quorum of votes - [Raft and leaders](/learn/clustering/raft-and-leaders.md): How a NATS cluster reaches agreement — RAFT groups, leaders and followers, and the election that picks a leader on a quorum of votes #### replication-and-r3 How a replicated stream commits a write by quorum, the consistency you get from it, and what quorum does and doesn't guarantee once the write reaches disk - [Replication and R=3](/learn/clustering/replication-and-r3.md): How a replicated stream commits a write by quorum, the consistency you get from it, and what quorum does and doesn't guarantee once the write reaches disk #### scaling-and-peers Add a peer with catchup, remove one safely, and keep quorum while you grow or shrink the ORDERS group - [Scaling and peer management](/learn/clustering/scaling-and-peers.md): Add a peer with catchup, remove one safely, and keep quorum while you grow or shrink the ORDERS group #### where-next Recap the clustering mechanism and point to siblings, Reference, and a production checklist - [Where to go next](/learn/clustering/where-next.md): Recap the clustering mechanism and point to siblings, Reference, and a production checklist ### core-nats Publish-subscribe, subjects, request-reply, queue groups, and scatter-gather — from the first connection to debugging delivery, built up step by step - [Core NATS Deep Dive](/learn/core-nats.md): Publish-subscribe, subjects, request-reply, queue groups, and scatter-gather — from the first connection to debugging delivery, built up step by step #### connecting One long-lived TCP connection that carries every publish and subscribe, named at connect time and kept alive with PING/PONG heartbeats - [Connecting](/learn/core-nats/connecting.md): One long-lived TCP connection that carries every publish and subscribe, named at connect time and kept alive with PING/PONG heartbeats #### connection-lifecycle What happens to your client and your in-flight messages when a connection drops, and how to observe the reconnect - [Connection lifecycle](/learn/core-nats/connection-lifecycle.md): What happens to your client and your in-flight messages when a connection drops, and how to observe the reconnect #### debugging-delivery Find out why a published message never arrived, using a wire tap, the subscription list, and message tracing - [Debugging delivery](/learn/core-nats/debugging-delivery.md): Find out why a published message never arrived, using a wire tap, the subscription list, and message tracing #### headers Attach key/value metadata to any message with the NATS/1.0 header format, set and read it, and see the status line the server uses in-band - [Message headers](/learn/core-nats/headers.md): Attach key/value metadata to any message with the NATS/1.0 header format, set and read it, and see the status line the server uses in-band #### publish-subscribe Fire-and-forget publish, the in-memory interest graph, and core NATS at-most-once delivery - [Publish-subscribe](/learn/core-nats/publish-subscribe.md): Fire-and-forget publish, the in-memory interest graph, and core NATS at-most-once delivery #### queue-groups Share one subject across a pool of subscribers, so each message is handled by exactly one of them - [Queue groups](/learn/core-nats/queue-groups.md): Share one subject across a pool of subscribers, so each message is handled by exactly one of them #### request-reply Build a reply on top of pub/sub with a private inbox, a timeout, and the no-responders signal - [Request-reply](/learn/core-nats/request-reply.md): Build a reply on top of pub/sub with a private inbox, a timeout, and the no-responders signal #### scatter-gather Fan one request to many responders and gather every reply by count or deadline - [Scatter-gather](/learn/core-nats/scatter-gather.md): Fan one request to many responders and gather every reply by count or deadline #### subject-mapping How the server rewrites a message's subject from its own config before routing — to rename a subject, split its traffic by weight, or shard it into stable buckets by a hashed token - [Subject mapping](/learn/core-nats/subject-mapping.md): How the server rewrites a message's subject from its own config before routing — to rename a subject, split its traffic by weight, or shard it into stable buckets by a hashed token #### subjects-and-wildcards How NATS addresses messages with dot-delimited subjects, and how subscribers match many of them at once with wildcards - [Subjects & wildcards](/learn/core-nats/subjects-and-wildcards.md): How NATS addresses messages with dot-delimited subjects, and how subscribers match many of them at once with wildcards #### where-next Recap the core NATS mental model and point to what comes after the foundation - [Where to go next](/learn/core-nats/where-next.md): Recap the core NATS mental model and point to what comes after the foundation ### deployment Take the Acme ORDERS cluster to production — size it, deploy it, configure it live, upgrade it safely, and harden it - [Deployment & Upgrades Deep Dive](/learn/deployment.md): Take the Acme ORDERS cluster to production — size it, deploy it, configure it live, upgrade it safely, and harden it #### config-management Split the cluster config into includes and reload it live with a SIGHUP — no downtime, no client reconnect - [Config management](/learn/deployment/config-management.md): Split the cluster config into includes and reload it live with a SIGHUP — no downtime, no client reconnect #### hardening Lock down the ORDERS cluster — TLS on every link, mounted credentials, and a sandboxed systemd unit - [Hardening](/learn/deployment/hardening.md): Lock down the ORDERS cluster — TLS on every link, mounted credentials, and a sandboxed systemd unit #### kubernetes Stand the ORDERS cluster up as a StatefulSet with the NATS Helm chart, then declare its streams and consumers as CRDs - [Kubernetes](/learn/deployment/kubernetes.md): Stand the ORDERS cluster up as a StatefulSet with the NATS Helm chart, then declare its streams and consumers as CRDs #### rolling-upgrades Roll a new server version through the ORDERS cluster with lame-duck mode and the right upgrade order, keeping the R3 stream available and clients connected - [Rolling upgrades](/learn/deployment/rolling-upgrades.md): Roll a new server version through the ORDERS cluster with lame-duck mode and the right upgrade order, keeping the R3 stream available and clients connected #### sizing-and-resources The four resources a NATS node spends, the JetStream defaults, and how account limits count R3 replication - [Sizing & resources](/learn/deployment/sizing-and-resources.md): The four resources a NATS node spends, the JetStream defaults, and how account limits count R3 replication #### where-next Recap the deployment runbook and collect every page's production checklist in one place - [Where to go next](/learn/deployment/where-next.md): Recap the deployment runbook and collect every page's production checklist in one place ### jetstream Streams, consumers, and the acknowledgment loop, taught step by step - [JetStream Deep Dive](/learn/jetstream.md): Streams, consumers, and the acknowledgment loop, taught step by step #### acknowledgment The four ways a client answers a message, and the server controls that drive redelivery. - [Ack responses and redelivery](/learn/jetstream/acknowledgment.md): The four ways a client answers a message, and the server controls that drive redelivery. #### advanced-publishing Async, atomic-batch, and fast-ingest publishing — what each is for, the order trap in async, and the code for every mode - [Advanced publishing](/learn/jetstream/advanced-publishing.md): Async, atomic-batch, and fast-ingest publishing — what each is for, the order trap in async, and the code for every mode #### altering-stream-state Delete a single message or purge the stream by hand, and what removal does to sequence numbers - [Altering stream state](/learn/jetstream/altering-stream-state.md): Delete a single message or purge the stream by hand, and what removal does to sequence numbers #### delivery-and-acknowledgment How a message is delivered and held in flight, what an ack and a double ack do, and how an unacked message comes back - [Delivery and acknowledgment](/learn/jetstream/delivery-and-acknowledgment.md): How a message is delivered and held in flight, what an ack and a double ack do, and how an unacked message comes back #### filtering Add a second consumer that reads only orders.shipped, and see consumers as independent views - [Filtering what you consume](/learn/jetstream/filtering.md): Add a second consumer that reads only orders.shipped, and see consumers as independent views #### get-direct Get one message or a bounded batch straight from the stream with Direct Get — no consumer, served by any replica - [Reading messages directly](/learn/jetstream/get-direct.md): Get one message or a bounded batch straight from the stream with Direct Get — no consumer, served by any replica #### message-ttl Make a single message expire ahead of the stream's MaxAge - [Per-message TTL](/learn/jetstream/message-ttl.md): Make a single message expire ahead of the stream's MaxAge #### mirrors-and-sources Copy one stream into another, or aggregate many streams into one - [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md): Copy one stream into another, or aggregate many streams into one #### ordered-consumer A fast, in-order read of a stream that cleans up after itself — what the library does for you, and the consumer config underneath - [Ordered consumers](/learn/jetstream/ordered-consumer.md): A fast, in-order read of a stream that cleans up after itself — what the library does for you, and the consumer config underneath #### pausing Stop delivery to a consumer until a deadline, then resume where it left off - [Pausing a consumer](/learn/jetstream/pausing.md): Stop delivery to a consumer until a deadline, then resume where it left off #### policies Every stream and consumer policy in one place, what each decides, and which ones you can still change - [Stream and consumer policies](/learn/jetstream/policies.md): Every stream and consumer policy in one place, what each decides, and which ones you can still change #### priority-groups Steer which client on a pull consumer gets messages, and when - [Priority groups](/learn/jetstream/priority-groups.md): Steer which client on a pull consumer gets messages, and when #### publishing Publish into a stream and understand the PubAck contract - [Publishing](/learn/jetstream/publishing.md): Publish into a stream and understand the PubAck contract #### pull-consumers Fetch a batch versus consume a continuous flow, and the fields that bound each pull - [Pull consumers in depth](/learn/jetstream/pull-consumers.md): Fetch a batch versus consume a continuous flow, and the fields that bound each pull #### reading-back Meet the producer, stream, and consumer; create a durable consumer and read everything the stream holds - [Reading back the stream](/learn/jetstream/reading-back.md): Meet the producer, stream, and consumer; create a durable consumer and read everything the stream holds #### retention-policies The three retention policies, and how to pick one for the kind of work a stream does - [Retention policies](/learn/jetstream/retention-policies.md): The three retention policies, and how to pick one for the kind of work a stream does #### shaping-the-stream Control stream size and age with limits, and decide what happens when a limit is reached - [Shaping the stream](/learn/jetstream/shaping-the-stream.md): Control stream size and age with limits, and decide what happens when a limit is reached #### subject-mapping Rewrite subjects on the way into a stream, and republish stored messages onto new subjects - [Subject mapping and transforms](/learn/jetstream/subject-mapping.md): Rewrite subjects on the way into a stream, and republish stored messages onto new subjects #### surviving-node-loss Why R=1 is a single point of failure, why R=3 is the production floor, and what storage durability means - [Surviving node loss](/learn/jetstream/surviving-node-loss.md): Why R=1 is a single point of failure, why R=3 is the production floor, and what storage durability means #### where-next Recap the JetStream model and point to what comes after this chapter - [Where to go next](/learn/jetstream/where-next.md): Recap the JetStream model and point to what comes after this chapter #### worker-pool Point several workers at one consumer to share the load, and what happens when one crashes mid-message - [Scaling a consumer](/learn/jetstream/worker-pool.md): Point several workers at one consumer to share the load, and what happens when one crashes mid-message #### your-first-stream Why you want a stream, then create the ORDERS stream and look at its configuration - [Your first stream](/learn/jetstream/your-first-stream.md): Why you want a stream, then create the ORDERS stream and look at its configuration ### key-value Buckets, keys, watching, revisions, and TTL, built on the stream you already have - [Key-Value Store](/learn/key-value.md): Buckets, keys, watching, revisions, and TTL, built on the stream you already have #### history-and-revisions Track a key's revisions, read its history, and decrement safely with compare-and-swap - [History and revisions](/learn/key-value/history-and-revisions.md): Track a key's revisions, read its history, and decrement safely with compare-and-swap #### ttl-and-limits Expire a single key with a per-key TTL, bound the bucket with limits, and watch a value disappear on its own - [TTL and limits](/learn/key-value/ttl-and-limits.md): Expire a single key with a per-key TTL, bound the bucket with limits, and watch a value disappear on its own #### under-the-hood The KV_INVENTORY stream behind the bucket, the direct read path, and delete versus purge - [Under the hood](/learn/key-value/under-the-hood.md): The KV_INVENTORY stream behind the bucket, the direct read path, and delete versus purge #### watching Watch a bucket for live stock changes — the initial snapshot, the end-of-initial-data signal, and wildcard filters - [Watching](/learn/key-value/watching.md): Watch a bucket for live stock changes — the initial snapshot, the end-of-initial-data signal, and wildcard filters #### where-next Recap the key-value mental model and point to what comes after this chapter - [Where to go next](/learn/key-value/where-next.md): Recap the key-value mental model and point to what comes after this chapter #### your-first-bucket Create the INVENTORY bucket, put a stock count, and get back an entry - [Your first bucket](/learn/key-value/your-first-bucket.md): Create the INVENTORY bucket, put a stock count, and get back an entry ### monitoring Watch a running NATS deployment — where the numbers come from, which ones matter, and how to turn them into alerts and dashboards - [Monitoring & Observability Deep Dive](/learn/monitoring.md): Watch a running NATS deployment — where the numbers come from, which ones matter, and how to turn them into alerts and dashboards #### advisories-and-events Subscribe to JetStream advisories and system events to learn about things you never polled for - [Advisories & events](/learn/monitoring/advisories-and-events.md): Subscribe to JetStream advisories and system events to learn about things you never polled for #### jetstream-health Read stream and consumer state, then compute lag, in-flight, and redelivery as numbers - [JetStream health](/learn/monitoring/jetstream-health.md): Read stream and consumer state, then compute lag, in-flight, and redelivery as numbers #### monitoring-endpoints The HTTP monitoring port :8222 and its on-demand JSON — /varz, /connz, /routez, and the /jsz JetStream lens - [Monitoring endpoints](/learn/monitoring/monitoring-endpoints.md): The HTTP monitoring port :8222 and its on-demand JSON — /varz, /connz, /routez, and the /jsz JetStream lens #### prometheus-and-dashboards Scrape the monitoring port into Prometheus time series, chart them in Grafana, and alert with nats server check thresholds - [Prometheus & dashboards](/learn/monitoring/prometheus-and-dashboards.md): Scrape the monitoring port into Prometheus time series, chart them in Grafana, and alert with nats server check thresholds #### where-next Recap the four monitoring lenses and point to the Operate siblings and Reference that take watching NATS further - [Where to go next](/learn/monitoring/where-next.md): Recap the four monitoring lenses and point to the Operate siblings and Reference that take watching NATS further ### mqtt Run nats-server as the MQTT broker for your devices, so their topics land on the NATS subjects your applications already use - [Connect MQTT devices to NATS](/learn/mqtt.md): Run nats-server as the MQTT broker for your devices, so their topics land on the NATS subjects your applications already use #### auth-and-clustering Restrict a user to MQTT connections, authenticate devices in operator mode, and run MQTT on the east cluster - [Auth and clustering](/learn/mqtt/auth-and-clustering.md): Restrict a user to MQTT connections, authenticate devices in operator mode, and run MQTT on the east cluster #### qos-sessions-and-retained What the server guarantees about delivery, what it stores between connections, and why NATS publishes always arrive as QoS 0 - [QoS, sessions, and retained messages](/learn/mqtt/qos-sessions-and-retained.md): What the server guarantees about delivery, what it stores between connections, and why NATS publishes always arrive as QoS 0 #### topics-and-subjects How the server converts MQTT topics into NATS subjects, what wildcards become, and which characters have no valid conversion - [Topics and subjects](/learn/mqtt/topics-and-subjects.md): How the server converts MQTT topics into NATS subjects, what wildcards become, and which characters have no valid conversion #### where-next Recap the MQTT model and point to the chapters and Reference that take it further - [Where to go next](/learn/mqtt/where-next.md): Recap the MQTT model and point to the chapters and Reference that take it further #### your-first-mqtt-client Enable the mqtt block on a local server, publish from an MQTT sensor, and receive the reading on the NATS side - [Your first MQTT client](/learn/mqtt/your-first-mqtt-client.md): Enable the mqtt block on a local server, publish from an MQTT sensor, and receive the reading on the NATS side ### object-store Buckets, objects, chunks, metadata, and links, built on the stream you already have - [Object Store](/learn/object-store.md): Buckets, objects, chunks, metadata, and links, built on the stream you already have #### chunking How a large invoice is split into chunks, reassembled, and verified by its digest - [Chunking](/learn/object-store/chunking.md): How a large invoice is split into chunks, reassembled, and verified by its digest #### metadata-and-links Attach a description, headers, and a metadata map to an object, then link one object to another - [Metadata and links](/learn/object-store/metadata-and-links.md): Attach a description, headers, and a metadata map to an object, then link one object to another #### under-the-hood See the OBJ_INVOICES stream, the chunk and metadata subjects, and rollup versus soft delete - [Under the hood](/learn/object-store/under-the-hood.md): See the OBJ_INVOICES stream, the chunk and metadata subjects, and rollup versus soft delete #### watching-and-listing List the INVOICES bucket and watch it for new objects with the analytics service - [Watching and listing](/learn/object-store/watching-and-listing.md): List the INVOICES bucket and watch it for new objects with the analytics service #### where-next Recap the object store mental model and point to what comes after this chapter - [Where to go next](/learn/object-store/where-next.md): Recap the object store mental model and point to what comes after this chapter #### your-first-object Put a file into the INVOICES bucket and get it back - [Your first object](/learn/object-store/your-first-object.md): Put a file into the INVOICES bucket and get it back ### resilient-clients The client connection as a state machine that detects faults, buffers through them, and recovers - [Resilient Clients Deep Dive](/learn/resilient-clients.md): The client connection as a state machine that detects faults, buffers through them, and recovers #### connecting Open order-svc with a name, a server pool, and a connect timeout, and read the connect handshake - [Connecting](/learn/resilient-clients/connecting.md): Open order-svc with a name, a server pool, and a connect timeout, and read the connect handshake #### connection-events Make the connection state machine observable — wire every connection event and read the state for health checks - [Connection Events](/learn/resilient-clients/connection-events.md): Make the connection state machine observable — wire every connection event and read the state for health checks #### drain-and-shutdown Exit a connection without dropping in-flight work by draining instead of closing - [Drain & Shutdown](/learn/resilient-clients/drain-and-shutdown.md): Exit a connection without dropping in-flight work by draining instead of closing #### reconnection Survive a server going away — reconnect with backoff and jitter across the pool, and buffer publishes through the gap - [Reconnection](/learn/resilient-clients/reconnection.md): Survive a server going away — reconnect with backoff and jitter across the pool, and buffer publishes through the gap #### request-reply-resilience Tell a slow responder apart from an absent one, then retry the request safely with backoff and idempotent IDs - [Request-Reply Resilience](/learn/resilient-clients/request-reply-resilience.md): Tell a slow responder apart from an absent one, then retry the request safely with backoff and idempotent IDs #### slow-consumers Bound a subscription's in-memory buffer so a slow handler surfaces backlog instead of silently exhausting memory - [Slow Consumers](/learn/resilient-clients/slow-consumers.md): Bound a subscription's in-memory buffer so a slow handler surfaces backlog instead of silently exhausting memory #### tls-and-auth Connect order-svc securely by consuming a credentials file and trusting the cluster CA over a TLS handshake - [TLS & Auth](/learn/resilient-clients/tls-and-auth.md): Connect order-svc securely by consuming a credentials file and trusting the cluster CA over a TLS handshake #### where-next Recap the connection lifecycle, point to the sibling deep dives, and collect every page's pitfalls into one production checklist - [Where Next](/learn/resilient-clients/where-next.md): Recap the connection lifecycle, point to the sibling deep dives, and collect every page's pitfalls into one production checklist ### security Authentication, authorization, and encryption, scoped per account and built up step by step - [Security Deep Dive](/learn/security.md): Authentication, authorization, and encryption, scoped per account and built up step by step #### accounts-and-multitenancy How an account isolates a tenant's subject space, and the two accounts every server already has - [Accounts and multitenancy](/learn/security/accounts-and-multitenancy.md): How an account isolates a tenant's subject space, and the two accounts every server already has #### auth-callout Delegate the authentication decision to an external service over $SYS.REQ.USER.AUTH, protected by a signed request and response - [Auth callout](/learn/security/auth-callout.md): Delegate the authentication decision to an external service over $SYS.REQ.USER.AUTH, protected by a signed request and response #### authentication-basics Centralized config-based authentication and the three credential types - [Authentication basics](/learn/security/authentication-basics.md): Centralized config-based authentication and the three credential types #### authorization Subject permissions, allow and deny lists, and the rule that an allow-list closes everything else - [Authorization](/learn/security/authorization.md): Subject permissions, allow and deny lists, and the rule that an allow-list closes everything else #### cross-account Open exactly one subject between two isolated accounts with an export/import pair - [Cross-account](/learn/security/cross-account.md): Open exactly one subject between two isolated accounts with an export/import pair #### decentralized-auth What the server verifies in the operator-account-user trust chain, and how scoped signing keys, revocation, and expiry control users - [Decentralized authentication](/learn/security/decentralized-auth.md): What the server verifies in the operator-account-user trust chain, and how scoped signing keys, revocation, and expiry control users #### encryption Secure the wire with TLS and mTLS, then encrypt the JetStream store on disk - [Encryption & TLS](/learn/security/encryption.md): Secure the wire with TLS and mTLS, then encrypt the JetStream store on disk #### operator-mode Build the ACME trust chain with nats auth, point the server at a resolver, and connect with a creds file - [Operator mode](/learn/security/operator-mode.md): Build the ACME trust chain with nats auth, point the server at a resolver, and connect with a creds file #### where-next The whole security model in one picture, and where to read further - [Where to go next](/learn/security/where-next.md): The whole security model in one picture, and where to read further ### services The micro framework that turns a request-reply responder into a named, versioned, discoverable service - [Services](/learn/services.md): The micro framework that turns a request-reply responder into a named, versioned, discoverable service #### discovery Learn what services exist and what they answer using the $SRV discovery verbs - [Discovery](/learn/services/discovery.md): Learn what services exist and what they answer using the $SRV discovery verbs #### endpoints-and-groups Give a service more than one endpoint, and organize endpoints under a group subject prefix - [Endpoints and groups](/learn/services/endpoints-and-groups.md): Give a service more than one endpoint, and organize endpoints under a group subject prefix #### observability Read the per-endpoint stats the framework keeps for you, and signal service errors that those stats record - [Observability](/learn/services/observability.md): Read the per-endpoint stats the framework keeps for you, and signal service errors that those stats record #### scaling Run N instances of a service and let the default queue group balance the load, then stop one cleanly - [Scaling](/learn/services/scaling.md): Run N instances of a service and let the default queue group balance the load, then stop one cleanly #### where-next Recap the Services mental model and point to the chapters and Reference that take it further - [Where to go next](/learn/services/where-next.md): Recap the Services mental model and point to the chapters and Reference that take it further #### your-first-service Promote the Core NATS inventory responder into a named, versioned OrderInventory service with one endpoint - [Your first service](/learn/services/your-first-service.md): Promote the Core NATS inventory responder into a named, versioned OrderInventory service with one endpoint ### topologies How NATS servers compose into bigger shapes, grown one deployment at a time - [Topologies Deep Dive](/learn/topologies.md): How NATS servers compose into bigger shapes, grown one deployment at a time #### jetstream-in-a-cluster What changes for JetStream once it runs on a cluster — the meta layer and replication - [JetStream in a cluster](/learn/topologies/jetstream-in-a-cluster.md): What changes for JetStream once it runs on a cluster — the meta layer and replication #### leaf-nodes Bridge a factory site to the east cluster with a leaf node that dials out and binds to an account - [Leaf nodes](/learn/topologies/leaf-nodes.md): Bridge a factory site to the east cluster with a leaf node that dials out and binds to an account #### putting-it-together Compose clusters, gateways, and leaf nodes into the full Acme topology, with isolation behind leaves and the same client code everywhere - [Putting it together](/learn/topologies/putting-it-together.md): Compose clusters, gateways, and leaf nodes into the full Acme topology, with isolation behind leaves and the same client code everywhere #### single-server One nats-server process clients connect to directly — when it's enough and where its ceiling is - [Single server](/learn/topologies/single-server.md): One nats-server process clients connect to directly — when it's enough and where its ceiling is #### super-clusters Join the east and west clusters with gateways, and keep order traffic local by default - [Super-clusters](/learn/topologies/super-clusters.md): Join the east and west clusters with gateways, and keep order traffic local by default #### where-next Recap the four composable shapes and point to the mechanics, operations, and protocol references beyond this chapter - [Where to go next](/learn/topologies/where-next.md): Recap the four composable shapes and point to the mechanics, operations, and protocol references beyond this chapter #### your-first-cluster Join three servers into a cluster with routes, then watch a client survive a server loss - [Your first cluster](/learn/topologies/your-first-cluster.md): Join three servers into a cluster with routes, then watch a client survive a server loss ### websocket Run the NATS protocol over a WebSocket transport, so browsers and anything behind an HTTP ingress can reach the same server - [Connect over WebSocket](/learn/websocket.md): Run the NATS protocol over a WebSocket transport, so browsers and anything behind an HTTP ingress can reach the same server #### browsers-and-origins Restrict which web pages may connect, and give a browser credentials when it has no credentials file - [Browsers and origins](/learn/websocket/browsers-and-origins.md): Restrict which web pages may connect, and give a browser credentials when it has no credentials file #### leaf-nodes-over-websocket Point a leaf node at a WebSocket listener instead of the leafnode port, and the differences that come with it - [Leaf nodes over WebSocket](/learn/websocket/leaf-nodes-over-websocket.md): Point a leaf node at a WebSocket listener instead of the leafnode port, and the differences that come with it #### tls-and-proxies Put a certificate on the listener or terminate TLS at an ingress, and configure a proxy that won't break a long-lived upgraded connection - [TLS and proxies](/learn/websocket/tls-and-proxies.md): Put a certificate on the listener or terminate TLS at an ingress, and configure a proxy that won't break a long-lived upgraded connection #### where-next Recap the WebSocket model and point to the chapters and Reference that take it further - [Where to go next](/learn/websocket/where-next.md): Recap the WebSocket model and point to the chapters and Reference that take it further #### your-first-websocket-connection Enable the websocket block on a local server, subscribe with the nats CLI over ws, then receive the same subject in a browser - [Your first WebSocket connection](/learn/websocket/your-first-websocket-connection.md): Enable the websocket block on a local server, subscribe with the nats CLI over ws, then receive the same subject in a browser ## reference NATS Protocol and API reference documentation. - [Reference](/reference.md): NATS Protocol and API reference documentation. ### config While the NATS server has many flags that allow for simple testing of features, the NATS server products provide a flexible configuration format that combines the best of traditional formats and newer styles such as JSON and YAML. - [Configuration](/reference/config.md): While the NATS server has many flags that allow for simple testing of features, the NATS server products provide a flexible configuration format that combines the best of traditional formats and newer styles such as JSON and YAML. #### accounts Static config-defined accounts. - [accounts](/reference/config/accounts.md): Static config-defined accounts. ##### default_permissions The default permissions applied to users within this account, - [default_permissions](/reference/config/accounts/default_permissions.md): The default permissions applied to users within this account, ###### allow_responses Types - [allow_responses](/reference/config/accounts/default_permissions/allow_responses.md): Types - [expires](/reference/config/accounts/default_permissions/allow_responses/expires.md): The amount of time the permission is valid. Values such - [max](/reference/config/accounts/default_permissions/allow_responses/max.md): The maximum number of response messages that can be published. ###### publish A single subject, list of subjects, or a allow-deny map of - [publish](/reference/config/accounts/default_permissions/publish.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/accounts/default_permissions/publish/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/accounts/default_permissions/publish/deny.md): List of subjects that are denied to the client. ###### subscribe A single subject, list of subjects, or a allow-deny map of - [subscribe](/reference/config/accounts/default_permissions/subscribe.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/accounts/default_permissions/subscribe/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/accounts/default_permissions/subscribe/deny.md): List of subjects that are denied to the client. ##### exports A list of exports for this account. - [exports](/reference/config/accounts/exports.md): A list of exports for this account. ###### accounts A list of account names that can import the stream or service. - [accounts](/reference/config/accounts/exports/accounts.md): A list of account names that can import the stream or service. ###### response_type Indicates if a response to a service request consists of a single - [response_type](/reference/config/accounts/exports/response_type.md): Indicates if a response to a service request consists of a single ###### service A subject or subject with wildcards that the account will subscribe to. - [service](/reference/config/accounts/exports/service.md): A subject or subject with wildcards that the account will subscribe to. ###### stream A subject or subject with wildcards that the account will publish to. - [stream](/reference/config/accounts/exports/stream.md): A subject or subject with wildcards that the account will publish to. ##### imports A list of imports for this account. - [imports](/reference/config/accounts/imports.md): A list of imports for this account. ###### prefix A local subject prefix mapping for the imported stream. - [prefix](/reference/config/accounts/imports/prefix.md): A local subject prefix mapping for the imported stream. ###### service Stream import source configuration. - [service](/reference/config/accounts/imports/service.md): Stream import source configuration. - [account](/reference/config/accounts/imports/service/account.md): Account name owning the export. - [subject](/reference/config/accounts/imports/service/subject.md): The subject under which the stream or service is made ###### stream Stream import source configuration. - [stream](/reference/config/accounts/imports/stream.md): Stream import source configuration. - [account](/reference/config/accounts/imports/stream/account.md): Account name owning the export. - [subject](/reference/config/accounts/imports/stream/subject.md): The subject under which the stream or service is made ###### to A local subject mapping for the imported service. - [to](/reference/config/accounts/imports/to.md): A local subject mapping for the imported service. ##### jetstream Types - [jetstream](/reference/config/accounts/jetstream.md): Types ###### disk_max_stream_bytes Maximum bytes any given file-based stream is allowed to be allocated. - [disk_max_stream_bytes](/reference/config/accounts/jetstream/disk_max_stream_bytes.md): Maximum bytes any given file-based stream is allowed to be allocated. ###### max_ack_pending The maximum ack pending count allowed to be set on any given consumer. - [max_ack_pending](/reference/config/accounts/jetstream/max_ack_pending.md): The maximum ack pending count allowed to be set on any given consumer. ###### max_bytes_required If true, requires all streams to have an explicit max bytes defined - [max_bytes_required](/reference/config/accounts/jetstream/max_bytes_required.md): If true, requires all streams to have an explicit max bytes defined ###### max_consumers The maximum number of consumers allowed. - [max_consumers](/reference/config/accounts/jetstream/max_consumers.md): The maximum number of consumers allowed. ###### max_file The maximum storage allowed across all file-based assets. - [max_file](/reference/config/accounts/jetstream/max_file.md): The maximum storage allowed across all file-based assets. ###### max_memory The maximum storage allowed across all memory-based assets. - [max_memory](/reference/config/accounts/jetstream/max_memory.md): The maximum storage allowed across all memory-based assets. ###### max_streams The maximum number of streams allowed. - [max_streams](/reference/config/accounts/jetstream/max_streams.md): The maximum number of streams allowed. ###### memory_max_stream_bytes Maximum bytes any given memory-based stream is allowed to be allocated. - [memory_max_stream_bytes](/reference/config/accounts/jetstream/memory_max_stream_bytes.md): Maximum bytes any given memory-based stream is allowed to be allocated. ##### limits Properties - [limits](/reference/config/accounts/limits.md): Properties ###### max_connections The maximum number of concurrent connections for this account. - [max_connections](/reference/config/accounts/limits/max_connections.md): The maximum number of concurrent connections for this account. ###### max_leafnodes The maximum number of concurrent leafnode connections allowed. - [max_leafnodes](/reference/config/accounts/limits/max_leafnodes.md): The maximum number of concurrent leafnode connections allowed. ###### max_payload The maximum payload size allowed for messages. - [max_payload](/reference/config/accounts/limits/max_payload.md): The maximum payload size allowed for messages. ###### max_subscriptions The maximum number of concurrent subscriptions for this account. - [max_subscriptions](/reference/config/accounts/limits/max_subscriptions.md): The maximum number of concurrent subscriptions for this account. ##### mappings Types - [mappings](/reference/config/accounts/mappings.md): Types ###### cluster If specified, the destination is cluster-scoped. Messages received - [cluster](/reference/config/accounts/mappings/cluster.md): If specified, the destination is cluster-scoped. Messages received ###### destination The subject mapping destination for the source subject. - [destination](/reference/config/accounts/mappings/destination.md): The subject mapping destination for the source subject. ###### weight A number between 0 and 100 (inclusive). The string form allows for - [weight](/reference/config/accounts/mappings/weight.md): A number between 0 and 100 (inclusive). The string form allows for ##### msg_trace Where this account's message traces are delivered, and how often - [msg_trace](/reference/config/accounts/msg_trace.md): Where this account's message traces are delivered, and how often ###### dest Subject the trace events are published to. - [dest](/reference/config/accounts/msg_trace/dest.md): Subject the trace events are published to. ###### sampling Percentage of traced messages to report, 1 to 100. - [sampling](/reference/config/accounts/msg_trace/sampling.md): Percentage of traced messages to report, 1 to 100. ##### nkey Public NKey that identifies this account (an A-prefixed public - [nkey](/reference/config/accounts/nkey.md): Public NKey that identifies this account (an A-prefixed public ##### users A list of users under this account. - [users](/reference/config/accounts/users.md): A list of users under this account. ###### allowed_connection_types If specified, the user is constrained to the specified connection - [allowed_connection_types](/reference/config/accounts/users/allowed_connection_types.md): If specified, the user is constrained to the specified connection ###### nkey Public NKey identifying the user. The value begins with a U - [nkey](/reference/config/accounts/users/nkey.md): Public NKey identifying the user. The value begins with a U ###### password Password of the user. This can be a free-text value - [password](/reference/config/accounts/users/password.md): Password of the user. This can be a free-text value ###### permissions Properties - [permissions](/reference/config/accounts/users/permissions.md): Properties - [allow_responses](/reference/config/accounts/users/permissions/allow_responses.md): Types - [expires](/reference/config/accounts/users/permissions/allow_responses/expires.md): The amount of time the permission is valid. Values such - [max](/reference/config/accounts/users/permissions/allow_responses/max.md): The maximum number of response messages that can be published. - [publish](/reference/config/accounts/users/permissions/publish.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/accounts/users/permissions/publish/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/accounts/users/permissions/publish/deny.md): List of subjects that are denied to the client. - [subscribe](/reference/config/accounts/users/permissions/subscribe.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/accounts/users/permissions/subscribe/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/accounts/users/permissions/subscribe/deny.md): List of subjects that are denied to the client. ###### username Name of the user. - [username](/reference/config/accounts/users/username.md): Name of the user. #### allow_non_tls Allow mixed TLS and non-TLS on the same port. - [allow_non_tls](/reference/config/allow_non_tls.md): Allow mixed TLS and non-TLS on the same port. #### authorization Static single or multi-user declaration. - [authorization](/reference/config/authorization.md): Static single or multi-user declaration. ##### auth_callout Enables the auth callout functionality. - [auth_callout](/reference/config/authorization/auth_callout.md): Enables the auth callout functionality. ###### account The name or public NKey of an account of the users which will - [account](/reference/config/authorization/auth_callout/account.md): The name or public NKey of an account of the users which will ###### issuer An account public NKey. - [issuer](/reference/config/authorization/auth_callout/issuer.md): An account public NKey. ###### key A public XKey that will encrypt server requests to the auth - [key](/reference/config/authorization/auth_callout/key.md): A public XKey that will encrypt server requests to the auth ###### users The names or public NKeys of users within the defined account - [users](/reference/config/authorization/auth_callout/users.md): The names or public NKeys of users within the defined account ##### default_permissions The default permissions applied to users, if permissions are - [default_permissions](/reference/config/authorization/default_permissions.md): The default permissions applied to users, if permissions are ###### allow_responses Types - [allow_responses](/reference/config/authorization/default_permissions/allow_responses.md): Types - [expires](/reference/config/authorization/default_permissions/allow_responses/expires.md): The amount of time the permission is valid. Values such - [max](/reference/config/authorization/default_permissions/allow_responses/max.md): The maximum number of response messages that can be published. ###### publish A single subject, list of subjects, or a allow-deny map of - [publish](/reference/config/authorization/default_permissions/publish.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/authorization/default_permissions/publish/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/authorization/default_permissions/publish/deny.md): List of subjects that are denied to the client. ###### subscribe A single subject, list of subjects, or a allow-deny map of - [subscribe](/reference/config/authorization/default_permissions/subscribe.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/authorization/default_permissions/subscribe/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/authorization/default_permissions/subscribe/deny.md): List of subjects that are denied to the client. ##### password Specifies a global password that clients can use to authenticate - [password](/reference/config/authorization/password.md): Specifies a global password that clients can use to authenticate ##### proxy_required Reject client connections that did not arrive through a PROXY - [proxy_required](/reference/config/authorization/proxy_required.md): Reject client connections that did not arrive through a PROXY ##### timeout Maximum number of seconds to wait for a client to authenticate. - [timeout](/reference/config/authorization/timeout.md): Maximum number of seconds to wait for a client to authenticate. ##### token Specifies a global token that clients can use to authenticate with - [token](/reference/config/authorization/token.md): Specifies a global token that clients can use to authenticate with ##### username Specifies a global user name that clients can use to authenticate - [username](/reference/config/authorization/username.md): Specifies a global user name that clients can use to authenticate ##### users A list of multiple users with different credentials. - [users](/reference/config/authorization/users.md): A list of multiple users with different credentials. ###### allowed_connection_types If specified, the user is constrained to the specified connection - [allowed_connection_types](/reference/config/authorization/users/allowed_connection_types.md): If specified, the user is constrained to the specified connection ###### nkey Public NKey identifying the user. The value begins with a U - [nkey](/reference/config/authorization/users/nkey.md): Public NKey identifying the user. The value begins with a U ###### password Password of the user. This can be a free-text value - [password](/reference/config/authorization/users/password.md): Password of the user. This can be a free-text value ###### permissions Properties - [permissions](/reference/config/authorization/users/permissions.md): Properties - [allow_responses](/reference/config/authorization/users/permissions/allow_responses.md): Types - [expires](/reference/config/authorization/users/permissions/allow_responses/expires.md): The amount of time the permission is valid. Values such - [max](/reference/config/authorization/users/permissions/allow_responses/max.md): The maximum number of response messages that can be published. - [publish](/reference/config/authorization/users/permissions/publish.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/authorization/users/permissions/publish/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/authorization/users/permissions/publish/deny.md): List of subjects that are denied to the client. - [subscribe](/reference/config/authorization/users/permissions/subscribe.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/authorization/users/permissions/subscribe/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/authorization/users/permissions/subscribe/deny.md): List of subjects that are denied to the client. ###### username Name of the user. - [username](/reference/config/authorization/users/username.md): Name of the user. #### client_advertise Advertised client :. Useful for cluster setups - [client_advertise](/reference/config/client_advertise.md): Advertised client :. Useful for cluster setups #### cluster Configuration for clustering a set of servers. - [cluster](/reference/config/cluster.md): Configuration for clustering a set of servers. ##### accounts A list of accounts to pin, each of which will have their own dedicated - [accounts](/reference/config/cluster/accounts.md): A list of accounts to pin, each of which will have their own dedicated ##### advertise Advertised cluster :. Useful for cluster setups since - [advertise](/reference/config/cluster/advertise.md): Advertised cluster :. Useful for cluster setups since ##### authorization Authorization map for configuring cluster routes. When a single username/password is used, it defines the authentication mechanism - [authorization](/reference/config/cluster/authorization.md): Authorization map for configuring cluster routes. When a single username/password is used, it defines the authentication mechanism ###### default_permissions The default permissions applied to users, if permissions are - [default_permissions](/reference/config/cluster/authorization/default_permissions.md): The default permissions applied to users, if permissions are - [allow_responses](/reference/config/cluster/authorization/default_permissions/allow_responses.md): Types - [expires](/reference/config/cluster/authorization/default_permissions/allow_responses/expires.md): The amount of time the permission is valid. Values such - [max](/reference/config/cluster/authorization/default_permissions/allow_responses/max.md): The maximum number of response messages that can be published. - [publish](/reference/config/cluster/authorization/default_permissions/publish.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/cluster/authorization/default_permissions/publish/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/cluster/authorization/default_permissions/publish/deny.md): List of subjects that are denied to the client. - [subscribe](/reference/config/cluster/authorization/default_permissions/subscribe.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/cluster/authorization/default_permissions/subscribe/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/cluster/authorization/default_permissions/subscribe/deny.md): List of subjects that are denied to the client. ###### password Password the connecting server authenticates with - [password](/reference/config/cluster/authorization/password.md): Password the connecting server authenticates with ###### timeout Maximum number of seconds to wait for a client to authenticate. - [timeout](/reference/config/cluster/authorization/timeout.md): Maximum number of seconds to wait for a client to authenticate. ###### username User name the connecting server authenticates with - [username](/reference/config/cluster/authorization/username.md): User name the connecting server authenticates with ##### compression Defines the type compression mode to use between routes. - [compression](/reference/config/cluster/compression.md): Defines the type compression mode to use between routes. ###### mode Types - [mode](/reference/config/cluster/compression/mode.md): Types ###### rtt_thresholds Applies only when the s2_auto mode is used and defines the round-trip time (RTT) - [rtt_thresholds](/reference/config/cluster/compression/rtt_thresholds.md): Applies only when the s2_auto mode is used and defines the round-trip time (RTT) ##### connect_backoff Back off between route reconnect attempts instead of retrying at a - [connect_backoff](/reference/config/cluster/connect_backoff.md): Back off between route reconnect attempts instead of retrying at a ##### connect_retries After how many failed connect attempts to give up establishing a connection to a discovered route. Default is 0, do not retry. - [connect_retries](/reference/config/cluster/connect_retries.md): After how many failed connect attempts to give up establishing a connection to a discovered route. Default is 0, do not retry. ##### host Host for cluster route connections. - [host](/reference/config/cluster/host.md): Host for cluster route connections. ##### listen Types - [listen](/reference/config/cluster/listen.md): Types ##### name Name of the cluster. - [name](/reference/config/cluster/name.md): Name of the cluster. ##### no_advertise If true, the server will not send or gossip its client URLs to other servers in the cluster, nor - [no_advertise](/reference/config/cluster/no_advertise.md): If true, the server will not send or gossip its client URLs to other servers in the cluster, nor ##### permissions Subject permissions applied to routes, limiting what this cluster - [permissions](/reference/config/cluster/permissions.md): Subject permissions applied to routes, limiting what this cluster ###### allow_responses Types - [allow_responses](/reference/config/cluster/permissions/allow_responses.md): Types - [expires](/reference/config/cluster/permissions/allow_responses/expires.md): The amount of time the permission is valid. Values such - [max](/reference/config/cluster/permissions/allow_responses/max.md): The maximum number of response messages that can be published. ###### publish A single subject, list of subjects, or a allow-deny map of - [publish](/reference/config/cluster/permissions/publish.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/cluster/permissions/publish/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/cluster/permissions/publish/deny.md): List of subjects that are denied to the client. ###### subscribe A single subject, list of subjects, or a allow-deny map of - [subscribe](/reference/config/cluster/permissions/subscribe.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/cluster/permissions/subscribe/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/cluster/permissions/subscribe/deny.md): List of subjects that are denied to the client. ##### ping_interval How often the server pings an idle route. - [ping_interval](/reference/config/cluster/ping_interval.md): How often the server pings an idle route. ##### ping_max Unanswered pings before the route is closed. - [ping_max](/reference/config/cluster/ping_max.md): Unanswered pings before the route is closed. ##### pool_size The size of the connection pool used to distribute load across non-pinned - [pool_size](/reference/config/cluster/pool_size.md): The size of the connection pool used to distribute load across non-pinned ##### port Port for cluster route connections. - [port](/reference/config/cluster/port.md): Port for cluster route connections. ##### routes A list of server URLs to cluster with. Self-routes are ignored. Should authentication via token or username/password - [routes](/reference/config/cluster/routes.md): A list of server URLs to cluster with. Self-routes are ignored. Should authentication via token or username/password ##### tls TLS configuration for securing cluster connections. - [tls](/reference/config/cluster/tls.md): TLS configuration for securing cluster connections. ###### allow_insecure_cipher_suites Permit cipher suites Go considers insecure to be named in - [allow_insecure_cipher_suites](/reference/config/cluster/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in ###### ca_file TLS certificate authority file. Defaults to system trust store. - [ca_file](/reference/config/cluster/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. ###### cert_file TLS certificate file. - [cert_file](/reference/config/cluster/tls/cert_file.md): TLS certificate file. ###### cert_store Load the certificate from an OS certificate store rather than a - [cert_store](/reference/config/cluster/tls/cert_store.md): Load the certificate from an OS certificate store rather than a ###### certs Multiple certificate/key pairs to serve, so one listener can - [certs](/reference/config/cluster/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can - [cert_file](/reference/config/cluster/tls/certs/cert_file.md): Certificate file for this pair. - [key_file](/reference/config/cluster/tls/certs/key_file.md): Key file for this pair. ###### cipher_suites When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [cipher_suites](/reference/config/cluster/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. ###### connection_rate_limit Types - [connection_rate_limit](/reference/config/cluster/tls/connection_rate_limit.md): Types ###### curve_preferences List of TLS cipher curves to use in order. - [curve_preferences](/reference/config/cluster/tls/curve_preferences.md): List of TLS cipher curves to use in order. ###### handshake_first Send the TLS handshake before the INFO protocol message rather - [handshake_first](/reference/config/cluster/tls/handshake_first.md): Send the TLS handshake before the INFO protocol message rather ###### insecure Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [insecure](/reference/config/cluster/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. ###### key_file TLS certificate key file. - [key_file](/reference/config/cluster/tls/key_file.md): TLS certificate key file. ###### min_version Lowest TLS version the server will negotiate. - [min_version](/reference/config/cluster/tls/min_version.md): Lowest TLS version the server will negotiate. ###### ocsp_peer Verify the peer's certificate against its OCSP responder and reject - [ocsp_peer](/reference/config/cluster/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject - [allowed_clockskew](/reference/config/cluster/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. - [ca_timeout](/reference/config/cluster/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. - [cache_ttl_when_next_update_unset](/reference/config/cluster/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. - [unknown_is_good](/reference/config/cluster/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. - [verify](/reference/config/cluster/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. - [warn_only](/reference/config/cluster/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. ###### pinned_certs List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [pinned_certs](/reference/config/cluster/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the ###### timeout TLS handshake timeout. - [timeout](/reference/config/cluster/tls/timeout.md): TLS handshake timeout. ###### verify If true, require and verify client certificates. Does not apply to monitoring. - [verify](/reference/config/cluster/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. ###### verify_and_map If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_and_map](/reference/config/cluster/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ###### verify_cert_and_check_known_urls Only used in a non-client context where verify is true, such as cluster and gateway configurations. - [verify_cert_and_check_known_urls](/reference/config/cluster/tls/verify_cert_and_check_known_urls.md): Only used in a non-client context where verify is true, such as cluster and gateway configurations. ##### write_deadline How long a route write may block before the connection is treated - [write_deadline](/reference/config/cluster/write_deadline.md): How long a route write may block before the connection is treated ##### write_timeout What to do when a route misses its write_deadline. - [write_timeout](/reference/config/cluster/write_timeout.md): What to do when a route misses its write_deadline. #### connect_error_reports Number of attempts at which a repeated failed route, gateway - [connect_error_reports](/reference/config/connect_error_reports.md): Number of attempts at which a repeated failed route, gateway #### debug If true, enables debug log messages. - [debug](/reference/config/debug.md): If true, enables debug log messages. #### default_js_domain Account to domain name mapping. - [default_js_domain](/reference/config/default_js_domain.md): Account to domain name mapping. #### default_sentinel User name applied to connections that authenticate with a sentinel - [default_sentinel](/reference/config/default_sentinel.md): User name applied to connections that authenticate with a sentinel #### disable_sublist_cache If true, disable subscription caches for all accounts. This saves - [disable_sublist_cache](/reference/config/disable_sublist_cache.md): If true, disable subscription caches for all accounts. This saves #### feature_flags Toggles for features that are not yet on by default. Names are - [feature_flags](/reference/config/feature_flags.md): Toggles for features that are not yet on by default. Names are #### gateway Configuration for setting up gateway connections - [gateway](/reference/config/gateway.md): Configuration for setting up gateway connections ##### advertise : to advertise how this server can be contacted by - [advertise](/reference/config/gateway/advertise.md): : to advertise how this server can be contacted by ##### authorization Authorization map for gateways. When a single username/password is - [authorization](/reference/config/gateway/authorization.md): Authorization map for gateways. When a single username/password is ###### default_permissions The default permissions applied to users, if permissions are - [default_permissions](/reference/config/gateway/authorization/default_permissions.md): The default permissions applied to users, if permissions are - [allow_responses](/reference/config/gateway/authorization/default_permissions/allow_responses.md): Types - [expires](/reference/config/gateway/authorization/default_permissions/allow_responses/expires.md): The amount of time the permission is valid. Values such - [max](/reference/config/gateway/authorization/default_permissions/allow_responses/max.md): The maximum number of response messages that can be published. - [publish](/reference/config/gateway/authorization/default_permissions/publish.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/gateway/authorization/default_permissions/publish/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/gateway/authorization/default_permissions/publish/deny.md): List of subjects that are denied to the client. - [subscribe](/reference/config/gateway/authorization/default_permissions/subscribe.md): A single subject, list of subjects, or a allow-deny map of - [allow](/reference/config/gateway/authorization/default_permissions/subscribe/allow.md): List of subjects that are allowed to the client. - [deny](/reference/config/gateway/authorization/default_permissions/subscribe/deny.md): List of subjects that are denied to the client. ###### password Password the connecting server authenticates with - [password](/reference/config/gateway/authorization/password.md): Password the connecting server authenticates with ###### timeout Maximum number of seconds to wait for a client to authenticate. - [timeout](/reference/config/gateway/authorization/timeout.md): Maximum number of seconds to wait for a client to authenticate. ###### username User name the connecting server authenticates with - [username](/reference/config/gateway/authorization/username.md): User name the connecting server authenticates with ##### connect_backoff Back off between gateway reconnect attempts instead of retrying at - [connect_backoff](/reference/config/gateway/connect_backoff.md): Back off between gateway reconnect attempts instead of retrying at ##### connect_retries After how many failed connect attempts to give up establishing - [connect_retries](/reference/config/gateway/connect_retries.md): After how many failed connect attempts to give up establishing ##### gateways List of gateway entries. - [gateways](/reference/config/gateway/gateways.md): List of gateway entries. ###### name Name of the gateway being connected to. - [name](/reference/config/gateway/gateways/name.md): Name of the gateway being connected to. ###### tls A TLS configuration map for creating a secure gateway connection. - [tls](/reference/config/gateway/gateways/tls.md): A TLS configuration map for creating a secure gateway connection. - [allow_insecure_cipher_suites](/reference/config/gateway/gateways/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in - [ca_file](/reference/config/gateway/gateways/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. - [cert_file](/reference/config/gateway/gateways/tls/cert_file.md): TLS certificate file. - [cert_store](/reference/config/gateway/gateways/tls/cert_store.md): Load the certificate from an OS certificate store rather than a - [certs](/reference/config/gateway/gateways/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can - [cert_file](/reference/config/gateway/gateways/tls/certs/cert_file.md): Certificate file for this pair. - [key_file](/reference/config/gateway/gateways/tls/certs/key_file.md): Key file for this pair. - [cipher_suites](/reference/config/gateway/gateways/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [connection_rate_limit](/reference/config/gateway/gateways/tls/connection_rate_limit.md): Types - [curve_preferences](/reference/config/gateway/gateways/tls/curve_preferences.md): List of TLS cipher curves to use in order. - [handshake_first](/reference/config/gateway/gateways/tls/handshake_first.md): Send the TLS handshake before the INFO protocol message rather - [insecure](/reference/config/gateway/gateways/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [key_file](/reference/config/gateway/gateways/tls/key_file.md): TLS certificate key file. - [min_version](/reference/config/gateway/gateways/tls/min_version.md): Lowest TLS version the server will negotiate. - [ocsp_peer](/reference/config/gateway/gateways/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject - [allowed_clockskew](/reference/config/gateway/gateways/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. - [ca_timeout](/reference/config/gateway/gateways/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. - [cache_ttl_when_next_update_unset](/reference/config/gateway/gateways/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. - [unknown_is_good](/reference/config/gateway/gateways/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. - [verify](/reference/config/gateway/gateways/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. - [warn_only](/reference/config/gateway/gateways/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. - [pinned_certs](/reference/config/gateway/gateways/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [timeout](/reference/config/gateway/gateways/tls/timeout.md): TLS handshake timeout. - [verify](/reference/config/gateway/gateways/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. - [verify_and_map](/reference/config/gateway/gateways/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_cert_and_check_known_urls](/reference/config/gateway/gateways/tls/verify_cert_and_check_known_urls.md): Only used in a non-client context where verify is true, such as cluster and gateway configurations. ###### url A single URL to connect to. - [url](/reference/config/gateway/gateways/url.md): A single URL to connect to. ###### urls A list of URLs to connect to (multiple servers in a cluster). - [urls](/reference/config/gateway/gateways/urls.md): A list of URLs to connect to (multiple servers in a cluster). ##### host Interface where the gateway will listen for incoming gateway - [host](/reference/config/gateway/host.md): Interface where the gateway will listen for incoming gateway ##### listen : format. Alternative to host/port. - [listen](/reference/config/gateway/listen.md): : format. Alternative to host/port. ##### name Name of this cluster. All gateway connections belonging to the - [name](/reference/config/gateway/name.md): Name of this cluster. All gateway connections belonging to the ##### port Port where the gateway will listen for incoming gateway connections. - [port](/reference/config/gateway/port.md): Port where the gateway will listen for incoming gateway connections. ##### reject_unknown_cluster If true, gateway will reject connections from cluster that are - [reject_unknown_cluster](/reference/config/gateway/reject_unknown_cluster.md): If true, gateway will reject connections from cluster that are ##### tls A tls configuration map for securing gateway connections. verify - [tls](/reference/config/gateway/tls.md): A tls configuration map for securing gateway connections. verify ###### allow_insecure_cipher_suites Permit cipher suites Go considers insecure to be named in - [allow_insecure_cipher_suites](/reference/config/gateway/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in ###### ca_file TLS certificate authority file. Defaults to system trust store. - [ca_file](/reference/config/gateway/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. ###### cert_file TLS certificate file. - [cert_file](/reference/config/gateway/tls/cert_file.md): TLS certificate file. ###### cert_store Load the certificate from an OS certificate store rather than a - [cert_store](/reference/config/gateway/tls/cert_store.md): Load the certificate from an OS certificate store rather than a ###### certs Multiple certificate/key pairs to serve, so one listener can - [certs](/reference/config/gateway/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can - [cert_file](/reference/config/gateway/tls/certs/cert_file.md): Certificate file for this pair. - [key_file](/reference/config/gateway/tls/certs/key_file.md): Key file for this pair. ###### cipher_suites When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [cipher_suites](/reference/config/gateway/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. ###### connection_rate_limit Types - [connection_rate_limit](/reference/config/gateway/tls/connection_rate_limit.md): Types ###### curve_preferences List of TLS cipher curves to use in order. - [curve_preferences](/reference/config/gateway/tls/curve_preferences.md): List of TLS cipher curves to use in order. ###### handshake_first Send the TLS handshake before the INFO protocol message rather - [handshake_first](/reference/config/gateway/tls/handshake_first.md): Send the TLS handshake before the INFO protocol message rather ###### insecure Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [insecure](/reference/config/gateway/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. ###### key_file TLS certificate key file. - [key_file](/reference/config/gateway/tls/key_file.md): TLS certificate key file. ###### min_version Lowest TLS version the server will negotiate. - [min_version](/reference/config/gateway/tls/min_version.md): Lowest TLS version the server will negotiate. ###### ocsp_peer Verify the peer's certificate against its OCSP responder and reject - [ocsp_peer](/reference/config/gateway/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject - [allowed_clockskew](/reference/config/gateway/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. - [ca_timeout](/reference/config/gateway/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. - [cache_ttl_when_next_update_unset](/reference/config/gateway/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. - [unknown_is_good](/reference/config/gateway/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. - [verify](/reference/config/gateway/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. - [warn_only](/reference/config/gateway/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. ###### pinned_certs List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [pinned_certs](/reference/config/gateway/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the ###### timeout TLS handshake timeout. - [timeout](/reference/config/gateway/tls/timeout.md): TLS handshake timeout. ###### verify If true, require and verify client certificates. Does not apply to monitoring. - [verify](/reference/config/gateway/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. ###### verify_and_map If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_and_map](/reference/config/gateway/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ###### verify_cert_and_check_known_urls Only used in a non-client context where verify is true, such as cluster and gateway configurations. - [verify_cert_and_check_known_urls](/reference/config/gateway/tls/verify_cert_and_check_known_urls.md): Only used in a non-client context where verify is true, such as cluster and gateway configurations. ##### write_deadline How long a gateway write may block before the connection is - [write_deadline](/reference/config/gateway/write_deadline.md): How long a gateway write may block before the connection is ##### write_timeout What to do when a gateway connection misses its write_deadline. - [write_timeout](/reference/config/gateway/write_timeout.md): What to do when a gateway connection misses its write_deadline. #### host Host for client connections. - [host](/reference/config/host.md): Host for client connections. #### http Listen specification : for server monitoring. - [http](/reference/config/http.md): Listen specification : for server monitoring. #### http_base_path Base path for monitoring endpoints. - [http_base_path](/reference/config/http_base_path.md): Base path for monitoring endpoints. #### http_port HTTP port for server monitoring. - [http_port](/reference/config/http_port.md): HTTP port for server monitoring. #### https Listen specification : for TLS server monitoring. - [https](/reference/config/https.md): Listen specification : for TLS server monitoring. #### https_port HTTPS port for server monitoring. - [https_port](/reference/config/https_port.md): HTTPS port for server monitoring. #### jetstream Types - [jetstream](/reference/config/jetstream.md): Types ##### cipher Defines the encryption algorithm to use if an encryption key is defined. - [cipher](/reference/config/jetstream/cipher.md): Defines the encryption algorithm to use if an encryption key is defined. ##### domain The JetStream domain the server is part of. - [domain](/reference/config/jetstream/domain.md): The JetStream domain the server is part of. ##### enabled If true, enables the JetStream subsystem. - [enabled](/reference/config/jetstream/enabled.md): If true, enables the JetStream subsystem. ##### encryption_key If defined, enables JetStream filestore encryption using the - [encryption_key](/reference/config/jetstream/encryption_key.md): If defined, enables JetStream filestore encryption using the ##### extension_hint Types - [extension_hint](/reference/config/jetstream/extension_hint.md): Types ##### info_queue_limit Queued stream-info requests allowed before new ones are rejected. - [info_queue_limit](/reference/config/jetstream/info_queue_limit.md): Queued stream-info requests allowed before new ones are rejected. ##### limits Default cross-account JetStream limits. - [limits](/reference/config/jetstream/limits.md): Default cross-account JetStream limits. ###### batch Ceilings on atomic batch publishing. - [batch](/reference/config/jetstream/limits/batch.md): Ceilings on atomic batch publishing. - [max_inflight_per_stream](/reference/config/jetstream/limits/batch/max_inflight_per_stream.md): Batches allowed in flight for one stream. - [max_inflight_total](/reference/config/jetstream/limits/batch/max_inflight_total.md): Batches allowed in flight across the server. - [max_msgs](/reference/config/jetstream/limits/batch/max_msgs.md): Messages allowed in a single batch. - [timeout](/reference/config/jetstream/limits/batch/timeout.md): How long an incomplete batch is held before it is discarded. ###### duplicate_window The maximum duplication window period allowed to be configured on - [duplicate_window](/reference/config/jetstream/limits/duplicate_window.md): The maximum duplication window period allowed to be configured on ###### max_ack_pending Defines the maximum number of in-flight messages allowed to be - [max_ack_pending](/reference/config/jetstream/limits/max_ack_pending.md): Defines the maximum number of in-flight messages allowed to be ###### max_ha_assets The maximum number of JetStream assets that can exist at any given - [max_ha_assets](/reference/config/jetstream/limits/max_ha_assets.md): The maximum number of JetStream assets that can exist at any given ###### max_request_batch The maximum request batch size allowed to be configured on pull - [max_request_batch](/reference/config/jetstream/limits/max_request_batch.md): The maximum request batch size allowed to be configured on pull ##### max_buffered_msgs Messages the server buffers for a stream whose storage is - [max_buffered_msgs](/reference/config/jetstream/max_buffered_msgs.md): Messages the server buffers for a stream whose storage is ##### max_buffered_size Byte ceiling for that same buffer. - [max_buffered_size](/reference/config/jetstream/max_buffered_size.md): Byte ceiling for that same buffer. ##### max_file_store Maximum size of the file storage. - [max_file_store](/reference/config/jetstream/max_file_store.md): Maximum size of the file storage. ##### max_memory_store Maximum size of the memory storage. - [max_memory_store](/reference/config/jetstream/max_memory_store.md): Maximum size of the memory storage. ##### max_outstanding_catchup Max in-flight bytes for stream catch-up. This was introduced to - [max_outstanding_catchup](/reference/config/jetstream/max_outstanding_catchup.md): Max in-flight bytes for stream catch-up. This was introduced to ##### meta_compact Metalayer Raft entries to accumulate before the log is compacted. - [meta_compact](/reference/config/jetstream/meta_compact.md): Metalayer Raft entries to accumulate before the log is compacted. ##### meta_compact_size Metalayer Raft log size that triggers compaction. - [meta_compact_size](/reference/config/jetstream/meta_compact_size.md): Metalayer Raft log size that triggers compaction. ##### meta_compact_sync Compact the metalayer Raft log synchronously rather than in the - [meta_compact_sync](/reference/config/jetstream/meta_compact_sync.md): Compact the metalayer Raft log synchronously rather than in the ##### prev_encryption_key Previous encryption key, kept so existing filestore data can be - [prev_encryption_key](/reference/config/jetstream/prev_encryption_key.md): Previous encryption key, kept so existing filestore data can be ##### request_queue_limit Queued JetStream API requests allowed before new ones are - [request_queue_limit](/reference/config/jetstream/request_queue_limit.md): Queued JetStream API requests allowed before new ones are ##### store_dir Directory to use for file-based storage. - [store_dir](/reference/config/jetstream/store_dir.md): Directory to use for file-based storage. ##### strict Reject API requests containing unknown fields instead of ignoring - [strict](/reference/config/jetstream/strict.md): Reject API requests containing unknown fields instead of ignoring ##### sync_interval Defines the internal to force sync file-based stream and consumer data - [sync_interval](/reference/config/jetstream/sync_interval.md): Defines the internal to force sync file-based stream and consumer data ##### tpm Seal the filestore encryption key in the machine's TPM, so it - [tpm](/reference/config/jetstream/tpm.md): Seal the filestore encryption key in the machine's TPM, so it ###### cipher Cipher used for the filestore once the key is unsealed. - [cipher](/reference/config/jetstream/tpm/cipher.md): Cipher used for the filestore once the key is unsealed. ###### encryption_password Password protecting the sealed key. - [encryption_password](/reference/config/jetstream/tpm/encryption_password.md): Password protecting the sealed key. ###### keys_file File the TPM-sealed key is stored in. - [keys_file](/reference/config/jetstream/tpm/keys_file.md): File the TPM-sealed key is stored in. ###### pcr Platform Configuration Register the key is sealed against, so it - [pcr](/reference/config/jetstream/tpm/pcr.md): Platform Configuration Register the key is sealed against, so it ###### srk_password Password for the TPM storage root key. - [srk_password](/reference/config/jetstream/tpm/srk_password.md): Password for the TPM storage root key. ##### unique_tag Defines a tag prefix as a constraint for placement of assets across - [unique_tag](/reference/config/jetstream/unique_tag.md): Defines a tag prefix as a constraint for placement of assets across #### lame_duck_duration Must be at least 30s. - [lame_duck_duration](/reference/config/lame_duck_duration.md): Must be at least 30s. #### lame_duck_grace_period This is the duration the server waits, after entering - [lame_duck_grace_period](/reference/config/lame_duck_grace_period.md): This is the duration the server waits, after entering #### leafnodes Configuration for setting up leaf node connections. - [leafnodes](/reference/config/leafnodes.md): Configuration for setting up leaf node connections. ##### advertise Hostport to advertise how this sever be contacted - [advertise](/reference/config/leafnodes/advertise.md): Hostport to advertise how this sever be contacted ##### authorization Authorization scoped to accepting leaf node connections. - [authorization](/reference/config/leafnodes/authorization.md): Authorization scoped to accepting leaf node connections. ###### account Account that leaf nodes authenticating with these - [account](/reference/config/leafnodes/authorization/account.md): Account that leaf nodes authenticating with these ###### nkey Public user nkey a connecting leaf node must sign for. - [nkey](/reference/config/leafnodes/authorization/nkey.md): Public user nkey a connecting leaf node must sign for. ###### password Password the connecting server authenticates with - [password](/reference/config/leafnodes/authorization/password.md): Password the connecting server authenticates with ###### proxy_required Reject leaf node connections that did not arrive through a - [proxy_required](/reference/config/leafnodes/authorization/proxy_required.md): Reject leaf node connections that did not arrive through a ###### timeout Maximum number of seconds to wait for a client to authenticate. - [timeout](/reference/config/leafnodes/authorization/timeout.md): Maximum number of seconds to wait for a client to authenticate. ###### username User name the connecting server authenticates with - [username](/reference/config/leafnodes/authorization/username.md): User name the connecting server authenticates with ###### users A list of multiple users with different credentials. - [users](/reference/config/leafnodes/authorization/users.md): A list of multiple users with different credentials. - [password](/reference/config/leafnodes/authorization/users/password.md): Password of the user. This can be a free-text value - [username](/reference/config/leafnodes/authorization/users/username.md): Name of the user. ##### compression on, ->accept) are closed and reconnect." /> - [compression](/reference/config/leafnodes/compression.md): on, ->accept) are closed and reconnect." /> ###### mode Types - [mode](/reference/config/leafnodes/compression/mode.md): Types ###### rtt_thresholds Applies only when the s2_auto mode is used and defines the round-trip time (RTT) - [rtt_thresholds](/reference/config/leafnodes/compression/rtt_thresholds.md): Applies only when the s2_auto mode is used and defines the round-trip time (RTT) ##### host Host name the server will listen on for incoming - [host](/reference/config/leafnodes/host.md): Host name the server will listen on for incoming ##### isolate_leafnode_interest Do not propagate subscription interest learned from one leaf - [isolate_leafnode_interest](/reference/config/leafnodes/isolate_leafnode_interest.md): Do not propagate subscription interest learned from one leaf ##### listen Types - [listen](/reference/config/leafnodes/listen.md): Types ##### min_version The minimum server version required of the connecting - [min_version](/reference/config/leafnodes/min_version.md): The minimum server version required of the connecting ##### no_advertise If true, the server will not be advertised to leaf nodes. - [no_advertise](/reference/config/leafnodes/no_advertise.md): If true, the server will not be advertised to leaf nodes. ##### port Port the server will listen for incoming leaf node - [port](/reference/config/leafnodes/port.md): Port the server will listen for incoming leaf node ##### reconnect Interval in seconds at which reconnect attempts to a - [reconnect](/reference/config/leafnodes/reconnect.md): Interval in seconds at which reconnect attempts to a ##### remotes List of entries specifiying servers where the leaf - [remotes](/reference/config/leafnodes/remotes.md): List of entries specifiying servers where the leaf ###### account Account name or public NKey identifying the local account - [account](/reference/config/leafnodes/remotes/account.md): Account name or public NKey identifying the local account ###### compression on, ->accept) are closed and reconnect." /> - [compression](/reference/config/leafnodes/remotes/compression.md): on, ->accept) are closed and reconnect." /> - [mode](/reference/config/leafnodes/remotes/compression/mode.md): Types - [rtt_thresholds](/reference/config/leafnodes/remotes/compression/rtt_thresholds.md): Applies only when the s2_auto mode is used and defines the round-trip time (RTT) ###### credentials Path to a credentials file. This is application when - [credentials](/reference/config/leafnodes/remotes/credentials.md): Path to a credentials file. This is application when ###### deny_exports Types - [deny_exports](/reference/config/leafnodes/remotes/deny_exports.md): Types ###### deny_imports Types - [deny_imports](/reference/config/leafnodes/remotes/deny_imports.md): Types ###### disabled Keep the remote in the configuration without connecting to it. - [disabled](/reference/config/leafnodes/remotes/disabled.md): Keep the remote in the configuration without connecting to it. ###### first_info_timeout How long to wait for the remote's INFO message before giving up - [first_info_timeout](/reference/config/leafnodes/remotes/first_info_timeout.md): How long to wait for the remote's INFO message before giving up ###### hub Types - [hub](/reference/config/leafnodes/remotes/hub.md): Types ###### ignore_discovered_servers Connect only to the URLs listed here, ignoring any additional - [ignore_discovered_servers](/reference/config/leafnodes/remotes/ignore_discovered_servers.md): Connect only to the URLs listed here, ignoring any additional ###### isolate_leafnode_interest Do not propagate subscription interest learned from this remote to - [isolate_leafnode_interest](/reference/config/leafnodes/remotes/isolate_leafnode_interest.md): Do not propagate subscription interest learned from this remote to ###### jetstream_cluster_migrate Types - [jetstream_cluster_migrate](/reference/config/leafnodes/remotes/jetstream_cluster_migrate.md): Types ###### nkey User nkey seed this remote authenticates with. - [nkey](/reference/config/leafnodes/remotes/nkey.md): User nkey seed this remote authenticates with. ###### no_randomize If true and more than one URL is specified, the first one - [no_randomize](/reference/config/leafnodes/remotes/no_randomize.md): If true and more than one URL is specified, the first one ###### proxy Reach the remote through an HTTP proxy. - [proxy](/reference/config/leafnodes/remotes/proxy.md): Reach the remote through an HTTP proxy. - [password](/reference/config/leafnodes/remotes/proxy/password.md): Password for proxy authentication. - [timeout](/reference/config/leafnodes/remotes/proxy/timeout.md): How long to wait for the proxy to establish the tunnel. - [url](/reference/config/leafnodes/remotes/proxy/url.md): Proxy URL, e.g. http3128. - [username](/reference/config/leafnodes/remotes/proxy/username.md): User name for proxy authentication. ###### request_isolation Ask the remote to isolate this connection's interest on its side. - [request_isolation](/reference/config/leafnodes/remotes/request_isolation.md): Ask the remote to isolate this connection's interest on its side. ###### tls TLS configuration for connecting/authenticating with - [tls](/reference/config/leafnodes/remotes/tls.md): TLS configuration for connecting/authenticating with - [allow_insecure_cipher_suites](/reference/config/leafnodes/remotes/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in - [ca_file](/reference/config/leafnodes/remotes/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. - [cert_file](/reference/config/leafnodes/remotes/tls/cert_file.md): TLS certificate file. - [cert_store](/reference/config/leafnodes/remotes/tls/cert_store.md): Load the certificate from an OS certificate store rather than a - [certs](/reference/config/leafnodes/remotes/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can - [cert_file](/reference/config/leafnodes/remotes/tls/certs/cert_file.md): Certificate file for this pair. - [key_file](/reference/config/leafnodes/remotes/tls/certs/key_file.md): Key file for this pair. - [cipher_suites](/reference/config/leafnodes/remotes/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [connection_rate_limit](/reference/config/leafnodes/remotes/tls/connection_rate_limit.md): Types - [curve_preferences](/reference/config/leafnodes/remotes/tls/curve_preferences.md): List of TLS cipher curves to use in order. - [handshake_first](/reference/config/leafnodes/remotes/tls/handshake_first.md): Force the leafnode connection to use a TLS-first handshake prior - [insecure](/reference/config/leafnodes/remotes/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [key_file](/reference/config/leafnodes/remotes/tls/key_file.md): TLS certificate key file. - [min_version](/reference/config/leafnodes/remotes/tls/min_version.md): Lowest TLS version the server will negotiate. - [ocsp_peer](/reference/config/leafnodes/remotes/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject - [allowed_clockskew](/reference/config/leafnodes/remotes/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. - [ca_timeout](/reference/config/leafnodes/remotes/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. - [cache_ttl_when_next_update_unset](/reference/config/leafnodes/remotes/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. - [unknown_is_good](/reference/config/leafnodes/remotes/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. - [verify](/reference/config/leafnodes/remotes/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. - [warn_only](/reference/config/leafnodes/remotes/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. - [pinned_certs](/reference/config/leafnodes/remotes/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [timeout](/reference/config/leafnodes/remotes/tls/timeout.md): TLS handshake timeout. - [verify](/reference/config/leafnodes/remotes/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. - [verify_and_map](/reference/config/leafnodes/remotes/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ###### url URL or list of URLs of a remote server accepting leaf - [url](/reference/config/leafnodes/remotes/url.md): URL or list of URLs of a remote server accepting leaf ###### ws_compression If true, and connecting with the WebSocket protocol, - [ws_compression](/reference/config/leafnodes/remotes/ws_compression.md): If true, and connecting with the WebSocket protocol, ###### ws_no_masking If true and connecting with the WebSocket protocol, - [ws_no_masking](/reference/config/leafnodes/remotes/ws_no_masking.md): If true and connecting with the WebSocket protocol, ##### tls TLS configuration for securing leaf node connections. - [tls](/reference/config/leafnodes/tls.md): TLS configuration for securing leaf node connections. ###### allow_insecure_cipher_suites Permit cipher suites Go considers insecure to be named in - [allow_insecure_cipher_suites](/reference/config/leafnodes/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in ###### ca_file TLS certificate authority file. Defaults to system trust store. - [ca_file](/reference/config/leafnodes/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. ###### cert_file TLS certificate file. - [cert_file](/reference/config/leafnodes/tls/cert_file.md): TLS certificate file. ###### cert_store Load the certificate from an OS certificate store rather than a - [cert_store](/reference/config/leafnodes/tls/cert_store.md): Load the certificate from an OS certificate store rather than a ###### certs Multiple certificate/key pairs to serve, so one listener can - [certs](/reference/config/leafnodes/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can - [cert_file](/reference/config/leafnodes/tls/certs/cert_file.md): Certificate file for this pair. - [key_file](/reference/config/leafnodes/tls/certs/key_file.md): Key file for this pair. ###### cipher_suites When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [cipher_suites](/reference/config/leafnodes/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. ###### connection_rate_limit Types - [connection_rate_limit](/reference/config/leafnodes/tls/connection_rate_limit.md): Types ###### curve_preferences List of TLS cipher curves to use in order. - [curve_preferences](/reference/config/leafnodes/tls/curve_preferences.md): List of TLS cipher curves to use in order. ###### handshake_first Force the leafnode connection to use a TLS-first handshake prior - [handshake_first](/reference/config/leafnodes/tls/handshake_first.md): Force the leafnode connection to use a TLS-first handshake prior ###### insecure Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [insecure](/reference/config/leafnodes/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. ###### key_file TLS certificate key file. - [key_file](/reference/config/leafnodes/tls/key_file.md): TLS certificate key file. ###### min_version Lowest TLS version the server will negotiate. - [min_version](/reference/config/leafnodes/tls/min_version.md): Lowest TLS version the server will negotiate. ###### ocsp_peer Verify the peer's certificate against its OCSP responder and reject - [ocsp_peer](/reference/config/leafnodes/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject - [allowed_clockskew](/reference/config/leafnodes/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. - [ca_timeout](/reference/config/leafnodes/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. - [cache_ttl_when_next_update_unset](/reference/config/leafnodes/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. - [unknown_is_good](/reference/config/leafnodes/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. - [verify](/reference/config/leafnodes/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. - [warn_only](/reference/config/leafnodes/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. ###### pinned_certs List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [pinned_certs](/reference/config/leafnodes/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the ###### timeout TLS handshake timeout. - [timeout](/reference/config/leafnodes/tls/timeout.md): TLS handshake timeout. ###### verify If true, require and verify client certificates. Does not apply to monitoring. - [verify](/reference/config/leafnodes/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. ###### verify_and_map If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_and_map](/reference/config/leafnodes/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ##### write_deadline How long a leaf node write may block before the connection is - [write_deadline](/reference/config/leafnodes/write_deadline.md): How long a leaf node write may block before the connection is ##### write_timeout What to do when a leaf node connection misses its - [write_timeout](/reference/config/leafnodes/write_timeout.md): What to do when a leaf node connection misses its #### listen : for a client connections. - [listen](/reference/config/listen.md): : for a client connections. #### logfile Log file name. - [logfile](/reference/config/logfile.md): Log file name. #### logfile_max_num Maximum number of rotated log files to keep. Older ones are - [logfile_max_num](/reference/config/logfile_max_num.md): Maximum number of rotated log files to keep. Older ones are #### logfile_size_limit Size in bytes after the log file rolls over to a new one. - [logfile_size_limit](/reference/config/logfile_size_limit.md): Size in bytes after the log file rolls over to a new one. #### logtime If false, log without timestamps. - [logtime](/reference/config/logtime.md): If false, log without timestamps. #### logtime_utc If true, log timestamps with be in UTC rather than the local timezone. - [logtime_utc](/reference/config/logtime_utc.md): If true, log timestamps with be in UTC rather than the local timezone. #### mappings Types - [mappings](/reference/config/mappings.md): Types ##### cluster If specified, the destination is cluster-scoped. Messages received - [cluster](/reference/config/mappings/cluster.md): If specified, the destination is cluster-scoped. Messages received ##### destination The subject mapping destination for the source subject. - [destination](/reference/config/mappings/destination.md): The subject mapping destination for the source subject. ##### weight A number between 0 and 100 (inclusive). The string form allows for - [weight](/reference/config/mappings/weight.md): A number between 0 and 100 (inclusive). The string form allows for #### max_closed_clients How many closed connections the server keeps for /connz - [max_closed_clients](/reference/config/max_closed_clients.md): How many closed connections the server keeps for /connz #### max_connections Maximum number of active client connections. - [max_connections](/reference/config/max_connections.md): Maximum number of active client connections. #### max_control_line Maximum length of a protocol line (including combined length of subject and queue group). Increasing this value may require client changes to be used. Applies to all traffic. - [max_control_line](/reference/config/max_control_line.md): Maximum length of a protocol line (including combined length of subject and queue group). Increasing this value may require client changes to be used. Applies to all traffic. #### max_payload Maximum number of bytes in a message payload. Reducing this size may force you to implement chunking in your clients. Applies to client and leafnode payloads. It is not recommended to use values over 8MB but maxpayload can be set up to 64MB. The max payload must be equal or smaller to the maxpending value. - [max_payload](/reference/config/max_payload.md): Maximum number of bytes in a message payload. Reducing this size may force you to implement chunking in your clients. Applies to client and leafnode payloads. It is not recommended to use values over 8MB but maxpayload can be set up to 64MB. The max payload must be equal or smaller to the maxpending value. #### max_pending Maximum number of bytes buffered for a connection Applies to client connections. Note that applications can also set PendingLimits (number of messages and total size) for their subscriptions. - [max_pending](/reference/config/max_pending.md): Maximum number of bytes buffered for a connection Applies to client connections. Note that applications can also set PendingLimits (number of messages and total size) for their subscriptions. #### max_subscription_tokens Types - [max_subscription_tokens](/reference/config/max_subscription_tokens.md): Types #### max_subscriptions Maximum numbers of subscriptions per client and leafnode accounts connection. A value of 0 means unlimited. - [max_subscriptions](/reference/config/max_subscriptions.md): Maximum numbers of subscriptions per client and leafnode accounts connection. A value of 0 means unlimited. #### max_traced_msg_len Set a limit to the trace of the payload of a message. - [max_traced_msg_len](/reference/config/max_traced_msg_len.md): Set a limit to the trace of the payload of a message. #### mqtt Configuration for enabling the MQTT interface. - [mqtt](/reference/config/mqtt.md): Configuration for enabling the MQTT interface. ##### ack_wait This is the amount of time after which a QoS 1 message sent to - [ack_wait](/reference/config/mqtt/ack_wait.md): This is the amount of time after which a QoS 1 message sent to ##### authorization Properties - [authorization](/reference/config/mqtt/authorization.md): Properties ###### password Specifies a global password that clients can use to authenticate - [password](/reference/config/mqtt/authorization/password.md): Specifies a global password that clients can use to authenticate ###### timeout Maximum number of seconds to wait for a client to authenticate. - [timeout](/reference/config/mqtt/authorization/timeout.md): Maximum number of seconds to wait for a client to authenticate. ###### token Specifies a global token that clients can use to authenticate with - [token](/reference/config/mqtt/authorization/token.md): Specifies a global token that clients can use to authenticate with ###### username Specifies a global user name that clients can use to authenticate - [username](/reference/config/mqtt/authorization/username.md): Specifies a global user name that clients can use to authenticate ##### consumer_inactive_threshold Set an explicit default inactive threshold for consumers used by MQTT. - [consumer_inactive_threshold](/reference/config/mqtt/consumer_inactive_threshold.md): Set an explicit default inactive threshold for consumers used by MQTT. ##### consumer_memory_storage Hold MQTT consumer state in memory rather than on disk. - [consumer_memory_storage](/reference/config/mqtt/consumer_memory_storage.md): Hold MQTT consumer state in memory rather than on disk. ##### consumer_replicas Deprecated. The server logs a warning and ignores the value; - [consumer_replicas](/reference/config/mqtt/consumer_replicas.md): Deprecated. The server logs a warning and ignores the value; ##### downgrade_qos2_subscribe Accept a QoS 2 subscription but serve it at QoS 1. - [downgrade_qos2_subscribe](/reference/config/mqtt/downgrade_qos2_subscribe.md): Accept a QoS 2 subscription but serve it at QoS 1. ##### host Types - [host](/reference/config/mqtt/host.md): Types ##### js_api_timeout How long MQTT waits for a JetStream API call before giving up. - [js_api_timeout](/reference/config/mqtt/js_api_timeout.md): How long MQTT waits for a JetStream API call before giving up. ##### js_domain If specified, sets an explicit JetStream domain to be used by MQTT. - [js_domain](/reference/config/mqtt/js_domain.md): If specified, sets an explicit JetStream domain to be used by MQTT. ##### listen Types - [listen](/reference/config/mqtt/listen.md): Types ##### max_ack_pending This is the amount of QoS 1 messages the server can send to - [max_ack_pending](/reference/config/mqtt/max_ack_pending.md): This is the amount of QoS 1 messages the server can send to ##### no_auth_user If no user name is provided when an MQTT client connects, will default - [no_auth_user](/reference/config/mqtt/no_auth_user.md): If no user name is provided when an MQTT client connects, will default ##### port Types - [port](/reference/config/mqtt/port.md): Types ##### reject_qos2_publish Reject QoS 2 publishes instead of accepting them, for clients that - [reject_qos2_publish](/reference/config/mqtt/reject_qos2_publish.md): Reject QoS 2 publishes instead of accepting them, for clients that ##### stream_replicas If specified, sets an explicit number of stream replicas to be used - [stream_replicas](/reference/config/mqtt/stream_replicas.md): If specified, sets an explicit number of stream replicas to be used ##### tls Properties - [tls](/reference/config/mqtt/tls.md): Properties ###### allow_insecure_cipher_suites Permit cipher suites Go considers insecure to be named in - [allow_insecure_cipher_suites](/reference/config/mqtt/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in ###### ca_file TLS certificate authority file. Defaults to system trust store. - [ca_file](/reference/config/mqtt/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. ###### cert_file TLS certificate file. - [cert_file](/reference/config/mqtt/tls/cert_file.md): TLS certificate file. ###### cert_store Load the certificate from an OS certificate store rather than a - [cert_store](/reference/config/mqtt/tls/cert_store.md): Load the certificate from an OS certificate store rather than a ###### certs Multiple certificate/key pairs to serve, so one listener can - [certs](/reference/config/mqtt/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can - [cert_file](/reference/config/mqtt/tls/certs/cert_file.md): Certificate file for this pair. - [key_file](/reference/config/mqtt/tls/certs/key_file.md): Key file for this pair. ###### cipher_suites When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [cipher_suites](/reference/config/mqtt/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. ###### connection_rate_limit Types - [connection_rate_limit](/reference/config/mqtt/tls/connection_rate_limit.md): Types ###### curve_preferences List of TLS cipher curves to use in order. - [curve_preferences](/reference/config/mqtt/tls/curve_preferences.md): List of TLS cipher curves to use in order. ###### handshake_first Send the TLS handshake before the INFO protocol message rather - [handshake_first](/reference/config/mqtt/tls/handshake_first.md): Send the TLS handshake before the INFO protocol message rather ###### insecure Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [insecure](/reference/config/mqtt/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. ###### key_file TLS certificate key file. - [key_file](/reference/config/mqtt/tls/key_file.md): TLS certificate key file. ###### min_version Lowest TLS version the server will negotiate. - [min_version](/reference/config/mqtt/tls/min_version.md): Lowest TLS version the server will negotiate. ###### ocsp_peer Verify the peer's certificate against its OCSP responder and reject - [ocsp_peer](/reference/config/mqtt/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject - [allowed_clockskew](/reference/config/mqtt/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. - [ca_timeout](/reference/config/mqtt/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. - [cache_ttl_when_next_update_unset](/reference/config/mqtt/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. - [unknown_is_good](/reference/config/mqtt/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. - [verify](/reference/config/mqtt/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. - [warn_only](/reference/config/mqtt/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. ###### pinned_certs List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [pinned_certs](/reference/config/mqtt/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the ###### timeout TLS handshake timeout. - [timeout](/reference/config/mqtt/tls/timeout.md): TLS handshake timeout. ###### verify If true, require and verify client certificates. Does not apply to monitoring. - [verify](/reference/config/mqtt/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. ###### verify_and_map If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_and_map](/reference/config/mqtt/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ###### verify_cert_and_check_known_urls Only used in a non-client context where verify is true, such as cluster and gateway configurations. - [verify_cert_and_check_known_urls](/reference/config/mqtt/tls/verify_cert_and_check_known_urls.md): Only used in a non-client context where verify is true, such as cluster and gateway configurations. #### no_auth_user Name of the user that non-authenticated clients - [no_auth_user](/reference/config/no_auth_user.md): Name of the user that non-authenticated clients #### no_fast_producer_stall Do not stall a fast producer when a consumer cannot keep up. The - [no_fast_producer_stall](/reference/config/no_fast_producer_stall.md): Do not stall a fast producer when a consumer cannot keep up. The #### no_header_support Disables support for message headers. - [no_header_support](/reference/config/no_header_support.md): Disables support for message headers. #### no_system_account Types - [no_system_account](/reference/config/no_system_account.md): Types #### ocsp OCSP Stapling is honored by default for certificates that have the - [ocsp](/reference/config/ocsp.md): OCSP Stapling is honored by default for certificates that have the ##### mode The OCSP stapling mode to adhere to. - [mode](/reference/config/ocsp/mode.md): The OCSP stapling mode to adhere to. ##### url An explicit override URL to request staples. - [url](/reference/config/ocsp/url.md): An explicit override URL to request staples. ##### urls An explicit list of override URLs to request staples. - [urls](/reference/config/ocsp/urls.md): An explicit list of override URLs to request staples. #### ocsp_cache Cache OCSP responses the server staples, so it does not query the - [ocsp_cache](/reference/config/ocsp_cache.md): Cache OCSP responses the server staples, so it does not query the ##### local_store Directory for the local cache. - [local_store](/reference/config/ocsp_cache/local_store.md): Directory for the local cache. ##### preserve_revoked Keep revoked responses in the cache rather than evicting them. - [preserve_revoked](/reference/config/ocsp_cache/preserve_revoked.md): Keep revoked responses in the cache rather than evicting them. ##### type Where stapled responses are cached. local writes them under the - [type](/reference/config/ocsp_cache/type.md): Where stapled responses are cached. local writes them under the #### operator One or more operator JWTs, either in files or inlined. - [operator](/reference/config/operator.md): One or more operator JWTs, either in files or inlined. #### pidfile Types - [pidfile](/reference/config/pidfile.md): Types #### ping_interval Duration at which pings are sent to clients, leaf nodes and routes. - [ping_interval](/reference/config/ping_interval.md): Duration at which pings are sent to clients, leaf nodes and routes. #### ping_max After how many unanswered pings the server will allow before closing - [ping_max](/reference/config/ping_max.md): After how many unanswered pings the server will allow before closing #### port Port for client connections. Use -1 for a - [port](/reference/config/port.md): Port for client connections. Use -1 for a #### ports_file_dir Types - [ports_file_dir](/reference/config/ports_file_dir.md): Types #### prof_block_rate Go block-profile sampling rate. Set only while profiling; it slows - [prof_block_rate](/reference/config/prof_block_rate.md): Go block-profile sampling rate. Set only while profiling; it slows #### prof_port Types - [prof_port](/reference/config/prof_port.md): Types #### proxy_protocol Accept the PROXY protocol header on client connections, so the - [proxy_protocol](/reference/config/proxy_protocol.md): Accept the PROXY protocol header on client connections, so the #### reconnect_error_reports Number of failed attempt to reconnect a route, gateway or - [reconnect_error_reports](/reference/config/reconnect_error_reports.md): Number of failed attempt to reconnect a route, gateway or #### remote_syslog Remote syslog address. - [remote_syslog](/reference/config/remote_syslog.md): Remote syslog address. #### resolver Takes precedence over the value obtained from - [resolver](/reference/config/resolver.md): Takes precedence over the value obtained from ##### allow_delete If true, allows JWTs to be deleted. Note, in full mode, this will - [allow_delete](/reference/config/resolver/allow_delete.md): If true, allows JWTs to be deleted. Note, in full mode, this will ##### dir The path to storing account JWT files when pushed to the server. - [dir](/reference/config/resolver/dir.md): The path to storing account JWT files when pushed to the server. ##### hard_delete If true, and the resolver is in full mode, deleted account JWTs will - [hard_delete](/reference/config/resolver/hard_delete.md): If true, and the resolver is in full mode, deleted account JWTs will ##### interval Defines the interval the resolver will randomly contact another server - [interval](/reference/config/resolver/interval.md): Defines the interval the resolver will randomly contact another server ##### limit If set, limit the number of stored JWTs. In full mode, new JWTs - [limit](/reference/config/resolver/limit.md): If set, limit the number of stored JWTs. In full mode, new JWTs ##### timeout Defines the request timeout for resolvers interacting with with - [timeout](/reference/config/resolver/timeout.md): Defines the request timeout for resolvers interacting with with ##### ttl If cache mode, defines how long an account JWT will be cached - [ttl](/reference/config/resolver/ttl.md): If cache mode, defines how long an account JWT will be cached ##### type Defines whether the resolver type. A full resolver stores all - [type](/reference/config/resolver/type.md): Defines whether the resolver type. A full resolver stores all #### resolver_pinned_accounts Types - [resolver_pinned_accounts](/reference/config/resolver_pinned_accounts.md): Types #### resolver_preload Map of account public key to the account JWT. - [resolver_preload](/reference/config/resolver_preload.md): Map of account public key to the account JWT. #### resolver_tls Properties - [resolver_tls](/reference/config/resolver_tls.md): Properties ##### allow_insecure_cipher_suites Permit cipher suites Go considers insecure to be named in - [allow_insecure_cipher_suites](/reference/config/resolver_tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in ##### ca_file TLS certificate authority file. Defaults to system trust store. - [ca_file](/reference/config/resolver_tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. ##### cert_file TLS certificate file. - [cert_file](/reference/config/resolver_tls/cert_file.md): TLS certificate file. ##### cert_store Load the certificate from an OS certificate store rather than a - [cert_store](/reference/config/resolver_tls/cert_store.md): Load the certificate from an OS certificate store rather than a ##### certs Multiple certificate/key pairs to serve, so one listener can - [certs](/reference/config/resolver_tls/certs.md): Multiple certificate/key pairs to serve, so one listener can ###### cert_file Certificate file for this pair. - [cert_file](/reference/config/resolver_tls/certs/cert_file.md): Certificate file for this pair. ###### key_file Key file for this pair. - [key_file](/reference/config/resolver_tls/certs/key_file.md): Key file for this pair. ##### cipher_suites When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [cipher_suites](/reference/config/resolver_tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. ##### connection_rate_limit Types - [connection_rate_limit](/reference/config/resolver_tls/connection_rate_limit.md): Types ##### curve_preferences List of TLS cipher curves to use in order. - [curve_preferences](/reference/config/resolver_tls/curve_preferences.md): List of TLS cipher curves to use in order. ##### handshake_first Send the TLS handshake before the INFO protocol message rather - [handshake_first](/reference/config/resolver_tls/handshake_first.md): Send the TLS handshake before the INFO protocol message rather ##### insecure Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [insecure](/reference/config/resolver_tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. ##### key_file TLS certificate key file. - [key_file](/reference/config/resolver_tls/key_file.md): TLS certificate key file. ##### min_version Lowest TLS version the server will negotiate. - [min_version](/reference/config/resolver_tls/min_version.md): Lowest TLS version the server will negotiate. ##### ocsp_peer Verify the peer's certificate against its OCSP responder and reject - [ocsp_peer](/reference/config/resolver_tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject ###### allowed_clockskew Clock skew tolerated when checking responder timestamps. - [allowed_clockskew](/reference/config/resolver_tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. ###### ca_timeout How long to wait for the OCSP responder. - [ca_timeout](/reference/config/resolver_tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. ###### cache_ttl_when_next_update_unset How long to cache a response that carries no next-update time. - [cache_ttl_when_next_update_unset](/reference/config/resolver_tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. ###### unknown_is_good Treat an unknown response from the responder as good. - [unknown_is_good](/reference/config/resolver_tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. ###### verify Enable OCSP revocation checking of the peer's certificate. - [verify](/reference/config/resolver_tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. ###### warn_only Log revocation failures instead of rejecting the connection. - [warn_only](/reference/config/resolver_tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. ##### pinned_certs List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [pinned_certs](/reference/config/resolver_tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the ##### timeout TLS handshake timeout. - [timeout](/reference/config/resolver_tls/timeout.md): TLS handshake timeout. ##### verify If true, require and verify client certificates. Does not apply to monitoring. - [verify](/reference/config/resolver_tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. ##### verify_and_map If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_and_map](/reference/config/resolver_tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ##### verify_cert_and_check_known_urls Only used in a non-client context where verify is true, such as cluster and gateway configurations. - [verify_cert_and_check_known_urls](/reference/config/resolver_tls/verify_cert_and_check_known_urls.md): Only used in a non-client context where verify is true, such as cluster and gateway configurations. #### server_metadata Arbitrary key/value pairs reported with this server in monitoring - [server_metadata](/reference/config/server_metadata.md): Arbitrary key/value pairs reported with this server in monitoring #### server_name The servers name, shows up in logging. Defaults to the generated - [server_name](/reference/config/server_name.md): The servers name, shows up in logging. Defaults to the generated #### server_tags One or more tags associated with the server. This is currently - [server_tags](/reference/config/server_tags.md): One or more tags associated with the server. This is currently #### store_dir Directory to use for file-based JetStream storage. - [store_dir](/reference/config/store_dir.md): Directory to use for file-based JetStream storage. #### syslog Log to syslog. - [syslog](/reference/config/syslog.md): Log to syslog. #### system_account Name or public key of the account that will be deemed the - [system_account](/reference/config/system_account.md): Name or public key of the account that will be deemed the #### tls TLS configuration for client and HTTP monitoring. - [tls](/reference/config/tls.md): TLS configuration for client and HTTP monitoring. ##### allow_insecure_cipher_suites Permit cipher suites Go considers insecure to be named in - [allow_insecure_cipher_suites](/reference/config/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in ##### ca_file TLS certificate authority file. Defaults to system trust store. - [ca_file](/reference/config/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. ##### cert_file TLS certificate file. - [cert_file](/reference/config/tls/cert_file.md): TLS certificate file. ##### cert_store Load the certificate from an OS certificate store rather than a - [cert_store](/reference/config/tls/cert_store.md): Load the certificate from an OS certificate store rather than a ##### certs Multiple certificate/key pairs to serve, so one listener can - [certs](/reference/config/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can ###### cert_file Certificate file for this pair. - [cert_file](/reference/config/tls/certs/cert_file.md): Certificate file for this pair. ###### key_file Key file for this pair. - [key_file](/reference/config/tls/certs/key_file.md): Key file for this pair. ##### cipher_suites When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [cipher_suites](/reference/config/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. ##### connection_rate_limit Types - [connection_rate_limit](/reference/config/tls/connection_rate_limit.md): Types ##### curve_preferences List of TLS cipher curves to use in order. - [curve_preferences](/reference/config/tls/curve_preferences.md): List of TLS cipher curves to use in order. ##### handshake_first Send the TLS handshake before the INFO protocol message rather - [handshake_first](/reference/config/tls/handshake_first.md): Send the TLS handshake before the INFO protocol message rather ##### insecure Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [insecure](/reference/config/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. ##### key_file TLS certificate key file. - [key_file](/reference/config/tls/key_file.md): TLS certificate key file. ##### min_version Lowest TLS version the server will negotiate. - [min_version](/reference/config/tls/min_version.md): Lowest TLS version the server will negotiate. ##### ocsp_peer Verify the peer's certificate against its OCSP responder and reject - [ocsp_peer](/reference/config/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject ###### allowed_clockskew Clock skew tolerated when checking responder timestamps. - [allowed_clockskew](/reference/config/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. ###### ca_timeout How long to wait for the OCSP responder. - [ca_timeout](/reference/config/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. ###### cache_ttl_when_next_update_unset How long to cache a response that carries no next-update time. - [cache_ttl_when_next_update_unset](/reference/config/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. ###### unknown_is_good Treat an unknown response from the responder as good. - [unknown_is_good](/reference/config/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. ###### verify Enable OCSP revocation checking of the peer's certificate. - [verify](/reference/config/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. ###### warn_only Log revocation failures instead of rejecting the connection. - [warn_only](/reference/config/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. ##### pinned_certs List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [pinned_certs](/reference/config/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the ##### timeout TLS handshake timeout. - [timeout](/reference/config/tls/timeout.md): TLS handshake timeout. ##### verify If true, require and verify client certificates. Does not apply to monitoring. - [verify](/reference/config/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. ##### verify_and_map If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_and_map](/reference/config/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ##### verify_cert_and_check_known_urls Only used in a non-client context where verify is true, such as cluster and gateway configurations. - [verify_cert_and_check_known_urls](/reference/config/tls/verify_cert_and_check_known_urls.md): Only used in a non-client context where verify is true, such as cluster and gateway configurations. #### trace If true, enables protocol trace log messages, - [trace](/reference/config/trace.md): If true, enables protocol trace log messages, #### trace_headers Trace message headers. Also enables trace. - [trace_headers](/reference/config/trace_headers.md): Trace message headers. Also enables trace. #### trace_verbose If true, enables protocol trace log messages, - [trace_verbose](/reference/config/trace_verbose.md): If true, enables protocol trace log messages, #### trusted_keys One or more operator public keys to trust. - [trusted_keys](/reference/config/trusted_keys.md): One or more operator public keys to trust. #### websocket Configuration for enabling the WebSocket interface. - [websocket](/reference/config/websocket.md): Configuration for enabling the WebSocket interface. ##### advertise Advertised client :. Useful for cluster setups - [advertise](/reference/config/websocket/advertise.md): Advertised client :. Useful for cluster setups ##### allowed_origins The list of accepted origins. When empty, and same_origin is false, - [allowed_origins](/reference/config/websocket/allowed_origins.md): The list of accepted origins. When empty, and same_origin is false, ##### authorization Properties - [authorization](/reference/config/websocket/authorization.md): Properties ###### password Specifies a global password that clients can use to authenticate - [password](/reference/config/websocket/authorization/password.md): Specifies a global password that clients can use to authenticate ###### timeout Maximum number of seconds to wait for a client to authenticate. - [timeout](/reference/config/websocket/authorization/timeout.md): Maximum number of seconds to wait for a client to authenticate. ###### token Specifies a global token that clients can use to authenticate with - [token](/reference/config/websocket/authorization/token.md): Specifies a global token that clients can use to authenticate with ###### username Specifies a global user name that clients can use to authenticate - [username](/reference/config/websocket/authorization/username.md): Specifies a global user name that clients can use to authenticate ##### compress This enables support for compressed websocket frames - [compress](/reference/config/websocket/compress.md): This enables support for compressed websocket frames ##### handshake_timeout This is the total time allowed for the server to - [handshake_timeout](/reference/config/websocket/handshake_timeout.md): This is the total time allowed for the server to ##### headers Extra HTTP headers returned on the WebSocket upgrade response. - [headers](/reference/config/websocket/headers.md): Extra HTTP headers returned on the WebSocket upgrade response. ##### host Types - [host](/reference/config/websocket/host.md): Types ##### jwt_cookie Name of the HTTP cookie, that, if present, will be used as a client JWT. - [jwt_cookie](/reference/config/websocket/jwt_cookie.md): Name of the HTTP cookie, that, if present, will be used as a client JWT. ##### listen Types - [listen](/reference/config/websocket/listen.md): Types ##### no_auth_user If no user name is provided when a WebSocket client connects, will - [no_auth_user](/reference/config/websocket/no_auth_user.md): If no user name is provided when a WebSocket client connects, will ##### no_tls Types - [no_tls](/reference/config/websocket/no_tls.md): Types ##### pass_cookie Name of the HTTP cookie holding the password. - [pass_cookie](/reference/config/websocket/pass_cookie.md): Name of the HTTP cookie holding the password. ##### ping_interval How often the server sends a WebSocket ping. Set 0 to disable. - [ping_interval](/reference/config/websocket/ping_interval.md): How often the server sends a WebSocket ping. Set 0 to disable. ##### port Port to accept WebSocket client connections on. There is no default: - [port](/reference/config/websocket/port.md): Port to accept WebSocket client connections on. There is no default: ##### same_origin This option is relevant for clients used within a Web Browser, such - [same_origin](/reference/config/websocket/same_origin.md): This option is relevant for clients used within a Web Browser, such ##### tls Properties - [tls](/reference/config/websocket/tls.md): Properties ###### allow_insecure_cipher_suites Permit cipher suites Go considers insecure to be named in - [allow_insecure_cipher_suites](/reference/config/websocket/tls/allow_insecure_cipher_suites.md): Permit cipher suites Go considers insecure to be named in ###### ca_file TLS certificate authority file. Defaults to system trust store. - [ca_file](/reference/config/websocket/tls/ca_file.md): TLS certificate authority file. Defaults to system trust store. ###### cert_file TLS certificate file. - [cert_file](/reference/config/websocket/tls/cert_file.md): TLS certificate file. ###### cert_store Load the certificate from an OS certificate store rather than a - [cert_store](/reference/config/websocket/tls/cert_store.md): Load the certificate from an OS certificate store rather than a ###### certs Multiple certificate/key pairs to serve, so one listener can - [certs](/reference/config/websocket/tls/certs.md): Multiple certificate/key pairs to serve, so one listener can - [cert_file](/reference/config/websocket/tls/certs/cert_file.md): Certificate file for this pair. - [key_file](/reference/config/websocket/tls/certs/key_file.md): Key file for this pair. ###### cipher_suites When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. - [cipher_suites](/reference/config/websocket/tls/cipher_suites.md): When set, only the specified TLS cipher suites will be allowed. Values must match the golang version used to build the server. ###### connection_rate_limit Types - [connection_rate_limit](/reference/config/websocket/tls/connection_rate_limit.md): Types ###### curve_preferences List of TLS cipher curves to use in order. - [curve_preferences](/reference/config/websocket/tls/curve_preferences.md): List of TLS cipher curves to use in order. ###### handshake_first Send the TLS handshake before the INFO protocol message rather - [handshake_first](/reference/config/websocket/tls/handshake_first.md): Send the TLS handshake before the INFO protocol message rather ###### insecure Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. - [insecure](/reference/config/websocket/tls/insecure.md): Skip certificate verification. This only applies to outgoing connections, NOT incoming client connections. not recommended. ###### key_file TLS certificate key file. - [key_file](/reference/config/websocket/tls/key_file.md): TLS certificate key file. ###### min_version Lowest TLS version the server will negotiate. - [min_version](/reference/config/websocket/tls/min_version.md): Lowest TLS version the server will negotiate. ###### ocsp_peer Verify the peer's certificate against its OCSP responder and reject - [ocsp_peer](/reference/config/websocket/tls/ocsp_peer.md): Verify the peer's certificate against its OCSP responder and reject - [allowed_clockskew](/reference/config/websocket/tls/ocsp_peer/allowed_clockskew.md): Clock skew tolerated when checking responder timestamps. - [ca_timeout](/reference/config/websocket/tls/ocsp_peer/ca_timeout.md): How long to wait for the OCSP responder. - [cache_ttl_when_next_update_unset](/reference/config/websocket/tls/ocsp_peer/cache_ttl_when_next_update_unset.md): How long to cache a response that carries no next-update time. - [unknown_is_good](/reference/config/websocket/tls/ocsp_peer/unknown_is_good.md): Treat an unknown response from the responder as good. - [verify](/reference/config/websocket/tls/ocsp_peer/verify.md): Enable OCSP revocation checking of the peer's certificate. - [warn_only](/reference/config/websocket/tls/ocsp_peer/warn_only.md): Log revocation failures instead of rejecting the connection. ###### pinned_certs List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the - [pinned_certs](/reference/config/websocket/tls/pinned_certs.md): List of hex-encoded SHA256 of DER-encoded public key fingerprints. When present, during the TLS handshake, the ###### timeout TLS handshake timeout. - [timeout](/reference/config/websocket/tls/timeout.md): TLS handshake timeout. ###### verify If true, require and verify client certificates. Does not apply to monitoring. - [verify](/reference/config/websocket/tls/verify.md): If true, require and verify client certificates. Does not apply to monitoring. ###### verify_and_map If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. - [verify_and_map](/reference/config/websocket/tls/verify_and_map.md): If true, require and verify client certificates and map certificate values for authentication. Does not apply to monitoring. ###### verify_cert_and_check_known_urls Only used in a non-client context where verify is true, such as cluster and gateway configurations. - [verify_cert_and_check_known_urls](/reference/config/websocket/tls/verify_cert_and_check_known_urls.md): Only used in a non-client context where verify is true, such as cluster and gateway configurations. ##### token_cookie Name of the HTTP cookie holding the auth token. - [token_cookie](/reference/config/websocket/token_cookie.md): Name of the HTTP cookie holding the auth token. ##### user_cookie Name of the HTTP cookie holding the user name, for clients that - [user_cookie](/reference/config/websocket/user_cookie.md): Name of the HTTP cookie holding the user name, for clients that #### write_deadline Maximum number of seconds the server will block when writing. Once - [write_deadline](/reference/config/write_deadline.md): Maximum number of seconds the server will block when writing. Once #### write_timeout What the server does when a connection misses its - [write_timeout](/reference/config/write_timeout.md): What the server does when a connection misses its ### jetstream JetStream is the persistence layer of NATS, providing message streaming, replay, and at-least-once delivery semantics. - [JetStream](/reference/jetstream.md): JetStream is the persistence layer of NATS, providing message streaming, replay, and at-least-once delivery semantics. #### advisory Advisories are system events published by JetStream servers to notify about important state changes and operational events. These events are published to specific subjects that can be subscribed to for monitoring and observability. - [JetStream Advisories](/reference/jetstream/advisory.md): Advisories are system events published by JetStream servers to notify about important state changes and operational events. These events are published to specific subjects that can be subscribed to for monitoring and observability. ##### api-audit JetStream API audit events. - [API Audit](/reference/jetstream/advisory/api-audit.md): JetStream API audit events. ##### api-limit-reached API rate limit reached events. - [API Limit Reached](/reference/jetstream/advisory/api-limit-reached.md): API rate limit reached events. ##### consumer-action Consumer lifecycle events. - [Consumer Action](/reference/jetstream/advisory/consumer-action.md): Consumer lifecycle events. ##### consumer-group-pinned Consumer group pinned to node. - [Consumer Group Pinned](/reference/jetstream/advisory/consumer-group-pinned.md): Consumer group pinned to node. ##### consumer-group-unpinned Consumer group unpinned from node. - [Consumer Group Unpinned](/reference/jetstream/advisory/consumer-group-unpinned.md): Consumer group unpinned from node. ##### consumer-leader-elected New consumer leader elected. - [Consumer Leader Elected](/reference/jetstream/advisory/consumer-leader-elected.md): New consumer leader elected. ##### consumer-pause Consumer paused or resumed. - [Consumer Pause](/reference/jetstream/advisory/consumer-pause.md): Consumer paused or resumed. ##### consumer-quorum-lost Consumer lost quorum. - [Consumer Quorum Lost](/reference/jetstream/advisory/consumer-quorum-lost.md): Consumer lost quorum. ##### domain-leader-elected New domain leader elected. - [Domain Leader Elected](/reference/jetstream/advisory/domain-leader-elected.md): New domain leader elected. ##### max-deliver Message exceeded max delivery attempts. - [Max Deliveries Exceeded](/reference/jetstream/advisory/max-deliver.md): Message exceeded max delivery attempts. ##### nak Message negatively acknowledged. - [Message Negative Acknowledgement](/reference/jetstream/advisory/nak.md): Message negatively acknowledged. ##### restore-complete Stream restore completed. - [Restore Complete](/reference/jetstream/advisory/restore-complete.md): Stream restore completed. ##### restore-create Stream restore initiated. - [Restore Started](/reference/jetstream/advisory/restore-create.md): Stream restore initiated. ##### server-out-of-space Server storage exhausted. - [Server Out of Space](/reference/jetstream/advisory/server-out-of-space.md): Server storage exhausted. ##### server-removed Server removed from cluster. - [Server Removed](/reference/jetstream/advisory/server-removed.md): Server removed from cluster. ##### snapshot-complete Stream snapshot completed. - [Snapshot Complete](/reference/jetstream/advisory/snapshot-complete.md): Stream snapshot completed. ##### snapshot-create Stream snapshot initiated. - [Snapshot Started](/reference/jetstream/advisory/snapshot-create.md): Stream snapshot initiated. ##### stream-action Stream lifecycle events. - [Stream Action](/reference/jetstream/advisory/stream-action.md): Stream lifecycle events. ##### stream-batch-abandoned An advisory sent when a stream abandons a batch. - [Stream Batch Abandoned](/reference/jetstream/advisory/stream-batch-abandoned.md): An advisory sent when a stream abandons a batch. ##### stream-leader-elected New stream leader elected. - [Stream Leader Elected](/reference/jetstream/advisory/stream-leader-elected.md): New stream leader elected. ##### stream-quorum-lost Stream lost quorum. - [Stream Quorum Lost](/reference/jetstream/advisory/stream-quorum-lost.md): Stream lost quorum. ##### terminated Message terminated. - [Message Terminated](/reference/jetstream/advisory/terminated.md): Message terminated. #### api The JetStream API provides programmatic access to manage and interact with NATS JetStream resources. These APIs enable you to create, configure, and manage streams and consumers, as well as perform administrative operations. - [JetStream API](/reference/jetstream/api.md): The JetStream API provides programmatic access to manage and interact with NATS JetStream resources. These APIs enable you to create, configure, and manage streams and consumers, as well as perform administrative operations. ##### account | Name | Subject | System Account | - [Account](/reference/jetstream/api/account.md): | Name | Subject | System Account | ###### info Retrieves JetStream account information. - [Account Info](/reference/jetstream/api/account/info.md): Retrieves JetStream account information. ###### purge Purges all data for an account. - [Account Purge](/reference/jetstream/api/account/purge.md): Purges all data for an account. ##### consumer | Name | Subject | System Account | - [Consumer](/reference/jetstream/api/consumer.md): | Name | Subject | System Account | ###### create Creates a new consumer. - [Create Consumer](/reference/jetstream/api/consumer/create.md): Creates a new consumer. ###### delete Deletes a consumer. - [Delete Consumer](/reference/jetstream/api/consumer/delete.md): Deletes a consumer. ###### get-next Gets next message(s) from a consumer. - [Get Next Message](/reference/jetstream/api/consumer/get-next.md): Gets next message(s) from a consumer. ###### info Retrieves consumer information. - [Consumer Info](/reference/jetstream/api/consumer/info.md): Retrieves consumer information. ###### leader-stepdown Initiates leader stepdown for a consumer. - [Leader Stepdown](/reference/jetstream/api/consumer/leader-stepdown.md): Initiates leader stepdown for a consumer. ###### list Lists consumers for a stream. - [List Consumers](/reference/jetstream/api/consumer/list.md): Lists consumers for a stream. ###### names Lists consumer names for a stream. - [Consumer Names](/reference/jetstream/api/consumer/names.md): Lists consumer names for a stream. ###### pause Pauses or resumes a consumer. - [Pause Consumer](/reference/jetstream/api/consumer/pause.md): Pauses or resumes a consumer. ###### unpin Unpins a consumer group. - [Unpin Consumer](/reference/jetstream/api/consumer/unpin.md): Unpins a consumer group. ##### headers This document provides a comprehensive reference for all headers used in JetStream operations. These headers are used for message publishing, delivery, and various JetStream features. - [JetStream API Headers](/reference/jetstream/api/headers.md): This document provides a comprehensive reference for all headers used in JetStream operations. These headers are used for message publishing, delivery, and various JetStream features. ##### meta | Name | Subject | System Account | - [Meta](/reference/jetstream/api/meta.md): | Name | Subject | System Account | ###### leader-stepdown Initiates meta-group leader stepdown. - [Meta Leader Stepdown](/reference/jetstream/api/meta/leader-stepdown.md): Initiates meta-group leader stepdown. ###### server-remove Removes a server from the JetStream cluster. - [Server Remove](/reference/jetstream/api/meta/server-remove.md): Removes a server from the JetStream cluster. ##### stream | Name | Subject | System Account | - [Stream](/reference/jetstream/api/stream.md): | Name | Subject | System Account | ###### create Creates a new stream with the specified configuration. - [Create Stream](/reference/jetstream/api/stream/create.md): Creates a new stream with the specified configuration. ###### delete Deletes an existing stream. - [Delete Stream](/reference/jetstream/api/stream/delete.md): Deletes an existing stream. ###### info Retrieves information about a stream. - [Stream Info](/reference/jetstream/api/stream/info.md): Retrieves information about a stream. ###### leader-stepdown Initiates leader stepdown for a stream. - [Leader Stepdown](/reference/jetstream/api/stream/leader-stepdown.md): Initiates leader stepdown for a stream. ###### list Lists all streams. - [List Streams](/reference/jetstream/api/stream/list.md): Lists all streams. ###### msg-delete Deletes a specific message from a stream. - [Delete Message](/reference/jetstream/api/stream/msg-delete.md): Deletes a specific message from a stream. ###### msg-get Retrieves a specific message from a stream. - [Get Message](/reference/jetstream/api/stream/msg-get.md): Retrieves a specific message from a stream. ###### names Lists stream names. - [Stream Names](/reference/jetstream/api/stream/names.md): Lists stream names. ###### pub-ack Publishes a message directly to a stream and receives an acknowledgement. - [Publish Acknowledgement](/reference/jetstream/api/stream/pub-ack.md): Publishes a message directly to a stream and receives an acknowledgement. ###### purge Purges messages from a stream. - [Purge Stream](/reference/jetstream/api/stream/purge.md): Purges messages from a stream. ###### remove-peer Removes a peer from a stream cluster. - [Remove Peer](/reference/jetstream/api/stream/remove-peer.md): Removes a peer from a stream cluster. ###### restore Restores a stream from a snapshot. - [Restore Stream](/reference/jetstream/api/stream/restore.md): Restores a stream from a snapshot. ###### snapshot Creates a snapshot of a stream. - [Snapshot Stream](/reference/jetstream/api/stream/snapshot.md): Creates a snapshot of a stream. ###### update Updates an existing stream configuration. - [Update Stream](/reference/jetstream/api/stream/update.md): Updates an existing stream configuration. #### errors This page contains a comprehensive list of all JetStream error codes and their descriptions, organized by category. - [JetStream Errors](/reference/jetstream/errors.md): This page contains a comprehensive list of all JetStream error codes and their descriptions, organized by category. #### metric Metrics are operational measurements published by JetStream to provide observability into message processing performance and behavior. - [JetStream Metrics](/reference/jetstream/metric.md): Metrics are operational measurements published by JetStream to provide observability into message processing performance and behavior. ##### consumer-ack Consumer acknowledgement metrics. - [Consumer Acknowledgement Metric](/reference/jetstream/metric/consumer-ack.md): Consumer acknowledgement metrics. ### protocols NATS uses text-based protocols for all communication between clients, servers, and clusters. These protocols are designed to be simple, efficient, and easy to implement. - [Protocols](/reference/protocols.md): NATS uses text-based protocols for all communication between clients, servers, and clusters. These protocols are designed to be simple, efficient, and easy to implement. #### client The NATS client protocol is used to communicate between the NATS server and clients. - [Client](/reference/protocols/client.md): The NATS client protocol is used to communicate between the NATS server and clients. #### gateway The NATS gateway protocol is used to connect multiple NATS clusters together. - [Gateway](/reference/protocols/gateway.md): The NATS gateway protocol is used to connect multiple NATS clusters together. #### leafnode The NATS leafnode protocol is used to create lightweight, unidirectional connections from edge servers to a central cluster. - [Leafnode](/reference/protocols/leafnode.md): The NATS leafnode protocol is used to create lightweight, unidirectional connections from edge servers to a central cluster. #### route The NATS route protocol is used to form a cluster of NATS servers. - [Route](/reference/protocols/route.md): The NATS route protocol is used to form a cluster of NATS servers. ### services The NATS Services API provides a framework for building observable microservices with built-in discovery, health checking, and monitoring. - [Services API](/reference/services.md): The NATS Services API provides a framework for building observable microservices with built-in discovery, health checking, and monitoring. #### info-response Service information and metadata response. - [Info Response](/reference/services/info-response.md): Service information and metadata response. #### ping-response Service health check response. - [Ping Response](/reference/services/ping-response.md): Service health check response. #### stats-response Service statistics and metrics response. - [Stats Response](/reference/services/stats-response.md): Service statistics and metrics response. ### system System reference for monitoring, observability, and system management. - [System Reference](/reference/system.md): System reference for monitoring, observability, and system management. #### advisory Server advisory messages are published by the NATS server to notify clients of significant events related to connections, accounts, and system state. - [Server Advisory](/reference/system/advisory.md): Server advisory messages are published by the NATS server to notify clients of significant events related to connections, accounts, and system state. ##### account-connections Account connection limit events. - [Account Connections](/reference/system/advisory/account-connections.md): Account connection limit events. ##### client-connect Client connection events. - [Client Connect](/reference/system/advisory/client-connect.md): Client connection events. ##### client-disconnect Client disconnection events. - [Client Disconnect](/reference/system/advisory/client-disconnect.md): Client disconnection events. #### errors NATS server non-JetStream errors for client, route, gateway, and leafnode connections - [System Errors](/reference/system/errors.md): NATS server non-JetStream errors for client, route, gateway, and leafnode connections #### metric Server metric messages provide telemetry data from the NATS server for monitoring and performance analysis. - [Server Metrics](/reference/system/metric.md): Server metric messages provide telemetry data from the NATS server for monitoring and performance analysis. ##### service-latency Service latency metrics. - [Service Latency](/reference/system/metric/service-latency.md): Service latency metrics. #### monitor Server monitoring endpoints provide detailed information about the NATS server state, performance metrics, and operational health. - [Server Monitoring](/reference/system/monitor.md): Server monitoring endpoints provide detailed information about the NATS server state, performance metrics, and operational health. ##### accountz Request Schema - [Accountz](/reference/system/monitor/accountz.md): Request Schema ##### accstatz Request Schema - [Accstatz](/reference/system/monitor/accstatz.md): Request Schema ##### connz Request Schema - [Connz](/reference/system/monitor/connz.md): Request Schema ##### gatewayz Request Schema - [Gatewayz](/reference/system/monitor/gatewayz.md): Request Schema ##### healthz Request Schema - [Healthz](/reference/system/monitor/healthz.md): Request Schema ##### idz Response Schema - [Idz](/reference/system/monitor/idz.md): Response Schema ##### ipqueuesz Request Schema - [IPQueuesz](/reference/system/monitor/ipqueuesz.md): Request Schema ##### jsz Request Schema - [JSz](/reference/system/monitor/jsz.md): Request Schema ##### leafz Request Schema - [Leafz](/reference/system/monitor/leafz.md): Request Schema ##### profilez Request Schema - [Profilez](/reference/system/monitor/profilez.md): Request Schema ##### raftz Request Schema - [Raftz](/reference/system/monitor/raftz.md): Request Schema ##### routez Request Schema - [Routez](/reference/system/monitor/routez.md): Request Schema ##### statsz Request Schema - [Statsz](/reference/system/monitor/statsz.md): Request Schema ##### subsz Request Schema - [Subsz](/reference/system/monitor/subsz.md): Request Schema ##### varz Request Schema - [Varz](/reference/system/monitor/varz.md): Request Schema ## search - [Search the documentation](/search.md) ## tutorials Hand-held, start-to-finish tutorials that get you a working NATS result fast - [Tutorials](/tutorials.md): Hand-held, start-to-finish tutorials that get you a working NATS result fast ### build-an-app Combine publish/subscribe, request/reply, and a stream into one working app - [7. Capstone: build a small NATS app](/tutorials/build-an-app.md): Combine publish/subscribe, request/reply, and a stream into one working app ### first-stream Create your first stream, publish to it, and replay stored messages - [4. Persist messages with JetStream](/tutorials/first-stream.md): Create your first stream, publish to it, and replay stored messages ### hello-nats Install NATS, then publish and subscribe your first message - [1. Hello NATS](/tutorials/hello-nats.md): Install NATS, then publish and subscribe your first message ### key-value Put, get, and watch values in a JetStream Key-Value bucket - [6. Store state in Key-Value](/tutorials/key-value.md): Put, get, and watch values in a JetStream Key-Value bucket ### request-reply Build a tiny responder service and call it with a request that gets one answer back - [2. Request and reply](/tutorials/request-reply.md): Build a tiny responder service and call it with a request that gets one answer back ### stream-consumer Read a stream with a durable consumer, acknowledge each message, and resume where you left off after a restart - [5. A consumer that survives restarts](/tutorials/stream-consumer.md): Read a stream with a durable consumer, acknowledge each message, and resume where you left off after a restart ### work-queue Use a queue group to split a workload across two workers, so each message goes to exactly one of them - [3. Share work across workers](/tutorials/work-queue.md): Use a queue group to split a workload across two workers, so each message goes to exactly one of them --- # Full Documentation Content [Skip to main content](#__docusaurus_skipToContent_fallback) [![NATS Logo](/img/nats-logo.svg)![NATS Logo](/img/nats-logo-dark.svg)](/index.md) [****](/index.md)[Concepts](/concepts/intro.md)[Learn](/learn/.md)[Tutorials](/tutorials/.md)[Reference](/reference/.md) [2.14 (latest)](/reference/.md) * [2.14 (latest)](/reference/.md) * [2.12](/reference/2.12/) * [2.11](/reference/2.11/) [GitHub](https://github.com/nats-io)[Slack](https://slack.nats.io) Search # Search the documentation Documentation * [Concepts](/concepts/intro.md) * [Learn](/learn/.md) * [Tutorials](/tutorials/.md) * [Reference](/reference/.md) Community * [Slack](https://slack.nats.io) * [Twitter](https://twitter.com/nats_io) * [Google Groups](https://groups.google.com/forum/#!forum/natsio) More * [GitHub](https://github.com/nats-io) * [Contribute to these docs](https://github.com/nats-io/nats.docs.v2) * [NATS.io](https://nats.io) * [NATS by Example](https://natsbyexample.com) Copyright © 2026 Synadia Communications, Inc. Built with Docusaurus. --- # Learn This section holds the long-form deep dives in the NATS documentation. Each deep dive walks through one subsystem from first encounter to working confidence, building up a single running scenario, the Acme `ORDERS` platform, step by step. Deep dives sit between two other layers of the docs: * **[Concepts](/concepts/intro.md)** Short primers and Getting Started. Read those first if you're new to NATS. * **[Reference](/reference/.md)** Exhaustive, versioned catalog of every configuration option, header, and wire-protocol detail. Deep dives link out to Reference for the full surface area, since Reference is a catalog rather than a teaching resource. Every deep dive follows the same pattern: it uses one running scenario, introduces at most two new concepts per page, provides runnable examples in CLI and the client libraries, and links out to Reference for the remaining detail. ## Develop Build applications with NATS. * **[Core NATS](/learn/core-nats/.md)** the foundation: connections, publish-subscribe, subjects and wildcards, request-reply, queue groups, scatter-gather, headers, and server-side subject mapping. * **[Services](/learn/services/.md)** the micro request-reply framework: endpoints, groups, discovery, observability, and scaling. * **[JetStream](/learn/jetstream/.md)** the persistence layer: streams, consumers, and the acknowledgment loop, built up from a single `ORDERS` stream. * **[Resilient Clients](/learn/resilient-clients/.md)** production-grade connections: reconnection, drain, slow consumers, request-reply resilience, and TLS plus auth. * **[Key-Value Store](/learn/key-value/.md)** buckets, watches, history and revisions, TTLs, and how KV sits on top of a stream. * **[Object Store](/learn/object-store/.md)** chunked blobs over JetStream, with metadata, links, and listing. ## Operate Run, scale, and secure NATS. * **[Topologies](/learn/topologies/.md)** Topological shapes: a single server growing into a cluster, a super-cluster, and leaf nodes at the edge. * **[Security](/learn/security/.md)** authentication, authorization, and accounts: centralized and decentralized auth, TLS, and auth callout. * **[Clustering & Replication](/learn/clustering/.md)** the mechanism under a cluster: routes and gossip, RAFT and leaders, `R=3` quorum, placement, and peer management. * **[Monitoring & Observability](/learn/monitoring/.md)** the monitoring endpoints, JetStream health, advisories and events, and Prometheus with dashboards. * **[Backup & Recovery](/learn/backup-recovery/.md)** stream snapshots, mirrors for disaster recovery, a recovery runbook, and backing up identity. * **[Deployment & Upgrades](/learn/deployment/.md)** production: sizing, Kubernetes, config management, rolling upgrades, and hardening. * **[MQTT](/learn/mqtt/.md)** running `nats-server` as the MQTT broker for your devices: topic-to-subject conversion, QoS, sessions and retained messages, and MQTT users on a cluster. * **[WebSocket](/learn/websocket/.md)** the NATS protocol over a WebSocket transport: browser clients, origin checking and cookie credentials, TLS behind a proxy, and leaf nodes over port 443. ## Where to start If you're new to NATS, start with the [Core NATS deep dive](/learn/core-nats/.md); it's the foundation every other chapter builds on. If you're already comfortable with the basics and here for persistence, go straight to the [JetStream deep dive](/learn/jetstream/.md). --- # Backup & Recovery Deep Dive The earlier chapters built things. JetStream gave you an `ORDERS` stream full of order events. Security gave you the `ACME` operator and the `ORDERS` and `ANALYTICS` accounts that decide who may touch it. Topologies spread it across the `east` cluster. This chapter doesn't build anything new. It keeps all of that alive when something goes wrong. Something will go wrong. A disk fills, a region drops off the network, a script deletes the wrong stream, a laptop holding the operator key is left in a taxi. Each of those is a different kind of loss, and each one needs a different kind of copy made ahead of time. This chapter is about making those copies and putting them back. ## The three things to protect Protecting a NATS platform means protecting three things, and none of them protects the others. Keep this in mind for the whole chapter: * **A snapshot** is a point-in-time copy of a stream (its messages, its config, and optionally its consumer state) written off-site. It's the point you can return to after a delete or a corruption. It answers "how much data can I afford to lose," your **recovery point**. * **A mirror** is a live, read-only copy of a stream running at a second site, kept current by replication. It's the failover destination when the first one dies. It answers "how long can recovery take," your **recovery time**. * **Identity** is the set of files that *are* your security layer: the operator and account JWTs, the nkeys that sign them, the user creds, and the server config. Without these, a restored stream is data nobody is allowed to read. They're the keys that prove who you are. A snapshot won't bring a dead site back quickly, a mirror won't save you from bad data (a corrupt write replicates to the mirror, and a mirror keeps no earlier state to rewind to), and neither one matters if you've lost the keys. You need all three, and this chapter takes them one at a time. ### Why R3 is not on the list You might expect replication to be the fourth item. A stream with three replicas (R3) survives a node dying without losing a message, so it's tempting to call it a backup, but it isn't. R3 is high availability: several copies of the same live stream, kept identical at all times. Being kept identical is the problem. When a bad write lands (an accidental delete, a logic error that corrupts a message), every replica applies it. The mistake is replicated the same way the good data is. R3 keeps the stream *available*; only a snapshot lets you go *back* to before the mistake. We treat R3 as availability throughout this chapter and never as a backup. The replication and leader-election mechanics live in the [Clustering & Replication](/learn/clustering/.md) deep dive. ## By the end you'll have * A dated, off-site snapshot of the `ORDERS` stream under `./backups/orders/`, plus a verified restore procedure that rebuilds it with matching message counts. * An `ORDERS_DR` mirror of `ORDERS` running at a second site, whose lag you can read before trusting it. * A runbook that picks the right recovery for each class of failure (restore the snapshot or promote the mirror), walked against the real objects. * An off-site, encrypted copy of the platform's identity (the `ACME` operator and the `ORDERS` and `ANALYTICS` accounts with every JWT and private seed, plus the server config), and a procedure that puts it all back in a clean-room rebuild and re-mints any lost creds files. ## Who this is for You've worked through the [JetStream deep dive](/learn/jetstream/.md), so you know what a stream and a consumer are. You've worked through the [Security deep dive](/learn/security/.md), so you know what an operator, an account, and a user are. Ideally you have the [Topologies deep dive](/learn/topologies/.md) behind you too, so the `east` cluster and a second site are familiar shapes. This chapter assumes you now run NATS for someone else. You're past whether it works and into what happens when it stops in production. It doesn't re-teach how a stream stores messages, how a mirror replicates them, or how an account trust chain validates. Instead, it links to the chapter that owns each of those and builds the operational layer on top. ## How to read it Each page introduces at most two new concepts and carries the same Acme ORDERS world forward. You keep the `east` cluster and the `ORDERS` stream running, take a snapshot of them, stand up `ORDERS_DR` beside them, and then walk the runbook against those exact objects, with no fresh example per page. Where a feature has a long list of knobs, error codes, or advisory subjects, the page covers only what you need to run the procedure and links to [Reference](/reference/.md) for the exhaustive detail. Two boundaries are deliberate, because another chapter already owns them. *How* a mirror replicates lives in [JetStream → Mirrors and sources](/learn/jetstream/mirrors-and-sources.md); here we only apply a mirror to disaster recovery and read its lag. *What* an operator or account is lives in [Security → Operator mode](/learn/security/operator-mode.md); here we only copy those files off-site and put them back. ## Map | Page | What you learn | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | [Stream backup and restore](/learn/backup-recovery/stream-backup-restore.md) | Take a point-in-time snapshot of `ORDERS`, restore it, and verify the counts match | | [Mirrors as a DR tool](/learn/backup-recovery/mirrors-and-sources.md) | Stand up `ORDERS_DR` at a second site, read its lag, and see why a mirror is not a backup | | [Disaster recovery](/learn/backup-recovery/disaster-recovery.md) | A runbook that picks restore or promotion per failure class, and how to promote a mirror | | [Config and JWT backup](/learn/backup-recovery/config-and-jwt-backup.md) | Back up and restore the operator, accounts, keys, and server config off-site | | [Where to go next](/learn/backup-recovery/where-next.md) | The whole game recapped, plus a single production checklist | ## Prerequisites You'll need the world the earlier chapters built, running locally: * A `nats-server` with **file** storage and JetStream enabled, holding the `ORDERS` stream. Memory streams can't be snapshotted, so file storage isn't optional here. * The `nats` CLI installed and pointed at that server. * The `nats auth` store from the Security chapter (the tree under `$XDG_DATA_HOME/nats`), holding the `ACME` operator and the `ORDERS` and `ANALYTICS` accounts. * A second place to put copies: a second site for the mirror, and any off-site location (another disk, a bucket) for snapshots and identity. Open a terminal, confirm `nats stream info ORDERS` returns your stream, and turn to [Stream backup and restore](/learn/backup-recovery/stream-backup-restore.md). ## See also * [JetStream → Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) — how a mirror actually replicates, which this chapter applies but doesn't re-teach. * [Security → Operator mode](/learn/security/operator-mode.md) — what the operator, accounts, and users are, the identity this chapter backs up. * [Clustering & Replication](/learn/clustering/.md) — the R3 mechanics this chapter deliberately leaves out. --- # Config and JWT backup The last three pages protected the data with a snapshot to return to, a mirror to fail over to, and a runbook to choose between them. But a restored `ORDERS` stream is useless if nobody is allowed to read it. The accounts that gate it, the operator that signs those accounts, and the creds `order-svc` connects with don't live in the stream. They live in a different set of files, and those files need their own backup. This page protects the **identity plane**. It does two things: it copies the files that *are* your security layer off-site, encrypted; then it puts them back in a clean-room rebuild and verifies that the platform runs again. This page doesn't teach what an operator, account, or user *is*; that model lives in [Security → Operator mode](/learn/security/operator-mode.md). Here you only learn which files carry that identity, how to get them off-site, and how to restore them. ## The files that carry identity Everything that proves who may touch the `ORDERS` platform reduces to files on disk. There are three groups, and losing any one of them breaks the platform in a different way. The first group is the **`nats auth` store**, the directory tree the Security chapter built under `$XDG_DATA_HOME/nats` (by default `~/.local/share/nats`). Two kinds of file live in it. A **JWT** is the signed identity token: one for the operator, one per account, one per user. A **seed** (an `.nk` file) is the private half of the nkey pair whose public key the JWT names as its subject. Each JWT is signed by the key one level up the chain: the operator signs the account JWTs, and each account signs its own users' JWTs. The layout for the Acme world: ``` $XDG_DATA_HOME/nats/nsc/ # store root; the on-disk layout is nsc-compatible ├── keys/ │ └── keys/ │ ├── O/BJ/OBJYFWT2….nk # operator identity seed │ ├── O/CY/OCYOEBDC….nk # operator signing-key seed │ ├── A/CX/ACXV2TF4….nk # ORDERS account seed (+ ANALYTICS, SYSTEM) │ └── U/A2/UA2YLJKK….nk # order-svc user seed (+ analytics-reader, admin) └── stores/ └── ACME/ ├── ACME.jwt # operator JWT └── accounts/ ├── ORDERS/ORDERS.jwt ├── ORDERS/users/order-svc.jwt ├── ANALYTICS/… # ANALYTICS.jwt, analytics-reader.jwt └── SYSTEM/… # SYSTEM.jwt, admin.jwt ``` The `stores` half holds only JWTs, which are public; they assert identity, they sign nothing. The `keys` half is secret. A leaked operator or account seed lets identity be forged, because those keys sign the JWTs one level down; a leaked user seed lets an attacker connect as that user; and a lost seed means that identity is lost. Protect the `keys` subtree with the same controls you apply to stored passwords. The second group is the **creds files**. A `.creds` file is a user's JWT and seed concatenated into one file: the thing a client points at to connect. They aren't inside the store — each one landed wherever `--credential` wrote it when the user was created, and the copies your services run with sit on the client machines. These are the files `order-svc` and `analytics-reader` present at connect time. The third group is the **server config**, the `server.conf` that `nats server generate` wrote for the running cluster. It isn't identity itself, but it anchors it: the operator JWT the server trusts, the `SYSTEM` account preload, and the resolver directory where account JWTs land. Restore the keys without the config and the server doesn't know to trust them. ``` # ./acme-server/server.conf — what ties the identity together operator: eyJ0eXAiOiJKV1Qi... # the full ACME operator JWT system_account: AAW27T5RB3M5GNDKLGEZ... # the SYSTEM account public key resolver_preload { # SYSTEM account JWT, baked into the config AAW27T5RB3M5GNDKLGEZ...: eyJ0eXAiOiJKV1Qi... } resolver { type: full dir: /var/lib/nats/resolver # the server's own copy of account JWTs } ``` The `resolver` block is the **account resolver**, the server component that verifies accounts when a user connects. Its `dir` holds the server's own copy of every pushed account JWT, and that copy matters at restore time. The full set of resolver options lives in [Reference → Resolver](/reference/config/resolver/.md). For now you only need to know that the server keeps its own copy. ## Backing up the files The store has a native backup command. One line captures the whole `ACME` subtree — every JWT and every private seed, operator, accounts, and users — in a single file: ``` nats auth operator backup ACME acme-operator.backup ``` ``` Wrote backup for ACME to acme-operator.backup WARNING: The output file is unencrypted and contains secrets, consider encrypting it with 'nats auth nkey seal' ``` The file is a single JSON document — treat it as an opaque blob and don't edit it by hand. Read the warning literally: the file holds the operator's private seed, every signing-key seed, and every account and user seed. Whoever holds this one file *is* the `ACME` operator. Never ship it anywhere unencrypted. The `--key` flag encrypts the backup with a curve key. Generate the key once, then point `--key` at the seed file — it takes a file path, not the key string: ``` nats auth nkey gen curve --output backup-curve.nk nats auth operator backup ACME acme-operator.backup --key backup-curve.nk ``` ``` Wrote backup for ACME to acme-operator.backup ``` No warning this time: the output is a sealed blob that only the curve seed can open. That makes `backup-curve.nk` the key to every future restore. Store it somewhere other than the location holding the backups; keeping an archive and its key in the same place is a single point of failure that defeats the backup. Two things the backup does *not* contain. It carries no creds files — those are re-minted from the restored store, as you'll see below. And it carries nothing from the server side: back up `server.conf` alongside it, because the `SYSTEM` preload inside it is what lets you repopulate a server after a disaster. The resolver directory itself needs no backup; re-pushing the accounts rebuilds it. Ship both files off-site, dated: ``` # Ship the sealed backup and the server config away from the live cluster. aws s3 cp acme-operator.backup \ s3://acme-dr/identity/acme-operator-2026-07-04.backup aws s3 cp ./acme-server/server.conf \ s3://acme-dr/identity/server-2026-07-04.conf ``` The date serves a purpose. If you rotate the operator key (re-sign the chain under a new operator identity), an older backup restores the *previous* operator, and a server rebuilt from it trusts a chain nobody signs anymore. Tag each backup with the day the identity was current so you can match a file to the operator version it belongs to. Run this on a schedule the same way you schedule the snapshot. A daily cron line keeps the identity copy as fresh as the data copy: ``` # /etc/cron.d/acme-identity-backup — daily at 02:30 30 2 * * * nats /usr/local/bin/backup-identity.sh ``` ## Restoring the files A clean-room restore is one command against the backup file, then a verification pass, then one server-side step that teams often miss. Pull the backup down and restore it. `--key` names the same curve seed file the backup was sealed with: ``` aws s3 cp s3://acme-dr/identity/acme-operator-2026-07-04.backup acme-operator.backup nats auth operator restore ACME acme-operator.backup --key backup-curve.nk ``` ``` Operator ACME (OBJYFWT2JMTZJBNNXZWQU5UDZSQYUKK2GQ6OGTDSBDN35WW3PXWPSSP6) Configuration: Name: ACME Subject: OBJYFWT2JMTZJBNNXZWQU5UDZSQYUKK2GQ6OGTDSBDN35WW3PXWPSSP6 Accounts: 3 System Account: SYSTEM (AAW27T5RB3M5GNDKLGEZR27S2HY5XHGL2PWVKOMBU7L4ZZYEPEGHOS7J) Signing Keys: OCYOEBDCJQKV3F6LWKCWVLGDOLMLZYO5LRLILZVRW376BDWBBZOO4RQX ``` The `Subject` is the same operator public key as before the loss: restore brings back the original keys, it doesn't mint new ones. That has a useful consequence — every creds file you handed out before the disaster keeps working, because nothing rotated. One caveat: restore refuses to run if the operator already exists in the store (`nats: error: operator ACME already exist`). It's for rebuilding a clean machine; to restore over a corrupted store, move the old store directory aside first. Confirm the chain is complete: ``` nats auth account ls nats auth user ls ORDERS ``` `account ls` should list `ANALYTICS`, `ORDERS`, and `SYSTEM` with one user each, and `user ls ORDERS` should show `order-svc`. If a service's creds file was lost along with the machine it lived on, mint a fresh one from the restored seeds: ``` nats auth user credential order-svc.creds order-svc ORDERS ``` ``` Wrote credential for order-svc to order-svc.creds ``` Next comes the step that a naive restore skips. Restoring the store rebuilds your workstation's copy of the chain, but the server validates connections against its *own* copy: the account JWTs in its resolver directory. If that directory survived, the server never notices your restore. If it didn't — a fresh machine, a wiped disk — the server starts from the saved `server.conf`, trusts `ACME`, and still rejects every user: ``` nats pub orders.new "hello" --creds order-svc.creds ``` ``` nats: error: nats: Authorization Violation ``` The resolver directory is empty, so the server can't find the `ORDERS` account JWT. Fill it by pushing each account, exactly as on first setup: ``` nats auth account push ORDERS --operator ACME --creds sys.creds nats auth account push ANALYTICS --operator ACME --creds sys.creds ``` ``` Updating account ORDERS (ACXV2TF4CTC575UWIFY75K4ZHLS337VNP2JKAD4E5IS346TATYFTDLYR) on 1 server(s) ✓ Update completed on acme-1 Success 1 Failed 0 Expected 1 ``` The push itself authenticates with the `SYSTEM` creds, and it can get in even though the resolver is empty because `server.conf` preloads the `SYSTEM` account JWT. That preload is the bootstrap path for the whole recovery — and the reason the config file belongs in the backup set. Finally, prove a real client can connect — identity plane and data plane together: ``` nats pub orders.new "back" --creds order-svc.creds ``` ``` 13:48:24 Published 4 bytes to "orders.new" ``` If `order-svc` authenticates with the same creds file it had before the disaster, and `nats stream info ORDERS --creds order-svc.creds` returns the restored stream, the full platform is back: the data the earlier pages protected, and now the identity that gates it. ## Pitfalls Three traps come up the first time teams back up identity rather than data. Each one stays inside this page's two jobs: backing the files up, and restoring them. **The backup file is the whole authority.** `operator backup` writes every private seed in the `ACME` subtree into one file; anyone who reads it can sign accounts and users as you. Treat it more carefully than any single key: always pass `--key` so it leaves your machine sealed, and keep the curve seed file away from the backups it opens. The flip side is that the curve seed is now load-bearing. Restore without it fails — ``` nats auth operator restore ACME acme-operator.backup ``` ``` nats: error: unmarshal failed: invalid character 'e' looking for beginning of value ``` — and there's no recovery path: no reset link, no support ticket that regenerates a seed. Losing both the store and the means to open its backup is losing the identity. Test-restore on a spare machine once so you know the file and the key actually pair up. **A restored store doesn't refill the server's resolver.** The store on your workstation and the resolver directory on the server are separate copies of the account JWTs. `operator restore` rebuilds only yours. A server that lost its resolver directory keeps rejecting users with `Authorization Violation` — the files on your side are correct, `account ls` looks right, yet nobody can connect. The fix is one `nats auth account push` per account, authenticated with the `SYSTEM` creds that the config's `resolver_preload` lets in. That works only while you still have `server.conf`; back it up with the identity, not as an afterthought. **An un-backed-up operator rotation orphans the archive.** If you rotate the `ACME` operator (re-sign the chain under a fresh operator key) and your last off-site backup predates the rotation, that backup restores an operator nobody signs accounts under anymore. Tag every backup with the operator version or timestamp, and take a fresh backup right after any rotation, so a backup and the live operator never drift apart. ## Where you are The identity plane is now recoverable. You have a sealed, off-site backup of the whole `ACME` subtree — the operator, `ORDERS`, `ANALYTICS`, and `SYSTEM` with all their JWTs and private seeds — plus the `server.conf` that anchors it, dated to the operator version it belongs to, with the curve seed stored apart. And you have a restore procedure that rebuilds the store, re-mints any lost creds, re-pushes the accounts into an empty resolver, and verifies a real client connects. Combined with the snapshot from [Stream backup and restore](/learn/backup-recovery/stream-backup-restore.md) and the `ORDERS_DR` mirror from [Mirrors as a DR tool](/learn/backup-recovery/mirrors-and-sources.md), the whole platform now survives a clean-room rebuild. The data comes back from a snapshot, the site comes back from the mirror, and the identity that gates both comes back from this backup. ## What's next Every protective copy is now in place: snapshot, mirror, runbook, and identity. The last page recaps the whole chapter and collects every page's pitfalls into one production checklist you run before you trust the platform with production traffic. Continue to [Where to go next](/learn/backup-recovery/where-next.md). ## See also * [Security → Operator mode](/learn/security/operator-mode.md) — what the operator, accounts, and users you backed up here actually are. * [Security → Cross-account](/learn/security/cross-account.md) — the `ORDERS`-to-`ANALYTICS` export/import that a cross-account mirror also depends on; it lives in the account JWTs, so the backup carries it. * [Reference → Resolver](/reference/config/resolver/.md) — the full set of account-resolver options, including the `dir` the push refills on restore. --- # Disaster recovery You now have two tools. A **snapshot** of `ORDERS` sits off-site under `./backups/orders/`, and a live **mirror**, `ORDERS_DR`, runs at `site2`. Each protects against a different failure, and reaching for the wrong one during an outage costs you either data or hours. This page is the **runbook**: an ordered procedure that names the failure, picks the right tool, and carries out the recovery against the real `ORDERS` deployment. It also covers the one operation the earlier pages set up but never performed: **promotion**, turning the read-only `ORDERS_DR` into a writable primary. ## Match the failure to the tool A runbook starts before the outage. The decision you don't want to make under pressure during an outage is *which tool*. Decide it ahead of time, per failure class. | What happened | Reach for | Why | | ---------------------------------------------------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The whole `east` cluster is gone | promote `ORDERS_DR` | The mirror already holds the data; promotion is minutes, a restore is hours. | | Someone deleted or purged `ORDERS` by mistake | recover from `ORDERS_DR`, or restore the snapshot | The mirror survives an upstream delete: it keeps every message it had copied and just stops updating, so it's usually the freshest intact copy. The snapshot is the older fallback. | | Messages on `ORDERS` are corrupt or wrong (a bad publisher) | restore a known-good snapshot, or purge the bad range | The bad data replicated to the mirror as well. The snapshot predates the corruption. | | A consumer lost its position (`shipping` redelivering from zero) | restore a `--consumers` snapshot | Only the snapshot captured the consumer's saved delivery position. | One principle runs through the whole table: a snapshot is the only copy that predates a mistake. A mirror follows the upstream's live writes, so a corrupt write replicates into it; it survives an upstream delete but only as a copy frozen at the break, never an earlier point you chose. That's why the chapter built both, and why [R3 replication](/learn/clustering/.md) is on neither row: replicating a bad write three times doesn't undo it. ## Failover: promote the mirror When the `east` cluster is gone, the goal is to make `ORDERS_DR` start accepting writes so `order-svc` and the consumers can carry on at `site2`. That's promotion, and it's a short, ordered sequence. The animation below shows it end to end. **Message flow — Promote a mirror on site loss (animated):** Disaster-recovery failover in five stages. site1 owns the writable ORDERS stream while ORDERS\_DR on site2 mirrors every message across the WAN. site1 goes dark, taking the primary and its clients with it; only the ORDERS\_DR mirror survives. Before failing over, an operator checks the mirror's lag and waits until ORDERS\_DR reports zero messages behind. The mirror is then promoted — reconfigured into a standalone, writable ORDERS stream on site2 — and order-svc and the consumers reconnect to site2 and resume from where the mirror left off. * orderSvc → order-svc (subject: publish) * order-svc → consumers (subject: deliver) * order-svc → site2 (subject: mirror) * orderSvc → site2 (subject: publish) * site2 → consumers (subject: deliver) ### Step 1 — verify the lag is zero A mirror trails its upstream by some **lag**: the count of messages it hasn't copied yet. Promote it while lag is non-zero and you start publishing on top of a stream that's still missing its tail. So the first runbook step is always the same: read the lag. #### CLI ``` #!/bin/bash # Before you promote ORDERS_DR, read how far it trails the upstream. # A mirror is eventually consistent: it follows its upstream over the # network, so at any instant it can be a few messages behind. The Lag # field is that gap, in messages. Promote a mirror with non-zero lag # and you publish on top of a stream that is still missing tail # messages. # Point at site2, where the ORDERS_DR mirror lives. nats --server nats://127.0.0.1:5222 stream info ORDERS_DR # Look at the Mirror block in the output: # # Mirror Information: # # Stream Name: ORDERS # Lag: 0 # Last Seen: 1.2s # # Lag: 0 means the mirror has caught up to every message the upstream # had at last contact — it is safe to promote. A non-zero Lag means # messages are still in flight; wait and re-run until it reaches 0. # # Last Seen is how long ago the mirror last heard from the upstream. If # the upstream site is already gone, Last Seen climbs and Lag freezes at # whatever it was when contact dropped — that frozen Lag is your data # loss, your actual RPO for this failover. ``` #### C ``` // Before you promote ORDERS_DR, read how far it trails the // upstream. Lag is that gap, in messages: promote a mirror with // non-zero lag and you publish on top of a stream that is still // missing tail messages. s = js_GetStreamInfo(&si, js, "ORDERS_DR", NULL, &jerr); if ((s == NATS_OK) && (si->Mirror != NULL)) { printf("Mirror of : %s\n", si->Mirror->Name); printf("Lag : %" PRIu64 " messages\n", si->Mirror->Lag); // Active is how long ago the mirror last heard from the // upstream. If the upstream site is already gone, this // climbs and Lag freezes at whatever it was when contact // dropped — that frozen Lag is your data loss. printf("Last seen : %.1fs ago\n", (double) si->Mirror->Active / 1E9); // Lag 0 means the mirror has caught up to every message the // upstream had at last contact: it is safe to promote. if (si->Mirror->Lag == 0) printf("Safe to promote\n"); else printf("Messages still in flight — wait and re-check\n"); } ``` If the `east` site is fully unreachable, the mirror can't reach its upstream and the lag stops at whatever it was when contact dropped. That stalled number is your real recovery point: the messages written to `ORDERS` after the last successful copy are lost. Note it, then proceed. Waiting for a lag that will never move to zero only extends the outage. ### Step 2 — drop the mirror config A mirror is **read-only** by design: it rejects direct publishes because its only job is to follow its upstream. To make `ORDERS_DR` writable, you remove the mirror relationship from its configuration. ``` # At site2: edit ORDERS_DR so it is no longer a mirror. # Removing the mirror source makes the stream a standalone primary. nats --server nats://site2:4222 stream edit ORDERS_DR --no-mirror ``` Once the mirror config is gone, the stream stops following `east`. It still holds every message it had copied. Promotion doesn't touch the data, only the relationship. ### Step 3 — clear the lost stream's assignment The subject bind in the next step fails if the old `ORDERS` assignment is still in the JetStream metadata. The server checks the subjects you're adding against every stream in the account, and losing the site doesn't remove the dead `ORDERS` on its own, so the edit comes back with `subjects overlap with an existing stream (10065)`. Clear the stale entry first: ``` # Remove the assignment for the lost ORDERS so its subjects are free. nats --server nats://site2:4222 stream rm ORDERS --force ``` This is the official promotion order: free the mirrored stream's subjects, drop the mirror config (step 2), then bind those subjects to the promoted stream (step 4). ### Step 4 — add the subjects so it accepts writes A mirror has no subjects of its own; it receives messages through the mirror mechanism, not by listening on `orders.>`. A writable primary needs to *bind* those subjects so publishers can reach it. Add them: ``` # Give the promoted stream the subjects ORDERS used to own. nats --server nats://site2:4222 stream edit ORDERS_DR --subjects "orders.>" ``` `ORDERS_DR` now captures `orders.created`, `orders.shipped`, `orders.canceled` — the same subjects the lost primary held. It now functions as a full primary. ### Step 5 — redirect publishers and consumers The last step redirects traffic. Point `order-svc` and the consumers at `site2` and they resume against the promoted stream: ``` # Publishers and consumers now connect to site2. nats --server nats://site2:4222 pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` The order JSON is byte-for-byte what `order-svc` always sent; only the server address changed. Failover is complete: the platform writes and reads at `site2`, and the data loss is exactly the stalled lag you noted in step 1. Two deployment preconditions sit under this whole sequence. First, every `stream rm` and `stream edit` here goes through the JetStream metadata group, so that group has to keep quorum after the site is lost — if the failed site held the meta majority, no edit succeeds until the cluster recovers. Second, the promoted stream must live where that quorum survives. That's why a DR mirror is normally placed in its own JetStream domain (a leaf node) or an independent cluster, rather than sharing one meta group that spans both sites. How the mirror replicated those messages in the first place (the config, the filters, the start position) is the JetStream chapter's job, covered in [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md). The runbook only reads the lag and changes the relationship. ## Recovery from a mistake: restore the snapshot The other rows of the table roll back rather than fail over. When corrupt messages were written to `ORDERS`, the mirror is no help, because it copied the bad writes; the snapshot is the intact copy, taken before the bad event. An accidental delete or purge is the exception: the mirror keeps everything it had already replicated, so it's often the freshest surviving copy — recover from it when it holds more than the last snapshot, and fall back to the snapshot otherwise. #### CLI ``` #!/bin/bash # Recovery for a hard site loss or an accidental delete: rebuild ORDERS # from the last off-site snapshot. Restore recreates the stream byte # for byte — same messages, same sequence numbers, same config, and # (because this snapshot was taken with --consumers) the same shipping # and analytics consumers at their saved delivery position. # # Do NOT recreate an empty ORDERS first. Restore creates the stream # itself; an existing stream of the same name makes the restore fail. # After an accidental delete, restore straight from the snapshot. nats --server nats://127.0.0.1:4222 stream restore ./backups/orders/2026-06-04 # Expected tail of the output: # # Starting restore of Stream "ORDERS" from ./backups/orders/2026-06-04 # ... # Restored stream "ORDERS" in 0.38s # # Information for Stream ORDERS # ... # Messages: 1,000 # First Sequence: 1 # Last Sequence: 1,000 # Active Consumers: 2 # # Then verify: the message count and last sequence must match what the # snapshot held. Everything published after the snapshot was taken is # gone — that gap is your recovery point. A snapshot you never test is # a guess, so rehearse this restore on a schedule, not during the # outage. ``` For an accidental delete, restore straight from the snapshot. Don't recreate an empty `ORDERS` first, because restore creates the stream itself and an existing stream of the same name makes it fail. For logical corruption, stop the publishers first so no new bad data races in, then either purge the corrupt sequence range or restore a known-good snapshot that predates it. For a lost consumer position, a snapshot taken with `--consumers` brings the durable consumer config and its delivery position back together. The mechanics of the snapshot itself (chunking, the `backup.json`, the restore name rule) live one page back on [Stream backup and restore](/learn/backup-recovery/stream-backup-restore.md). Here it's one step in a larger procedure. ## Pitfalls The runbook fails most often on the order of the steps and the assumptions around them rather than on the commands themselves. Four common mistakes happen mid-outage. **Never promote a mirror before lag reaches zero.** Promotion makes the mirror writable. If you do it while messages are still in flight from the upstream, those tail messages are lost and new writes land on top of the gap. Always run the lag check first, and only proceed at `Lag: 0` or with a stalled lag you've consciously accepted as your recovery point. Read the lag and don't skip step 1. You can make that check a gate. Run the lag check from [step 1 above](#step-1--verify-the-lag-is-zero), decide on the number, then promote, never the reverse. **R3 replication will not save you from a mistake.** A three-replica stream survives a node loss, but an accidental delete or a bad publish replicates to all three copies at once. R3 is availability, not a backup. Don't put it on the mistake rows of the table; that's what snapshots are for. **Stop publishers before purging corrupted messages.** Purging a bad sequence range while `order-svc` is still writing lets new corrupt data arrive behind you, so you keep purging against a tail that keeps growing. Stop the publishers, purge or restore, then resume. **An untested snapshot is unverified.** A healthy `nats stream info` on the live stream tells you the live stream is healthy; it proves nothing about the archive in `./backups/orders/`. Rehearse the restore on a schedule (quarterly is a reasonable minimum) into a throwaway stream or server, so the first time you run it isn't during the outage. ## Where you are You can now name a NATS failure and reach for the right tool quickly. A lost site means promote `ORDERS_DR`: verify lag, drop the mirror config, add the subjects, redirect traffic. A mistake (delete, corruption, or a lost consumer position) means restore the snapshot, the copy that predates the mistake — a mirror copies a corrupt write and only freezes at an upstream delete. R3 is on neither path; it's availability, not recovery. The data plane is now fully covered: a snapshot you can restore to, and a site where you can promote the mirror. ## What's next One layer is still unprotected. If a laptop full of keys is lost, or the servers are rebuilt clean, the data is recoverable but no one can prove who they are: the operator JWT, the account JWTs, the nkeys, and the creds are gone. The next page backs up the **identity** plane. Continue to [Config and JWT backup](/learn/backup-recovery/config-and-jwt-backup.md). ## See also * [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) — how a mirror replicates, the mechanism the runbook only reads the lag of. * [Stream backup and restore](/learn/backup-recovery/stream-backup-restore.md) — the snapshot internals the restore step depends on. * [Clustering](/learn/clustering/.md) — R3 leader election, the availability story that is not a recovery story. --- # Mirrors as a DR tool The last page gave you a **snapshot**, a point-in-time copy of `ORDERS` you can restore from. A snapshot gets back the data as it stood at the moment you took it. The data written *since* that moment is gone, and a restore takes as long as it takes to stream the archive back in. This page adds the second tool of the triad: a **mirror**. A mirror is a read-only live copy of `ORDERS` running at a second site, kept current by replication. A snapshot answers "what point can I return to", and a mirror answers "what site can I promote to take over". You'll stand one up, watch how far it trails the original, and learn the one statement that keeps a mirror from being treated as more than it is. This page applies a mirror to disaster recovery. It doesn't teach how a mirror replicates internally (the start position, the subject handling, the fan-in rules). That mechanism is taught in full at [Mirrors & sources](/learn/jetstream/mirrors-and-sources.md), and this page links to it rather than repeating it. ## A mirror is a live copy at a second site So far the `ORDERS` stream lives on the `east` cluster (`n1-east`/`n2-east`/`n3-east`). That cluster is one site. If the whole site fails (a power loss, a network partition that strands the region, a data-center failure), every replica of `ORDERS` fails with it. The R3 replication that protects you from losing *one node* does nothing when you lose the *whole cluster*. A mirror lives somewhere else. Call the second site `site2`: a separate cluster, reachable from the primary over a gateway or leaf-node link you built in the Topologies chapter. On `site2` you create a stream named `ORDERS_DR` whose only job is to copy `ORDERS`. It accepts no direct writes from your services. It receives every message the upstream stores and stores it too. The stream `ORDERS_DR` copies from is its **upstream stream** (here, `ORDERS` on `east`). The copy flows one way: upstream to mirror, never back. Create the mirror on `site2`, pointing it at the upstream: ``` # Run against a server in site2, where ORDERS_DR will live. nats --server nats://site2:4222 stream add ORDERS_DR \ --mirror ORDERS --defaults ``` The `--mirror ORDERS` flag is the whole DR setup from the data side; `--defaults` just accepts the standard storage and retention answers instead of prompting for them. The server on `site2` opens a replication link to the upstream and begins pulling messages. Every order that lands in `ORDERS` on `east` shows up in `ORDERS_DR` on `site2` a short time later. Decide the topology before you run that command, because a mirror's configuration is fixed once the stream exists: you can't re-point a running mirror at a different upstream or change what it copies in place. Changing it means deleting `ORDERS_DR` and recreating it, after which the messages re-replicate from the upstream. Plan the upstream, the site, and the subjects you want once, upfront. The full set of mirror configuration (start position, subject filtering, sourcing from many streams) is covered at [Mirrors & sources](/learn/jetstream/mirrors-and-sources.md). For disaster recovery you only need the plain 1:1 copy above. **Message flow — Mirror as a hot standby (animated):** Replication as a steady-state DR picture, with no promotion. order-svc publishes to ORDERS on site1, which holds the authoritative copy. ORDERS\_DR on site2 is a mirror that pulls new messages across the link continuously, with no client involvement. A lag counter — how many messages ORDERS\_DR is behind ORDERS — trends toward zero as the mirror works through the backlog. When lag reaches 0, ORDERS\_DR holds a complete, up-to-date copy and site2 stands ready as a hot standby while site1 still serves all traffic. * order-svc → site1 · ORDERS * site1 · ORDERS → site2 · ORDERS\_DR The animation shows the steady state: `order-svc` writes the canonical order to `ORDERS` on the `east` cluster, and each message replicates across to `ORDERS_DR` on `site2`. Watch the `Lag` counter: it's the next concept, and it's the number that determines the outcome of a failover. ## Lag is how far the mirror trails Replication isn't instant or synchronous. The upstream stores a message and acknowledges the publisher *before* the mirror has it. The mirror catches up a moment later. That gap is called **lag**: how many messages the mirror trails behind its upstream. You read lag from the mirror's own stream info: ``` nats --server nats://site2:4222 stream info ORDERS_DR ``` The output carries a `Mirror` section the upstream stream doesn't have: ``` Mirror Information: Stream Name: ORDERS Lag: 0 Last Seen: 1.20s ``` Three fields are relevant here: `Stream Name`, `Lag`, and `Last Seen`. `Stream Name` confirms the upstream this mirror copies: `ORDERS`. If it says anything else, the mirror points at the wrong source. `Lag` is the count of messages the upstream has that the mirror doesn't have yet. `Lag: 0` means the mirror holds every message the upstream holds. Any number above zero is the data you'd lose if the primary vanished this instant. `Last Seen` is how long ago the mirror last heard from its upstream. A small number, a second or two, means the link is healthy. A growing `Last Seen` means the mirror is no longer keeping up, and the `Lag` you read is already stale. These two numbers are your **RPO**, the recovery point objective: how much data you can afford to lose. A mirror at `Lag: 0` gives you an RPO of zero messages; a mirror that trails by thousands gives you an RPO of thousands. Read this number *before* you ever trust the mirror in a real failover. The disaster-recovery page makes "is lag zero?" the first step of promotion for exactly this reason. A failover isn't the only time to look. A mirror that quietly stops keeping up is worth catching long before the day you need it, so watch `Lag` continuously rather than checking it once. The server surfaces the same `Lag` and `Last Seen` fields through its monitoring endpoints for an alert to scrape (in the JSON the last-seen field is named `active`). Wiring that up is the [Monitoring](/learn/monitoring/.md) chapter's job. ## A mirror is not a backup One statement captures the limit of a mirror: > A mirror follows the upstream's live writes, so a corrupt write lands in the mirror too — and a mirror keeps no earlier state to rewind to. A mirror gives you **availability**. Lose the whole `east` site and `ORDERS_DR` on `site2` still holds your orders, ready to take over. That's a matter of recovery *time*: your **RTO**, how long recovery takes. A mirror's RTO is short, because the data is already there. A mirror does not give you a **recovery point** you can rewind to. It tracks the upstream's live tail: write a corrupt batch and the corruption replicates into `ORDERS_DR` as faithfully as a good write. The mirror holds only what the upstream has now, never an earlier state, and its own retention limits can age messages out on their own schedule. An upstream delete or purge behaves differently from what people expect. Delete the upstream `ORDERS` and `ORDERS_DR` is *not* deleted with it: it keeps every message it had already copied, stops receiving new ones, and records the fault in the `Error` field of its Mirror Information. Purge a range on the upstream and the messages the mirror already stored stay put; the purge only caps what the mirror will still receive, because it detects the sequence gap and skips the missing messages. So a mirror can outlive a mistake on the upstream — but only as a stale copy frozen at the break, never as a point in time you chose. A snapshot differs here. It's fixed at the moment you took it, so a delete or a corruption that happens *after* the snapshot can't affect it. Because it's fixed, you can rewind to it. So the two tools cover two different failures, and you need both: * **Mirror** → the site failed. Promote the copy and redirect traffic to it. Short RTO, no data loss if lag was zero. * **Snapshot** → the data is wrong (deleted, purged, corrupted). Restore the point in time before it went wrong. Bounded RPO, restore-length RTO. Don't let a healthy mirror be your reason to stop taking snapshots. A mirror doesn't protect against the failure snapshots exist for. And don't reach for R3 replication here either. R3 keeps `ORDERS` available across the loss of one node in the `east` cluster: availability inside one site, not a backup and not a second site. A corrupt write replicates across all three R3 replicas just as faithfully as it lands in a mirror. Why R3 is availability and never a backup is taken up on the [disaster-recovery](/learn/backup-recovery/disaster-recovery.md) page, and leader election is covered at [Clustering](/learn/clustering/.md). ## Pitfalls A mirror is easy to create and easy to over-trust. Each trap below is scoped to this page's two ideas: the mirror as a DR copy, and the distinction between a mirror and a backup. **A mirror is not a backup.** It's the main point of this page and the most common mistake. Corrupt the upstream and the corruption replicates into the mirror as faithfully as a good write; delete or purge the upstream and the mirror survives only as a stale copy frozen where replication stopped. Either way it gives you no earlier state to rewind to. Don't run a mirror *instead of* snapshots. Pair them: the mirror for site failure, the snapshot for bad data. **Read `Lag` before you trust the copy.** Replication is eventually consistent, not synchronous, so a mirror can trail its upstream by an unknown amount at any moment. If you've never checked a mirror, you don't know how far it trails. Read `Lag` and `Last Seen` from the mirror's stream info and confirm the link is current before you depend on it: ``` # Confirm ORDERS_DR is caught up. Lag should read 0, and Last Seen # should be a small, recent number. A growing Last Seen means the # Lag you just read is already stale — the link is falling behind. nats --server nats://site2:4222 stream info ORDERS_DR | grep -A4 "Mirror Information" ``` If `Lag` is non-zero or `Last Seen` keeps climbing, the mirror is behind. Diagnose the link before a failover, never during one. **A mirror's config is effectively locked after creation.** As the setup section warned, you can't re-point a running mirror or change what it copies in place. Don't treat `ORDERS_DR` as something you'll tune later; settle the upstream, the site, and the subjects upfront. To change any of them, delete the mirror and recreate it, and the messages re-replicate from the upstream. **Avoid a Work Queue upstream under a mirror.** A Work Queue stream is built to hand each message to exactly one consumer. To replicate, a mirror creates a hidden internal consumer on the upstream, and that consumer is a *direct* consumer that bypasses the work queue's subject-overlap check. So a regular worker and the mirror's consumer can both receive the same message, which defeats precisely the single-consumer guarantee the work queue exists to enforce. Use a `Limits` upstream instead; the retention policies are covered at [Mirrors & sources](/learn/jetstream/mirrors-and-sources.md). ## Where you are You now have: * An `ORDERS_DR` mirror running at `site2`, copying `ORDERS` from the `east` cluster one message at a time. * A way to read its `Lag` and `Last Seen` fields, and the knowledge that `Lag: 0` is the condition that makes a failover safe. * The distinction between the two DR tools: a mirror gives you a short RTO for a site failure; a snapshot gives you a recovery point for bad data. Neither replaces the other, and R3 replaces neither. What you *don't* have yet is the procedure for actually using the mirror when the primary fails: verifying lag, promoting the copy to a writable primary, and redirecting your publishers and consumers to `site2`. That procedure is the runbook. ## What's next The next page is the disaster-recovery **runbook**: which tool to reach for per failure class, and the exact steps to **promote** `ORDERS_DR` into a writable `ORDERS` when the `east` site is gone. Continue to [Disaster recovery](/learn/backup-recovery/disaster-recovery.md). ## See also * [Mirrors & sources](/learn/jetstream/mirrors-and-sources.md) — how a mirror replicates: start position, subject handling, and sourcing. * [Cross-account export & import](/learn/security/cross-account.md) — what a cross-account mirror's export/import is, which you must back up too. * [Super-clusters](/learn/topologies/super-clusters.md) — the gateway links that connect `east` to `site2`. --- # Stream backup and restore The `ORDERS` stream holds every order Acme has ever taken. Replication keeps it available when a node dies, but it doesn't protect you from a mistake: an accidental `nats stream purge`, a bad migration, or a logic bug that deletes the wrong messages. To recover from those you need a copy the cluster can't touch — a point in time you can return to. This page makes that copy and proves it works. It introduces two operations and nothing else: taking a snapshot of `ORDERS`, and restoring the stream from one. ## A snapshot is a point-in-time copy A **snapshot** is a complete copy of a stream as it exists at one instant: every message, the stream's configuration, and, if you ask for it, the state of its consumers. The server takes the snapshot and streams it to you in chunks; the CLI writes those chunks to a directory. That directory holds two things. A `backup.json` file records the stream's configuration and state: its subjects, retention, limits, and sequence range. Alongside it, `stream.tar.s2` is the messages themselves, packed into a tarball and compressed with S2. Together they're everything you need to recreate the stream from scratch. Take a snapshot of `ORDERS` into a dated, off-site directory: #### CLI ``` #!/bin/bash # Take a point-in-time snapshot of the ORDERS stream and write it to a # dated, off-site directory. The snapshot is two files: a backup.json # (the stream config + state) and a chunked, S2-compressed tarball # (stream.tar.s2) carrying the messages. # # --consumers includes the durable consumer config and delivery # position, so a restore brings back not just the messages but the # shipping and analytics consumers exactly where they were. nats stream backup ORDERS ./backups/orders/2026-06-04 --consumers # Expected tail of the output: # # Starting backup of Stream "ORDERS" with 1 data file # ... # Received 4 MiB compressed data in 128 chunks for stream "ORDERS" # in 0.41s, 4.0 MiB uncompressed # Backup of "ORDERS" took 0.41s # # After this, ./backups/orders/2026-06-04/ holds backup.json and # stream.tar.s2. Ship that directory off-site (see the cron + encrypt # script on the page) — a snapshot left next to the live cluster does # not survive the event that takes the cluster down. ``` The directory name carries the date on purpose. A snapshot is a point in time, and naming it `2026-06-04` makes that explicit. Tomorrow's snapshot goes in `2026-06-05`, and you keep the dated snapshots under `./backups/orders/`. We make that automatic later; here it's one command. Consumer state matters more than it looks. By default the snapshot records each durable consumer's config and delivery position: the **consumer state**. Restore that snapshot and the `shipping` and `analytics` consumers come back exactly where they left off, not at the start of the stream. Pass `--no-consumers` and the snapshot carries the messages but forgets the consumers reading them, so keep the default when you back up a production stream. ## How the snapshot streams off the server A snapshot doesn't arrive as one big download. The server cuts the tarball into chunks and pushes them to an inbox subject, keeping up to a window's worth of unacknowledged chunks in flight at once — 8 MiB by default, which is 64 of the default 128 KiB chunks. Each client ack frees a slot for the next chunk, and if no ack arrives for about five seconds the backup aborts. That windowed backpressure keeps a large stream from overwhelming a slow disk or a high-latency link. **Message flow — Stream snapshot streaming (animated):** Backing up the ORDERS stream with nats stream backup. The backup client sends a snapshot request to the ORDERS stream and opens an ephemeral inbox to receive the data. The server answers with the stream's config and state, then drains its store as an S2-compressed tar archive and streams stream.tar.s2 to the inbox as a sequence of chunks. The client returns a flow-control ack per chunk; the server keeps a window of unacknowledged chunks in flight and uses the acks as backpressure so a slow disk or link can't be overrun. When the last chunk is acked, backup.json (config and state) and stream.tar.s2 (the messages) land in the off-site backup store. * backup client → ORDERS stream (subject: snapshot request) * ORDERS stream → backup client (subject: config + state) * ORDERS stream → \_INBOX deliver (subject: stream.tar.s2 chunk) * \_INBOX deliver → ORDERS stream (subject: flow-control ack) * \_INBOX deliver → snapshot request The request lands on the snapshot API, the server answers with the config and state, and then the message chunks flow to the inbox with an ack per chunk until the tarball and `backup.json` are written to the backup store. The two settings that govern this, chunk size and window size, have sensible defaults, and you only change them when the defaults time out. We cover them in the [Pitfalls](#pitfalls). The full set of snapshot request options is documented in [Reference → Snapshot Stream](/reference/jetstream/api/stream/snapshot.md). We only need the behavior here. ## Restore rebuilds the stream A snapshot becomes useful when you turn it back into a stream. **Restore** reads a snapshot directory and recreates the stream from it: same messages, same sequence numbers, same configuration. If the snapshot included consumer state, restore brings the consumers back too. Restore the directory you just wrote: #### CLI ``` #!/bin/bash # Rebuild the ORDERS stream from a snapshot directory. Restore streams # the tarball back into JetStream and recreates the stream byte for # byte: same messages, same sequence numbers, same config, and — since # this snapshot was taken with --consumers — the same shipping and # analytics consumers at their saved delivery position. # # The stream must not already exist. Restore recreates it; it does not # merge into a live stream. nats stream restore ./backups/orders/2026-06-04 # Expected tail of the output: # # Starting restore of Stream "ORDERS" from ./backups/orders/2026-06-04 # ... # Restored stream "ORDERS" in 0.38s # # Information for Stream ORDERS # ... # Messages: 1,000 # First Sequence: 1 # Last Sequence: 1,000 # # Note: the stream name comes from backup.json. You cannot rename the # stream on restore — if you need a copy under a new name, restore to # the original name and then mirror or source it. ``` One rule shapes how you use restore: **the stream name cannot change on restore.** The name lives in `backup.json`, and the server rejects a restore that would land under a different name. A snapshot of `ORDERS` restores as `ORDERS`, never as `ORDERS_COPY`. That keeps a restore unambiguous: it rebuilds one stream under its own name rather than forking it into a new one. If you do need a second copy under a new name, restore to `ORDERS` first and then mirror or source it, which [Mirrors and sources](/learn/backup-recovery/mirrors-and-sources.md) covers. Restore also expects the stream not to already exist. It recreates the stream; it doesn't merge a snapshot into a live one. So a real recovery is: confirm the broken stream is gone (or remove it), then restore. ## Verify the counts A restore you didn't check is unverified. The last step is always to read the rebuilt stream's state back and confirm it matches the source. #### CLI ``` #!/bin/bash # A restore you did not verify is a restore you do not have. After # rebuilding ORDERS, read its state back and confirm the message count # and last sequence match what the snapshot claimed. # # nats stream info prints both halves of the stream: the configuration # you asked for and the state actually on disk. The numbers under State # are the ones to check against the source. nats stream info ORDERS # Look at the State block: # # State: # # Messages: 1,000 # Bytes: 4.0 MiB # First Sequence: 1 # Last Sequence: 1,000 # Active Consumers: 2 # # Messages and Last Sequence must match the live stream you snapshotted. # Active Consumers should show shipping and analytics back (because the # snapshot used --consumers). If Messages is lower than expected, the # snapshot was taken under live writes that did not finish — snapshot a # quiesced stream, or accept the snapshot's point in time as your RPO. ``` #### C ``` // After rebuilding ORDERS, read its state back and confirm the // message count and last sequence match what the snapshot held. s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { printf("Messages : %" PRIu64 "\n", si->State.Msgs); printf("First Sequence : %" PRIu64 "\n", si->State.FirstSeq); printf("Last Sequence : %" PRIu64 "\n", si->State.LastSeq); // With a --consumers snapshot, shipping and analytics are // back, so this should report 2. printf("Active Consumers: %d\n", (int) si->State.Consumers); } ``` Look at the `State` block. `Messages` and `Last Sequence` must match the stream you snapshotted, and `Active Consumers` shows `shipping` and `analytics` back if you used `--consumers`. Matching counts are what prove the archive is real; the backup command exiting zero does not prove it on its own. ## Pitfalls Four common pitfalls, each scoped to this page's two operations: snapshot and restore. **Memory streams cannot be snapshotted.** A snapshot reads a stream's on-disk files, so a stream with `Storage: Memory` has nothing to read. The backup fails with `memory streams do not support snapshots`. Any stream you want to back up needs file storage: set `Storage: File` when you create it. Check the storage type before you rely on a snapshot, rather than finding out during an incident: #### CLI ``` #!/bin/bash # Take a point-in-time snapshot of the ORDERS stream and write it to a # dated, off-site directory. The snapshot is two files: a backup.json # (the stream config + state) and a chunked, S2-compressed tarball # (stream.tar.s2) carrying the messages. # # --consumers includes the durable consumer config and delivery # position, so a restore brings back not just the messages but the # shipping and analytics consumers exactly where they were. nats stream backup ORDERS ./backups/orders/2026-06-04 --consumers # Expected tail of the output: # # Starting backup of Stream "ORDERS" with 1 data file # ... # Received 4 MiB compressed data in 128 chunks for stream "ORDERS" # in 0.41s, 4.0 MiB uncompressed # Backup of "ORDERS" took 0.41s # # After this, ./backups/orders/2026-06-04/ holds backup.json and # stream.tar.s2. Ship that directory off-site (see the cron + encrypt # script on the page) — a snapshot left next to the live cluster does # not survive the event that takes the cluster down. ``` If that command errors on a memory stream, recreate the stream with file storage before it holds anything you can't lose. **The stream name cannot change on restore.** As above, the server rejects a restore that would rename the stream with `stream name may not be changed during restore`. Restore to the original name. If you need a copy under a different name, restore first and then mirror or source it; [Mirrors and sources](/learn/backup-recovery/mirrors-and-sources.md) covers that. **Flow control can time out on slow disks or distant links.** The server waits a few seconds for the client to acknowledge each chunk. On a slow disk or a high-latency link the acknowledgment can arrive late, and the backup aborts with a flow-control timeout (`408 No Flow Response`). The fix is to send smaller chunks and a smaller window so each round trip is cheaper: ``` # Smaller chunks and a smaller window survive a slow or distant link. nats stream backup ORDERS ./backups/orders/2026-06-04 \ --consumers --chunk-size 64k --window-size 1m ``` **`--no-consumers` silently drops consumer state.** Consumer state is included by default; pass `--no-consumers` and the snapshot carries messages only. The restore rebuilds the stream with no consumers, and nothing warns you until `shipping` is missing in production. Keep the default unless you plan to recreate every consumer by hand. ## Where you are You now have: * A dated, off-site snapshot of `ORDERS` under `./backups/orders/`: `backup.json` plus a compressed `stream.tar.s2`. * A restore procedure that rebuilds the stream byte-identical, under its original name, with its consumers intact. * A verification step that proves the restore by matching message counts. This is your recovery point: the instant you can return to after an accidental delete or a logic error. On its own, though, it won't protect you from losing the whole site. For that you need a live copy somewhere else. ## What's next The next page stands up that live copy. A **mirror** of `ORDERS` at a second site keeps a continuously updated copy you can fail over to, and it distinguishes a snapshot (your recovery point) from a mirror (your recovery time). Continue to [Mirrors and sources](/learn/backup-recovery/mirrors-and-sources.md). ## See also * [Reference → Snapshot Stream](/reference/jetstream/api/stream/snapshot.md) — every snapshot request option and its valid range. * [Reference → Restore Stream](/reference/jetstream/api/stream/restore.md) — the restore request schema. * [Reference → Snapshot Create Advisory](/reference/jetstream/advisory/snapshot-create.md) — the event the server emits when a snapshot starts, for backup alerting. --- # Where to go next You started this chapter with a built but unprotected platform: an `ORDERS` stream full of real orders, the `ACME` operator and its accounts deciding who may touch them, all spread across the `east` cluster, with no backup or recovery in place. You end it with a dated snapshot under `./backups/orders/`, an `ORDERS_DR` mirror running at a second site, and an encrypted off-site copy of every key that proves who you are. That covers the full progression. This page doesn't teach anything new. It collects the model you built into one place and points you at the chapters and Reference that take it further. ## The three copies Every page in this chapter covered the same three ideas. They protect against different failures, and they don't protect each other. These are the key points to retain. A **snapshot** is the point you can return to. It's a point-in-time copy of a stream (its messages, its config, and optionally its consumer state) written off-site. When a delete or a corruption occurs, the snapshot is the only thing that lets you go *back* to before the mistake. It answers how much data you can afford to lose: your **recovery point**. A **mirror** is the site you can fail over to. It's a live, read-only copy of `ORDERS` running at a second site, kept current by replication and tracked by its `Lag`. It doesn't give you a point in the past; it gives you a place to keep running when the first site fails. It answers how long recovery may take: your **recovery time**. **Identity** is the proof of who you are. It's the set of files that *are* your security layer: the operator and account JWTs, the nkeys that sign them, the user creds, and the server config. A restored stream that nobody is allowed to read is not a recovery. Without these files the data plane comes back but the platform stays unusable. The three are snapshot, mirror, and identity. The item that is *not* on the list: **R3 is availability, not a backup.** Three replicas keep a stream alive through a node loss, but a bad write replicates to all three just as a good one does. R3 keeps the stream up; only a snapshot lets you undo a mistake. ## Where the details live The chapter is unversioned and concept-first. The exact flags, defaults, and ranges live in **Reference**, which is versioned and exhaustive. When you need the precise chunk-size clamp on a snapshot request, the full restore schema, or the advisory subjects to alert on, that's where to look. The [Reference root](/reference/.md) is the entry point. The handoff phrases throughout this chapter ("the full set of options is documented in Reference") all point into it. The snapshot and restore request schemas live at [Snapshot Stream](/reference/jetstream/api/stream/snapshot.md) and [Restore Stream](/reference/jetstream/api/stream/restore.md); the resolver knobs behind identity restore live at [resolver config](/reference/config/resolver/.md). ## Sibling deep dives This chapter sits in the Operate half of Learn, downstream of the chapters that *built* the platform. The others around it go deeper than any single page here could. The [JetStream deep dive](/learn/jetstream/.md) is where the data this chapter protects comes from, and where the mirror mechanism is taught in full. When you want to know *how* a mirror replicates (start position, filters, fan-in), read [JetStream → Mirrors and sources](/learn/jetstream/mirrors-and-sources.md). This chapter only applies a mirror to disaster recovery and reads its lag. The [Security deep dive](/learn/security/.md) is where the identity this chapter backs up comes from. When you want to know *what* an operator, account, or signing key is, and how a trust chain validates, read [Security → Operator mode](/learn/security/operator-mode.md). This chapter only copies those files off-site and puts them back. The [Clustering & Replication deep dive](/learn/clustering/.md) owns the R3 story this chapter deliberately set aside: how a stream elects a leader, how placement works, and why replication is availability rather than a recovery point. The [Monitoring deep dive](/learn/monitoring/.md) covers the other half of running this safely: watching the snapshot advisories fire, alerting on a mirror whose lag stops trending to zero, and proving a restore worked before you need it. The [Deployment deep dive](/learn/deployment/.md) covers sizing the disks and store directories that a restored stream lands on, the capacity question this chapter assumes you've already answered. ## Where you are This is the end of the chapter: the triad is complete, and this page introduces no new scenario state. The `east` cluster, the `ORDERS` stream, the `ORDERS_DR` mirror at the second site, and the off-site identity backup are all exactly as you left them on the previous page. You can rehearse a failover against them, schedule a test restore, or tear the practice copies down when you're done. You have the operational model: a snapshot for the point you can return to, a mirror for the site you can fail over to, identity backup for the keys that prove who you are — and R3 as availability, never as any of those three. ## Production checklist Every content page in this chapter closed with a Pitfalls section. This page collects the action items from all of them in one place: a last pass before you trust this platform with real orders and a real outage. Each group links back to the page that explains the why. ### Stream backup and restore — see [Pitfalls](/learn/backup-recovery/stream-backup-restore.md#pitfalls) * Use file storage for any stream you must snapshot; a memory stream cannot be backed up and the attempt fails outright. * Restore to the original stream name; the name may not change during restore, so mirror or source afterward if you need a copy under a new name. * Reduce `--chunk-size` and `--window-size` on a slow disk or a high-latency link before the flow-control timeout fires and aborts the snapshot. * Keep consumer state in the snapshot; it is included by default, and passing `--no-consumers` silently drops durable consumer config and delivery position. ### Mirrors as a DR tool — see [Pitfalls](/learn/backup-recovery/mirrors-and-sources.md#pitfalls) * Pair every mirror with snapshots; a mirror is not a backup — a corrupt write replicates to the copy, and a mirror keeps no earlier state to rewind to. * Plan the mirror topology upfront; a mirror's config is effectively locked after creation, so changing it means delete and recreate. * Read the `Lag` field before trusting a mirror; replication is eventually consistent, not synchronous. * Avoid Work Queue retention on a mirrored upstream; the mirror's internal consumer bypasses the work queue's subject-overlap check and breaks the single-consumer guarantee, so use Limits. ### Disaster recovery — see [Pitfalls](/learn/backup-recovery/disaster-recovery.md#pitfalls) * Verify lag has reached 0 before promoting a mirror; promote early and you publish on top of a stream still missing its tail. * Reach for a snapshot, not R3, against an accidental delete or a logical error; the bad write replicates, so availability cannot undo it. * Test restore on a schedule, quarterly at least; a green `nats stream info` on the live stream proves nothing about the archive. * Stop publishers before purging corrupted messages; purging under live writes races new bad data in. ### Config and JWT backup — see [Pitfalls](/learn/backup-recovery/config-and-jwt-backup.md#pitfalls) * Seal every `nats auth operator backup` with `--key` and store the curve seed apart from the backups; the file carries every private seed, and losing store plus key together is losing the identity. * Re-push each account after an identity restore if the server lost its resolver directory; the restore rebuilds only your store, and the push needs the `server.conf` whose `SYSTEM` preload lets it in — so back that file up too. * Tag each identity backup with the operator version or timestamp; an unrecorded operator rotation leaves the backup pointing at a dead operator. ## See also * [Reference](/reference/.md) — every config field, flag, default, and error code behind this chapter, versioned and exhaustive. * [JetStream → Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) — how a mirror actually replicates, the mechanism this chapter applies but never re-teaches. * [Clustering & Replication deep dive](/learn/clustering/.md) — the R3 story this chapter deliberately set aside. --- # Clustering & Replication Deep Dive A NATS cluster involves more than three servers wired together. Beneath that wiring is a mechanism by which the servers find each other, elect leaders, agree on every write, and keep replicas in step. This chapter covers that mechanism, taught one layer at a time the way you'd learn it by running a real cluster and watching it work. The [Topologies](/learn/topologies/.md) chapter stood up the shapes (one server growing into the `east` cluster, then a super-cluster, then leaf nodes) and deliberately left the internals for here. The [JetStream](/learn/jetstream/.md) chapter gave you one page on [surviving node loss](/learn/jetstream/surviving-node-loss.md): set `R=3`, lose a server, keep serving. This chapter explains how both of those work. We go beneath the shapes to the agreement and replication that make them work. ## By the end you'll have * A live three-server cluster (`n1-east`, `n2-east`, `n3-east`, the same `east` cluster from Topologies) that discovered itself from a single seed route. * The `ORDERS` stream running at `R=3`, with a write you can trace from `order-svc` publishing `orders.created`, through the leader's log, to a quorum commit that survives one server dying. * A working mental model of the five moving parts: **routes** form the mesh, **RAFT groups** agree, a **quorum** commits each write, **placement** decides where replicas live, and **peer management** grows or shrinks the set safely. * The commands to inspect all of it (leaders, replicas, lag) and the failure modes to watch for when you operate it for real. ## Who this is for You've read the [Core Concepts](/concepts/what-is-nats.md) primers and, ideally, the [JetStream](/learn/jetstream/.md) and [Topologies](/learn/topologies/.md) deep dives. You know what a stream, a consumer, a route, and a cluster are. This chapter doesn't re-teach them. You don't need to know anything about consensus or replication specifically. We start from "what is a route and how does one server find another" and build up to placement and peer management from there. ## How to read it Each page introduces at most two new concepts and carries the cluster forward. You stand up `n1-east`, `n2-east`, and `n3-east` on the first page and keep them running: later pages elect a leader on that same cluster, replicate a write to it, place the stream on tagged servers, and add or remove a peer. The server names, the `ORDERS` stream, and the payload never change. The vocabulary here is dense: leader, follower, term, quorum, commit, apply. Every term is defined before it's used, so read the pages in order rather than jumping. Where a feature has a long list of knobs or timer values, the page teaches only what you need to understand the mechanism and links to [Reference](/reference/.md) for the rest. This chapter is replication *within one cluster*. Replication *across* clusters (gateways, geo-affinity, super-cluster traffic) stays in [Super-clusters](/learn/topologies/super-clusters.md). ## Map | Page | What you learn | | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | [Forming a cluster](/learn/clustering/forming-a-cluster.md) | Routes (explicit seed vs implicit gossip) and how one seed grows into a full mesh | | [Raft and leaders](/learn/clustering/raft-and-leaders.md) | RAFT groups, the meta leader and stream leaders, and how an election picks one | | [Replication and R=3](/learn/clustering/replication-and-r3.md) | How a quorum commits a write, then followers apply it, and the consistency you get | | [Placement](/learn/clustering/placement.md) | Constrain replicas to a cluster and tagged servers, and move a stream's leader to a chosen server | | [Scaling and peer management](/learn/clustering/scaling-and-peers.md) | Grow the group with catchup, move a replica off a server safely, and never lose quorum doing it | | [Where to go next](/learn/clustering/where-next.md) | A recap of the whole mechanism and a production checklist | In summary, servers form a mesh, elect leaders, replicate every write, place the replicas where you want them, and let you scale the peer set without losing agreement. ## Prerequisites You'll need: * Three local `nats-server` processes with JetStream enabled, which the first page stands up as the `east` cluster. A single server isn't enough here: RAFT needs a majority, so the mechanism only appears once three servers are talking. * The `nats` CLI installed. Most of this chapter is server config and `nats` operator commands; a few stream operations also show the JavaScript, Go, Python, Java, Rust, and C# client form, since every client sets the same replica count and placement. * A system account (`$SYS`) to run the `nats server` operator commands against. These configs don't set one up; add a system user (the [Security deep dive](/learn/security/.md) covers it) and connect with its credentials before the first `nats server list`, `info`, or `report`. Open a terminal and turn to [Forming a cluster](/learn/clustering/forming-a-cluster.md). ## See also * [Topologies deep dive](/learn/topologies/.md) — the shapes this chapter runs beneath, including the `east` cluster it reuses. * [JetStream → surviving node loss](/learn/jetstream/surviving-node-loss.md) — the one-page replica intro this chapter goes deeper than. * [Reference](/reference/.md) — the exhaustive config, protocol, and monitoring detail behind every mechanism here. --- # Forming a cluster The [Topologies chapter](/learn/topologies/your-first-cluster.md) wired three servers into the cluster `east` and showed a client surviving a server loss. It covered the `cluster {}` block and the shape it produces. It did not cover the *how*: how a server you pointed at one peer ends up holding a route to every peer. This page covers that. It's the first mechanism of the chapter, the one every later page builds on. Before servers can elect a leader or replicate a write, they have to find each other. Two ideas do that work: a route, the connection one server opens to another, and gossip, the way servers tell each other who else is in the cluster. We use the same `east` cluster the whole way: `n1-east`, `n2-east`, and `n3-east`, client ports 4222/4223/4224, route ports 6222/6223/6224. By the end of the page they're running and have discovered themselves from a single seed. ## A route is a server-to-server connection A **route** is the connection one `nats-server` opens to another so the two act as one cluster. It isn't a client connection. A client connects on the client port (4222); a route connects on a separate route port (6222). The two never share a port, and that distinction has practical consequences: half of cluster-formation bugs are a route pointed at the client port. A route is bidirectional once open. Whichever server dialed, both ends afterward send and receive over the same link: subscription interest in one direction, the messages that match it in the other. With three servers fully connected, every server holds a route to the other two, so each is exactly one hop from all the rest. Routes come in two kinds, and this page covers how to tell them apart. An **explicit route** is one you configured. You wrote its address into the `routes` list in `nats.conf`, and the server dials it on startup. This is the seed: the address a fresh server uses to find the cluster at all. An **implicit route** is one the server opened on its own, to a peer it was *not* configured to know. The server learned that peer existed and dialed it without you writing its address anywhere. Implicit routes are how a cluster fills in the connections you didn't configure, and they come from gossip. ## Gossip turns one seed into a full mesh You configured `n2-east` and `n3-east` with a single explicit route each, both pointing at `n1-east`. Yet the running cluster has every server connected to every other. The connections you never wrote appear on their own through **gossip**. Gossip is route discovery by INFO redistribution. When two servers form a route, each sends the other an **INFO** message, a small protocol frame carrying its own address. The receiver learns that peer exists and dials it. A server then forwards that INFO to the other peers it already holds routes to, so each of them learns the new peer and dials it too. Those self-opened connections are implicit routes. Trace it on `east`. `n1-east` boots and waits. `n2-east` boots, dials its explicit route to `n1-east`, and the two exchange INFO. `n3-east` boots, dials *its* explicit route to `n1-east`, and now `n1-east` knows about both newcomers. On the next INFO exchange, `n1-east` tells `n2-east` about `n3-east`. `n2-east` has no route to `n3-east`, so it opens one — an implicit route that completes the mesh. **Message flow — One seed grows into a full mesh (animated):** n1-east boots as the seed and waits. n2-east and n3-east each boot and dial their one explicit route to n1-east, and each pair exchanges an INFO frame carrying the peer's address. n1-east now knows both joiners, so on the next INFO exchange it tells n2-east about n3-east; n2-east has no route to n3-east yet, so it opens one — an implicit route that completes the mesh. Each server needed only a single seed address; gossip discovered the rest. * n1-east → n2-east (subject: explicit route) * n2-east → n1-east (subject: INFO (gossip)) * n1-east → n3-east (subject: implicit route) * n2-east → n3-east (subject: implicit route) This is why each server only needs one seed address. You point them all at `n1-east`; gossip discovers the rest. Adding a fourth server later means giving it one route, to `n1-east`, and nothing else. The wire-level detail of the INFO frame and the route handshake (every field a server advertises, the protocol verbs) lives in [Reference → Route protocol](/reference/protocols/route.md). We only need the behavior here: each INFO announces one peer's address, and a server forwards a new peer's INFO to the routes it already holds so they dial it too. ## Stand up the seed and two joiners These are the `east` configs from Topologies, with JetStream still enabled — the later pages replicate a stream on it. We read them here for what they tell us about routes, not as a shape to choose; that choice belongs to [Topologies → Your first cluster](/learn/topologies/your-first-cluster.md). One detail differs from that page: here `n1-east` carries no `routes` of its own, so the pure seed pattern stands out. `n1-east` is the seed. It carries no `routes` of its own; the others find it. n1-east.conf ``` server_name: n1-east listen: 127.0.0.1:4222 jetstream { store_dir: "./js/n1-east" } cluster { name: east listen: 127.0.0.1:6222 } ``` Three things in the `cluster {}` block control formation. `name` is the cluster identifier, `east`. Every server that should join must set the exact same name. A route to a server whose name differs is rejected the moment the names are compared: the server logs `Rejecting connection, cluster name ... does not match ...` and closes the route, so the odd server forms a separate cluster of its own. See [Pitfalls](#pitfalls) for how to catch it in the log. `listen` is the route port this server accepts routes on: `6222`, one above the client port and never the same as it. This is the port peers dial, not clients. `routes` is the list of seed addresses to dial on startup. `n1-east` has none, so it only waits. The joiners carry one each. `n2-east` is the same pattern with its own ports and a single explicit route back to the seed: n2-east.conf ``` server_name: n2-east listen: 127.0.0.1:4223 jetstream { store_dir: "./js/n2-east" } cluster { name: east listen: 127.0.0.1:6223 routes: [ nats://127.0.0.1:6222 ] } ``` `n3-east` repeats it, one port higher, pointing at the same seed. It does **not** list `n2-east`, and it doesn't need to: gossip will supply that route. n3-east.conf ``` server_name: n3-east listen: 127.0.0.1:4224 jetstream { store_dir: "./js/n3-east" } cluster { name: east listen: 127.0.0.1:6224 routes: [ nats://127.0.0.1:6222 ] } ``` Start all three, each with its own config file: ``` nats-server -c n1-east.conf & nats-server -c n2-east.conf & nats-server -c n3-east.conf & ``` `n1-east` comes up first. As the joiners dial their explicit route to it, they exchange INFO, learn about each other, and open the implicit routes that complete the mesh, all within a moment of the last server starting. The full set of `cluster {}` fields (`pool_size`, `compression`, `authorization`, the per-link `tls {}` block) is documented in [Reference → Cluster config](/reference/config/cluster/.md). We only need `name`, `listen`, and `routes` to form the cluster. ## Confirm the mesh formed Check the cluster's state from the outside. The `nats server` commands here query the system account (`$SYS`), which these configs don't set up; add one (the [Security deep dive](/learn/security/.md) covers it) and connect with its credentials before you run them. ``` nats server list ``` The list shows all three servers as one cluster, each with its route count: ``` ╭─────────────────────────────────────────────────────────────────────────╮ │ Server Overview │ ├─────────┬─────────┬──────┬─────────┬─────┬───────┬──────┬────────┬────────┤ │ Name │ Cluster │ Host │ Version │ JS │ Conns │ Subs │ Routes │ Uptime │ ├─────────┼─────────┼──────┼─────────┼─────┼───────┼──────┼────────┼────────┤ │ n1-east │ east │ ... │ 2.x.x │ yes │ 0 │ 9 │ 8 │ 1m2s │ │ n2-east │ east │ ... │ 2.x.x │ yes │ 0 │ 9 │ 8 │ 58s │ │ n3-east │ east │ ... │ 2.x.x │ yes │ 0 │ 9 │ 8 │ 55s │ ╰─────────┴─────────┴──────┴─────────┴─────┴───────┴──────┴────────┴────────╯ ``` Every row's `Cluster` column reads `east`, so the three joined one cluster and not three separate ones. The `Routes` column counts route connections, not peers: each link to a peer is a small pool of connections (three by default) plus a dedicated system-account route, so a three-server cluster shows several per server. What confirms the mesh is that the count is the same on every row and non-zero. You configured one explicit route per joiner; gossip supplied the rest. For one server's own view of its routes, query it directly: ``` nats server info n1-east ``` This prints `n1-east`'s perspective: its client port, the routes it holds, and the cluster name it belongs to. The route list there is the explicit seed plus every implicit route gossip added. ## Pitfalls A cluster is straightforward to form, but three details have to be correct. Each one leaves you with a running server that isn't in the cluster you meant — sometimes with nothing but a line in the server log to tell you. **A mismatched `cluster.name` forms two clusters.** The name is what binds servers together. Set `name: east` on two servers and `name: eest` on the third, and the third's route is rejected: its log reads `Rejecting connection, cluster name "east" does not match "eest"` and it forms its own one-server cluster that never merges. Unless you read that line, you're left with two clusters that look like one until a message fails to cross. Set the identical `name` on every server, then confirm they joined as one before trusting the cluster: ``` nats server list ``` If every row shows `east`, the mesh is complete. A stray name, or a server missing from the list, means it formed a separate cluster. Fix the `name` and restart it. **Pointing `routes` at the client port (4222) never forms the mesh.** The route port (6222) and the client port (4222) are different listeners. A `routes` entry of `nats://127.0.0.1:4222` aims at the client listener, which speaks the client protocol, not the route protocol. The route never establishes and the server runs alone. Always point `routes` at a peer's route port: `6222`, not `4222`. **One seed is enough, but list two or three anyway.** Gossip means a single seed route is sufficient to discover the whole cluster. The risk is boot ordering: if every joiner seeds off `n1-east` alone and `n1-east` happens to be down when they start, none of them can find the cluster. Listing two or three seed routes lets formation survive any one seed being unavailable at boot. Don't rely on a single seed in production: ``` cluster { name: east listen: 127.0.0.1:6223 routes: [ nats://127.0.0.1:6222 nats://127.0.0.1:6224 ] } ``` ## Where you are The `east` cluster is running and has discovered itself from one seed: * `n1-east`, `n2-east`, and `n3-east` are up on client ports 4222/4223/4224 and route ports 6222/6223/6224. * Each joiner carried one explicit route to `n1-east`; gossip opened the implicit routes that complete the mesh, so every server holds a route to the other two. * `nats server list` shows all three under cluster `east`, each with the same non-zero route count. The servers can now reach each other. What they can't yet do is *agree*: decide together which server owns a stream, and keep that decision when one of them fails. RAFT handles that, and it's the subject of the next page. ## What's next A cluster whose servers can reach each other still can't reach agreement. The next page introduces **RAFT groups** and **leader election**: how the servers in `east` pick a leader for the cluster and for each stream, and how they pick a new one when a leader is lost. Continue to [Raft and leaders](/learn/clustering/raft-and-leaders.md). ## See also * [Reference → Cluster config](/reference/config/cluster/.md) — every field of the `cluster {}` block. * [Reference → Route protocol](/reference/protocols/route.md) — the wire-level route handshake and the INFO frame gossip uses. * [Topologies → Your first cluster](/learn/topologies/your-first-cluster.md) — the same `east` cluster as a deployment shape. --- # Placement By now the `ORDERS` stream runs `R=3` on the `east` cluster, and the meta leader chose which three servers hold it. So far you haven't had a say in that choice: the meta leader picked any three servers with room. This page lets you control that choice. It constrains *where* a stream's replicas land: onto a named cluster, or onto servers carrying labels you assign. Two concepts do all the work, and nothing here changes the payload `order-svc` publishes or the subjects it uses. ## Placement constrains which servers hold the replicas **Placement** is a rule attached to a stream that limits which servers may hold its replicas. Without it, the meta leader is free to put the three copies of `ORDERS` on any servers in `east` that have capacity. With it, the meta leader must honor your constraint or refuse to create the stream. Placement has two levers. The first is the **cluster**: name a cluster and every replica must live there. In a single cluster like `east` this is a no-op, since there's only one cluster to choose. It's useful across clusters, where a stream is pinned to one region; that cross-cluster case is covered in [Super-clusters](/learn/topologies/super-clusters.md), not here. The second lever is the one that matters inside `east`, namely **tags**. ## Tags label servers; placement matches them A **tag** is a label you attach to a server in its configuration. The server advertises its tags to the rest of the cluster, and placement uses them to pick servers. A tag is freeform text: a region, a disk class, a hardware tier, whatever distinction you want placement to respect. You set tags with `server_tags` in each server's config. Give the three production servers a region tag and a disk-class tag: ``` # n1-east.conf — tag this server for placement server_name: n1-east listen: "0.0.0.0:4222" server_tags: ["region:us-east", "disk:ssd"] cluster { name: east listen: "0.0.0.0:6222" routes: [ "nats://127.0.0.1:6223" "nats://127.0.0.1:6224" ] } jetstream { store_dir: "/data/n1-east" } ``` Repeat the same `server_tags` line on `n2-east` and `n3-east`, keeping their own `server_name`, ports, and `store_dir`. After a restart, confirm a server actually carries the tags you expect. `nats server info` queries the system account (`$SYS`), so connect with a system user (see [Forming a cluster](/learn/clustering/forming-a-cluster.md)); name the server you mean, or the command answers for whichever server replies to the `$SYS` ping first: ``` nats server info n1-east ``` The `Tags` line in the output lists the server's tags. Read them back rather than assuming the config took. A typo in `server_tags` is silent until a placement asks for a tag no server advertises. ### Placing the stream on tagged servers With the servers tagged, constrain `ORDERS` to land only on servers carrying both `region:us-east` and `disk:ssd`. The CLI flag is `--tag`, passed once per required tag; the client libraries set `Placement.Tags` to a list. The example also names the cluster with `--cluster east`, a no-op in a single cluster, shown so the syntax is familiar when you place across clusters later: #### CLI ``` #!/bin/bash # Place the ORDERS stream on servers carrying specific tags. # # This assumes the 3-node "east" cluster is running with JetStream # enabled, and that n1-east, n2-east, n3-east each advertise the tags # region:us-east and disk:ssd via server_tags in their config. See the # server config block on the placement page for the tag setup. # Create ORDERS at R=3, constrained to servers carrying BOTH tags. # --tag is passed once per required tag; the match is an intersection, # so every listed tag must be present on a server for it to qualify. # Tag matching folds case (ssd == SSD) but spelling is exact. # # --cluster names the cluster every replica must live in. In a single # cluster like "east" it is a no-op (there is only one cluster), but it # is shown here so the syntax is familiar when you place across clusters # later. The match is an intersection of cluster AND tags. nats --server nats://127.0.0.1:4222 stream add ORDERS \ --subjects "orders.>" \ --replicas 3 \ --cluster east \ --tag region:us-east \ --tag disk:ssd \ --defaults # If ORDERS already exists, change its placement instead of recreating # it. The same flags apply on edit; the meta leader re-assigns the # replicas to servers matching the new constraint. # # nats --server nats://127.0.0.1:4222 stream edit ORDERS \ # --cluster east --tag region:us-east --tag disk:ssd # Read the result. The Cluster block names the leader and the two other # peers — every one of them is a server you tagged. nats --server nats://127.0.0.1:4222 stream info ORDERS # Publish the canonical order to confirm the placed stream accepts # writes exactly as before — placement changes where, not what. nats --server nats://127.0.0.1:4222 pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` #### C ``` // Create ORDERS at R=3, constrained to servers carrying BOTH tags. // Placement.Tags is a list and the match is an intersection: every // listed tag must be present on a server for it to qualify. Tag // matching folds case (ssd == SSD) but spelling is exact. // // Placement.Cluster names the cluster every replica must live in. // In a single cluster like "east" it is a no-op (there is only one // cluster), but it is shown here so the syntax is familiar when you // place across clusters later. The match is an intersection of // cluster AND tags. jsStreamConfig cfg; jsPlacement placement; const char *subjects[] = {"orders.>"}; const char *tags[] = {"region:us-east", "disk:ssd"}; jsPlacement_Init(&placement); placement.Cluster = "east"; placement.Tags = tags; placement.TagsLen = 2; jsStreamConfig_Init(&cfg); cfg.Name = "ORDERS"; cfg.Subjects = subjects; cfg.SubjectsLen = 1; cfg.Replicas = 3; cfg.Placement = &placement; s = js_AddStream(&si, js, &cfg, NULL, &jerr); if (jerr == JSStreamNameExistErr) { // ORDERS already exists: change its placement instead of // recreating it. The same config applies on update; the meta // leader re-assigns the replicas to servers matching the new // constraint. s = js_UpdateStream(&si, js, &cfg, NULL, &jerr); } // Read the result. The Cluster block names the leader and the two // other peers — every one of them is a server you tagged. if ((s == NATS_OK) && (si->Cluster != NULL)) { printf("Cluster: %s, leader: %s\n", si->Cluster->Name, si->Cluster->Leader); for (i = 0; i < si->Cluster->ReplicasLen; i++) printf("Replica: %s\n", si->Cluster->Replicas[i]->Name); } // Publish the canonical order to confirm the placed stream accepts // writes exactly as before — placement changes where, not what. if (s == NATS_OK) s = natsConnection_PublishString(conn, "orders.created", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"); ``` The meta leader now picks three servers that carry *both* tags. Read the result in the `Cluster` block of `nats stream info ORDERS`: the leader and the two other peers are all servers you tagged. ## Tag matching is an intersection When placement lists more than one tag, a server qualifies only if it carries *every* tag in the list. The match is an intersection, not a union: `region:us-east` **and** `disk:ssd`, never either-or. The match folds case: `disk:ssd`, `disk:SSD`, and `disk:Ssd` are the same tag. Spelling, though, is exact, so `disk:sdd` matches nothing. The problem to watch for is typos rather than case. Ask for a tag that no server carries (a misspelling, or a tag you meant to add but didn't) and the intersection is empty. No server qualifies, and the meta leader refuses the stream. The error names the tag no server carried, in brackets: ``` nats: error: could not create Stream: no suitable peers for placement, tags not matched ['disk:sdd'] (10005) ``` The same error appears if you ask for three replicas but only two servers carry the required tags. Placement doesn't relax the constraint to fit the replica count; it fails so you notice. The full set of placement and server-tag options is documented in [Reference](/reference/config/server_tags.md). We only need the cluster constraint and the tag intersection here. ## Asking a chosen server to lead Placement decides which servers hold the replicas. It does not decide which of them leads. When you create a placed stream, the meta leader picks the initial leader itself, and there's no placement field that names one: set a `preferred` server in a stream's placement and the server rejects the config with `preferred server not permitted in placement`. No client library exposes such a field either — `Placement` carries only a cluster and tags. What you can do is ask the current leader to hand off to a named peer once the group is running. That's a **stepdown** request with a preferred target: ``` nats stream cluster step-down ORDERS --preferred n2-east ``` The leader yields and the group runs an election that tries to place the new leader on `n2-east`. It's a request, not a lock: the quorum election still decides, so read `nats stream info ORDERS` afterward to see who won. `--preferred` needs NATS Server 2.11 or newer. Its full syntax lives in [Reference](/reference/jetstream/api/stream/.md). ## Pitfalls Two mistakes are common the first time you place a stream. Both come from treating placement as more forgiving than it is. **Tags are an intersection; a missing tag fails the placement.** Asking for a tag no server carries leaves the meta leader with nothing to pick: the stream fails with `no suitable peers for placement` rather than falling back to any server. Matching folds case, so `ssd` and `SSD` are the same tag, but spelling is exact — `sdd` matches nothing. Don't guess at tag spelling. Read the tags back from the servers first, then place against exactly what they advertise. Verify the tags exist before you trust a placement, and watch the placement either succeed or name the missing tag: #### CLI ``` #!/bin/bash # Verify server tags BEFORE placing a stream, then place against exactly # what the servers advertise — so a typo fails loudly instead of silently # placing nowhere. # # This assumes the 3-node "east" cluster is running with JetStream # enabled. Tag matching folds case (ssd == SSD) but is matched as an exact # intersection: every requested tag must be present, spelled correctly, on a # server for it to qualify. # Read the tags each server actually advertises. Do not assume the config # took — a typo in server_tags is silent until a placement asks for a tag # no server carries. The Tags line in the output is the source of truth. # # `nats server info` queries the system account ($SYS), so connect with a # system user. Name the server you want, or the command answers for whichever # server replies to the $SYS ping first. nats server info n1-east nats server info n2-east nats server info n3-east # Now place ORDERS against exactly the tags you just read back. nats --server nats://127.0.0.1:4222 stream add ORDERS \ --subjects "orders.>" \ --replicas 3 \ --tag region:us-east \ --tag disk:ssd \ --defaults # If a requested tag is misspelled or missing on every server, the # intersection is empty and the create fails — it does NOT fall back to # any server. The error names the tag no server carried, in brackets: # # nats: error: could not create Stream: no suitable peers for placement, tags not matched ['disk:sdd'] (10005) # # Fix the spelling to match what the servers advertise (case does not # matter) and re-run. Confirm the placement landed where you intended: nats --server nats://127.0.0.1:4222 stream info ORDERS ``` #### C ``` // Read the tags each server actually advertises. Do not assume the // config took — a typo in server_tags is silent until a placement // asks for a tag no server carries. There is no dedicated client // API for this: it is a plain request on the system account's // $SYS.REQ.SERVER.PING.VARZ subject, filtered by server name. The // "tags" field of the reply is the source of truth. const char *names[] = {"n1-east", "n2-east", "n3-east"}; for (i = 0; (s == NATS_OK) && (i < 3); i++) { natsMsg *reply = NULL; char req[64]; snprintf(req, sizeof(req), "{\"server_name\":\"%s\"}", names[i]); s = natsConnection_RequestString(&reply, sysConn, "$SYS.REQ.SERVER.PING.VARZ", req, 2000); if (s == NATS_OK) { // Print just the "tags" list out of the VARZ reply. const char *tags = strstr(natsMsg_GetData(reply), "\"tags\":"); const char *end = (tags != NULL) ? strchr(tags, ']') : NULL; if (end != NULL) printf("%s %.*s]\n", names[i], (int)(end - tags), tags); natsMsg_Destroy(reply); } } // Now place ORDERS against exactly the tags you just read back. // If a requested tag is misspelled or missing on every server, the // intersection is empty and the create fails — it does NOT fall // back to any server. The error names the tag no server carried: // // no suitable peers for placement, tags not matched ['disk:sdd'] // (error code 10005) if (s == NATS_OK) { jsStreamConfig cfg; jsPlacement placement; const char *subjects[] = {"orders.>"}; const char *tags[] = {"region:us-east", "disk:ssd"}; jsPlacement_Init(&placement); placement.Tags = tags; placement.TagsLen = 2; jsStreamConfig_Init(&cfg); cfg.Name = "ORDERS"; cfg.Subjects = subjects; cfg.SubjectsLen = 1; cfg.Replicas = 3; cfg.Placement = &placement; s = js_AddStream(&si, js, &cfg, NULL, &jerr); if (jerr == JSStreamNameExistErr) s = js_UpdateStream(&si, js, &cfg, NULL, &jerr); } // Confirm the placement landed where you intended: the leader and // both replicas are servers that advertised the tags. if ((s == NATS_OK) && (si->Cluster != NULL)) { printf("Leader: %s\n", si->Cluster->Leader); for (i = 0; i < si->Cluster->ReplicasLen; i++) printf("Replica: %s\n", si->Cluster->Replicas[i]->Name); } ``` **You can't name a leader at placement time.** Placement constrains which servers hold the replicas; it can't say which one leads, and setting a `preferred` server in placement is rejected outright. Don't build an operational assumption ("the leader is always `n1-east`") on where the group first landed; the moment a leader dies, the next election is quorum-based and picks a leader from the survivors. If you need leadership on a specific server, ask for it explicitly with `nats stream cluster step-down --preferred ` (see [Raft and leaders](/learn/clustering/raft-and-leaders.md)), then re-check with `nats stream info` — it's a request the election can still overrule. ## Where you are The `ORDERS` stream is no longer placed on whichever servers the meta leader chose freely. You tagged `n1-east`, `n2-east`, and `n3-east`, and constrained the stream to servers carrying both `region:us-east` and `disk:ssd`. You know the tag match is an intersection that folds case, that a missing or misspelled tag yields `no suitable peers for placement`, and that placement can't name a leader — you move leadership after the fact with a stepdown request. The cluster is still three servers. Nothing on this page changed the peer count. ## What's next Changing the peer count is the next page. **Scaling and peer management** grows the group by raising the replica count, watches a new peer catch up before you lean on it, and moves a replica off a server without ever losing the majority that keeps `ORDERS` writable. Continue to [Scaling and peer management](/learn/clustering/scaling-and-peers.md). ## See also * [Reference → Server tags](/reference/config/server_tags.md) — every server-tag and placement option and its syntax. * [Reference → Meta API](/reference/jetstream/api/meta/.md) — how the meta leader assigns a placed stream to servers. * [Super-clusters](/learn/topologies/super-clusters.md) — placing replicas across clusters for geo-affinity. --- # Raft and leaders The [previous page](/learn/clustering/forming-a-cluster.md) left you with a live three-server cluster `east` (`n1-east`, `n2-east`, and `n3-east`) that discovered itself from one seed route. The servers know about each other and forward messages. But knowing about each other isn't the same as *agreeing* with each other. Plain `orders.created` traffic needs no agreement: a publish lands on one server, gets forwarded, and is gone. Stored data is different. When the `ORDERS` stream keeps three copies of every order, the three servers holding those copies must agree on which orders exist and in what order, even while one of them is restarting or unreachable. This page is about that agreement. It introduces two ideas: a **RAFT group** is a set of servers that keep an identical log by consensus, and a **leader election** is how a group picks the one server that drives the log. Everything the next page does with replication rests on these two. ## RAFT groups NATS reaches agreement with **RAFT**, a consensus algorithm. A **RAFT group** is a fixed set of servers (its **peers**) that maintain a single shared log that all of them agree on. One peer is the **leader**; the rest are **followers**. The leader is the only peer that accepts new entries; it copies each entry to the followers, and the group advances together. A cluster runs many RAFT groups at once, layered. The first is the **meta group**: one cluster-wide RAFT group whose peers are the servers themselves. Its log holds the cluster's *assignments* rather than your data: which streams exist, how many replicas each has, and which servers hold them. Its leader is the **meta leader**, the server that decides where new streams and consumers are placed. Every server in `east` is a peer of the meta group. The rest are **per-asset groups**: every replicated stream gets its own RAFT group, and so does every replicated consumer. The `ORDERS` stream running with three copies is one RAFT group whose three peers are the three servers holding those copies. Its leader, the **stream leader**, is the server that accepts writes to `ORDERS` and replicates them. So `east` holding one replicated `ORDERS` stream runs at least two RAFT groups at once: the meta group across all three servers, and the `ORDERS` stream group across its three peers. They're independent. The meta leader and the `ORDERS` stream leader may be the same server or different servers, and a change to one doesn't move the other. You can see both. The JetStream report names the meta group's leader: ``` nats server report jetstream ``` Its `RAFT Meta Group Information` table lists every server and marks the one that holds meta leadership. To see a stream's group, ask the stream (you stand `ORDERS` up in the exercise below): ``` nats stream info ORDERS ``` The `Cluster Information` block names the stream's RAFT group, listing its `Leader` and each `Replica` peer: ``` Cluster Information: Name: east Leader: n1-east Replica: n2-east, current, seen 0.05s ago Replica: n3-east, current, seen 0.07s ago ``` `Leader` is the stream leader. Each `Replica` line shows a follower peer, whether it's `current` (caught up), and how recently the leader heard from it. This is the `ORDERS` RAFT group, viewed from the outside. For a live view of a group's RAFT state (current term, who the leader is, each peer's status), check the `/raftz` monitoring endpoint. The full set of RAFT internals it exposes is documented in [Reference → /raftz](/reference/system/monitor/raftz.md): log compaction, the `$NRG.*` subjects peers vote over, snapshot timing. We only need the group, the leader, and the followers here. ## Leader election A group has one leader at a time. When the leader is healthy, the steady-state behavior is straightforward: the leader sends a periodic **heartbeat** to its followers (by default about once a second), and as long as that heartbeat arrives, the followers stay followers and do nothing but accept the leader's entries. The interesting case is when the heartbeat stops, because the leader crashed, hung, or got cut off by the network. The followers can't tell *why* the heartbeat stopped, only that it did. So they don't wait forever. Each follower runs an **election timer**, and if no heartbeat arrives before it fires, that follower assumes the leader is gone and starts an **election** to replace it. An election needs one more idea: the **term**, a number that counts elections and only ever goes up. Every entry and every vote is stamped with a term, so the group can always tell a stale message from a current one. A leader from an older term is automatically obsolete the moment a newer term exists. Here's the sequence when a follower's election timer fires. The follower becomes a **candidate**, the third RAFT role. It increments the term to one higher than any it's seen, votes for itself, and asks every other peer to vote for it in this new term. Each peer grants its vote if it hasn't already voted in this term and the candidate's log is at least as up to date as its own. A peer votes for at most one candidate per term, which is what stops two leaders from emerging at once. The candidate becomes the leader the instant it collects a **quorum** of votes: a majority of the group's peers, `(N+1)/2`. For the three-peer `ORDERS` group that's two: the candidate's own vote plus one more. With the majority in hand, the new leader immediately starts sending heartbeats, the other peers return to being followers, and the group is whole again under the new term. **Message flow — A follower wins a leader election (animated):** A three-peer RAFT group whose leader sends a heartbeat each term. The leader stops — a crash, a hang, or a network cut — so its heartbeat no longer arrives. A follower's election timer fires, so it becomes a candidate, increments the term to T5, votes for itself, and asks the other peers for their votes. Each peer grants at most one vote per term, so once the candidate collects a quorum — two of three — it becomes leader and starts sending heartbeats in the new term, and the others return to being followers. * n3 → n2 (subject: Vote T5) The quorum rule is why a majority must survive for a group to elect a leader at all. A three-peer group keeps a leader as long as two peers are up; lose two and the survivor can't reach a majority, so it can't become leader and the group goes leaderless until a peer returns. This is the consensus math behind the odd-server-count advice the [Topologies chapter](/learn/topologies/your-first-cluster.md) gives as a deployment choice: an even count buys no extra majority. ## Observing an election You can observe this directly. This exercise needs the replicated `ORDERS` stream; the [next page](/learn/clustering/replication-and-r3.md) creates it in full, but one command stands it up now: ``` nats stream add ORDERS --subjects 'orders.>' --replicas=3 --defaults ``` With `east` running and `ORDERS` replicated, find the current stream leader, kill it, and watch the survivors elect a new one. First, find the leader: ``` nats stream info ORDERS | grep Leader ``` Say it reports `Leader: n1-east`. Stop that server: Ctrl-C its terminal, or kill its process. For a moment the `ORDERS` group has no leader: its heartbeat has stopped and the two survivors are running their election timers. Within a few seconds, ask one of the survivors: ``` nats stream info ORDERS --server nats://127.0.0.1:4223 ``` The `Cluster Information` block now names a different leader, `n2-east` or `n3-east`, and the term has advanced. The remaining two peers held a quorum (two of three), so they elected a new leader and `ORDERS` is writable again, even with `n1-east` down. ## Moving a leader manually Sometimes you want to move leadership without killing anything: to drain a server before maintenance, or to rebalance after a restart. A **stepdown** is a leader voluntarily yielding its role so the group elects a new one. For a stream leader, ask the stream's group to step down: ``` nats stream cluster step-down ORDERS ``` The current leader yields, the group runs an election, and a different peer takes over. The meta group has its own stepdown, scoped to the whole cluster: ``` nats server cluster step-down ``` That moves the *meta* leader, independently of any stream leader. Use the stream form to move a single stream, the server form to move cluster-wide assignment duty. ## Pitfalls RAFT is robust, but its timing and its layering are common sources of confusion. Each pitfall below is scoped to this page's two concepts: groups and elections. **An election takes seconds, not milliseconds.** The election timer fires between four and nine seconds after the last heartbeat, deliberately staggered so two followers don't become candidates at the exact same instant. So when you kill a leader, expect a short window where `nats stream info` shows no leader and writes are refused. That window is RAFT working as designed, not a bug. Don't build a client that treats a brief "no leader" as a fatal error. Have it retry, since a new leader arrives within seconds. The correct handling is to retry the write rather than fail it. A `nats stream info` during the gap confirms what's happening: ``` # During the election window, the leader line is briefly empty: nats stream info ORDERS | grep Leader # Leader: # Re-run a few seconds later and a new leader appears: nats stream info ORDERS | grep Leader # Leader: n2-east ``` **Stepdown moves leadership, but the election still picks the successor.** `nats stream cluster step-down` makes the current leader yield, and the *next* leader is chosen by a quorum election among the remaining peers. You can name a preferred successor with `nats stream cluster step-down --preferred ` (NATS Server 2.11+), but that's a request, not a guarantee — the election can still land elsewhere. So run stepdown to *move leadership off* the current server, then read `nats stream info` to learn who actually won. Stream placement (cluster and tags) can't name a leader at all; that's covered on [Placement](/learn/clustering/placement.md). **The meta leader and a stream leader are different groups.** Losing the meta leader doesn't lose the `ORDERS` stream leader, and vice versa; they're separate RAFT groups with separate elections. A common mistake is to see "the leader is down," panic, and assume the stream is unavailable when only the meta leader moved (or the reverse). Check the right group: `nats server report jetstream` for the meta leader, `nats stream info ORDERS` for the stream leader. ## Where you are Your cluster now has names for its moving parts: * The meta group spans all three servers and holds the cluster's assignments; its meta leader decides placement. * The `ORDERS` stream is its own RAFT group of three peers with its own stream leader, independent of the meta leader. * You've killed a leader and watched the survivors run a leader election (a candidate, a bumped term, and a quorum of votes), and you've moved a leader on purpose with stepdown. What you haven't done yet is follow a single write through the group: how the leader gets an order onto all three peers and decides it's safe. ## What's next The next page traces exactly that. It follows one `orders.created` write from the leader's log to a quorum of peers, shows where the write **commits**, and explains the consistency you get from `R=3`: [Replication and R=3](/learn/clustering/replication-and-r3.md). ## See also * [Reference → /raftz](/reference/system/monitor/raftz.md) — the RAFT group monitoring endpoint and its full field set. * [Surviving node loss](/learn/jetstream/surviving-node-loss.md) — the one-page operator view of replicas riding through a server loss. * [Topologies → Your first cluster](/learn/topologies/your-first-cluster.md) — where the odd-server-count and shape choices live. --- # Replication and R=3 The last page elected leaders, so the cluster now has a meta leader and, once you create a stream, a leader for that stream's RAFT group. This page uses those leaders: it follows a single order from `order-svc` into the `ORDERS` stream and shows exactly when that write becomes safe to lose a server over. The [surviving node loss](/learn/jetstream/surviving-node-loss.md) page in the JetStream chapter gave you the one-line version: `R=3` keeps three copies, so the loss of one server costs nothing. This page explains the mechanism behind that guarantee. It introduces two ideas: **quorum commit**, how the leader turns one write into a committed entry across the group, and the **consistency** you get back from it. It also covers a narrower gap worth knowing before you rely on either: what quorum commit does, and doesn't, guarantee once the write reaches disk. ## R=3 means three peers in one RAFT group A stream's replica count is the number of copies the cluster keeps. `R=3` keeps three. Each copy lives on a different server, and the three servers holding the copies form a single RAFT group, the same kind of consensus group the last page elected a leader for. You set the count when you create the stream. On the `east` cluster, one flag turns the single-server `ORDERS` of the JetStream chapter into a three-peer stream. (If you ran the election exercise on [Raft and leaders](/learn/clustering/raft-and-leaders.md), `ORDERS` already runs at `R=3`; restart any server you stopped there, and read this as the create that stood it up.) #### CLI ``` #!/bin/bash # Create the ORDERS stream as an R=3 stream on the `east` cluster, then publish # one order and read back which server leads the stream and which servers hold # the copies. # # This assumes the three `east` servers from the Topologies chapter are running, # each with JetStream enabled: # nats-server -c n1-east.conf # nats-server -c n2-east.conf # nats-server -c n3-east.conf # # Point the CLI at any server in the cluster. The cluster routes the request to # the stream leader wherever it currently lives. export NATS_URL="nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224" # Create ORDERS with three replicas. --replicas=3 is the only line that differs # from the single-server create in the JetStream chapter. If ORDERS already # exists at R=1, raise it instead: nats stream edit ORDERS --replicas=3 nats stream add ORDERS \ --subjects "orders.>" \ --replicas=3 \ --defaults # Publish one order as order-svc would. --jetstream makes this a JetStream # publish that waits for a PubAck; the PubAck returns only after the leader # has the write committed to a quorum (itself plus one follower). Plain # `nats pub` is a core publish and would not wait for one. nats pub --jetstream orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # Confirm the replica count and the cluster layout that R=3 produced. nats stream info ORDERS # Expected Cluster Information section: # # Cluster Information: # # Name: east # Cluster Group: S-R3F-xK2p9aLm # Leader: n1-east # Replica: n2-east, current, seen 0.00s ago # Replica: n3-east, current, seen 0.00s ago ``` #### C ``` // Create ORDERS with three replicas. Replicas=3 is the only line // that differs from the single-server create in the JetStream // chapter. If ORDERS already exists at R=1, raise it in place with // js_UpdateStream instead. jsStreamConfig cfg; const char *subjects[] = {"orders.>"}; jsPubAck *pa = NULL; jsStreamConfig_Init(&cfg); cfg.Name = "ORDERS"; cfg.Subjects = subjects; cfg.SubjectsLen = 1; cfg.Replicas = 3; s = js_AddStream(NULL, js, &cfg, NULL, &jerr); if (jerr == JSStreamNameExistErr) s = js_UpdateStream(NULL, js, &cfg, NULL, &jerr); // Publish one order as order-svc would. A JetStream publish waits // for a PubAck; the PubAck returns only after the leader has the // write committed to a quorum (itself plus one follower). A core // publish (natsConnection_Publish) would not wait for one. const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; if (s == NATS_OK) s = js_Publish(&pa, js, "orders.created", order, (int) strlen(order), NULL, &jerr); if (s == NATS_OK) { printf("Stored in %s, sequence %" PRIu64 "\n", pa->Stream, pa->Sequence); jsPubAck_Destroy(pa); } // Confirm the replica count and the cluster layout that R=3 // produced: one leader and two follower replicas, each on its own // server of the east cluster. if (s == NATS_OK) s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if ((s == NATS_OK) && (si->Cluster != NULL)) { printf("Cluster: %s, leader: %s\n", si->Cluster->Name, si->Cluster->Leader); for (i = 0; i < si->Cluster->ReplicasLen; i++) { jsPeerInfo *peer = si->Cluster->Replicas[i]; printf("Replica: %s, %s, seen %.2fs ago\n", peer->Name, peer->Current ? "current" : "outdated", (double) peer->Active / 1E9); } } ``` `--replicas=3` is the whole change. Starting from the JetStream chapter's `R=1` stream instead, raise it in place with `nats stream edit ORDERS --replicas=3`. The application code doesn't move: `order-svc` publishes the same payload to the same subject whether the stream is `R=1` or `R=3`. ``` {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} ``` What changes is underneath. The stream now has a **leader** (one of the three peers — the meta leader picks which, and an election may have moved it) and two **followers**. Every write goes through the leader. The followers never take writes directly; they receive them from the leader. Three is the minimum for production, and a stream supports at most `R=5`. The reasoning for *which* odd count to choose belongs to [surviving node loss](/learn/jetstream/surviving-node-loss.md). Here we follow what one write does once the count is three. ## A write commits by quorum When `order-svc` publishes `orders.created`, the message reaches the stream leader, `n1-east`. The leader does not return success on storing it locally; it runs a short sequence first. The leader **appends** the write to its own log: an ordered, append-only record of every operation the group has agreed on. Appending is local and not yet durable across the group: only `n1-east` has the entry so far. The leader then sends an **append entry** to each follower: the replication message that says "add this entry to your log at this position." `n2-east` and `n3-east` receive it, write it to their own logs, and reply with an ack. The leader counts acks. A write is **committed** once a **quorum**, a majority of the peers, holds the entry. For `R=3`, a quorum is two of three. The leader is itself one of the two, so it needs just one follower's ack to reach quorum. The instant the first follower acks, the entry is committed. Commit is the point at which durability is reached. A committed entry survives the loss of any single server, because it already lives on a majority. This is what makes the `PubAck` that `order-svc` receives a real guarantee: the leader returns it only after the write commits, so a `PubAck` means the order already survived the chance of a single-node failure before you heard back. The third peer isn't on the critical path. `n3-east` may ack a moment later, or be briefly behind; the write committed without waiting for it. That's the point of a quorum: the group makes progress as long as a majority is reachable, even if not all peers are. ## Followers apply what the leader commits Committing records that a quorum *has* the entry. It doesn't yet put the order into each peer's copy of the stream. That last step is **apply**: copying a committed entry from the log into the stream store, where consumers can read it. The leader tracks a **commit index**, the position up to which entries are committed. It is included on the next append entry or heartbeat, so followers learn "everything up to here is committed; apply it." Each follower then applies those entries to its own stream store in the same order the leader did. Order is what the group guarantees here. Every peer applies the same entries in the same sequence, so all three copies of `ORDERS` converge on the identical message log. A follower can lag the leader by a few entries, but it never reorders them and never skips one. Here's one write from `order-svc` moving through that whole sequence (publish, append entry, ack, commit at quorum, apply): **Message flow — A write commits by quorum (animated):** One write through an R=3 stream group. order-svc publishes orders.created to the leader n1-east, which appends the entry to its own log (write WAL) and sends an AppendEntry to the two followers n2-east and n3-east. Each follower writes the entry and replies with an ack. The write commits the instant a quorum — the leader plus one follower — holds it, so the PubAck can return; the third peer applies the entry a moment later. Every peer applies committed entries in the same order, so all three copies converge on the identical log. * order-svc → n1-east (leader) * n1-east (leader) → n1-east (leader) (subject: write WAL) * n1-east (leader) → n2-east * n1-east (leader) → n3-east (subject: AppendEntry) You'll find the full set of RAFT replication parameters (append-entry batching, heartbeat intervals, log compaction) in [Reference](/reference/system/monitor/raftz.md). All you need here is the append → quorum → commit → apply shape. ## The consistency you get Quorum commit gives a specific, nameable consistency, and you should know its boundaries before you build on it. Reads from the leader are read-after-write. The leader holds every committed entry and assigns every sequence number, so once a `PubAck` returns, a read from the leader sees that order. There's no window where your own just-acked write is missing. Reads from a follower can lag. A follower applies committed entries slightly after the leader does, so a direct read from `n2-east` or `n3-east` might not yet show the most recent order, even though that order is already committed and safe. The data is correct but slightly behind. For read-after-write, read from the leader. This is the trade `R=3` makes on purpose. Rather than promising that every copy is identical at every instant, it promises that every copy *converges*, in order, and that a committed write survives one server loss. When you need to confirm where the copies actually stand, the `Cluster` block of `nats stream info` reports each replica's status and how far behind it is, which is exactly the first trap under Pitfalls, below. ## Syncing data to disk Quorum commit protects against losing a server, not against losing a server's disk. For file storage, applying a committed entry writes it to disk — but JetStream doesn't `fsync` every write immediately. It batches disk syncs on a timer, `sync_interval`, which defaults to 2 minutes. Until the next sync, an applied write can sit in the OS's write cache without reaching physical disk. That gap means a `PubAck` only confirms quorum; the disk sync still happens later, on the next timer tick. Two peers can hold a write in memory and still lose it if the underlying hardware fails before either one's next sync: * Multiple peers suffer an OS failure within the same `sync_interval` window, before any of them synced the write. * A peer that lost unsynced data in an OS crash rejoins the group and, together with a peer that never held the write, reaches a new quorum — one that never saw it. A shorter `sync_interval` narrows this window, at the cost of write throughput. Setting it to `always` closes the window entirely: every write syncs to disk before the leader returns its `PubAck`. Combined with peers spread across availability zones, `always` gives you the strongest durability guarantee available, at the cost of the slowest writes. The 2-minute default balances that risk against performance for a typical production deployment spread across multiple availability zones. Here's how narrow the failure window actually is. Take `ORDERS` at `R=3`, its three peers spread across three zones. For its log to diverge, all of this has to happen, in order: * One of the three peers is already offline, isolated, or partitioned, and never gets the write. * A second peer's OS crashes and loses the write: it was one of only two peers holding it, and neither had synced yet. * The leader — the third peer, the one that did have the write — goes down or gets isolated too. * The offline peer reconnects, but the leader isn't reachable anymore. * The crashed peer comes back and reaches the reconnected peer, but not the leader. Now two peers are up, neither holding the write, and together they form a new quorum. They start accepting new writes, silently dropping some of what the old leader had acked. This is a real way for a stream's log to diverge, but it takes several independent faults landing in exactly this order, across separate availability zones. One mitigation: keep the crashed server offline instead of restarting it right away. A NATS server rejoins its cluster automatically on restart, so the safeguard is in when you restart it — wait until `nats stream info` shows the remaining peers are fully caught up before bringing it back. Or remove the crashed peer and let it rejoin as a wiped, empty peer that resyncs over the network — safer, though expensive if there's a lot of data to resync. If minimizing loss matters more than throughput, use `sync_interval: always`. It costs performance cluster-wide, even for streams that don't need the extra durability, so weigh it against your own durability, cost, and performance requirements before you turn it on everywhere. You don't have to pick one setting for the whole deployment. Run most clusters on the default `sync_interval`, and add a separate cluster tagged for `sync_interval: always`. Then place only the streams that need the strongest durability there, using [placement tags](/learn/clustering/placement.md). ``` # Configure a cluster that's dedicated to always sync writes. server_tags: ["sync:always"] jetstream { sync_interval: always } ``` Create a replicated stream pinned to that cluster, so only the writes that need this level of durability pay for it. ``` nats stream add --replicas 3 --tag sync:always ``` ## Pitfalls These are three common mistakes the first time you trust a replicated stream. **`R=1` has no copy.** A stream at `R=1` lives on exactly one server. There's no second peer, so there's no quorum and nothing to commit *to* beyond the one log. If that server's disk is lost, the order is lost too, with no failover and no recovery. Only `R≥3` survives a node loss. Don't run real orders at `R=1`; the why-three reasoning is covered on [surviving node loss](/learn/jetstream/surviving-node-loss.md). **A follower may lag, so a follower read can be stale.** A committed write is safe, but it reaches each follower's stream store slightly after the leader applies it. A direct read aimed at a follower can therefore return data that's correct but not the newest. Don't assume any peer is current just because the write was acked. For read-after-write, read from the leader. Check the leader and each replica's lag when you need to know how far behind the group's followers are running, not as a way to decide whether a particular read is fresh: #### CLI ``` #!/bin/bash # Read the Cluster block of `nats stream info` to check that every copy of # ORDERS is current before trusting the stream. This is the handling step for # the "a replica may lag" pitfall: do not assume all three copies hold the same # data — read the lag and confirm it. # # Assumes the `east` cluster is running and ORDERS is an R=3 stream (see # createR3.sh). Point the CLI at any server in the cluster. export NATS_URL="nats://127.0.0.1:4222,nats://127.0.0.1:4223,nats://127.0.0.1:4224" # Ask the cluster who leads ORDERS and how its followers are doing. The Cluster # Information block names the LEADER (every write lands there) and lists each # Replica with its status. nats stream info ORDERS # Read the Cluster Information section carefully: # # Cluster Information: # # Name: east # Cluster Group: S-R3F-xK2p9aLm # Leader: n1-east # Replica: n2-east, current, seen 0.00s ago # Replica: n3-east, current, seen 0.00s ago # # "current" means the follower has applied every committed entry — it is up to # date with the leader. A healthy R=3 stream shows every Replica "current" with # a small "seen" age. # # A follower that is catching up shows "outdated" and a non-zero lag instead: # # Replica: n3-east, outdated, seen 0.12s ago, 4,512 operations behind # # If a replica is outdated, do not treat it as a current copy. Read from the # leader for read-after-write, and wait for "current" before trusting that copy # to survive a node loss. A persistent lag points at a slow disk or a saturated # route between peers. ``` #### C ``` // Ask the cluster who leads ORDERS and how its followers are // doing. The Cluster info names the leader (every write lands // there) and lists each replica with its status. s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if ((s == NATS_OK) && (si->Cluster != NULL)) { bool allCurrent = true; printf("Cluster: %s, leader: %s\n", si->Cluster->Name, si->Cluster->Leader); for (i = 0; i < si->Cluster->ReplicasLen; i++) { jsPeerInfo *peer = si->Cluster->Replicas[i]; // Current means the follower has applied every committed // entry — it is up to date with the leader. A follower // that is catching up shows Current false and a non-zero // Lag (in operations) instead. Active is how long ago the // leader last heard from it. printf("Replica: %s, %s, seen %.2fs ago", peer->Name, peer->Current ? "current" : "outdated", (double) peer->Active / 1E9); if (peer->Lag > 0) printf(", %" PRIu64 " operations behind", peer->Lag); printf("\n"); if (!peer->Current) allCurrent = false; } // A healthy R=3 stream shows every replica current with a // small seen age. If a replica is outdated, do not treat it as // a current copy: read from the leader for read-after-write, // and wait for current before trusting that copy to survive a // node loss. A persistent lag points at a slow disk or a // saturated route between peers. if (allCurrent) printf("All replicas current\n"); else printf("A replica is catching up — wait before leaning on it\n"); } ``` **A `PubAck` proves quorum, not full replication.** The leader returns the `PubAck` the instant a quorum holds the entry: for `R=3`, the leader plus one follower. The third peer may still be catching up at that moment. That's correct and safe: the write already survives one node loss. But don't read a `PubAck` as "all three copies are identical right now." If you need every copy current (say, before deliberately taking a server down), verify each replica shows `current` in `nats stream info` first. ## Where you are The `ORDERS` stream now runs `R=3` on the `east` cluster: a leader on one of `n1-east`, `n2-east`, or `n3-east` and followers on the other two, all carrying the same order log. What changed is your model of a write: * A write appends to the leader's log, replicates as an append entry to followers, and commits once a quorum holds it: two of three for `R=3`. * Followers apply committed entries in order, so all three copies converge on the identical log. * A `PubAck` means the order survived the loss of one server before you heard back. * Read-after-write comes from the leader; follower reads may lag. ## What's next The stream is replicated, but the cluster chose *where* its three copies landed. The next page makes that choice yours: placement constrains a stream's replicas to a cluster and to servers carrying matching tags, and lets you ask a chosen server to take leadership. Continue to [Placement](/learn/clustering/placement.md). ## See also * [Surviving node loss](/learn/jetstream/surviving-node-loss.md) — the one-page operator intro to `R=3` and storage durability. * [Reference → Stream Configuration](/reference/jetstream/api/stream/.md) — the full `StreamConfig`, including every replica option. * [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) — copying a stream's data on purpose, across clusters and for DR. --- # Scaling and peer management The `ORDERS` stream runs at `R=3` on `n1-east`, `n2-east`, and `n3-east`, placed where the [Placement](/learn/clustering/placement.md) page pinned it. That peer set isn't frozen. You can grow it — raise the replica count so a copy lands on another server — or move a replica off a server to retire it, without taking the stream down or losing the agreement the rest of this chapter built. This page changes the membership of a RAFT group while it keeps serving. A **peer** here is a RAFT-group member, as it has been since [Raft and leaders](/learn/clustering/raft-and-leaders.md): one server's role inside the `ORDERS` group. Growing or shrinking that set is **peer management**, and it comes in two halves: **growing the group** so a new peer catches up, and **moving a replica off** a server. **Message flow — Add a peer, then remove one (animated):** Growing and shrinking the ORDERS group. A fourth server, n4-east, is added and streams the entries it is missing (catchup) until its lag reaches zero and stream info shows it current; while it holds no history it cannot win an election, so you do not rely on it as a data-bearing replica until it is caught up. Later a peer is removed: the meta leader re-places its replica onto another server and the dropped peer lets go of its RAFT subscriptions while the rest keep serving. * n1-east (leader) → n2-east * n2-east → n3-east * n1-east (leader) → n3-east * n1-east (leader) → RAFT (subject: AddPeer) * n1-east (leader) → RAFT (subject: catchup) * n1-east (leader) → RAFT (subject: lag->0) * n2-east → RAFT * n3-east → RAFT * n1-east (leader) → RAFT (subject: RemovePeer) * n1-east (leader) → RAFT (subject: dropped) The animation shows both halves: a fourth server joins the group and streams the entries it's missing until it's caught up; later a replica is moved off a server, which drops its RAFT subscriptions while the rest carry on. ## Growing the group: a new peer catches up You grow a stream's group by raising its replica count. The fourth server has to be a running member of `east` already, so start `n4-east` with the same cluster config the [first page](/learn/clustering/forming-a-cluster.md) used, and let it join the mesh. Then raise `ORDERS` from three replicas to four: ``` nats --server nats://127.0.0.1:4222 stream edit ORDERS --replicas=4 ``` You don't name the new server. The meta leader assigns the extra replica to a server that qualifies under the stream's placement (the tags from the [Placement](/learn/clustering/placement.md) page), records the new peer set in its assignment log, and the `ORDERS` group picks it up. `n4-east` is now a member. It isn't a useful one yet, because a brand-new peer holds none of the stream's history. So it **catches up** first. Catchup is how a new or behind peer streams the entries it's missing: the leader feeds it the log from where it's short, the peer applies each entry into its stream store, and its lag shrinks toward zero. Lag here is just a count: how many entries behind the leader's log the peer still is. Adding the peer changes the quorum right away: an `R=4` group commits once three peers hold a write, not two. What the new peer can't do while its log is empty is win an election — it stays an observer until the leader's first entries reach it, so it never campaigns on state it doesn't have. The practical rule is simpler: don't lean on `n4-east` as a data-bearing replica until `stream info` shows it `current`, because until then only the peers that already hold the data can serve it. You watch the catchup in the same place you read everything else about the group, the `Cluster` block of `nats stream info`: ``` nats --server nats://127.0.0.1:4222 stream info ORDERS ``` The `Replicas` list now shows a fourth entry, and its lag counts down as catchup proceeds: ``` Cluster Information: Name: east Leader: n1-east Replica: n2-east, current, seen 0.12s ago Replica: n3-east, current, seen 0.20s ago Replica: n4-east, outdated, seen 0.18s ago, 14,231 operations behind ``` `outdated` and the operations-behind count are catchup in progress. When `n4-east` reads `current` with no lag, it holds the full stream and pulls its weight as a replica like any other peer. ## Moving a replica off a server To retire a server, or move a stream off one, you **remove a peer**. This doesn't shrink the stream: it evicts the replica from the named server, and the meta leader re-places it on another server that qualifies, so `ORDERS` stays at its replica count. The command names the stream and the peer to drop: ``` nats --server nats://127.0.0.1:4222 stream cluster peer-remove ORDERS n4-east ``` The meta leader picks a replacement peer, updates the stream's assignment, and the evicted server lets go of its RAFT subscriptions for the group. The replacement then catches up the same way a grown peer does. If the evicted peer held leadership, the group elects a new leader first, so leadership lands on a peer that stays. If no other server qualifies — placement leaves nowhere to put the replica — an R>1 stream still loses the peer: the server evicts it and returns `peer remap failed`, leaving the group a replica short. Only a single-replica stream is spared, since removing its last peer would brick it (the pitfall below covers that case). To change the replica *count* — shrink `R=3` to `R=1`, say — edit the stream instead: `nats stream edit ORDERS --replicas=1`. `peer-remove` moves a replica between servers; `--replicas` sets how many replicas there are. Removing a server from the JetStream **meta** group is a different command, `nats server cluster peer-remove`, and that one allows only one change at a time: ask for a second while one is in flight and it answers `cluster member change is in progress`. Let one finish before the next. After any `peer-remove`, re-read the group before you touch it again: the evicted peer is gone, its replacement is catching up, and a named leader is in place. The `Cluster` block of `nats stream info` shows all three. The full set of peer-management and stream-assignment operations is documented in [Reference](/reference/jetstream/api/meta/.md). We only need grow, move, and the verify step here. ## Pitfalls Three mistakes are common the first time you resize a live group. All three come from this page's two concepts: growing a group with catchup, and moving a replica off a server. **Don't stack membership changes before the replacement catches up.** A `peer-remove` evicts a healthy replica and its replacement starts empty, so for a while only the peers that already held the data can serve it. Fire a second change — another `peer-remove`, or a `--replicas` edit — before that replacement is `current`, and you can drop the number of peers holding the data below the majority the group needs, and it stops committing. Make one change, wait for a named leader and a caught-up replacement, then the next. The handling is the verify step itself. Make exactly one change, then read the `Cluster` block back before going further: #### CLI ``` #!/bin/bash # Move a replica off one server, safely, then verify the new peer set # before trusting the change. # # This assumes the east cluster (n1-east on 4222, n2-east on 4223, # n3-east on 4224, plus a fourth server n4-east) is running and ORDERS # holds a replica on n4-east. peer-remove does NOT shrink the stream: it # evicts the replica from n4-east and the meta leader re-places it on # another qualifying server, so the replica count stays the same. Make # ONE change at a time and wait for a leader and a caught-up replacement # before the next — stacking changes can drop the peers holding the data # below a majority and the stream stops committing. # First, read the current peer set. The Cluster block lists the leader # and every replica with its lag. Confirm there is a leader and that # every replica's lag is 0 before you change anything — a peer mid # catchup is not safe to lean on. nats --server nats://127.0.0.1:4222 stream info ORDERS # Evict the replica from one server by name. The meta leader picks a # replacement server, updates the stream assignment, and n4-east drops # its RAFT subscriptions. If no other server qualifies, an R>1 stream is # still left a peer short: the server evicts the peer and returns "peer # remap failed" (only a single-replica stream is refused outright). # (To change the replica COUNT, use: nats stream edit ORDERS --replicas=N) nats --server nats://127.0.0.1:4222 stream cluster peer-remove ORDERS n4-east # Verify. Re-read the Cluster block and confirm three things: # - n4-east is gone from the Replicas list, # - there is still a named Leader, # - the replacement replica is catching up (and reaches lag 0). # # Only when a leader is back and the replacement is current is it safe to # make the next change. If the stream shows "no leader", stop — you have # lost quorum and must restore a peer, not make another change. nats --server nats://127.0.0.1:4222 stream info ORDERS ``` #### C ``` // First, read the current peer set. Confirm there is a leader and // that every replica's lag is 0 before you change anything — a // peer mid catchup is not safe to lean on. natsMsg *reply = NULL; s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { printCluster(si); jsStreamInfo_Destroy(si); si = NULL; } // Evict the replica from one server by name. nats.c has no // peer-remove helper, so send the JetStream API request directly — // the same request the CLI's `stream cluster peer-remove` sends. // The meta leader picks a replacement server and updates the // stream assignment; n4-east drops its RAFT subscriptions. If no // other server qualifies, an R>1 stream is still left a peer // short: the reply carries "peer remap failed" (only a // single-replica stream is refused outright). // (To change the replica COUNT, update the stream's Replicas // field with js_UpdateStream instead.) if (s == NATS_OK) s = natsConnection_RequestString(&reply, conn, "$JS.API.STREAM.PEER.REMOVE.ORDERS", "{\"peer\":\"n4-east\"}", 5000); if (s == NATS_OK) { printf("%.*s\n", natsMsg_GetDataLength(reply), natsMsg_GetData(reply)); natsMsg_Destroy(reply); } // Verify. Re-read the cluster info and confirm three things: // - n4-east is gone from the replicas list, // - there is still a named leader, // - the replacement replica is catching up (and reaches lag 0). // // Only when a leader is back and the replacement is current is it // safe to make the next change. If the stream shows no leader, // stop — you have lost quorum and must restore a peer, not make // another change. if (s == NATS_OK) s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) printCluster(si); ``` If that second `stream info` shows `no leader`, stop. You've lost quorum, and the fix is to restore a peer, not make another change. **A freshly added peer isn't safe until its lag is zero.** When you raise the replica count, the new peer joins the set immediately but holds none of the stream's history. It can't win an election and can't serve a read while it catches up. Kill another server mid-catchup and you can drop below the peers that actually hold the data and stall the group. Don't treat a new peer as a working replica until `nats stream info` shows it `current` with zero lag, which is when catchup is done. **Removing the only peer needs `--force`, and forcing it doesn't move the data.** The CLI refuses to `peer-remove` the last peer of a stream without `--force` (`removing the only peer on a stream will result in data loss`). Even forced, there's nowhere to re-place the replica, so the server refuses to drop it and answers `peer remap failed` rather than brick the stream. The danger to avoid is forcing the removal in the belief the data will follow — it won't. Know the current replica count from `nats stream info` first, and change the count with `nats stream edit --replicas` rather than by removing the last peer. ## Where you are You can now resize a live RAFT group without taking the stream down: * You grew the `ORDERS` group by raising `--replicas`, watched the new peer catch up, and learned not to lean on it until `stream info` shows it `current`. * You moved a replica off a server with `nats stream cluster peer-remove`, saw the meta leader re-place it to keep the replica count, and confirmed a leader was back before touching it again. * You know to make one membership change at a time, and why stacking a second before the replacement catches up is the way to lose quorum. The `ORDERS` stream is back on `n1-east`, `n2-east`, and `n3-east`, the same three peers it started on — but now you can grow or shrink that set on purpose. ## What's next You've walked the whole mechanism: routes form the mesh, RAFT groups agree, a quorum commits each write, placement decides where replicas live, and peer management grows the set safely. The last page collects the recap, points to where the exhaustive detail lives, and gathers every page's Pitfalls into one production checklist. Continue to [Where to go next](/learn/clustering/where-next.md). ## See also * [Raft and leaders](/learn/clustering/raft-and-leaders.md) — election and `leader-stepdown`, which a `peer-remove` triggers when it drops the leader. * [Reference → meta API](/reference/jetstream/api/meta/.md) — the full set of peer-management and stream-assignment operations. * [Backup & recovery](/learn/backup-recovery/.md) — take a backup before a risky resize, so a lost replica is recoverable. --- # Where to go next You started this chapter with nothing running, and by the end you have three servers (`n1-east`, `n2-east`, `n3-east`) that found each other from a single seed route, elected leaders for every RAFT group, and hold the `ORDERS` stream at `R=3`. A write from `order-svc` now lands on the leader, commits once a quorum has it, and survives one server dying. That covers the full arc of the chapter. This page collects the mechanism you built into one place and points you at the chapters and Reference that take it further. ## The five core ideas Every page in this chapter turned on the same five ideas. **Routes** are the server-to-server connections that form the cluster. You configure one explicit seed route, and gossip does the rest: each server shares the peers it knows in its INFO, so one seed grows into a full mesh without you listing every server. **RAFT groups** are how the cluster agrees. There's one meta group across the whole cluster plus one group per stream, and each group runs an election to pick a single leader. The leader is the only member that accepts writes; the followers replicate from it. A **quorum** is a majority of a group's peers: for `R=3`, two of three. The leader appends a write to its log and commits it the moment a quorum has the entry. Quorum is what lets the group make progress while a minority is down, and lose nothing when it returns. **Placement** decides where the replicas live. You constrain a stream to a cluster and to servers carrying matching tags, and the meta leader assigns the peers from the servers that qualify. **Peer management** grows or shrinks the set. You raise the replica count and wait for the new peer's catchup to bring its lag to zero before you lean on it; you move a replica off a server one change at a time so the group never drops below a majority. Those five ideas are routes, RAFT, quorum, placement, and peers. Everything else in this chapter (terms, elections, append entries, apply, preferred leader, migration) is a refinement of those five. ## Where the details live now The chapter is unversioned and concept-first. The exact election timers, the WAL format, the full `cluster {}` field list, and every `StreamConfig` option live in **Reference**, which is versioned and exhaustive. When you need the precise type of a config field or the wire format of a route, that's where to look. The [Reference root](/reference/.md) is the entry point. The handoff phrases throughout this chapter ("the full set of options is documented in Reference") all point into it. The pages you'll reach for most: * [`cluster {}` config](/reference/config/cluster/.md) and the [route protocol](/reference/protocols/route.md): every field behind forming a cluster * [Stream API](/reference/jetstream/api/stream/.md) and [server tags](/reference/config/server_tags.md): replicas and placement * [`/jsz`](/reference/system/monitor/jsz.md) and [`/raftz`](/reference/system/monitor/raftz.md): the monitoring endpoints behind every cluster field you read with the CLI ## Sibling deep dives This chapter is the mechanism beneath two others, so it depends on them and they depend on it. The [Topologies deep dive](/learn/topologies/.md) covers the shapes: when to run a single server, when to grow into a cluster, when to reach for a super-cluster or leaf nodes. This chapter ran beneath its `east` cluster; Topologies decides *what shape* to build, and this chapter explains *how* the shape agrees and replicates once it's built. The [JetStream deep dive](/learn/jetstream/.md) created the `ORDERS` stream this chapter replicated. Its page on [surviving node loss](/learn/jetstream/surviving-node-loss.md) is the one-page operator intro to `R=3`; this chapter went deeper, into the election and the quorum commit that make "lose a node, keep serving" actually work. For copies *across* clusters, JetStream's [mirrors and sources](/learn/jetstream/mirrors-and-sources.md) page covers the DR story this chapter leaves to it. The [Deployment deep dive](/learn/deployment/.md) covers running this on real infrastructure: Kubernetes, rolling upgrades, and sizing the servers you formed a cluster from here. The [Backup & Recovery deep dive](/learn/backup-recovery/.md) covers the operational protection: snapshotting a stream before a risky peer change, and restoring it if a migration goes wrong. ## Where you are This is the end of the chapter. The arc is complete, and this page adds no new scenario state. The `east` cluster, its elected leaders, and the `ORDERS` stream at `R=3` are still running in your session exactly as you left them on the previous page. You can keep experimenting (kill a server and watch a re-election, add a fifth peer, move placement) or tear it all down with `nats stream rm ORDERS` and stop the three servers when you're done. You hold the core model: routes form the mesh, RAFT groups agree, a quorum commits each write, placement decides where the replicas live, and peer management grows the set without losing agreement. That model is the minimum you need for operating any NATS cluster in production. ## Production checklist Every page in this chapter closed with a Pitfalls section. This collects the action items from all of them in one place: a last pass before you trust a cluster with real orders. Each group links back to the page that explains the why. ### Forming a cluster — see [Pitfalls](/learn/clustering/forming-a-cluster.md#pitfalls) * Give every server the same `cluster.name`; a mismatch silently forms two clusters that look like one until a message fails to cross. * Point `routes` at the route port (6222), not the client port (4222); aiming at the client listener never establishes the route. * List two or three seed routes so the cluster still forms if one seed server is down at boot; gossip needs only one to reach, but only if that one is up. ### Raft and leaders — see [Pitfalls](/learn/clustering/raft-and-leaders.md#pitfalls) * Treat a brief "no leader" window during failover as normal; the election timer is 4–9s, so let the client retry instead of failing the write. * Use `nats stream cluster step-down` to move leadership off a server, not to pick its successor; the next election is still quorum-based, so read `nats stream info` to learn who won. * Track the meta leader and a stream leader as different RAFT groups; check `nats server report jetstream` for one and `nats stream info ORDERS` for the other, because losing one is not losing the other. ### Replication and R=3 — see [Pitfalls](/learn/clustering/replication-and-r3.md#pitfalls) * Run real orders at `R≥3`, never `R=1`; a single copy is gone with its server's disk, with no failover and no recovery. * Read from the leader when you need read-after-write; a follower can lag, so a Direct Get from one returns data that's correct but not the newest. * Read a `PubAck` as quorum held, not full replication; before deliberately taking a server down, verify each replica shows `current` in `nats stream info`. ### Placement — see [Pitfalls](/learn/clustering/placement.md#pitfalls) * Read a server's tags back before placing against them; tags are an intersection and a missing one fails with `no suitable peers for placement` rather than falling back to any server. * Spell tags exactly; matching folds case (`ssd` equals `SSD`) but `sdd` matches nothing. * Remember placement can't name a leader — a `preferred` server in placement is rejected; to move leadership to a specific server use `nats stream cluster step-down --preferred` (NATS Server 2.11+), and treat it as a request the quorum election can still overrule. ### Scaling and peer management — see [Pitfalls](/learn/clustering/scaling-and-peers.md#pitfalls) * Make one membership change at a time; stacking a second before the replacement is `current` can drop the peers holding the data below a majority and the stream stops committing. Make one change, wait for a named leader and a caught-up replacement, then the next. * Wait for a freshly added peer to show `current` with zero lag before trusting it; while it catches up it can't win an election or serve a read, so don't kill another server mid-catchup. * Change the replica count with `nats stream edit --replicas`, not by removing the last peer; the CLI blocks removing the only peer without `--force`, and even forced the server answers `peer remap failed` rather than move the data. ## See also * [Reference](/reference/.md): every config field, flag, default, and error code behind this chapter, versioned and exhaustive * [Topologies deep dive](/learn/topologies/.md): the shapes this chapter runs beneath, including the `east` cluster it reuses * [JetStream → surviving node loss](/learn/jetstream/surviving-node-loss.md): the one-page replica intro this chapter goes deeper than --- # Core NATS Deep Dive Core NATS is the foundation everything else is built on. It's a publish-subscribe system: a publisher publishes a message to a subject, and every subscriber interested in that subject receives a copy. There's no broker queue, no storage, and no acknowledgment. The message goes to whoever is listening right now, and it goes to nobody else. That single idea, subjects plus interest, is enough to build four communication patterns and the addressing they share. This chapter walks through them the way you'd discover them while writing real code: start with one publisher and one subscriber, then add addressing, replies, load balancing, and reply gathering, one page at a time. The chapter starts a step before that, with the connection every client opens first, and continues past the patterns into message headers, server-side subject mapping, connection lifecycle, and debugging delivery. **Message flow — Publish / Subscribe (animated):** Animated publish/subscribe: a publisher emits messages; NATS delivers a copy to every matching subscriber. * Publisher → NATS (subject: updates) * NATS → Subscriber 1 (subject: updates) * NATS → Subscriber 2 (subject: updates) ## Core NATS is ephemeral The one property to hold in your head for the whole chapter: **core NATS is at-most-once**. A message reaches every interested subscriber that's connected at the moment of publish, at most once. If a subscriber is offline, restarting, or not subscribed yet, it never sees that message. The server does not store it for later. That behavior is intentional. It keeps core NATS small and fast, and it's exactly right when each message is superseded by the next one, such as a live price, a current temperature, or a cache invalidation. When you need messages to wait for a subscriber, survive a restart, or be replayed later, you add a stream. That's [JetStream](/learn/jetstream/.md), the persistence layer that sits on top of core NATS. This chapter is everything that happens *before* you reach for it. ## The running scenario Every page builds the same example: the Acme ORDERS platform, shown at its foundation, before it adds any persistence. The order services talk over core NATS only. Three things happen to an order: it's created, shipped, or canceled. Each one shows up as a message on a subject: ``` orders.created orders.shipped orders.canceled ``` The payload is a small JSON object, the same shape across every example in this chapter: ``` { "order_id": "ord_8w2k", "customer": "acme-co", "total_cents": 4200, "ts": "2026-05-22T10:14:22Z" } ``` Several services care about these messages. A `warehouse` process packs the box on `orders.created`. A `notifications` service emails the customer when an order ships. An `analytics` pipeline counts everything. Later pages add regional analytics and audit views on wildcard subjects, an `inventory` service that answers requests, a pool of `packers` that share the load, and three shipping-quote providers that each reply to the same question. You keep a single local `nats-server` through the whole chapter — two later pages restart it with a flag or a config file — and add subscribers and services as you go. ## Who this is for You've read the [Core Concepts → Publish & Subscribe](/concepts/pub-sub-basics.md) primer, or you're otherwise comfortable with the idea of subjects and subscribers. Rather than re-teaching the *what*, this chapter shows the *how*: the mechanism on the wire, the trade-off behind each pattern, and a runnable session you build up command by command. Each page introduces one or two new ideas and builds on the one before it. ## Map | Page | What you learn | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | [Connecting](/learn/core-nats/connecting.md) | One long-lived connection per client, the connect URL and handshake, connection names, and PING/PONG heartbeats | | [Publish-subscribe](/learn/core-nats/publish-subscribe.md) | Fire-and-forget publish, subscribing and unsubscribing, the interest graph, at-most-once delivery, and the 1 MB max payload | | [Subjects & wildcards](/learn/core-nats/subjects-and-wildcards.md) | Dot-delimited subject hierarchies and the `*` and `>` subscriber wildcards | | [Request-reply](/learn/core-nats/request-reply.md) | The `_INBOX` reply subject, timeouts, and the no-responders signal | | [Queue groups](/learn/core-nats/queue-groups.md) | Load balancing where each message goes to exactly one group member | | [Scatter-gather](/learn/core-nats/scatter-gather.md) | Fan one request to many responders and gather the replies | | [Message headers](/learn/core-nats/headers.md) | Key/value metadata in the `NATS/1.0` format, setting and reading it, and the status codes the server sends in-band | | [Subject mapping](/learn/core-nats/subject-mapping.md) | Server-side rewriting of a subject before routing, to rename it, split its traffic by weight, or shard it by a hashed token | | [Connection lifecycle](/learn/core-nats/connection-lifecycle.md) | What a dropped connection does to your client and in-flight messages, the client-owned reconnect, and the lifecycle callbacks that watch it | | [Debugging delivery](/learn/core-nats/debugging-delivery.md) | Why a published message never arrived, found with a wire tap, the server's subscription list, and message tracing | | [Where to go next](/learn/core-nats/where-next.md) | The four-idea recap and a map of what's beyond the foundation | ## Prerequisites You'll need: * A single local `nats-server`. The default build is all you need; core NATS requires no flags. Start it with `nats-server`. * The `nats` CLI installed and pointed at your server. Every example leads with the CLI; most pages show the same operation in JavaScript, Go, Python, Java, Rust, and C# as well. The two server-side pages (Subject mapping, Debugging delivery) are CLI-only. * One later page, [Subject mapping](/learn/core-nats/subject-mapping.md), starts the server from a config file with `nats-server -c server.conf`; that page gives you the file. * [Debugging delivery](/learn/core-nats/debugging-delivery.md) reads the server's monitoring port, which you turn on with `nats-server -m 8222`. Open a terminal, run `nats-server`, and continue to the next page. ## What's next Start with [Connecting](/learn/core-nats/connecting.md): the one long-lived connection every client opens before it can publish or subscribe, and the handshake that opens it and the heartbeats that keep it alive. ## See also * [Core Concepts → Publish & Subscribe](/concepts/pub-sub-basics.md) — the five-minute overview of the same material. * [JetStream Deep Dive](/learn/jetstream/.md) — the persistence layer this chapter stops short of. --- # Connecting A NATS application talks to the network through a **client**: a library you embed in your program, or the `nats` command-line tool, that opens a connection to a server and moves messages over it. Before any pattern in this chapter works, the client has to connect. This page is about that connection itself: what it is, how you open one, and the few choices you make when you do. You need one local `nats-server` running for this chapter. The default build needs no configuration: ``` nats-server ``` Leave it running. It listens on port 4222, and every page in this chapter connects to it. ## One connection carries everything A **connection** is a single, long-lived TCP connection between the client and the server. The client opens it once, when your application starts, and keeps it open for the application's lifetime. Everything the client does travels over that one connection. Every message you publish, and every **subscription** you register (a standing request to receive messages addressed to a given subject), is multiplexed onto the same TCP connection and tagged so both ends can tell the streams apart. The client doesn't open a new connection per subject or per message; one connection carries them all. That's why you connect once and reuse the result. A single connection handles thousands of subscriptions and a high message rate, and sharing it across every publish and subscribe is the pattern the clients are built for. ## The connect URL The client opens a connection by dialing a **connect URL**: the address of the server, in the form `nats://host:port`. Servers listen on port 4222 by default, so a server on your own machine is `nats://127.0.0.1:4222`. That's also what a client dials when you don't give it a URL, and what the `nats` CLI uses until you point it elsewhere with `--server` or a saved context. The scheme names the transport. `tls://` is the same connection encrypted, and `ws://` or `wss://` carries the protocol over WebSocket — the transport a browser has to use, and the one that gets through a network allowing only outbound 443. [Connect over WebSocket](/learn/websocket/.md) covers that case. One URL names one server. A production client usually passes several URLs so it can fail over when one server is unreachable; that list, and how the client works through it, belong to [Resilient clients → Connecting](/learn/resilient-clients/connecting.md). Here, one local server is all you need. ## The connect handshake When the client reaches the server, the two run a short exchange, the **connect handshake**, before any message flows. The server sends first. It immediately sends an `INFO` message announcing itself and the limits it enforces, among them `max_payload` (the largest message it accepts, 1 MB by default) and whether it supports message headers. The client reads `INFO`, then replies with a `CONNECT` message that declares what it wants: its name, the protocol features it supports, and any credentials. The server accepts, and the connection is ready to carry messages. You can watch the first half of that exchange directly. Point any raw TCP tool at the server and it prints the `INFO` line the instant it connects: #### CLI ``` #!/bin/bash # The server sends first. The moment a TCP client connects to port 4222, the # server sends an INFO line -- plain text, ending in CRLF -- describing itself # and the limits it enforces. Any raw TCP tool shows it; here we use nc against # the running nats-server. Ctrl-C to quit. nc localhost 4222 # The server immediately prints a line like: # # INFO {"server_id":"ND2X...","version":"2.14.0","proto":1,"headers":true,"max_payload":1048576,...} # # headers:true means the server supports message headers, and max_payload # (1048576 bytes, 1 MB) is the largest message it will accept. A real client # reads this INFO, then replies with its own CONNECT line declaring its name # and the features it wants -- the exchange your client library runs for you. ``` #### C ``` // Raw POSIX/BSD sockets -- Unix-like systems (Linux, macOS) only. // The server sends first. Open a raw TCP connection to port 4222 -- // no NATS library involved -- and the server immediately sends its // INFO line: plain text, ending in CRLF, describing itself and the // limits it enforces (headers support, max_payload, ...). struct sockaddr_in addr; char buf[4096]; ssize_t n; int fd = socket(AF_INET, SOCK_STREAM, 0); memset(&addr, 0, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_port = htons(4222); inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr); if (connect(fd, (struct sockaddr *) &addr, sizeof(addr)) != 0) { perror("connect"); return 2; } // Read what the server sent on connect and print it. This is the // first half of the handshake; a real client would now reply with // its own CONNECT line, which the library does for you. n = recv(fd, buf, sizeof(buf) - 1, 0); if (n > 0) { buf[n] = '\0'; printf("%s", buf); } ``` These messages are plain text, each line ending in a carriage return and line feed, the same wire format as the `PUB` and `SUB` lines that carry your messages and the `PING`/`PONG` described below. You never write them yourself; the client library runs the whole handshake when you call connect. The step-by-step walkthrough, and every field the two sides exchange, is in [Resilient clients → Connecting](/learn/resilient-clients/connecting.md). ## Naming the connection One field in that `CONNECT` message is worth setting yourself: the **connection name**. By default a client connects without a meaningful name, and server monitoring shows it with no name at all (the `nats` CLI substitutes a generic `NATS CLI Version …` label). Give it a name and that name identifies the connection instead. The `nats` CLI sets the name with the global `--connection-name` flag. Name a connection after the service that owns it. Here it's `warehouse`, the first Acme service, which opens a connection and confirms it can reach the server: #### CLI ``` #!/bin/bash # Open a named connection to the server and measure the round trip. `nats rtt` # dials the default URL (nats://127.0.0.1:4222), sends a PING, times the PONG # that comes back, and by default averages five such round trips. # --connection-name labels this connection "warehouse" so it is identifiable # in server monitoring instead of the generic default name the CLI uses when # you leave it unset. nats rtt --connection-name warehouse # You will see the average round trip per server address, something like: # # nats://127.0.0.1:4222: # # nats://127.0.0.1:4222: 187µs # # A printed time means the connect handshake succeeded and the server answered # the PING. The same --connection-name flag works on every nats command (sub, # pub, request), so name each long-lived client after the service that owns it. ``` #### C ``` // Name the connection "warehouse" so server monitoring identifies it, // instead of showing a connection with no name at all. if (s == NATS_OK) s = natsOptions_SetName(opts, "warehouse"); // Open the connection. This runs the connect handshake: the server // sends INFO, the client answers with CONNECT carrying the name. if (s == NATS_OK) s = natsConnection_Connect(&conn, opts); // Measure the round trip: one PING sent to the server, timed until // its PONG comes back. A printed time means the handshake succeeded // and the server is answering. if (s == NATS_OK) { int64_t rtt = 0; s = natsConnection_GetRTT(conn, &rtt); if (s == NATS_OK) printf("round trip: %dus\n", (int) (rtt / 1000)); } ``` The name then surfaces wherever the server lists its connections: `nats server report connections` and the [monitoring endpoint](/learn/monitoring/monitoring-endpoints.md). When you're staring at a list of connected clients, the name tells you which application each one is, instead of leaving you to guess from an address and a number. The report command answers fully once you connect with system-account credentials; the monitoring endpoint (`nats-server -m 8222`) is the no-credentials alternative. ## Receiving your own messages Another connect-time choice is **echo**. By default a connection receives its own messages: if one client publishes to a subject and also holds a subscription on that same subject, the server delivers a copy back to that client, exactly as it delivers to any other subscriber. Most of the time this goes unnoticed, because a client subscribes to subjects that other clients publish. It matters when a single client does both on one subject, such as a service that publishes an update and also listens for updates: it receives what it just published. Turning echo off at connect time (clients call the option `NoEcho`) tells the server to skip the originating connection when it delivers, so a client never gets a copy of its own message. Like the name, it's fixed for the life of the connection: you choose it when you open the connection, not while messages flow. ## Staying connected with PING/PONG An open connection can sit idle between messages. To notice when the peer at the other end has gone away, crashed or cut off from the network, without waiting for the next message to fail, both ends exchange heartbeats. The mechanism is **PING/PONG**. Each side periodically sends a `PING` and expects a `PONG` back. When too many PINGs go unanswered, that side treats the peer as dead and closes the connection. The client and the server each run this on their own, so either end can detect a peer that stopped responding. The defaults match on both sides: a PING every two minutes, and the connection is declared dead after two unanswered PINGs. The `nats rtt` command you ran above triggers this exchange on demand. Each round trip it reports is one `PING` sent to the server and the `PONG` the server sent back, and by default it averages five of them. The heartbeat only detects the drop. Recovering from it, by reconnecting and buffering publishes while the client is disconnected, is [Connection lifecycle](/learn/core-nats/connection-lifecycle.md) later in this chapter, and [Resilient clients → Reconnection](/learn/resilient-clients/reconnection.md) for production tuning. ## Pitfalls **Opening a new connection per message.** The connection is meant to be opened once and shared. Connecting, publishing a single message, and disconnecting repeats the whole handshake every time and discards a connection built to carry thousands of messages. Open one connection when your service starts, hold it, and reuse it for every publish and subscribe. **A client that publishes and subscribes on one subject receives its own messages.** With echo on (the default), a client's own published messages come back to its matching subscriptions. If that isn't what you want, open the connection with echo off (`NoEcho`) so the originating connection is skipped. **Exiting before buffered publishes are sent.** A publish hands the message to the client's write buffer and returns right away, so a short-lived program that exits immediately can quit before the buffer reaches the server. Flush before you exit. The [Publish-subscribe pitfall](/learn/core-nats/publish-subscribe.md#pitfalls) covers this in full; it's the same buffer whether you publish one message or many. ## Where you are The Acme world now has a running server and a client that can reach it: * One local `nats-server` is up, listening on `nats://127.0.0.1:4222`. * You understand a connection as one long-lived TCP connection that multiplexes every publish and subscription. * You open it by dialing a URL, and the connect handshake exchanges the server's `INFO` and the client's `CONNECT` before any message moves. * The `warehouse` connection carries a name, set with `--connection-name`, so the server can identify it. * Echo delivers a client its own messages by default, and `PING`/`PONG` heartbeats let both ends notice a dead peer. ## What's next The connection is open. Now put it to work: publish a message to a subject and have other clients receive a copy. [Publish-subscribe](/learn/core-nats/publish-subscribe.md) introduces the one operation the rest of core NATS is built on, and the interest graph that decides who gets each message. ## See also * [Core Concepts → What is NATS?](/concepts/what-is-nats.md) — the client, the server, and the messaging model in five minutes. * [Resilient clients → Connecting](/learn/resilient-clients/connecting.md) — connection options, server pools, connect-timeout tuning, and the full handshake walkthrough for production. * [Reference → Client protocol](/reference/protocols/client.md) — the wire-level `INFO`, `CONNECT`, `PING`, and `PONG` this page describes at the model level. --- # Connection lifecycle The [Connecting](/learn/core-nats/connecting.md) page opened a client's connection: the handshake, a connection name, and the heartbeats that tell client and server the link is still alive. A connection doesn't stay up forever, though. The server it's attached to restarts, or a network blip cuts the socket, and the connection drops. This page answers one question: what happens to your client, and to your messages, when that drop happens. The client handles most of it for you. Knowing exactly what it does, and what it can't do, tells you which failures you can ignore and which you have to design around. Two ideas carry the page. First, the client owns reconnection: it re-dials and rebuilds its own state without your help. Second, you watch the whole lifecycle through a few callbacks. After that, two shorter sections point at where the tuning and hardening live, in the [Resilient clients](/learn/resilient-clients/.md) chapter. ## The client owns reconnection When the socket drops, the client reconnects for you rather than surfacing the failure and stopping. It re-dials a server, replays the connect handshake, and re-sends every subscription it had, so messages start flowing again. This is on by default in every client library. The client has to re-send its subscriptions because the server keeps no per-client memory. A subscription exists only as an entry in the server's in-memory interest graph, tied to one socket. When that socket dies the entry is gone, and to the server the reconnecting client is a brand-new connection. So a packer's interest in `orders.created` isn't restored by the server; the client library restores it by subscribing again. That leaves the messages, and the client treats the ones you send and the ones you'd receive very differently. The publishes *you* make while disconnected are held. A publish call during the gap doesn't fail; the client writes it to a **reconnect buffer**, an outbound queue that keeps your publishes during the outage and flushes them, in order, once the link returns. That buffer is bounded (8 MB by default), and a publish that would overflow it returns an error rather than growing without limit. Sizing it, and handling that overflow, belongs to [Reconnection](/learn/resilient-clients/reconnection.md). The messages *other* publishers send while you're away are gone, and no buffer on your side can change that. Core NATS is [at-most-once](/learn/core-nats/publish-subscribe.md#at-most-once-delivery): while your subscription is missing, an `orders.created` message from another service finds no interest to match, so the server discards it. When you reconnect your subscription is back, but that message was never kept and isn't coming. That gap is the reason to reach for a server-side store when missed messages actually matter, such as an order that must be handled even if the subscriber was restarting. That store is [JetStream](/learn/jetstream/.md), the layer this chapter stops short of. ## Observing the lifecycle A connection moves through states a single publish or subscribe call never shows you: CONNECTED, then RECONNECTING after a drop, then CONNECTED again, and eventually CLOSED. To see those transitions you register **lifecycle callbacks** when you open the connection, functions the client runs as the connection changes state. Three of them mark the transitions: * a **disconnect handler** runs when the link drops and reconnection begins, * a **reconnect handler** runs when the client re-establishes a connection, * a **closed handler** runs once, when the connection is finished for good and won't come back. A disconnect followed by a reconnect is routine and needs nothing from you beyond a log line. The closed handler is the one that matters: it means the client gave up retrying, or you closed the connection deliberately, and this connection is over. A fourth callback isn't about state at all. The **async error handler** is where the client reports errors that don't belong to any single call you made, such as the server rejecting a subscription because your credentials don't permit that subject. An error a publish call can detect comes back from the call itself, but these arrive with no call to attach them to, so the client routes them all to this one handler. Wire it up in every real service; without it, those errors are invisible. Registering these four is the first thing a production connection does, before it publishes or subscribes a single message. The [Reconnection](/learn/resilient-clients/reconnection.md) page shows a service registering a callback that logs every failed reconnect attempt through a long outage. A connection also reaches CLOSED the clean way, when you shut down on purpose; draining it first so in-flight work finishes is [Drain & Shutdown](/learn/resilient-clients/drain-and-shutdown.md). ## Force a reconnect Reconnection usually happens *to* a client: the server drops the link and the client reacts. A client can also start it deliberately. **Force reconnect** is a call that closes the current connection and immediately re-dials the pool, running the same path a real disconnect would: subscriptions are restored and buffered publishes flush, exactly as if the server had gone away. You use it when the connection is healthy but you want to move it. After you scale a cluster out, clients stay on the servers they first dialed and a new server sits idle; forcing a reconnect across the fleet spreads clients onto it. It's also how a client leaves a server you know is about to go down, on your schedule instead of waiting for the socket to close. nats.go exposes this as `Conn.ForceReconnect()`, and several other clients offer the same call under their own naming. There's nothing new to learn in the mechanics; it's the ordinary reconnect from the top of this page, started by you. ## Lame-duck mode A server doesn't always vanish without warning. When an operator takes one down for an upgrade, the server can enter **lame-duck mode**: it stops accepting new connections, announces to every connected client that it's about to go away, and then closes the existing ones gradually rather than all at once. From the client's side this is a gentler disconnect. A client that watches for the notice can reconnect to another server in the pool *before* its socket is cut — the lame-duck callback wired to a forced reconnect — and move without ever seeing a hard drop. Even a client that does nothing special is disconnected in a spread-out wave, instead of every client on the server reconnecting at the same instant. An operator triggers it with a signal to the server process: #### CLI ``` #!/bin/bash # Lame-duck mode: tell a running server to step down gracefully instead of # dropping every client at once. Keep the local nats-server running, with a # subscriber (nats sub orders.created) attached in another terminal. # # Signal the server to enter lame-duck mode. With a single nats-server on the # box you don't pass a pid -- the CLI finds the one running process: nats-server --signal ldm # The running server logs that it is draining and stops taking new clients: # # [INF] Entering lame duck mode, stop accepting new clients # # It then closes existing clients gradually rather than all at once. Across a # cluster, a client that watches for the lame-duck notice can move to another # server before its socket is cut; the rest are disconnected in a spread-out # wave and reconnect elsewhere on their own. Against this single server # there is nowhere else to go, so the subscriber is disconnected gracefully # and reconnects once you start the server again -- the same stop and start as # any restart. # # The grace period before clients are closed, and the window the disconnects # are spread over, are server settings an operator tunes. See the server-side # walkthrough in Deployment -> Rolling upgrades. ``` #### C ``` // Register a callback for the lame-duck notice. When an operator takes // the server down gracefully (nats-server --signal ldm), the server // stops accepting new connections and announces to every connected // client that it is about to go away; this callback is that notice. if (s == NATS_OK) s = natsOptions_SetLameDuckModeCB(opts, onLameDuck, NULL); if (s == NATS_OK) s = natsConnection_Connect(&conn, opts); // Keep the client running. Signal the server from another terminal // with: nats-server --signal ldm if (s == NATS_OK) { printf("connected, waiting for the lame-duck notice...\n"); while (natsConnection_Status(conn) != NATS_CONN_STATUS_CLOSED) nats_Sleep(100); } ``` The server-side details, the signal itself and the timing it uses to spread the disconnects, are an operations task covered in [Deployment → Rolling upgrades](/learn/deployment/rolling-upgrades.md). Lame-duck mode matters most across a cluster, where the client has another server to move to. Against the single local server in this chapter there's nowhere else to go, so the client waits out the graceful shutdown and reconnects when the server comes back, the same stop and start as any other restart. ## Try it You can watch the client reconnect against the single local server. Subscribe in one terminal, then stop and restart the server underneath it. The subscriber logs the drop, reconnects on its own, and resumes, and you never restart the subscriber: #### CLI ``` #!/bin/bash # Watch the client reconnect on its own. Keep the local nats-server from # earlier in the chapter running in its own terminal. # # In this terminal, subscribe as the warehouse service on orders.created: nats sub orders.created # Now stop the nats-server (Ctrl-C in its terminal) and start it again. The # subscriber keeps running the whole time. It logs the drop -- the reason is # EOF because the server closed the socket cleanly on the way down: # # 14:02:11 Subscribing on orders.created # 14:02:19 >>> Disconnected due to: EOF, will attempt reconnect # # You never restart the subscriber. Once the server is back, publish an order # and the subscriber prints it, because the client re-dialed and re-sent the # subscription itself. Add --trace to log an explicit reconnect line too, # along with the initial connect and each retry delay: # # nats sub orders.created --trace # 14:02:21 >>> Reconnected to nats://127.0.0.1:4222 (127.0.0.1:4222) # # Publishing WHILE the server is down fails instead: nats pub has no server # to connect to, and nothing queues the message server-side. The at-most-once # loss happens on a live server: a message that arrives before the # subscription is restored finds no interest and is discarded. ``` #### C ``` // Register the lifecycle callbacks before connecting: one for the // drop, one for the recovery. if (s == NATS_OK) s = natsOptions_SetDisconnectedCB(opts, onDisconnected, NULL); if (s == NATS_OK) s = natsOptions_SetReconnectedCB(opts, onReconnected, NULL); if (s == NATS_OK) s = natsConnection_Connect(&conn, opts); // Subscribe as the warehouse service. Now stop and restart the // nats-server: the client logs the drop, re-dials, and re-sends this // subscription on its own -- you never restart this program. Publish // an orders.created message once the server is back and it prints, // proving the subscription was restored on the new connection. if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "orders.created", onMsg, NULL); if (s == NATS_OK) { printf("subscribed to orders.created, restart the server underneath me\n"); while (natsConnection_Status(conn) != NATS_CONN_STATUS_CLOSED) nats_Sleep(100); } ``` Publish an `orders.created` message once the server is back and the subscriber prints it, which proves the subscription was restored on the new connection. Publishing *while* the server is down fails instead: `nats pub` has no server to connect to, and nothing queues the message server-side. The at-most-once loss from earlier in the page happens on a live server: a message that arrives in the gap before your subscription is restored finds no interest, and the server discards it. ## Pitfalls Each of these turns the automatic reconnect into a quiet way to lose data or stall a service. **Trusting the reconnect buffer as a delivery guarantee.** The buffer holds your publishes through a disconnect, but it lives in your client's memory. It isn't written anywhere durable, and if the process itself dies, everything in the buffer dies with it. Treat it as a short-term hold over a brief outage, not as storage you can rely on. A publish that must survive regardless needs the server-side store from earlier in the page. **Doing real work inside a connection callback.** The disconnect, reconnect, closed, and error handlers run on the client's own thread. Block one with a database write or a slow HTTP call, and every other callback on that connection waits behind it; in some clients the connection's processing does too. Keep the handlers to a log line or a signal on a channel, and do the actual work elsewhere. **Letting infinite reconnect hide a dead configuration.** A long-lived service should retry forever, but "forever" also means a client pointed at a wrong address, or refused by auth, retries that mistake silently and looks healthy while it never connects. Log the drop from the disconnect handler and every failed attempt from the reconnect-error callback the Reconnection page wires up, so a misconfiguration shows up as constant errors in your logs instead of silence. The retry-limit setting and its defaults are in [Reconnection](/learn/resilient-clients/reconnection.md). ## Where you are You now know what a dropped connection does to a client: * The client owns reconnection. It re-dials, replays the handshake, and re-subscribes on its own, because the server keeps no per-client state. * Your outgoing publishes are buffered through the gap and flushed on reconnect; messages others published while you were away are gone, which is at-most-once at work. * You observe the whole cycle through the disconnect, reconnect, closed, and async error callbacks, wired up before the first message. * A client can force its own reconnect, and a server in lame-duck mode lets clients leave gracefully before it stops. ## What's next A connection can drop and heal without you noticing, which is usually what you want, but it also means a delivery problem can hide behind a healthy-looking client. The next page, [Debugging delivery](/learn/core-nats/debugging-delivery.md), is how you tell a message that reached no subscriber from one that never left the client at all, using the server's own view of the connection. ## See also * [Resilient clients → Reconnection](/learn/resilient-clients/reconnection.md) — tuning backoff, jitter, retry limits, and the reconnect buffer for a long-lived service. * [Resilient clients → Drain & Shutdown](/learn/resilient-clients/drain-and-shutdown.md) — closing a connection cleanly so in-flight work isn't dropped. * [Deployment → Rolling upgrades](/learn/deployment/rolling-upgrades.md) — the server side of lame-duck mode: the signal, the grace period, and the timing. * [Publish-subscribe → At-most-once delivery](/learn/core-nats/publish-subscribe.md#at-most-once-delivery) — why a message published with no interest is gone for good. --- # Debugging delivery You publish an `orders.created` message and no packer packs the box. The publish returned without an error, so the message left your client, but nothing arrived. In core NATS that has one of three causes: * Nobody was subscribed to that subject when you published, so the server had nowhere to deliver the message and dropped it. * Something was subscribed, but not to the subject you actually published on. A typo or an extra token routes the message past every subscriber. * The subscriber was running, but its connection had dropped, so its interest was already out of the server's graph when you published. A plain publish gives you no feedback to tell these apart. It's [fire-and-forget](/learn/core-nats/publish-subscribe.md): the publisher can't see how many subscribers received the message, or whether any did. A request at least surfaces an empty subject as a [no responders](/learn/core-nats/request-reply.md#no-responders) signal, but a one-way publish stays silent. This page covers three tools that show you what actually happened, each aimed at a different cause. ## See every message with a wire tap The fastest way to learn whether your message went out, and on which subject, is to watch the traffic directly. Subscribe to the multi-token wildcard `>` and the server sends you a copy of every message published in your account. A `>` subscription used this way, to observe traffic rather than to drive application logic, is a **wire tap**. #### CLI ``` #!/bin/bash # Wire tap: subscribe to the multi-token wildcard > and the server sends you # a copy of every message published in your account, on its exact subject. # This is the fastest way to answer "did my message go out at all, and on # which subject?" -- the subject printed is the literal one the publisher # used, so a stray token or a typo shows up the moment you publish. # # nats sub prints one line per message: [#N] Received on "", then the # payload. Request-reply traffic shows the reply inbox too, as # [#N] Received on "" with reply "_INBOX...". nats sub ">" # On a busy account, a > tap receives every message in it. Narrow the tap to # the subject family you're investigating so you receive only what matters -- # here, everything under orders at any depth, and nothing else. nats sub "orders.>" ``` Each line names the exact subject the message was published on: ``` 12:30:25 Subscribing on > [#1] Received on "orders.us.created" {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} ``` If you expected `orders.created` and the tap shows `orders.us.created`, the subject is your problem, not interest. This is how you catch a stray token or the [whitespace that silently misroutes a publish](/learn/core-nats/subjects-and-wildcards.md#pitfalls). If the tap shows nothing at all when you publish, the message never left your publisher. The tap also shows request-reply traffic, including the [inbox](/learn/core-nats/request-reply.md#the-inbox) subjects clients generate for replies, because an `_INBOX` subject is an ordinary subject on the wire. The subject lines alone show the round trip: ``` [#2] Received on "orders.inventory.check" with reply "_INBOX.9tQ3pZ.aQ1" [#3] Received on "_INBOX.9tQ3pZ.aQ1" ``` Message #2 is the request, carrying the reply subject; message #3 is the answer coming back on that inbox. Seeing both confirms the round trip happened. On a busy account, `>` is a lot of traffic to receive, and a subscriber that can't keep up gets cut off as a slow consumer. Narrow the tap to the family you're debugging, such as `orders.>`, so you see only the subjects that matter. ## List the subscriptions the server holds If the message went out on the right subject but still didn't arrive, the question moves to interest. The server tracks every subscription in its in-memory [interest graph](/learn/core-nats/publish-subscribe.md#the-interest-graph); you just need to read it. On your local server, the plain-HTTP [monitoring port](/learn/monitoring/monitoring-endpoints.md) is the way in. Start the server with one (`nats-server -m 8222`) and its `/subsz` endpoint lists the subscriptions it holds right now. #### CLI ``` #!/bin/bash # See where interest actually lives, over the plain-HTTP monitoring port. # This needs the server started with a monitoring port: nats-server -m 8222 # /subsz lists the subscriptions the server holds right now. subs=1 adds the # per-subscription detail (account, subject, cid) on top of the summary counts. # acc=$G scopes the counts to your own account; without it num_subscriptions # spans every account, including the system account. Even scoped, a few # $SYS.REQ.* service subscriptions the server keeps in every account show up. curl -s 'http://localhost:8222/subsz?subs=1&acc=$G' | jq # The test parameter answers "who would receive a publish to THIS subject?" # It returns only the subscriptions whose subject would match a message # published to orders.us.created -- wildcard subscriptions (orders.>, # orders.*.created) included. An empty subscriptions_list means nobody is # subscribed, so a message published now would be dropped. curl -s 'http://localhost:8222/subsz?subs=1&acc=$G&test=orders.us.created' | jq # /connz shows the connections behind those subscriptions, so you can tell a # missing subscription from a subscriber whose connection was cut. The # Monitoring chapter documents its full parameter set. curl -s 'http://localhost:8222/connz?subs=true' | jq ``` `subs=1` adds a `subscriptions_list`, one entry per subscription, with the account, the subject, and the connection id (`cid`) behind it. `acc=$G` scopes the counts to your own account; without it, `num_subscriptions` spans every account, including the system account's. Even scoped, the list carries a few `$SYS.REQ.*` service subscriptions the server keeps in every account — they show up next to your own. The `test` parameter then answers the delivery question directly: give it the literal subject you published, and `/subsz` returns only the subscriptions that a message on that subject would match, wildcards included. ``` { "num_subscriptions": 6, "total": 1, "offset": 0, "limit": 1024, "subscriptions_list": [ { "account": "$G", "subject": "orders.>", "sid": "5", "msgs": 0, "cid": 8 } ] } ``` Here `num_subscriptions` shows the `$G` account holds six subscriptions, and `total` reports that one of them matches a publish to `orders.us.created`: the audit service's `orders.>` on connection `8`. An empty `subscriptions_list` means nobody was subscribed, and a message published then would be dropped. If Acme needs that order kept until a subscriber is ready, that's the job of a stream, which [JetStream](/learn/jetstream/.md) adds on top of these same subjects. To tell a missing subscription apart from a subscriber whose connection was cut, look at the connections. The `/connz` endpoint lists them, and with `?subs=true` it shows each connection's subscriptions; the [monitoring endpoints](/learn/monitoring/monitoring-endpoints.md) page covers it in full. A subscriber that believes it's connected but is absent from `/connz` has lost its connection, and its interest left the graph with it. The `nats` CLI can read the same state over NATS itself, with `nats server request subscriptions` and `nats server request connections`. Both ride `$SYS` system-account requests. A default `nats-server` has the `$SYS` account but no user in it, and the system request subjects only answer for clients connected to `$SYS`, so on a plain local server `nats server request subscriptions` fails with `server request failed, ensure the account used has system privileges and appropriate permissions`. `nats server request connections` still answers there, but only with your own account's connections. You get the full server-wide view once you connect with credentials for the [system account](/learn/security/.md), which is how production and clustered servers are run. ## Trace one message's path The wire tap shows subjects and the subscription list shows interest. **Message tracing** reports both for one subject: every subscriber the message matches and every hop it takes to get there. Run `nats trace` with the subject. #### CLI ``` #!/bin/bash # Trace the path of one message to a subject. Requires NATS Server 2.11 or # newer. nats trace publishes a special traced message and prints every hop it # takes and every subscriber it matches, including why it matched. # # Without --deliver the traced message is NOT handed to matching subscribers: # the trace reports who WOULD receive it without actually delivering it. It's a # probe, not a real publish, so it never triggers your subscribers' side effects. nats trace orders.us.created # Add --deliver to also deliver the traced message to matching subscribers, # turning the probe into a real publish that they receive. nats trace orders.us.created --deliver ``` `nats trace` needs NATS Server 2.11 or newer. It publishes a special traced message and prints the route the server computed for it. When nothing is subscribed to the subject, the trace says so: ``` Tracing message route to subject orders.us.created Client "NATS CLI Version development" cid:16 server:"n1" version:"2.11.6" --X No active interest Legend: Client: --C Router: --> Gateway: ==> Leafnode: ~~> JetStream: --J Error: --X ``` When a subscriber does match, a line beginning `--C` names it and ends with the subject its subscription matched on, such as `subject:"orders.>"`. One `--C` line appears per matching subscriber, so the trace tells you exactly who a real publish would reach. One property makes trace safe to run against a live system: without the `--deliver` flag, the traced message is not handed to those matching subscribers. The trace reports who would receive it without delivering anything, so it won't fire your subscribers' side effects. Add `--deliver` only when you want the traced message delivered for real. ## Which tool answers which symptom Start from what you can see and pick the tool that narrows it down. | Symptom | Tool | | ------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | Unsure the publish left your client, or unsure of the exact subject | Wire tap: `nats sub ">"`, then publish and watch | | Message goes out, but a subscriber you expected gets nothing | Subscription list: `/subsz?subs=1&acc=$G&test=` to see what matches | | A subscriber insists it's subscribed yet receives nothing | Connections: `/connz?subs=true` to check its connection is still open | | You want the authoritative list of who one publish would reach | Trace: `nats trace ` | If `test=` and `nats trace` both show a matching subscriber but the message still didn't arrive, the subscriber was absent at the instant you published and is present now. That timing gap is [at-most-once delivery](/learn/core-nats/publish-subscribe.md#at-most-once-delivery): core NATS delivers only to subscribers connected at the moment of publish, and never retries. ## Pitfalls **A `>` tap on a production account receives every message in it.** On a busy system that can overwhelm the tap and get its connection cut as a slow consumer. Scope the tap to the subject family you're investigating, like `orders.>`, and leave it running no longer than you need. **A trace is its own message, not proof about a past publish.** `nats trace` publishes a fresh traced message and reports the route that one takes. It tells you nothing about a message you published earlier, and without `--deliver` it never reaches your subscribers at all. Read its output as what a publish to this subject would do now, not what your last publish did. **The monitoring port is unauthenticated by default.** Anyone who can reach `:8222` can read your subjects and connections off `/subsz` and `/connz`. That's fine on your laptop, but don't expose the port on a shared or public network. Locking it down is a [security](/learn/security/.md) concern; name it now so you don't leave it open by accident. ## Where you are When an Acme order message doesn't arrive, you have a tool for each cause: * A wire tap (`nats sub ">"`) shows whether the message went out and on which subject. * The subscription list (`/subsz`, with `test=` for a specific subject) shows whether any interest matches, and `/connz` shows whether the subscriber's connection is still open. * A message trace (`nats trace`) gives the authoritative list of who a publish to one subject would reach, without delivering anything unless you ask. All three read the same interest graph the server uses to route every message; they just show it to you instead of leaving you to guess. ## What's next You've reached the end of the core NATS patterns and the tools to debug them. [Where to go next](/learn/core-nats/where-next.md) collects the whole model into one place and points to the chapters that build on it, starting with the persistence layer that keeps the messages core NATS discards. ## See also * [Publish-subscribe](/learn/core-nats/publish-subscribe.md) — the interest graph and at-most-once delivery these tools inspect. * [Monitoring → monitoring endpoints](/learn/monitoring/monitoring-endpoints.md) — `/connz`, `/varz`, and the rest of the monitoring port in depth. * [Request-reply](/learn/core-nats/request-reply.md) — the inboxes a wire tap sees and the no-responders signal a request gives you. --- # Message headers Every message so far has been a subject and a payload. NATS lets you attach a third thing: **headers**, key/value metadata that rides with the message alongside the payload. The payload is the business data. A header is anything you want the receiver to know about that data without reading it out of the body: who sent it, what format it's in, or an ID to trace it by. You've already seen a header without naming it. When a request finds [no responders](/learn/core-nats/request-reply.md#no-responders), the server answers with a header and no payload. This page shows the whole mechanism behind that signal. ## The header format A header block is text, and it looks like HTTP. It opens with a version line, `NATS/1.0`, followed by one or more `Key: Value` pairs, each on its own line: ``` NATS/1.0 Content-Type: application/json Acme-Request-Id: req_7f3c9a ``` Each pair is one key and one value. A key can appear on more than one line to carry several values, and the receiver reads back all of them. So a header is really a key mapped to a list of values, even though most keys hold just one. NATS keeps the case of your keys exactly as you write them and treats them as case-sensitive, so `Content-Type` and `content-type` are two different keys. Many HTTP libraries fold keys to a single canonical case; NATS doesn't, so pick one spelling for a key and use it on both ends. The header block travels as part of the message but separately from the payload, so the body a receiver reads is exactly the bytes you published. A service that ignores headers sees the same JSON it always did. ## Setting and reading headers From the CLI, `-H` (or `--header`) attaches a header as a `Key:Value` pair, and you repeat the flag for each one you want. `nats pub` takes it: #### CLI ``` #!/bin/bash # Publish an orders.created message with two headers attached. -H (or # --header) takes a Key:Value pair; repeat it to add more than one header. # The headers travel alongside the JSON body, they are not part of it. # # Content-Type tells receivers how to read the body # Acme-Request-Id a value your own code sets, here the upstream request # that produced this order nats pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ -H 'Content-Type:application/json' \ -H 'Acme-Request-Id:req_7f3c9a' ``` #### C ``` // Build an orders.created message with two headers attached. The // headers travel alongside the JSON body, they are not part of it: // Content-Type tells receivers how to read the body // Acme-Request-Id a value your own code sets, here the upstream // request that produced this order const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; if (s == NATS_OK) s = natsMsg_Create(&msg, "orders.created", NULL, order, (int) strlen(order)); if (s == NATS_OK) s = natsMsgHeader_Set(msg, "Content-Type", "application/json"); if (s == NATS_OK) s = natsMsgHeader_Set(msg, "Acme-Request-Id", "req_7f3c9a"); if (s == NATS_OK) s = natsConnection_PublishMsg(conn, msg); ``` On the receiving side, `nats sub` prints each header as a `Key: Value` line above the body, so you can see exactly what the publisher attached: #### CLI ``` #!/bin/bash # Subscribe to orders.created and watch the headers arrive. nats sub prints # each header as a Key: Value line, then a blank line, then the body. Run the # publish snippet in another terminal to produce output like this (the two # header lines can print in either order): # # [#1] Received on "orders.created" # Content-Type: application/json # Acme-Request-Id: req_7f3c9a # # {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} # # Add --headers-only to print just the headers and skip the body, handy for # inspecting metadata on a busy subject. nats sub orders.created ``` #### C ``` // Print each message's headers as Key: Value lines, then a blank line, // then the body -- so you can see exactly what the publisher attached. static void onMsg(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { const char* *keys = NULL; int nkeys = 0; int i; if (natsMsgHeader_Keys(msg, &keys, &nkeys) == NATS_OK) { for (i = 0; i < nkeys; i++) { const char *val = NULL; if (natsMsgHeader_Get(msg, keys[i], &val) == NATS_OK) printf("%s: %s\n", keys[i], val); } free((void*) keys); } printf("\n%.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); natsMsg_Destroy(msg); } // Subscribe to orders.created and watch the headers arrive. Run the // headers publish snippet in another terminal to produce output. if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "orders.created", onMsg, NULL); ``` Client libraries expose the same thing as a header type on the message. You build it with `Set` to give a key a value or `Add` to append another value to a key, and read it back with `Get` for the first value of a key or `Values` for the whole list. `Content-Type` in the example is the one header nearly everyone uses: it tells receivers how to read the body, `application/json` here. The rest of the keys are yours to name. Keep the `Nats-` prefix out of your own keys, though, since NATS reserves it for system headers; some of those drive [JetStream](/learn/jetstream/.md) and are out of scope for core NATS. ## A request ID and a trace ID Request-reply is a natural place for headers. When the warehouse asks the inventory service [whether an item is in stock](/learn/core-nats/request-reply.md), the question and the answer are two messages, and headers let you tag them without touching the order payload. Attach an `Acme-Request-Id` to name this one call, and an `Acme-Trace-Id`, a **correlation ID**: a value every message in the same logical operation carries, so you can follow one order across services. The responder reads the trace ID off the request and copies it onto the reply: #### CLI ``` #!/bin/bash # Carry a trace id through a request and watch it come back on the reply. # # Terminal 1 - an echo responder on orders.inventory.check. --echo reflects # each request straight back and copies its headers onto the reply, so the # trace id you send returns to you. A real inventory service would instead # copy just the trace id onto its own {"in_stock":...} answer. # nats reply orders.inventory.check --echo # # Terminal 2 - send the check with a trace id and a request id attached. # nats request prints the reply's headers, so you will see Acme-Trace-Id come # back (alongside a NATS-Reply-Counter the echo responder adds). nats request orders.inventory.check \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ -H 'Acme-Trace-Id:trace_5e21' \ -H 'Acme-Request-Id:req_7f3c9a' \ --timeout 2s ``` #### C ``` // Carry a trace id through a request. Run a responder in another // terminal first, e.g.: nats reply orders.inventory.check --echo // (--echo reflects the request back and copies its headers onto the // reply, so the trace id you send returns to you). const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; if (s == NATS_OK) s = natsMsg_Create(&req, "orders.inventory.check", NULL, order, (int) strlen(order)); if (s == NATS_OK) s = natsMsgHeader_Set(req, "Acme-Trace-Id", "trace_5e21"); if (s == NATS_OK) s = natsMsgHeader_Set(req, "Acme-Request-Id", "req_7f3c9a"); // Send the check and wait up to 2 seconds for the reply. if (s == NATS_OK) s = natsConnection_RequestMsg(&reply, conn, req, 2000); // The trace id comes back on the reply's headers, so you can match // this answer to the operation that asked the question. if (s == NATS_OK) { const char *traceID = NULL; if (natsMsgHeader_Get(reply, "Acme-Trace-Id", &traceID) == NATS_OK) printf("Acme-Trace-Id: %s\n", traceID); printf("%.*s\n", natsMsg_GetDataLength(reply), natsMsg_GetData(reply)); } ``` Because the reply comes back carrying the trace ID you sent, you can match this answer to the order you asked about, even with many checks in flight at once. Your payload stayed a clean order document the whole way. ## Status codes from the server The server uses headers as well. It signals a client in-band with a **status code** placed right on the version line. Instead of `NATS/1.0` alone, the line reads `NATS/1.0 503`, and the message carries no payload. You've met exactly this. The [no responders](/learn/core-nats/request-reply.md#no-responders) signal is a reply with the header line `NATS/1.0 503` and an empty body; your client recognizes the code and turns it into a distinct no-responders error. That's the whole mechanism: a status on the header line and no payload, read by the client before your code ever sees a message. ## Header support is negotiated at connect Headers work only when both ends agree to them, and they settle that when the connection opens. The server advertises header support in the INFO line it sends first, a `headers` field that's on by default. The client reads that and declares its own support in the CONNECT it sends back. By the time you publish anything, both sides know headers are in play. Every current server and client supports headers, so this negotiation succeeds quietly and you never think about it. It has one visible failure mode: a server with `no_header_support` set in its configuration, or one old enough to predate the feature. Against such a server the client sees no header support advertised, and any publish that carries a header fails on the client with `headers not supported by this server`, before the message reaches the wire. The [Connecting](/learn/core-nats/connecting.md) page covers the INFO and CONNECT handshake. The same agreement gates the previous section: because the no-responders signal arrives as a header, a connection without header support can't receive it either. ## Try it Open two terminals against your running server. ``` # Terminal 1 — watch orders.created with headers rendered nats sub orders.created ``` ``` # Terminal 2 — publish an order tagged with a content type and a request ID nats pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ -H 'Content-Type:application/json' \ -H 'Acme-Request-Id:req_7f3c9a' ``` Terminal 1 prints the headers above the body (the two header lines can print in either order): ``` [#1] Received on "orders.created" Content-Type: application/json Acme-Request-Id: req_7f3c9a {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} ``` Now add `--headers-only` to the subscriber and publish again. You get the header lines and no body, which is how you inspect metadata on a busy subject without printing every payload. ## Pitfalls **Reading a key with the wrong case.** Header keys are case-sensitive. If a publisher sends `Content-Type` and you read `content-type`, you get an empty value and no error. Match the exact spelling the sender used, and settle on one convention across your services. **Headers count against the payload limit.** The server checks the combined size of the header block and the body against `max_payload`, the same 1 MB ceiling the [payload faces on its own](/learn/core-nats/publish-subscribe.md#the-1-mb-payload-limit). A large header block leaves less room for the body, and if the two together cross the limit the server rejects the publish. **Putting real data in headers.** Headers are for small metadata that describes the message: an ID, a content type, a flag. Large or structured data belongs in the payload, where receivers expect it. Data stuffed into headers inflates every message and competes with the body for `max_payload`. Keep headers short and keep the body authoritative. ## Where you are The Acme world now tags its messages with metadata that travels beside the payload: * A publisher attaches headers with `-H` (or a client's `Set`/`Add`); a subscriber reads them with `nats sub` (or `Get`/`Values`). * A request carries a request ID and a trace ID, and the responder copies the trace ID onto the reply so you can correlate the two. * The server uses headers in-band as well, which is how the `NATS/1.0 503` no-responders signal reaches you. * Header support is agreed at connect, and every current server and client has it. ## What's next Headers travel with a message to wherever its subject leads. Sometimes you need to change where that is: rewrite a subject as it enters the server, so `orders.created` can also arrive under another name without the publisher knowing. That's subject mapping, on the [next page](/learn/core-nats/subject-mapping.md). ## See also * [Request-reply → No responders](/learn/core-nats/request-reply.md#no-responders) — the status header this page generalizes. * [Connecting](/learn/core-nats/connecting.md) — the INFO and CONNECT handshake that negotiates header support. * [Reference → Client protocol](/reference/protocols/client.md) — the wire-level `HPUB`/`HMSG` frames and the exact header format. --- # Publish-subscribe Core NATS has one fundamental operation: a client publishes a message to a subject, and every client subscribed to that subject right now gets a copy. This page shows how that behaves on the wire: where the copies come from, what happens when nobody is listening, and what core NATS promises about delivery. ## The scenario Acme runs an order platform. Three things happen to an order, and each one is a message on its own subject: ``` orders.created orders.shipped orders.canceled ``` Every message in this chapter carries the same small JSON payload: ``` { "order_id": "ord_8w2k", "customer": "acme-co", "total_cents": 4200, "ts": "2026-05-22T10:14:22Z" } ``` Three services care about these messages. The `warehouse` service packs the box when an order is created. The `notifications` service emails the customer. The `analytics` service counts everything. None of them knows the others exist. [Connecting](/learn/core-nats/connecting.md) already started one local `nats-server`. Leave it running for the rest of this chapter. If you landed here directly, start one now: ``` nats-server ``` That's the whole deployment, with no flags, no persistence, and no cluster. Add services to it as the chapter grows. ## Publishing a message A **publisher** is a client that sends a message to a subject. The warehouse doesn't subscribe to anything yet, so start by publishing one `orders.created` message: #### CLI ``` #!/bin/bash # Publish one order to the orders.created subject. The publish is # fire-and-forget: nats pub hands the message to the server and exits. # It does not wait for, or report, any subscriber. nats pub orders.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` #### JavaScript/TypeScript ``` // Publish one order to the orders.created subject. Publishing is // fire-and-forget: the call hands the message to the server and returns. const order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'; nc.publish("orders.created", order); ``` #### Go ``` // Publish one order to the orders.created subject. Publishing is // fire-and-forget: the call hands the message to the server and returns. order := `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}` nc.Publish("orders.created", []byte(order)) ``` #### Python ``` # Publish one order to the orders.created subject. Publishing is # fire-and-forget: the call hands the message to the server and returns. order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' await client.publish("orders.created", order.encode()) ``` #### Java ``` // Publish one order to the orders.created subject. Publishing is // fire-and-forget: the call hands the message to the server and returns. byte[] order = ("{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," + "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}").getBytes(StandardCharsets.UTF_8); nc.publish("orders.created", order); ``` #### Rust ``` // Publish one order to the orders.created subject. Publishing is // fire-and-forget: the call hands the message to the server and returns. let order = r#"{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}"#; client.publish("orders.created", order.into()).await?; ``` #### C#/.NET ``` // Publish one order to the orders.created subject. Publishing is // fire-and-forget: the call hands the message to the server and returns. // The client serializes the Order record to JSON by default. var order = new Order( OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:22Z", CultureInfo.InvariantCulture)); await client.PublishAsync("orders.created", order); ``` #### C ``` // Publish one order to the orders.created subject. Publishing is // fire-and-forget: the call hands the message to the client and // returns. It does not wait for, or report, any subscriber. if (s == NATS_OK) s = natsConnection_PublishString(conn, "orders.created", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"); ``` The publish call returns immediately. It doesn't wait for a subscriber, and it doesn't tell you how many subscribers received the message, or whether any did. This is **fire-and-forget**: the publisher hands the message to the server and moves on. A publisher always names a fully-qualified subject. `orders.created` is a concrete subject, not a pattern. Wildcards belong to subscribers, and we cover them on the [next page](/learn/core-nats/subjects-and-wildcards.md). ## Subscribing to a subject A **subscriber** is a client that registers interest in a subject and receives a copy of every matching message. Start the warehouse service as a subscriber on `orders.created`: #### CLI ``` #!/bin/bash # Subscribe as the warehouse service to orders.created. Each matching # message is printed as it arrives; Ctrl-C to stop. To receive every # order subject at once instead, subscribe to the wildcard orders.> nats sub orders.created ``` #### JavaScript/TypeScript ``` // Subscribe as the warehouse service to orders.created. Each matching // message is delivered to this subscription as it is published. const sub = nc.subscribe("orders.created"); for await (const msg of sub) { console.log(`warehouse received: ${msg.string()}`); } ``` #### Go ``` // Subscribe as the warehouse service to orders.created. The callback runs // for each matching message as it is published. nc.Subscribe("orders.created", func(m *nats.Msg) { fmt.Printf("warehouse received: %s\n", string(m.Data)) }) ``` #### Python ``` # Subscribe as the warehouse service to orders.created. Each matching # message is delivered to this subscription as it is published. async with await client.subscribe("orders.created") as subscription: async for message in subscription: print(f"warehouse received: {message.data.decode()}") ``` #### Java ``` // Subscribe as the warehouse service to orders.created. Each matching // message is delivered to this dispatcher as it is published. nc.createDispatcher(msg -> System.out.println("warehouse received: " + new String(msg.getData(), StandardCharsets.UTF_8)) ).subscribe("orders.created"); ``` #### Rust ``` // Subscribe as the warehouse service to orders.created. Each matching // message is delivered to this subscription as it is published. let mut sub = client.subscribe("orders.created").await?; while let Some(msg) = sub.next().await { println!( "warehouse received: {}", String::from_utf8_lossy(&msg.payload) ); } ``` #### C#/.NET ``` // Subscribe as the warehouse service to orders.created. Each matching // message is delivered to this subscription as it is published and // deserialized from JSON into an Order record. await foreach (var msg in client.SubscribeAsync("orders.created")) { output.WriteLine($"warehouse received: {msg.Data}"); } ``` #### C ``` // The callback runs for each matching message as it is published. static void onMsg(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("warehouse received: %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); natsMsg_Destroy(msg); } // Subscribe as the warehouse service to orders.created. if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "orders.created", onMsg, NULL); ``` Now publish again with the snippet above. The warehouse receives the message. Run a second subscriber for `notifications` and a third for `analytics`, each on `orders.created`, and every one of them receives its own copy of the next publish. Each subscriber gets an independent copy; subscribing isn't taking from a queue, and one subscriber receiving a message takes nothing from another. If you want one subscriber to see *all* order subjects at once, it can subscribe to `orders.>` instead of a single subject. That `>` is a wildcard, and the [next page](/learn/core-nats/subjects-and-wildcards.md) is where we explain it. ## Unsubscribing When a subscriber unsubscribes, the server removes its interest in that subject as soon as it processes the request. Any message published after that doesn't reach it, and there's nothing to clean up on the server side. You can also have a subscription end itself after a fixed number of messages, without tracking the count yourself. This is **auto-unsubscribe**: a client calls its auto-unsubscribe method (`AutoUnsubscribe(N)` in Go), which tells the server to end the subscription once it has delivered N messages. The CLI's `nats sub --count N` gets the same result by counting messages itself and unsubscribing when it hits the limit. It fits a take-exactly-N flow, such as reading the next three orders and stopping. #### CLI ``` #!/bin/bash # Subscribe to orders.created and stop automatically after three messages. # --count makes nats sub quit once it has received that many, giving a # take-exactly-N read instead of an open-ended subscription you Ctrl-C. nats sub orders.created --count 3 ``` #### C ``` // Subscribe to orders.created and stop automatically after three // messages. AutoUnsubscribe tells the client to remove the // subscription once that many have been delivered, giving a // take-exactly-N read instead of an open-ended subscription. if (s == NATS_OK) s = natsConnection_SubscribeSync(&sub, conn, "orders.created"); if (s == NATS_OK) s = natsSubscription_AutoUnsubscribe(sub, 3); // Read the three orders, waiting up to 10 seconds for each. if (s == NATS_OK) { int i; for (i = 0; (s == NATS_OK) && (i < 3); i++) { natsMsg *msg = NULL; s = natsSubscription_NextMsg(&msg, sub, 10000); if (s == NATS_OK) { printf("warehouse received: %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); natsMsg_Destroy(msg); } } } ``` ## The interest graph The server tracks who's subscribed to what in an in-memory structure called the **interest graph**. Each subscription adds an entry; each unsubscribe or disconnect removes it. **Message flow — Publish / Subscribe (animated):** Animated publish/subscribe: a publisher emits messages; NATS delivers a copy to every matching subscriber. * Publisher → NATS (subject: updates) * NATS → Subscriber 1 (subject: updates) * NATS → Subscriber 2 (subject: updates) When a message arrives, the server looks up the subject in the interest graph, finds the matching subscribers, and sends each one a copy. With three services subscribed to `orders.created`, one publish produces three deliveries. The publisher did nothing different; the fan-out happened entirely on the server. The interest graph is the source of the decoupling. The publisher holds no list of subscribers. It publishes to a subject, and the server resolves interest at the moment the message arrives. This is also why services can come and go freely. Start a fourth subscriber and it joins the graph; the next publish reaches it. Stop one and its entry is gone; the next publish skips it. Nobody coordinates, and the publisher never changes. ## When nobody is listening Publish to `orders.created` while no service is subscribed. The publish still succeeds, and the message is dropped. The server finds no matching entry in the interest graph, so there's nothing to deliver to, and the message is discarded. A publish with no interest is a silent no-op: no error, no stored backlog. The publisher can't tell "delivered to three subscribers" from "delivered to nobody" — both look like a successful publish. If the warehouse is restarting when an `orders.created` message is published, that message is gone. ## At-most-once delivery Core NATS delivers each message **at-most-once**. A subscriber that's connected and interested when the message is published gets it once. A subscriber that's absent, slow, or disconnected at that instant gets it zero times. There's no second attempt. Core NATS doesn't retry a missed message, doesn't detect or suppress duplicates, and doesn't guarantee that two subscribers see messages in the same order under load. Each is a property you add with [JetStream](/learn/jetstream/.md), not something core provides. At-most-once is the right guarantee for a large class of messages: telemetry you sample, cache invalidations, a live dashboard feed. For those, a missed message costs nothing because another is already on the way. For an order that must be packed exactly once, it isn't enough. That's the boundary of core NATS. ## The 1 MB payload limit A message payload has a maximum size. By default the server caps it at **1 MB** (`max_payload`, `1048576` bytes). The server announces this limit to every client when the connection opens, so the client knows the ceiling before it ever publishes. Because the client knows the ceiling, it checks the payload before sending. An official client fails the publish call with `nats: maximum payload exceeded` and leaves the connection up, so the message never leaves the client. The server is the backstop for a client that sends an oversized message anyway: it replies `-ERR 'Maximum Payload Violation'` and closes the connection. The Acme order payload is a few hundred bytes, well under the ceiling, but a service that tries to ship a large blob inside a message will hit it. For large data, publish a reference (an object-store key or a URL) and let the receiver fetch the bytes out of band. ## Try it in two terminals Observe fire-and-forget and at-most-once directly. Open two terminals against the running server. ``` # Terminal 1 — the warehouse subscribes nats sub orders.created ``` ``` # Terminal 2 — publish three orders nats pub orders.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' nats pub orders.created '{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}' nats pub orders.created '{"order_id":"ord_5kq1","customer":"initech","total_cents":1500,"ts":"2026-05-22T10:14:29Z"}' ``` Terminal 1 prints each message the instant it's published. Now stop the subscriber in Terminal 1 with Ctrl-C, publish a fourth message, and restart the subscriber. The fourth message never appears. It was published into an empty interest graph and discarded, which is at-most-once delivery in action. ## Pitfalls **Publishing over the limit fails the publish.** An official client checks the payload against `max_payload` and fails the call with `nats: maximum payload exceeded` before anything goes out; the connection stays up. Keep payloads under the limit and pass a reference for anything large. #### CLI ``` #!/bin/bash # Ask your connection for its limits before sizing a message. The # "Maximum Payload" row comes from the INFO the server sends at connect # (1 MB by default), so a plain no-auth connection can read it. An # official client checks this ceiling and fails an oversized publish # locally; the server rejects and closes the connection of any client # that sends a larger PUB anyway. nats account info # A safe order publish stays far under the ceiling. nats pub orders.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # Publishing over the ceiling fails immediately, client-side, before the # message reaches the server. Try a 2 MB payload: # # head -c 2000000 /dev/zero | tr '\0' x | nats pub orders.created --force-stdin # nats: error: nats: maximum payload exceeded # # So keep payloads small and pass a reference (an object-store key or a URL) # for anything large, rather than discovering the limit the hard way. ``` #### C ``` // Ask the connection for its limit before sizing a message. The value // comes from the INFO the server sent at connect (1 MB by default). if (s == NATS_OK) printf("maximum payload: %d bytes\n", (int) natsConnection_GetMaxPayload(conn)); // A safe order publish stays far under the ceiling. if (s == NATS_OK) s = natsConnection_PublishString(conn, "orders.created", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"); // Publishing over the ceiling fails immediately, client-side, before // the message reaches the server. Try a 2 MB payload: if (s == NATS_OK) { int bigLen = 2 * 1024 * 1024; char *big = malloc(bigLen); memset(big, 'x', bigLen); s = natsConnection_Publish(conn, "orders.created", big, bigLen); if (s == NATS_MAX_PAYLOAD) { printf("oversized publish rejected: %s\n", natsStatus_GetText(s)); s = NATS_OK; } free(big); } ``` **Exiting before the publish leaves the client drops it.** The publish call returns immediately because the client buffers the message and sends it in the background. A short-lived publisher that exits right after the call can quit before that buffered message reaches the server, and the message is gone. Flush (or drain) before you exit. The client waits for a round trip to the server, so everything buffered before it has arrived: ``` nc.Publish("orders.created", payload) nc.Flush() // wait until the server has the message, then exit nc.Close() ``` **A connection receives the messages it publishes.** When one connection both publishes and subscribes to the same subject, it gets a copy of its own messages, because echo is on by default for every connection. A service that publishes status updates and also subscribes to them can process its own updates in a loop. [Connecting](/learn/core-nats/connecting.md) covers turning it off. **A slow subscriber gets cut off.** A subscriber that can't keep up with the rate of matching messages builds a backlog on the server. Past a threshold the server stops protecting it, logs `Slow Consumer Detected`, and closes its connection. The other subscribers are unaffected. The fix lives in the client: process messages fast enough, or hand them to a worker. [Resilient clients → Slow consumers](/learn/resilient-clients/slow-consumers.md) covers the tuning and recovery. ## Where you are You have one local `nats-server` running, and the Acme order services talking over core publish-subscribe: * A publisher sends `orders.created` messages fire-and-forget. * The `warehouse`, `notifications`, and `analytics` subscribers each receive their own copy. * A message published with no interest is discarded, and delivery is at-most-once. ## What's next Right now every service subscribes to one exact subject. The next page, [Subjects & wildcards](/learn/core-nats/subjects-and-wildcards.md), shows how subjects form a hierarchy and how a subscriber uses `*` and `>` to match many subjects at once, including regional orders like `orders.us.created`. ## See also * [Core Concepts → Publish-subscribe](/concepts/pub-sub-basics.md) — the five-minute overview of this pattern. * [Learn → JetStream → Why a stream](/learn/jetstream/your-first-stream.md#why-a-stream) — the layer that keeps the messages core NATS discards. * [Reference → Client protocol](/reference/protocols/client.md) — the wire-level `PUB`/`SUB`/`MSG` details. --- # Queue groups So far every subscriber on a subject gets a copy of every message. That's what you want for `notifications` and `analytics`: each of them needs to see every order. It's not what you want for `warehouse`. The warehouse does real work for each order: it prints a label and packs the box. One process can't keep up with a busy day. You want a pool of packers, and you want each order packed by exactly one of them, not zero and not two. Plain pub/sub can't do that. Run three copies of the `warehouse` subscriber and all three pack the same order. This page introduces the NATS answer: the queue group. ## What a queue group is A **queue group** is a set of subscribers on the same subject that share a name. The server treats the whole group as one logical subscriber: for each message, it picks exactly one member of the group and delivers to that member alone. The shared name is the **queue group name**. A subscriber joins a group by subscribing to a subject *and* naming a group at the same time. Any subscriber that names the same group, on the same subject, is in the same group. There's nothing to configure on the server. Your application picks the queue group name, and it arrives with the subscription. The server learns a group exists the moment its first member subscribes. **Message flow — Queue group (animated):** Animated queue group: a publisher emits messages; NATS load-balances each message to exactly one worker in the queue group. * Publisher → Worker 1 * Publisher → Worker 2 * Publisher → Worker 3 ## Add the packers pool Our running scenario carries over one `nats-server` from earlier pages, with the `notifications` and `analytics` subscribers still running. Now replace the single `warehouse` subscriber with a pool. Each packer subscribes to `orders.created` and names the queue group `packers`. The `--queue` flag is what turns a plain subscription into a queue-group membership: #### CLI ``` #!/bin/bash # Subscribe to orders.created as a member of the "packers" queue group. # The --queue flag names the group; every subscriber that names the same # group on the same subject shares the load. Each published order is # delivered to exactly one member of the group, chosen at random. # # Run this command in several terminals to watch the load balance: each # order lands in only one of the running copies. nats sub orders.created --queue packers ``` #### JavaScript/TypeScript ``` // Join the "packers" queue group on orders.created. Every subscriber that // names the same group shares the load: each order is delivered to exactly // one member. Run this in several processes to watch the load balance. const sub = nc.subscribe("orders.created", { queue: "packers" }); for await (const msg of sub) { console.log(`packer handling: ${msg.string()}`); } ``` #### Go ``` // Join the "packers" queue group on orders.created. Every subscriber that // names the same group shares the load: each order is delivered to exactly // one member. Run this in several processes to watch the load balance. nc.QueueSubscribe("orders.created", "packers", func(m *nats.Msg) { fmt.Printf("packer handling: %s\n", string(m.Data)) }) ``` #### Python ``` # Join the "packers" queue group on orders.created. Every subscriber that # names the same group shares the load: each order is delivered to exactly # one member. Run this in several processes to watch the load balance. async with await client.subscribe("orders.created", queue="packers") as subscription: async for message in subscription: print(f"packer handling: {message.data.decode()}") ``` #### Java ``` // Join the "packers" queue group on orders.created. Every subscriber that // names the same group shares the load: each order is delivered to exactly // one member. Run this in several processes to watch the load balance. nc.createDispatcher(msg -> System.out.println("packer handling: " + new String(msg.getData(), StandardCharsets.UTF_8)) ).subscribe("orders.created", "packers"); ``` #### Rust ``` // Join the "packers" queue group on orders.created. Every subscriber that // names the same group shares the load: each order is delivered to exactly // one member. Run this in several processes to watch the load balance. let mut sub = client .queue_subscribe("orders.created", "packers".to_string()) .await?; while let Some(msg) = sub.next().await { println!("packer handling: {}", String::from_utf8_lossy(&msg.payload)); } ``` #### C#/.NET ``` // Join the "packers" queue group on orders.created. Every subscriber that // names the same group shares the load: each order is delivered to exactly // one member. Run this in several processes to watch the load balance. await foreach (var msg in client.SubscribeAsync("orders.created", queueGroup: "packers")) { output.WriteLine($"packer handling: {msg.Data}"); } ``` #### C ``` // Join the "packers" queue group on orders.created. Every subscriber that // names the same group shares the load: each order is delivered to exactly // one member. Run this in several processes to watch the load balance. static void onOrder(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("packer handling: %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); natsMsg_Destroy(msg); } if (s == NATS_OK) s = natsConnection_QueueSubscribe(&sub, conn, "orders.created", "packers", onOrder, NULL); ``` The queue group name here is `packers`. It's a plain string the application chooses. Pick a name that says what the group does; `packers` reads better in logs than `q1`. ## See the load balancing Open three terminals and run the same `nats sub` in each, so you have three packers in the `packers` group: ``` # Terminal 1, 2, and 3 — three members of the packers group nats sub orders.created --queue packers ``` In a fourth terminal, publish a few orders: ``` nats pub orders.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' nats pub orders.created '{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}' nats pub orders.created '{"order_id":"ord_4kp1","customer":"acme-co","total_cents":1500,"ts":"2026-05-22T10:14:29Z"}' ``` Each message lands in exactly one of the three terminals. Publish a dozen more and they spread across the three packers. No packer sees a message that another packer already took. ## How the server picks a member The server keeps the live members of a group in a list. For each message, it picks a random index into that list and delivers to that member. On a single server the selection is uniform-random across the available members; a cluster adds a locality preference, covered below. Random selection has one consequence: the server doesn't rotate fairly through the members the way round-robin would. The same packer can be chosen twice in a row, and over a handful of messages the split can look lopsided. Over many messages it evens out. ## Membership is dynamic A packer joins the group by subscribing and leaves by unsubscribing or disconnecting. Both happen with no configuration and no coordination step. Start a fourth packer while orders are flowing. It starts receiving its share immediately, because the server includes it in the next random pick. Stop a packer and the server drops it from the list; the next message goes to one of the survivors. This is what makes a queue group a good fit for an autoscaler. A pool that grows to ten packers under load and shrinks to two overnight needs no broker reconfiguration. Each instance subscribes on start and the group resizes itself. One limit belongs here: core NATS is at-most-once, so if the server picks a packer and it dies *after* delivery, that message is gone — the server won't retry it with another member. Work that must survive a worker crash belongs in a durable work queue in [JetStream](/learn/jetstream/.md). ## Queue members and plain subscribers coexist A queue group and a plain subscriber can share the same subject without interfering. `analytics` subscribes to `orders.created` with no queue group; the three packers subscribe in the `packers` group. For each published order: * `analytics` receives it: plain subscribers always get every message. * exactly one packer receives it: the group gets one copy, shared. The server runs the two distributions independently, so the same subject carries both behaviors with no extra configuration — analytics counts every order while packing happens once per order. You can confirm it. Keep the three `packers` terminals open, and in another terminal subscribe plain: ``` # A plain subscriber alongside the packers group nats sub orders.created ``` Publish an order. The plain terminal prints it, and exactly one of the three packers prints it too. ## A group only shares within one subject Membership is evaluated *after* subject matching. The server first finds the subscriptions whose subject matches the message, then, among those, applies the group pick. The name alone doesn't pull in members on a different subject. Two packers in a group named `packers`, one subscribed to `orders.created` and one to `orders.shipped`, don't share load. They match different subjects, so a message to `orders.created` is only ever considered for the first one. The shared name does nothing across different subjects. A queue group can subscribe with a wildcard, and then it load-balances across everything that wildcard matches. A `packers` group on `orders.*.created` (the regional subjects from the [subjects page](/learn/core-nats/subjects-and-wildcards.md)) would spread `orders.us.created` and `orders.eu.created` across its members. The group shares load across the subjects its own subscription matches, and only those. ## A note on placement across regions When the same queue group has members in several clusters, the server prefers a member in the publisher's own cluster before reaching across to another region. That keeps work local and cuts cross-region traffic. That behavior, geo-affinity for queue groups, belongs to multi-cluster deployments, which this chapter doesn't set up. Our scenario is a single local server, so every packer is equally local. See [Topologies → Super-clusters](/learn/topologies/super-clusters.md) for how it works once you span regions. ## Pitfalls **A typo in the queue group name makes a second group.** The server matches members by the exact name string, so `packers` and `packer` are two separate groups on the same subject. Both subscriptions succeed with no warning, and each published order goes to one member of *each* group: the work is double-handled instead of load-balanced. Give every member the byte-for-byte identical name. #### CLI ``` #!/bin/bash # A typo in the queue group name silently creates a SECOND group. # The server matches members by the exact name string, so "packers" and # "packer" are two different groups on the same subject. There is no # warning -- both subscriptions succeed. # # To see it, run these two subscribers in separate terminals. Note the # mismatched names: one says "packers", the other "packer". nats sub orders.created --queue packers # terminal 1: group "packers" nats sub orders.created --queue packer # terminal 2: group "packer" (typo) # Now publish the same test order three times from a third terminal: # nats pub orders.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' --count 3 # # Each order is delivered to ONE member of EACH group, so BOTH terminals # print every order -- the work is double-handled, not load-balanced. # Fix: give every member the byte-for-byte identical group name. ``` #### C ``` // A typo in the queue group name silently creates a SECOND group. The // server matches members by the exact name string, so "packers" and // "packer" are two different groups on the same subject. Both // subscriptions succeed with no warning. static void onOrder(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("group %s handling: %.*s\n", (const char *) closure, natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); natsMsg_Destroy(msg); } if (s == NATS_OK) s = natsConnection_QueueSubscribe(&sub1, conn, "orders.created", "packers", onOrder, (void *) "packers"); if (s == NATS_OK) s = natsConnection_QueueSubscribe(&sub2, conn, "orders.created", "packer", onOrder, (void *) "packer"); // typo! // Publish an order and BOTH callbacks print it: one member of EACH // group receives it, so the work is double-handled instead of // load-balanced. Fix: give every member the byte-for-byte identical // group name. ``` **Don't expect ordering or an even split.** Selection is random per message, not round-robin, so a short burst can look lopsided. If work must be strictly ordered for a customer, keep it on a single subscriber, not a queue group. **Make a packer's work safe to repeat.** A packer that's slow or briefly cut off can still be doing work the publisher assumes was lost, and core NATS won't send it again. Write each packer so handling the same order twice is harmless — pack by `order_id`, skip one already packed. ## Where you are Your running session now looks like this: * One local `nats-server`, carried from earlier pages. * A `packers` queue group on `orders.created`, load-balancing each order to exactly one packer, with members you can add and remove live. * `analytics` still a plain subscriber on `orders.created`, seeing every order, undisturbed by the group. You can do load-balanced *work* now, and the same mechanics apply to request-reply: a queue group of responders on one subject answers each request exactly once. The next pattern is the opposite case — one request where you *want* every responder to answer, and you gather all the replies. ## What's next The next page is [Scatter-gather](/learn/core-nats/scatter-gather.md): one request, many responders, and gathering all the replies instead of taking the first. You'll query three `shipping.quote` providers and pick the best. ## See also * [Core Concepts → Queue Groups](/concepts/queue-groups.md) — the five-minute overview of the same material. * [JetStream → Learn](/learn/jetstream/.md) — durable work queues, where a message survives a worker crash and is retried to another member. * [Topologies → Super-clusters](/learn/topologies/super-clusters.md) — geo-affinity that keeps queue-group work in the publisher's region. --- # Request-reply Pub/sub is one-way. A publisher publishes to `orders.created`, and a copy goes to every interested subscriber. The publisher never hears back. Acme needs the other direction too. When an order arrives, the warehouse wants to ask one question and get one answer: *is this item in stock?* That's a request and a reply, not a broadcast. This page builds an **inventory** service that answers that question on the subject `orders.inventory.check`. Along the way it shows the two things that make request-reply work: the private reply subject the client sets up for itself, and what happens when nobody is there to answer. ## How request-reply uses pub/sub Request-reply isn't a new protocol; it's the pub/sub you already know, used twice. Here are all the steps. The client invents a fresh, unique subject to receive the answer on. It subscribes to that subject. It then publishes the request, and includes that reply subject as a field on the message. The responder reads the request, sees the reply subject, and publishes its answer there. The client's subscription receives it. **Message flow — Request / Reply:** Request-reply pattern where a client sends a request and waits for a response from a service * Client → NATS (subject: get.user.150) * NATS → Service (subject: get.user.150) * Service → NATS (subject: \_INBOX.\) * NATS → Client (subject: \_INBOX.\) The orange arrow is the request traveling out on the request subject. The green dashed arrow is the reply traveling back on the private subject the client made for this one call. Every NATS client wraps those steps in a single `request()` call, so you never write the subscribe-publish-wait by hand. ## The inbox The private reply subject is called an **inbox**, and clients generate it under the reserved `_INBOX.` prefix: something like `_INBOX.nQ4k2v8...` with a random unique tail. The inbox is per-request: each `request()` uses a fresh inbox, so two in-flight requests never get each other's replies. The `_INBOX.` prefix is reserved, so it never collides with the subjects you pick yourself; the client owns that namespace. A client doesn't open a subscription for every `request()`. On its first request it subscribes once to a wildcard that stays fixed for the connection — a subject shaped like `_INBOX..*` — and reuses that one subscription for every request after. Each request adds only its own final token to that prefix, so the per-request fresh inbox amounts to that fresh token, and the client routes each reply back to the request waiting on its token. That's why thousands of concurrent requests cost one subscription, not thousands: every reply arrives on the same wildcard, and its final token identifies which request it belongs to. (Clients can be configured to fall back to one subscription per request; the shared one is the default.) There's a size budget on the reply subject. The server limits the length of a single protocol line, subject plus reply subject combined, to 4 KB by default (`max_control_line`). Generated inbox names sit far under that, so this only matters if you hand-build unusually long subjects. The default `_INBOX.` prefix is configurable per connection — a client option, exposed in natscli as the global `--inbox-prefix` flag. It exists for permissions: with a distinct prefix per application, an operator can grant each one its own reply-subject namespace instead of a shared `_INBOX.>`. Subject permissions are covered on the [Authorization](/learn/security/authorization.md) page. ## The inventory service Now make the responder real. The inventory service subscribes to `orders.inventory.check` and replies to each request with an in-stock answer. From the CLI, `nats reply` does exactly this: it subscribes to the subject (joining the default queue group `NATS-RPLY-22`) and publishes a reply to whatever inbox each request carries. #### CLI ``` #!/bin/bash # The inventory service: subscribe to orders.inventory.check and answer # every request. `nats reply` subscribes to the subject and publishes the # given body to whatever inbox each request carries. # # Leave this running in its own terminal. It answers every request with # an in-stock reply until you stop it with Ctrl-C. nats reply orders.inventory.check '{"in_stock":true,"warehouse":"us-east"}' ``` #### JavaScript/TypeScript ``` // The inventory service: subscribe to orders.inventory.check and answer // every request by responding on the reply subject it carries. const sub = nc.subscribe("orders.inventory.check"); for await (const msg of sub) { msg.respond('{"in_stock":true,"warehouse":"us-east"}'); } ``` #### Go ``` // The inventory service: subscribe to orders.inventory.check and answer // every request by responding on the reply subject it carries. nc.Subscribe("orders.inventory.check", func(m *nats.Msg) { m.Respond([]byte(`{"in_stock":true,"warehouse":"us-east"}`)) }) ``` #### Python ``` # The inventory service: subscribe to orders.inventory.check and answer # every request by publishing back to the reply subject each one carries. async with await client.subscribe("orders.inventory.check") as subscription: async for message in subscription: if message.reply: await client.publish(message.reply, b'{"in_stock":true,"warehouse":"us-east"}') ``` #### Java ``` // The inventory service: subscribe to orders.inventory.check and answer // every request by publishing back to the reply subject it carries. nc.createDispatcher(msg -> { if (msg.getReplyTo() != null) { nc.publish(msg.getReplyTo(), "{\"in_stock\":true,\"warehouse\":\"us-east\"}".getBytes(StandardCharsets.UTF_8)); } }).subscribe("orders.inventory.check"); ``` #### Rust ``` // The inventory service: subscribe to orders.inventory.check and answer // every request by publishing back to the reply subject it carries. let mut sub = client.subscribe("orders.inventory.check").await?; while let Some(msg) = sub.next().await { if let Some(reply) = msg.reply { client .publish(reply, r#"{"in_stock":true,"warehouse":"us-east"}"#.into()) .await?; } } ``` #### C#/.NET ``` // The inventory service: subscribe to orders.inventory.check and answer // every request by replying on the subject each one carries. await foreach (var msg in client.SubscribeAsync("orders.inventory.check")) { await msg.ReplyAsync(new InventoryReply(InStock: true, Warehouse: "us-east")); } ``` #### C ``` // The inventory service: subscribe to orders.inventory.check and answer // every request by publishing a reply on the reply subject it carries. static void onCheck(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { if (natsMsg_GetReply(msg) != NULL) natsConnection_PublishString(nc, natsMsg_GetReply(msg), "{\"in_stock\":true,\"warehouse\":\"us-east\"}"); natsMsg_Destroy(msg); } if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "orders.inventory.check", onCheck, NULL); ``` Leave that running. It's now the one service in the Acme world that answers questions instead of just receiving messages. The warehouse, notifications, and analytics subscribers from the earlier pages keep running unchanged, and request-reply runs alongside them. ## Sending a request In a second terminal, ask the question. The warehouse sends the order payload to `orders.inventory.check` and waits for the inventory service to answer. #### CLI ``` #!/bin/bash # Ask the inventory service whether an order's item is in stock. # `nats request` creates a private inbox, subscribes to it, publishes the # order payload to orders.inventory.check with the inbox attached, and # prints the first reply it receives. # # --timeout bounds the wait: it is the longest the client waits for an answer # before giving up. The CLI defaults it to 5s; here we set 2s. If no service is # subscribed, the server sends a "no responders" signal immediately and the CLI # logs "No responders are available" instead of waiting out the timeout. nats request orders.inventory.check \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --timeout 2s ``` #### JavaScript/TypeScript ``` // Ask the inventory service whether an order's item is in stock. The client // creates a private inbox, sends the request, and waits up to the timeout for // one reply. A missing service surfaces immediately as NoRespondersError; a // slow one as TimeoutError. const order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'; try { const reply = await nc.request("orders.inventory.check", order, { timeout: 2000, }); console.log(`inventory replied: ${reply.string()}`); } catch (err) { if (err instanceof RequestError && err.isNoResponders()) { console.log("no inventory service is running"); } else if (err instanceof TimeoutError) { console.log("inventory service did not answer in time"); } else { throw err; } } ``` #### Go ``` // Ask the inventory service whether an order's item is in stock. The client // creates a private inbox, sends the request, and waits up to the timeout // for one reply. A missing service surfaces immediately as // nats.ErrNoResponders; a slow one as nats.ErrTimeout. order := `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}` msg, err := nc.Request("orders.inventory.check", []byte(order), 2*time.Second) switch err { case nats.ErrNoResponders: fmt.Println("no inventory service is running") case nats.ErrTimeout: fmt.Println("inventory service did not answer in time") case nil: fmt.Printf("inventory replied: %s\n", string(msg.Data)) default: fmt.Printf("request failed: %v\n", err) } ``` #### Python ``` # Ask the inventory service whether an order's item is in stock. The client # creates a private inbox, sends the request, and waits up to the timeout # for one reply. A missing service surfaces immediately as # NoRespondersError; a slow one surfaces as TimeoutError. order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' try: reply = await client.request("orders.inventory.check", order.encode(), timeout=2.0) print(f"inventory replied: {reply.data.decode()}") except NoRespondersError: print("no inventory service is running") except TimeoutError: print("inventory service did not answer in time") ``` #### Java ``` // Ask the inventory service whether an order's item is in stock. A missing // service surfaces immediately as a canceled request; a slow one as a // timeout. byte[] order = ("{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," + "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}").getBytes(StandardCharsets.UTF_8); CompletableFuture future = nc.request("orders.inventory.check", order); try { Message reply = future.get(2, TimeUnit.SECONDS); System.out.println("inventory replied: " + new String(reply.getData(), StandardCharsets.UTF_8)); } catch (TimeoutException e) { System.out.println("inventory service did not answer in time"); } catch (ExecutionException e) { // reportNoResponders() surfaces a missing service as a 503 status if (e.getCause() instanceof JetStreamStatusException) { System.out.println("no inventory service is running"); } else { System.out.println("request failed: " + e.getMessage()); } } ``` #### Rust ``` // Ask the inventory service whether an order's item is in stock. The client // creates a private inbox, sends the request, and waits for one reply. A // missing service surfaces as NoResponders; a slow one as TimedOut. let order = r#"{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}"#; match client.request("orders.inventory.check", order.into()).await { Ok(response) => { println!( "inventory replied: {}", String::from_utf8_lossy(&response.payload) ); } Err(err) => match err.kind() { RequestErrorKind::NoResponders => println!("no inventory service is running"), RequestErrorKind::TimedOut => println!("inventory service did not answer in time"), _ => eprintln!("request failed: {}", err), }, } ``` #### C#/.NET ``` // Ask the inventory service whether an order's item is in stock. The client // creates a private inbox, sends the request, and waits for one reply. // RequestAsync throws NatsNoRespondersException immediately when nothing is // subscribed on the subject. var order = new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:22Z", CultureInfo.InvariantCulture)); try { var reply = await client.RequestAsync("orders.inventory.check", order); output.WriteLine($"inventory replied: {reply.Data}"); } catch (NatsNoRespondersException) { output.WriteLine("no inventory service is running"); } ``` #### C ``` // Ask the inventory service whether an order's item is in stock. The // client creates a private inbox, sends the request, and waits up to // the timeout for one reply. A missing service surfaces immediately as // NATS_NO_RESPONDERS; a slow one as NATS_TIMEOUT. const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; natsMsg *reply = NULL; s = natsConnection_RequestString(&reply, conn, "orders.inventory.check", order, 2000); switch (s) { case NATS_NO_RESPONDERS: printf("no inventory service is running\n"); break; case NATS_TIMEOUT: printf("inventory service did not answer in time\n"); break; case NATS_OK: printf("inventory replied: %.*s\n", natsMsg_GetDataLength(reply), natsMsg_GetData(reply)); natsMsg_Destroy(reply); break; default: printf("request failed: %s\n", natsStatus_GetText(s)); break; } ``` You should see the reply printed back. Behind that one line, the client picked a fresh inbox subject, published your payload with the inbox attached, and matched the answer back to this call by its token on the subscription it already holds for its inbox prefix. ## Every request needs a timeout A request can fail to come back — the responder might be slow or busy, or the reply lost in flight. Replies are [at-most-once](/learn/core-nats/publish-subscribe.md#at-most-once-delivery) like everything else: one that doesn't arrive is gone, not retried. So every request carries a **timeout**: the longest the client will wait for the answer before giving up. The CLI sets `--timeout` for you (five seconds by default), and the request snippet above makes it explicit with `--timeout 2s`. Pick a value that covers the responder's work plus the network round-trip. In a client library you pass the timeout on every `request()` call, so a request can't wait indefinitely. When the timeout expires with no reply, the call returns a timeout error. Your code decides what to do next: retry, fall back, or fail the caller. Core NATS won't make that decision for you, and it won't deliver the answer late. A timeout tells you the answer didn't arrive in time, not *why* — the responder might be slow, or not there at all. The next section tells those apart. ## No responders Waiting two seconds to discover that nobody is even listening is wasteful. NATS has a faster signal for that exact case. When you send a request to a subject with zero subscribers, the server knows immediately that nobody can answer. Rather than let your timeout run, it sends back a **no responders** signal right away: a reply carrying a `503` status. Your client surfaces it as a distinct no-responders error, not a timeout. This is the difference between "the inventory service is slow" (you get a timeout after 2s) and "the inventory service isn't running at all" (you get no responders in milliseconds). One is a latency problem; the other is a deployment problem. The signal lets your code react correctly to each. See it for yourself. Stop the inventory service from the first terminal, then send the request again: ``` nats request orders.inventory.check \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --timeout 2s ``` ``` 14:02:31 Sending request on "orders.inventory.check" 14:02:31 No responders are available ``` The no-responders signal comes back instantly, not after two seconds. The CLI prints the line and exits cleanly; a client library surfaces the same case as a distinct error you can branch on (`ErrNoResponders` in the Go client, with an equivalent in each language). Start the service again and the same request succeeds. The signal rides the message header mechanism: the server delivers a reply with the header line `NATS/1.0 503`. A client needs header support to receive it, which every current client enables. ## Headers That `503` mechanism points to a wider capability: NATS messages can carry **headers**, key/value metadata that travels alongside the payload in a format that looks like HTTP. A request can attach headers, and so can a reply. You won't need them for the inventory call, so this page doesn't build with them. [Message headers](/learn/core-nats/headers.md) shows how to set and read them; the full wire format is in [Reference](/reference/.md). Reach for them when you want metadata that isn't part of the business payload: a request ID, a trace context, a content type. ## Request-reply, queue groups, and many answers The inventory service has one responder. Two questions follow naturally, and each is its own page. What if you run several inventory instances for capacity, and want exactly one of them to handle each request? That's a **queue group**, and the [next page](/learn/core-nats/queue-groups.md) builds one. What if you want *every* responder on a subject to answer the same request, and you collect all the replies? That's [scatter-gather](/learn/core-nats/scatter-gather.md), two pages on. If your request-reply services start to grow real endpoints, discovery, and stats, you're describing the [Services framework](/learn/services/.md), a layer built on exactly the request-reply and queue-group primitives in this chapter. This chapter stays on the primitives. ## Pitfalls **A request without a timeout can wait forever.** Pass a deadline on every `request()` call, sized to the responder's work plus the round-trip — long enough not to give up on a merely-slow reply. **Treating no responders as a hang.** No responders comes back in milliseconds, not as a slow timeout. Branch on it separately: no responders means nothing is deployed, a timeout means it's deployed but slow. **Assuming exactly one reply.** A plain `request()` returns the first reply and discards the rest. If two inventory instances both answer on `orders.inventory.check`, the second answer is lost silently, with no indication that it ever arrived. When more than one service may answer, ask for it explicitly and gather by count or deadline: #### CLI ``` #!/bin/bash # A plain request returns only the FIRST reply and discards the rest. # If more than one service answers on a subject, the extras are lost # silently. Make the expectation explicit with --replies. # # --replies N waits for up to N replies instead of stopping at the first. # --replies 0 collects every reply until --timeout ends the call. nats request orders.inventory.check \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --replies 0 --timeout 2s ``` #### JavaScript/TypeScript ``` // Gather more than one reply to a single request. A plain request() returns // only the first reply, so when several services may answer, subscribe to your // own inbox, publish the request with that inbox as the reply subject, and // collect replies until they stop arriving. const order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'; const inbox = createInbox(); const sub = nc.subscribe(inbox); nc.publish("orders.inventory.check", order, { reply: inbox }); const replies: string[] = []; const gather = (async () => { for await (const msg of sub) { replies.push(msg.string()); } })(); // Stop once no further reply is expected. setTimeout(() => sub.unsubscribe(), 300); await gather; console.log(`gathered ${replies.length} replies`); ``` #### Go ``` // Gather more than one reply to a single request. A plain Request returns // only the first reply, so when several services may answer, subscribe to // your own inbox, publish the request with that inbox as the reply subject, // and collect replies until they stop arriving. order := `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}` inbox := nats.NewInbox() sub, _ := nc.SubscribeSync(inbox) nc.PublishRequest("orders.inventory.check", inbox, []byte(order)) var replies [][]byte for { // Stop once no further reply arrives within the gap deadline. msg, err := sub.NextMsg(300 * time.Millisecond) if err != nil { break } replies = append(replies, msg.Data) } fmt.Printf("gathered %d replies\n", len(replies)) ``` #### Python ``` # Gather more than one reply to a single request. A plain request() returns # only the first reply, so when several services may answer, subscribe to # your own inbox and collect replies until they stop arriving. order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' inbox = f"_INBOX.{uuid.uuid4().hex}" async with await client.subscribe(inbox) as subscription: await client.publish("orders.inventory.check", order.encode(), reply=inbox) replies = [] while True: try: # Stop once no further reply arrives within the gap deadline. message = await subscription.next(timeout=0.3) replies.append(message.data.decode()) except TimeoutError: break print(f"gathered {len(replies)} replies: {replies}") ``` #### Java ``` // Gather more than one reply to a single request. A plain request returns // only the first reply, so when several services may answer, subscribe to // your own inbox, publish the request with that inbox as the reply subject, // and collect replies until they stop arriving. byte[] order = ("{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," + "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}").getBytes(StandardCharsets.UTF_8); String inbox = nc.createInbox(); Subscription sub = nc.subscribe(inbox); nc.publish("orders.inventory.check", inbox, order); List replies = new ArrayList<>(); Message m = sub.nextMessage(Duration.ofMillis(300)); while (m != null) { replies.add(new String(m.getData(), StandardCharsets.UTF_8)); m = sub.nextMessage(Duration.ofMillis(300)); } System.out.println("gathered " + replies.size() + " replies"); ``` #### Rust ``` // Gather more than one reply to a single request. A plain request() returns // only the first reply, so when several services may answer, subscribe to // your own inbox, publish the request with that inbox as the reply subject, // and collect replies until they stop arriving. let order = r#"{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}"#; let inbox = client.new_inbox(); let mut sub = client.subscribe(inbox.clone()).await?; client .publish_with_reply("orders.inventory.check", inbox, order.into()) .await?; client.flush().await?; let mut replies = Vec::new(); // Stop once no further reply arrives within the gap deadline. while let Ok(Some(msg)) = tokio::time::timeout(Duration::from_millis(300), sub.next()).await { replies.push(String::from_utf8_lossy(&msg.payload).to_string()); } println!("gathered {} replies", replies.len()); ``` #### C#/.NET ``` // Gather more than one reply to a single request. A plain request returns // only the first reply, so when several services may answer, subscribe to // your own inbox, publish the request with that inbox as the reply subject, // and collect replies until they stop arriving. var order = new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:22Z", CultureInfo.InvariantCulture)); var inbox = client.Connection.NewInbox(); await using var sub = await client.Connection.SubscribeCoreAsync(inbox); await client.PublishAsync("orders.inventory.check", order, replyTo: inbox); var replies = new List(); using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300)); try { // Stop once no further reply arrives within the gap deadline. await foreach (var msg in sub.Msgs.ReadAllAsync(cts.Token)) { replies.Add(msg.Data!); } } catch (OperationCanceledException) { } output.WriteLine($"gathered {replies.Count} replies"); ``` #### C ``` // Gather more than one reply to a single request. A plain Request // returns only the first reply, so when several services may answer, // subscribe to your own inbox, publish the request with that inbox as // the reply subject, and collect replies until they stop arriving. const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; int replies = 0; if (s == NATS_OK) s = natsInbox_Create(&inbox); if (s == NATS_OK) s = natsConnection_SubscribeSync(&sub, conn, (const char *) inbox); if (s == NATS_OK) s = natsConnection_PublishRequestString(conn, "orders.inventory.check", (const char *) inbox, order); while (s == NATS_OK) { natsMsg *reply = NULL; // Stop once no further reply arrives within the gap deadline. s = natsSubscription_NextMsg(&reply, sub, 300); if (s != NATS_OK) break; replies++; natsMsg_Destroy(reply); } printf("gathered %d replies\n", replies); ``` When you actually want every responder to answer, that's [scatter-gather](/learn/core-nats/scatter-gather.md), not a bug. When you want exactly one of several instances to handle each request, that's a [queue group](/learn/core-nats/queue-groups.md). **Doing slow work inside the responder.** A responder that runs a slow lookup before replying serializes every request behind it, so one expensive call adds latency to all the callers waiting in line. Keep the reply path fast, or run several instances in a [queue group](/learn/core-nats/queue-groups.md) so the load spreads across them instead of stacking on one. ## Where you are The Acme world now has its first two-way conversation: * An inventory service answers on `orders.inventory.check`, built with `nats reply` (or a client's `respond()`). * The warehouse asks with `nats request` (or `request()`), every call bounded by a timeout. * A missing responder surfaces instantly as no responders, not as a slow timeout. * Replies are at-most-once like everything else in core NATS: not retried, not held. ## What's next The inventory service is a single process. To scale it, you run several copies and let NATS hand each request to exactly one of them. That's a queue group: built-in load balancing with no broker in the middle. Build one on the next page: [Queue groups](/learn/core-nats/queue-groups.md). ## See also * [Core Concepts → Request-reply](/concepts/request-reply.md) — the five-minute overview of the same pattern. * [Learn → Services](/learn/services/.md) — the framework that turns request-reply responders into discoverable services. * [Reference → Client protocol](/reference/protocols/client.md) — the wire-level `PUB`/`SUB`/`MSG` and header format. --- # Scatter-gather The inventory service answered one request with one reply, which is the common case of a single question producing a single answer. Some questions have several answers. "What would it cost to ship this order?" is one of them. Acme works with three carriers, and each one quotes a different price. The order service wants all three quotes, then picks the cheapest. That's **scatter-gather**: fan one request out to every responder, then gather the replies that come back. This page builds it on `shipping.quote` with three quote providers. **Message flow — Request / Reply — scatter-gather:** Request to multiple services - all respond (scatter-gather pattern) * Client → NATS (subject: get.status) * NATS → Service A * NATS → Service B * NATS → Service C * Service A → NATS * Service B → NATS * Service C → NATS * NATS → Client (subject: 3 replies) ## Why one request can produce many replies Recall how request-reply works: the client subscribes to a unique `_INBOX` subject and publishes the request carrying that inbox as its reply subject. Nothing in that mechanism limits the number of responders. If three providers subscribe to `shipping.quote`, all three receive the request (plain publish-subscribe) and all three can reply to the inbox. The single reply you saw earlier was just one responder plus a client that stopped after the first answer. This works only when the responders are not in a queue group — a queue group hands each request to one member, which is load balancing, not scatter-gather. Every responder must subscribe plainly. The CLI demo below makes that explicit, because the CLI's default does the opposite. ## Set up three quote providers A provider subscribes to `shipping.quote`, reads the order, and replies with a price. Run three of them, each quoting a different number. #### CLI ``` #!/bin/bash # Three shipping-quote providers, each answering on shipping.quote. # # IMPORTANT: `nats reply` subscribes inside a queue group by default # (NATS-RPLY-22). Three providers left on that default would share ONE # queue group, so only one would ever answer. To scatter the request to # all three, give each provider its OWN queue group name with --queue. # A queue group of one member behaves like a plain subscriber. # # Run each line in its own terminal. # Terminal 1 — carrier A quotes 1500 cents. nats reply shipping.quote --queue carrier-a '{"carrier":"carrier-a","quote_cents":1500}' # Terminal 2 — carrier B quotes 1200 cents. nats reply shipping.quote --queue carrier-b '{"carrier":"carrier-b","quote_cents":1200}' # Terminal 3 — carrier C quotes 1800 cents. nats reply shipping.quote --queue carrier-c '{"carrier":"carrier-c","quote_cents":1800}' ``` #### JavaScript/TypeScript ``` // A shipping-quote provider. Subscribe plainly to shipping.quote (NOT in a // queue group, so every provider sees each request) and reply with a price. // Run several copies, each quoting a different number. const sub = nc.subscribe("shipping.quote"); for await (const msg of sub) { msg.respond('{"carrier":"carrier-a","quote_cents":1500}'); } ``` #### Go ``` // A shipping-quote provider. Subscribe plainly to shipping.quote (NOT a // queue group, so every provider sees each request) and reply with a price. // Run several copies, each quoting a different number. nc.Subscribe("shipping.quote", func(m *nats.Msg) { m.Respond([]byte(`{"carrier":"carrier-a","quote_cents":1500}`)) }) ``` #### Python ``` # A shipping-quote provider. Subscribe plainly to shipping.quote (NOT in a # queue group, so every provider sees each request) and reply with a price. # Run several copies, each quoting a different number. async with await client.subscribe("shipping.quote") as subscription: async for message in subscription: if message.reply: await client.publish(message.reply, b'{"carrier":"carrier-a","quote_cents":1500}') ``` #### Java ``` // A shipping-quote provider. Subscribe plainly to shipping.quote (NOT in a // queue group, so every provider sees each request) and reply with a price. // Run several copies, each quoting a different number. nc.createDispatcher(msg -> { if (msg.getReplyTo() != null) { nc.publish(msg.getReplyTo(), "{\"carrier\":\"carrier-a\",\"quote_cents\":1500}".getBytes(StandardCharsets.UTF_8)); } }).subscribe("shipping.quote"); ``` #### Rust ``` // A shipping-quote provider. Subscribe plainly to shipping.quote (NOT in a // queue group, so every provider sees each request) and reply with a price. // Run several copies, each quoting a different number. let mut sub = client.subscribe("shipping.quote").await?; while let Some(msg) = sub.next().await { if let Some(reply) = msg.reply { client .publish( reply, r#"{"carrier":"carrier-a","quote_cents":1500}"#.into(), ) .await?; } } ``` #### C#/.NET ``` // A shipping-quote provider. Subscribe plainly to shipping.quote (NOT in a // queue group, so every provider sees each request) and reply with a price. // Run several copies, each quoting a different number. await foreach (var msg in client.SubscribeAsync("shipping.quote")) { await msg.ReplyAsync(new ShippingQuote(Carrier: "carrier-a", QuoteCents: 1500)); } ``` #### C ``` // A shipping-quote provider. Subscribe plainly to shipping.quote (NOT a // queue group, so every provider sees each request) and reply with a // price. Run several copies, each quoting a different number. static void onQuote(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { if (natsMsg_GetReply(msg) != NULL) natsConnection_PublishString(nc, natsMsg_GetReply(msg), "{\"carrier\":\"carrier-a\",\"quote_cents\":1500}"); natsMsg_Destroy(msg); } if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "shipping.quote", onQuote, NULL); ``` There's one trap to know about with `nats reply`. By default, the CLI subscribes inside a queue group named `NATS-RPLY-22`. Three `nats reply` instances left on that default would form one queue group, and only one of them would ever answer. That's the load-balancing behavior from the previous page, not what we want here. To make each provider an independent responder, give each one its own queue group name with `--queue`. A queue group of one member behaves like a plain subscriber: it receives every matching request. The CLI source for the snippet above runs the three providers with distinct names (`carrier-a`, `carrier-b`, `carrier-c`), so all three see each request. The client library form has no such trap. A library subscribes plainly unless you ask for a queue group, so three plain subscribers on `shipping.quote` already scatter correctly. ## Gather by count Now the gather side. The client sends one request to `shipping.quote` and collects replies until it's heard from every provider. #### CLI ``` #!/bin/bash # Send one request to shipping.quote and gather replies from every provider. # Gather by count: stop after 3 replies have arrived. nats request shipping.quote \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --replies 3 --timeout 2s # Gather by deadline instead: --replies 0 collects every reply that arrives # during the full --timeout window, then returns. It is a fixed, predictable # budget; --reply-timeout has no effect in this mode. Use this when you do # not know how many providers are running. nats request shipping.quote \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --replies 0 --timeout 2s ``` #### JavaScript/TypeScript ``` // Scatter one request to every shipping-quote provider and gather the replies. // Subscribe to a private inbox, publish the request with that inbox as the // reply subject, then collect quotes until they stop arriving and pick the // cheapest. const order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'; const inbox = createInbox(); const sub = nc.subscribe(inbox); nc.publish("shipping.quote", order, { reply: inbox }); const quotes: string[] = []; const gather = (async () => { for await (const msg of sub) { quotes.push(msg.string()); } })(); // Stop once no further reply is expected. setTimeout(() => sub.unsubscribe(), 300); await gather; console.log(`gathered ${quotes.length} quotes: ${JSON.stringify(quotes)}`); ``` #### Go ``` // Scatter one request to every shipping-quote provider and gather the // replies. Subscribe to a private inbox, publish the request with that // inbox as the reply subject, then collect quotes until they stop arriving // and pick the cheapest. order := `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}` inbox := nats.NewInbox() sub, _ := nc.SubscribeSync(inbox) nc.PublishRequest("shipping.quote", inbox, []byte(order)) var quotes []string for { msg, err := sub.NextMsg(300 * time.Millisecond) if err != nil { break } quotes = append(quotes, string(msg.Data)) } fmt.Printf("gathered %d quotes: %v\n", len(quotes), quotes) ``` #### Python ``` # Scatter one request to every shipping-quote provider and gather the # replies. Subscribe to a private inbox, publish the request with that inbox # as the reply subject, then collect quotes until they stop arriving and # pick the cheapest. order = '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' inbox = f"_INBOX.{uuid.uuid4().hex}" async with await client.subscribe(inbox) as subscription: await client.publish("shipping.quote", order.encode(), reply=inbox) quotes = [] while True: try: message = await subscription.next(timeout=0.3) quotes.append(message.data.decode()) except TimeoutError: break print(f"gathered {len(quotes)} quotes: {quotes}") ``` #### Java ``` // Scatter one request to every shipping-quote provider and gather the // replies. Subscribe to a private inbox, publish the request with that // inbox as the reply subject, then collect quotes until they stop arriving // and pick the cheapest. byte[] order = ("{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," + "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}").getBytes(StandardCharsets.UTF_8); String inbox = nc.createInbox(); Subscription sub = nc.subscribe(inbox); nc.publish("shipping.quote", inbox, order); List quotes = new ArrayList<>(); Message m = sub.nextMessage(Duration.ofMillis(300)); while (m != null) { quotes.add(new String(m.getData(), StandardCharsets.UTF_8)); m = sub.nextMessage(Duration.ofMillis(300)); } System.out.println("gathered " + quotes.size() + " quotes"); ``` #### Rust ``` // Scatter one request to every shipping-quote provider and gather the // replies. Subscribe to a private inbox, publish the request with that inbox // as the reply subject, then collect quotes until they stop arriving and // pick the cheapest. let order = r#"{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}"#; let inbox = client.new_inbox(); let mut sub = client.subscribe(inbox.clone()).await?; client .publish_with_reply("shipping.quote", inbox, order.into()) .await?; client.flush().await?; let mut quotes = Vec::new(); while let Ok(Some(msg)) = tokio::time::timeout(Duration::from_millis(300), sub.next()).await { quotes.push(String::from_utf8_lossy(&msg.payload).to_string()); } println!("gathered {} quotes: {:?}", quotes.len(), quotes); ``` #### C#/.NET ``` // Scatter one request to every shipping-quote provider and gather the // replies. Subscribe to a private inbox, publish the request with that inbox // as the reply subject, then collect quotes until they stop arriving and // pick the cheapest. var order = new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:22Z", CultureInfo.InvariantCulture)); var inbox = client.Connection.NewInbox(); await using var sub = await client.Connection.SubscribeCoreAsync(inbox); await client.PublishAsync("shipping.quote", order, replyTo: inbox); var quotes = new List(); using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300)); try { await foreach (var msg in sub.Msgs.ReadAllAsync(cts.Token)) { quotes.Add(msg.Data!); } } catch (OperationCanceledException) { } output.WriteLine($"gathered {quotes.Count} quotes"); ``` #### C ``` // Scatter one request to every shipping-quote provider and gather the // replies. Subscribe to a private inbox, publish the request with that // inbox as the reply subject, then collect quotes until they stop // arriving and pick the cheapest. const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; int quotes = 0; if (s == NATS_OK) s = natsInbox_Create(&inbox); if (s == NATS_OK) s = natsConnection_SubscribeSync(&sub, conn, (const char *) inbox); if (s == NATS_OK) s = natsConnection_PublishRequestString(conn, "shipping.quote", (const char *) inbox, order); while (s == NATS_OK) { natsMsg *quote = NULL; // Stop once no further quote arrives within the gap deadline. s = natsSubscription_NextMsg("e, sub, 300); if (s != NATS_OK) break; printf("quote: %.*s\n", natsMsg_GetDataLength(quote), natsMsg_GetData(quote)); quotes++; natsMsg_Destroy(quote); } printf("gathered %d quotes\n", quotes); ``` The CLI uses `--replies 3`: read from the inbox until three replies arrive, then stop — or until replies stop arriving, whichever comes first. With the three providers running, the output shows three quotes, one per carrier. The client compares the prices and keeps the lowest. A library does the same thing, and how much you hand-roll depends on the client. A plain `request()` returns only the first reply — on its own it never returns a list, because the client can't know how many responders exist. Some clients ship a gather helper that returns many: nats.js has `requestMany` and orbit.go has `RequestMany` (both follow ADR-47, "Request Many", with count, stall, and sentinel stop conditions), and the .NET client has `RequestManyAsync`. Where no helper exists, you build the loop yourself: subscribe to a fresh inbox, publish the request with that inbox as the reply subject, then read from the subscription and append each reply to a list until your count or deadline is reached, then unsubscribe. ## Gather by deadline Counting replies assumes you know how many providers there are. Often you don't. Carriers come and go; one might be down. Write that gather yourself with no read deadline, and waiting for a fixed count of three blocks forever if only two answer — the read for the third reply never returns. The safer approach gathers by **deadline** instead. Collect every reply that arrives within a time budget, then act on whatever you have. From the CLI, `--replies 0` switches to deadline mode: `--timeout` becomes the whole collection window. The command reads replies for the full window and returns when it closes, so `--replies 0 --timeout 2s` gathers everything that arrives in those two seconds. It's a fixed, predictable budget, and `--reply-timeout` has no effect in this mode — it only bounds the gap between replies when you gather by count. ``` nats request shipping.quote \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --replies 0 --timeout 2s ``` With three providers up, all three quotes arrive within milliseconds, but the command still holds open for the full two seconds before returning the set. With one provider down, you get the two quotes that answered and wait out the same two-second window. If no provider is subscribed at all, the command returns right away with a no responders signal instead of waiting — though a provider that's subscribed but slow to answer still costs the full `--timeout`. Either way the wait is bounded; it never blocks forever. A deadline changes the rule from waiting for every responder to waiting only for whoever answers in time, which is the only safe assumption when the responder set isn't fixed. There's a third way to end a gather: a **sentinel**. With `--wait-for-empty`, the command keeps collecting until a reply arrives with an empty payload, which a responder sends to mark the end of the set. This counts replies rather than watching a single deadline, so if the sentinel never comes the `--reply-timeout` bounds the gap between replies and the wait still ends. ## Delivery guarantees Scatter-gather is [at-most-once](/learn/core-nats/publish-subscribe.md#at-most-once-delivery) like everything else: a reply dropped in transit is just absent from the gathered set, nothing redelivers it, and arrival order carries no meaning. Treat the set as whatever answers happened to arrive, not a ranked list. That's fine for a shipping quote — re-asking is cheap. When each reply must survive a crash, that's [JetStream](/learn/jetstream/.md), not a core gather. ## Pitfalls **Taking only the first reply.** A plain `nats request` stops after one reply, because its `--replies` flag defaults to `1`. Point it at three providers and you get whichever carrier answered first; the other two quotes are discarded and you never learn there were more. A plain `request()` in a client library does the same. When you mean to gather, use `--replies 0` from the CLI, reach for your client's gather helper where it has one (nats.js `requestMany`, orbit.go `RequestMany`), or subscribe to the inbox yourself and read in a loop. #### CLI ``` #!/bin/bash # The "first reply only" trap, and the fix. # # `nats request` defaults to --replies 1: it reads ONE reply and stops. # With three providers up, you get whichever carrier answered first and # the other two quotes are silently discarded. That is a single request, # not a scatter-gather. # WRONG for scatter-gather — takes only the first quote that lands. nats request shipping.quote \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # RIGHT — gather every quote, then pick the cheapest. --replies 0 collects # every reply that arrives during the full --timeout window, then returns — # a fixed, predictable budget. If a carrier is down, its quote is simply # absent from the set; --reply-timeout has no effect in this mode. nats request shipping.quote \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --replies 0 --timeout 2s ``` #### C ``` // The "first reply only" trap: a plain request reads ONE reply and // stops. With three providers up, you get whichever carrier answered // first and the other two quotes are silently discarded. const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; natsMsg *first = NULL; if (s == NATS_OK) s = natsConnection_RequestString(&first, conn, "shipping.quote", order, 2000); if (s == NATS_OK) { printf("first quote only: %.*s\n", natsMsg_GetDataLength(first), natsMsg_GetData(first)); natsMsg_Destroy(first); } // The fix: gather every quote yourself. Subscribe to a private inbox, // publish the request with that inbox as the reply subject, and read // replies until they stop arriving. int quotes = 0; if (s == NATS_OK) s = natsInbox_Create(&inbox); if (s == NATS_OK) s = natsConnection_SubscribeSync(&sub, conn, (const char *) inbox); if (s == NATS_OK) s = natsConnection_PublishRequestString(conn, "shipping.quote", (const char *) inbox, order); while (s == NATS_OK) { natsMsg *quote = NULL; s = natsSubscription_NextMsg("e, sub, 300); if (s != NATS_OK) break; printf("quote: %.*s\n", natsMsg_GetDataLength(quote), natsMsg_GetData(quote)); quotes++; natsMsg_Destroy(quote); } printf("gathered %d quotes\n", quotes); ``` **No deadline, so you wait for replies that never come.** A hand-rolled gather that reads with no deadline blocks forever if a provider is down — the read for the missing reply never returns. Bound the wait: gather by deadline (`--replies 0 --timeout 2s`) so a missing carrier just leaves its quote out of the set. **Reading the gathered set as ranked.** Arrival order isn't priority and a short set isn't an error. Compare every reply on its merits (here, the lowest `quote_cents`). ## Where you are The Acme ORDERS world now talks in four shapes over one local `nats-server`: * `notifications` and `analytics` each receive their own copy of every `orders.created` message (publish-subscribe). * Regional analytics spans `orders.us.created` and `orders.eu.created` with `orders.*.created`, and an audit service reads the whole hierarchy on `orders.>`. * The `inventory` service answers single requests on `orders.inventory.check` (request-reply). * A `packers` queue group shares the work on `orders.created`, one packer per order. * Three providers answer one `shipping.quote` request, and the client gathers every reply within a deadline and picks the cheapest. That's the whole of core NATS: subjects, interest, reply inboxes, and queue groups. Everything is ephemeral and at-most-once, and nothing is remembered after it's delivered. ## What's next The next page adds a third piece to every message, beside its subject and payload: headers, a set of key/value metadata. You'll put a request id and a trace id on the orders flow and read the status header the server uses to signal no responders. Continue to [Message headers](/learn/core-nats/headers.md). ## See also * [Concepts → Request-reply](/concepts/request-reply.md) — the five-minute overview of the pattern this page extends. * [Learn → Services](/learn/services/.md) — when many responders become a managed service that aggregates results for you. * [Reference](/reference/.md) — gather helper signatures per client library. --- # Subject mapping Every message so far has kept the subject its publisher gave it: a client publishes `orders.created`, and the server routes it to whoever subscribed to `orders.created`. Subject mapping breaks that one-to-one link. The server rewrites a message's subject the moment it arrives, before it looks for interested subscribers, and it does the rewrite from its own configuration — the publisher sends the same command and never sees the change. Acme has a use for that today. Its old mobile app still publishes new orders to `orders.placed`, but every service moved to `orders.created` several pages ago. Instead of shipping a new app build, Acme tells the server to map `orders.placed` onto `orders.created`. The app keeps publishing `orders.placed`; the packers and analytics keep subscribing to `orders.created`; the server joins the two. This is the account-level subject mapping that the JetStream chapter points back to. It's different from a stream's subject transform or its republish setting: those rewrite subjects as a message enters or leaves a JetStream stream, and you configure them on the stream. Server-side mapping runs earlier, on plain core subjects, before any stream is involved. For the stream side, see [Subject mapping and transforms](/learn/jetstream/subject-mapping.md); this page stays on the core, server-config side. ## Rename a subject A **subject mapping** is a rule in server configuration that rewrites one subject to another. You give the server a source subject and a destination subject; when a message arrives on the source, the server delivers it as if the publisher had named the destination. Mapping lives in the server's configuration file, so this is the first page in the chapter that needs one. Earlier pages started the server with a bare `nats-server`; from here you write a small config file and start the server with `nats-server -c server.conf`. A config file that renames Acme's legacy subject looks like this: ``` # server.conf mappings { orders.placed: orders.created } ``` ``` nats-server -c server.conf ``` Now a publisher to `orders.placed` reaches every subscriber on `orders.created`. The publisher's command doesn't change, and it gets no signal that a mapping exists — the rewrite happens on the server, on the way in. The top-level `mappings` block belongs to the server's built-in account, so on a plain server with no accounts or authentication it covers every client. Mapping is always scoped to an account: with several accounts configured, each has its own mappings and they never cross the account boundary. The `mappings` block is reloadable. Edit `server.conf` and run `nats-server --signal reload` to apply the change without restarting; the signal tells the running server to re-read its config, and open connections stay up. ## Preview a mapping before you apply it You don't have to change a running server to see what a mapping does. `nats server mappings` takes a source, a destination, and a subject, runs the same transform code the server uses, and prints the result — with no server and no connection. #### CLI ``` #!/bin/bash # Check a mapping without a running server. `nats server mappings` takes a # source pattern, a destination pattern, and a subject, runs the same # transform code the server uses, and prints the result. It opens no # connection, so you can try a mapping before you ever put it in config. # A literal rename: orders.placed always maps to orders.created. nats server mappings "orders.placed" "orders.created" orders.placed # A token reference: {{wildcard(1)}} pulls the token the first * matched into # the destination, so orders.legacy.us becomes orders.us.created. nats server mappings "orders.legacy.*" "orders.{{wildcard(1)}}.created" orders.legacy.us ``` ``` orders.created orders.us.created ``` The second command shows a **token reference**: `{{wildcard(1)}}` in the destination stands for the token the first single-token wildcard matched in the source. Source `orders.legacy.*` matched `us`, so the destination `orders.{{wildcard(1)}}.created` filled in as `orders.us.created`. You number wildcards left to right, so `{{wildcard(2)}}` would be the second `*`. ## Split traffic by weight A mapping can send a fraction of matched messages to a different subject. A **weighted mapping** lists several destinations, each with a weight from 0 to 100, and the server picks one per message at random in proportion to the weights. Acme is testing a new packer build. It wants a tenth of new orders handled by the canary and the rest handled normally, so it maps `orders.created` and gives the canary a weight of 10: ``` # server.conf mappings { orders.created: [ { destination: orders.created.canary, weight: 10 } ] } ``` The weights you list add up to 10, not 100. The server sends the remaining 90 out of every 100 messages to the source subject unchanged, so the packers already on `orders.created` keep handling most orders while the canary on `orders.created.canary` samples the rest. Whenever your listed weights total less than 100, the leftover share stays on the original subject. To split between two new subjects instead, list both and make the weights total 100 — 50 and 50, say, for an even A/B test where nothing stays on the original. If you want the leftover share dropped rather than kept — to test how a subscriber copes with loss — list the source subject itself as a destination, which tells the server your weights are final and stops it topping them up. This works because the source here is a literal subject. The split is per-message and random, the same way a queue group picks a member: over a few messages it can look lopsided, and over thousands it converges on the weights you set. Each weight must be 100 or less, and the weights for one source must total 100 or less, or the server rejects the config. ## Partition by a token A weighted mapping spreads messages at random across its destinations. **Partitioning** spreads them deterministically instead: it routes each message by hashing one of its tokens into a fixed bucket, so the same token value always lands in the same bucket. In a destination, `{{partition(n, 1)}}` stands for that bucket — it hashes the token the first single-token wildcard matched into one of `n` buckets, numbered `0` to `n-1`. Acme wants each order handled by one fixed packer pool, so everything about an order stays together: one pool's cache, no coordination between pools. The order id decides the pool, which means the id has to be in the subject. So the app publishes new orders with the id as the last token, like `orders.created.ord_8w2k`, and Acme splits them into three buckets by hashing that id: ``` # server.conf mappings { "orders.created.*": "orders.created.{{partition(3, 1)}}.{{wildcard(1)}}" } ``` The source `orders.created.*` matches any order id in the last token. The destination builds a new subject: `{{partition(3, 1)}}` becomes the bucket for that id, and `{{wildcard(1)}}` carries the id itself through. So `orders.created.ord_8w2k` becomes `orders.created.0.ord_8w2k`. The bucket is deterministic, so `nats server mappings` tells you exactly where an id lands before you commit the config: ``` nats server mappings "orders.created.*" "orders.created.{{partition(3, 1)}}.{{wildcard(1)}}" orders.created.ord_8w2k ``` ``` orders.created.0.ord_8w2k ``` Change the id and the bucket changes with it: `ord_7mn3` maps to `orders.created.1.ord_7mn3`, and `ord_2zr9` to `orders.created.2.ord_2zr9`. Feed the same id twice and you get the same bucket both times — that's what makes it safe to route on. Each pool subscribes to its own bucket. The packers for bucket 0 subscribe to `orders.created.0.*` under a shared queue group; buckets 1 and 2 have their own pools on `orders.created.1.*` and `orders.created.2.*`. [Queue groups](/learn/core-nats/queue-groups.md) share the work inside a bucket; the partition decides which bucket. Because the mapping rewrites the subject before the server routes it, the pools subscribe to the bucket subjects, not to `orders.created.*` — a subscriber still on the pre-map `orders.created.ord_8w2k` receives nothing, because the server moved the message to a bucket subject before matching interest. `partition` and `wildcard` cover almost every core mapping. The transform language has more functions — `split`, `slicefromleft`, and others reshape a single token — and streams use the same set, so the full list lives with [the transform language](/learn/jetstream/subject-mapping.md#the-transform-language). ## Cluster-scoped destinations A destination can carry a `cluster` field. The mapping then applies only to messages published through a server in that named cluster, and a server falls back to the unscoped mapping when none matches its cluster. That lets one source map differently in different regions, so it only matters once you run more than one cluster — see [Super-clusters](/learn/topologies/super-clusters.md). A single server has one scope and doesn't need it. ## Try it Put the partition mapping on a real server and watch orders fall into buckets. Save the config and start the server with it: ``` # server.conf mappings { "orders.created.*": "orders.created.{{partition(3, 1)}}.{{wildcard(1)}}" } ``` ``` nats-server -c server.conf ``` Then open one subscriber per bucket and publish three orders: #### CLI ``` #!/bin/bash # Deterministic partitioning across three packer pools. # # The server (server.conf) maps: # orders.created.* -> orders.created.{{partition(3, 1)}}.{{wildcard(1)}} # so every order id hashes to a fixed bucket: 0, 1, or 2. The same id always # lands in the same bucket. # # Open one subscriber per bucket, each in its own terminal. A real pool would # add --queue to share a bucket across several packers; one subscriber per # bucket is enough to see the split. # Terminal 1 — bucket 0 nats sub "orders.created.0.*" # Terminal 2 — bucket 1 nats sub "orders.created.1.*" # Terminal 3 — bucket 2 nats sub "orders.created.2.*" # Terminal 4 — publish three orders with the id as the last token. # ord_8w2k hashes to bucket 0, ord_7mn3 to bucket 1, ord_2zr9 to bucket 2. nats pub orders.created.ord_8w2k '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' nats pub orders.created.ord_7mn3 '{"order_id":"ord_7mn3","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}' nats pub orders.created.ord_2zr9 '{"order_id":"ord_2zr9","customer":"initech","total_cents":1500,"ts":"2026-05-22T10:14:29Z"}' ``` Each order arrives at exactly one bucket: `ord_8w2k` at bucket 0, `ord_7mn3` at bucket 1, `ord_2zr9` at bucket 2. Publish `orders.created.ord_8w2k` again and it returns to bucket 0 — the hash doesn't drift. ## Pitfalls **A mapping quietly changes who receives what.** The server rewrites the subject before it checks interest, so a mapping can pull messages away from subscribers still listening on the original subject. Map `orders.created` entirely onto a new subject and the packers on `orders.created` go silent — every order lands somewhere else, with no error and no warning. Before you map a subject that services already use, check what subscribes to it, and keep a share on the original (the weighted remainder is one way) if those subscribers still need it. **Weights under 100 keep the rest, they don't drop it.** List one destination at weight 10 and it's easy to assume the other 90 percent disappears. It doesn't: the server routes that 90 percent to the source subject. If you really want the remainder dropped — say, to test loss — list the source subject itself as a destination so the server treats your weights as final. This only works for a literal source like `orders.created`. **A partition count is part of the subject contract.** Which bucket an order lands in depends on `n` in `partition(n, ...)`. Raise `partition(3, ...)` to `partition(4, ...)` later and the same order id can hash to a different bucket, so a pool's subject filter silently starts covering a different set of orders. Pick the bucket count once, up front, the way you would for any sharded system. ## Where you are Your running session picked up its first configuration file: * The one local `nats-server` now starts with `nats-server -c server.conf` and reloads with `nats-server --signal reload`. * A mapping bridges the legacy `orders.placed` publisher onto `orders.created`, with nothing changed in the app. * The server can also split a subject's traffic by weight, or shard it into fixed buckets by a hashed token, all before any subscriber is chosen. The subject a publisher writes is no longer always the subject a subscriber matches — the server config sits in between. ## What's next Every publish and subscribe so far assumed the connection to the server was just there. The next page, [Connection lifecycle](/learn/core-nats/connection-lifecycle.md), looks at that connection directly: what happens to your client and its in-flight messages when the link drops, and how to watch it reconnect. ## See also * [Subject mapping and transforms](/learn/jetstream/subject-mapping.md) — the stream side: transforms that rewrite subjects into a stream, and republish that emits them back out. * [Reference → mappings](/reference/config/mappings/.md) — the config keys `destination`, `weight`, and `cluster`, and their exact rules. * [Queue groups](/learn/core-nats/queue-groups.md) — the pools that share the work inside each partition bucket. * [Super-clusters](/learn/topologies/super-clusters.md) — cluster-scoped destinations across regions. --- # Subjects & wildcards On the previous page Acme published `orders.created` and three services subscribed to it. That subject was a flat name. This page gives the subject some structure and lets a subscriber match a whole family of them at once. Acme is opening regional fulfillment. Orders no longer all look the same: a US order and an EU order need to land somewhere a regional service can tell them apart. The subject is where that distinction lives. ## A subject is a sequence of tokens A **subject** is a string the server uses to match publishers to subscribers. You already used one: `orders.created`. The `.` (dot) splits the subject into **tokens**. `orders.created` is two tokens, `orders` then `created`. The server treats each token as a separate unit when it matches. That splitting turns a flat name into a hierarchy. Acme can put the region in the middle: ``` orders.us.created orders.eu.created orders.us.canceled orders.eu.canceled ``` Each of those is three tokens. The first token groups them all under `orders`. The second token says which region. The third says what happened. The server isn't configured to know "region" means anything; the meaning lives entirely in the names Acme chose. A few rules govern what a token may contain. Subjects are **case-sensitive**. `Orders.created` and `orders.created` are two different subjects. A publisher to one won't reach a subscriber on the other. Tokens are split by single dots only. Spaces, tabs, and line breaks aren't allowed anywhere in a subject. Stick to letters, digits, `-`, and `_` inside a token and you'll never be surprised. ## Subjects cost almost nothing to create Acme just invented four new subjects without telling the server first — that's allowed, and it costs almost nothing. The interest graph from the previous page holds an entry for a subject only once something subscribes to it, so a subject nobody listens to has no presence on the server at all. You never declare or clean up subjects, and a system can use millions of them without slowing the server down, because matching walks the token tree rather than scanning every subscription. ## Wildcards: subscribe to many subjects at once A publisher always names one full subject. `nats pub` to `orders.*.created` isn't "publish to every region": it would publish to the literal subject containing a `*`, which nobody wants. Wildcards are a **subscriber-only** tool. A **wildcard** is a token in a subscription that matches more than one literal subject. NATS has exactly two of them, and they differ in how many tokens they match. **Message flow — Subject wildcards (animated):** Animated subject wildcards: messages on different subjects (orders.us.created, orders.eu.created, …) are routed by matching wildcard subscriptions. * Publisher 1 → NATS (subject: orders.retail.placed) * Publisher 2 → NATS (subject: orders.retail.shipped) * Publisher 3 → NATS (subject: orders.wholesale.placed) * NATS → Subscriber (subject: orders.retail.\*) * NATS → Subscriber ### The single-token wildcard `*` The **single-token wildcard** `*` matches exactly one token — not zero, not two. Acme wants one analytics view of created orders across every region. Instead of subscribing to `orders.us.created` and `orders.eu.created` separately, it subscribes once to `orders.*.created`: #### CLI ``` #!/bin/bash # Regional analytics: catch created orders from every region with one # subscription. The single-token wildcard * matches exactly one token in # the region position, so orders.us.created and orders.eu.created both # match, while orders.created and orders.us.west.created do not. nats sub "orders.*.created" ``` #### JavaScript/TypeScript ``` // Regional analytics: one subscription catches created orders from every // region. The single-token wildcard * matches exactly one token, so both // orders.us.created and orders.eu.created match, while orders.created and // orders.us.west.created do not. const sub = nc.subscribe("orders.*.created"); for await (const msg of sub) { console.log(`analytics: new order on ${msg.subject}`); } ``` #### Go ``` // Regional analytics: one subscription catches created orders from every // region. The single-token wildcard * matches exactly one token, so // orders.us.created and orders.eu.created both match, while orders.created // and orders.us.west.created do not. nc.Subscribe("orders.*.created", func(m *nats.Msg) { fmt.Printf("analytics: new order on %s\n", m.Subject) }) ``` #### Python ``` # Regional analytics: one subscription catches created orders from every # region. The single-token wildcard * matches exactly one token, so both # orders.us.created and orders.eu.created match, while orders.created and # orders.us.west.created do not. async with await client.subscribe("orders.*.created") as subscription: async for message in subscription: print(f"analytics: new order on {message.subject}") ``` #### Java ``` // Regional analytics: one subscription catches created orders from every // region. The single-token wildcard * matches exactly one token, so both // orders.us.created and orders.eu.created match, while orders.created and // orders.us.west.created do not. nc.createDispatcher(msg -> System.out.println("analytics: new order on " + msg.getSubject()) ).subscribe("orders.*.created"); ``` #### Rust ``` // Regional analytics: one subscription catches created orders from every // region. The single-token wildcard * matches exactly one token, so both // orders.us.created and orders.eu.created match, while orders.created and // orders.us.west.created do not. let mut sub = client.subscribe("orders.*.created").await?; while let Some(msg) = sub.next().await { println!("analytics: new order on {}", msg.subject); } ``` #### C#/.NET ``` // Regional analytics: one subscription catches created orders from every // region. The single-token wildcard * matches exactly one token, so both // orders.us.created and orders.eu.created match, while orders.created and // orders.us.west.created do not. await foreach (var msg in client.SubscribeAsync("orders.*.created")) { output.WriteLine($"analytics: new order on {msg.Subject}"); } ``` #### C ``` // Regional analytics: one subscription catches created orders from every // region. The single-token wildcard * matches exactly one token, so // orders.us.created and orders.eu.created both match, while orders.created // and orders.us.west.created do not. static void onOrder(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("analytics: new order on %s\n", natsMsg_GetSubject(msg)); natsMsg_Destroy(msg); } if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "orders.*.created", onOrder, NULL); ``` The `*` sits in the region position. Walk through what it catches: * `orders.us.created` matches: `*` takes the single token `us`. * `orders.eu.created` matches: `*` takes the single token `eu`. * `orders.created` does not match: there's no token in the region position, and `*` needs exactly one. * `orders.us.west.created` does not match: two tokens sit where `*` allows only one. The position of `*` is fixed; the token in it is free. You can also use more than one: `orders.*.*` matches any three-token subject under `orders`, with both middle and last tokens free. ### The multi-token wildcard `>` The **multi-token wildcard** `>` matches one or more tokens, and it must be the last token in the pattern. Acme is adding an audit service that wants every order message, at every depth, regardless of region or action. One subscription covers it: #### CLI ``` #!/bin/bash # Audit service: catch every order message at any depth. The multi-token # wildcard > matches one or more tokens and must be the last token, so # orders.> matches orders.created, orders.us.created, and # orders.us.west.created alike. nats sub "orders.>" ``` #### JavaScript/TypeScript ``` // Audit service: catch every order message at any depth. The multi-token // wildcard > matches one or more tokens and must be the last token, so // orders.> matches orders.created, orders.us.created, and // orders.us.west.created alike. const sub = nc.subscribe("orders.>"); for await (const msg of sub) { console.log(`audit: ${msg.subject}`); } ``` #### Go ``` // Audit service: catch every order message at any depth. The multi-token // wildcard > matches one or more tokens and must be the last token, so // orders.> matches orders.created, orders.us.created, and // orders.us.west.created alike. nc.Subscribe("orders.>", func(m *nats.Msg) { fmt.Printf("audit: %s\n", m.Subject) }) ``` #### Python ``` # Audit service: catch every order message at any depth. The multi-token # wildcard > matches one or more tokens and must be the last token, so # orders.> matches orders.created, orders.us.created, and # orders.us.west.created alike. async with await client.subscribe("orders.>") as subscription: async for message in subscription: print(f"audit: {message.subject}") ``` #### Java ``` // Audit service: catch every order message at any depth. The multi-token // wildcard > matches one or more tokens and must be the last token, so // orders.> matches orders.created, orders.us.created, and // orders.us.west.created alike. nc.createDispatcher(msg -> System.out.println("audit: " + msg.getSubject()) ).subscribe("orders.>"); ``` #### Rust ``` // Audit service: catch every order message at any depth. The multi-token // wildcard > matches one or more tokens and must be the last token, so // orders.> matches orders.created, orders.us.created, and // orders.us.west.created alike. let mut sub = client.subscribe("orders.>").await?; while let Some(msg) = sub.next().await { println!("audit: {}", msg.subject); } ``` #### C#/.NET ``` // Audit service: catch every order message at any depth. The multi-token // wildcard > matches one or more tokens and must be the last token, so // orders.> matches orders.created, orders.us.created, and // orders.us.west.created alike. await foreach (var msg in client.SubscribeAsync("orders.>")) { output.WriteLine($"audit: {msg.Subject}"); } ``` #### C ``` // Audit service: catch every order message at any depth. The multi-token // wildcard > matches one or more tokens and must be the last token, so // orders.> matches orders.created, orders.us.created, and // orders.us.west.created alike. static void onOrder(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("audit: %s\n", natsMsg_GetSubject(msg)); natsMsg_Destroy(msg); } if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "orders.>", onOrder, NULL); ``` `orders.>` reaches the entire hierarchy under `orders`: * `orders.created` matches: `>` takes the one token `created`. * `orders.us.created` matches: `>` takes the two tokens `us.created`. * `orders.us.west.created` matches: `>` takes all three remaining tokens. * `orders` does not match: `>` needs at least one token after the prefix. Because `>` matches a token *and everything after it*, it only makes sense at the end. `orders.>.created` is invalid: the server rejects the subscription with an Invalid Subject error. There's no way to anchor a tail wildcard in the middle and still know where it stops. This is the difference to keep: `*` is a placeholder for one token in a known shape; `>` is "everything from here down." ## A wildcard subscriber behaves like any subscriber A wildcard doesn't change the delivery model from the last page. It changes which subjects count as a match, nothing else. The audit service on `orders.>` is just another interested subscriber. When Acme publishes `orders.us.created`, every matching subscriber gets its own copy: the warehouse on `orders.created` does *not* (different subject now), the regional analytics on `orders.*.created` does, and the audit service on `orders.>` does. The server fans one publish out to all of them. Delivery is still [at-most-once](/learn/core-nats/publish-subscribe.md#at-most-once-delivery): a wildcard lets a service that joins now see everything published *from now on*, not what it missed before joining. Capturing that backlog is what [JetStream](/learn/jetstream/.md) adds. ## Reserved prefixes to avoid Acme can name subjects almost anything, but two prefixes are reserved. Subjects beginning with `$` belong to the server and its subsystems: `$SYS` for system events, and `$JS`, `$KV`, `$O`, and `$SRV` for the JetStream, Key-Value, Object-Store, and Services subsystems. Don't publish application messages under `$`. The `_INBOX` prefix is reserved for reply subjects that clients generate automatically. You don't pick `_INBOX` names yourself, and you don't publish business messages there. The next page, on request-reply, shows exactly what `_INBOX` is for. ## Try it in two terminals With `nats-server` running, watch a wildcard catch messages it was never told about by name: ``` # Terminal 1 — the audit service: every order message, any depth nats sub "orders.>" ``` ``` # Terminal 2 — publish to two regions and a flat subject nats pub orders.us.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' nats pub orders.eu.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' nats pub orders.shipped '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` All three arrive in Terminal 1. Now restart Terminal 1 with `nats sub "orders.*.created"` and re-run Terminal 2: only the two `*.created` messages arrive; `orders.shipped` no longer matches. ## Pitfalls **`>` only works as the last token.** Use `*` for a free token in the middle: subscribe to `orders.*.created`, not the invalid `orders.>.created`, which the server rejects. **Publishers can't publish "to a wildcard."** Wildcards are a subscriber-only tool. A publisher always names one fully-qualified subject. The trap is that publishing to `orders.*.created` doesn't produce an error: the `*` is taken as a literal character, so the message lands on the odd literal subject `orders.*.created`. A service subscribed to the exact subject `orders.us.created` never sees it, while wildcard subscribers like the regional analytics on `orders.*.created` and the audit service on `orders.>` do get it — a subscription's `*` matches any single token, including a literal `*` — which makes the mistake confusing to debug. Publish the real subject (`orders.us.created`); reserve `*` and `>` for `nats sub` and the subscribe call in your client. **An over-broad `orders.>` pulls more than you want.** It's tempting to subscribe to `orders.>` and filter in code, but that subscriber then receives *every* order message at every depth, for all regions and actions, including subjects created later. Subscribe to the narrowest pattern that covers your need: `orders.*.created` for regional new-order analytics, the exact subject for a single concern. Narrow interest keeps unwanted traffic off the wire entirely, rather than having your code discard it after delivery. **Whitespace is never allowed in a subject.** A token can't contain a space, tab, or line break: on the wire, a space separates the subject from the reply subject and byte count, so a space inside a subject would be read as a boundary. The CLI and most clients catch this before sending — publishing to `orders.us created` fails with `nats: invalid subject`, and nothing is sent. A client that skips the check (nats.py's `publish` does) writes the space straight into the `PUB` line, and the server silently misroutes: `orders.us` becomes the subject and `created` a reply subject. Don't rely on the check; keep spaces out of the subject: #### CLI ``` #!/bin/bash # A subject token can't contain whitespace. On the wire, a space separates the # subject from the reply subject and byte count, so a modern client rejects the # subject before anything is sent. This publish therefore FAILS with # nats: error: nats: invalid subject # and exits non-zero -- the message never reaches the server. nats pub "orders.us created" '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # (An older client -- nats.go before v1.48.0 -- or a raw-protocol writer skips # this check and would instead misroute silently: the server would read # "orders.us" as the subject and "created" as a reply subject.) # The fix is one token per dot, no spaces: orders.us.created nats pub "orders.us.created" '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` #### C ``` // A subject token can't contain whitespace: on the wire, a space // separates the subject from the reply subject and byte count. The C // client's publish skips the check, so this call does NOT fail: the // space goes straight into the PUB line and the server silently // misroutes: "orders.us" becomes the subject and "created" a reply // subject. Nobody subscribed to orders.us.created ever sees it. const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; s = natsConnection_PublishString(conn, "orders.us created", order); printf("publish with a space returned: %s\n", natsStatus_GetText(s)); // Don't rely on a client-side check; keep spaces out of the subject. // The fix is one token per dot: orders.us.created. if (s == NATS_OK) s = natsConnection_PublishString(conn, "orders.us.created", order); ``` ## Where you are Acme's order traffic now has a shape: * Orders publish to structured subjects: `orders.created`, `orders.us.created`, `orders.eu.created`, and so on. * Regional analytics subscribes to `orders.*.created` to catch every region's new orders with one subscription. * An audit service subscribes to `orders.>` to catch the whole hierarchy. Subjects address messages; wildcards let one subscriber match a family of them. That's the addressing layer everything else in core NATS uses. ## What's next So far every message flows one way: a publisher sends, subscribers receive, and nobody sends a reply back. The next page, [Request-reply](/learn/core-nats/request-reply.md), adds a reply path: Acme builds an inventory service that *answers* a question on `orders.inventory.check`, using a reserved `_INBOX` subject you just met. ## See also * [Core Concepts → Subjects](/concepts/subjects.md) — the short overview of tokens and wildcards. * [Learn → Security](/learn/security/.md) — granting or denying access per subject and wildcard. * [Learn → Super-clusters](/learn/topologies/super-clusters.md) — how subject interest propagates across servers and regions. --- # Where to go next You started this chapter by opening one connection, then a single publish to `orders.created` with no guarantee anyone was subscribed. You end it with `notifications` and `analytics` reading `orders.created`, a regional analytics view on `orders.*.created` and an audit service on `orders.>`, an `inventory` service answering requests on `orders.inventory.check`, a `packers` queue group sharing the load on `orders.created`, and three `shipping.quote` providers answering one scatter-gather request. That's the whole Acme ORDERS world, built on core NATS alone. This page doesn't teach anything new. It collects the model you built into one place and points you at the chapters and Reference that take it further. ## The four core ideas Every pattern in this chapter circled the same four ideas. A **subject** is the address. A publisher publishes to a subject and never names a receiver. The subject is the only thing the publisher and the subscriber agree on, and wildcards let one subscription match a whole hierarchy of subjects at once. **Interest** is what makes a message move. The server keeps an in-memory graph of which subscribers want which subjects, and a published message goes to every interested subscriber, or to nobody, in which case it's discarded. With no interest, no copy is made and nothing is recorded. A **reply subject** turns one-way publish into a two-way exchange. The client subscribes to a unique `_INBOX` subject, sends it alongside the request, and the responder publishes the answer back to it. The same mechanism surfaces failure as a no-responders signal when no responder is subscribed. A **queue group** is how many subscribers share one subject's load. Each message goes to exactly one member of the group, chosen by the server, with no broker or coordinator deciding for you. The four ideas are subject, interest, reply subject, and queue group. Scatter-gather is not a fifth idea. It is a reply subject with the first-answer-wins step removed, gathering every responder instead of one. Every messaging pattern in core NATS is those four mechanics arranged differently. ## What core NATS does not store Core NATS does not store messages: a message exists only while it's in flight to a live subscriber, then it's gone. That's the **at-most-once** delivery you met on the first page, and it's a deliberate choice that keeps a publisher from blocking on slow or offline subscribers. For Acme, though, an order that arrives while the warehouse is restarting is lost. The [JetStream deep dive](/learn/jetstream/.md) adds a server-side store on top of the same subjects. Start with [Why a stream](/learn/jetstream/your-first-stream.md#why-a-stream): it picks up the Acme ORDERS world where this chapter leaves it. ## Where the details live now This chapter is unversioned and concept-first. For exact flags, defaults, and the byte-level `PUB`/`SUB`/`MSG` frames, see [Reference → Client protocol](/reference/protocols/client.md) and the [Reference root](/reference/.md). ## Sibling deep dives Core NATS is the foundation. The other chapters build directly on the four mechanics you just learned. The [Services deep dive](/learn/services/.md) takes the [request-reply](/learn/core-nats/request-reply.md) and [queue group](/learn/core-nats/queue-groups.md) patterns and wraps them in a framework that adds discovery, versioning, and built-in metrics. If you find yourself hand-rolling many request-reply responders, that's the chapter to read next. Start with [your first service](/learn/services/your-first-service.md). The [Resilient clients deep dive](/learn/resilient-clients/.md) picks up where [Connection lifecycle](/learn/core-nats/connection-lifecycle.md) left off. That page introduced the drop-and-reconnect cycle; Resilient clients tunes it for production, covering reconnect backoff and retry limits, sizing the reconnect buffer, draining a connection cleanly, and handling a slow consumer. Its pages on [reconnection](/learn/resilient-clients/reconnection.md) and [slow consumers](/learn/resilient-clients/slow-consumers.md) matter the moment you move off a single local server. The [Topologies deep dive](/learn/topologies/.md) explains how the interest graph you met in [publish-subscribe](/learn/core-nats/publish-subscribe.md) stretches across clustered and geographically separated servers, including how a queue group prefers a local member when the same group spans regions — see [Super-clusters](/learn/topologies/super-clusters.md). The [Security deep dive](/learn/security/.md) covers who's allowed to publish or subscribe to which subjects. The subject hierarchy from [subjects & wildcards](/learn/core-nats/subjects-and-wildcards.md) is also the unit of permission, so the addressing you designed is the same thing you secure. ## Where you are This is the end of the chapter. The whole arc is complete, and this page adds no new scenario state. Your local `nats-server`, the `notifications` and `analytics` subscribers, the regional analytics and audit views, the `inventory` service, the `packers` queue group, and the `shipping.quote` providers are all still as you left them. Keep experimenting, or stop the server when you're done: core NATS held nothing on disk, so there's nothing to clean up. ## What's next The most important link to follow is the [JetStream deep dive](/learn/jetstream/.md), which is what core NATS becomes when a message needs to survive. It resumes the same Acme ORDERS story right where you are now. ## Production checklist The pitfalls scattered across this chapter collapse into one list. Run through it before you put a core NATS service in front of real traffic. Each group links back to the page that explains the why. ### Connecting — see [Pitfalls](/learn/core-nats/connecting.md#pitfalls) * Open one connection per process and share it; don't connect per message or per request. * Decide about echo up front: a service that publishes and subscribes on the same subject receives its own messages unless it connects with echo off. ### Publish-subscribe — see [Pitfalls](/learn/core-nats/publish-subscribe.md#pitfalls) * Ask the server for `max_payload` and keep payloads under it; pass a reference for anything large. * Flush or drain before a short-lived publisher exits, so buffered messages reach the server. * Process messages fast enough (or hand off to a worker) so a slow subscriber isn't cut off. ### Subjects & wildcards — see [Pitfalls](/learn/core-nats/subjects-and-wildcards.md#pitfalls) * Put `>` only as the last token; use `*` for a free token in the middle. * Publish a fully-qualified subject; never publish "to a wildcard." * Subscribe to the narrowest pattern that covers your need, not a catch-all `orders.>`. * Keep whitespace out of subjects, and stay clear of `$` and `_INBOX` prefixes. ### Request-reply — see [Pitfalls](/learn/core-nats/request-reply.md#pitfalls) * Pass a timeout on every request, sized to the responder's work plus the round-trip. * Branch on no responders separately from a timeout. * Gather explicitly when more than one service may answer; don't assume exactly one reply. * Keep the reply path fast, or spread responders across a queue group. ### Queue groups — see [Pitfalls](/learn/core-nats/queue-groups.md#pitfalls) * Give every member the byte-for-byte identical queue group name. * Don't expect ordering or an even split; keep strictly-ordered work on a single subscriber. * Make a member's work safe to repeat, since core NATS does not send a message again after a hand-off. ### Scatter-gather — see [Pitfalls](/learn/core-nats/scatter-gather.md#pitfalls) * Gather replies in a loop instead of taking the first reply. * Bound the wait with a deadline so a missing responder costs only the timeout. * Treat the gathered set as unordered; compare every reply on its merits, never by arrival order. ### Message headers — see [Pitfalls](/learn/core-nats/headers.md#pitfalls) * Treat header keys as case-sensitive; read them back with the exact case you set. * Budget headers into `max_payload`; keep bulk data in the payload, not the headers. ### Subject mapping — see [Pitfalls](/learn/core-nats/subject-mapping.md#pitfalls) * Dry-run every mapping with `nats server mappings` before you deploy it. * Make weighted destinations sum to 100 unless you intend to drop a share. * Treat the partition count as part of the subject contract; changing it reshuffles every key. ### Connection lifecycle — see [Pitfalls](/learn/core-nats/connection-lifecycle.md#pitfalls) * Wire the disconnect, reconnect, and async error callbacks first, and keep them to a log line or a signal. * Log the drop from the disconnect handler and every failed attempt from the reconnect-error callback. * Treat the reconnect buffer as best-effort client memory, not a delivery guarantee. ### Debugging delivery — see [Pitfalls](/learn/core-nats/debugging-delivery.md#pitfalls) * Don't leave a `>` wire tap running against production traffic. * Remember `nats trace` publishes its own test message; it doesn't prove an earlier publish arrived. * Keep the monitoring port off the public network; it answers without authentication. ## See also * [JetStream deep dive](/learn/jetstream/.md) — the next chapter, which adds persistence to the subjects you already use. * [Services deep dive](/learn/services/.md) — request-reply and queue groups wrapped in a framework with discovery and metrics. * [Reference](/reference/.md) — every flag, default, and the wire protocol, versioned and exhaustive. --- # Deployment & Upgrades Deep Dive The [Topologies deep dive](/learn/topologies/.md) designed a shape: a three-node NATS cluster, `east`, carrying the `ORDERS` stream. This chapter takes that exact cluster and runs it in production. Topologies covered the *shapes*: one server, a cluster, a super-cluster, leaf nodes. This chapter covers the *operations* that turn a shape into a running system. Those operations are the resources a node needs, the manifests that stand it up, the SIGHUP that reloads its config without dropping a connection, the lame-duck mode that upgrades it without losing the stream, and the systemd flags that restrict its access. This is the runbook rather than the theory. Each page names the mechanism, gives the exact command or manifest you run, and links the *why* out to the chapter that covers it. **Message flow — Lame-duck drain during a rolling upgrade (animated):** A single node draining gracefully during a rolling upgrade. The operator sends SIGUSR2 to nats-0 to enter lame-duck mode; nats-0 broadcasts an updated INFO with ldm:true so connected clients learn it is leaving, hands its Raft leadership to nats-1, and lets JetStream rebalance the ORDERS replicas onto nats-1 and nats-2 at full replication. Drained, nats-0 closes connections gracefully and the warehouse client reconnects to nats-1 without losing a message; nats-0 restarts on the new version and rejoins as a non-leader. Upgrade the followers first, the meta-leader last. * operator → nats-2 * nats-2 → warehouse * nats-2 → nats1 * nats-2 → nats2 * nats-2 → nats1 (subject: ORDERS replica) * nats1 → nats2 * warehouse → nats1 * nats1 → nats-2 (subject: rejoin (follower)) ## By the end you'll have A production runbook for the same Acme ORDERS cluster you've followed through every other Learn chapter: * A **sizing baseline** for the workload: the four resources a node spends (CPU, memory, disk, file descriptors), and how an R3 stream counts against the `ORDERS` account limits. * The cluster **running on Kubernetes** as a three-replica StatefulSet with pods `nats-0`, `nats-1`, `nats-2` (the same `n1-east`, `n2-east`, `n3-east` cluster you designed in Topologies; the Kubernetes page spells out that mapping), with the `ORDERS` stream and the `shipping` and `analytics` consumers declared as CRDs. * The ability to **change config live**: split into per-account and per-region includes, reloaded with a SIGHUP, with zero downtime and no client reconnect. * A safe **rolling upgrade** procedure that walks a new server version through `nats-0..2` while the R3 ORDERS stream stays available and clients stay connected. * A **hardened** cluster: TLS on every link, the `ACME` credentials mounted, a locked-down systemd unit, and the monitor port closed to the internet. ## Who this is for You're an operator standing the cluster up for real. You've read the [Topologies deep dive](/learn/topologies/.md) and know the shapes, and you've skimmed the [JetStream](/learn/jetstream/.md) and [Security](/learn/security/.md) deep dives. This chapter reuses the `ORDERS` stream and the `ACME` operator rather than re-introducing them. This chapter is deliberately the runbook half. It divides the work with four sibling chapters, and each owns part of it: * **Which shape to deploy** is [Topologies](/learn/topologies/.md). This chapter assumes the three-node ORDERS cluster shape is already decided. * **How Raft and replication work underneath** is [Clustering](/learn/clustering/.md). This chapter *triggers* a leadership transfer during an upgrade; it doesn't explain how Raft elects a leader. * **The auth model** (operators, accounts, users, JWTs) is [Security](/learn/security/.md). This chapter *mounts* the credentials and *turns on* TLS; the model is taught there. * **What to watch once it's live** is [Monitoring](/learn/monitoring/.md). This chapter *sets up* the cluster; monitoring watches it. If a sentence here would be at home verbatim in one of those four, it belongs there, and this chapter links to it instead. ## How to read it Each page introduces at most two new concepts and carries the same cluster forward. You [size it](/learn/deployment/sizing-and-resources.md), [deploy it](/learn/deployment/kubernetes.md), [edit its config live](/learn/deployment/config-management.md), [upgrade it](/learn/deployment/rolling-upgrades.md), and [harden it](/learn/deployment/hardening.md). The node names (`n1-east`/`n2-east`/`n3-east`), the `ORDERS` stream, the `order-svc` publisher, and the `ACME` operator stay fixed throughout, so you keep a mental picture of one Acme system getting production-ready rather than a new example each time. Every page covers only the configuration keys this deployment needs. The full set of server configuration options lives in [Reference → Configuration](/reference/config/.md); each page points there for the exhaustive knob list. ## Map | Page | What you learn | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | [Deployment & Upgrades](/learn/deployment/.md) | What this chapter operates and the order of the runbook | | [Sizing & resources](/learn/deployment/sizing-and-resources.md) | The four resources a node spends, and how an R3 stream counts against account limits | | [Kubernetes](/learn/deployment/kubernetes.md) | Stand the cluster up as a StatefulSet with the NATS Helm chart, and declare streams as CRDs | | [Config management](/learn/deployment/config-management.md) | Split config into includes and reload it live with a SIGHUP: no downtime, no reconnect | | [Rolling upgrades](/learn/deployment/rolling-upgrades.md) | Use lame-duck mode and the right upgrade order to roll a new version through the cluster | | [Hardening](/learn/deployment/hardening.md) | TLS on every link, mounted credentials, and a locked-down systemd unit | | [Where to go next](/learn/deployment/where-next.md) | The production checklist and a map of what's beyond running the cluster | ## Prerequisites You'll need: * A running three-node ORDERS cluster, or the willingness to stand one up while you read. The [Topologies deep dive](/learn/topologies/.md) builds it step by step. * The `nats` CLI installed and pointed at the cluster, for the inspection commands (`nats account info`, `nats server info`, `nats stream info`) that anchor each page. * For the Kubernetes pages, a cluster you can `kubectl` against and `helm` installed. The [NATS Helm chart](/learn/deployment/kubernetes.md) and the NACK controller are the canonical path this chapter follows. Open a terminal and turn to [Sizing & resources](/learn/deployment/sizing-and-resources.md). ## See also * [Topologies deep dive](/learn/topologies/.md) — the shape this chapter runs. * [Clustering & Replication](/learn/clustering/.md) — the Raft mechanics this chapter triggers but doesn't teach. * [Security deep dive](/learn/security/.md) — the auth model behind the credentials this chapter mounts. --- # Config management The `ORDERS` cluster is running on Kubernetes as the three pods `nats-0`, `nats-1`, `nats-2`. Now the inevitable happens: you need to change something. Raise an account limit, rotate a TLS certificate, add a user for a new service. One option is to restart the process. The better option is to **reload** it: apply the new config to the running server without dropping a connection. Two mechanisms make live config change safe. First, an **include** splits one giant config file into small files you can own per account and per region. Second, **live reload** applies a changed file to the running server through a SIGHUP, after a dry-run validates it. Together they let you change `order-svc`'s limits or the cluster's certificates while `ORDERS` keeps flowing. ## Includes split the config A single `nats.conf` for a production cluster grows large. Put accounts, users, per-region routing, TLS paths, and JetStream limits all in one file, and it becomes hard to review and hard to hand to different owners. An include pulls another file into the config at the point of the directive. The keyword is `include`, and the path is **relative to the directory of the config file that contains it**, not to the directory you launch the server from: ``` # /etc/nats-config/nats.conf — the main config for nats-0..2 # The Helm chart generates the identity keys — server_name and the cluster # routes — uniquely per pod. You own the includes below. server_name: $SERVER_NAME listen: "0.0.0.0:4222" cluster { name: "east" listen: "0.0.0.0:6222" routes: [ "nats://nats-0.nats-headless:6222" "nats://nats-1.nats-headless:6222" "nats://nats-2.nats-headless:6222" ] } jetstream { store_dir: "/data/jetstream" } # Pull in the shared TLS material and the per-region routing files. include "tls.conf" include "regions/us.conf" include "regions/eu.conf" ``` A JetStream cluster won't boot without both `server_name` and `routes`, so the chart fills them in per pod; a dry-run that reports the file valid still can't start without them (see [Validate, then reload](#validate-then-reload)). The per-region files hold the routing each region owns, so a region's configuration can be reviewed and changed on its own without touching the main config or another region's file. Because the path is relative to the config file's directory, `include "regions/us.conf"` resolves to `/etc/nats-config/regions/us.conf`. Launch the server from `/root` or from `/`, and it still resolves the same way. The [Pitfalls](#pitfalls) section shows what happens when you forget this. An account splits into its own file the same way. This cluster runs under the `ACME` operator, which keeps accounts as JWTs in a resolver — you edit those with `nats account edit` and push them live, not through a config include. A deployment that keeps accounts in the config instead wraps each one in an `accounts` block so the server parses it: ``` # /etc/nats-config/accounts/orders.conf — config-based accounts (no operator) # Reloadable: edit and SIGHUP. accounts { ORDERS: { jetstream: { max_memory: 256MB, max_file: 10GB } users: [ { user: "order-svc", password: $ORDER_SVC_PASS } ] } } ``` The `$ORDER_SVC_PASS` reference is unquoted on purpose: an unquoted `$NAME` token resolves a config-defined variable and then an environment variable, and an unset one is a parse error the dry-run catches. Quoting it (`"$ORDER_SVC_PASS"`) would store the literal string `$ORDER_SVC_PASS` instead — which is why bcrypt hashes, full of `$`, are the values you do quote. ## Reloadable versus non-reloadable keys Not every key can change on a running server. Knowing which keys are reloadable determines whether a change is a zero-downtime reload or requires a restart. **Reloadable** keys take effect on a reload, in place, with no reconnect: * Account, user, and permission definitions: add `analytics-reader`, tighten `order-svc`'s subjects. * Connection and message limits: `max_connections`, `max_payload`, `max_control_line`. (Not `max_subscriptions` — a reload rejects a change to it.) * Most JetStream account limits, and the `jetstream` enable flag itself (a reload turns JetStream on or off). * TLS certificate and key paths: the server re-reads the files when it reloads. * Cluster routes and logging settings. (Gateway routes are not reloadable; a reload can only refresh their TLS material.) **Non-reloadable** keys need a process restart, because they define the server's identity: * `port` / `listen`: the address the server binds. * The cluster's listen host and port: the address peers route to. * The JetStream `store_dir`: the server won't move its storage directory on a reload. A reload can change *policy* (who connects, what they may do, how much they may store), but it can't change *identity* (the addresses the server and its cluster bind, or where JetStream keeps its data). Changing one of those requires a [rolling upgrade](/learn/deployment/rolling-upgrades.md) rather than a reload. The full set of reloadable keys is in [Reference → Configuration](/reference/config/.md); here we cover only the ones this deployment reloads. ## Validate, then reload A reload that fails is a problem only if it leaves the server in a broken state. NATS validates the new config first, and on a parse or validation failure the old config stays active. The reload is atomic: either the new config applies cleanly, or nothing changes. You still validate before you signal, because catching a typo at your terminal is preferable to catching it in the server log. The dry-run parses a config file and exits without starting a server: ``` nats-server -c /etc/nats/nats.conf -t ``` A clean config prints `nats-server: configuration file ... is valid` and exits zero. A broken one prints the parse error and the offending line, and exits non-zero, so you can gate the reload on it in a script. The dry-run checks syntax and validation, not whether the server can start: a JetStream cluster missing `server_name` or `routes` passes `-t` yet still fails to boot, which is why the chart supplies those keys. With the config validated, trigger the reload. The mechanism is a **SIGHUP** to the `nats-server` process. The systemd unit wires `systemctl reload` to send exactly that signal: ``` # Validate the edited config, then reload only if it is valid. nats-server -c /etc/nats/nats.conf -t && systemctl reload nats-server ``` The server re-reads its config, applies the reloadable changes in place, and logs `Reloaded server configuration`. Open connections, including `order-svc`'s, stay up the whole time, and no client reconnects. That is the advantage of reload over restart. ## The reloader sidecar on Kubernetes On Kubernetes there's no shell to run `systemctl reload` in. The config arrives as a ConfigMap mounted into the pod, and editing the ConfigMap updates the file on disk, but nothing tells `nats-server` to re-read it. That's the job of the **config reloader sidecar**. It runs alongside `nats-server` in each of the `nats-0..2` pods, watches the mounted config file with inotify, and on any change reads the server PID from `/var/run/nats/nats.pid` and sends it a SIGHUP. The NATS Helm chart includes the reloader by default, so a ConfigMap edit becomes a live reload across all three pods automatically: ``` # values.yaml — the reloader ships enabled in the Helm chart reloader: enabled: true ``` The animation below traces the whole path: a config file change, the reloader detecting it and sending the SIGHUP, the server reloading in place, and the `order-svc` connection staying open while the peers pick up the new server info. **Message flow — Config reload without a reconnect (animated):** A config change flows from file to reloaded server without dropping a connection. A new nats.conf is written — a ConfigMap update or an on-disk edit. The reloader sidecar, watching the file with inotify, sees the change and sends SIGHUP to the nats-server process, which re-reads and applies the new config in place, with no restart. The order-svc client connection stays open the whole time, and the server gossips its updated INFO to its cluster peers. * nats.conf → reloader * reloader → nats-server * nats-server → nats-server (subject: reload in place) * nats-server → order-svc The reloader retries if the server is briefly unreachable (30 retries by default, four seconds apart), so a reload issued during a momentary blip still lands. Where a node's filesystem doesn't deliver inotify events, add the reloader's `--force-poll` flag through `reloader.merge`, which replaces the container's args wholesale. ## Secrets as mounted files Credentials and TLS material never belong inline in the config. They're mounted as **files**: a Kubernetes Secret projected into the pod, or a creds file on disk for a systemd deployment. The config references the path; the secret lives outside the ConfigMap. ``` # A TLS path the config points at — the file itself is a mounted secret. tls { cert_file: "/etc/nats-certs/server-cert.pem" key_file: "/etc/nats-certs/server-key.pem" ca_file: "/etc/nats-certs/ca.pem" } ``` This split matters for reload. Rotating a certificate means replacing the file behind `cert_file` and `key_file`, then reloading; the server re-reads the certificates when it reloads, and new handshakes present the new certificate while open connections keep their session. The config text never changes; only the file it points at does. The auth model behind these credentials (operators, accounts, JWTs) is taught in [Security](/learn/security/.md); here you only mount and reference them. ## Pitfalls A few mistakes can turn a routine reload into an outage. Each is scoped to this page's two ideas: includes and live reload. **Include paths are relative to the config file, not your shell.** The `include "regions/us.conf"` directive resolves against the directory of the file that contains it, so a server launched from `/root` and one launched from `/etc/nats-config` both find `/etc/nats-config/regions/us.conf`. The trap is assuming the path is relative to your current directory, moving the main config, and watching the includes fail to resolve. Use absolute paths when in doubt, and validate before you trust it. **A reload during a rebalance can interrupt a leadership transfer.** If you SIGHUP a node while JetStream is moving `ORDERS` replicas or handing off Raft leadership, the reload competes with that work. Don't reload mid-rebalance. Wait for the cluster to settle, then apply the change. That's the same graceful window the [rolling upgrades](/learn/deployment/rolling-upgrades.md) page builds its procedure around. **Lowering a store limit on reload does not evict data already stored.** Drop the `ORDERS` account's `max_file` below what the stream already holds, reload, and the existing messages stay, but new writes fail until an admin trims the stream back under the limit. The reload changes the *ceiling*, not the *contents*. Raise limits freely; lower them only after checking what the stream currently stores. **Rotating a TLS certificate late fails every new handshake.** The server re-reads `cert_file` and `key_file` when it reloads, so a reload swaps the certificate cleanly and connections already open keep the session they negotiated regardless of the old cert's expiry. What breaks is letting the certificate expire before you rotate: once it's expired the server presents a dead cert, and every new connection and every client reconnect fails its TLS handshake until you drop in a valid file and reload. Do: rotate well before expiry, and track the certificate's expiry date so the swap is never an emergency. Don't: wait for the alert that the cert already expired. The auth model behind these certificates lives in [Security](/learn/security/.md); here the rule is operational: replace the file, reload, and rotate with margin to spare. The do-this for all four is the same: never SIGHUP an unvalidated config. The dry-run parses the file and exits without touching the running server, so a typo never reaches it. Gate the reload on it: ``` # Validate first; only signal the running server if the config is valid. if nats-server -c /etc/nats/nats.conf -t; then systemctl reload nats-server echo "reload sent" else echo "config invalid — running server left untouched" >&2 exit 1 fi ``` Because the server also validates internally and keeps the old config on failure, even a reload that slips through the dry-run can't leave the cluster broken. The worst case is that no change applies, rather than a half-applied one. ## Where you are The `ORDERS` config is now split into per-account and per-region includes, each ownable on its own. You can change a limit, add a user, or rotate a certificate, validate it with a dry-run, and apply it to the running cluster with a SIGHUP: no downtime, no client reconnect. On Kubernetes the reloader sidecar does the signaling for you whenever the ConfigMap changes. What a reload can't do is change the server's identity: its ports, the addresses its cluster binds, or where JetStream keeps its data. Those need a process restart, rolled through the cluster one node at a time. ## What's next That controlled restart is the next mechanism: a **rolling upgrade**. Lame-duck mode drains a node gracefully, transfers its Raft leadership, and lets the next version take its place, all while the R3 `ORDERS` stream stays available and clients stay connected. Continue to [Rolling upgrades](/learn/deployment/rolling-upgrades.md). ## See also * [Reference → Configuration](/reference/config/.md) — the full set of reloadable and non-reloadable keys. * [Rolling upgrades](/learn/deployment/rolling-upgrades.md) — the procedure for the non-reloadable changes a SIGHUP can't apply. * [Security](/learn/security/.md) — the auth model behind the credentials this page mounts as files. --- # Hardening The cluster is sized, deployed, configurable, and upgradable. Right now it's also unsecured. Routes between `n1-east`, `n2-east`, and `n3-east` run in plaintext, the monitor port responds to any request, and the `nats-server` process can read and write the whole filesystem. This page closes those gaps. It puts TLS on every link the cluster connects over, and it wraps the process in a hardened systemd unit that removes everything it doesn't need. Both are operator-side controls. The auth model that issues the credentials (the operator `ACME`, the accounts, the users) is taught in [Security](/learn/security/.md); here you *mount* those credentials and *turn on* the transport security around them. ## TLS on every link A NATS server connects to more than one kind of peer, and each kind has its own TLS block. There are three: * **Client TLS**: the top-level `tls {}` block, securing client connections like `order-svc` publishing to `orders.created`. * **Cluster TLS**: a separate `cluster { tls {} }` block, securing the routes between `n1-east`, `n2-east`, and `n3-east`. * **Gateway TLS**: a `gateway { tls {} }` block, securing supercluster links. These blocks are independent. Turning on TLS for clients leaves the cluster routes plaintext until you configure the cluster block too. This per-link split is a common hardening mistake. An operator secures clients and sees the encrypted client connection, then ships a cluster whose inter-node Raft traffic, including replicated `ORDERS` data, is still unencrypted. Here's a server config that secures both the client link and the cluster link. Client TLS sits at the top level; cluster TLS sits inside the `cluster {}` block: ``` # nats.conf on n1-east — TLS for clients AND for cluster routes listen: "0.0.0.0:4222" # Client-facing TLS tls { cert_file: "/var/lib/nats/certs/server-cert.pem" key_file: "/var/lib/nats/certs/server-key.pem" ca_file: "/var/lib/nats/certs/ca.pem" verify: true } cluster { name: "east" listen: "0.0.0.0:6222" routes: [ "nats://n2-east:6222" "nats://n3-east:6222" ] # Cluster-route TLS — separate from the client block above tls { cert_file: "/var/lib/nats/certs/server-cert.pem" key_file: "/var/lib/nats/certs/server-key.pem" ca_file: "/var/lib/nats/certs/ca.pem" verify: true } } ``` In the top-level `tls {}` block, `verify: true` is what makes the client link **mTLS** (mutual TLS): without it the server proves itself to the client but never checks the client's certificate; with it every client must present a certificate that chains to `ca_file`. Cluster and gateway routes work differently — the server forces mutual verification on them whether or not you set `verify`, because each end acts as both client and server on the route. So the stray-node protection on `east` holds either way: a node can't join just by knowing the route address; it must present a certificate that chains to `ca_file`. Use `verify_and_map: true` on the client block when you want the client certificate's subject to *be* the NATS user; the certificate identity mechanism is covered in [Security → Encryption & TLS](/learn/security/encryption.md). The server re-reads `cert_file` and `key_file` when it reloads its configuration, so rotating certificates is drop-in-new-files plus a SIGHUP, not a restart. After the reload, new handshakes present the new certificate while existing connections keep their session. Drop the new files in place and send the SIGHUP you learned on the [config management](/learn/deployment/config-management.md) page: ``` # After dropping new cert/key files at the same paths, reload in place. # Existing connections keep their session; new handshakes pick up the new cert. systemctl reload nats-server ``` The full set of TLS keys (cipher suites, curve preferences, and `pinned_certs` for certificate pinning) is documented in [Reference → TLS](/reference/config/tls/.md). We use only `cert_file`, `key_file`, `ca_file`, and `verify` here. ## Mount the credentials and verify the link TLS encrypts the link, and credentials identify the user on it. The `ACME` operator from the [Security deep dive](/learn/security/operator-mode.md) issues a `.creds` file for the `order-svc` user. You don't create it here; you mount it as a file the server and client can read. On Kubernetes that file is a Secret; on a host it lives under a path only the `nats` user can read. With the CA file and the creds file both available, one publish confirms the whole hardened path works. The client trusts the CA (so the link encrypts), presents the `order-svc` credentials (so the server authenticates the user), and publishes one canonical order to `orders.created`: #### CLI ``` #!/bin/bash # Prove that auth and TLS are both live on the hardened ORDERS cluster. # This single command exercises the whole hardened path: the client trusts # the CA that signed the server certificate (--tlsca), so the link encrypts; # and it presents the ACME-issued order-svc credentials (--creds), so the # server authenticates the user before accepting the publish. # # Connect as order-svc in the ORDERS account and publish one canonical order. nats pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --server tls://nats.acme.internal:4222 \ --tlsca /etc/nats/certs/ca.pem \ --creds /etc/nats/creds/order-svc.creds # Expected on a hardened cluster: the publish succeeds, which proves three # things at once -- the TLS handshake completed (encryption is live), the # CA was trusted (the server identity verified), and the credentials were # accepted (the user authenticated). # # Drop --creds and the server rejects the connection with an authorization # error. Drop --tlsca (or point the client at nats:// instead of tls://) and # the handshake fails before authentication is even attempted. Either failure # is the hardened cluster doing its job. ``` #### C ``` // Prove that auth and TLS are both live on the hardened ORDERS // cluster. The client trusts the CA that signed the server // certificate, so the link encrypts; and it presents the // ACME-issued order-svc credentials, so the server authenticates // the user before accepting the publish. if (s == NATS_OK) s = natsOptions_SetURL(opts, (url == NULL) ? "tls://nats.acme.internal:4222" : url); if (s == NATS_OK) s = natsOptions_SetSecure(opts, true); if (s == NATS_OK) s = natsOptions_LoadCATrustedCertificates(opts, "/etc/nats/certs/ca.pem"); if (s == NATS_OK) s = natsOptions_SetUserCredentialsFromFiles(opts, "/etc/nats/creds/order-svc.creds", NULL); // Connect as order-svc in the ORDERS account and publish one // canonical order. Success proves three things at once: the TLS // handshake completed, the server identity verified against the // CA, and the credentials authenticated the user. if (s == NATS_OK) s = natsConnection_Connect(&conn, opts); if (s == NATS_OK) s = natsConnection_PublishString(conn, "orders.created", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"); if (s == NATS_OK) s = natsConnection_FlushTimeout(conn, 2000); if (s == NATS_OK) printf("published one order over the hardened path\n"); ``` The payload is the same Acme order shape you've carried through every chapter: ``` {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} ``` If the publish succeeds, encryption and authentication are both on. If it fails at the handshake, TLS is misconfigured; if it fails with an authorization error, the credentials are wrong or unmounted, and this one command distinguishes the two cases. ## A hardened systemd unit TLS protects the cluster from the network, while the systemd unit protects the host from the cluster by running `nats-server` as an unprivileged, sandboxed process that can access only what it needs. The NATS distribution ships a hardened unit (`nats-server-hardened.service`), which is the second and last concept of the page, and you adapt it rather than write it from scratch. The unit does three jobs: it raises the file descriptor limit so a busy cluster doesn't run out of sockets, it sandboxes the filesystem and kernel surface so a compromised process can't escape, and it drops every Linux capability the server doesn't need: ``` # /etc/systemd/system/nats-server.service (hardened) [Service] ExecStart=/usr/local/bin/nats-server -c /var/lib/nats/nats.conf ExecReload=/bin/kill -s HUP $MAINPID # 1. File descriptors: 2 FDs per stream plus gossip and client sockets. # A large cluster exhausts the default 1024 quickly. LimitNOFILE=800000 # 2. Filesystem and kernel sandbox. ProtectSystem=strict ReadWritePaths=/var/lib/nats MemoryDenyWriteExecute=true ProtectKernelTunables=true ProtectProc=invisible PrivateDevices=true # 3. Drop all capabilities and filter syscalls down to a server profile. CapabilityBoundingSet= SystemCallFilter=@system-service ~@privileged ~@resources ``` Two flags matter most for an operator. `LimitNOFILE=800000` lifts the file descriptor (FD) ceiling far above the default 1024. Each stream costs roughly two FDs, and inter-node gossip plus client sockets add many more, so a real `ORDERS` cluster needs the headroom. `ProtectSystem=strict` mounts the entire filesystem read-only *except* the paths in `ReadWritePaths`, which is why the TLS certificates and the JetStream store both live under `/var/lib/nats`, a path the server is explicitly allowed to write. The full set of sandboxing directives (`PrivateUsers`, `RestrictNamespaces`, `ProtectClock`, and the rest) ships in the distribution's hardened unit; copy that file and adjust only `ExecStart`, `ReadWritePaths`, and `LimitNOFILE` for your layout. The server-configuration keys these flags wrap are documented in [Reference → Configuration](/reference/config/.md). ## Close the monitor port The server listens on four ports: **4222** for clients, **6222** for cluster routes, **7222** for gateways, and **8222** for the HTTP monitor. The first three carry TLS once you configure it. The monitor port does not: it serves `/varz`, `/healthz`, and the rest in plaintext, and its `/varz` output leaks the server version, connected-client count, and memory usage to anyone who can reach it. On a host, bind it to localhost so only an on-host agent can read it, and let a firewall handle the rest: ``` # nats.conf — monitor reachable only from the host itself http: "127.0.0.1:8222" ``` Don't do this on the chapter's Kubernetes deployment. The kubelet's startup, readiness, and liveness probes connect to the pod's IP, not its loopback, so a `127.0.0.1` bind fails every probe and the pods never go ready. There, keep the chart's default bind and restrict the port with a NetworkPolicy instead. ``` # Firewall: clients in, cluster routes between nodes only, monitor never. # Open 4222 to clients, 6222 to the other east nodes, deny 8222 outright. ufw allow 4222/tcp ufw allow from 10.0.0.0/24 to any port 6222 proto tcp ufw deny 8222/tcp ``` What to actually scrape from `/varz` and `/healthz`, and how to read it, is the monitoring discipline taught in [Monitoring → Monitoring endpoints](/learn/monitoring/monitoring-endpoints.md). At the hardening stage, you only need to make sure the port isn't open to the world. ## Pitfalls A few traps affect teams the first time they harden a NATS cluster. Each one comes from this page's work: the sandboxed systemd unit, and locking down the ports that TLS now protects. **`ProtectSystem=strict` blocks writes outside `ReadWritePaths`, not reads.** The sandbox mounts the filesystem read-only, so reading rotated certificates from `/etc/nats-certs` on a SIGHUP works fine: an external process writes the new files, and the server only reads them. What genuinely fails under `strict` is the server *writing* to a path not listed — the JetStream `store_dir` (and the pid and ports-file directories) must sit inside `ReadWritePaths`, or the server can't come up. Keep `ReadWritePaths` covering the JetStream store; certificates need only read access, which `strict` already allows. **A memory cap set below the working set gets the process OOM-killed under load.** The hardened unit can cap the process with `MemoryMax=` (systemd) or `GOMEMLIMIT` (the Go runtime). Neither reserves memory at startup: `max_memory_store` is an accounting limit checked as messages arrive, and `GOMEMLIMIT` is a soft target that only makes GC more aggressive. The failure shows up at runtime — set `MemoryMax` below what the node actually uses (memory-store data plus connection and route buffers plus GC headroom) and the cgroup OOM-kills the process once usage grows, which systemd logs as `Main process killed (oom-kill)`. Set `MemoryMax` above expected use, and `GOMEMLIMIT` somewhat below `MemoryMax` so GC reins memory in before the hard cap. ``` # Cap the process above what it uses at peak, not at the config number. # With jetstream { max_memory_store: 4Gi }, size for the store plus buffers. MemoryMax=6G Environment=GOMEMLIMIT=5500MiB ``` **The monitor port exposed to the internet leaks operational detail.** A reachable `:8222/varz` hands out the server version, the client count, and the memory footprint, which is useful information for an attacker performing reconnaissance. Don't leave `http:` bound to `0.0.0.0`. Bind it to `127.0.0.1` and let the firewall deny 8222 from everywhere else. **A firewall blocking cluster port 6222 leaves nodes unable to form quorum.** Hardening locks ports down, and it's easy to deny 6222 to the world while forgetting to allow it *between* the east nodes. When that happens the route handshakes never complete: `n1-east`, `n2-east`, and `n3-east` each come up alone, can't reach each other, and show as orphans that never join the `east` cluster. Don't deny 6222 globally. Allow it explicitly from the cluster subnet, and only then deny it elsewhere: ``` # Allow cluster routes between the east nodes; deny 6222 from anywhere else. ufw allow from 10.0.0.0/24 to any port 6222 proto tcp ufw deny 6222/tcp ``` Once the ports are open and TLS is on every link, the same authenticated publish from [Mount the credentials and verify the link](#mount-the-credentials-and-verify-the-link) confirms the whole hardened path end to end. A successful publish confirms the client link encrypts and the credentials authenticate. Run that check from a node on the cluster network to confirm the routes came up. If a node still shows as an orphan after you allow 6222, its route handshake is failing on the firewall or on a missing or mismatched cluster certificate. ## Where you are The `ORDERS` cluster now runs locked down. TLS protects every link: the client connection for `order-svc` and the cluster routes that replicate the stream between `n1-east`, `n2-east`, and `n3-east`. The `ACME` credentials are mounted as files, and one publish confirms auth and encryption are both on. The process runs under a hardened systemd unit that raises the FD limit and sandboxes the filesystem, and the monitor port responds only from localhost. That completes the runbook. The cluster is sized, deployed on Kubernetes, configurable without downtime, upgradable in place, and hardened. ## What's next The last page recaps the whole runbook (size, deploy, configure, upgrade, harden) and collects every page's pitfalls into a single production checklist you can run down before you call the cluster ready. Continue to [Where to go next](/learn/deployment/where-next.md). ## See also * [Security → Encryption & TLS](/learn/security/encryption.md) — the certificate-as-identity model behind `verify_and_map`. * [Security → Operator mode](/learn/security/operator-mode.md) — how the `ACME` operator issues the credentials this page mounts. * [Monitoring → Monitoring endpoints](/learn/monitoring/monitoring-endpoints.md) — what to scrape from the monitor port once it's locked to localhost. --- # Kubernetes The [previous page](/learn/deployment/sizing-and-resources.md) gave the ORDERS workload a sizing baseline: an R3 file stream that fits a 10 Gi volume, with `order-svc` running comfortably in \~128 Mi. This page stands that cluster up on Kubernetes. The three-node cluster `east` you built in Topologies (`n1-east`, `n2-east`, `n3-east`) becomes three pods. On Kubernetes those pods are named by ordinal: **`nats-0`, `nats-1`, `nats-2`**. They're the same three nodes carrying the same R3 `ORDERS` stream; only the names change. The rest of this chapter uses the pod names, because lame-duck, disruption-budget, and rolling-update language is all ordinal-based. This page introduces two ideas: the NATS Helm chart, which deploys the cluster as a StatefulSet, and the NACK controller, which lets you declare the `ORDERS` stream and its consumers as Kubernetes resources instead of running CLI commands. ## The NATS Helm chart deploys a StatefulSet A NATS cluster is stateful, unlike a stateless web app. Each node owns a slice of the R3 stream on its own disk, and node identity has to survive a restart: `nats-1` must come back as `nats-1`, with its volume, not as a fresh replica. That requirement rules out a Deployment and calls for a **StatefulSet**: the Kubernetes workload that gives each pod a stable name, a stable network identity, and a stable volume across restarts. The **NATS Helm chart** installs that StatefulSet for you. Its `values.yaml` describes the cluster, and Helm renders the StatefulSet, the headless service, the ConfigMap, and the probes from it. A minimal values file for the three-node ORDERS cluster: ``` # values.yaml — the three-node ORDERS cluster config: cluster: enabled: true replicas: 3 # nats-0, nats-1, nats-2 jetstream: enabled: true fileStore: pvc: size: 10Gi # the sizing baseline from the previous page ``` Install it with the chart's release name `nats`, which is what gives the pods their `nats-N` names: ``` helm repo add nats https://nats-io.github.io/k8s/helm/charts/ helm install nats nats/nats -f values.yaml ``` Two things are worth naming. The chart already sets `podManagementPolicy: Parallel` on the StatefulSet, so you don't add it to `values.yaml`. It starts all three pods together instead of one at a time; the ordered default would wait for each pod to become ready before starting the next, and that deadlocks a NATS cluster, because no single node is ready until it can see its peers. Parallel lets them find each other and form the cluster in one pass. The second is the **headless service**. It gives each pod a stable DNS name of the form `nats-0.nats-headless.default.svc.cluster.local`, and the nodes use those names to route to each other. Clients don't dial the headless service: the chart also creates a regular ClusterIP service named `nats`, and that's the address a client inside the Kubernetes cluster connects to — the one readiness pulls a not-ready pod out of. The chart has many more values, with sensible defaults for almost all of them. Every server configuration option is documented in [Reference → Configuration](/reference/config/.md); here we cover only the values this deployment needs. ## The three probes on the monitor port Kubernetes decides a pod's lifecycle from its probes, and a JetStream node has three distinct states worth probing. The chart wires all three against the monitor port (8222) and the `/healthz` endpoint: ``` # rendered into the StatefulSet by the chart — shown for reference startupProbe: httpGet: { path: /healthz, port: 8222 } failureThreshold: 90 # ~900s window for the node to boot + sync readinessProbe: httpGet: { path: "/healthz?js-server-only=true", port: 8222 } livenessProbe: httpGet: { path: "/healthz?js-enabled-only=true", port: 8222 } ``` The **startup probe** guards the boot window. A node rejoining a cluster may need to catch its stream replicas up from its peers before it's useful, and that can take minutes. The generous failure threshold keeps Kubernetes from killing a pod that's doing legitimate startup work. The **readiness probe** asks whether this server is ready to serve clients, so Kubernetes takes a not-ready pod out of the service rotation. The **liveness probe** asks only whether JetStream is enabled at all; it's the last resort that restarts a truly wedged process. What each `/healthz` query parameter reports, and which advisories to alert on, belongs to [Monitoring](/learn/monitoring/jetstream-health.md). Here the probes are just the configuration that reports each pod's state to the StatefulSet. ## Declare streams as CRDs with the NACK controller You could now open a shell in the chart's `nats-box` pod and run `nats stream add ORDERS` by hand. That works, but it's imperative: the stream exists because someone ran a command, and nothing brings it back if it's deleted. Kubernetes prefers declarative state, and NATS has a controller for exactly that. The **NACK controller** runs in the cluster and reconciles **CRDs** (Custom Resource Definitions) against the NATS cluster. You write a `Stream` resource describing the `ORDERS` stream you want, apply it with `kubectl`, and the controller calls the JetStream API to make the cluster match. (The term *controller* here is the Kubernetes piece; the security *operator* `ACME` is a different thing, taught in [Security](/learn/security/.md).) Here's the `ORDERS` stream as a CRD, the same R3 file stream from every other chapter, now declarative: ``` # orders-stream.yaml — the ORDERS stream as a declarative resource apiVersion: jetstream.nats.io/v1beta2 kind: Stream metadata: name: orders spec: name: ORDERS subjects: ["orders.>"] storage: file replicas: 3 ``` And the two consumers that read it: the `shipping` pull consumer and the `analytics` consumer filtering `orders.shipped`. ``` # orders-consumers.yaml — consumers as declarative resources apiVersion: jetstream.nats.io/v1beta2 kind: Consumer metadata: name: shipping spec: streamName: ORDERS durableName: shipping --- apiVersion: jetstream.nats.io/v1beta2 kind: Consumer metadata: name: analytics spec: streamName: ORDERS durableName: analytics filterSubject: orders.shipped ``` Apply them and the controller does the rest: ``` kubectl apply -f orders-stream.yaml -f orders-consumers.yaml ``` The controller watches the CRD, calls the JetStream API on the cluster, creates the R3 stream across `nats-0..2`, and writes the result back into the resource's `.status`. From then on the stream's desired state lives in version control, not in someone's terminal history. **Message flow — NACK reconcile and self-heal (animated):** The NACK control loop reconciling a Stream CRD, then self-healing drift. An admin runs kubectl apply on a Stream CRD describing ORDERS with 3 replicas, storing the desired state in the Kubernetes API. The NACK controller watches the API, sees the new resource, and calls the JetStream API on the NATS cluster to create the R3 ORDERS stream across nats-0, nats-1, and nats-2, then writes the result back to the CRD's .status. When someone deletes the stream by hand, reality drifts from the declared state; the controller detects the deletion on its next reconcile and recreates the R3 stream automatically. * kubectl → K8s API / etcd * K8s API / etcd → NACK controller * NACK controller → nats-1 * NACK controller → K8s API / etcd * NACK controller → nats-1 (subject: stream deleted — drift!) The reconcile is a closed loop. If someone deletes the stream by hand, the controller detects the drift and recreates it from the CRD. The declared state is the source of truth, and the controller keeps changing the cluster back to match it. The `Stream` resource has many more fields: `maxBytes`, `maxAge`, `placement`, `mirror`, `sources`, and the rest of the stream configuration. The full set is documented in [Reference → Configuration](/reference/config/.md); this page sets only the fields the ORDERS stream needs. ## Pitfalls A few traps catch teams the first time they run NATS on Kubernetes. Each is scoped to this page's two concepts: the StatefulSet and the CRDs. **A pod stuck Pending is usually an unbound volume.** A StatefulSet pod can't start until its persistent volume is bound, and on a cluster with no default storage class the claim hangs forever: `nats-0` sits in `Pending` and the whole cluster waits on it. The Helm chart provisions volumes through `volumeClaimTemplates` by default, so each pod gets its own claim; the fix when a pod hangs is to confirm a storage class exists, not to delete and retry the pod. **A ConfigMap edit does not reload the server by itself.** Editing the config the NATS Helm chart renders changes the file, but the running `nats-server` keeps its old config until something sends it a SIGHUP. The chart includes the **config reloader sidecar** (the `nats-server-config-reloader` container, enabled by default) to send that signal for you; without it, a config change sits inert until the pod restarts. Turning a ConfigMap change into a live reload is its own subject, covered on [Config management](/learn/deployment/config-management.md). **Replica catch-up shows up in the startup probe, not readiness.** The chart points the readiness probe at `/healthz?js-server-only=true`, which checks only that the server and its JetStream subsystem are up — it deliberately skips every stream, consumer, and meta-assignment check. So a pod catching its R3 replicas up after a restart still reports ready and keeps serving clients. The plain `/healthz` the startup probe uses is the strict one: it checks the meta layer and every stream and consumer asset, which is why the chart sets its `failureThreshold` to 90 — a wide window for a rebooting node to finish syncing before Kubernetes gives up on it. To widen that window further, override the probe fields through the chart's container `merge`/`patch`; there's no named `failureThreshold` value. **Never mix CLI and CRD management of the same stream.** The NACK controller reconciles the `Stream` CRD against the cluster, but what it enforces depends on its mode. By default it re-creates a stream that's been deleted (it notices on its \~30-second resync), yet it does *not* revert a manual `nats stream edit`: a config change sticks until the CRD itself next changes. Run the controller in its `--control-loop` mode and it also enforces config drift, reverting manual edits on about a one-minute cycle. Either way, pick one owner per stream — let the CRD own it or the CLI own it, never both. Whichever owner you pick, verify the stream the controller created is actually the R3 stream you declared. Open a shell in `nats-box` and read it back: #### CLI ``` #!/bin/bash # Confirm the stream the NACK controller created from a Stream CRD is # really the R3 ORDERS stream you declared — not a half-applied or # single-replica copy. Run this from the nats-box pod that the NATS Helm # chart deploys alongside the StatefulSet, so it resolves nats-0..2 over # the headless service. # # Open a shell in nats-box first: # kubectl exec -it deploy/nats-box -- sh # then run the commands below against the cluster. # List every stream the cluster holds. After the Stream CRD reconciles, # ORDERS appears here even though no human ran `nats stream add` — the # controller created it from the declarative resource. nats stream ls # Read ORDERS back in full. The line that matters for a CRD-created # stream is Replicas: 3 — proof the controller honoured `replicas: 3` # from the CRD spec and placed a copy on each of nats-0, nats-1, nats-2. nats stream info ORDERS # Publish one order to prove the CRD-created stream actually accepts # writes across all three replicas. The payload is the canonical Acme # order shape, unchanged from every other chapter. nats pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # Expected (trimmed): the Configuration block shows # Replicas: 3 # Storage: File # and the State block shows the message you just published, with the # Cluster section listing nats-0 as leader and nats-1/nats-2 as # replicas. If Replicas reads 1, the CRD spec is missing `replicas: 3` # or the controller has not finished reconciling yet — re-run after the # .status block on the CRD reports ready. ``` #### C ``` // List every stream the cluster holds. After the Stream CRD // reconciles, ORDERS appears here even though no human ran // `nats stream add` -- the NACK controller created it from the // declarative resource. if (s == NATS_OK) s = js_Streams(&list, js, NULL, &jerr); if (s == NATS_OK) { for (i = 0; i < list->Count; i++) printf("stream: %s\n", list->List[i]->Config->Name); } // Read ORDERS back in full. The number that matters for a // CRD-created stream is Replicas: 3 -- proof the controller // honoured `replicas: 3` from the CRD spec and placed a copy on // each of nats-0, nats-1, nats-2. if (s == NATS_OK) s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { printf("ORDERS replicas: %" PRId64 "\n", si->Config->Replicas); if (si->Cluster != NULL) printf("ORDERS leader: %s\n", si->Cluster->Leader); } // Publish one order to prove the CRD-created stream actually // accepts writes across all three replicas. if (s == NATS_OK) s = js_Publish(&ack, js, "orders.created", order, (int) strlen(order), NULL, &jerr); if (s == NATS_OK) printf("stored in %s, sequence %" PRIu64 "\n", ack->Stream, ack->Sequence); ``` The line that matters is `Replicas: 3`. If it reads `1`, the CRD spec is missing `replicas: 3`, or the controller hasn't finished reconciling; check the resource's `.status` before assuming the stream is wrong. ## Where you are The ORDERS cluster now runs on Kubernetes: * three pods (`nats-0`, `nats-1`, `nats-2`) managed by a StatefulSet, each with a stable name, DNS entry, and volume; they're the same `n1-east`/`n2-east`/`n3-east` nodes from Topologies * the NATS Helm chart rendering the StatefulSet, the headless service, the ConfigMap, and the three `/healthz` probes * the `ORDERS` stream and the `shipping` and `analytics` consumers declared as CRDs, reconciled by the NACK controller, with their desired state in version control ## What's next The cluster is running, but its config is baked into a ConfigMap you can't yet change without restarting pods. The next page splits that config into includes and reloads it live (limits, TLS paths, accounts) with a SIGHUP and zero downtime. Continue to [Config management](/learn/deployment/config-management.md). ## See also * [Reference → Configuration](/reference/config/.md) — every server configuration option, including the full `Stream` CRD field set * [Topologies → Your first cluster](/learn/topologies/your-first-cluster.md) — the `n1-east`/`n2-east`/`n3-east` cluster these pods deploy * [Monitoring → JetStream health](/learn/monitoring/jetstream-health.md) — what the `/healthz` probes report and what to alert on --- # Rolling upgrades The previous page changed the cluster's config without dropping a connection. A new server *version* is a harder problem, because the binary itself has to change, which means the process has to restart. If a restart is done carelessly, it drops every client on that node and can leave the `ORDERS` stream a node short of quorum. This page rolls a new version through `nats-0`, `nats-1`, and `nats-2` one node at a time, with the cluster staying up the whole way. It needs two ideas: **lame-duck mode**, the graceful way a single node steps out, and **upgrade order**, the sequence across the three nodes that protects the stream. ## Lame-duck mode A node restart is abrupt. The process stops, and every client connected to it discovers the loss only when its next write fails. Any stream replica that node was leading goes leaderless until the cluster elects a replacement. Lame-duck mode makes that stop orderly instead of abrupt. A node in lame-duck mode broadcasts that it is going away, hands off its work, and lets clients move *before* the process exits. The mechanism is specified in the server's ADR-5; here you only need what the operator triggers and what it does. You enter lame-duck mode by sending the running process the `SIGUSR2` signal: ``` # Signal nats-0 to enter lame-duck mode (run on the node, or via the pod) kill -SIGUSR2 $(cat /var/run/nats/nats.pid) ``` That one signal kicks off a sequence inside the node: 1. It closes its client listener, so no *new* connection lands on a node that's on its way out. 2. It transfers any Raft leadership it holds to another replica, so no stream is left leaderless. 3. It shuts down its JetStream assets cleanly, flushing to disk. 4. It broadcasts `INFO ldm:true` to its routes and its connected clients. The update drops this node from each client's server pool, so the next reconnect lands elsewhere; the client takes no other action on the INFO itself. 5. After a short grace period it closes the remaining client connections, spread over the lame-duck duration rather than all at once. Each client sees its connection close and reconnects to another node through its normal reconnect logic. Only after that does the process exit. By then leadership has transferred and the clients have moved off as their connections closed, so the stream never lost a quorum. Two settings control the timing. `lame_duck_grace_period` (default `10s`) is how long the node waits before it starts kicking clients. `lame_duck_duration` (default `2m`, minimum `30s`) is the total window over which it spreads those kicks. The grace period must be shorter than the duration. ``` # nats-server.conf — lame-duck timing lame_duck_grace_period: "10s" lame_duck_duration: "2m" ``` Set the duration to comfortably cover how long your clients take to reconnect *and* how long JetStream needs to move leadership off this node. A duration shorter than the rebalance drops clients before the stream has caught up. We'll come back to that trap in [Pitfalls](#pitfalls). On Kubernetes you don't send the signal by hand. The NATS Helm chart wires lame-duck mode into the pod's `preStop` hook, so a normal `kubectl rollout restart` triggers it for you: ``` # Pod preStop hook (set by the NATS Helm chart) — enter lame-duck before exit lifecycle: preStop: exec: command: - "nats-server" - "-sl=ldm=/var/run/nats/nats.pid" ``` The `-sl=ldm=...` form reads the pid file, signals the running server to enter lame-duck mode, and returns — identical to the `SIGUSR2` you sent above. Kubernetes then sends SIGTERM, which the server ignores while it's draining, so the drain is protected only by `terminationGracePeriodSeconds`: when that expires the kubelet sends SIGKILL. The chart defaults `lame_duck_duration` to `30s` and `terminationGracePeriodSeconds` to `60s`. If you raise the duration, raise the grace period above `lame_duck_duration` plus shutdown overhead too, or the kubelet SIGKILLs the node mid-drain. ## Upgrade order Lame-duck mode makes *one* node leave gracefully. Rolling a new version across all three depends on the *order* you take them in, and that order follows a specific rule. One node in the cluster is the **meta-leader**: the Raft leader for the cluster's own metadata, the node that coordinates where streams and consumers live. The other two are non-leaders. Stepping the meta-leader down forces a metadata election, and while that election runs, stream and consumer *operations* (create, update, leadership moves) pause until a new leader wins — typically about 5 to 10 seconds with default timeouts if the node was killed outright, or roughly a second if it handed leadership off first. So the rule is: **upgrade the non-leaders first, and the meta-leader last.** By the time you reach the meta-leader, the other two nodes are already on the new version and ready to take over, so the one unavoidable metadata election is short and happens once. Before you start, read the cluster's current shape and confirm the stream is at full R3: #### CLI ``` #!/bin/bash # Read the R3 ORDERS stream's replicas and current leader. # Run this BEFORE you start the upgrade, and again AFTER each node rejoins, # to confirm the stream stayed at 3 replicas and the leader moved as expected. nats stream info ORDERS \ --server tls://nats.acme.internal:4222 \ --creds /etc/nats/creds/order-svc.creds # Look for the Cluster section in the output: # Replicas: 3 the R3 ORDERS stream has 3 copies # Leader: nats-1 the node currently coordinating writes # Replica: nats-0, current, ... a healthy follower, caught up # Replica: nats-2, current, ... a healthy follower, caught up # # Upgrade rule: every replica must read "current" before you take the next # node down. A replica that lags ("outdated") is still catching up, so # pausing on it keeps the stream at full R3 the whole way through. ``` #### C ``` // Read the R3 ORDERS stream's replicas and current leader. Run // this BEFORE you start the upgrade, and again AFTER each node // rejoins, to confirm the stream stayed at 3 replicas and the // leader moved as expected. if (s == NATS_OK) s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { printf("replicas: %" PRId64 "\n", si->Config->Replicas); if (si->Cluster != NULL) { printf("leader: %s\n", si->Cluster->Leader); for (i = 0; i < si->Cluster->ReplicasLen; i++) { jsPeerInfo *peer = si->Cluster->Replicas[i]; // Upgrade rule: every replica must read "current" // before you take the next node down. A lagging // replica is still catching up, so pausing on it // keeps the stream at full R3 the whole way through. printf("replica: %s, %s, lag %" PRIu64 "\n", peer->Name, peer->Current ? "current" : "outdated", peer->Lag); } } } ``` That output shows the `ORDERS` stream's own Raft leader and each replica's status — the per-node "current" gate you'll use between steps. The **meta-leader** is a separate Raft group and may sit on a different node; find it with `nats server report jetstream`, which marks the node leading the cluster's metadata: ``` nats server report jetstream ``` The walkthrough below assumes `nats-1` is the meta-leader. The procedure for each node is the same three steps: ``` # For each NON-leader node (nats-0, then nats-2): # 1. Enter lame-duck mode and let it drain. kill -SIGUSR2 $(cat /var/run/nats/nats.pid) # on nats-0 # 2. Restart the process on the new server version. systemctl restart nats-server # picks up the new binary # 3. Wait until the stream reports this node as a "current" replica again # before moving to the next node — re-run nats stream info ORDERS. ``` Step 3 is the gate that controls when you proceed. A restarted node isn't done until its `ORDERS` replica has caught up, because taking the next node down while this one is still syncing leaves the stream one healthy replica short. Take the meta-leader (`nats-1`) **last**, with the same three steps. When it enters lame-duck mode it transfers metadata leadership to one of the two already-upgraded nodes, the brief election runs once, and the whole cluster is on the new version. There's one more guardrail on Kubernetes. A `PodDisruptionBudget` (PDB) caps how many pods may be down at once. With `minAvailable: 2`, Kubernetes won't voluntarily evict two of the three pods together, so a node drain can never cost the R3 stream its quorum: ``` # pod-disruption-budget.yaml — never let the cluster drop below 2 nodes apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: nats spec: minAvailable: 2 selector: matchLabels: app.kubernetes.io/name: nats ``` The flow below shows the whole sequence for one node: the signal, the `INFO ldm:true` broadcast, the leadership transfer, the client reconnect, and the rejoin on the new version. **Message flow — Lame-duck drain during a rolling upgrade (animated):** A single node draining gracefully during a rolling upgrade. The operator sends SIGUSR2 to nats-0 to enter lame-duck mode; nats-0 broadcasts an updated INFO with ldm:true so connected clients learn it is leaving, hands its Raft leadership to nats-1, and lets JetStream rebalance the ORDERS replicas onto nats-1 and nats-2 at full replication. Drained, nats-0 closes connections gracefully and the warehouse client reconnects to nats-1 without losing a message; nats-0 restarts on the new version and rejoins as a non-leader. Upgrade the followers first, the meta-leader last. * operator → nats-2 * nats-2 → warehouse * nats-2 → nats1 * nats-2 → nats2 * nats-2 → nats1 (subject: ORDERS replica) * nats1 → nats2 * warehouse → nats1 * nats1 → nats-2 (subject: rejoin (follower)) The two keys this page touches, `lame_duck_duration` and `lame_duck_grace_period`, are the only ones you must set for a safe roll. The full set of server configuration options is documented in [Reference → Configuration](/reference/config/.md). We only cover the keys this deployment needs here. ## Client reconnection during the upgrade A node leaving in lame-duck mode requires no action from a correctly configured client. The `INFO ldm:true` broadcast tells the client the node is going away (and fires an optional lame-duck callback if the app registered one); the client keeps running on its existing connection, and because the node stops accepting new connections, no client dials it again. When the server later closes that connection during the staggered kick, the client's normal reconnect logic dials another node in the cluster, resubscribes, and resumes. You can watch this happen. Subscribe `warehouse` to `orders.created` in one terminal, publish an order from `order-svc` in another, and roll a node mid-stream. The subscriber logs a reconnect and keeps printing messages; the published order still lands in the `ORDERS` stream, because the two nodes still up hold a quorum: #### CLI ``` #!/bin/bash # Prove a client keeps working THROUGH a rolling upgrade. # Run these two commands in two terminals and leave them running while you # roll the nodes. The subscriber reconnects when its node enters lame-duck # mode, and the publisher's order still lands in the ORDERS stream. # Terminal 1 — warehouse subscribes to new orders and stays connected. nats sub "orders.created" \ --server tls://nats.acme.internal:4222 \ --creds /etc/nats/creds/order-svc.creds # Terminal 2 — order-svc publishes one order. It is captured by ORDERS and # delivered to the warehouse subscriber even if a node is mid-upgrade. nats pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \ --server tls://nats.acme.internal:4222 \ --creds /etc/nats/creds/order-svc.creds # What you should see: when a node enters lame-duck mode the subscriber logs # a reconnect to another node, then keeps printing messages. No order is lost, # because the R3 ORDERS stream still has a quorum on the two nodes that are up. ``` #### C ``` // Prove a client keeps working THROUGH a rolling upgrade. The // reconnect callback logs the moment the client's node enters // lame-duck mode and the connection moves to another node. if (s == NATS_OK) s = natsOptions_SetReconnectedCB(opts, onReconnected, NULL); if (s == NATS_OK) s = natsConnection_Connect(&conn, opts); // warehouse subscribes to new orders and stays connected. if (s == NATS_OK) s = natsConnection_Subscribe(&sub, conn, "orders.created", onMsg, NULL); // order-svc publishes one order. It is captured by ORDERS and // delivered to the warehouse subscriber even if a node is // mid-upgrade, because the R3 stream still has a quorum on the // two nodes that are up. if (s == NATS_OK) s = natsConnection_PublishString(conn, "orders.created", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"," "\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"); // Leave the subscriber running and roll a node: it logs a // reconnect and keeps printing messages. No order is lost. if (s == NATS_OK) { printf("subscribed to orders.created, roll a node while I run\n"); while (natsConnection_Status(conn) != NATS_CONN_STATUS_CLOSED) nats_Sleep(100); } ``` This is why the order and the timing matter: get them right and a version upgrade has no observable effect on `warehouse`, `notifications`, and `analytics`. ## Pitfalls A few traps turn a clean rolling upgrade into an outage. All of them come back to this page's two ideas: lame-duck timing and upgrade order. **A `lame_duck_duration` shorter than the rebalance drops clients early.** If you set the duration to `30s` but JetStream needs `45s` to move the `ORDERS` leadership and resync replicas off the node, the node kicks its clients and exits while the stream is still catching up. Measure how long a real drain takes on your cluster first, then set the duration above it with margin, rather than defaulting to the minimum value. **Upgrading the meta-leader without draining it stalls stream ops until a new leader is elected.** Restart the meta-leader directly and the cluster has no metadata leader until the election timeout fires and a follower wins — about 5 to 10 seconds with default timeouts — and during that window every stream and consumer operation stalls. Draining it first transfers leadership in roughly a second instead. Always enter lame-duck mode so leadership transfers *before* the process stops, and always do the non-leaders first so the meta-leader's one election is short. Check which node leads before you touch anything, and re-check after each node, so you never take down two replicas at once: #### CLI ``` #!/bin/bash # Read the R3 ORDERS stream's replicas and current leader. # Run this BEFORE you start the upgrade, and again AFTER each node rejoins, # to confirm the stream stayed at 3 replicas and the leader moved as expected. nats stream info ORDERS \ --server tls://nats.acme.internal:4222 \ --creds /etc/nats/creds/order-svc.creds # Look for the Cluster section in the output: # Replicas: 3 the R3 ORDERS stream has 3 copies # Leader: nats-1 the node currently coordinating writes # Replica: nats-0, current, ... a healthy follower, caught up # Replica: nats-2, current, ... a healthy follower, caught up # # Upgrade rule: every replica must read "current" before you take the next # node down. A replica that lags ("outdated") is still catching up, so # pausing on it keeps the stream at full R3 the whole way through. ``` #### C ``` // Read the R3 ORDERS stream's replicas and current leader. Run // this BEFORE you start the upgrade, and again AFTER each node // rejoins, to confirm the stream stayed at 3 replicas and the // leader moved as expected. if (s == NATS_OK) s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { printf("replicas: %" PRId64 "\n", si->Config->Replicas); if (si->Cluster != NULL) { printf("leader: %s\n", si->Cluster->Leader); for (i = 0; i < si->Cluster->ReplicasLen; i++) { jsPeerInfo *peer = si->Cluster->Replicas[i]; // Upgrade rule: every replica must read "current" // before you take the next node down. A lagging // replica is still catching up, so pausing on it // keeps the stream at full R3 the whole way through. printf("replica: %s, %s, lag %" PRIu64 "\n", peer->Name, peer->Current ? "current" : "outdated", peer->Lag); } } } ``` **Evicting pods without a PDB can drain all three at once.** A node drain, a cluster autoscaler, or a careless `kubectl delete` can take two or three pods together if nothing stops it, and two nodes down costs the R3 stream its quorum. Set a `PodDisruptionBudget` with `minAvailable: 2` so Kubernetes refuses to voluntarily evict past one pod. Don't rely on doing the steps slowly by hand — make the budget enforce it. **A reconnect storm comes from abrupt kills, not from lame-duck.** Clients take no action on the `INFO ldm:true` broadcast itself — lame-duck mode staggers the connection closes over the duration precisely so their reconnects spread out instead of arriving as one burst that could overwhelm the two nodes still up. The storm risk is what you get *without* that: killing a node outright, or setting a duration so short the spread collapses. Keep the duration comfortable, and roll one node at a time, finishing one and letting it rejoin before starting the next. ## Where you are The `ORDERS` cluster can now take a new server version without losing the stream or dropping a client: * Each node leaves through lame-duck mode: `SIGUSR2` (or the K8s `preStop` hook) makes it broadcast `INFO ldm:true`, transfer leadership, flush JetStream, and move clients before exiting. * You roll the non-leaders first and the meta-leader last, gating on each node returning as a `current` replica. * A `PodDisruptionBudget` of `minAvailable: 2` keeps the R3 stream's quorum safe even under involuntary eviction. Clients reconnect on their own, so `warehouse`, `notifications`, and `analytics` continue through the upgrade without code changes. ## What's next The cluster is sized, deployed, configurable, and upgradable. The last operational step is **hardening** it: TLS on every link, the `ACME` credentials mounted, a locked-down systemd unit, and the monitor port closed to the internet. Continue to [Hardening](/learn/deployment/hardening.md). ## See also * [Clustering → Raft and leaders](/learn/clustering/raft-and-leaders.md) — how leadership transfer and re-election actually work underneath. * [Surviving node loss](/learn/jetstream/surviving-node-loss.md) — why an R3 stream survives one node leaving, and what a quorum buys you. * [Core NATS → Connection lifecycle](/learn/core-nats/connection-lifecycle.md) — the client's side of lame-duck mode: what the notice means and how the client ends up on another node. --- # Sizing & resources Topologies decided the shape: a three-node cluster called `east` (`n1-east`, `n2-east`, `n3-east`) carrying the R3 `ORDERS` stream. This chapter runs that shape for real, and running it starts with the question of how much of each resource this cluster needs. A node spends a small, fixed set of resources, the JetStream defaults are knowable numbers, and the account limits the server enforces are readable with one command. This page turns those into a baseline for the ORDERS workload. You'll learn two things here: the four resources a node spends (and their JetStream defaults), and how account limits count R3 replication against the storage ceiling. ## The four resources a node spends Every NATS node spends the same four resources. Once you size each one, you've sized the node. **CPU** handles moving messages. Core NATS routing is cheap; TLS handshakes and JetStream replication are where cycles go. There's no hard CPU limit to set, so the rule is headroom: overprovision CPU by 20–30% above steady state so a node has cycles spare for a rebalance when a peer leaves. **Memory** holds connections, subscriptions, and (for memory-storage streams) message data. The ORDERS stream uses file storage, so its messages live on disk, not in RAM. The `order-svc` publisher and the `warehouse`, `notifications`, and `analytics` subscribers are light clients; a few hundred connections fit comfortably in a few hundred megabytes. Budget `order-svc` at roughly 128 MiB. **Disk** holds file-storage streams. This is the resource the ORDERS stream actually spends, and the one most likely to run out. We'll size it below. **File descriptors (FDs)** are the per-process limit on open files and sockets. Connections, routes, and streams each consume FDs. JetStream spends roughly two FDs per stream. On a small cluster the default per-process limit is plenty; on a large one it isn't, which is why the hardened service unit on the [hardening](/learn/deployment/hardening.md) page raises it to `LimitNOFILE=800000`. ### The JetStream defaults Left unset, JetStream sizes its limits from the system it's on. **Memory storage** defaults to 75% of system RAM (capped by `GOMEMLIMIT` if set), falling back to **256 MB** only when the server can't read system memory. **File storage** defaults to 75% of the disk space actually available under `store_dir`, falling back to **1 TB** only when the platform can't report disk size (Windows and a few others, or a failed `statfs`). On a Linux container with a 10 GiB volume the file-storage default is therefore about 7.5 GiB, not 1 TB. Because the default tracks the real disk, the container hazard isn't the default itself — it's setting `max_file_store` *larger* than the volume, or running on a shared or thin-provisioned volume where "available" disk overstates what this stream may safely use. Either way the node accepts writes it can't ultimately store, and the publish that crosses the real boundary fails. Pin `max_file_store` to what the volume can actually hold: ``` # n1-east.conf — pin JetStream storage to the real volume size jetstream { store_dir: "/var/lib/nats/jetstream" max_memory_store: 256MB max_file_store: 10GB } ``` The ORDERS stream is an R3 file stream. At R3 it keeps three copies, one per node, so each node stores the full stream once. A 10 GiB `max_file_store` per node leaves ample room for an ORDERS stream sized to fit the default 10 GiB volume the Kubernetes chapter provisions. The Helm chart already defaults `max_file_store` to the PVC size, so on Kubernetes this pin is usually set for you. The full set of JetStream limit keys is documented in [Reference → Configuration](/reference/config/jetstream/.md). We only cover `max_memory_store` and `max_file_store` here. ## Account limits and how replication counts The server config sizes the *node*, while **account limits** size the *tenant*. The ORDERS account has its own ceilings (`MaxMemory`, `MaxStore`, `MaxStreams`, `MaxConsumers`), and the server enforces them no matter how much disk the node has. Read them live before you size: #### CLI ``` #!/bin/bash # Read the LIVE account limits for the ORDERS account before you size anything. # This is the source of truth: it shows the tier and the ceilings the server # will actually enforce, not what you hope the config says. # # Connect as order-svc in the ORDERS account, then ask the server. nats account info \ --server tls://nats.acme.internal:4222 \ --creds /etc/nats/creds/order-svc.creds # Look for these rows in the output: # Memory: the MaxMemory ceiling for the account # Storage: the MaxStore ceiling (file storage) for the account # Streams: MaxStreams — how many streams the account may create # Consumers: MaxConsumers — how many consumers per stream # # Sizing rule: on an UN-tiered account an R3 stream counts as # replicas x bytes against MaxStore, so a 10 GiB ORDERS stream at R3 # spends 30 GiB of the account's storage limit. A tiered account bakes # replication into the tier, so the number you see is the usable bytes. ``` #### C ``` // Read the LIVE account limits for the ORDERS account before you // size anything. This is the source of truth: the ceilings the // server will actually enforce, not what you hope the config says. if (s == NATS_OK) s = js_GetAccountInfo(&ai, js, NULL, &jerr); if (s == NATS_OK) { // Current usage, then the ceilings. A value of -1 means // unlimited. printf("memory used: %" PRIu64 " bytes\n", ai->Memory); printf("storage used: %" PRIu64 " bytes\n", ai->Store); printf("streams: %" PRId64 "\n", ai->Streams); printf("consumers: %" PRId64 "\n", ai->Consumers); printf("max memory: %" PRId64 "\n", ai->Limits.MaxMemory); printf("max storage: %" PRId64 "\n", ai->Limits.MaxStore); printf("max streams: %" PRId64 "\n", ai->Limits.MaxStreams); printf("max consumers: %" PRId64 "\n", ai->Limits.MaxConsumers); // Sizing rule: on an UN-tiered account an R3 stream counts as // replicas x bytes against max storage, so a 10 GiB ORDERS // stream at R3 spends 30 GiB of the account's limit. A tiered // account bakes replication in, so its number is usable bytes. if (ai->TiersLen > 0) printf("tiered account: %d tier(s), limits are per tier\n", ai->TiersLen); } ``` One subtlety in that output decides your storage math: how replication counts against `MaxStore`. There are two cases. On an **un-tiered** account, an R3 stream counts as `replicas × bytes`. A 10 GiB ORDERS stream at R3 spends 30 GiB of the account's `MaxStore`, because the limit measures total bytes stored across all replicas. Forget the multiplier and the third replica fails to place when the account hits its ceiling. On a **tiered** account, replication is baked into the tier. The bytes the limit reports are the usable bytes: the R3 multiplier is already accounted for, so a 10 GiB tier holds a 10 GiB R3 stream. The durability R3 buys (surviving the loss of a node) is the JetStream chapter's subject, covered on [Surviving node loss](/learn/jetstream/surviving-node-loss.md). Here you only need the cost: on an un-tiered account, three copies cost three times the bytes. ## Connection, subscription, and payload limits Three more limits round out the node, and one command reads them all: #### CLI ``` #!/bin/bash # Read the LIVE server limits for one node of the east cluster. # These are per-server ceilings: payload size, connection cap, and the # JetStream memory/store limits the node was started with. # # `nats server info` is a system-account request, so authenticate with the # system account's creds, not the ORDERS-account user creds. nats server info n1-east \ --server tls://nats.acme.internal:4222 \ --creds /etc/nats/creds/sys.creds # Look for these in the output: # Maximum Payload: max_payload (default 1.0 MiB) — the largest single message # Maximum Connections: max_connections (default 64K, i.e. 65,536) # JetStream: Max Memory and Max Storage configured on this node # # Sizing rule: max_payload must be <= max_pending. Keep max_pending at # >= 10x your peak message size so a burst of large orders does not stall # the connection. The ORDERS payload is well under 1 KiB, so the 1 MiB # default has ample headroom here. ``` #### C ``` // Read the LIVE server limits for one node of the east cluster. // There is no dedicated client API for this: it is a plain // request on the system account's $SYS.REQ.SERVER.PING.VARZ // subject, filtered to the n1-east node by name. The VARZ reply // carries the per-server ceilings: max_payload (default 1 MiB), // max_connections (default 64K), and the JetStream memory/store // limits the node was started with. if (s == NATS_OK) s = natsConnection_RequestString(&reply, conn, "$SYS.REQ.SERVER.PING.VARZ", "{\"server_name\":\"n1-east\"}", 2000); if (s == NATS_OK) printf("%.*s\n", natsMsg_GetDataLength(reply), natsMsg_GetData(reply)); ``` `max_connections` caps how many clients a node accepts (default 64K, i.e. 65,536; it's reloadable, and overflow disconnects immediately). `max_subscriptions` caps subscriptions per connection (default unlimited). `max_payload` caps a single message at 1 MB by default, and it must stay `≤ max_pending`. The ORDERS payload is well under a kilobyte, so the defaults have room to spare: ``` {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} ``` The full set of connection and payload keys is documented in [Reference → Configuration](/reference/config/max_payload.md). We only name the ones a sizing baseline needs here. ## Pitfalls A few sizing mistakes only surface in production under load, where they're expensive to fix. Each is scoped to this page's two concepts: the node's resources and the account's limits. **Setting `max_file_store` beyond the real disk.** A `max_file_store` larger than the mounted volume — or a shared or thin-provisioned volume whose "available" space the default over-reads — lets the node accept writes it can't store. The failure is a publish error mid-stream, not a startup warning. Set `max_file_store` to the volume size, test small with a `10GB` value, and watch the real disk with `df -h`. Don't trust the config number over what the device reports. **`max_payload` larger than `max_pending`.** If `max_payload` exceeds `max_pending`, the server refuses to start. Keep `max_pending` at `≥ 10× peak message size` so a burst of large messages can't stall a connection. Don't raise `max_payload` in isolation. **File-descriptor exhaustion on a big cluster.** Each stream costs about two FDs, and routes and gossip add more. On a large cluster the default per-process FD limit runs out, and the symptom is connection refusals that look like a network fault. Raise the limit (`ulimit -n 800000`) before the process starts; the hardened unit on the [hardening](/learn/deployment/hardening.md) page does exactly this. **JWT account limits change only when the account JWT does.** Operator-mode accounts enforce their limits through the account JWT the resolver holds, not through server config, so raising `MaxStore` or `MaxStreams` means editing and pushing the JWT — a server reload or restart won't move them. Keep the cluster's server versions aligned as you roll upgrades so every node reads the same JWT limit fields (tiered R1/R3 limits, for instance, need servers new enough to understand them). Read the active tier with `nats account info` before you size, so you plan against the limits the cluster actually applies. The runnable fix for all four is the same first step: read the live limits before you size, so the numbers you plan against are the numbers the server enforces. #### CLI ``` #!/bin/bash # Read the LIVE account limits for the ORDERS account before you size anything. # This is the source of truth: it shows the tier and the ceilings the server # will actually enforce, not what you hope the config says. # # Connect as order-svc in the ORDERS account, then ask the server. nats account info \ --server tls://nats.acme.internal:4222 \ --creds /etc/nats/creds/order-svc.creds # Look for these rows in the output: # Memory: the MaxMemory ceiling for the account # Storage: the MaxStore ceiling (file storage) for the account # Streams: MaxStreams — how many streams the account may create # Consumers: MaxConsumers — how many consumers per stream # # Sizing rule: on an UN-tiered account an R3 stream counts as # replicas x bytes against MaxStore, so a 10 GiB ORDERS stream at R3 # spends 30 GiB of the account's storage limit. A tiered account bakes # replication into the tier, so the number you see is the usable bytes. ``` #### C ``` // Read the LIVE account limits for the ORDERS account before you // size anything. This is the source of truth: the ceilings the // server will actually enforce, not what you hope the config says. if (s == NATS_OK) s = js_GetAccountInfo(&ai, js, NULL, &jerr); if (s == NATS_OK) { // Current usage, then the ceilings. A value of -1 means // unlimited. printf("memory used: %" PRIu64 " bytes\n", ai->Memory); printf("storage used: %" PRIu64 " bytes\n", ai->Store); printf("streams: %" PRId64 "\n", ai->Streams); printf("consumers: %" PRId64 "\n", ai->Consumers); printf("max memory: %" PRId64 "\n", ai->Limits.MaxMemory); printf("max storage: %" PRId64 "\n", ai->Limits.MaxStore); printf("max streams: %" PRId64 "\n", ai->Limits.MaxStreams); printf("max consumers: %" PRId64 "\n", ai->Limits.MaxConsumers); // Sizing rule: on an UN-tiered account an R3 stream counts as // replicas x bytes against max storage, so a 10 GiB ORDERS // stream at R3 spends 30 GiB of the account's limit. A tiered // account bakes replication in, so its number is usable bytes. if (ai->TiersLen > 0) printf("tiered account: %d tier(s), limits are per tier\n", ai->TiersLen); } ``` ## Where you are You now have a sizing baseline for the ORDERS cluster: * The four resources a node spends: CPU (20–30% headroom), memory (`order-svc` \~128 MiB), disk (`max_file_store` pinned to the volume), and FDs (two per stream). * The JetStream defaults named (memory and file storage both default to 75% of RAM and disk; 256 MB / 1 TB are only the can't-read fallbacks) and `max_file_store` pinned to the real 10 GiB volume. * The account limits read live with `nats account info`, and the rule that an un-tiered R3 stream costs `replicas × bytes`. The ORDERS R3 file stream fits the default 10 GiB volume, and `order-svc` fits in \~128 MiB. That baseline is what the next page deploys. ## What's next With the resources sized, the next page stands the cluster up: the NATS Helm chart, the StatefulSet that maps `nats-0..2` to `n1-east..n3-east`, and the NACK controller that declares the `ORDERS` stream as a CRD. Continue to [Kubernetes](/learn/deployment/kubernetes.md). ## See also * [Reference → Configuration](/reference/config/jetstream/.md) — every JetStream limit key and its default. * [Reference → max\_payload](/reference/config/max_payload.md) — the payload and pending limits in full. * [Surviving node loss](/learn/jetstream/surviving-node-loss.md) — what R3 durability buys for the bytes it costs. --- # Where to go next You started this chapter with a cluster design: the three-node `east` cluster the Topologies chapter designed, carrying the R3 `ORDERS` stream. You end it with that same cluster sized, running on Kubernetes, reconfigurable without downtime, upgradable without losing the stream, and locked down with TLS on every link. Those steps take a cluster design and produce a running system. This page introduces no new material. It collects the runbook you built into one place and points you at the chapters and Reference that take it further. ## The five operational steps Every page in this chapter advanced the same cluster through one operational step, and the order they run in matters. You **size** it first. A node uses four resources (CPU, memory, disk, and file descriptors), and an R3 stream counts three times against the `ORDERS` account limits. Sizing means reading those limits before you commit to a PVC, rather than estimating them after the disk fills. You **deploy** it second. The NATS Helm chart stands the cluster up as a StatefulSet whose three pods, `nats-0`/`nats-1`/`nats-2`, are the same `n1-east`/`n2-east`/`n3-east` nodes from Topologies. The NACK controller turns the `ORDERS` stream and its consumers into declarative CRDs. You **configure** it third. An include splits the config into per-account and per-region files, and a SIGHUP reloads the reloadable keys in place without a restart or a client reconnect. The reloader sidecar turns a ConfigMap edit into that SIGHUP. You **upgrade** it fourth. Lame-duck mode lets a node drain its clients and transfer Raft leadership before it stops, and the upgrade order (non-leaders first, the meta-leader last, quorum protected by a PodDisruptionBudget) keeps the R3 ORDERS stream available the whole way. You **harden** it fifth. TLS goes on every link, the `ACME` credentials mount as files, a locked-down systemd unit strips the process of every capability it doesn't need, and the monitor port closes to everything but localhost. The five steps are size, deploy, configure, upgrade, and harden, and everything else in this chapter is a detail of one of those five. ## Where the exhaustive details are documented The chapter is unversioned and concept-first. The exact keys, defaults, and ranges live in **Reference**, which is versioned and exhaustive. When you need the precise type of a config field or the full list of TLS or JetStream limit options, that's where to look. The full set of server configuration options lives in [Reference → Configuration](/reference/config/.md). The handoff phrases throughout this chapter ("we only cover the keys this deployment needs") all point into it. ## Sibling deep dives This chapter is the runbook, and five other chapters continue from where it ends. Each of those chapters covers a separate job, so the cluster you built here carries straight into them. The [Topologies deep dive](/learn/topologies/.md) covers the **shape**. It's where the three-node `east` cluster came from, and where you go to choose a different shape (a super-cluster across regions, or leaf nodes at the edge) before you size and deploy it. The [Clustering & Replication deep dive](/learn/clustering/.md) covers the **Raft mechanics** this chapter only triggers. When a rolling upgrade transfers leadership, [raft-and-leaders](/learn/clustering/raft-and-leaders.md) explains how the meta-leader is elected and how R3 stays consistent through the change. The [Security deep dive](/learn/security/.md) covers the **auth model** behind the credentials this chapter mounts. This chapter turns TLS on and points the server at a creds file; [operator-mode](/learn/security/operator-mode.md) is where you design the operator `ACME`, the accounts, and the JWTs that the creds file carries. The [Monitoring deep dive](/learn/monitoring/.md) covers **what to watch** once the cluster is live. This chapter exposes `/healthz` and runs `nats server report` as one-off operational checks; [monitoring-endpoints](/learn/monitoring/monitoring-endpoints.md) is where you turn those signals into ongoing alerting. The [Backup & Recovery deep dive](/learn/backup-recovery/.md) covers **disaster recovery** (snapshotting the ORDERS stream and restoring it), the one production concern this chapter's hardening doesn't cover. ## Where you are This is the end of the chapter. The Acme ORDERS cluster is now sized, deployed as a StatefulSet, split into includes you can reload live, upgradable through lame-duck mode, and hardened with TLS and a locked-down systemd unit. This page introduces no new scenario state; the cluster is running exactly as you left it on the hardening page. You now have the whole runbook: take a topology shape, size its resources, stand it up declaratively, change it without downtime, roll it forward safely, and lock it down. That runbook is the minimum baseline for operating any NATS cluster in production, not only this one. ## Production checklist Every page in this chapter closed with a Pitfalls section. This collects the action items from all of them in one place as a last pass before you run real orders through the cluster. Each group links back to the page that explains the why. ### Sizing & resources — see [Pitfalls](/learn/deployment/sizing-and-resources.md#pitfalls) * Set `max_file_store` to a size the disk can actually hold; an oversized limit lets JetStream error mid-publish instead of failing fast. Test small (`10GB`) and watch `df -h`. * Keep `max_payload` at or below `max_pending`; a `max_payload` larger than `max_pending` refuses the server start. Hold `max_pending` at `≥ 10×` your peak message size. * Raise the file-descriptor limit before the process starts (`ulimit -n 800000`); a big cluster spends about two FDs per stream plus routes and gossip and exhausts the default cap. * Keep server versions aligned as you roll operator-mode upgrades, so every node reads the same JWT limit fields; account limits change only when you edit and push the account JWT, not on a server reload. * Read the live limits with `nats account info` before sizing, so you plan against the limits the server actually enforces. ### Kubernetes — see [Pitfalls](/learn/deployment/kubernetes.md#pitfalls) * Use `volumeClaimTemplates` (the Helm default) so each PVC binds before its StatefulSet replica starts; an unbound PVC leaves the pod Pending. * Run the config reloader sidecar; a ConfigMap edit does not reload the server on its own. * Raise the readiness failure threshold so the probe doesn't flap not-ready during a healthy JetStream rebalance. * Never mix `nats` CLI mutations with a CRD-owned stream; NACK re-creates a deleted stream on its \~30s resync, and in `--control-loop` mode it also reverts manual config edits on about a one-minute cycle. * Confirm the CRD-created stream is R3 with `nats stream info ORDERS` from nats-box before trusting the declarative path. ### Config management — see [Pitfalls](/learn/deployment/config-management.md#pitfalls) * Write include paths as absolute paths; an include resolves relative to the config file's directory, not the working directory. * Fit a SIGHUP inside the graceful window; a reload during a rebalance can interrupt leadership transfer. * Monitor TLS cert expiry as you rotate it; a rotation without monitoring leaves old connections hung on an expired cert. * Trim oversized streams by hand after lowering `max_file`; a reload changes the ceiling, not the contents, so new writes fail until an admin trims. * Dry-run every config change with `nats-server -c nats.conf -t` before you reload it. ### Rolling upgrades — see [Pitfalls](/learn/deployment/rolling-upgrades.md#pitfalls) * Measure rebalance time before setting `lame_duck_duration`; a duration shorter than the rebalance drops clients before replicas sync. * Transfer leadership before killing the meta-leader; upgrading it directly stalls stream ops until a new leader is elected (about 5 to 10 seconds with default timeouts). Do non-leaders first. * Set a PodDisruptionBudget with `minAvailable: 2`; without it an eviction can drain all three pods and lose quorum. * Stagger lame-duck start across ordinals; firing it on every node at once triggers a reconnect storm. * Read replicas and leader with `nats stream info ORDERS` before and after the upgrade to confirm the stream stayed R3. ### Hardening — see [Pitfalls](/learn/deployment/hardening.md#pitfalls) * Keep the JetStream `store_dir` (and the pid/ports-file dirs) inside `ReadWritePaths`; `ProtectSystem=strict` blocks writes there, not the cert reads a reload does. * Set `MemoryMax` above the node's real peak usage (memory store plus buffers plus GC headroom); a cap below the working set gets the process OOM-killed under load, not at startup. * Open cluster port 6222 between nodes; a firewall that blocks it leaves nodes unable to form quorum and showing as orphans. * Bind the monitor port 8222 to localhost; exposed to the internet it leaks version, client count, and memory. * Connect with `--creds` and `--tlsca` and publish one order to prove auth and TLS are live before you call the cluster hardened. ## See also * [Reference → Configuration](/reference/config/.md) — every config key, flag, default, and limit, versioned and exhaustive. * [Topologies deep dive](/learn/topologies/.md) — the shape this chapter runs. * [Monitoring deep dive](/learn/monitoring/.md) — what to watch once the cluster is live. --- # JetStream Deep Dive JetStream is the part of NATS that stores messages. Once a message is stored, you can read it again later, and replay it as many times as you need. Delivery is tracked per reader: a message counts as handled once it's acknowledged. This chapter builds that up one page at a time, using a single running example: the Acme `ORDERS` platform. The same server runs across the whole chapter, and its data carries over from one page to the next. ## Who this is for You've read the [Core Concepts → JetStream](/concepts/jetstream.md) primer, or you're otherwise comfortable with NATS basics: publishing, subscribing, subjects, and queue groups. ## Map | Page | What you learn | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------- | | [Your first stream](/learn/jetstream/your-first-stream.md) | Why a stream, then create the `ORDERS` stream and read its anatomy | | [Publishing](/learn/jetstream/publishing.md) | Publish into a stream and understand the `PubAck` contract | | [Reading back the stream](/learn/jetstream/reading-back.md) | Read stored messages back with a durable consumer | | [Filtering what you consume](/learn/jetstream/filtering.md) | Add a second consumer that reads only `orders.shipped` | | [Delivery and acknowledgment](/learn/jetstream/delivery-and-acknowledgment.md) | In-flight, ack, double ack, and redelivery | | [Ack responses and redelivery](/learn/jetstream/acknowledgment.md) | ack, nak, term, in-progress, and redelivery timing | | [Pull consumers in depth](/learn/jetstream/pull-consumers.md) | fetch vs consume, and the knobs that bound a pull | | [Scaling a consumer](/learn/jetstream/worker-pool.md) | Many workers split the load of one consumer | | [Ordered consumers](/learn/jetstream/ordered-consumer.md) | A throwaway in-order read of a stream, and the config behind it | | [Priority groups](/learn/jetstream/priority-groups.md) | Steer which client gets served: overflow, pinned\_client, and prioritized | | [Pausing a consumer](/learn/jetstream/pausing.md) | Stop delivery for a window, then resume | | [Shaping the stream](/learn/jetstream/shaping-the-stream.md) | Tune retention limits and discard behavior | | [Retention policies](/learn/jetstream/retention-policies.md) | Limits, Interest, and WorkQueue retention | | [Altering stream state](/learn/jetstream/altering-stream-state.md) | Delete a message or purge the stream, by hand | | [Surviving node loss](/learn/jetstream/surviving-node-loss.md) | Replicas, leaders, and storage durability | | [Advanced publishing](/learn/jetstream/advanced-publishing.md) | Async, atomic-batch, and fast-ingest publishing | | [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) | Copy one stream, or aggregate many | | [Reading messages directly](/learn/jetstream/get-direct.md) | Get one message or a batch straight from the stream, no consumer | | [Subject mapping and transforms](/learn/jetstream/subject-mapping.md) | Rewrite subjects on the way into a stream, and republish stored messages | | [Per-message TTL](/learn/jetstream/message-ttl.md) | Expire individual messages ahead of the stream | | [Stream and consumer policies](/learn/jetstream/policies.md) | Every stream and consumer policy, and which are fixed at creation | | [Where to go next](/learn/jetstream/where-next.md) | A map of what's beyond this chapter | ## Prerequisites You'll need: * A running `nats-server` with JetStream turned on. The simplest way to get one is `nats-server -js` (see [Getting Started](/concepts/getting-started/.md)). * The `nats` CLI installed and pointed at your server. --- # Ack responses and redelivery The `shipping` consumer was created with `AckPolicy=explicit`. That choice means every message it delivers must be answered. A message is not done until the client says so. This page covers the four responses a client can give — ack, nak, term ("give up on this one"), and in-progress ("still working") — and the server-side controls that decide what happens when an answer is late or never comes. ## Why the server waits for an answer A message stays in flight from the moment the server delivers it until the consumer answers. The server keeps a copy on the pending list and starts a timer. If the answer never comes, the server assumes the worker stopped and delivers the message again. This is the redelivery loop from the previous page. It has two parts. The first part is the timer. Its length is AckWait, and it defaults to 30 seconds. The second part is the answer itself. The answer takes one of four forms. ## The four responses A client answers a delivered message in exactly one of four ways. **ack**: the acknowledgment. The work succeeded. The server removes the message from the pending list and never delivers it again. This is the answer you send when a message is handled. **nak**: a negative acknowledgment. The work failed, redeliver this message. The server puts it back for another attempt. A plain nak asks for redelivery right away, or after a delay you set — see [below](#negative-ack-with-a-delay). **term**: stop trying. This message can never be handled, so don't deliver it again to anyone. The server drops it from the pending list as it does for an ack, but the work was never done. **in-progress**: still working. This is not a final answer. It resets the AckWait timer so a long job doesn't trip redelivery. The client then keeps going and answers for real later. ack, nak, and term are final. Each one closes out a delivery. in-progress extends the timer instead. **Message flow — The four ack responses (animated):** The four responses shown as consequences on a real stream of deliveries. The consumer delivers message #1 and the client acks it, so the consumer hands over the next message, #2. The client naks #2 and the same message is redelivered immediately; the retry is then acked and delivery moves on to #3. The client terms #3: it is dropped and turns red, and the next message, #4, is delivered at once. For #4 the client sends in-progress, which is not a final answer — it refills the Ack Wait window to keep the slow message in flight — then finally acks it and moves on to #5. In short: ack advances to the next message, nak redelivers the same message immediately, term drops the message and advances to the next, and in-progress extends the Ack Wait window. The rest of this page takes the three non-trivial answers — nak, term, and the controls behind them — one at a time. ## Negative ack with a delay A plain nak redelivers immediately. That's rarely what you want. A failure is often temporary. A service it calls is briefly down, a row is locked, or a rate limit is hit. Redelivering in the same instant fails again right away, and the message retries over and over with no pause. To avoid that, nak with a delay. The client tells the server to redeliver the message but wait a given time first. The server holds the message for that delay, then puts it back. Passing that delay is a client-library call — the CLI's `--nak` only asks for immediate redelivery — so there's no CLI tab here. (To space out redeliveries from the CLI, set a consumer [backoff](#backoff-a-growing-delay-between-attempts) instead.) #### JavaScript/TypeScript ``` // Bind to the existing durable and pull one message with next(). nak() tells the // server to redeliver instead of advancing. Passing a delay (in milliseconds) // holds the redelivery for that long, which backs off a downstream that isn't // ready yet. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); const m = await c.next(); if (m) { console.log(`${m.subject}: ${m.string()}`); m.nak(10_000); // ask for redelivery after 10 seconds } else { console.log("nothing to read"); } ``` #### Go ``` // Bind to the durable "shipping" consumer created earlier. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Fetch a single message from the consumer. msg, err := cons.Next() if err != nil { panic(err) } fmt.Printf("Received on %s: %s\n", msg.Subject(), string(msg.Data())) // Negatively acknowledge with a delay. The server holds the message and // redelivers it after the delay instead of right away, which is useful when // a downstream dependency needs time to recover. if err := msg.NakWithDelay(10 * time.Second); err != nil { panic(err) } fmt.Println("Message NAK'd, redelivery delayed by 10s") ``` #### Python ``` # Bind to the durable consumer and pull a single message. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") msgs = await psub.fetch(batch=1, timeout=5) msg = msgs[0] print(f"{msg.subject}: {msg.data.decode()}") # Negative ack with a delay: redeliver this message after 10 seconds. await msg.nak(delay=10) ``` #### Java ``` // Pull one message and negatively acknowledge it with a delay. // The server holds the message and redelivers it after 10 seconds, // which backs off retries instead of looping immediately. Message m = cc.next(Duration.ofSeconds(5)); if (m == null) { System.out.println("Nothing to read."); return; } System.out.printf("subject=%s data=%s%n", m.getSubject(), new String(m.getData(), StandardCharsets.UTF_8)); m.nakWithDelay(Duration.ofSeconds(10)); System.out.println("Nak sent; redelivery delayed 10s."); ``` #### Rust ``` // Bind to the existing durable consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // Fetch a single message and negatively acknowledge it with a delay. let mut messages = consumer.fetch().max_messages(1).messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("{}: {}", msg.subject, std::str::from_utf8(&msg.payload)?); // Tell the server to redeliver this message, but wait 10 seconds // before doing so. Use this to back off on a transient failure. msg.ack_with(AckKind::Nak(Some(std::time::Duration::from_secs(10)))) .await?; } ``` #### C#/.NET ``` // Bind to the existing durable consumer var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // Pull one message var msg = await consumer.NextAsync(); if (msg is { } order) { output.WriteLine($"{order.Subject}: {order.Data}"); // Negative ack with a delay: ask the server to redeliver after 10 seconds await order.NakAsync(new AckOpts { NakDelay = TimeSpan.FromSeconds(10) }); } ``` #### C ``` // Bind to the durable "shipping" consumer created earlier. jsSubOptions so; natsMsgList list = {NULL, 0}; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); // Fetch a single message from the consumer. if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, 1, 5000, &jerr); if ((s == NATS_OK) && (list.Count > 0)) { natsMsg *msg = list.Msgs[0]; printf("Received on %s: %.*s\n", natsMsg_GetSubject(msg), natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); // Negatively acknowledge with a delay. The server holds the // message and redelivers it after the delay instead of right // away, which is useful when a downstream dependency needs // time to recover. s = natsMsg_NakWithDelay(msg, 10000, NULL); if (s == NATS_OK) printf("Message NAK'd, redelivery delayed by 10s\n"); } natsMsgList_Destroy(&list); ``` A nak returns the message to the consumer, not to the worker that nak'd it. If several workers share one consumer, the redelivery can land on a different worker (see [worker pool](/learn/jetstream/worker-pool.md)). Each nak also raises a nak advisory on `$JS.EVENT.ADVISORY.CONSUMER.MSG_NAKED.ORDERS.shipping`; its fields are in [Reference → Nak advisory](/reference/jetstream/advisory/nak.md). A delayed nak sets the wait one redelivery at a time, and the client chooses it. A **backoff** on the consumer grows the wait automatically, but it only shapes redeliveries that fire when the AckWait timer runs out — it doesn't slow a nak. Backoff is covered below. ## Term: the poison message path Some failures aren't temporary. A message with a broken payload, or one that fails a check that will never pass, is a poison message. Redelivering it just wastes delivery attempts. For these, the client answers term. The message leaves the pending list and the server never delivers it again, no matter how many attempts remain. #### CLI ``` #!/bin/bash # This message can never be processed (a poison message: malformed # payload, a validation that will never pass). Terminate it so the # server drops it from the pending list and never redelivers it, # regardless of how many delivery attempts remain. # Pull the next message and term it. nats consumer next ORDERS shipping --term ``` #### JavaScript/TypeScript ``` // Bind to the existing durable and pull one message with next(). term() stops // delivery for good: the server advances past the message and never redelivers // it. Use it for a poison message that will fail every time, so it doesn't loop // forever. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); const m = await c.next(); if (m) { console.log(`${m.subject}: ${m.string()}`); m.term(); } else { console.log("nothing to read"); } ``` #### Go ``` // Bind to the durable "shipping" consumer created earlier. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Fetch a single message from the consumer. msg, err := cons.Next() if err != nil { panic(err) } fmt.Printf("Received on %s: %s\n", msg.Subject(), string(msg.Data())) // Terminate the message. This is for a poison message the consumer can never // process. The server stops redelivering it and advances past it for good. if err := msg.Term(); err != nil { panic(err) } fmt.Println("Message terminated, it will not be redelivered") ``` #### Python ``` # Bind to the durable consumer and pull a single message. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") msgs = await psub.fetch(batch=1, timeout=5) msg = msgs[0] print(f"{msg.subject}: {msg.data.decode()}") # Terminate a poison message: stop delivery for good, no redelivery. await msg.term() ``` #### Java ``` // Pull one message that can never be processed (a poison message) // and terminate it. The server stops redelivering without marking // it as successfully processed. Message m = cc.next(Duration.ofSeconds(5)); if (m == null) { System.out.println("Nothing to read."); return; } System.out.printf("subject=%s data=%s%n", m.getSubject(), new String(m.getData(), StandardCharsets.UTF_8)); m.term(); System.out.println("Message terminated; no more redeliveries."); ``` #### Rust ``` // Bind to the existing durable consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // Fetch a single message and terminate it. A terminated message is one // this consumer can never process, so the server stops redelivering it. let mut messages = consumer.fetch().max_messages(1).messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("{}: {}", msg.subject, std::str::from_utf8(&msg.payload)?); msg.ack_with(AckKind::Term).await?; } ``` #### C#/.NET ``` // Bind to the existing durable consumer var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // Pull one message var msg = await consumer.NextAsync(); if (msg is { } order) { output.WriteLine($"{order.Subject}: {order.Data}"); // Terminate: stop redelivery of a poison message that will never succeed await order.AckTerminateAsync(); } ``` #### C ``` // Bind to the durable "shipping" consumer created earlier. jsSubOptions so; natsMsgList list = {NULL, 0}; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); // Fetch a single message from the consumer. if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, 1, 5000, &jerr); if ((s == NATS_OK) && (list.Count > 0)) { natsMsg *msg = list.Msgs[0]; printf("Received on %s: %.*s\n", natsMsg_GetSubject(msg), natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); // Terminate the message. This is for a poison message the // consumer can never process. The server stops redelivering // it and advances past it for good. s = natsMsg_Term(msg, NULL); if (s == NATS_OK) printf("Message terminated, it will not be redelivered\n"); } natsMsgList_Destroy(&list); ``` **After a term**, the message is gone from this consumer but not from the stream. Term runs through the same path as an ack: the pending entry clears and the acknowledgment floor moves past the message, so it's never redelivered to this consumer. The message itself stays in the stream under the default `Limits` retention — other consumers still see it, and it ages out with the stream's limits like any other message. On a `WorkQueue` or `Interest` stream, where a handled message is removed, a term removes it just as an ack would; see [Retention policies](/learn/jetstream/retention-policies.md). The difference from an ack is that the work never happened, so the server records the give-up. It publishes a **terminated advisory** on `$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.ORDERS.shipping`, carrying the stream and consumer sequence, the delivery count, and an optional reason you can attach to the term. Watch it the way the pitfall below watches the max-deliveries advisory, so a terminated `order_id` doesn't disappear without a trace. The advisory's fields are in [Reference → Terminated advisory](/reference/jetstream/advisory/terminated.md). Use term only when the code can tell that no future attempt will succeed. When in doubt, nak with a delay and let the delivery limit below decide. ## The server controls The four responses are the client's side. The server has two settings that work with them, both on the consumer. **AckWait** is the timer. If a delivery is not ack'd, nak'd, or kept alive with in-progress before AckWait runs out, the server treats it as a silent failure and redelivers. The default is 30 seconds. Shorten it for fast work, lengthen it for slow work. **MaxDeliver** is the limit on attempts. It caps how many times the server will deliver one message before giving up. The default is `-1`, which means no limit. A message can be redelivered forever. These two cover the two ways a delivery can fail. AckWait handles the case where no answer arrives. MaxDeliver caps the case where a worker keeps sending a nak on the same message. A timeout and a nak both cause redelivery, but they're timed differently. The backoff schedule below only spaces out redeliveries that fire when the AckWait timer runs out. A bare nak redelivers right away, and a configured backoff doesn't slow it — to delay a nak, the client attaches the delay to the nak itself. Set both on the consumer with `nats consumer edit`: #### CLI ``` #!/bin/bash # Set the two server-side ack controls on the shipping consumer. # # --ack=explicit each message must be answered on its own # --wait=10s AckWait: redeliver if no answer within 10 seconds # --max-deliver=5 cap delivery attempts at 5 (default -1 = unlimited) nats consumer edit ORDERS shipping \ --ack=explicit \ --wait=10s \ --max-deliver=5 # Read the controls back. Look for: Ack Wait, Maximum Deliveries. nats consumer info ORDERS shipping ``` Read them back from `nats consumer info ORDERS shipping`: ``` Configuration: Ack Policy: Explicit Ack Wait: 10.00s Replay Policy: Instant Maximum Deliveries: 5 Max Ack Pending: 1,000 ``` With `--max-deliver=5`, a message that fails five times stops being delivered. Without a term path, that message is dropped after the fifth attempt. With a term path, your code retires the poison message itself, before the limit is reached. ## Backoff: a growing delay between attempts A flat AckWait waits the same amount of time before every redelivery. A backoff makes that wait grow. The server holds a list of delays, one per attempt: wait one second before the second delivery, five seconds before the third, 30 before the fourth. The wait between attempts grows each round instead of staying the same. The CLI builds the list for you from a range: ``` nats consumer edit ORDERS shipping --backoff=linear --backoff-steps=5 --backoff-min=1s --backoff-max=30s ``` If the list has fewer entries than MaxDeliver allows, the server reuses the last entry for the remaining attempts. Setting a backoff replaces AckWait: the first entry in the list becomes the wait before the first redelivery, so it's also the ack deadline for the first delivery. Here `--backoff-min=1s` drops the effective AckWait to 1 second, overriding the 10 seconds set earlier. Pick a `--backoff-min` at least as long as normal processing takes, or a slow job trips redelivery while it's still running. `nats consumer edit --help` lists every backoff flag. We use only a linear range here. ## Ack policy: the other values This page assumed `explicit`, the policy `shipping` was created with. AckPolicy has three more values. `explicit` answers each message on its own. It's what `shipping` uses and the right default for work that must not be lost — everything on this page depends on it. `none` requires no answer at all: the server treats a message as done the moment it's delivered, so there's no pending list, no Ack Wait, and no redelivery, and nothing on this page applies. `all` lets one ack answer every earlier message too — cheaper, but it only fits a consumer that processes strictly in order, since acking message 10 also retires 1 through 9. Strict order is a requirement you have to create, not something a consumer does by default: [Delivery and acknowledgment](/learn/jetstream/delivery-and-acknowledgment.md) showed that a redelivery arrives after later messages unless `MaxAckPending` is 1. On a consumer without that setting, acking message 10 also retires a message 7 that failed and was waiting to come back — silent data loss. A fourth value, `flow_control`, is for the push consumers the server creates for durable mirrors and sources: acks ride the flow-control responses and behave like `all`. You won't set it on a work consumer like `shipping`. The full list of available policies is in [Reference → Consumer configuration](/reference/jetstream/api/consumer/create.md). This page uses `explicit`. ## Pitfalls Each response and control is simple on its own. Most traps come from how they combine. **A plain nak retries with no delay.** A nak with no delay asks for redelivery in the same instant. A temporary failure then retries right away, fails again, and ties up one worker on one message. Don't send a bare nak for a temporary failure. Nak with a delay so the redelivery waits before it retries (covered above). A consumer backoff won't help here — it spaces out AckWait timeouts, not naks. **A poison message with no term path uses every attempt.** Without term, a broken payload is nak'd over and over until MaxDeliver gives up, using the full set of attempts and holding up the messages behind it. When the code can tell no future attempt will succeed, answer term so the message leaves the pending list at once instead of working through the limit. **MaxDeliver drops a message with no dead-letter.** When a message hits the delivery limit, the server removes it from the consumer's pending list and never delivers it again. The message stays in the stream, but the `shipping` consumer's normal output says nothing, so the drop is easy to miss. JetStream has no built-in dead-letter queue. You can still catch the drop: the server publishes a max-deliveries advisory on `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping` the moment a message goes past its limit ([Reference → Max-deliveries advisory](/reference/jetstream/advisory/max-deliver.md)). Subscribe to it so a poison `order_id` isn't dropped without notice: #### CLI ``` #!/bin/bash # A message that hits MaxDeliver leaves the consumer with no dead-letter # queue. It is not lost from the stream, but the shipping consumer stops # delivering it -- and nothing in the consumer's normal output says so. # # The drop is observable: the server publishes an advisory the moment a # message exceeds its delivery limit. Watch for it so a poison message # does not vanish unnoticed. # Subscribe to the max-deliveries advisory for the shipping consumer. # The server publishes here when a message (e.g. order ord_8w2k) is # delivered MaxDeliver times without a final ack. nats sub '$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping' # Or watch every JetStream advisory in one stream, including this one: # nats events --js-advisory --no-srv-advisory ``` #### JavaScript/TypeScript ``` // The server publishes a max-deliveries advisory when a message on the shipping // consumer hits its delivery limit without an ack. It's a plain core subject, so // subscribe with the core client to watch for poison messages. Each advisory is // JSON describing the stream, consumer, and sequence that gave up. const sub = nc.subscribe( "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping", ); for await (const m of sub) { console.log(`max deliveries reached: ${m.string()}`); } ``` #### Go ``` // Max-deliveries advisories are plain core NATS messages, so subscribe with // the core client instead of a JetStream consumer. The server publishes one // each time a message on the "shipping" consumer hits its delivery limit. subject := "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping" sub, err := nc.SubscribeSync(subject) if err != nil { panic(err) } defer sub.Unsubscribe() // Print each advisory as it arrives. for { msg, err := sub.NextMsg(nats.DefaultTimeout) if err == nats.ErrTimeout { // No advisory yet; keep waiting. continue } if err != nil { panic(err) } fmt.Printf("Max-deliveries advisory: %s\n", string(msg.Data)) } ``` #### Python ``` # JetStream publishes an advisory when a message hits its max delivery limit. # Subscribe with the core API to watch them for the shipping consumer. subject = "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping" async def handler(msg): print(f"max deliveries advisory: {msg.data.decode()}") await nc.subscribe(subject, cb=handler) print(f"Watching {subject}") # Keep the subscription open to receive advisories as they arrive. await asyncio.sleep(60) ``` #### Java ``` // Watch the max-deliveries advisory for this consumer. The server // publishes one of these whenever a message hits MaxDeliver and is // dropped, which is your signal to route it to a dead-letter flow. String advisory = "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping"; Dispatcher d = nc.createDispatcher(msg -> System.out.println("max-deliveries advisory: " + new String(msg.getData(), StandardCharsets.UTF_8))); d.subscribe(advisory); // Keep the subscription open so advisories can arrive. Thread.sleep(60_000); ``` #### Rust ``` // Subscribe to the max-deliveries advisory for the shipping consumer. // The server publishes here when a message hits the consumer's // MaxDeliver limit, which is how you find poison messages. let mut advisories = client .subscribe("$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping") .await?; while let Some(advisory) = advisories.next().await { println!( "max deliveries: {}", std::str::from_utf8(&advisory.payload)? ); } ``` #### C#/.NET ``` // Watch for messages a consumer gave up on after hitting MaxDeliver var subject = "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping"; await foreach (var advisory in client.SubscribeAsync(subject, cancellationToken: cts.Token)) { output.WriteLine($"max deliveries reached: {advisory.Data}"); } ``` #### C ``` // Max-deliveries advisories are plain core NATS messages, so // subscribe with the core client instead of a JetStream consumer. // The server publishes one each time a message on the "shipping" // consumer hits its delivery limit. const char *subject = "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.shipping"; s = natsConnection_SubscribeSync(&sub, conn, subject); // Print each advisory as it arrives. while (s == NATS_OK) { natsMsg *msg = NULL; s = natsSubscription_NextMsg(&msg, sub, 60000); if (s == NATS_OK) { printf("Max-deliveries advisory: %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); natsMsg_Destroy(msg); } } ``` **AckWait shorter than real processing time causes double work.** If a job often takes longer than AckWait and the worker never sends in-progress, the server decides the worker stopped and redelivers a message that's still being handled, so two workers run the same order. Either raise AckWait to cover the slow case, or send in-progress to reset the timer while a long job runs (both covered above). ## Where you are The `shipping` consumer is unchanged in shape (still pull, still `AckPolicy=explicit`), but now you understand it fully. You know the four answers a client gives, and the two server controls, AckWait and MaxDeliver, that decide when a message comes back and when it stops. A poison message has a clear exit through term. ## What's next One worker pulls one order at a time. The next page covers the two ways a client drives a pull consumer — [fetching a batch versus consuming a continuous flow](/learn/jetstream/pull-consumers.md) — and when to reach for each. ## See also * [Reference → Consumer configuration](/reference/jetstream/api/consumer/create.md) — AckWait, MaxDeliver, backoff arrays, and every other field. * [Reference → Terminated advisory](/reference/jetstream/advisory/terminated.md), [Nak advisory](/reference/jetstream/advisory/nak.md), and [Max-deliveries advisory](/reference/jetstream/advisory/max-deliver.md) — the events the server raises on term, nak, and a delivery-limit drop. --- # Advanced publishing [Publishing](/learn/jetstream/publishing.md) sent one order at a time and waited for each `PubAck`. That's the right default, and most services never need anything else. Two things change that: publishing at high volume, where waiting for each ack in turn is too slow, and writing a group of orders that only make sense together. JetStream has three publish modes for those cases. This page covers each one — what it does, when to reach for it, and what it costs. The `ORDERS` stream you've used all chapter doesn't change; these are choices the publisher makes. ## Async publish A normal publish blocks until the `PubAck` comes back, so a service that sends a thousand orders pays a thousand round trips end to end. An **async publish** doesn't wait: you fire each publish and keep going, then collect the acks afterward. The round trips overlap, so the same thousand orders take a fraction of the wall-clock time. The contract is unchanged — one `PubAck` per message, at-least-once storage — so you still have to check every ack. What's new is when you check it: later, in a batch, instead of right after each call. ### The order trap Async publish has one failure mode a synchronous publish doesn't, and it's the reason to understand the mode before using it. The server numbers messages in the order they arrive and get stored. When you fire orders 1 through 6 async, they're stored at sequences 1 through 6. Now say order 3's ack fails — a timeout, a dropped connection — while 4, 5, and 6 succeeded. The reorder is an after-effect of the retry. A failed publish doesn't land, so order 3 is simply missing — and you send it again, automatically or by hand. By then 4, 5, and 6 are already stored, so the re-sent order takes a higher sequence and ends up *after* the orders you sent next. **Message flow — A late async retry lands out of order (animated):** Async publishing doesn't wait for each PubAck before sending the next order, so publishes can finish out of order. The server numbers five orders as they land, but order 3's ack fails. Nothing is resent on its own — order 3 is simply missing until your code re-publishes it. By the time the retry runs, orders 4, 5, and 6 are already stored, so the re-published order 3 lands last, at sequence 7. An async publish you never check is a lost write, and a late retry reorders the stream. There are two fixes, for two different problems: * **If the ack was lost but the message was actually stored**, re-publishing stores a second copy. Give each publish a stable `Nats-Msg-Id` (the same header from [Avoiding duplicate writes](/learn/jetstream/publishing.md#avoiding-duplicate-writes)), and the server drops the repeat instead of storing it twice. * **If the order itself matters**, set `Nats-Expected-Last-Subject-Sequence` on each publish. The server stores the message only when the subject's last sequence is the one you expect, and rejects it otherwise. An out-of-order retry then fails fast — you handle the rejection — instead of silently landing in the wrong place. One rule covers the rest: **an async publish you never check is a lost write.** Firing publishes without reading the acks gives up the only guarantee a JetStream publish offers. Collect every ack and confirm it. Async publish is a client-library feature — there's no stream setting to turn it on — and the API differs by language. Each one below fires several orders without awaiting each ack, then collects and checks them all afterward: #### JavaScript/TypeScript ``` // Async publish: call publish() for every order WITHOUT awaiting each one, so // the round trips overlap. Collect the promises, await them together, then // check each result -- a rejected promise is a failed publish you must retry. const orders = [ `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200}`, `{"order_id":"ord_2zr9","customer":"globex","total_cents":7800}`, `{"order_id":"ord_5t1m","customer":"initech","total_cents":1500}`, `{"order_id":"ord_9p3x","customer":"hooli","total_cents":9900}`, ]; const pending = orders.map((order) => js.publish("orders.created", order)); const results = await Promise.allSettled(pending); results.forEach((result, i) => { if (result.status === "fulfilled") { console.log(`order ${i + 1} stored at sequence ${result.value.seq}`); } else { console.log(`order ${i + 1} failed, re-publish it: ${result.reason}`); } }); ``` #### Go ``` // Async publish: fire every order without waiting for its PubAck, then // collect the futures and check each one. The round trips overlap, so this // is far faster than publishing one at a time -- but you still have to // confirm every ack, because a publish whose ack never arrives is a lost // order, not a stored one. orders := []string{ `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200}`, `{"order_id":"ord_2zr9","customer":"globex","total_cents":7800}`, `{"order_id":"ord_5t1m","customer":"initech","total_cents":1500}`, `{"order_id":"ord_9p3x","customer":"hooli","total_cents":9900}`, } // PublishAsync returns immediately, before the server replies. futures := make([]jetstream.PubAckFuture, 0, len(orders)) for _, order := range orders { f, err := js.PublishAsync("orders.created", []byte(order)) if err != nil { panic(err) } futures = append(futures, f) } // Wait until every publish has been answered, or give up after a timeout. select { case <-js.PublishAsyncComplete(): case <-time.After(5 * time.Second): panic(fmt.Sprintf("timed out with %d publishes unconfirmed", js.PublishAsyncPending())) } // Check each ack. A publish whose ack failed has to be re-published. for i, f := range futures { select { case ack := <-f.Ok(): fmt.Printf("order %d stored at sequence %d\n", i+1, ack.Sequence) case err := <-f.Err(): fmt.Printf("order %d failed, re-publish it: %v\n", i+1, err) } } ``` #### Python ``` # nats.py has no dedicated async-publish API: publish() already returns a # coroutine, so the async-publish pattern is to start them all and gather # the results instead of awaiting one at a time. The round trips overlap, # but the client does not bound how many are in flight -- check every ack, # and add your own limit for large bursts. orders = [ b'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200}', b'{"order_id":"ord_2zr9","customer":"globex","total_cents":7800}', b'{"order_id":"ord_5t1m","customer":"initech","total_cents":1500}', b'{"order_id":"ord_9p3x","customer":"hooli","total_cents":9900}', ] pending = [js.publish("orders.created", order) for order in orders] results = await asyncio.gather(*pending, return_exceptions=True) for i, result in enumerate(results, start=1): if isinstance(result, Exception): print(f"order {i} failed, re-publish it: {result}") else: print(f"order {i} stored at sequence {result.seq}") ``` #### Java ``` // Async publish: publishAsync returns a CompletableFuture immediately, // before the server replies, so the round trips overlap. Collect the // futures, then read each ack -- a future that completes exceptionally // is a failed publish you must re-send. String[] orders = { "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200}", "{\"order_id\":\"ord_2zr9\",\"customer\":\"globex\",\"total_cents\":7800}", "{\"order_id\":\"ord_5t1m\",\"customer\":\"initech\",\"total_cents\":1500}", "{\"order_id\":\"ord_9p3x\",\"customer\":\"hooli\",\"total_cents\":9900}" }; List> futures = new ArrayList<>(); for (String order : orders) { futures.add(js.publishAsync("orders.created", order.getBytes(StandardCharsets.UTF_8))); } for (int i = 0; i < futures.size(); i++) { try { PublishAck ack = futures.get(i).get(); System.out.printf("order %d stored at sequence %d%n", i + 1, ack.getSeqno()); } catch (Exception e) { System.out.printf("order %d failed, re-publish it: %s%n", i + 1, e.getMessage()); } } ``` #### Rust ``` // Async publish: the first await only sends the message and returns a // PublishAckFuture. Collect the futures without awaiting them, so the round // trips overlap. Then await each future to read its ack -- a future that // resolves to an error is a failed publish you must re-publish. let orders = [ r#"{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200}"#, r#"{"order_id":"ord_2zr9","customer":"globex","total_cents":7800}"#, r#"{"order_id":"ord_5t1m","customer":"initech","total_cents":1500}"#, r#"{"order_id":"ord_9p3x","customer":"hooli","total_cents":9900}"#, ]; let mut acks = Vec::with_capacity(orders.len()); for order in orders { let future = js.publish("orders.created", order.into()).await?; acks.push(future); } // Now await each future to confirm the order was stored. for (i, future) in acks.into_iter().enumerate() { match future.await { Ok(ack) => println!("order {} stored at sequence {}", i + 1, ack.sequence), Err(err) => println!("order {} failed, re-publish it: {}", i + 1, err), } } ``` #### C#/.NET ``` // Async publish: PublishConcurrentAsync sends the message and returns a // future without waiting for the ack, so the round trips overlap. Collect // the futures, then await each one and call EnsureSuccess -- a future whose // ack reports an error is a failed publish you must re-send. Order[] orders = [ new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200), new Order(OrderId: "ord_2zr9", Customer: "globex", TotalCents: 7800), new Order(OrderId: "ord_5t1m", Customer: "initech", TotalCents: 1500), new Order(OrderId: "ord_9p3x", Customer: "hooli", TotalCents: 9900), ]; var futures = new List(); foreach (var order in orders) { futures.Add(await js.PublishConcurrentAsync("orders.created", order)); } for (var i = 0; i < futures.Count; i++) { await using var future = futures[i]; var ack = await future.GetResponseAsync(); ack.EnsureSuccess(); output.WriteLine($"order {i + 1} stored at sequence {ack.Seq}"); } ``` #### C ``` // Async publish: fire every order without waiting for its PubAck, // then wait for all acks at once. The round trips overlap, so this // is far faster than publishing one at a time -- but you still have // to confirm every ack, because a publish whose ack never arrives // is a lost order, not a stored one. const char *orders[] = { "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200}", "{\"order_id\":\"ord_2zr9\",\"customer\":\"globex\",\"total_cents\":7800}", "{\"order_id\":\"ord_5t1m\",\"customer\":\"initech\",\"total_cents\":1500}", "{\"order_id\":\"ord_9p3x\",\"customer\":\"hooli\",\"total_cents\":9900}"}; int i; jsPubOptions wait; // js_PublishAsync returns immediately, before the server replies. // A failed ack is reported to the ErrHandler registered on the // JetStream context (see asyncPubErr above). for (i = 0; (s == NATS_OK) && (i < 4); i++) s = js_PublishAsync(js, "orders.created", orders[i], (int) strlen(orders[i]), NULL); // Wait until every publish has been answered, or give up after // a timeout. jsPubOptions_Init(&wait); wait.MaxWait = 5000; if (s == NATS_OK) s = js_PublishAsyncComplete(js, &wait); if (s == NATS_TIMEOUT) { // Publishes still unconfirmed after the wait are lost writes: // take them back and re-publish or report each one. natsMsgList pending = {NULL, 0}; if (js_PublishAsyncGetPendingList(&pending, js) == NATS_OK) printf("timed out with %d publishes unconfirmed\n", pending.Count); natsMsgList_Destroy(&pending); } else if (s == NATS_OK) printf("all 4 orders stored\n"); ``` A note per language: Go, Java, .NET, and Rust have a dedicated async-publish call that hands back a future you collect; nats.js does it by not awaiting each `publish()` and gathering the promises; nats.py has no first-class async publish, so the example approximates it with `asyncio.gather` and you add your own limit on how many run at once. On the CLI, the everyday `nats pub` is synchronous; the async path lives in the benchmark, `nats bench js pub async orders.created --batch 1000`. ## Atomic batch publish An **atomic batch** stores a group of messages all-or-nothing. Either the whole batch commits, or none of it does. Use it when several messages only make sense together — the line items of one order, where a half-written order would leave the data inconsistent. **Message flow — An atomic batch commits all-or-nothing (animated):** An atomic batch commits all of its messages together or none at all. You open a batch and the server stages each message — nothing is visible in ORDERS yet. Add the second and third and all three sit in the staging buffer, still uncommitted. On commit, the whole batch lands in ORDERS at once, at sequences 1, 2, and 3, visible together and never half-written. If the batch is abandoned, every staged message is discarded and ORDERS is unchanged. The stream opts in with `AllowAtomicPublish`. The client opens a batch with a `Nats-Batch-Id`, tags each message with an increasing `Nats-Batch-Sequence`, and marks the last one with `Nats-Batch-Commit`. The server holds the messages in a staging buffer and writes them as a unit only on commit. The committing `PubAck` carries two extra fields, `batch` and `count`, so you can confirm the whole group landed. A batch is bounded, and it can be abandoned. By default it's capped at 1,000 messages and a stream allows at most 50 batches in flight — both are operator-configurable server limits, not fixed protocol caps. A sequence gap or an over-limit batch is rejected with an error `PubAck`, so the publisher hears about it. A batch that goes ten seconds without a message is dropped with no error reply — the server raises a `stream_batch_abandoned` advisory instead of committing a partial group. Treat the final `PubAck` as the only proof the batch committed. Atomic batch was added in server 2.12. Opt the stream in with `AllowAtomicPublish`, then open a batch, stage messages, and commit them as a unit. The CLI and nats.js have this in the core client; Go, Java, Rust, and .NET reach it through the [Synadia Orbit](https://github.com/synadia-io) companion libraries; nats.py drives it with the `Nats-Batch-*` headers directly. Each example commits three line items of one order: #### CLI ``` #!/bin/bash # Atomic batch publish from the CLI. The stream must be created with # --allow-batch (AllowAtomicPublish). The --atomic flag reads one message per # line from STDIN and commits the whole batch at end of input: either all three # line items land together, or none of them do. printf '%s\n' \ '{"order_id":"ord_8w2k","line":"sku-1"}' \ '{"order_id":"ord_8w2k","line":"sku-2"}' \ '{"order_id":"ord_8w2k","line":"sku-3"}' \ | nats pub --atomic --send-on=newline --force-stdin orders.created # Prints e.g. `Wrote batch ID: heTjQHWT0emRc98Os9nUtx Messages: 3 Sequence: 3`. # The batch is committed as a unit; a sequence gap or a stall abandons it whole. ``` #### JavaScript/TypeScript ``` // Atomic batch: all three line items of order ord_8w2k reach the stream // together, or none of them do. startBatch() opens the batch with the first // item, add() stages the next without its own round trip, and commit() // publishes the last item and seals the batch. The server stores every staged // message at once and answers with a single ack for the whole batch. const batch = await js.startBatch("orders.created", lineItems[0]); batch.add("orders.created", lineItems[1]); const ack = await batch.commit("orders.created", lineItems[2]); console.log(`batch ${ack.batch} stored ${ack.count} line items`); ``` #### Go ``` // Publish the three line items of order ORD-42 as one atomic batch: // all three land in the ORDERS stream, or none do. batch, err := jetstreamext.NewBatchPublisher(js) if err != nil { log.Fatalf("create batch publisher: %v", err) } if err := batch.Add("orders.created", []byte(`{"order":"ORD-42","sku":"COFFEE-1KG","qty":2}`)); err != nil { log.Fatalf("add line item: %v", err) } if err := batch.Add("orders.created", []byte(`{"order":"ORD-42","sku":"FILTER-100","qty":1}`)); err != nil { log.Fatalf("add line item: %v", err) } // Commit sends the final line item and atomically commits the batch. ack, err := batch.Commit(ctx, "orders.created", []byte(`{"order":"ORD-42","sku":"MUG-WHITE","qty":4}`)) if err != nil { log.Fatalf("commit batch: %v", err) } log.Printf("batch %q committed: %d line items stored", ack.BatchID, ack.BatchSize) ``` #### Python ``` # Atomic batch (ADR-50): tag each line item with one batch id and a rising # sequence, then mark the last one committed. The server stores all three or # none, and sends a single ack -- on the commit -- for the whole batch. batch = "ord_8w2k-batch" await js.client.request( "orders.created", b'{"sku":"TEE"}', headers={NATS_BATCH_ID: batch, NATS_BATCH_SEQUENCE: "1"} ) await js.client.publish( "orders.created", b'{"sku":"MUG"}', headers={NATS_BATCH_ID: batch, NATS_BATCH_SEQUENCE: "2"} ) ack = await js.publish( "orders.created", b'{"sku":"CAP"}', headers={NATS_BATCH_ID: batch, NATS_BATCH_SEQUENCE: "3", NATS_BATCH_COMMIT: NATS_BATCH_COMMIT_FINAL}, ) print(f"committed batch {ack.batch_id}: {ack.batch_size} messages stored") ``` #### Java ``` // One order, three line items, stored as a single atomic batch: // either all three messages land in the stream, or none do. BatchPublisher publisher = BatchPublisher.builder() .connection(nc) .batchId(BATCH_ID) .build(); publisher.add(SUBJECT, "{\"sku\":\"NATS-TEE\",\"qty\":2}".getBytes()); publisher.add(SUBJECT, "{\"sku\":\"NATS-MUG\",\"qty\":1}".getBytes()); PublishAck ack = publisher.commit(SUBJECT, "{\"sku\":\"NATS-CAP\",\"qty\":1}".getBytes()); System.out.println("Committed batch [" + publisher.getBatchId() + "]" + " of " + ack.getBatchSize() + " line items" + " at stream sequence " + ack.getSeqno() + "."); ``` #### Rust ``` // Open a batch, add the first two line items, then commit with the third. // Every message carries the same batch id; the order is stored only if the // commit succeeds. let mut batch = js.batch_publish().build(); batch .add("orders.created", r#"{"sku":"NATS-TEE","qty":2}"#.into()) .await?; batch .add("orders.created", r#"{"sku":"NATS-MUG","qty":1}"#.into()) .await?; let ack = batch .commit("orders.created", r#"{"sku":"NATS-CAP","qty":3}"#.into()) .await?; println!( "batch {} committed {} line items", ack.batch_id, ack.batch_size ); ``` #### C#/.NET ``` await using var batch = new NatsJSBatchPublisher(js); // Three line items of one order. They all land, or none do. await batch.AddAsync("orders.created", "{\"sku\":\"COFFEE-1KG\",\"qty\":2}"); await batch.AddAsync("orders.created", "{\"sku\":\"FILTER-100\",\"qty\":1}"); // The final item commits the batch and returns one ack for the whole order. NatsJSBatchAck ack = await batch.CommitAsync("orders.created", "{\"sku\":\"MUG-350ML\",\"qty\":4}"); Console.WriteLine($"Committed batch {ack.BatchId}: {ack.BatchSize} messages into {ack.Stream}"); ``` #### C ``` // Publish the three line items of order ORD-42 as one atomic batch: // all three land in the ORDERS stream, or none do. jsPubAck *ack = NULL; // Start the batch with the first line item. s = js_BatchPublishStart(&batch, NULL, js, msg1, NULL, &jerr); // Add the second line item to the batch. if (s == NATS_OK) s = js_BatchPublishAdd(NULL, batch, msg2, NULL, &jerr); // Commit sends the final line item and atomically commits the // batch. The ack carries the batch ID and how many messages the // batch stored. if (s == NATS_OK) s = js_BatchPublishCommit(&ack, batch, msg3, NULL, &jerr); if (s == NATS_OK) { printf("batch \"%s\" committed: %" PRIu64 " line items stored\n", ack->Batch, ack->Count); jsPubAck_Destroy(ack); } jsAtomicBatchCtx_Destroy(batch); ``` The wire protocol is the same underneath — the `Nats-Batch-Id`, `Nats-Batch-Sequence`, and `Nats-Batch-Commit` headers — so a client without an Orbit helper can drive a batch with raw headers, as the Python tab shows. See [ADR-50](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-50.md) for the full header set and limits. ## Fast-ingest batch publish A **fast-ingest batch** moves data into a stream at high speed with the server setting the pace. It's built to replace async publish: instead of the client guessing how fast to push and paying to track every ack, the client opens one channel and the server runs flow control over it. The server acks in batches and tells each publisher how fast it may go — ramping up while it keeps up, slowing down under load — so many concurrent fast publishers stay balanced. **Message flow — Fast ingest widens its ack window (animated):** Fast ingest raises publish throughput by acking batches of messages instead of every one. The server starts slow, acking each message while it gauges the load, then widens its window so one ack covers two messages, then four — fewer acks, higher throughput. The trade-off shows when a message is dropped in flight: it is never acked, so it leaves a gap the client has to notice for itself. It trades away atomicity, and that trade is the choice you make per batch. A batch can run unbounded, and a dropped message means one of two things: * **`gap: fail`** abandons the batch on the first gap, so what's stored is in order with no holes. Use it for ordered data, like the chunks of an object. * **`gap: ok`** reports the gap and keeps going. Use it where a hole is acceptable, like a stream of metrics. The stream opts in with `AllowBatchPublish`. Fast-ingest was added in server 2.14, and client support is still landing: | Client | Fast-ingest publish | | ------------------ | -------------------------------------------------------------- | | CLI | `nats bench js pub fast` (benchmark only) | | Go | Synadia Orbit — `jetstreamext.NewFastPublisher` | | Rust | Synadia Orbit — `jetstream_extra`'s `fast_publish` | | nats.js | Synadia Orbit — `@synadiaorbit/fastingest` (`startFastIngest`) | | Python, Java, .NET | `AllowBatchPublish` stream flag; Orbit publishers catching up | Because there's no stable public publisher in most clients, this page doesn't show per-language code for it. When you need fast ingest today, the practical paths are the Orbit libraries for Go, Rust, and JavaScript; the rest will follow. The flow-control protocol is in [ADR-50](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-50.md). ## Choosing a mode Most services should publish one at a time and check each `PubAck`. Reach past that only when one of these is true: | Mode | Reach for it when | All-or-nothing | Opt-in | | ------------- | ------------------------------------------------------------------------ | -------------- | -------------------- | | One at a time | The default — simple, ordered, safe to retry | n/a | none | | Async | You publish at volume and one-at-a-time is too slow | no | none | | Atomic batch | A group of messages must land together or not at all | yes | `AllowAtomicPublish` | | Fast-ingest | You need maximum sustained throughput and can tolerate (or fail on) gaps | no | `AllowBatchPublish` | ## Pitfalls A few things separate these modes from a plain publish. **An async publish you never check is a lost write.** Firing publishes without reading the `PubAcks` gives up the one guarantee a JetStream publish offers. Collect and check every ack — and if order matters, add `Nats-Expected-Last-Subject-Sequence` so a retry fails fast instead of landing out of order. **An atomic batch can be abandoned.** A sequence gap or a batch over the size limit (1,000 messages by default) comes back as an error `PubAck`. A batch that goes ten seconds without a message is dropped with no error reply — only an advisory. Either way the whole batch is dropped, so treat the final `PubAck` as the only proof it committed; don't assume a half-sent batch landed. **`AllowAtomicPublish` and async persistence don't mix.** A stream set to persist asynchronously (`PersistMode: async`) rejects atomic publishing, because the atomicity depends on the synchronous write path. Fast-ingest batches are fine on such a stream. **Fast-ingest gaps lose data in `gap: ok` mode.** That mode keeps going past a dropped message on purpose. Use it only when a hole is acceptable, like metrics; for anything you can't lose, use `gap: fail` or an atomic batch. ## Where you are Nothing about `ORDERS` changed on this page. You now have the three publish modes beyond one-at-a-time, and the code for the one most services actually reach for: * **Async** overlaps round trips for throughput. You collect and check every `PubAck` yourself, and watch the order trap on retries. * **Atomic batch** commits a group all-or-nothing, gated by `AllowAtomicPublish`. * **Fast-ingest batch** trades atomicity for server-paced speed, gated by `AllowBatchPublish`. The default one-at-a-time publish from the [publishing page](/learn/jetstream/publishing.md) still fits most services. ## What's next The next page covers copying a stream's data elsewhere: **mirrors and sources**, the building blocks for read-replicas, aggregation, and disaster recovery across regions. ## See also * [ADR-50: JetStream Batch Publishing](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-50.md) — the full atomic and fast-ingest protocols, headers, and limits. * [Reference → Create Stream](/reference/jetstream/api/stream/create.md) — the `allow_atomic` and `allow_batched` stream settings. * [Reference → JetStream Headers](/reference/jetstream/api/headers.md) — `Nats-Expected-Last-Subject-Sequence` and the batch headers. * [Reference → Publish Acknowledgement](/reference/jetstream/api/stream/pub-ack.md) — the `batch` and `count` fields a batch `PubAck` carries. --- # Altering stream state Retention limits remove messages for you, on a schedule the server runs. Sometimes you need to remove something by hand: one bad message a buggy producer wrote, or every message in a stream you're clearing out for a fresh test run. JetStream gives you two manual tools for that. You can delete a single message, or purge many at once. Both change what the stream holds without touching its config or its consumers, and both leave the sequence numbers alone. How the numbers behave is worth understanding before you use either tool. ## Delete one message Every stored message has a sequence number, the one the `PubAck` gave back when you published it. To remove a specific message, give the server that number: #### CLI ``` #!/bin/bash # Remove one stored message by its sequence number. `nats stream rmm` securely # erases it: the server overwrites the stored bytes so the old contents can't # be read back. It asks for confirmation first; add --force to skip the prompt. nats stream rmm ORDERS 2 ``` #### C ``` // Remove the message stored at sequence 2 from ORDERS. This marks // it erased but leaves the stored bytes in place until they are // later overwritten, which is cheap. s = js_DeleteMsg(js, "ORDERS", 2, NULL, &jerr); if (s == NATS_OK) printf("Deleted message 2 from ORDERS\n"); // For a message that held data it shouldn't have, use js_EraseMsg // instead: the server overwrites the stored bytes right away, the // way `nats stream rmm` does, so the old contents can't be read // back. It is slower for it. // // s = js_EraseMsg(js, "ORDERS", 2, NULL, &jerr); ``` `rmm` is "remove message." It takes the stream and the sequence to drop, and it asks for confirmation first (`Really remove message 2 from Stream ORDERS`). Add `--force` to skip the prompt in a script. `nats stream rmm` securely removes the message: the server overwrites the stored bytes so the old contents can't be read back. That's the right default for a message that held data it shouldn't have, such as a card number a producer logged by mistake. **Message flow — Deleting a message leaves a gap (animated):** Deleting a message removes one sequence without renumbering the rest. ORDERS holds five messages, #1 through #5. The server overwrites sequence 3 — its bytes can't be read back — but the surrounding sequences keep their numbers, so a gap opens at 3. New orders keep arriving; the next one lands at #6, never reusing 3. From a client library you choose that cost yourself. `DeleteMsg(seq)` marks the message erased but leaves its bytes in place until they're later overwritten, which is cheap. `SecureDeleteMsg(seq)` overwrites them right away the way the CLI does, and is slower for it. The only difference the server sees is a single `no_erase` flag on the delete request; the full request is in [Reference](/reference/jetstream/api/stream/msg-delete.md). ## Purge the stream To clear a whole stream at once, purge it: #### CLI ``` #!/bin/bash # Clear every message from ORDERS. The stream itself stays — same config, same # consumers, same name — only the messages go. It asks for confirmation first; # add --force to skip the prompt in a script. nats stream purge ORDERS ``` #### C ``` // Clear every message from ORDERS. The stream itself stays -- same // config, same consumers, same name -- only the messages go. s = js_PurgeStream(js, "ORDERS", NULL, &jerr); if (s == NATS_OK) printf("Purged ORDERS\n"); ``` That removes every message in `ORDERS` and reports how many it dropped (`Purged N messages from ORDERS`, for whatever the stream held). Like `rmm`, it asks for confirmation first; add `--force` to skip the prompt in a script. The stream itself stays: same config, same consumers, same name — only the messages are removed. (Don't confuse it with `nats stream rm ORDERS`, which deletes the whole stream, config and consumers included.) **Message flow — Purge clears data, not the counter (animated):** Purging empties a stream but never rewinds its sequence counter. ORDERS holds five messages, first sequence 1; a purge removes all five, the stream stays, and its first sequence is now 6. The next publish lands at #6, not #1 — purge clears the data, not the numbering. A bare purge removes everything. Three optional flags narrow it: #### CLI ``` #!/bin/bash # A bare purge removes everything. Three optional flags narrow it. # Remove only the shipped events, leave everything else. nats stream purge ORDERS --subject orders.shipped # Remove everything up to but not including sequence 100 (keep 100 onward). nats stream purge ORDERS --seq 100 # Keep only the most recent 50 messages. nats stream purge ORDERS --keep 50 ``` #### C ``` // A bare purge removes everything. Three optional purge options // narrow it. jsOptions o; // Remove only the shipped events, leave everything else. jsOptions_Init(&o); o.Stream.Purge.Subject = "orders.shipped"; s = js_PurgeStream(js, "ORDERS", &o, &jerr); // Remove everything up to but not including sequence 100 // (keep 100 onward). jsOptions_Init(&o); o.Stream.Purge.Sequence = 100; if (s == NATS_OK) s = js_PurgeStream(js, "ORDERS", &o, &jerr); // Keep only the most recent 50 messages. jsOptions_Init(&o); o.Stream.Purge.Keep = 50; if (s == NATS_OK) s = js_PurgeStream(js, "ORDERS", &o, &jerr); ``` `--subject` limits the purge to one subject. `--seq` counts from the bottom (drop everything older than a sequence); `--keep` counts from the top (retain a number of the newest). You can't use `--seq` and `--keep` together, since one works from each end, but either one can pair with `--subject` to count within a single subject. ## Sequence numbers never backfill Delete a message or purge a range and you leave a hole. Sequence `2` is gone; `1` and `3` stay exactly where they were. The server does not renumber `3` down to close the gap, and it never hands `2` out again to a future message. This is the rule the [publishing page](/learn/jetstream/publishing.md) promised: sequence numbers only ever climb. After a full purge of `ORDERS`, the next message you publish doesn't start over at `1`. It gets the number after the last one the stream ever held, because purge sets the stream's first sequence to one past its last. **Message flow — A deletion leaves a permanent gap (animated):** Sequence numbers are assigned once and never reused, so a deletion leaves a permanent gap. Sequence 2 was deleted earlier, so ORDERS holds 1, 3, 4, and 5 with a hole at 2. New orders don't backfill the hole: the next one lands at #6, the next number up. Consumers handle the gaps without trouble. A consumer reading `ORDERS` skips the sequences that are gone; it never blocks waiting for a deleted message, and a missing `2` is not redelivered. This means a stored sequence is a stable address. If you saved "order `ord_8w2k` is at sequence 2" alongside your business record, that pointer either still points at the same message or points at nothing. It never points at a different message, because `2` never gets reused. ## Manual removal versus automatic Everything on this page is manual: you ran a command to remove something deliberately. The server also removes messages on its own, and that lives on two other pages: * [Shaping the stream](/learn/jetstream/shaping-the-stream.md) — `MaxAge`, `MaxBytes`, `MaxMsgs`, and discard policy trim the stream to stay within its limits. * [Per-message TTL](/learn/jetstream/message-ttl.md) — a single message expires on its own clock. Use delete and purge for one-off removals. Use limits and TTL for the steady state. ## Pitfalls Some ways manual removal goes wrong. **`purge` empties the stream; `rm` destroys it.** The two commands look similar but do different things. `nats stream purge ORDERS` removes the messages and leaves the stream ready for more. `nats stream rm ORDERS` deletes the stream itself, including its config and every consumer. Use `purge` when you only mean to clear messages; don't type `rm` for that, and read the confirmation prompt before answering it. **A deleted sequence is gone, not renumbered.** Code that assumes messages run `1..N` with no holes breaks the first time anyone removes one. Do treat a sequence as a stable address that may or may not still hold a message; don't work out "the next message" as `count + 1`, and don't read a message count as the highest sequence. **A purge can't be undone.** The messages are gone the moment the command returns. On a stream that matters, set `DenyPurge` in its config so a stray call can't wipe it, and keep a [mirror](/learn/jetstream/mirrors-and-sources.md) if the data is worth recovering. Setting `DenyPurge` is permanent: the server refuses any later update that turns it off, so the only way to purge that stream again is to delete and recreate it. Try it on a throwaway stream, not `ORDERS`. ``` # Guard a stream so purge requests are refused nats stream add GUARDED --subjects "guarded.>" --defaults nats stream edit GUARDED --deny-purge # A purge now fails instead of silently emptying the stream nats stream purge GUARDED ``` ## Where you are You can now remove messages from `ORDERS` by hand: one at a time with `nats stream rmm`, or many at once with `nats stream purge` and its `--subject`, `--seq`, and `--keep` flags. You saw that removal leaves gaps the server never fills back in, and that the sequence counter only ever climbs. If you ran a full `nats stream purge ORDERS` above, the stream is empty now; republish the orders from the [publishing page](/learn/jetstream/publishing.md) to repopulate `ORDERS` for the next page. The new orders get fresh, higher sequence numbers, not `1`, `2`, `3` — purge left the counter where it was, so publishing continues from one past the stream's last sequence. Everything else about the stream, its config and its consumers, is exactly as the previous page left it. ## What's next Manual and automatic removal are both covered, so the stream's contents are fully in your hands on a single server. The next page asks what happens to all of it when the server itself goes away. ## See also * [Reference → Stream Purge](/reference/jetstream/api/stream/purge.md) — the exact purge request, `filter`, `seq`, and `keep`. * [Reference → Delete Message](/reference/jetstream/api/stream/msg-delete.md) — the delete request and the `no_erase` flag. * [Shaping the stream](/learn/jetstream/shaping-the-stream.md) — the limits that remove messages without you asking. --- # Delivery and acknowledgment The [reading-back](/learn/jetstream/reading-back.md) and [filtering](/learn/jetstream/filtering.md) pages created consumers — `billing` to read the stream back, then `analytics` to filter it — and acked each message on the happy path, where every message succeeded. This page is the part those pages took for granted: what an acknowledgment actually does, and what happens to a message that's delivered but never acked. That's the **ack/redeliver loop**, and it's one of the pieces that give JetStream its **at-least-once** delivery: a message stays available and is redelivered until a reader confirms it, so it's handled at least once even when things fail. A reader can crash mid-message, a handler can throw, a process can be killed — and the message it was working on comes back instead of vanishing. (At-least-once also leans on the durable stream from the earlier pages; the ack loop is the consumer half.) You already have the orders in `ORDERS` and the `billing` and `analytics` consumers from the previous pages. ## A consumer to experiment with The demos below deliberately skip an ack to watch a message come back, so they're easier to follow on a consumer of their own. Create one named `shipping` — a pull consumer with explicit ack, the same kind you made on the previous pages: #### CLI ``` #!/bin/bash # Create a durable pull consumer named `shipping` on the ORDERS stream. # --pull: reader fetches messages on demand (pull consumer) # --ack explicit: every delivered message must be individually acknowledged nats consumer add ORDERS shipping --pull --ack explicit --defaults ``` #### JavaScript/TypeScript ``` // Create a durable pull consumer on the ORDERS stream. The durable keeps its // position under a fixed name, so a reader can come back later and pick up where // it left off. ack_policy Explicit means the server only advances that position // once a reader acks each message. deliver_policy All starts from the first // stored message. add() is idempotent: calling it again with the same config is // a no-op. const jsm = await jetstreamManager(nc); await jsm.consumers.add("ORDERS", { durable_name: "shipping", ack_policy: AckPolicy.Explicit, deliver_policy: DeliverPolicy.All, }); console.log("Created durable consumer: shipping"); ``` #### Go ``` // Create a durable pull consumer named "shipping" on the ORDERS stream. // DeliverAllPolicy starts from the first message in the stream, and // AckExplicitPolicy means every message must be acknowledged by hand. cons, err := js.CreateOrUpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{ Durable: "shipping", DeliverPolicy: jetstream.DeliverAllPolicy, AckPolicy: jetstream.AckExplicitPolicy, }) if err != nil { panic(err) } // Confirm the consumer is ready. fmt.Printf("Created consumer: %s\n", cons.CachedInfo().Name) ``` #### Python ``` # Create a durable pull consumer that reads the whole stream from the start. # add_consumer is idempotent: calling it again with the same config is a no-op. await js.add_consumer( "ORDERS", ConsumerConfig( durable_name="shipping", ack_policy=AckPolicy.EXPLICIT, deliver_policy=DeliverPolicy.ALL, ), ) print("Created durable consumer shipping on stream ORDERS") ``` #### Java ``` // Create a durable pull consumer that delivers every stored message. // AckPolicy.Explicit means each message must be acknowledged. ConsumerContext cc = sc.createOrUpdateConsumer( ConsumerConfiguration.builder() .durable("shipping") .deliverPolicy(DeliverPolicy.All) .ackPolicy(AckPolicy.Explicit) .build()); System.out.println("Created durable consumer: " + cc.getConsumerName()); ``` #### Rust ``` // Create a durable pull consumer that starts at the first stored message // and acknowledges each message explicitly, so the server tracks progress. let stream = js.get_stream("ORDERS").await?; let consumer = stream .create_consumer(pull::Config { durable_name: Some("shipping".to_string()), deliver_policy: DeliverPolicy::All, ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; println!("Created durable consumer: {}", consumer.cached_info().name); ``` #### C#/.NET ``` // Create a durable pull consumer that delivers every stored message from the start var consumer = await js.CreateOrUpdateConsumerAsync("ORDERS", new ConsumerConfig("shipping") { AckPolicy = ConsumerConfigAckPolicy.Explicit, DeliverPolicy = ConsumerConfigDeliverPolicy.All, }); output.WriteLine($"Created durable consumer {consumer.Info.Config.Name} on stream ORDERS"); ``` #### C ``` // Create a durable pull consumer named "shipping" on the ORDERS stream. // js_DeliverAll starts from the first message in the stream, and // js_AckExplicit means every message must be acknowledged by hand. jsConsumerConfig cfg; jsConsumerConfig_Init(&cfg); cfg.Durable = "shipping"; cfg.DeliverPolicy = js_DeliverAll; cfg.AckPolicy = js_AckExplicit; s = js_AddConsumer(&ci, js, "ORDERS", &cfg, NULL, &jerr); if (s == NATS_OK) { // Confirm the consumer is ready. printf("Created consumer: %s\n", ci->Name); } ``` It starts at the beginning of `ORDERS`, so it has the stored orders to work through. ## The delivery lifecycle A message moves through a few states on a consumer: 1. **Delivered, in flight.** The consumer hands the message to a reader. The message stays in the stream, and the consumer's **acknowledgment floor** — how far it has confirmed handling — hasn't moved past it. The message is in flight: out with a reader, not yet confirmed. 2. **Acked.** The reader processes the message and sends an **ack**. The floor advances past it; the message is done as far as this consumer is concerned. 3. **Redelivered.** If no ack arrives within the **Ack Wait** deadline, the server assumes the reader failed and delivers the message again — to the same reader or another one on the same consumer. **Message flow — Consumers with independent cursors (animated):** Three consumers read one 8-message stream from independent positions. Inventory, Email, and Analytics each start at a different point and move at their own speed, and each keeps its own cursor — one consumer catching up never moves another's position. The stream holds a single shared copy of every message and serves each consumer from where it left off. The gap between what's been delivered and what's been acked is the set of in-flight messages. The rest of this page opens and closes that gap on purpose. ## Acknowledge a message Pull one message from `shipping` and ack it: #### CLI ``` #!/bin/bash # Pull one message from the `shipping` consumer and acknowledge it. # --ack (default) acks each received message, advancing the durable cursor. nats consumer next ORDERS shipping --ack ``` #### JavaScript/TypeScript ``` // Bind to the existing durable and pull one message with next(). Handle it, then // ack() so the server advances the consumer past it and won't redeliver it. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); const m = await c.next(); if (m) { console.log(`${m.subject}: ${m.string()}`); m.ack(); } else { console.log("nothing to read"); } ``` #### Go ``` // Bind to the durable "shipping" consumer created earlier. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Fetch a single message from the consumer. msg, err := cons.Next() if err != nil { panic(err) } fmt.Printf("Received on %s: %s\n", msg.Subject(), string(msg.Data())) // Acknowledge so the server advances the consumer past this message. if err := msg.Ack(); err != nil { panic(err) } ``` #### Python ``` # Bind to the durable consumer and pull a single message. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") msgs = await psub.fetch(batch=1, timeout=5) msg = msgs[0] print(f"{msg.subject}: {msg.data.decode()}") # Explicit ack removes the message from this consumer's pending list. await msg.ack() ``` #### Java ``` // Pull one message, process it, then acknowledge so the server // advances the consumer and never redelivers this message. Message m = cc.next(Duration.ofSeconds(5)); if (m == null) { System.out.println("Nothing to read."); return; } System.out.printf("subject=%s data=%s%n", m.getSubject(), new String(m.getData(), StandardCharsets.UTF_8)); m.ack(); ``` #### Rust ``` // Bind to the existing durable consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // Fetch a single message, process it, then acknowledge it. let mut messages = consumer.fetch().max_messages(1).messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("{}: {}", msg.subject, std::str::from_utf8(&msg.payload)?); // Acknowledge so the server records this message as handled. msg.ack().await?; } ``` #### C#/.NET ``` // Bind to the existing durable consumer var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // Pull one message and process it var msg = await consumer.NextAsync(); if (msg is { } order) { output.WriteLine($"{order.Subject}: {order.Data}"); // Acknowledge the message so the consumer advances past it await order.AckAsync(); } ``` #### C ``` // Bind to the durable "shipping" consumer created earlier. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); // Fetch a single message from the consumer. if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, 1, 5000, &jerr); if (s == NATS_OK) { natsMsg *msg = list.Msgs[0]; printf("Received on %s: %.*s\n", natsMsg_GetSubject(msg), natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); // Acknowledge so the server advances the consumer past this message. s = natsMsg_Ack(msg, NULL); } ``` You get the first stored order, and the ack confirms it: ``` [10:14:52] subj: orders.created / tries: 1 / cons seq: 1 / str seq: 1 / pending: 2 {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} Acknowledged message ``` `tries: 1` is the delivery count — this is the first time the message has gone out. Check the consumer: ``` State: Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 Acknowledgment Floor: Consumer sequence: 1 Stream sequence: 1 Outstanding Acks: 0 out of maximum 1,000 Redelivered Messages: 0 ``` Both cursors advanced to sequence 1: delivered, then acked, with nothing left in flight. You read the `tries`, sequences, and pending count off the delivered message itself — no `nats consumer info` call needed, as the previous pages noted. The full `State` block is here only to show the floor moving. ## Double ack: make sure the ack landed A plain ack is fire-and-forget. The client sends it and moves on without waiting to hear that the server recorded it. Almost always that's fine. But if the ack is lost on the way to the server — a connection drops at the wrong moment — the server never advances the floor, Ack Wait elapses, and the message is redelivered. The reader handled it once and will see it again. A **double ack** closes that window. The client sends the ack and waits for the server to confirm it before treating the message as done. Reach for it when reprocessing a message would be harmful and you can't make the handler idempotent — a payment capture, say. It costs a round-trip per message, so it's a deliberate choice, not the default. **Message flow — Plain ack vs double ack (animated):** Plain ack versus double ack. In both, the server (the consumer) delivers a message to the client and the client sends an ack back. With a plain ack the client moves on the instant it sends the ack (fire-and-forget). With a double ack the ack is a request: the client waits for the server to confirm the ack landed before treating the message as done. There's no CLI flag for it — a double ack is a client-library call. Pull a message and double-ack it: #### JavaScript/TypeScript ``` // Bind to the existing durable and pull one message with next(). ackAck() sends // the ack and waits for the server to confirm it landed, so you know the // consumer position advanced before moving on. Plain ack() is fire-and-forget. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); const m = await c.next(); if (m) { console.log(`${m.subject}: ${m.string()}`); await m.ackAck(); } else { console.log("nothing to read"); } ``` #### Go ``` // Bind to the durable "shipping" consumer created earlier. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Fetch a single message from the consumer. msg, err := cons.Next() if err != nil { panic(err) } fmt.Printf("Received on %s: %s\n", msg.Subject(), string(msg.Data())) // DoubleAck blocks until the server confirms it recorded the acknowledgment. // Use it when losing an ack would be worse than the extra round trip. if err := msg.DoubleAck(ctx); err != nil { panic(err) } fmt.Println("Acknowledgment confirmed by the server") ``` #### Python ``` # Bind to the durable consumer and pull a single message. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") msgs = await psub.fetch(batch=1, timeout=5) msg = msgs[0] print(f"{msg.subject}: {msg.data.decode()}") # Double ack: wait for the server to confirm the ack was recorded. await msg.ack_sync() ``` #### Java ``` // Pull one message and acknowledge it with a confirmed ack. // ackSync waits for the server to confirm the ack was stored, so // you know the consumer advanced before moving on. Message m = cc.next(Duration.ofSeconds(5)); if (m == null) { System.out.println("Nothing to read."); return; } System.out.printf("subject=%s data=%s%n", m.getSubject(), new String(m.getData(), StandardCharsets.UTF_8)); m.ackSync(Duration.ofSeconds(1)); System.out.println("Ack confirmed by server."); ``` #### Rust ``` // Bind to the existing durable consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // Fetch a single message and confirm the ack with the server. let mut messages = consumer.fetch().max_messages(1).messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("{}: {}", msg.subject, std::str::from_utf8(&msg.payload)?); // double_ack waits for the server to confirm the acknowledgment, // so you know the message will not be redelivered. msg.double_ack().await?; } ``` #### C#/.NET ``` // Bind to the existing durable consumer var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // Pull one message and process it var msg = await consumer.NextAsync(); if (msg is { } order) { output.WriteLine($"{order.Subject}: {order.Data}"); // Double-ack: wait for the server to confirm the acknowledgment was stored await order.AckAsync(new AckOpts { DoubleAck = true }); } ``` #### C ``` // Bind to the durable "shipping" consumer created earlier. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); // Fetch a single message from the consumer. if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, 1, 5000, &jerr); if (s == NATS_OK) { natsMsg *msg = list.Msgs[0]; printf("Received on %s: %.*s\n", natsMsg_GetSubject(msg), natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); // AckSync blocks until the server confirms it recorded the // acknowledgment. Use it when losing an ack would be worse than // the extra round trip. s = natsMsg_AckSync(msg, NULL, &jerr); if (s == NATS_OK) printf("Acknowledgment confirmed by the server\n"); } ``` The call is named differently across clients — `DoubleAck` in Go, `ackAck` in JavaScript, `ack_sync` in Python, `ackSync` in Java, `AckAsync` (with the double-ack option) in .NET, `double_ack` in Rust — but each waits for the server to confirm the ack before returning. ## Redelivery: when a message isn't acked To see redelivery, pull a message but skip the ack: #### CLI ``` #!/bin/bash # Pull one message from the `shipping` consumer but skip the ack. # --no-ack leaves the message in flight; after Ack Wait elapses it is redelivered. nats consumer next ORDERS shipping --no-ack ``` #### JavaScript/TypeScript ``` // Bind to the existing durable and pull one message with next(). Here we handle // it but never ack. The message stays in flight: once Ack Wait passes with no // ack, the server redelivers it to a reader of this consumer. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); const m = await c.next(); if (m) { console.log(`${m.subject}: ${m.string()}`); // no ack here — the message stays in flight and is redelivered after Ack Wait } else { console.log("nothing to read"); } ``` #### Go ``` // Bind to the durable "shipping" consumer created earlier. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Fetch a single message from the consumer. msg, err := cons.Next() if err != nil { panic(err) } fmt.Printf("Received on %s: %s\n", msg.Subject(), string(msg.Data())) // We deliberately do not acknowledge this message. It stays in flight, and // the server redelivers it once the consumer's Ack Wait window elapses. ``` #### Python ``` # Bind to the durable consumer and pull a single message. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") msgs = await psub.fetch(batch=1, timeout=5) msg = msgs[0] print(f"{msg.subject}: {msg.data.decode()}") # No ack here. The message stays in flight and is redelivered after Ack Wait. ``` #### Java ``` // Pull one message but do not acknowledge it. Because there is no // ack, the message stays in flight and the server redelivers it // after the consumer's Ack Wait window expires. Message m = cc.next(Duration.ofSeconds(5)); if (m == null) { System.out.println("Nothing to read."); return; } System.out.printf("subject=%s data=%s%n", m.getSubject(), new String(m.getData(), StandardCharsets.UTF_8)); // No m.ack() — the message is redelivered after Ack Wait. ``` #### Rust ``` // Bind to the existing durable consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // Fetch a single message and read it, but do not acknowledge it. let mut messages = consumer.fetch().max_messages(1).messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("{}: {}", msg.subject, std::str::from_utf8(&msg.payload)?); // Without an ack the message stays in flight and the server // redelivers it once the Ack Wait period expires. } ``` #### C#/.NET ``` // Bind to the existing durable consumer var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // Pull one message and look at it var msg = await consumer.NextAsync(); if (msg is { } order) { output.WriteLine($"{order.Subject}: {order.Data}"); // No Ack here: the message stays in flight and is redelivered after Ack Wait } ``` #### C ``` // Bind to the durable "shipping" consumer created earlier. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); // Fetch a single message from the consumer. if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, 1, 5000, &jerr); if (s == NATS_OK) { natsMsg *msg = list.Msgs[0]; printf("Received on %s: %.*s\n", natsMsg_GetSubject(msg), natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); } // We deliberately do not acknowledge this message. It stays in flight, // and the server redelivers it once the consumer's Ack Wait window // elapses. ``` This delivers the next order, sequence 2, and leaves it in flight. The state now shows the gap: ``` State: Last Delivered Message: Consumer sequence: 2 Stream sequence: 2 Acknowledgment Floor: Consumer sequence: 1 Stream sequence: 1 Outstanding Acks: 1 out of maximum 1,000 Redelivered Messages: 0 ``` Last Delivered moved to 2, but the Acknowledgment Floor stayed at 1. That one message is in flight: `Outstanding Acks: 1`. Wait out the Ack Wait deadline — 30 seconds by default — pull again, and the server hands you sequence 2 a second time, now with `tries: 2`. It came back because you never acked it. A redelivery doesn't slot back into stream order. A consumer usually has several messages in flight at once — the `out of maximum 1,000` in the state above is **MaxAckPending**, the cap on how many it delivers before they're acked. While one message waits to be redelivered, the consumer keeps handing out later ones, so the repeat can arrive *after* messages with higher sequence numbers. **Message flow — Out-of-order redelivery (animated):** Redelivery is in delivery order, not stream order. A consumer delivers messages 1 through 5; the client acks 1, 2, 4, and 5 but skips 3. While 3 sits unacked an Ack Wait timer fills; when it completes the server redelivers message 3, so it arrives after 4 and 5 — out of stream order — and the client acks it that second time, before the consumer continues with message 6. If a consumer must process strictly in order, set MaxAckPending to 1. It then delivers one message at a time and won't move past it until it's acked, so a redelivery always comes back before anything new: ``` nats consumer edit ORDERS shipping --max-pending 1 ``` That ordering guarantee costs throughput — one message in flight means no overlap between handlers — so use it only when order actually matters. A reader doesn't have to wait out the deadline, and a stuck message doesn't have to retry forever. A reader can answer a message in other ways — ask for an immediate retry, give up on a message that can never be handled, or signal that it's still working — and the server has controls that decide how long it waits between retries and when it stops. Those responses and controls are the [next page](/learn/jetstream/acknowledgment.md). ## Pitfalls **Forgetting to ack.** A handler that processes a message but never acks looks identical to a crashed reader: the message stays in flight, Ack Wait elapses, and it's redelivered — and with the default unlimited delivery limit, that repeats forever. Always ack on the success path. The [next page](/learn/jetstream/acknowledgment.md) covers how to retire a message that genuinely can't be processed, so it stops coming back. **Acking the same message twice.** Once you ack a message, the floor moves past it. A second ack does nothing useful. Some clients (Go, Python) reject it locally with *"message was already acknowledged"*; others ignore or resend it, and the server ignores the duplicate either way. Ack each delivery exactly once, in one place in your handler. ## Where you are `shipping` is a durable pull consumer with explicit ack, and you've seen the core of the loop: * a message is **delivered** and held **in flight** until it's confirmed * an **ack** (or a **double ack**, when a lost ack would hurt) advances the acknowledgment floor past it * an unacked message is **redelivered** after the Ack Wait deadline The delivery count (`tries`) rides along on each message, so a reader always knows how many times it's seen one. ## What's next You've seen the happy path and that an unacked message comes back. The next page goes deep on the four ways a client answers a message — ack, nak, term, and in-progress — and the server controls (Ack Wait, MaxDeliver, backoff) that decide when a message comes back and when it stops. ## See also * [Reference → Create Consumer](/reference/jetstream/api/consumer/create.md) — every consumer option, the four ack policies, and push consumers. --- # Filtering what you consume The `billing` consumer from the previous page reads every message in the `ORDERS` stream — no filter, the whole log. A reporting job needs only one thing: when an order ships. It has no use for `orders.created` or `orders.canceled`, so delivering those messages to it would be wasted work on both sides. This page adds a second consumer that reads only `orders.shipped`, and shows why one consumer doesn't interfere with another. ## What a filter does A **filter** is a subject pattern attached to a consumer. The consumer receives only the messages whose subject matches the filter; the rest of the stream is skipped. The pattern can be a literal subject like `orders.shipped`, or a wildcard: a filter of `orders.*` matches every order event, while `orders.shipped` matches only the ships. The `*` and `>` wildcards behave exactly as they do for a [core subscription](/learn/core-nats/subjects-and-wildcards.md). The stream still captures all of `orders.>`; nothing about the stream changes. The filter lives on the consumer and decides which of the stored messages this consumer receives. Create the `analytics` consumer with a filter of `orders.shipped`: #### CLI ``` #!/bin/bash # Add a second consumer to the ORDERS stream that reads only orders.shipped. # The --filter flag pins the consumer to a single subject. nats consumer add ORDERS analytics \ --filter "orders.shipped" \ --pull \ --ack explicit \ --defaults # Inspect it — the config now shows a Filter Subject line. nats consumer info ORDERS analytics # Pull from analytics: only orders.shipped messages come back. # orders.created and orders.canceled are skipped for this consumer. nats consumer next ORDERS analytics --count 5 ``` #### JavaScript/TypeScript ``` // ORDERS already holds orders.created and orders.shipped messages. Create a // durable pull consumer that only sees one of those subjects: filter_subject // "orders.shipped" tells the server to skip everything else. ack_policy Explicit // means a reader acks each delivered message. add() is idempotent. const jsm = await jetstreamManager(nc); await jsm.consumers.add("ORDERS", { durable_name: "analytics", ack_policy: AckPolicy.Explicit, filter_subject: "orders.shipped", }); console.log("Created filtered consumer: analytics (orders.shipped)"); // Bind to it and pull a small batch. Only orders.shipped come back — the filter // drops orders.created before it ever reaches this consumer. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "analytics"); const msgs = await c.fetch({ max_messages: 5, expires: 2000 }); for await (const m of msgs) { console.log(m.subject); await m.ack(); } ``` #### Go ``` // Create a durable pull consumer named "analytics" on the ORDERS stream. // FilterSubject narrows delivery to "orders.shipped", so the consumer never // sees "orders.created" messages even though the stream stores both. cons, err := js.CreateOrUpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{ Durable: "analytics", FilterSubject: "orders.shipped", DeliverPolicy: jetstream.DeliverAllPolicy, AckPolicy: jetstream.AckExplicitPolicy, }) if err != nil { panic(err) } fmt.Printf("Created consumer: %s\n", cons.CachedInfo().Name) fmt.Printf("Filter: %s\n", cons.CachedInfo().Config.FilterSubject) // Fetch up to 5 messages with a short expiry. Only "orders.shipped" // messages come back; the filter excludes everything else. msgs, err := cons.Fetch(5, jetstream.FetchMaxWait(2*time.Second)) if err != nil { panic(err) } for msg := range msgs.Messages() { fmt.Printf("Received on %s\n", msg.Subject()) msg.Ack() } if err := msgs.Error(); err != nil { panic(err) } ``` #### Python ``` # Create a durable pull consumer that only sees orders.shipped. # The filter is applied server-side: orders.created never reaches this consumer. await js.add_consumer( "ORDERS", ConsumerConfig( durable_name="analytics", ack_policy=AckPolicy.EXPLICIT, filter_subject="orders.shipped", ), ) print("Created filtered consumer analytics on stream ORDERS") # Fetch a small batch. Only orders.shipped messages come back. psub = await js.pull_subscribe_bind("analytics", stream="ORDERS") msgs = await psub.fetch(batch=5, timeout=2) for msg in msgs: print(f"got {msg.subject}") await msg.ack() ``` #### Java ``` // Create a durable pull consumer that only sees orders.shipped. // The filter subject narrows the stream's subjects down to one. ConsumerContext cc = sc.createOrUpdateConsumer( ConsumerConfiguration.builder() .durable("analytics") .filterSubject("orders.shipped") .ackPolicy(AckPolicy.Explicit) .build()); System.out.println("Created filtered consumer: " + cc.getConsumerName()); // Fetch a small batch. Only orders.shipped messages come back; // orders.created is filtered out before it reaches this consumer. try (FetchConsumer fc = cc.fetch( FetchConsumeOptions.builder().maxMessages(5).expiresIn(2000).build())) { Message m; while ((m = fc.nextMessage()) != null) { System.out.println("subject=" + m.getSubject()); m.ack(); } } ``` #### Rust ``` // Create a durable pull consumer that only sees orders.shipped messages. // The filter subject restricts delivery to one subject in the stream. let stream = js.get_stream("ORDERS").await?; let consumer = stream .create_consumer(pull::Config { durable_name: Some("analytics".to_string()), filter_subject: "orders.shipped".to_string(), ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; println!("Created filtered consumer: {}", consumer.cached_info().name); // Fetch a small batch with a short expiry. Only orders.shipped come back. let mut messages = consumer .fetch() .max_messages(5) .expires(std::time::Duration::from_secs(2)) .messages() .await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("got: {}", msg.subject); msg.ack().await?; } ``` #### C#/.NET ``` // Create a durable pull consumer that only sees orders.shipped var consumer = await js.CreateOrUpdateConsumerAsync("ORDERS", new ConsumerConfig("analytics") { AckPolicy = ConsumerConfigAckPolicy.Explicit, FilterSubject = "orders.shipped", }); output.WriteLine($"Created durable consumer {consumer.Info.Config.Name} filtered on orders.shipped"); // Fetch a small batch; only orders.shipped comes back await foreach (var msg in consumer.FetchAsync(opts: new NatsJSFetchOpts { MaxMsgs = 5, Expires = TimeSpan.FromSeconds(2) })) { output.WriteLine($"{msg.Subject}: {msg.Data}"); subjects.Add(msg.Subject); await msg.AckAsync(); } ``` #### C ``` // Create a durable pull consumer named "analytics" on the ORDERS // stream. FilterSubject narrows delivery to "orders.shipped", so // the consumer never sees "orders.created" messages even though // the stream stores both. jsConsumerConfig cc; jsSubOptions so; natsMsgList list = {NULL, 0}; int i; jsConsumerConfig_Init(&cc); cc.Durable = "analytics"; cc.FilterSubject = "orders.shipped"; cc.DeliverPolicy = js_DeliverAll; cc.AckPolicy = js_AckExplicit; s = js_AddConsumer(&ci, js, "ORDERS", &cc, NULL, &jerr); if (s == NATS_OK) { printf("Created consumer: %s\n", ci->Name); printf("Filter: %s\n", ci->Config->FilterSubject); } // Bind to the consumer and fetch up to 5 messages with a short // expiry. Only "orders.shipped" messages come back; the filter // excludes everything else. jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "analytics"; if (s == NATS_OK) s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, 5, 2000, &jerr); for (i = 0; (s == NATS_OK) && (i < list.Count); i++) { printf("Received on %s\n", natsMsg_GetSubject(list.Msgs[i])); natsMsg_Ack(list.Msgs[i], NULL); } natsMsgList_Destroy(&list); ``` The new flag is `--filter`. It ties the consumer to a single filter subject. A message on `orders.shipped` reaches `analytics`; a message on `orders.created` or `orders.canceled` does not. Ask the server to describe the consumer: ``` nats consumer info ORDERS analytics ``` The configuration block now carries a line the `billing` consumer didn't have: ``` Configuration: Name: analytics Pull Mode: true Filter Subject: orders.shipped Deliver Policy: All Ack Policy: Explicit Ack Wait: 30.00s Replay Policy: Instant Max Ack Pending: 1,000 Max Waiting Pulls: 512 ``` `Filter Subject: orders.shipped` is the line that matters. The `billing` consumer has no filter, so its info output omits this line. No filter line means every subject in the stream. ## Two consumers with separate positions The `analytics` consumer and the `billing` consumer read the same stream, but each tracks its own position in it. From the previous page, a consumer keeps a cursor: the sequence number of the last message it delivered and saw acknowledged. That cursor belongs to the consumer, not to the stream. Two consumers on one stream have two separate cursors. The server stores the cursor alongside the consumer's config and ack state, separate from the stream's messages. When `analytics` advances its cursor past sequence `3`, `billing`'s position does not change. Both consumers read the same stored messages from their own cursor. **Message flow — Two consumers, separate positions (animated):** Two consumers read one ORDERS stream from independent positions. The billing consumer has no filter and delivers every order; the analytics consumer filters to orders.shipped and delivers only the shipped messages, skipping orders.created. Each keeps its own cursor, so reading from one never moves the other: billing advances through all six messages and reaches #6, while analytics has delivered only the two shipped orders and sits at #5. The stream keeps one shared copy of every message and serves each consumer from its own position. `billing` reads every order and advances through all of them; `analytics` delivers only the `orders.shipped` messages and skips the rest, so the two cursors come to rest at different positions. Neither one moves the other. Pull from `analytics` and see what comes back: ``` nats consumer next ORDERS analytics --count 5 ``` `analytics` sees only the `orders.shipped` message stored on the publishing page, sequence `3`. The `orders.created` messages at sequences `1` and `2` don't appear for this consumer. They're still in the stream; the filter just hides them from `analytics`. `billing` stays wherever you left it. Reading from `analytics` did not move `billing`'s cursor, and it did not consume or delete any message from the stream. ## A consumer is a view A consumer is an independent **view** over the stored messages, with its own filter, cursor, and ack state. The stream holds the one shared copy of every message, and each consumer reads it from its own position. Because consumers are independent, a filter is a cheap way to send the same messages to more than one reader. Adding `analytics` cost one command. It did not copy any data, it did not slow down `billing`, and it can start, stop, or fall behind without affecting any other consumer. The server keeps one copy of each message and serves every consumer from it. This differs from the core NATS [queue group](/learn/core-nats/queue-groups.md) you met in core NATS. A queue group splits one subject's live traffic across workers that share the load. Here, each consumer gets its own full view of the stored stream, filtered to what it asked for. Sharing load within one consumer — the [worker-pool pattern](/learn/jetstream/worker-pool.md) — comes later in the chapter. ## Other filtering options The `analytics` consumer filters on a single subject. A consumer can also filter on several subjects at once. That goes beyond what this scenario needs. For the full set of consumer filtering options, including multiple filter subjects, see [Reference → Create Consumer](/reference/jetstream/api/consumer/create.md). We use only a single `Filter Subject` here. ## Pitfalls A filter is a small piece of config, but a wrong one fails quietly. **A filter that matches nothing.** The server accepts any filter subject, even one that matches no message in the stream. A typo like `orders.shiped` creates a valid consumer that never receives anything. There's no error and no warning, just an empty pull. Don't assume an empty pull means the stream is empty; first confirm the filter matches a subject the stream actually stores. #### CLI ``` #!/bin/bash # A filter that matches no subject in the stream is accepted without error. # Here the typo "orders.shiped" matches nothing in ORDERS (orders.>). nats consumer add ORDERS analytics-typo \ --filter "orders.shiped" \ --pull \ --ack explicit \ --defaults # The consumer exists and its config looks fine. nats consumer info ORDERS analytics-typo # But pulling delivers nothing — the request just times out. # No error tells you the filter was wrong; the consumer is simply silent. nats consumer next ORDERS analytics-typo --count 5 --timeout 2s # Confirm the filter never matched: Delivered shows 0 of the stored orders. nats consumer info ORDERS analytics-typo | grep -A1 "Delivery counts" ``` #### JavaScript/TypeScript ``` // The filter_subject here has a typo: "orders.shiped" matches no subject ORDERS // actually stores. The server still accepts the consumer — a filter that matches // nothing is valid, just empty. const jsm = await jetstreamManager(nc); await jsm.consumers.add("ORDERS", { durable_name: "analytics-typo", ack_policy: AckPolicy.Explicit, filter_subject: "orders.shiped", }); console.log("Created filtered consumer: analytics-typo (orders.shiped)"); // Try to pull. The fetch waits out its short expiry and returns nothing — no // error, no message. A wrong filter fails silently: the pull just times out // empty because no stored subject matches. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "analytics-typo"); const msgs = await c.fetch({ max_messages: 5, expires: 2000 }); let count = 0; for await (const m of msgs) { count++; await m.ack(); } if (count === 0) { console.log("pull returned nothing: filter matched no stored subject"); } ``` #### Go ``` // Create a durable pull consumer named "analytics-typo" on the ORDERS // stream. The filter "orders.shiped" has a typo (one "p"), so it matches // no subject the stream actually stores. cons, err := js.CreateOrUpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{ Durable: "analytics-typo", FilterSubject: "orders.shiped", DeliverPolicy: jetstream.DeliverAllPolicy, AckPolicy: jetstream.AckExplicitPolicy, }) if err != nil { panic(err) } // Fetch with a short expiry. The fetch succeeds with no error, but the // channel yields zero messages because the filter matched no stored subject. msgs, err := cons.Fetch(5, jetstream.FetchMaxWait(2*time.Second)) if err != nil { panic(err) } count := 0 for range msgs.Messages() { count++ } if err := msgs.Error(); err != nil { panic(err) } // A wrong filter fails silently: the pull returned nothing, and no error // was raised. The consumer is healthy, it just never matches a message. fmt.Printf("Received %d messages: the filter matched no stored subject, so the pull returned nothing (no error).\n", count) ``` #### Python ``` # Note the typo: "orders.shiped" matches no subject in the stream. # codespell:ignore shiped # JetStream accepts the consumer anyway. The wrong filter fails silently. await js.add_consumer( "ORDERS", ConsumerConfig( durable_name="analytics-typo", ack_policy=AckPolicy.EXPLICIT, filter_subject="orders.shiped", # codespell:ignore shiped ), ) psub = await js.pull_subscribe_bind("analytics-typo", stream="ORDERS") try: await psub.fetch(batch=5, timeout=2) except nats.errors.TimeoutError: # No error from the server. The pull simply returned nothing because the # filter matched no stored subject. A wrong filter looks like an empty stream. print("Fetch timed out: the filter matched no stored subject") ``` #### Java ``` // The filter subject has a typo: "orders.shiped" matches no stored // subject. The consumer is still created without error. ConsumerContext cc = sc.createOrUpdateConsumer( ConsumerConfiguration.builder() .durable("analytics-typo") .filterSubject("orders.shiped") .ackPolicy(AckPolicy.Explicit) .build()); // The fetch blocks until the expiry, then returns nothing. A wrong // filter fails silently: no messages, no error. try (FetchConsumer fc = cc.fetch( FetchConsumeOptions.builder().maxMessages(5).expiresIn(2000).build())) { Message m = fc.nextMessage(); if (m == null) { System.out.println("Pull returned nothing: filter matched no stored subject."); } } ``` #### Rust ``` // Create a durable pull consumer whose filter has a typo: "orders.shiped" // matches no stored subject, so the consumer is valid but empty. let stream = js.get_stream("ORDERS").await?; let consumer = stream .create_consumer(pull::Config { durable_name: Some("analytics-typo".to_string()), filter_subject: "orders.shiped".to_string(), ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; // The fetch waits up to the expiry, then returns with no messages and no // error. A wrong filter fails silently: nothing matches, nothing arrives. let mut messages = consumer .fetch() .max_messages(5) .expires(std::time::Duration::from_secs(2)) .messages() .await?; let mut count = 0; while let Some(msg) = messages.next().await { msg?; count += 1; } if count == 0 { println!("Pull returned nothing: the filter matched no stored subject."); } ``` #### C#/.NET ``` // The filter has a typo: "orders.shiped" matches no subject the stream stores var consumer = await js.CreateOrUpdateConsumerAsync("ORDERS", new ConsumerConfig("analytics-typo") { AckPolicy = ConsumerConfigAckPolicy.Explicit, FilterSubject = "orders.shiped", }); // The fetch waits out its expiry and returns nothing, with no error await foreach (var msg in consumer.FetchAsync(opts: new NatsJSFetchOpts { MaxMsgs = 5, Expires = TimeSpan.FromSeconds(2) })) { received++; await msg.AckAsync(); } output.WriteLine($"Pull returned {received} messages: the filter matched no stored subject, so a wrong filter fails silently"); ``` #### C ``` // Create a durable pull consumer named "analytics-typo" on the ORDERS // stream. The filter "orders.shiped" has a typo (one "p"), so it matches // no subject the stream actually stores. if (s == NATS_OK) { jsSubOptions_Init(&so); so.Stream = "ORDERS"; s = js_PullSubscribe(&sub, js, "orders.shiped", "analytics-typo", NULL, &so, &jerr); } // Fetch with a short wait. The fetch times out with zero messages // because the filter matched no stored subject; that is not an error. if (s == NATS_OK) { s = natsSubscription_Fetch(&list, sub, 5, 2000, &jerr); if (s == NATS_TIMEOUT) s = NATS_OK; } // A wrong filter fails silently: the pull returned nothing, and no error // was raised. The consumer is healthy, it just never matches a message. if (s == NATS_OK) printf("Received %d messages: the filter matched no stored subject, so the pull returned nothing (no error).\n", list.Count); ``` When a pull comes back empty, run `nats consumer info` and check the `Filter Subject` line against the stream's subjects. A filter outside `orders.>` can never match. **Expecting a filter to delete from the stream.** A filter narrows one consumer's view; it never removes messages. After `analytics` reads `orders.shipped`, every `orders.created` and `orders.canceled` message is still stored and still readable by `billing`. Don't use a filter to prune a stream. What stays and what ages out is controlled by the stream's limits, covered in [Shaping the stream](/learn/jetstream/shaping-the-stream.md), not by any consumer. **Overlapping filters within one consumer.** Overlap *between* consumers is fine on limits and interest streams: two separate consumers whose filters match the same subject each get their own full copy of those messages. That's the kind of sharing this page relies on. The exception is work-queue retention, where consumers' filters must not overlap each other. See [Retention policies](/learn/jetstream/retention-policies.md). Overlap *inside* one consumer is different. You can give a single consumer several filter subjects, but if one of those subjects already covers another, like `orders.>` next to `orders.shipped`, the create call fails with `consumer subject filters cannot overlap`. Filters that only partly overlap, where neither covers the other, are accepted. ## Where you are The `ORDERS` stream now has two consumers reading it: * `billing`: no filter, reads every order; the reader from the previous page * `analytics`: filtered to `orders.shipped`, sees only ships Both read the same stored messages. Neither consumer's progress affects the other, and the stream itself is untouched by either read. ## What's next Both `billing` and `analytics` read on the happy path: pull a message, ack it, move on. The next page is about what that acknowledgment actually does — how a message is held in flight until it's confirmed, what a double ack adds, and how an unacked message is redelivered. ## See also * [Reference → Create Consumer](/reference/jetstream/api/consumer/create.md) — every consumer config field, including multiple filter subjects. * [Reading back the stream](/learn/jetstream/reading-back.md) — where you met the consumer cursor this page builds on. --- # Reading messages directly [Reading back the stream](/learn/jetstream/reading-back.md) built a consumer to walk the whole log in order. That's the right tool when a reader works through every message and keeps its place between runs. Some reads need less than that: one message by its sequence, or the latest order on a subject, with no consumer to create and no position to track. JetStream has a lighter read for those. You ask the stream for a message and the server returns it straight from the store. ## Get one message Every stored message has a sequence number, the one the `PubAck` returned when you published it. Give the server that number and it hands the message back: #### CLI ``` #!/bin/bash # Get one stored message by its sequence number — the number the PubAck # returned when the message was published. No consumer, no ack, no cursor: # the server reads the message straight from the stream's store and returns it. nats stream get ORDERS 2 ``` #### JavaScript/TypeScript ``` // Read the message at stream sequence 2 through the regular get API, which is // served by the stream leader. const m = await jsm.streams.getMessage("ORDERS", { seq: 2 }); console.log(`subject: ${m?.subject}`); console.log(`payload: ${m?.string()}`); ``` #### Go ``` // Read the message stored at stream sequence 2. This is the regular get, // served by the stream leader. msg, err := stream.GetMsg(ctx, 2) if err != nil { panic(err) } fmt.Printf("Subject: %s\n", msg.Subject) fmt.Printf("Payload: %s\n", string(msg.Data)) ``` #### Python ``` # Get one stored message by its sequence number — the number the PubAck # returned when it was published. This regular get is served by the # stream's leader, so it always sees the latest write. msg = await js.get_msg("ORDERS", seq=2) print(f"seq {msg.seq} on {msg.subject}: {msg.data.decode()}") ``` #### Java ``` // Read the message stored at stream sequence 2. Without Direct Get // enabled this is a regular get, served by the stream leader. MessageInfo mi = sc.getMessage(2); System.out.println("Subject: " + mi.getSubject()); System.out.println("Payload: " + new String(mi.getData(), StandardCharsets.UTF_8)); ``` #### Rust ``` // Read the message stored at stream sequence 2. This raw-message get always // goes to the stream leader. let message = stream.get_raw_message(2).await?; println!("Subject: {}", message.subject); println!("Payload: {}", String::from_utf8_lossy(&message.payload)); ``` #### C#/.NET ``` // Fetch one specific message by its stream sequence. This is a regular // get, served by the stream leader. var response = await stream.GetAsync(new StreamMsgGetRequest { Seq = 2 }); var message = response.Message; var payload = Encoding.UTF8.GetString(message.Data.Span); output.WriteLine($"Subject: {message.Subject}"); output.WriteLine($"Payload: {payload}"); ``` #### C ``` // Read the message stored at stream sequence 2. This is the regular get, // served by the stream leader. if (s == NATS_OK) s = js_GetMsg(&msg, js, "ORDERS", 2, NULL, &jerr); if (s == NATS_OK) { printf("Subject: %s\n", natsMsg_GetSubject(msg)); printf("Payload: %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); } ``` ``` Item: ORDERS#2 received 2026-05-22 10:14:25 +0000 UTC (36h2m11s) on Subject orders.created {"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"} ``` More often you don't have the sequence; you have the subject and want its most recent message. `--last-for` returns the last message stored on a subject: #### CLI ``` #!/bin/bash # Get the most recent message stored on a subject, when you know the subject # but not the sequence. This is the read a key-value lookup is built on: # "the latest value for a key" is "the last message on its subject". nats stream get ORDERS --last-for orders.shipped ``` #### JavaScript/TypeScript ``` // Read the last message stored on subject orders.shipped through the regular // get API, which is served by the stream leader. const m = await jsm.streams.getMessage("ORDERS", { last_by_subj: "orders.shipped", }); console.log(`subject: ${m?.subject}`); console.log(`payload: ${m?.string()}`); ``` #### Go ``` // Read the last message stored on subject orders.shipped. This is the // regular get, served by the stream leader. msg, err := stream.GetLastMsgForSubject(ctx, "orders.shipped") if err != nil { panic(err) } fmt.Printf("Subject: %s\n", msg.Subject) fmt.Printf("Payload: %s\n", string(msg.Data)) ``` #### Python ``` # Get the most recent message stored on a subject, when you know the # subject but not the sequence. This is the read a key-value lookup uses. msg = await js.get_last_msg("ORDERS", "orders.shipped") print(f"seq {msg.seq} on {msg.subject}: {msg.data.decode()}") ``` #### Java ``` // Read the most recent message on a subject. Without Direct Get // enabled this is a regular get, served by the stream leader. MessageInfo mi = sc.getLastMessage("orders.shipped"); System.out.println("Subject: " + mi.getSubject()); System.out.println("Payload: " + new String(mi.getData(), StandardCharsets.UTF_8)); ``` #### Rust ``` // Read the last message stored on subject `orders.shipped`. This raw-message // get always goes to the stream leader. let message = stream .get_last_raw_message_by_subject("orders.shipped") .await?; println!("Subject: {}", message.subject); println!("Payload: {}", String::from_utf8_lossy(&message.payload)); ``` #### C#/.NET ``` // Fetch the most recent message on a subject. This is a regular get, // served by the stream leader. var response = await stream.GetAsync(new StreamMsgGetRequest { LastBySubj = "orders.shipped" }); var message = response.Message; var payload = Encoding.UTF8.GetString(message.Data.Span); output.WriteLine($"Subject: {message.Subject}"); output.WriteLine($"Payload: {payload}"); ``` #### C ``` // Read the last message stored on subject orders.shipped. This is the // regular get, served by the stream leader. if (s == NATS_OK) s = js_GetLastMsg(&msg, js, "ORDERS", "orders.shipped", NULL, &jerr); if (s == NATS_OK) { printf("Subject: %s\n", natsMsg_GetSubject(msg)); printf("Payload: %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); } ``` ``` Item: ORDERS#3 received 2026-05-22 10:14:31 +0000 UTC (36h2m05s) on Subject orders.shipped {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:31Z"} ``` This is the read a key-value lookup is built on. A KV bucket is a stream keyed by subject, and "get the value for this key" is "get the last message on this subject." The [Key-Value deep dive](/learn/key-value/.md) uses exactly this read underneath. Both forms ask the stream's leader, the one server that holds the latest write, so a `nats stream get` right after a publish always sees it. ## Direct Get reads from any replica `nats stream get` goes to the leader. That's fine for the occasional lookup. For a read you run often, or from far away, JetStream offers **Direct Get**: the same by-sequence and by-subject reads, answered by *any* server that holds a copy of the stream, not only the leader. **Message flow — Direct Get reads from any replica (animated):** Direct Get reads one message by sequence from any copy of a replicated stream, not only the leader. ORDERS is replicated across three servers — a leader and two replicas, each holding a full copy. A reader fires a stream of Direct Gets, each for a different sequence number; every read is answered by whichever copy serves it — sometimes the leader, sometimes a replica — and a per-server tally shows the read load spreading across all three. Direct Get is the stream's `allow_direct` setting. Check it with `nats stream info ORDERS`: its output carries a `Direct Get: true` line when the setting is on, and no such line when it's off. The CLI enables it for new streams, so `ORDERS` already has it — turn it on for a stream that doesn't: #### CLI ``` #!/bin/bash # Turn on Direct Get for a stream that doesn't have it. The CLI enables it # for new streams, so ORDERS already shows "Direct Get: true" in stream info; # run this only if a stream shows "Direct Get: false". nats stream edit ORDERS --allow-direct ``` #### JavaScript/TypeScript ``` // Turn on direct access by setting allow_direct on the stream config and // applying the update. const config = info.config; config.allow_direct = true; const updated = await jsm.streams.update("ORDERS", config); console.log(`allow_direct: ${updated.config.allow_direct}`); ``` #### Go ``` // Read the current config, turn on direct access, and push the update. // AllowDirect lets any replica serve single-message gets. info, err := stream.Info(ctx) if err != nil { panic(err) } cfg := info.Config cfg.AllowDirect = true updated, err := js.UpdateStream(ctx, cfg) if err != nil { panic(err) } fmt.Printf("AllowDirect: %v\n", updated.CachedInfo().Config.AllowDirect) ``` #### Python ``` # Turn on Direct Get for the stream. With allow_direct enabled, get-message # requests can be served by any replica instead of only the stream leader. config.allow_direct = True updated = await js.update_stream(config) print(f"allow_direct is now {updated.config.allow_direct}") ``` #### Java ``` // Turn on Direct Get for ORDERS. Reads can then be served by any // replica or mirror instead of only the stream leader. StreamConfiguration updated = StreamConfiguration.builder(current) .allowDirect(true) .build(); StreamInfo info = jsm.updateStream(updated); System.out.println("AllowDirect: " + info.getConfiguration().getAllowDirect()); ``` #### Rust ``` // Turn on direct get for the stream, then push the updated config. config.allow_direct = true; let info = js.update_stream(&config).await?; println!("allow_direct is now: {}", info.config.allow_direct); ``` #### C#/.NET ``` // Turn on direct get by updating the stream config. Read the current // config, set AllowDirect, and send the update back. var config = stream.Info.Config; config.AllowDirect = true; var updated = await js.UpdateStreamAsync(config); output.WriteLine($"AllowDirect: {updated.Info.Config.AllowDirect}"); ``` #### C ``` // Read the current config, turn on direct access, and push the update. // AllowDirect lets any replica serve single-message gets. if (s == NATS_OK) s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { si->Config->AllowDirect = true; s = js_UpdateStream(&updated, js, si->Config, NULL, &jerr); } if (s == NATS_OK) printf("AllowDirect: %s\n", updated->Config->AllowDirect ? "true" : "false"); ``` With it on, read directly with the Direct Get API. This fetches the message at sequence 1 from any replica that holds it, not just the leader: #### CLI ``` #!/bin/bash # Read directly from the stream's store with the Direct Get API. --direct # routes the read to any server holding a copy of the stream, not just the # leader. This fetches the message at sequence 1 from whichever replica # answers, marked "(direct)" to show it came over the Direct Get API. nats sub --stream ORDERS --direct --start-sequence 1 --count 1 ``` #### JavaScript/TypeScript ``` // Read a single message at sequence 1 through the Direct Get API. This needs // allow_direct on the stream and can be served by any replica, not just the // leader. const m = await jsm.direct.getMessage("ORDERS", { seq: 1 }); console.log(`subject: ${m?.subject}`); console.log(`payload: ${m?.string()}`); ``` #### Go ``` // Read the message at sequence 1. Because the stream has AllowDirect // enabled, GetMsg is served by the Direct Get API, so any replica can // answer it, not only the leader. msg, err := stream.GetMsg(ctx, 1) if err != nil { panic(err) } fmt.Printf("Subject: %s\n", msg.Subject) fmt.Printf("Payload: %s\n", string(msg.Data)) ``` #### Python ``` # Read directly from the stream's store with the Direct Get API. With # direct=True the read is served by any server holding a copy of the # stream, not just the leader (the stream must have allow_direct set). msg = await js.get_msg("ORDERS", seq=1, direct=True) print(f"seq {msg.seq} on {msg.subject} (direct): {msg.data.decode()}") ``` #### Java ``` // Read the message at stream sequence 1. Because ORDERS has // AllowDirect enabled, the client sends this over the Direct Get // API, so any replica or mirror can answer instead of the leader. MessageInfo mi = sc.getMessage(1); System.out.println("Subject: " + mi.getSubject()); System.out.println("Payload: " + new String(mi.getData(), StandardCharsets.UTF_8)); ``` #### Rust ``` // Read the message at stream sequence 1 with the Direct Get API. This can be // served by any replica, not just the leader. let message = stream.direct_get(1).await?; println!("Subject: {}", message.subject); println!("Payload: {}", String::from_utf8_lossy(&message.payload)); ``` #### C#/.NET ``` // Read sequence 1 with a direct get. Any replica can serve this, not // just the stream leader. The original subject comes back as a header. var msg = await stream.GetDirectAsync(new StreamMsgGetRequest { Seq = 1 }); msg.Headers!.TryGetLastValue("Nats-Subject", out var subject); output.WriteLine($"Subject: {subject}"); output.WriteLine($"Payload: {msg.Data}"); ``` #### C ``` // Read the message at sequence 1. Because the stream has AllowDirect // enabled, the Direct Get API serves the read, so any replica can // answer it, not only the leader. if (s == NATS_OK) { jsDirectGetMsgOptions_Init(&dgo); dgo.Sequence = 1; s = js_DirectGetMsg(&msg, js, "ORDERS", NULL, &dgo); } if (s == NATS_OK) { printf("Subject: %s\n", natsMsg_GetSubject(msg)); printf("Payload: %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); } ``` ``` Subscribing to JetStream Stream (direct) holding messages with subject orders.> starting with sequence 1 [#1] Received JetStream message (direct): stream: ORDERS seq 1 / subject: orders.created / time: 2026-05-22 10:14:22 ``` The `(direct)` marks the path: the message came from a server's local store over the Direct Get API, not from the leader through the regular read. On a replicated stream that spreads read load across the servers, and when a mirror stream sets `mirror_direct`, the mirror's servers also answer Direct Get requests for the origin, so a reader can be served by a nearby [mirror](/learn/jetstream/mirrors-and-sources.md) instead of the origin. The trade-off is freshness. A replica or mirror can sit a moment behind the leader, so Direct Get is **not read-after-write coherent**: a message you just published might not show up on a direct read for a beat, where `nats stream get` against the leader always sees it. For a latest-value lookup that tolerates being a little stale — a cache, a dashboard — that's a fair trade for spreading the load. When you must see your own most recent write, read the leader with `nats stream get`. ## Get a batch in one request A single request can return more than one message. Ask Direct Get for a range and the server streams the matching messages back over one request, instead of a round trip each. This reads three messages starting at sequence 1: #### CLI ``` #!/bin/bash # Direct Get can return a batch of messages over a single request. Ask for three # starting at sequence 1; the server streams them back without a round trip # each, every message carrying a Nats-Num-Pending header that counts down to 0 # on the last one. nats sub --stream ORDERS --direct --start-sequence 1 --count 3 ``` #### JavaScript/TypeScript ``` // Fetch up to 3 messages starting at stream sequence 1 in a single Direct Get // request. This needs allow_direct on the stream and can be served by any // replica, not just the leader. getBatch returns an iterator of stored // messages in stream order. const iter = await jsm.direct.getBatch("ORDERS", { seq: 1, batch: 3 }); for await (const m of iter) { console.log(`seq ${m.seq} on ${m.subject}`); } ``` #### Go ``` // Fetch up to 3 messages from ORDERS in a single Direct Get request, // starting at stream sequence 1, and iterate them in order. msgs, err := jetstreamext.GetBatch(ctx, js, "ORDERS", 3, jetstreamext.GetBatchSeq(1)) if err != nil { log.Fatalf("get batch: %v", err) } for msg, err := range msgs { if err != nil { log.Fatalf("read message: %v", err) } fmt.Printf("seq %d on %s\n", msg.Sequence, msg.Subject) } ``` #### Java ``` // Batch Direct Get: in one request, read up to 3 messages from the // ORDERS stream starting at stream sequence 1, then iterate in order. DirectBatchContext direct = new DirectBatchContext(nc, STREAM); MessageBatchGetRequest request = MessageBatchGetRequest.batch(">", 3, 1); List messages = direct.fetchMessageBatch(request); for (MessageInfo mi : messages) { System.out.println("sequence " + mi.getSeq() + " | subject " + mi.getSubject()); } ``` #### Rust ``` // Fetch up to 3 messages starting at stream sequence 1, all in one request. let mut messages = js.get_batch("ORDERS", 3).sequence(1).send().await?; // The batch arrives as a stream of messages in sequence order. while let Some(msg) = messages.next().await { let msg = msg?; println!("seq {} on subject {}", msg.sequence, msg.subject); } ``` #### C#/.NET ``` // Fetch up to three messages starting at stream sequence 1, in one request. var request = new StreamMsgBatchGetRequest { Seq = 1, Batch = 3, }; // The server streams the messages back in sequence order. await foreach (NatsMsg msg in js.GetBatchDirectAsync("ORDERS", request)) { // On a direct get the stream sequence and original subject come back in // headers, not in msg.Subject (which is the reply subject). msg.Headers!.TryGetValue("Nats-Sequence", out var sequence); msg.Headers!.TryGetValue("Nats-Subject", out var subject); Console.WriteLine($"seq {sequence}: {subject}"); } ``` Each message carries a `Nats-Num-Pending` header counting how many still match after it, so the client knows when the batch is complete — the count reaches `0` on the last message: ``` [#1] ... seq 1 ... Nats-Num-Pending: 2 [#2] ... seq 2 ... Nats-Num-Pending: 1 [#3] ... seq 3 ... Nats-Num-Pending: 0 ``` **Message flow — Batch Direct Get over one request (animated):** Batch Direct Get returns many messages over a single request, and any copy of a replicated stream can serve it. ORDERS is replicated across three servers — a leader and two replicas. Three batches run in turn, each served by a different copy, so batch reads spread across all three. Within a batch the serving copy streams the messages back one after another, each carrying a Nats-Num-Pending header that counts down — 2, then 1, then 0 on the last message — so the reader knows the batch is complete. That makes Direct Get a cheap way to pull a slice of the log without standing up a consumer: a range from a sequence, the latest message on each of several subjects (`--last-per-subject`), or a point-in-time snapshot across subjects. A batch is bounded by a count or a byte budget; the request fields (`batch`, `max_bytes`, `multi_last`) are in the [reference](/reference/jetstream/api/stream/msg-get.md). `nats.js` sends a batched Direct Get directly; Go, Rust, Java, and C# reach it through the [Synadia Orbit](https://github.com/synadia-io) helper libraries. A batch read is still a one-shot snapshot, not a subscription. It returns what's stored when you ask and stops. To keep receiving new orders as they arrive, that's a consumer's job. ## Direct Get or a consumer? Both read a stream; they answer different needs. * Reach for **Direct Get** (or `nats stream get`) for a point read: one message by sequence, the latest value on a subject, or a bounded batch snapshot. Nothing is acked, no position is kept, and any replica can answer. * Reach for a **[consumer](/learn/jetstream/reading-back.md)** to work through the log: read every message in order, keep a durable position across restarts, ack as you go, and pick up new messages as they're published. A direct read never moves a consumer's position and never removes a message. It only reads what's stored. ## Pitfalls A direct read is simple, but a few assumptions trip people up. **Direct Get is off, so the read hangs.** If `allow_direct` isn't set, no server answers the Direct Get subject and the request times out with nothing returned. Check `nats stream info` for `Direct Get: true`, or set it with `nats stream edit ORDERS --allow-direct`, before you point a direct reader at a stream. **A direct read can be stale.** Direct Get answers from any replica or mirror, which may trail the leader. Don't use it for read-after-write checks — confirming a publish landed, reading a value you wrote a moment ago. Use `nats stream get` against the leader for those, and keep Direct Get for high-volume or far-away reads that tolerate a little lag. **A batch isn't a subscription.** A direct batch returns the stored messages that match and stops. Code that expects to keep receiving new messages from it reads the backlog once and then sits idle. Use a consumer when you need to follow the stream as it grows. ## Where you are `ORDERS` is unchanged. You now have a second way to read it, alongside the consumer from [reading back](/learn/jetstream/reading-back.md): * `nats stream get` for one message, by sequence or by the last on a subject, answered by the leader and always current. * **Direct Get** (`--direct`) for the same reads answered by any replica or mirror, and for pulling a bounded batch in one request — at the cost of read-after-write freshness. * The split between a point read (Direct Get) and working through the log (a consumer). ## What's next You've read messages by sequence and by subject. The next page changes the subjects themselves: [subject mapping](/learn/jetstream/subject-mapping.md) rewrites a message's subject on the way into a stream and on the way back out. ## See also * [ADR-31: JetStream Direct Get](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-31.md) — the authoritative spec for `allow_direct`, the by-sequence, by-subject, batch, and multi-subject request forms, and the response headers. * [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md) — the `allow_direct` and `mirror_direct` fields alongside every other stream setting. * [Reading back the stream](/learn/jetstream/reading-back.md) — the durable consumer that walks the whole log and keeps its place. * [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) — mirrors that can answer Direct Get reads near a client. --- # Per-message TTL The [Shaping the stream](/learn/jetstream/shaping-the-stream.md) page capped `ORDERS` at a 7-day `MaxAge`, so every message in the stream lives the same seven days, then is removed. That's one lifespan for the whole stream. Sometimes one message needs a different lifespan. ## When one message should expire sooner Consider an `orders.canceled` message that only matters for an hour. A service that reads from the stream has 60 minutes to react to a cancellation. After that the message is no longer useful, and you don't want it in the stream for the full seven days. `MaxAge` applies one deadline to every message in the stream, so it can't expire this one ahead of the rest. A **per-message TTL** handles this case. TTL is short for time-to-live. It's a lifespan attached to one message, telling the server to delete that message after a stated time even when the stream's `MaxAge` would keep it longer. You set it with a header named `Nats-TTL` when you publish. The value is a length of time: `1h`, `5s`, `30m`. The server deletes the message that long after it was stored in the stream. ## The stream must opt in Per-message TTL is off by default. A stream rejects a `Nats-TTL` header until you turn the feature on. The switch is a stream setting named `AllowMsgTTL`. Turn it on for `ORDERS` now: ``` nats stream edit ORDERS --allow-msg-ttl ``` Confirm it landed: ``` nats stream info ORDERS ``` The `Allows Per-Message TTL` line in the settings block flips from `false` to `true`: ``` Allows Per-Message TTL: true ``` This switch comes with two catches. It can't be reversed. You can turn `AllowMsgTTL` on for an existing stream, but you can't turn it off again. The server refuses to take it back off. It needs server 2.11 or newer. On older servers the setting doesn't exist and the edit has no effect. The rest of this page assumes 2.11+. ## Publish a short-lived message With the feature on, publish an `orders.canceled` message that expires in 60 seconds. The TTL travels along as the `Nats-TTL` header: #### CLI ``` #!/bin/bash # Publish an orders.canceled message that expires 60 seconds after it # is stored. The per-message TTL rides along as the Nats-TTL header. # The stream must already have AllowMsgTTL enabled (nats stream edit # ORDERS --allow-msg-ttl) or this publish is rejected. # -J makes it a JetStream publish, so the server returns a PubAck # (Stored in Stream ... Sequence ...) instead of a fire-and-forget send. nats pub orders.canceled -J \ --header "Nats-TTL:60s" \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` #### JavaScript/TypeScript ``` // Publish one order with a per-message TTL. The `ttl` option sets the // `Nats-TTL` header ("60s"), so the server deletes this message 60 seconds // after it is stored, even if the stream would otherwise keep it forever. const pa = await js.publish("orders.cancelled", "order ord_8w2k cancelled", { ttl: "60s", }); console.log(`Stored in ${pa.stream} at sequence ${pa.seq}`); console.log("This message is deleted 60s after it was stored"); ``` #### Go ``` // Publish with a 60-second TTL. WithMsgTTL sets the Nats-TTL header, so the // server deletes this single message 60s after it is stored, ahead of the // stream's MaxAge. ack, err := js.Publish(ctx, "orders.canceled", []byte("order 42 canceled"), jetstream.WithMsgTTL(60*time.Second)) if err != nil { panic(err) } fmt.Printf("Stored in %s at sequence %d\n", ack.Stream, ack.Sequence) ``` #### Python ``` # Publish with a Nats-TTL header so the server deletes this message 60 # seconds after it's stored, no matter what the stream would otherwise keep. ack = await js.publish( "orders.canceled", b'{"id": "A-1001", "reason": "customer"}', headers={"Nats-TTL": "60s"}, ) print(f"stored in {ack.stream} at sequence {ack.seq}") ``` #### Java ``` // Attach a per-message TTL with the Nats-TTL header. The server keeps // this message for 60 seconds after it is stored, then deletes it. Headers headers = new Headers(); headers.add("Nats-TTL", "60s"); PublishAck ack = js.publish("orders.canceled", headers, "order 4242 canceled".getBytes(StandardCharsets.UTF_8)); System.out.println("Stored in stream: " + ack.getStream()); System.out.println("At sequence: " + ack.getSeqno()); ``` #### Rust ``` // Publish a cancellation that carries its own time-to-live. The `Nats-TTL` // header tells the server to delete this one message 60 seconds after it is // stored, even if the rest of the stream never expires. "60s" is a duration // string; the server also accepts an integer number of seconds (minimum 1s). // (A typed shortcut also exists: `PublishMessage::build().ttl(Duration)`.) let mut headers = async_nats::HeaderMap::new(); headers.insert("Nats-TTL", "60s"); let ack = js .publish_with_headers("orders.canceled", headers, "order 4242 canceled".into()) .await? .await?; println!( "Stored on stream {} at sequence {}", ack.stream, ack.sequence ); // The message is live now and is removed 60 seconds after it was stored. ``` #### C#/.NET ``` // Give one message its own lifetime with the `Nats-TTL` header. The // server deletes this message 60 seconds after it's stored, while the // rest of the stream keeps its normal retention. var headers = new NatsHeaders { ["Nats-TTL"] = "60s" }; var ack = await js.PublishAsync( subject: "orders.canceled", data: new OrderCancellation(OrderId: "ord_8w2k", Reason: "customer_request"), headers: headers); output.WriteLine($"Stored in {ack.Stream} at sequence {ack.Seq}"); ``` #### C ``` // Publish with a 60-second TTL. MsgTTL sets the Nats-TTL header, so the // server deletes this single message 60s after it is stored, ahead of // the stream's MaxAge. if (s == NATS_OK) { jsPubOptions_Init(&po); po.MsgTTL = 60000; // in milliseconds s = js_Publish(&ack, js, "orders.cancelled", payload, (int) strlen(payload), &po, &jerr); } if (s == NATS_OK) printf("Stored in %s at sequence %" PRIu64 "\n", ack->Stream, ack->Sequence); ``` The publish returns a normal `PubAck`. The message is stored in the stream like any other, with a sequence number. The one difference is that the server now holds a deletion time for it: stored time plus 60 seconds. Right after publishing, `nats stream info ORDERS` counts the message. Wait past the minute, ask again, and the count drops back. The message expired on its own deadline, while every other message in `ORDERS` keeps its full 7-day life. ## How TTL and MaxAge interact A message lives until the *first* deadline that arrives. The per-message TTL and the stream `MaxAge` are both deadlines, and the earlier one applies. **Message flow — Per-message TTL versus MaxAge (animated):** Three stored messages with different lifespans on one timeline, with a 'now' marker sweeping left to right. orders.cancelled carries a 1-hour per-message TTL and expires first, well before the stream's 7-day MaxAge. orders.created has no TTL, so it lives until MaxAge. orders.schema carries Nats-TTL: never and outlives even MaxAge. The earlier deadline always wins — except never, which has no deadline at all. For the cancellation above, the 60-second TTL arrives long before the 7-day `MaxAge`. The TTL applies, and the message expires early. A per-message TTL only ever makes a message expire *sooner* than `MaxAge` would, never later, with one exception covered next. ## How to make a message permanent Sometimes a single message has to outlive everything around it: a schema definition the rest of the stream depends on, or a baseline snapshot you replay new consumers against. You want that one message held past the stream's `MaxAge`, not just held a little longer. The exact value `never` does this. A message published with `Nats-TTL: never` never expires, not even at the stream's `MaxAge`. It stays until something deletes it on purpose. ## Storage versus delivery A per-message TTL decides how long a message stays *stored in the stream*. It says nothing about whether a consumer has read the message. This is the same split between storage and delivery from the publishing page, seen from the other end. A short TTL puts a deadline on the stored copy. If no consumer reads and acks the message before that deadline, the message expires unread. The server deletes it on schedule either way. If a consumer needs to learn that a message expired rather than just find it gone, the stream's `SubjectDeleteMarkerTTL` setting can leave a delete marker in its place, but only when the expired message was the last one on its subject. On an active subject with newer messages, no marker is placed. The setting also has its own conditions; the [reference](/reference/jetstream/api/stream/create.md) covers them. Size the TTL to the work. A 60-second TTL on a cancellation only makes sense if the consumer that cares about cancellations reads within that minute. Set the TTL shorter than the time in which the message still matters, but long enough for a healthy consumer to keep up. ## Two ways a TTL publish can fail The TTL has to be at least one second. The server rejects a sub-second or zero `Nats-TTL` with an *invalid-TTL* error (`err_code` 10165) and stores nothing. The message never lands. A valid `Nats-TTL` on a stream that hasn't opted in fails differently. The server rejects it with a *TTL-disabled* error (`err_code` 10166). The result is the same: no message stored. But the cause differs, so the two error codes are worth telling apart when you read a failed `PubAck`. In both cases the server never quietly drops the header. On a stream like `ORDERS`, which has no `SubjectDeleteMarkerTTL` set, a TTL publish either honors the TTL or returns an error. ## Pitfalls A few ways per-message TTL goes wrong in practice. **A TTL header on a stream that never opted in fails the publish. It does not store the message without a TTL instead.** The server rejects the `Nats-TTL` header with a `per-message TTL is disabled` error (`err_code` 10166), and nothing is stored. The danger is assuming the message landed with its TTL when it never landed at all. Do check the `Allows Per-Message TTL` line in `nats stream info ORDERS` before you rely on the header. Don't publish a TTL message and walk away on a stream you haven't confirmed opted in. #### CLI ``` #!/bin/bash # A Nats-TTL header on a stream that hasn't opted in is rejected. The # JetStream publish fails loudly (err_code 10166) and stores nothing — # the message is not quietly kept forever without a TTL. # A stream that never enabled AllowMsgTTL. Its config has no # "Allows Per-Message TTL" line (the line only appears once it's on). nats stream add ORDERS_NO_TTL --subjects "no-ttl.>" --defaults # A JetStream publish (-J) with a TTL header is rejected: nats pub exits # non-zero and prints the error, and nothing is stored. A plain core # publish would hide this — it returns no PubAck, so you'd never see the # rejection. nats pub no-ttl.msg -J \ --header "Nats-TTL:60s" \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # -> nats: error: per-message TTL is disabled (10166) # The fix is to opt the stream in once (a one-way switch), then republish. nats stream edit ORDERS_NO_TTL --allow-msg-ttl ``` #### JavaScript/TypeScript ``` // Publish with a TTL to a stream that has per-message TTLs disabled. The server // rejects the message with err 10166 ("per-message TTL is disabled") and stores // nothing. Enabling `allow_msg_ttl` on the stream is the fix. try { await js.publish("no-ttl.msg", "order ord_8w2k cancelled", { ttl: "60s" }); } catch (err) { if (err instanceof JetStreamApiError) { console.log(`Rejected (err ${err.code}): ${err.message}`); console.log("Nothing was stored"); } else { throw err; } } ``` #### Go ``` // Publishing with a TTL to a stream that hasn't enabled AllowMsgTTL fails. // The server returns "per-message TTL is disabled" (err_code 10166) and // stores nothing. The fix is to opt the stream in with AllowMsgTTL: true. _, err = js.Publish(ctx, "no-ttl.msg", []byte("order 42 canceled"), jetstream.WithMsgTTL(60*time.Second)) if err != nil { fmt.Printf("Publish rejected, message not stored: %s\n", err) return } ``` #### Python ``` # Publishing with a Nats-TTL header fails when the stream doesn't allow # per-message TTLs. The server rejects it with err_code 10166 and stores # nothing. try: await js.publish( "no-ttl.msg", b'{"id": "A-1002"}', headers={"Nats-TTL": "60s"}, ) except APIError as e: # "per-message TTL is disabled" — the fix is to create or update the # stream with allow_msg_ttl=True. print(f"publish rejected: {e.description} (err_code {e.err_code})") ``` #### Java ``` // Publishing a Nats-TTL header to a stream without AllowMsgTTL is // rejected by the server, and nothing is stored. Headers headers = new Headers(); headers.add("Nats-TTL", "60s"); try { js.publish("no-ttl.canceled", headers, "order 4242 canceled".getBytes(StandardCharsets.UTF_8)); } catch (JetStreamApiException e) { // Error 10166: "per-message TTL is disabled". Enabling // AllowMsgTTL on the stream is the fix. System.out.println("Publish rejected [" + e.getApiErrorCode() + "]: " + e.getErrorDescription()); } ``` #### Rust ``` // Try to publish a message carrying a `Nats-TTL` header to a stream that does // not allow per-message TTLs. The server refuses the write with error 10166, // "per-message TTL is disabled", so nothing is ever stored. let mut headers = async_nats::HeaderMap::new(); headers.insert("Nats-TTL", "60s"); match js .publish_with_headers("no-ttl.event", headers, "should be rejected".into()) .await? .await { Ok(ack) => println!("Unexpected: message stored at sequence {}", ack.sequence), Err(err) => { println!("Publish rejected, nothing stored: {err}"); // The fix is to enable TTLs on the stream: create (or update) it with // `allow_message_ttl: true`, as in the publish-with-TTL example. } } ``` #### C#/.NET ``` // The `Nats-TTL` header only works on a stream created with // AllowMsgTTL = true. Publishing it to a stream that hasn't enabled the // feature is rejected instead of silently storing the message forever. var headers = new NatsHeaders { ["Nats-TTL"] = "60s" }; try { var ack = await js.PublishAsync( subject: "no-ttl.canceled", data: new OrderCancellation(OrderId: "ord_8w2k", Reason: "customer_request"), headers: headers); // PublishAsync returns the server's answer; EnsureSuccess turns a // rejection into an exception instead of leaving it unchecked. ack.EnsureSuccess(); } catch (NatsJSApiException ex) { // 10166: per-message TTL is disabled. The fix is to recreate the // stream with AllowMsgTTL = true so it accepts the `Nats-TTL` header. rejection = ex; output.WriteLine($"Rejected ({ex.Error.Code}): {ex.Error.Description}"); } ``` #### C ``` // Publishing with a TTL to a stream that hasn't enabled AllowMsgTTL // fails. The server returns "per-message TTL is disabled" (err_code // 10166) and stores nothing. The fix is to opt the stream in with // AllowMsgTTL: true. if (s == NATS_OK) { jsPubOptions_Init(&po); po.MsgTTL = 60000; // in milliseconds s = js_Publish(NULL, js, "no-ttl.msg", payload, (int) strlen(payload), &po, &jerr); if ((s != NATS_OK) && (jerr == JSMessageTTLDisabledErr)) { printf("Publish rejected, message not stored: %s (err_code %d)\n", nats_GetLastError(NULL), (int) jerr); s = NATS_OK; } } ``` **A short TTL deletes the message whether or not a consumer read it.** The TTL is a deadline on the *stored copy*, not a guarantee about delivery. It counts down regardless of what any consumer is doing. If the TTL runs out while the consumer that handles cancellations is down, backed up, or reading slowly mid-batch, the server deletes the message and nobody processes the cancellation. A consumer that already holds a message doesn't pause that message's TTL. Do size the TTL to outlast the lag of your slowest healthy consumer. Don't set a TTL shorter than the time in which the message still has to be acted on. **Turning `AllowMsgTTL` on can't be undone.** You can turn the feature on with `nats stream edit ORDERS --allow-msg-ttl`, but the server refuses to turn it back off. Do turn it on on purpose, knowing the stream keeps the feature for life. Don't flip it on to test something and expect to undo it. ## Where you are `ORDERS` now has `AllowMsgTTL` turned on, a switch you can't turn back off. You published an `orders.canceled` message with a 60-second `Nats-TTL` and saw it expire while the rest of the stream remained. The earlier of TTL and `MaxAge` always applies. The stream still holds its earlier messages under the 7-day `MaxAge` set on the [Shaping the stream](/learn/jetstream/shaping-the-stream.md) page. Nothing else changed. ## What's next That's the last of the stream and consumer mechanics. Before the chapter closes, [Stream and consumer policies](/learn/jetstream/policies.md) lines up every policy you've met, and the few you haven't, in one place. ## See also * [Reference → Per-Message TTL](/reference/jetstream/api/headers.md) — the `Nats-TTL` header and the delete-marker headers in full. * [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md) — `AllowMsgTTL` and `SubjectDeleteMarkerTTL` alongside every other stream field. --- # Mirrors and sources So far the running example has been a single `ORDERS` stream. This page covers the two ways to build one stream from another. A **mirror** is a read-only copy of a single stream. Sources aggregate many streams into one. Both behaviors are specified in [ADR-59](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-59.md), the authoritative document for stream sourcing and mirroring. ## What a mirror is A mirror is a stream that continuously copies every message from one upstream stream. **Message flow — A mirror catches up and stays current (animated):** A mirror keeps its own copy of another stream by continuously pulling from it. ORDERS already holds five orders and a publisher keeps adding more; the mirror copies faster than the publisher writes, so its lag falls until it catches up. Once caught up, each new order is mirrored the moment it lands and the lag stays at zero. The copy is exact. A message in the mirror keeps the same sequence number, the same timestamp, and the same subject it had upstream. If `orders.created` was sequence `1` in `ORDERS`, it's sequence `1` in the mirror too. A mirror is read-only. You can't publish to it directly, because it listens on no subjects of its own. Its only job is to follow the upstream. A publish lands in whatever stream owns the subject, not the mirror. A mirror keeps its own retention. The upstream might keep messages for seven days while the mirror keeps them forever — the mirror's own limits decide what it stores, independent of the upstream's. Its configuration is fixed at creation. You can't point a mirror at a different upstream or add a filter later; to change any of that you delete it and create it again. That's cheap, because the upstream still holds the data and the new mirror catches up on its own. ## What sources are A **source** is the inverse of a mirror. Where a mirror copies from one upstream, a stream with sources pulls from several upstreams at once and merges them into a single stream. Consider three regional order streams — `ORDERS-US`, `ORDERS-EU`, `ORDERS-APAC`. A stream that lists all three as sources becomes one combined `ALL-ORDERS` view, fed by every region. **Message flow — Sources merge many streams into one (animated):** A sourced stream pulls from several upstreams at once into one. ALL-ORDERS sources from US, EU, and APAC; each upstream keeps its own order, and across them the messages interleave by arrival time, so one stream carries the merged flow of all three regions. The merge interleaves. Messages from one upstream keep their own order, but across upstreams there's no ordering guarantee, and the aggregate gives them fresh sequence numbers as they arrive — it doesn't preserve each upstream's the way a mirror does. A sourced stream can also listen on its own subjects. Unlike a mirror, it may accept direct publishes alongside the messages it pulls in, so one stream can hold both what it gathered and what was published straight to it. Sources can also change after creation. You add an upstream, drop one, or adjust a filter by updating the stream config — no need to delete and recreate. ## Mirror or source? The two solve different problems. A mirror is one stream copied exactly; a source is many streams merged into one. | | Mirror | Source | | ------------------------------ | ------------------------ | --------------------------------- | | Upstreams | exactly one | one or many | | Sequence numbers | kept from the upstream | fresh, interleaved across sources | | Own subjects, direct publishes | no — read-only | yes, optional | | Change the config later | no — delete and recreate | yes — add, drop, or edit sources | Reach for a **mirror** when you want a second copy of one stream: a read replica close to a remote region, a stream that survives the loss of the upstream's cluster, or a long-retention archive of a short-retention stream. Reach for **sources** when you want to combine many streams into one: merging per-region or per-tenant streams for reporting, or building a derived view that draws from several streams. ## Build them ### Build the ORDERS-ARCHIVE mirror Create a second stream that mirrors `ORDERS`. Call it `ORDERS-ARCHIVE`, and give it no limits, so it becomes a permanent record of every order: #### CLI ``` #!/bin/bash # Create ORDERS-ARCHIVE as a read-only mirror of the ORDERS stream. # A mirror takes no --subjects of its own; it follows the upstream. nats stream add ORDERS-ARCHIVE --mirror ORDERS # Confirm the mirror caught up. The Mirror Information section reports # the upstream stream name, the replication Lag, and the Last Seen time. nats stream info ORDERS-ARCHIVE ``` #### JavaScript/TypeScript ``` // Create ORDERS-ARCHIVE as a read-only mirror of ORDERS. A mirror takes // no subjects of its own; it follows the upstream stream. const jsm = await jetstreamManager(nc); const info = await jsm.streams.add({ name: "ORDERS-ARCHIVE", mirror: { name: "ORDERS" }, }); console.log(`Created mirror ${info.config.name} of ${info.config.mirror?.name}`); ``` #### Go ``` // Create ORDERS-ARCHIVE as a read-only mirror of ORDERS. A mirror takes // no subjects of its own; it follows the upstream stream. stream, err := js.CreateStream(ctx, jetstream.StreamConfig{ Name: "ORDERS-ARCHIVE", Mirror: &jetstream.StreamSource{Name: "ORDERS"}, }) if err != nil { panic(err) } // Confirm: the new stream mirrors ORDERS. cfg := stream.CachedInfo().Config fmt.Printf("Created mirror %s of %s\n", cfg.Name, cfg.Mirror.Name) ``` #### Python ``` # Create ORDERS-ARCHIVE as a read-only mirror of ORDERS. A mirror takes # no subjects of its own; it follows the upstream stream. info = await js.add_stream(name="ORDERS-ARCHIVE", mirror=StreamSource(name="ORDERS")) print(f"Created mirror {info.config.name} of {info.config.mirror.name}") ``` #### Java ``` // Create ORDERS-ARCHIVE as a read-only mirror of ORDERS. A mirror // takes no subjects of its own; it follows the upstream stream. JetStreamManagement jsm = nc.jetStreamManagement(); StreamInfo streamInfo = jsm.addStream(StreamConfiguration.builder() .name("ORDERS-ARCHIVE") .mirror(Mirror.builder().sourceName("ORDERS").build()) .storageType(StorageType.File) .build()); // Confirm: the new stream mirrors ORDERS StreamConfiguration config = streamInfo.getConfiguration(); System.out.println("Created mirror " + config.getName() + " of " + config.getMirror().getSourceName()); ``` #### Rust ``` // Create ORDERS-ARCHIVE as a read-only mirror of ORDERS. A mirror takes // no subjects of its own; it follows the upstream stream. let stream = js .create_stream(jetstream::stream::Config { name: "ORDERS-ARCHIVE".to_string(), mirror: Some(Source { name: "ORDERS".to_string(), ..Default::default() }), ..Default::default() }) .await?; // Confirm: the new stream mirrors ORDERS. let cfg = &stream.cached_info().config; println!( "Created mirror {} of {}", cfg.name, cfg.mirror.as_ref().map_or("unknown", |m| m.name.as_str()) ); ``` #### C#/.NET ``` // Create ORDERS-ARCHIVE as a read-only mirror of ORDERS. A mirror takes // no subjects of its own; it follows the upstream stream. var stream = await js.CreateStreamAsync(new StreamConfig(name: "ORDERS-ARCHIVE", subjects: []) { Mirror = new StreamSource { Name = "ORDERS" }, }); // Confirm: the new stream mirrors ORDERS output.WriteLine($"Created mirror {stream.Info.Config.Name} of {stream.Info.Config.Mirror!.Name}"); ``` #### C ``` // Create ORDERS-ARCHIVE as a read-only mirror of ORDERS. A mirror takes // no subjects of its own; it follows the upstream stream. if (s == NATS_OK) { jsStreamConfig_Init(&mc); jsStreamSource_Init(&mirror); mirror.Name = "ORDERS"; mc.Name = "ORDERS-ARCHIVE"; mc.Mirror = &mirror; s = js_AddStream(&si, js, &mc, NULL, &jerr); } // Confirm: the new stream mirrors ORDERS. if (s == NATS_OK) printf("Created mirror %s of %s\n", si->Config->Name, si->Config->Mirror->Name); ``` The `--mirror ORDERS` flag tells the server this new stream is a mirror of `ORDERS` rather than a normal stream. You don't give it `--subjects`, because a mirror listens on no subjects of its own. The CLI exposes only the simplest mirror; for one that filters or rewrites subjects you supply a JSON config, covered in the Reference. Right after creation the mirror catches up. Within moments it holds the same three orders that `ORDERS` does, reported in a section a normal stream doesn't have: ``` Mirror Information: Stream Name: ORDERS Lag: 0 Last Seen: 1.20s ``` **Stream Name** is the upstream the mirror follows. **Lag** is how many messages it's still behind — `0` means fully caught up, and a lag that climbs and stays high means it can't keep pace. **Last Seen** is the time since the last message or heartbeat from the upstream; a small, steady value is healthy. Publish a fourth order into `ORDERS`, then re-run `nats stream info ORDERS-ARCHIVE`. The mirror picks it up on its own, with no consumer and no client code involved, and the lag ticks back to `0`. ### Build the ALL-ORDERS source A source needs streams to aggregate. Create three regional streams, each owning its own subjects, then create `ALL-ORDERS` to source all three at once: #### CLI ``` #!/bin/bash # Sources aggregate many streams into one. First, the three regional streams # ALL-ORDERS will pull from — each owns its own subjects: nats stream add ORDERS-US --subjects 'us.orders.>' nats stream add ORDERS-EU --subjects 'eu.orders.>' nats stream add ORDERS-APAC --subjects 'apac.orders.>' # Create ALL-ORDERS as an aggregate of all three. A sourced stream can list # several upstreams (a mirror takes exactly one). It needs no --subjects of # its own. nats stream add ALL-ORDERS --source ORDERS-US --source ORDERS-EU --source ORDERS-APAC # Confirm. The Source Information section lists each upstream, with its own Lag. nats stream info ALL-ORDERS # Source Information: # # Stream Name: ORDERS-US # Lag: 0 # Last Seen: 1.20s # # Stream Name: ORDERS-EU # Lag: 0 # Last Seen: 1.20s # # Stream Name: ORDERS-APAC # Lag: 0 # Last Seen: 1.20s ``` #### JavaScript/TypeScript ``` // Create ALL-ORDERS as an aggregate that sources the three regional streams // into one. Unlike a mirror, a stream can list several sources. const info = await jsm.streams.add({ name: "ALL-ORDERS", sources: [ { name: "ORDERS-US" }, { name: "ORDERS-EU" }, { name: "ORDERS-APAC" }, ], }); console.log(`Created ${info.config.name} sourcing ${info.config.sources?.length} streams`); ``` #### Go ``` // Create ALL-ORDERS as an aggregate that sources the three regional streams // into one. Unlike a mirror, a stream can list several sources. stream, err := js.CreateStream(ctx, jetstream.StreamConfig{ Name: "ALL-ORDERS", Sources: []*jetstream.StreamSource{ {Name: "ORDERS-US"}, {Name: "ORDERS-EU"}, {Name: "ORDERS-APAC"}, }, }) if err != nil { panic(err) } cfg := stream.CachedInfo().Config fmt.Printf("Created %s sourcing %d streams\n", cfg.Name, len(cfg.Sources)) ``` #### Python ``` # Create ALL-ORDERS as an aggregate that sources the three regional streams # into one. Unlike a mirror, a stream can list several sources. info = await js.add_stream( name="ALL-ORDERS", sources=[ StreamSource(name="ORDERS-US"), StreamSource(name="ORDERS-EU"), StreamSource(name="ORDERS-APAC"), ], ) print(f"Created {info.config.name} sourcing {len(info.config.sources)} streams") ``` #### Java ``` // Create ALL-ORDERS as an aggregate that sources the three regional // streams into one. Unlike a mirror, a stream can list several sources. StreamInfo streamInfo = jsm.addStream(StreamConfiguration.builder() .name("ALL-ORDERS") .sources( Source.builder().sourceName("ORDERS-US").build(), Source.builder().sourceName("ORDERS-EU").build(), Source.builder().sourceName("ORDERS-APAC").build()) .storageType(StorageType.File) .build()); // Confirm: the aggregate lists three sources StreamConfiguration config = streamInfo.getConfiguration(); System.out.println("Created " + config.getName() + " sourcing " + config.getSources().size() + " streams"); ``` #### Rust ``` // Create ALL-ORDERS as an aggregate that sources the three regional streams // into one. Unlike a mirror, a stream can list several sources. let stream = js .create_stream(jetstream::stream::Config { name: "ALL-ORDERS".to_string(), sources: Some(vec![ Source { name: "ORDERS-US".to_string(), ..Default::default() }, Source { name: "ORDERS-EU".to_string(), ..Default::default() }, Source { name: "ORDERS-APAC".to_string(), ..Default::default() }, ]), ..Default::default() }) .await?; let cfg = &stream.cached_info().config; println!( "Created {} sourcing {} streams", cfg.name, cfg.sources.as_ref().map_or(0, |s| s.len()) ); ``` #### C#/.NET ``` // Create ALL-ORDERS as an aggregate that sources the three regional // streams into one. Unlike a mirror, a stream can list several sources. var stream = await js.CreateStreamAsync(new StreamConfig(name: "ALL-ORDERS", subjects: []) { Sources = [ new StreamSource { Name = "ORDERS-US" }, new StreamSource { Name = "ORDERS-EU" }, new StreamSource { Name = "ORDERS-APAC" }, ], }); // Confirm: the aggregate lists three sources output.WriteLine($"Created {stream.Info.Config.Name} sourcing {stream.Info.Config.Sources!.Count} streams"); ``` #### C ``` // Create ALL-ORDERS as an aggregate that sources the three regional // streams into one. Unlike a mirror, a stream can list several sources. jsStreamConfig sc; jsStreamSource us, eu, apac; jsStreamSource *sources[] = {&us, &eu, &apac}; jsStreamSource_Init(&us); us.Name = "ORDERS-US"; jsStreamSource_Init(&eu); eu.Name = "ORDERS-EU"; jsStreamSource_Init(&apac); apac.Name = "ORDERS-APAC"; jsStreamConfig_Init(&sc); sc.Name = "ALL-ORDERS"; sc.Sources = sources; sc.SourcesLen = 3; s = js_AddStream(&si, js, &sc, NULL, &jerr); if (s == NATS_OK) printf("Created %s sourcing %d streams\n", si->Config->Name, si->Config->SourcesLen); ``` `ALL-ORDERS` takes no `--subjects` of its own here; it just lists its upstreams. Its info carries a Source Information section — one block per upstream, because each source replicates on its own: ``` Source Information: Stream Name: ORDERS-US Lag: 0 Last Seen: 1.20s Stream Name: ORDERS-EU Lag: 0 Last Seen: 1.20s Stream Name: ORDERS-APAC Lag: 0 Last Seen: 1.20s ``` Each upstream has its own Lag and Last Seen. `ALL-ORDERS` now holds every order from every region, interleaved in arrival order. Add or drop a region later by updating the stream — no recreation needed. ## Filters, transforms, and reach A mirror or source can copy a subset of subjects with a filter, rewrite subjects with a subject transform, or reach a stream in another account or JetStream domain. Each is one extra field on the mirror or source configuration. Reaching across an account or domain involves three subjects, each with a required export type. Setting one wrong is a common mistake — the Pitfalls below cover which type each subject needs and what goes wrong. The full set of mirror and source options (`filter_subject`, `subject_transforms`, `opt_start_seq`, `external`, and the rest) is documented in [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md). We use only the plain `--mirror` and `--source` forms here. Using mirrors for disaster recovery (switching over to a mirror when the primary cluster is lost) is its own operational topic, covered in [Operate → Backup & Recovery](/learn/backup-recovery/mirrors-and-sources.md). ## Pitfalls Mirrors and sources add little configuration, but a few of their rules are easy to get wrong the first time. **Treating a mirror as writable.** A mirror listens on no subjects of its own, so no subject routes a publish to it. Publish `orders.shipped` and the message lands in the origin `ORDERS` stream that owns that subject — never in `ORDERS-ARCHIVE`. A plain publish reaches the origin unless subject mapping or a cross-domain setup redirects it, in which case it fails. Force the mirror by name with a `Nats-Expected-Stream: ORDERS-ARCHIVE` header and the server rejects the publish with `expected stream does not match` (error `10060`), because the subject still routed to `ORDERS`, not the mirror. A mirror is read-only either way: publish to the upstream `ORDERS` and let the mirror copy the message on its own. #### CLI ``` #!/bin/bash # A mirror is read-only. It captures no subjects of its own, so a publish # aimed at the mirror name reaches no stream. A JetStream publish waits for # a PubAck that never comes and fails with "no responders available". # (A plain `nats pub` would instead report "Published N bytes" and the # message would silently go nowhere.) nats pub --jetstream ORDERS-ARCHIVE '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # Published 88 bytes to "ORDERS-ARCHIVE" # nats: error: nats: no responders available for request # Publish to the upstream ORDERS stream instead. The mirror copies the # message on its own, with no client code involved. nats pub --jetstream orders.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # Published 88 bytes to "orders.created" # Stored in Stream: ORDERS Sequence: 4 ``` **Treating mirror contents as real-time.** A mirror is eventually consistent: the server copies the upstream stream continuously, so the mirror can run slightly behind. During a burst of writes, its `Lag` climbs above `0` until it catches up. Don't assume a message in `ORDERS` is already in `ORDERS-ARCHIVE` the instant it lands. Read the `Lag` field first, and treat a lag that climbs and stays high as a sign the mirror can't keep pace. #### CLI ``` #!/bin/bash # A mirror is eventually consistent, not real-time. Read the Lag field # before you trust the mirror to hold what the upstream just received. # Lag 0 means fully caught up; a non-zero Lag means messages are still # in flight from the upstream. nats stream info ORDERS-ARCHIVE # Mirror Information: # # Stream Name: ORDERS # Lag: 0 # Last Seen: 1.20s ``` #### JavaScript/TypeScript ``` // A mirror is eventually consistent. Read its lag before trusting it to // hold what the upstream just received: 0 means fully caught up. const jsm = await jetstreamManager(nc); const info = await jsm.streams.info("ORDERS-ARCHIVE"); console.log(`Upstream: ${info.mirror?.name}`); console.log(`Lag: ${info.mirror?.lag}`); console.log(`Last seen: ${info.mirror?.active} ns ago`); ``` #### Go ``` // A mirror is eventually consistent. Read its Lag before trusting it to // hold what the upstream just received: 0 means fully caught up. stream, err := js.Stream(ctx, "ORDERS-ARCHIVE") if err != nil { panic(err) } info, err := stream.Info(ctx) if err != nil { panic(err) } fmt.Printf("Upstream: %s\n", info.Mirror.Name) fmt.Printf("Lag: %d\n", info.Mirror.Lag) fmt.Printf("Last seen: %s ago\n", info.Mirror.Active) ``` #### Python ``` # A mirror is eventually consistent. Read its lag before trusting it to # hold what the upstream just received: 0 means fully caught up. info = await js.stream_info("ORDERS-ARCHIVE") print(f"Upstream: {info.mirror.name}") print(f"Lag: {info.mirror.lag}") print(f"Last seen: {info.mirror.active}") ``` #### Java ``` // A mirror is eventually consistent. Read its lag before trusting // it to hold what the upstream just received: 0 means caught up. StreamInfo streamInfo = jsm.getStreamInfo("ORDERS-ARCHIVE"); MirrorInfo mirror = streamInfo.getMirrorInfo(); System.out.println("Upstream: " + mirror.getName()); System.out.println("Lag: " + mirror.getLag()); System.out.println("Last seen: " + mirror.getActive()); ``` #### Rust ``` // A mirror is eventually consistent. Read its lag before trusting it to // hold what the upstream just received: 0 means fully caught up. let mut stream = js.get_stream("ORDERS-ARCHIVE").await?; let info = stream.info().await?; let Some(mirror) = info.mirror.as_ref() else { println!("stream is not a mirror"); return Ok(()); }; println!("Upstream: {}", mirror.name); println!("Lag: {}", mirror.lag); println!("Last seen: {:?} ago", mirror.active); ``` #### C#/.NET ``` // A mirror is eventually consistent. Read its lag before trusting it to // hold what the upstream just received: 0 means fully caught up. var stream = await js.GetStreamAsync("ORDERS-ARCHIVE"); var mirror = stream.Info.Mirror!; output.WriteLine($"Upstream: {mirror.Name}"); output.WriteLine($"Lag: {mirror.Lag}"); output.WriteLine($"Last seen: {mirror.Active}"); ``` #### C ``` // A mirror is eventually consistent. Read its Lag before trusting it to // hold what the upstream just received: 0 means fully caught up. s = js_GetStreamInfo(&si, js, "ORDERS-ARCHIVE", NULL, &jerr); if ((s == NATS_OK) && (si->Mirror != NULL)) { printf("Upstream: %s\n", si->Mirror->Name); printf("Lag: %" PRIu64 "\n", si->Mirror->Lag); printf("Last seen: %.2fs ago\n", (double) si->Mirror->Active / 1E9); } ``` **Combining a filter with a transform on one source.** On a single source or mirror entry, you can set `filter_subject` or `subject_transforms`, but not both: the server rejects a config that sets both. Use `filter_subject` when you only need to select a subset of subjects, and `subject_transforms` when you also need to rename them (a transform filters and renames in one step). Don't reach for both fields on the same entry. Pick the one that fits. **Cross-domain config that fails silently.** Reaching a stream in another account or JetStream domain needs the `external` block plus matching exports and imports on both sides, and each of the three subjects has a required type. The consumer API and flow-control subjects are *service* exports, because they work as request and reply. The delivery subject is a *stream* export, because the messages flow one way. Get a type wrong and replication doesn't fail with an error; the mirror never catches up. Check each import type against [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md). Setting up cross-account and cross-domain access is part of configuring accounts and authorization. ## Where you are You now have: * an `ORDERS` stream, unchanged * an `ORDERS-ARCHIVE` mirror — an exact, read-only copy of it * an `ALL-ORDERS` aggregate that sources three regional streams into one ## What's next The next page covers [reading messages directly](/learn/jetstream/get-direct.md): getting one message or a batch straight from the stream, with no consumer, served by any replica or mirror. After that, [subject mapping](/learn/jetstream/subject-mapping.md), [per-message TTL](/learn/jetstream/message-ttl.md), and [stream and consumer policies](/learn/jetstream/policies.md), then [Where to go next](/learn/jetstream/where-next.md) recaps the chapter. ## See also * [Reference → ADR-59](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-59.md) — the authoritative spec for mirroring and sourcing behavior. * [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md) — every mirror and source field and its valid values. * [Operate → Backup & Recovery](/learn/backup-recovery/mirrors-and-sources.md) — using mirrors for disaster recovery. --- # Ordered consumers Most of this chapter built consumers that stick around. `billing` keeps its place so it can resume after a restart. `shipping` is a durable consumer a whole pool of workers shares. Both are made to last. Some reads aren't. Finance wants to total every order from launch to now, in the order they happened, to reconcile the books. You read the whole `ORDERS` log once, top to bottom, and then you never need that reader again. There's no work to share, no ack to send, and nothing worth keeping once the pass is done. An **ordered consumer** is built for exactly that: a fast, in-order read of a stream that cleans up after itself. ## What you write You ask the library for an ordered consumer over the stream and read it like any other consumer — iterate, take each message, move on. There's no name to pick, no ack to send, and no consumer to delete when you finish. The library hands you every message in stream order, from your start point to the end, as one continuous flow. #### JavaScript/TypeScript ``` // Ask for an ordered consumer over the stream: consumers.get() with no consumer // name returns one. There's no ack to send — the library runs the consumer for // you and recreates it if it ever misses a message, so you read every order in // stream order. const js = jetstream(nc); const c = await js.consumers.get("ORDERS"); // Read the whole log once, in order, stopping when caught up (pending 0). const messages = await c.consume(); for await (const m of messages) { console.log(`order ${m.string()}`); if (m.info.pending === 0) { break; } } ``` #### Go ``` // Ask for an ordered consumer over the stream. There's no name to manage and // no ack to send: the library runs the consumer for you and recreates it if // it ever misses a message, so you read every order in stream order. An // empty config starts from the first order. cons, err := js.OrderedConsumer(ctx, "ORDERS", jetstream.OrderedConsumerConfig{}) if err != nil { panic(err) } iter, err := cons.Messages() if err != nil { panic(err) } defer iter.Stop() // Read the whole log once, in order, stopping when caught up (NumPending 0). for { msg, err := iter.Next() if err != nil { break } meta, err := msg.Metadata() if err != nil { panic(err) } fmt.Printf("order %s\n", string(msg.Data())) if meta.NumPending == 0 { break } } ``` #### Python ``` # Ask for an ordered consumer with ordered_consumer=True. There's no ack to # send: nats.py runs the consumer for you and recreates it if it ever misses # a message, so you read every order in stream order. deliver_policy=ALL # starts from the first order. psub = await js.subscribe( "orders.>", stream="ORDERS", ordered_consumer=True, deliver_policy=DeliverPolicy.ALL, ) # Read the whole log once, in order, stopping when caught up (num_pending 0). while True: msg = await psub.next_msg(timeout=5) print(f"order {msg.data.decode()}") if msg.metadata.num_pending == 0: break ``` #### Java ``` // Ask for an ordered consumer over the stream. There's no ack to // send: the library runs the consumer for you and recreates it if it // ever misses a message, so you read every order in stream order. // DeliverPolicy.All starts from the first order. OrderedConsumerContext occ = sc.createOrderedConsumer( new OrderedConsumerConfiguration().deliverPolicy(DeliverPolicy.All)); // Read the whole log once, in order, stopping when caught up. Message m; while ((m = occ.next(Duration.ofSeconds(5))) != null) { System.out.println("order " + new String(m.getData(), StandardCharsets.UTF_8)); if (m.metaData().pendingCount() == 0) { break; } } ``` #### Rust ``` // Ask for an ordered consumer over the stream. There's no name to manage and // no ack to send: the library runs the consumer for you and recreates it if // it ever misses a message, so you read every order in stream order. let stream = js.get_stream("ORDERS").await?; let consumer = stream .create_consumer(consumer::pull::OrderedConfig { deliver_policy: consumer::DeliverPolicy::All, ..Default::default() }) .await?; // Read the whole log once, in order, stopping when caught up (pending 0). let mut messages = consumer.messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("order {}", std::str::from_utf8(&msg.payload)?); if msg.info()?.pending == 0 { break; } } ``` #### C#/.NET ``` // Ask for an ordered consumer over the stream. There's no ack to send: // the library runs the consumer for you and recreates it if it ever // misses a message, so you read every order in stream order. var consumer = await js.CreateOrderedConsumerAsync("ORDERS"); // Read the whole log once, in order, stopping when caught up. await foreach (var msg in consumer.ConsumeAsync()) { output.WriteLine($"order {msg.Data}"); read++; if (msg.Metadata?.NumPending == 0) { break; } } ``` #### C ``` // Ask for an ordered consumer over the stream. There's no name to // manage and no ack to send: the library runs the consumer for you and // recreates it if it ever misses a message, so you read every order in // stream order, starting from the first one. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Ordered = true; s = js_SubscribeSync(&sub, js, "orders.>", NULL, &so, &jerr); // Read the whole log once, in order, stopping when caught up // (NumPending 0). while (s == NATS_OK) { natsMsg *msg = NULL; jsMsgMetaData *meta = NULL; uint64_t pending = 0; s = natsSubscription_NextMsg(&msg, sub, 5000); if (s != NATS_OK) break; s = natsMsg_GetMetaData(&meta, msg); if (s == NATS_OK) { pending = meta->NumPending; printf("order %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); jsMsgMetaData_Destroy(meta); } natsMsg_Destroy(msg); if ((s == NATS_OK) && (pending == 0)) break; } // An empty stream simply times out waiting for the first message. if (s == NATS_TIMEOUT) s = NATS_OK; ``` That's the whole client-side surface: ask for an ordered consumer, loop until the log is drained. The ordered consumer is a client-library construct, so there's no CLI form. ## What the library does The simplicity hides a loop. Under the cover, the library creates a consumer with a short **inactivity threshold** — a span of idle time after which the server deletes the consumer — and tracks the stream sequence of each message it delivers. As long as the sequences arrive in order, it passes them straight to you. When a sequence goes missing, or the consumer goes quiet — its heartbeats stop because it was deleted, lost on a reconnect, or dropped in a node restart — the library throws the old consumer away and creates a fresh one starting at the next sequence it expected. Each new consumer gets its own name (`prefix_1`, `prefix_2`, and so on). You see one unbroken, in-order stream through all of it; the recovery is invisible. That recovery is what lets an ordered consumer drop acks and still keep order. Without acks there's nothing to trigger a redelivery, so a plain no-ack consumer that missed a message would skip it and read on with a gap. The ordered consumer instead spots the gap and rebuilds itself from the missing sequence — the message is still in the stream — so the order holds with no acks at all. ## The config underneath An ordered consumer isn't a separate kind of consumer on the server. It's an ordinary consumer the library configures a specific way, and every choice fits a throwaway in-order read: * **No acks** (`AckPolicy=none`). You're reading straight through, not processing work that has to land exactly once. Nothing to acknowledge, no redelivery to reason about. * **Memory storage.** The cursor lives in memory, not on disk. It's faster, and a one-shot read has no reason to make its position survive a restart. * **One replica.** A single copy of the consumer's state. Cheap, and there's no position worth replicating for a read you can simply run again. * **A short inactivity threshold** (five minutes). The server deletes the consumer once it sits idle that long, so it disappears on its own after the pass. This is the part that cleans up after you. * **Start where you ask.** The first consumer uses the start point you pick — the whole log, a specific sequence, or a point in time. After a recovery, the library pins the replacement consumer to the next sequence it expected. These are the same knobs any consumer has. The ordered consumer just sets them all toward speed and disposability instead of durability. The full set of fields is in [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md). ## What you give up The trade is in what you can't do: * **No per-message acks.** You can't mark one order handled and another not, so this isn't the tool for processing that has to happen exactly once. That's a named consumer with explicit ack — the `shipping` pattern. * **No sharing.** Each reader creates its own ordered consumer, so two processes can't split one between them the way a pool shares one `shipping` consumer. * **No parallelism.** Delivery is single-threaded and in order, one message at a time. In return you get a gap-free, in-order read with no ack bookkeeping and nothing left behind. ## Where ordered consumers show up Ordered consumers power features you may use elsewhere in NATS: a [Key-Value](/learn/key-value/.md) watch and an [Object Store](/learn/object-store/.md) read are both ordered consumers under the cover — each walks a stream straight through in order. Reach for one whenever you want to read a stream top to bottom without coordinating readers or tracking acks. ## Pitfalls **Using it for work that must be processed once.** An ordered consumer doesn't ack, so it can't mark individual messages handled — there's no way to record that order 5 shipped but order 6 didn't. For processing where each message must land exactly once, use a named consumer with explicit ack and let redelivery cover failures. **Expecting to share progress across processes.** Each reader gets its own ordered consumer with its own position, so two processes reading this way both read the whole stream — they don't split it. To share work, use a named consumer and a [worker pool](/learn/jetstream/worker-pool.md). **Counting on the position to survive.** Memory storage, one replica, and a short inactivity threshold make the cursor disposable on purpose. Don't build a long-running job on an ordered consumer expecting it to resume where it left off after a crash — it starts over. ## What's next An ordered consumer is one reader walking the log alone. Back at the worker pool, several readers share one `shipping` consumer — and the next page steers that split: [priority groups](/learn/jetstream/priority-groups.md) can send all the work to one client until it fails, or keep a standby idle until the pool falls behind. ## See also * [ADR-17: Ordered Consumer](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-17.md) — the behavior the clients implement. * [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md) — every field the library sets under the cover. * [Key-Value](/learn/key-value/.md) and [Object Store](/learn/object-store/.md) — ordered consumers at work. --- # Pausing a consumer Acme's shipping carrier sends a notice: the API the `shipping` workers call to book each shipment goes down for maintenance this Sunday, 03:00 to 05:00. For those two hours no shipment can be booked. Left running, the pool keeps pulling orders, fails every booking, and churns through redeliveries against a dead API. You could delete the `shipping` consumer and recreate it after the window, but that throws away everything it tracked: how far it has read and which orders it has acked. Pausing fits exactly here. A **paused** consumer stops receiving messages until a deadline you set, and keeps all of its progress until then. You pause `shipping` until 05:00; orders keep landing in `ORDERS`, and the pool resumes on its own when the carrier is back, picking up right where it stopped. ## What pausing keeps Pausing changes one thing: the server stops delivering messages to the consumer. Everything else stays the same. The cursor, the sequence number the consumer has read up to, doesn't move. Acknowledged messages stay acknowledged. The count of how often each message has been redelivered stays as it was. When the pause ends, delivery resumes from the position it stopped at. This is the difference between pausing and deleting a consumer. A deleted consumer loses all of this; a paused consumer keeps it. ## Pause until a deadline You pause a consumer *until* a moment in time. When that moment arrives, the consumer resumes on its own, with no second command needed. Pause the `shipping` consumer for one hour: #### CLI ``` #!/bin/bash # Pause the shipping consumer for one hour. The deadline accepts a # duration ("1h", "30m") meaning "from now", or an RFC3339 timestamp # like "2026-05-22 14:30:00" for an exact wall-clock time. nats consumer pause ORDERS shipping "1h" --force # Check the pause state and how much time is left on the deadline. nats consumer info ORDERS shipping # Resume early, before the deadline. Delivery picks up at the same # cursor position the consumer held when it was paused. nats consumer resume ORDERS shipping --force ``` #### C ``` // Pause the shipping consumer for one hour. The server stores a fixed // deadline: now plus one hour, in nanoseconds since the epoch. jsConsumerPauseResponse *resp = NULL; uint64_t deadline = (uint64_t) nats_NowInNanoSeconds() + 3600ULL * 1000000000ULL; s = js_PauseConsumer(&resp, js, "ORDERS", "shipping", deadline, NULL, &jerr); if (s == NATS_OK) { printf("Paused: %s, remaining: %.0fs\n", resp->Paused ? "true" : "false", (double) resp->PauseRemaining / 1E9); jsConsumerPauseResponse_Destroy(resp); resp = NULL; } // Resume early, before the deadline: a pause request with no deadline // lifts the pause. Delivery picks up at the same cursor position the // consumer held when it was paused. if (s == NATS_OK) s = js_PauseConsumer(&resp, js, "ORDERS", "shipping", 0, NULL, &jerr); if (s == NATS_OK) { printf("Paused: %s\n", resp->Paused ? "true" : "false"); jsConsumerPauseResponse_Destroy(resp); } ``` The CLI accepts two forms for the deadline. A duration like `1h` or `30m` means "from now." A time like `2026-05-22 14:30:00` means that exact clock time. Either way the server stores a fixed deadline. Clients can also pause a consumer the moment they create it. Set the `PauseUntil` field in the consumer config to a future time, and the consumer starts out paused until that deadline. It's the same fixed deadline the CLI sets, just given when the consumer is first created. See [Reference → Consumer configuration](/reference/jetstream/api/consumer/create.md) for the field. The command confirms the pause and the time remaining: ``` Paused ORDERS > shipping until 2026-05-22 11:14:22 (59m58s) ``` While the consumer is paused, the stream keeps accepting new messages as normal. They pile up behind the cursor, waiting. The pause stops delivery to the consumer, not storage in the stream. ## Check the pause from consumer info `nats consumer info` reports the pause state: ``` nats consumer info ORDERS shipping ``` A paused consumer shows the deadline in the Configuration section, and the cursor in the State section below it: ``` Configuration: Paused Until Deadline: 2026-05-22 11:14:22 (57m11s remaining) State: Last Delivered Message: Consumer sequence: ... Stream sequence: ... Acknowledgment Floor: Consumer sequence: ... Stream sequence: ... ``` The `Paused Until Deadline` line sits under Configuration, with the deadline and how long is left. The cursor values under State — last delivered and acknowledgment floor — stay wherever they were; whatever they are, the pause doesn't move them. ## Resume early The deadline auto-resumes the consumer. If you want it back sooner, the `resume` command lifts the pause immediately: ``` nats consumer resume ORDERS shipping --force ``` The server confirms, and delivery picks up at once: ``` Consumer ORDERS > shipping was resumed while previously paused until 2026-05-22 11:14:22 ``` Resuming before the deadline and letting the deadline expire end up the same way: a running consumer at the same cursor. The only difference is whether you or the deadline decides the timing. ## When to pause The carrier window is the first of two common reasons. A **maintenance window** is any planned time when a system the consumer feeds into goes offline, so delivering messages no worker can handle yet only wastes attempts. The second is **backpressure**: a downstream system is overloaded and needs room to recover, so you pause to stop the flow and resume once it catches up. In both cases the cursor doesn't move, so you stop delivery without losing the consumer's place: nothing is lost and nothing is handled twice. ## Availability Pause runs on the consumer's leader, the server that holds the timer for the deadline. If that leader changes while a consumer is paused, the new leader takes over the deadline and resumes on time. You don't have to pause again. Pausing consumers needs NATS Server 2.11 or later. An older server rejects the command with a clear message. The full `PauseUntil` API and the pause notification the server sends are documented in [Reference → Pause consumer](/reference/jetstream/api/consumer/pause.md) and the [pause advisory](/reference/jetstream/advisory/consumer-pause.md). We use only the pause and resume commands here. ## Pitfalls **A deadline in the past does nothing.** Pause stores a fixed moment in time. If that moment has already passed, the server leaves the consumer running. The CLI tells you this instead of acting as if the pause took effect. Pause with a duration like `1h` instead, since it's always measured from now and can't land in the past. #### CLI ``` #!/bin/bash # A deadline in the past is a no-op: the consumer is not paused. The CLI # tells you exactly that instead of silently doing nothing. nats consumer pause ORDERS shipping "2020-01-01 00:00:00" --force # Output: consumer failed to pause, perhaps a time in the past was given # Use a future deadline. A duration like "1h" is always "from now", so it # can never land in the past. nats consumer pause ORDERS shipping "1h" --force # Confirm the pause took, with the deadline and time remaining. nats consumer info ORDERS shipping ``` #### C ``` // Pause stores a fixed moment in time. A deadline already in the past // is a no-op: the server leaves the consumer running and reports it // as not paused. jsConsumerPauseResponse *resp = NULL; uint64_t past = 1577836800000000000ULL; // 2020-01-01 00:00:00 UTC s = js_PauseConsumer(&resp, js, "ORDERS", "shipping", past, NULL, &jerr); if (s == NATS_OK) { // Prints "Paused: false" — the deadline had already passed. printf("Paused: %s\n", resp->Paused ? "true" : "false"); jsConsumerPauseResponse_Destroy(resp); resp = NULL; } // Use a deadline measured from now instead — it can't land in the // past. if (s == NATS_OK) { uint64_t deadline = (uint64_t) nats_NowInNanoSeconds() + 3600ULL * 1000000000ULL; // one hour from now s = js_PauseConsumer(&resp, js, "ORDERS", "shipping", deadline, NULL, &jerr); } if (s == NATS_OK) { printf("Paused: %s, remaining: %.0fs\n", resp->Paused ? "true" : "false", (double) resp->PauseRemaining / 1E9); jsConsumerPauseResponse_Destroy(resp); } ``` **Pausing does not stop publishers.** Pause is a consumer setting, so it doesn't change the stream. New messages keep arriving while the `shipping` consumer is paused, and they count toward the stream's storage limits. A long pause on a stream with a tight `MaxMsgs` or `MaxBytes` can drop the oldest orders before the consumer reads them. Size the stream for the longest pause you expect, or keep pauses short. See [Shaping the stream](/learn/jetstream/shaping-the-stream.md) for how the limits apply. ## Where you are The `shipping` consumer has been paused until a deadline and then resumed. Its cursor stayed where it was: no messages were lost and none were delivered twice by mistake. The stream is unchanged, and the consumer is running again at the same place it held before the pause. ## What's next With consumers covered, the next page returns to the stream itself: the [limits and discard policy](/learn/jetstream/shaping-the-stream.md) that decide what `ORDERS` keeps and what it drops once it fills up. ## See also * [Reference → Pause consumer](/reference/jetstream/api/consumer/pause.md) — the `PauseUntil` field and the pause response. * [Reference → Consumer pause advisory](/reference/jetstream/advisory/consumer-pause.md) — the notification the server sends when a consumer pauses or resumes. * [Pull consumers in depth](/learn/jetstream/pull-consumers.md) — the consumer state that pausing preserves. --- # Stream and consumer policies Much of what you configured in this chapter came down to policies. A **policy** is a config field that takes one value from a short, fixed list, and each policy answers one question about how the stream or the consumer behaves. You met most of them one page at a time as the `ORDERS` walkthrough grew; this page lines up all nine in one place. Two things are true of every policy. Each has a default, and the defaults are safe to start with: `ORDERS` ran for most of the chapter without you touching them. And five of the nine are fixed the moment the stream or consumer exists. The server refuses the update, and the only way to change your mind is to recreate the object. The table near the end shows which ones. ## Stream policies Five policies shape a stream. **Retention** decides when a message leaves the stream. `limits` keeps messages until a cap removes them. `interest` and `workqueue` remove messages once consumers have acked them. `ORDERS` stayed on `limits`, and the `FULFILLMENT` work queue ran on `workqueue`. Choosing among the three is covered on [Retention policies](/learn/jetstream/retention-policies.md). **Discard** decides what happens when a stream at its limit receives one more message. `old` deletes the oldest messages to make room; `new` refuses the publish. `ORDERS` kept the default, `old`; [Shaping the stream](/learn/jetstream/shaping-the-stream.md) covers when `new` is worth the rejected publishes. **Storage** decides where messages live. `file` writes them to disk and they survive a server restart. `memory` keeps them only in RAM: faster, and gone on restart. The choice applies to the whole stream, replicas included. How storage and replicas together decide what survives a failure is covered on [Surviving node loss](/learn/jetstream/surviving-node-loss.md). **Compression** is one this chapter didn't need. Setting it to `s2` (from the default `none`) makes a file-storage stream compress message blocks on disk, spending CPU to cut disk usage. Repetitive payloads such as JSON compress well; already-compressed payloads such as images don't. Set it at creation with `--compression s2` on `nats stream add`. Editing it later is allowed, but the new setting waits until the stream's store restarts, on a server restart or a leader change, and blocks already on disk stay as they are. **Persist mode** decides when the server acknowledges a publish relative to writing it out. By default, the message is flushed to storage first and the `PubAck` follows. On `async`, the server acknowledges first and flushes in the background. That buys a higher ingest rate in exchange for a window in which a crash can lose messages the server already acked, so it fits data that arrives in volume and tolerates a lost tail, such as metrics — not an order log. `async` is only accepted on a file-storage stream with a single replica, and an `async` stream refuses the atomic batch publishing from [Advanced publishing](/learn/jetstream/advanced-publishing.md). Every stream field, with each policy's values described inline, is in [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md). ## Consumer policies Four policies shape a consumer. **Deliver policy** decides where a consumer starts reading, once, at creation. `billing` was created with `all`, which is also the default: start at the stream's first message. `last` starts at the newest message and continues live. `new` delivers only messages that arrive after the consumer is created. `by_start_sequence` and `by_start_time` start at a sequence or a point in time (`--deliver 1000`, `--deliver 1h` for an hour back). The remaining value, `last_per_subject`, starts with the newest message *for each subject* the consumer matches. That gives a latest-state view per subject, and it's what [Key-Value](/learn/key-value/.md) watches are built on. Don't confuse it with the `--last-per-subject` flag of [Reading messages directly](/learn/jetstream/get-direct.md). The two are different features that share a name: one is a consumer's start position, the other a one-shot read. **Ack policy** decides what the server needs before it considers a delivered message handled. `shipping` used `explicit`; the others are `none`, `all`, and `flow_control`, covered in [Ack responses and redelivery](/learn/jetstream/acknowledgment.md). **Replay policy** decides the pace of delivery. Every consumer in this chapter ran on the default, `instant`, which [Reading back the stream](/learn/jetstream/reading-back.md) introduced: messages arrive as fast as the reader takes them. The other value, `original`, spaces deliveries to match the gaps between the original timestamps. A long quiet stretch in the stream is replayed as a long quiet wait. **Priority policy** decides which of a consumer's clients gets served. The default, `none`, applies no steering: pulls are served as they come. `overflow`, `pinned_client`, and `prioritized` each steer differently, and they're covered on [Priority groups](/learn/jetstream/priority-groups.md). The consumer fields, including the start-position options the deliver policy uses, are in [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md). ## What you can change later Five of the nine policies are fixed at creation. The server enforces this on update, so settle these before you create anything you'll keep: | Stream policy | On a live stream | | ------------- | ---------------------------------------------------------------------------------------------------------------------- | | Retention | `limits` ↔ `interest` allowed, though the switch re-applies to messages already stored; to or from `workqueue` refused | | Discard | Can change | | Storage | Fixed at creation | | Compression | Can change; takes effect after a server or leader restart | | Persist mode | Fixed at creation | | Consumer policy | On a live consumer | | --------------- | ------------------------------------------------------------------------------------------ | | Deliver policy | Fixed at creation | | Ack policy | Fixed at creation | | Replay policy | Fixed at creation | | Priority policy | Can change; `nats consumer edit` has no flag for it, so pass a config file with `--config` | ## Pitfalls **Changing a fixed policy means recreating the object.** The server is explicit about it: editing a stream's storage type fails with `stream configuration update can not change storage type`, and updating a consumer's start position fails with `deliver policy can not be updated`. Recreating a stream means moving the data too. A [mirror](/learn/jetstream/mirrors-and-sources.md) can copy the messages across, but a mirror is read-only, and turning it into a publishable replacement takes steps of its own. Recreating a consumer loses the saved position: the new consumer starts wherever its deliver policy says, not where the old one stopped. Do settle the fixed policies before creating anything durable. Don't count on `nats stream edit` or `nats consumer edit` to fix them later. **Expecting `new` to skip the backlog on every restart.** A deliver policy applies once, when the consumer is created. A durable created with `new` starts with the messages that arrive after that moment, and it keeps a saved position like any other durable. When your client restarts and attaches to it, it resumes from that position, backlog included. Do use `new` to define where a consumer's history begins. Don't use it hoping each reconnect jumps to live traffic. **Two features answer to "last per subject".** The deliver policy value gives a consumer a standing latest-per-subject view; the Direct Get flag fetches the same shape once, with no consumer. Do check which feature a flag or value belongs to before copying it. Don't assume the phrase means the same thing in both places. ## Where you are `ORDERS` and its consumers are unchanged. You now have: * the nine policies mapped, most to the page where you used them * compression and persist mode placed, the two `ORDERS` never set * the five policies that are fixed at creation, and what the server says when you try to change one ## What's next That's the full policy map. [Where to go next](/learn/jetstream/where-next.md) recaps the model you built and points to the chapters that take it further. ## See also * [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md) — every stream field, its type, range, and default. * [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md) — every consumer field, its type, range, and default. --- # Priority groups The [worker pool](/learn/jetstream/worker-pool.md) shared work evenly. Every worker on the `shipping` consumer pulled, and the server delivered messages to whichever worker asked. Some workloads need a different split. You might want one client to handle all the work until it fails, or a far-away client to stay idle unless the near ones fall behind. Even work sharing can't do either of those. **Priority groups** let a pull consumer ask for those behaviors. They're designed in [ADR-42](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-42.md). This page covers all three policies the server offers. ## What a priority group is A priority group is a name on a pull consumer, plus a policy that decides how the server hands out messages for that name. You set two fields when you create the consumer: * **`PriorityGroups`**: the list of group names the consumer supports. Today a consumer uses exactly one group; naming more than one is accepted but only the first takes effect (see Pitfalls). * **`PriorityPolicy`**: the rule the server applies, one of `overflow`, `pinned_client`, or `prioritized`. Two of the three policies, overflow and pinned\_client, need the consumer to acknowledge its messages, so those examples use `--ack explicit`. They decide what to do from counts the server keeps per client: overflow looks at how many messages are waiting and unacknowledged, pinned\_client at which client is still pulling, and the server only keeps those counts when the consumer acks. Prioritized just sorts pulls by a number and needs no acks. Once a consumer has a policy, every pull must name its group. A pull that leaves the group out is rejected with `Bad Request - Priority Group missing`. The group on the pull and the group on the consumer must match. So a priority group is a name on the consumer, a policy that governs it, and pulls that join the group by name. The rest of this page covers the three policies and the problem each one solves. ## The overflow policy Two regions can both process orders. `us-east` is close to the stream and cheap to serve. `us-west` works too, but every message it pulls crosses the country, costs more, and arrives slower. You want `us-west` to stay idle unless `us-east` falls behind. The **overflow** policy does this. Workers in `us-east` pull with no threshold, so they always get messages. Workers in `us-west` pull with a `min_pending` threshold: the server answers their pull only when the consumer has at least that many messages waiting. Below the threshold their pull gets nothing, the same as if the stream were empty. **Message flow — Overflow priority policy (animated):** Overflow priority policy. A near worker (us-east) pulls with no threshold and always drains the ORDERS backlog; a far worker (us-west) pulls with a min\_pending threshold and is served only while the backlog sits above it. A burst pushes the backlog over the line so us-west takes the overflow, and once the backlog falls back under, us-west goes idle again. Create an overflow consumer on the `ORDERS` stream from a config file: ``` nats consumer add ORDERS dispatch --config overflow-consumer.json ``` where `overflow-consumer.json` sets the policy and its single group: ``` { "durable_name": "dispatch", "ack_policy": "explicit", "priority_policy": "overflow", "priority_groups": ["regions"] } ``` The intended shorthand is `nats consumer add ORDERS dispatch --overflow-groups regions --ack explicit --pull`, but through natscli v0.4.x that flag sets the policy without attaching the group, so the server rejects the create with error `10159`. Use `--config` until the fix ships. The pinned and prioritized creates later on this page use their flags normally. Confirm the consumer: ``` nats consumer info ORDERS dispatch ``` The configuration now carries the two priority fields: ``` Configuration: Pull Mode: true Ack Policy: Explicit Priority Policy: Overflow Priority Groups: regions ``` The threshold goes on the pull request, not on the consumer. A near-region worker pulls with no threshold. A far-region worker adds `min_pending`: deliver only when the consumer has backed up past that many waiting messages. `min_ack_pending` is a related threshold, counted against unacknowledged messages instead; meeting either one triggers delivery. The `nats consumer next` command issues a plain pull and has no flag for these thresholds, so the overflow pull below comes from a client library: #### CLI ``` #!/bin/bash # Overflow policy: a standby region pulls only when the consumer has # backed up past a min_pending threshold. # Create an overflow pull consumer on ORDERS. --overflow-groups sets the # policy to overflow and names the single group "regions". Overflow # requires explicit acks. nats consumer add ORDERS dispatch \ --overflow-groups regions \ --pull \ --ack explicit \ --defaults # Inspect it — the configuration now shows Priority Policy: overflow and # Priority Groups: [regions]. nats consumer info ORDERS dispatch # The min_pending threshold lives on the pull request, which natscli's # `nats consumer next` does not expose a flag for. A near-region worker # pulls plainly and always gets messages: nats consumer next ORDERS dispatch --count 5 ``` #### C ``` // Bind to the existing overflow consumer. Every pull on a // priority-group consumer must name its group. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "dispatch"; s = js_PullSubscribe(&sub, js, NULL, "dispatch", NULL, &so, &jerr); // A near-region worker pulls with no threshold, so it always gets // messages. if (s == NATS_OK) { jsFetchRequest req; natsMsgList list = {NULL, 0}; jsFetchRequest_Init(&req); req.Batch = 5; req.Expires = 2 * 1000000000LL; // 2 seconds, in nanoseconds req.Group = "regions"; s = natsSubscription_FetchRequest(&list, sub, &req); if (s == NATS_OK) { printf("near region got %d messages\n", list.Count); for (i = 0; i < list.Count; i++) natsMsg_Ack(list.Msgs[i], NULL); natsMsgList_Destroy(&list); } else if (s == NATS_TIMEOUT) s = NATS_OK; // nothing waiting right now } // A far-region worker adds MinPending: the server answers its pull // only when the consumer has backed up past that many waiting // messages. Below the threshold it gets nothing, the same as if the // stream were empty. if (s == NATS_OK) { jsFetchRequest req; natsMsgList list = {NULL, 0}; jsFetchRequest_Init(&req); req.Batch = 5; req.Expires = 2 * 1000000000LL; req.Group = "regions"; req.MinPending = 100; s = natsSubscription_FetchRequest(&list, sub, &req); if (s == NATS_OK) { printf("far region got %d overflow messages\n", list.Count); for (i = 0; i < list.Count; i++) natsMsg_Ack(list.Msgs[i], NULL); natsMsgList_Destroy(&list); } else if (s == NATS_TIMEOUT) { // Backlog below 100: the far region stays idle. printf("far region got nothing\n"); s = NATS_OK; } } ``` The near-region worker, pulling without a threshold, empties the backlog as fast as it processes. The far-region worker gets messages only when the backlog crosses its `min_pending` threshold, takes the overflow, and goes idle again once the near worker catches up. ## The pinned\_client policy The overflow policy spreads work under load. The **pinned\_client** policy sends all work to one client and keeps a standby ready to take over. Consider an order pipeline that must process messages strictly in arrival order. Two clients run so that one can take over if the other fails, but only one may work at a time, or the ordering breaks. You want one active client and one standby. The server picks one waiting pull and **pins** it. That client becomes the one that receives messages. Every other client's pull waits as a standby. If the pinned client stops pulling, because it crashed or went quiet longer than the pin timeout allows, the server pins a standby instead. **Message flow — Pinned-client priority policy (animated):** Pinned\_client priority policy. The server pins one worker and sends it every message while the others stand by. The pinned worker goes quiet; once PinnedTTL elapses the server pins a standby instead and stamps its messages with a new Nats-Pin-Id, and the old worker's next pull carrying the stale id comes back 423 — it clears the id and rejoins the standby pool. Create a pinned consumer: ``` nats consumer add ORDERS sequencer --pinned-groups ordered --pinned-ttl 90s --ack explicit --pull --defaults ``` Two flags do the work. `--pinned-groups ordered` sets the policy to `pinned_client` and names the group `ordered`. `--pinned-ttl 90s` sets how long the server waits for a pull from the pinned client before it gives up and pins someone else. The pin timeout must sit comfortably above the pull's `expires` value. The pinned client needs time to pull, get its batch or time out, process, then pull again, all before the timeout fires and costs it the pin. The server's default timeout is two minutes; keep `expires` under a minute and the whole cycle fits. The pinned client earns and keeps the pin like this: #### CLI ``` #!/bin/bash # Pinned_client policy: one active client gets all the work, with a # standby ready to take over. # Create a pinned consumer. --pinned-groups sets the policy to # pinned_client and names the group "ordered". --pinned-ttl is how long # the server waits for a pull from the pinned client before pinning a # standby instead. Pinned consumers require explicit acks. nats consumer add ORDERS sequencer \ --pinned-groups ordered \ --pinned-ttl 90s \ --pull \ --ack explicit \ --defaults # Inspect it — the configuration shows Priority Policy: pinned_client, # Priority Groups: [ordered], and Pinned TTL: 1m30s. The State block # shows which client (if any) currently holds the pin. nats consumer info ORDERS sequencer # The Nats-Pin-Id handshake happens on the pull request, which natscli's # `nats consumer next` does not drive. Pull a message to see delivery: nats consumer next ORDERS sequencer --count 1 # Force the server to pick a new pinned client: nats consumer unpin ORDERS sequencer ordered ``` #### C ``` // Pull from the pinned consumer with the async helper. The library // runs the pin handshake for you: it stores the Nats-Pin-Id header // from the first delivered message, sends it back on every later pull, // and clears it to rejoin the standby pool if the server unpins this // client. jsOptions jsOpts; jsSubOptions so; jsOptions_Init(&jsOpts); jsOpts.PullSubscribeAsync.Group = "ordered"; // the consumer's priority group jsOpts.PullSubscribeAsync.Timeout = 30000; // stop after 30 seconds jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "sequencer"; so.ManualAck = true; // pinned_client requires explicit acks s = js_PullSubscribeAsync(&sub, js, NULL, "sequencer", onMsg, NULL, &jsOpts, &so, &jerr); // If the server pins this client, every message flows here; if a // standby client runs the same code, its pulls wait until the pin // moves to it. while ((s == NATS_OK) && natsSubscription_IsValid(sub)) nats_Sleep(100); ``` The client and server agree on the pin through a header. When the server pins a client, the first message it delivers carries a `Nats-Pin-Id` header. The client reads that ID and sends it back on every later pull. The server keeps serving the client that presents the matching ID and parks the others. The client gives up the pin in one of two ways. If it falls silent past the timeout, the server pins a standby, and the old client's next pull, still carrying the now-stale ID, comes back with a `423` status. The client clears its stored ID and pulls without it, joining the standby pool again. The `423` rules and the pinned and unpinned advisories are in the [Consumer API reference](/reference/jetstream/api/consumer/info.md). The other way is an operator forcing a switch. `nats consumer unpin` clears the current pin and makes the server choose again: ``` nats consumer unpin ORDERS sequencer ordered -f ``` The command takes the stream, the consumer, and the group name; `-f` skips the confirmation prompt. It reports the client it dropped: ``` Unpinned client from Priority Group ORDERS > sequencer > ordered ``` To check who's pinned without forcing a change, read the consumer's state. `nats consumer info ORDERS sequencer` shows the live pin in its `State` block: ``` State: Priority Groups: ordered: pinned at 2026-06-02 12:14:22 ``` A group with no active client reads `No client`. To list every fully pinned consumer at once, run `nats consumer find ORDERS --pinned`. :::note Client support varies The pinned-client steps (storing `Nats-Pin-Id`, sending it back, handling the `423`) run in the Go, Java, JavaScript/TypeScript, and .NET clients today. Rust and Python let you set the configuration fields but don't yet run the client-side pinning loop. Check your client's reference before relying on it. ::: ## The prioritized policy The overflow policy makes a standby wait for a backlog to build. Sometimes you want the opposite: hand work to the next region the instant the closer one stops asking, with no threshold and no delay. `us-east` is close and cheap but resource-constrained. Whenever it has capacity it should take the work; when it doesn't, `us-west` should pick up right away, and `eu-west` only when neither US region is pulling. That's a hierarchy, not a threshold. The **prioritized** policy serves pulls in priority order. Each pull carries a `priority` from `0` to `9`, and the server hands messages to the lowest number present first; pulls at the same priority share round-robin. So `us-east` pulls at priority `0`, `us-west` at `1`, and `eu-west` at `2`: work goes to `us-east` whenever it's asking, falls to `us-west` the moment `us-east` isn't, and reaches `eu-west` only when neither is pulling. **Message flow — Prioritized priority policy (animated):** Prioritized priority policy. Three regions pull at priority 0, 1 and 2. The server serves the lowest priority that is currently pulling, with no delay: us-east (0) gets everything while it pulls; the moment it goes quiet the work falls to us-west (1), then eu-west (2); when us-east returns the work snaps straight back to priority 0. Create a prioritized consumer: ``` nats consumer add ORDERS dispatch --prioritized-groups regions --pull --defaults ``` `--prioritized-groups regions` sets the policy to `prioritized` and names the single group `regions`. Unlike the other two policies, prioritized keeps no per-client counts, so it doesn't require explicit acks. This reuses the `dispatch` name from the overflow example on purpose: overflow and prioritized are two ways to run the same regional-dispatch consumer, so you pick one. The server does let you switch a live consumer's policy, though `nats consumer edit` has no flag for it; you'd pass a full config with `--config`. The priority rides on the pull request, the same place overflow's thresholds go, so `nats consumer next` can't set it and the pull comes from a client library: #### CLI ``` #!/bin/bash # Prioritized policy: each pull carries a 0-9 priority and the server serves # the lowest number first, so nearer workers get first refusal and farther # ones pick up the instant the nearer ones go quiet. # Create a prioritized pull consumer on ORDERS. --prioritized-groups sets the # policy to prioritized and names the single group "regions". Prioritized # sorts pulls by number and tracks no per-client counts, so unlike overflow # and pinned_client it needs no explicit acks. nats consumer add ORDERS dispatch \ --prioritized-groups regions \ --pull \ --defaults # Inspect it — the configuration shows Priority Policy: prioritized and # Priority Groups: [regions]. nats consumer info ORDERS dispatch # The priority rides on the pull request, which natscli's `nats consumer next` # does not expose a flag for. A worker pulls plainly (priority 0) and is # served ahead of any higher-numbered pull: nats consumer next ORDERS dispatch --count 5 ``` #### C ``` // Bind to the existing prioritized consumer. Every pull names the // group and carries a priority from 0 to 9; the server serves the // lowest number present first. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "dispatch"; s = js_PullSubscribe(&sub, js, NULL, "dispatch", NULL, &so, &jerr); // us-east pulls at priority 0, so it's served ahead of any // higher-numbered pull. if (s == NATS_OK) { jsFetchRequest req; natsMsgList list = {NULL, 0}; jsFetchRequest_Init(&req); req.Batch = 5; req.Expires = 2 * 1000000000LL; // 2 seconds, in nanoseconds req.Group = "regions"; req.Priority = 0; s = natsSubscription_FetchRequest(&list, sub, &req); if (s == NATS_OK) { printf("us-east got %d messages\n", list.Count); for (i = 0; i < list.Count; i++) natsMsg_Ack(list.Msgs[i], NULL); natsMsgList_Destroy(&list); } else if (s == NATS_TIMEOUT) s = NATS_OK; // nothing waiting right now } // us-west pulls the same way at priority 1: it's served the moment no // priority-0 pull is waiting, with no threshold and no delay. if (s == NATS_OK) { jsFetchRequest req; natsMsgList list = {NULL, 0}; jsFetchRequest_Init(&req); req.Batch = 5; req.Expires = 2 * 1000000000LL; req.Group = "regions"; req.Priority = 1; s = natsSubscription_FetchRequest(&list, sub, &req); if (s == NATS_OK) { printf("us-west got %d messages\n", list.Count); for (i = 0; i < list.Count; i++) natsMsg_Ack(list.Msgs[i], NULL); natsMsgList_Destroy(&list); } else if (s == NATS_TIMEOUT) s = NATS_OK; } ``` This is the immediate counterpart to overflow. Overflow waits for the backlog to cross a threshold before a standby gets anything, which avoids churn but makes far regions wait. Prioritized shifts work the moment a higher-priority puller goes quiet, with no delay, at the cost of some flip-flop as work moves between regions. The full set of priority-group options, the `423` protocol, the `PriorityGroupState` fields, and the advisories live in [Reference → Consumer API](/reference/jetstream/api/consumer/info.md) and in [ADR-42](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-42.md). ## Pitfalls Priority groups have a small surface, but a few details are easy to miss. **One group per consumer.** A consumer uses exactly one priority group today. The `--overflow-groups` and `--pinned-groups` flags take a comma list, so passing two looks legal, and the server accepts it, but it uses only the first group and ignores the rest. Multiple groups per consumer is planned for a future server release. To split work by region or tier now, run separate consumers on the same stream, each with its own group. #### CLI ``` #!/bin/bash # A consumer acts on exactly one priority group. The --overflow-groups # and --pinned-groups flags accept a comma list, so it is easy to pass # two by accident. # NOT THIS: two group names. The server accepts the create without error, # but it uses only the first group (regions) and silently ignores the # rest. Multiple groups per consumer is reserved for a future release. nats consumer add ORDERS dispatch \ --overflow-groups regions,backup \ --pull \ --ack explicit \ --defaults # DO THIS: name a single group. To split work by region or tier, run # separate consumers, each with its own group, on the same stream. nats consumer add ORDERS dispatch \ --overflow-groups regions \ --pull \ --ack explicit \ --defaults ``` #### C ``` // A consumer acts on exactly one priority group, but PriorityGroups is // a list, so it's easy to pass two by accident. jsConsumerConfig cc; const char *twoGroups[] = {"regions", "backup"}; const char *oneGroup[] = {"regions"}; // NOT THIS: two group names. The server accepts the create without // error, but it uses only the first group (regions) and silently // ignores the rest. Multiple groups per consumer is reserved for a // future release. jsConsumerConfig_Init(&cc); cc.Durable = "dispatch"; cc.AckPolicy = js_AckExplicit; cc.PriorityPolicy = "overflow"; cc.PriorityGroups = twoGroups; cc.PriorityGroupsLen = 2; s = js_AddConsumer(&ci, js, "ORDERS", &cc, NULL, &jerr); if (s == NATS_OK) { jsConsumerInfo_Destroy(ci); ci = NULL; // The policy and groups are fixed at creation, so delete the // consumer before recreating it correctly. s = js_DeleteConsumer(js, "ORDERS", "dispatch", NULL, &jerr); } // DO THIS: name a single group. To split work by region or tier, run // separate consumers, each with its own group, on the same stream. if (s == NATS_OK) { cc.PriorityGroups = oneGroup; cc.PriorityGroupsLen = 1; s = js_AddConsumer(&ci, js, "ORDERS", &cc, NULL, &jerr); } if (s == NATS_OK) printf("dispatch uses group %s\n", ci->Config->PriorityGroups[0]); ``` **The pin does not give one client sole ownership.** The server can switch the pinned client while that client still believes it holds the pin, so a long-running handler can finish work the server already gave to someone else. A pull that carries a now-stale `Nats-Pin-Id` comes back with a `423`; clear the stored ID and pull without it to join the standby pool again. If the same message must never be processed twice, use explicit acks and handlers that are safe to run more than once, not the pin alone. **A quiet pinned client keeps the pin.** The pin only resets when the pinned client pulls again within `--pinned-ttl`. A client that holds the pin but stops pulling, for example because it's stuck on a slow handler, keeps every other client parked until the timeout fires. Keep each pull's `expires` comfortably under the pin timeout so the client always pulls again in time to renew. Which node in a cluster serves these pulls is covered in [clustering](/learn/clustering/.md). ## Where you are A priority group is always one group plus a policy that decides how the server hands out pulls: reach for `overflow` to spill work to a standby only under load, `pinned_client` for one active worker with a standby ready to take over, and `prioritized` for an immediate, lowest-first hierarchy across regions. ## What's next A consumer doesn't have to be running at all. The next page pauses a consumer, stopping it from delivering for a set window, and shows when that's the right tool. ## See also * [Reference → Consumer API](/reference/jetstream/api/consumer/info.md) — the priority-group config fields, `PriorityGroupState`, the `423` protocol, and the pinned/unpinned advisories. * [ADR-42](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-42.md) — the design of pull consumer priority groups. * [A pool of workers](/learn/jetstream/worker-pool.md) — the even work-sharing this page steers away from. --- # Publishing The `ORDERS` stream is empty. This page puts messages into it. Publishing into a stream works like a normal NATS publish, with one difference: the server sends back a confirmation that it stored the message. This page covers that confirmation, called a `PubAck`, and how to make a publish safe to retry. **Message flow — Publish and PubAck (animated):** Animated PubAck flow: a publisher sends a message to a subject; the ORDERS stream accepts it (it listens on orders.>), stores it and assigns a sequence number, then returns a PubAck with the stream name and sequence. * Publisher → orders.created (subject: orders.created) ## Publish from the CLI Start in the terminal. A plain `nats pub` is a core NATS publish: the client sends the message and the server returns nothing. When you publish into a stream you usually want a confirmation, so add the `--jetstream` flag: ``` nats pub --jetstream orders.created '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` The server appends the message to `ORDERS` (the stream captures `orders.>`) and replies on the same call: ``` Stored in Stream: ORDERS Sequence: 1 ``` That reply is the `PubAck`. It contains the name of the stream that stored the message and the sequence number the stream gave it. Sequence numbers start at `1` and only increase. The server gives each new message the next number and never reuses one. Publish two more messages so the stream has some content. Each one is acknowledged with the next sequence number, `2` and then `3`: ``` nats pub --jetstream orders.created '{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}' nats pub --jetstream orders.shipped '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:31Z"}' ``` You can check the whole stream with `nats stream info ORDERS`. It now reports three messages: ``` State: Messages: 3 Bytes: 404 B First Sequence: 1 @ 2026-05-22 10:14:22 Last Sequence: 3 @ 2026-05-22 10:14:31 Active Consumers: 0 Number of Subjects: 2 ``` You didn't have to run that command to know the writes succeeded. Each publish already reported its sequence number when it returned. ## Publish from a client library On the CLI, the `PubAck` is printed to the screen. In application code it is a return value: the same confirmation from the server, as an object your program can read. The `PubAck` is the main thing you work with when publishing to JetStream, and the next section covers what it contains. Here are the same three publishes from a client library: #### JavaScript/TypeScript ``` // Publish three orders and read each PubAck, which confirms the stream that // stored the message and the sequence it was assigned const orders = [ { subject: "orders.created", data: `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}`, }, { subject: "orders.created", data: `{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}`, }, { subject: "orders.shipped", data: `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:31Z"}`, }, ]; for (const order of orders) { const pa = await js.publish(order.subject, order.data); console.log(`Stored in ${pa.stream}, sequence ${pa.seq}`); } ``` #### Go ``` // Publish three order messages into the ORDERS stream. Each Publish blocks // until the server replies with a PubAck confirming where it was stored. orders := []struct { subject string payload string }{ {"orders.created", `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}`}, {"orders.created", `{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}`}, {"orders.shipped", `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:31Z"}`}, } for _, order := range orders { ack, err := js.Publish(ctx, order.subject, []byte(order.payload)) if err != nil { panic(err) } fmt.Printf("Stored in %s, sequence %d\n", ack.Stream, ack.Sequence) } ``` #### Python ``` # Publish three orders. Each publish waits for the server's acknowledgement, # which carries the stream name and the assigned sequence number. ack = await js.publish( "orders.created", b'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}', ) print(f"Stored in {ack.stream}, sequence {ack.seq}") ack = await js.publish( "orders.created", b'{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}', ) print(f"Stored in {ack.stream}, sequence {ack.seq}") ack = await js.publish( "orders.shipped", b'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:31Z"}', ) print(f"Stored in {ack.stream}, sequence {ack.seq}") ``` #### Java ``` // Publish three orders to the "ORDERS" stream and read each ack PublishAck ack1 = js.publish("orders.created", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}".getBytes(StandardCharsets.UTF_8)); System.out.printf("Stored in %s at sequence %d%n", ack1.getStream(), ack1.getSeqno()); PublishAck ack2 = js.publish("orders.created", "{\"order_id\":\"ord_2zr9\",\"customer\":\"globex\",\"total_cents\":7800,\"ts\":\"2026-05-22T10:14:25Z\"}".getBytes(StandardCharsets.UTF_8)); System.out.printf("Stored in %s at sequence %d%n", ack2.getStream(), ack2.getSeqno()); PublishAck ack3 = js.publish("orders.shipped", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:31Z\"}".getBytes(StandardCharsets.UTF_8)); System.out.printf("Stored in %s at sequence %d%n", ack3.getStream(), ack3.getSeqno()); ``` #### Rust ``` // Publish three orders into the ORDERS stream, reading each ack as it returns. let messages = [ ( "orders.created", r#"{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}"#, ), ( "orders.created", r#"{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}"#, ), ( "orders.shipped", r#"{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:31Z"}"#, ), ]; for (subject, payload) in messages { // The first await sends the message; the second waits for the server ack. let ack = js.publish(subject, payload.into()).await?.await?; println!("Stored in {} at sequence {}", ack.stream, ack.sequence); } ``` #### C#/.NET ``` // Publish each order and read the ack the stream returns var ack1 = await js.PublishAsync( subject: "orders.created", data: new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:22Z", CultureInfo.InvariantCulture))); output.WriteLine($"Stored in {ack1.Stream} at sequence {ack1.Seq}"); var ack2 = await js.PublishAsync( subject: "orders.created", data: new Order(OrderId: "ord_2zr9", Customer: "globex", TotalCents: 7800, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:25Z", CultureInfo.InvariantCulture))); output.WriteLine($"Stored in {ack2.Stream} at sequence {ack2.Seq}"); var ack3 = await js.PublishAsync( subject: "orders.shipped", data: new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:31Z", CultureInfo.InvariantCulture))); output.WriteLine($"Stored in {ack3.Stream} at sequence {ack3.Seq}"); ``` #### C ``` // Publish three order messages into the ORDERS stream. Each js_Publish // blocks until the server replies with an ack confirming where it was // stored. const char *subjects[] = {"orders.created", "orders.created", "orders.shipped"}; const char *payloads[] = { "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}", "{\"order_id\":\"ord_2zr9\",\"customer\":\"globex\",\"total_cents\":7800,\"ts\":\"2026-05-22T10:14:25Z\"}", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:31Z\"}"}; for (int i = 0; (i < 3) && (s == NATS_OK); i++) { jsPubAck *ack = NULL; s = js_Publish(&ack, js, subjects[i], payloads[i], (int) strlen(payloads[i]), NULL, &jerr); if (s == NATS_OK) { printf("Stored in %s, sequence %" PRIu64 "\n", ack->Stream, ack->Sequence); jsPubAck_Destroy(ack); } } ``` Two details show up in every one of these snippets: 1. You publish to a subject, not to a stream. The server finds the stream that captures the subject and stores the message there. Your code doesn't need to know which stream is bound to which subject. 2. The publish call returns a `PubAck` instead of nothing. That return value is how you know the message reached a stream. The next section reads it. ## What a PubAck contains A `PubAck` has three fields you use regularly: * **stream**: the stream that stored the message. Useful in tests and logs; in normal code you already know it. * **sequence**: the sequence number the stream gave the message. It's the number shown by `nats stream info`. If you save it next to your business record, you can replay the stream from that point later. * **duplicate**: `false` for a new message, `true` if the server recognized the message as a repeat. Duplicate detection is covered in [Avoiding duplicate writes](#avoiding-duplicate-writes) below. A `PubAck` can also include a few situational fields, such as a `domain` for multi-tenant or leaf-node setups. The full list is in [Reference](/reference/jetstream/api/stream/pub-ack.md). The three above are the ones day-to-day publishing code reads. The snippets in the previous section already return this object; reading its fields is all there is to it. The rule: the `PubAck` is your only proof that the message was stored. When a publish fails, what you know depends on how it failed. A subject that no stream captures fails immediately (a "no responders" error), and nothing was stored. But a network timeout means no confirmation, not no write: the server may have stored the message and the ack got lost on the way back. So handle a failed publish by retrying it or reporting the failure — and make the retry safe with the `Nats-Msg-Id` header covered in [Avoiding duplicate writes](#avoiding-duplicate-writes) below. ## Stored versus delivered A `PubAck` confirms that the server stored the message. It does not confirm that any consumer has received it. Storage and delivery are separate. In core NATS, a message is delivered at almost the same moment the publish completes, so the two are easy to treat as one thing. In JetStream they happen at different times: 1. The publisher publishes. The server stores the message and returns a `PubAck`. 2. Later (possibly minutes, hours, or days) a client reads the message through a consumer. 3. After processing it, the client acknowledges it. Only then is the message considered handled. The rest of this chapter covers steps 2 and 3. This page has done step 1. ## Avoiding duplicate writes A real publisher retries when a publish fails. But a failed publish doesn't mean the message wasn't stored: a timeout can fire after the server stored the message, with only the ack lost. A plain retry then stores the same message twice. To prevent that, tag the publish with a `Nats-Msg-Id` header. The server refuses to store the same ID twice within the stream's duplicate-tracking window (the two-minute setting from the previous page's config). A blocked duplicate is the `duplicate: true` case in the `PubAck`. On the CLI, set the header with `--header` and publish with `nats pub --jetstream` so the `PubAck` is shown: ``` nats pub --jetstream orders.created \ --header "Nats-Msg-Id:ord_8w2k-created" \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' ``` Run that command twice. The first call stores the message and prints its sequence number (`Stored in Stream: ORDERS Sequence: …`). The second call prints the same sequence number with `Duplicate: true`, and nothing new is stored. The same header from a client library: #### JavaScript/TypeScript ``` // Publish the same order twice with a stable msgID. Within the stream's // duplicate window the server stores it once: the second PubAck reports the // same sequence and `duplicate: true` const data = `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}`; const first = await js.publish("orders.created", data, { msgID: "ord_8w2k-created", }); console.log(`First: sequence ${first.seq}, duplicate ${first.duplicate}`); const second = await js.publish("orders.created", data, { msgID: "ord_8w2k-created", }); console.log(`Second: sequence ${second.seq}, duplicate ${second.duplicate}`); ``` #### Go ``` // Publish the same message twice with a message ID. The stream uses the ID // to detect duplicates within its dedupe window. first, err := js.Publish(ctx, "orders.created", []byte(payload), jetstream.WithMsgID("ord_8w2k-created")) if err != nil { panic(err) } fmt.Printf("First: sequence %d, duplicate %t\n", first.Sequence, first.Duplicate) // Re-publish with the same ID. The server recognizes it and reports the // original sequence instead of storing the message again. second, err := js.Publish(ctx, "orders.created", []byte(payload), jetstream.WithMsgID("ord_8w2k-created")) if err != nil { panic(err) } fmt.Printf("Second: sequence %d, duplicate %t\n", second.Sequence, second.Duplicate) ``` #### Python ``` # Set "Nats-Msg-Id" so the server can detect duplicates within the stream's # duplicate window. payload = b'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' headers = {"Nats-Msg-Id": "ord_8w2k-created"} # Publish the same message twice with the same id. first = await js.publish("orders.created", payload, headers=headers) print(f"First: sequence {first.seq}, duplicate {first.duplicate}") second = await js.publish("orders.created", payload, headers=headers) print(f"Second: sequence {second.seq}, duplicate {second.duplicate}") ``` #### Java ``` // Tag the message with a stable id so a retry is deduplicated PublishOptions options = PublishOptions.builder() .messageId("ord_8w2k-created") .build(); // First publish stores the message PublishAck first = js.publish("orders.created", body, options); System.out.printf("First: seq=%d duplicate=%b%n", first.getSeqno(), first.isDuplicate()); // Re-publishing the same id returns the original sequence, marked duplicate PublishAck second = js.publish("orders.created", body, options); System.out.printf("Second: seq=%d duplicate=%b%n", second.getSeqno(), second.isDuplicate()); ``` #### Rust ``` // Publish the same order twice, tagging both with the same message id. // The server stores the first and detects the second as a duplicate. let first = js .send_publish( "orders.created", PublishMessage::build() .payload(payload.into()) .message_id("ord_8w2k-created"), ) .await? .await?; println!( "First: sequence {}, duplicate {}", first.sequence, first.duplicate ); let second = js .send_publish( "orders.created", PublishMessage::build() .payload(payload.into()) .message_id("ord_8w2k-created"), ) .await? .await?; println!( "Second: sequence {}, duplicate {}", second.sequence, second.duplicate ); ``` #### C#/.NET ``` var order = new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:22Z", CultureInfo.InvariantCulture)); // Tag the message with a unique id. The stream uses it to detect duplicates. var opts = new NatsJSPubOpts { MsgId = "ord_8w2k-created" }; // First publish: the stream stores the message var ack1 = await js.PublishAsync(subject: "orders.created", data: order, opts: opts); output.WriteLine($"First: seq={ack1.Seq} duplicate={ack1.Duplicate}"); // Republish with the same id: the stream recognizes it and stores nothing new var ack2 = await js.PublishAsync(subject: "orders.created", data: order, opts: opts); output.WriteLine($"Second: seq={ack2.Seq} duplicate={ack2.Duplicate}"); ``` #### C ``` // Publish the same message twice with a message ID. The stream uses // the ID to detect duplicates within its dedupe window. jsPubAck *first = NULL; jsPubAck *second = NULL; jsPubOptions po; const char *payload = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; jsPubOptions_Init(&po); po.MsgId = "ord_8w2k-created"; s = js_Publish(&first, js, "orders.created", payload, (int) strlen(payload), &po, &jerr); if (s == NATS_OK) { printf("First: sequence %" PRIu64 ", duplicate %s\n", first->Sequence, first->Duplicate ? "true" : "false"); jsPubAck_Destroy(first); // Re-publish with the same ID. The server recognizes it and // reports the original sequence instead of storing the message // again. s = js_Publish(&second, js, "orders.created", payload, (int) strlen(payload), &po, &jerr); } if (s == NATS_OK) { printf("Second: sequence %" PRIu64 ", duplicate %s\n", second->Sequence, second->Duplicate ? "true" : "false"); jsPubAck_Destroy(second); } ``` Give every publish you might retry a stable `Nats-Msg-Id` that the producer can recompute, such as an order ID, a request ID, or a hash of the payload. The full set of publish headers is in [Reference → JetStream Headers](/reference/jetstream/api/headers.md). The tracking window is a stream setting, not a header: it's the `Duplicate Window` covered on the [stream page](/learn/jetstream/your-first-stream.md). This page uses only `Nats-Msg-Id`. ## What we've skipped A few things this page leaves out. The other publishing modes get their own page later in the chapter; the rest is in Reference: * **Async publish** — fire many publishes and collect the `PubAcks` later, for higher throughput. See [Advanced publishing](/learn/jetstream/advanced-publishing.md). * **Atomic batch publish** — store a group of messages all-or-nothing (server 2.12+). See [Advanced publishing](/learn/jetstream/advanced-publishing.md). * **Fast-ingest batch publish** — flow-controlled, high-throughput ingest without atomicity (server 2.14+). See [Advanced publishing](/learn/jetstream/advanced-publishing.md). * **Expected-stream and expected-sequence headers** — fail a publish unless the stream is in a specific state, for optimistic concurrency. See [Reference → JetStream Headers](/reference/jetstream/api/headers.md). ## Pitfalls Three mistakes are common on a first publish into a stream. Each follows from something above. **Ignoring the `PubAck`.** A publish returns a `PubAck`, and code that discards it can't tell a stored message from a lost one. This is easy to miss on the CLI: a plain `nats pub` is a core publish, so it prints `Published N bytes` whether or not a stream captured the subject. That line doesn't prove the message was stored. Read the `PubAck` instead: in code, check the return value; on the CLI, publish with `nats pub --jetstream`, which reports the stream and sequence number. #### CLI ``` #!/bin/bash # Plain `nats pub` is a core NATS publish: it reports "Published" whether # or not a stream stored the message. Publish to a subject nothing # captures and it still looks fine: nats pub invoices.created "test" # 17:31:21 Published 4 bytes to "invoices.created" # The same publish with --jetstream reads the PubAck -- and now the miss # is visible, because no stream captured the subject: nats pub --jetstream invoices.created "test" # 17:31:21 Published 4 bytes to "invoices.created" # nats: error: nats: no responders available for request # On a captured subject, the PubAck confirms the write with the stream # name and the assigned sequence: nats pub --jetstream orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # 17:31:21 Published 91 bytes to "orders.created" # 17:31:21 Stored in Stream: ORDERS Sequence: 4 ``` #### JavaScript/TypeScript ``` // `publish` resolves only once the server has persisted the message. Awaiting // the PubAck and reading its stream and sequence confirms the order is stored const data = `{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}`; const pa = await js.publish("orders.created", data); console.log(`Stored in ${pa.stream} at sequence ${pa.seq}`); ``` #### Go ``` // Publish one order and confirm it was stored. A failed publish returns an // error rather than losing the message silently, so check err first. ack, err := js.Publish(ctx, "orders.created", []byte(payload)) if err != nil { panic(err) } // The returned PubAck is proof the message is on disk in the stream. fmt.Printf("Stored in %s at sequence %d\n", ack.Stream, ack.Sequence) ``` #### Python ``` # Publish one order. The publish only resolves once the server has stored # the message, so reading the acknowledgement confirms it is durable. ack = await js.publish( "orders.created", b'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}', ) # The stream name and sequence number prove the message is stored. print(f"Confirmed stored in {ack.stream} at sequence {ack.seq}") ``` #### Java ``` // Publish one order; the returned ack confirms it was stored PublishAck ack = js.publish("orders.created", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}".getBytes(StandardCharsets.UTF_8)); // A non-null ack means the server persisted the message System.out.printf("Confirmed stored in %s at sequence %d%n", ack.getStream(), ack.getSeqno()); ``` #### Rust ``` // Publish, then await the ack. The `?` turns a failed store into an error // instead of letting the program continue as if the message was saved. let ack = js.publish("orders.created", payload.into()).await?.await?; // Reaching this line means the server persisted the message. println!( "Confirmed stored in {} at sequence {}", ack.stream, ack.sequence ); ``` #### C#/.NET ``` // Publish an order and read the ack var ack = await js.PublishAsync( subject: "orders.created", data: new Order(OrderId: "ord_8w2k", Customer: "acme-co", TotalCents: 4200, Timestamp: DateTimeOffset.Parse("2026-05-22T10:14:22Z", CultureInfo.InvariantCulture))); // Throw if the stream rejected the message; otherwise the ack confirms storage ack.EnsureSuccess(); output.WriteLine($"Confirmed: stored in {ack.Stream} at sequence {ack.Seq}"); ``` #### C ``` // Publish one order and confirm it was stored. A failed publish // returns an error status rather than losing the message silently, // so check s first. jsPubAck *ack = NULL; const char *payload = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\",\"total_cents\":4200,\"ts\":\"2026-05-22T10:14:22Z\"}"; s = js_Publish(&ack, js, "orders.created", payload, (int) strlen(payload), NULL, &jerr); if (s == NATS_OK) { // The returned ack is proof the message is on disk in the stream. printf("Stored in %s at sequence %" PRIu64 "\n", ack->Stream, ack->Sequence); jsPubAck_Destroy(ack); } ``` The plain publish reports `Published` even on a subject no stream captures. The `--jetstream` version surfaces the miss as a "no responders" error, and confirms a real write with `Stored in Stream … Sequence …`. **Retrying without a `Nats-Msg-Id`.** A publisher that retries after a timeout (which any well-built publisher does) can store the message twice: the timed-out publish may have been stored with only its ack lost, and the retry adds a second copy unless it carries the same `Nats-Msg-Id`. The duplicate-tracking window is two minutes by default, so a retry that arrives after that also stores a second copy. Don't retry a bare publish. Give every retryable publish a stable `Nats-Msg-Id`, as shown in [Avoiding duplicate writes](#avoiding-duplicate-writes) above. **Treating "published" as "delivered".** A `PubAck` means the stream stored the message, not that a consumer processed it. Code that marks an order shipped as soon as the `PubAck` returns is acting on a write that no shipping logic has seen. Keep business outcomes separate from the publish. The delivery-and-ack half of the story is on the [next page](/learn/jetstream/reading-back.md) and in [delivery and acknowledgment](/learn/jetstream/delivery-and-acknowledgment.md). ## Where you are The `ORDERS` stream now holds three messages, each confirmed by a `PubAck`, and a repeated publish no longer stores a second copy. No consumer has read these messages yet. The next page, [Reading back the stream](/learn/jetstream/reading-back.md), reads them back. ## See also * [Reference → JetStream Headers](/reference/jetstream/api/headers.md) — every publish-side header, including dedup, expected state, and batch. * [Reference → Publish Acknowledgement](/reference/jetstream/api/stream/pub-ack.md) — the exact fields returned by a publish. --- # Pull consumers in depth The `shipping` consumer delivers the next message to a worker when the worker asks. That ask is a **pull**. The page that created the consumer treated a pull as a single "give me one message." A worker often wants more than one message at a time. It might want a handful, process them, and come back. It might want a continuous flow where new messages arrive as soon as they reach the stream. This page covers both patterns and the fields that bound them. The `shipping` consumer doesn't change. It stays a pull consumer with explicit ack. What changes is how your code drives it. ## Two ways to pull There are two pull patterns, and every client library names them the same way. **Fetch** asks for a batch of up to *N* messages. The call returns when the batch is full or when a timeout expires, whichever comes first. You get a finite set of messages, you process them, and the call is done. To keep going, you fetch again. **Consume** sets up a continuous flow. You pass it a function. The library sends pull requests in the background and adds new ones as messages arrive. It calls your function for each message. It runs until you stop it instead of returning after a batch. Use fetch when your code wants control over each round, such as a cron job that drains what's queued or a request handler that takes a few messages. Use consume when you want a long-running worker that processes messages as fast as they arrive. Most services use consume. ## Fetch a batch A fetch names a batch size and a timeout. Here's a worker asking for up to ten messages and waiting up to two seconds for them: #### CLI ``` #!/bin/bash # Fetch a batch of up to 10 messages from the shipping pull consumer, # waiting up to 2 seconds for them. nats consumer next is a single # fetch; --count is the batch size, --wait is the timeout. nats consumer next ORDERS shipping --count 10 --wait 2s ``` #### JavaScript/TypeScript ``` // Bind to the durable "shipping" consumer. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); // Fetch a batch of up to 10 orders, waiting up to 2 seconds for them. The call // returns when the batch is full or the wait elapses, whichever comes first. // Process and ack each, then fetch again to keep going. const msgs = await c.fetch({ max_messages: 10, expires: 2000 }); for await (const m of msgs) { console.log(`shipping ${m.string()}`); m.ack(); } ``` #### Go ``` // Bind to the durable "shipping" consumer. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Fetch a batch of up to 10 orders, waiting up to 2 seconds for them. The // call returns when the batch is full or the wait elapses, whichever comes // first. Process and ack each one, then fetch again to keep going. msgs, err := cons.Fetch(10, jetstream.FetchMaxWait(2*time.Second)) if err != nil { panic(err) } for msg := range msgs.Messages() { fmt.Printf("shipping %s\n", string(msg.Data())) msg.Ack() } if msgs.Error() != nil { panic(msgs.Error()) } ``` #### Python ``` # Bind to the durable "shipping" consumer. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") # Fetch a batch of up to 10 orders, waiting up to 2 seconds for them. The # call returns the messages it has when the batch is full or the timeout # passes. Process and ack each, then fetch again to keep going. msgs = await psub.fetch(batch=10, timeout=2) for msg in msgs: print(f"shipping {msg.data.decode()}") await msg.ack() ``` #### Java ``` // Fetch a batch of up to 10 orders, waiting up to 2 seconds for // them. The fetch ends when the batch is full or the wait elapses, // whichever comes first. Process and ack each, then fetch again. try (FetchConsumer fc = cc.fetch( FetchConsumeOptions.builder().maxMessages(10).expiresIn(2000).build())) { Message m; while ((m = fc.nextMessage()) != null) { System.out.println("shipping " + new String(m.getData(), StandardCharsets.UTF_8)); m.ack(); } } ``` #### Rust ``` // Bind to the durable "shipping" consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // Fetch a batch of up to 10 orders, waiting up to 2 seconds for them. The // call returns when the batch is full or the wait elapses, whichever comes // first. Process and ack each, then fetch again to keep going. let mut messages = consumer .fetch() .max_messages(10) .expires(Duration::from_secs(2)) .messages() .await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("shipping {}", std::str::from_utf8(&msg.payload)?); msg.ack().await?; } ``` #### C#/.NET ``` // Bind to the durable "shipping" consumer. var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // Fetch a batch of up to 10 orders, waiting up to 2 seconds for them. The // call returns the messages it has when the batch is full or the wait // elapses. Process and ack each, then fetch again to keep going. await foreach (var msg in consumer.FetchAsync( opts: new NatsJSFetchOpts { MaxMsgs = 10, Expires = TimeSpan.FromSeconds(2) })) { output.WriteLine($"shipping {msg.Data}"); await msg.AckAsync(); shipped++; } ``` #### C ``` // Bind to the durable "shipping" consumer. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); if (s == NATS_OK) { // Fetch a batch of up to 10 orders, waiting up to 2 seconds for // them. The call returns when the batch is full or the wait // elapses, whichever comes first. Process and ack each one, then // fetch again to keep going. natsMsgList list = {NULL, 0}; s = natsSubscription_Fetch(&list, sub, 10, 2000, &jerr); if ((s == NATS_OK) || (s == NATS_TIMEOUT)) { for (int i = 0; i < list.Count; i++) { printf("shipping %s\n", natsMsg_GetData(list.Msgs[i])); natsMsg_Ack(list.Msgs[i], NULL); } natsMsgList_Destroy(&list); s = NATS_OK; } } ``` Two outcomes are normal. If ten messages are queued, the call returns all ten immediately. The worker processes and acks them, then fetches again. If only three messages are queued, the call returns those three and then waits up to two seconds for a fourth. When the two seconds pass, it returns the three it has. The timeout is the ceiling, so a fetch always returns within it. The CLI has no single batch fetch. `nats consumer next --count N` retrieves N messages by issuing N single-message pulls in a row, each bounded by `--timeout`, so it approximates a fetch loop rather than one batch request: #### CLI ``` #!/bin/bash # A CLI fetch. --count retrieves up to 5 messages by issuing 5 single # pulls in a row, not one batch request. Run this again to walk the # stream a batch at a time. nats consumer next ORDERS shipping --count 5 ``` Run it twice and you walk the stream a batch at a time. The consumer's cursor advances as messages are acked, the same way it did one message at a time on the consumer page. ## Consume a continuous flow A long-running worker shouldn't loop on fetch by hand. The consume pattern does the looping for you. It keeps pull requests open so a new message is delivered as soon as it's stored in the stream: #### CLI ``` #!/bin/bash # Consume a continuous flow from the CLI. nats consumer next exits after # --timeout of quiet, so wrap it in a loop to keep going as new messages # land. --ack acknowledges each message as it is received. Ctrl-C stops # the loop. while true; do nats consumer next ORDERS shipping --count 100 --ack --timeout 5s || sleep 1 done ``` #### JavaScript/TypeScript ``` // Bind to the durable "shipping" consumer. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); // consume() sets up a continuous flow: the library keeps pull requests open and // yields each order as soon as it lands in the stream. It runs until you stop // it, no fetch loop to write by hand. const messages = await c.consume(); for await (const m of messages) { console.log(`shipping ${m.string()}`); m.ack(); } ``` #### Go ``` // Bind to the durable "shipping" consumer. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Consume sets up a continuous flow: the library keeps pull requests open // in the background and runs the handler for each order as soon as it lands // in the stream. No fetch loop to write by hand; it runs until you stop it. consCtx, err := cons.Consume(func(msg jetstream.Msg) { fmt.Printf("shipping %s\n", string(msg.Data())) msg.Ack() }) if err != nil { panic(err) } defer consCtx.Stop() // Keep consuming until interrupted (Ctrl-C). sig := make(chan os.Signal, 1) signal.Notify(sig, os.Interrupt) <-sig ``` #### Python ``` # Bind to the durable "shipping" consumer. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") # nats.py drives a pull consumer with fetch, so a continuous flow is a fetch # loop: pull a batch, process it, come back. fetch times out when nothing is # waiting; keep looping so new orders are picked up as soon as they land. while True: try: msgs = await psub.fetch(batch=10, timeout=5) except nats.errors.TimeoutError: continue for msg in msgs: print(f"shipping {msg.data.decode()}") await msg.ack() ``` #### Java ``` // consume() sets up a continuous flow: the library keeps pull // requests open and runs the handler for each order as soon as it // lands in the stream. It runs until you stop it. try (MessageConsumer mc = cc.consume(msg -> { System.out.println("shipping " + new String(msg.getData(), StandardCharsets.UTF_8)); msg.ack(); })) { // Keep consuming until the process is stopped. Thread.sleep(Long.MAX_VALUE); } ``` #### Rust ``` // Bind to the durable "shipping" consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // messages() sets up a continuous flow: the library keeps pull requests // open and yields each order as soon as it lands in the stream. It runs // until you stop it, no fetch loop to write by hand. let mut messages = consumer.messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("shipping {}", std::str::from_utf8(&msg.payload)?); msg.ack().await?; } ``` #### C#/.NET ``` // Bind to the durable "shipping" consumer. var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // ConsumeAsync sets up a continuous flow: it keeps pull requests open and // yields each order as soon as it lands in the stream. It runs until you // stop it, no fetch loop to write by hand. await foreach (var msg in consumer.ConsumeAsync()) { output.WriteLine($"shipping {msg.Data}"); await msg.AckAsync(); // A real consumer runs forever; stop once the backlog is clear so the // example returns. if (++shipped == 2) { break; } } ``` #### C ``` // The handler runs for each order as soon as it lands in the stream. static void onOrder(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("shipping %s\n", natsMsg_GetData(msg)); natsMsg_Ack(msg, NULL); natsMsg_Destroy(msg); } // Bind to the durable "shipping" consumer. The library keeps pull // requests open in the background and dispatches messages to the // handler. No fetch loop to write by hand; it runs until you stop it. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; so.ManualAck = true; s = js_PullSubscribeAsync(&sub, js, NULL, NULL, onOrder, NULL, NULL, &so, &jerr); ``` Your function runs once per message and acks on success. The library handles the pull requests, sends new ones as the old ones empty, and keeps going until you stop it. Most order-processing workers use this pattern. ## The two fields that bound a pull Both patterns issue the same underlying pull request, and two fields on that request decide how much a single pull returns: * **batch**: the maximum number of messages this pull may return. A bigger batch means fewer round trips and higher throughput. A smaller batch means lower latency per message and less work lost if the worker dies mid-batch. * **expires**: how long the server holds the pull open waiting for messages before it returns what it has. This is the timeout from the fetch above. It bounds latency on a quiet stream. Client libraries set defaults for both, and the consume pattern keeps a batch and an expiry in flight for you, so a plain consume loop behaves well without tuning. For the full set of pull request fields, see [Reference → Get next message](/reference/jetstream/api/consumer/get-next.md). This page uses only `batch` and `expires`. ## Pitfalls A couple of defaults trip people up once `shipping` carries real order traffic. **An empty fetch is normal.** When no orders are queued, a fetch returns nothing once `expires` elapses. The server replies with a `408 Request Timeout` status (a no-wait fetch with no messages gets `404 No Messages` instead), and every client reports that as an empty batch (the CLI exits non-zero). A worker that treats an empty fetch as a failure fails on a quiet stream. An empty result means nothing is available right now, so keep looping: wait and fetch again. #### CLI ``` #!/bin/bash # A fetch on a drained consumer returns nothing. nats consumer next exits # non-zero when the pull times out with no messages waiting. Treat that as # "nothing right now," not a failure: sleep and fetch again. if nats consumer next ORDERS shipping --count 10 --timeout 2s; then echo "processed a batch" else echo "no orders waiting, will retry" sleep 1 fi ``` #### JavaScript/TypeScript ``` // Bind to the durable "shipping" consumer. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); // A fetch on a drained consumer ends empty once the wait elapses, not with an // error. Treat "nothing right now" as normal: if no orders came back, wait and // fetch again instead of failing. const msgs = await c.fetch({ max_messages: 10, expires: 2000 }); let count = 0; for await (const m of msgs) { console.log(`shipping ${m.string()}`); m.ack(); count++; } if (count === 0) { console.log("no orders waiting, will retry"); } ``` #### Go ``` // Bind to the durable "shipping" consumer. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // A fetch on a drained consumer returns an empty batch once the wait // elapses, not an error. Treat "nothing right now" as normal: if no orders // came back, wait and fetch again instead of failing. msgs, err := cons.Fetch(10, jetstream.FetchMaxWait(2*time.Second)) if err != nil { panic(err) } count := 0 for msg := range msgs.Messages() { fmt.Printf("shipping %s\n", string(msg.Data())) msg.Ack() count++ } if msgs.Error() != nil { panic(msgs.Error()) } if count == 0 { fmt.Println("no orders waiting, will retry") time.Sleep(time.Second) } ``` #### Python ``` # Bind to the durable "shipping" consumer. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") # On a drained consumer, fetch raises TimeoutError once the timeout passes. # Treat that as "nothing right now," not a failure: catch it, wait, and # fetch again instead of crashing. try: msgs = await psub.fetch(batch=10, timeout=2) except nats.errors.TimeoutError: print("no orders waiting, will retry") msgs = [] for msg in msgs: print(f"shipping {msg.data.decode()}") await msg.ack() ``` #### Java ``` // On a drained consumer the fetch ends with no messages once the // expiry passes, and nextMessage() returns null right away. Treat // that as "nothing right now," not a failure: note it and fetch // again instead of erroring. try (FetchConsumer fc = cc.fetch( FetchConsumeOptions.builder().maxMessages(10).expiresIn(2000).build())) { Message m = fc.nextMessage(); if (m == null) { System.out.println("no orders waiting, will retry"); } while (m != null) { System.out.println("shipping " + new String(m.getData(), StandardCharsets.UTF_8)); m.ack(); m = fc.nextMessage(); } } ``` #### Rust ``` // Bind to the durable "shipping" consumer. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // A fetch on a drained consumer ends empty once the wait elapses, not with // an error. Treat "nothing right now" as normal: if no orders came back, // wait and fetch again instead of failing. let mut messages = consumer .fetch() .max_messages(10) .expires(Duration::from_secs(2)) .messages() .await?; let mut count = 0; while let Some(msg) = messages.next().await { let msg = msg?; println!("shipping {}", std::str::from_utf8(&msg.payload)?); msg.ack().await?; count += 1; } if count == 0 { println!("no orders waiting, will retry"); } ``` #### C#/.NET ``` // Bind to the durable "shipping" consumer. var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // On a drained consumer the fetch ends with no messages once the expiry // passes, not with an error. Treat "nothing right now" as normal: if no // orders came back, wait and fetch again instead of failing. await foreach (var msg in consumer.FetchAsync( opts: new NatsJSFetchOpts { MaxMsgs = 10, Expires = TimeSpan.FromSeconds(2) })) { output.WriteLine($"shipping {msg.Data}"); await msg.AckAsync(); shipped++; } if (shipped == 0) { output.WriteLine("no orders waiting, will retry"); } ``` #### C ``` // Bind to the durable "shipping" consumer. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); if (s == NATS_OK) { // A fetch on a drained consumer comes back empty with // NATS_TIMEOUT once the wait elapses. Treat "nothing right now" // as normal: if no orders came back, wait and fetch again // instead of failing. natsMsgList list = {NULL, 0}; s = natsSubscription_Fetch(&list, sub, 10, 2000, &jerr); if ((s == NATS_OK) || (s == NATS_TIMEOUT)) { for (int i = 0; i < list.Count; i++) { printf("shipping %s\n", natsMsg_GetData(list.Msgs[i])); natsMsg_Ack(list.Msgs[i], NULL); } if (list.Count == 0) { printf("no orders waiting, will retry\n"); nats_Sleep(1000); } natsMsgList_Destroy(&list); s = NATS_OK; } } ``` **A raw fetch with no expiry can stall.** A pull request with `expires` set to zero never times out: the server holds it until the batch fills. Client libraries protect you from that with a default of about 30 seconds, so a fetch from client code returns control on a quiet stream even when you don't set `expires`. Set it yourself when you want a specific ceiling instead of relying on the default. The CLI bounds each pull with `--timeout`. **`MaxAckPending` set too low limits throughput.** This is the limit on un-acked messages the consumer hands out before it waits for acks. If you set it well below your batch size (a limit of ten against a batch of 100), the server delivers ten orders, then stops until your worker acks, no matter how large a batch you ask for. Keep it at or above your batch size. The default is 1000; lower it only when you know the in-flight count you want. The worker pool shares this single limit across every worker, so it matters even more there: see [the worker pool page](/learn/jetstream/worker-pool.md). **A batch set too large uses more memory than expected.** `batch` counts messages, not bytes, so a large batch against large orders can pull more into memory in one round than you expect. Most clients let you bound a pull by total size instead — a `max_bytes` option on fetch or consume — so you can cap memory directly; whichever limit is hit first ends the pull. ## Where you are You still have one stream, `ORDERS`, and one pull consumer, `shipping`. What changed is how you drive it. You can fetch a bounded batch when your code wants each round, or consume a continuous flow when you want a long-running worker. In both cases you bound a single pull with `batch` and `expires`. ## What's next The next page puts several workers on the `shipping` consumer at once and shows the server splitting the stream between them: a pool of workers sharing one cursor. That's also where `MaxAckPending`, the limit on un-acked messages across the whole consumer, starts to matter, since the pool shares one limit between every worker. ## See also * [Reference → Get next message](/reference/jetstream/api/consumer/get-next.md) — every field of a pull request, including `no_wait` and the `min_pending` controls this page left out. * [The worker pool page](/learn/jetstream/worker-pool.md) — sharing one pull consumer across many workers. --- # Reading back the stream The `ORDERS` stream holds the orders you published on the previous page. So far you've only seen them through `nats stream info`, which counts the messages but doesn't show their contents. This page reads them back through Acme's **billing** service. Billing acts on every order — it charges the customer when one is created, captures the payment when it ships, refunds a cancellation — so when it first comes up, or restarts after downtime, it has to work through the whole stream from the first order and miss none. That's a durable consumer reading the log from sequence 1, which is what this page builds. First, the three pieces that move a message from a publisher to a reader. You might have run the publish commands once, or a few times while experimenting, so the stream could hold three messages or thirty. None of the steps below assume a number: they read back whatever is there. ## How a producer, a stream, and a consumer fit together Three pieces move a message through JetStream: * A **producer** publishes messages to a subject, reads the `PubAck` and retries on failure. * A **stream** stores the messages, in order. As it stores each one it assigns a **stream sequence**, the message's position in the store. * A **consumer** is a server-side cursor on a stream. A **client** connects to it to receive messages; the consumer tracks how far that client has progressed and hands out the next one. Each consumer keeps its own **consumer sequence**, a counter the server bumps on every delivery — redeliveries included — so it counts deliveries, not distinct messages. **Message flow — Producer, stream, and consumer (animated):** Animated JetStream pipeline: a producer publishes into the ORDERS stream, where each message gets a fixed stream sequence; one consumer then reads the stored messages in order, advancing its own consumer sequence, and hands each to a client. The stream sequence is the message's position in the log; the consumer sequence is how many messages this consumer has read. A stream can feed many consumers at once, each reading at its own pace and each with its own position. This page uses one consumer; later pages add more. ## Streams keep messages In core NATS, a message published to a subject exists only as long as it takes to reach whoever is subscribed right now. The [Why a stream](/learn/jetstream/your-first-stream.md#why-a-stream) page covers why that's too short for orders. A stream keeps its messages. They're durable records with fixed stream sequences. They don't disappear when you read them, and they don't move. Reading message 1 leaves message 1 where it was, available to the next reader. You can re-read a stream whenever you want: a new consumer created a month from now starts at the same sequence 1 you're about to read today. ## A consumer is a server-side object To read a stream you create a consumer. A consumer isn't part of your application; it's an object on the server, sitting next to the stream. Your application is the **client**: it connects to the consumer to receive messages. The stored messages and the read position both stay on the server, so the client can disconnect and reconnect without losing its place or tracking any sequence numbers itself. **Message flow — A consumer is server-side (animated):** Architecture diagram: the NATS server contains two entities, the ORDERS stream (the stored messages) and the billing consumer (a server-side cursor over the stream). A separate client application connects from outside the server: it pulls messages from the consumer and receives them. The stored messages and the read position both live on the server, not in the client. Create a consumer named `billing`: #### CLI ``` #!/bin/bash # Create a named, durable consumer on the ORDERS stream. # A durable consumer has a name you choose. The server keeps its position on # disk under that name, so a reader resumes where it left off after a # disconnect or a server restart instead of starting over. # --deliver all start at the first message in the stream (sequence 1) # --pull the reader asks for messages when it's ready # --ack explicit acknowledge each message; the position advances only once # a message is acked (the default for a pull consumer) nats consumer add ORDERS billing \ --deliver all \ --pull \ --ack explicit \ --defaults ``` #### JavaScript/TypeScript ``` // Create a durable consumer on the ORDERS stream. A durable keeps its position // under a fixed name, so a reader can come back later and pick up where it left // off. With explicit acks the server only advances that position once a reader // acks each message. add() is idempotent: calling it again with the same config // is a no-op. const jsm = await jetstreamManager(nc); await jsm.consumers.add("ORDERS", { durable_name: "billing", ack_policy: AckPolicy.Explicit, deliver_policy: DeliverPolicy.All, }); console.log("Created durable consumer: billing"); ``` #### Go ``` // Create a durable consumer on the ORDERS stream. The durable name lets a // reader bind to the same consumer later and pick up where it left off. cons, err := js.CreateOrUpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{ Durable: "billing", DeliverPolicy: jetstream.DeliverAllPolicy, AckPolicy: jetstream.AckExplicitPolicy, }) if err != nil { panic(err) } // Confirm the consumer is ready. fmt.Printf("Created consumer: %s\n", cons.CachedInfo().Name) ``` #### Python ``` # Create a durable consumer that reads the whole stream from the start. # add_consumer is idempotent: calling it again with the same config is a no-op. await js.add_consumer( "ORDERS", ConsumerConfig( durable_name="billing", ack_policy=AckPolicy.EXPLICIT, deliver_policy=DeliverPolicy.ALL, ), ) print("Created durable consumer billing on stream ORDERS") ``` #### Java ``` // Create a durable consumer that delivers every stored message. // AckPolicy.Explicit means each message must be acknowledged. ConsumerContext cc = sc.createOrUpdateConsumer( ConsumerConfiguration.builder() .durable("billing") .deliverPolicy(DeliverPolicy.All) .ackPolicy(AckPolicy.Explicit) .build()); System.out.println("Created durable consumer: " + cc.getConsumerName()); ``` #### Rust ``` // Create a durable pull consumer that starts at the first stored message // and acknowledges each message explicitly, so the server tracks progress. let stream = js.get_stream("ORDERS").await?; let consumer = stream .create_consumer(pull::Config { durable_name: Some("billing".to_string()), deliver_policy: DeliverPolicy::All, ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; println!("Created durable consumer: {}", consumer.cached_info().name); ``` #### C#/.NET ``` // Create a durable consumer that delivers every stored message from the start var consumer = await js.CreateOrUpdateConsumerAsync("ORDERS", new ConsumerConfig("billing") { AckPolicy = ConsumerConfigAckPolicy.Explicit, DeliverPolicy = ConsumerConfigDeliverPolicy.All, }); output.WriteLine($"Created durable consumer {consumer.Info.Config.Name} on stream ORDERS"); ``` #### C ``` // Create a durable consumer on the ORDERS stream. The durable name // lets a reader bind to the same consumer later and pick up where it // left off. jsConsumerInfo *ci = NULL; jsConsumerConfig cc; jsConsumerConfig_Init(&cc); cc.Durable = "billing"; cc.DeliverPolicy = js_DeliverAll; cc.AckPolicy = js_AckExplicit; s = js_AddConsumer(&ci, js, "ORDERS", &cc, NULL, &jerr); if (s == NATS_OK) { // Confirm the consumer is ready. printf("Created consumer: %s\n", ci->Name); jsConsumerInfo_Destroy(ci); } ``` Three settings define how it reads: * **Deliver all** starts the consumer at the first message in the stream, sequence 1, so it reads the whole log. * **Pull** means the client asks the server for messages when it's ready, rather than having the server push them on its own. Pull is the default for new consumers; an older push model exists, but the consumers you create in this chapter are pull. * **Ack explicit** means the client acknowledges each message it handles, and the consumer's position advances only as acks arrive. It's the default for a pull consumer. On this page you read and ack on the happy path; a [later page](/learn/jetstream/delivery-and-acknowledgment.md) digs into what acknowledgment buys you — the redelivery loop that makes delivery reliable. Look at the consumer before it has read anything: ``` nats consumer info ORDERS billing ``` ``` State: Last Delivered Message: Consumer sequence: 0 Stream sequence: 0 Acknowledgment Floor: Consumer sequence: 0 Stream sequence: 0 Outstanding Acks: 0 out of maximum 1,000 Redelivered Messages: 0 Unprocessed Messages: 3 Waiting Pulls: 0 of maximum 512 ``` `Unprocessed Messages` is how many stored messages the consumer hasn't delivered yet: the whole stream, since it hasn't started. Yours shows however many you published. The acknowledgment fields below it stay at zero until the consumer starts delivering and acking; the next page works through them. ## Read the stored messages Now read. Binding to the consumer reuses its position, delivers everything it hasn't seen, and acknowledges each message as it goes: #### CLI ``` #!/bin/bash # Read every message the stream currently holds, in order, then stop. # Binding 'nats sub' to the durable consumer reuses its on-disk position. # --stream ORDERS --durable billing bind to the durable consumer # --terminate-at-end stop once all stored messages are read # --ack acknowledge each message it delivers # # This makes no assumption about how many messages are stored: it reads # whatever is there and stops when 'pending' reaches 0. nats sub --stream ORDERS --durable billing --terminate-at-end --ack ``` #### JavaScript/TypeScript ``` // Bind to the existing durable and read every stored message in order. Ask the // consumer how many are waiting (num_pending) instead of guessing a count, then // fetch exactly that many. Ack each message after handling it so the server // advances the consumer past it and won't redeliver it. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "billing"); const info = await c.info(); const n = info.num_pending; if (n === 0) { console.log("nothing to read"); } else { const msgs = await c.fetch({ max_messages: Number(n) }); for await (const m of msgs) { console.log( `stream seq ${m.info.streamSequence}, delivery seq ${m.info.deliverySequence}: ${m.string()}`, ); await m.ack(); } } ``` #### Go ``` // Bind to the durable consumer created earlier. cons, err := js.Consumer(ctx, "ORDERS", "billing") if err != nil { panic(err) } // Ask the consumer how many messages are still waiting, then read exactly // that many. This drains everything stored without assuming a count. info, err := cons.Info(ctx) if err != nil { panic(err) } pending := info.NumPending if pending == 0 { fmt.Println("nothing to read") return } msgs, err := cons.Fetch(int(pending)) if err != nil { panic(err) } for msg := range msgs.Messages() { // Metadata carries the position of this message in the stream and in // the consumer's own delivery sequence. meta, err := msg.Metadata() if err != nil { panic(err) } fmt.Printf("stream seq=%d consumer seq=%d payload=%s\n", meta.Sequence.Stream, meta.Sequence.Consumer, string(msg.Data())) // Acknowledge so the server advances the consumer past this message. if err := msg.Ack(); err != nil { panic(err) } } if msgs.Error() != nil { panic(msgs.Error()) } ``` #### Python ``` # Bind to the durable consumer created earlier and pull every stored message. psub = await js.pull_subscribe_bind("billing", stream="ORDERS") # Fetch in batches until a fetch times out, which means the stream is drained. # The consumer uses explicit ack, so acknowledge each message after handling it. while True: try: msgs = await psub.fetch(batch=100, timeout=1) except nats.errors.TimeoutError: break for msg in msgs: print( f"stream seq {msg.metadata.sequence.stream}, " f"consumer seq {msg.metadata.sequence.consumer}: " f"{msg.data.decode()}" ) await msg.ack() ``` #### Java ``` // Ask the consumer how many messages are still waiting, then fetch // exactly that many so the loop ends without guessing a count. long pending = cc.getConsumerInfo().getNumPending(); if (pending == 0) { System.out.println("Nothing to read."); return; } try (FetchConsumer fc = cc.fetch( FetchConsumeOptions.builder().maxMessages((int) pending).build())) { Message m; while ((m = fc.nextMessage()) != null) { System.out.printf("stream_seq=%d consumer_seq=%d data=%s%n", m.metaData().streamSequence(), m.metaData().consumerSequence(), new String(m.getData(), StandardCharsets.UTF_8)); // Acknowledge each message so the server advances the consumer. m.ack(); } } ``` #### Rust ``` // Bind to the existing durable consumer. let stream = js.get_stream("ORDERS").await?; let mut consumer: PullConsumer = stream.get_consumer("billing").await?; // Ask the consumer how many messages are still waiting, then fetch exactly // that many. This reads everything in order without assuming a count. let pending = consumer.info().await?.num_pending; if pending == 0 { println!("nothing to read"); return Ok(()); } let mut messages = consumer .fetch() .max_messages(usize::try_from(pending).unwrap_or(usize::MAX)) .messages() .await?; while let Some(msg) = messages.next().await { let msg = msg?; let info = msg.info()?; println!( "stream {} consumer {}: {}", info.stream_sequence, info.consumer_sequence, std::str::from_utf8(&msg.payload)? ); // Acknowledge so the server records this message as read. msg.ack().await?; } ``` #### C#/.NET ``` // Bind to the existing durable consumer var consumer = await js.GetConsumerAsync("ORDERS", "billing"); // Read exactly what the stream is holding, no count assumed up front var pending = (long)consumer.Info.NumPending; if (pending == 0) { output.WriteLine("nothing to read"); } else { await foreach (var msg in consumer.FetchAsync(opts: new NatsJSFetchOpts { MaxMsgs = (int)pending })) { output.WriteLine($"stream {msg.Metadata?.Sequence.Stream} consumer {msg.Metadata?.Sequence.Consumer}: {msg.Data}"); read++; // Acknowledge each message so the consumer advances past it await msg.AckAsync(); } } ``` #### C ``` // Ask the consumer how many messages are still waiting, then read // exactly that many. This drains everything stored without assuming // a count. jsConsumerInfo *ci = NULL; uint64_t pending = 0; s = js_GetConsumerInfo(&ci, js, "ORDERS", "billing", NULL, &jerr); if (s == NATS_OK) { pending = ci->NumPending; jsConsumerInfo_Destroy(ci); } if ((s == NATS_OK) && (pending == 0)) printf("nothing to read\n"); if ((s == NATS_OK) && (pending > 0)) { // Bind to the durable consumer created earlier and fetch the // pending messages. jsSubOptions so; natsMsgList list = {NULL, 0}; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "billing"; s = js_PullSubscribe(&sub, js, NULL, NULL, NULL, &so, &jerr); if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, (int) pending, 5000, &jerr); if (s == NATS_OK) { for (int i = 0; i < list.Count; i++) { // Metadata carries the position of this message in the // stream and in the consumer's own delivery sequence. jsMsgMetaData *meta = NULL; s = natsMsg_GetMetaData(&meta, list.Msgs[i]); if (s != NATS_OK) break; printf("stream seq=%" PRIu64 " consumer seq=%" PRIu64 " payload=%s\n", meta->Sequence.Stream, meta->Sequence.Consumer, natsMsg_GetData(list.Msgs[i])); jsMsgMetaData_Destroy(meta); // Acknowledge so the server advances the consumer past // this message. natsMsg_Ack(list.Msgs[i], NULL); } natsMsgList_Destroy(&list); } } ``` The messages arrive oldest first, each with its stream sequence: ``` [#1] Received JetStream message: stream: ORDERS seq: 1 / pending: 2 / subject: orders.created / time: 2026-05-22 10:14:22 {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"} [#2] Received JetStream message: stream: ORDERS seq: 2 / pending: 1 / subject: orders.created / time: 2026-05-22 10:14:25 {"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"} [#3] Received JetStream message: stream: ORDERS seq: 3 / pending: 0 / subject: orders.shipped / time: 2026-05-22 10:14:31 {"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:31Z"} ``` The `pending` count is how many stored messages are still waiting. It falls by one with each message and the read stops when it reaches `0`, whether the stream held three messages or three hundred. The stream sequences (`seq: 1`, `2`, `3`) match what `nats stream info` reported on the previous page. You're reading the same append-only log, in order. ## Stream sequence versus consumer sequence Read the consumer again, now that it has caught up: ``` nats consumer info ORDERS billing ``` ``` State: Last Delivered Message: Consumer sequence: 3 Stream sequence: 3 Acknowledgment Floor: Consumer sequence: 3 Stream sequence: 3 Unprocessed Messages: 0 ``` `Last Delivered Message` carries the two sequence numbers side by side, and they're worth telling apart: * **Stream sequence** is the message's fixed position in the log. The server assigned it when the message was published, and it never changes. It's the same number you saw in the `PubAck` and in `nats stream info`. * **Consumer sequence** is this consumer's own counter. The server adds one every time the consumer delivers a message, redeliveries included, so it tracks how many deliveries this consumer has made, not how many distinct messages it has seen. Here both read `3` because `billing` started at the beginning and read straight through, so its third delivery was stream message 3. They drift apart whenever a consumer delivers a different set of messages than the stream stores in order: one that starts partway through, one that [filters](/learn/jetstream/filtering.md) to a subset of subjects, or one that has a message [redelivered](/learn/jetstream/delivery-and-acknowledgment.md). The consumer sequence counts deliveries; the stream sequence stays pinned to the message. You read these numbers with `nats consumer info`, but in code you rarely need that call. Every message a consumer delivers carries the same state in its **metadata** — its stream and consumer sequence, how many messages are still pending, how many times it's been delivered. That metadata rides along with the delivery for free, while `nats consumer info` is a separate request to the server: fine for a one-off check, too costly to call for every message. In a handler, read the state off the message — the read example above already does this, pulling the stream and consumer sequence straight from each delivery. ## The consumer remembers where it stopped The position you just saw is on disk. To see it work, publish one more order: ``` nats pub --jetstream orders.created \ '{"order_id":"ord_5xk1","customer":"initech","total_cents":1500,"ts":"2026-05-22T11:02:00Z"}' ``` Then read again with the same command. The consumer delivers only the new message, not the whole log: ``` [#1] Received JetStream message: stream: ORDERS seq: 4 / pending: 0 / subject: orders.created / time: 2026-05-22 11:02:00 {"order_id":"ord_5xk1","customer":"initech","total_cents":1500,"ts":"2026-05-22T11:02:00Z"} ``` It resumed from where it left off. A disconnect or a server restart wouldn't change that: the position is durable, tied to the name `billing`. An unnamed reader wouldn't have it, which the next section gets to. ## Reading once without a named consumer Creating a durable consumer is the right move when a reader comes back. For a throwaway look at a stream you don't need one. Passing `nats sub` a JetStream flag like `--all` switches it from a live core subscription to a stream read: the CLI builds an **ephemeral** consumer, reads the backlog, and discards the consumer when the command exits: #### CLI ``` #!/bin/bash # Replay every message stored in the ORDERS stream, from sequence 1. # 'nats sub' against a stream-bound subject uses an ephemeral consumer # with no ack by default. # --all start at the beginning of the stream # --terminate-at-end stop once all stored messages have been read nats sub "orders.>" --all --terminate-at-end ``` An ephemeral consumer keeps no position you can return to. It's fine for a one-off read, but for anything you need to resume after an interruption, use the durable consumer from this page: an ephemeral one is removed once it goes idle, and a reconnect then starts over from the beginning. In client code, the same one-pass read has a dedicated form — see [Ordered consumers](/learn/jetstream/ordered-consumer.md). ## The stream is unchanged Run `nats stream info ORDERS` again. The `State` block holds the same message count and sequences as before the read (plus the one order you just published). Reading a stream doesn't remove its data. Unlike taking an item off a queue, the messages stay in place for the next reader. ## Pitfalls Reading a stream back is safe, but a few habits cause trouble the first time you point a reader at real data. **Re-running a caught-up reader waits.** The read command stops once it has delivered everything stored, but if there's nothing new, there's nothing to stop after, so it waits for the next message instead of returning. That's correct for a live subscription and surprising for a one-shot read. To read the current backlog and exit, run it when there's something to read; to watch for new messages, leave it running and let them arrive. **Replaying a huge stream from the beginning.** `Deliver all` starts at sequence 1 and reads the entire log. On a three-message `ORDERS` stream that's instant. On a stream holding millions of orders it takes time and moves a lot of data. Read the whole history only when you want it. To start elsewhere, create the consumer with a different delivery policy: the most recent messages (`--deliver last`), messages from a window back to now (`--deliver 1h`), or a known sequence (`--deliver 1000`). The full set is in [Reference → Create Consumer](/reference/jetstream/api/consumer/create.md). Delivery policy sets *where* a consumer starts; a separate **replay policy** sets the *pace* once it's reading. The default, `instant`, hands messages over as fast as the client reads them. The other value, `original`, spaces deliveries out to match the gaps between the messages' original timestamps, replaying recorded traffic at something like its real speed. This chapter uses `instant` throughout; both are listed in [Reference → Create Consumer](/reference/jetstream/api/consumer/create.md). **Reusing a durable name with a different config.** A durable consumer is identified by its name. Create `billing` again with different settings and the server returns *consumer already exists*; it won't silently reconfigure a consumer a reader is using. Edit the consumer (`nats consumer edit`) instead, or pick a new name. ## Where you are The `ORDERS` stream is unchanged by reading. You now have: * the `ORDERS` stream bound to `orders.>`, holding the messages you published * a durable consumer, `billing`, with a position saved on the server * a clear split between the **stream sequence** (the message's fixed slot) and the **consumer sequence** (how far a reader has progressed) ## What's next You've read the log with one consumer and acked each message. The next page adds a second consumer with a filter and shows how several consumers read the same stream as independent views, each at its own position. ## See also * [Reference → Consumer Configuration](/reference/jetstream/api/consumer/.md) — every consumer option, including delivery policies and ordered consumers. * [Filtering what you consume](/learn/jetstream/filtering.md) — add a second consumer that reads only the subjects it needs. --- # Retention policies The previous page shaped `ORDERS` with limits: how many messages it keeps, for how long, in how many bytes. Those limits decide when a message leaves the stream because the stream ran out of room. There's a second, separate question: should a message ever leave the stream because a consumer finished with it? Limits don't ask that. A message capped only by age or size stays until its limit hits, read or unread, acked or not. Some workloads need the other behavior. A job queue wants a message gone once a worker completes it. A fan-out wants a message gone once every interested consumer has seen it. Limits can't express either. The **retention policy** is the field that does. It decides what makes a message ready to leave the stream. You picked one already, by accepting a default, when you created `ORDERS`. ## The three policies A stream has exactly one retention policy, fixed by the `retention` field. There are three values. **Limits** is the default, and the one `ORDERS` has. Messages stay until a limit is reached: `MaxMsgs`, `MaxBytes`, or `MaxAge`, whichever comes first. Consumers reading and acking messages has no effect on what the stream keeps. The stream is a log, and the log holds everything inside its limits. **Message flow — Limits retention — acks keep the message (animated):** Limits retention. A consumer reads and acks every order in the ORDERS stream, and each one stays in place — acking never removes a message here. Only a limit (MaxAge, MaxBytes, or MaxMsgs) removes one. The read cursor sweeps the stored messages while all of them remain. **Interest** keeps a message only while some consumer still wants it. A message is removed once *every* consumer whose filter covers it has acked it. If no consumer is interested in a subject, a message on that subject is removed right away. **Message flow — Interest retention — all-ack and no-interest (animated):** Interest retention, both behaviors. When an order is published on orders.shipped, a subject both consumers subscribe to, it is stored and removed once every consumer has acked it. When an order is published on orders.archived, a subject no consumer subscribes to, it is dropped the instant it is published — no interest, nothing stored. **WorkQueue** keeps a message only until *one* consumer acks it. The first ack removes the message for everyone. Each message is delivered once and then removed. **Message flow — WorkQueue retention — first ack drains it (animated):** WorkQueue retention. Each order is delivered to exactly one worker; the first ack removes it for everyone, so the stream drains back to empty. Workers take turns: an order is published, one worker pulls and acks it, and the message is gone. The three policies differ in who decides a message is finished. Under Limits, the limits decide. Under Interest, every consumer must ack before the message is removed. Under WorkQueue, the first consumer to ack removes it. Limits still apply under all three. Retention removes a message when consumers are done with it; the stream's limits remove it when the stream grows too old or too large. On an Interest or WorkQueue stream the limits are the backstop that keeps it bounded when consumers fall behind — retention doesn't replace limits, it adds a second way a message can leave. ## Pick the policy from the kind of work Choose a retention policy from the kind of work the stream does. The policy follows from the work, not the other way around. **An audit log or event history → Limits.** You want to keep every message for a window of time no matter who read it, and you want to replay from any point. `ORDERS` is this kind of stream. Late consumers, re-reads, and the replay on the reading-back page all depend on messages staying after they're consumed. Limits is the only policy that allows that, which is why it's the default and why `ORDERS` keeps it. **A fan-out where every consumer must process each message → Interest.** Several independent services each need to handle every order once, and once they all have, the message is no longer needed. The stream stays small because it drops a message once the last interested consumer is done. You get fan-out delivery without an ever-growing log. **A job queue where each message is work for one worker → WorkQueue.** This is the home for Acme's shipping workers from [Scaling a consumer](/learn/jetstream/worker-pool.md). Each message is an order to ship. One shipping worker claims it, ships it, and acks, and then the task is removed so no one ships the same order twice. The queue drains as the workers keep up, where a log would only grow. ## A WorkQueue stream for the shipping work `ORDERS` stays Limits; don't change it — it's the record of what customers did. But Acme also has *work* to do on each order: when one is paid, a [shipping worker](/learn/jetstream/worker-pool.md) has to ship it, exactly once. Ship the same order twice and a second parcel goes out the door. That's a job queue, not a log — each task goes to one worker and is gone once it's done. Give it its own `FULFILLMENT` stream with WorkQueue retention: #### CLI ``` #!/bin/bash # A WorkQueue stream for the shipping work. Leave ORDERS alone — that's # the record. FULFILLMENT is a separate queue of paid orders waiting to # ship, worked by the shipping workers from the Scaling a consumer page. # --retention work sets the WorkQueue policy. natscli accepts "work" and # "workq" as aliases. Under WorkQueue a message is removed the moment the # first consumer acks it, so the queue drains as the workers keep up. nats stream add FULFILLMENT \ --subjects "fulfill.>" \ --retention work \ --defaults # Inspect it — the Options block now reads Retention: WorkQueue, where # ORDERS reads Retention: Limits. nats stream info FULFILLMENT # Enqueue one order to ship, then have a shipping worker pull and ack it. # After the ack the task is gone — the message count drops back to zero, # which no limit on a Limits stream like ORDERS would ever do. nats pub fulfill.us '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' nats consumer add FULFILLMENT shippers --pull --ack explicit --defaults nats consumer next FULFILLMENT shippers --count 1 --ack # Confirm the ack removed the task: Messages is back to 0. nats stream info FULFILLMENT ``` #### JavaScript/TypeScript ``` // FULFILLMENT is the queue of paid orders waiting to ship. WorkQueue retention // means the stream holds each task only until a worker acks it, then drops it. // That is the opposite of a Limits stream like ORDERS, which keeps the record. const info = await jsm.streams.add({ name: "FULFILLMENT", subjects: ["fulfill.>"], retention: RetentionPolicy.Workqueue, }); console.log(`Created stream FULFILLMENT, retention: ${info.config.retention}`); // Queue one order to ship in the US region. const order = `{"order_id":"ord_8w2k","customer":"acme-co"}`; const pa = await js.publish("fulfill.us", order); console.log(`Queued ${pa.stream} seq ${pa.seq}`); // A durable pull consumer drains the queue. WorkQueue streams require explicit // ack, so the worker acks each task it finishes. await jsm.consumers.add("FULFILLMENT", { durable_name: "shippers", ack_policy: AckPolicy.Explicit, }); // Pull the order, ship it, and ack. ackAck waits for the server to confirm the // ack landed, so the next read sees the result. const c = await js.consumers.get("FULFILLMENT", "shippers"); const msgs = await c.fetch({ max_messages: 1, expires: 5000 }); for await (const m of msgs) { console.log(`Shipping ${m.subject}: ${m.string()}`); await m.ackAck(); } // The ack removed the task, so the stream is now empty. A Limits stream would // still hold the message; a WorkQueue stream drains to zero. const after = await jsm.streams.info("FULFILLMENT"); console.log(`Messages in FULFILLMENT after ack: ${after.state.messages}`); ``` #### Go ``` // FULFILLMENT is the queue of paid orders awaiting shipment. With WorkQueue // retention the server keeps each message only until a consumer acks it, // then deletes it from the stream. stream, err := js.CreateStream(ctx, jetstream.StreamConfig{ Name: "FULFILLMENT", Subjects: []string{"fulfill.>"}, Retention: jetstream.WorkQueuePolicy, }) if err != nil { panic(err) } fmt.Printf("FULFILLMENT retention: %s\n", stream.CachedInfo().Config.Retention) // Queue one order to ship. if _, err := js.Publish(ctx, "fulfill.us", []byte(`{"order_id":"ord_8w2k","customer":"acme-co"}`)); err != nil { panic(err) } // A shipping worker binds to a durable pull consumer with explicit ack. cons, err := stream.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{ Durable: "shippers", AckPolicy: jetstream.AckExplicitPolicy, }) if err != nil { panic(err) } // Fetch the order and ack it once the worker has shipped it. msgs, err := cons.Fetch(1) if err != nil { panic(err) } for msg := range msgs.Messages() { fmt.Printf("shipping order: %s\n", string(msg.Data())) if err := msg.Ack(); err != nil { panic(err) } } if msgs.Error() != nil { panic(msgs.Error()) } // The ack removed the task from the queue, so the stream is now empty. A // Limits stream like ORDERS would still hold the message; a WorkQueue // stream drains to zero as workers finish. info, err := stream.Info(ctx) if err != nil { panic(err) } fmt.Printf("messages left in FULFILLMENT: %d\n", info.State.Msgs) ``` #### Python ``` # FULFILLMENT is the queue of paid orders awaiting shipment. WorkQueue # retention delivers each order to a single worker; the first ack removes # it for everyone, so the stream drains to empty. info = await js.add_stream( StreamConfig( name="FULFILLMENT", subjects=["fulfill.>"], retention=RetentionPolicy.WORK_QUEUE, ) ) print(f"FULFILLMENT retention is {info.config.retention}") # A paid order arrives and needs shipping. await js.publish( "fulfill.us", json.dumps({"order_id": "ord_8w2k", "customer": "acme-co"}).encode(), ) # Shipping workers pull from a durable consumer with explicit ack. await js.add_consumer( "FULFILLMENT", ConsumerConfig(durable_name="shippers", ack_policy=AckPolicy.EXPLICIT), ) # A worker takes one order and acks it once the order has shipped. psub = await js.pull_subscribe_bind("shippers", stream="FULFILLMENT") msgs = await psub.fetch(batch=1, timeout=5) await msgs[0].ack() print(f"shipped {msgs[0].data.decode()}") # The ack removed the task, so the WorkQueue stream is now empty. A Limits # stream like ORDERS would still hold the message after the ack. info = await js.stream_info("FULFILLMENT") print(f"messages left in FULFILLMENT: {info.state.messages}") ``` #### Java ``` // FULFILLMENT is the queue of paid orders awaiting shipment, separate // from the ORDERS record stream. WorkQueue retention delivers each task // to one consumer; the first ack removes it, so the stream drains to empty. StreamInfo info = jsm.addStream(StreamConfiguration.builder() .name("FULFILLMENT") .subjects("fulfill.>") .retentionPolicy(RetentionPolicy.WorkQueue) .build()); System.out.println("Retention policy: " + info.getConfiguration().getRetentionPolicy()); // Queue one paid order for a US shipper to pick up. js.publish("fulfill.us", "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"}".getBytes(StandardCharsets.UTF_8)); // A durable pull consumer with explicit ack: a shipping worker must // acknowledge each task once it has handled the order. jsm.addOrUpdateConsumer("FULFILLMENT", ConsumerConfiguration.builder() .durable("shippers") .ackPolicy(AckPolicy.Explicit) .build()); // Fetch one task and ack it, as a shipping worker would. ConsumerContext shippers = js.getConsumerContext("FULFILLMENT", "shippers"); try (FetchConsumer fc = shippers.fetchMessages(1)) { Message task = fc.nextMessage(); System.out.println("Shipping: " + new String(task.getData(), StandardCharsets.UTF_8)); task.ack(); } // The ack removed the task, so the WorkQueue stream is now empty. // A Limits stream like ORDERS would still hold the message. long count = jsm.getStreamInfo("FULFILLMENT").getStreamState().getMsgCount(); System.out.println("Messages remaining in FULFILLMENT: " + count); ``` #### Rust ``` // FULFILLMENT is the queue of paid orders waiting to ship, separate from the // ORDERS record. WorkQueue retention means each message is delivered to one // consumer, and the first ack removes it for everyone, draining the stream. let mut stream = js .create_stream(Config { name: "FULFILLMENT".to_string(), subjects: vec!["fulfill.>".to_string()], retention: RetentionPolicy::WorkQueue, ..Default::default() }) .await?; println!("retention is {:?}", stream.cached_info().config.retention); // Queue one paid order for a shipper to pick up. js.publish( "fulfill.us", r#"{"order_id":"ord_8w2k","customer":"acme-co"}"#.into(), ) .await? .await?; // Shipping workers drain the queue through a durable pull consumer that // acknowledges each task explicitly. let consumer = stream .create_consumer(pull::Config { durable_name: Some("shippers".to_string()), ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; // Take one task and ack it once the order is shipped. let mut messages = consumer.fetch().max_messages(1).messages().await?; while let Some(message) = messages.next().await { let message = message?; println!("shipping {}", std::str::from_utf8(&message.payload)?); message.ack().await?; } // The ack removed the task from the WorkQueue stream, so the count is now 0. // A Limits stream like ORDERS would still hold the message after ack. let count = stream.info().await?.state.messages; println!("messages left in FULFILLMENT: {count}"); ``` #### C#/.NET ``` // FULFILLMENT is the queue of paid orders waiting to ship. WorkQueue // retention means an order leaves the stream the moment a worker acks it. var stream = await js.CreateStreamAsync(new StreamConfig(name: "FULFILLMENT", subjects: ["fulfill.>"]) { Retention = StreamConfigRetention.Workqueue, }); output.WriteLine($"FULFILLMENT retention is {stream.Info.Config.Retention}"); // One paid order waiting to ship to a US address await js.PublishAsync(subject: "fulfill.us", data: new Order(OrderId: "ord_8w2k", Customer: "acme-co")); // A durable pull consumer the shipping workers share var consumer = await js.CreateOrUpdateConsumerAsync("FULFILLMENT", new ConsumerConfig("shippers") { AckPolicy = ConsumerConfigAckPolicy.Explicit, }); // Fetch the one order and ack it. DoubleAck waits for the server to confirm // the ack, so the WorkQueue removal has happened before we read the count. await foreach (var msg in consumer.FetchAsync(opts: new NatsJSFetchOpts { MaxMsgs = 1 })) { output.WriteLine($"shipping {msg.Data}"); await msg.AckAsync(new AckOpts { DoubleAck = true }); } // The ack removed the task, so a WorkQueue stream drains back to empty. A // Limits stream like ORDERS would still be holding this order. var info = await js.GetStreamAsync("FULFILLMENT"); output.WriteLine($"Messages left in FULFILLMENT: {info.Info.State.Messages}"); ``` #### C ``` // FULFILLMENT is the queue of paid orders awaiting shipment. With // WorkQueue retention the server keeps each message only until a // consumer acks it, then deletes it from the stream. jsStreamConfig sc; const char *subjects[] = {"fulfill.>"}; const char *order = "{\"order_id\":\"ord_8w2k\",\"customer\":\"acme-co\"}"; jsStreamConfig_Init(&sc); sc.Name = "FULFILLMENT"; sc.Subjects = subjects; sc.SubjectsLen = 1; sc.Retention = js_WorkQueuePolicy; s = js_AddStream(&si, js, &sc, NULL, &jerr); if (s == NATS_OK) printf("FULFILLMENT retention: %s\n", (si->Config->Retention == js_WorkQueuePolicy) ? "workqueue" : "other"); // Queue one order to ship. if (s == NATS_OK) s = js_Publish(NULL, js, "fulfill.us", order, (int) strlen(order), NULL, &jerr); // A shipping worker binds to a durable pull consumer with explicit ack. if (s == NATS_OK) { jsConsumerConfig cc; jsConsumerConfig_Init(&cc); cc.Durable = "shippers"; cc.AckPolicy = js_AckExplicit; s = js_AddConsumer(NULL, js, "FULFILLMENT", &cc, NULL, &jerr); } if (s == NATS_OK) { jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "FULFILLMENT"; so.Consumer = "shippers"; s = js_PullSubscribe(&sub, js, NULL, "shippers", NULL, &so, &jerr); } // Fetch the order and ack it once the worker has shipped it. if (s == NATS_OK) s = natsSubscription_Fetch(&list, sub, 1, 5000, &jerr); for (i = 0; (s == NATS_OK) && (i < list.Count); i++) { printf("shipping order: %s\n", natsMsg_GetData(list.Msgs[i])); s = natsMsg_AckSync(list.Msgs[i], NULL, &jerr); } natsMsgList_Destroy(&list); // The ack removed the task from the queue, so the stream is now // empty. A Limits stream like ORDERS would still hold the message; a // WorkQueue stream drains to zero as workers finish. if (s == NATS_OK) s = js_GetStreamInfo(&info, js, "FULFILLMENT", NULL, &jerr); if (s == NATS_OK) printf("messages left in FULFILLMENT: %d\n", (int) info->State.Msgs); ``` The `--retention work` flag is the only change from how you built `ORDERS`. `nats stream info FULFILLMENT` shows it in the `Options` block: ``` nats stream info FULFILLMENT ``` ``` Subjects: fulfill.> Options: Retention: WorkQueue Discard Policy: Old ``` Enqueue one order to ship and have a shipping worker ack it, and the stream's message count drops back to zero. The ack removed the task, which no limit on `ORDERS` does. The worker publishes `orders.shipped` back to `ORDERS` as it finishes, so the record lands in the log while the task drains from the queue. Those shipping workers run as a pool sharing **one** consumer on `FULFILLMENT`, exactly as on [Scaling a consumer](/learn/jetstream/worker-pool.md): WorkQueue hands each order to the consumer, one worker ships it, and the ack clears it. You scale by adding workers to that one consumer, not by adding consumers — WorkQueue won't let two consumers claim the same order (the pitfalls below cover why). ## Switching retention on a live stream Set retention when you create the stream, and leave it there. The server allows exactly one live change: it swaps Limits and Interest, in either direction. Anything involving WorkQueue is locked — see the pitfall below. And even the allowed swap applies to messages already stored, right away. Say a stream has been collecting an audit history under Limits and you switch it to Interest. From that moment the server removes any message every consumer has already acked, and any message on a subject no consumer is interested in — including history you meant to keep. Treat the policy as fixed at creation. If you want a different policy than the stream has, create a new stream with the right policy rather than editing the running one. `ORDERS` was created as Limits on purpose, and it stays Limits. ## How Interest and WorkQueue can go wrong Interest and WorkQueue each have a way they can go wrong. Know it before you use them. **Interest can fill the disk.** A message is removed only when every consumer whose filter covers it has acked it. A slow consumer holds up cleanup for every message it still owes an ack on. If a consumer stalls (a stuck worker, a service that's down), its unacked messages never become ready to leave, and the stream grows until it hits its limits or runs out of room. Interest retention still needs limits set, and it makes watching consumer health more important. **WorkQueue delivers each message once.** The first ack removes the message for everyone, so no two consumers can claim the same message — the server rejects overlapping consumers outright (the pitfall below). You can still run more than one consumer if their filters *partition* the subjects, but each then handles only its slice; none sees the whole stream. For several consumers that each see every message, use Interest or Limits instead. To scale a single workload, use a worker *pool* sharing one consumer (the worker-pool page), not multiple consumers. The full set of retention behavior, including how Interest and WorkQueue interact with stream republish, mirrors, and sources, is in [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md). This page uses only the three `retention` values. ## Pitfalls Both of these are WorkQueue constraints. The server checks them when you create or edit a stream, so you find out right away rather than in production. **Retention to or from WorkQueue is locked after creation.** The earlier section covered the Limits–Interest swap the server allows, and how even that rewrites your history. The rule underneath it is stricter: the server rejects any change that adds or removes WorkQueue. A stream that isn't WorkQueue at creation can't become one, and a WorkQueue stream can't change to another policy. Don't plan a migration path that edits retention into or out of WorkQueue. Create a new stream with the policy you want and move the data. The edit below is rejected with `stream configuration update can not change retention policy to/from workqueue`. #### CLI ``` #!/bin/bash # Pitfall: you cannot switch a stream's retention to or from WorkQueue # on a live stream. Limits and Interest can swap, but WorkQueue is fixed # at creation. Try to move FULFILLMENT (WorkQueue) to Limits and the # server refuses: # stream configuration update can not change retention policy to/from # workqueue # The command exits non-zero — FULFILLMENT keeps its WorkQueue policy. nats stream edit FULFILLMENT --retention limits --force # The safe move is a new stream with the policy you want, then migrate. # ORDERS itself stays Limits — never edit a live stream's retention to # WorkQueue hoping to convert it. nats stream info FULFILLMENT ``` #### JavaScript/TypeScript ``` // Retention is fixed at creation. Read the live config, flip it from WorkQueue // to Limits, and submit the update. The server rejects the change with err // 10052: it won't move a stream to or from workqueue retention. To change // retention, create a new stream instead. const fulfillment = await jsm.streams.info("FULFILLMENT"); fulfillment.config.retention = RetentionPolicy.Limits; try { await jsm.streams.update("FULFILLMENT", fulfillment.config); } catch (err) { if (err instanceof JetStreamApiError) { console.log(`Rejected (err ${err.code}): ${err.message}`); } else { throw err; } } ``` #### Go ``` // Load FULFILLMENT and take its current config. stream, err := js.Stream(ctx, "FULFILLMENT") if err != nil { panic(err) } cfg := stream.CachedInfo().Config // Try to turn the WorkQueue stream into a Limits stream by flipping // retention and pushing the update back. cfg.Retention = jetstream.LimitsPolicy if _, err := js.UpdateStream(ctx, cfg); err != nil { // The server refuses: a stream's retention policy is fixed for its // lifetime (err 10052: stream configuration update can not change // retention policy to/from workqueue). To change it, recreate the stream. fmt.Printf("retention switch rejected: %v\n", err) return } fmt.Println("update unexpectedly succeeded") ``` #### Python ``` # Read the current config and try to switch FULFILLMENT from WorkQueue to # Limits retention. The server refuses: retention is fixed at create time, # so you cannot turn a queue into a record (or back) on an existing stream. info = await js.stream_info("FULFILLMENT") try: await js.update_stream(info.config, retention=RetentionPolicy.LIMITS) print("retention changed (unexpected)") except APIError as e: print(f"retention switch rejected: {e}") ``` #### Java ``` // You can't change a stream's retention policy after it's created. // Switching FULFILLMENT from WorkQueue to Limits is rejected; to change // retention you would delete the stream and recreate it. try { jsm.updateStream(StreamConfiguration.builder() .name("FULFILLMENT") .subjects("fulfill.>") .retentionPolicy(RetentionPolicy.Limits) .build()); } catch (JetStreamApiException e) { System.out.println("Rejected: " + e.getMessage()); // stream configuration update can not change retention policy to/from workqueue [10052] } ``` #### Rust ``` // Retention is fixed once a stream exists. Trying to switch FULFILLMENT from // WorkQueue to Limits is rejected by the server (err 10052: stream // configuration update can not change retention policy to/from workqueue). match js .update_stream(Config { name: "FULFILLMENT".to_string(), subjects: vec!["fulfill.>".to_string()], retention: RetentionPolicy::Limits, ..Default::default() }) .await { Ok(_) => println!("unexpected: retention switch was accepted"), Err(err) => println!("rejected: {err}"), } ``` #### C#/.NET ``` // Try to turn FULFILLMENT into a plain Limits stream after the fact. var stream = await js.GetStreamAsync("FULFILLMENT"); var config = stream.Info.Config; config.Retention = StreamConfigRetention.Limits; try { await js.UpdateStreamAsync(config); output.WriteLine("update accepted"); } catch (NatsJSApiException e) { // The server won't switch retention into or out of WorkQueue on a live // stream. To change it, recreate the stream. output.WriteLine($"rejected (err {e.Error.ErrCode}): {e.Error.Description}"); } ``` #### C ``` // Load FULFILLMENT and take its current config. s = js_GetStreamInfo(&si, js, "FULFILLMENT", NULL, &jerr); if (s == NATS_OK) { // Try to turn the WorkQueue stream into a Limits stream by // flipping retention and pushing the update back. jsStreamInfo *updated = NULL; si->Config->Retention = js_LimitsPolicy; s = js_UpdateStream(&updated, js, si->Config, NULL, &jerr); if (s != NATS_OK) { // The server refuses: a stream's retention policy is fixed // for its lifetime (err 10052: stream configuration update // can not change retention policy to/from workqueue). To // change it, recreate the stream. printf("retention switch rejected: %s\n", nats_GetLastError(NULL)); s = NATS_OK; } else { printf("update unexpectedly succeeded\n"); jsStreamInfo_Destroy(updated); } } ``` **WorkQueue rejects consumers that overlap.** The first ack removes a message for everyone, so the server won't let two consumers claim the same message. Adding a second unfiltered consumer, or two consumers whose filters overlap, fails the create: `multiple non-filtered consumers not allowed on workqueue stream`, or `filtered consumer not unique on workqueue stream` for overlapping filters. Give each consumer a filter that splits the subjects between them, so no message belongs to two consumers. A worker *pool* sharing one consumer is the other valid setup; see [A pool of workers](/learn/jetstream/worker-pool.md). #### CLI ``` #!/bin/bash # Pitfall: a WorkQueue stream rejects a second consumer whose subjects # overlap an existing one. The server won't let two consumers fight over # the same task, so it refuses the create up front. # FULFILLMENT is the WorkQueue stream from earlier on this page. Add one # unfiltered consumer — fine, it owns the whole queue. nats consumer add FULFILLMENT shippers --pull --ack explicit --defaults # Now try to add a second unfiltered consumer. The server rejects it: # multiple non-filtered consumers not allowed on workqueue stream # (error 10099). The command exits non-zero. nats consumer add FULFILLMENT eu-shippers --pull --ack explicit --defaults # The fix is non-overlapping filters so each task belongs to exactly one # consumer — here, one shipper consumer per region. Delete the broad # consumer, then split by subject. nats consumer rm FULFILLMENT shippers --force nats consumer add FULFILLMENT us-shippers --pull --ack explicit --filter "fulfill.us" --defaults nats consumer add FULFILLMENT eu-shippers --pull --ack explicit --filter "fulfill.eu" --defaults # These two coexist because fulfill.us and fulfill.eu never collide. # A wildcard like fulfill.> would overlap both and be rejected with # "filtered consumer not unique on workqueue stream" (error 10100). ``` #### JavaScript/TypeScript ``` // One unfiltered consumer reads every subject in the queue. On a WorkQueue // stream that is fine on its own. await jsm.consumers.add("FULFILLMENT", { durable_name: "shippers", ack_policy: AckPolicy.Explicit, }); console.log("Created consumer: shippers"); // A WorkQueue stream hands each task to exactly one consumer, so two unfiltered // consumers would both claim the same subjects. The server rejects the second // one with err 10099. try { await jsm.consumers.add("FULFILLMENT", { durable_name: "eu-shippers", ack_policy: AckPolicy.Explicit, }); } catch (err) { if (err instanceof JetStreamApiError) { console.log(`Rejected (err ${err.code}): ${err.message}`); } else { throw err; } } // Give each worker its own slice of the queue instead. Drop the unfiltered // consumer, then create one consumer per region. Their filters do not overlap, // so both are allowed. await jsm.consumers.delete("FULFILLMENT", "shippers"); await jsm.consumers.add("FULFILLMENT", { durable_name: "us-shippers", ack_policy: AckPolicy.Explicit, filter_subject: "fulfill.us", }); await jsm.consumers.add("FULFILLMENT", { durable_name: "eu-shippers", ack_policy: AckPolicy.Explicit, filter_subject: "fulfill.eu", }); console.log("Created filtered consumers: us-shippers, eu-shippers"); ``` #### Go ``` // One unfiltered consumer covers every order in the queue. if _, err := stream.CreateConsumer(ctx, jetstream.ConsumerConfig{ Durable: "shippers", AckPolicy: jetstream.AckExplicitPolicy, }); err != nil { panic(err) } fmt.Println("created unfiltered consumer: shippers") // A WorkQueue stream delivers each message to exactly one consumer, so it // refuses a second consumer whose reach overlaps the first. Two unfiltered // consumers both cover fulfill.> , so this is rejected // (err 10099: multiple non-filtered consumers not allowed on workqueue stream). if _, err := stream.CreateConsumer(ctx, jetstream.ConsumerConfig{ Durable: "eu-shippers", AckPolicy: jetstream.AckExplicitPolicy, }); err != nil { fmt.Printf("second unfiltered consumer rejected: %v\n", err) } // Drop the catch-all consumer so we can split the queue by region instead. if err := stream.DeleteConsumer(ctx, "shippers"); err != nil { panic(err) } // Filtered consumers are allowed as long as their subjects don't overlap. // us-shippers takes fulfill.us, eu-shippers takes fulfill.eu, and each // order still goes to exactly one worker. if _, err := stream.CreateConsumer(ctx, jetstream.ConsumerConfig{ Durable: "us-shippers", FilterSubject: "fulfill.us", AckPolicy: jetstream.AckExplicitPolicy, }); err != nil { panic(err) } if _, err := stream.CreateConsumer(ctx, jetstream.ConsumerConfig{ Durable: "eu-shippers", FilterSubject: "fulfill.eu", AckPolicy: jetstream.AckExplicitPolicy, }); err != nil { panic(err) } fmt.Println("created filtered consumers: us-shippers (fulfill.us), eu-shippers (fulfill.eu)") ``` #### Python ``` # One unfiltered consumer covers every fulfill.> subject. That is allowed. await js.add_consumer( "FULFILLMENT", ConsumerConfig(durable_name="shippers", ack_policy=AckPolicy.EXPLICIT), ) print("added unfiltered consumer shippers") # A WorkQueue stream gives each message to exactly one consumer, so two # consumers whose subjects overlap would compete for the same orders. A # second unfiltered consumer overlaps with the first and is rejected. try: await js.add_consumer( "FULFILLMENT", ConsumerConfig(durable_name="eu-shippers", ack_policy=AckPolicy.EXPLICIT), ) except APIError as e: print(f"second unfiltered consumer rejected: {e}") # Drop the unfiltered consumer so the subject space is free again. await js.delete_consumer("FULFILLMENT", "shippers") # Filtered consumers work as long as their subjects do not overlap. Split # the queue by region: one worker pool per destination. await js.add_consumer( "FULFILLMENT", ConsumerConfig( durable_name="us-shippers", filter_subject="fulfill.us", ack_policy=AckPolicy.EXPLICIT, ), ) await js.add_consumer( "FULFILLMENT", ConsumerConfig( durable_name="eu-shippers", filter_subject="fulfill.eu", ack_policy=AckPolicy.EXPLICIT, ), ) print("added filtered consumers us-shippers and eu-shippers") ``` #### Java ``` // One unfiltered consumer on a WorkQueue stream is fine: it owns every // subject, so each task still has exactly one owner. jsm.addOrUpdateConsumer("FULFILLMENT", ConsumerConfiguration.builder() .durable("shippers") .ackPolicy(AckPolicy.Explicit) .build()); System.out.println("Added consumer: shippers"); // A second unfiltered consumer would let two workers claim the same // task, so the WorkQueue stream rejects it. try { jsm.addOrUpdateConsumer("FULFILLMENT", ConsumerConfiguration.builder() .durable("eu-shippers") .ackPolicy(AckPolicy.Explicit) .build()); } catch (JetStreamApiException e) { System.out.println("Rejected: " + e.getMessage()); // multiple non-filtered consumers not allowed on workqueue stream [10099] } // To split the work, drop the catch-all consumer... jsm.deleteConsumer("FULFILLMENT", "shippers"); // ...and give each region its own subject filter. The filters don't // overlap, so every task still maps to exactly one consumer. jsm.addOrUpdateConsumer("FULFILLMENT", ConsumerConfiguration.builder() .durable("us-shippers") .filterSubject("fulfill.us") .ackPolicy(AckPolicy.Explicit) .build()); jsm.addOrUpdateConsumer("FULFILLMENT", ConsumerConfiguration.builder() .durable("eu-shippers") .filterSubject("fulfill.eu") .ackPolicy(AckPolicy.Explicit) .build()); System.out.println("Added filtered consumers: us-shippers (fulfill.us), eu-shippers (fulfill.eu)"); ``` #### Rust ``` // One unfiltered consumer claims every order on the queue. That is allowed. stream .create_consumer(pull::Config { durable_name: Some("shippers".to_string()), ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; // A WorkQueue stream gives each message to exactly one consumer, so two // unfiltered consumers would both claim the same orders. The server rejects // the second one (err 10099: multiple non-filtered consumers not allowed on // workqueue stream). match stream .create_consumer(pull::Config { durable_name: Some("eu-shippers".to_string()), ack_policy: AckPolicy::Explicit, ..Default::default() }) .await { Ok(_) => println!("unexpected: second unfiltered consumer was accepted"), Err(err) => println!("rejected: {err}"), } // Split the work by subject instead. Drop the unfiltered consumer, then add // one consumer per region. Their filters do not overlap, so both succeed. stream.delete_consumer("shippers").await?; stream .create_consumer(pull::Config { durable_name: Some("us-shippers".to_string()), filter_subject: "fulfill.us".to_string(), ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; stream .create_consumer(pull::Config { durable_name: Some("eu-shippers".to_string()), filter_subject: "fulfill.eu".to_string(), ack_policy: AckPolicy::Explicit, ..Default::default() }) .await?; println!("us-shippers and eu-shippers created"); ``` #### C#/.NET ``` // One unfiltered consumer over the whole queue is fine. await js.CreateOrUpdateConsumerAsync("FULFILLMENT", new ConsumerConfig("shippers") { AckPolicy = ConsumerConfigAckPolicy.Explicit, }); // A second unfiltered consumer would see the same orders, so two workers // could each be handed the same task. A WorkQueue stream refuses it. try { await js.CreateOrUpdateConsumerAsync("FULFILLMENT", new ConsumerConfig("eu-shippers") { AckPolicy = ConsumerConfigAckPolicy.Explicit, }); } catch (NatsJSApiException e) { output.WriteLine($"rejected (err {e.Error.ErrCode}): {e.Error.Description}"); } // Give each worker its own slice of the subjects instead. Drop the // catch-all consumer first... await js.DeleteConsumerAsync("FULFILLMENT", "shippers"); // ...then create two consumers whose filters don't overlap. Both succeed, // because no order can be claimed by more than one of them. await js.CreateOrUpdateConsumerAsync("FULFILLMENT", new ConsumerConfig("us-shippers") { AckPolicy = ConsumerConfigAckPolicy.Explicit, FilterSubject = "fulfill.us", }); await js.CreateOrUpdateConsumerAsync("FULFILLMENT", new ConsumerConfig("eu-shippers") { AckPolicy = ConsumerConfigAckPolicy.Explicit, FilterSubject = "fulfill.eu", }); output.WriteLine("Created us-shippers and eu-shippers on non-overlapping filters"); ``` #### C ``` // One unfiltered consumer covers every order in the queue. jsConsumerConfig cc; jsConsumerConfig_Init(&cc); cc.Durable = "shippers"; cc.AckPolicy = js_AckExplicit; s = js_AddConsumer(NULL, js, "FULFILLMENT", &cc, NULL, &jerr); if (s == NATS_OK) printf("created unfiltered consumer: shippers\n"); // A WorkQueue stream delivers each message to exactly one consumer, // so it refuses a second consumer whose reach overlaps the first. // Two unfiltered consumers both cover fulfill.>, so this is rejected // (err 10099: multiple non-filtered consumers not allowed on // workqueue stream). if (s == NATS_OK) { jsConsumerConfig_Init(&cc); cc.Durable = "eu-shippers"; cc.AckPolicy = js_AckExplicit; if (js_AddConsumer(NULL, js, "FULFILLMENT", &cc, NULL, &jerr) != NATS_OK) printf("second unfiltered consumer rejected: %s\n", nats_GetLastError(NULL)); } // Drop the catch-all consumer so we can split the queue by region // instead. if (s == NATS_OK) s = js_DeleteConsumer(js, "FULFILLMENT", "shippers", NULL, &jerr); // Filtered consumers are allowed as long as their subjects don't // overlap. us-shippers takes fulfill.us, eu-shippers takes // fulfill.eu, and each order still goes to exactly one worker. if (s == NATS_OK) { jsConsumerConfig_Init(&cc); cc.Durable = "us-shippers"; cc.FilterSubject = "fulfill.us"; cc.AckPolicy = js_AckExplicit; s = js_AddConsumer(NULL, js, "FULFILLMENT", &cc, NULL, &jerr); } if (s == NATS_OK) { jsConsumerConfig_Init(&cc); cc.Durable = "eu-shippers"; cc.FilterSubject = "fulfill.eu"; cc.AckPolicy = js_AckExplicit; s = js_AddConsumer(NULL, js, "FULFILLMENT", &cc, NULL, &jerr); } if (s == NATS_OK) printf("created filtered consumers: us-shippers (fulfill.us), eu-shippers (fulfill.eu)\n"); ``` ## Where you are `ORDERS` is unchanged. It's a Limits stream that holds its order history and lets late and repeat consumers replay. You now have: * The three retention policies (Limits, Interest, WorkQueue) and the one question that separates them: who decides a message is finished. * Which policy fits which kind of work: audit log → Limits, fan-out → Interest, job queue → WorkQueue. * A `FULFILLMENT` WorkQueue stream — the shipping workers' queue — that dropped a task on ack while `ORDERS` kept the record. * The rule that retention is fixed at creation, not switched on a live stream. ## What's next Retention removes messages on a schedule the server runs. The next page covers removing them by hand: [deleting a single message](/learn/jetstream/altering-stream-state.md) and purging the stream. ## See also * [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md) — the `retention` field, its three values, and how each interacts with limits, republish, and mirrors. * [Shaping the stream](/learn/jetstream/shaping-the-stream.md) — the limits that govern a Limits stream. * [A pool of workers](/learn/jetstream/worker-pool.md) — the worker pool that shares one consumer, the pattern that fits WorkQueue. --- # Shaping the stream The `ORDERS` stream you created back on the [Your first stream](/learn/jetstream/your-first-stream.md) page has no limits. It keeps every message forever, on however much disk the server has. That was fine for learning, but not for production. Without a limit, the stream keeps growing until it fills the disk and takes the server down with it. This page covers two settings. The **limit** is the cap that decides when the stream is full. The **Discard policy** decides which message the server discards when the stream hits that limit: the new one or the oldest stored one. ## The limit A stream under the default **Limits** retention policy keeps messages until a limit forces it to discard them. You saw that policy in the config printout on the [Your first stream](/learn/jetstream/your-first-stream.md) page. A limit is a ceiling on the stream. You set it with one of three options, depending on how you want to measure the stream: * **MaxAge** caps how old a message may get. Set it to seven days and a message is discarded roughly seven days after it was stored. It fits when only recent events matter — a live order stream rarely needs one from last quarter. * **MaxBytes** caps how much disk the stream may use. Set it to one gigabyte and the stream never grows past a gigabyte, no matter how old or new the messages are. This option protects the server itself. * **MaxMsgs** caps how many messages the stream may hold. Set it to one million and the millionth-and-first message forces a discard. Use this when message count, rather than size or age, is what you reason about. The three options work separately, and all of them are active at once. Whichever one is reached first triggers a discard. You don't have to set all three. Set the ones that match how you think about the stream, and leave the rest unlimited. MaxAge evicts by the clock; MaxMsgs evicts by the count. Same discard, two different triggers: **Message flow — MaxAge expires messages by age (animated):** MaxAge expires messages once they pass an age limit, no matter how full the stream is. Orders arrive a couple a day against a 7-day window; each order lives until it reaches 7 days old, then the server discards it, while everything still inside the window stays. The stream holds a rolling seven days of orders rather than a fixed count. **Message flow — MaxMsgs keeps a fixed message count (animated):** MaxMsgs caps how many messages a stream keeps. The stream is full at MaxMsgs=5, so each new order discards the oldest to fit; the count stays at five and the stream holds only the most recent orders. ## Cap the ORDERS stream Give `ORDERS` a seven-day age limit and a one-gigabyte ceiling. The `nats stream edit` command changes an existing stream in place, so the messages already stored stay where they are. #### CLI ``` #!/bin/bash # Cap the ORDERS stream so it cannot grow without bound. # # `nats stream edit` changes an existing stream in place. The three # messages already stored stay where they are; only the configuration # changes. The command prints a diff of what will change and asks for # confirmation before applying it. # # --max-age sets MaxAge: the oldest a message may get. 7d means a # message is removed roughly seven days after it was stored. # --max-bytes sets MaxBytes: the most disk the stream may occupy on # disk. 1GiB caps it at one gibibyte (about a gigabyte); the stream # never grows past it. nats reports the cap back as "1.0 GiB". # # MaxMsgs is left unset, so message count stays unlimited. Discard # policy is left at its default (Old), so when a limit is hit the # server drops the oldest messages to make room. nats stream edit ORDERS \ --max-age=7d \ --max-bytes=1GiB ``` #### JavaScript/TypeScript ``` // Cap ORDERS with a seven-day age limit and a 1 GiB byte ceiling. streams.update // takes just the fields you're changing and leaves the rest, and the stored // messages, in place. max_age is in nanoseconds. const jsm = await jetstreamManager(nc); await jsm.streams.update("ORDERS", { max_age: 7 * 24 * 60 * 60 * 1_000_000_000, // 7 days max_bytes: 1024 * 1024 * 1024, // 1 GiB }); console.log("ORDERS capped at 7d age and 1 GiB"); ``` #### Go ``` // Cap ORDERS so it can't grow without bound. Fetch the current config, add // a seven-day age limit and a 1 GiB byte ceiling, and update the stream in // place. Editing limits leaves the messages already stored alone. s, err := js.Stream(ctx, "ORDERS") if err != nil { panic(err) } cfg := s.CachedInfo().Config cfg.MaxAge = 7 * 24 * time.Hour cfg.MaxBytes = 1 << 30 // 1 GiB if _, err := js.UpdateStream(ctx, cfg); err != nil { panic(err) } fmt.Println("ORDERS capped at 7d age and 1 GiB") ``` #### Python ``` # Cap ORDERS with a seven-day age limit and a 1 GiB byte ceiling. Fetch the # current config, set the limits, and update the stream in place; the stored # messages stay put. max_age is in seconds. info = await js.stream_info("ORDERS") config = info.config config.max_age = 7 * 24 * 3600 # 7 days config.max_bytes = 1024 * 1024 * 1024 # 1 GiB await js.update_stream(config) print("ORDERS capped at 7d age and 1 GiB") ``` #### Java ``` // Cap ORDERS with a seven-day age limit and a 1 GiB byte ceiling. // Build from the current config so other fields, and the stored // messages, are left in place, then update the stream. StreamInfo si = jsm.getStreamInfo("ORDERS"); StreamConfiguration cfg = StreamConfiguration.builder(si.getConfiguration()) .maxAge(Duration.ofDays(7)) .maxBytes(1L << 30) // 1 GiB .build(); jsm.updateStream(cfg); System.out.println("ORDERS capped at 7d age and 1 GiB"); ``` #### Rust ``` // Cap ORDERS so it can't grow without bound. Fetch the current config, add // a seven-day age limit and a 1 GiB byte ceiling, and update the stream in // place. Editing limits leaves the messages already stored alone. let mut stream = js.get_stream("ORDERS").await?; let mut config = stream.info().await?.config.clone(); config.max_age = Duration::from_secs(7 * 24 * 3600); config.max_bytes = 1 << 30; // 1 GiB js.update_stream(config).await?; println!("ORDERS capped at 7d age and 1 GiB"); ``` #### C#/.NET ``` // Cap ORDERS with a seven-day age limit and a 1 GiB byte ceiling. Fetch // the current config, set the limits, and update the stream in place; // the stored messages stay put. var stream = await js.GetStreamAsync("ORDERS"); var config = stream.Info.Config; config.MaxAge = TimeSpan.FromDays(7); config.MaxBytes = 1L << 30; // 1 GiB await js.UpdateStreamAsync(config); output.WriteLine("ORDERS capped at 7d age and 1 GiB"); ``` #### C ``` // Cap ORDERS so it can't grow without bound. Fetch the current // config, add a seven-day age limit and a 1 GiB byte ceiling, and // update the stream in place. Editing limits leaves the messages // already stored alone. s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { si->Config->MaxAge = (int64_t) 7 * 24 * 60 * 60 * 1000000000LL; // 7 days in ns si->Config->MaxBytes = 1024 * 1024 * 1024; // 1 GiB s = js_UpdateStream(NULL, js, si->Config, NULL, &jerr); } if (s == NATS_OK) printf("ORDERS capped at 7d age and 1 GiB\n"); ``` `nats stream edit` shows the change and asks for confirmation before applying it. Confirm, then read the stream back: ``` nats stream info ORDERS ``` The configuration block now reports the two limits instead of `unlimited`: ``` Options: Retention: Limits Acknowledgments: true Discard Policy: Old Duplicate Window: 2m0s Limits: Maximum Messages: unlimited Maximum Per Subject: unlimited Maximum Bytes: 1.0 GiB Maximum Age: 7d0h0m0s Maximum Message Size: unlimited Maximum Consumers: unlimited ``` `Maximum Messages` is still `unlimited`, because you set only age and bytes. (`Maximum Message Size`, also in the block, is a different limit — a cap on a single message rather than the whole stream — and stays unlimited here.) The stream now has clear bounds: it can't grow past a gigabyte, and it can't hold anything older than a week. ## The Discard policy The Discard policy controls what happens at the moment a new message would push the stream past a limit: the server discards the new message or the oldest stored one. It has two settings. **Discard Old** is the default, and it's what you have right now. When a limit is hit, the server discards the *oldest* messages to make room for the new one. The publish always succeeds: the newest message is stored, the oldest is discarded. **Message flow — Discard Old drops the oldest when full (animated):** With Discard Old (the default), a full stream makes room by dropping its oldest message. Orders flow toward a stream at its limit; each new order is stored while the oldest is discarded to fit, so the publish always succeeds and the stream holds a moving window of the most recent messages. **Discard New** is the opposite. When a size or count limit is hit — `MaxMsgs`, `MaxBytes`, or a per-subject cap — the server discards the *new* message: it rejects the publish, which fails with an error, and nothing already stored is dropped to make room. `MaxAge` isn't a Discard-policy choice: it expires stored messages on its own timer under either policy. The rule from the top of the page still holds — **the first limit to hit applies.** **Message flow — Discard New rejects writes when full (animated):** With Discard New, a full stream rejects new writes instead of dropping old ones. Orders flow toward a stream already at its message limit; each new order is rejected with maximum messages exceeded, and the messages already stored are left untouched. The publisher feels the limit as a failed publish rather than silent data loss. For `ORDERS`, Discard Old is the right choice. A live order stream wants the most recent week of events. If disk pressure forces a trade-off, the order from eight days ago is the one to discard, not today's. Leave the default in place. Use Discard New when discarding an old message would lose data you're required to keep, and you'd rather slow the publisher down than lose history. The publisher then has to handle the rejected publish, which is why it's the less common choice. ## Limits apply to the stream, not the consumer A limit discards a message for everyone. When MaxAge discards a message, it's gone from the stream, and every consumer reading that stream loses access to it at once. Limits and consumers are separate decisions. The `shipping` consumer's position and the `analytics` consumer's filter don't protect a message from the stream's limits. If a consumer is too slow and a message ages out before that consumer reads it, the message is gone. The next page returns to that risk, where the retention policy itself changes. ## Per-subject limits A stream can also limit messages *per subject* rather than across the whole stream, which is useful when `orders.>` should keep, say, the last hundred messages for each individual subject. That's the `MaxMsgsPerSubject` option. The full set of stream limit options is documented in [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md). We use only MaxAge, MaxBytes, and Discard here. ## Pitfalls Limits are easy to set and easy to misread. Two traps account for most of the surprises. **Discard Old discards the oldest message silently.** Discard Old never fails a publish. When a size or count limit is hit, the server drops the oldest message and the publish succeeds — right for a rolling window, but the publisher gets no warning and the oldest order is gone. To keep history and push back on the publisher, switch to Discard New, which rejects the publish (`maximum bytes exceeded` or `maximum messages exceeded`) instead of dropping a stored message. Discard New still leaves `MaxAge` in force — **the first limit to hit applies** — so if you must keep history, remove or raise the age limit too: #### CLI ``` #!/bin/bash # Switch ORDERS from Discard Old to Discard New so a full stream pushes # backpressure to the publisher instead of silently dropping old orders. # # Under Discard Old (the default) a publish that exceeds a limit always # succeeds, because the server drops the oldest message to make room and # tells the publisher nothing. If you need to keep history, that silent # drop is data loss you never see. # # --discard new flips the trade: when a limit is hit, the server rejects # the new message and the publish fails with "maximum bytes exceeded" (or # "maximum messages exceeded"). The publisher now feels the limit. # Force the full condition so you can see the rejection. Discard New never # drops messages that are already stored, so switching ORDERS to it and # capping it at one message leaves the orders from earlier pages in place and # puts the stream instantly over its limit -- both in one edit. nats stream edit ORDERS --discard new --max-msgs 1 # ORDERS is now full. Under Discard New this publish fails instead of # succeeding silently, so the publisher can retry, alert, or shed load: nats pub orders.created \ '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # -> nats: maximum messages exceeded # Put ORDERS back the way it was: Discard Old and no message-count cap. The # 7-day age and 1 GiB byte limits set earlier stay in place. nats stream edit ORDERS --discard old --max-msgs -1 ``` #### JavaScript/TypeScript ``` // Switch ORDERS to Discard New. Discard New never drops stored messages, so // capping it at one message leaves the existing orders in place and puts the // stream instantly over its limit; the next publish is rejected. const jsm = await jetstreamManager(nc); await jsm.streams.update("ORDERS", { discard: DiscardPolicy.New, max_msgs: 1 }); // This publish hits the full stream and rejects with "maximum messages // exceeded" instead of succeeding silently. Handle it in the publisher. const js = jetstream(nc); try { await js.publish("orders.created", JSON.stringify({ order_id: "ord_8w2k" })); } catch (err) { console.log(`publish rejected: ${(err as Error).message}`); } // Put ORDERS back: Discard Old, no message cap (age and byte limits stay). await jsm.streams.update("ORDERS", { discard: DiscardPolicy.Old, max_msgs: -1 }); ``` #### Go ``` // Switch ORDERS to Discard New. Discard New never drops messages already // stored, so capping it at one message leaves the existing orders in place // and puts the stream instantly over its limit. The next publish is // rejected rather than evicting an older order. s, err := js.Stream(ctx, "ORDERS") if err != nil { panic(err) } cfg := s.CachedInfo().Config cfg.Discard = jetstream.DiscardNew cfg.MaxMsgs = 1 if _, err := js.UpdateStream(ctx, cfg); err != nil { panic(err) } // This publish hits the full stream and fails with "maximum messages // exceeded" instead of succeeding silently. Handle it in the publisher. if _, err := js.Publish(ctx, "orders.created", []byte(`{"order_id":"ord_8w2k"}`)); err != nil { fmt.Printf("publish rejected: %v\n", err) } // Put ORDERS back: Discard Old, no message cap (age and byte limits stay). cfg.Discard = jetstream.DiscardOld cfg.MaxMsgs = -1 if _, err := js.UpdateStream(ctx, cfg); err != nil { panic(err) } ``` #### Python ``` # Switch ORDERS to Discard New. Discard New never drops stored messages, so # capping it at one message leaves the existing orders in place and puts the # stream instantly over its limit; the next publish is rejected. info = await js.stream_info("ORDERS") config = info.config config.discard = DiscardPolicy.NEW config.max_msgs = 1 await js.update_stream(config) # This publish hits the full stream and is rejected instead of succeeding # silently. Handle it in the publisher. try: await js.publish("orders.created", b'{"order_id":"ord_8w2k"}') except APIError as err: print(f"publish rejected: {err.description}") # Put ORDERS back: Discard Old, no message cap (age and byte limits stay). config.discard = DiscardPolicy.OLD config.max_msgs = -1 await js.update_stream(config) ``` #### Java ``` // Switch ORDERS to Discard New. Discard New never drops stored // messages, so capping it at one message leaves the existing orders // in place and puts the stream instantly over its limit; the next // publish is rejected. StreamInfo si = jsm.getStreamInfo("ORDERS"); jsm.updateStream(StreamConfiguration.builder(si.getConfiguration()) .discardPolicy(DiscardPolicy.New) .maxMessages(1) .build()); // This publish hits the full stream and is rejected instead of // succeeding silently. Handle it in the publisher. try { js.publish("orders.created", "{\"order_id\":\"ord_8w2k\"}".getBytes()); } catch (JetStreamApiException e) { System.out.println("publish rejected: " + e.getMessage()); } // Put ORDERS back: Discard Old, no message cap (age and byte limits // stay). jsm.updateStream(StreamConfiguration.builder(si.getConfiguration()) .discardPolicy(DiscardPolicy.Old) .maxMessages(-1) .build()); ``` #### Rust ``` // Switch ORDERS to Discard New. Discard New never drops messages already // stored, so capping it at one message leaves the existing orders in place // and puts the stream instantly over its limit. The next publish is // rejected rather than evicting an older order. let mut stream = js.get_stream("ORDERS").await?; let mut config = stream.info().await?.config.clone(); config.discard = DiscardPolicy::New; config.max_messages = 1; js.update_stream(config.clone()).await?; // This publish hits the full stream; the ack returns an error instead of // the publish succeeding silently. Handle it in the publisher. let ack = js .publish("orders.created", r#"{"order_id":"ord_8w2k"}"#.into()) .await?; if let Err(err) = ack.await { println!("publish rejected: {err}"); } // Put ORDERS back: Discard Old, no message cap (age and byte limits stay). config.discard = DiscardPolicy::Old; config.max_messages = -1; js.update_stream(config).await?; ``` #### C#/.NET ``` // Switch ORDERS to Discard New. Discard New never drops stored messages, // so capping it at one message leaves the existing orders in place and // puts the stream instantly over its limit; the next publish is rejected. var stream = await js.GetStreamAsync("ORDERS"); var config = stream.Info.Config; config.Discard = StreamConfigDiscard.New; config.MaxMsgs = 1; await js.UpdateStreamAsync(config); // This publish hits the full stream and is rejected instead of // succeeding silently. Handle it in the publisher. try { var ack = await js.PublishAsync(subject: "orders.created", data: new Order(OrderId: "ord_5k1m")); // PublishAsync returns the server's answer; EnsureSuccess turns a // rejection into an exception instead of leaving it unchecked. ack.EnsureSuccess(); } catch (NatsJSApiException e) { output.WriteLine($"publish rejected: {e.Message}"); rejected = true; } // Put ORDERS back: Discard Old, no message cap (age and byte limits stay). config.Discard = StreamConfigDiscard.Old; config.MaxMsgs = -1; await js.UpdateStreamAsync(config); ``` #### C ``` // Switch ORDERS to Discard New. Discard New never drops messages // already stored, so capping it at one message leaves the existing // orders in place and puts the stream instantly over its limit. The // next publish is rejected rather than evicting an older order. const char *order = "{\"order_id\":\"ord_8w2k\"}"; s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { si->Config->Discard = js_DiscardNew; si->Config->MaxMsgs = 1; s = js_UpdateStream(NULL, js, si->Config, NULL, &jerr); } // This publish hits the full stream and fails with "maximum messages // exceeded" instead of succeeding silently. Handle it in the // publisher. if (s == NATS_OK) { if (js_Publish(NULL, js, "orders.created", order, (int) strlen(order), NULL, &jerr) != NATS_OK) printf("publish rejected: %s\n", nats_GetLastError(NULL)); } // Put ORDERS back: Discard Old, no message cap (age and byte limits // stay). if (s == NATS_OK) { si->Config->Discard = js_DiscardOld; si->Config->MaxMsgs = -1; s = js_UpdateStream(NULL, js, si->Config, NULL, &jerr); } ``` The same quiet discard applies to MaxAge and MaxBytes together. The two limits work separately, so whichever is reached first triggers the discard. A seven-day MaxAge does not guarantee seven days of history. If traffic spikes, MaxBytes can be reached first and discard messages that are only hours old. Set MaxBytes for your busiest traffic, not your average, if the age window matters to you. **Whole-stream limits don't balance across subjects.** MaxMsgs, MaxBytes, and MaxAge measure `ORDERS` as a whole, across every subject under `orders.>`. A high volume of `orders.created` counts toward the same ceiling as `orders.shipped`, so Discard Old can discard a shipped order to make room for a created one. When each subject needs its own limit, add a per-subject ceiling with `MaxMsgsPerSubject`: #### CLI ``` #!/bin/bash # Add a per-subject ceiling to ORDERS so one noisy subject cannot evict # another subject's messages. # # MaxMsgs, MaxBytes, and MaxAge all measure the stream as a whole, # across every subject under orders.>. If orders.created floods in, its # messages count toward the same ceiling as orders.shipped, and Discard # Old can drop a shipped order to make room for a created one. # # --max-msgs-per-subject sets MaxMsgsPerSubject: a separate ceiling # applied to each individual subject. Set it to 100000 and every subject # keeps its own most-recent 100000 messages, independent of how loud its # neighbors are. nats stream edit ORDERS --max-msgs-per-subject=100000 # Read the stream back; the per-subject limit now appears alongside the # whole-stream limits. nats stream info ORDERS ``` #### JavaScript/TypeScript ``` // Add a per-subject ceiling so one noisy subject can't evict another's // messages. max_msgs_per_subject keeps the most recent N messages for every // subject independently, alongside the whole-stream limits. const jsm = await jetstreamManager(nc); await jsm.streams.update("ORDERS", { max_msgs_per_subject: 100000 }); console.log("ORDERS now keeps 100000 messages per subject"); ``` #### Go ``` // Add a per-subject ceiling so one noisy subject can't evict another's // messages. MaxMsgsPerSubject keeps the most recent N messages for every // subject independently, alongside the whole-stream limits. s, err := js.Stream(ctx, "ORDERS") if err != nil { panic(err) } cfg := s.CachedInfo().Config cfg.MaxMsgsPerSubject = 100000 if _, err := js.UpdateStream(ctx, cfg); err != nil { panic(err) } fmt.Println("ORDERS now keeps 100000 messages per subject") ``` #### Python ``` # Add a per-subject ceiling so one noisy subject can't evict another's # messages. max_msgs_per_subject keeps the most recent N messages for every # subject independently, alongside the whole-stream limits. info = await js.stream_info("ORDERS") config = info.config config.max_msgs_per_subject = 100000 await js.update_stream(config) print("ORDERS now keeps 100000 messages per subject") ``` #### Java ``` // Add a per-subject ceiling so one noisy subject can't evict // another's messages. maxMessagesPerSubject keeps the most recent N // messages for every subject independently, alongside the // whole-stream limits. StreamInfo si = jsm.getStreamInfo("ORDERS"); StreamConfiguration cfg = StreamConfiguration.builder(si.getConfiguration()) .maxMessagesPerSubject(100000) .build(); jsm.updateStream(cfg); System.out.println("ORDERS now keeps 100000 messages per subject"); ``` #### Rust ``` // Add a per-subject ceiling so one noisy subject can't evict another's // messages. max_messages_per_subject keeps the most recent N messages for // every subject independently, alongside the whole-stream limits. let mut stream = js.get_stream("ORDERS").await?; let mut config = stream.info().await?.config.clone(); config.max_messages_per_subject = 100_000; js.update_stream(config).await?; println!("ORDERS now keeps 100000 messages per subject"); ``` #### C#/.NET ``` // Add a per-subject ceiling so one noisy subject can't evict another's // messages. MaxMsgsPerSubject keeps the most recent N messages for every // subject independently, alongside the whole-stream limits. var stream = await js.GetStreamAsync("ORDERS"); var config = stream.Info.Config; config.MaxMsgsPerSubject = 100000; await js.UpdateStreamAsync(config); output.WriteLine("ORDERS now keeps 100000 messages per subject"); ``` #### C ``` // Add a per-subject ceiling so one noisy subject can't evict // another's messages. MaxMsgsPerSubject keeps the most recent N // messages for every subject independently, alongside the // whole-stream limits. s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { si->Config->MaxMsgsPerSubject = 100000; s = js_UpdateStream(NULL, js, si->Config, NULL, &jerr); } if (s == NATS_OK) printf("ORDERS now keeps 100000 messages per subject\n"); ``` Under Discard Old, a per-subject ceiling discards the oldest message *for that subject* once it fills. Discard New doesn't change that on its own: by default the per-subject limit still rolls, discarding the subject's oldest message rather than rejecting the publish. Making a full subject reject takes a second setting, `DiscardNewPerSubject` (the `discard_new_per_subject` config field), on top of Discard New. With both, a publish past the per-subject ceiling fails with `maximum messages per subject exceeded`, a third rejection string alongside the whole-stream `maximum bytes exceeded` and `maximum messages exceeded`. The field is in [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md). ## Where you are You now have: * an `ORDERS` stream capped at a seven-day MaxAge and a 1 GiB MaxBytes ceiling * the messages from earlier still stored (editing limits doesn't discard messages that already sit within them) * Discard Old in place, so a future limit discards the oldest order, never the newest ## What's next You set *limits* under the default Limits retention policy. The next page covers the policy choice itself: the three retention policies, Limits versus Interest versus WorkQueue, and which one to use when. ## See also * [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md) — every limit option, its type, range, and default. --- # Subject mapping and transforms A [filter](/learn/jetstream/filtering.md) picks which stored messages a consumer sees without ever changing a subject. **Subject mapping** rewrites the subject itself, in three places: * **On the way in** — a stream's *subject transform* rewrites a message's subject as it's stored. * **On the way out** — *republish* re-emits each stored message onto a new subject, so plain core subscribers can watch a stream without a consumer. * **While copying** — a source or a mirror can transform subjects as it pulls from another stream (see [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md)). All three use the same small transform language. ## The transform language A transform is a `source → destination` pair. The source is a subject filter with the usual `*` and `>` wildcards; the destination is a subject template that pulls matched tokens back in by position: * `{{wildcard(1)}}` — the token the first `*` matched (`{{wildcard(2)}}` the second, and so on). The older `$1` form means the same thing. * `{{partition(n, 1)}}` — hash the token the first `*` matched into one of `n` buckets, `0`…`n-1`. The same value always lands in the same bucket, so it's a stable way to shard. * A trailing `>` in the source carries across to a `>` in the destination unchanged — that's how `orders.>` maps to `dash.orders.>`, with every token after `orders` riding along. * `{{split(1, -)}}`, `{{splitfromleft(1, 3)}}` — reshape a single matched token: split it wherever a bare delimiter appears (written without quotes, and it can't be `.`, which already separates tokens), or cut it at a character position. ADR-30 (linked below) has the complete list. You'll mostly reach for `wildcard` and `partition`; `split`, `splitfromleft`/`splitfromright`, and `slicefromleft`/`slicefromright` chop a single token when you need it. You can try a transform without a stream. `nats server mappings` takes the source, the destination, and a subject, and prints what it maps to: ``` nats server mappings "orders.*" "orders.{{wildcard(1)}}.archived" orders.created ``` ``` orders.created.archived ``` ## Rewrite subjects on the way in A stream's **subject transform** rewrites the subject a message is stored under. The stream keeps listening on all its configured subjects: a message whose subject matches the transform's source is stored under the rewritten subject, and any other message is stored under its original subject. A transform rewrites subjects; it never drops a message. **Message flow — A stream's subject transform (animated):** A stream's subject transform rewrites the subject a message is stored under. Messages arrive on ingest.\; as each passes through the transform orders.{{partition(3,1)}}.{{wildcard(1)}}, its subject is rewritten — the customer token is hashed into one of three buckets and carried into the new subject. The same customer always hashes to the same bucket, so acme and globex both land in bucket 1 while hooli goes to 0 and wayne to 2, which lets consumers split the load by bucket. A common use is deterministic partitioning: hash a token into a fixed set of buckets so consumers can split the load by bucket. Leave `ORDERS` alone — this is a throwaway stream that ingests on `ingest.*` and shards each message into one of three buckets by hashing the customer token: #### CLI ``` #!/bin/bash # A subject transform rewrites the subject a message is STORED under, which # is separate from the subjects the stream listens on. Leave ORDERS alone: # this is a throwaway stream that ingests on ingest.* and shards each # message into one of three buckets by hashing the customer token. # --transform-source which incoming subjects to match (wildcards ok) # --transform-destination the stored subject, pulling matched tokens in # {{partition(3,1)}} hash the 1st token into bucket 0, 1, or 2 # {{wildcard(1)}} the 1st * token from the source nats stream add ORDERS-SHARDED \ --subjects "ingest.*" \ --transform-source "ingest.*" \ --transform-destination "orders.{{partition(3,1)}}.{{wildcard(1)}}" \ --defaults ``` #### JavaScript/TypeScript ``` // Create ORDERS-SHARDED with a subject transform. Every message on `ingest.*` // is rewritten before it is stored: `partition(3,1)` shards each customer // (the first wildcard token) into one of three buckets, and `wildcard(1)` // keeps the customer id, so `ingest.acme` becomes `orders.<0-2>.acme`. const info = await jsm.streams.add({ name: "ORDERS-SHARDED", subjects: ["ingest.*"], subject_transform: { src: "ingest.*", dest: "orders.{{partition(3,1)}}.{{wildcard(1)}}", }, }); console.log(`Created stream: ${info.config.name}`); ``` #### Go ``` // Create a stream that rewrites subjects as it stores them. The transform // shards each customer into one of three buckets: partition(3,1) hashes the // first wildcard token into 0, 1, or 2, and wildcard(1) keeps that token. // So "ingest.alice" might land on "orders.2.alice". stream, err := js.CreateStream(ctx, jetstream.StreamConfig{ Name: "ORDERS-SHARDED", Subjects: []string{"ingest.*"}, SubjectTransform: &jetstream.SubjectTransformConfig{ Source: "ingest.*", Destination: "orders.{{partition(3,1)}}.{{wildcard(1)}}", }, }) if err != nil { panic(err) } fmt.Printf("Created stream: %s\n", stream.CachedInfo().Config.Name) ``` #### Python ``` # Create a stream that rewrites subjects as it stores them. Incoming # ingest. messages become orders.., where # partition(3,1) hashes the first wildcard token into one of three # buckets — sharding each customer into a fixed partition. info = await js.add_stream( StreamConfig( name="ORDERS-SHARDED", subjects=["ingest.*"], subject_transform=SubjectTransform( src="ingest.*", dest="orders.{{partition(3,1)}}.{{wildcard(1)}}", ), ) ) print(f"created stream {info.config.name}") ``` #### Java ``` // Rewrite each ingested subject as it lands in the stream. The template // shards each customer into one of three buckets: partition(3,1) hashes // the first wildcard token into 0, 1 or 2, and wildcard(1) keeps that // same token as the trailing part of the stored subject. SubjectTransform st = new SubjectTransform( "ingest.*", "orders.{{partition(3,1)}}.{{wildcard(1)}}"); StreamConfiguration sc = StreamConfiguration.builder() .name("ORDERS-SHARDED") .subjects("ingest.*") .subjectTransform(st) .build(); StreamInfo info = jsm.addStream(sc); System.out.println("Created stream: " + info.getConfiguration().getName()); ``` #### Rust ``` // Create a stream that rewrites each incoming subject as it is stored. Every // message published to `ingest.*` is transformed into // `orders..`, where `partition(3, 1)` hashes the first // wildcard token into one of three buckets (0, 1, or 2). This spreads orders // across three shard subjects so consumers can filter by bucket. let stream = js .create_stream(stream::Config { name: "ORDERS-SHARDED".to_string(), subjects: vec!["ingest.*".to_string()], subject_transform: Some(stream::SubjectTransform { source: "ingest.*".to_string(), destination: "orders.{{partition(3,1)}}.{{wildcard(1)}}".to_string(), }), ..Default::default() }) .await?; println!("Created stream: {}", stream.cached_info().config.name); ``` #### C#/.NET ``` // Ingest orders on `ingest.` and rewrite the subject on the way // in. `partition(3, 1)` hashes the first wildcard token into one of three // buckets, so each customer always shards to the same bucket. var stream = await js.CreateStreamAsync(new StreamConfig(name: "ORDERS-SHARDED", subjects: ["ingest.*"]) { SubjectTransform = new SubjectTransform { Src = "ingest.*", Dest = "orders.{{partition(3,1)}}.{{wildcard(1)}}", }, }); output.WriteLine($"Created {stream.Info.Config.Name}"); ``` #### C ``` // Create a stream that rewrites subjects as it stores them. The // transform shards each customer into one of three buckets: // partition(3,1) hashes the first wildcard token into 0, 1, or 2, // and wildcard(1) keeps that token. So "ingest.alice" might land on // "orders.2.alice". jsStreamConfig sc; const char *subjects[] = {"ingest.*"}; jsStreamConfig_Init(&sc); sc.Name = "ORDERS-SHARDED"; sc.Subjects = subjects; sc.SubjectsLen = 1; sc.SubjectTransform.Source = "ingest.*"; sc.SubjectTransform.Destination = "orders.{{partition(3,1)}}.{{wildcard(1)}}"; s = js_AddStream(&si, js, &sc, NULL, &jerr); if (s == NATS_OK) printf("Created stream: %s\n", si->Config->Name); ``` Publish a few customers' orders, and each lands under `orders..`: ``` nats pub ingest.acme "an order" nats pub ingest.globex "an order" nats pub ingest.wayne "an order" nats stream subjects ORDERS-SHARDED ``` ``` Subject │ Count orders.1.acme │ 1 orders.1.globex │ 1 orders.2.wayne │ 1 ``` Each customer hashes to the same bucket every time, so a consumer filtered to one bucket — `orders.1.>` here — always reads the same share of customers. Add or remove the transform on an existing stream with `nats stream edit --transform-source/--transform-destination`, or `--no-transform` to clear it. The transform rewrites only what gets stored from the moment it's set. It doesn't touch messages already in the stream; the Pitfalls cover what that means. ## Republish to live subjects **Republish** re-emits every message a stream stores onto a second subject, in real time. Core subscribers listen on that subject and see the data flow by without creating a consumer or replaying anything — a low-cost way to feed a dashboard or a monitor from a stored stream. #### CLI ``` #!/bin/bash # Republish re-emits every message the stream stores onto a second subject, # in real time. A plain core subscriber on that subject sees the data flow # by without creating a consumer. This adds republish to ORDERS; clear it # later with: nats stream edit ORDERS --no-republish nats stream edit ORDERS \ --republish-source "orders.>" \ --republish-destination "dash.orders.>" ``` #### JavaScript/TypeScript ``` // Add a republish rule so every message stored under `orders.>` is also // published live to a matching `dash.orders.>` subject that dashboards can // subscribe to, then apply the update. const config = info.config; config.republish = { src: "orders.>", dest: "dash.orders.>", // headers_only: true, // republish only the headers, not the message body }; const updated = await jsm.streams.update("ORDERS", config); console.log(`republish dest: ${updated.config.republish?.dest}`); ``` #### Go ``` // Read the current config, add a republish rule, and push the update. // Every message stored on "orders.>" is also echoed live to "dash.orders.>" // so a dashboard can subscribe without touching the stream. cfg := stream.CachedInfo().Config cfg.RePublish = &jetstream.RePublish{ Source: "orders.>", Destination: "dash.orders.>", // HeadersOnly: true, // republish only the headers, not the body } updated, err := js.UpdateStream(ctx, cfg) if err != nil { panic(err) } fmt.Printf("Republish destination: %s\n", updated.CachedInfo().Config.RePublish.Destination) ``` #### Python ``` # Republish every stored orders.> message onto a dash.orders.> subject, so # core subscribers can watch the stream live without a consumer. # Set headers_only=True to republish just the headers, not the body. config.republish = RePublish( src="orders.>", dest="dash.orders.>", # headers_only=True, ) updated = await js.update_stream(config) print(f"republishing to {updated.config.republish.dest}") ``` #### Java ``` // Republish every message ORDERS stores onto a parallel subject that // ordinary core subscribers can watch, without them touching the stream. Republish rp = Republish.builder() .source("orders.>") .destination("dash.orders.>") // .headersOnly(true) // republish only the headers, not the body .build(); StreamConfiguration updated = StreamConfiguration.builder(current) .republish(rp) .build(); StreamInfo info = jsm.updateStream(updated); System.out.println("Republish destination: " + info.getConfiguration().getRepublish().getDestination()); ``` #### Rust ``` // Republish a copy of every stored message onto a second subject so a // dashboard can subscribe live without touching the stream. Set // `headers_only: true` to republish just the headers (subject, sequence, // size) and skip the payload. config.republish = Some(async_nats::jetstream::stream::Republish { source: "orders.>".to_string(), destination: "dash.orders.>".to_string(), headers_only: false, }); let info = js.update_stream(&config).await?; println!( "Republishing to: {}", info.config .republish .as_ref() .map_or("", |r| r.destination.as_str()) ); ``` #### C#/.NET ``` // Mirror every stored order onto a `dash.orders.>` subject as it lands. // Read the current config, add the republish rule, and send it back. var config = stream.Info.Config; config.Republish = new Republish { Src = "orders.>", Dest = "dash.orders.>", // HeadersOnly = true, }; var updated = await js.UpdateStreamAsync(config); output.WriteLine($"Republishing to {updated.Info.Config.Republish!.Dest}"); ``` #### C ``` // Read the current config, add a republish rule, and push the // update. Every message stored on "orders.>" is also echoed live to // "dash.orders.>" so a dashboard can subscribe without touching the // stream. jsRePublish rp; s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { jsRePublish_Init(&rp); rp.Source = "orders.>"; rp.Destination = "dash.orders.>"; // rp.HeadersOnly = true; // republish only the headers, not the body si->Config->RePublish = &rp; s = js_UpdateStream(&updated, js, si->Config, NULL, &jerr); } if (s == NATS_OK) printf("Republish destination: %s\n", updated->Config->RePublish->Destination); ``` Now a plain core subscription sees each order as it lands: ``` nats sub "dash.orders.>" ``` ``` [#1] Received on "dash.orders.created" Nats-Stream: ORDERS Nats-Subject: orders.created Nats-Sequence: 5 Nats-Time-Stamp: 2026-05-22T11:02:00Z Nats-Last-Sequence: 4 {"order_id":"ord_5xk1","customer":"initech","total_cents":1500,"ts":"2026-05-22T11:02:00Z"} ``` Each republished message carries five headers describing where it came from: `Nats-Stream`, `Nats-Subject` (the original stored subject), `Nats-Sequence`, `Nats-Time-Stamp`, and `Nats-Last-Sequence` — the stream sequence of the previous message *on the same subject*, or `0` if there wasn't one, so a subscriber following one subject can tell it missed something. Any headers the publisher set are carried through too. `--republish-headers` sends the headers without the bodies, for subscribers that only need to know something changed; each message then also carries a `Nats-Msg-Size` header with the omitted body's byte count. Clear republish with `nats stream edit ORDERS --no-republish`. ## Transform while copying When a stream **sources** from or **mirrors** another stream, each source can carry its own subject transform, so you can re-namespace messages as you aggregate them — for example prefixing every region's orders as they merge into one stream. That belongs with the copying mechanics, so it's covered on [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md). ## Not the same as account subject mapping NATS also has [account-level subject mapping](/learn/core-nats/subject-mapping.md), configured on the server, not on a stream. It reroutes *core* subjects before they're ever published into a stream. That's a server-configuration topic, separate from the stream transforms on this page. ## Pitfalls A few ways subject mapping trips people up. **A transform that drops a token a consumer filters on.** A destination template keeps only the tokens you name. Rewrite `orders.*` to `orders.archived` and drop the `{{wildcard(1)}}`, and every order is stored under the one subject `orders.archived`, where a consumer can no longer filter by type. Keep every token a downstream filter needs in the destination. **Republish is not a consumer.** A republished subject is plain core NATS: fire-and-forget, no storage, no acks, no replay. A subscriber that's down misses whatever was republished while it was away, and nothing redelivers it. Use republish to watch a stream live; use a consumer when a reader has to catch up on what it missed. **A republish destination that loops back into the stream.** The destination can't overlap the stream's own subjects. Point a stream that ingests `orders.>` at destination `orders.dash.>` and the server rejects it as a cycle (error `10052`). Send republished messages to a separate subject space — `dash.orders.>`, not something under `orders.>` — and watch for loops that span two streams in one account, which the server can't always catch. **Editing a transform doesn't rewrite what's already stored.** Adding or changing a subject transform only affects messages stored after the change. Messages already in the stream keep the subjects they were stored under. To re-namespace existing data, copy it into a new stream that sources from the old one with the transform applied. **A partition count is fixed once consumers depend on it.** Consumers filter on bucket numbers (`orders.0.>`, `orders.1.>`, …). Change `partition(3, …)` to `partition(4, …)` later and the same customer can hash to a different bucket, so a consumer's filter quietly starts covering a different set. Pick the bucket count up front, the way you would for any partitioned system. ## Where you are `ORDERS` is unchanged by the transform work — that ran on a throwaway `ORDERS-SHARDED` stream. If you added republish to `ORDERS` above, that's the one change; it's additive, and `nats stream edit ORDERS --no-republish` clears it. You saw the three places JetStream rewrites a subject: * a **subject transform** that changes the stored subject on the way in * **republish** that re-emits stored messages onto a live subject on the way out * the per-source transforms that rewrite subjects while copying, over on [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) ## What's next The next page covers [per-message TTL](/learn/jetstream/message-ttl.md): giving a single message a shorter lifespan than the rest of the stream. ## See also * [Reference → Create Stream](/reference/jetstream/api/stream/create.md) — the `subject_transform`, `republish`, and per-source `subject_transforms` fields. * [ADR-36: Subject Mapping Transforms in Streams](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-36.md) — where a transform attaches to a stream, a source, or a mirror. * [ADR-30: Subject Transform](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-30.md) — the transform template functions and their exact behavior. * [ADR-28: RePublish](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-28.md) — the republish headers and the loop-prevention rules. * [Filtering what you consume](/learn/jetstream/filtering.md) — narrowing a consumer's view without changing subjects. * [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) — transforms applied while aggregating streams. --- # Surviving node loss Every page so far ran against a single `nats-server` on your laptop. That's good for learning, but it's not how you run orders in production. A single server is a single point of failure. If that machine loses its disk, crashes, or reboots, your `ORDERS` stream is at risk. This page explains what protects against that, and why you have to turn the protection on yourself. It introduces two ideas: **replicas** (how many copies of the stream exist) and **storage durability** (whether a copy survives a restart). ## Replicas `nats stream info ORDERS` reports `Replicas: 1`. You saw this on the [Your first stream](/learn/jetstream/your-first-stream.md) page and haven't changed it since. `Replicas: 1` means the stream exists on exactly one server. There's one copy of the message log and nothing else. This is **R=1**. R=1 has no fault tolerance. Lose that one server and you lose the stream: every message and every consumer's position. On a laptop that's fine. In production it risks data loss. The fix is more copies. Set the stream to keep three copies across three servers, and the loss of any one server costs you nothing. This is **R=3**, and it's the production floor. R=3 tolerates one server failure. With three copies, losing one leaves two, and two out of three is a **majority**. The stream keeps serving reads and writes through the failure, with no data loss and no manual recovery. **Message flow — R=3 survives losing one server (animated):** A stream's replication factor decides whether it survives a server loss. R=1 keeps one copy on one server; R=3 keeps three copies across three servers. When a server goes down, the R=1 stream had no second copy, so it is gone, while the R=3 stream still holds a majority of its copies and keeps serving through the loss. The majority matters because the replicas have to agree on the order of messages, and they settle that order by majority vote — so the group keeps working as long as more than half its members are reachable. That voting rule is called Raft. The [Clustering & Replication](/learn/clustering/.md) deep dive walks through it on a real cluster. You can go higher. **R=5** keeps five copies and tolerates two simultaneous server failures. Five is the maximum a stream supports. Most production streams run R=3; R=5 is for state you can't afford to re-derive. ## The stream has a leader With more than one copy, one of them is in charge. The replicas elect a **leader**, and the other copies are **followers**. Every write goes through the leader. When you publish into `ORDERS`, the leader assigns the sequence number, stores the message, and only returns the `PubAck` once a majority of replicas have the message. That's what makes the `PubAck` on an R=3 stream a real durability promise: the message survives the loss of any single server. Reads work differently from writes. Writes flow through the stream leader, but each consumer has a leader of its own that can sit on any of the stream's replicas, so delivery spreads across those replica servers instead of piling on one. A [Direct Get](/learn/jetstream/get-direct.md) spreads load further: any of the stream's replicas can answer it directly, leader or not. The full picture — stream leaders, consumer leaders, and the cluster-wide meta leader — is the [Clustering & Replication](/learn/clustering/.md) deep dive's job. If the leader's server dies, the remaining replicas elect a new leader from among themselves, automatically. Writes pause for the short window of that election, then resume. No acked message is lost, because every message that received its `PubAck` was already stored on a majority before the old leader failed. **Message flow — A follower takes over when the leader fails (animated):** A replicated stream keeps one leader that takes writes and copies them to its followers. The leader copies each write to followers n2 and n3; once two of the three hold it — a majority — the PubAck returns and the order is safe. When the leader fails, the followers elect n2 as the new leader and writes resume, with no acked order lost. A publish that was still on its way when the old leader crashed is a different case. The message had been sent, but it hadn't reached a majority of replicas yet, so it was never acked. That write is lost, and the client never gets a `PubAck` for it. The fix is the client's job: when a `PubAck` doesn't come back, treat the publish as failed and send it again. The durability promise covers acked messages, not ones that were still on their way when a server died. One more failure case is worth covering. If so many servers are down that no majority remains (two of three gone), the group can't elect a leader. Writes are blocked until enough replicas come back. The stream would rather stop than accept writes it couldn't safely copy to a majority. ## Storage durability The replica count sets how many copies exist. Storage type sets whether a copy survives a server restart. `nats stream info ORDERS` reports `Storage: File`. This is the default, and it's the durable one. File storage writes messages to disk, so a server can reboot and read its copy back intact. The alternative is **Memory** storage. A memory stream keeps its messages in RAM only. It's faster, but it's not durable: restart that server and its copy is gone. Memory storage suits data you can afford to lose on a restart. It's not suitable for an order log. Storage type is a property of the whole stream, not of individual replicas. An R=3 stream is all file or all memory; you can't mix a disk copy with two RAM copies. Replicas and storage are independent choices that combine. R=3 file storage is the durable, fault-tolerant default for important streams: three copies, each on disk. R=3 memory storage survives a single server crash through its replicas but loses everything if the whole group restarts at once. File storage doesn't sync every write to disk right away, either. A write can still sit unsynced after it gets a `PubAck`, long enough that an OS crash could lose it, no full restart required. [Replication and R=3](/learn/clustering/replication-and-r3.md) covers how a write gets that `PubAck`, and this gap in what it guarantees. ## Consumers replicate too A consumer also has state worth protecting: how far it has read and which messages are still waiting for an ack. On an R=3 stream, the `shipping` consumer's state is copied the same way, so a worker pool keeps its place through a server failure. By default a durable consumer takes its stream's replica count. On a limits-retention stream like `ORDERS` you can give a consumer fewer replicas than its stream when its state is cheap to rebuild, but never more. On interest and work-queue streams the consumer's replica count must match the stream's. The full set of consumer replica and storage options is documented in [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md). Here the consumer just takes its count from the stream. ## What replicas buy, and what they cost Replicas are a durability control. Raising the count buys fault tolerance and costs load. Be exact about what it does and doesn't do, because adding replicas is not how you scale throughput. **Stream replicas** (R=3, R=5): * Survive node loss. More copies tolerate more failures. * Spread reads. Writes flow through the stream leader, but a consumer's own leader can sit on any of the stream's replicas, so delivery spreads across them; and a [Direct Get](/learn/jetstream/get-direct.md) is answered by any of the stream's replicas directly. Read work doesn't funnel through the stream leader the way writes do. * Cost load across the cluster. Every replica stores the full log, and every write is copied to a majority before its `PubAck`. R=3 is roughly three times the storage and write traffic of R=1. * Don't scale writes. Every write still goes through the one leader, so a higher count lowers peak write throughput rather than raising it. **Consumer replicas:** * Survive node loss. A replicated consumer keeps its read position and pending acks when its leader's server dies, and the group elects a new consumer leader. * Don't scale delivery. One consumer leader does all the work; the followers only stand by to take over. Extra replicas add replication load without adding throughput. When you need more throughput, the tool isn't a replica: * To scale a consumer, add workers — [Scaling a consumer](/learn/jetstream/worker-pool.md). * To scale writes past one leader, split subjects across streams — [Subject mapping](/learn/jetstream/subject-mapping.md). * To spread those streams across servers, see [Clustering & Replication](/learn/clustering/.md). ## Turning R=3 on On a cluster you raise the replica count with one command: #### CLI ``` #!/bin/bash # Raise the ORDERS stream to three replicas (R=3), the production floor. # Three copies live on three servers; the loss of any one server costs # no data, and the stream keeps serving reads and writes. # # This command REQUIRES a clustered NATS deployment of at least three # servers. On a single-node server it is rejected, because there are no # three servers to hold the three copies. nats stream edit ORDERS --replicas=3 # Inspect the result. On a cluster, the report gains a Cluster section # naming the leader and the replicas, alongside the Replicas count. nats stream info ORDERS ``` #### C ``` // Raise the ORDERS stream to three replicas (R=3). This needs a real // cluster behind it: a single server rejects Replicas=3 because there // aren't three servers to hold the three copies. jsStreamConfig cfg; const char *subjects[] = {"orders.>"}; jsStreamConfig_Init(&cfg); cfg.Name = "ORDERS"; cfg.Subjects = subjects; cfg.SubjectsLen = 1; cfg.Replicas = 3; s = js_UpdateStream(&si, js, &cfg, NULL, &jerr); if (s == NATS_OK) { printf("Stream %s now has %" PRId64 " replicas\n", si->Config->Name, si->Config->Replicas); } ``` This command needs a real cluster behind it. A single server rejects `--replicas=3`, because there aren't three servers to hold the three copies. Running it on your laptop returns an error rather than a three-copy stream, which is expected. Standing up the three-node cluster, watching a leader get elected, and killing a server to see the failover is covered separately in the [Clustering & Replication](/learn/clustering/.md) deep dive, which picks up where this page leaves off. The full set of placement controls (which servers a stream lands on, tag-based steering, and per-account replica limits) is documented in the [Clustering & Replication](/learn/clustering/.md) deep dive. We change only the replica count here. ## Pitfalls These are the failures that show up when a stream first loses a server. **Trusting R=1 in production.** An R=1 stream has exactly one copy. Lose that server's disk and the `ORDERS` stream is gone: every message, every consumer's position. There's no recovery, because there was no second copy to recover from. R=3 is the production floor; R=1 belongs on a laptop. Before you trust a stream with real orders, check its replica count rather than assuming it. #### CLI ``` #!/bin/bash # Before trusting ORDERS in production, confirm how many copies exist. # nats stream info reports the Replicas count; on a single-node laptop # this is 1, which means no fault tolerance. nats stream info ORDERS # Pull out just the replica count for a fast, scriptable check. A value # of 1 is R=1 — a single point of failure. Treat anything below 3 as a # warning sign for a stream that holds real orders. nats stream info ORDERS --json | grep '"num_replicas"' ``` #### C ``` // Before trusting ORDERS in production, confirm how many copies // exist. Stream info reports the replica count; on a single-node // laptop this is 1, which means no fault tolerance. s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { // A value of 1 is R=1, a single point of failure. Treat // anything below 3 as a warning sign for a stream that holds // real orders. printf("ORDERS replicas: %d\n", (int) si->Config->Replicas); if (si->Config->Replicas < 3) printf("warning: fewer than 3 replicas, no fault tolerance\n"); } ``` **Setting an even replica count.** Fault tolerance comes from a majority, and a majority needs an odd number. R=2 still has a single point of failure: lose either copy and you're left with one server out of two, which can't form a majority, so writes block. R=4 tolerates only one loss, the same as R=3, while paying for a fourth copy. Use odd counts: R=3 for the production floor, R=5 for state you can't re-derive. Five is the maximum a stream supports. **Reading failover from a single-server demo.** A clean run on one server proves nothing about fault tolerance: replicas only exist across servers, so a one-server laptop can't elect a leader or survive a server loss. Don't conclude a stream is fault-tolerant until you've proven failover on a real cluster, which the [Clustering & Replication](/learn/clustering/.md) deep dive walks through end to end. ## Where you are Nothing about your local `ORDERS` stream changed on this page. It's still R=1 file storage on one server, exactly as you've run it throughout the chapter. What changed is your mental model: * **R=1** is a single point of failure: fine for learning, dangerous in production. * **R=3** is the production floor: three copies, tolerates one server loss, no data lost. * **File** storage survives a restart; **Memory** storage doesn't. * The stream has a **leader** that all writes flow through, and a new one is elected automatically when a server dies. * Replicas are a **durability** knob, not a throughput one: they survive node loss but don't scale writes or delivery. Throughput comes from workers and partitioning, not from more copies. ## What's next The next page returns to the publisher: **advanced publishing** — the async, atomic-batch, and fast-ingest modes for when one-at-a-time publishing isn't enough. ## See also * [Operate → Clustering & Replication](/learn/clustering/.md) — stand up a real three-node cluster and watch leader election and failover. * [Reference → Stream Configuration](/reference/jetstream/api/stream/create.md) — every storage and replica option and its valid range. * [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md) — consumer-level replica and storage overrides. --- # Where to go next This chapter started with a publisher and no one guaranteed to be listening. It ends with an `ORDERS` stream, a handful of consumers reading it at their own pace, and a mirror keeping a permanent copy. This page gathers the model you built into one place. It then points you at the chapters and Reference that take it further. ## The core model Every page in this chapter built on the same three ideas. A **stream** is a log of messages kept on the server. You publish to a subject. The server adds the message to any stream that captures that subject. The message stays there, ready to be read again, until a limit removes it. A consumer is a position marker over a stream. It tracks which messages a reader has seen, separately from every other consumer. Two consumers on the same stream can read the same messages at different positions without affecting each other. An ack is the reader's way of saying a message is handled. Until the consumer acks, the message stays open, and the server sends it again after a timeout. A stored message has not yet been processed. Everything else in this chapter builds on stream, consumer, and ack. That includes filtering, worker pools, retention, and mirrors. ## Where the details live This chapter teaches the ideas and is not tied to a server version. The exact flags, defaults, and ranges live in **Reference**, which is tied to a version and covers every option. When you need the precise type of a config field, or the full list of consumer options, look there. The [Reference root](/reference/.md) is the way in. The pointers to Reference throughout this chapter all lead into it. ## Sibling deep dives This was the first deep dive. The others build on the same foundation, so the stream, consumer, and ack model carries into them. The [Key-Value deep dive](/learn/key-value/.md) shows how a key-value bucket is a stream underneath. The keys become subjects. The history becomes sequence numbers. A watch is a consumer. Everything you learned about retention and limits applies directly. The [Object Store deep dive](/learn/object-store/.md) does the same for large files. An object is split across many messages in a stream, then put back together when you read it. The stream is still where the data lives. The [Clustering & Replication deep dive](/learn/clustering/.md) goes deeper than this chapter's single page on surviving node loss. It covers how `R=3` picks a leader, how the server decides where to place the stream, and what happens when a node fails. The [Monitoring deep dive](/learn/monitoring/.md) covers how to watch a stream and its consumers in production. It walks through advisories, health endpoints, and the numbers that tell you a consumer is falling behind. The [Backup & Recovery deep dive](/learn/backup-recovery/.md) covers the day-to-day operations: saving a stream to a snapshot, restoring it, and using the mirrors you met on the [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) page to recover from a disaster. ## Where you are This is the end of the chapter. This page adds nothing new to the running example. The `ORDERS` stream, its consumers, and its mirror are still running in your session as you left them on the previous page. You can keep experimenting with them, or remove them with `nats stream rm ORDERS` when you're done. You now have the core model. A stream stores messages. A consumer reads them at its own pace. An ack confirms a message was handled. That model sits under every other JetStream feature. ## Production checklist Every page in this chapter closed with a Pitfalls section. This gathers the action items from all of them in one place. Each group links back to the page that explains it. ### Your first stream — see [Pitfalls](/learn/jetstream/your-first-stream.md#pitfalls) * Stay on plain pub-sub when the next message supersedes the last; reach for a stream only when a missed message has consequences. * Set at least one limit (`MaxAge`, `MaxBytes`, or `MaxMsgs`) so an unbounded stream never fills the disk. * Pick the stream name deliberately the first time; there's no rename, only delete-and-recreate. * Choose the retention policy before messages flow; switching to or from WorkQueue on a live stream is rejected. ### Publishing — see [Pitfalls](/learn/jetstream/publishing.md#pitfalls) * Read the `PubAck` back; a plain `nats pub` line isn't proof the message was stored. * Give every retryable publish a stable `Nats-Msg-Id` so a retry doesn't double-store. * Wait for delivery and ack before acting on a business outcome; a `PubAck` means stored, not processed. ### Reading back — see [Pitfalls](/learn/jetstream/reading-back.md#pitfalls) * Reach for `--all` only when you want the whole history; sample the tail with `--last`, `--since`, or `--start-sequence`. * Use a named, durable consumer for any read you must resume after a disconnect; an ephemeral one restarts from sequence 1. * Edit a durable consumer to change it; recreating with a new config is rejected as already-exists. * Confirm `--all` versus `--new` matches the question (backlog or live traffic) before you run the command. * Pair `--all` with `--terminate-at-end` for a one-shot replay; on its own it drains the backlog then blocks waiting for more. ### Delivery and acknowledgment — see [Pitfalls](/learn/jetstream/delivery-and-acknowledgment.md#pitfalls) * Set Ack Wait longer than your slowest handler, with headroom, to avoid a redelivery storm. * Ack on every success path, and term a genuinely unprocessable message so it stops coming back. * Ack each delivery exactly once, in one place in your handler; a second ack is a no-op the server ignores. ### Filtering — see [Pitfalls](/learn/jetstream/filtering.md#pitfalls) * Confirm the filter matches a subject the stream stores before assuming an empty pull means an empty stream. * Decide retention through stream limits, not filters; a filter narrows a view, it never deletes messages. * Keep multiple filter subjects on one consumer disjoint; the server rejects overlap inside a single consumer. ### Ack responses and redelivery — see [Pitfalls](/learn/jetstream/acknowledgment.md#pitfalls) * Nak a transient failure with a delay, or set a backoff, instead of a bare nak that loops at network speed. * Term a poison message the moment the code knows no attempt will succeed, rather than burning the delivery budget. * Subscribe to the max-deliveries advisory so a dropped message doesn't vanish unnoticed; JetStream has no dead-letter queue. * Raise AckWait or send in-progress for long jobs so a slow handler doesn't trigger double work. ### Pull consumers — see [Pitfalls](/learn/jetstream/pull-consumers.md#pitfalls) * Treat an empty fetch as "nothing right now" and loop; never as an error that crashes the worker. * Always set an `expires` on a fetch so a quiet stream returns control instead of stalling. * Keep `MaxAckPending` at or above your batch size so it doesn't throttle throughput. * Pair `batch` with `max_bytes` so a single pull is bounded by size as well as count. ### Scaling a consumer — see [Pitfalls](/learn/jetstream/worker-pool.md#pitfalls) * Key every side effect by `order_id` so a redelivered message is a no-op, not a double shipment. * Size `MaxAckPending` to at least your worker count, with headroom; the cap is shared across the whole pool. * Tune `AckWait` to real processing time so a crashed worker's message recovers without redelivering honest work. ### Ordered consumers — see [Pitfalls](/learn/jetstream/ordered-consumer.md#pitfalls) * Reach for an ordered consumer only for a read you can restart from the top; for work that must land once, use a named consumer with explicit ack. * Split work across processes with a named consumer and a worker pool; two ordered consumers each read the whole stream instead of sharing it. * Don't build a resumable job on an ordered consumer; its cursor is disposable by design and starts over after a crash. ### Priority groups — see [Pitfalls](/learn/jetstream/priority-groups.md#pitfalls) * Run one priority group per consumer; passing more than one silently uses only the first. * Drive failover with `min_pending` or `min_ack_pending`; the ADR-42 `failover` timer isn't shipped yet. * Lean on explicit acks and idempotent handlers, not the pin, to keep work from doubling up; the pin is not a lock. * Keep each pull's `expires` comfortably under `--pinned-ttl` so a pinned client renews in time. ### Pausing a consumer — see [Pitfalls](/learn/jetstream/pausing.md#pitfalls) * Pause with a duration like `1h` so the deadline can never land in the past and no-op. * Size the stream for the longest pause you expect; publishes keep landing and count against the limits while a consumer sleeps. ### Shaping the stream — see [Pitfalls](/learn/jetstream/shaping-the-stream.md#pitfalls) * Switch to Discard New when you need backpressure; Discard Old drops the oldest message silently. * Size `MaxBytes` for your peak, not your average, when the age window matters; either limit can fire first. * Add `MaxMsgsPerSubject` when each subject deserves its own retention; whole-stream limits let one noisy subject starve a quiet one. ### Retention policies — see [Pitfalls](/learn/jetstream/retention-policies.md#pitfalls) * Create a new stream to move to or from WorkQueue; the server locks that change on a live stream. * Give each WorkQueue consumer a disjoint filter, or share one consumer as a pool; overlapping consumers are rejected. * Watch disk on an Interest stream with a stopped consumer; interest means messages wait for every registered consumer's ack. * Treat a live switch from Limits to Interest as destructive; it re-applies to stored messages and can drop already-acked history. ### Per-message TTL — see [Pitfalls](/learn/jetstream/message-ttl.md#pitfalls) * Confirm `Allows Per-Message TTL` is on before relying on `Nats-TTL`; a header on an opted-out stream fails the publish. * Size a TTL to outlast the slowest healthy consumer's lag; the clock deletes the stored copy whether or not it was read. * Set `SubjectDeleteMarkerTTL` only when consumers must learn a value expired, and keep it at or below your shortest TTL. ### Altering stream state — see [Pitfalls](/learn/jetstream/altering-stream-state.md#pitfalls) * Use `nats stream purge` to clear messages and `nats stream rm` to delete the stream; don't swap the two. * Treat a sequence as a stable address that may hold no message; never assume messages run `1..N` without gaps. * Set `DenyPurge` on a stream that matters, and keep a mirror; a purge can't be undone. ### Surviving node loss — see [Pitfalls](/learn/jetstream/surviving-node-loss.md#pitfalls) * Confirm the replica count before trusting a stream with real orders; R=1 has no copy to recover from. * Use odd replica counts: R=3 for the production floor, R=5 for state you can't re-derive. * Prove failover on a real cluster; a green single-node run can't show leader election or a node loss. ### Advanced publishing — see [Pitfalls](/learn/jetstream/advanced-publishing.md#pitfalls) * Collect and check every `PubAck` on an async publish; add `Nats-Expected-Last-Subject-Sequence` when order matters so a retry fails fast. * Treat the final `PubAck` as the only proof an atomic batch committed; a sequence gap, an oversized batch, or a ten-second idle drops the whole batch. * Keep atomic publishing off a stream set to `PersistMode: async`; the server rejects it, though fast-ingest batches are fine there. * Choose `gap: ok` only when a lost message is acceptable; use `gap: fail` or an atomic batch for anything you can't lose. ### Mirrors and sources — see [Pitfalls](/learn/jetstream/mirrors-and-sources.md#pitfalls) * Publish to the upstream stream, not the mirror; a mirror captures no subjects, so a plain publish lands in the origin, and forcing the mirror by name errors with `expected stream does not match`. * Read the `Lag` field before assuming a mirror is current; a mirror is eventually consistent, not synchronous. * Pick `filter_subject` or `subject_transforms` on one entry, never both; the server rejects a config that sets both. * Verify each export type for cross-domain sourcing; a wrong type lets the mirror silently never catch up. ### Reading messages directly — see [Pitfalls](/learn/jetstream/get-direct.md#pitfalls) * Confirm `Direct Get: true`, or set `--allow-direct`, before a direct read; without it the request just times out. * Keep direct reads off read-after-write checks; they can answer from a trailing replica, so read with `nats stream get` against the leader there. * Use a consumer, not a direct batch, when you need to follow the stream; a batch reads the backlog once and stops. ### Subject mapping — see [Pitfalls](/learn/jetstream/subject-mapping.md#pitfalls) * Keep every token a downstream consumer filters on in the transform's destination; dropping a `{{wildcard}}` collapses messages under one subject. * Use a consumer when a reader has to catch up on what it missed; republish is fire-and-forget core NATS with no storage or replay. * Point a republish destination at a separate subject space, not one under the stream's own subjects; an overlap is rejected as a cycle (`10052`). * Re-namespace existing data by sourcing it into a new stream with the transform; editing a transform leaves already-stored messages untouched. * Pick the partition count up front; raising it later re-hashes keys, so a consumer's filter quietly starts covering a different set. ### Stream and consumer policies — see [Pitfalls](/learn/jetstream/policies.md#pitfalls) * Settle the fixed policies — storage, persist mode, deliver, ack, and replay — before creating anything durable; `stream edit` and `consumer edit` can't change them. * Plan the data move when a fixed policy must change; recreating a stream needs a mirror or a re-publish, and a recreated consumer starts fresh, not where the old one stopped. * Treat `--deliver new` as a one-time start position, not a per-restart skip; a durable resumes from its saved position on reconnect. * Check whether "last per subject" means the deliver policy or the Direct Get flag before copying a command. ## See also * [Reference](/reference/.md) — every config field, flag, default, and error code, versioned and exhaustive. * [Key-Value deep dive](/learn/key-value/.md) — the next chapter built on the same stream foundation. * [Clustering & Replication deep dive](/learn/clustering/.md) — the detail behind "surviving node loss." --- # Scaling a consumer On the [previous page](/learn/jetstream/pull-consumers.md), one worker consumed the `shipping` consumer: a continuous pull loop that ships each order and acks. That kept up while Acme shipped a few orders an hour. Then Acme's order volume climbed. Orders arrive faster than one worker can ship them, and the unshipped ones pile up in `ORDERS`. You clear that backlog by adding workers. Point several processes at the same `shipping` consumer and they share the load: the server hands each stored order to exactly one worker. You reuse the `ORDERS` stream and `shipping` consumer you already have. ## One consumer, many workers Each worker names the same `shipping` consumer and runs the same pull loop. Start that loop in several processes at once: #### CLI ``` #!/bin/bash # A worker loop for the shipping consumer. # Run this same loop in three separate terminals to form a pool of # three workers. All three share the one "shipping" consumer, and the # server splits the stored messages across them. # Each pass: pull one message, "process" it, and acknowledge it. # --count 1 pulls a single message at a time. # --ack acknowledges the message after it is received. while true; do nats consumer next ORDERS shipping --count 1 --ack done # Publish work from a fourth terminal and watch it spread across the # three running loops: # nats pub orders.shipped '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' # # To see in-flight redelivery: stop one worker (Ctrl-C) right after it # pulls a message but before it acks. After AckWait (30s default), the # server redelivers that message to one of the surviving workers. ``` #### JavaScript/TypeScript ``` // Bind to the durable "shipping" consumer created earlier. const js = jetstream(nc); const c = await js.consumers.get("ORDERS", "shipping"); // consume() yields the orders the server hands this worker. Run this same // program in several processes: they all share the one "shipping" consumer, and // the server splits the stored orders across them, one order to one worker. const messages = await c.consume(); for await (const m of messages) { console.log(`shipping ${m.string()}`); m.ack(); } ``` #### Go ``` // Bind to the durable "shipping" consumer created earlier. cons, err := js.Consumer(ctx, "ORDERS", "shipping") if err != nil { panic(err) } // Consume runs the handler for every order the server hands this worker. // Start this same program in several processes: they all share the one // "shipping" consumer, and the server splits the stored orders across them, // one order to one worker. consCtx, err := cons.Consume(func(msg jetstream.Msg) { fmt.Printf("shipping %s\n", string(msg.Data())) msg.Ack() }) if err != nil { panic(err) } defer consCtx.Stop() // Keep this worker running until interrupted (Ctrl-C). sig := make(chan os.Signal, 1) signal.Notify(sig, os.Interrupt) <-sig ``` #### Python ``` # Bind to the durable "shipping" consumer created earlier. Run this same # program in several processes: they all share the one consumer, and the # server splits the stored orders across them, one order to one worker. psub = await js.pull_subscribe_bind("shipping", stream="ORDERS") # Pull one order at a time and ack it. fetch times out when nothing is # waiting; keep looping so the worker stays ready for the next order. while True: try: msgs = await psub.fetch(batch=1, timeout=5) except nats.errors.TimeoutError: continue for msg in msgs: print(f"shipping {msg.data.decode()}") await msg.ack() ``` #### Java ``` // consume() runs the handler for every order the server hands this // worker. Run this same program in several processes: they all share // the one "shipping" consumer, and the server splits the stored // orders across them, one order to one worker. try (MessageConsumer mc = cc.consume(msg -> { System.out.println("shipping " + new String(msg.getData(), StandardCharsets.UTF_8)); msg.ack(); })) { // Keep this worker running until the process is stopped. Thread.sleep(Long.MAX_VALUE); } ``` #### Rust ``` // Bind to the durable "shipping" consumer created earlier. let stream = js.get_stream("ORDERS").await?; let consumer: PullConsumer = stream.get_consumer("shipping").await?; // messages() is a never-ending stream of orders the server hands this // worker. Run this same program in several processes: they all share the // one "shipping" consumer, and the server splits the stored orders across // them, one order to one worker. let mut messages = consumer.messages().await?; while let Some(msg) = messages.next().await { let msg = msg?; println!("shipping {}", std::str::from_utf8(&msg.payload)?); msg.ack().await?; } ``` #### C#/.NET ``` // Bind to the durable "shipping" consumer created earlier. var consumer = await js.GetConsumerAsync("ORDERS", "shipping"); // ConsumeAsync yields the orders the server hands this worker. Run this // same program in several processes: they all share the one "shipping" // consumer, and the server splits the stored orders across them, one // order to one worker. await foreach (var msg in consumer.ConsumeAsync()) { output.WriteLine($"shipping {msg.Data}"); await msg.AckAsync(); // A real worker loops forever; stop once the backlog is clear so the // example returns. if (++shipped == 3) { break; } } ``` #### C ``` // The handler runs for every order the server hands this worker. static void onOrder(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("shipping %.*s\n", natsMsg_GetDataLength(msg), natsMsg_GetData(msg)); natsMsg_Ack(msg, NULL); natsMsg_Destroy(msg); } // Bind to the durable "shipping" consumer created earlier and deliver // messages to the handler. Start this same program in several processes: // they all share the one "shipping" consumer, and the server splits the // stored orders across them, one order to one worker. jsSubOptions so; jsSubOptions_Init(&so); so.Stream = "ORDERS"; so.Consumer = "shipping"; so.ManualAck = true; s = js_PullSubscribeAsync(&sub, js, NULL, NULL, onOrder, NULL, NULL, &so, &jerr); // Keep this worker running until interrupted (Ctrl-C). while ((s == NATS_OK) && !done) nats_Sleep(100); ``` Open three terminals and run that loop in each. The `ORDERS` stream already holds the orders from earlier pages; now add a fresh handful of `orders.shipped` messages from a fourth terminal so all three workers have something to compete for: ``` nats pub --jetstream orders.shipped '{"order_id":"ord_9x3f","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T11:01:50Z"}' nats pub --jetstream orders.shipped '{"order_id":"ord_7p4d","customer":"globex","total_cents":7800,"ts":"2026-05-22T11:01:55Z"}' nats pub --jetstream orders.shipped '{"order_id":"ord_5xk1","customer":"initech","total_cents":1500,"ts":"2026-05-22T11:02:00Z"}' ``` Each `--jetstream` publish returns a `PubAck` confirming the order landed in `ORDERS`. The server hands each waiting order to one worker, serving pull requests in the order they arrived. No two workers get the same order, so the three split the backlog roughly evenly, because each loop pulls one order at a time: **Message flow — Workers sharing one consumer (animated):** Several workers share one pull consumer. The ORDERS stream holds a backlog of stored orders and the single shipping consumer has one read position that sweeps through them. Each order is handed to exactly one worker, rotating round-robin across the workers that are asking, so three workers end up with an even share. Acked orders stay in the stream — the workers share a read position, not the messages. A worker only takes a turn while it's actually asking. One that's still shipping an order has no pull request open, so the server skips it and gives the order to the next worker in line. Distribution follows demand: a faster worker pulls more often and ships more. The consumer still tracks a single position through all of this. Ask the server and the acked count climbs as one number: ``` nats consumer info ORDERS shipping ``` ``` State: Last Delivered Message: Consumer sequence: ... Stream sequence: ... Acknowledgment Floor: Consumer sequence: ... Stream sequence: ... Outstanding Acks: 0 out of maximum 1,000 ``` Your exact sequence numbers depend on the earlier pages; what matters is the shape. The two Acknowledgment Floor numbers advance together as workers ack, and the Last Delivered pair sits at or ahead of them. The position belongs to the consumer, not to each worker, so it advances the same whether one process pulls or three. ## This works on the log you already have `ORDERS` is the same Limits-retention stream from page one; nothing about it changed to make this work. Sharing is a property of the consumer, not the stream: point many workers at one consumer and the server splits the work on any stream. One consequence matters here. An ack advances the position, it doesn't remove the order. The order stays in `ORDERS` for `billing`, `analytics`, and any consumer that reads the log later. The workers share a *position*, not the messages. ## How this differs from a queue group If you read the Core Concepts, this looks like a queue group, but the two balance different things. A queue group splits **live** messages across core NATS subscribers as each one arrives, with no storage behind it. A subscriber that's offline when a message arrives misses it for good. Workers sharing a consumer split **stored** messages against the stream. A message waits in the stream until some worker pulls it and acks it, so a worker that's offline just leaves its share for the others. Only the stream-backed split survives a worker dropping out or a restart. ## When a worker crashes mid-message A worker pulls an order and starts shipping it. Before it acks, the process dies. On the consumer that order is still in progress: the server delivered it and is waiting on the ack. When `AckWait` runs out (30 seconds by default), the server hands the order to another worker. This is the redelivery loop from the [acknowledgment page](/learn/jetstream/acknowledgment.md), now spread across the pool. **Message flow — Crash mid-message, then redelivery (animated):** One order is delivered to a worker that crashes before it acks. The order stays in progress on the shipping consumer while the AckWait timer runs; the server can't see the crash, only the missing ack. When AckWait elapses the server redelivers the same order to a surviving worker, which ships it and acks. The order is handled exactly once even though the first worker failed. Watch it happen. The loop above acks the instant it pulls, so there's no window to interrupt — a real ship takes time. Add a brief delay before the ack (a `sleep 5` in the shell loop, or a pause in your handler), then kill that worker during the delay. After `AckWait` (30 seconds by default), the order reappears on a surviving worker and ships once, because some worker eventually acks it. If the dead worker had already shipped the order before crashing, it ships twice; the pitfall below covers keying side effects by `order_id`. ## Capping how many orders are in progress Each worker can hold an order in progress, so more workers mean more in progress at once. There's a ceiling on that. The ceiling is `MaxAckPending`: how many delivered-but-unacked messages the consumer allows at once, default 1000. Hit it and the server stops delivering new orders until some get acked. The cap is shared across the whole consumer, not per worker: five workers get 1000 between them, not 1000 each. You set it when you create or update the consumer: ``` nats consumer edit ORDERS shipping --max-pending 1000 ``` Set it too low and workers sit idle waiting for a slot. Set it too high and a slow ack leaves a big in-progress backlog that all redelivers at once if many workers die together. ## Pitfalls Running several workers makes two consumer settings, `AckWait` and `MaxAckPending`, matter in practice. **A redelivered order can arrive at a second worker.** When one worker crashes mid-order, the server gives that order to another worker after `AckWait`, so the work still gets done. The same order can then run twice, so key side effects by `order_id`. A redelivery shows up in the message's delivery count. The pool shares work by demand, so you don't choose which worker gets an order. For that control, use [priority groups](/learn/jetstream/priority-groups.md): send everything to one worker until it fails, or keep a standby worker idle until the pool falls behind. **A low `MaxAckPending` starves a large set of workers.** The cap is shared across the whole consumer, not per worker. Set it to 3 and only three messages are ever in progress, so ten workers leave seven of them idle no matter how much is stored. Set the cap to at least your worker count, with room to spare. **A crashed worker holds its order until `AckWait`.** The server can't tell a crash from slow work; it only knows the ack hasn't come. Until the timer runs out (30 seconds by default), that order stays in progress and goes to no one else. Set `AckWait` to your normal processing time: too short redelivers while a healthy worker is still working, too long leaves real failures stuck. The full set of in-progress and redelivery options (`AckWait`, `MaxDeliver`, backoff arrays) is in [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md). ## Where you are You now have: * One `shipping` consumer, unchanged from earlier pages. * Several workers pulling from it, splitting the stored messages between them. * A name for the ceiling on how much runs at once: `MaxAckPending`, shared across every worker. You scale by starting more processes; the consumer is unchanged. And because `ORDERS` is a log, it keeps every order after a worker handles it. ## What's next Every consumer so far — `billing`, `analytics`, `shipping` — was built to last: named, durable, resumable. The next page is the opposite kind of read: an [ordered consumer](/learn/jetstream/ordered-consumer.md), a fast single pass over the whole log, in order, that cleans up after itself — for a job like totaling every order once, with no position to keep. ## See also * [Reference → Consumer Configuration](/reference/jetstream/api/consumer/create.md) — `MaxAckPending`, `AckWait`, backoff, and every other consumer field. * [Core Concepts → Queue Groups](/concepts/queue-groups.md) — the core NATS balancing this page contrasts with. --- # Your first stream This page creates the `ORDERS` stream with one CLI command, then looks at it with another. ## Why a stream The running example for this chapter is a small online store, the Acme `ORDERS` platform. Each thing that happens to an order shows up as a message on a [subject](/concepts/subjects.md): * `orders.created` — a new order comes in * `orders.shipped` — the order leaves the warehouse * `orders.canceled` — the order is called off Services react to those messages: * a warehouse service packs the box on `orders.created` * a notification service emails the customer on `orders.shipped` * an analytics service counts everything Plain core NATS drops any of these messages the moment no service is listening. A **stream** saves them instead. A stream is a store that runs on the server and keeps every message on the subjects you choose. You can read the saved messages again whenever you need them, minutes or a month after they arrived. ## Prerequisites A running `nats-server` with JetStream enabled. If you haven't yet: ``` nats-server -js ``` The `-js` flag turns on JetStream. Without it, the next command won't run. ## Create the stream In another terminal: #### CLI ``` #!/bin/bash # Create the ORDERS stream, capturing every subject under orders.> # --defaults fills in the remaining config with sensible starting values. nats stream add ORDERS --subjects "orders.>" --defaults # Expected: output ending with # Stream ORDERS was created ``` #### JavaScript/TypeScript ``` // Get a JetStream manager and create the ORDERS stream, which captures // every Acme order subject under `orders.` const jsm = await jetstreamManager(nc); const info = await jsm.streams.add({ name: "ORDERS", subjects: ["orders.>"], }); console.log(`Created stream: ${info.config.name}`); ``` #### Go ``` // Create the ORDERS stream, capturing every subject under 'orders.>'. stream, err := js.CreateStream(ctx, jetstream.StreamConfig{ Name: "ORDERS", Subjects: []string{"orders.>"}, }) if err != nil { panic(err) } // Confirm the stream was created. fmt.Printf("Created stream: %s\n", stream.CachedInfo().Config.Name) ``` #### Python ``` # Create the "ORDERS" stream, capturing every subject under "orders.". info = await js.add_stream(name="ORDERS", subjects=["orders.>"]) # Confirm success by printing the stream name the server returned. print(f"Created stream: {info.config.name}") ``` #### Java ``` // Create a stream named "ORDERS" that captures any subject under `orders.` JetStreamManagement jsm = nc.jetStreamManagement(); StreamInfo streamInfo = jsm.addStream(StreamConfiguration.builder() .name("ORDERS") .subjects("orders.>") .storageType(StorageType.File) .build()); // Confirm the stream was created System.out.println("Created stream: " + streamInfo.getConfiguration().getName()); ``` #### Rust ``` // Create the ORDERS stream, capturing every subject under `orders.`. let stream = js .create_stream(jetstream::stream::Config { name: "ORDERS".to_string(), subjects: vec!["orders.>".to_string()], storage: StorageType::File, ..Default::default() }) .await?; // Confirm success by printing the name the server assigned. println!("Created stream: {}", stream.cached_info().config.name); ``` #### C#/.NET ``` // Create the ORDERS stream, capturing every subject under `orders.` var stream = await js.CreateStreamAsync(new StreamConfig(name: "ORDERS", subjects: ["orders.>"])); // Confirm the stream was created output.WriteLine($"Created stream: {stream.Info.Config.Name}"); ``` #### C ``` // Create the ORDERS stream, capturing every subject under 'orders.>'. jsStreamConfig cfg; const char *subjects[] = {"orders.>"}; jsStreamConfig_Init(&cfg); cfg.Name = "ORDERS"; cfg.Subjects = subjects; cfg.SubjectsLen = 1; s = js_AddStream(&si, js, &cfg, NULL, &jerr); if (s == NATS_OK) { // Confirm the stream was created. printf("Created stream: %s\n", si->Config->Name); } ``` Two parts of that command matter. The first is the **stream name**: `ORDERS`. Stream names are case-sensitive. They can't contain dots, `*`, `>`, spaces, or slashes, so a subject like `orders.created` won't work as a name. The name shows up in every command and every error message in this chapter. The second is the **subjects** the stream keeps: `orders.>`. That's a [wildcard](/concepts/subjects.md#wildcards). Any subject that starts with `orders.` goes into this stream. `orders.created`, `orders.shipped`, and `orders.canceled` all match. So would `orders.refunded` next month, with no change to the stream. The `--defaults` flag tells `nats` not to prompt you for the other settings. The server fills them in with its standard defaults. We'll look at what those values are in a moment. For now, the defaults are fine. You should see output ending with something like: ``` Stream ORDERS was created ``` If the command fails instead with `no responders available for request`, your server started without `-js`: JetStream isn't running, so nothing answers the request. Restart it with the flag and try again. ## Checking the created stream Look at what the server just created: #### CLI ``` #!/bin/bash # Inspect the stream: its configuration (what you asked for plus the # defaults the server filled in) and its state (what's stored right now). nats stream info ORDERS ``` #### JavaScript/TypeScript ``` // Get a JetStream manager and look up the ORDERS stream, then print the // key fields from its configuration and current state const jsm = await jetstreamManager(nc); const info = await jsm.streams.info("ORDERS"); console.log(`Stream: ${info.config.name}`); console.log(`Subjects: ${info.config.subjects?.join(", ")}`); console.log(`Messages: ${info.state.messages}`); ``` #### Go ``` // Look up the ORDERS stream and fetch its latest info from the server. stream, err := js.Stream(ctx, "ORDERS") if err != nil { panic(err) } info, err := stream.Info(ctx) if err != nil { panic(err) } // Print key fields: name, captured subjects, and current message count. fmt.Printf("Name: %s\n", info.Config.Name) fmt.Printf("Subjects: %v\n", info.Config.Subjects) fmt.Printf("Messages: %d\n", info.State.Msgs) ``` #### Python ``` # Fetch the latest information about the "ORDERS" stream. info = await js.stream_info("ORDERS") # Print key fields: name and subjects come from the config, # the live message count comes from the stream state. print(f"Stream: {info.config.name}") print(f"Subjects: {info.config.subjects}") print(f"Messages: {info.state.messages}") ``` #### Java ``` // Fetch info for the "ORDERS" stream and print key fields StreamInfo streamInfo = jsm.getStreamInfo("ORDERS"); StreamConfiguration config = streamInfo.getConfiguration(); StreamState state = streamInfo.getStreamState(); System.out.println("Name: " + config.getName()); System.out.println("Subjects: " + config.getSubjects()); System.out.println("Messages: " + state.getMsgCount()); ``` #### Rust ``` // Fetch the ORDERS stream and read its current info from the server. let mut stream = js.get_stream("ORDERS").await?; let info = stream.info().await?; // Print the key fields: name, captured subjects, and message count. println!("Stream name: {}", info.config.name); println!("Subjects: {:?}", info.config.subjects); println!("Message count: {}", info.state.messages); ``` #### C#/.NET ``` // Fetch information about the ORDERS stream var stream = await js.GetStreamAsync("ORDERS"); // Read key fields: name and subjects come from the config, the // message count comes from the live stream state var info = stream.Info; var subjects = string.Join(", ", info.Config.Subjects ?? []); output.WriteLine($"Name: {info.Config.Name}"); output.WriteLine($"Subjects: {subjects}"); output.WriteLine($"Messages: {info.State.Messages}"); ``` #### C ``` // Look up the ORDERS stream and fetch its latest info from the server. s = js_GetStreamInfo(&si, js, "ORDERS", NULL, &jerr); if (s == NATS_OK) { int i; // Print key fields: name, captured subjects, and current message count. printf("Name: %s\n", si->Config->Name); printf("Subjects:"); for (i = 0; i < si->Config->SubjectsLen; i++) printf(" %s", si->Config->Subjects[i]); printf("\n"); printf("Messages: %" PRIu64 "\n", si->State.Msgs); } ``` The output has two halves. The **configuration** half shows what you asked for and what the defaults filled in — the header plus the `Options` and `Limits` sections: ``` Information for Stream ORDERS Subjects: orders.> Replicas: 1 Storage: File Options: Retention: Limits Acknowledgments: true Discard Policy: Old Duplicate Window: 2m0s Limits: Maximum Messages: unlimited Maximum Per Subject: unlimited Maximum Bytes: unlimited Maximum Age: unlimited Maximum Message Size: unlimited Maximum Consumers: unlimited ``` The `Options` section also lists a `Direct Get` line and a run of `Allows …` feature toggles. The CLI turns direct get on by default, so that line shows here; client libraries leave it off unless you set it. They're trimmed and covered where each feature comes up. The **state** half shows what's in the stream right now: ``` State: Messages: 0 Bytes: 0 B First Sequence: 0 Last Sequence: 0 Active Consumers: 0 ``` A new stream is empty: zero messages, zero bytes, no consumers. The first message you publish gets sequence `1`. ## The defaults You didn't set any of the configuration values above. The server filled them in with its standard defaults, the values a stream gets whenever a field is left unset. Here is what each one means. * **Replicas: 1**. The stream lives on one server. If that server goes down, the stream goes down with it. That's fine on a laptop, but risky in production. We come back to this on the [Surviving node loss](/learn/jetstream/surviving-node-loss.md) page. * **Storage: File**. Messages are written to disk. The other option is memory, which is faster but lost on restart. * **Retention: Limits**. The stream keeps messages until it hits a limit (size, age, or count). The other options are `Interest` and `WorkQueue`, which delete messages once a consumer has read them. We cover the three policies on the [Retention policies](/learn/jetstream/retention-policies.md) page. * **Discard Policy: Old**. When the stream finally hits a limit, the oldest messages are deleted to make room. The other option is `New`, which turns away new messages when the stream is full. * **Maximum Messages / Bytes / Age / Message Size: unlimited**. No upper bound today. On a real cluster you'd always set at least one of these. We do that on the [Shaping the stream](/learn/jetstream/shaping-the-stream.md) page. * **Duplicate Window: 2m0s**. For two minutes after a message is stored, the server turns away a second message that carries the same [`Nats-Msg-Id`](/reference/jetstream/api/headers.md) header. This is what lets you publish the same message twice without storing it twice. The [Publishing](/learn/jetstream/publishing.md) page uses it. The full set of stream configuration options is listed in [Reference → Create Stream](/reference/jetstream/api/stream/create.md). We use only the defaults here. ## Subjects bind to exactly one stream Only one stream can keep a given subject. If you try to create a second stream whose subjects overlap `orders.>`, the server turns it down: ``` nats stream add ARCHIVE --subjects "orders.*" --defaults ``` ``` nats: error: could not create Stream: subjects overlap with an existing stream (10065) ``` The check covers any overlap, not just the identical filter: `ORDERS` keeps `orders.>`, so a new stream asking for `orders.*`, or even the single subject `orders.created`, is turned down the same way. This is on purpose. When a message lands on a subject, JetStream always knows which stream it goes into. To keep overlapping subjects, you use **mirrors** and **sources**. A mirror keeps a copy of one other stream. Sources pull messages from several streams into one. The running scenario doesn't need either, so we don't set one up here. The [Mirrors and sources](/learn/jetstream/mirrors-and-sources.md) page builds both from start to finish, and [Reference → Create Stream](/reference/jetstream/api/stream/create.md) lists the `mirror` and `sources` configuration fields. ## Pitfalls A few things catch people on a first stream. **Unlimited defaults grow forever.** With `--defaults`, `Maximum Messages`, `Maximum Bytes`, and `Maximum Age` are all `unlimited`. The `ORDERS` stream then keeps every order it ever stored until the disk fills up, and a full disk takes the server down with it. That's fine while you're learning on a laptop, but a production stream needs at least one limit so old orders age out before the disk does. Setting limits is the [Shaping the stream](/learn/jetstream/shaping-the-stream.md) page; here the defaults are deliberately left unbounded. **A stream name is permanent.** There's no rename. `nats stream edit` has no `--name` flag, and the server turns down any update that changes an existing stream's name with `stream configuration name must match original`. The only way to "rename" `ORDERS` is to delete it and create a new stream, which loses every order already stored. So pick the name carefully the first time. If you do outgrow a name, the safe move is to add a new stream on new subjects and let the old one age out. ## Where you are You now have: * an `ORDERS` stream bound to `orders.>` * zero messages in it * a configuration of default values The next page, [Publishing](/learn/jetstream/publishing.md), sends the first few messages and shows what the server returns. ## See also * [Reference → Create Stream](/reference/jetstream/api/stream/create.md): every configuration option and its valid range * [Reference → JetStream API](/reference/jetstream/api/.md): the full list of stream and consumer operations --- # Key-Value Store NATS gives you a key-value store. It is not a separate database: a bucket is a JetStream stream named `KV_` whose subjects are `$KV..>`; a key is the last token of that subject, and a value is a message on it. Everything in this chapter is that one stream presented through a key-value API. Keep that framing in mind from the start. A `put` of a value appends a message, a `get` of a value reads the last message for a subject, and a `watch` of a bucket opens a consumer. The API (`put`, `get`, `watch`, `create`, `update`, TTL) is what you use day to day, and the stream underneath is what makes it work. This chapter teaches the abstraction first and describes the stream last. The first four pages teach you the key-value API on its own terms, so you can be productive without memorizing JetStream internals. [Under the hood](/learn/key-value/under-the-hood.md) shows you the stream that the API has been using all along. ## By the end you will have * An `INVENTORY` bucket whose keys are SKUs (`widget-blue`, `widget-red`, `gadget-pro`) and whose values are stock counts the inventory service reads and decrements. * A warehouse dashboard that **watches** the bucket: it receives the current count of every key as a snapshot, then live updates as counts change. * A safe decrement of `widget-blue` from 41 to 40 using **compare-and-swap**, so two concurrent sales never lose a write. * A `flash-sale` key with a per-key **TTL** that expires on its own, and a feel for the bucket-wide limits that bound the whole thing. * A clear picture of the `KV_INVENTORY` stream underneath: the subjects, the direct read path, and the difference between delete and purge. ## Who this is for You've read the [JetStream Deep Dive](/learn/jetstream/.md) or the [Core Concepts → JetStream](/concepts/jetstream.md) primer, so the sentence "a stream stores messages" already means something to you. This chapter doesn't re-teach what a stream is, how a consumer tracks its position, or how acknowledgment works. Where you'd want those, it names the gap and links back to [JetStream](/learn/jetstream/.md). You don't need to know anything about key-value specifically. We start from "what is a bucket and why would you want one" and grow from there. ## How to read it Each page introduces at most two new concepts and builds on the one before it: the same `INVENTORY` bucket carries through, and you can keep one terminal open through the whole chapter without resetting state. You create the bucket on [Your first bucket](/learn/key-value/your-first-bucket.md), add a watcher on [Watching](/learn/key-value/watching.md), decrement a key safely on [History and revisions](/learn/key-value/history-and-revisions.md), give a key a TTL on [TTL and limits](/learn/key-value/ttl-and-limits.md), and inspect the stream underneath on [Under the hood](/learn/key-value/under-the-hood.md). Key-value has many configuration options, covering bucket limits, watch options, and headers on the wire. Where a feature has a long list, the page covers only what you need to understand the concept. Because a bucket is created as a stream, the full set of bucket configuration options lives in [Reference → Create Stream](/reference/jetstream/api/stream/create.md). ## Map | Page | What you learn | | ------------------------------------------------------------------ | ------------------------------------------------------------------------- | | [Your first bucket](/learn/key-value/your-first-bucket.md) | Create `INVENTORY`, put and get `widget-blue`, and read its status | | [Watching](/learn/key-value/watching.md) | Receive a snapshot of every key, then live changes as they happen | | [History and revisions](/learn/key-value/history-and-revisions.md) | Track revisions, read history, and decrement safely with compare-and-swap | | [TTL and limits](/learn/key-value/ttl-and-limits.md) | Expire a single key with a per-key TTL, and bound the bucket with limits | | [Under the hood](/learn/key-value/under-the-hood.md) | See the `KV_INVENTORY` stream, the direct read, and delete versus purge | | [Where to go next](/learn/key-value/where-next.md) | A map of what's beyond key-value, and one pre-production checklist | ## Prerequisites You'll need: * A working `nats-server` with JetStream enabled. The key-value store is built on JetStream, so JetStream must be on. The simplest way is `nats-server -js`. * The `nats` CLI installed and pointed at your server. The first page uses only the CLI. Later pages add JavaScript, Go, Python, Java, Rust, and C# examples for the same operations. Open a terminal, run `nats-server -js`, and continue to the first page. --- # History and revisions So far every write to a key has been a `put`: the new value replaces the old one, and you never had to think about who wrote last. That works until two writers touch the same key at the same time. This page adds the two ideas that make concurrent writes safe. The first is the revision: a number the bucket assigns to every write, in sequence, and `history` keeps a key's prior ones. The second is compare-and-swap (CAS): a write that only succeeds if the key still holds the revision you read. Together they let the inventory service decrement `widget-blue` from 41 to 40 without ever losing a sale. You still have the `INVENTORY` bucket from the first page. After the live update on the watching page, `widget-blue` holds 41, and the warehouse dashboard from that page is still attached. This page builds on both of those. ## Revisions and history A **revision** is the number the server assigns to a write. The bucket keeps one counter across all of its keys, and every write — to any key — takes the next number. So a revision always increases when you write a key, but not by one each time: writes to other keys advance the counter in between. You don't set the revision; the server assigns it, and `get` returns it alongside the value as part of the **entry**. The entry you got back on the first page already carried it. Your put of `widget-blue 42` landed at revision 1, the first write to an empty bucket. The put of `41` over it on the watching page landed at revision 2. The puts to `widget-red` and `gadget-pro` that followed took their own numbers, so `widget-blue`'s next write won't be revision 3. What matters for a safe write is only that the revision you read names the exact version you saw, which you'll use in a moment. The bucket also keeps the **history**: the prior revisions of a key, up to the bucket's history depth. You set that depth when you created the bucket with `--history`, and `INVENTORY` was created with `--history 1`, which keeps only the latest value of each key. Raise it now so a key remembers where it's been: ``` nats kv edit INVENTORY --history 10 ``` The depth caps at 64. Raising it keeps every future write to a key, up to ten revisions deep. The values written before the raise are already gone, because at depth 1 each new write dropped the one before it. ## Compare-and-swap: writing without locks Consider a problem `put` can't solve. Two copies of the inventory service both sell a `widget-blue`. Each reads the count as 41, computes 40, and puts 40. Two sales happened, but the count only dropped by one. One write overwrote the other, and a unit of stock was lost from the count. The fix is **optimistic concurrency**, which uses no locks and no waiting. Instead, each writer reads the current revision, then writes *on the condition* that the revision hasn't changed since. If another writer got there first, the condition fails and the write is rejected; nothing is overwritten. The rejected writer reads the fresh value and tries again. The mechanism behind that condition is **compare-and-swap (CAS)**: you hand the server the revision you expect the key to be at, and the server swaps in your new value only if the key is still at that revision. NATS exposes CAS through two operations: * **create** writes a key only if it doesn't exist yet. It's CAS against the expectation "this key is at revision 0" (no value present). * **update** writes a key only if it's at the revision you name. It's CAS against "this key is still at the revision I read." To decrement `widget-blue` safely, the inventory service reads the entry, takes its revision, computes the new count, and calls update with that revision. If the key is still at the revision it read, the write lands. If not, the write is rejected and nothing is lost. #### CLI ``` #!/bin/bash # Decrement widget-blue from 41 to 40 safely with compare-and-swap. # Read the current entry, take its revision, then update on the condition # that the key is still at that revision. If another writer got there # first, the update is rejected and nothing is overwritten. # Read the current revision from a get. The get prints a header line # "... revision: N created @ ..." followed by the value, so pull N from it. REVISION=$(nats kv get INVENTORY widget-blue | sed -n 's/.*revision: \([0-9]*\).*/\1/p') # Update succeeds only if widget-blue is still at REVISION. nats kv update INVENTORY widget-blue 40 "$REVISION" ``` #### C ``` // Decrement widget-blue from 41 to 40 safely with compare-and-swap. kvEntry *entry = NULL; uint64_t revision = 0; uint64_t newRev = 0; // Read the current entry; it carries the value and its revision. s = kvStore_Get(&entry, kv, "widget-blue"); if (s == NATS_OK) { revision = kvEntry_Revision(entry); printf("widget-blue is %s at revision %" PRIu64 "\n", kvEntry_ValueString(entry), revision); kvEntry_Destroy(entry); // Update succeeds only if widget-blue is still at that revision. // If another writer got there first, the update is rejected and // nothing is overwritten. s = kvStore_UpdateString(&newRev, kv, "widget-blue", "40", revision); } if (s == NATS_OK) printf("decremented widget-blue to 40 at revision %" PRIu64 "\n", newRev); ``` This is the read-modify-write you use whenever the new value depends on the old one, such as a decrement, an increment, or a status transition. The revision is what makes the write safe. **Message flow — KV compare-and-swap retry (animated):** Optimistic concurrency on the KV bucket KV\_INVENTORY. The inventory service gets widget-blue and reads revision 7, then sends an update that only applies if the key is still at revision 7. A concurrent writer commits first and bumps widget-blue to revision 8, so the compare-and-set is rejected on the revision mismatch. The service re-gets widget-blue at revision 8, reapplies its change with the fresh revision, and this update is accepted. No write is lost and the service never holds a lock. * inventory → KV\_INVENTORY * other writer → KV\_INVENTORY The animation shows the conflict: the service gets `widget-blue` at revision 7 and calls update expecting 7, but a concurrent writer has already bumped the key to revision 8. The server rejects the update on the revision mismatch. The service re-gets (now revision 8) and retries with the fresh revision, which the server accepts. No write was lost, and the service never held a lock. ## Read the history back With the depth raised, the trail behind `widget-blue` is now visible. Read its history: #### CLI ``` #!/bin/bash # Read the kept history of a key. With history raised to 10 on this page, # the trail behind widget-blue is now visible. Each row is one revision: # the key, its revision number, the operation, when it was written, its # length, and the value. nats kv history INVENTORY widget-blue # Expected output after the CAS decrement — two revisions kept: # # History for INVENTORY > widget-blue # # Key Revision Op Created Length Value # widget-blue 2 PUT ... 2 41 # widget-blue 5 PUT ... 2 40 # # The revision numbers aren't consecutive — writes to widget-red and # gadget-pro took the numbers in between — because the revision counter is # bucket-wide, not per key. The value 42 written before the raise is gone; # at history 1 each write dropped the one before it. ``` #### C ``` // Read the kept history of a key, oldest first. Each entry is one // revision: the key, its revision number, the operation, and the // value. The revision numbers aren't consecutive — the counter is // bucket-wide, not per key. kvEntryList list; int i; s = kvStore_History(&list, kv, "widget-blue", NULL); if (s == NATS_OK) { printf("History for INVENTORY > widget-blue\n"); for (i = 0; i < list.Count; i++) { kvEntry *e = list.Entries[i]; printf(" %s rev %" PRIu64 " %s %s\n", kvEntry_Key(e), kvEntry_Revision(e), (kvEntry_Operation(e) == kvOp_Put) ? "PUT" : "MARKER", (kvEntry_Value(e) != NULL) ? kvEntry_ValueString(e) : ""); } kvEntryList_Destroy(&list); } ``` You see two revisions: `41` from the watching page and `40` from the decrement you just made. Their revision numbers aren't consecutive — the puts to `widget-red` and `gadget-pro` took the numbers in between — which is the revision counter being bucket-wide, not per key. Each entry is one line: the key, its revision, the operation, when it was written, and the value. The `42` written before the raise isn't here; depth 1 had already dropped it. History holds the prior revisions of a single key, up to the depth. It isn't an audit log of the whole bucket, and it doesn't grow without bound: once a key has more revisions than the depth allows, the oldest one is removed. The full set of bucket configuration options is documented in [Reference → Create Stream](/reference/jetstream/api/stream/create.md). ## Pitfalls Two mistakes are common the first time you rely on revisions. Both come from treating a CAS write like an unconditional one. **A rejected update is dropped rather than queued, and you must retry.** Optimistic concurrency doesn't wait. When update finds the key at a different revision than you named, the server returns an error and your value is not written. If you fire-and-forget an update, a conflict silently loses the write. Don't assume an update succeeded; check the result, and on a revision mismatch, re-get the key and try again with the fresh revision. That re-get-and-retry loop is what CAS exists to support. Without that loop you have the same lost-write bug `put` had. Here's the handling: get the current revision, attempt the update, and on a mismatch re-get and retry once with the new revision. #### CLI ``` #!/bin/bash # CAS retry loop. A rejected update is dropped, not queued, so on a # revision mismatch re-get the key and retry with the fresh revision. # Here the inventory service decrements widget-blue by one, safely. # Read the value AND its revision from a SINGLE get, so the pair is # consistent. Two separate gets could pair a stale value with a fresh # revision if a concurrent write landed in between. The get prints a header # line "... revision: N created @ ..." then a blank line then the value. ENTRY=$(nats kv get INVENTORY widget-blue) VALUE=$(printf '%s\n' "$ENTRY" | tail -n1) REVISION=$(printf '%s\n' "$ENTRY" | sed -n 's/.*revision: \([0-9]*\).*/\1/p') NEW_VALUE=$((VALUE - 1)) # Try the update; if a concurrent writer bumped the revision, re-get and retry once. if nats kv update INVENTORY widget-blue "$NEW_VALUE" "$REVISION"; then echo "decremented widget-blue to $NEW_VALUE" else echo "revision conflict, re-getting and retrying" ENTRY=$(nats kv get INVENTORY widget-blue) VALUE=$(printf '%s\n' "$ENTRY" | tail -n1) REVISION=$(printf '%s\n' "$ENTRY" | sed -n 's/.*revision: \([0-9]*\).*/\1/p') NEW_VALUE=$((VALUE - 1)) nats kv update INVENTORY widget-blue "$NEW_VALUE" "$REVISION" fi ``` #### C ``` // CAS retry loop. A rejected update is dropped, not queued, so on a // revision mismatch re-get the key and retry with the fresh revision. // Here the inventory service decrements widget-blue by one, safely. kvEntry *entry = NULL; uint64_t revision = 0; uint64_t newRev = 0; char newValue[32]; // Read the value AND its revision from a single get, so the pair is // consistent. Two separate gets could pair a stale value with a fresh // revision if a concurrent write landed in between. s = kvStore_Get(&entry, kv, "widget-blue"); if (s == NATS_OK) { revision = kvEntry_Revision(entry); snprintf(newValue, sizeof(newValue), "%d", atoi(kvEntry_ValueString(entry)) - 1); kvEntry_Destroy(entry); // Try the update; if a concurrent writer bumped the revision, // re-get and retry once. s = kvStore_UpdateString(&newRev, kv, "widget-blue", newValue, revision); if (s != NATS_OK) { printf("revision conflict, re-getting and retrying\n"); s = kvStore_Get(&entry, kv, "widget-blue"); if (s == NATS_OK) { revision = kvEntry_Revision(entry); snprintf(newValue, sizeof(newValue), "%d", atoi(kvEntry_ValueString(entry)) - 1); kvEntry_Destroy(entry); s = kvStore_UpdateString(&newRev, kv, "widget-blue", newValue, revision); } } } if (s == NATS_OK) printf("decremented widget-blue to %s\n", newValue); ``` **Do not use put for a read-modify-write.** Put is unconditional: it writes no matter what the key holds now, so it will overwrite a value a concurrent writer just stored. Put is correct when the new value doesn't depend on the old one, such as setting a SKU's count to a known absolute number from an inventory recount. When the new value is computed *from* the current value, like a decrement on a sale, use update with the revision instead. Use put to set a key to a given value, and update to change it from the value it currently holds. ## Where you are You now have: * An `INVENTORY` bucket whose `widget-blue` key has been decremented from 41 to 40 with a CAS update, not a blind put. * The ability to read a key's history and reason about its revision. * A working model of optimistic concurrency: read the revision, write on that condition, retry on mismatch. The warehouse dashboard from the watching page saw that decrement arrive live, the same as any other change. ## What's next The next page gives a key its own lifetime with a per-key **TTL**, and bounds the whole bucket with limits. Continue to [TTL and limits](/learn/key-value/ttl-and-limits.md). ## See also * [Reference → Create Stream](/reference/jetstream/api/stream/create.md) — every bucket configuration option, including history depth. * [Watching](/learn/key-value/watching.md) — how a watcher receives the changes you make with update. --- # TTL and limits Every key in `INVENTORY` so far lives until you overwrite or delete it. That's the right default for a stock count, but it's not the only thing a bucket holds. Some values should clean themselves up, such as a flash-sale price, a short-lived session token, or a "this SKU is locked for the next 30 minutes" flag. This page adds two ways to bound the bucket by time. The first is the per-key TTL, a single key that expires on its own. The second is the set of bucket limits that bound the whole thing: total size, value size, history depth. When a key expires, the server leaves a marker so the warehouse dashboard learns the value is gone, the same way it learned about every other change. You still have the `INVENTORY` bucket from the previous pages, with `widget-blue` decremented to 40 and the warehouse dashboard watching. You'll add a key with a TTL to it. ## A per-key TTL expires a single value A **TTL** (time-to-live) is how long a value stays in the bucket before the server removes it. A **per-key TTL** attaches that clock to one key. The key lives for its TTL, then disappears on its own, with no service having to remember to delete it. Per-key TTL is set at **create** time, and only at create time. You hand the TTL to create alongside the value, and the key starts its countdown the moment it lands. This is a deliberate restriction: a TTL belongs to the value you're writing now, not to a value that might be put over it later. One setup step comes first. Per-key TTLs ride on a bucket feature called **limit markers**, the same mechanism that leaves a trace when a value expires. A bucket needs limit markers enabled before any key in it can carry a TTL. `INVENTORY` was created without them, so you turn them on once, then create the timed key: #### CLI ``` #!/bin/bash # Give a single key its own lifetime with a per-key TTL. # # Per-key TTL rides on the bucket's Limit Markers, so the bucket must have # them enabled first. INVENTORY was created without them, so turn them on # once (the duration is how long the bucket keeps the expiry marker): nats kv edit INVENTORY --marker-ttl 1h # Now create flash-sale with a 30-minute TTL. The value expires on its own # 30 minutes after this create; no service has to remember to clean it up. # Note: TTL is accepted on CREATE only. nats kv create INVENTORY flash-sale 99 --ttl 30m # Expected: create echoes the value it stored, and the key counts down on # its own: # # 99 # # Per-key TTL requires nats-server 2.11 or newer. On an older server the # create is rejected because the bucket cannot enable Limit Markers. # --- Per-key TTL is create-only: to change it, delete then create --------- # # Neither put nor update takes a --ttl; the CLI rejects the flag. Writing # flash-sale again with put or update appends a new value with no TTL, so # the key stops expiring rather than keeping its clock. To give flash-sale # a different lifetime, remove it and create it again with the new TTL: nats kv del INVENTORY flash-sale --force nats kv create INVENTORY flash-sale 99 --ttl 10m # The key now expires in 10 minutes instead of 30. ``` #### C ``` // Per-key TTL rides on the bucket's limit markers, so the bucket // needs them enabled. The C client turns them on when the bucket is // created: a LimitMarkerTTL > 0 enables per-key TTLs and is how long // the bucket keeps an expiry marker. Requires nats-server 2.11+. kvStore *kv = NULL; uint64_t rev = 0; kvConfig cfg; kvConfig_Init(&cfg); cfg.Bucket = "INVENTORY"; cfg.History = 10; cfg.LimitMarkerTTL = 60LL * 60 * 1000000000LL; // 1 hour, in nanoseconds s = js_CreateKeyValue(&kv, js, &cfg); // Create flash-sale with a 30-minute TTL (in milliseconds). The value // removes itself 30 minutes later; no service has to clean it up. // A TTL is accepted on create only: neither put nor update takes one. if (s == NATS_OK) s = kvStore_CreateStringWithTTL(&rev, kv, "flash-sale", "99", 30 * 60 * 1000); if (s == NATS_OK) printf("created flash-sale at revision %" PRIu64 "\n", rev); // A per-key TTL is create-only, and writing the key again appends a // value with no TTL of its own. To change a TTL, delete the key and // create it again with the new one. if (s == NATS_OK) s = kvStore_Delete(kv, "flash-sale"); if (s == NATS_OK) s = kvStore_CreateStringWithTTL(&rev, kv, "flash-sale", "99", 10 * 60 * 1000); if (s == NATS_OK) printf("flash-sale now expires in 10 minutes\n"); ``` The `flash-sale` key now holds `99` and will remove itself 30 minutes later. Because the TTL is stored with the value, this happens without a cron job, a separate cleanup service, or a sweep. Per-key TTL needs **nats-server 2.11 or newer**; that's the release that added limit markers. On an older server, enabling markers on the bucket is rejected, and the timed create fails with it. If you're on 2.11, the feature is available. ## Bucket limits bound the whole bucket Where a per-key TTL bounds one value, **bucket limits** bound the whole bucket. They keep a key-value store from growing without end, and you set them when you create the bucket. Three of them matter most: * **Max bucket size**: the total bytes the bucket may hold across every key and every kept revision. The bucket won't grow past it. * **Max value size**: the largest a single value may be. A put of something bigger is rejected. Key-value values are meant to be small; large values belong in the [Object Store](/learn/object-store/.md). * **History depth**: how many prior revisions each key keeps, which you already met on the previous page. It caps at 64, and it doubles as a per-key cap: it's the most messages any single key may hold. Here those limits go on a throwaway `CACHE` bucket, so the numbers stand on their own and don't imply anything about `INVENTORY`: #### CLI ``` #!/bin/bash # Create a throwaway CACHE bucket to show the three bucket-level limits. # We use CACHE, not the pinned INVENTORY bucket, on purpose: INVENTORY's # TTLs are per-key, not bucket-wide, and we do not want to imply the whole # INVENTORY bucket expires. # # --ttl 1h bucket-wide max age: every value older than 1h is # removed, regardless of key. This is the bucket's # MaxAge, not a per-key TTL. # --max-bucket-size the bucket's total size cap, in bytes. # --max-value-size the largest a single value may be, in bytes. nats kv add CACHE \ --history 1 \ --ttl 1h \ --max-bucket-size 16MB \ --max-value-size 64KB # Expected output is the bucket's status. The CLI parses MB and KB as binary # units (1 MB = 1 MiB, 1 KB = 1 KiB), so 16MB shows as 16 MiB and 64KB as # 64 KiB (labels abbreviated here): # # Information for Key-Value Store Bucket CACHE created