NATS is a simple, secure and high performance open source data layer for cloud native applications, IoT messaging, and microservices architectures.

We feel that it should be the backbone of your communication between services. It doesn't matter what language, protocol, or platform you are using; NATS is the best way to connect your services.

10,000 foot view

• Publish and subscribe to messages at millions of messages per second. At most once delivery.

• Supports fan-in/out delivery patterns

• Request/reply

• Every major language is supported

• Persistence via JetStream

  • at least once delivery or exactly once delivery

  • work queues

  • stream processing

  • data replication

  • data retention

  • data deduplication

  • Higher order data structures

    • Key/Value with watchers, versioning, and TTL

    • Object storage with versioning

• Security

  • TLS

  • JWT-based zero trust security

• Clustering

  • High availability

  • Fault tolerance

  • Auto-discovery

• Protocols supported

  • TCP

  • MQTT

  • WebSockets

All of this in a single binary that is easy to deploy and manage. No external dependencies, just drop it in and add a configuration file to point to other NATS servers and you are ready to go. In fact, you can even embed NATS in your application (for Go users)!

Guided tour

Contribute

Additional questions?

Thank you from the entire NATS Team of Maintainers for your interest in NATS!

NATS is a system for publishing and listening for messages on named communication channels we call Subjects. Fundamentally, NATS is an interest-based messaging system, where the listener has to subscribe to a subset of subjects.

Location transparency Through subject-based addressing, NATS provides location transparency across a (large) cloud of routed NATS servers.

• Subject subscriptions are automatically propagated within the server cloud.

• Messages will be automatically routed to all interested subscribers, independent of location.

Wildcards

NATS provides two wildcards that can take the place of one or more elements in a dot-separated subject. Publishers will always send a message to a fully specified subject, without the wildcard. While subscribers can use these wildcards to listen to multiple subjects with a single subscription.

Subject hierarchies

The . character is used to create a subject hierarchy. For example, a world clock application might define the following to logically group related subjects:

Subject usage best practices

There is no hard limit to subject size, but it is recommended to keep the maximum number of tokens in your subjects to a reasonable value. E.g. a maximum of 16 tokens and the subject length to less than 256 characters.

Number of subjects

NATS can manage 10s of millions of subjects efficiently, therefore, you can use fine-grained addressing for your business entities. Subjects are ephemeral resources, which will disappear when no longer subscribed to.

Still, subject subscriptions need to be cached by the server in memory. Consider when increasing your subscribed subject count to more than one million you will need more than 1GB of server memory and it will grow linearly from there.

Subject-based filtering and security

The message subject can be filtered with various means and through various configuration elements in your NATS server cluster. For example, but not limited to:

• Security - allow/deny per user

• Import/export between accounts

• Automatic transformations

• When inserting messages into JetStream streams

• When sourcing/mirroring JetStream streams

• When connecting leaf nodes (NATS edge servers)

• ...

A well-designed subject hierarchy will make the job a lot easier for those tasks.

Naming things

A subject hierarchy is a powerful tool for addressing your application resources. Most NATS users therefore encode business semantics into the subject name. You are free to choose a structure fit for your purpose, but you should refrain from over-complicating your subject design at the start of the project.

Some guidelines:

• Use the first token(s) to establish a general namespace.

• Use the final token(s)for identifiers

• A subject should be used for more than one message.

• Subscriptions should be stable (exist for receiving more than one message).

• Use wildcard subscriptions over subscribing to individual subjects whenever feasible.

• Name business or physical entities. Refrain from encoding too much data into the subject.

• Encode (business) intent into the subject, not technical details.

Pragmatic:

Maybe not so useful:

• NATS messages support headers. These can be used for additional metadata. There are subscription modes, which deliver headers only, allowing for efficient scanning of metadata in the message flow.

Matching a single token

The first wildcard is * which will match a single token. For example, if an application wanted to listen for eastern time zones, they could subscribe to time.*.east, which would match time.us.east and time.eu.east. Note that * can not match a substring within a token time.New*.east.

Matching multiple tokens

The second wildcard is > which will match one or more tokens, and can only appear at the end of the subject. For example, time.us.> will match time.us.east and time.us.east.atlanta, while time.us.* would only match time.us.east since it can't match more than one token.

Monitoring and wire taps

Subject to your security configuration, wildcards can be used for monitoring by creating something called a wire tap. In the simplest case, you can create a subscriber for >. This application will receive all messages -- again, subject to security settings -- sent on your NATS cluster.

Mixing wildcards

The wildcard * can appear multiple times in the same subject. Both types can be used as well. For example, *.*.east.> will receive time.us.east.atlanta.

Characters allowed and recommended for subject names

For compatibility across clients and ease of maintaining configuration files, we recommend using alphanumeric characters, - (dash) and _ (underscore) ASCII characters for subject and other entity names created by the user.

UTF-8 (UTF8) characters are supported in subjects. Please use UTF-8 characters at your own risk. Using multilingual names for technical entities can create many issues for editing, configuration files, display, and cross-border collaboration.

The rules and recommendations here apply to ALL system names, subjects, streams, durables, buckets, keys (in key-value stores), as NATS will create API subjects that contain those names. NATS will enforce these constraints in most cases, but we recommend not relying on this.

• Allowed characters: Any Unicode character except null, space, ., * and >

• Recommended characters: (a - z), (A - Z), (0 - 9), - and _ (names are case sensitive, and cannot contain whitespace).

• Naming Conventions If you want to delimit words, use either CamelCase as in MyServiceOrderCreate or - and _ as in my-service-order-create

• Special characters: The period . (which is used to separate the tokens in the subject) and * and also > (the * and > are used as wildcards) are reserved and cannot be used.

• Reserved names: By convention subject names starting with a $ are reserved for system use (e.g. subject names starting with $SYS or $JS or $KV, etc...). Many system subjects also use _ (underscore) (e.g. _INBOX , KV_ABC, OBJ_XYZ etc.)

Good names

Deprecated subject names

Forbidden stream names

Pedantic mode

By default, for the sake of efficiency, subject names are not verified during message publishing. In particular, when generating subjects programmatically, this will result in illegal subjects which cannot be subscribed to. E.g. subjects containing wildcards may be ignored.

To enable subject name verification, activate pedantic mode in the client connection options.

What is NATS? NATS is a connective technology that powers modern distributed systems. A connective technology is responsible for addressing, discovery and exchanging of messages that drive the common patterns in distributed systems; asking and answering questions, aka services/microservices, and making and processing statements, or stream processing.

Challenges faced by modern distributed systems Modern distributed systems are defined by an ever increasing number of hyper-connected moving parts and the additional data they generate. They employ both services and streams to drive business value. They are also being defined by location independence and mobility, and not just for things we would typically recognize as front end technologies. Today’s systems and the backend processes, microservices and stream processing are being asked to be location independent and mobile as well, all while being secure.

These modern systems present challenges to technologies that have been used to connect mobile front ends to fairly static backends. These incumbent technologies typically manage addressing and discovery via hostname (DNS) or IP and port, utilize a 1:1 communication pattern, and have multiple different security patterns for authentication and authorization. Although not perfect, incumbent technologies have been good enough in many situations, but times are changing quickly. As microservices, functions, and stream processing are being asked to move to the edge, these technologies and the assumptions they make are being challenged.

What makes the NATS connective technology unique for these modern systems?

Effortless M:N connectivity: NATS manages addressing and discovery based on subjects and not hostname and ports. Defaulting to M:N communications, which is a superset of 1:1, meaning it can do 1:1 but can also do so much more. If you have a 1:1 system that is successful in development, ask how many other moving parts are required for production to work around the assumption of 1:1? Things like load balancers, log systems, and network security models, as well as proxies and sidecars. If your production system requires all of these things just to get around the fact that the connective technology being used, e.g. HTTP or gRPC, is 1:1, it’s time to give NATS.io a look.

Deploy anywhere: NATS can be deployed nearly anywhere; on bare metal, in a VM, as a container, inside K8S, on a device, or whichever environment you choose. NATS runs well within deployment frameworks or without.

Secure: Similarly, NATS is secure by default and makes no requirements on network perimeter security models. When you start considering mobilizing your backend microservices and stream processors, many times the biggest roadblock is security.

Scalable, Future-Proof Deployments

NATS infrastructure and clients communicate all topology changes in real-time. This means that NATS clients do not need to change when NATS deployments change. Having to change clients with deployments would be like having to reboot your phone every time your cell provider added or changed a cell tower. This sounds ridiculous of course, but think about how many systems today have their front ends tied so closely to the backend, that any change requires a complete front end reboot or at least a reconfiguration. NATS clients and applications need no such change when backend servers are added and removed and changed. Even DNS is only used to bootstrap first contact, after that, NATS handles endpoint locations transparently.

Hybrid Deployments

Another advantage to utilizing a NATS is that it allows a hybrid mix of SaaS/Utility computing with separately owned and operated systems. Meaning you can have a shared NATS service with core microservices, streams and stream processing be extended by groups or individuals who have a need to run their own NATS infrastructure. You are not forced to choose one or the other.

Adaptability

Today’s systems will fall short with new demands. As modern systems continue to evolve and utilize more components and process more data, supporting patterns beyond 1:1 communications, with addressing and discovery tied to DNS is critical. Foundational technologies like NATS promise the most return on investment. Incumbent technologies will not work as modern systems unify cloud, Edge, IoT and beyond. NATS does.

Use Cases

NATS can run anywhere, from large servers and cloud instances, through edge gateways and even IoT devices. Use cases for NATS include:

• Cloud Messaging

  • Services (microservices, service mesh)

  • Event/Data Streaming (observability, analytics, ML/AI)

• Command and Control

  • IoT and Edge

  • Telemetry / Sensor Data / Command and Control

• Augmenting or Replacing Legacy Messaging Systems

This guide is tailored for existing NATS users upgrading from NATS version 2.9.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement.

Upgrade considerations

Client versions

Although all existing client versions will work, new client versions will expose additional options used to leverage new features. The minimum client versions that have full 2.10.0 support include:

• nats.net.v2 - Coming soon!

• nats.py - Coming soon!

• nats.c - Coming soon!

Helm charts

Downgrade warnings

For critical infrastructure like NATS, zero downtime upgrades are table stakes. Although the best practice for all infrastructure like this is for users to thoroughly test a new release against your specific workloads, inevitably there are cases where an upgrade occurs in production followed by a decision to downgrade. This is never recommended and can cause more harm than good for most infrastructure and data systems.

Below are a few important considerations if downgrading is required.

Storage format changes

2.10.0 brings on-disk storage changes which bring significant performance improvements. These are not compatible with previous versions of the NATS Server. If an upgrade is performed to a server with existing stream data on disk, followed by a downgrade, the older version server will not understand the stream data in the new format.

However, being mindful of the possibility of the need to downgrade, a special version of the 2.9.x series was released with awareness of key changes in the new storage format, allowing it to startup properly.

The takeaway is that if a downgrade is the only resort, it must be to 2.9.22 or later to ensure storage format changes are handled appropriately.

Stream and consumer config options

There are new stream and consumer configuration options that could be problematic if a downgrade occurs since previous versions of the server have no awareness of them. Examples include:

• Multi-filter consumers - Downgrading would result in no filter being applied since the new field is configured as a list rather than a single string.

• Subject-transform on streams - Downgrading would result in the subject transform not being applied since the server has no awareness of it.

• Compression on streams - Downgrading when compression is enabled on streams will cause those streams to become unloadable since the older server versions will not understand the compression being used.

Features

Platforms

Reload

JetStream

Subject mapping

• The requirement to use all wildcard tokens in subject mapping or transforms has been relaxed. This can be applied to config or account-based subject mapping, stream subject transforms, and stream republishing, but not on subject mappings that are associated with stream and service import/export between accounts.

Streams

• A ts field has been added to stream info responses indicating the server time of the snapshot. This was added to allow for local time calculations relying on the local clock.

• An array of subject-transforms (subject filter + subject transform destination) can be added to a mirror or source configuration (can not use the single subject filter/subject transform destination fields at the same time as the array).

• A stream configured with sources can source from the same stream multiple times when distinct filter+transform options are used, allowing for some messages of a stream to be sourced more than once.

Consumers

• A ts field has been added to consumer info responses indicating the server time of the snapshot. This was added to allow for local time calculations without relying on the local clock.

Key-value

• A bucket configured as a mirror or sourcing from other buckets

Object store

Authn/Authz

Monitoring

MQTT

Clustering

Leafnodes

Windows

Improvements

Reload

Streams

Consumers

• A new header has been added on a fetch response that indicates to clients the fetch has been fulfilled without requiring clients to rely on heartbeats. It avoids some conditions in which the client would issue fetch requests that could go over limits or have more fetch requests pending than required.

Leafnodes

• Previously, a leafnode configured with two or more remotes binding to the same hub account would be rejected. This restriction has been relaxed since each remote could be binding to a different local account.

MQTT

Prerequisite: enabling JetStream

If you are running a local nats-server stop it and restart it with JetStream enabled using nats-server -js (if that's not already done)

You can then check that JetStream is enabled by using

nats account info

Account Information

                           User: 
                        Account: $G
                        Expires: never
                      Client ID: 5
                      Client IP: 127.0.0.1
                            RTT: 128µs
              Headers Supported: true
                Maximum Payload: 1.0 MiB
                  Connected URL: nats://127.0.0.1:4222
              Connected Address: 127.0.0.1:4222
            Connected Server ID: NAMR7YBNZA3U2MXG2JH3FNGKBDVBG2QTMWVO6OT7XUSKRINKTRFBRZEC
       Connected Server Version: 2.11.0-dev
                 TLS Connection: no

JetStream Account Information:

Account Usage:

                        Storage: 0 B
                         Memory: 0 B
                        Streams: 0
                      Consumers: 0

Account Limits:

            Max Message Payload: 1.0 MiB

  Tier: Default:

      Configuration Requirements:

        Stream Requires Max Bytes Set: false
         Consumer Maximum Ack Pending: Unlimited

      Stream Resource Usage Limits:

                               Memory: 0 B of Unlimited
                    Memory Per Stream: Unlimited
                              Storage: 0 B of Unlimited
                   Storage Per Stream: Unlimited
                              Streams: 0 of Unlimited
                            Consumers: 0 of Unlimited

If you see the below then JetStream is not enabled

JetStream Account Information:

   JetStream is not supported in this account

1. Creating a stream

Let's start by creating a stream to capture and store the messages published on the subject "foo".

Enter nats stream add <Stream name> (in the examples below we will name the stream "my_stream"), then enter "foo" as the subject name and hit return to use the defaults for all the other stream attributes:

nats stream add my_stream

? Subjects foo
? Storage file
? Replication 1
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
Stream my_stream was created

Information for Stream my_stream created 2024-06-07 12:29:36

              Subjects: foo
              Replicas: 1
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 0
         Last Sequence: 0
      Active Consumers: 0

You can then check the information about the stream you just created:

nats stream info my_stream

Information for Stream my_stream created 2024-06-07 12:29:36

              Subjects: foo
              Replicas: 1
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 0
         Last Sequence: 0
      Active Consumers: 0

2. Publish some messages into the stream

Let's now start a publisher

nats pub foo --count=1000 --sleep 1s "publication #{{.Count}} @ {{.TimeStamp}}"

As messages are being published on the subject "foo" they are also captured and stored in the stream, you can check that by using nats stream info my_stream and even look at the messages themselves using nats stream view my_stream or nats stream get my_stream

3. Creating a consumer

Now at this point if you create a 'Core NATS' (i.e. non-streaming) subscriber to listen for messages on the subject 'foo', you will only receive the messages being published after the subscriber was started, this is normal and expected for the basic 'Core NATS' messaging. In order to receive a 'replay' of all the messages contained in the stream (including those that were published in the past) we will now create a 'consumer'

We can administratively create a consumer using the 'nats consumer add ' command, in this example we will name the consumer "pull_consumer", and we will leave the delivery subject to 'nothing' (i.e. just hit return at the prompt) because we are creating a 'pull consumer' and select all for the start policy, you can then just use the defaults and hit return for all the other prompts. The stream the consumer is created on should be the stream 'my_stream' we just created above.

nats consumer add

? Consumer name pull_consumer
? Delivery target (empty for Pull Consumers) 
? Start policy (all, new, last, subject, 1h, msg sequence) all
? Acknowledgment policy explicit
? Replay policy instant
? Filter Stream by subjects (blank for all) 
? Maximum Allowed Deliveries -1
? Maximum Acknowledgments Pending 0
? Deliver headers only without bodies No
? Add a Retry Backoff Policy No
? Select a Stream my_stream
Information for Consumer my_stream > pull_consumer created 2024-06-07T12:32:09-05:00

Configuration:

                    Name: pull_consumer
               Pull Mode: true
          Deliver Policy: All
              Ack Policy: Explicit
                Ack Wait: 30.00s
           Replay Policy: Instant
         Max Ack Pending: 1,000
       Max Waiting Pulls: 512

State:

  Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
    Acknowledgment Floor: Consumer sequence: 0 Stream sequence: 0
        Outstanding Acks: 0 out of maximum 1,000
    Redelivered Messages: 0
    Unprocessed Messages: 74
           Waiting Pulls: 0 of maximum 512

You can check on the status of any consumer at any time using nats consumer info or view the messages in the stream using nats stream view my_stream or nats stream get my_stream, or even remove individual messages from the stream using nats stream rmm

3. Subscribing from the consumer

Now that the consumer has been created and since there are messages in the stream we can now start subscribing to the consumer:

nats consumer next my_stream pull_consumer --count 1000

This will print out all the messages in the stream starting with the first message (which was published in the past) and continuing with new messages as they are published until the count is reached.

Note that in this example we are creating a pull consumer with a 'durable' name, this means that the consumer can be shared between as many consuming processes as you want. For example instead of running a single nats consumer next with a count of 1000 messages you could have started two instances of nats consumer each with a message count of 500 and you would see the consumption of the messages from the consumer distributed between those instances of nats

Replaying the messages again

Once you have iterated over all the messages in the stream with the consumer, you can get them again by simply creating a new consumer or by deleting that consumer (nats consumer rm) and re-creating it (nats consumer add).

4. Cleaning up

You can clean up a stream (and release the resources associated with it (e.g. the messages stored in the stream)) using nats stream purge

You can also delete a stream (which will also automatically delete all of the consumers that may be defined on that stream) using nats stream rm

JetStream, the persistence layer of NATS, not only allows for the higher qualities of service and features associated with 'streaming', but it also enables some functionalities not found in messaging systems.

Managing a Key Value store

1. Create a bucket, which corresponds to a stream in the underlying storage. Define KV/Stream limits as appropriate

2. Use the operation below.

Map style operations

You can use KV buckets to perform the typical operations you would expect from an immediately consistent key/value store:

• put: associate a value with a key

• get: retrieve the value associated with a key

• delete: clear any value associated with a key

• purge: clear all the values associated with all keys

• keys: get a copy of all of the keys (with a value or operation associated with it)

Atomic operations used for locking and concurrency control

• create: associate the value with a key only if there is currently no value associated with that key (i.e. compare to null and set)

• update: compare and set (aka compare and swap) the value for a key

Limiting size, TTL etc.

You can set limits for your buckets, such as:

• the maximum size of the bucket

• the maximum size for any single value

• a TTL: how long the store will keep values for

Treating the Key Value store as a message stream

Finally, you can even do things that typically can not be done with a Key/Value Store:

• watch: watch for changes happening for a key, which is similar to subscribing (in the publish/subscribe sense) to the key: the watcher receives updates due to put or delete operations on the key pushed to it in real-time as they happen

• watch all: watch for all the changes happening on all the keys in the bucket

• history: retrieve a history of the values (and delete operations) associated with each key over time (by default the history of buckets is set to 1, meaning that only the latest value/operation is stored)

Notes

JetStream, the persistence layer of NATS, not only allows for the higher qualities of service and features associated with 'streaming', but it also enables some functionalities not found in messaging systems.

One such feature is the Object store functionality, which allows client applications to create buckets (corresponding to streams) that can store a set of files. Files are stored and transmitted in chunks, allowing files of arbitrary size to be transferred safely over the NATS infrastructure.

Note: Object store is not a distributed storage system. All files in a bucket will need to fit on the target file system.

Basic Capabilities

The Object Store implements a chunking mechanism, allowing you to for example store and retrieve files (i.e. the object) of any size by associating them with a path or file name as the key.

• add a bucket to hold the files.

• put Add a file to the bucket

• get Retrieve the file and store it to a designated location

• del Delete a file

Advanced Capabilities

• watch Subscribe to changes in the bucket. Will receive notifications on successful put and del operations.

This guide is tailored for existing NATS users upgrading from NATS version v2.10.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement.

Features

Observability

• Distributed message tracing: Users can now trace messages as they move through the system by setting a Nats-Trace-Dest header to an inbox subject. Servers on the message path will return events to the provided subject that report each time a message enters or leaves a server, by which connection type, when subject mappings occur, or when messages traverse an account import/export boundary. Additionally, the Nats-Trace-Only header (if set to true) will allow tracing events to propagate on a specific subject without delivering them to subscribers of that subject.

Streams

• Subject delete markers on MaxAge: The SubjectDeleteMarkerTTL stream configuration option now allows for the placement of delete marker messages in the stream when the configured MaxAge limit causes the last message for a given subject to be deleted. The delete markers include a Nats-Marker-Reason header explaining which limit was responsible for the deletion.

• Stream ingest rate limiting: New options max_buffered_size and max_buffered_msgs in the jetstream configuration block enable rate limiting on Core NATS publishing into JetStream streams, protecting the system from overload.

Consumers

• Pull consumer priority groups: Pull consumers now support priority groups with pinning and overflow, enabling flexible failover and priority management when multiple clients are pulling from the same consumer. Configurable policies based on the number of pending messages on the consumer, or the number of pending acks, can control when messages overflow from one client to another, enabling new design patterns or regional awareness.

• Consumer pausing: Message delivery to consumers can be temporarily suspended using the new pause API endpoint (or the PauseUntil configuration option when creating), ideal for maintenance or migrations. Message delivery automatically resumes once the configured deadline has passed. Consumer clients continue to receive heartbeat messages as usual to ensure that they do not surface errors during the pause.

Operations

• Replication traffic in asset accounts: Raft replication traffic can optionally be moved into the same account in which replicated assets live on a per-account basis, rather than being sent and received in the system account. When combined with multiple route connections, this can help to reduce latencies and avoid head-of-line blocking issues that may occur in heavily-loaded multi-tenant or multi-account deployments.

• TLS first on leafnode connections: A new handshake_first in the leafnode tls block allows setting up leafnode connections that perform TLS negotiation first, before any other protocol handshakes take place.

• Configuration state digest: A new -t command line flag on the server binary can generate a hash of the configuration file. The config_digest item in varz displays the hash of the currently running configuration file, making it possible to check whether a configuration file has changed on disk compared to the currently running configuration.

• TPM encryption on Windows: When running on Windows, the filestore can now store encryption keys in the TPM, useful in environments where physical access may be a concern.

MQTT

• SparkplugB: The built-in MQTT support is now compliant with SparkplugB Aware, with support for NBIRTH and NDEATH messages.

Improvements

• Replicated delete proposals: Message removals in clustered interest-based or workqueue streams are now propagated via Raft to guarantee consistent removal order across replicas, reducing a number of possible ways that a cluster failure can result in de-synced streams.

• Metalayer, stream and consumer consistency: A new leader now only responds to read/write requests after synchronizing with its Raft log, preventing desynchronization between KV key updates and the stream during leader changes.

• Replicated consumer reliability: Replicated consumers now consistently redeliver unacknowledged messages after a leader change.

• Consumer starting sequence: The consumer starting sequence is now always respected, except for internal hidden consumers for sources/mirrors.

Upgrade Considerations

Stream ingest rate limiting

The NATS Server can now return a 429 error with type JSStreamTooManyRequests when too many messages have been queued up for a stream. It should not generally be possible to hit this limit while using JetStream publishes and waiting for PubAcks, but may trigger if trying to publish into JetStream using Core NATS publishes without waiting for PubAcks, which is not advised.

The new max_buffered_size and max_buffered_msgs options control how many messages can be queued for each stream before the rate limit is hit, therefore if needed, you can increase these limits on your deployments. The default values for max_buffered_size and max_buffered_msgs are 128MB and 10,000 respectively, whereas in v2.10 these were unlimited.

You can detect in the server logs whether running into a queue limit with the following warning:

If your application starts to log the above warnings then you can first try to increase the limits to higher values while investigating the fast publishers, for example:

Replicated delete proposals

Since stream deletes are now replicated through group proposals in a replicated stream, there may be a slight increase in replication traffic on this version.

JetStream healthcheck

The js-server-only healthcheck no longer checks for the health of the metaleader on v2.11.0. Since this healthcheck was designed to detect the server readiness (or in k8s for the readiness probe) checking the metaleader would sometimes cause a NATS server to be considered unhealthy when restarting the servers. In v2.11, this should no longer be an issue. If the previous behavior from v2.10 is preferred, there is a new healthcheck option js-meta-only which can be used to check whether the meta group is healthy.

Exit code

Earlier versions of the NATS Server would return an exit code 1 when gracefully shut down, i.e. after SIGTERM. From v2.11, an exit code of 0 (zero) will now be returned instead.

Server, cluster and gateway names

Configurations that have server, cluster, and gateway names with spaces are now considered invalid, as this can cause problems at the protocol level. A server running NATS v2.11 will fail to start with spaces configured in these names. Please ensure that spaces are not used in server, cluster or gateway names.

Downgrade Considerations

Stream state

When downgrading from v2.11 to v2.10, the stream state files on disk will be rebuilt due to a change in the format of these files in v2.11. This requires re-scanning all stream message blocks, which may use higher CPU than usual and will likely take longer for the restarted node to report healthy. This will only happen on the first restart after downgrading and will not result in data loss.

NATS 2.0 was the largest feature release since the original code base for the server was released. NATS 2.0 was created to allow a new way of thinking about NATS as a shared utility, solving problems at scale through distributed security, multi-tenancy, larger networks, and secure sharing of data.

Rationale

NATS 2.0 was created to address problems in large scale distributed computing.

It is difficult at best to combine identity management end-to-end (or end-to-edge), with data sharing, while adhering to policy and compliance. Current distributed systems increase significantly in operational complexity as they scale upward. Problems arise around service discovery, connectivity, scaling for volume, and application onboarding and updates. Disaster recovery is difficult, especially as systems have evolved to operate in silos defined by technology rather than business needs. As complexity increases, systems become expensive to operate in terms of time and money. They become fragile making it difficult to deploy services and applications hindering innovation, increasing time to value and total cost of ownership.

We decided to:

• Reduce total cost of ownership: Users want reduced TCO for their

  distributed systems. This is addressed by an easy to use technology that

  can operate at global scale with simple configuration and a resilient

  and cloud-native architecture.

• Decrease Time to Value: As systems scale, time to value increases.

  Operations resist change due to risk in touching a complex and fragile

  system. Providing isolation contexts can help mitigate this.

• Support manageable large scale deployments: No data silos defined by

  software, instead easily managed through software to provide exactly what the

  business needs. We wanted to provide easy to configure disaster recovery.

• Decentralize security: Provide security supporting one

  technology end-to-end where organizations may self-manage making it

  easier to support a massive number of endpoints.

To achieve this, we added a number of new features that are transparent to existing clients with 100% backward client compatibility.

Accounts

Accounts are securely isolated communication contexts that allow multi-tenancy spanning a NATS deployment. Accounts allow users to bifurcate technology from business driven use cases, where data silos are created by design, not software limitations. When a client connects, it specifies an account or will default to authentication with a global account.

At least some services need to share data outside of their account. Data can be securely shared between accounts with secure services and streams. Only mutual agreement between account owners permit data flow, and the import account has complete control over its own subject space.

This means within an account, limitations may be set and subjects can be used without worry of collisions with other groups or organizations. Development groups choose any subjects without affecting the rest of the system, and open up accounts to export or import only the services and streams they need.

Accounts are easy, secure, and cost effective. There is one NATS deployment to manage, but organizations and development teams can self manage with more autonomy reducing time to value with faster, more agile development practices.

Service and Streams

Services and streams are mechanisms to share messages between accounts.

Think of a service as an RPC endpoint into an account. Behind that account there might be many microservices working in concert to handle requests, but from outside the account there is simply one subject exposed.

Service definitions share an endpoint:

• Export a service to allow other accounts to import

• Import a service to allow requests to be sent securely and seamlessly to another account

Use cases include most applications - anything that accepts a request and returns a response.

Stream definitions allow continuous data flow between accounts:

• Export a stream to allow egress

• Import a stream to allow ingress

Use cases include Observability, Metrics, and Data analytics. Any application or endpoint reading a stream of data.

Note that services and streams operate with zero client configuration or API changes. Services may even move between accounts, entirely transparent to end clients.

System Accounts

The system account publishes system messages under established subject patterns. These are internal NATS system messages that may be useful to operators.

Server initiated events and data include:

• Client connection events

• Account connection status

• Authentication errors

• Leaf node connection events

• Server stats summary

Tools and clients with proper privileges can request:

• Service statistics

• Server discovery and metrics

Account servers will also publish messages when an account changes.

With this information and system metadata you can build useful monitoring and anomaly detection tools.

Global Deployments

NATS 2.0 supports global deployments, allowing for global topologies that optimize for WANs while extend to the edge or devices.

Self Healing

While self healing features have been part of NATS 1.X releases, we ensured they continue to work in global deployments. These include:

• Client and server connections automatically reconnect

• Auto-Discovery where servers exchange server topology changes with each

  other and with clients, in real time with zero configuration changes and

  zero downtime while being entirely transparent to clients. Clients can

  failover to servers they were not originally configured with.

• NATS server clusters dynamically adjust to new or removed servers allowing

  for seamless rolling upgrades and scaling up or down.

Superclusters

Conceptually, superclusters are clusters of NATS clusters. Create superclusters to deploy a truly global NATS network. Superclusters use a novel spline based technology with a unique approach to topology, keeping one hop semantics and optimizing WAN traffic through optimistic sends with interest graph pruning. Superclusters provide transparent, intelligent support for geo-distributed queue subscribers.

Disaster Recovery

Superclusters inherently support disaster recovery. With geo-distributed queue subscribers, local clients are preferred, then an RTT is used to find the lowest latency NATS cluster containing a matching queue subscriber in the supercluster.

What does this mean?

Let's say you have a set of load balanced services in US East Coast (US-EAST), another set in the EU (EU-WEST), and a supercluster consisting of a NATS cluster in US-EAST connected to a NATS cluster in EU-WEST. Clients in the US would connect to a US-EAST, and services connected to that cluster would service those clients. Clients in Europe would automatically use services connected to EU-WEST. If the services in US-EAST disconnect, clients in US-EAST will begin using services in EU-WEST.

Once the Eastern US services have reconnected to US-EAST, those services will immediately begin servicing the Eastern US clients since they're local to the NATS cluster. This is automatic and entirely transparent to the client. There is no extra configuration in NATS servers.

This is zero configuration disaster recovery.

Leaf Nodes

Leaf nodes are NATS servers running in a special configuration, allowing hub and spoke topologies to extend superclusters.

Leaf nodes can also bridge separate security domains. e.g. IoT, mobile, web. They are ideal for edge computing, IoT hubs, or data centers that need to be connected to a global NATS deployment. Local applications that communicate using the loopback interface with physical VM or Container security can leverage leaf nodes as well.

Leaf nodes:

• Transparently and securely bind to a remote NATS account

• Securely bridge specific local data to a wider NATS deployment

• Are 100% transparent to clients which remain simple, lightweight, and easy to develop

• Allow for a local security scheme while using new NATS security features globally

• Can create a DMZ between a local NATS deployment and external NATS cluster or supercluster.

Decentralized Security

Operators, Accounts, and Users

NATS 2.0 Security consists of defining Operators, Accounts, and Users within a NATS deployment.

• An Operator provides the root of trust for the system, may represent

  a company or enterprise

  • Creates Accounts for account administrators. An account represents

    an organization, business unit, or service offering with a secure context

    within the NATS deployment, for example an IT system monitoring group, a

    set of microservices, or a regional IoT deployment. Account creation

    would likely be managed by a central group.

• Accounts define limits and may securely expose services and streams.

  • Account managers create Users with permissions

• Users have specific credentials and permissions.

Trust Chain

• Operators are represented by a self signed JWT and is the only thing that

  is required to be configured in the server. This JWT is usually signed by a

  master key that is kept offline. The JWT will contain valid signing keys that

  can be revoked with the master updating this JWT.

  • Operators will sign Account JWTs with various signing keys.

  • Accounts sign User JWTs, again with various signing keys.

• Clients or leaf nodes present User credentials and a signed nonce when connecting.

  • The server uses resolvers to obtain JWTs and verify the client trust chain.

This allows for rapid change of permissions, authentication and limits, to a secure multi-tenant NATS system.

NATS 2.2 is the largest feature release since version 2.0. The 2.2 release provides highly scalable, highly performant, secure and easy-to-use next generation streaming in the form of JetStream, allows remote access via websockets, has simplified NATS account management, native MQTT support, and further enables NATS toward our goal of securely democratizing streams and services for the hyperconnected world we live in.

Next Generation Streaming

JetStream is the next generation streaming platform for NATS, highly resilient, highly available, and easy to use. We’ve spent a long time listening to our community, learning from our experiences, looking at the needs of today, and thinking deeply about the needs of tomorrow. We built JetStream to address these needs.

JetStream:

• is easy to deploy and manage, built into the NATS server

• simplifies and accelerates development

• supports wildcard subjects

• supports at least once delivery and exactly once within a window

• is horizontally scalable at runtime with no interruptions

• persists data via streams and delivers or replays via consumers

• supports multiple patterns to consume data on the same stream

• supports push and pull modes when consuming messages

• is account aware

• allows for detailed granularity of security, by stream, by consumer, by function

Security and Simplified Account Management

Account management just became much easier. This version of NATS has a built-in account management system, eliminating the need to set up an account manager when not using the memory account resolver. With automated default system account generation, and the ability to preload accounts, simply enable a set of servers in your deployment to be account resolvers or account resolver caches, and they will handle public account information provided to the NATS system through the NATS nsc tooling. Have enterprise-scale account management up and running in minutes.

CIDR Block Account Restrictions

By specifying a CIDR block restriction for a user, policy can be applied to limit connections from clients within a certain range or set of IP addresses. Use this as another layer of security atop user credentials to better secure your distributed system. Ensure your applications can only connect from within a specific cloud, enterprise, geographic location, virtual or physical network.

Time-Based Account Restrictions

Default User Permissions

WebSockets

Native MQTT Support

Seamlessly integrate existing IoT deployments using MQTT 3.1.1 with a cloud-native NATS deployment. Add a leaf node that is MQTT enabled and instantly send and receive messages to your MQTT applications and devices from a NATS deployment whether it be edge, single-cloud, multi-cloud, on-premise, or any combination thereof.

Build Better Systems

We’ve added a variety of features to allow you to build a more resilient, secure, and simply better system at scale.

Message Headers

We’ve added the ability to optionally use headers, following the HTTP semantics familiar to developers. Headers naturally apply overhead, which was why we resisted adding them for so long. By creating new internal protocol messages transparent to developers, we maintain the extremely fast processing of simple NATS messages that we have always had while supporting headers for those who would like to leverage them. Adding headers to messages allows you to provide application-specific metadata, such as compression or encryption-related information, without touching the payload. We also provide some NATS specific headers for use in JetStream and other features.

Seamless Maintenance with Lame Duck Notifications

React Quicker with No-Responder Notifications

Why wait for timeouts when services aren’t available? When a request is made to a service (request-reply) and the NATS Server knows there are no services available the server will short circuit the request. A “no-responders” protocol message will be sent back to the requesting client which will break from blocking API calls. This allows applications to immediately react which further enables building a highly responsive system at scale, even in the face of application failures and network partitions.

Subject Mapping and Traffic Shaping

Account Monitoring - More Meaningful Metrics

Software applications and services need to exchange data. NATS is an infrastructure that allows such data exchange, segmented in the form of messages. We call this a "message oriented middleware".

With NATS, application developers can:

• Effortlessly build distributed and scalable client-server applications.

• Store and distribute data in realtime in a general manner. This can flexibly be achieved across various environments, languages, cloud providers and on-premises systems.

NATS Client Applications

Developers use one of the NATS client libraries in their application code to allow them to publish, subscribe, request and reply between instances of the application or between completely separate applications. Those applications are generally referred to as 'client applications' or sometimes just as 'clients' throughout this manual (since from the point of view of the NATS server, they are clients).

NATS Service Infrastructure

The NATS services are provided by one or more NATS server processes that are configured to interconnect with each other and provide a NATS service infrastructure. The NATS service infrastructure can scale from a single NATS server process running on an end device (the nats-server process is less than 20 MB in size!) all the way to a public global super-cluster of many clusters spanning all major cloud providers and all regions of the world such as Synadia's NGS.

Connecting NATS Client applications to the NATS servers

To connect a NATS client application with a NATS service, and then subscribe or publish messages to subjects, it only needs to be configured with:

Simple messaging design

NATS makes it easy for applications to communicate by sending and receiving messages. These messages are addressed and identified by subject strings, and do not depend on network location.

Data is encoded and framed as a message and sent by a publisher. The message is received, decoded, and processed by one or more subscribers.

With this simple design, NATS lets programs share common message-handling code, isolate resources and interdependencies, and scale by easily handling an increase in message volume, whether those are service requests or stream data.

NATS Quality of service (QoS)

NATS offers multiple qualities of service, depending on whether the application uses just the Core NATS functionality or also leverages the added functionalities enabled by NATS JetStream (JetStream is built into nats-server but may not be enabled on all service infrastructures).

• At most once QoS: Core NATS offers an at most once quality of service. If a subscriber is not listening on the subject (no subject match), or is not active when the message is sent, the message is not received. This is the same level of guarantee that TCP/IP provides. Core NATS is a fire-and-forget messaging system. It will only hold messages in memory and will never write messages directly to disk.

This feature comparison is a summary of a few of the major components in several of the popular messaging technologies of today. This is by no means an exhaustive list and each technology should be investigated thoroughly to decide which will work best for your implementation.

In this comparison, we will be featuring NATS, Apache Kafka, RabbitMQ, Apache Pulsar, and gRPC.

Language and Platform Coverage

Built-in Patterns

Delivery Guarantees

Multi-tenancy and Sharing

AuthN

AuthZ

Message Retention and Persistence

High Availability and Fault Tolerance

Deployment

Monitoring

Management

Integrations

References

Roadmap for future releases

Server release v2.11.0

Check out the:

Server release v2.10.0

Check out the:

Server release v2.9.0

Server release v2.8.0

LeafNode

Support for a min_version in the leafnodes{} that would reject servers with a lower version. Note that this would work only for servers that are v2.8.0 and above.

Monitoring

• Server version in monitoring landing page.

• Logging to /healthz endpoint when failure occurs.

• MQTT and Websocket blocks in the /varz endpoint.

JetStream

• Consumer check added to healthz endpoint.

• Max stream bytes checks.

• Ability to limit a consumer's MaxAckPending value.

• Allow streams and consumers to migrate between clusters. This feature is considered "beta".

• New unique_tag option in jetstream{} configuration block to prevent placing a stream in the same availability zone twice.

• Stream Alternates field in StreamInfo response. They provide a priority list of mirrors and the source in relation to where the request originated.

• Deterministic subject tokens to partition mapping.

For full release information, see links below;

Server release v2.7.0

Notice for JetStream Users

Configuration

Ability to configure account limits (max_connections, max_subscriptions, max_payload, max_leafnodes) in server configuration file.

JetStream

• Overflow placement for streams. A stream can now be placed in the closest cluster from the origin request if it can be placed there.

• Support for ephemeral Pull consumers (client libraries will need to be updated to allow those).

• New consumer configuration options

  • For Pull Consumers: MaxRequestBatch to limit the batch size any client can request MaxRequestExpires to limit the expiration any client can request

  • For ephemeral consumers: InactiveThreshold duration that instructs the server to cleanup ephemeral consumers that are inactive for that long.

• Ability to configure max_file_store and max_memory_store in the jetstream{} block as strings with the following suffixes K, M, G and T, for instance: max_file_store: "256M".

• Support for the JWT field MaxBytesRequired, which defines a per-account maximum bytes for assets.

MQTT

Support for websocket protocol. MQTT clients must connect to the opened websocket port and add /mqtt to the URL path.

TLS

Ability to rate-limit the clients connections by adding the connection_rate_limit: <number of connections per seconds> in the tls{} top-level block.

For full release information, see links below;

Server release v2.6.0

Notice for JetStream Users

Notice for MQTT Users

Monitoring

• JetStream's reserved memory and memory used from accounts with reservations in /jsz and /varz endpoints

• Hardened systemd service

For full release information, see links below;

Server release v2.5.0

Notice for JetStream Users

MQTT/Monitoring

• MQTTClient in the /connz connections report and system events CONNECT and DISCONNECT. Ability to select on mqtt_client.

MQTT Improvement

• Sessions are now all stored inside a single stream, as opposed to individual streams, reducing resources usage.

MQTT Update

• Due to the aforementioned improvement described above, when an MQTT client connects for the first time after an upgrade to this server version, the server will migrate all individual $MQTT_sess_<xxxx> streams to a new $MQTT_sess stream for the user's account.

For full release information, see links below;

Server release v2.4.0

Notice for JetStream Users

With the latest release of the NATS server, we have fixed bugs around queue subscriptions and have restricted undesired behavior that could be confusing or introduce data loss by unintended/undefined behavior of client applications. If you are using queue subscriptions on a JetStream Push Consumer or have created multiple push subscriptions on the same consumer, you may be affected and need to upgrade your client version along with the server version. We’ve detailed the behavior with different client versions below.

With a NATS Server prior to v2.4.0 and client libraries prior to these versions: NATS C client v3.1.0, Go client v1.12.0, Java client 2.12.0-SNAPSHOT, NATS.js v2.2.0, NATS.ws v1.3.0, NATS.deno v1.2.0, NATS .NET 0.14.0-pre2:

• It was possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is not correct since each instance will receive the same copy of a message and acknowledgment is therefore meaningless since the first instance to acknowledge the message will prevent other instances to control if/when a message should be acknowledged.

• Similar to the first issue, it was possible to create many different queue groups for one single JetStream consumer.

• For queue subscriptions, if no consumer nor durable name was provided, the libraries would create ephemeral JetStream consumers, which meant that each member of the same group would receive the same message as the other members, which was not the expected behavior. Users assumed that 2 members subscribing to “foo” with the queue group named “bar” would load-balance the consumption of messages from the stream/consumer.

• It was possible to create a queue subscription on a JetStream consumer configured with heartbeat and/or flow control. This does not make sense because by definition, queue members would receive some (randomly distributed) messages, so the library would think that heartbeats are missed, and flow control would also be disrupted.

If above client libraries are not updated to the latest but the NATS Server is upgraded to v2.4.0:

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. Since the check is performed by the library (with the help of a new field called PushBound in the consumer information object set by the server), this misbehavior is still possible.

• Queue subscriptions will not receive any message. This is because the server now has a new field DeliverGroup in the consumer configuration, which won’t be set for existing JetStream consumers and by the older libraries, and detects interest (and starts delivering) only when a subscription on the deliver subject for a queue subscription matching the “deliver group” name is found. Since the JetStream consumer is thought to be a non-deliver-group consumer, the opposite happens: the server detects a core NATS queue subscription on the “deliver subject”, therefore does not trigger delivery on the JetStream consumer’s “deliver subject”.

The 2 other issues are still present because those checks are done in the updated libraries.

If the above client libraries are updated to the latest version, but the NATS Server is still to version prior to v2.4.0 (that is, up to v2.3.4):

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is because the JetStream consumer’s information retrieved by the library will not have the PushBound boolean set by the server, therefore will not be able to alert the user that they are trying to create multiple subscription instances for the same JetStream consumer.

• Queue subscriptions will fail because the consumer information returned will not contain the DeliverGroup field. The error will be likely to the effect that the user tries to create a queue subscription to a non-queue JetStream consumer. Note that if the application creates a queue subscription for a non-yet created JetStream consumer, then this call will succeed, however, adding new members or restarting the application with the now existing JetStream consumer will fail.

• Creating queue subscriptions without a named consumer/durable will now result in the library using the queue name as the durable name.

• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

For completeness, using the latest client libraries and NATS Server v2.4.0:

• Trying to start multiple non-queue subscriptions instances for the same JetStream consumer will now return an error to the effect that the user is trying to create a “duplicate subscription”. That is, there is already an active subscription on that JetStream consumer. It is now only possible to create a queue group for a JetStream consumer created for that group. The DeliverGroup field will be set by the library or need to be provided when creating the consumer externally.

• Trying to create a queue subscription without a durable nor consumer name results in the library creating/using the queue group as the JetStream consumer’s durable name.

• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

Note that if the server v2.4.0 recovers existing JetStream consumers that were created prior to v2.4.0 (and with older libraries), none of them will have a DeliverGroup, so none of them can be used for queue subscriptions. They will have to be recreated.

JetStream

• Domain to the content of a PubAck protocol

• PushBound boolean in ConsumerInfo to indicate that a push consumer is already bound to an active subscription

• DeliverGroup string in ConsumerConfig to specify which deliver group (or queue group name) the consumer is created for

• Warning log statement in situations where catchup for a stream resulted in an error

Monitoring

• The ability for normal accounts to access scoped connz information

Misc

• Operator option resolver_pinned_accounts to ensure users are signed by certain accounts

For full release information, see links below;

Server release v2.3.0

JetStream

• Richer API errors. JetStream errors now contain an ErrCode that uniquely describes the error.

• Ability to send more advanced Stream purge requests that can purge all messages for a specific subject

• Stream can now be configured with a per-subject message limit

• Encryption of JetStream data at rest

For full release information, see links below;

Server release v2.2.0

Server release v2.1.7

Monitoring Endpoints Available via System Services

Monitoring endpoints as listed in the table below are accessible as system services using the following subject pattern:

• $SYS.REQ.SERVER.<id>.<endpoint-name> (request server monitoring endpoint corresponding to endpoint name.)

• $SYS.REQ.SERVER.PING.<endpoint-name> (from all server request server monitoring endpoint corresponding to endpoint name - will return multiple messages)

Addition of no_auth_user Configuration

Configuration of no_auth_user allows you to refer to a configured user/account when no credentials are provided.

For full release information, see links below;

Server release v2.1.6

TLS Configuration for Account Resolver

This release adds the ability to specify TLS configuration for the account resolver.

Additional Trace & Debug Verbosity Options

Subscription Details in Monitoring Endpoints

We've added the option to include subscription details in monitoring endpoints /routez and /connz. For instance /connz?subs=detail will now return not only the subjects of the subscription, but the queue name (if applicable) and some other details.

Server release v2.1.4

Log Rotation

Server release v2.1.2

Queue Permissions

Server release v2.1.0

Service Latency Tracking

As services and service mesh functionality has become prominent, we have been looking at ways to make running scalable services on NATS.io a great experience. One area we have been looking at is observability. With publish/subscribe systems, everything is inherently observable, however we realized it was not as simple as it could be. We wanted the ability to transparently add service latency tracking to any given service with no changes to the application. We also realized that global systems, such as those NATS.io can support, needed something more than a single metric. The solution was to allow any sampling rate to be attached to an exported service, with a delivery subject for all collected metrics. We collect metrics that show the requestor’s view of latency, the responder’s view of latency and the NATS subsystem itself, even when requestor and responder are in different parts of the world and connected to different servers in a NATS supercluster.

Server release v2.0.4

Response Only Permissions

For services, the authorization for responding to requests usually included wildcards for _INBOX.> and possibly $GR.> with a supercluster for sending responses. What we really wanted was the ability to allow a service responder to only respond to the reply subject it was sent.

Response Types

Exported Services were originally tied to a single response. We added the type for the service response and now support singletons (default), streams and chunked. Stream responses represent multiple response messages, chunked represents a single response that may have to be broken up into multiple messages.

Request-Reply is a common pattern in modern distributed systems. A request is sent, and the application either waits on the response with a certain timeout, or receives a response asynchronously.

NATS makes Request-Reply simple and powerful

• NATS supports the Request-Reply pattern using its core communication mechanism — publish and subscribe. A request is published on a given subject using a reply subject. Responders listen on that subject and send responses to the reply subject. Reply subjects are called "inbox". These are unique subjects that are dynamically directed back to the requester, regardless of the location of either party.

• Multiple NATS responders can form dynamic queue groups. Therefore, it's not necessary to manually add or remove subscribers from the group for them to start or stop being distributed messages. It’s done automatically. This allows responders to scale up or down as per demand.

• NATS applications "drain before exiting" (processing buffered messages before closing the connection). This allows the applications to scale down without dropping requests.

• Since NATS is based on publish-subscribe, observability is as simple as running another application that can view requests and responses to measure latency, watch for anomalies, direct scalability and more.

• The power of NATS even allows multiple responses, where the first response is utilized and the system efficiently discards the additional ones. This allows for a sophisticated pattern to have multiple responders, reduce response latency and jitter.

The pattern

No responders

Most clients will represent this case by raising or returning an error. For example:

m, err := nc.Request("foo", nil, time.Second);
# err == nats.ErrNoResponders

We have provided Walkthroughs for you to try NATS (and JetStream) on your own. In order to follow along with the walkthroughs, you could choose one of these options:

• The nats CLI tool must be installed, and a local NATS server must be installed (or you can use a remote server you have access to).

• You can use Synadia's NGS.

• You could even use the demo server from where you installed NATS. This is accessible via nats://demo.nats.io (this is a NATS connection URL; not a browser URL. You pass it to a NATS client application).

Installing the NATS server locally (if needed)

Alternatively if you already know how to use NATS on a remote server, you only need to pass the server URL to nats using the -s option or preferably create a context using nats context add, to specify the server URL(s) and credentials file containing your user JWT.

Start the NATS server (if needed)

To start a simple demonstration server locally, simply run:

nats-server

(or nats-server -m 8222 if you want to enable the HTTP monitoring functionality)

When the server starts successfully, you will see the following messages:

[14524] 2021/10/25 22:53:53.525530 [INF] Starting nats-server
[14524] 2021/10/25 22:53:53.525640 [INF]   Version:  2.6.1
[14524] 2021/10/25 22:53:53.525643 [INF]   Git:      [not set]
[14524] 2021/10/25 22:53:53.525647 [INF]   Name:     NDAUZCA4GR3FPBX4IFLBS4VLAETC5Y4PJQCF6APTYXXUZ3KAPBYXLACC
[14524] 2021/10/25 22:53:53.525650 [INF]   ID:       NDAUZCA4GR3FPBX4IFLBS4VLAETC5Y4PJQCF6APTYXXUZ3KAPBYXLACC
[14524] 2021/10/25 22:53:53.526392 [INF] Starting http monitor on 0.0.0.0:8222
[14524] 2021/10/25 22:53:53.526445 [INF] Listening for client connections on 0.0.0.0:4222
[14524] 2021/10/25 22:53:53.526684 [INF] Server is ready

The NATS server listens for client connections on TCP Port 4222.

Core NATS is the foundational functionality in a NATS system. It operates on a publish-subscribe model using subject/topic-based addressing. This model offers two significant advantages: location independence and a default many-to-many (M:N) communication pattern. These fundamental concepts enable powerful and innovative solutions for common development patterns, such as microservices, without requiring additional technologies like load balancers, API gateways, or DNS configuration.

Publish-Subscribe

NATS implements a publish-subscribe message distribution model for one-to-many communication. A publisher sends a message on a subject and any active subscriber listening on that subject receives the message. Subscribers can also register interest in wildcard subjects that work a bit like a regular expression (but only a bit). This one-to-many pattern is sometimes called a fan-out.

Messages

Messages are composed of:

1. A subject.

2. A payload in the form of a byte array.

3. Any number of header fields.

4. An optional 'reply' address field.

Messages have a maximum size (which is set in the server configuration with max_payload). The size is set to 1 MB by default, but can be increased up to 64 MB if needed (though we recommend keeping the max message size to something more reasonable like 8 MB).

NATS Pub/Sub Walkthrough

This simple walkthrough demonstrates some ways in which subscribers listen on subjects, and publishers send messages on specific subjects.

Walkthrough prerequisites

1. Create Subscriber 1

In a shell or command prompt session, start a client subscriber program.

nats sub <subject>

Here, <subject> is a subject to listen on. It helps to use unique and well thought-through subject strings because you need to ensure that messages reach the correct subscribers even when wildcards are used.

For example:

nats sub msg.test

You should see the message: Listening on [msg.test]

2. Create a Publisher and publish a message

In another shell or command prompt, create a NATS publisher and send a message.

nats pub <subject> <message>

Where <subject> is the subject name and <message> is the text to publish.

For example:

nats pub msg.test "NATS MESSAGE"

3. Verify message publication and receipt

You'll notice that the publisher sends the message and prints: Published [msg.test] : 'NATS MESSAGE'.

The subscriber receives the message and prints: [#1] Received on [msg.test]: 'NATS MESSAGE'.

If the receiver does not get the message, you'll need to check if you are using the same subject name for the publisher and the subscriber.

4. Try publishing another message

nats pub msg.test "NATS MESSAGE 2"

You'll notice that the subscriber receives the message. Note that a message count is incremented each time your subscribing client receives a message on that subject.

5. Create Subscriber 2

In a new shell or command prompt, start a new NATS subscriber.

nats sub msg.test

6. Publish another message using the publisher client

nats pub msg.test "NATS MESSAGE 3"

Verify that both subscribing clients receive the message.

7. Create Subscriber 3

In a new shell or command prompt session, create a new subscriber that listens on a different subject.

nats sub msg.test.new

8. Publish another message

nats pub msg.test "NATS MESSAGE 4"

Subscriber 1 and Subscriber 2 receive the message, but Subscriber 3 does not. Why? Because Subscriber 3 is not listening on the message subject used by the publisher.

9. Alter Subscriber 3 to use a wildcard

Change the last subscriber to listen on msg.* and run it:

nats sub msg.*

Note: NATS supports the use of wildcard characters for message subscribers only. You cannot publish a message using a wildcard subject.

10. Publish another message

nats pub msg.test "NATS MESSAGE 5"

This time, all three subscribing clients should receive the message.

Do try out a few more variations of substrings and wildcards to test your understanding.

See Also

Publish-subscribe pattern with the NATS CLI

Streams with source and mirror configurations are best managed through a client API. If you intend to create such a configuration from command line with NATS CLI you should use a JSON configuration.

nats stream add --config stream_with_sources.json

Example stream configuration with two sources

Minimal example

{
  "name": "SOURCE_TARGET",
  "subjects": [
    "foo1.ext.*",
    "foo2.ext.*"
  ],
  "discard": "old",
  "duplicate_window": 120000000000,
  "sources": [
    {
      "name": "SOURCE1_ORIGIN",
    },
  ],
  "deny_delete": false,
  "sealed": false,
  "max_msg_size": -1,
  "allow_rollup_hdrs": false,
  "max_bytes": -1,
  "storage": "file",
  "allow_direct": false,
  "max_age": 0,
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "num_replicas": 1,
  "name": "SOURCE_TARGET",
  "deny_purge": false,
  "compression": "none",
  "max_msgs": -1,
  "retention": "limits",
  "mirror_direct": false
}

With additional options

{
  "name": "SOURCE_TARGET",
  "subjects": [
    "foo1.ext.*",
    "foo2.ext.*"
  ],
  "discard": "old",
  "duplicate_window": 120000000000,
  "sources": [
    {
      "name": "SOURCE1_ORIGIN",
      "filter_subject": "foo1.bar",
      "opt_start_seq": 42,
      "external": {
        "deliver": "",
        "api": "$JS.domainA.API"
      },
      
    },
    {
      "name": "SOURCE2_ORIGIN",
      "filter_subject": "foo2.bar"
    }
  ],
  "consumer_limits": {
    
  },
  "deny_delete": false,
  "sealed": false,
  "max_msg_size": -1,
  "allow_rollup_hdrs": false,
  "max_bytes": -1,
  "storage": "file",
  "allow_direct": false,
  "max_age": 0,
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "num_replicas": 1,
  "name": "SOURCE_TARGET",
  "deny_purge": false,
  "compression": "none",
  "max_msgs": -1,
  "retention": "limits",
  "mirror_direct": false
}

Example stream configuration with mirror

Minimal example

{
  "name": "MIRROR_TARGET"
  "discard": "old",
  "mirror": {
    "name": "MIRROR_ORIGIN"
  },
  "deny_delete": false,
  "sealed": false,
  "max_msg_size": -1,
  "allow_rollup_hdrs": false,
  "max_bytes": -1,
  "storage": "file",
  "allow_direct": false,
  "max_age": 0,
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "num_replicas": 1,
  "name": "MIRROR_TARGET",
  "deny_purge": false,
  "compression": "none",
  "max_msgs": -1,
  "retention": "limits",
  "mirror_direct": false
}

With additional options

{
  "name": "MIRROR_TARGET"
  "discard": "old",
  "mirror": {
    "opt_start_time": "2024-07-11T08:57:20.4441646Z",
    "external": {
      "deliver": "",
      "api": "$JS.domainB.API"
    },
    "name": "MIRROR_ORIGIN"
  },
  "consumer_limits": {
    
  },
  "deny_delete": false,
  "sealed": false,
  "max_msg_size": -1,
  "allow_rollup_hdrs": false,
  "max_bytes": -1,
  "storage": "file",
  "allow_direct": false,
  "max_age": 0,
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "num_replicas": 1,
  "name": "MIRROR_TARGET",
  "deny_purge": false,
  "compression": "none",
  "max_msgs": -1,
  "retention": "limits",
  "mirror_direct": false
}

Walkthrough prerequisites

1. Start the first member of the queue group

The nats reply instances don't just subscribe to the subject but also automatically join a queue group ("NATS-RPLY-22" by default)

nats reply foo "service instance A Reply# {{Count}}"

2. Start a second member of the queue-group

In a new window

nats reply foo "service instance B Reply# {{Count}}"

3. Start a third member of the queue-group

In a new window

nats reply foo "service instance C Reply# {{Count}}"

4. Publish a NATS message

nats request foo "Simple request"

5. Verify message publication and receipt

You should see that only one of the my-queue group subscribers receives the message and replies it, and you can also see which one of the available queue-group subscribers processed the request from the reply message received (i.e. service instance A, B or C)

6. Publish another message

nats request foo "Another simple request"

You should see that a different queue group subscriber receives the message this time, chosen at random among the 3 queue group members.

You can also send any number of requests back-to-back. From the received messages, you'll see the distribution of those requests amongst the members of the queue-group. For example: nats request foo --count 10 "Request {{Count}}"

7. Stop/start queue-group members

You can at any time start yet another service instance, or kill one and see how the queue-group automatically takes care of adding/removing those instances from the group.

See Also

Queue groups using the NATS CLI

Prerequisites

Walkthrough

Start two terminal sessions. These will be used to run the NATS request and reply clients.

In one terminal, run the reply client listener

nats reply help.please 'OK, I CAN HELP!!!'

You should see the message: Listening on [help.please]

This means that the NATS receiver client is listening for request messages on the "help.please" subject. In NATS, the receiver is a subscriber.

In the other terminal, run the request client

nats request help.please 'I need help!'

The NATS requestor client makes a request by sending the message "I need help!" on the “help.please” subject.

The NATS receiver client receives the message, formulates the reply ("OK, I CAN HELP!!!"), and sends it to the inbox of the requester.

When subscribers register themselves to receive messages from a publisher, the 1:N fan-out pattern of messaging ensures that any message sent by a publisher, reaches all subscribers that have registered. NATS provides an additional feature named "queue", which allows subscribers to register themselves as part of a queue. Subscribers that are part of a queue, form the "queue group".

How queue groups function

As an example, consider message delivery occurring in the 1:N pattern to all subscribers based on the subject name (delivery happens even to subscribers that are not part of a queue group). If a subscriber is registered based on a queue name, it will always receive messages it is subscribed to, based on the subject name. However, if more subscribers are added to the same queue name, they become a queue group, and only one randomly chosen subscriber of the queue group will consume a message each time a message is received by the queue group. Such distributed queues are a built-in load balancing feature that NATS provides.

Advantages

• Ensures application fault tolerance

• Workload processing can be scaled up or down

• Scale your consumers up or down without duplicate messages

• No extra configuration required

• Queue groups are defined by the application and their queue subscribers, rather than the server configuration

Queue subscribers are ideal for scaling services. Scale up is as simple as running another application, scale down is terminating the application with a signal that drains the in flight requests. This flexibility and lack of any configuration changes makes NATS an excellent service communication technology that can work with all platform technologies.

No responder

When a request is made to a service (request/reply) and the NATS Server knows there are no services available (since there are no client applications currently subscribing to the subject in a queue-group) the server will send a “no-responders” protocol message back to the requesting client which will break from blocking API calls. This allows applications to react immediately. This further enables building a highly responsive system at scale, even in the face of application failures and network partitions.

Stream as a queue

Queuing geo-affinity

When connecting to a globally distributed NATS super-cluster, there is an automatic service geo-affinity due to the fact that a service request message will only be routed to another cluster (i.e. another region) if there are no listeners on the cluster available to handle the request locally.

Tutorial

JetStream is built into nats-server. If you have a cluster of JetStream-enabled servers you can enable data replication and thus guard against failures and service disruptions.

JetStream was created to address the problems identified with streaming technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, or supports multiple deployment models. No other technology that we are aware of can scale from edge to cloud using the same security context while having complete deployment observability for operations.

Additional capabilities enabled by JetStream

The JetStream persistence layer enables additional use cases typically not found in messaging systems. Being built on top of JetStream they inherit the core capabilities of JetStream, replication, security, routing limits, and mirroring.

Key/Value and File transfer are capabilities are commonly found in in-memory databases or deployment tools. While NATS does not intend to compete with the feature set of such tools, it is our goal to provide the developer with reasonable complete set of data storage and replications features for use cases like micro service, edge deployments and server management.

Configuration

To configure a nats-server with JetStream refer to:

Examples

Goals

JetStream was developed with the following goals in mind:

• The system must be easy to configure and operate and be observable.

• The system must be secure and operate well with NATS 2.0 security models.

• The system must scale horizontally and be applicable to a high ingestion rate.

• The system must support multiple use cases.

• The system must self-heal and always be available.

• The system must allow NATS messages to be part of a stream as desired.

• The system must display payload agnostic behavior.

• The system must not have third party dependencies.

JetStream capabilities

Streaming: temporal decoupling between the publishers and subscribers

One of the tenets of basic publish/subscribe messaging is that there is a required temporal coupling between the publishers and the subscribers: subscribers only receive the messages that are published when they are actively connected to the messaging system (i.e. they do not receive messages that are published while they are not subscribing or not running or disconnected). The traditional way for messaging systems to provide temporal decoupling of the publishers and subscribers is through the 'durable subscriber' functionality or sometimes through 'queues', but neither one is perfect:

• durable subscribers need to be created before the messages get published

• queues are meant for workload distribution and consumption, not to be used as a mechanism for message replay.

However, in many use cases, you do not need to 'consume exactly once' functionality but rather the ability to replay messages on demand, as many times as you want. This need has led to the popularity of some 'streaming' messaging platforms.

Replay policies

JetStream consumers support multiple replay policies, depending on whether the consuming application wants to receive either:

• all of the messages currently stored in the stream, meaning a complete 'replay' and you can select the 'replay policy' (i.e. the speed of the replay) to be either:

  • instant (meaning the messages are delivered to the consumer as fast as it can take them).

  • original (meaning the messages are delivered to the consumer at the rate they were published into the stream, which can be very useful for example for staging production traffic).

• the last message stored in the stream, or the last message for each subject (as streams can capture more than one subject).

• starting from a specific sequence number.

• starting from a specific start time.

Retention policies and limits

JetStream enables new functionalities and higher qualities of service on top of the base 'Core NATS' functionality. However, practically speaking, streams can't always just keep growing 'forever' and therefore JetStream supports multiple retention policies as well as the ability to impose size limits on streams.

Limits

You can impose the following limits on a stream

• Maximum message age.

• Maximum total stream size (in bytes).

• Maximum number of messages in the stream.

• Maximum individual message size.

• You can also set limits on the number of consumers that can be defined for the stream at any given point in time.

You must also select a discard policy which specifies what should happen once the stream has reached one of its limits and a new message is published:

• discard old means that the stream will automatically delete the oldest message in the stream to make room for the new messages.

• discard new means that the new message is discarded (and the JetStream publish call returns an error indicating that a limit was reached).

Retention policy

You can choose what kind of retention you want for each stream:

• limits (the default) is to provide a replay of messages in the stream.

• work queue (the stream is used as a shared queue and messages are removed from it as they are consumed) is to provide the exactly-once consumption of messages in the stream.

• interest (messages are kept in the stream for as long as there are consumers that haven't delivered the message yet) is a variation of work queue that only retains messages if there is interest (consumers currently defined on the stream) for the message's subject.

Note that regardless of the retention policy selected, the limits (and the discard policy) always apply.

Subject mapping transformations

JetStream also enables the ability to apply subject mapping transformations to messages as they are ingested into a stream.

Persistent and Consistent distributed storage

You can choose the durability as well as the resilience of the message storage according to your needs.

• Memory storage.

• File storage.

• Replication (1 (none), 2, 3) between nats servers for Fault Tolerance.

JetStream can also provide encryption at rest of the messages being stored.

Stream replication factor

A stream's replication factor (R, often referred to as the number 'Replicas') determines how many places it is stored allowing you to tune to balance risk with resource usage and performance. A stream that is easily rebuilt or temporary might be memory-based with a R=1 and a stream that can tolerate some downtime might be file-based R-1.

Typical usage to operate in typical outages and balance performance would be a file-based stream with R=3. A highly resilient, but less performant and more expensive configuration is R=5, the replication factor limit.

Rather than defaulting to the maximum, we suggest selecting the best option based on the use case behind the stream. This optimizes resource usage to create a more resilient system at scale.

• Replicas=1 - Cannot operate during an outage of the server servicing the stream. Highly performant.

• Replicas=2 - No significant benefit at this time. We recommend using Replicas=3 instead.

• Replicas=3 - Can tolerate the loss of one server servicing the stream. An ideal balance between risk and performance.

• Replicas=4 - No significant benefit over Replicas=3 except marginally in a 5 node cluster.

• Replicas=5 - Can tolerate simultaneous loss of two servers servicing the stream. Mitigates risk at the expense of performance.

Mirroring and Sourcing between streams

JetStream also allows server administrators to easily mirror streams, for example between different JetStream domains in order to offer disaster recovery. You can also define a stream that 'sources' from one or more other streams.

De-coupled flow control

JetStream provides decoupled flow control over streams, the flow control is not 'end to end' where the publisher(s) are limited to publish no faster than the slowest of all the consumers (i.e. the lowest common denominator) can receive but is instead happening individually between each client application (publishers or consumers) and the nats server.

When using the JetStream publish calls to publish to streams there is an acknowledgment mechanism between the publisher and the NATS server, and you have the choice of making synchronous or asynchronous (i.e. 'batched') JetStream publish calls.

On the subscriber side, the sending of messages from the NATS server to the client applications receiving or consuming messages from streams is also flow controlled.

Exactly once semantics

Because publications to streams using the JetStream publish calls are acknowledged by the server the base quality of service offered by streams is 'at least once', meaning that while reliable and normally duplicate free there are some specific failure scenarios that could result in a publishing application believing (wrongly) that a message was not published successfully and therefore publishing it again, and there are failure scenarios that could result in a client application's consumption acknowledgment getting lost and therefore in the message being re-sent to the consumer by the server. Those failure scenarios while being rare and even difficult to reproduce do exist and can result in perceived 'message duplication' at the application level.

Therefore, JetStream also offers an 'exactly once' quality of service. For the publishing side, it relies on the publishing application attaching a unique message or publication ID in a message header and on the server keeping track of those IDs for a configurable rolling period of time in order to detect the publisher publishing the same message twice. For the subscribers a double acknowledgment mechanism is used to avoid a message being erroneously re-sent to a subscriber by the server after some kinds of failures.

Consumers

Fast push consumers

Client applications can choose to use fast un-acknowledged push (ordered) consumers to receive messages as fast as possible (for the selected replay policy) on a specified delivery subject or to an inbox. Those consumers are meant to be used to 'replay' rather than 'consume' the messages in a stream.

Horizontally scalable pull consumers with batching

Client applications can also use and share pull consumers that are demand-driven, support batching and must explicitly acknowledge message reception and processing which means that they can be used to consume (i.e. use the stream as a distributed queue) as well as process the messages in a stream.

Pull consumers can and are meant to be shared between applications (just like queue groups) in order to provide easy and transparent horizontal scalability of the processing or consumption of messages in a stream without having (for example) to worry about having to define partitions or worry about fault-tolerance.

Note: using pull consumers doesn't mean that you can't get updates (new messages published into the stream) 'pushed' in real-time to your application, as you can pass a (reasonable) timeout to the consumer's Fetch call and call it in a loop.

Consumer acknowledgments

While you can decide to use un-acknowledged consumers trading quality of service for the fastest possible delivery of messages, most processing is not idem-potent and requires higher qualities of service (such as the ability to automatically recover from various failure scenarios that could result in some messages not being processed or being processed more than once) and you will want to use acknowledged consumers. JetStream supports more than one kind of acknowledgment:

• Some consumers support acknowledging all the messages up to the sequence number of the message being acknowledged, some consumers provide the highest quality of service but require acknowledging the reception and processing of each message explicitly as well as the maximum amount of time the server will wait for an acknowledgment for a specific message before re-delivering it (to another process attached to the consumer).

• You can also send back negative acknowledgements.

• You can even send in progress acknowledgments (to indicate that you are still processing the message in question and need more time before acking or nacking it).

Key Value Store

The JetStream persistence layer enables the Key Value store: the ability to store, retrieve and delete value messages associated with a key into a bucket.

Watch and History

You can subscribe to changes in a Key Value on the bucket or individual key level with watch and optionally retrieve a history of the values (and deletions) that have happened on a particular key.

Atomic updates and locking

The Key Value store supports atomic create and update operations. This enables pessimistic locks (by creating a key and holding on to it) and optimistic locks (using CAS - compare and set).

Object Store

The Object Store is similar to the Key Value Store. The key being replaced by a file name and value being designed to store arbitrarily large objects (e.g. files, even if they are very large) rather than 'values' that are message-sized (i.e. limited to 1Mb by default). This is achieved by chunking messages.

Legacy

When a stream is configured with a source or mirror, it will automatically and asynchronously replicate messages from the origin stream.

source or mirror are designed to be robust and will recover from a loss of connection. They are suitable for geographic distribution over high latency and unreliable connections. E.g. even a leaf node starting and connecting intermittently every few days will still receive or send messages over the source/mirror link.

There are several options available when declaring the configuration.

• Name - Name of the origin stream to source messages from.

• StartSeq - An optional start sequence of the origin stream to start mirroring from.

• StartTime - An optional message start time to start mirroring from. Any messages that are equal to or greater than the start time will be included.

• FilterSubject - An optional filter subject which will include only messages that match the subject, typically including a wildcard. Note, this cannot be used with SubjectTransforms.

• Domain - An optional JetStream domain of where the origin stream exists. This is commonly used in a hub cluster and leafnode topology.

The stream using a source or mirror configuration can have its own retention policy, replication, and storage type.

General behavior

• All configurations are made on the receiving side. The stream from which data is sourced and mirrored does not need to be configured. No cleanup is required on the origin side if the receiver disappears.

• A stream can be the origin (source) for multiple streams. This is useful for geographic distribution or for designing "fan out" topologies where data needs to be distributed reliable to a large number (up to millions) of client connections.

• Leaf nodes and leaf node domains are explicitly supported through the API prefix

Source specific

A stream defining Sources is a generalized replication mechanism and allows for sourcing data from one or more streams concurrently. A stream with sources can still act as a regular stream allowing direct write/publish by local clients to the stream. Essentially the source streams and local client writes are aggregated into a single interleaved stream. Combined with subject transformation and filtering sourcing allows to design sophisticated data distribution architectures.

Mirror specific

A mirror can source its messages from exactly one stream and a clients can not directly write to the mirror. Although messages cannot be published to a mirror directly by clients, messages can be deleted on-demand (beyond the retention policy), and consumers have all capabilities available on regular streams.

Expected behavior in edge conditions

• Source and mirror contracts are designed with one-way (geographic) data replication in mind. Neither configuration provides a full synchronization between streams, which would include deletes or replication of other stream attributes.

• The content of the stream from which a source or mirror is drawn needs to be reasonable stable. Quickly deleting messages after publishing them may result in inconsistent replication due to the asynchronous nature of the replication process.

• Sources and Mirror try to be be efficient in replicating messages and are lenient towards the source/mirror origin being unreachable (event for extended periods of time), e.g. when using leaf nodes, which are connected intermittently. For sake of efficiency the recovery interval in case of a disconnect is 10-20s.

• Mirror and source agreements do not create a visible consumer in the origin stream.

WorkQueue retention

Source and mirror work with origin stream with workqueue retention. The source/mirror will act as a consumer removing messages from the origin stream.

The intended usage is for moving messages conveniently from one location to another (e.g. from a leaf node). It does not trivially implement a distributed workqueue (where messages are distributed to multiple streams sourcing from the same origin), although with the help of subject filtering a distributed workqueue can be approximated.

Interest base retention

A consumer is a stateful view of a stream. It acts as an interface for clients to consume a subset of messages stored in a stream and will keep track of which messages were delivered and acknowledged by clients.

While Streams are responsible for storing the published messages, the consumer is responsible for tracking the delivery and acknowledgments. This tracking ensures that if a message is not acknowledged (un-acked or 'nacked'), the consumer will automatically attempt to re-deliver it. JetStream consumers support various acknowledgment types and policies. If a message is not acknowledged within a user-specified number of delivery attempts, an advisory notification is emitted.

Dispatch type - Pull / Push

Consumers can be push-based where messages will be delivered to a specified subject or pull-based which allows clients to request batches of messages on demand. The choice of what kind of consumer to use depends on the use-case.

If there is a need to process messages in an application controlled manner and easily scale horizontally, you would use a 'pull consumer'. A simple client application that wants a replay of messages from a stream sequentially you would use an 'ordered push consumer'. An application that wants to benefit from load balancing or acknowledge messages individually will use a regular push consumer.

Ordered Consumers

Ordered consumers are the convenient default type of push & pull consumers designed for applications that want to efficiently consume a stream for data inspection or analysis.

• Always ephemeral

• No acknowledgements (if gap is detected, consumer is recreated)

• Automatic flow control/pull processing

• Single-threaded dispatching

• No load balancing

Persistence - Durable / Ephemeral

In addition to the choice of being push or pull, a consumer can also be ephemeral or durable. A consumer is considered durable when an explicit name is set on the Durable field when creating the consumer, or when InactiveThreshold is set.

Durables and ephemeral have the same message delivery semantics but an ephemeral consumer will not have persisted state or fault tolerance (server memory only) and will be automatically cleaned up (deleted) after a period of inactivity, when no subscriptions are bound to the consumer.

By default, consumers will have the same replication factor as the stream they consume, and will remain even when there are periods of inactivity (unless InactiveThreshold is set explicitly). Consumers can recover from server and client failure.

Configuration

Below are the set of consumer configuration options that can be defined. The Version column indicates the version of nats-server in which the option was introduced. The Editable column indicates the option can be edited after the consumer is created.

General

AckPolicy

The policy choices include:

• AckExplicit: The default policy. Each individual message must be acknowledged. Recommended for most reliability and functionality.

• AckNone: No acknowledgment needed; the server assumes acknowledgment on delivery.

• AckAll: Acknowledge only the last message received in a series; all previous messages are automatically acknowledged. Will acknowledge all pending messages for all subscribers for Pull Consumer.

If an acknowledgment is required but not received within the AckWait window, the message will be redelivered.

Warning: The server may consider an acknowledgment arriving out of the window. For instance, in a queue situation, if a first process fails to acknowledge within the window and the message has been redelivered to another consumer, the acknowledgment from the first consumer will be considered.

DeliverPolicy

The policy choices include:

• DeliverAll: Default policy. Start receiving from the earliest available message in the stream.

• DeliverLast: Start with the last message added to the stream, or the last message matching the consumer's filter subject if defined.

• DeliverLastPerSubject: Start with the latest message for each filtered subject currently in the stream.

• DeliverNew: Start receiving messages created after the consumer was created.

• DeliverByStartSequence: Start at the first message with the specified sequence number. The consumer must specify OptStartSeq defining the sequence number.

• DeliverByStartTime: Start with messages on or after the specified time. The consumer must specify OptStartTime defining the start time.

MaxAckPending

The MaxAckPending capability provides flow control and applies to both push and pull consumers. For push consumers, MaxAckPending is the only form of flow control. For pull consumers, client-driven message delivery creates implicit one-to-one flow control with subscribers.

For high throughput, set MaxAckPending to a high value. For applications with high latency due to external services, use a lower value and adjust AckWait to avoid re-deliveries.

FilterSubjects

A filter subject provides server-side filtering of messages before delivery to clients.

For example, a stream factory-events with subject factory-events.*.* can have a consumer factory-A with a filter factory-events.A.* to deliver only events for factory A.

A consumer can have a singular FilterSubject or plural FilterSubjects. Multiple filters can be applied, such as [factory-events.A.*, factory-events.B.*] or specific event types [factory-events.*.item_produced, factory-events.*.item_packaged].

Warning: For granular consumer permissions, a single filter uses $JS.API.CONSUMER.CREATE.{stream}.{consumer}.{filter} to restrict users to specific filters. Multiple filters use the general $JS.API.CONSUMER.DURABLE.CREATE.{stream}.{consumer}, which does not include the {filter} token. Use a different strategy for granular permissions.

Pull-specific

Push-specific

The diagram above shows the concept of storing all ORDERS.* in the Stream even though there are many types of order related messages. We’ll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider.

Streams can consume many subjects. Here we have ORDERS.* but we could also consume SHIPPING.state into the same Stream should that make sense.

Stream limits and message retention