The official NATS documentation

NATS is a simple, secure and high performance open source data layer for cloud native applications, IoT messaging, and microservices architectures.

We feel that it should be the backbone of your communication between services. It doesn't matter what language, protocol, or platform you are using; NATS is the best way to connect your services.

10,000 foot view

• Publish and subscribe to messages at millions of messages per second. At most once delivery.

• Supports fan-in/out delivery patterns

• Request/reply

• Every major language is supported

• Persistence via JetStream

  • at least once delivery or exactly once delivery

  • work queues

  • stream processing

  • data replication

  • data retention

  • data deduplication

  • Higher order data structures

    • Key/Value with watchers, versioning, and TTL

    • Object storage with versioning

• Security

  • TLS

  • JWT-based zero trust security

• Clustering

  • High availability

  • Fault tolerance

  • Auto-discovery

• Protocols supported

  • TCP

  • MQTT

  • WebSockets

All of this in a single binary that is easy to deploy and manage. No external dependencies, just drop it in and add a configuration file to point to other NATS servers and you are ready to go. In fact, you can even embed NATS in your application (for Go users)!

Guided tour

1. In general we recommend trying to solve your problems first using Core NATS.

2. If you need to share state between services, take a look at the KV or Object Store in JetStream.

3. When you need lower level access to persistence streams, move on to using JetStream directly for more advanced messaging patterns.

4. Learn about deployment strategies

5. Secure your deployments with zero trust security

Contribute

NATS is Open Source as is this documentation. Please let us know if you have updates and/or suggestions for these docs. You can also create a Pull Request using the Edit on GitHub link on each page.

Additional questions?

Feel free to chat with us on Slack slack.nats.io.

Thank you from the entire NATS Team of Maintainers for your interest in NATS!

This guide is tailored for existing NATS users upgrading from NATS version 2.9.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement.

Upgrade considerations

Client versions

Although all existing client versions will work, new client versions will expose additional options used to leverage new features. The minimum client versions that have full 2.10.0 support include:

• CLI - v0.1.0

• nats.go - v1.30.0

• nats.rs - v0.32.0

• nats.deno - v1.17.0

• nats.js - v2.17.0

• nats.ws - v1.18.0

• nats.java - v2.17.0

• nats.net - v1.1.0

• nats.net.v2 - Coming soon!

• nats.py - Coming soon!

• nats.c - Coming soon!

Helm charts

• k8s/nats - v1.1.0

• k8s/nack - v0.24.0

Downgrade warnings

For critical infrastructure like NATS, zero downtime upgrades are table stakes. Although the best practice for all infrastructure like this is for users to thoroughly test a new release against your specific workloads, inevitably there are cases where an upgrade occurs in production followed by a decision to downgrade. This is never recommended and can cause more harm than good for most infrastructure and data systems.

Below are a few important considerations if downgrading is required.

Storage format changes

2.10.0 brings on-disk storage changes which bring significant performance improvements. These are not compatible with previous versions of the NATS Server. If an upgrade is performed to a server with existing stream data on disk, followed by a downgrade, the older version server will not understand the stream data in the new format.

However, being mindful of the possibility of the need to downgrade, a special version of the 2.9.x series was released with awareness of key changes in the new storage format, allowing it to startup properly.

The takeaway is that if a downgrade is the only resort, it must be to 2.9.22 or later to ensure storage format changes are handled appropriately.

Stream and consumer config options

There are new stream and consumer configuration options that could be problematic if a downgrade occurs since previous versions of the server have no awareness of them. Examples include:

• Multi-filter consumers - Downgrading would result in no filter being applied since the new field is configured as a list rather than a single string.

• Subject-transform on streams - Downgrading would result in the subject transform not being applied since the server has no awareness of it.

• Compression on streams - Downgrading when compression is enabled on streams will cause those streams to become unloadable since the older server versions will not understand the compression being used.

Features

Platforms

• Experimental support for IBM z/OS

• Experimental support for NetBSD

Reload

• A server reload can now be performed by sending a message on $SYS.REQ.SERVER.<server-id>.RELOAD by a client authenticated in the system account.

JetStream

• A new sync_interval server config option has been added to change the default sync interval of stream data when written to disk, including allowing all writes to be flushed immediately. This option is only relevant if you need to modify durability guarantees.

Subject mapping

• Subject mappings can now be cluster-scoped and weighted, enabling the ability to have different mappings or weights on a per cluster basis.

• The requirement to use all wildcard tokens in subject mapping or transforms has been relaxed. This can be applied to config or account-based subject mapping, stream subject transforms, and stream republishing, but not on subject mappings that are associated with stream and service import/export between accounts.

Streams

• A subject_transform field has been added enabling per-stream subject transforms. This applies to standard streams, mirrors, and sourced streams.

• A metadata field has been added to stream configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field.

• A first_seq field has been added to stream configuration enabling explicitly setting the initial sequence on stream creation.

• A compression field has been added to stream configuration enabling on-disk compression for file-based streams.

• The ability to edit the republish config option on a stream after stream creation was added.

• A Nats-Time-Stamp header is now included in republished messages containing the original message's timestamp.

• A ts field has been added to stream info responses indicating the server time of the snapshot. This was added to allow for local time calculations relying on the local clock.

• An array of subject-transforms (subject filter + subject transform destination) can be added to a mirror or source configuration (can not use the single subject filter/subject transform destination fields at the same time as the array).

• A stream configured with sources can source from the same stream multiple times when distinct filter+transform options are used, allowing for some messages of a stream to be sourced more than once.

Consumers

• A filter_subjects field has been added which enables applying server-side filtering against multiple disjoint subjects, rather than only one.

• A metadata field has been added to consumer configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field.

• A ts field has been added to consumer info responses indicating the server time of the snapshot. This was added to allow for local time calculations without relying on the local clock.

Key-value

• A metadata field has been added to key-value configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field.

• A bucket configured as a mirror or sourcing from other buckets

Object store

• A metadata field has been added to object store configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field.

Authn/Authz

• A pluggable server extension, referred to as auth callout, has been added. This provides a mechanism for delegating authentication checks against a bring-your-own (BYO) provider and, optionally, dynamically declaring permissions for the authenticated user.

Monitoring

• A unique_tag field has been added to the /varz and /jsz HTTP endpoint responses, corresponding to the value of unique_tag defined in the server config.

• A slow_consumer_stats field has been added to the /varz HTTP endpoint providing a count of slow consumers for clients, routes, gateways, and leafnodes.

• A raft=1 query parameter has been added to the /jsz HTTP endpoint which adds stream_raft_group and consumer_raft_groups fields to the response.

• A num_subscriptions field has been added to the $SYS.REQ.SERVER.PING.STATZ NATS endpoint responses.

• A system account responder for $SYS.REQ.SERVER.PING.IDZ has been added which returns info for the server that the client is connected to.

• A system account responder for $SYS.REQ.SERVER.PING.PROFILEZ has been added and works even if a profiling port is not enabled in the server configuration.

• A user account responder for $SYS.REQ.USER.INFO has been added which allows a connected user to query for the account they are in and permissions they have.

MQTT

• Support for QoS2 has been added. Check out the new MQTT implementation details overview.

Clustering

• When defining routes between servers, a handful of optimizations have been introduced including a pool of TCP connections between servers, optional pinning of accounts to connections, and optional compression of traffic. There is quite a bit to dig into, so check out the v2 routes page for details.

Leafnodes

• A handshake_first config option has been added enabling TLS-first handshakes for leafnode connections.

Windows

• The NATS_STARTUP_DELAY environment variable has been added to allow changing the default startup for the server of 10 seconds

Improvements

Reload

• The nats-server --signal command now supports a glob expression on the <pid> argument which would match a subset of all nats-server instances running on the host.

Streams

• Prior to 2.10, setting republish configuration on mirrors would result in an error. On sourcing streams, only messages that were actively between stored matching configured subjects would be republished. The behavior has been relaxed to allow republishing on mirrors and includes all messages on sourcing streams.

Consumers

• A new header has been added on a fetch response that indicates to clients the fetch has been fulfilled without requiring clients to rely on heartbeats. It avoids some conditions in which the client would issue fetch requests that could go over limits or have more fetch requests pending than required.

Leafnodes

• Previously, a leafnode configured with two or more remotes binding to the same hub account would be rejected. This restriction has been relaxed since each remote could be binding to a different local account.

MQTT

• Previously a dot . in an MQTT topic was not supported, however now it is! Check out the topic-subject conversion table for details.

This guide is tailored for existing NATS users upgrading from NATS version v2.10.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement.

Features

Observability

• Distributed message tracing: Users can now trace messages as they move through the system by setting a Nats-Trace-Dest header to an inbox subject. Servers on the message path will return events to the provided subject that report each time a message enters or leaves a server, by which connection type, when subject mappings occur, or when messages traverse an account import/export boundary. Additionally, the Nats-Trace-Only header (if set to true) will allow tracing events to propagate on a specific subject without delivering them to subscribers of that subject.

Streams

• JetStream per-message TTLs: It is now possible to age out individual messages using a per-message TTL. The Nats-TTL header, in either string or integer format (in seconds) allows for individual message expiration independent of stream limits. This can be combined with other limits in place on the stream. More information is available in ADR-43.

• Subject delete markers on MaxAge: The SubjectDeleteMarkerTTL stream configuration option now allows for the placement of delete marker messages in the stream when the configured MaxAge limit causes the last message for a given subject to be deleted. The delete markers include a Nats-Marker-Reason header explaining which limit was responsible for the deletion.

• Stream ingest rate limiting: New options max_buffered_size and max_buffered_msgs in the jetstream configuration block enable rate limiting on Core NATS publishing into JetStream streams, protecting the system from overload.

Consumers

• Pull consumer priority groups: Pull consumers now support priority groups with pinning and overflow, enabling flexible failover and priority management when multiple clients are pulling from the same consumer. Configurable policies based on the number of pending messages on the consumer, or the number of pending acks, can control when messages overflow from one client to another, enabling new design patterns or regional awareness.

• Consumer pausing: Message delivery to consumers can be temporarily suspended using the new pause API endpoint (or the PauseUntil configuration option when creating), ideal for maintenance or migrations. Message delivery automatically resumes once the configured deadline has passed. Consumer clients continue to receive heartbeat messages as usual to ensure that they do not surface errors during the pause.

Operations

• Replication traffic in asset accounts: Raft replication traffic can optionally be moved into the same account in which replicated assets live on a per-account basis, rather than being sent and received in the system account. When combined with multiple route connections, this can help to reduce latencies and avoid head-of-line blocking issues that may occur in heavily-loaded multi-tenant or multi-account deployments.

• TLS first on leafnode connections: A new handshake_first in the leafnode tls block allows setting up leafnode connections that perform TLS negotiation first, before any other protocol handshakes take place.

• Configuration state digest: A new -t command line flag on the server binary can generate a hash of the configuration file. The config_digest item in varz displays the hash of the currently running configuration file, making it possible to check whether a configuration file has changed on disk compared to the currently running configuration.

• TPM encryption on Windows: When running on Windows, the filestore can now store encryption keys in the TPM, useful in environments where physical access may be a concern.

MQTT

• SparkplugB: The built-in MQTT support is now compliant with SparkplugB Aware, with support for NBIRTH and NDEATH messages.

Improvements

• Replicated delete proposals: Message removals in clustered interest-based or workqueue streams are now propagated via Raft to guarantee consistent removal order across replicas, reducing a number of possible ways that a cluster failure can result in de-synced streams.

• Metalayer, stream and consumer consistency: A new leader now only responds to read/write requests after synchronizing with its Raft log, preventing desynchronization between KV key updates and the stream during leader changes.

• Replicated consumer reliability: Replicated consumers now consistently redeliver unacknowledged messages after a leader change.

• Consumer starting sequence: The consumer starting sequence is now always respected, except for internal hidden consumers for sources/mirrors.

Upgrade Considerations

Stream ingest rate limiting

The NATS Server can now return a 429 error with type JSStreamTooManyRequests when too many messages have been queued up for a stream. It should not generally be possible to hit this limit while using JetStream publishes and waiting for PubAcks, but may trigger if trying to publish into JetStream using Core NATS publishes without waiting for PubAcks, which is not advised.

The new max_buffered_size and max_buffered_msgs options control how many messages can be queued for each stream before the rate limit is hit, therefore if needed, you can increase these limits on your deployments. The default values for max_buffered_size and max_buffered_msgs are 128MB and 10,000 respectively, whereas in v2.10 these were unlimited.

You can detect in the server logs whether running into a queue limit with the following warning:

[WRN] Dropping messages due to excessive stream ingest rate on 'account' > 'my-stream': IPQ len limit reached

If your application starts to log the above warnings then you can first try to increase the limits to higher values while investigating the fast publishers, for example:

jetstream {
  max_buffered_msgs: 50000
  max_buffered_size: 256mib
}

Replicated delete proposals

Since stream deletes are now replicated through group proposals in a replicated stream, there may be a slight increase in replication traffic on this version.

JetStream healthcheck

The js-server-only healthcheck no longer checks for the health of the metaleader on v2.11.0. Since this healthcheck was designed to detect the server readiness (or in k8s for the readiness probe) checking the metaleader would sometimes cause a NATS server to be considered unhealthy when restarting the servers. In v2.11, this should no longer be an issue. If the previous behavior from v2.10 is preferred, there is a new healthcheck option js-meta-only which can be used to check whether the meta group is healthy.

Exit code

Earlier versions of the NATS Server would return an exit code 1 when gracefully shut down, i.e. after SIGTERM. From v2.11, an exit code of 0 (zero) will now be returned instead.

Server, cluster and gateway names

Configurations that have server, cluster, and gateway names with spaces are now considered invalid, as this can cause problems at the protocol level. A server running NATS v2.11 will fail to start with spaces configured in these names. Please ensure that spaces are not used in server, cluster or gateway names.

Downgrade Considerations

Stream state

When downgrading from v2.11 to v2.10, the stream state files on disk will be rebuilt due to a change in the format of these files in v2.11. This requires re-scanning all stream message blocks, which may use higher CPU than usual and will likely take longer for the restarted node to report healthy. This will only happen on the first restart after downgrading and will not result in data loss.

NATS 2.2 is the largest feature release since version 2.0. The 2.2 release provides highly scalable, highly performant, secure and easy-to-use next generation streaming in the form of JetStream, allows remote access via websockets, has simplified NATS account management, native MQTT support, and further enables NATS toward our goal of securely democratizing streams and services for the hyperconnected world we live in.

Next Generation Streaming

JetStream is the next generation streaming platform for NATS, highly resilient, highly available, and easy to use. We’ve spent a long time listening to our community, learning from our experiences, looking at the needs of today, and thinking deeply about the needs of tomorrow. We built JetStream to address these needs.

JetStream:

• is easy to deploy and manage, built into the NATS server

• simplifies and accelerates development

• supports wildcard subjects

• supports at least once delivery and exactly once within a window

• is horizontally scalable at runtime with no interruptions

• persists data via streams and delivers or replays via consumers

• supports multiple patterns to consume data on the same stream

• supports push and pull modes when consuming messages

• is account aware

• allows for detailed granularity of security, by stream, by consumer, by function

Get started with JetStream.

Security and Simplified Account Management

Account management just became much easier. This version of NATS has a built-in account management system, eliminating the need to set up an account manager when not using the memory account resolver. With automated default system account generation, and the ability to preload accounts, simply enable a set of servers in your deployment to be account resolvers or account resolver caches, and they will handle public account information provided to the NATS system through the NATS nsc tooling. Have enterprise-scale account management up and running in minutes.

CIDR Block Account Restrictions

By specifying a CIDR block restriction for a user, policy can be applied to limit connections from clients within a certain range or set of IP addresses. Use this as another layer of security atop user credentials to better secure your distributed system. Ensure your applications can only connect from within a specific cloud, enterprise, geographic location, virtual or physical network.

Time-Based Account Restrictions

Scoped to the user, you can now specify a specific block of time during the day when applications can connect. For example, permit certain users or applications to access the system during specified business hours, or protect business operations during the busiest parts of the day from batch driven back-office applications that could adversely impact the system when run at the wrong time.

Default User Permissions

Now you can specify default user permissions within an account. This significantly reduces efforts around policy, reduces chances for error in permissioning, and simplifies the provisioning of user credentials.

WebSockets

Connect mobile and web applications to any NATS server using WebSockets. Built to more easily traverse firewalls and load balancers, NATS WebSocket support provides even more flexibility to NATS deployments and makes it easier to communicate to the edge and endpoints. This is currently supported in NATS server leaf nodes, nats.ts, nats.deno, and the nats.js clients.

Native MQTT Support

With the Adaptive Edge architecture and the ease with which NATS can extend a cloud deployment to the edge, it makes perfect sense to leverage existing investments in IoT deployments. It’s expensive to update devices and large edge deployments. Our goal is to enable the hyperconnected world, so we added first-class support for MQTT 3.1.1 directly into the NATS Server.

Seamlessly integrate existing IoT deployments using MQTT 3.1.1 with a cloud-native NATS deployment. Add a leaf node that is MQTT enabled and instantly send and receive messages to your MQTT applications and devices from a NATS deployment whether it be edge, single-cloud, multi-cloud, on-premise, or any combination thereof.

Build Better Systems

We’ve added a variety of features to allow you to build a more resilient, secure, and simply better system at scale.

Message Headers

We’ve added the ability to optionally use headers, following the HTTP semantics familiar to developers. Headers naturally apply overhead, which was why we resisted adding them for so long. By creating new internal protocol messages transparent to developers, we maintain the extremely fast processing of simple NATS messages that we have always had while supporting headers for those who would like to leverage them. Adding headers to messages allows you to provide application-specific metadata, such as compression or encryption-related information, without touching the payload. We also provide some NATS specific headers for use in JetStream and other features.

Seamless Maintenance with Lame Duck Notifications

When taking down a server for maintenance, servers can be signaled to enter Lame Duck Mode where they do not accept new connections and evict existing connections over a period of time. Maintainer supported clients will notify applications that a server has entered this state and will be shutting down, allowing a client to smoothly transition to another server or cluster and better maintain business continuity during scheduled maintenance periods.

React Quicker with No-Responder Notifications

Why wait for timeouts when services aren’t available? When a request is made to a service (request-reply) and the NATS Server knows there are no services available the server will short circuit the request. A “no-responders” protocol message will be sent back to the requesting client which will break from blocking API calls. This allows applications to immediately react which further enables building a highly responsive system at scale, even in the face of application failures and network partitions.

Subject Mapping and Traffic Shaping

Reduce risk when onboarding new services. Canary deployments, A/B testing, and transparent teeing of data streams are now fully supported in NATS. The NATS Server allows accounts to form subject mappings from one subject to another for both client inbound and service import invocations and allows weighted sets for the destinations. Map any percentage - 1 to 100 percent of your traffic - to other subjects, and change this at runtime with a server configuration reload. You can even artificially drop a percentage of traffic to introduce chaos testing into your system. See Configuring Subject Mapping and Traffic Shaping in NATS Server configuration for more details.

Account Monitoring - More Meaningful Metrics

NATS now allows for fine-grained monitoring to identify usage metrics tied to a particular account. Inspect messages and bytes sent or received and various connection statistics for a particular account. Accounts can represent anything - a group of applications, a team or organization, a geographic location, or even roles. If NATS is enabling your SaaS solution you could use NATS account scoped metrics to bill users.

What is NATS? NATS is a connective technology that powers modern distributed systems. A connective technology is responsible for addressing, discovery and exchanging of messages that drive the common patterns in distributed systems; asking and answering questions, aka services/microservices, and making and processing statements, or stream processing.

Challenges faced by modern distributed systems Modern distributed systems are defined by an ever increasing number of hyper-connected moving parts and the additional data they generate. They employ both services and streams to drive business value. They are also being defined by location independence and mobility, and not just for things we would typically recognize as front end technologies. Today’s systems and the backend processes, microservices and stream processing are being asked to be location independent and mobile as well, all while being secure.

These modern systems present challenges to technologies that have been used to connect mobile front ends to fairly static backends. These incumbent technologies typically manage addressing and discovery via hostname (DNS) or IP and port, utilize a 1:1 communication pattern, and have multiple different security patterns for authentication and authorization. Although not perfect, incumbent technologies have been good enough in many situations, but times are changing quickly. As microservices, functions, and stream processing are being asked to move to the edge, these technologies and the assumptions they make are being challenged.

What makes the NATS connective technology unique for these modern systems?

Effortless M:N connectivity: NATS manages addressing and discovery based on subjects and not hostname and ports. Defaulting to M:N communications, which is a superset of 1:1, meaning it can do 1:1 but can also do so much more. If you have a 1:1 system that is successful in development, ask how many other moving parts are required for production to work around the assumption of 1:1? Things like load balancers, log systems, and network security models, as well as proxies and sidecars. If your production system requires all of these things just to get around the fact that the connective technology being used, e.g. HTTP or gRPC, is 1:1, it’s time to give NATS.io a look.

Deploy anywhere: NATS can be deployed nearly anywhere; on bare metal, in a VM, as a container, inside K8S, on a device, or whichever environment you choose. NATS runs well within deployment frameworks or without.

Secure: Similarly, NATS is secure by default and makes no requirements on network perimeter security models. When you start considering mobilizing your backend microservices and stream processors, many times the biggest roadblock is security.

Scalable, Future-Proof Deployments

NATS infrastructure and clients communicate all topology changes in real-time. This means that NATS clients do not need to change when NATS deployments change. Having to change clients with deployments would be like having to reboot your phone every time your cell provider added or changed a cell tower. This sounds ridiculous of course, but think about how many systems today have their front ends tied so closely to the backend, that any change requires a complete front end reboot or at least a reconfiguration. NATS clients and applications need no such change when backend servers are added and removed and changed. Even DNS is only used to bootstrap first contact, after that, NATS handles endpoint locations transparently.

Hybrid Deployments

Another advantage to utilizing a NATS is that it allows a hybrid mix of SaaS/Utility computing with separately owned and operated systems. Meaning you can have a shared NATS service with core microservices, streams and stream processing be extended by groups or individuals who have a need to run their own NATS infrastructure. You are not forced to choose one or the other.

Adaptability

Today’s systems will fall short with new demands. As modern systems continue to evolve and utilize more components and process more data, supporting patterns beyond 1:1 communications, with addressing and discovery tied to DNS is critical. Foundational technologies like NATS promise the most return on investment. Incumbent technologies will not work as modern systems unify cloud, Edge, IoT and beyond. NATS does.

Use Cases

NATS can run anywhere, from large servers and cloud instances, through edge gateways and even IoT devices. Use cases for NATS include:

• Cloud Messaging

  • Services (microservices, service mesh)

  • Event/Data Streaming (observability, analytics, ML/AI)

• Command and Control

  • IoT and Edge

  • Telemetry / Sensor Data / Command and Control

• Augmenting or Replacing Legacy Messaging Systems

NATS 2.0 was the largest feature release since the original code base for the server was released. NATS 2.0 was created to allow a new way of thinking about NATS as a shared utility, solving problems at scale through distributed security, multi-tenancy, larger networks, and secure sharing of data.

Rationale

NATS 2.0 was created to address problems in large scale distributed computing.

It is difficult at best to combine identity management end-to-end (or end-to-edge), with data sharing, while adhering to policy and compliance. Current distributed systems increase significantly in operational complexity as they scale upward. Problems arise around service discovery, connectivity, scaling for volume, and application onboarding and updates. Disaster recovery is difficult, especially as systems have evolved to operate in silos defined by technology rather than business needs. As complexity increases, systems become expensive to operate in terms of time and money. They become fragile making it difficult to deploy services and applications hindering innovation, increasing time to value and total cost of ownership.

We decided to:

• Reduce total cost of ownership: Users want reduced TCO for their

  distributed systems. This is addressed by an easy to use technology that

  can operate at global scale with simple configuration and a resilient

  and cloud-native architecture.

• Decrease Time to Value: As systems scale, time to value increases.

  Operations resist change due to risk in touching a complex and fragile

  system. Providing isolation contexts can help mitigate this.

• Support manageable large scale deployments: No data silos defined by

  software, instead easily managed through software to provide exactly what the

  business needs. We wanted to provide easy to configure disaster recovery.

• Decentralize security: Provide security supporting one

  technology end-to-end where organizations may self-manage making it

  easier to support a massive number of endpoints.

To achieve this, we added a number of new features that are transparent to existing clients with 100% backward client compatibility.

Accounts

Accounts are securely isolated communication contexts that allow multi-tenancy spanning a NATS deployment. Accounts allow users to bifurcate technology from business driven use cases, where data silos are created by design, not software limitations. When a client connects, it specifies an account or will default to authentication with a global account.

At least some services need to share data outside of their account. Data can be securely shared between accounts with secure services and streams. Only mutual agreement between account owners permit data flow, and the import account has complete control over its own subject space.

This means within an account, limitations may be set and subjects can be used without worry of collisions with other groups or organizations. Development groups choose any subjects without affecting the rest of the system, and open up accounts to export or import only the services and streams they need.

Accounts are easy, secure, and cost effective. There is one NATS deployment to manage, but organizations and development teams can self manage with more autonomy reducing time to value with faster, more agile development practices.

Service and Streams

Services and streams are mechanisms to share messages between accounts.

Think of a service as an RPC endpoint into an account. Behind that account there might be many microservices working in concert to handle requests, but from outside the account there is simply one subject exposed.

Service definitions share an endpoint:

• Export a service to allow other accounts to import

• Import a service to allow requests to be sent securely and seamlessly to another account

Use cases include most applications - anything that accepts a request and returns a response.

Stream definitions allow continuous data flow between accounts:

• Export a stream to allow egress

• Import a stream to allow ingress

Use cases include Observability, Metrics, and Data analytics. Any application or endpoint reading a stream of data.

Note that services and streams operate with zero client configuration or API changes. Services may even move between accounts, entirely transparent to end clients.

System Accounts

The system account publishes system messages under established subject patterns. These are internal NATS system messages that may be useful to operators.

Server initiated events and data include:

• Client connection events

• Account connection status

• Authentication errors

• Leaf node connection events

• Server stats summary

Tools and clients with proper privileges can request:

• Service statistics

• Server discovery and metrics

Account servers will also publish messages when an account changes.

With this information and system metadata you can build useful monitoring and anomaly detection tools.

Global Deployments

NATS 2.0 supports global deployments, allowing for global topologies that optimize for WANs while extend to the edge or devices.

Self Healing

While self healing features have been part of NATS 1.X releases, we ensured they continue to work in global deployments. These include:

• Client and server connections automatically reconnect

• Auto-Discovery where servers exchange server topology changes with each

  other and with clients, in real time with zero configuration changes and

  zero downtime while being entirely transparent to clients. Clients can

  failover to servers they were not originally configured with.

• NATS server clusters dynamically adjust to new or removed servers allowing

  for seamless rolling upgrades and scaling up or down.

Superclusters

Conceptually, superclusters are clusters of NATS clusters. Create superclusters to deploy a truly global NATS network. Superclusters use a novel spline based technology with a unique approach to topology, keeping one hop semantics and optimizing WAN traffic through optimistic sends with interest graph pruning. Superclusters provide transparent, intelligent support for geo-distributed queue subscribers.

Disaster Recovery

Superclusters inherently support disaster recovery. With geo-distributed queue subscribers, local clients are preferred, then an RTT is used to find the lowest latency NATS cluster containing a matching queue subscriber in the supercluster.

What does this mean?

Let's say you have a set of load balanced services in US East Coast (US-EAST), another set in the EU (EU-WEST), and a supercluster consisting of a NATS cluster in US-EAST connected to a NATS cluster in EU-WEST. Clients in the US would connect to a US-EAST, and services connected to that cluster would service those clients. Clients in Europe would automatically use services connected to EU-WEST. If the services in US-EAST disconnect, clients in US-EAST will begin using services in EU-WEST.

Once the Eastern US services have reconnected to US-EAST, those services will immediately begin servicing the Eastern US clients since they're local to the NATS cluster. This is automatic and entirely transparent to the client. There is no extra configuration in NATS servers.

This is zero configuration disaster recovery.

Leaf Nodes

Leaf nodes are NATS servers running in a special configuration, allowing hub and spoke topologies to extend superclusters.

Leaf nodes can also bridge separate security domains. e.g. IoT, mobile, web. They are ideal for edge computing, IoT hubs, or data centers that need to be connected to a global NATS deployment. Local applications that communicate using the loopback interface with physical VM or Container security can leverage leaf nodes as well.

Leaf nodes:

• Transparently and securely bind to a remote NATS account

• Securely bridge specific local data to a wider NATS deployment

• Are 100% transparent to clients which remain simple, lightweight, and easy to develop

• Allow for a local security scheme while using new NATS security features globally

• Can create a DMZ between a local NATS deployment and external NATS cluster or supercluster.

Decentralized Security

Operators, Accounts, and Users

NATS 2.0 Security consists of defining Operators, Accounts, and Users within a NATS deployment.

• An Operator provides the root of trust for the system, may represent

  a company or enterprise

  • Creates Accounts for account administrators. An account represents

    an organization, business unit, or service offering with a secure context

    within the NATS deployment, for example an IT system monitoring group, a

    set of microservices, or a regional IoT deployment. Account creation

    would likely be managed by a central group.

• Accounts define limits and may securely expose services and streams.

  • Account managers create Users with permissions

• Users have specific credentials and permissions.

Trust Chain

• Operators are represented by a self signed JWT and is the only thing that

  is required to be configured in the server. This JWT is usually signed by a

  master key that is kept offline. The JWT will contain valid signing keys that

  can be revoked with the master updating this JWT.

  • Operators will sign Account JWTs with various signing keys.

  • Accounts sign User JWTs, again with various signing keys.

• Clients or leaf nodes present User credentials and a signed nonce when connecting.

  • The server uses resolvers to obtain JWTs and verify the client trust chain.

This allows for rapid change of permissions, authentication and limits, to a secure multi-tenant NATS system.

NATS is a system for publishing and listening for messages on named communication channels we call Subjects. Fundamentally, NATS is an interest-based messaging system, where the listener has to subscribe to a subset of subjects.

Location transparency Through subject-based addressing, NATS provides location transparency across a (large) cloud of routed NATS servers.

• Subject subscriptions are automatically propagated within the server cloud.

• Messages will be automatically routed to all interested subscribers, independent of location.

Wildcards

NATS provides two wildcards that can take the place of one or more elements in a dot-separated subject. Publishers will always send a message to a fully specified subject, without the wildcard. While subscribers can use these wildcards to listen to multiple subjects with a single subscription.

Subject hierarchies

The . character is used to create a subject hierarchy. For example, a world clock application might define the following to logically group related subjects:

Subject usage best practices

There is no hard limit to subject size, but it is recommended to keep the maximum number of tokens in your subjects to a reasonable value. E.g. a maximum of 16 tokens and the subject length to less than 256 characters.

Number of subjects

NATS can manage 10s of millions of subjects efficiently, therefore, you can use fine-grained addressing for your business entities. Subjects are ephemeral resources, which will disappear when no longer subscribed to.

Still, subject subscriptions need to be cached by the server in memory. Consider when increasing your subscribed subject count to more than one million you will need more than 1GB of server memory and it will grow linearly from there.

Subject-based filtering and security

The message subject can be filtered with various means and through various configuration elements in your NATS server cluster. For example, but not limited to:

• Security - allow/deny per user

• Import/export between accounts

• Automatic transformations

• When inserting messages into JetStream streams

• When sourcing/mirroring JetStream streams

• When connecting leaf nodes (NATS edge servers)

• ...

A well-designed subject hierarchy will make the job a lot easier for those tasks.

Naming things

A subject hierarchy is a powerful tool for addressing your application resources. Most NATS users therefore encode business semantics into the subject name. You are free to choose a structure fit for your purpose, but you should refrain from over-complicating your subject design at the start of the project.

Some guidelines:

• Use the first token(s) to establish a general namespace.

• Use the final token(s)for identifiers

• A subject should be used for more than one message.

• Subscriptions should be stable (exist for receiving more than one message).

• Use wildcard subscriptions over subscribing to individual subjects whenever feasible.

• Name business or physical entities. Refrain from encoding too much data into the subject.

• Encode (business) intent into the subject, not technical details.

Pragmatic:

Maybe not so useful:

• NATS messages support headers. These can be used for additional metadata. There are subscription modes, which deliver headers only, allowing for efficient scanning of metadata in the message flow.

Matching a single token

The first wildcard is * which will match a single token. For example, if an application wanted to listen for eastern time zones, they could subscribe to time.*.east, which would match time.us.east and time.eu.east. Note that * can not match a substring within a token time.New*.east.

Matching multiple tokens

The second wildcard is > which will match one or more tokens, and can only appear at the end of the subject. For example, time.us.> will match time.us.east and time.us.east.atlanta, while time.us.* would only match time.us.east since it can't match more than one token.

Monitoring and wire taps

Subject to your security configuration, wildcards can be used for monitoring by creating something called a wire tap. In the simplest case, you can create a subscriber for >. This application will receive all messages -- again, subject to security settings -- sent on your NATS cluster.

Mixing wildcards

The wildcard * can appear multiple times in the same subject. Both types can be used as well. For example, *.*.east.> will receive time.us.east.atlanta.

Characters allowed and recommended for subject names

For compatibility across clients and ease of maintaining configuration files, we recommend using alphanumeric characters, - (dash) and _ (underscore) ASCII characters for subject and other entity names created by the user.

UTF-8 (UTF8) characters are supported in subjects. Please use UTF-8 characters at your own risk. Using multilingual names for technical entities can create many issues for editing, configuration files, display, and cross-border collaboration.

The rules and recommendations here apply to ALL system names, subjects, streams, durables, buckets, keys (in key-value stores), as NATS will create API subjects that contain those names. NATS will enforce these constraints in most cases, but we recommend not relying on this.

• Allowed characters: Any Unicode character except null, space, ., * and >

• Recommended characters: (a - z), (A - Z), (0 - 9), - and _ (names are case sensitive, and cannot contain whitespace).

• Naming Conventions If you want to delimit words, use either CamelCase as in MyServiceOrderCreate or - and _ as in my-service-order-create

• Special characters: The period . (which is used to separate the tokens in the subject) and * and also > (the * and > are used as wildcards) are reserved and cannot be used.

• Reserved names: By convention subject names starting with a $ are reserved for system use (e.g. subject names starting with $SYS or $JS or $KV, etc...). Many system subjects also use _ (underscore) (e.g. _INBOX , KV_ABC, OBJ_XYZ etc.)

Good names

Deprecated subject names

Forbidden stream names

Pedantic mode

By default, for the sake of efficiency, subject names are not verified during message publishing. In particular, when generating subjects programmatically, this will result in illegal subjects which cannot be subscribed to. E.g. subjects containing wildcards may be ignored.

To enable subject name verification, activate pedantic mode in the client connection options.

We have provided Walkthroughs for you to try NATS (and JetStream) on your own. In order to follow along with the walkthroughs, you could choose one of these options:

• The nats CLI tool must be installed, and a local NATS server must be installed (or you can use a remote server you have access to).

• You can use Synadia's NGS.

• You could even use the demo server from where you installed NATS. This is accessible via nats://demo.nats.io (this is a NATS connection URL; not a browser URL. You pass it to a NATS client application).

Installing the NATS server locally (if needed)

Alternatively if you already know how to use NATS on a remote server, you only need to pass the server URL to nats using the -s option or preferably create a context using nats context add, to specify the server URL(s) and credentials file containing your user JWT.

Start the NATS server (if needed)

To start a simple demonstration server locally, simply run:

(or nats-server -m 8222 if you want to enable the HTTP monitoring functionality)

When the server starts successfully, you will see the following messages:

The NATS server listens for client connections on TCP Port 4222.

Software applications and services need to exchange data. NATS is an infrastructure that allows such data exchange, segmented in the form of messages. We call this a "message oriented middleware".

With NATS, application developers can:

• Effortlessly build distributed and scalable client-server applications.

• Store and distribute data in realtime in a general manner. This can flexibly be achieved across various environments, languages, cloud providers and on-premises systems.

NATS Client Applications

Developers use one of the NATS client libraries in their application code to allow them to publish, subscribe, request and reply between instances of the application or between completely separate applications. Those applications are generally referred to as 'client applications' or sometimes just as 'clients' throughout this manual (since from the point of view of the NATS server, they are clients).

NATS Service Infrastructure

The NATS services are provided by one or more NATS server processes that are configured to interconnect with each other and provide a NATS service infrastructure. The NATS service infrastructure can scale from a single NATS server process running on an end device (the nats-server process is less than 20 MB in size!) all the way to a public global super-cluster of many clusters spanning all major cloud providers and all regions of the world such as Synadia's NGS.

Connecting NATS Client applications to the NATS servers

To connect a NATS client application with a NATS service, and then subscribe or publish messages to subjects, it only needs to be configured with:

Simple messaging design

NATS makes it easy for applications to communicate by sending and receiving messages. These messages are addressed and identified by subject strings, and do not depend on network location.

Data is encoded and framed as a message and sent by a publisher. The message is received, decoded, and processed by one or more subscribers.

With this simple design, NATS lets programs share common message-handling code, isolate resources and interdependencies, and scale by easily handling an increase in message volume, whether those are service requests or stream data.

NATS Quality of service (QoS)

NATS offers multiple qualities of service, depending on whether the application uses just the Core NATS functionality or also leverages the added functionalities enabled by NATS JetStream (JetStream is built into nats-server but may not be enabled on all service infrastructures).

• At most once QoS: Core NATS offers an at most once quality of service. If a subscriber is not listening on the subject (no subject match), or is not active when the message is sent, the message is not received. This is the same level of guarantee that TCP/IP provides. Core NATS is a fire-and-forget messaging system. It will only hold messages in memory and will never write messages directly to disk.

Core NATS is the foundational functionality in a NATS system. It operates on a publish-subscribe model using subject/topic-based addressing. This model offers two significant advantages: location independence and a default many-to-many (M:N) communication pattern. These fundamental concepts enable powerful and innovative solutions for common development patterns, such as microservices, without requiring additional technologies like load balancers, API gateways, or DNS configuration.

NATS Pub/Sub Walkthrough

This simple walkthrough demonstrates some ways in which subscribers listen on subjects, and publishers send messages on specific subjects.

Walkthrough prerequisites

1. Create Subscriber 1

In a shell or command prompt session, start a client subscriber program.

Here, <subject> is a subject to listen on. It helps to use unique and well thought-through subject strings because you need to ensure that messages reach the correct subscribers even when wildcards are used.

For example:

You should see the message: Listening on [msg.test]

2. Create a Publisher and publish a message

In another shell or command prompt, create a NATS publisher and send a message.

Where <subject> is the subject name and <message> is the text to publish.

For example:

3. Verify message publication and receipt

You'll notice that the publisher sends the message and prints: Published [msg.test] : 'NATS MESSAGE'.

The subscriber receives the message and prints: [#1] Received on [msg.test]: 'NATS MESSAGE'.

If the receiver does not get the message, you'll need to check if you are using the same subject name for the publisher and the subscriber.

4. Try publishing another message

You'll notice that the subscriber receives the message. Note that a message count is incremented each time your subscribing client receives a message on that subject.

5. Create Subscriber 2

In a new shell or command prompt, start a new NATS subscriber.

6. Publish another message using the publisher client

Verify that both subscribing clients receive the message.

7. Create Subscriber 3

In a new shell or command prompt session, create a new subscriber that listens on a different subject.

8. Publish another message

Subscriber 1 and Subscriber 2 receive the message, but Subscriber 3 does not. Why? Because Subscriber 3 is not listening on the message subject used by the publisher.

9. Alter Subscriber 3 to use a wildcard

Change the last subscriber to listen on msg.* and run it:

Note: NATS supports the use of wildcard characters for message subscribers only. You cannot publish a message using a wildcard subject.

10. Publish another message

This time, all three subscribing clients should receive the message.

Do try out a few more variations of substrings and wildcards to test your understanding.

See Also

Publish-subscribe pattern with the NATS CLI

Publish-Subscribe

NATS implements a publish-subscribe message distribution model for one-to-many communication. A publisher sends a message on a subject and any active subscriber listening on that subject receives the message. Subscribers can also register interest in wildcard subjects that work a bit like a regular expression (but only a bit). This one-to-many pattern is sometimes called a fan-out.

Messages

Messages are composed of:

1. A subject.

2. A payload in the form of a byte array.

3. Any number of header fields.

4. An optional 'reply' address field.

Messages have a maximum size (which is set in the server configuration with max_payload). The size is set to 1 MB by default, but can be increased up to 64 MB if needed (though we recommend keeping the max message size to something more reasonable like 8 MB).

When subscribers register themselves to receive messages from a publisher, the 1:N fan-out pattern of messaging ensures that any message sent by a publisher, reaches all subscribers that have registered. NATS provides an additional feature named "queue", which allows subscribers to register themselves as part of a queue. Subscribers that are part of a queue, form the "queue group".

How queue groups function

As an example, consider message delivery occurring in the 1:N pattern to all subscribers based on the subject name (delivery happens even to subscribers that are not part of a queue group). If a subscriber is registered based on a queue name, it will always receive messages it is subscribed to, based on the subject name. However, if more subscribers are added to the same queue name, they become a queue group, and only one randomly chosen subscriber of the queue group will consume a message each time a message is received by the queue group. Such distributed queues are a built-in load balancing feature that NATS provides.

Advantages

• Ensures application fault tolerance

• Workload processing can be scaled up or down

• Scale your consumers up or down without duplicate messages

• No extra configuration required

• Queue groups are defined by the application and their queue subscribers, rather than the server configuration

Queue subscribers are ideal for scaling services. Scale up is as simple as running another application, scale down is terminating the application with a signal that drains the in flight requests. This flexibility and lack of any configuration changes makes NATS an excellent service communication technology that can work with all platform technologies.

No responder

When a request is made to a service (request/reply) and the NATS Server knows there are no services available (since there are no client applications currently subscribing to the subject in a queue-group) the server will send a “no-responders” protocol message back to the requesting client which will break from blocking API calls. This allows applications to react immediately. This further enables building a highly responsive system at scale, even in the face of application failures and network partitions.

Stream as a queue

Queuing geo-affinity

When connecting to a globally distributed NATS super-cluster, there is an automatic service geo-affinity due to the fact that a service request message will only be routed to another cluster (i.e. another region) if there are no listeners on the cluster available to handle the request locally.

Tutorial

The NATS.io team is continually working to bring you features that enhance your NATS experience. Below, you will find summaries of new NATS implementations. Release notes for the latest patch releases are available on GitHub Releases

Roadmap for future releases

See https://nats.io/about/#roadmap

Server release v2.11.0

Check out the:

• Upgrade guide

• Release notes

Server release v2.10.0

Check out the:

• Upgrade guide

• Podcast EP06: The journey and features of the NATS.io 2.10 release

• Release notes

Server release v2.9.0

Please check out the announcement post on the blog and the detailed release notes in the server repo.

Server release v2.8.0

LeafNode

Support for a min_version in the leafnodes{} that would reject servers with a lower version. Note that this would work only for servers that are v2.8.0 and above.

Monitoring

• Server version in monitoring landing page.

• Logging to /healthz endpoint when failure occurs.

• MQTT and Websocket blocks in the /varz endpoint.

JetStream

• Consumer check added to healthz endpoint.

• Max stream bytes checks.

• Ability to limit a consumer's MaxAckPending value.

• Allow streams and consumers to migrate between clusters. This feature is considered "beta".

• New unique_tag option in jetstream{} configuration block to prevent placing a stream in the same availability zone twice.

• Stream Alternates field in StreamInfo response. They provide a priority list of mirrors and the source in relation to where the request originated.

• Deterministic subject tokens to partition mapping.

For full release information, see links below;

• Release notes 2.8.0

• Full list of Changes 2.7.4...2.8.0

Server release v2.7.0

Notice for JetStream Users

See important note if using LeafNode regarding domains.

Configuration

Ability to configure account limits (max_connections, max_subscriptions, max_payload, max_leafnodes) in server configuration file.

JetStream

• Overflow placement for streams. A stream can now be placed in the closest cluster from the origin request if it can be placed there.

• Support for ephemeral Pull consumers (client libraries will need to be updated to allow those).

• New consumer configuration options

  • For Pull Consumers: MaxRequestBatch to limit the batch size any client can request MaxRequestExpires to limit the expiration any client can request

  • For ephemeral consumers: InactiveThreshold duration that instructs the server to cleanup ephemeral consumers that are inactive for that long.

• Ability to configure max_file_store and max_memory_store in the jetstream{} block as strings with the following suffixes K, M, G and T, for instance: max_file_store: "256M".

• Support for the JWT field MaxBytesRequired, which defines a per-account maximum bytes for assets.

MQTT

Support for websocket protocol. MQTT clients must connect to the opened websocket port and add /mqtt to the URL path.

TLS

Ability to rate-limit the clients connections by adding the connection_rate_limit: <number of connections per seconds> in the tls{} top-level block.

For full release information, see links below;

• Release notes 2.7.0

• Full list of Changes 2.6.6...2.7.0

Server release v2.6.0

Notice for JetStream Users

See important note if upgrading from a version prior to NATS Server v2.4.0.

Notice for MQTT Users

See important notes if upgrading from a version prior to v2.5.0.

Monitoring

• JetStream's reserved memory and memory used from accounts with reservations in /jsz and /varz endpoints

• Hardened systemd service

For full release information, see links below;

• Release notes 2.6.0

• Full list of Changes 2.5.0...2.6.0

Server release v2.5.0

Notice for JetStream Users

See important note if upgrading from a version prior to NATS Server v2.4.0.

MQTT/Monitoring

• MQTTClient in the /connz connections report and system events CONNECT and DISCONNECT. Ability to select on mqtt_client.

MQTT Improvement

• Sessions are now all stored inside a single stream, as opposed to individual streams, reducing resources usage.

MQTT Update

• Due to the aforementioned improvement described above, when an MQTT client connects for the first time after an upgrade to this server version, the server will migrate all individual $MQTT_sess_<xxxx> streams to a new $MQTT_sess stream for the user's account.

For full release information, see links below;

• Release notes 2.5.0

• Full list of Changes 2.4.0...2.5.0

Server release v2.4.0

Notice for JetStream Users

With the latest release of the NATS server, we have fixed bugs around queue subscriptions and have restricted undesired behavior that could be confusing or introduce data loss by unintended/undefined behavior of client applications. If you are using queue subscriptions on a JetStream Push Consumer or have created multiple push subscriptions on the same consumer, you may be affected and need to upgrade your client version along with the server version. We’ve detailed the behavior with different client versions below.

With a NATS Server prior to v2.4.0 and client libraries prior to these versions: NATS C client v3.1.0, Go client v1.12.0, Java client 2.12.0-SNAPSHOT, NATS.js v2.2.0, NATS.ws v1.3.0, NATS.deno v1.2.0, NATS .NET 0.14.0-pre2:

• It was possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is not correct since each instance will receive the same copy of a message and acknowledgment is therefore meaningless since the first instance to acknowledge the message will prevent other instances to control if/when a message should be acknowledged.

• Similar to the first issue, it was possible to create many different queue groups for one single JetStream consumer.

• For queue subscriptions, if no consumer nor durable name was provided, the libraries would create ephemeral JetStream consumers, which meant that each member of the same group would receive the same message as the other members, which was not the expected behavior. Users assumed that 2 members subscribing to “foo” with the queue group named “bar” would load-balance the consumption of messages from the stream/consumer.

• It was possible to create a queue subscription on a JetStream consumer configured with heartbeat and/or flow control. This does not make sense because by definition, queue members would receive some (randomly distributed) messages, so the library would think that heartbeats are missed, and flow control would also be disrupted.

If above client libraries are not updated to the latest but the NATS Server is upgraded to v2.4.0:

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. Since the check is performed by the library (with the help of a new field called PushBound in the consumer information object set by the server), this misbehavior is still possible.

• Queue subscriptions will not receive any message. This is because the server now has a new field DeliverGroup in the consumer configuration, which won’t be set for existing JetStream consumers and by the older libraries, and detects interest (and starts delivering) only when a subscription on the deliver subject for a queue subscription matching the “deliver group” name is found. Since the JetStream consumer is thought to be a non-deliver-group consumer, the opposite happens: the server detects a core NATS queue subscription on the “deliver subject”, therefore does not trigger delivery on the JetStream consumer’s “deliver subject”.

The 2 other issues are still present because those checks are done in the updated libraries.

If the above client libraries are updated to the latest version, but the NATS Server is still to version prior to v2.4.0 (that is, up to v2.3.4):

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is because the JetStream consumer’s information retrieved by the library will not have the PushBound boolean set by the server, therefore will not be able to alert the user that they are trying to create multiple subscription instances for the same JetStream consumer.

• Queue subscriptions will fail because the consumer information returned will not contain the DeliverGroup field. The error will be likely to the effect that the user tries to create a queue subscription to a non-queue JetStream consumer. Note that if the application creates a queue subscription for a non-yet created JetStream consumer, then this call will succeed, however, adding new members or restarting the application with the now existing JetStream consumer will fail.

• Creating queue subscriptions without a named consumer/durable will now result in the library using the queue name as the durable name.

• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

For completeness, using the latest client libraries and NATS Server v2.4.0:

• Trying to start multiple non-queue subscriptions instances for the same JetStream consumer will now return an error to the effect that the user is trying to create a “duplicate subscription”. That is, there is already an active subscription on that JetStream consumer. It is now only possible to create a queue group for a JetStream consumer created for that group. The DeliverGroup field will be set by the library or need to be provided when creating the consumer externally.

• Trying to create a queue subscription without a durable nor consumer name results in the library creating/using the queue group as the JetStream consumer’s durable name.

• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

Note that if the server v2.4.0 recovers existing JetStream consumers that were created prior to v2.4.0 (and with older libraries), none of them will have a DeliverGroup, so none of them can be used for queue subscriptions. They will have to be recreated.

JetStream

• Domain to the content of a PubAck protocol

• PushBound boolean in ConsumerInfo to indicate that a push consumer is already bound to an active subscription

• DeliverGroup string in ConsumerConfig to specify which deliver group (or queue group name) the consumer is created for

• Warning log statement in situations where catchup for a stream resulted in an error

Monitoring

• The ability for normal accounts to access scoped connz information

Misc

• Operator option resolver_pinned_accounts to ensure users are signed by certain accounts

For full release information, see links below;

• Release notes 2.4.0

• Full list of Changes 2.3.4...2.4.0

Server release v2.3.0

• OCSP support

JetStream

• Richer API errors. JetStream errors now contain an ErrCode that uniquely describes the error.

• Ability to send more advanced Stream purge requests that can purge all messages for a specific subject

• Stream can now be configured with a per-subject message limit

• Encryption of JetStream data at rest

For full release information, see links below;

• Release notes 2.3.0

• Full list of Changes 2.2.6...2.3.0

Server release v2.2.0

See NATS 2.2 for new features.

Server release v2.1.7

Monitoring Endpoints Available via System Services

Monitoring endpoints as listed in the table below are accessible as system services using the following subject pattern:

• $SYS.REQ.SERVER.<id>.<endpoint-name> (request server monitoring endpoint corresponding to endpoint name.)

• $SYS.REQ.SERVER.PING.<endpoint-name> (from all server request server monitoring endpoint corresponding to endpoint name - will return multiple messages)

For more information on monitoring endpoints see NATS Server Configurations System Events.

Addition of no_auth_user Configuration

Configuration of no_auth_user allows you to refer to a configured user/account when no credentials are provided.

For more information and examples, see Securing NATS

For full release information, see links below;

• Release notes 2.1.7

• Full list of Changes 2.1.6...2.1.7

Server release v2.1.6

TLS Configuration for Account Resolver

This release adds the ability to specify TLS configuration for the account resolver.

resolver_tls {
  cert_file: ...
  key_file: ...
  ca_file: ...
}

Additional Trace & Debug Verbosity Options

trace_verbose and command line parameters -VV and -DVV added. See NATS Logging Configuration

Subscription Details in Monitoring Endpoints

We've added the option to include subscription details in monitoring endpoints /routez and /connz. For instance /connz?subs=detail will now return not only the subjects of the subscription, but the queue name (if applicable) and some other details.

• Release notes 2.1.6

• Full list of Changes 2.1.4...2.1.6

Server release v2.1.4

Log Rotation

NATS introduces logfile_size_limit allowing auto-rotation of log files when the size is greater than the configured limit set in logfile_size_limit as a number of bytes. You can provide the size with units, such as MB, GB, etc. The backup files will have the same name as the original log file with the suffix .yyyy.mm.dd.hh.mm.ss.micros. For more information see Configuring Logging in the NATS Server Configuration section.

• Release notes 2.1.4

• Full list of Changes 2.1.2...2.1.4

Server release v2.1.2

Queue Permissions

Queue Permissions allow you to express authorization for queue groups. As queue groups are integral to implementing horizontally scalable microservices, control of who is allowed to join a specific queue group is important to the overall security model. Original PR - https://github.com/nats-io/nats-server/pull/1143

More information on Queue Permissions can be found in the Developing with NATS section.

Server release v2.1.0

Service Latency Tracking

As services and service mesh functionality has become prominent, we have been looking at ways to make running scalable services on NATS.io a great experience. One area we have been looking at is observability. With publish/subscribe systems, everything is inherently observable, however we realized it was not as simple as it could be. We wanted the ability to transparently add service latency tracking to any given service with no changes to the application. We also realized that global systems, such as those NATS.io can support, needed something more than a single metric. The solution was to allow any sampling rate to be attached to an exported service, with a delivery subject for all collected metrics. We collect metrics that show the requestor’s view of latency, the responder’s view of latency and the NATS subsystem itself, even when requestor and responder are in different parts of the world and connected to different servers in a NATS supercluster.

• Release notes 2.1.0

• Full list of Changes 2.0.4...2.1.0

Server release v2.0.4

Response Only Permissions

For services, the authorization for responding to requests usually included wildcards for _INBOX.> and possibly $GR.> with a supercluster for sending responses. What we really wanted was the ability to allow a service responder to only respond to the reply subject it was sent.

Response Types

Exported Services were originally tied to a single response. We added the type for the service response and now support singletons (default), streams and chunked. Stream responses represent multiple response messages, chunked represents a single response that may have to be broken up into multiple messages.

• Release notes 2.0.4

• Full list of Changes 2.0.2...2.0.4

NATS has a built-in persistence engine called JetStream which enables messages to be stored and replayed at a later time. Unlike NATS Core which requires you to have an active subscription to process messages as they happen, JetStream allows the NATS server to capture messages and replay them to consumers as needed. This functionality enables a different quality of service for your NATS messages, and enables fault-tolerant and high-availability configurations.

JetStream is built into nats-server. If you have a cluster of JetStream-enabled servers you can enable data replication and thus guard against failures and service disruptions.

JetStream was created to address the problems identified with streaming technology today - complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is truly multi-tenant, horizontally scalable, or supports multiple deployment models. No other technology that we are aware of can scale from edge to cloud using the same security context while having complete deployment observability for operations.

Additional capabilities enabled by JetStream

The JetStream persistence layer enables additional use cases typically not found in messaging systems. Being built on top of JetStream they inherit the core capabilities of JetStream, replication, security, routing limits, and mirroring.

• Key Value Store A map (associative array) with atomic operations

• Object Store File transfer, replications and storage API. Uses chunked transfers for scalability.

Key/Value and File transfer are capabilities are commonly found in in-memory databases or deployment tools. While NATS does not intend to compete with the feature set of such tools, it is our goal to provide the developer with reasonable complete set of data storage and replications features for use cases like micro service, edge deployments and server management.

Configuration

To configure a nats-server with JetStream refer to:

• Configuring JetStream

• JetStream Clustering

Examples

For runnable JetStream code examples, refer to NATS by Example.

Goals

JetStream was developed with the following goals in mind:

• The system must be easy to configure and operate and be observable.

• The system must be secure and operate well with NATS 2.0 security models.

• The system must scale horizontally and be applicable to a high ingestion rate.

• The system must support multiple use cases.

• The system must self-heal and always be available.

• The system must allow NATS messages to be part of a stream as desired.

• The system must display payload agnostic behavior.

• The system must not have third party dependencies.

JetStream capabilities

Streaming: temporal decoupling between the publishers and subscribers

One of the tenets of basic publish/subscribe messaging is that there is a required temporal coupling between the publishers and the subscribers: subscribers only receive the messages that are published when they are actively connected to the messaging system (i.e. they do not receive messages that are published while they are not subscribing or not running or disconnected). The traditional way for messaging systems to provide temporal decoupling of the publishers and subscribers is through the 'durable subscriber' functionality or sometimes through 'queues', but neither one is perfect:

• durable subscribers need to be created before the messages get published

• queues are meant for workload distribution and consumption, not to be used as a mechanism for message replay.

However, in many use cases, you do not need to 'consume exactly once' functionality but rather the ability to replay messages on demand, as many times as you want. This need has led to the popularity of some 'streaming' messaging platforms.

JetStream provides both the ability to consume messages as they are published (i.e. 'queueing') as well as the ability to replay messages on demand (i.e. 'streaming'). See retention policies below.

Replay policies

JetStream consumers support multiple replay policies, depending on whether the consuming application wants to receive either:

• all of the messages currently stored in the stream, meaning a complete 'replay' and you can select the 'replay policy' (i.e. the speed of the replay) to be either:

  • instant (meaning the messages are delivered to the consumer as fast as it can take them).

  • original (meaning the messages are delivered to the consumer at the rate they were published into the stream, which can be very useful for example for staging production traffic).

• the last message stored in the stream, or the last message for each subject (as streams can capture more than one subject).

• starting from a specific sequence number.

• starting from a specific start time.

Retention policies and limits

JetStream enables new functionalities and higher qualities of service on top of the base 'Core NATS' functionality. However, practically speaking, streams can't always just keep growing 'forever' and therefore JetStream supports multiple retention policies as well as the ability to impose size limits on streams.

Limits

You can impose the following limits on a stream

• Maximum message age.

• Maximum total stream size (in bytes).

• Maximum number of messages in the stream.

• Maximum individual message size.

• You can also set limits on the number of consumers that can be defined for the stream at any given point in time.

You must also select a discard policy which specifies what should happen once the stream has reached one of its limits and a new message is published:

• discard old means that the stream will automatically delete the oldest message in the stream to make room for the new messages.

• discard new means that the new message is discarded (and the JetStream publish call returns an error indicating that a limit was reached).

Retention policy

You can choose what kind of retention you want for each stream:

• limits (the default) is to provide a replay of messages in the stream.

• work queue (the stream is used as a shared queue and messages are removed from it as they are consumed) is to provide the exactly-once consumption of messages in the stream.

• interest (messages are kept in the stream for as long as there are consumers that haven't delivered the message yet) is a variation of work queue that only retains messages if there is interest (consumers currently defined on the stream) for the message's subject.

Note that regardless of the retention policy selected, the limits (and the discard policy) always apply.

Subject mapping transformations

JetStream also enables the ability to apply subject mapping transformations to messages as they are ingested into a stream.

Persistent and Consistent distributed storage

You can choose the durability as well as the resilience of the message storage according to your needs.

• Memory storage.

• File storage.

• Replication (1 (none), 2, 3) between nats servers for Fault Tolerance.

JetStream uses a NATS optimized RAFT distributed quorum algorithm to distribute the persistence service between NATS servers in a cluster while maintaining immediate consistency (as opposed to eventual consistency) even in the face of failures.

For writes (publications to a stream), the formal consistency model of NATS JetStream is Linearizable. On the read side (listening to or replaying messages from streams) the formal models don't really apply because JetStream does not support atomic batching of multiple operations together (so the only kind of 'transaction' is the persisting, replicating and voting of a single operation on the stream) but in essence, JetStream is serializable because messages are added to a stream in one global order (which you can control using compare and publish).

Do note, while we do guarantee immediate consistency when it comes to monotonic writes and monotonic reads. We don't guarantee read your writes at this time, as reads through direct get requests may be served by followers or mirrors. More consistent results can be achieved by sending get requests to the stream leader.

JetStream can also provide encryption at rest of the messages being stored.

In JetStream the configuration for storing messages is defined separately from how they are consumed. Storage is defined in a Stream and consuming messages is defined by multiple Consumers.

Stream replication factor

A stream's replication factor (R, often referred to as the number 'Replicas') determines how many places it is stored allowing you to tune to balance risk with resource usage and performance. A stream that is easily rebuilt or temporary might be memory-based with a R=1 and a stream that can tolerate some downtime might be file-based R-1.

Typical usage to operate in typical outages and balance performance would be a file-based stream with R=3. A highly resilient, but less performant and more expensive configuration is R=5, the replication factor limit.

Rather than defaulting to the maximum, we suggest selecting the best option based on the use case behind the stream. This optimizes resource usage to create a more resilient system at scale.

• Replicas=1 - Cannot operate during an outage of the server servicing the stream. Highly performant.

• Replicas=2 - No significant benefit at this time. We recommend using Replicas=3 instead.

• Replicas=3 - Can tolerate the loss of one server servicing the stream. An ideal balance between risk and performance.

• Replicas=4 - No significant benefit over Replicas=3 except marginally in a 5 node cluster.

• Replicas=5 - Can tolerate simultaneous loss of two servers servicing the stream. Mitigates risk at the expense of performance.

Mirroring and Sourcing between streams

JetStream also allows server administrators to easily mirror streams, for example between different JetStream domains in order to offer disaster recovery. You can also define a stream that 'sources' from one or more other streams.

De-coupled flow control

JetStream provides decoupled flow control over streams, the flow control is not 'end to end' where the publisher(s) are limited to publish no faster than the slowest of all the consumers (i.e. the lowest common denominator) can receive but is instead happening individually between each client application (publishers or consumers) and the nats server.

When using the JetStream publish calls to publish to streams there is an acknowledgment mechanism between the publisher and the NATS server, and you have the choice of making synchronous or asynchronous (i.e. 'batched') JetStream publish calls.

On the subscriber side, the sending of messages from the NATS server to the client applications receiving or consuming messages from streams is also flow controlled.

Exactly once semantics

Because publications to streams using the JetStream publish calls are acknowledged by the server the base quality of service offered by streams is 'at least once', meaning that while reliable and normally duplicate free there are some specific failure scenarios that could result in a publishing application believing (wrongly) that a message was not published successfully and therefore publishing it again, and there are failure scenarios that could result in a client application's consumption acknowledgment getting lost and therefore in the message being re-sent to the consumer by the server. Those failure scenarios while being rare and even difficult to reproduce do exist and can result in perceived 'message duplication' at the application level.

Therefore, JetStream also offers an 'exactly once' quality of service. For the publishing side, it relies on the publishing application attaching a unique message or publication ID in a message header and on the server keeping track of those IDs for a configurable rolling period of time in order to detect the publisher publishing the same message twice. For the subscribers a double acknowledgment mechanism is used to avoid a message being erroneously re-sent to a subscriber by the server after some kinds of failures.

Consumers

JetStream consumers are 'views' on a stream, they are subscribed to (or pulled) by client applications to receive copies of (or to consume if the stream is set as a working queue) messages stored in the stream.

Fast push consumers

Client applications can choose to use fast un-acknowledged push (ordered) consumers to receive messages as fast as possible (for the selected replay policy) on a specified delivery subject or to an inbox. Those consumers are meant to be used to 'replay' rather than 'consume' the messages in a stream.

Horizontally scalable pull consumers with batching

Client applications can also use and share pull consumers that are demand-driven, support batching and must explicitly acknowledge message reception and processing which means that they can be used to consume (i.e. use the stream as a distributed queue) as well as process the messages in a stream.

Pull consumers can and are meant to be shared between applications (just like queue groups) in order to provide easy and transparent horizontal scalability of the processing or consumption of messages in a stream without having (for example) to worry about having to define partitions or worry about fault-tolerance.

Note: using pull consumers doesn't mean that you can't get updates (new messages published into the stream) 'pushed' in real-time to your application, as you can pass a (reasonable) timeout to the consumer's Fetch call and call it in a loop.

Consumer acknowledgments

While you can decide to use un-acknowledged consumers trading quality of service for the fastest possible delivery of messages, most processing is not idem-potent and requires higher qualities of service (such as the ability to automatically recover from various failure scenarios that could result in some messages not being processed or being processed more than once) and you will want to use acknowledged consumers. JetStream supports more than one kind of acknowledgment:

• Some consumers support acknowledging all the messages up to the sequence number of the message being acknowledged, some consumers provide the highest quality of service but require acknowledging the reception and processing of each message explicitly as well as the maximum amount of time the server will wait for an acknowledgment for a specific message before re-delivering it (to another process attached to the consumer).

• You can also send back negative acknowledgements.

• You can even send in progress acknowledgments (to indicate that you are still processing the message in question and need more time before acking or nacking it).

Key Value Store

The JetStream persistence layer enables the Key Value store: the ability to store, retrieve and delete value messages associated with a key into a bucket.

• Concepts

• Walkthrough

• API and details

Watch and History

You can subscribe to changes in a Key Value on the bucket or individual key level with watch and optionally retrieve a history of the values (and deletions) that have happened on a particular key.

Atomic updates and locking

The Key Value store supports atomic create and update operations. This enables pessimistic locks (by creating a key and holding on to it) and optimistic locks (using CAS - compare and set).

Object Store

The Object Store is similar to the Key Value Store. The key being replaced by a file name and value being designed to store arbitrarily large objects (e.g. files, even if they are very large) rather than 'values' that are message-sized (i.e. limited to 1Mb by default). This is achieved by chunking messages.

• Concepts

• Walkthrough

• API and details

Legacy

Note that JetStream completely replaces the STAN legacy NATS streaming layer.

NATS supports request-reply messaging. In this tutorial you explore how to exchange point-to-point messages using NATS.

Prerequisites

If you have not already done so, you need to install the nats CLI Tool and optionally, the nats-server on your machine.

Walkthrough

Start two terminal sessions. These will be used to run the NATS request and reply clients.

In one terminal, run the reply client listener

nats reply help.please 'OK, I CAN HELP!!!'

You should see the message: Listening on [help.please]

This means that the NATS receiver client is listening for request messages on the "help.please" subject. In NATS, the receiver is a subscriber.

In the other terminal, run the request client

nats request help.please 'I need help!'

The NATS requestor client makes a request by sending the message "I need help!" on the “help.please” subject.

The NATS receiver client receives the message, formulates the reply ("OK, I CAN HELP!!!"), and sends it to the inbox of the requester.

This feature comparison is a summary of a few of the major components in several of the popular messaging technologies of today. This is by no means an exhaustive list and each technology should be investigated thoroughly to decide which will work best for your implementation.

In this comparison, we will be featuring NATS, Apache Kafka, RabbitMQ, Apache Pulsar, and gRPC.

Language and Platform Coverage

Built-in Patterns

Delivery Guarantees

Multi-tenancy and Sharing

AuthN

AuthZ

Message Retention and Persistence

High Availability and Fault Tolerance

Deployment

Monitoring

Management

Integrations

References

1. https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.0/bk_planning-your-deployment/content/ch_hardware-sizing.html

2. https://pulsar.apache.org/docs/v1.21.0-incubating/deployment/cluster/

3. https://github.com/grpc-ecosystem

Request-Reply is a common pattern in modern distributed systems. A request is sent, and the application either waits on the response with a certain timeout, or receives a response asynchronously.

The increased complexity of modern systems necessitates features like location transparency, scale-up and scale-down, observability (measuring a system's state based on the data it generates) and more. In order to implement this feature-set, various other technologies needed to incorporate additional components, sidecars (processes or services that support the primary application) and proxies. NATS on the other hand, implemented Request-Reply much more easily.

NATS makes Request-Reply simple and powerful

• NATS supports the Request-Reply pattern using its core communication mechanism — publish and subscribe. A request is published on a given subject using a reply subject. Responders listen on that subject and send responses to the reply subject. Reply subjects are called "inbox". These are unique subjects that are dynamically directed back to the requester, regardless of the location of either party.

• Multiple NATS responders can form dynamic queue groups. Therefore, it's not necessary to manually add or remove subscribers from the group for them to start or stop being distributed messages. It’s done automatically. This allows responders to scale up or down as per demand.

• NATS applications "drain before exiting" (processing buffered messages before closing the connection). This allows the applications to scale down without dropping requests.

• Since NATS is based on publish-subscribe, observability is as simple as running another application that can view requests and responses to measure latency, watch for anomalies, direct scalability and more.

• The power of NATS even allows multiple responses, where the first response is utilized and the system efficiently discards the additional ones. This allows for a sophisticated pattern to have multiple responders, reduce response latency and jitter.

The pattern

Try NATS request-reply on your own, using a live server by walking through the request-reply walkthrough.

No responders

When a request is sent to a subject that has no subscribers, it can be convenient to know about it right away. For this use-case, a NATS client can opt-into no_responder messages. This requires a server and client that support headers. When enabled, a request sent to a subject with no subscribers will immediately receive a reply that has no body, and a 503 status.

Most clients will represent this case by raising or returning an error. For example:

m, err := nc.Request("foo", nil, time.Second);
# err == nats.ErrNoResponders

NATS supports a form of load balancing using queue groups. Subscribers register a queue group name. A single subscriber in the group is randomly selected to receive the message.

Walkthrough prerequisites

If you have not already done so, you need to install the nats CLI Tool and optionally the nats-server on your machine.

1. Start the first member of the queue group

The nats reply instances don't just subscribe to the subject but also automatically join a queue group ("NATS-RPLY-22" by default)

nats reply foo "service instance A Reply# {{Count}}"

2. Start a second member of the queue-group

In a new window

nats reply foo "service instance B Reply# {{Count}}"

3. Start a third member of the queue-group

In a new window

nats reply foo "service instance C Reply# {{Count}}"

4. Publish a NATS message

nats request foo "Simple request"

5. Verify message publication and receipt

You should see that only one of the my-queue group subscribers receives the message and replies it, and you can also see which one of the available queue-group subscribers processed the request from the reply message received (i.e. service instance A, B or C)

6. Publish another message

nats request foo "Another simple request"

You should see that a different queue group subscriber receives the message this time, chosen at random among the 3 queue group members.

You can also send any number of requests back-to-back. From the received messages, you'll see the distribution of those requests amongst the members of the queue-group. For example: nats request foo --count 10 "Request {{Count}}"

7. Stop/start queue-group members

You can at any time start yet another service instance, or kill one and see how the queue-group automatically takes care of adding/removing those instances from the group.

See Also

Queue groups using the NATS CLI

Streams are message stores, each stream defines how messages are stored and what the limits (duration, size, interest) of the retention are. Streams consume normal NATS subjects, any message published on those subjects will be captured in the defined storage system. You can do a normal publish to the subject for unacknowledged delivery, though it’s better to use the JetStream publish calls instead as the JetStream server will reply with an acknowledgement that it was successfully stored.

The diagram above shows the concept of storing all ORDERS.* in the Stream even though there are many types of order related messages. We’ll show how you can selectively consume subsets of messages later. Relatively speaking the Stream is the most resource consuming component so being able to combine related data in this manner is important to consider.

Streams can consume many subjects. Here we have ORDERS.* but we could also consume SHIPPING.state into the same Stream should that make sense.

Stream limits and message retention

Streams support various retention policies which define when messages in the stream can be automatically deleted, such as when stream limits are hit (like max count, size or age of messages), or also more novel options that apply on top of the limits such as interest-based retention or work-queue semantics (see Retention Policy).

Upon reaching message limits, the server will automatically discard messages either by removing the oldest messages to make room for new ones (DiscardOld) or by refusing to store new messages (DiscardNew). For more details, see Discard Policy.

Streams support deduplication using a Nats-Msg-Id header and a sliding window within which to track duplicate messages. See the Message Deduplication section.

For examples on how to configure streams with your preferred NATS client, see NATS by Example.

Configuration

Below are the set of stream configuration options that can be defined. The Version column indicates the version of the server the option was introduced. The Editable column indicates the option can be edited after the stream created. See client-specific examples here.

StorageType

The storage types include:

• File (default) - Uses file-based storage for stream data.

• Memory - Uses memory-based storage for stream data.

Subjects

Note: a stream configured as a mirror cannot be configured with a set of subjects. A mirror implicitly sources a subset of the origin stream (optionally with a filter), but does not subscribe to additional subjects.

If no explicit subject is specified, the default subject will be the same name as the stream. Multiple subjects can be specified and edited over time. Note, if messages are stored by a stream on a subject that is subsequently removed from the stream config, consumers will still observe those messages if their subject filter overlaps.

RetentionPolicy

The retention options include:

• LimitsPolicy (default) - Retention based on the various limits that are set including: MaxMsgs, MaxBytes, MaxAge, and MaxMsgsPerSubject. If any of these limits are set, whichever limit is hit first will cause the automatic deletion of the respective message(s). See a full code example.

• WorkQueuePolicy - Retention with the typical behavior of a FIFO queue. Each message can be consumed only once. This is enforced by only allowing one consumer to be created per subject for a work-queue stream (i.e. the consumers' subject filter(s) must not overlap). Once a given message is ack’ed, it will be deleted from the stream. See a full code example.

• InterestPolicy - Retention based on the consumer interest in the stream and messages. The base case is that there are zero consumers defined for a stream. If messages are published to the stream, they will be immediately deleted so there is no interest. This implies that consumers need to be bound to the stream ahead of messages being published to the stream. Once a given message is ack’ed by all consumers filtering on the subject, the message is deleted (same behavior as WorkQueuePolicy). See a full code example.

DiscardPolicy

The discard behavior applies only for streams that have at least one limit defined. The options include:

• DiscardOld (default) - This policy will delete the oldest messages in order to maintain the limit. For example, if MaxAge is set to one minute, the server will automatically delete messages older than one minute with this policy.

• DiscardNew - This policy will reject new messages from being appended to the stream if it would exceed one of the limits. An extension to this policy is DiscardNewPerSubject which will apply this policy on a per-subject basis within the stream.

Placement

Refers to the placement of the stream assets (data) within a NATS deployment, be it a single cluster or a supercluster. A given stream, including all replicas (not mirrors), are bound to a single cluster. So when creating or moving a stream, a cluster will be chosen to host the assets.

Without declaring explicit placement for a stream, by default, the stream will be created within the cluster that the client is connected to assuming it has sufficient storage available.

By declaring stream placement, where these assets are located can be controlled explicitly. This is generally useful to co-locate with the most active clients (publishers or consumers) or may be required for data sovereignty reasons.

Placement is supported in all client SDKs as well as the CLI. For example, adding a stream via the the CLI to place a stream in a specific cluster looks like this:

nats stream add --cluster aws-us-east1-c1

For this to work, all servers in a given cluster must define the name field within the cluster server configuration block.

cluster {
  name: aws-us-east1-c1
  # etc..
}

If you have multiple clusters that form a supercluster, then each is required to have a different name.

Another placement option are tags. Each server can have its own set of tags, defined in configuration, typically describing properties of geography, hosting provider, sizing tiers, etc. In addition, tags are often used in conjunction with the jetstream.unique_tag config option to ensure that replicas must be placed on servers having different values for the tag.

For example, a server A, B, and C in the above cluster might all the same configuration except for the availability zone they are deployed to.

// Server A
server_tags: ["cloud:aws", "region:us-east1", "az:a"]

jetstream: {
  unique_tag: "az"
}

// Server B
server_tags: ["cloud:aws", "region:us-east1", "az:b"]

jetstream: {
  unique_tag: "az"
}

// Server C
server_tags: ["cloud:aws", "region:us-east1", "az:c"]

jetstream: {
  unique_tag: "az"
}

Now we can create a stream by using tags, for example indicating we want a stream in us-east1.

nats stream add --tag region:us-east1

If we had a second cluster in Google Cloud with the same region tag, the stream could be placed in either the AWS or GCP cluster. However, the unique_tag constraint ensures each replica will be placed in a different AZ in the cluster that was selected implicitly by the placement tags.

Although less common, note that both the cluster and tags can be used for placement. This would be used if a single cluster contains servers have different properties.

Sources and Mirrors

When a stream is configured with a source or mirror, it will automatically and asynchronously replicate messages from the origin stream. There are several options when declaring the configuration.

A source or mirror stream can have its own retention policy, replication, and storage type. Changes to to the source or mirror,e.g. deleting messages or publishing, do not reflect on the origin stream.

Stream sources

A stream defining Sources is a generalized replication mechanism and allows for sourcing data from one or more streams concurrently as well as allowing direct write/publish by clients. Essentially the source streams and client writes are aggregated into a single interleaved stream. Subject transformation and filtering allow for powerful data distribution architectures.

Mirrors

A mirror can source its messages from exactly one stream and a clients can not directly write to the mirror. Although messages cannot be published to a mirror directly by clients, messages can be deleted on-demand (beyond the retention policy), and consumers have all capabilities available on regular streams.

For details see:

• Source and Mirror

AllowRollup

If enabled, the AllowRollup stream option allows for a published message having a Nats-Rollup header indicating all prior messages should be purged. The scope of the purge is defined by the header value, either all or sub.

The Nats-Rollup: all header will purge all prior messages in the stream. Whereas the sub value will purge all prior messages for a given subject.

A common use case for rollup is for state snapshots, where the message being published has accumulated all the necessary state from the prior messages, relative to the stream or a particular subject.

RePublish

If enabled, the RePublish stream option will result in the server re-publishing messages received into a stream automatically and immediately after a successful write, to a distinct destination subject.

For high scale needs where, currently, a dedicated consumer may add too much overhead, clients can establish a core NATS subscription to the destination subject and receive messages that were appended to the stream in real-time.

The fields for configuring republish include:

• Source - An optional subject pattern which is a subset of the subjects bound to the stream. It defaults to all messages in the stream, e.g. >.

• Destination - The destination subject messages will be re-published to. The source and destination must be a valid subject mapping.

• HeadersOnly - If true, the message data will not be included in the re-published message, only an additional header Nats-Msg-Size indicating the size of the message in bytes.

For each message that is republished, a set of headers are automatically added.

SubjectTransform

If configured, the SubjectTransform will perform a subject transform to matching subjects of messages received by the stream and transform the subject, before storing it in the stream. The transform configuration specifies a Source and Destination field, following the rules of subject transform.

Consider this architecture

While it is an incomplete architecture it does show a number of key points:

• Many related subjects are stored in a Stream

• Consumers can have different modes of operation and receive just subsets of the messages

• Multiple Acknowledgement modes are supported

A new order arrives on ORDERS.received, gets sent to the NEW Consumer who, on success, will create a new message on ORDERS.processed. The ORDERS.processed message again enters the Stream where a DISPATCH Consumer receives it and once processed it will create an ORDERS.completed message which will again enter the Stream. These operations are all pull based meaning they are work queues and can scale horizontally. All require acknowledged delivery ensuring no order is missed.

All messages are delivered to a MONITOR Consumer without any acknowledgement and using Pub/Sub semantics - they are pushed to the monitor.

As messages are acknowledged to the NEW and DISPATCH Consumers, a percentage of them are Sampled and messages indicating redelivery counts, ack delays and more, are delivered to the monitoring system.

Example Configuration

JetStream, the persistence layer of NATS, not only allows for the higher qualities of service and features associated with 'streaming', but it also enables some functionalities not found in messaging systems.

Managing a Key Value store

1. Create a bucket, which corresponds to a stream in the underlying storage. Define KV/Stream limits as appropriate

2. Use the operation below.

Map style operations

You can use KV buckets to perform the typical operations you would expect from an immediately consistent key/value store:

• put: associate a value with a key

• get: retrieve the value associated with a key

• delete: clear any value associated with a key

• purge: clear all the values associated with all keys

• keys: get a copy of all of the keys (with a value or operation associated with it)

Atomic operations used for locking and concurrency control

• create: associate the value with a key only if there is currently no value associated with that key (i.e. compare to null and set)

• update: compare and set (aka compare and swap) the value for a key

Limiting size, TTL etc.

You can set limits for your buckets, such as:

• the maximum size of the bucket

• the maximum size for any single value

• a TTL: how long the store will keep values for

Treating the Key Value store as a message stream

Finally, you can even do things that typically can not be done with a Key/Value Store:

• watch: watch for changes happening for a key, which is similar to subscribing (in the publish/subscribe sense) to the key: the watcher receives updates due to put or delete operations on the key pushed to it in real-time as they happen

• watch all: watch for all the changes happening on all the keys in the bucket

• history: retrieve a history of the values (and delete operations) associated with each key over time (by default the history of buckets is set to 1, meaning that only the latest value/operation is stored)

Notes

Prerequisite: enabling JetStream

If you are running a local nats-server stop it and restart it with JetStream enabled using nats-server -js (if that's not already done)

You can then check that JetStream is enabled by using

If you see the below then JetStream is not enabled

1. Creating a stream

Let's start by creating a stream to capture and store the messages published on the subject "foo".

Enter nats stream add <Stream name> (in the examples below we will name the stream "my_stream"), then enter "foo" as the subject name and hit return to use the defaults for all the other stream attributes:

You can then check the information about the stream you just created:

2. Publish some messages into the stream

Let's now start a publisher

As messages are being published on the subject "foo" they are also captured and stored in the stream, you can check that by using nats stream info my_stream and even look at the messages themselves using nats stream view my_stream or nats stream get my_stream

3. Creating a consumer

Now at this point if you create a 'Core NATS' (i.e. non-streaming) subscriber to listen for messages on the subject 'foo', you will only receive the messages being published after the subscriber was started, this is normal and expected for the basic 'Core NATS' messaging. In order to receive a 'replay' of all the messages contained in the stream (including those that were published in the past) we will now create a 'consumer'

We can administratively create a consumer using the 'nats consumer add ' command, in this example we will name the consumer "pull_consumer", and we will leave the delivery subject to 'nothing' (i.e. just hit return at the prompt) because we are creating a 'pull consumer' and select all for the start policy, you can then just use the defaults and hit return for all the other prompts. The stream the consumer is created on should be the stream 'my_stream' we just created above.

You can check on the status of any consumer at any time using nats consumer info or view the messages in the stream using nats stream view my_stream or nats stream get my_stream, or even remove individual messages from the stream using nats stream rmm

3. Subscribing from the consumer

Now that the consumer has been created and since there are messages in the stream we can now start subscribing to the consumer:

This will print out all the messages in the stream starting with the first message (which was published in the past) and continuing with new messages as they are published until the count is reached.

Note that in this example we are creating a pull consumer with a 'durable' name, this means that the consumer can be shared between as many consuming processes as you want. For example instead of running a single nats consumer next with a count of 1000 messages you could have started two instances of nats consumer each with a message count of 500 and you would see the consumption of the messages from the consumer distributed between those instances of nats

Replaying the messages again

Once you have iterated over all the messages in the stream with the consumer, you can get them again by simply creating a new consumer or by deleting that consumer (nats consumer rm) and re-creating it (nats consumer add).

4. Cleaning up

You can clean up a stream (and release the resources associated with it (e.g. the messages stored in the stream)) using nats stream purge

You can also delete a stream (which will also automatically delete all of the consumers that may be defined on that stream) using nats stream rm

Streams with source and mirror configurations are best managed through a client API. If you intend to create such a configuration from command line with NATS CLI you should use a JSON configuration.

Example stream configuration with two sources

Minimal example

With additional options

Example stream configuration with mirror

Minimal example

With additional options

When a stream is configured with a source or mirror, it will automatically and asynchronously replicate messages from the origin stream.

source or mirror are designed to be robust and will recover from a loss of connection. They are suitable for geographic distribution over high latency and unreliable connections. E.g. even a leaf node starting and connecting intermittently every few days will still receive or send messages over the source/mirror link.

There are several options available when declaring the configuration.

• Name - Name of the origin stream to source messages from.

• StartSeq - An optional start sequence of the origin stream to start mirroring from.

• StartTime - An optional message start time to start mirroring from. Any messages that are equal to or greater than the start time will be included.

• FilterSubject - An optional filter subject which will include only messages that match the subject, typically including a wildcard. Note, this cannot be used with SubjectTransforms.

• Domain - An optional JetStream domain of where the origin stream exists. This is commonly used in a hub cluster and leafnode topology.

The stream using a source or mirror configuration can have its own retention policy, replication, and storage type.

General behavior

• All configurations are made on the receiving side. The stream from which data is sourced and mirrored does not need to be configured. No cleanup is required on the origin side if the receiver disappears.

• A stream can be the origin (source) for multiple streams. This is useful for geographic distribution or for designing "fan out" topologies where data needs to be distributed reliable to a large number (up to millions) of client connections.

• Leaf nodes and leaf node domains are explicitly supported through the API prefix

Source specific

A stream defining Sources is a generalized replication mechanism and allows for sourcing data from one or more streams concurrently. A stream with sources can still act as a regular stream allowing direct write/publish by local clients to the stream. Essentially the source streams and local client writes are aggregated into a single interleaved stream. Combined with subject transformation and filtering sourcing allows to design sophisticated data distribution architectures.

Mirror specific

A mirror can source its messages from exactly one stream and a clients can not directly write to the mirror. Although messages cannot be published to a mirror directly by clients, messages can be deleted on-demand (beyond the retention policy), and consumers have all capabilities available on regular streams.

Expected behavior in edge conditions

• Source and mirror contracts are designed with one-way (geographic) data replication in mind. Neither configuration provides a full synchronization between streams, which would include deletes or replication of other stream attributes.

• The content of the stream from which a source or mirror is drawn needs to be reasonable stable. Quickly deleting messages after publishing them may result in inconsistent replication due to the asynchronous nature of the replication process.

• Sources and Mirror try to be be efficient in replicating messages and are lenient towards the source/mirror origin being unreachable (event for extended periods of time), e.g. when using leaf nodes, which are connected intermittently. For sake of efficiency the recovery interval in case of a disconnect is 10-20s.

• Mirror and source agreements do not create a visible consumer in the origin stream.

WorkQueue retention

Source and mirror work with origin stream with workqueue retention. The source/mirror will act as a consumer removing messages from the origin stream.

The intended usage is for moving messages conveniently from one location to another (e.g. from a leaf node). It does not trivially implement a distributed workqueue (where messages are distributed to multiple streams sourcing from the same origin), although with the help of subject filtering a distributed workqueue can be approximated.

Interest base retention

A consumer is a stateful view of a stream. It acts as an interface for clients to consume a subset of messages stored in a stream and will keep track of which messages were delivered and acknowledged by clients.

While Streams are responsible for storing the published messages, the consumer is responsible for tracking the delivery and acknowledgments. This tracking ensures that if a message is not acknowledged (un-acked or 'nacked'), the consumer will automatically attempt to re-deliver it. JetStream consumers support various acknowledgment types and policies. If a message is not acknowledged within a user-specified number of delivery attempts, an advisory notification is emitted.

Dispatch type - Pull / Push

Consumers can be push-based where messages will be delivered to a specified subject or pull-based which allows clients to request batches of messages on demand. The choice of what kind of consumer to use depends on the use-case.

If there is a need to process messages in an application controlled manner and easily scale horizontally, you would use a 'pull consumer'. A simple client application that wants a replay of messages from a stream sequentially you would use an 'ordered push consumer'. An application that wants to benefit from load balancing or acknowledge messages individually will use a regular push consumer.

Ordered Consumers

Ordered consumers are the convenient default type of push & pull consumers designed for applications that want to efficiently consume a stream for data inspection or analysis.

• Always ephemeral

• No acknowledgements (if gap is detected, consumer is recreated)

• Automatic flow control/pull processing

• Single-threaded dispatching

• No load balancing

Persistence - Durable / Ephemeral

In addition to the choice of being push or pull, a consumer can also be ephemeral or durable. A consumer is considered durable when an explicit name is set on the Durable field when creating the consumer, or when InactiveThreshold is set.

Durables and ephemeral have the same message delivery semantics but an ephemeral consumer will not have persisted state or fault tolerance (server memory only) and will be automatically cleaned up (deleted) after a period of inactivity, when no subscriptions are bound to the consumer.

By default, consumers will have the same replication factor as the stream they consume, and will remain even when there are periods of inactivity (unless InactiveThreshold is set explicitly). Consumers can recover from server and client failure.

Configuration

Below are the set of consumer configuration options that can be defined. The Version column indicates the version of nats-server in which the option was introduced. The Editable column indicates the option can be edited after the consumer is created.

General

AckPolicy

The policy choices include:

• AckExplicit: The default policy. Each individual message must be acknowledged. Recommended for most reliability and functionality.

• AckNone: No acknowledgment needed; the server assumes acknowledgment on delivery.

• AckAll: Acknowledge only the last message received in a series; all previous messages are automatically acknowledged. Will acknowledge all pending messages for all subscribers for Pull Consumer.

If an acknowledgment is required but not received within the AckWait window, the message will be redelivered.

Warning: The server may consider an acknowledgment arriving out of the window. For instance, in a queue situation, if a first process fails to acknowledge within the window and the message has been redelivered to another consumer, the acknowledgment from the first consumer will be considered.

DeliverPolicy

The policy choices include:

• DeliverAll: Default policy. Start receiving from the earliest available message in the stream.

• DeliverLast: Start with the last message added to the stream, or the last message matching the consumer's filter subject if defined.

• DeliverLastPerSubject: Start with the latest message for each filtered subject currently in the stream.

• DeliverNew: Start receiving messages created after the consumer was created.

• DeliverByStartSequence: Start at the first message with the specified sequence number. The consumer must specify OptStartSeq defining the sequence number.

• DeliverByStartTime: Start with messages on or after the specified time. The consumer must specify OptStartTime defining the start time.

MaxAckPending

The MaxAckPending capability provides flow control and applies to both push and pull consumers. For push consumers, MaxAckPending is the only form of flow control. For pull consumers, client-driven message delivery creates implicit one-to-one flow control with subscribers.

For high throughput, set MaxAckPending to a high value. For applications with high latency due to external services, use a lower value and adjust AckWait to avoid re-deliveries.

FilterSubjects

A filter subject provides server-side filtering of messages before delivery to clients.

For example, a stream factory-events with subject factory-events.*.* can have a consumer factory-A with a filter factory-events.A.* to deliver only events for factory A.

A consumer can have a singular FilterSubject or plural FilterSubjects. Multiple filters can be applied, such as [factory-events.A.*, factory-events.B.*] or specific event types [factory-events.*.item_produced, factory-events.*.item_packaged].

Warning: For granular consumer permissions, a single filter uses $JS.API.CONSUMER.CREATE.{stream}.{consumer}.{filter} to restrict users to specific filters. Multiple filters use the general $JS.API.CONSUMER.DURABLE.CREATE.{stream}.{consumer}, which does not include the {filter} token. Use a different strategy for granular permissions.

Pull-specific

Push-specific

NATS is a client/server system in the fact that you have 'NATS client applications' (applications using one of the NATS client libraries) that connect to 'NATS servers' that provide the NATS service. The NATS servers work together to provide a NATS service infrastructure to their client applications.

NATS is extremely flexible and scalable and allows the service infrastructure to be as small as a single process running locally on your local machine and as large as an 'Internet of NATS' of Leaf Nodes, and Leaf Node clusters all interconnected in a secure way over a global shared NATS super-cluster.

Regardless of the size and complexity of the NATS service infrastructure being used, the only configuration needed by the client applications being the location (NATS URLs) of one or more NATS servers and depending on the required security, their credentials.

Note that if your application is written in Golang then you even have the option of embedding the NATS server functionality into the application itself (however you need to then configure your application instances with nats-server configuration information).

The Evolution of your NATS service infrastructure

You will typically start by running a single instance of nats-server on your local development machine, and have your applications connect to it while you do your application development and local testing.

Next you will probably want to start testing and running those applications and servers in a VPC, or a region or in some on-prem location, so you will deploy either single NATS server or clusters of NATS servers in your VPCs/regions/on-prem/etc... locations and in each location have the applications connect their local nats-server or nats-server cluster. You can then connect those local nats-servers or local nats-server clusters together by making them leaf nodes connecting to a 'backbone' cluster or super-cluster, or by connecting them directly together via gateway connections.

If you have very many client applications (i.e. applications deployed on end-user devices all over the Internet, or for example many IoT devices) or many servers in many locations you will then scale your NATS service infrastructure by deploying clusters of NATS servers in multiple locations and multiple cloud providers and VPCs, and connecting those clusters together into a global super-cluster and then devise a scheme to intelligently direct your client applications to the right 'closest' NATS server cluster.

Running your own NATS service infrastructure

You can deploy and run your own NATS service infrastructure of nats-server instances, composed of servers, clusters of servers, super-cluster and leaf node NATS servers.

Virtualization and containerization considerations

JetStream, the persistence layer of NATS, not only allows for the higher qualities of service and features associated with 'streaming', but it also enables some functionalities not found in messaging systems.

One such feature is the Object store functionality, which allows client applications to create buckets (corresponding to streams) that can store a set of files. Files are stored and transmitted in chunks, allowing files of arbitrary size to be transferred safely over the NATS infrastructure.

Note: Object store is not a distributed storage system. All files in a bucket will need to fit on the target file system.

Basic Capabilities

The Object Store implements a chunking mechanism, allowing you to for example store and retrieve files (i.e. the object) of any size by associating them with a path or file name as the key.

• add a bucket to hold the files.

• put Add a file to the bucket

• get Retrieve the file and store it to a designated location

• del Delete a file

Advanced Capabilities

• watch Subscribe to changes in the bucket. Will receive notifications on successful put and del operations.

The Key/Value Store is a JetStream feature, so we need to verify it is enabled by

which may return

In this case, you should enable JetStream.

Prerequisite: enabling JetStream

If you are running a local nats-server stop it and restart it with JetStream enabled using nats-server -js (if that's not already done)

You can then check that JetStream is enabled by using

Creating a KV bucket

A 'KV bucket' is like a stream; you need to create it before using it, as in nats kv add <KV Bucket Name>:

Storing a value

Now that we have a bucket, we can assign, or 'put', a value to a specific key:

which should return the key's value Value1

Getting a value

We can fetch, or 'get', the value for a key "Key1":

Deleting a value

You can always delete a key and its value by using

It is harmless to delete a non-existent key (check this!!).

Atomic operations

K/V Stores can also be used in concurrent design patterns, such as semaphores, by using atomic 'create' and 'update' operations.

E.g. a client wanting exclusive use of a file can lock it by creating a key, whose value is the file name, with create and deleting this key after completing use of that file. A client can increase the reslience against failure by using a timeout for the bucket containing this key. The client can use update with a revision number to keep the bucket alive.

Updates can also be used for more fine-grained concurrency control, sometimes known as optimistic locking, where multiple clients can try a task, but only one can successfully complete it.

Create (aka exclusive locking)

Create a lock/semaphore with the create operation.

Only one create can succeed. First come, first serve. All concurrent attempts will result in an error until the key is deleted

Update with CAS (aka optimistic locking)

We can also atomically update, sometimes known as a CAS (compare and swap) operation, a key with an additional parameter revision

A second attempt with the same revision 13, will fail

Watching a K/V Store

An unusual functionality of a K/V Store is being able to 'watch' a bucket, or a specific key in that bucket, and receive real-time updates to changes in the store.

For the example above, run nats kv watch my-kv. This will start a watcher on the bucket we have just created earlier. By default, the KV bucket has a history size of one, and so it only remembers the last change. In our case, the watcher should see a delete of the value associated with the key "Key1":

If we now concurrently change the value of 'my-kv' by

The watcher will see that change:

Cleaning up

When you are finished using a bucket, you can delete the bucket, and its resources, by using the rm operator:

If you are running a local nats-server stop it and restart it with JetStream enabled using nats-server -js (if that's not already done)

You can then check that JetStream is enabled by using

Which should output something like:

If you see the below instead then JetStream is not enabled

Creating an Object Store bucket

Just like you need to create streams before you can use them you need to first create an Object Store bucket

which outputs

Putting a file in the bucket

Putting a file in the bucket by providing a name

By default the full file path is used as a key. Provide the key explicitly (e.g. a relative path ) with --name

Listing the objects in a bucket

Getting an object from the bucket

Getting an object from the bucket with a specific output path

By default, the file will be stored relative to the local path under its name (not the full path). To specify an output path use --output

Removing an object from the bucket

Getting information about the bucket

Watching for changes to a bucket

Sealing a bucket

You can seal a bucket, meaning that no further changes are allowed on that bucket

Deleting a bucket

Using nats object rm myobjbucket will delete the bucket and all the files stored in it.

Message headers are used in a variety of JetStream contexts, such de-duplication, auto-purging of messages, metadata from republished messages, and more.

Publish

Headers that can be set by a client when a message being published.

RePublish

Headers set messages that are republished.

Sources

Headers that are implicitly added to messages sourced from other streams.

Headers-only

Headers added to messages when the consumer is configured to be "headers only" omitting the body.

Mirror

Headers used for internal flow-control messages for a mirror.