Managing a NATS Server is simple, typical lifecycle operations include:

• Using the nats CLI tool to check server cluster connectivity and latencies, as well as get account information, and manage and interact with streams (and other NATS applications). Try the following examples to learn about the most common ways to use nats.

  • nats cheat

  • nats cheat server

  • nats stream --help to monitor, manage and interact with streams

  • nats consumer --help to monitor, manage stream consumers

  • nats context --help if you need to switch between servers, clusters or user credentials

• Using the CLI tool when using JWT based authentication and authorization, to create, revoke operators, accounts, and user (i.e. client applications) JWTs and keys.

• to a server to reload a configuration or rotate log files

• a server (or cluster)

• Understanding

• Monitoring the server via:

  • The monitoring and tools like

  • By subscribing to

• Gracefully shut down a server with

Server Metrics

JetStream has a /jsz HTTP endpoint and advisories available.

Advisories

JetStream publishes a number of advisories that can inform operations about the health and the state of the Streams. These advisories are published to normal NATS subjects below $JS.EVENT.ADVISORY.> and one can store these advisories in JetStream Streams if desired.

The command nats event --js-advisory can view all these events on your console. The Golang package can consume and render these events and have data types for each of these events.

All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the nats schema show <schema kind> command.

Dashboards

Check out .

Repeat this procedure for all nodes of the cluster, one at a time:

1. Stop the server by putting it into Lame Duck Mode

2. Replace the binary or Docker image with the new version.

3. Restart the server.

4. Wait until the /healthz endpoint returns HTTP 200 OK before moving on to the next cluster node.

Downgrading

Although the NATS server goes through rigorous testing for each release, there may be a need to revert to the previous version if you observe a performance regression for your workload. The support policy for the server is the current release as well as one patch version release prior. For example, if the latest is 2.8.4, a downgrade to 2.8.3 is supported. Downgrades to earlier versions may work, but is not recommended.

Fortunately, the downgrade path is the same as the upgrade path as noted above. Swap the binary and do a rolling restart.

Once the server is running it's time to use the management tool. Please refer to the installation section in the readme.

nats --help
nats cheat

We'll walk through the above scenario and introduce features of the CLI and of JetStream as we recreate the setup above.

Throughout this example, we'll show other commands like nats pub and nats sub to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS.

We'll touch on some additional features but please review the section on the design model to understand all possible permutations.

Stream, Consumer (durable name), and Account names are used in both the subject namespace used by JetStream and the filesystem backing JetStream persistence. This means that when naming streams, consumers, and accounts, names must adhere to subject naming rules as well as being friendly to the file system.

We recommend the following guideline for stream, consumer, and account names:

• Alphanumeric values are recommended.

• Spaces, tabs, period (.), greater than (>

Account Information

JetStream is multi-tenant so you will need to check that your account is enabled for JetStream and is not limited. You can view your limits as follows:

nats account info

Connection Information:
               Client ID: 8
               Client IP: 127.0.0.1
                     RTT: 178.545µs
       Headers Supported: true
         Maximum Payload: 1.0 MiB
           Connected URL: nats://localhost:4222
       Connected Address: 127.0.0.1:4222
     Connected Server ID: NCCOHA6ONXJOGAEZP4WPU4UJ3IQP2VVXEPRKTQCGBCW4IL4YYW4V4KKL
JetStream Account Information:
           Memory: 0 B of 5.7 GiB
          Storage: 0 B of 11 GiB
          Streams: 0 of Unlimited
   Max Consumers: unlimited

In production we recommend that a server is shut down with ​lame duck mode​ as a graceful way to slowly evict clients. With large deployments this mitigates the "thundering herd" situation that will place CPU pressure on servers as TLS enabled clients reconnect.

Server

Lame duck mode is initiated by signaling the server:

nats-server --signal ldm

After entering lame duck mode, the server will stop accepting new connections, wait for a 10 second grace period, then begin to evict clients over a period of time configurable by the lame_duck_duration configuration option. This period defaults to 2 minutes.

Clients

When entering lame duck mode, the server will send a message to clients. Some maintainer supported clients will invoke an optional callback indicating that a server is entering lame duck mode. This is used for cases where an application can benefit from preparing for the short outage between the time it is evicted and automatically reconnected to another server.

Command Line

On Unix systems, the NATS server responds to the following signals:

The nats-server binary can be used to send these signals to running NATS servers using the --signal/-sl flag. It supports the following commands:

Quit the server

Stop the server

Lame duck mode the server

Reopen log file for log rotation

Reload server configuration

Multiple processes

If there are multiple nats-server processes running, or if pgrep isn't available, you must either specify a PID or the absolute path to a PID file:

As of NATS v2.10.0, a glob expression can be used to match one or more process IDs, such as:

Windows

See the section for information on signaling the NATS server on Windows.

In the event of unrecoverable JetStream message persistence on one (or more) server nodes, there are two recovery scenarios:

• Automatic recovery from intact quorum nodes

• Manual recovery from existing stream snapshots (backups)

Automatic Recovery

NATS will create replacement stream replicas automatically under the following conditions:

• Impacted stream is of replica configuration R3 (or greater)

• Remaining intact nodes (stream replicas) meet minimum RAFT quorum: floor(R/2) + 1

• Available node(s) in the stream's cluster for new replica(s)

• Impacted node(s) removed from the stream's domain RAFT Meta group (e.g. nats server cluster peer-remove

Manual Recovery

Snapshots (also known as backups) can pro-actively be made of any stream regardless of replication configuration.

The backup includes (by default):

• Stream configuration and state

• Stream durable consumer configuration and state

• All message payload data including metadata like timestamps and headers

Backup

The nats stream backup CLI command is used to create snapshots of a stream and its durable consumers.

Output

During a backup operation, the stream is placed in a status where it's configuration cannot change and no data will be evicted based on stream retention policies.

Restore

An existing backup (as above) can be restored to the same or a new NATS server (or cluster) using the nats stream restore command.

Output

The /data/js-backup/ORDERS.tgz file can also be extracted into the data dir of a stopped NATS Server.

Progress using the terminal bar can be disabled using --no-progress, it will then issue log lines instead.

Interactive CLI

In environments where the nats CLI is used interactively to configure the server you do not have a desired state to recreate the server from. This is not the ideal way to administer the server, we recommend Configuration Management, but many will use this approach.

Here you can back up the configuration into a directory from where you can recover the configuration later. The data for File backed stores can also be backed up.

This backs up Stream and Consumer configuration.

During the same process the data can also be backed up by passing --data, this will create files like /data/js-backup/stream_ORDERS.tgz.

Later the data can be restored, for Streams we support editing the Stream configuration in place to match what was in the backup.

The nats account restore tool does not support restoring data, the same process using nats stream restore, as outlined earlier, can be used which will also restore Stream and Consumer configurations and state.

To support resiliency and high availability, NATS provides built-in mechanisms to automatically prune the registered listener interest graph that is used to keep track of subscribers, including slow consumers and lazy listeners. NATS automatically handles a slow consumer. If a client is not processing messages quick enough, the NATS server cuts it off. To support scaling, NATS provides for auto-pruning of client connections. If a subscriber does not respond to ping requests from the server within the ping-pong interval, the client is cut off (disconnected). The client will need to have reconnect logic to reconnect with the server.

In core NATS, consumers that cannot keep up are handled differently from many other messaging systems: NATS favors the approach of protecting the system as a whole over accommodating a particular consumer to ensure message delivery.

What is a slow consumer?

A slow consumer is a subscriber that cannot keep up with the message flow delivered from the NATS server. This is a common case in distributed systems because it is often easier to generate data than it is to process it. When consumers cannot process data fast enough, back pressure is applied to the rest of the system. NATS has mechanisms to reduce this back pressure.

NATS identifies slow consumers in the client or the server, providing notification through registered callbacks, log messages, and statistics in the server's monitoring endpoints.

What happens to slow consumers?

When detected at the client, the application is notified and messages are dropped to allow the consumer to continue and reduce potential back pressure. When detected in the server, the server will disconnect the connection with the slow consumer to protect itself and the integrity of the messaging system.

Slow consumers identified in the client

A on a local connection and notify the application through use of the asynchronous error callback. It is better to catch a slow consumer locally in the client rather than to allow the server to detect this condition. This example demonstrates how to define and register an asynchronous error handler that will handle slow consumer errors.

With this example code and default settings, a slow consumer error would generate output something like this:

Note that if you are using a synchronous subscriber, Subscription.NextMsg(timeout time.Duration) will also return an error indicating there was a slow consumer and messages have been dropped.

Slow consumers identified by the server

When a client does not process messages fast enough, the server will buffer messages in the outbound connection to the client. When this happens and the server cannot write data fast enough to the client, in order to protect itself, it will designate a subscriber as a "slow consumer" and may drop the associated connection.

When the server initiates a slow consumer error, you'll see the following in the server output:

The server will also keep count of the number of slow consumer errors encountered, available through the monitoring varz endpoint in the slow_consumers field.

Handling slow consumers

Apart from using or optimizing your consuming application, there are a few options available: scale, meter, or tune NATS to your environment.

Scaling with queue subscribers

This is ideal if you do not rely on message order. Ensure your NATS subscription belongs to a , then scale as required by creating more instances of your service or application. This is a great approach for microservices - each instance of your microservice will receive a portion of the messages to process, and simply add more instances of your service to scale. No code changes, configuration changes, or downtime whatsoever.

Create a subject namespace that can scale

You can distribute work further through the subject namespace, with some forethought in design. This approach is useful if you need to preserve message order. The general idea is to publish to a deep subject namespace, and consume with wildcard subscriptions while giving yourself room to expand and distribute work in the future.

For a simple example, if you have a service that receives telemetry data from IoT devices located throughout a city, you can publish to a subject namespace like Sensors.North, Sensors.South, Sensors.East and Sensors.West. Initially, you'll subscribe to Sensors.> to process everything in one consumer. As your enterprise grows and data rates exceed what one consumer can handle, you can replace your single consumer with four consuming applications to subscribe to each subject representing a smaller segment of your data. Note that your publishing applications remain untouched.

Meter the publisher

A less favorable option may be to meter the publisher. There are several ways to do this varying from simply slowing down your publisher to a more complex approach periodically issuing a blocking request-reply to match subscriber rates.

Tune NATS through configuration

The NATS server can be tuned to determine how much data can be buffered before a consumer is considered slow, and some officially supported clients allow buffer sizes to be adjusted. Decreasing buffer sizes will let you identify slow consumers more quickly. Increasing buffer sizes is not typically recommended unless you are handling temporary bursts of data. Often, increasing buffer capacity will only postpone slow consumer problems.

Server Configuration

The NATS server has a write deadline it uses to write to a connection. When this write deadline is exceeded, a client is considered to have a slow consumer. If you are encountering slow consumer errors in the server, you can increase the write deadline to buffer more data.

The write_deadline configuration option in the NATS server configuration file will tune this:

Tuning this parameter is ideal when you have bursts of data to accommodate. Be sure you are not just postponing a slow consumer error.

Client Configuration

Most officially supported clients have an internal buffer of pending messages and will notify your application through an asynchronous error callback if a local subscription is not catching up. Receiving an error locally does not necessarily mean that the server will have identified a subscription as a slow consumer.

This buffer can be configured through setting the pending limits after a subscription has been created:

The default subscriber pending message limit is 65536, and the default subscriber pending byte limit is 65536*1024

If the client reaches this internal limit, it will drop messages and continue to process new messages. This is aligned with NATS at most once delivery. It is up to your application to detect the missing messages and recover from this condition.

Supported since NATS server version 2.3.0

TPM is supported on Windows since NATS Server version 2.11.0

The NATS server can be configured to encrypt message blocks which includes message headers and payloads. Other metadata files are encrypted as well, such as the stream metadata file and consumer metadata files.

Two choices of ciphers are currently supported:

• chachapoly -

• aes -

Enabling encryption is done through the jetstream on the server.

It is recommended to provide the encryption key through an environment variable at runtime, such as $JS_KEY, so it will not be persisted in a file.

The variable can be exported in the environment or passed when the server starts up.

TPM (Windows only)

Changing encryption settings

Enabling with existing data

Enabling encryption on a server with existing data is supported. Do note that existing unencrypted message blocks will not be re-encrypted, however any new blocks that are stored will be encrypted going forward.

If it is desired to encrypt the existing blocks, the stream can be backed up and restored (which decrypts on backup and then re-encrypts when restoring it).

Disabling or changing the key

If encryption was enabled on the server and the server is restarted with a different key or disabled all together, the server will fail to decrypt messages when attempting to load them from the store. If this happens, you’ll see log messages like the following:

Note, that this will impact JetStream functionality, but the server will still support core NATS functionality.

Changing the cipher

It is possible to change the cipher, however the same key must be used. The server will properly encrypt new message blocks with the new cipher and decrypt existing messages blocks with the existing cipher.

Performance considerations

Performance considerations: As expected, encryption is likely to decrease performance, but by how much is hard to define. In some performance tests on a MacbookPro 2.8 GHz Intel Core i7 with SSD, we have observed as little as 1% decrease to more than 30%. In addition to CPU cycles required for encryption, the encrypted files may be larger, which results in more data being stored or read.

The first step is to set up storage for our ORDERS related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year.

Creating

You can get prompted interactively for missing information as above, or do it all on one command. Pressing ? in the CLI will help you map prompts to CLI options:

Additionally one can store the configuration in a JSON file, the format of this is the same as $ nats str info ORDERS -j | jq .config

Messages are read or consumed from the Stream by Consumers. We support pull and push-based Consumers and the example scenario has both, let's walk through that.

Creating Pull-Based Consumers

The NEW and DISPATCH Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability.

Pull-based Consumers are created the same as push-based Consumers, you just don't specify a delivery target.

Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store.

Here we have 2 main streams - ORDERS and RETURNS - these streams are clustered across 3 nodes. These Streams have short retention periods and are memory based.

We create a ARCHIVE stream that has 2 sources set, the ARCHIVE will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly.

Finally, we create a REPORT stream mirrored from ARCHIVE that is not clustered and retains data for a month. The REPORT Stream does not listen for any incoming messages, it can only consume data from ARCHIVE.

Mirrors

A mirror copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A mirror does not listen on a subject for any data to be added. A mirror can filter by subject and the Start Sequence and Start Time can be set. A stream can only have 1 mirror and if it is a mirror it cannot also have any source.

Sources

A source is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result.

A Stream with sources may also listen on subjects, but could have no listening subject. When using the nats CLI to create sourced streams use --subjects to supply subjects to listen on.

A source can have Start Time or Start Sequence and can filter by a subject.

Configuration

The ORDERS and RETURNS streams as normal, I will not show how to create them.

We now add the ARCHIVE:

And we add the REPORT:

When configured we'll see some additional information in a nats stream info output:

Output extract

Here the Lag is how far behind we were reported as being last time we saw a message.

We can confirm all our setup using a nats stream report:

We then create some data in both ORDERS and RETURNS:

We can now see from a Stream Report that the data has been replicated:

Here we also pass the --dot replication.dot argument that writes a GraphViz format map of the replication setup.

If you are using the JWT model of authentication to secure your NATS infrastructure or implementing an Auth callout service, you can administer authentication and authorization without having to change the servers' configuration files.

You can use the nsc CLI tool to manage identities. Identities take the form of nkeys. Nkeys are a public-key signature system based on Ed25519 for the NATS ecosystem. The nkey identities are associated with NATS configuration in the form of a JSON Web Token (JWT). The JWT is digitally signed by the private key of an issuer forming a chain of trust. The nsc tool creates and manages these identities and allows you to deploy them to a JWT account server, which in turn makes the configurations available to nats-servers.

You can also use nk CLI tool and library to manage keys.

Creating, updating and managing JWTs programmatically

You can create, update and delete accounts and users programmatically using the following libraries:

• Golang: see and .

• Java: see and

Integrating with your existing authentication/authorization system

You can integrate NATS with your existing authentication/authorization system or create your own custom authentication using the .

Examples

See under "Authentication and Authorization" for JWT and Auth callout server implementation examples.

User JWTs

Account JWTs

Golang example from https://natsbyexample.com/examples/auth/nkeys-jwts/go

Notes

You can see the key (and any signing keys) of your operator using nsc list keys --show-seeds, you should use a 'signing key' to create the account JWTs (as singing keys can be revoked/rotated easily)

To delete accounts use the "$SYS.REQ.CLAIMS.DELETE" (see ) and make sure to enable JWT deletion in your nats-server resolver (config allow_delete: true in the resolver stanza of the server configuration).

The system is just like any other account, the only difference is that it is listed as system account in the operator's JWT (and the server config).

Monitoring NATS

To monitor the NATS messaging system, nats-server provides a lightweight HTTP server on a dedicated monitoring port. The monitoring server provides several endpoints, providing statistics and other information about the following:

• General Server Information (/varz)

All endpoints return a JSON object.

Note that info from these monitoring endpoints is also available through

The NATS monitoring endpoints support and , making it easy to create single page monitoring web applications. Part of the NATS ecosystem is a tool called that visualizes data from these endpoints on the command line.

Enabling monitoring

Monitoring can be enabled in or as a server . The conventional port is 8222.

As server configuration:

As a command-line option:

Once the server is running using one of the two methods, go to to browse the available endpoints detailed below.

Alternatively, if you have System account enabled, monitoring endpoints available as

Monitoring Endpoints

General Information (/varz)

The /varz endpoint returns general information about the server state and configuration.

Arguments

N/A

Example

Response

Connection Information (/connz)

The /connz endpoint reports more detailed information on current and recently closed connections. It uses a paging mechanism which defaults to 1024 connections.

Arguments

The server will default to holding the last 10,000 closed connections.

Sort Options

Examples

Get up to 1024 connections:

Control limit and offset: .

Get closed connection information: .

You can also report detailed subscription information on a per connection basis using subs=1. For example: .

Response

Route Information (/routez)

The /routez endpoint reports information on active routes for a cluster. Routes are expected to be low, so there is no paging mechanism with this endpoint.

Arguments

As noted above, the routez endpoint does support the subs argument from the /connz endpoint. For example:

Example

• Get route information:

Response

Gateway Information (/gatewayz)

The /gatewayz endpoint reports information about gateways used to create a NATS supercluster. Like routes, the number of gateways are expected to be low, so there is no paging mechanism with this endpoint.

Arguments

Examples

• Retrieve Gateway Information:

Response

Leaf Node Information (/leafz)

The /leafz endpoint reports detailed information about the leaf node connections.

Arguments

As noted above, the leafz endpoint does support the subs argument from the /connz endpoint. For example:

Example

• Get leaf nodes information:

Response

Subscription Routing Information (/subsz)

The /subsz endpoint reports detailed information about the current subscriptions and the routing data structure. It is not normally used.

Arguments

Example

• Get subscription routing information:

Response

Account Information (/accountz)

The /accountz endpoint reports information on a server's active accounts. The default behavior is to return a list of all accounts known to the server.

Example

• Get list of all accounts:

• Get details for specific account $G:

Response

Default behavior:

Retrieve specific account:

Account Statistics (/accstatz)

The /accstatz endpoint reports per-account statistics such as the number of connections, messages/bytes in/out, etc.

Arguments

Examples

• Accounts with active connections -

• Include ones without any connections (in this case $SYS)-

Response

JetStream Information (/jsz)

The /jsz endpoint reports more detailed information on JetStream. For accounts, it uses a paging mechanism that defaults to 1024 connections.

Note: If you're in a clustered environment, it is recommended to retrieve the information from the stream's leader in order to get the most accurate and up-to-date data.

Arguments

Examples

Get basic JetStream information:

Request accounts and control limit and offset: .

You can also report detailed consumer information on a per connection basis using consumer=true. For example: .

Response

Health (/healthz)

The /healthz endpoint returns OK if the server is able to accept connections.

Arguments

Example

• Default -

• Expect JetStream -

Response

Creating Monitoring Applications

NATS monitoring endpoints support and . You can easily create single page web applications for monitoring. To do this you simply pass the callback query parameter to any endpoint.

For example:

Here is a JQuery example implementation:

Monitoring Tools

In addition to writing custom monitoring tools, you can monitor nats-server in Prometheus. The allows you to configure the metrics you want to observe and store in Prometheus, and there are Grafana dashboards available for you to visualize the server metrics.

See the for more details.

When investigating and debugging a performance issue with the NATS Server (i.e. unexpectedly high CPU or RAM utilisation), it may be necessary for you to collect and provide profiles from your deployment for troublshooting. These profiles are crucial to understand where CPU time and memory are being spent.

Note that profiling is an advanced operation for development purposes only. Server operators should use the monitoring port instead for monitoring day-to-day runtime statistics.

Via the NATS CLI

The NATS CLI can request profiles from the NATS Server when connected to the system account only. Profiles will be written out to the current working directory by default as files, which can then either be sent onwards or inspected using go tool pprof.

Memory profile

The --name, --tags and --cluster selectors can be used either individually or combined in order to request profiles from specific servers. Memory profiles are returned instantly. Examples include:

CPU profile

The --name, --tags and --cluster selectors can be used either individually or combined in order to request profiles from specific servers. The --timeout option can also be provided as a means of specifying how long the CPU profile should run for. The default is 5 seconds. Examples include:

Via the Profiling Port

The NATS Server can expose a HTTP pprof profiling port, although it must be enabled by setting the prof_port in your NATS Server configuration file. Note that the profiling port is not authenticated and should not be exposed to clients, to the internet etc. For example, to enable the profiling port on TCP/65432:

Note that this option does not support , so the server must be restarted for the config change to take effect.

Once the profiling port has been enabled, you can download profiles as per the following sections. These profiles can be inspected using .

To see all available profiles, open

Memory profile

http://localhost:65432/debug/pprof/allocs

This endpoint will return instantly.

For example, to download an allocation profile from NATS running on the same machine:

The profile will be saved into mem.prof.

CPU profile

http://localhost:65432/debug/pprof/profile?seconds=30

This endpoint will block for the specified duration and then return. You can specify a different duration by adjusting ?seconds= in the URL if you want to sample a shorter or longer period of time.

For example, to download a CPU profile from NATS running on the same machine with a 30 second window:

The profile will be saved into cpu.prof.

This document provides a step by step deep dive into JWT usage within NATS. Starting with related concepts, it will introduce JWTs and how they can be used in NATS. This will NOT list every JWT/nsc option, but will focus on the important options and concepts.

• Concepts

  • What are Accounts?

To exercise listed examples please have the following installed:

• nats-server:

• nats (cli):

• nk (cli & library):

• nsc (cli):

To install nats-server, nats, nk, nsc:

To practice the examples below:

Save the configuration below in a file named say server.conf, and start nats server via:

So that the server started. Later when we change the configuration we can do a reload like this:

Concepts

What are Accounts?

Accounts are the NATS isolation context.

Messages published in one account won't be received in another.

Listen for any message on account a:

Publish a message from account b:

Note that you do not see this message received by your subscriber.

Now publish a messages from account a:

This time the message is received by the subscriber:

The above example shows no message flow between user a associated with account A and user b in account B. Messages are delivered only within the same account. That is, unless you explicitly define it.

Below is a similar example, this time with messages crossing explicit account boundaries.

Modify server.conf and run nats-server --signal reload

Subscribe to everything as user 'a'

Publish on 'foo' as user 'b':

This time the message is received by the subscriber:

Accounts are a lot more powerful than what has been demonstrated here. Take a look at the complete documentation of and the associated with them. All of this is in a plain NATS config file. (Copy the above config and try it using this command: nats-server -c <filename>) In order to make any changes, every participating nats-server config file in the same security domain has to change. This configuration is typically controlled by one organization or the administrator.

Key Takeaways

• Accounts are isolated from each other.

• One can selectively combine accounts,

• Need to modify a config file to add/remove/modify accounts and users,

• The config file can be applied to take effect via nats-server --signal reload

What are NKEYs?

NKEYs are decorated, Base32 encoded, CRC16 check-summed, keys.

Ed25519 is:

• a public key signature system. (can sign and verify signatures)

• resistant to side channel attacks (no conditional jumps in algorithm)

NATS server can be configured with public NKEYs as user (identities). When a client connects the nats-server sends a challenge for the client to sign in order to prove it is in possession of the corresponding private key. The nats-server then verifies the signed challenge. Unlike with a password based scheme, the secret never left the client.

To assist with knowing what type of key one is looking at, in config or logs, the keys are decorated as follows:

• Public Keys, have a one byte prefix: O, A, U for various types, O for operator, A for account, and U meaning user.

• Private Keys, have a two byte prefix SO, SA

NKEYs are generated as follows:

To view the key:

Create another key:

View the key:

Replacing the user/password with NKEY in account config example:

Simple example:

Subscribe with nats -s nats://localhost:4222 sub --nkey=a.nk ">"

Publish a message using nats -s nats://localhost:4222 pub --nkey=b.nk foo nkey the subscriber should receive it.

When the nats-server was started with -V tracing, you can see the signature in the CONNECT message (formatting added manually):

On connect, clients are instantly sent the nonce to sign as part of the INFO message (formatting added manually). Since telnet will not authenticate, the server closes the connection after hitting the timeout.

Key Takeaways

• NKEYS are a secure way to authenticate clients,

• Private keys are never accessed or stored by the NATS server,

• The public key still needs to be configured in NATS server.

JSON Web Tokens (JWT)

Motivation for JWT

In a large organization the centralized configuration approach can lead to less flexibility and more resistance to change when controlled by one entity. Alternatively, instead of operating one infrastructure, it can be deployed more often (say per team) thus making import/export relationships harder as they have to bridge separate systems. In order to make accounts truly powerful, they should ideally be configured separately from the infrastructure, only constrained by limits. This is similar for user. An account contains the user but this relationship could be a reference as well, such that alterations to user do not alter the account. Users of the same account should be able to connect from anywhere in the same infrastructure and be able to exchange messages as long as they are in the same authentication domain.

Key Takeaways

• JWT splits a nats-server configuration into separate artifacts manageable by different entities.

• Management of Accounts, Configuration, and Users are separated.

• Accounts do NOT correspond to infrastructure, they correspond to teams or applications.

• Connect to any cluster in the same infrastructure and be able to communicate with all other users in your account.

Decentralized Authentication/Authorization using JWT

Account and User creation managed as separate artifacts in a decentralized fashion using NKEYs. Relying upon a hierarchical chain of trust between three distinct NKEYs and associated roles:

1. Operator: corresponds to operator of a set of NATS servers in the same authentication domain (entire topology, crossing gateways and leaf nodes),

2. Account: corresponds to the set of a single account's configuration,

3. User: corresponds to one user's configuration.

Each NKEY is referenced, together with additional configuration, in a JWT document. Each JWT has a subject field and its value is the public portion of an NKEY and serves as identity. Names exist in JWT but as of now are only used by tooling, nats-server does not read this value. The referenced NKEY's role determines the JWT content:

1. Operator JWTs contain server applicable throughout all operated NATS servers,

2. Account JWTs contain Account specific such as exports, imports, limits, and default user permissions,

3. User JWTs contain user specific such as permissions and limits.

In addition, JWTs can contain settings related to their decentralized nature, such as expiration/revocation/signing. At no point do JWTs contain the private portion of an NKEY, only signatures that can be verified with public NKEY. JWT content can be viewed as public, although it's content may reveal which subjects/limits/permissions exist.

Key Takeaways

• JWTs are hierarchically organized in operator, account and user,

• They carry corresponding configuration and the configuration has been dedicated to the decentralized nature of NATS JWT usage.

NATS JWT Hierarchy

Decentralized Chain of Trust

A nats-server is configured to trust an operator. Meaning, the Operator JWT is part of its server configuration and requires a restart or nats-server --signal reload once the configuration changed. It is also configured with a way to obtain account JWT in one of three ways (explained below).

Clients provide a User JWT when connecting. An Account JWT is not used by clients talking to a nats-server. The clients also possess the private NKEY corresponding to the JWT identity, so that they can prove their identity as described .

The issuer field of the User JWT identifies the Account, and the nats-server then independently obtains the current Account JWT from its configured source. The server can then verify that signature on the User JWT was issued by an NKEY of the claimed Account, and in turn that the Account has an issuer of the Operator and that an NKEY of the Operator signed the Account JWT. The entire three-level hierarchy is verified.

Obtain an Account JWT

To obtain an Account JWT, the nats-server is configured with one of three types. Which one to pick depends upon your needs:

• : Very few or very static accounts

  • You are comfortable changing the server config if the operator or any accounts changed,

  • You can generate a user programmatically using NKEYs and a JWT library (more about that later),

JWT nats-resolver is recommended to use in production environment. With JWT nats-resolver, you can manage huge accounts and users without server reloading. But before adopting JWT nat-resolver, make sure you understand correctly how it works. You can make use of static account settings(probably with NKEYs) and memory-resolver as the necessary steps forwarding JWT nats-resolver fully understanding.

JWT and Chain of Trust Verification

Each JWT document has a subject(sub) it represents. This is the public identity NKEY represented by the JWT document. JWT documents contain an issued at (iat) time of signing. This time is in seconds since Unix epoch. It is also used to determine which of two JWTs for the same subject is more recent. Furthermore JWT documents have an issuer, this may be an (identity) NKEY or a dedicated signing NKEY of an item one level above it in the trust hierarchy. A key is a signing key if it is listed as such in the JWT (above). Signing NKEYs adhere to same NKEY roles and are additional keys that unlike identity NKEY may change over time. In the hierarchy, signing keys can only be used to sign JWT for the role right below them. User JWTs have no signing keys for this reason. To modify one role's set of signing keys, the identity NKEY needs to be used.

Each JWT is signed as below:

If a JWT is valid, the JWT above it is validated as well. If all of them are valid, the chain of trust between them is tested top down as follows:

This is a conceptual view. While all these checks happen, the results of earlier evaluations might be cached: if the Operator/Account is trusted already and the JWT did not change since, then there is no reason to re-evaluate.

Below are examples of decoded JWT (iss == issuer, sub == subject, iat == issuedAt):

nsc describe account -n demo-test --json:

nsc describe user -a demo-test -n alpha --json:

Obtain a User JWT - Client Connect

When a client connects, the steps below have to succeed. The following nats-server configuration is used (for ease of understanding, we are using url-resolver):

1. Client connects and the nats-server responds with INFO () and a containing nonce.

   For ease of use, the NATS CLI uses a creds file that is the concatenation of JWT and private user identity/NKEY.

2. The Client responds with a CONNECT message (formatting added manually), containing a JWT and signed nonce. (output copied from nats-server started with -V)

Key Takeaways

• JWTs are secure,

• JWTs carry configuration appropriate to their role as Operator/Accounts/User,

• JWTs provide a basis for operating one single NATS infrastructure which serves separate, yet optionally connected, entities,

• Account resolvers are a way to obtain unknown Account JWTs,

Deployment Models Enabled by Chain of Trust

Depending on which entity has access to private Operator/Account identity or signing NKEYs, different deployment models are enabled. When picking one, it is important to pick the simplest deployment model that enables what you need it to do. Everything beyond just results in unnecessary configuration and steps.

1. Centralized config: one (set of) user(s) has access to all private operator and account NKEYs,

   Administrators operating the shared infrastructure call all the shots

2. Decentralized config (with multiple nsc environments, explained later):

   1. Administrator/Operator(s) have access to private operator NKEYs to sign accounts. By signing or not signing an account JWT, Administrators can enforce constraints (such as limits).

Signing keys can not only be used by individuals in one or more nsc environments, but also by programs facilitating and libraries. This allows the implementation of sign-up services.

• Account signing key enabled on the fly:

  • user generation (explained later),

  • export activation generation (explained later).

• Operator signing key enables on the fly account generation.

Key Takeaways

• JWTs and the associated chain of trust allows for centralized, decentralized, or self-service account configuration,

• It is important to pick the deployment model that fits your needs, NOT the most complicated one,

• Distributing Operator/Account JWT NKEYs between Administrators and teams enables these deployment models,

Accounts Re-visited

A deeper understanding of accounts will help you to best setup NATS JWT based security.

• What entity do accounts correspond to:

  Our official suggestion is to scope accounts by application/service offered.

  This is very fine grained and will require some configuration.

  This is why some users gravitate to accounts per team. One account for all Applications of a team.

  It is possible to start out with less granular accounts and as applications grow in importance or scale become more fine grained.

• Compared to file based config, Imports and Exports change slightly.

Key Takeaways

• Accounts can be arbitrarily scoped, from Application to Team,

• Account Exports can be restricted by requiring use of activation tokens,

• Receiving a more recent Account JWT causes the nats-server to apply changes and re evaluate existing connections.

Tooling And Key Management

This section will introduce nsc cli to generate and manage operator/accounts/user. Even if you intend to primarily generate your Accounts/User programmatically, in all likelihood, you won't do so for an operator or all accounts. Key Management and how to do so using nsc will also be part of this section.

nsc

Environment

nsc is a tool that uses the and libraries to create NKEYs (if asked to) and all types of JWT. It then stores these artifacts in separate directories.

It keeps track of the last operator/account used. Because of this, commands do not need to reference operator/accounts but can be instructed to do so (recommended for scripts). It supports an interactive mode when -i is provided. When used, referencing accounts/keys is easier.

nsc env will show where NKEYS/JWT are stored and what current defaults are. For testing you may want to switch between nsc environments: Changing the (JWT) store directory: nsc env --store <different folder> Changing the (NKEY) store directory by having an environment variable set: export NKEYS_PATH=<different folder>

Subsequent sections will refer to different environments in context of different . As such you can skip over all mentions for modes not of interest to you. The mixed deployment mode is not mentioned and left as an exercise to the reader.

Backup

NKEYS store directory

Possessing NKEYS gives access to the system. Backups should therefore best be offline and access to them should be severely restricted. In cases where regenerating all/parts of the operator/accounts is not an option, signing NKEYs must be used and identity NKEYs should be archived and then removed from the original store directory, so that in the event of a data breach you can recover without a flag-day change-over of identities. Thus, depending on your scenario, relevant identity NKEYS need to only exist in very secure offline backup(s).

JWT store directory

The store directory contains JWTs for operators, accounts, and users. It does not contain private keys. Therefore it is ok to back these up or even store them in a VCS such as git. But be aware that depending on content, JWT may reveal which permissions/subjects/public-nkeys exist. Knowing the content of a JWT does not grant access; only private keys will. However, organizations may not wish to make those public outright and thus have to make sure that these external systems are secured appropriately.

When restoring an older version, be aware that:

• All changes made since will be lost, specifically revocations may be undone,

• Time has moved on and thus JWTs that were once valid at the time of the backup or commit may be expired now. Thus you may have to be edit them to match your expectations again,

• NKEYS are stored in a separate directory, so not to restore a JWT for which the NKEY has been deleted since:

Names in JWT

JWTs allow you to specify names. But names do NOT represent an identity, they are only used to ease referencing of identities in our tooling. At no point are these names used to reference each other, instead, the public identity NKEY is used for that. The nats-server does not read them at all. Because names do not relate to identity, they may collide. Therefore, when using nsc, these names need to be keep unique.

Setup an Operator

Create/Edit Operator - Operator Environment - All Deployment modes

Create operator with system account and system account user:

The command nsc edit operator [flags] can subsequently be used to modify the operator. For example if you are setting the account server url (used by url-resolver and nats-resolver), nsc does not require them being specified on subsequent commands. nsc edit operator --account-jwt-server-url "nats://localhost:4222"

Note that if you update an operator JWT that is installed on a server you will need to manually update the operator JWT and reload the server While nsc is able to update accounts, it never updates the operator.

We always recommend using signing keys for an operator. Generate one for an operator (-o) and store it in the key directory (--store). The output will display the public portion of the signing key, use that to assign it to the operator (--sk O...). nsc generate nkey -o --store followed by nsc edit operator --sk OB742OV63OE2U55Z7UZHUB2DUVGQHRA5QVR4RZU6NXNOKBKJGKF6WRTZ. To pick the operator signing key for account generation, provide the -i option when doing so.

The system account is the account under which nats-server offers system services as will be explained below in the section. To access these services a user with credentials for the system account is needed. Unless this user is restricted with appropriate permissions, this user is essentially the admin user. They are created like any other user.

For cases where signing keys are generated and immediately added --sk generate will create an NKEY on the fly and assign it as signing NKEY.

Import Operator - Non Operator/Administrator Environment - Decentralized/Self Service Deployment Modes

In order to import an Operator JWT, say the one just created, into a separate nsc environment maintained by a different entity/team, the following has to happen:

1. Obtain the operator JWT using: nsc describe operator --raw and store the output in a file named operator.jwt. The option --raw causes the raw JWT to be emitted,

2. Exchange that file or it's content in any way you like, email works fine (as there are no credentials in the JWT),

3. Import the operator JWT into the second environment with: nsc add operator -u operator.jwt

If the operator should been changed and an update is required, simply repeat these steps but provide the --force option in the last step. This will overwrite the stored operator JWT.

Import Operator - Self Service Deployment Modes

In addition to the , self service deployments require an operator signing key and a system account user. Ideally you would want an operator signing key per entity to distribute a signing key too. Simply repeat the command shown but:

1. Perform nsc generate nkey -o --store in this environment instead,

2. Exchange the public key with the Administrator/Operator via a way that assures you sent the public key and not someone elses,

3. Perform nsc edit operator --sk in the operator environment,

To import the system account user needed for administrative purposes as well as monitoring, perform these steps:

1. Perform nsc describe account -n SYS --raw and store the output in a file named SYS.jwt.

   The option -n specifies the (system) account named SYS.

2. Exchange the file,

As a result of these operations, your operator environment should have these keys and signing keys:

And your account should have the following ones:

Between the two outputs, compare the Stored column.

Alternatively if the administrator is willing to exchange private keys and the exchange can be done securely, a few of these steps fall away. The signing key and system account user can be generated in the administrator/operator environment, omitting --store to avoid unnecessary key copies. Then the public/private signing NKEYS are exchanged together with the system account user as creds file. A creds file can be generated with nsc generate creds -a SYS -n sys-non-op and imported into this environment with nsc import user --file sys.jwt. If the signing key is generated before the operator is imported into this environment, operator update falls away.

Setup an Account

Create/Edit Account - All Environments - All Deployment modes

Create an account as follows:

In case you have multiple operator signing keys -i will prompt you to select one. nsc edit account [flags] can subsequently be used to modify the account. (Edit is also applicable to the system account)

Similar to the operator signing keys are recommended. Generate signing key for an account (-a) and store it in the key directory maintained by nsc (--store) The output will display the public portion of the signing key, use that to assign it to the account (--sk A...) nsc generate nkey -a --store nsc edit account --sk ACW2QC262CIQUX4ACGOOS5XLKSZ2BY2QFBAAOF3VOP7AWAVI37E2OQZX To pick the signing key for user generation, provide the -i option when doing so.

Export Account - Non Operator/Administrator Environment - Decentralized Deployment Modes

In this mode, the created account is self-signed. To have it signed by the operator perform these steps:

1. In this environment export the created account as a JWT like this nsc describe account -n <account name> --raw.

   Store the output in a file named import.jwt.

2. Exchange the file with the Administrator/Operator via a way that assures it is your JWT and not someone elses.

3. In the operator environment import the account with nsc import account --file import.jwt

If the account should be changed and an update is required, simply repeat these steps but provide the --force option during the last step. This will overwrite the stored account JWT.

Export Account - Non Operator/Administrator Environment - Self Service Deployment Modes

This environment is set up with a signing key, thus the account is already . The only step that is needed is to push the Account into the NATS network. However, this depends on your ability to do so. If you have no permissions, you have to perform the same steps as for the . The main difference is that upon import, the account won't be re-signed.

Publicize an Account with Push - Operator Environment/Environment with push permissions - All Deployment Modes

How accounts can be publicized wholly depends on the resolver you are using:

• : The operator has to have all accounts imported and generate a new config,

• : nsc push will send an HTTP POST request to the hosting webserver or nats-account-server,

• nats-resolver: Every environment with a system account user that has permissions to send properly signed account JWT as requests to:

nsc generate config <resolver-type> is an utility that generates the relevant NATS config. Where <resolver-type> can be --mem-resolver or --nats-resolver for the corresponding resolver. Typically the generated output is stored in a file that is then by the NATS config. Every server within the same authentication domain needs to be configured with this configuration.

nats-resolver setup and push example - Operator Environment/Environment with push permissions - All Deployment Modes

This is a quick demo of the nats-based resolver from operator creation to publishing a message. Please be aware that the ability to push only relates to permissions to do so and does not require an account keys. Thus, how accounts to be pushed into the environment (outright creation/import) does not matter. For simplicity, this example uses the operator environment.

Operator Setup:

Inspect the setup:

nsc describe operator:

nsc describe account:

Generate the config and start the server in the background. Also, inspect the generated config. It consists of the mandatory operator, explicitly lists the system account and corresponding JWT:

Add an account and a user for testing:

Without having pushed the account the user can't be used yet.

Doesn't work

Push the account, or push all accounts:

For the NATS resolver, each nats-server that responds will be listed. In case you get fewer responses than you have servers or a server reports an error, it is best practice to resolve this issue and retry. The NATS resolver will gossip missing JWTs in an eventually consistent way. Servers without a copy will perform a lookup from servers that do. If during an initial push only one server responds there is a window where this server goes down or worse, loses its disk. During that time the pushed account is not available to the network at large. Because of this, it is important to make sure that initially, more servers respond than what you are comfortable with losing in such a way at once.

Once the account is pushed, its user can be used:

Setup User

Create/Edit Account - All Environments - All Deployment modes

Create a user as follows: nsc add user --account <account name> --name <user name> -i nsc edit user [flags] can subsequently be used to modify the user. In case you have multiple account signing keys, for either command, -i will prompt you to select one.

In case you generate a user on behalf of another entity that has no nsc environment, you may want to consider not exchanging the NKEY. 1. To do this, have the other entity generate a user NKEY pair like this: nsc generate nkey -u (--store is omitted so as to not have an unnecessary copy of the key) 2. Exchange the public key printed by the command via a way that assures what is used is not someone elses. 3. Create the user by providing (-k) the exchanged public key nsc add user --account SYS -n sys-non-op -k UDJKPL7H6QY4KP4LISNHENU6Z434G6RLDEXL2C64YZXDABNCEOAZ4YY2 in your environment. () 4. If desired edit the user 5. Export the user nsc describe user --account SYS -n sys-non-op --raw from your environment and store the output in a JWT file. 6. Exchange the JWT file 7. Use the JWT file and the NKEY pair in your application.

Automated sign up services - JWT and NKEY libraries

nsc essentially uses the and libraries to generate operator/accounts/users. You can use these libraries to generate the necessary artifacts as well. Because there is only one, generating the operator this way makes little sense. Accounts only if you need them dynamically, say for everyone of your customer. Dynamically provision user and integrate that process with your existing infrastructure, say LDAP, is the most common use case for these libraries.

The next sub sections demonstrate dynamic user generation. The mechanisms shown are applicable to dynamic account creation as well. For dynamic user/account creation, signing keys are highly recommended.

By generating users or accounts dynamically, it becomes YOUR RESPONSIBILITY to properly authenticate incoming requests for these users or accounts

For sign up service issued JWTs, ALWAYS set the SHORTEST POSSIBLE EXPIRATION

Simple user creation

This example illustrates the linear flow of the algorithm and how to use the generated artifacts. In a real world application you would want this algorithm to be distributed over multiple processes. For simplicity of the examples, keys may be hard coded and error handling is omitted.

Create user NKEY

Create user JWT

Inspect the for all available properties/limits/permissions to set. When using an instead, you can dynamically generate accounts. Additional steps are to push the new account as outlined . Depending on your needs, you may want to consider exchanging the accounts identity NKEY in a similar way that the users key is exchanged in the .

Distributed User Creation

As mentioned earlier this example needs to be distributed. This example makes uses of Go channels to encode the same algorithm, uses closures to encapsulate functionalities and Go routines to show which processes exist. Sending and receiving from channels basically illustrates the information flow. To realize this, you can pick HTTP, NATS itself etc... (For simplicity, properly closing channels, error handling, waiting for Go routines to finish is omitted.)

The above example did not need authentication mechanisms, RequestUser possessed the signing key. How you decide to trust an incoming request is completely up to you. Here are a few examples:

• everyone

• username/password

• 3rd party authentication token

In this example, this logic is encapsulated as placeholder closures ObtainAuthorizationToken and IsTokenAuthorized that do nothing.

In this example the users NKEY is generated by the requesting process and the public key is sent to the user sign up service. This way the service does not need to know or send the private key. Furthermore, any process receiving the initial request or even response, may have the user JWT but will not be able to proof possession of private NKEY. However, you can have the provisioning service generate the NKEY pair and respond with the NKEY pair and the user JWT. This is less secure but would enable a less complicated protocol where permissable.

User creation using NATS

The used Go channels to demonstrate data flows. You can use all sorts of protocols to achieve this data flow and pick whatever fits best in your existing infrastructure. However, you can use NATS for this purpose as well.

Straight forward Setup

You can replace send and receive <- with nats publish and subscribe or - for added redundancy on the sign up service - queue subscribe. To do so, you will need connections that enable the sign up service as well as the requestor to exchange messages. The sign up service uses the same connection all of the time and (queue) subscribes to a well known subject. The requestor uses the connection and sends a request to the well known subject. Once the response is received the first connection is closed and the obtained JWT is used to establish a new connection.

Here in lies a chicken and and egg problem. The first connection to request the JWT itself needs credentials. The simplest approach is to set up a different NATS server/cluster that does not require authentication, connect first to cluster 1 and keep requesting the user JWT. Once obtained disconnect from cluster 1 and connect to cluster 2 using the obtained JWT.

Account based Setup

The can be simplified by using accounts instead of separate server/clusters. But a JWT/operator based setup requires JWT authentication. Thus, would be, connections to a different cluster are replaced by connections to the same cluster but different accounts.

• Cluster 1 translates to connections to a signup account.

• Cluster 2 translates to connections to accounts to who's signing keys have been used to sign the user JWT. (This happens the first setup as well)

Connections to the signup accounts use two kinds of credentials. 1. Sign up service(s) use(s) credentials generated for it/them. 2. All requestors use the same JWT and NKEY, neither of which are used for actual authentication.

• That JWT is probably generated using nsc itself.

• Do not use this JWT/NKEY for anything else but contacting the sign up service.

• You want to allow publish only to the well known subject.

Stamping JWT in languages other than Go

The NKEY library does exist or is incorporated in all languages where NATS supports NKEY. The NATS JWT library on the other hand is written in Go. This may not be your language of choice. Other than encoding JWTs, most of what the that library does is maintain the NATS JWT schema. If you use nsc to generate a user as a template for the sign up service and work off of that template you don't need the JWT library. The sample shows how a program that takes an account identity NKEY and account signing NKEY as arguments and outputs a valid creds file.

If .NET is your language of choice, you can also use the package.

System Account

The system account is the account under which nats-server offer services. To use it either the operator JWT has to specify it, which happens during nsc init or when providing --sys to nsc add operator. Alternatively you can encode it in the server configuration by providing system_account with the public NKEY of the account you want to be the system account:

It is NOT recommended to use this account to facilitate communication between your own applications. Its sole purpose is to facilitate communication with and between nats-server.

Event Subjects

Events are published as they happen. But you MUST NOT rely on a particular ordering or due to the possibility of loss, events matching. Say, CONNECT for a client always matching a DISCONNECT for the same client. Your subscriber may simply be disconnected when either event happens. Some messages carry aggregate data and are periodically emitted. There, missing a message for one reason or another is compensated by the next one.

The subject $SYS.SERVER.ACCOUNT.<account-id>.CONNS is still used but it is recommended to subscribe to it's new name $SYS.ACCOUNT.<account-id>.SERVER.CONNS.

Service Subjects

Subjects always available

Each of the subjects can be used without any input. However, for each request type (STATZ, VARZ, SUBSZ, CONNS, ROUTEZ, GATEWAYZ, LEAFZ, ACCOUNTZ, JSZ) a json with type specific options can be sent. Furthermore all subjects allow for filtering by providing these values as json:

Subjects available when using NATS-based resolver

Old Subjects

Leaf Node Connections - Outgoing

It is important to understand that leaf nodes do not multiplex between accounts. Every account that you wish to connect across a leaf node connection needs to be explicitly listed. Thus, the system account is not automatically connected, even if both ends of a leaf node connection use the same system account. For leaf nodes connecting into a cluster or super cluster, the system account needs to be explicitly connected as separate remote to the same URL(s) used for the other account(s). The system account user used by providing credentials can be heavily restricted and for example, only allow publishing on some subjects. This also holds true when you don't use the system account yourself, but indirectly need it for NATS based account resolver or centralized monitoring.

Examples in sub sections below assume that the cluster to connect into is in operator mode.

Non Operator Mode

The outgoing connection is not in Operator mode, thus the system account may differ from the user account. This example shows how to configure a user account and the system account in a leaf node. Credentials files provided have to contain credentials that are valid server/cluster reachable by url. In the example, no accounts are explicitly configured, yet some are referenced. These are the default Account $G and the default system account $SYS

Operator Mode

Outgoing connection is in operator mode as well. This example assumes usage of the same operator and thus system account. However, using a different operator would look almost identical. Only the credentials would be issued by accounts of the other operator.

Connecting Accounts

As shown in , they can be connected via exports and imports. While in configuration files this is straight forward, this becomes a bit more complicated when using JWTs. In part this is due to the addition of new concepts such as public/private/activation tokens that do not make sense in a config based context.

Exports

Add an export with: nsc add export --name <export name> --subject <export subject> This will export a public stream that can be imported by any account. To alter the export to be a service add --service.

To have more control over which account is allowed to import provide the option --private. When doing so only accounts for which you generate tokens can add the matching import. A token can be generated and stored in a file as follows: nsc generate activation --account <account name> --subject <export subject> --output-file <token file> --target-account <account identity public NKEY> The resulting file can then be exchanged with the importer.

Imports

To add an import for a public export use nsc add import --account <account name> --src-account <account identity public NKEY> --remote-subject <subject of export>. To import a service provide the option --service.

To add an import for a private export use nsc add import --account <account name> --token <token file or url> If your nsc environment contains operator and account signing NKEYs, nsc add import -i will generate token to embed on the fly

Import Subjects

Between export/import/activation tokens there are many subjects in use. Their relationship is as follows:

• Import subject is identical to or a subset of the exported subject.

• An activation token's subject is identical to or a subset of the exported subject.

• An activation token's subject is also identical to or a subset of the import subject of the account it is embedded in.

Import Remapping

In order to be independent of subject names chosen by the exporter, importing allows to remap the imported subject. To do so provide the option --remote-subject <subject name> to the import command.

This example will change the subject name the importing account uses locally from the exporter picked subject foo to bar.

Visualizing Export/Import Relationships

NSC can generate diagrams of inter account relationships using: nsc generate diagram component --output-file test.uml The generated file contains a component diagram of all accounts connected through their exports/imports. To turn the file into a .png execute: plantuml -tpng test.uml If the diagram is cut off, increase available memory and image size limit with these options: -Xmx2048m -DPLANTUML_LIMIT_SIZE=16384

Managing Keys

Identity keys are extremely important, so you may want to keep them safe and instead hand out more easily replaceable signing keys to operators. Key importance generally follows the chain of trust with operator keys being more important than account keys. Furthermore, identity keys are more important than signing keys.

There are instances where regenerating a completely new identity key of either type is not a feasible option. For example, you might have an extremely large deployment (IoT) where there is simply too much institutional overhead. In this case, we suggest you securely backup identity keys offline and use exchangeable signing keys instead. Depending on which key was compromised, you may have to exchange signing keys and re-sign all JWTs signed with the compromised key. The compromised key may also have to be revoked.

Whether you simply plan to regenerate new NKEY/JWT or exchange signing NKEYs and re-sign JWTs, in either case, you need to prepare and try this out beforehand and not wait until disaster strikes.

Protect Identity NKEYs

Usage of signing keys for Operator and Account has been shown in the section. This shows how to take an identity key offline. Identity NKEY of the operator/account is the only one allowed to modify the corresponding JWT and thus add/remove signing keys. Thus, initial signing keys are best created and assigned prior to removing the private identity NKEY.

Basic strategy: take them offline & delete in NKEY directory.

Use nsc env to determine your NKEY directory. (Assuming ~/.nkeys for this example) nsc list keys --all lists all keys under your operator and indicates if they are present and if they are signing keys.

Keys for your Operator/Account can be found under <nkyesdir>/keys/O/../<public-nkey>.nk or <nkyesdir>/keys/A/../<public-nkey>.nk. The operator identity NKEY ODMFND7EIJ2MBHNPO2JHCKOZIAY6NAK7OT4V2ZT2C5O6LEB3DPKYV3QL would reside under ~/.nkeys/keys/O/DM/ODMFND7EIJ2MBHNPO2JHCKOZIAY6NAK7OT4V2ZT2C5O6LEB3DPKYV3QL.nk.

Please note that key storage is sharded by the 2nd and 3rd letter in the key

Once these files are backed up and deleted nsc list keys --all will show them as not stored. You can continue as normal, nsc will pick up signing keys instead.

Since you typically distribute user keys or creds files to your applications, there is no need for nsc to hold on to them in the first place. Credentials files are a concatenated user JWT and the corresponding private key, so don't forget to delete that as well.

Key and creds can be found under <nkyesdir>/keys/U/../<public-nkey>.nk and <nkyesdir>/creds/<operator-name>/<account-name>/<user-name>.creds

Reissue Identity NKEYs

If you can easily re-deploy all necessary keys and JWTs, simply by re-generating a new account/user (possibly operator) this will be the simplest solution. The steps necessary are identical to the initial setup, which is why it would be preferred. In fact, for user NKEYs and JWT, generating and distributing new ones to affected applications is the best option.

Even if regeneration of an account or operator is not your first choice, it may be your method of last resort. Below sections outline the steps this would entail.

Operator

In order to reissue an operator identity NKEY use nsc reissue operator. It will generate a new identity NKEY and use it to sign the operator. nsc will also re-sign all accounts signed by the original identity NKEY. Accounts signed by operator signing keys will remain untouched.

The altered operator JWT will have to be deployed to all affected nats-server (one server at a time). Once all nats-server have been restarted with the new operator, push the altered accounts. Depending on your deployment mode you may have to distribute the operator JWT and altered account JWT to all other environments.

This process will be a lot easier when operator signing keys were used throughout and no account will be re-signed because of this. If they were not, you can convert the old identity NKEY into a signing key using nsc reissue operator --convert-to-signing-key. On your own time - you can then remove the then signing NKEY using nsc edit operator --rm-sk O.. and redeploy the operator JWT to all nats-server.

Account

Unlike with the operator, account identity NKEYs can not be changed as easily. User JWT explicitly reference the account identity NKEY such that the nats-server can download them via a resolver. This complicates reissuing these kind of NKEYs, which is why we strongly suggest sticking to signing keys.

The basic approach is to:

1. generate a new account with similar settings - including signing NKEYs,

2. re-sign all users that used to be signed by the old identity NKEY,

3. push the account and,

4. deploy the new user JWT to all programs running inside the account.

When signing keys were used, the account identity NKEY would only be needed to self-sign the account JWT exchange with an administrators/operators environment.

Revocations

JWTs for user, activations and accounts can be explicitly revoked. Furthermore, signing keys can be removed, thus invalidating all JWTs signed by the removed NKEY.

User

To revoke all JWTs for a user in a account issue nsc revocations add-user --account <account name> --name <user name>.

With the argument --at you can specify a time different than now. Use nsc revocations list-users --account <account name> to inspect the result or nsc revocations delete-user --account <account name> --name <user name> to remove the revocation.

Please note that the revocation created only applies to JWTs issued before the time listed. Users created or updated after revocation will be valid as they are outside of the revocation time. Also, please be aware that adding a revocation will modify the account and therefore has to be pushed in order to publicize the revocation.

Activations

To revoke all activations of the export, identified by --account and --subject (--stream if the export is a stream), issued for a given Account identity NKEY use:

Use nsc revocations list-activations --account SYS to inspect the result or:

to remove the revocation.

Please note the revocation created only applies to JWTs issued before the time listed. Activations created or edited after, will be valid as they are outside of the revocation time. Also be aware that adding a revocation will modify the account and therefore has to be pushed in order to publicize the revocation.

Accounts

Account identity NKEYS can not be revoked like user or activations. Instead lock out all users by setting the connection count to 0 using nsc edit account --name <account name> --conns 0 and pushing the change using nsc push --all.

Alternatively you can also remove the account using nsc delete account --name and keep it from found by the account resolver. How to do this depends on your resolver type:

• :

  Remove the JWT from the configuration field resolver_preload and restart all nats-server

• :

  Manually delete the JWT from the nats-account-server store directory.

Signing keys

Accounts, Activations, and Users can be revoked in bulk by removing the respective signing key.

Remove an operator signing key: nsc edit operator --rm-sk <signing key> As a modification of the operator, in order to take effect, all dependent installations as well as nats-server will need this new version of the operator JWT.

Remove an account signing key: nsc edit account --name <account name> --rm-sk <signing key>. In order to take effect, a modification of an account needs to be pushed: nsc push --all.