NATS Server Clustering

NATS supports running each server in clustered mode. You can cluster servers together for high volume messaging systems and resiliency and high availability.

NATS servers achieve this by gossiping about and connecting to, all of the servers they know, thus dynamically forming a full mesh. Once clients connect or re-connect to a particular server, they are informed about current cluster members. Because of this behavior, a cluster can grow, shrink and self heal. The full mesh does not necessarily have to be explicitly configured either.

Note that NATS clustered servers have a forwarding limit of one hop. This means that each nats-server instance will only forward messages that it has received from a client to the immediately adjacent nats-server instances to which it has routes. Messages received from a route will only be distributed to local clients.

For the cluster to successfully form a full mesh and NATS to function as intended and described throughout the documentation - temporary errors permitting - it is necessary that servers can connect to each other and that clients can connect to each server in the cluster.

Cluster URLs

In addition to a port to listen for clients, nats-server listens on a "cluster" URL (the -cluster option). Additional nats-server servers can then add that URL to their -routes argument to join the cluster. These options can also be specified in a config file, but only the command-line version is shown in this overview for simplicity.

Running a Simple Cluster

Here is a simple cluster running on the same machine:

Server A - the 'seed server'

nats-server -p 4222 -cluster nats://localhost:4248 --cluster_name test-cluster

Server B

nats-server -p 5222 -cluster nats://localhost:5248 -routes nats://localhost:4248 --cluster_name test-cluster

Check the output of the server for the selected client and route ports.

Server C

nats-server -p 6222 -cluster nats://localhost:6248 -routes nats://localhost:4248 --cluster_name test-cluster

Check the output of the server for the selected client and route ports.

Each server has a client and cluster port specified. Servers with the routes option establish a route to the seed server. Because the clustering protocol gossips members of the cluster, all servers are able to discover other server in the cluster. When a server is discovered, the discovering server will automatically attempt to connect to it in order to form a full mesh. Typically only one instance of the server will run per machine, so you can reuse the client port (4222) and the cluster port (4248), and simply the route to the host/port of the seed server.

Similarly, clients connecting to any server in the cluster will discover other servers in the cluster. If the connection to the server is interrupted, the client will attempt to connect to all other known servers.

There is no explicit configuration for seed server. They simply serve as the starting point for server discovery by other members of the cluster as well as clients. As such these are the servers that clients have in their list of connect urls and cluster members have in their list of routes. They reduce configuration as not every server needs to be in these lists. But the ability for other server and clients to successfully connect depends on seed server running. If multiple seed server are used, they make use of the routes option as well, so they can establish routes to one another.

Command Line Options

The following cluster options are supported:

--routes [rurl-1, rurl-2]     Routes to solicit and connect
--cluster nats://host:port    Cluster URL for solicited routes

When a NATS server routes to a specified URL, it will advertise its own cluster URL to all other servers in the route effectively creating a routing mesh to all other servers.

Note: when using the -routes option, you must also specify a -cluster option.

Clustering can also be configured using the server config file.

Three Server Cluster Example

The following example demonstrates how to run a cluster of 3 servers on the same host. We will start with the seed server and use the -D command line parameter to produce debug information.

nats-server -p 4222 -cluster nats://localhost:4248 -D

Alternatively, you could use a configuration file, let's call it seed.conf, with a content similar to this:

# Cluster Seed Node

listen: 127.0.0.1:4222
http: 8222

cluster {
  listen: 127.0.0.1:4248
}

And start the server like this:

nats-server -config ./seed.conf -D

This will produce an output similar to:

[83329] 2020/02/12 16:04:52.369039 [INF] Starting nats-server version 2.1.4
[83329] 2020/02/12 16:04:52.369130 [DBG] Go build version go1.13.6
[83329] 2020/02/12 16:04:52.369133 [INF] Git commit [not set]
[83329] 2020/02/12 16:04:52.369360 [INF] Starting http monitor on 127.0.0.1:8222
[83329] 2020/02/12 16:04:52.369436 [INF] Listening for client connections on 127.0.0.1:4222
[83329] 2020/02/12 16:04:52.369441 [INF] Server id is NDSGCS74MG5ZUMBOVWOUJ5S3HIOW
[83329] 2020/02/12 16:04:52.369443 [INF] Server is ready
[83329] 2020/02/12 16:04:52.369534 [INF] Listening for route connections on 127.0.0.1:4248

It is also possible to specify the hostname and port independently. At the minimum, the port is required. If you leave the hostname off it will bind to all the interfaces ('0.0.0.0').

cluster {
  host: 127.0.0.1
  port: 4248
}

Now let's start two more servers, each one connecting to the seed server.

nats-server -p 5222 -cluster nats://localhost:5248 -routes nats://localhost:4248 -D

When running on the same host, we need to pick different ports for the client connections -p, and for the port used to accept other routes -cluster. Note that -routes points to the -cluster address of the seed server (localhost:4248).

Here is the log produced. See how it connects and registers a route to the seed server (...GzM).

[83330] 2020/02/12 16:05:09.661047 [INF] Starting nats-server version 2.1.4
[83330] 2020/02/12 16:05:09.661123 [DBG] Go build version go1.13.6
[83330] 2020/02/12 16:05:09.661125 [INF] Git commit [not set]
[83330] 2020/02/12 16:05:09.661341 [INF] Listening for client connections on 0.0.0.0:5222
[83330] 2020/02/12 16:05:09.661347 [INF] Server id is NAABC2CKRVPZBIECMLZZA6L3PK
[83330] 2020/02/12 16:05:09.661349 [INF] Server is ready
[83330] 2020/02/12 16:05:09.662429 [INF] Listening for route connections on localhost:5248
[83330] 2020/02/12 16:05:09.662676 [DBG] Trying to connect to route on localhost:4248
[83330] 2020/02/12 16:05:09.663308 [DBG] 127.0.0.1:4248 - rid:1 - Route connect msg sent
[83330] 2020/02/12 16:05:09.663370 [INF] 127.0.0.1:4248 - rid:1 - Route connection created
[83330] 2020/02/12 16:05:09.663537 [DBG] 127.0.0.1:4248 - rid:1 - Registering remote route "NDSGCS74MG5ZUMBOVWOUJ5S3HIOW"
[83330] 2020/02/12 16:05:09.663549 [DBG] 127.0.0.1:4248 - rid:1 - Sent local subscriptions to route

From the seed's server log, we see that the route is indeed accepted:

[83329] 2020/02/12 16:05:09.663386 [INF] 127.0.0.1:62941 - rid:1 - Route connection created
[83329] 2020/02/12 16:05:09.663665 [DBG] 127.0.0.1:62941 - rid:1 - Registering remote route "NAABC2CKRVPZBIECMLZZA6L3PK"
[83329] 2020/02/12 16:05:09.663681 [DBG] 127.0.0.1:62941 - rid:1 - Sent local subscriptions to route

Finally, let's start the third server:

nats-server -p 6222 -cluster nats://localhost:6248 -routes nats://localhost:4248 -D

Again, notice that we use a different client port and cluster address, but still point to the same seed server at the address nats://localhost:4248:

[83331] 2020/02/12 16:05:12.838022 [INF] Listening for client connections on 0.0.0.0:6222
[83331] 2020/02/12 16:05:12.838029 [INF] Server id is NBE7SLUDLFIMHS2U6347N3DQEJ
[83331] 2020/02/12 16:05:12.838031 [INF] Server is ready
...
[83331] 2020/02/12 16:05:12.839203 [INF] Listening for route connections on localhost:6248
[83331] 2020/02/12 16:05:12.839453 [DBG] Trying to connect to route on localhost:4248
[83331] 2020/02/12 16:05:12.840112 [DBG] 127.0.0.1:4248 - rid:1 - Route connect msg sent
[83331] 2020/02/12 16:05:12.840198 [INF] 127.0.0.1:4248 - rid:1 - Route connection created
[83331] 2020/02/12 16:05:12.840324 [DBG] 127.0.0.1:4248 - rid:1 - Registering remote route "NDSGCS74MG5ZUMBOVWOUJ5S3HIOW"
[83331] 2020/02/12 16:05:12.840342 [DBG] 127.0.0.1:4248 - rid:1 - Sent local subscriptions to route
[83331] 2020/02/12 16:05:12.840717 [INF] 127.0.0.1:62946 - rid:2 - Route connection created
[83331] 2020/02/12 16:05:12.840906 [DBG] 127.0.0.1:62946 - rid:2 - Registering remote route "NAABC2CKRVPZBIECMLZZA6L3PK"
[83331] 2020/02/12 16:05:12.840915 [DBG] 127.0.0.1:62946 - rid:2 - Sent local subscriptions to route

First a route is created to the seed server (...IOW) and after that, a route from ...3PK - which is the ID of the second server - is accepted.

The log from the seed server shows that it accepted the route from the third server:

[83329] 2020/02/12 16:05:12.840111 [INF] 127.0.0.1:62945 - rid:2 - Route connection created
[83329] 2020/02/12 16:05:12.840350 [DBG] 127.0.0.1:62945 - rid:2 - Registering remote route "NBE7SLUDLFIMHS2U6347N3DQEJ"
[83329] 2020/02/12 16:05:12.840363 [DBG] 127.0.0.1:62945 - rid:2 - Sent local subscriptions to route

And the log from the second server shows that it connected to the third.

[83330] 2020/02/12 16:05:12.840529 [DBG] Trying to connect to route on 127.0.0.1:6248
[83330] 2020/02/12 16:05:12.840684 [DBG] 127.0.0.1:6248 - rid:2 - Route connect msg sent
[83330] 2020/02/12 16:05:12.840695 [INF] 127.0.0.1:6248 - rid:2 - Route connection created
[83330] 2020/02/12 16:05:12.840814 [DBG] 127.0.0.1:6248 - rid:2 - Registering remote route "NBE7SLUDLFIMHS2U6347N3DQEJ"
[83330] 2020/02/12 16:05:12.840827 [DBG] 127.0.0.1:6248 - rid:2 - Sent local subscriptions to route

At this point, there is a full mesh cluster of NATS servers.

Testing the Cluster

Now, the following should work: make a subscription to the first server (port 4222). Then publish to each server (ports 4222, 5222, 6222). You should be able to receive messages without problems.

Testing server A

nats sub -s "nats://127.0.0.1:4222" hello &
nats pub -s "nats://127.0.0.1:4222" hello world_4222

23:34:45 Subscribing on hello
23:34:45 Published 10 bytes to "hello"

[#1] Received on "hello"
world_4222

Testing server B

nats pub -s "nats://127.0.0.1:5222" hello world_5222

[#2] Received on "hello"
23:36:09 Published 10 bytes to "hello"
world_5222

Testing server C

nats pub -s "nats://127.0.0.1:6222" hello world_6222

23:38:40 Published 10 bytes to "hello"
[#3] Received on "hello"
world_6222

Testing using seed (i.e. A, B and C) server URLs

nats pub -s "nats://127.0.0.1:4222,nats://127.0.0.1:5222,nats://127.0.0.1:6222" hello whole_world

[#4] Received on "hello"
23:39:16 Published 11 bytes to "hello"
whole_world

The cluster configuration map has the following configuration options:

cluster {
  name: example

  # host/port for inbound route connections from other server
  listen: localhost:4244

  # Authorization for route connections
  # Other server can connect if they supply the credentials listed here
  # This server will connect to discovered routes using this user
  authorization {
    user: route_user
    password: pwd
    timeout: 0.5
  }

  # This server establishes routes with these server.
  # This server solicits new routes and Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above.
  routes = [
    nats://route_user:pwd@127.0.0.1:4245
    nats://route_user:pwd@127.0.0.1:4246
  ]
}

Introduced in NATS v2.10.0

Connection pooling

Before the v2.10.0 release, two servers in a cluster had only one connection to transmit all messages for all accounts, which could lead to slow down and increased memory usage when data was not transmitted fast enough.

The v2.10.0 release introduces the ability to have multiple route connections between two servers. By default, without any configuration change, clustering two v2.10.0+ servers together will create 3 route connections. This can of course be configured to a different value by explicitly configuring the pool size.

cluster {
  pool_size: 3
}

Pool size requirements

Each of those connections will handle a specific subset of the accounts, and the assignment of an account to a specific connection index in the pool is the same in any server in the cluster. It is required that each server in the cluster have the same pool size value, otherwise, clustering will fail to be established with an error similar to:

[ERR] 127.0.0.1:6222 - rid:6 - Mismatch route pool size: 3 vs 4

Handling loss of connection

In the event that a given connection of the pool breaks, automatic reconnection occurs as usual. However, while the disconnection is happening, traffic for accounts handled by that connection is stopped (that is, traffic is not routed through other connections), the same way that it was when there was a single route connection.

Configuration reload

Note that although the cluster's pool_size configuration parameter can be changed through a configuration reload, connections between servers will likely break since there will be a mismatch between servers. It is possible though to do a rolling reload by setting the same value on all servers. Client and other connections (including dedicated account routes - see next section) will not be closed.

Monitoring

When monitoring is enabled, you will see that the /routez page now has more connections than before, which is expected.

Disabling connection pooling

It is possible to disable route connection pooling by setting the pool_size configuration parameter to the value -1. When that is the case, a server will behave as a server pre-v2.10.0 and create a single route connection between its peers.

Note that in that mode, no accounts list can be defined (see "Accounts Pinning" below).

Account pinning

In addition to connection pooling, the release v2.10.0 has introduced the ability to configure a list of accounts that will have a dedicated route connection.

cluster {
  accounts: [acc1, acc2]
}

Note that by default, the server will create a dedicated route for the system account (no specific configuration is needed).

Having a dedicated route improves performance and reduces latency, but another benefit is that since the route is dedicated to an account, the account name does not need to be added to the message route protocols. Since account names can be quite long, this reduces the number of bytes that need to be transmitted in all other cases.

Handling loss of connection

In the event that an account route connection breaks, automatic reconnection occurs as usual. However, while the disconnection is happening, traffic for this account is stopped.

Configuration reload

The accounts list can be modified and a configuration signal be sent to the server. All servers need to have the same list, however, it is possible to perform a rolling configuration reload.

For instance, adding an account to the list of server A and issuing a configuration reload will not produce an error, even though the other server in the cluster does not have that account in the list yet. A dedicated connection will not yet be established, but traffic for this account in the pooled connection currently handling it will stop. When the configuration reload happens on the other server, a dedicated connection will then be established and this account’s traffic will resume.

When removing an account from the list and issuing a configuration reload, the connection for this account will be closed, and traffic for this account will stop. Other server(s) that still have this account configured with a dedicated connection will fail to reconnect. When they are also sent the configuration reload (with updated accounts configuration), the account traffic will now be handled by a connection in the pool.

Note that configuration reload of changes in the accounts list do not affect existing pool connections, and therefore should not affect traffic for other accounts.

Monitoring

When monitoring is enabled, the route connection's information now has a new field called account that displays the name of the account this route is for.

Disabling connection pooling

As indicated in the connection pooling section, if pool_size is set to -1, the accounts list cannot be configured nor would be in use.

Connecting to an older server

Although it is recommended that all servers part of the same cluster be at the same version number, a v2.10.0 server will be able to connect to an older server and in that case create a single route to that server. This allows for an easy deployment of a v2.10.0 release into an existing cluster running an older release.

Compression

Release v2.10.0 introduces the ability to configure compression between servers having route connections. The current compression algorithm used is S2, an extension of Snappy. By default, routes do not use compression and it needs to be explicitly enabled.

cluster {
  compression: {
    mode: accept
  }
}

Compression Modes

There are several modes of compression and there is no requirement to have the same mode between routed servers.

• off - Explicitly disables compression for any route between the server and a peer.

• accept (default) - Does not initiate compression, but will accept the compression mode of the peer it connects to.

• s2_fast - Applies compression, but optimizes for speed over compression ratio.

• s2_better - Applies compression, providing a balance of speed and compression ratio.

• s2_best - Applies compression and optimizes for compression ratio over speed.

• s2_auto - Choose the appropriate s2_* mode relative to the round-trip time (RTT) measured between the server and the peer. See rtt_thresholds below.

Round-trip time thresholds

When s2_auto compression is used, it relies on a rtt_thresholds option, which is a list of three latency thresholds that dictate increasing or decreasing the compression mode.

cluster {
  compression: {
    mode: s2_auto
    rtt_thresholds: [10ms, 50ms, 100ms]
  }
}

The default rtt_thresholds value is [10ms, 50ms, 100ms]. The way to read this is that if the RTT is under 10ms, no compression is applied. Once 10ms is reached, s2_fast is applied and so on with the remaining two thresholds for s2_better and s2_best.

Configuration reload

The compression configuration can be changed through configuration reload. If the value is changed from off to anything else, then connections are closed and recreated, same as if the compression mode was set to something and disabled by setting it to off.

For all other compression modes, the mode is changed dynamically without a need to close the route connections.

Monitoring

When monitoring is enabled, the route connection's information now has a new field called compression that displays the current compression mode. It can be off or any other mode described above. Note that s2_auto is not displayed, instead, what will be displayed is the current mode, say s2_best or s2_uncompressed.

If connected to an older server, the compression field will display not supported.

Connecting to an older server

It is possible to have a v2.10.0+ server, with a compression mode configured, connect to an older server that does not support compression. The connection will simply not use compression.

Clustering in JetStream is required for a highly available and scalable system. Behind clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up JetStream clusters.

RAFT

JetStream uses a NATS optimized RAFT algorithm for clustering. Typically RAFT generates a lot of traffic, but the NATS server optimizes this by combining the data plane for replicating messages with the messages RAFT would normally use to ensure consensus. Each server participating requires an unique server_name (only applies within the same domain).

RAFT Groups

The RAFT groups include API handlers, streams, consumers, and an internal algorithm designates which servers handle which streams and consumers.

The RAFT algorithm has a few requirements:

• A log to persist state

• A quorum for consensus

The Quorum

In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two JetStream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth.

RAFT Groups

Meta Group - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement.

Stream Group - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages.

Consumer Group - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have their own group.

Cluster Size

Generally, we recommend 3 or 5 JetStream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are JetStream enabled you would want two servers in one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating.

Mixing JetStream enabled servers with standard NATS servers

This is possible and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the JetStream enabled servers would handle JetStream traffic.

Configuration

To configure JetStream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any JetStream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each JetStream node must specify a server name and cluster name.

Below are explicitly listed server configuration for a three-node cluster across three machines, n1-c1, n2-c1, and n3-c1.

Server password configuration

A user and password under the system account ($SYS) should be configured. The following configuration uses a bcrypted password: a very long s3cr3t! password.

Server 1 (host_a)

server_name=n1-c1
listen=4222

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "$2a$11$DRh4C0KNbNnD8K/hb/buWe1zPxEHrLEiDmuq1Mi0rRJiH/W25Qidm"
      }
    ]
  }
}

jetstream {
   store_dir=/nats/storage
}

cluster {
  name: C1
  listen: 0.0.0.0:6222
  routes: [
    nats://host_b:6222
    nats://host_c:6222
  ]
}

Server 2 (host_b)

server_name=n2-c1
listen=4222

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "$2a$11$DRh4C0KNbNnD8K/hb/buWe1zPxEHrLEiDmuq1Mi0rRJiH/W25Qidm"
      }
    ]
  }
}

jetstream {
   store_dir=/nats/storage
}

cluster {
  name: C1
  listen: 0.0.0.0:6222
  routes: [
    nats://host_a:6222
    nats://host_c:6222
  ]
}

Server 3 (host_c)

server_name=n3-c1
listen=4222

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "$2a$11$DRh4C0KNbNnD8K/hb/buWe1zPxEHrLEiDmuq1Mi0rRJiH/W25Qidm"
      }
    ]
  }
}

jetstream {
   store_dir=/nats/storage
}

cluster {
  name: C1
  listen: 0.0.0.0:6222
  routes: [
    nats://host_a:6222
    nats://host_b:6222
  ]
}

Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SSD, and launch each server. After two servers are running you'll be ready to use JetStream.

Once a JetStream cluster is operating interactions with the CLI and with nats CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1.

Account Level

Within an account there are operations and reports that show where users data is placed and which allow them some basic interactions with the RAFT system.

Creating clustered streams

When adding a stream using the nats CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored on multiple nodes in your cluster using the RAFT protocol as above.

nats str add ORDERS --replicas 3

Example output extract:

....
Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
....
Configuration:
....
             Replicas: 3

Cluster Information:

                 Name: C1
               Leader: n1-c1
              Replica: n4-c1, current, seen 0.07s ago
              Replica: n3-c1, current, seen 0.07s ago

Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using nats stream info.

Here we have a stream in the NATS cluster C1, its current leader is a node n1-c1 and it has 2 followers - n4-c1 and n3-c1.

The current indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently.

The replica count cannot be edited once configured.

Viewing Stream Placement and Stats

Users can get overall statistics about their streams and also where these streams are placed:

nats stream report

Obtaining Stream stats
+----------+-----------+----------+--------+---------+------+---------+----------------------+
| Stream   | Consumers | Messages | Bytes  | Storage | Lost | Deleted | Cluster              |
+----------+-----------+----------+--------+---------+------+---------+----------------------+
| ORDERS   | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
| ORDERS_3 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
| ORDERS_4 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
| ORDERS_5 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1, n2-c1, n3-c1* |
| ORDERS_2 | 4         | 1,385    | 13 MiB | File    | 0    | 1       | n1-c1, n2-c1, n3-c1* |
| ORDERS_0 | 4         | 1,561    | 14 MiB | File    | 0    | 0       | n1-c1, n2-c1*, n3-c1 |
+----------+-----------+----------+--------+---------+------+---------+----------------------+

Forcing Stream and Consumer leader election

Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time.

Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election.

nats stream cluster step-down ORDERS

14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group
14:32:18 New leader elected "n4-c1"

Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
...
Cluster Information:

                 Name: c1
               Leader: n4-c1
              Replica: n1-c1, current, seen 0.12s ago
              Replica: n3-c1, current, seen 0.12s ago

The same is true for consumers, nats consumer cluster step-down ORDERS NEW.

System Level

Systems users can view state of the Meta Group - but not individual Stream or Consumers.

Viewing the cluster state

We have a high level report of cluster state:

nats server report jetstream --user admin --password s3cr3t!

+--------------------------------------------------------------------------------------------------+
|                                        JetStream Summary                                         |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
| Server | Cluster | Streams | Consumers | Messages | Bytes  | Memory | File   | API Req | API Err |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
| n3-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 1       | 0       |
| n3-c1  | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 3       | 0       |
| n2-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 3       | 0       |
| n1-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 14      | 2       |
| n2-c1  | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 15      | 0       |
| n1-c1* | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 31      | 0       |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
|        |         | 18      | 72        | 8,838    | 80 MiB | 0 B    | 80 MiB | 67      | 2       |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
+---------------------------------------------------+
|            RAFT Meta Group Information            |
+-------+--------+---------+---------+--------+-----+
| Name  | Leader | Current | Offline | Active | Lag |
+-------+--------+---------+---------+--------+-----+
| n1-c1 | yes    | true    | false   | 0.00s  | 0   |
| n1-c2 |        | true    | false   | 0.05s  | 0   |
| n2-c1 |        | true    | false   | 0.05s  | 0   |
| n2-c2 |        | true    | false   | 0.05s  | 0   |
| n3-c1 |        | true    | false   | 0.05s  | 0   |
| n3-c2 |        | true    | false   | 0.05s  | 0   |
+-------+--------+---------+---------+--------+-----+

This is a full cluster wide report, the report can be limited to a specific account using --account.

Here we see the distribution of streams, messages, api calls etc by across 2 super clusters and an overview of the RAFT meta group.

In the Meta Group report the server n2-c1 is not current and has not been seen for 9 seconds, it's also behind by 2 raft operations.

This report is built using raw data that can be obtained from the monitor port on the /jsz url, or over nats using:

nats server req jetstream --user admin --password s3cr3t! --help

usage: nats server request jetstream [<flags>] [<wait>]

Show JetStream details

Flags:
  -h, --help                    Show context-sensitive help (also try --help-long and --help-man).
      --version                 Show application version.
  -s, --server=NATS_URL         NATS server urls
      --user=NATS_USER          Username or Token
      --password=NATS_PASSWORD  Password
      --creds=NATS_CREDS        User credentials
      --nkey=NATS_NKEY          User NKEY
      --tlscert=NATS_CERT       TLS public certificate
      --tlskey=NATS_KEY         TLS private key
      --tlsca=NATS_CA           TLS certificate authority chain
      --timeout=NATS_TIMEOUT    Time to wait on responses from NATS
      --js-api-prefix=PREFIX    Subject prefix for access to JetStream API
      --js-event-prefix=PREFIX  Subject prefix for access to JetStream Advisories
      --js-domain=DOMAIN        JetStream domain to access
      --context=CONTEXT         Configuration context
      --trace                   Trace API interactions
      --limit=2048              Limit the responses to a certain amount of records
      --offset=0                Start at a certain record
      --name=NAME               Limit to servers matching a server name
      --host=HOST               Limit to servers matching a server host name
      --cluster=CLUSTER         Limit to servers matching a cluster name
      --tags=TAGS ...           Limit to servers with these configured tags
      --account=ACCOUNT         Show statistics scoped to a specific account
      --accounts                Include details about accounts
      --streams                 Include details about Streams
      --consumer                Include details about Consumers
      --config                  Include details about configuration
      --leader                  Request a response from the Meta-group leader only
      --all                     Include accounts, streams, consumers and configuration

Args:
  [<wait>]  Wait for a certain number of responses

nats server req jetstream --user admin --password s3cr3t! --leader

This will produce a wealth of raw information about the current state of your cluster - here requesting it from the leader only.

Forcing Meta Group leader election

Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a SYSTEM user.

nats server raft step-down --user admin --password s3cr3t!

17:44:24 Current leader: n2-c2
17:44:24 New leader: n1-c2

Evicting a peer

Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down.

There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers.

After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream.

nats stream cluster peer-remove ORDERS

? Select a Peer n4-c1
14:38:50 Removing peer "n4-c1"
14:38:50 Requested removal of peer "n4-c1"

At this point the stream and all consumers will have removed n4-c1 from the group, they will all start new peer selection and data replication.

$ nats stream info ORDERS

....
Cluster Information:

                 Name: c1
               Leader: n3-c1
              Replica: n1-c1, current, seen 0.02s ago
              Replica: n2-c1, outdated, seen 0.42s ago

We can see a new replica was picked, the stream is back to replication level of 3 and n4-c1 is not active any more in this Stream or any of its Consumers.

Diagnosing problems in NATS JetStream clusters requires:

• knowledge of JetStream concepts

• knowledge of the NATS Command Line Interface (CLI)

The following tips and commands (while not an exhaustive list) can be useful when diagnosing problems in NATS JetStream clusters:

Troubleshooting tips

1. Look at nats-server logs. By default, only warning and error logs are produced, but debug and trace logs can be turned on from the command line using -D and -DV, respectively. Alternatively, enabling debug or trace in the server config.

2. Make sure that in the NATS JetStream configuration, at least one system user is configured in this section: { $SYS { users } }.

nats account commands

Basic nats server commands

nats server report commands

nats server request commands

nats server cluster commands

Experimental commands

Further troubleshooting references

• Testing your setup