Administration
Once a JetStream cluster is operating interactions with the CLI and with
nats CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1.Account Level
Within an account there are operations and reports that show where users data is placed and which allow them some basic interactions with the RAFT system.
Creating clustered streams
When adding a stream using the
nats CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored on multiple nodes in your cluster using the RAFT protocol as above.1
nats str add ORDERS --replicas 3
Copied!
Example output extract:
1
....
2
Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
3
....
4
Configuration:
5
....
6
Replicas: 3
7
​
8
Cluster Information:
9
​
10
Name: C1
11
Leader: n1-c1
12
Replica: n4-c1, current, seen 0.07s ago
13
Replica: n3-c1, current, seen 0.07s ago
Copied!
Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using
nats stream info.Here we have a stream in the NATS cluster
C1, its current leader is a node n1-c1 and it has 2 followers - n4-c1 and n3-c1.The
current indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently.The replica count cannot be edited once configured.
Viewing Stream Placement and Stats
Users can get overall statistics about their streams and also where these streams are placed:
1
nats stream report
Copied!
1
Obtaining Stream stats
2
+----------+-----------+----------+--------+---------+------+---------+----------------------+
3
| Stream | Consumers | Messages | Bytes | Storage | Lost | Deleted | Cluster |
4
+----------+-----------+----------+--------+---------+------+---------+----------------------+
5
| ORDERS | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 |
6
| ORDERS_3 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 |
7
| ORDERS_4 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1*, n2-c1, n3-c1 |
8
| ORDERS_5 | 4 | 0 | 0 B | File | 0 | 0 | n1-c1, n2-c1, n3-c1* |
9
| ORDERS_2 | 4 | 1,385 | 13 MiB | File | 0 | 1 | n1-c1, n2-c1, n3-c1* |
10
| ORDERS_0 | 4 | 1,561 | 14 MiB | File | 0 | 0 | n1-c1, n2-c1*, n3-c1 |
11
+----------+-----------+----------+--------+---------+------+---------+----------------------+
Copied!
Forcing Stream and Consumer leader election
Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time.
Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election.
1
nats stream cluster step-down ORDERS
Copied!
1
14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group
2
14:32:18 New leader elected "n4-c1"
3
​
4
Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
5
...
6
Cluster Information:
7
​
8
Name: c1
9
Leader: n4-c1
10
Replica: n1-c1, current, seen 0.12s ago
11
Replica: n3-c1, current, seen 0.12s ago
Copied!
The same is true for consumers,
nats consumer cluster step-down ORDERS NEW.System Level
Systems users can view state of the Meta Group - but not individual Stream or Consumers.
Viewing the cluster state
We have a high level report of cluster state:
1
nats server report jetstream --user system
Copied!
1
+--------------------------------------------------------------------------------------------------+
2
| JetStream Summary |
3
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
4
| Server | Cluster | Streams | Consumers | Messages | Bytes | Memory | File | API Req | API Err |
5
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
6
| n3-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 1 | 0 |
7
| n3-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 3 | 0 |
8
| n2-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 3 | 0 |
9
| n1-c2 | c2 | 0 | 0 | 0 | 0 B | 0 B | 0 B | 14 | 2 |
10
| n2-c1 | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 15 | 0 |
11
| n1-c1* | c1 | 6 | 24 | 2,946 | 27 MiB | 0 B | 27 MiB | 31 | 0 |
12
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
13
| | | 18 | 72 | 8,838 | 80 MiB | 0 B | 80 MiB | 67 | 2 |
14
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
15
+---------------------------------------------------+
16
| RAFT Meta Group Information |
17
+-------+--------+---------+---------+--------+-----+
18
| Name | Leader | Current | Offline | Active | Lag |
19
+-------+--------+---------+---------+--------+-----+
20
| n1-c1 | yes | true | false | 0.00s | 0 |
21
| n1-c2 | | true | false | 0.05s | 0 |
22
| n2-c1 | | true | false | 0.05s | 0 |
23
| n2-c2 | | true | false | 0.05s | 0 |
24
| n3-c1 | | true | false | 0.05s | 0 |
25
| n3-c2 | | true | false | 0.05s | 0 |
26
+-------+--------+---------+---------+--------+-----+
Copied!
This is a full cluster wide report, the report can be limited to a specific account using
--account.Here we see the distribution of streams, messages, api calls etc by across 2 super clusters and an overview of the RAFT meta group.
In the Meta Group report the server
n2-c1 is not current and has not been seen for 9 seconds, it's also behind by 2 raft operations.This report is built using raw data that can be obtained from the monitor port on the
/jsz url, or over nats using:1
nats server req jetstream --help
Copied!
1
usage: nats server request jetstream [<flags>] [<wait>]
2
​
3
Show JetStream details
4
​
5
Flags:
6
-h, --help Show context-sensitive help (also try --help-long and --help-man).
7
--version Show application version.
8
-s, --server=NATS_URL NATS server urls
9
--user=NATS_USER Username or Token
10
--password=NATS_PASSWORD Password
11
--creds=NATS_CREDS User credentials
12
--nkey=NATS_NKEY User NKEY
13
--tlscert=NATS_CERT TLS public certificate
14
--tlskey=NATS_KEY TLS private key
15
--tlsca=NATS_CA TLS certificate authority chain
16
--timeout=NATS_TIMEOUT Time to wait on responses from NATS
17
--js-api-prefix=PREFIX Subject prefix for access to JetStream API
18
--js-event-prefix=PREFIX Subject prefix for access to JetStream Advisories
19
--js-domain=DOMAIN JetStream domain to access
20
--context=CONTEXT Configuration context
21
--trace Trace API interactions
22
--limit=2048 Limit the responses to a certain amount of records
23
--offset=0 Start at a certain record
24
--name=NAME Limit to servers matching a server name
25
--host=HOST Limit to servers matching a server host name
26
--cluster=CLUSTER Limit to servers matching a cluster name
27
--tags=TAGS ... Limit to servers with these configured tags
28
--account=ACCOUNT Show statistics scoped to a specific account
29
--accounts Include details about accounts
30
--streams Include details about Streams
31
--consumer Include details about Consumers
32
--config Include details about configuration
33
--leader Request a response from the Meta-group leader only
34
--all Include accounts, streams, consumers and configuration
35
​
36
Args:
37
[<wait>] Wait for a certain number of responses
Copied!
1
nats server req jetstream --leader
Copied!
This will produce a wealth of raw information about the current state of your cluster - here requesting it from the leader only.
Forcing Meta Group leader election
Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a
SYSTEM user.1
nats server raft step-down --user system
Copied!
1
17:44:24 Current leader: n2-c2
2
17:44:24 New leader: n1-c2
Copied!
Evicting a peer
Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function. A 5 node cluster can withstand 2 nodes being down.
There might be a case though where you know a machine will never return, and you want to signal to JetStream that the machine will not return. This will remove it from the Stream in question and all it's Consumers.
After the node is removed the cluster will notice that the replica count is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream.
1
nats stream cluster peer-remove ORDERS
Copied!
1
? Select a Peer n4-c1
2
14:38:50 Removing peer "n4-c1"
3
14:38:50 Requested removal of peer "n4-c1"
Copied!
At this point the stream and all consumers will have removed
n4-c1 from the group, they will all start new peer selection and data replication.1
$ nats stream info ORDERS
Copied!
1
....
2
Cluster Information:
3
​
4
Name: c1
5
Leader: n3-c1
6
Replica: n1-c1, current, seen 0.02s ago
7
Replica: n2-c1, outdated, seen 0.42s ago
Copied!
We can see a new replica was picked, the stream is back to replication level of 3 and
n4-c1 is not active any more in this Stream or any of its Consumers.Last modified 1mo ago
Export as PDF
Copy link
Edit on GitHub