Powered By GitBook
In Depth JWT Guide
This document provides a step by step deep dive into JWT usage within NATS. Starting with related concepts, it will introduce JWTs and how they can be used in NATS. This will NOT list every JWT/nsc option, but will focus on the important options and concepts.
To exercise listed examples please have the following installed:

Concepts

What are Accounts?

Accounts are the NATS isolation context.
1
accounts: {
2
A: {
3
users: [{user: a, password: a}]
4
},
5
B: {
6
users: [{user: b, password: b}]
7
},
8
}
Copied!
Messages published in one account won't be received in another.
1
> nats -s nats://a:[email protected]:4222 sub ">" &
2
[1] 28199
3
17:56:40 Subscribing on >
4
> nats -s nats://b:[email protected]:4222 pub "foo" "user b"
5
17:56:56 Published 6 bytes to "foo"
6
> nats -s nats://a:[email protected]:4222 pub "foo" "user a"
7
17:57:06 [#1] Received on "foo"
8
user a
9
10
17:57:06 Published 6 bytes to "foo"
11
>
Copied!
As indicated by the absence of a Received on print, the above example shows no message flow between user a associated with account A and user b in account B. Messages are delivered only within the same account. That is, unless you explicitly define it.
1
accounts: {
2
A: {
3
users: [{user: a, password: a}]
4
imports: [{stream: {account: B, subject: "foo"}}]
5
},
6
B: {
7
users: [{user: b, password: b}]
8
exports: [{stream: "foo"}]
9
},
10
}
Copied!
Here is a similar example, this time with messages crossing explicit account boundaries.
1
> nats -s nats://a:[email protected]:4222 sub ">" &
2
[1] 28552
3
18:28:18 Subscribing on >
4
> nats -s nats://b:[email protected]:4222 pub "foo" "user b"
5
18:28:25 [#1] Received on "foo"
6
user b
7
8
18:28:25 Published 6 bytes to "foo"
9
> nats -s nats://a:[email protected]:4222 pub "foo" "user a"
10
18:28:30 [#2] Received on "foo"
11
user a
12
13
18:28:30 Published 6 bytes to "foo"
14
>
Copied!
Accounts are a lot more powerful than what has been demonstrated here. Take a look at the complete documentation of accounts and the users associated with them. All of this is in a plain NATS config file. (Copy the above config and try it using this command: nats-server -c <filename>) In order to make any changes, every participating nats-server config file in the same security domain has to change. This configuration is typically controlled by one organization or the administrator.

Key Takeaways

    Accounts are isolated from each other
    One can selectively combine accounts
    Need to modify a config file to add/remove/modify accounts and user

What are NKEYs?

NKEYs are decorated, Base32 encoded, CRC16 check-summed, Ed25519 keys.
Ed25519 is:
    a public key signature system. (can sign and verify signatures)
    resistant to side channel attacks (no conditional jumps in algorithm)
NATS server can be configured with public NKEYs as user (identities). When a client connects the nats-server sends a challenge for the client to sign in order to prove it is in possession of the corresponding private key. The nats-server then verifies the signed challenge. Unlike with a password based scheme, the secret never left the client.
To assist with knowing what type of key one is looking at, in config or logs, the keys are decorated as follows:
    Public Keys, have a one byte prefix: O, A, U for various types. U meaning user.
    Private Keys, have a two byte prefix SO, SA, SU. S stands for seed. The remainder is the same as in public keys.
NKEYs are generated as follows:
1
> nk -gen user -pubout > a.nk
2
> cat a.nk
3
SUAAEZYNLTEA2MDTG7L5X7QODZXYHPOI2LT2KH5I4GD6YVP24SE766EGPA
4
UC435ZYS52HF72E2VMQF4GO6CUJOCHDUUPEBU7XDXW5AQLIC6JZ46PO5
5
> nk -gen user -pubout > b.nk
6
> cat b.nk
7
SUANS4XLL5NWBTM57GSVHLN4TMFW55WGGWNI5YXXSIOYFJQYFVNHJK5GFY
8
UARZVI6JAV7YMJTPRANXANOOW4K3ZCD45NYP6S7C7XKCBHPVN2TFZ7ZC
9
>
Copied!
Replacing the user/password with NKEY in account config example:
1
accounts: {
2
A: {
3
users: [{nkey:UC435ZYS52HF72E2VMQF4GO6CUJOCHDUUPEBU7XDXW5AQLIC6JZ46PO5}]
4
imports: [{stream: {account: B, subject: "foo"}}]
5
},
6
B: {
7
users: [{nkey:UARZVI6JAV7YMJTPRANXANOOW4K3ZCD45NYP6S7C7XKCBHPVN2TFZ7ZC}]
8
exports: [{stream: "foo"}]
9
},
10
}
Copied!
Simple example:
1
> nats -s nats://localhost:4222 sub --nkey=a.nk ">" &
2
[1] 94745
3
11:50:41 Subscribing on >
4
>nats -s nats://localhost:4222 pub --nkey=b.nk "foo" "nkey"
5
11:56:30 [#1] Received on "foo"
6
nkey
7
8
11:56:30 Published 4 bytes to "foo"
9
>
Copied!
When the nats-server was started with -V tracing, you can see the signature in the CONNECT message (formatting added manually).
1
[95184] 2020/10/26 12:15:44.350577 [TRC] [::1]:55551 - cid:2 - <<- [CONNECT {
2
"echo": true,
3
"headers": true,
4
"lang": "go",
5
"name": "NATS CLI",
6
"nkey": "UC435ZYS52HF72E2VMQF4GO6CUJOCHDUUPEBU7XDXW5AQLIC6JZ46PO5",
7
"no_responders": true,
8
"pedantic": false,
9
"protocol": 1,
10
"sig": "lopzgs98JBQYyRdw1zT_BoBpSFRDCfTvT4le5MYSKrt0IqGWZ2OXhPW1J_zo2_sBod8XaWgQc9oWohWBN0NdDg",
11
"tls_required": false,
12
"verbose": false,
13
"version": "1.11.0"
14
}]
Copied!
On connect, clients are instantly sent the nonce to sign as part of the INFO message (formatting added manually). Since telnet will not authenticate, the server closes the connection after hitting the authorization timeout.
1
> telnet localhost 4222
2
Trying ::1...
3
Connected to localhost.
4
Escape character is '^]'.
5
INFO {
6
"auth_required": true,
7
"client_id": 3,
8
"client_ip": "::1",
9
"go": "go1.14.1",
10
"headers": true,
11
"host": "0.0.0.0",
12
"max_payload": 1048576,
13
"nonce": "-QPTE1Jsk8kI3rE",
14
"port": 4222,
15
"proto": 1,
16
"server_id": "NBSHIXACRHUODC4FY2Z3OYXSZSRUBRH6VWIKQNGVPKOTA7H4YTXWJRTO",
17
"server_name": "NBSHIXACRHUODC4FY2Z3OYXSZSRUBRH6VWIKQNGVPKOTA7H4YTXWJRTO",
18
"version": "2.2.0-beta.26"
19
}
20
-ERR 'Authentication Timeout'
21
Connection closed by foreign host.
Copied!

Key Takeaways

    NKEYS are a secure way to authenticate clients
    Private keys are never accessed or stored by the NATS server
    The public key still needs to be configured

JSON Web Tokens (JWT)

Motivation for JWT

In a large organization the centralized configuration approach can lead to less flexibility and more resistance to change when controlled by one entity. Alternatively, instead of operating one infrastructure, it can be deployed more often (say per team) thus making import/export relationships harder as they have to bridge separate systems. In order to make accounts truly powerful, they should ideally be configured separately from the infrastructure, only constrained by limits. This is similar for user. An account contains the user but this relationship could be a reference as well, such that alterations to user do not alter the account. Users of the same account should be able to connect from anywhere in the same infrastructure and be able to exchange messages as long as they are in the same authentication domain.

Key Takeaways

    JWT splits a nats-server configuration into separate artifacts manageable by different entities.
    Management of Accounts, Configuration, and Users are separated.
    Accounts do NOT correspond to infrastructure, they correspond to teams or applications.
    Connect to any cluster in the same infrastructure and be able to communicate with all other users in your account.
    Infrastructure and its topology have nothing to do with Accounts and where an Account's User connects from.

Decentralized Authentication/Authorization using JWT

Account and User creation managed as separate artifacts in a decentralized fashion using NKEYs. Relying upon a hierarchical chain of trust between three distinct NKEYs and associated roles:
    1.
    Operator: corresponds to operator of a set of NATS servers in the same authentication domain (entire topology, crossing gateways and leaf nodes)
    2.
    Account: corresponds to the set of a single account's configuration
    3.
    User: corresponds to one user's configuration
Each NKEY is referenced, together with additional configuration, in a JWT document. Each JWT has a subject field and its value is the public portion of an NKEY and serves as identity. Names exist in JWT but as of now are only used by tooling, nats-server does not read this value. The referenced NKEY's role determines the JWT content.
    1.
    Operator JWTs contain server configuration applicable throughout all operated NATS servers
    2.
    Account JWTs contain Account specific configuration such as exports, imports, limits, and default user permissions
    3.
    User JWTs contain user specific configuration such as permissions and limits
In addition, JWTs can contain settings related to their decentralized nature, such as expiration/revocation/signing. At no point do JWTs contain the private portion of an NKEY, only signatures that can be verified with public NKEY. JWT content can be viewed as public, although it's content may reveal which subjects/limits/permissions exist.

Key Takeaways

    JWTs are hierarchically organized in operator, account and user.
    They carry corresponding configuration and config dedicated to the decentralized nature of NATS JWT usage.

NATS JWT Hierarchy

Decentralized Chain of Trust

A nats-server is configured to trust an operator. Meaning, the Operator JWT is part of its server configuration and requires a restart or nats-server --signal reload once changed. It is also configured with a way to obtain account JWT in one of three ways (explained below).
Clients provide a User JWT when connecting. An Account JWT is not used by clients talking to a nats-server. The clients also possess the private NKEY corresponding to the JWT identity, so that they can prove their identity as described above.
The issuer field of the User JWT identifies the Account, and the nats-server then independently obtains the current Account JWT from its configured source. The server can then verify that signature on the User JWT was issued by an NKEY of the claimed Account, and in turn that the Account has an issuer of the Operator and that an NKEY of the Operator signed the Account JWT. The entire three-level hierarchy is verified.

Obtain an Account JWT

To obtain an Account JWT, the nats-server is configured with one of three resolver types. Which one to pick depends upon your needs:
    mem-resolver: Very few or very static accounts
      You are comfortable changing the server config if the operator or any accounts change.
      You can generate a user programmatically using NKEYs and a JWT library (more about that later).
      Users do not need to be known by nats-server.
    url-resolver: Very large volume of accounts
      Same as mem-resolver, except you do not have to modify server config if accounts are added/changed.
      Changes to the operator still require reloading (only a few operations require that).
      Will download Accounts from a web server.
        Allows for easy publication of account JWTs programmatically generated using NKEYs and the JWT library.
        The nats-account-server is such a webserver. When set up correctly, it will inform nats-server of Account JWT changes.
      Depending on configuration, requires read and/or write access to persistent storage.
    nats-resolver: Same as url-resolver, just uses NATS instead of http
      No separate binary to run/config/monitor.
      Easier clustering when compared to nats-account-server. Will eventually converge on the union of all account JWTs known to every participating nats-server.
      Requires persistent storage in the form of a NON-NTFS directory for nats-server to exclusively write into.
      Optionally, directly supports Account JWT removal.
      Between nats-resolver and url-resolver, the nats-resolver is the clear recommendation.
If your setup has few Accounts and Users and/or you are comfortable reloading server configs when accounts/users change, then save yourself the complexity and do not use JWT. Regular config -- possibly with NKEYs -- will work just fine for you.

JWT and Chain of Trust Verification

Each JWT document has a subject it represents. This is the public identity NKEY represented by the JWT document. JWT documents contain an issued at (iat) time of signing. This time is in seconds since Unix epoch. It is also used to determine which of two JWTs for the same subject is more recent. Furthermore JWT documents have an issuer, this may be an (identity) NKEY or a dedicated signing NKEY of an item one level above it in the trust hierarchy. A key is a signing key if it is listed as such in the JWT (above). Signing NKEYs adhere to same NKEY roles and are additional keys that unlike identity NKEY may change over time. In the hierarchy, signing keys can only be used to sign JWT for the role right below them. User JWTs have no signing keys for this reason. To modify one role's set of signing keys, the identity NKEY needs to be used.
Each JWT is signed as follows: jwt.sig = sign(hash(jwt.header+jwt.body), private-key(jwt.issuer)) (jwt.issuer is part of jwt.body) If a JWT is valid, the JWT above it is validated as well. If all of them are valid, the chain of trust between them is tested top down as follows:
Type
Trust Rule
Obtained
Operator
jwt.issuer == jwt.subject (self signed)
configured to trust
Account
jwt.issuer == trusted issuing operator (signing/identity) key
configured to obtain
User
jwt.issuer == trusted issuing account (signing/identity) key && jwt.issuedAt > issuing account revocations[jwt.subject]
provided on connect
This is a conceptual view. While all these checks happen, the results of earlier evaluations might be cached: if the Operator/Account is trusted already and the JWT did not change since, then there is no reason to re-evaluate.
Below are examples of decoded JWT. (iss == issuer, sub == subject, iat == issuedAt)
1
> nsc describe operator --json
2
{
3
"iat": 1603473819,
4
"iss": "OBU5O5FJ324UDPRBIVRGF7CNEOHGLPS7EYPBTVQZKSBHIIZIB6HD66JF",
5
"jti": "57BWRLW67I6JTVYMQAZQF54G2G37DJB5WG5IFIPVYI4PEYNX57ZQ",
6
"name": "DEMO",
7
"nats": {
8
"account_server_url": "nats://localhost:4222",
9
"system_account": "AAAXAUVSGK7TCRHFIRAS4SYXVJ76EWDMNXZM6ARFGXP7BASNDGLKU7A5"
10
},
11
"sub": "OBU5O5FJ324UDPRBIVRGF7CNEOHGLPS7EYPBTVQZKSBHIIZIB6HD66JF",
12
"type": "operator"
13
}
14
> nsc describe account -n demo-test --json
15
{
16
"iat": 1603474600,
17
"iss": "OBU5O5FJ324UDPRBIVRGF7CNEOHGLPS7EYPBTVQZKSBHIIZIB6HD66JF",
18
"jti": "CZDE4PM7MGFNYHRZSE6INTP6QDU4DSLACVHPQFA7XEYNJT6R6LLQ",
19
"name": "demo-test",
20
"nats": {
21
"limits": {
22
"conn": -1,
23
"data": -1,
24
"exports": -1,
25
"imports": -1,
26
"leaf": -1,
27
"payload": -1,
28
"subs": -1,
29
"wildcards": true
30
}
31
},
32
"sub": "ADKGAJU55CHYOIF5H432K2Z2ME3NPSJ5S3VY5Q42Q3OTYOCYRRG7WOWV",
33
"type": "account"
34
}
35
> nsc describe user -a demo-test -n alpha --json
36
{
37
"iat": 1603475001,
38
"iss": "ADKGAJU55CHYOIF5H432K2Z2ME3NPSJ5S3VY5Q42Q3OTYOCYRRG7WOWV",
39
"jti": "GOOPXCFDWVMEU3U6I6MT344Z56MGBYIS42GDXMUXDFA3NYDR2RUQ",
40
"name": "alpha",
41
"nats": {
42
"pub": {},
43
"sub": {}
44
},
45
"sub": "UC56LV5NNMP5FURQZ7HZTGWCRRTWSMHZNNELQMHDLH3DCYNGX57B2TN6",
46
"type": "user"
47
}
48
>
Copied!

Obtain a User JWT - Client Connect

When a client connects, the steps below have to succeed. The following nats-server configuration is used (for ease of understanding, we are using url-resolver):
1
operator: ./trustedOperator.jwt
2
resolver: URL(http://localhost:9090/jwt/v1/accouts/)
Copied!
    1.
    Client connects and the nats-server responds with INFO (identical to NKEYs) and a containing nonce.
    1
    > telnet localhost 4222
    2
    Trying 127.0.0.1...
    3
    Connected to localhost.
    4
    Escape character is '^]'.
    5
    INFO {
    6
    "auth_required": true,
    7
    "client_id": 5,
    8
    "client_ip": "127.0.0.1",
    9
    "go": "go1.14.1",
    10
    "headers": true,
    11
    "host": "localhost",
    12
    "max_payload": 1048576,
    13
    "nonce": "aN9-ZtS7taDoAZk",
    14
    "port": 4222,
    15
    "proto": 1,
    16
    "server_id": "NCIK6FX5MRIEPMEK22YL2ECLIWVJBH2SWFD5EQWSI5XRDQPKZXWKX3VP",
    17
    "server_name": "NCIK6FX5MRIEPMEK22YL2ECLIWVJBH2SWFD5EQWSI5XRDQPKZXWKX3VP",
    18
    "tls_required": true,
    19
    "version": "2.2.0-beta.26"
    20
    }
    21
    Connection closed by foreign host.
    Copied!
    For ease of use, the NATS CLI uses a creds file that is the concatenation of JWT and private user identity/NKEY.
    1
    > cat user.creds
    2
    -----BEGIN NATS USER JWT-----
    3
    eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJXNkFYSFlSS1RHVTNFUklQM0dSRDdNV0FQTzQ2VzQ2Vzc3R1JNMk5SWFFIQ0VRQ0tCRjJRIiwiaWF0IjoxNjAzNDczNzg4LCJpc3MiOiJBQUFYQVVWU0dLN1RDUkhGSVJBUzRTWVhWSjc2RVdETU5YWk02QVJGR1hQN0JBU05ER0xLVTdBNSIsIm5hbWUiOiJzeXMiLCJzdWIiOiJVRE5ZMktLUFRJQVBQTk9OT0xBVE5SWlBHTVBMTkZXSFFQS1VYSjZBMllUQTQ3Tk41Vk5GSU80NSIsInR5cGUiOiJ1c2VyIiwibmF0cyI6eyJwdWIiOnt9LCJzdWIiOnt9fX0.ae3OvcapjQgbXhI2QbgIs32AWr3iBb2UFRZbXzIg0duFHNPQI5LsprR0OQoSlc2tic6e3sn8YM5x0Rt34FryDA
    4
    ------END NATS USER JWT------
    5
    6
    ************************* IMPORTANT *************************
    7
    NKEY Seed printed below can be used to sign and prove identity.
    8
    NKEYs are sensitive and should be treated as secrets.
    9
    10
    -----BEGIN USER NKEY SEED-----
    11
    SUAAZU5G7UOUR7VXQ7DBD5RQTBW54O2COGSXAVIYWVZE4GCZ5C7OCZ5JLY
    12
    ------END USER NKEY SEED------
    13
    14
    *************************************************************
    Copied!
    1
    > nats -s localhost:4222 "--creds=user.creds" pub "foo" "hello world"
    Copied!
    2.
    The Client responds with a CONNECT message (formatting added manually), containing a JWT and signed nonce. (output copied from nats-server started with -V)
    1
    [98019] 2020/10/26 16:07:53.861612 [TRC] 127.0.0.1:56830 - cid:4 - <<- [CONNECT {
    2
    "echo": true,
    3
    "headers": true,
    4
    "jwt": "eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJXNkFYSFlSS1RHVTNFUklQM0dSRDdNV0FQTzQ2VzQ2Vzc3R1JNMk5SWFFIQ0VRQ0tCRjJRIiwiaWF0IjoxNjAzNDczNzg4LCJpc3MiOiJBQUFYQVVWU0dLN1RDUkhGSVJBUzRTWVhWSjc2RVdETU5YWk02QVJGR1hQN0JBU05ER0xLVTdBNSIsIm5hbWUiOiJzeXMiLCJzdWIiOiJVRE5ZMktLUFRJQVBQTk9OT0xBVE5SWlBHTVBMTkZXSFFQS1VYSjZBMllUQTQ3Tk41Vk5GSU80NSIsInR5cGUiOiJ1c2VyIiwibmF0cyI6eyJwdWIiOnt9LCJzdWIiOnt9fX0.ae3OvcapjQgbXhI2QbgIs32AWr3iBb2UFRZbXzIg0duFHNPQI5LsprR0OQoSlc2tic6e3sn8YM5x0Rt34FryDA",
    5
    "lang": "go",
    6
    "name": "NATS CLI",
    7
    "no_responders": true,
    8
    "pedantic": false,
    9
    "protocol": 1,
    10
    "sig": "VirwM--xq5i2RI9VEQiFYv_6JBs-IR4oObypglR7qVxYtXDUtIKIr1qXW_M54iHFB6Afu698J_in5CfBRjuVBg",
    11
    "tls_required": true,
    12
    "verbose": false,
    13
    "version": "1.11.0"
    14
    }]
    Copied!
    3.
    Server verifies if a JWT returned is a user JWT and if it is consistent: sign(jwt.sig, jwt.issuer) == hash(jwt.header+jwt.body) (issuer is part of body)
    4.
    Server verifies if nonce matches JWT.subject, thus proving client's possession of private user NKEY.
    5.
    Server either knows referenced account or downloads it from http://localhost:9090/jwt/v1/accouts/AAAXAUVSGK7TCRHFIRAS4SYXVJ76EWDMNXZM6ARFGXP7BASNDGLKU7A5
    6.
    Server verifies downloaded JWT is an account JWT and if it is consistent: sign(jwt.sig, jwt.issuer) == hash(jwt.header+jwt.body) (issuer is part of body).
    7.
    Server verifies if an account JWT issuer is in configured list of trusted operator keys (derived from operator JWT in configuration).
    8.
    Server verifies that a user JWT subject is not in the account's revoked list, or if jwt.issuedAt field has a higher value.
    9.
    Server verifies that a user JWT issuer is either identical to the account JWT subject or part of the account JWT signing keys.
    10.
    If all of the above holds true, the above invocation will succeed, only if the user JWT does not contain permissions or limits restricting the operation otherwise.
    1
    > nats -s localhost:4222 "--creds=user.creds" pub "foo" "hello world"
    2
    > 16:56:02 Published 11 bytes to "foo"
    Copied!
    11.
    Output if user.creds were to contain a JWT where the maximum message payload is limited to 5 bytes
    1
    > nats -s localhost:4222 "--creds=user.creds" pub "foo" "hello world"
    2
    nats: error: nats: Maximum Payload Violation, try --help
    3
    >
    Copied!

Key Takeaways

    JWTs are secure
    JWTs carry configuration appropriate to their role as Operator/Accounts/User
    JWTs provide a basis for operating one single NATS infrastructure which serves separate, yet optionally connected, entities
    Account resolvers are a way to obtain unknown Account JWTs
    On connect clients provide only the User JWT and use the NKEY for the JWT to authenticate.
    JWTs can be issued programmatically

Deployment Models Enabled by Chain of Trust

Depending on which entity has access to private Operator/Account identity or signing NKEYs, different deployment models are enabled. When picking one, it is important to pick the simplest deployment model that enables what you need it to do. Everything beyond just results in unnecessary configuration and steps.
    1.
    Centralized config: one (set of) user(s) has access to all private operator and account NKEYs.
    Administrators operating the shared infrastructure call all the shots
    2.
    Decentralized config (with multiple nsc environments, explained later):
      1.
      Administrator/Operator(s) have access to private operator NKEYs to sign accounts. By signing or not signing an account JWT, Administrators can enforce constraints (such as limits).
      2.
      Other sets of users (teams) have access to their respective private account identity/signing NKEYs and can issue/sign a user JWT.
    This can also be used by a single entity to not mix up nsc environments as well.
    3.
    Self-service, decentralized config (shared dev cluster):
    Is similar to 2, but sets of users 2.i have access to an operator private signing NKEY.
    This allows teams to add/modify their own accounts.
    Since administrators give up control over limits, there should be at least one organizational mechanism to prevent unchecked usage.
    Administrators operating the infrastructure can add/revoke access by controlling the set of operator signing keys.
    4.
    Mix of the above - as needed: separate sets of users (with multiple nsc environments).
    For some user/teams the Administrator operates everything.
Signing keys can not only be used by individuals in one or more nsc environments, but also by programs facilitating JWT and NKEY libraries. This allows the implementation of sign-up services.
    Account signing key enabled on the fly:
      user generation (explained later)
      export activation generation (explained later)
    Operator signing key enables on the fly account generation.

Key Takeaways

    JWTs and the associated chain of trust allows for centralized, decentralized, or self-service account configuration
    It is important to pick the deployment model that fits your needs, NOT the most complicated one
    Distributing Operator/Account JWT NKEYs between Administrators and teams enables these deployment models
    Sign-up services for Accounts/Users can be implemented by programs in possession of the parent type's signing keys

Accounts Re-visited

A deeper understanding of accounts will help you to best setup NATS JWT based security.
    What entity do accounts correspond to:
    Our official suggestion is to scope accounts by application/service offered.
    This is very fine grained and will require some configuration.
    This is why some users gravitate to accounts per team. One account for all Applications of a team.
    It is possible to start out with less granular accounts and as applications grow in importance or scale become more fine grained.
    Compared to file based config, Imports and Exports change slightly.
    To control who gets to import an export, activation tokens are introduced.
    These are JWTs that an importer can embed.
    They comply to similar verification rules as user JWT, thus enabling a nats-server to check if the exporting account gave explicit consent.
    Due to the use of a token, the exporting account's JWT does not have to be modified for each importing account.
    Updates of JWTs are applied as nats-server discover them.
      How this is done depends on the resolver.
        mem-resolver require nats-server --signal reload to re-read all configured account JWTs.
          url-resolver and nats-resolver listen on a dedicated update subject of the system account and applied if the file is valid.
        nats-resolver will also also update the corresponding JWT file and compensate in case the update message was not received due to temporary disconnect.
      User JWTs only depend on the issuing Account NKEY, they do NOT depend on a particular version of an Account JWT.
      Depending on the change, the internal Account representation will be updated and existing connections re-evaluated.
    The System Account is the account under which nats-server offers (administrative) services and monitoring events.

Key Takeaways

    Accounts can be arbitrarily scoped, from Application to Team
    Account Exports can be restricted by requiring use of activation tokens
    Receiving a more recent Account JWT causes the nats-server to apply changes and re evaluate existing connections.

Tooling And Key Management

This section will introduce nsc cli to generate and manage operator/accounts/user. Even if you intend to primarily generate your Accounts/User programmatically, in all likelihood, you won't do so for an operator or all accounts. Key Management and how to do so using nsc will also be part of this section.

nsc

Environment

nsc is a tool that uses the JWT and NKEY libraries to create NKEYs (if asked to) and all types of JWT. It then stores these artifacts in separate directories.
It keeps track of the last operator/account used. Because of this, commands do not need to reference operator/accounts but can be instructed to do so. (recommended for scripts) It supports an interactive mode when -i is provided. When used, referencing accounts/keys is easier.
nsc env will show where NKEYS/JWT are stored and what current defaults are. For testing you may want to switch between nsc environments: Changing the (JWT) store directory: nsc env --store <different folder> Changing the (NKEY) store directory by having an environment variable set: export NKEYS_PATH=<different folder>
Subsequent sections will refer to different environments in context of different deployment modes. As such you can skip over all mentions for modes not of interest to you. The mixed deployment mode is not mentioned and left as an exercise to the reader.

Backup

NKEYS store directory

Possessing NKEYS gives access to the system. Backups should therefore best be offline and access to them should be severely restricted. In cases where regenerating all/parts of the operator/accounts is not an option, signing NKEYs must be used and identity NKEYs should be archived and then removed from the original store directory, so that in the event of a data breach you can recover without a flag-day change-over of identities. Thus, depending on your scenario, relevant identity NKEYS need to only exist in very secure offline backup(s).

JWT store directory

The store directory contains JWTs for operators, accounts, and users. It does not contain private keys. Therefore it is ok to back these up or even store them in a VCS such as git. But be aware that depending on content, JWT may reveal which permissions/subjects/public-nkeys exist. Knowing the content of a JWT does not grant access; only private keys will. However, organizations may not wish to make those public outright and thus have to make sure that these external systems are secured appropriately.
When restoring an older version, be aware that:
    All changes made since will be lost, specifically revocations may be undone.
    Time has moved on and thus JWTs that were once valid at the time of the backup or commit may be expired now. Thus you may have to be edit them to match your expectations again.
    NKEYS are stored in a separate directory, so to not restore a JWT for which the NKEY has been deleted since:
      Either keep all keys around; or
      Restore the NKEY directory in tandem

Names in JWT

JWTs allow you to specify names. But names do NOT represent an identity, they are only used to ease referencing of identities in our tooling. At no point are these names used to reference each other. Only the public identity NKEY is used for that. The nats-server does not read them at all. Because names do not relate to identity, they may collide. Therefore, when using nsc, these names need to be keep unique.

Setup an Operator

Create/Edit Operator - Operator Environment - All Deployment modes

Create operator with system account and system account user: nsc add operator -n <operator-name> --sys The command nsc edit operator [flags] can subsequently be used to modify the operator. For example if you are setting the account server url (used by url-resolver and nats-resolver), nsc does not require them being specified on subsequent commands. nsc edit operator --account-jwt-server-url "nats://localhost:4222"
We always recommend using signing keys for an operator. Generate one for an operator (-o) and store it in the key directory (--store) The output will display the public portion of the signing key, use that to assign it to the operator (--sk O...). nsc generate nkey -o --store followed by nsc edit operator --sk OB742OV63OE2U55Z7UZHUB2DUVGQHRA5QVR4RZU6NXNOKBKJGKF6WRTZ. To pick the operator signing key for account generation, provide the -i option when doing so.
The system account is the account under which nats-server offers system services as will be explained below in the system-account section. To access these services a user with credentials for the system account is needed. Unless this user is restricted with appropriate permissions, this user is essentially the admin user. They are created like any other user.
For cases where signing keys are generated and immediately added --sk generate will create an NKEY on the fly and assign it as signing NKEY.

Import Operator - Non Operator/Administrator Environment - Decentralized/Self Service Deployment Modes

In order to import an Operator JWT, such as the one just created, into a separate nsc environment maintained by a different entity/team, the following has to happen:
    1.
    Obtain the operator JWT using: nsc describe operator --raw and store the output in a file named operator.jwt. The option --raw causes the raw JWT to be emitted.
    2.
    Exchange that file or it's content any way you like, email works fine (as there are no credentials in the JWT).
    3.
    Import the operator JWT into the second environment with: nsc add operator -u operator.jwt
Should the operator change and an update is required, simply repeat these steps but provide the --force option during the last step. This will overwrite the stored operator JWT.

Import Operator - Self Service Deployment Modes

In addition to the previous step, self service deployments require an operator signing key and a system account user. Ideally you would want an operator signing key per entity to distribute a signing key too. Simply repeat the command shown earlier but: 1. Perform nsc generate nkey -o --store in this environment instead 2. Exchange the public key with the Administrator/Operator via a way that assures you sent the public key and not someone elses. 3. Perform nsc edit operator --sk in the operator environment 4. Refresh the operator JWT in this environment by performing the import steps using --force
To import the system account user needed for administrative purposes as well as monitoring, perform these steps: 1. Perform nsc describe account -n SYS --raw and store the output in a file named SYS.jwt. The option -n specifies the (system) account named SYS. 2. Exchange the file. 3. Import the account nsc import account --file SYS.jwt 4. Perform nsc generate nkey -u --store in this environment 5. Exchange the public key printed by the command with the Administrator/Operator via a way that assures you sent the public key and not someone elses. 6. Create a system account user named (-n) any way you like (here named sys-non-op) providing (-k) the exchanged public key nsc add user -a SYS -n sys-non-op -k UDJKPL7H6QY4KP4LISNHENU6Z434G6RLDEXL2C64YZXDABNCEOAZ4YY2 in the operator environment. (-a references the Account SYS.) 7. If desired edit the user 8. Export the user nsc describe user -a SYS -n sys-non-op --raw from the operator environment and store it in a file named sys.jwt. (-n references the user sys-non-op) 9. Exchange the file 10. Import the user in this environment using nsc import user --file sys.jwt
As a result of these operations, your operator environment should have these keys and signing keys:
1
> nsc list keys --all
2
+------------------------------------------------------------------------------------------------+
3
| Keys |
4
+--------------+----------------------------------------------------------+-------------+--------+
5
| Entity | Key | Signing Key | Stored |
6
+--------------+----------------------------------------------------------+-------------+--------+
7
| DEMO | OD5FHU4LXGDSGDHO7UNRMLW6I36QX5VPJXRQHFHMRUIKSHOPEDSHVPBB | | * |
8
| DEMO | OBYAIG4T4PVR6GVYDERN74RRW7VBKRWBTI7ULLMM6BRHUID4AAQL7SGA | * | |
9
| ACC | ADRB4JJYFDLWKIMX4DH6MX2DMKA3TENJWGMNVM5ILYLZTT6BN7QIF5ZX | | |
10
| SYS | AAYVLZJC2ULKSH5HNSKMIKFMCEHCNU5VOV5KG56IRL7ENHLBUGZ27CZT | | * |
11
| sys | UBVZYLLCAFMHBXBUDKKKFKH62T4AW7Q5MAAE3R3KKAIRCZNYITZPDQZ3 | | * |
12
| sys-non-op | UDJKPL7H6QY4KP4LISNHENU6Z434G6RLDEXL2C64YZXDABNCEOAZ4YY2 | | |
13
+--------------+----------------------------------------------------------+-------------+--------+
Copied!
And your account should have the following ones:
1
> nsc list keys --all
2
+------------------------------------------------------------------------------------------------+
3
| Keys |
4
+--------------+----------------------------------------------------------+-------------+--------+
5
| Entity | Key | Signing Key | Stored |
6
+--------------+----------------------------------------------------------+-------------+--------+
7
| DEMO | OD5FHU4LXGDSGDHO7UNRMLW6I36QX5VPJXRQHFHMRUIKSHOPEDSHVPBB | | |
8
| DEMO | OBYAIG4T4PVR6GVYDERN74RRW7VBKRWBTI7ULLMM6BRHUID4AAQL7SGA | * | * |
9
| SYS | AAYVLZJC2ULKSH5HNSKMIKFMCEHCNU5VOV5KG56IRL7ENHLBUGZ27CZT | | |
10
| sys-non-op | UDJKPL7H6QY4KP4LISNHENU6Z434G6RLDEXL2C64YZXDABNCEOAZ4YY2 | | * |
11
+--------------+----------------------------------------------------------+-------------+--------+
Copied!
Between the two outputs, compare the Stored column.
Alternatively if the administrator is willing to exchange private keys and the exchange can be done securely, a few of these steps fall away. The signing key and system account user can be generated in the administrator/operator environment, omitting --store to avoid unnecessary key copies. Then the public/private signing NKEYS are exchanged together with the system account user as creds file. A creds file can be generated with nsc generate creds -a SYS -n sys-non-op and imported into this environment with nsc import user --file sys.jwt. If the signing key is generated before the operator is imported into this environment, operator update falls away.

Setup an Account

Create/Edit Account - All Environments - All Deployment modes

Create an account as follows: nsc add account -n <account name> -i In case you have multiple operator signing keys -i will prompt you to select one. nsc edit account [flags] can subsequently be used to modify the account. (Edit is also applicable to the system account)
Similar to the operator signing keys are recommended. Generate signing key for an account (-a) and store it in the key directory maintained by nsc (--store) The output will display the public portion of the signing key, use that to assign it to the account (--sk A...) nsc generate nkey -a --store nsc edit account --sk ACW2QC262CIQUX4ACGOOS5XLKSZ2BY2QFBAAOF3VOP7AWAVI37E2OQZX To pick the signing key for user generation, provide the -i option when doing so.

Export Account - Non Operator/Administrator Environment - Decentralized Deployment Modes

In this mode, the created account is self-signed. To have it signed by the operator perform these steps: 1. In this environment export the created account as a JWT like this nsc describe account -n <account name> --raw. Store the output in a file named import.jwt. 2. Exchange the file with the Administrator/Operator via a way that assures it is your JWT and not someone elses. 3. In the operator environment import the account with nsc import account --file import.jwt. This step also re-signs the JWT so that it is no longer self-signed. 4. The Administrator/operator can now modify the account with nsc edit account [flags]
Should the account change and an update is required, simply repeat these steps but provide the --force option during the last step. This will overwrite the stored account JWT.

Export Account - Non Operator/Administrator Environment - Self Service Deployment Modes

This environment is set up with a signing key, thus the account is already created properly signed. The only step that is needed is to push the Account into the NATS network. However, this depends on your ability to do so. If you have no permissions, you have to perform the same steps as for the decentralized deployment mode. The main difference is that upon import, the account won't be re-signed.

Publicize an Account with Push - Operator Environment/Environment with push permissions - All Deployment Modes

How accounts can be publicized wholly depends on the resolver you are using:
    mem-resolver: The operator has to have all accounts imported and generate a new config.
    url-resolver: nsc push will send an HTTP POST request to the hosting webserver or nats-account-server.
    nats-resolver: Every environment with a system account user that has permissions to send properly signed account JWT as requests to:
      $SYS.REQ.CLAIMS.UPDATE can upload and update all accounts. Currently, nsc push uses this subject.
      $SYS.REQ.ACCOUNT.*.CLAIMS.UPDATE can upload and update specific accounts.
nsc generate config <resolver-type> as a utility that generates the relevant NATS config. Where <resolver-type> can be --mem-resolver or --nats-resolver for the corresponding resolver. Typically the generated output is stored in a file that is then included by the NATS config. Every server within the same authentication domain needs to be configured with this configuration.

nats-resolver setup and push example - Operator Environment/Environment with push permissions - All Deployment Modes

This is a quick demo of the nats-based resolver from operator creation to publishing a message. Please be aware that the ability to push only relates to permissions to do so and does not require an account keys. Thus, how accounts to be pushed came to be in the environment (outright creation/import) does not matter. For simplicity, this example uses the operator environment.
Operator Setup
1
> nsc add operator -n DEMO --sys
2
[ OK ] generated and stored operator key "ODHUVOUVUA3XIBV25XSQS2NM2UN4IKJYLAMCGLWRFAV7F7KUWADCM4K6"
3
[ OK ] added operator "DEMO"
4
[ OK ] created system_account: name:SYS id:AA6W5MRDIFIQWE6UE6D4YWQT5L4YZG7ZRHSKYCPF2VIEMUHRZH3VQZ27
5
[ OK ] created system account user: name:sys id:UABM73CE5F3ZYFNC3ZDODAF7GIB62W2WXV5DOLMYLGEW4MEHYBC46PN4
6
[ OK ] system account user creds file stored in `~/test/demo/env1/keys/creds/DEMO/SYS/sys.creds`
7
> nsc edit operator --account-jwt-server-url nats://localhost:4222
8
[ OK ] set account jwt server url to "nats://localhost:4222"
9
[ OK ] edited operator "DEMO"
Copied!
Inspect the setup
1
> nsc list keys --all
2
+------------------------------------------------------------------------------------------+
3
| Keys |
4
+--------+----------------------------------------------------------+-------------+--------+
5
| Entity | Key | Signing Key | Stored |
6
+--------+----------------------------------------------------------+-------------+--------+
7
| DEMO | ODHUVOUVUA3XIBV25XSQS2NM2UN4IKJYLAMCGLWRFAV7F7KUWADCM4K6 | | * |
8
| SYS | AA6W5MRDIFIQWE6UE6D4YWQT5L4YZG7ZRHSKYCPF2VIEMUHRZH3VQZ27 | | * |
9
| sys | UABM73CE5F3ZYFNC3ZDODAF7GIB62W2WXV5DOLMYLGEW4MEHYBC46PN4 | | * |
10
+--------+----------------------------------------------------------+-------------+--------+
11
> nsc describe operator
12
+-------------------------------------------------------------------------------+
13
| Operator Details |
14
+--------------------+----------------------------------------------------------+
15
| Name | DEMO |
16
| Operator ID | ODHUVOUVUA3XIBV25XSQS2NM2UN4IKJYLAMCGLWRFAV7F7KUWADCM4K6 |
17
| Issuer ID | ODHUVOUVUA3XIBV25XSQS2NM2UN4IKJYLAMCGLWRFAV7F7KUWADCM4K6 |
18
| Issued | 2020-11-04 19:25:25 UTC |
19
| Expires | |
20
| Account JWT Server | nats://localhost:4222 |
21
| System Account | AA6W5MRDIFIQWE6UE6D4YWQT5L4YZG7ZRHSKYCPF2VIEMUHRZH3VQZ27 |
22
+--------------------+----------------------------------------------------------+
23
> nsc describe account
24
+--------------------------------------------------------------------------------------+
25
| Account Details |
26
+---------------------------+----------------------------------------------------------+
27
| Name | SYS |
28
| Account ID | AA6W5MRDIFIQWE6UE6D4YWQT5L4YZG7ZRHSKYCPF2VIEMUHRZH3VQZ27 |
29
| Issuer ID | ODHUVOUVUA3XIBV25XSQS2NM2UN4IKJYLAMCGLWRFAV7F7KUWADCM4K6 |
30
| Issued | 2020-11-04 19:24:41 UTC |
31
| Expires | |
32
+---------------------------+----------------------------------------------------------+
33
| Max Connections | Unlimited |
34
| Max Leaf Node Connections | Unlimited |
35
| Max Data | Unlimited |
36
| Max Exports | Unlimited |
37
| Max Imports | Unlimited |
38
| Max Msg Payload | Unlimited |
39
| Max Subscriptions | Unlimited |
40
| Exports Allows Wildcards | True |
41
+---------------------------+----------------------------------------------------------+
42
| Imports | None |
43
| Exports | None |
44
+---------------------------+----------------------------------------------------------+
45
>
Copied!
Generate the config and start the server in the background. Also, inspect the generated config. It consists of the mandatory operator, explicitly lists the system account and corresponding JWT.
1
> nsc generate config --nats-resolver > nats-res.cfg
2
> nats-server -c nats-res.cfg --addr localhost --port 4222 &
3
[2] 30129
4
[30129] 2020/11/04 14:30:14.062132 [INF] Starting nats-server version 2.2.0-beta.26
5
[30129] 2020/11/04 14:30:14.062215 [INF] Git commit [not set]
6
[30129] 2020/11/04 14:30:14.062219 [INF] Using configuration file: nats-res.cfg
7
[30129] 2020/11/04 14:30:14.062220 [INF] Trusted Operators
8
[30129] 2020/11/04 14:30:14.062224 [INF] System : ""
9
[30129] 2020/11/04 14:30:14.062226 [INF] Operator: "DEMO"
10
[30129] 2020/11/04 14:30:14.062241 [INF] Issued : 2020-11-04 14:25:25 -0500 EST
11
[30129] 2020/11/04 14:30:14.062244 [INF] Expires : 1969-12-31 19:00:00 -0500 EST
12
[30129] 2020/11/04 14:30:14.062652 [INF] Managing all jwt in exclusive directory /demo/env1/jwt
13
[30129] 2020/11/04 14:30:14.065888 [INF] Listening for client connections on localhost:4222
14
[30129] 2020/11/04 14:30:14.065896 [INF] Server id is NBQ6AG5YIRC6PRCUPCAUSVCSCQWAAWW2XQXIM6UPW5AFPGZBUKZJTRRS
15
[30129] 2020/11/04 14:30:14.065898 [INF] Server name is NBQ6AG5YIRC6PRCUPCAUSVCSCQWAAWW2XQXIM6UPW5AFPGZBUKZJTRRS
16
[30129] 2020/11/04 14:30:14.065900 [INF] Server is ready
17
>
Copied!
Add an account and a user for testing.
1
> nsc add account -n TEST
2
[ OK ] generated and stored account key "ADXDDDR2QJNNOSZZX44C2HYBPRUIPJSQ5J3YG2XOUOOEOPOBNMMFLAIU"
3
[ OK ] added account "TEST"
4
> nsc add user -a TEST -n foo
5
[ OK ] generated and stored user key "UA62PGBNKKQQWDTILKP5U4LYUYF3B6NQHVPNHLS6IZIPPQH6A7XSRWE2"
6
[ OK ] generated user creds file `/DEMO/TEST/foo.creds`
7
[ OK ] added user "foo" to account "TEST"
8
>
Copied!
Without having pushed the account the user can't be used yet.
1
> nats -s nats://localhost:4222 pub --creds=/DEMO/TEST/foo.creds "hello" "world"
2
nats: error: read tcp 127.0.0.1:60061->127.0.0.1:4222: i/o timeout, try --help
3
[9174] 2020/11/05 16:49:34.331078 [WRN] Account [ADI4H2XRYMT5ENVBBS3UKYC2FBLGB3NF4VV5L57HUZIO4AMYROB4LMYF] fetch took 2.000142625s
4
[9174] 2020/11/05 16:49:34.331123 [WRN] Account fetch failed: fetching jwt timed out
5
[9174] 2020/11/05 16:49:34.331182 [ERR] 127.0.0.1:60061 - cid:5 - "v1.11.0:go:NATS CLI Version development" - authentication error
6
[9174] 2020/11/05 16:49:34.331258 [WRN] 127.0.0.1:60061 - cid:5 - "v1.11.0:go:NATS CLI Version development" - Readloop processing time: 2.000592801s
Copied!
Push the account, or push all accounts
1
> nsc push -a TEST
2
[ OK ] push to nats-server "nats://localhost:4222" using system account "SYS" user "sys":
3
[ OK ] push TEST to nats-server with nats account resolver:
4
[ OK ] pushed "TEST" to nats-server NBQ6AG5YIRC6PRCUPCAUSVCSCQWAAWW2XQXIM6UPW5AFPGZBUKZJTRRS: jwt updated
5
[ OK ] pushed to a total of 1 nats-server
6
> nsc push --all
7
[ OK ] push to nats-server "nats://localhost:4222" using system account "SYS" user "sys":
8
[ OK ] push SYS to nats-server with nats account resolver:
9
[ OK ] pushed "SYS" to nats-server NBENVYIBPNQGYVP32Y3P6WLGBOISORNAZYHA6SCW6LTBE42ORTIQMWHX: jwt updated
10
[ OK ] pushed to a total of 1 nats-server
11
[ OK ] push TEST to nats-server with nats account resolver:
12
[ OK ] pushed "TEST" to nats-server NBENVYIBPNQGYVP32Y3P6WLGBOISORNAZYHA6SCW6LTBE42ORTIQMWHX: jwt updated
13
[ OK ] pushed to a total of 1 nats-server
Copied!
For the NATS resolver, each nats-server that responds will be listed. In case you get fewer responses than you have servers or a server reports an error, it is best practice to resolve this issue and retry. The NATS resolver will gossip missing JWTs in an eventually consistent way. Servers without a copy will perform a lookup from servers that do. If during an initial push only one server responds there is a window where this server goes down or worse, loses its disk. During that time the pushed account is not available to the network at large. Because of this, it is important to make sure that initially, more servers respond than what you are comfortable with losing in such a way at once.
Once the account is pushed, its user can be used:
1
> nats -s nats://localhost:4222 pub --creds=/DEMO/TEST/foo.creds "hello" "world"
2
16:50:51 Published 5 bytes to "hello"
3
>
Copied!

Setup User

Create/Edit Account - All Environments - All Deployment modes