Creating a NATS Super Cluster in Digital Ocean with Helm
Let's create a super cluster using NATS Gateways. First let's create 3 different clusters in NYC, Amsterdam, and San Francisco:
1
doctl kubernetes cluster create nats-k8s-nyc1 --count 3 --region nyc1
2
doctl kubernetes cluster create nats-k8s-sfo2 --count 3 --region sfo2
3
doctl kubernetes cluster create nats-k8s-ams3 --count 3 --region ams3
Copied!
Next, open up the firewall across the 3 regions to be able to access the client, leafnode and gateways ports:
1
for firewall in `doctl compute firewall list | tail -n 3 | awk '{print $1}'`; do
2
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:4222,address:0.0.0.0/0
3
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7422,address:0.0.0.0/0
4
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7522,address:0.0.0.0/0
5
done
Copied!
For this setup, we will create a super cluster using the external IPs from the nodes of the 3 clusters. For a production type of setup, it is recommended to use a DNS entry and an A record for each one of the servers.
1
for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
2
echo "name: $ctx"
3
for externalIP in `kubectl --context $ctx get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'`; do
4
echo "- nats://$externalIP:7522";
5
done
6
echo
7
done
Copied!
The Helm definition would look as follows for the 3 clusters:
1
# super-cluster.yaml
2
nats:
3
externalAccess: true
4
logging:
5
debug: false
6
trace: false
7
​
8
cluster:
9
enabled: true
10
​
11
gateway:
12
enabled: true
13
​
14
# NOTE: defined via --set gateway.name="$ctx"
15
# name: $ctx
16
​
17
gateways:
18
- name: do-ams3-nats-k8s-ams3
19
urls:
20
- nats://142.93.251.181:7522
21
- nats://161.35.12.245:7522
22
- nats://161.35.2.153:7522
23
​
24
- name: do-nyc1-nats-k8s-nyc1
25
urls:
26
- nats://142.93.251.181:7522
27
- nats://161.35.12.245:7522
28
- nats://161.35.2.153:7522
29
​
30
- name: do-sfo2-nats-k8s-sfo2
31
urls:
32
- nats://142.93.251.181:7522
33
- nats://161.35.12.245:7522
34
- nats://161.35.2.153:7522
35
​
36
natsbox:
37
enabled: true
Copied!
Let's deploy the super cluster with Helm using the name of cluster as the name of the gateway:
1
for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
2
helm --kube-context $ctx install nats nats/nats -f super-cluster.yaml --set gateway.name=$ctx
3
done
Copied!
That's it! It should now be possible to send some messages across regions:
1
# Start subscription in Amsterdam
2
nats-box:~# kubectl --context do-ams3-nats-k8s-ams3 exec -it nats-box -- /bin/sh -l
3
nats-box:~# nats sub -s nats hello
4
​
5
# Send messages from San Francisco region
6
nats-box:~# kubectl --context do-sfo2-nats-k8s-sfo2 exec -it nats-box -- /bin/sh -l
7
nats-box:~# nats pub -s nats hello 'Hello World!'
8
​
9
# From outside of k8s can use the external IPs
10
nats sub -s 142.93.251.181 hello
11
nats pub -s 161.35.2.153 hello 'Hello World!'
Copied!
Using leafnodes and NATS super clusters to communicate across regions
You can also create a multi-region NATS topology by using leafnodes connecting to a NATS super cluster (which could also be a much simpler way!).
1
doctl kubernetes cluster create nats-k8s-nyc1 --count 3 --region nyc1
2
doctl kubernetes cluster create nats-k8s-sfo2 --count 3 --region sfo2
3
doctl kubernetes cluster create nats-k8s-ams3 --count 3 --region ams3
Copied!
Next, open up the firewall across the 3 regions to be able to access the client, leafnode and gateways ports:
1
for firewall in `doctl compute firewall list | tail -n 3 | awk '{print $1}'`; do
2
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:4222,address:0.0.0.0/0
3
done
Copied!
The Helm definition would look as follows for the 3 clusters:
1
# nats.yaml
2
leafnodes:
3
enabled: true
4
remotes:
5
- url: tls://connect.ngs.global:7422
6
credentials:
7
secret:
8
name: ngs-creds
9
key: NGS.creds
10
​
11
natsbox:
12
enabled: true
Copied!
Let's deploy the super cluster with Helm using the name of cluster as the name of the gateway:
1
for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
2
kubectl --context $ctx create secret generic ngs-creds --from-file $HOME/.nkeys/creds/synadia/NGS/NGS.creds
3
helm --kube-context $ctx install nats nats/nats -f nats.yaml
4
done
Copied!
It should now be possible to send some messages across regions:
1
# Start subscription in Amsterdam
2
nats-box:~# kubectl --context do-ams3-nats-k8s-ams3 exec -it nats-box -- /bin/sh -l
3
nats-box:~# nats-sub -s nats hello
4
​
5
# Send messages from San Francisco region
6
nats-box:~# kubectl --context do-sfo2-nats-k8s-sfo2 exec -it nats-box -- /bin/sh -l
7
nats-box:~# nats-pub -s nats hello 'Hello World!'
Copied!
Or from outside of k8s can use the external IPs:
1
# Find the external ips from the nodes
2
$ for ctx in do-ams3-nats-k8s-ams3 do-nyc1-nats-k8s-nyc1 do-sfo2-nats-k8s-sfo2; do
3
kubectl --context $ctx get nodes -o wide
4
done
5
​
6
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
7
nats-k8s-ams3-default-pool-3cifn Ready <none> 38m v1.17.5 10.133.7.58 188.166.32.235 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
8
nats-k8s-ams3-default-pool-3ciq9 Ready <none> 38m v1.17.5 10.133.19.19 188.166.40.159 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
9
nats-k8s-ams3-default-pool-3ciqz Ready <none> 38m v1.17.5 10.133.31.211 188.166.43.34 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
10
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
11
nats-k8s-nyc1-default-pool-3ciq4 Ready <none> 37m v1.17.5 10.136.75.202 161.35.125.69 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
12
nats-k8s-nyc1-default-pool-3ciqh Ready <none> 38m v1.17.5 10.136.90.125 161.35.125.70 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
13
nats-k8s-nyc1-default-pool-3ciqk Ready <none> 37m v1.17.5 10.136.65.137 161.35.125.66 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
14
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
15
nats-k8s-sfo2-default-pool-3ciq0 Ready <none> 37m v1.17.5 10.138.20.132 206.189.79.122 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
16
nats-k8s-sfo2-default-pool-3ciqd Ready <none> 37m v1.17.5 10.138.4.194 64.225.124.243 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
17
nats-k8s-sfo2-default-pool-3ciqv Ready <none> 37m v1.17.5 10.138.148.237 206.189.79.131 Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64 docker://18.9.2
Copied!
Send a message from Amsterdam to SFO via the super cluster connected with leafnodes:
1
$ nats-sub -s 188.166.32.235 hello # From Amsterdam
2
$ nats-pub -s 206.189.79.131 hello world # To SFO
Copied!
Export as PDF
Copy link
Edit on GitHub