CNCF and Synadia Align on Securing the Future of the NATS.io Project. Read the joint press release.
NATS Docs
NATS.ioNATS by ExampleGitHubSlackTwitter
  • Welcome
  • Release Notes
    • What's New!
      • NATS 2.11
      • NATS 2.10
      • NATS 2.2
      • NATS 2.0
  • NATS Concepts
    • Overview
      • Compare NATS
    • What is NATS
      • Walkthrough Setup
    • Subject-Based Messaging
    • Core NATS
      • Publish-Subscribe
        • Pub/Sub Walkthrough
      • Request-Reply
        • Request-Reply Walkthrough
      • Queue Groups
        • Queueing Walkthrough
    • JetStream
      • Streams
      • Source and Mirror Streams
        • Example
      • Consumers
        • Example
      • JetStream Walkthrough
      • Key/Value Store
        • Key/Value Store Walkthrough
      • Object Store
        • Object Store Walkthrough
      • Headers
    • Subject Mapping and Partitioning
    • NATS Service Infrastructure
      • NATS Adaptive Deployment Architectures
    • Security
    • Connectivity
  • Using NATS
    • NATS Tools
      • nats
        • nats bench
      • nk
      • nsc
        • Basics
        • Streams
        • Services
        • Signing Keys
        • Revocation
        • Managed Operators
      • nats-top
        • Tutorial
    • Developing With NATS
      • Anatomy of a NATS application
      • Connecting
        • Connecting to the Default Server
        • Connecting to a Specific Server
        • Connecting to a Cluster
        • Connection Name
        • Authenticating with a User and Password
        • Authenticating with a Token
        • Authenticating with an NKey
        • Authenticating with a Credentials File
        • Encrypting Connections with TLS
        • Setting a Connect Timeout
        • Ping/Pong Protocol
        • Turning Off Echo'd Messages
        • Miscellaneous functionalities
        • Automatic Reconnections
          • Disabling Reconnect
          • Set the Number of Reconnect Attempts
          • Avoiding the Thundering Herd
          • Pausing Between Reconnect Attempts
          • Listening for Reconnect Events
          • Buffering Messages During Reconnect Attempts
        • Monitoring the Connection
          • Listen for Connection Events
          • Slow Consumers
      • Receiving Messages
        • Synchronous Subscriptions
        • Asynchronous Subscriptions
        • Unsubscribing
        • Unsubscribing After N Messages
        • Replying to a Message
        • Wildcard Subscriptions
        • Queue Subscriptions
        • Draining Messages Before Disconnect
        • Receiving Structured Data
      • Sending Messages
        • Including a Reply Subject
        • Request-Reply Semantics
        • Caches, Flush and Ping
        • Sending Structured Data
      • Building Services
      • JetStream
        • JetStream Model Deep Dive
        • Managing Streams and consumers
        • Consumer Details
        • Publishing to Streams
        • Using the Key/Value Store
        • Using the Object Store
      • Tutorials
        • Advanced Connect and Custom Dialer in Go
    • Running Workloads on NATS
      • Getting Started
        • Installing Nex
        • Building a Service
        • Starting a Node
        • Deploying Services
        • Building a Function
        • Deploying Functions
      • Host Services
        • Javascript | V8
      • Nex Internals
        • Architecture Overview
        • Node Process
        • Nex Agent
        • No Sandbox Mode
        • Root File System
        • Control Interface
      • FAQ
  • Running a NATS service
    • Installing, running and deploying a NATS Server
      • Installing a NATS Server
      • Running and deploying a NATS Server
      • Windows Service
      • Flags
    • Environmental considerations
    • NATS and Docker
      • Tutorial
      • Docker Swarm
      • Python and NGS Running in Docker
      • JetStream
      • NGS Leaf Nodes
    • NATS and Kubernetes
    • NATS Server Clients
    • Configuring NATS Server
      • Configuring JetStream
        • Configuration Management
          • NATS Admin CLI
          • Terraform
          • GitHub Actions
          • Kubernetes Controller
      • Clustering
        • Clustering Configuration
        • v2 Routes
        • JetStream Clustering
          • Administration
          • Troubleshooting
      • Super-cluster with Gateways
        • Configuration
      • Leaf Nodes
        • Configuration
        • JetStream on Leaf Nodes
      • Securing NATS
        • Enabling TLS
        • Authentication
          • Tokens
          • Username/Password
          • TLS Authentication
            • TLS Authentication in clusters
          • NKeys
          • Authentication Timeout
          • Decentralized JWT Authentication/Authorization
            • Account lookup using Resolver
            • Memory Resolver Tutorial
            • Mixed Authentication/Authorization Setup
        • Authorization
        • Multi Tenancy using Accounts
        • OCSP Stapling
        • Auth Callout
      • Logging
      • Enabling Monitoring
      • MQTT
        • Configuration
      • Configuring Subject Mapping
      • System Events
        • System Events & Decentralized JWT Tutorial
      • WebSocket
        • Configuration
    • Managing and Monitoring your NATS Server Infrastructure
      • Monitoring
        • Monitoring JetStream
      • Managing JetStream
        • Account Information
        • Naming Streams, Consumers, and Accounts
        • Streams
        • Consumers
        • Data Replication
        • Disaster Recovery
        • Encryption at Rest
      • Managing JWT Security
        • In Depth JWT Guide
      • Upgrading a Cluster
      • Slow Consumers
      • Signals
      • Lame Duck Mode
      • Profiling
  • Reference
    • FAQ
    • NATS Protocols
      • Protocol Demo
      • Client Protocol
        • Developing a Client
      • NATS Cluster Protocol
      • JetStream wire API Reference
    • Roadmap
    • Contributing
  • Legacy
    • nats-account-server
Powered by GitBook
On this page
  • Via the NATS CLI
  • Via the Profiling Port

Was this helpful?

Edit on GitHub
Export as PDF
  1. Running a NATS service
  2. Managing and Monitoring your NATS Server Infrastructure

Profiling

PreviousLame Duck ModeNextFAQ

Last updated 27 days ago

Was this helpful?

When investigating and debugging a performance issue with the NATS Server (i.e. unexpectedly high CPU or RAM utilisation), it may be necessary for you to collect and provide profiles from your deployment for troublshooting. These profiles are crucial to understand where CPU time and memory are being spent.

Note that profiling is an advanced operation for development purposes only. Server operators should use the instead for monitoring day-to-day runtime statistics.

Via the NATS CLI

The NATS CLI can request profiles from the NATS Server when connected to the system account only. Profiles will be written out to the current working directory by default as files, which can then either be sent onwards or inspected using .

Memory profile

The --name, --tags and --cluster selectors can be used either individually or combined in order to request profiles from specific servers. Memory profiles are returned instantly. Examples include:

Command
Description

nats server request profile allocs

Request a memory profile from all servers in the system

nats server request profile allocs ./profiles

Request a memory profile from all servers in the system and write to the profiles directory

nats server request profile allocs --name=servername1

Request a memory profile from servername1 only

nats server request profile allocs --tags=aws

Request a memory profile from all servers tagged as aws

nats server request profile allocs --cluster=aws-useast2

Request a memory profile from all servers in the cluster named aws-useast2 only

CPU profile

The --name, --tags and --cluster selectors can be used either individually or combined in order to request profiles from specific servers. The --timeout option can also be provided as a means of specifying how long the CPU profile should run for. The default is 5 seconds. Examples include:

Command
Description

nats server request profile cpu

Request a CPU profile from all servers in the system

nats server request profile cpu ./profiles

Request a CPU profile from all servers in the system and write to the profiles directory

nats server request profile cpu --timeout=10s

Request a CPU profile from all servers in the system over a 10 second period

nats server request profile cpu --name=servername1

Request a CPU profile from servername1 only

nats server request profile cpu --tags=aws

Request a CPU profile from all servers tagged as aws

nats server request profile cpu --cluster=aws-useast2

Request a CPU profile from all servers in the cluster named aws-useast2 only

Via the Profiling Port

nats-server does not have authentication/authorization for the profiling endpoint. When you plan to open your nats-server to the internet make sure to not expose the profiling port as well. By default, profiling binds to every interface 0.0.0.0 so consider setting profiling to localhost or have appropriate firewall rules.

The NATS Server can expose a HTTP pprof profiling port, although it must be enabled by setting the prof_port in your NATS Server configuration file. Note that the profiling port is not authenticated and should not be exposed to clients, to the internet etc. For example, to enable the profiling port on TCP/65432:

prof_port = 65432

Memory profile

http://localhost:65432/debug/pprof/allocs

This endpoint will return instantly.

For example, to download an allocation profile from NATS running on the same machine:

curl -o mem.prof http://localhost:65432/debug/pprof/allocs

The profile will be saved into mem.prof.

CPU profile

http://localhost:65432/debug/pprof/profile?seconds=30

This endpoint will block for the specified duration and then return. You can specify a different duration by adjusting ?seconds= in the URL if you want to sample a shorter or longer period of time.

For example, to download a CPU profile from NATS running on the same machine with a 30 second window:

curl -o cpu.prof http://localhost:65432/debug/pprof/profile?seconds=30

The profile will be saved into cpu.prof.

Note that this option does not support , so the server must be restarted for the config change to take effect.

Once the profiling port has been enabled, you can download profiles as per the following sections. These profiles can be inspected using .

To see all available profiles, open

monitoring port
go tool pprof
go tool pprof
http://localhost:65432/debug/pprof/
configuration-reloading