Last updated
Was this helpful?
Last updated
Was this helpful?
Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store.
Here we have 2 main streams - ORDERS and RETURNS - these streams are clustered across 3 nodes. These Streams have short retention periods and are memory based.
We create a ARCHIVE stream that has 2 sources set, the ARCHIVE will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly.
Finally, we create a REPORT stream mirrored from ARCHIVE that is not clustered and retains data for a month. The REPORT Stream does not listen for any incoming messages, it can only consume data from ARCHIVE.
A mirror copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A mirror does not listen on a subject for any data to be added. A mirror can filter by subject and the Start Sequence and Start Time can be set. A stream can only have 1 mirror and if it is a mirror it cannot also have any source.
A source is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result.
A Stream with sources may also listen on subjects, but could have no listening subject. When using the nats
CLI to create sourced streams use --subjects
to supply subjects to listen on.
A source can have Start Time or Start Sequence and can filter by a subject.
The ORDERS and RETURNS streams as normal, I will not show how to create them.
We now add the ARCHIVE:
And we add the REPORT:
When configured we'll see some additional information in a nats stream info
output:
Output extract
Here the Lag
is how far behind we were reported as being last time we saw a message.
We can confirm all our setup using a nats stream report
:
We then create some data in both ORDERS and RETURNS:
We can now see from a Stream Report that the data has been replicated:
Here we also pass the --dot replication.dot
argument that writes a GraphViz format map of the replication setup.