Clustering & replication

Enterprise Across machines since 1.1

How replication works

JoinCluster(seedNodes) makes a node a cluster member. Replication is peer-to-peer: there is no primary, and every node accepts writes. Each node holds a full copy of the replicated collections - there is no sharding across nodes. Changes are delivered asynchronously and every node converges on the same state.

builder.Services
    .AddNetlyDB(options =>
    {
        options.SelfAddress = "10.0.0.11";               // how peers reach this node
        options.Port = 9024;                             // control channel (replication uses GrpcPort, default 9025)
        options.Security.ClusterDatabaseId = databaseId; // the same on every member
        options.Security.ClusterSecret = clusterSecret;  // pre-shared; required
        options.Security.EnableClusterTls = true;        // required on a non-loopback address
        options.Security.CertificatePath = "/etc/netlydb/node.pfx";
    })
    .AddNetlyDBCollection<Order>()
    .JoinCluster("10.0.0.12", "10.0.0.13");

What replication guarantees

• Convergence: given time and connectivity, every node holding a collection converges on the same contents.
• Per-collection ordering from one source: changes made on one node to one collection arrive in order.
• Recognised redelivery: every change carries its provenance, so a change delivered twice is recognised as the same change.
• Progress survives restart: receivers remember what they applied; a durable outbox knows what is still owed.
• Deletes replicate as retained tombstones.
• Read-your-writes on request: a write can return a token that a later read on another node waits for.

What it does not guarantee

• Concurrent updates to one document on two nodes lose one of the two values, silently. Conflicts resolve last-write-wins under a deterministic order that every node applies the same way.
• A local write receipt says nothing about other nodes. There is no synchronous or quorum replication.
• No atomicity across collections or nodes; no global ordering; no linearizable reads across nodes.
• No consensus-based membership, and no automatic failover.
• File collections resolve conflicts on wall-clock time, without clock-skew hardening.
• A local truncate is not replicated, and vectors in external vector indexes are not replicated.
• Replication is not a backup: a delete or bad write replicates like any other change.
• All nodes must run the same replication protocol version; there is no rolling upgrade.
• Rebuilding a replica from a peer is not exposed as an operator command in the current release.

Security

• Peers authenticate with a pre-shared Security.ClusterSecret, required unless explicitly disabled for in-process test clusters.
• Security.ClusterDatabaseId, set to the same value on every intended member, makes a node refuse a peer from a different database instead of merging two stores that share collection names.
• From 1.1, a node listening on anything other than loopback refuses to start without TLS (Security.EnableClusterTls), which then covers both the control and replication channels. Without TLS, anything on the network path can read replicated data and the cluster secret.

Replication and the engine's state

From 1.2.0, replication follows the engine's lifecycle. A node pauses outbound and inbound replication while its engine is starting, recovering or requires recovery, starts no new replication work while shutting down, and keeps replicating while degraded. A change a peer sends while the node is paused is refused and applied once the node is writable again. A node also reports its own transport impairment as Degraded, which its health probes show.