Persistence

Where data is stored

Everything NetlyDB persists lives under StorageRoot, a directory you choose. It is resolved once at startup, so two databases in one process cannot share files. The default storage provider is the local file system; see Storage providers.

The write-ahead log

Every write travels one pipeline - admission, a bounded queue, a log sequence number, the apply, and a durable commit. The write-ahead log (WAL) is framed, versioned, checksummed and ordered, and each record carries the identity of the mutation that produced it. Concurrent writers waiting at the same moment share one flush to disk (group commit).

Each collection has one WAL file. It is not segmented or rotated; it is compacted when a new snapshot is published.

Snapshot generations

A snapshot is a generation: it is published atomically next to the previous one, never written over it. Recovery replays the WAL over the last published generation. SnapshotRetention sets how many superseded generations are kept (PreviousGenerations, default 1) and for how long (MaxAge).

Snapshots are per collection. There is no cross-collection point-in-time snapshot.

Data classes

How aggressively an unreadable artifact may be discarded and rebuilt depends on how replaceable the data is. DataClasses declares that per collection: Critical, Reconstructable, Regenerable or Disposable. Unclassified collections are treated as Critical, and ambiguity in a critical collection stops writable startup rather than discarding anything.

builder.Services.AddNetlyDB(o =>
{
    o.Durability = DurabilityMode.Synchronous;  // acknowledge after the durable commit (default)
    o.CommitTimeout = TimeSpan.FromSeconds(30); // how long a caller waits for it

    o.SnapshotRetention.PreviousGenerations = 1;      // superseded generations to keep
    o.DataClasses.Default = DataClass.Critical;       // how replaceable this store's data is
});

Compression at rest

Optional, transparent compression of snapshots, the write-ahead log and vector data. It composes with encryption (compress, then encrypt), and reads always decompress automatically, so data stays readable after you turn compression off. File-collection content is always stored compressed.

builder.Services.AddNetlyDB(o =>
{
    o.Compression.Enabled = true;                        // compress new writes at rest
    o.Compression.Algorithm = CompressionAlgorithm.GZip; // or Brotli (smaller, more CPU)
    o.Compression.MinimumSizeBytes = 1024;               // store tiny payloads uncompressed
});

How writes are acknowledged, and what recovery does at startup, is on Durability & recovery.