Replication and durability

Documentation home · Architecture · RWO ownership

Each volume generation has one Raft group and one elected serving coordinator. Workers store ordinary files, journal state and checkpoints on their local-CSI PVCs. Background file copying alone is not the replication protocol: ordering, retry identity, serving fences and verified replay establish authority.

Quorum and acknowledgement

No witness or automatic quorum reduction exists. A replica can discover/redirect to a coordinator, but cannot acknowledge conflicting independent writes. A Ready Pod or local leader flag is not sufficient authority.

A successful mutation means its command is durably committed to a majority's journals and applied by the serving coordinator. A follower may materialize that command later; its durable journal supplies replay data. Reads establish current leader/quorum authority and wait for the committed prefix to finish applying. The first operation in a leadership term establishes an application barrier.

fsync also synchronizes the serving derived object after verifying authority. It does not wait for all followers to materialize the file. Flush and close do not turn a failed write into success. Durability depends on the entire local filesystem, host and storage stack honoring synchronization; replication does not protect against loss of more durable copies than the majority model permits.

Retries and uncertain results

Requests bind the immutable volume/generation, an operation and a retry ID. Duplicate mutations return the saved result only for the same input. Normal clients generate an expiring identity with a one-minute deadline and reuse its serialized bytes across attempts within the call's 15-second budget.

A timeout, disconnect or storage error after submission can follow commitment. It is an uncertain result, not proof the write failed. Do not repeat an append or another non-idempotent mutation under a fresh identity merely because the caller saw an error. A serving worker can fail materialization after the command committed; journal replay is still authoritative after recovery.

The replicated monotonic expiry floor removes expired deduplication results without permitting an expired request to execute again after restart or clock regression. Checkpoints retain unexpired results and the floor. Legacy identities without expiry retain results until volume deletion and consume bounded metadata. Synchronized clocks are required for availability; expired identities return ESTALE rather than silently creating a different operation.

Backing files and restart

The backing PVC supplies an existing filesystem. The driver neither formats it nor provisions a second local-storage layer. The journal's stable term, member identity and checkpoint boundary remain authority even if derived files appear newer or larger. Never reset a journal beside existing files or copy another volume's backing tree into a live worker.

A materialization failure is sticky: the worker stops serving and withdraws so a healthy majority can elect a coordinator. Restart replays the valid authoritative prefix before service resumes. Persistent storage errors still need repair; a Running Pod does not establish successful replay. Checksums and identity checks fail closed on corrupt checkpoints, inconsistent boundaries and uncovered gaps.

Checkpoints and compaction

Current checkpoints capture namespace/object metadata, allocated sparse extents, retry state, expiry floors, durable handles, link relationships, RWO ownership and serving retirement state. The capture holds the FSM lock while copying a consistent working set into an immutable spool. Persistence/transfer can then overlap new work. Capture may pause file I/O; it is not a CSI snapshot API or a latency guarantee.

The driver uses the linked Raft library's default periodic checkpoint cadence and retains 16 trailing raw logs after local compaction. Journal/history pressure can request an earlier checkpoint before admitting further ordinary mutations. The default journal window is 256 MiB, configurable down to 16 MiB; this is a compaction/admission bound, not a lifetime write allowance or storage quota.

Older POC baselines retained full command history. Current checkpoints represent current state and transfer sparse allocated data rather than all historical file bytes. Their reader retains supported historical formats without promising a new format can be opened by an older runtime. Close descriptors and quiesce workloads before upgrades; active mixed versions and format downgrade are unsupported.

Transfer and rebuild

A complete checkpoint and each chunk have checksums. The receiver stores one fsynced partial stream bound to the complete-stream identity. The sender verifies its prefix before resuming; a different checkpoint or mismatched prefix resets the partial candidate. Only verified Raft snapshot installation may publish recovered authority. A partial file never authorizes service.

Transfer uses bounded chunk buffers, one outbound rebuild and a shared 32 MiB/s outbound limiter per worker. A receiver admits one rebuild actor. Current published checkpoints, partial receive state and capture/send spools all consume real disk space. Spool cleanup and journal page reuse do not imply that Bolt's high-water file allocation shrinks immediately.

Controlled member replacement

Replica count remains the creation-time target. Replacement requires a distinct eligible spare storage node, current quorum, a recorded learner identity and verified catch-up. The learner is promoted, the original member durably records its serving fence, and the old member is then removed. A temporary extra voter can increase the votes needed during transition; membership is never weakened just to make an unavailable group appear healthy.

Stable endpoint aliases and authenticated successor hints support clients whose original endpoints remain mounted. Retired workers persist their fence and cannot resume file service after restart. A permanently unreachable original cannot supply the required acknowledgement; replacement/reclamation remain blocked until original authority returns. Node deletion or name reuse is not a fence. See lifecycle and recovery.

Bounded failure behavior

Client/worker admission, encoded data, metadata, objects, extents, connections, queues and transfer concurrency are finite. Full disks and permission failures are reported separately from no quorum and invalid requests. Context cancellation cannot interrupt an already blocked kernel disk syscall. Local quorum/health checks therefore do not promise uninterrupted I/O or a successful-latency bound. See resource bounds and metrics interpretation.