Architecture

Documentation home · Replication · Lifecycle

The driver combines Kubernetes lifecycle management with a direct replicated filesystem protocol. Existing file operations go between a FUSE mount owner and replica workers; neither Kubernetes objects nor the manager are a per-write log.

Components

The following describes the 4.1.0 shared runtime, awaiting staging observations; no deployment is included in the release. Release 4.0.1 still uses generated node-controller and replica Pods.

The replica host is an additional container/process in the existing NODE DaemonSet Pod, with separate health/metrics and its own API client. It needs no additional per-node Pod. The standalone replica image remains for legacy volumes. CSI provisioner/registrar sidecars retain their usual lifecycle roles.

Pod footprint and resource bounds

For N storage nodes and replica counts R(v), release 4.0.1 generates N + sum(R(v)) controller/worker Pods. The 4.1.0 manager generates neither kind for new shared volumes; worker count increases in-process state instead. Individual local-CSI backing PVCs remain sum(R(v)). RF1 follows the same implementation and has no redundant copy. Legacy volumes keep their workers until an explicit transition, so deploying new code alone does not remove them.

The manager normally provisions new generations with --replica-runtime=shared; pod remains an explicit legacy option. Shared placement requires a ready replica-host container on an eligible local-storage node. The manager retires owned old API-only controller Pods and performs allocation itself.

The default node limit is 128 workers, four concurrent starts, 32 shared HTTP operations, two checkpoint captures and two outbound rebuilds. Outbound rebuilds share 32 MiB/s. Per-worker request/retry limits still apply. Exceeding admission returns an explicit pending/unavailable condition, not a smaller replica count. These are resource bounds, not storage quotas or measured capacity guarantees. The source NODE patch starts at 100m CPU / 128 MiB requested and 2 CPU / 768 MiB limited per replica host; operators must size that shared budget for the working set.

Storage and routing contract

The manager owns PVC allocation, selected-node binding and lifecycle intent. The NODE host owns local publication and worker-specific EndpointSlices. Each replica keeps its PVC, journal, generation and original Service DNS identity. Selectorless Services route to independent listener ports on the NODE Pod, so TLS roots, SANs and bearer credentials stay per-volume. There is no shared CA. Endpoints stay reachable for Raft before volume quorum readiness; health is a separate bounded runtime observation and the worker still verifies authority.

Local CSI must enable --consumer-mount-root and expose the same host directory with bidirectional propagation. The NODE host consumes it with HostToContainer propagation. Publication beneath that root is exclusive, recorded durably by local CSI before mounting, and blocks ordinary publication and deletion until normal unpublication completes. The host waits for the bind to appear in its own mount namespace before reading authority or initializing a worker. Dedicated publication locks also exclude an overlapping supervisor given a different private state root. No synthetic kubelet Pod paths or direct disk provisioning are used. Ordinary Pod volume lists cannot be extended in place; see Kubernetes Pod updates.

The host retains PVC/PV/node identity and initialization history on persistent host storage, takes process/publication locks, and protects active claims with a backing-use finalizer. Workers close before unpublication; the manager waits for an identity-bound release acknowledgement before removing that finalizer and requesting normal reclamation. API uncertainty leaves storage preserved.

Transition and failure policy

A generation-guarded shared-runtime request requires quiesced consumers and normal unpublish/unstage. The manager normally deletes legacy workers and waits for their absence on the original Ready node. Local CSI then affirmatively rejects acquisition while any other mount survives. The runtime selection is one-way; bindings, original trust, Raft identities and backing data remain intact. See the transition procedure.

The operator accepts shared process failures and explicit recovery to reduce footprint on 4–16 GB / 4–16 vCPU VM nodes; see the operating scope. A whole NODE Pod replacement also loses FUSE connections even with separate containers. Quotas and enterprise isolation remain outside this consolidation.

Worker failures stay local where possible; OOM/panic can still stop the shared process. A recorded replica is never freshly initialized when retained authority is missing. Missing/truncated retained journal or identity state is reported as ReplicaStateLost with application-data extent explicitly unknown. This is a known loss of replica authority, not a claim that every application byte is lost. Node/API absence reports unavailability or uncertainty, not confirmed data loss.

Data and control paths

flowchart LR
    K[Kubernetes API] --> M[Manager]
    M --> H[NODE replica host]
    H --> W[Independent replica workers]
    A[Application / kernel] --> F[FUSE mount owner]
    N[Kubelet / CSI frontend] --> F
    F --> W
    W <--> P[Raft peers]
    W --> V[Ordinary local-CSI PVC]

The manager supplies lifecycle and trust information. Workers establish actual quorum authority for reads, mutations, membership transitions and fences. Authenticated clients may discover a new coordinator without waiting for the manager to rewrite an application mount. Credentials are refreshed through a background API-dependent path; sufficiently long API outages can eventually prevent refresh and deny service.

Identity and authority

Kubernetes records desired lifecycle intent. The journal/checkpoint chain and current quorum determine authoritative file state. A Running Pod, largest file, newest timestamp or absent PV is not a substitute for those authorities.

Provisioning flow

The provisioner supplies the original PVC metadata. The manager records immutable access mode, count, requested capacity and backing class, chooses distinct eligible local-storage nodes, and creates trust/configuration, backing claims and owned workers. A successful creation exposes one filesystem volume, not separate application-visible copies. A backing replica count of one follows the same implementation and has no redundant copy.

File operation flow

The application kernel sends a FUSE request to the mount owner. The client keeps one serialized operation/retry identity across endpoint changes. The elected worker coordinator establishes current quorum, orders mutations in Raft, and applies committed state to ordinary backing files. Reads wait for authority and applied state. Other members catch up from ordered journal entries or a verified checkpoint and tail. An unsuccessful call can still have committed; see uncertain results.

Failure boundaries

A surviving replica majority may elect a coordinator and continue I/O. Losing quorum returns errors rather than silently weakening membership. Replacing the CSI frontend can retain a surviving mount owner's connections. Losing that owner, the kernel or the host requires normal unpublish/unstage/remount; transparent open-descriptor recovery is not promised. Permanently lost original-node repair and reclamation remain blocked under the current contract.