Volume lifecycle

Documentation home · Getting started · Operator runbook

Kubernetes records desired lifecycle intent and observed progress. File contents, request-result state and replication journals stay on backing filesystems. Lifecycle API objects are not a store for ordinary file writes.

Immutable volume authority

A namespaced ReplicatedVolume in the driver namespace owns generated resources and carries a cleanup finalizer. Its UID is the protocol generation.

Keep the original CRD schema, identity, trust and backing records through Retain and reinstall. Missing Kubernetes metadata is not authority to adopt another disk or generate a new CA for existing data.

Provisioning and reconciliation

The external-provisioner must supply original claim metadata. The manager checks that source identity, validates the backing class and chooses one through four distinct eligible local-storage nodes. It creates per-volume credentials, configuration, Services and ordinary backing PVCs. In shared mode, the NODE replica host supervises workers and publishes their EndpointSlices; legacy mode uses owned worker Pods. The manager performs API-only replica reconciliation and retires only positively owned legacy node-controller Pods. The local driver owns physical allocation, filesystem behavior and reclamation.

A backing node must have the expected Ready local-CSI Pod/service account and matching registration. The backing class uses the local driver, delayed binding and Delete reclaim. Application-only mount nodes do not need backing assignments. There is no raw-device discovery or implicit reduction when too few nodes qualify.

Creation records win over later retries: replica overrides are resolved against the original PVC at initial creation, then remain frozen. Stale schema pruning of RWO intent or a mismatch against the original source claim blocks allocation. Children are matched by immutable ownership/specification, not names alone. Foreign children are conflicts rather than adoption opportunities.

The effective replica count is returned in CSI volume context and eventually annotated on the outer PV under csi.replicated.tak-os.org/replicas. The manager matches the full source claim reference and volume handle and uses UID/resource- version guards; unrelated metadata and foreign PVs remain untouched.

A cancelled CSI request ends that wait, not the durable creation intent. Retry and elected reconciliation resume from recorded state. Dependency failures, invalid requests, quorum loss and storage/permission errors remain distinct. Manager/node-controller leadership uses Kubernetes Leases; storage serving safety still comes from committed volume/member fences.

Delete policy

Deletion first records idempotent intent, then establishes authoritative fencing. Once committed, the tombstone denies file operations, including replay of an old successful request. API endpoint removal, a missing worker Pod or a Node marked NotReady does not prove that an isolated writer stopped.

Before any worker Pod can be created, an activation marker is persisted. A never-activated cancelled creation can clean up without a nonexistent quorum. An activated generation cannot be reclassified as never used simply because its workers disappeared; it may need worker recovery to establish the deletion fence.

After fencing, cleanup still verifies original Node UID/readiness, local-driver identity and recorded backing bindings. It normally stops owned workers, deletes only matching backing claims, waits for the recorded PVs/local-CSI reclamation, and removes remaining owned metadata. A resource with another finalizer remains a blocker. The manager removes its own volume cleanup finalizer only after its lifecycle conditions are complete; it never strips child protection finalizers.

A lost original storage node can block Delete indefinitely. A missing PV or an accepted API DELETE alone does not prove physical reclamation.

Retain and ambiguous orphans

An outer PV using Retain normally causes no CSI DeleteVolume request when its claim is released. Preserve its ReplicatedVolume, trust, assignments and backing storage. Do not garbage-collect that authority merely because its source PVC/PV is gone. Retain is not an external backup.

An explicit later administrative DeleteVolume request is destructive intent; the controller also honors deletion of the original ReplicatedVolume through its normal cleanup finalizer. Retain does not make those operations harmless. Establish exact ownership and close consumers before normal disposal. Do not make direct backing PVC removal, allocation edits or finalizer removal the disposal procedure.

Only never-exposed creation records whose exact source UID is gone and which have no published PV are automatically eligible for conservative cancellation. Exposed generations need explicit deletion even when outer metadata disappears: that absence could represent removed retained metadata.

Uninstall and reinstall

1. Stop new provisioning and application writers through the managed deployment.
2. Close application consumers and complete normal CSI unpublish/unstage.
3. Choose normal Delete or deliberate Retain for each exact volume generation.
4. Keep manager, NODE replica hosts, CSI mount components, remaining legacy cleanup components, RBAC, CRD, Secrets and local CSI installed while Delete cleanup or Retain preservation is unresolved.
5. Remove replicated runtime components only when owned volumes are reclaimed or their complete retained authority is deliberately preserved for reinstall.
6. Keep the external local driver and retained storage intact. Reinstall with compatible schema, original identities/trust and the documented role settings.

Do not delete the CRD to hide blocked volumes. Same names without original UIDs are not a retained reinstall. A versioned historical cleanup observation is not proof of a different deployment's disk state.

Operational visibility

Status and transition-only Events identify blocked lifecycle stages. Structured logs carry component, operation, volume/generation, replica/node, correlation, outcome, duration and error category. Metrics aggregate inventory freshness, desired/ready counts and bounded cleanup reasons without paths or per-volume labels. Follow the runbook for diagnosis and guarded repair.

Shared-worker ownership in 4.1.0

The shared runtime preserves each backing local-CSI PVC and its immutable claim/PV/node binding. The manager marks its selected node and protects it with replicated.csi.tak-os.org/backing-use; no worker Pod is required to trigger binding. The NODE host uses exclusive local-CSI consumer publication and records initialization before publishing endpoints. Original volume Service names, trust, Raft identities and journals survive an explicitly requested legacy transition. See shared runtime operations.

Retirement/fenced deletion stops the corresponding worker, completes CSI unpublication and records a generation/replica-bound release on the PVC. The manager then removes its own finalizer and lets the original local provisioner perform normal reclamation. It does not clear unrelated finalizers. An exposed volume still requires its durable volume fence. A never-exposed shared creation can be cancelled without bootstrapping a quorum solely for deletion; worker stop and backing release remain required. Retain preserves storage and authority. API absence or cancellation alone never authorizes removal of uncertain data.