Operator runbook

Documentation home · Administrator guide · Configuration · Lifecycle

Use this guide for existing volumes. Repair intent, application routing and storage reclamation are distinct operations. Preserve original ownership and quorum authority throughout an incident.

Select the cluster and tools

Use an explicitly selected, authorized kubeconfig for every operation. Context names alone are not unique across kubeconfig files. Confirm actual API endpoint, Node UIDs and storage placement before making changes.

CSI_KUBECONFIG=/absolute/path/to/authorized-kubeconfig
CSI_CONTEXT=your-context-name
kubectl --kubeconfig "$CSI_KUBECONFIG" --context "$CSI_CONTEXT" get nodes \
  -o custom-columns=NAME:.metadata.name,UID:.metadata.uid

The Python support/repair tools below run from a source checkout with Python and kubectl available; they are not inside the scratch role images. These tools accept --context, so explicitly bind their kubeconfig using KUBECONFIG on that invocation. This is operational tooling, not a build or test workflow.

Diagnostics

KUBECONFIG="$CSI_KUBECONFIG" python3 tools/support-bundle.py \
  --context "$CSI_CONTEXT" --namespace kube-system \
  --volume "$CSI_VOLUME" --output /private/path/incident-volume.json

Set CSI_VOLUME to the exact rv-… handle; omit --volume for the driver's allowlisted inventory. The exclusive mode-0600 output records source claim, generation, assignments, backing identities, ownership, aliases and cleanup state. It collects no Secrets, ConfigMap payloads, environments, file contents or application logs. Review infrastructure identity before sharing the file.

Record the running role image identities and build information. The current CSI vendor-version string still shows 3.2.0 for the 4.1.0 image release; it is not sufficient deployment provenance. See configuration. Correlate metrics and JSON logs by process epoch and volume/generation/request attributes. Do not print full Secrets or broad CI/environment dumps.

Unavailable or degraded volume

1. Establish configured membership and required majority. RF1 has no redundant storage and RF2 requires both current members.
2. Inspect actual worker readiness/restarts, leader/quorum observations, journal and local-CSI errors, backing bindings, space and original Node UIDs.
3. Restore the missing original dependency where possible. New Pod/Node names do not adopt an uncertain journal or prove an old writer stopped.
4. Interpret post-submission errors as uncertain results. Do not repeat a non-idempotent write under a new identity just to hide a timeout.
5. Keep configured replica count and quorum unchanged. A surviving majority may continue I/O while repair remains pending because no spare or fence exists.

Request controlled replacement

A cordon or explicit replacement request may initiate learner replacement. The original member must affirmatively persist its serving fence; permanently lost nodes cannot complete that contract. Provide a distinct Ready eligible local storage node and observe current quorum before requesting repair.

Set the volume, generation and replica variables from the original inventory, and keep one explicit request identity for the entire operation:

CSI_REPAIR_ID=your-change-record-id
KUBECONFIG="$CSI_KUBECONFIG" python3 tools/request-repair.py \
  --context "$CSI_CONTEXT" --namespace kube-system \
  --volume "$CSI_VOLUME" --generation "$CSI_GENERATION" \
  --operation replace --replica "$CSI_REPLICA" --request-id "$CSI_REPAIR_ID"

The default renders a UID/resource-version guarded patch without submitting it. Review the target and intent, then repeat the same command with --apply if the operation is authorized. A request accepted by Kubernetes does not mean catch-up, promotion, fencing or cleanup has finished. Do not edit member allocations or remove finalizers to complete it.

Lag or stalled rebuild

Local apply distance includes Raft metadata and is not bytes or elapsed time behind the coordinator. A disconnected replica cannot know the leader's newest commit index. Compare fresh worker/process identities and complete progress signals, not an isolated gauge or old leader flag.

Use checkpoint capture-active/duration, storage stage timing, committed/applied indices, verified transfer/resume bytes, readiness reasons, CPU/RSS and actual backing free space. Capture holds the FSM lock and may pause I/O. A completed send does not prove receiver publication. Cancellation cannot interrupt a stuck kernel disk syscall; restarting a worker does not repair the storage device. See observability and capacity.

Shared NODE runtime

This requires replicated CSI 4.1.0 and the local CSI 2.1.0 consumer extension. Deliver matching manager/NODE images, updated CRD and reviewed NODE/RBAC/mount configuration before selecting shared mode. No such rollout is included in the source change. --replica-runtime=shared selects new generations; existing legacy volumes keep their worker Pods. The manager retires only positively owned API-only controllers.

For an existing volume, stop application consumers and complete normal unpublish/unstage. Preserve the generation, backing claims, journal and original Secret. Render a guarded intent using the ordinary repair tool:

KUBECONFIG="$CSI_KUBECONFIG" python3 tools/request-repair.py \
  --context "$CSI_CONTEXT" --namespace kube-system \
  --volume "$CSI_VOLUME" --generation "$CSI_GENERATION" \
  --operation shared-runtime --request-id "$CSI_REPAIR_ID"

After review and separate operational authorization, repeat with --quiesced --apply. The acknowledgement records operator intent; it is not proof that a partitioned writer stopped. Normal legacy Pod deletion must finish on the original node, and local CSI rejects shared acquisition while another mount survives. Never force-delete workers or remove backing-use finalizers to make a transition progress. Resume consumers after shared workers report ready through their identity-bound EndpointSlices and ordinary mounting succeeds.

Inspect replicated_csi_host_workers{state=…}, correlated replica-host/worker logs, per-replica EndpointSlices and ReplicatedVolume status. The support bundle includes the runtime and shared endpoint observations without credential data. A healthy host Pod can contain an unavailable worker. Endpoint reachability permits Raft bootstrap and is distinct from worker quorum readiness.

ReplicaStateLost reports known missing/truncated retained replica authority; application-data extent remains unknown. It does not infer total file loss from an unavailable node or lost redundant copy. Preserve surviving journals and use normal diagnosis/backup recovery; never initialize an empty journal to clear the condition. A blocked kernel operation retains its publication until the worker really closes. Local CSI records and the backing-use finalizer protect cleanup across process crashes; restart with the same host state/mount roots to resume it.

Delete/Retain and uninstall still preserve original authority. For shared reclamation the node stops the worker, unpublishes through local CSI and records release on the exact PVC; only then does the manager remove its backing-use finalizer and request normal PVC/PV deletion. Retain keeps the backing claims. Keep the NODE/local-CSI cleanup components until those publications are released.

Mount recovery

Replacing only the CSI frontend can preserve a separate healthy owner. Losing the owner or host can leave a disconnected kernel FUSE mount. Stop affected application consumers and let kubelet perform normal unpublish/unstage before remounting. Retain exact volume/session identity and inspect kernel references.

A matching filesystem type, source volume, device/root and publication-reference check is required before teardown. Do not force/lazy-unmount foreign or busy paths, delete staging directories, or remove RWO session files to manufacture a clean state. An expired RWO epoch requires normal unstage/re-stage; it cannot silently reacquire ownership while stale references remain.

Cleanup blockers

Even after the volume fence commits, original local-storage authority can block reclamation. Keep the owned resources and cleanup record while blocked. API absence alone does not prove disk deletion. Permanent-node recovery/disposal is parked; external application-file restore into new authority does not dispose of the old volume. See disaster recovery.

Credentials and configuration changes

Expired volume certificates

volume leaf is expired or outside immutable trust occurs during local certificate validation, before a CSI client contacts a replica. It can mean an expired or not-yet-valid certificate, expired CA or invalid trust chain. Check public certificate validity dates and clock accuracy, exact volume/generation, Secret ownership, and bounded manager logs. Do not dump the Secret or private keys/tokens. A worker may reject the same credentials at startup even while an old ReplicatedVolume status still says Ready.

Read-only 2026-11-01 observations on TLSPU found expired leaves and repeated client rate-limit/deadline errors before credential renewal. The 4.0.0 manager shares a 30-second budget across the entire volume list; later volumes can be skipped on every pass. Release 4.0.1 runs credential renewal in an independent manager-leader loop with dedicated API clients/rate-limit budgets. It starts immediately, pauses one minute between passes, begins renewal with eighteen hours remaining, and continues after individual errors with a fifteen-second budget per volume. It does not wait for lifecycle reconciliation, controller setup, readiness or quorum. Leadership loss and shutdown still cancel it. Lifecycle work separately gets thirty seconds per volume and status reporting ten seconds, with existing credentials checked before PVC/PV bookkeeping. This change has not been deployed or observed running in that cluster.

Restore renewal through an authorized delivery of the correction, preserving the owned Secret, volume generation and backing claims. Existing renewal can issue a fresh leaf after expiry if the original CA remains valid, its matching signer is available and the prior overlap has ended. Then observe the Secret update, worker projection/restart recovery, correct status and normal mounting. A rotation annotation alone cannot fix a loop that never reaches renewal. Missing or invalid original trust requires deliberate recovery; recreating the auth Secret with an unrelated CA is not a mount repair.

Request early rotation

Normal reconciliation renews leaf/key and bearer credentials under the original CA. To request early rotation, choose one stable rotation request ID and render an intent with the same repair tool:

KUBECONFIG="$CSI_KUBECONFIG" python3 tools/request-repair.py \
  --context "$CSI_CONTEXT" --namespace kube-system \
  --volume "$CSI_VOLUME" --generation "$CSI_GENERATION" \
  --operation rotate-credentials --request-id "$CSI_ROTATION_ID"

Review before repeating with --apply. The tool does not read credential bytes. Observe completion, projection/refresh and eventual previous-token cutoff. A new rotation identity is refused while an overlap is active. Normal renewal does not revoke a compromised CA; isolate compromised authority and use a deliberate recovery procedure. See security.

DNS-domain or binary changes require a quiescent rollout and normal remount. Keep worker/node-controller settings consistent with the Orbit render. Use startup/readiness checks for operational state; do not kill a worker with a liveness probe merely because quorum or backing I/O is temporarily unavailable.

Uninstall order

Stop new provisioning, close consumers, choose normal Delete or deliberate Retain for every generation, and preserve the local driver and complete retained identity. Keep cleanup roles/RBAC/CRD/Secrets until ownership is resolved. Only then remove replicated runtime resources through GitOps. Follow the complete lifecycle sequence; deleting a namespace or CRD to hide blocked cleanup is not a supported uninstall procedure.