Administrator guide

TAK-OS Local CSI is a Linux directory-backed CSI driver named csi.local.tak-os.org. It does not provision disks, format filesystems, replicate data, or relocate volumes after node failure. This guide covers fresh installation and operation of the source in this checkout. See the documentation index for release and image-version caveats.

Architecture and storage contract

One DaemonSet pod per storage node contains driver, provisioner, and registrar containers. There is no central controller Deployment. Each driver serves both controller and node RPCs for its own node. The external provisioner uses node deployment mode with delayed binding and strict topology.

1. A WaitForFirstConsumer claim waits for scheduler selection of a node.
2. That node's provisioner calls CreateVolume with the selected topology.
3. The driver persists ownership metadata and creates a data directory.
4. Returned topology becomes the PV's required node affinity.
5. Kubelet publishes a bind mount only after the driver validates ownership and existing data. Publishing never allocates a missing data directory.

The topology key is csi.local.tak-os.org/node. Its value is the first 16 bytes of the SHA-256 hash of the Kubernetes node name, encoded as 32 hexadecimal characters. Kubelet sets the topology label through registration and NodeGetInfo. Preserve node names; neither relabeling a different node nor editing PV affinity transfers data ownership.

Prepare and install

Production manifests live in tak-os-orbit-deployments/tak-os, under default/100-Local-CSI. Production rollout uses the TAK-OS Orbit Deploy pipeline.

Before rollout:

1. Prepare the intended persistent filesystem on each eligible Linux node. Mount any separate disk before starting the driver and arrange boot/service ordering so the driver cannot start against an unmounted directory.
2. On a fresh node, create /var/lib/tak-os/local-volumes as a root-owned real directory, for example sudo install -d -o root -g root -m 0750 /var/lib/tak-os/local-volumes. Do not clear an existing store during an update. Parent paths must not contain symlinks. The store and .metadata must belong to the driver user and must not be writable by group or others.
3. Verify that the kubelet backing mount supports shared propagation. The kubelet mount in the driver uses Bidirectional; the data mount uses None. A directory's existence alone does not prove that the intended disk is mounted.
4. Confirm admission policy permits the privileged driver and its hostPath mounts. Keep its socket, storage parents, and optional consumer root under trusted administrator control.
5. Review node eligibility. The supplied DaemonSet selects all Linux nodes and tolerates all taints, including control-plane taints. Narrow its selector if only some nodes should provide storage, and prepare the disk on every selected node.
6. Decide whether tak-os-local should be the default class. The supplied manifest marks it as default; remove that annotation if inappropriate.

For a new image, build and publish an explicitly chosen release tag from the repository root (replace YOUR_REGISTRY and YOUR_VERSION):

docker build -t YOUR_REGISTRY/tak-os-local-csi:YOUR_VERSION .
docker push YOUR_REGISTRY/tak-os-local-csi:YOUR_VERSION

Set images.tak-os-local-csi in the external deployment's deployment.json to the published image, then use the established Orbit pipeline to render and apply the deployment. Review its sidecar images and arguments together. The supplied image map pins external-provisioner v5.3.0 and node-driver-registrar v2.13.0. The deployment supplies:

• ServiceAccount and RBAC for PV/PVC, node, event, registration, and capacity work.
• CSIDriver with attachRequired: false, storageCapacity: true, and persistent lifecycle mode.
• DaemonSet tak-os-local-csi in kube-system, with a Unix socket shared by all three containers and the registrar's kubelet registration directory.
• StorageClass tak-os-local with WaitForFirstConsumer, Delete, and allowVolumeExpansion: false.

Do not replace distributed provisioning with immediate binding or add a central controller instance pointing at a node's store.

Driver configuration

For a nonstandard kubelet root, change the driver flag, kubelet hostPath and mountPath, plugin socket hostPath, registrar registration path, and plugin registry hostPath together. The default host registration socket is /var/lib/kubelet/plugins/csi.local.tak-os.org/csi.sock; the driver and sidecars see it at /csi/csi.sock.

Changing --volume-path on an existing installation requires a coordinated offline move of the entire store and mount configuration. Changing node identity is not an adoption or recovery mechanism.

StorageClasses and retention

Use the Retain example to provide a separate class without changing the default:

kubectl apply -f docs/examples/storageclass-retain.yaml

New PVCs opt in with storageClassName: tak-os-local-retain. A StorageClass's reclaim policy is copied to a PV at provisioning time; creating this class does not protect existing PVs. To preserve an existing volume before claim deletion, identify and patch its PV:

pv=$(kubectl -n APP_NAMESPACE get pvc CLAIM_NAME -o jsonpath='{.spec.volumeName}')
kubectl patch pv "$pv" --type=merge -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
kubectl get pv "$pv" -o yaml

Replace the namespace and claim placeholders first. Verify the policy before deleting the claim. A retained PV and its directory need deliberate administrator recovery or disposal. Automatic adoption of arbitrary directories is unsupported.

If defining additional classes, keep the same provisioner, delayed binding, and disabled expansion. Optional mountOptions may contain only the supported flags. Do not set fsType, disk-format options, or driver-specific parameters.

Optional trusted consumer publications

The source includes an opt-in extension for trusted node services, used by the replicated CSI shared NODE runtime. Local CSI volumes remain node-local and unreplicated. This extension publishes an already provisioned local volume; it does not allocate raw disks or replace its PVC/PV ownership.

The base deployment does not enable it. To integrate it in the external deployment repository:

1. Use an image containing the extension; confirm release pipeline publication.
2. Configure --consumer-mount-root to an absolute, clean, persistent dedicated host path, for example /var/lib/tak-os/local-csi-consumers. It must be outside both storage and kubelet roots, and must not contain either root. Symlinks and / are rejected.
3. Mount that path at the identical host/container path in local CSI with Bidirectional propagation. Its backing mount must support propagation. The trusted replica host consumes that path with HostToContainer propagation.
4. Keep the original root configured and available until all publications are normally unpublished. Limit access to the privileged consumer and its CSI socket.

Accepted targets have this exact structure:

<consumer-root>/<generation>/<replica>/mount

generation is 1–128 ASCII letters, digits, or hyphens. replica is r0 through r3, optionally followed by a hyphen and 12 lowercase hexadecimal characters. The service must use the original local volume handle and node identity through CSI NodePublishVolume and NodeUnpublishVolume; ordinary workloads keep using kubelet paths.

Before mounting, the driver durably records exclusive ownership as .metadata/<volume-id>.consumer.json. A consumer cannot take over a volume with another surviving mount. While claimed, a different consumer target, ordinary kubelet publication, and deletion are blocked. Retries for the original target can resume after a crash. Failed publication can leave ownership recorded; normal unpublication for the original target releases it even if no bind exists.

To transition ownership, stop the current user and complete normal unpublication before publishing to the new target. Do not remove ownership files to bypass a conflict. If the configured root changes or disappears, restore the original configuration and inspect mounts before retrying. Shared-runtime migration, recovery, and resource consumption still require staging qualification.

Verify installation and monitor

kubectl -n kube-system rollout status daemonset/tak-os-local-csi --timeout=240s
kubectl -n kube-system get pods -l app=tak-os-local-csi -o wide
kubectl get csidriver csi.local.tak-os.org -o yaml
kubectl get csinodes
kubectl get nodes -L csi.local.tak-os.org/node
kubectl -n kube-system get csistoragecapacities
kubectl get storageclass tak-os-local -o yaml

Check each eligible node's CSINode for this driver and topology key. Use the user guide to verify provisioning, permissions, and persistence in an isolated namespace. Inspect the resulting PV's node affinity against its owner's topology label.

The driver listens on TCP port 9898:

Counters increment only when --consumer-mount-root is enabled. They count all publish/unpublish requests in that mode, including regular kubelet targets, and reset on driver restart. JSON diagnostics include operation, volume, node, outcome, duration, request ID, and errors. The registrar separately exposes registration health on port 9809. The manifests do not install a metrics Service or scraper configuration.

For a chosen driver pod, inspect each component and optionally forward metrics:

kubectl -n kube-system logs DRIVER_POD -c driver --tail=100
kubectl -n kube-system logs DRIVER_POD -c provisioner --tail=100
kubectl -n kube-system logs DRIVER_POD -c registrar --tail=100
kubectl -n kube-system port-forward pod/DRIVER_POD 9898:9898
# In another terminal:
curl http://127.0.0.1:9898/metrics

Monitor the backing filesystem's free bytes and inodes with host monitoring; df -h /var/lib/tak-os/local-volumes and df -i are useful diagnostics on the owner. Capacity objects are snapshots, not reservations. The driver has no volume stats RPC or per-volume quota enforcement.

On-disk state, backup, and recovery

Default layout:

/var/lib/tak-os/local-volumes/
  .metadata/
    driver.lock
    v1.<node-hash>.<name-hash>.json
    v1.<node-hash>.<name-hash>.consumer.json  # only while consumer-owned
  v1.<node-hash>.<name-hash>/
    application files

The node hash is described above; the name hash is the full SHA-256 of the CSI creation name, not necessarily the PVC's human-readable name. Use PV.spec.csi.volumeHandle to identify the directory. Volume records contain the original name, node, requested capacity, capabilities, parameters, and lifecycle state (creating, ready, or deleting). Metadata writes use atomic renames and syncs. A process lock prevents two driver processes owning one store.

Back up application data with an application-consistent procedure. For a store backup intended to restore original volumes, preserve .metadata, ownership, permissions, and Kubernetes PVC/PV definitions alongside data. Quiesce writers and coordinate lifecycle operations so files and metadata form a consistent backup. CSI snapshots are unsupported. A consumer ownership record is durable control state; copying it does not recreate a live mount in another namespace.

Do not rename volume directories, copy them onto another node and alter affinity, or delete metadata to make errors disappear. Recovery must preserve original ownership or use new volumes and application-level restore.

Maintenance, upgrades, and removal

Cordoning or draining an owning node does not relocate its volumes. Plan application downtime: pods evicted from that node remain Pending until it is available again. Applications needing availability across node failure require their own replication or another storage layer.

Before upgrading, confirm backups and image availability, review the changelog, and retain the original node identity, store, kubelet path, and any consumer root. The supplied DaemonSet updates one unavailable pod at a time with maxSurge: 0, avoiding overlapping driver processes on the same store. Watch rollout, logs, registration, and workload persistence. No automatic legacy-store conversion is documented; do not point a new installation at unidentified existing directories. Do not assume a rollback supports newer metadata or consumer state.

To remove the driver, first inventory all PVs using csi.local.tak-os.org, back up required data, and decide which volumes to retain. Stop users and release all normal and trusted consumer mounts. For Delete volumes, allow the owning provisioner to finish reclamation while it is still running. For retained volumes, record ownership and arrange later recovery or disposal. Then remove deployment resources through their owning Orbit workflow. Removing manifests alone does not clean host data or recover volumes, and removing the driver early can leave PV deletion waiting. Some external manifests carry k8s.tlspu.com/nodelete annotations; account for the deployment system's handling of those resources.

Troubleshooting reference

Inspect the PVC/pod events first, then the PV, original node, and all three containers on that node. The user guide has namespace-level commands.

Qualification tools

Every Docker image build runs the formatting check, go vet, and go test -race in its builder stage before compiling the static driver. The final image uses scratch and includes the driver, CA certificates, mount/umount, their shared libraries, and required directories. It has no shell or package manager. The runtime assembly stage checks that both mount tools start in the assembled root.

From the repository root, run the build with:

docker build -t tak-os-local-csi:local .

For direct checks outside Docker:

go vet ./...
go test -race ./...
CGO_ENABLED=0 GOOS=linux go build -o /tmp/tak-os-local-csi .
bash scripts/test-mounts.sh

The mount script needs Linux user/mount namespace support, mount, and umount. It runs selected integration tests in a private namespace. For a disposable end-to-end cluster, install Docker, kind, kubectl, and Python 3, ensure the external orbit-deployment directory resolves, then run:

bash scripts/test-e2e.sh

The script builds an image, creates its own kind cluster/context, renders the external manifests, and checks topology, data persistence across pod/driver replacement, read-only enforcement, Pending behavior on a cordoned owner, and owner-local deletion. It deletes its own cluster on exit; use KEEP_E2E_CLUSTER=1 to retain it. Its local image tag is currently hard-coded to 2.0.0; this test does not establish registry publication or qualify the optional trusted consumer integration.