Security and resource bounds
The transport authenticates a volume generation, not an individual application user. Workers use TLS 1.3 with a distinct immutable CA per volume and verified client certificates. Every client verifies the original volume Service SAN even when a replacement Service becomes the connection target. Redirects and foreign leader hint domains are rejected. File RPC and Raft/member/rebuild control use independent random 256-bit bearer credentials. Possession of a file credential does not grant membership control. Tokens and file contents must never enter logs or metric labels.
A Kubernetes administrator, privileged CSI node, or manager is trusted for the volumes it controls. The mount owner necessarily has host mount privileges; the replica worker has a read-only image, no service-account token, and all Linux capabilities dropped. The volume Secret retains a CA signing key for renewal. Worker Secret projections exclude that key. The CSI credential provider retains only the client certificate, key, CA, file token, and its overlap fields. However, Kubernetes Secret GET is not field-scoped: the trusted node service account can retrieve the whole Secret from its configured namespace. This is not isolation against a compromised privileged node or per-worker certificate revocation. Kubernetes API transport uses the normal service-account CA and rotating token.
New generations have a ten-year immutable CA and a 24-hour leaf certificate. The 4.0.0 manager renews the leaf, private key and both independent bearer tokens when twelve hours remain, through ordinary lifecycle reconciliation. Release 4.0.1 starts renewal with eighteen hours remaining and gives it an independent manager-leader loop with dedicated Kubernetes API clients and rate-limit budgets. It runs immediately on leadership and pauses one minute between passes. Each volume has a fifteen-second budget; one failure does not stop other volumes or later passes. Controller setup, workload readiness, quorum, PVC/PV bookkeeping and ordinary reconciliation failures do not gate this loop. Leadership loss or shutdown cancels it. The original CA/signer validation, ownership guards and resource-version updates still apply; API outages or invalid trust remain explicit failures that are retried, never bypassed. This correction is not yet deployed or runtime-verified in the affected cluster. See the expired-certificate runbook. The previous tokens are accepted for at most ten minutes after publication. Outbound clients and peers use the old token for the first five minutes, allowing kubelet Secret projections to catch up, and then switch to the new token. CSI mount owners refresh credentials every 30 seconds in a background control-plane task, outside file I/O. Workers reload the leaf on TLS handshakes and token files on authenticated operations; existing Raft connections recheck credential generation on transport reads and writes. Client idle connections are closed after refresh. Expired client certificates are rejected on HTTP requests as well as new handshakes. Secret propagation or API outages longer than the overlap can cause explicit permission/unavailability errors; the driver does not bypass authentication to hide those failures.
An operator may request early renewal by setting this annotation on the exact ReplicatedVolume, using immutable UID and resourceVersion guards:
csi.replicated.tak-os.org/rotate-credentials: <unique request identity>
The manager records the request identity in an owned Secret annotation using a resourceVersion update. Retrying the same identity does nothing. A different request is refused while the prior overlap is active, preventing a second rotation from invalidating still-valid previous credentials. Identities are at most 128 bytes and are included in structured renewal logs. The operator should observe successful continued I/O through the final ten-minute cutoff. Never print the Secret or pass token values in a shell command or support bundle. Legacy generations without their original CA signing key continue to use their valid existing credentials but require offline renewal when the leaf approaches expiry. Missing or malformed active trust is never replaced automatically. CA rollover, emergency compromise recovery, and hostile-node isolation remain production security work; ordinary renewal does not claim to provide these.
Symlinks are namespace metadata and inert regular-file materialization on the backing volume. Worker RPC never follows them through the host filesystem. Readlink returns the original target for the FUSE kernel to interpret in the application's mount namespace. Absolute links therefore have normal application namespace semantics; they do not give the worker access to host paths. RPC names are single path components and reject slash, dot-dot, and absolute traversal. CSI stage/publish paths validate all components and reject symlink parents.
Resource limits are protocol and concurrency bounds, not storage quotas:
- A client admits at most 64 concurrent calls, fails excess calls before sending, sends at most 1 MiB data per request, and accepts at most 4 MiB response bytes. Each HTTP attempt has a three-second deadline inside the total fifteen-second call deadline. Every retry retains identical request bytes and identity.
- A worker admits at most 16 file/member operations before decoding a file body, serializes mutations, and bounds each request body and operation deadline. Its data listener accepts 128 connections, Raft 32, and metrics 16. Headers are capped at 16 KiB, read/write/idle timeouts are explicit, and HTTP/2 is disabled on the file/control server to avoid extra per-connection stream queues.
- A worker runs one outbound checkpoint rebuild at a time. Chunk buffers are 1 MiB and a shared limiter permits at most 32 MiB/s of outgoing rebuild data. Waiting for a transfer or bandwidth is cancellable. The receiver admits one rebuild actor and persists only verified chunks before advancing its resume prefix. Transfer spools are files, not complete in-memory copies.
- Local journal/checkpoint/object/extent bounds remain explicit. Full backing storage or failed persistence produces an error, not acknowledged success. Cancellation cannot interrupt an already blocked kernel disk syscall. Queues and published readiness remain bounded while the syscall completes; no latency guarantee or recovery of a failed single-copy backing disk is implied.
The runtime's trusted-node model is distinct from hostile multi-tenant isolation. Normal renewal is not emergency CA revocation. Historical observations have their recorded source boundaries; use release policy and actual image build metadata when identifying the code deployed in an incident.