External backup and disaster recovery
Replication preserves ordered acknowledged writes while the configured majority and durable authority survive. It is not a backup. A quorum-lost group cannot identify the latest committed state by selecting the largest file, newest mtime, Running Pod, surviving checkpoint or apparent longest journal. Restoring a stale backup deliberately loses changes since that backup. Record its time, digest, source identity, application consistency point and the accepted loss before application cutover.
The supported recovery direction is application files from a verified external backup into a new PVC/volume/generation, using the ordinary local-backed CSI path. Never reset a journal, copy another generation's backing tree into a worker, change Node identity, reduce voting membership, or edit finalizers/allocation metadata to make uncertain state serve. Old journal/checkpoint authority is not imported by the external tools.
Backup contract
tools/backup-volume.py and tools/restore-volume.py are standalone Python 3.11+
operator tools, not a snapshot API or a new backup controller. Run them against a
mounted application filesystem. They do not call Kubernetes or inspect worker
backing paths. The platform's backup scheduler, encryption, retention, access
control and independent storage are external responsibilities.
Stop application writers, close descriptors and complete their acknowledged writes first. An application database needs its own consistent logical export; file-backed databases/locks/mmap remain unsupported on this filesystem. A walk of a live tree is not a transaction snapshot. Unlinked open files, dedup state, leases, inode/object IDs, ctime, credentials and replication history are not backed up. A new volume has new object identities. Quorum must be healthy to create a fresh backup; loss of quorum requires a previously completed backup.
The version-1 TAKBAK1 format stores a bounded checksummed JSON manifest and
checksummed regular-file extents. It preserves relative UTF-8 paths, file bytes,
directories, UID/GID/basic mode, nanosecond atime/mtime, regular-file hardlink
relationships and opaque symlink targets. Zero 1MiB chunks need no payload;
restore writes offset extents and truncates to the original logical size. Exact
physical block allocation is not promised. Symbolic links are never traversed
for writes. ACLs, xattrs, special mode bits and special files are refused rather
than silently omitted. Metadata restoration requires appropriate ownership and
permission privileges; archive/restore processes need the appropriate access
to the mounted filesystem, not raw disks or Kubernetes credentials.
Limits default to 100,000 entries, 64MiB manifest, 65,536 extents per file, 1TiB logical file size, 8GiB stored payload and a 600-second operation deadline. Flags can reduce entry/payload limits and select a deadline up to one hour. Memory is bounded by the manifest, finite metadata inventory and 1MiB data buffers; logical zero ranges still require bounded-time hashing. An empty target with enough real backing capacity is required; requested PVC capacity remains unenforced.
The archive is published exclusively with mode 0600 outside the source tree,
fsynced and never overwritten. Copy it outside the cluster/source failure
domain, verify SHA256 and complete --verify before treating it as a backup.
A helper's emptyDir alone is not an external backup. The format is plaintext and its checksums are not signatures. Verify against a
trusted independently recorded digest, and protect/encrypt/authenticate the
external artifact through the operator's backup system.
Current source reports deadline_exceeded for the tool's configured alarm
and filesystem_timeout for a filesystem ETIMEDOUT. Failure JSON adds
the bounded operation phase, failure source and numeric errno/name; permission
and disk-full errors have separate categories. It excludes paths, file contents
and exception text. Earlier tool versions used deadline_exceeded for both
timeout sources, so that historical category alone cannot identify the cause.
Check the actual tool source/version and exit status; an incomplete operation
is not a verified backup, even if an archive file exists.
Explicit operator records
Both tools require a small identity JSON containing exactly volume,
generation, claimUID. Resolve these from the exact bound PVC/PV and owned
ReplicatedVolume, never from a reused name. The backup record is:
{
"version": 1,
"purpose": "backup-quiesced",
"source": {"volume": "rv-source", "generation": "original-generation", "claimUID": "original-claim"},
"recordID": "incident-or-change-id",
"operator": "responsible-operator",
"method": "documented application stop and descriptor close",
"evidenceReference": "private audited evidence location",
"operatorAcknowledged": true
}
Restore requires the same source in a record with purpose restore-isolated and
a new destination identity whose three fields all differ. This record is an
operator audit acknowledgement, not cryptographic or automatic proof of
physical fencing. The operator must supply real isolation evidence. For a
controlled recovery, record exact old consumer Pod absence, no old FUSE
mounts on every original Ready node, and separate immutable old/new identities.
For unavailable/hostile nodes or an irretrievably lost cluster, use verified
infrastructure isolation, revoked application access and a deliberate routing
cutover. API Node/Pod deletion and NotReady alone do not prove old clients or
storage stopped. Do not promote the restored application until that isolation
and accepted recovery point are reviewed.
Commands and completion
The Python tools are source files, not executables bundled in the scratch role
images. Provide them in an approved Python environment with the application PVC
and external archive path mounted. Example /tools paths below refer to that
operator-provided environment. The commands do not create helper Pods or modify
a deployment by themselves.
Inside a bounded helper, with the quiesced source mounted at /data and an
independent export path at /backup:
python3 /tools/backup-volume.py --root /data --archive /backup/source.takbak \
--identity /backup/source-identity.json --record /backup/quiescence.json
python3 /tools/backup-volume.py --verify --archive /backup/source.takbak
After copying to independently protected storage, run --verify there too. Copy
that external artifact into a fresh restore helper. Keep all target application
clients stopped; mount only the new empty PVC and run:
python3 /tools/restore-volume.py --root /data --archive /backup/source.takbak \
--identity /backup/target-identity.json --record /backup/isolation.json
The whole archive is validated before destination changes. The destination must
be empty; no merge or overwrite is available. Creation first fsyncs an incomplete
marker and root directory. Final data and inode metadata, child directories and
root namespace are fsynced; the marker is removed last and root is synced again.
Actual destination hashes, metadata and link relationships are checked before
PASS. Failure leaves the new volume unusable for cutover. A failure after marker
unlink is explicitly restore_publication_uncertain_keep_clients_stopped, even
if marker recreation itself fails. Never interpret missing marker alone as
success. Preserve the failed target for diagnosis or delete it normally and
allocate another fresh target; the tool will not resume a partial tree.
JSON results and optional exclusive Prometheus textfiles expose bounded operation/result/duration, aggregate sizes and archive hashes, without paths, file contents, operator-record contents or credentials. A PASS does not prove application-level consistency, physical old-node fencing or a backup scheduler's retention policy. Independently read back the restored application files from a new mount before cutover and archive its evidence.
Permanent node loss and cleanup
Current replacement still needs affirmative original-member serving fencing. Deletion also preserves original Node UID/local CSI/backing authority and waits for actual local CSI reclamation. If the original Node/disk is permanently gone, these operations remain blocked; returning the original authority is the normal supported completion path. External restoration into a new volume can recover application service without adopting that uncertain authority, but it does not reclaim an unreachable old disk or authorize removing its finalizers.
Closing that gap requires a reviewed infrastructure fencing/lost-disk attestation contract and backing-driver cleanup support. Permanent-node recovery/disposal remains PARK-09. Old recovery reports retain their historical scope; external restoration does not reclaim old storage or authorize force-delete operations.