Performance and capacity guidance
The 2026-10-26 performance report records the source-bound staging measurements and unresolved limitations. They cover four volumes with RF1/2/3/4, ten backing workers and two independent clients per volume on four co-hosted-role VMs. They do not qualify a larger installation, maximum file size, production SLO or physical disk performance. The candidate support contract is the operating boundary; 3.x remains staging-only. These are historical measurements rather than a future benchmark requirement. Release packaging and explicit operator decisions remain required.
What the samples imply
On candidate eb1dc38, small metadata create+write+fsync+close p50 was
0.143/0.206/0.201/1.539 seconds for RF1/2/3/4. A single 4 MiB synchronous write
achieved 1.209/0.721/0.687/0.217 MiB/s respectively. The larger RF3 32 MiB
write+fsync+close took 46.632 seconds (0.686 MiB/s). These are observed whole
filesystem operations, not target RPC latency or a throughput floor.
RF4 became slower than the baseline in this run; it must not be presented as an unqualified performance improvement. The small metadata sample has only 16 observations per replica count, so reported p95 is its maximum, not a credible production tail estimate. Recent-write readbacks have uncontrolled page-cache state. All benchmark sets had successful independent hash, mode, namespace and link readback; correctness evidence does not establish low latency.
A serving coordinator serializes mutations and establishes fresh authority. Replication waits for a durable majority and serving-coordinator application; it does not require all followers to have materialized every acknowledged mutation. The replica/quorum choices have different availability costs:
| Data replicas | Durable majority | Unavailable current members tolerated for quorum I/O |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
RF2 and RF3 both require two votes; measurements need not worsen monotonically with replica count. RF4 has four copies and a three-vote majority. Placement, VM contention, concurrent work and checkpoint phase also matter. The existing samples do not isolate their individual contribution. Choose a replica count for its recovery boundary before assessing the workload cost; never reduce quorum as a performance workaround.
Capacity is allocated filesystem space, not the PVC request
Backing storage is an ordinary local-CSI PVC for each replica. Requested outer PVC size does not impose a quota. There is no fixed per-volume budget, automatic capacity reservation, reserved rebuild space or admission guarantee. Quotas remain parked pending local-CSI support. A representation bound is not a claim that that amount of data is usable or has been tested.
For each replica, account for all of the following on its backing filesystem:
- The current materialized file set, metadata and namespace. Sparse holes may use little space; writes that fill them change allocated space without changing the logical file length.
- The physical journal and its allocation history. The default 256 MiB journal setting bounds a logical post-checkpoint command window; it is not a cap on the physical bbolt file or on the volume's total storage use.
- Up to two retained published checkpoints, plus a new checkpoint capture and publication stage while old authority still exists. Capture streams allocated extents into a separate spool while holding the filesystem state lock.
- Rebuild send and receive spools, incoming authority/materialization and recovery overlap with existing files. The sender spools a complete stream before transfer. A new verified authority must exist before obsolete state can be discarded safely.
- New application writes, metadata growth and failure headroom during that overlap. A failed or restarted process may temporarily leave unpublished partial files until its owned startup cleanup can run.
These components can coexist. Their exact peak depends on workload and state; adding one fixed multiple of the requested PVC size is not justified by the current measurements. The sparse checkpoint format skips holes, but does not eliminate the duplicated allocated-data cost. Do not delete authority files or edit allocation metadata to recover free space.
Two published checkpoints and one outbound rebuild slot per worker are local bounds. They do not establish a cluster-wide limit on concurrent volume rebuilds, ordinary writes, receive materialization or their combined disk use. Practical peak recovery concurrency and growth have not yet been qualified. The 32 MiB/s rebuild limiter is an upper throttle on chunks, not promised rebuild throughput or a space reservation.
The observed ten backing directories occupied 577.273 MiB after the baseline and 923.316 MiB at the candidate soak-start sample. Those windows contain different benchmark and recovery histories and the beginning of soak. They are not an overhead ratio or a steady-state growth rate. Terminal candidate soak/resource/end-space receipts remain pending in the versioned report.
Bounds that help diagnose saturation
The current source implements these distinct bounds; none increases available backing capacity:
| Boundary | Current value and interpretation |
|---|---|
| Ordinary file request | 1 MiB data maximum; 15-second complete client budget, three-second HTTP attempts. A large file operation consists of many requests. A timeout may have committed and must retain its request identity. |
| Admission | 64 requests per protocol client; 16 worker operation slots, one mutation lane and one quorum-barrier lane. Queued requests remain deadline bounded. |
| Checkpoint cadence | Raft default threshold 8192 log entries and two-minute check interval. Pressure can trigger earlier checkpointing; this is not a guarantee that capture completes within two minutes. |
| Metadata | 100,000 objects, retained dedup results and worker handle records; 64 MiB checkpoint metadata encoding ceiling. Directory iteration is limited to 4096 entries. |
| Client handles | 512 per FUSE client; durable 90-second handle leases, renewed every 20 seconds. |
| File representation | 1 TiB maximum encoded file length. This is not a supported-capacity, quota or tested-file-size promise. |
The worker,
checkpoint,
snapshot store,
rebuild and
protocol implement these boundaries.
Hitting the logical journal window or checkpoint/metadata limit can deny new
work even with free bytes. Conversely, physical ENOSPC can occur before a
logical limit. Diagnose both rather than treating every no_space result as
a physical-disk measurement.
Interpreting capacity and load
Start with the supported small workload and preserve an external correctness witness. Collect per-operation distributions and error counts together with worker CPU/RSS, journal allocation, allocated backing-directory bytes and filesystem free bytes. Observe checkpoint capture duration/active state separately from publication, plus transfer duration/bytes and coordinator changes. Aggregate by worker and node: co-hosted volume recovery competes for the same disk and CPU.
Measure a steady application working set, normal churn across multiple
checkpoint cycles, and a separately identified rebuild/recovery window. Record
initial/final allocated bytes on the same original node/backing identities.
Do not substitute file logical lengths, a single df sample or peak RSS from
an incomplete observer window for that evidence. Include source/image hashes,
actual workload sizes, elapsed window, cache conditions and failed attempts.
Use zero independent-witness mismatches as a correctness requirement, and investigate sustained operation failures, quorum loss, capture stalls and low free space using the operational alerts. Select any latency or free-space threshold from representative normal and recovery measurements for that installation. No numeric production p99, availability target, minimum IOPS, fixed reserve or maximum supported capacity is established by these samples. Keep backups outside the replica failure domain and keep a documented restoration path into a new volume; adding replicas does not replace that recovery path.