Observability

Documentation home

All four roles use internal/observe for JSON logs and a private Prometheus registry. Ordinary successful file operations stay at DEBUG. Failures are WARN; a successful file RPC requiring more than one HTTP attempt emits an INFO recovery summary. Raft diagnostics retain the existing WARN threshold. Log emission does not change quorum, replay identity or operation deadlines.

Library/caller fields named time, level, msg or component are preserved as diagnostic_time, diagnostic_level, diagnostic_msg or diagnostic_component, so they cannot replace the shared envelope.

Every production record has time, level, msg, component, operation, outcome, duration_seconds and request_id. Request-bound operations use a context correlation ID; file calls use the exact replay ID sent to workers. Asynchronous/background events use the explicit request_id="background" sentinel and zero duration when no operation duration exists. That sentinel is not an invented Raft request trace. Relevant calls additionally include volume, generation, replica or node identity. Failure records include bounded error_category and retry_decision; the latter describes diagnostic policy, not authorization to replay an arbitrary mutation.

Repeated WARN/ERROR and retry-recovery INFO records allow two records per 30-second window for each severity, operation, error category and operational identity. Diagnostic message classes also distinguish unrelated Raft/HTTP failures. Request IDs and changing error strings do not create new operation buckets. The limiter keeps at most 128 context buckets and one overflow bucket. The first emitted record in a subsequent window carries suppressed_count; suppression_scope="overflow" explicitly means a count shared across overflow contexts. If no later record arrives, the Prometheus suppression counter still records the suppressed events. Operation and retry metrics count every event, including events whose logs were suppressed. This policy bounds repeated failures, rather than hiding a stuck dependency behind an unlimited log stream.

Structured sensitive keys (authorization, token, password, credentials, private_key, tls.key, payload, data, file_contents) are redacted, including nested groups. Callers must still never place credentials or file contents in free-text messages, errors or arbitrary objects: key redaction is not a free-text secret detector. No endpoint, path, request ID, volume or generation is a metric label.

Retry summaries include actual attempt/switch counts and the four preceding failure-reason counts. A terminal permission, I/O or no-space error after a redirect is failed, not a claim that the deadline was exhausted. Observability does not alter the protocol's replay identity, endpoint selection, terminal admission errors or total deadline. The context hook receives one immutable summary value per call and does not introduce shared mutable per-client state. Counters are process-local and reset with the process. Non-client roles expose zero client counters; a scrape target supplies the process/Pod identity.

The 4.0.1 manager records the independent renewal pass as operation="reconcile_credentials" and each credential check/update as operation="ensure_credentials" in the operation counter and duration histogram. Failure logs correlate volume/generation and request IDs without credential contents; successful checks remain at DEBUG and actual renewals log at INFO. Renewal continues independently when lifecycle reconciliation or readiness fails. These operations are not available in the existing 4.0.0 deployment.

Replica progress and health

Replica-specific metrics below have no volume, generation, member, path or request ID labels. Identify a worker by its scrape target and process/Pod epoch. Counters reset when the worker process restarts; a counter delta across different epochs is not evidence of completed work.

stage means an accepted snapshot sink writing its private validation spool. It includes both locally generated baselines and incoming transfers; it is not a pure network-receive counter. Success requires validated durable publication. Cancel records an abandoned sink, including an interrupted incoming transfer; validation or publication failure on close records an error. Admission failures before a sink is created are not stage operations.

send measures bytes read by the outbound InstallSnapshot transport and its accepted/rejected result. Consumed bytes are not TCP acknowledgements or proof that a peer durably published them. restore measures bytes consumed and commands replayed while rebuilding an isolated candidate. Success requires validation and publication; discarded candidate work still contributes to attempted progress. There is no percentage-complete or resumable-transfer claim. Baselines retain the complete bounded command history; these metrics do not change membership, lifetime limits or authority recovery rules.

Completed-state gauges use atomic mirrors. Their values advance after a mutation or baseline publication completes, while progress counters remain available during replay. A metrics scrape does not acquire the filesystem mutex. Compare source/target commit and baseline indexes, validated operation outcomes and actual file results when qualifying catch-up; neither Ready nor a byte counter alone proves it.

Journal and snapshot I/O timing

Workers expose the following storage timing metrics. They were introduced after a historical no-fault failure; their presence and later captured samples do not establish that failure's cause or a performance improvement.

Journal operations are journal_first_index, journal_last_index, journal_get_log, journal_store_log, journal_store_logs, journal_delete_range, journal_get, journal_set, journal_get_uint64 and journal_set_uint64. The wrapper preserves the concrete underlying store's optional methods, return values, error identities, transaction ordering and durability. It adds no retry or deadline. Startup authority validation uses the original store; these timings cover subsequent Raft/FSM journal accesses.

Snapshot operations are snapshot_open_published, snapshot_validate_published, snapshot_validate_staged, snapshot_copy, snapshot_publish_wait and snapshot_publish_close. The first measures HashiCorp FileSnapshotStore Open, including its complete CRC scan before returning the reader. Application-format validation of that returned stream is timed separately. Staged-candidate validation includes rewinding its spool; copy measures the underlying snapshot stream write. Publication wait measures acquisition of the snapshot-store mutex, and publication close measures the underlying durability/publication/ reaping call while that mutex is held. A local journal_delete_range interval can expose later log-compaction delay. These spans are components of existing work; do not sum them with their containing baseline-stage duration as if they were independent operations.

Histogram finite bounds are 0.001, 0.01, 0.1, 1, 3, 5, 15, 30, 60 and 120 seconds. A blocked call appears in the active gauge before its histogram or outcome counter changes. Scrapes do not acquire journal, filesystem or snapshot-store locks. The inner timing hooks perform no synchronous JSON log writes, because their caller can already hold those locks; correlate existing logical-operation logs and their replay IDs with same-epoch metrics instead. Keys, paths, filenames, volume IDs, mount IDs and credentials are never labels.

Interpreting replication lag

replicated_csi_replica_apply_lag_entries is local apply lag, not the distance behind another replica. An isolated follower can report zero after applying everything it knows, while the coordinator continues committing new work. For a leader-relative estimate, compare a fresh coordinator commit index with each same-generation worker's completed command/baseline index. The staging catch-up harness captures those actual same-group indexes alongside transfer outcomes and file checks; it does not treat the local gauge as cluster-wide lag.

The following bounded instant query uses the existing qualification collector's namespace, pod and role target labels. Replace the example volume ID in both selectors. Before interpreting it, verify exactly one fresh, ready leader for that fixed group and unchanged Pod/container and generation identities. It omits down/stale targets rather than reporting them as caught up. volume is derived only in this query from the immutable worker Pod name; it is not a new exported label, recording rule or alert.

clamp_min(
  max by (namespace, volume) (
    label_replace(
      (
        replicated_csi_raft_commit_index{namespace="kube-system",role="replica",pod=~"rv-0123456789abcdef01234567-r[0-3]"}
        and (timestamp(replicated_csi_raft_commit_index) > time() - 30)
        and on (namespace, pod) (replicated_csi_raft_leader == 1)
        and on (namespace, pod) (replicated_csi_ready{component="replica"} == 1)
        and on (namespace, pod) (up == 1)
      ),
      "volume", "$1", "pod", "(rv-[0-9a-f]{24})-r[0-3]"
    )
  )
  - on (namespace, volume) group_right
  label_replace(
    (
      replicated_csi_replica_applied_command_index{namespace="kube-system",role="replica",pod=~"rv-0123456789abcdef01234567-r[0-3]"}
      and (timestamp(replicated_csi_replica_applied_command_index) > time() - 30)
      and on (namespace, pod) (up == 1)
    ),
    "volume", "$1", "pod", "(rv-[0-9a-f]{24})-r[0-3]"
  ),
  0
)

The result retains each worker's target labels and measures an entry-index distance, not bytes, elapsed replication time or missing file writes. Barrier and no-op entries, independent scrape times and publication timing affect it; clamping a negative sample to zero does not establish synchronization. Fresh scrapes and sampled readiness also do not establish current quorum. During an election or ambiguous leader observation, defer this comparison and inspect the captured identities and protocol outcomes.

Readiness and liveness

The bounded readiness reasons are starting, ready, no_leader, quorum_unavailable, catching_up, journal_unavailable, restoring, retired, materialization_failure, check_expired, check_busy and shutdown. check_busy means a concurrent publication/state check prevented a safe completed answer; check_expired means the bounded readiness check expired. The replicated_csi_ready{component="replica"} gauge and /readyz represent the last published outcome. Reason transitions also produce structured operation logs; IDs remain log attributes, and repeated identical reasons do not generate logs.

The serving loop initiates checks on a one-second ticker and gives each check a two-second deadline. It retains the last completed result while the next check runs, but an independent expiry callback publishes unready even if a synchronous filesystem check stalls. Readiness publication and reason updates do not wait for a blocked diagnostic log writer. Checks are serialized; a slow check can delay the next tick. /healthz remains process liveness and returns 200 during missing quorum, catch-up or restore. Sticky materialization failure causes the serving worker to stop; lack of quorum alone does not trigger a restart.

A ready leader must establish current quorum and complete its committed prefix. A follower uses leader contact within two seconds and locally completed state; its readiness can briefly outlive quorum loss and never authorizes direct file I/O. File requests still require the verified coordinator. Retired, restoring, failed or expired workers are unready. These diagnostics are sampled operational evidence, not an alternative to the protocol's read/write authority checks.

Monitoring pack and inventory

The schema-1 monitoring pack supplies bounded recording rules, alert definitions and a Grafana dashboard. Load them into the platform monitoring stack and configure its existing notification routing; the driver does not deploy a collector or external receiver.

Every manager, including lease followers, refreshes a read-only API inventory every five seconds with a ten-second deadline. Lifecycle writes remain elected. A failed or partial inventory leaves its previous gauges and freshness stamp unchanged; it never publishes zero as a successful observation. This prevents an old leader's stale positive counts persisting indefinitely after takeover. HA recording rules use maximum per namespace, rather than summing duplicate inventories. A separate stale-inventory alert makes API uncertainty visible.

Replica metrics expose current process state, not unique global object counts; never sum replica copies to infer logical files or compare deltas across Pod restarts. Exact volume drill-down uses the GET-only support bundle, preserving bounded exported application labels.

Shared NODE workers in 4.1.0

The replica host exposes replicated_csi_host_workers{state} with the fixed states starting, ready, unavailable, uncertain, replica_state_lost and stopping. Volume/generation/member identities remain log fields, not metric labels. Per-worker registries remain separate and are reachable through each replica Service's metrics port; update service discovery accordingly. Host readiness never means all hosted workers have quorum.

Operation counters/durations cover replica_host_reconcile, replica_open, replica_serve, replica_endpoint_refresh, backing_publish, backing_unpublish, runtime_transition and retire_node_controllers. Known retained-state loss uses error_category="data_loss" and the volume status reason ReplicaStateLost; no exact application-file loss is invented. The optional local-CSI consumer extension exposes bounded publish/unpublish outcome counters at its existing health listener's /metrics endpoint and emits JSON diagnostics.