Operating Keel

Flags

Run keel --help for the full list. The daemonset already supplies the relevant values; the notable ones are:

Observability

• Prometheus metrics are exposed on --metrics-addr (default :8080) via controller-runtime (controller metrics, workqueue depth, etc.).
• Keel adds counters keel_nodeconfig_runs_total, keel_nodeconfig_changed_total, keel_nodeconfig_errors_total, and histogram keel_nodeconfig_run_duration_seconds, all labelled by handler type. Duration includes handler execution and restarts; interval skips and excluded nodes do not count as runs. Errors include validation/template/handler/restart failures; status API failures are exposed through controller-runtime metrics/logs. Counters reset when the pod restarts. Prometheus scraping must be configured separately; no ServiceMonitor is installed.
• Kubernetes Events attached to each NodeConfig report RunStarted, RunSucceeded (Normal), and RunFailed (Warning). The node name appears in messages; Events are best-effort, aggregated by Kubernetes and expire according to cluster retention. Inspect them with kubectl -n tak-os describe nodeconfig <name>.
• Structured logs are emitted with controller-runtime's zap logger. Each CR run logs node, result, changed, and message.
• Healthz (/healthz) is a simple liveness ping; readyz (/readyz) reports not-ready until the facts store has been populated at least once.

Status and troubleshooting

Per-node run outcomes are in the CR status:

kubectl -n tak-os get nodeconfigs -o yaml
kubectl get nodeconfigs -A

lastResult: error with a message is the primary diagnostic for a failing CR. Compare each target node's observedGeneration with metadata.generation; the top-level status generation does not establish cluster-wide completion. Old entries can remain for excluded or removed nodes, and steady no-op runs do not advance their persisted timestamps. An error before handler execution, such as a failed selector or ConfigMap API read, may appear only in agent logs.

Common causes:

• unknown type "..." — the CR references a type this build does not have.
• Templating errors — a param references a missing fact or malformed template. Facts only exist after the first successful collection, and the reconciler waits for facts before running.
• file handler errors — path escapes the host root, bad mode, or write permission problems.
• package handler errors — no supported package manager detected, or the install command failed (apt update/install, dnf, etc.). Packages are installed against the host via chroot /host, so the host's own package sources apply.

Scheduling and intervals

• A matching CR runs on creation or spec-generation change and then every spec.interval (default --default-interval), once initial facts are ready. Metadata-only edits do not bypass the interval.
• If a node was offline when its interval lapsed, the run happens on the next reconcile after the node is back (facts re-collected, controller catches up).
• Successful no-op runs keep their actual execution time in memory without rewriting status. After a process restart, scheduling falls back to persisted per-node lastRunTime, so an extra idempotent run is possible.
• Spec generation changes and referenced ConfigMap create/update/delete events bypass the interval; status-only updates do not. Failed runs retry using exponential controller backoff and preserve pending service restarts.
• Node label changes reevaluate selection promptly. Selection never removes host state previously applied by a configuration.
• Runs across different CRs are serialized per node agent. Overlapping desired targets must have one owner to avoid alternating configurations.

Security considerations

Keel is privileged by design — it maintains the host OS. Mitigations to consider in your environment:

• Restrict who can create NodeConfig CRs (RBAC on the nodeconfigs resource).
• Keep spec.interval sane for change-heavy CRs to bound restart churn.
• Prefer file + services over custom command types where possible.
• The host root is mounted read-write; do not expose it to anything except the Keel daemonset.

See security.md for the privilege review and resource-sizing procedure, admission.md for validation/TLS operations, and local-cli.md for one-off runs.

For diagnosis by symptom, see troubleshooting. Installation lifecycle, configuration-author RBAC and recovery are in the administrator guide.

Adding types

See adding-a-type.md.