Troubleshooting

Start with the affected NodeConfig and node. An API write succeeding, an agent being Ready and a host configuration succeeding are separate observations. Commands below use tak-os; use the NodeConfig's actual namespace when different.

Collect evidence

KEEL_CONFIG='replace-with-config-name'
KEEL_POD='replace-with-agent-pod-on-affected-node'
kubectl config current-context
kubectl -n tak-os get nodeconfig "$KEEL_CONFIG" -o yaml
kubectl -n tak-os describe nodeconfig "$KEEL_CONFIG"
kubectl -n tak-os get pods -l app.kubernetes.io/name=keel -o wide
kubectl -n tak-os logs "$KEEL_POD" --since=30m
kubectl -n tak-os describe pod "$KEEL_POD"
kubectl -n tak-os get events --sort-by=.metadata.creationTimestamp

Replace the two variable values with the affected resource names. The pod list shows which node each agent runs on.

Compare metadata.generation with the target node's status.nodes[<node-name>].observedGeneration. Read lastResult, message, changed and restartPending. Old status entries can remain after nodes cease matching the selector. Unchanged successful runs can leave lastRunTime old; use logs, Events and metrics to assess ongoing execution. Events expire and are best-effort, so collect relevant evidence promptly. Redact sensitive configuration and command output before sharing logs or snapshots.

Installation and readiness

The supplied role must allow list/watch on Nodes, Namespaces, ConfigMaps and NodeConfigs, plus NodeConfig status patching and Event writes. A cached client can need list/watch even when a particular operation looks like a single get. If permitted to impersonate service accounts, an administrator can verify:

kubectl auth can-i list nodes --as=system:serviceaccount:tak-os:keel
kubectl auth can-i list namespaces --as=system:serviceaccount:tak-os:keel
kubectl auth can-i watch configmaps --all-namespaces --as=system:serviceaccount:tak-os:keel
kubectl auth can-i patch nodeconfigs.keel.tlspu.io --subresource=status -n tak-os --as=system:serviceaccount:tak-os:keel

Restore the reviewed role from deploy/rbac.yaml rather than granting general cluster-admin access to the agent.

Accepted configuration with no result

Check that the selector matches the current Node labels and that an agent runs there. Confirm the NodeConfig namespace and check all namespaces with kubectl get nodeconfigs -A. Look for facts-not-ready, selector, ConfigMap read or status-patch errors in that node's agent logs. Failures before handler execution can be logged without a new per-node status outcome.

If the same spec and ConfigMap digest already succeeded, Keel waits for the interval. Metadata-only NodeConfig edits do not force application. A referenced ConfigMap change or a spec change can request an earlier run. ConfigMaps must be in the NodeConfig's namespace; their data never becomes template input.

Validation and templates

unknown type means the executing image lacks the named handler. Use file or package, or deploy the custom build consistently to agents and webhook. Missing parameters must use the exact handler keys; all values must be strings. Use quoted octal modes and a positive Go duration such as 30m or 24h.

Template parse errors indicate malformed syntax or an unknown function. Execution errors indicate a missing field, missing direct map key or incorrect argument type. See facts and templates for field names and index with fallbacks. Admission validates syntax only; it cannot prove a template will render on each node. Empty known facts do not necessarily cause an execution error.

Files and packages

For file errors, use an absolute host filename, omit the /host prefix, check octal mode syntax, parent paths, mount writability, disk space and filesystem behavior. Replacement needs a temporary file in the same directory. Parent symlinks and special files need particular care. Specify mode explicitly; replacing changed contents without one uses 0644. File-format validation and ownership management are outside this handler.

For package errors, inspect the host /etc/os-release ID, installed package tools, repositories, network/DNS access, locks and free space. Commands run against the host through chroot. Version strings must match the backend's installed representation and be available from its repositories. A successful install command is not followed by version verification, so verify important pins on the host. The handler reference lists exact backends and their limitations. Use the host's normal package-repair procedure for interrupted transactions; repeated agent restarts do not repair a package database.

Service restart pending

applied but service restart failed means the host configuration changed before systemctl failed. Check the host unit name, systemctl status and journal, unit syntax, dependencies and access to the running host systemd. The unit may be called ssh on one distribution and sshd on another.

Correct the service or configuration and watch restartPending clear. Keel retries the current service list even if the file or package is already in the desired state. Each attempt performs daemon-reload and can restart units that previously succeeded. Removing services can clear the pending state without performing the originally intended restart; that is not service recovery. Keel does not restore earlier host state automatically.

Admission rejects or blocks API writes

An explicit field-validation error requires correcting the NodeConfig. A webhook connection or TLS error requires repairing the optional admission server:

kubectl -n tak-os get deployment keel-webhook
kubectl -n tak-os get pods -l app.kubernetes.io/name=keel-webhook
kubectl -n tak-os logs deployment/keel-webhook --since=30m
kubectl -n tak-os get certificate keel-webhook
kubectl -n tak-os get endpointslices -l kubernetes.io/service-name=keel-webhook
kubectl get validatingwebhookconfiguration keel-nodeconfig -o yaml

The certificate command applies to the cert-manager installation. Check DNS names, certificate readiness, mounted TLS Secret, Service endpoints, CA injection and API-server reachability to port 443 (target 9443). Do not print the private key while collecting diagnostics.

The registration uses failurePolicy: Fail; an outage blocks NodeConfig creates and updates. Existing agent reconciliation and status updates are separate from that admission path. If an administrator chooses to disable admission during recovery, remove deploy/webhook-admission.yaml first as documented in admission, then restore the registration after the server is healthy. Runtime validation remains on the agents, but admission-time rejection is unavailable while the registration is absent.

Unexpected repeated changes

Look for multiple NodeConfigs or external host tools managing the same path or package, dynamic template values, file drift or failed restarts. Compare per-node results rather than the shared top-level generation. Do not install multiple Keel agents on one node to improve availability: their locks are process-local. The administrator guide explains stopping reconciliation and restoring previous desired state.