Keel Design

Keel Design

Keel maintains the OS configuration of every Kubernetes node from inside the cluster. It runs as a daemonset, so there is one instance per node, and each instance only acts on the node it runs on.

Components

                +---------------------+
                |  DaemonSet (per node)|
                |                     |
   NodeConfig   |  +----------------+ |
   CR (watch) -->|  |  Controller    | |    chroot /host
                |  |  (reconciler)   |-----------> handlers (file, package, ...)
                |  +----------------+ |            + services (systemctl)
                |                     |
   Node object  |  +----------------+ |
   (watch)  --->|  | Facts collector | |---> facts.Store (async, cached)
   Node /etc    |  |  (background)   | |
   os-release  ->|  +----------------+ |
                |                     |
                +---------------------+
  • Controller (internal/controllers): watches NodeConfig CRs, schedules each on its interval, and applies it to the local node.
  • Facts collector (internal/facts): asynchronously refreshes a thread-safe facts store with node and cluster data. It does not block CR runs; the controller simply reads whatever snapshot is current.
  • Templating (internal/templating): resolves CR params against the facts store using Go text/template with missingkey=error so unknown references fail the run.
  • Handlers (internal/handlers): one Handler per configuration type, selected by NodeConfig.spec.type through a small registry.

The reconcile loop

For each NodeConfig, the controller:

  1. Loads the CR and computes its interval (defaulting to --default-interval).
  2. Waits (requeues short) until facts are ready.
  3. Checks the node label selector and skips excluded nodes. For matching nodes, skips until the interval expires unless the spec generation or watched ConfigMap digest changed. Successful no-op run times are also tracked in memory to avoid runs caused by other nodes updating status.
  4. Looks up the handler for spec.type; unknown types are reported to status.
  5. Validates spec.params.
  6. Renders spec.params and spec.services through templating.
  7. Runs handler.Ensure(ctx, hostRoot, params), which returns changed bool.
  8. If changed and spec.services is non-empty, restarts the services via systemctl (chrooted to the host root).
  9. Writes the per-node result into status.nodes[nodeName] and requeues for the interval.

RequeueAfter: interval makes execution periodic. Creation, spec-generation changes and changed ConfigMap digests bypass the remaining successful interval; metadata-only updates do not. Pending restarts are retried even without a new handler change. Failures return an error for controller backoff.

Status model

Because every node runs an instance, each pod must not clobber the others' status. status is therefore a map keyed by node name, updated with a per-node JSON merge patch (status.nodes[<node>]) rather than a whole-object replace, so concurrent pods do not conflict. Status persists the first result and changes to outcome, generation, trigger digest or restart-pending state. Identical steady-state no-op results skip the write.

status:
  observedGeneration: 3
  nodes:
    node-01:
      lastRunTime: "2026-10-15T12:00:00Z"
      lastResult: success        # success | error
      changed: true
      restartPending: false
      message: configuration applied, services restarted

restartPending is set when a change was applied but a service restart failed; it is retried on the next run even if nothing changed.

Facts

The facts collector refreshes on --facts-interval (default 5m), on startup, and when the local Node informer reports an add, update or delete. Informer callbacks coalesce notifications into a one-element channel; a managed background runnable performs API reads and host I/O without blocking the informer. Node changes refresh facts promptly; applying templates still follows CR scheduling unless a spec or ConfigMap trigger requests an earlier run. Data sources:

  • Node object (Kubernetes API): name, addresses (internal/external IP, hostname), labels, conditions, capacity/allocatable, kubelet version, container runtime, machine/boot IDs, provider ID, architecture.
  • Host filesystem (under --host-root, default /host): /etc/os-release for OS identity/version; /proc/sys/kernel/osrelease for the kernel.
  • Cluster (Kubernetes API): kube-apiserver version, platform, server URL, and a cluster identifier derived from the kube-system namespace UID. A friendly name can be set via the keel.tlspu.io/cluster-name annotation on the kube-system namespace.

Cluster facts are cached after the first successful cluster collection; node facts refresh on the schedule above. See facts and templates for the full field reference and cache limitations.

Facts are a snapshot cache with copied maps and slices on reads and writes; templates resolve against the latest snapshot. The readiness probe reports not-ready until the first successful collection.

Handler interface

A handler is a single interface, so a new type needs no core changes:

type Handler interface {
    Name() string
    Validate(params map[string]string) error
    Ensure(ctx context.Context, root string, params map[string]string) (changed bool, err error)
}

Handlers act on the host through handlers.MakeRunner(root), which runs commands via chroot(root) so package managers and systemctl affect the host OS rather than the container. changed=true means the node's state was modified; it gates service restarts. See adding-a-type.md.

Concurrency and safety

  • A per-agent mutex covers the complete host transaction, including service restarts and status persistence, across all NodeConfigs. There is no parallel modification by different CRs within a node agent.
  • Serialization prevents concurrent writes but cannot reconcile contradictory desired values. Assign each file/package/service one owner. Two CRs maintaining different contents for the same file will alternate values on their intervals.
  • The guard is process-local. Run one Keel agent per node; separate installations or overlapping manually started processes require external coordination.
  • Status updates use JSON merge patches scoped to the current node. A merge patch includes explicit empty optional values to clear stale errors/digests.
  • File writes use a temporary file and rename, with lexical root path checks. Host-root access is a privileged trust boundary; see security.md.