Deploying Keel

Deploying Keel

Keel is a daemonset: one privileged pod per node that maintains the node's OS configuration from inside the cluster.

Prerequisites

  • A Kubernetes cluster with Linux nodes. Service restarts require host systemd; package operations require a supported host backend.
  • Permission and cluster admission policy allowing privileged root pods, hostPID and a writable hostPath mount of / in the management namespace.
  • Administrative permission to install the cluster-scoped CRD and RBAC, or those resources pre-provisioned by the platform administrator.
  • The NodeConfig CRD installed (applied before the daemonset, or the pods will fail to watch a non-existent resource).
  • The host root mount path (--host-root, default /host) must exist and be a hostPath mount of /.

Install

kubectl apply -f config/crd/bases/
kubectl apply -f deploy/namespace.yaml
kubectl apply -f deploy/rbac.yaml
kubectl apply -f deploy/daemonset.yaml

Wait for agents before creating configurations:

kubectl -n tak-os rollout status daemonset/keel --timeout=180s

Follow the user guide for a single-node first configuration. The supplied examples change /etc/motd, request an sshd restart and install htop on all agent nodes. Review their service/package names and add approved node selectors before applying them:

kubectl apply -f deploy/example/nodeconfigs.yaml
kubectl -n tak-os get nodeconfigs

Daemonset layout

deploy/daemonset.yaml provides:

  • hostPID: true — host PID namespace visibility. The command runner uses chroot; it does not call nsenter to enter other host namespaces.
  • privileged: true, runAsUser: 0 — required to manipulate the host OS and its mounts.
  • A hostPath volume mounting / at /host — the host root used by the file handler and the chroot target for package/systemctl commands.
  • tolerations: [operator: Exists] — runs on every node, including control plane nodes with taints.
  • nodeSelector: {kubernetes.io/os: linux} — excludes non-Linux nodes. Narrow this selector or add affinity to restrict the agent to approved pools.
  • NODE_NAME from the downward API (spec.nodeName) passed to --node-name.
  • Liveness (/healthz) and readiness (/readyz) probes on :8081. Readiness stays not-ready until node facts have been collected.

Building the image

make docker-build          # builds tak-os/keel:latest

The default image reference in deploy/daemonset.yaml is registry.gitlab.com/tlspu/tak-os-containers/tak-os-keel:0.2.0; update it for your registry before applying. The CI pipeline (.gitlab-ci.yml) builds the container via the tlspu/ci-utils container build template.

The image built by make docker-build has a different local tag from the deployment manifest. Push an image to a registry reachable by the nodes and update the manifest, or explicitly load and select that local image in an isolated test cluster. Pin the reviewed tag or digest; building locally alone does not update a running deployment. Configure image pull credentials if needed.

Running under Orbit Deploy

When Keel is deployed with Orbit Deploy, the daemonset and its supporting resources can be supplied as part of the application's resources. The recommended layout:

  • The CRD is a cluster-scoped resource and should be installed once, before the namespace-scoped application resources (apply it as part of the platform or bootstrap step).
  • The namespace, ServiceAccount, ClusterRole/ClusterRoleBinding, and DaemonSet are applied per deployment target.
  • NodeConfig CRs are namespace-scoped and can live in the same namespace as the application or a dedicated management namespace.

Note that the ClusterRole/ClusterRoleBinding are cluster-scoped: an Orbit Deploy tenant must have permission to create them, or the RBAC must be pre-provisioned cluster-wide and only the namespace-scoped pieces shipped per target.

These are integration steps for the supplied Kubernetes manifests; this repository does not include a dedicated Orbit Deploy application definition. Have the platform bootstrap layer own the cluster-scoped pieces and reference the reviewed namespaced manifests from the application's deployment source. Keep CRD installation, agent readiness and NodeConfig creation in ordered steps. For optional admission, use the separate ordering described below.

When changing the namespace from tak-os, update namespaced object metadata, ClusterRoleBinding subjects, webhook Service references, certificate DNS names and the admission CA-injection annotation together. Namespaced NodeConfigs are still watched cluster-wide; application namespaces do not isolate host access.

Verifying

kubectl -n tak-os get daemonset keel
kubectl -n tak-os get pods -o wide -l app.kubernetes.io/name=keel
kubectl -n tak-os logs -l app.kubernetes.io/name=keel --prefix --tail=100
kubectl -n tak-os get nodeconfigs -o yaml
kubectl get nodeconfigs -A

Each pod's logs show per-CR runs with result/changed/message, and the CR status shows per-node outcomes.

Admission and rollout choices

The node agents operate without the optional webhook. For admission-time validation, install the dedicated webhook Deployment and TLS resources, wait for readiness, then register it as described in admission.md. Under Orbit Deploy, keep admission registration in a later ordered deployment step so a fail-closed webhook cannot block NodeConfig creation before its endpoints exist. Keep the CRD, admission registration and certificate infrastructure under platform-level ownership when tenants cannot create cluster-scoped resources.

Run one installation of the node DaemonSet per node. Use selectors on NodeConfigs for canary rollout; removing a selector match does not revert existing host configuration. Review the security model before granting authorship or deploying package configurations. The kind suite exercises an isolated cluster and does not target an Orbit Deploy environment.

For ongoing access management, upgrades, recovery and removal, use the administrator guide.