Administrator guide

Administrator guide

Administrators install Keel, control who can submit host configuration and operate the node agents and optional admission service. Configuration authors should start with the user guide.

Plan the installation

Use the deployment guide for installation commands and Orbit Deploy integration. Review the security model and resource inventory before deployment.

The supplied DaemonSet targets Linux nodes, tolerates all taints and can run on control-plane nodes. It runs as privileged root with host PID access and writable host / mounted at /host. Cluster policy must permit that deployment. Package managers and systemctl run inside the host root; the image's Alpine package manager is not used to manage nodes. Systemd is required for service restarts; file-only tasks do not invoke it. Non-systemd service backends are not implemented.

Choose one Keel node-agent installation per node. Agents watch NodeConfigs across all namespaces, so deploying a second DaemonSet in another namespace does not isolate it. Its independent host operations could conflict with the first.

Before installation, decide:

  • Which node pools and control-plane nodes should run Keel. Narrow the DaemonSet's pod nodeSelector or affinity if necessary.
  • Which trusted administrators own NodeConfigs, trigger ConfigMaps, release images, host recovery and optional webhook certificates.
  • Which immutable release image to use for both agents and webhook, and how nodes will pull it (including registry credentials if needed).
  • Whether admission validation is required, and who supplies cert-manager/TLS.
  • How desired-state manifests, monitoring and host backups will be maintained.

The sample resource requests and limits are starting values. Measure workloads before changing them; security describes sizing. Package subprocesses, caches and host writes need particular attention.

Install and verify

Follow the ordered steps in deployment, then check:

kubectl -n tak-os rollout status daemonset/keel --timeout=180s
kubectl -n tak-os get daemonset keel
kubectl -n tak-os get pods -l app.kubernetes.io/name=keel -o wide
kubectl get nodeconfigs -A

Compare desired, current and ready agent counts with the intended node pool. Initial readiness requires facts collection, including Kubernetes API access. A ready pod does not establish that every NodeConfig succeeded; inspect status on each target node as described in the user guide.

If using admission, install the server and certificate first, wait for healthy endpoints and a CA bundle, then register it. The exact procedure is in admission. Keep the webhook and node agents on the same release.

Use a harmless file configuration on one approved node for an initial canary. The supplied deploy/example/nodeconfigs.yaml instead modifies /etc/motd, requests an sshd restart and installs htop on all agent nodes. Review and add selectors before applying it; service and package names vary by host.

Grant configuration access

The supplied keel ClusterRole is for the agent, not configuration authors. Grant authors separate namespaced access. For example, save and review the following as keel-config-access.yaml, replacing the group name with your identity provider's trusted node-administrator group:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: keel-config-author
  namespace: tak-os
rules:
  - apiGroups: [keel.tlspu.io]
    resources: [nodeconfigs]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [""]
    resources: [events]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: keel-config-authors
  namespace: tak-os
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: keel-config-author
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: replace-with-trusted-node-admin-group

kubectl apply -f keel-config-access.yaml

This grants host administration through NodeConfigs on any matching agent node, not just hosts associated with tak-os. Kubernetes RBAC does not constrain the contents of spec.nodeSelector or host paths. The webhook validates syntax and required values; it does not enforce tenant isolation or approve the operational safety of a change. Audit NodeConfig write grants in every namespace, including broader roles. A read-only observer can use a separate Role with only get, list, watch on NodeConfigs and Events; reading a NodeConfig also exposes its parameters and file contents.

Grant ConfigMap trigger writes separately only where needed. Changing a watched ConfigMap can cause immediate host runs. Authors do not need agent service-account tokens or permission to patch NodeConfig status. Kubernetes API audit logging, when configured by the platform, can record who changed desired state; transient Keel Events do not replace that audit trail.

Routine operation

Monitor agent availability, restarts, reconciliation errors, pending restarts and host storage. Configure metric scraping separately; the repository installs no ServiceMonitor. Listener defaults, metric names and scheduling rules are in operations. Keep unauthenticated metrics and health endpoints on trusted networks.

Apply host changes to canaries before wider node pools. Keel does not drain nodes, respect workload disruption budgets for service restarts, reboot hosts or coordinate cluster-wide sequencing. Use a separate maintenance procedure for changes affecting kubelet, networking, storage, SSH or other critical services. Avoid overlapping file/package owners across NodeConfigs and other host managers.

Upgrade or roll back the agent

  1. Save desired-state manifests and export the current configuration as described below. Review CRD, RBAC, parameter and handler changes in the target release.
  2. Test the release against representative hosts and existing NodeConfigs. The testing guide covers developer verification; its kind suite does not establish compatibility with every host package manager.
  3. Apply any required CRD/RBAC updates from that release. If using admission, upgrade its image and wait for rollout before submitting configurations that need new validation or handlers.
  4. Update the DaemonSet image in the deployment source, apply it, and observe the rolling update and per-node outcomes. Keep the optional webhook image aligned.
  5. Expand configuration changes after agents have converged on the release.

Use a pinned tag or digest in the reviewed manifests. For an installation managed by direct kubectl commands, the equivalent image change is:

KEEL_IMAGE='registry.example.com/team/keel:replace-with-reviewed-release'
# Run these two commands only if the optional webhook is installed.
kubectl -n tak-os set image deployment/keel-webhook webhook="$KEEL_IMAGE"
kubectl -n tak-os rollout status deployment/keel-webhook --timeout=180s
kubectl -n tak-os set image daemonset/keel keel="$KEEL_IMAGE"
kubectl -n tak-os rollout status daemonset/keel --timeout=180s

Replace the image placeholder first. For Orbit Deploy or another desired-state manager, make the change in that manager's source to keep it persistent. A DaemonSet rollout only sequences agent pod replacement; it does not sequence the host changes requested by a NodeConfig across nodes.

If an image must be rolled back, restore the previous pinned image and manifests after checking compatibility with current NodeConfigs and the installed CRD. Rolling back the agent does not restore files or packages it already changed. Keel provides no automatic CRD downgrade or host-state rollback procedure.

Recovery and backups

Keep deployment manifests, NodeConfigs, previous file contents, package version requirements and trigger ConfigMap definitions in version control. Protect sensitive contents according to your existing secret-handling practice; Keel does not fetch Secrets into templates.

For a diagnostic snapshot:

kubectl get nodeconfigs -A -o yaml > keel-nodeconfigs-snapshot.yaml
kubectl -n tak-os get daemonset keel -o yaml > keel-daemonset-snapshot.yaml
kubectl -n tak-os get events --sort-by=.metadata.creationTimestamp > keel-events-snapshot.txt

Exports include server metadata and status and are records for investigation. Restore from the authored manifests, not by blindly applying snapshots. Preserve relevant ConfigMap manifests separately. Back up original host files and package recovery information using your normal host backup process; Keel stores no pre-change host backup.

For a faulty configuration, correct or restore its desired manifest and verify every target node. If repeated application must stop, remove that NodeConfig or change its selector to exclude the affected nodes, accounting for in-progress operations. Keel has no suspend flag. Stopping the agent DaemonSet stops all its future reconciliation once pods terminate; a desired-state manager can recreate it unless its source is also changed. Interrupting package operations can leave the host package database needing repair.

Investigate restartPending on the host and fix the unit or its configuration. The agent retries the pending service list; already successful units can restart again. Use the troubleshooting guide for evidence collection. Local CLI execution has no persisted restart recovery, so a failed local restart requires explicit operator recovery.

Remove Keel

First choose whether to restore host changes or leave them in place. Restore and verify while agents still run if that is the intent. Export configuration and coordinate removal with your deployment manager so it does not reinstall Keel.

From the manifests belonging to the installed release:

# Only if optional admission was installed: registration must go first.
kubectl delete -f deploy/webhook-admission.yaml
kubectl delete -f deploy/webhook.yaml

# Stop agents before removing their permissions.
kubectl delete -f deploy/daemonset.yaml
kubectl -n tak-os wait --for=delete pod -l app.kubernetes.io/name=keel --timeout=180s
kubectl delete -f deploy/rbac.yaml

Remove author Roles/RoleBindings and trigger ConfigMaps you created when no longer needed. The commands above retain NodeConfigs and the CRD. Delete retained NodeConfigs individually after backup if desired. Removing the CRD deletes all NodeConfigs across all namespaces; do that only as an intentional final platform cleanup. Keep tak-os if it contains other applications. Cert-manager-generated TLS Secrets may remain depending on its owner-reference configuration; remove the dedicated keel-webhook-tls Secret if retained and no longer needed.

Uninstalling Keel never deletes managed host files, uninstalls packages, restores permissions or reverses service restarts. Complete any remaining host cleanup through the approved host-maintenance process.