Administrator guide

Architecture and operational limits

The Python controller loads in-cluster credentials when KUBERNETES_PORT is set; otherwise it uses the current kubeconfig. It reads /crd.yaml in the container or crd.yaml relative to the working directory for local execution. At startup it lists CRDs and creates its definition if no CRD with kind Crate exists. It does not update an existing definition and its existence check compares only the kind name, not the API group.

One controller watches crates.k8s.tlspu.com across all namespaces. An ADDED event creates the StatefulSet, client Service, and headless Service, in that order. A DELETED event attempts to delete those three resources individually. MODIFIED events are ignored.

Account for these limits when operating it:

• There is no periodic reconciliation, resource adoption, leader election, or automatic repair. Run a single controller replica for this API group.
• spec.created is a string marker, not status. It is set to "True" even if creation failed. Existing marked requests are skipped when seen again.
• Generated resources have no owner references. There is no finalizer or durable deletion queue; do not rely on cleanup for requests deleted while the controller is unavailable.
• There is no built-in scaling, upgrade, storage expansion, backup, restore, database credential, or TLS workflow.
• API failures are printed to standard output. There is no health endpoint, metrics endpoint, or custom-resource status subresource.

Prerequisites

• A Kubernetes cluster supporting apiextensions.k8s.io/v1 CRDs, apps/v1 StatefulSets, and persistent volume claims.
• Administrative credentials to install a CRD, ClusterRole, ClusterRoleBinding, namespace, and Deployment.
• A default StorageClass or compatible pre-provisioned storage for ReadWriteOnce claims. The operator cannot select a StorageClass.
• Worker nodes and admission policies that permit a privileged BusyBox init container to run sysctl -w vm.max_map_count=262144. This changes a host kernel setting; the workload cannot run unchanged under a policy that forbids it.
• Cluster DNS with the cluster.local suffix, which is embedded in discovery arguments, and network connectivity between database pods on port 4300.
• Registry access for the operator, the selected CrateDB image, and busybox.

The generated database container requests 100m CPU and 512Mi memory, sets CRATE_HEAP_SIZE=256m, and has no resource limits. There are no probes, placement constraints, anti-affinity rules, disruption budgets, or explicit pod security settings beyond the privileged init container. Capacity planning, database security, backup policy, and availability requirements need separate review. The repository does not define a tested Kubernetes or CrateDB version matrix.

Build and install

Run these commands from the repository root. Substitute a registry and version tag that your cluster can pull. Building and pushing require Docker and registry credentials.

export OPERATOR_IMAGE=registry.example.com/platform/crate-operator:1.0.0
docker build -t "$OPERATOR_IMAGE" .
docker run --rm --entrypoint python "$OPERATOR_IMAGE" -c 'import kubernetes, yaml'
docker push "$OPERATOR_IMAGE"

The Dockerfile uses python:alpine and installs the Kubernetes Python client without an upper version bound. It does not pin a dependency set; the import check above checks that the built image includes both runtime libraries.

The checked-in deployment template still grants access to the legacy crate-operator.limbicly.com group. Render a local installation manifest with the current group and your image:

sed \
  -e 's/crate-operator\.limbicly\.com/k8s.tlspu.com/g' \
  -e "s|{{IMAGE}}|${OPERATOR_IMAGE}|g" \
  manifests/01-operator.yaml.jinja2 > /tmp/crate-operator-install.yaml

kubectl apply -f crd.yaml
kubectl wait --for=condition=Established \
  crd/crates.k8s.tlspu.com --timeout=60s
kubectl apply -f /tmp/crate-operator-install.yaml
kubectl rollout status deployment/crate-controller \
  -n crate-operator --timeout=300s
kubectl logs -n crate-operator deployment/crate-controller

Review the rendered manifest before applying it in your environment. Applying the CRD explicitly makes its schema available before the controller starts. The Deployment is named crate-controller, runs one replica, and uses the Recreate strategy. Its log should reach Waiting for Crate Cluster requests.

Keep the rendered manifest, image reference, and any environment-specific changes in your deployment configuration. The commands above create a local file; they do not change the checked-in template. The legacy test fixture also needs its API version changed to k8s.tlspu.com/v1 before use. Prefer the complete example in the user guide.

Controller permissions

The template binds ClusterRole cc-clusterrole to the default ServiceAccount in namespace crate-operator. The controller needs:

The template grants some additional verbs. If you use a dedicated ServiceAccount or reduce permissions, change both the Deployment and binding consistently. The controller does not directly create pods or claims; the StatefulSet controller creates them. Application users need separate namespace-level access to custom resources and whatever inspection or port forwarding their role needs.

For example, an administrator allowed to impersonate ServiceAccounts can check:

kubectl auth can-i watch crates.k8s.tlspu.com --all-namespaces \
  --as=system:serviceaccount:crate-operator:default
kubectl auth can-i update crates.k8s.tlspu.com -n test-crate \
  --as=system:serviceaccount:crate-operator:default
kubectl auth can-i create statefulsets.apps -n test-crate \
  --as=system:serviceaccount:crate-operator:default

Validate an installation

Create the three-node request from the user guide in a disposable namespace. Check that the CRD reports the current API group and that all resources appear:

kubectl get crd crates.k8s.tlspu.com
kubectl get crates.k8s.tlspu.com simple -n test-crate -o yaml
kubectl get statefulset/simple -n test-crate
kubectl get pods -n test-crate -l app=simple -o wide
kubectl get service/simple service/simple-headless -n test-crate
kubectl get pvc -n test-crate
kubectl logs -n crate-operator deployment/crate-controller

Verify three database pods, bound claims, and Service endpoints. Use port forwarding as described in the user guide to verify database access and inspect node membership. A successful rollout alone is insufficient because no database readiness probe is configured. Delete the test request while the controller is running, verify workload cleanup, and separately remove test data only when it is no longer needed.

Maintenance and data lifecycle

Controller updates

Build and validate a new image, update OPERATOR_IMAGE, render the installation manifest again, and apply it. Watch the Deployment rollout and controller logs. Applying a new controller image does not upgrade existing database containers. Schema changes must be reviewed and applied explicitly with crd.yaml; startup does not migrate an existing CRD. Avoid deleting requests during controller downtime because cleanup depends on observed deletion events.

Database changes

Changes to a Crate spec do not reach existing resources. Plan database upgrades, scaling, or storage changes as manual administrative work with a tested backup and recovery procedure. Changing StatefulSet replicas alone also leaves the generated EXPECTED_NODES environment variable and bootstrap node list at their original values. Review CrateDB's requirements for the deployed version before changing these settings.

There is no supported configuration field for heap size, CPU/memory settings, StorageClass, authentication, TLS, or probes. Such changes require maintaining controller templates or explicitly managing the generated resources. Keep any manual changes in deployment records, since the request no longer fully describes the workload.

Retained data and deletion

The controller deletes the StatefulSet and Services but never explicitly deletes PVCs. Its StatefulSet template does not set a PVC retention policy; under the default retention behavior the claims remain. Check the live workload and your storage policies, including admission-time changes, before relying on retention. Deleting a PVC can also delete its backing volume according to the volume's reclaim policy.

For simple in test-crate, retained claims are named data-simple-<ordinal>. Inspect them explicitly before cleanup:

kubectl get pvc -n test-crate
kubectl get pv

If a request is recreated with the same name, the StatefulSet may reuse retained claims. This reuses existing data; it is not a fresh database. Verify data ownership, image compatibility, and recovery requirements first. Do not delete and recreate a request as a routine upgrade or retry mechanism.

Troubleshooting

Useful diagnostics for the user-guide example:

kubectl logs -n crate-operator deployment/crate-controller
kubectl describe pod simple-0 -n test-crate
kubectl logs simple-0 -n test-crate -c init-sysctl
kubectl logs simple-0 -n test-crate -c simple
kubectl get endpoints simple simple-headless -n test-crate
kubectl get events -n test-crate --sort-by=.metadata.creationTimestamp

For a partial creation, preserve the request, logs, resource definitions, and claims while determining what succeeded. Fix the underlying API, admission, image, or storage problem, then plan a targeted repair against the controller's templates. Replaying creation against existing resources can fail with AlreadyExists: the StatefulSet is created first, and an error there prevents Service creation. Setting created to "False" does not trigger repair because the resulting modification event is ignored. Do not use marker changes as a health check or recovery procedure.

Local controller execution

For development, use a Python virtual environment with kubernetes and PyYAML installed, a valid kubeconfig, and the permissions described above. Run from the repository root so that crd.yaml is found:

python3 -m venv .venv
.venv/bin/pip install kubernetes PyYAML
.venv/bin/python -u controller.py

Ensure KUBERNETES_PORT is unset when running outside a pod. Stop the deployed controller before running a local instance against the same cluster. Local execution uses the current kubeconfig context and has the same provisioning and deletion effects as the deployed controller.

Remove the operator

While the controller is still running, inventory requests across namespaces, back up data as required, and delete the requests whose clusters should be removed. Verify StatefulSet and Service cleanup, and decide separately how to handle each retained claim.

kubectl get crates.k8s.tlspu.com --all-namespaces

When no requests remain and data handling is complete, remove the controller's installation resources. These names follow the supplied template:

kubectl delete deployment crate-controller -n crate-operator
kubectl delete clusterrolebinding cc-binding
kubectl delete clusterrole cc-clusterrole
kubectl delete crd crates.k8s.tlspu.com

Deleting the CRD removes its custom resources. If requests remain, do not assume their workloads will be cleaned up as part of CRD removal. Delete namespace crate-operator only after confirming it contains no resources you need to keep.