Orbit Deploy Application Logic

1. Overview:
    - Orbit Deploy is a tool for automating Kubernetes deployments.
    - Each configuration is called an Orbit Deployment.
    - An Orbit Deployment is linked to a single Git repository and optionally:
        - A folder path within the repository (default is "default").
        - A Git reference (branch or tag).
        - Replacement strings for token substitution.
        - A secret containing credentials for the Git repository. HTTP(S) Git uses
            "username" and "password"; SSH Git uses "sshPrivateKey" and may also use
            "sshPasshrase". If "sshKnownHosts" is omitted or empty, SSH host validation
            is skipped; otherwise the configured known-host entries are used for
            validation.

2. Folder Structure:
   - The folder path in the repository is called an Orbit Source.
   - The Orbit Source contains YAML-formatted Kubernetes manifests.
   - The Orbit Source may contain arbitrarily deep folders.
   - Files that do not end with ".yaml" are ignored.

3. Replacement Tokens:
    - Tokens take the form `<!key!>` and are resolved recursively at the point of use.
    - Default Replacement Tokens (Lowest priority):
        - Environmental Default Replacement Tokens:
            - domain - Extracted from the hostname, or "localhost"
            - addresses - The IP addresses of the nodes in the cluster as a list
            - external_addresses - The external IP addresses of the nodes in the cluster (if any are defined on the node objects)
        - Config Map Default Replacement Tokens:
            - From a config map called orbit-deploy in the kube-system namespace every key/value pair is loaded as a Default Replacement Token.
    - Primary Replacement (gitops) Tokens:
        - Defined in `deployment.json` files within the folder tree.
        - Consolidated as you traverse deeper into the folder tree (deeper values override higher ones).
    - Secondary Replacement (local) Tokens:
        - Defined in the `spec.replacements` field of the Orbit Deployment.
        - Have the highest precedence.
    - Image Tokens (Highest priority):
        - Dynamically created from the `images` field in `deployment.json`.
        - Format: `<!image-key!>` resolves to the local OCI registry URL.
        - Format: `<!image-key-hash!>` resolves to the SHA256 hash of the image manifest.

4. OCI Registry Integration:
    - Images specified in `deployment.json` are added as replacement tokens "image-key" for use in manifests.
    - Each image also has its current hash recorded as a replacement token "image-key-hash"
    - The "image-key-hash" token ensures that deployments automatically update pods when the image changes.
    - When a manifest references an image token, Orbit Deploy stamps an informational label on the object so the original
      reference remains visible even though the image in the manifest is rewritten to a content digest:
        - `k8s.tlspu.com/original-image` - the last path component of the image name (typically the project name) together
          with the original image tag, hyphen-separated (e.g. `access-controller-1.1.0`). Multi-image manifests are joined
          with a hyphen. Only images the object actually references are included.
      If the label is already present on the object it is replaced with the value for the image currently deployed. No
      separate version label is added.
    - As a final safety net the value is re-validated immediately before it is written (at most 63 bytes and no characters
      invalid in a label value). If either check fails the label is skipped entirely and an error is logged in the form
      `Failed to add original version label. Got [<value>]`.

5. Manifest Processing:
    - Manifests are split into individual YAML documents.
    - Token replacement is performed on each document.
    - Processing rules:
        - If the object already exists and the annotation `orbit-deploy-latest` matches the processed YAML, the document is skipped.
        - If the object exists and the annotation `com.tlspu.k8s.orbit-deploy/no-update` is present, the document is skipped.
        - Otherwise, the YAML is applied to the Kubernetes API.
        - If the manifest is successfully applied, the object metadata (e.g., `apiVersion`, `kind`, `namespace`, `name`) is added to the `managedObjects` field of the Orbit Deployment and any error message storeed is removed
        - If there is a processing error then record the error message return by K8S API in errorMessages, replacing any previous error message for this object.
        - Ownership arbitration: before creating or updating an object, the shared ownership map is consulted. A different
          OrbitDeployment with priority greater than or equal to the incoming manifest blocks the action. The owner record is
          only honoured while the object still exists in the cluster: if the object is absent (for example, a previous owner
          was deleted and its cleanup failed to remove its owner record), the record is stale and is ignored, and the create
          proceeds. If you have deleted the object, you are no longer the owner.

6. Reconciliation:
    - After processing all manifests for an Orbit Deployment:
        - Compare the list of objects in `managedObjects` with the objects present in the Git repository.
        - If an object in `managedObjects` is no longer present in the repository, if the object does not have the annotation "com.tlspu.k8s.orbitdeploy/no-delete", delete it from the Kubernetes cluster.
        - Remove the deleted object from the `managedObjects` list and from the errors list.
    - If an object fails to delete, log the error with prefix `X!== Error deleting` and continue with the next object.
    - Successful deletes are logged with prefix `<=== Delete`; blocked deletes with `!=== No-Delete`; objects already
      absent with `!=== Already-Deleted`.
    - When an OrbitDeployment is deleted, all the objects it manages should be deleted, ignoring errors

7. Processing Flow:
    - For each Orbit Deployment:
        - Clone the Git repository and check out the specified branch/tag and folder.
        - Traverse the folder tree, consolidating `deployment.json` values.
        - Process all YAML manifests in the folder tree.
        - When a manifest is the same OrbitDeployment as the worker, applied spec fields are synced into memory after
          create/replace; status-only writes at end of run avoid reverting spec from the run-start snapshot.
        - Status merge-patch sends null for keys removed from errors or managedObjects since load (omitted keys are not
          deleted by merge-patch).
        - Reconcile objects by cleaning up resources no longer present in the repository.
        - Report progress heartbeats during long steps (load, git fetch, image cache, each manifest, deletes, store).
    - Sleep for 30 seconds before the next loop.
    - While a reconcile run is in progress, deferred reload of the OrbitDeployment CR from watch events is applied at
      end of run.
    - A background watchdog (Orbit-Worker-Watchdog) restarts workers that stop reporting progress mid-run. Configure with
      ORBIT_WORKER_WATCHDOG (default true), ORBIT_WORKER_WATCHDOG_INTERVAL (default 60s), and ORBIT_WORKER_STUCK_SECONDS
      (default 1800s). See README.md for details.

8. Deployment.json Format:
    - Each folder may contain a `deployment.json` file with the following structure:
        {
            "replacements": {
                "key1": "value1",
                "key2": "value2"
            },
            "images": {
                "app1": "registry.example.com/app1:tag"
                "app2": "registry.example.com/app2:tag2"
            }
        }
    - Replacements and images consolidate as you traverse deeper into the folder tree.
    - Values from deeper folders override earlier ones.
    - Excluding image tokesn, replacements from the Orbit Deployment object have the highest precedence.

9. Error Handling:
    - On failure to apply a manifest, the error is recorded in the Orbit Deployment's `errors` attribute under the key `processing-name`.
    - On success, the error is cleared, and the processed YAML is stored in `lastConfig` under the key `processing-name`.

10. Additional Notes:
    - The tool loops through all defined Orbit Deployments continuously.
    - After all deployments are processed, the tool sleeps for 30 seconds before starting the next loop.