The k3s Home Lab, Part 3: Closing the Loop with Flux

After the Docker-to-k3s migration, the cluster was declarative in theory but still imperative in practice. Manifests lived on a workstation, and the last person to run kubectl apply held context that Git did not. Rebuilding the cluster meant finding the right files, remembering their order, and distinguishing intended configuration from experiments.

Flux solved the ownership problem. It watches a source, builds the declared configuration, applies it with server-side reconciliation, and reports whether the live cluster converged. More importantly, it continuously corrects drift. Git became the reviewed record; the cluster became a consumer of that record.

Bootstrap Is a Trust Decision

Flux bootstrap installs the controllers, commits their manifests to the repository, and creates the source and Kustomization objects that let Flux manage itself. That circular-looking model is the feature: after bootstrap, Flux upgrades and configuration changes follow the same Git workflow as applications.

Before running it, I decided what identity Flux would use, whether it needed read-only or write access, and which repository path represented this cluster. A controller that can apply cluster resources is privileged even when its Git key is read-only. Repository protection, narrow deploy credentials, controller network policy, and review rules are part of the cluster security boundary.

Repository Structure Expresses Dependency

clusters/
  home-lab/
    flux-system/
    infrastructure.yaml
    apps.yaml
infrastructure/
  controllers/
  networking/
  storage/
apps/
  base/
  home-lab/

I separated cluster entry points, shared infrastructure, and applications. The exact folder names matter less than one property: a new operator should be able to tell what applies to this cluster and in what order.

Flux Kustomization objects made that order explicit. Namespaces and controllers reconcile before resources that depend on their custom resource definitions. Networking and storage foundations settle before applications. Health checks and dependsOn turn ordering into an observed condition rather than a sleep timer.

Reconciliation Boundaries Matter

One giant Kustomization is easy to start and painful to diagnose. A malformed application can block unrelated infrastructure, while broad permissions make every reconciliation more powerful than it needs to be. I split the repository into boundaries that matched ownership and failure impact.

Pruning was enabled deliberately. It is what makes deletion in Git delete the live object, but it also turns a misplaced path or rename into a destructive instruction. Small commits, dependency-aware boundaries, and reviewing the rendered difference reduced that risk.

Secrets Need Their Own Workflow

Plain Kubernetes Secret manifests are encoding, not encryption. I did not want credentials in Git history, but I also did not want “create these secrets by hand” to become the one undocumented step that prevented recovery.

The durable pattern is encrypted secrets or an external secret store, with decryption available only inside the intended cluster. Regardless of tool, recovery testing must include the key path. A perfectly versioned encrypted file is useless if the decryption identity disappears with the failed cluster.

I also separated Git read access from any optional image automation write access. Flux's image controllers can scan registries, select tags according to policy, and commit updates, but that deserves a distinct identity and branch protection rather than silently expanding the bootstrap credential.

Operations Changed More Than Deployment

The normal change path became edit, validate, review, merge, reconcile. Emergency changes could still be made with Kubernetes tools, but Flux would expose and eventually correct the drift. That changed the question during troubleshooting from “what did someone run?” to “which revision is this cluster trying to realize?”

Observability remained essential. A Git commit being correct does not mean the source fetched, the build passed, the health checks succeeded, or the application became reachable. Flux conditions and events joined Kubernetes events, controller logs, ingress checks, and application monitoring as one diagnostic chain.

What Flux Actually Installed

The bootstrap process deployed a set of cooperating controllers rather than one monolithic agent. The source controller retrieves Git or other artifacts. The kustomize controller builds and applies manifests. The helm controller manages Helm releases. The notification controller connects reconciliation events to external systems. Optional image controllers discover registry tags and write approved updates back to Git.

Understanding those boundaries made troubleshooting more deterministic. If a repository revision was not available, I looked at the source object and source-controller conditions. If the artifact existed but resources failed to apply, the Kustomization and kustomize-controller events were the next layer. If a Helm release stalled, the chart source, values, release conditions, and helm-controller logs formed their own chain.

From One Kustomization to a Dependency Graph

The first working layout can be deceptively flat: one root Kustomization recursively includes everything. That approach hides dependencies and expands blast radius. I moved toward separate Flux Kustomization resources with explicit source paths, intervals, timeouts, pruning behavior, health checks, and dependencies.

Custom resource definitions are a classic ordering problem. A controller's CRDs must exist before Flux can successfully apply custom resources of that kind. Namespaces should exist before namespaced configuration. An ingress route should not be considered healthy merely because its YAML applied if the ingress controller itself is unavailable.

dependsOn expresses the coarse graph, while health checks determine whether a dependency is actually ready. This is different from file order. Git directories are for human organization; reconciliation objects are the platform's execution model.

I avoided making the graph too granular. One reconciliation object per manifest creates operational noise and a complicated dependency web. The useful boundary is a unit with a shared lifecycle, permission scope, and failure impact.

Kustomize and Helm Play Different Roles

Kustomize worked well for manifests I owned and for environment-specific overlays. Base resources captured common application intent, while the home-lab overlay supplied hostnames, storage classes, replica counts, and other cluster-specific values. The rendered output remained ordinary Kubernetes objects that could be validated before merge.

HelmRelease resources were useful for third-party controllers distributed as charts. Flux still kept the release declarative: chart source, version range, values, remediation, and upgrade behavior lived in Git. I pinned versions or controlled version ranges rather than silently consuming whatever a repository considered latest.

Values files can become an unstructured dumping ground, so I kept overrides minimal and documented why the cluster differed from chart defaults. A large copied default values file makes upgrades harder because it preserves old assumptions and hides meaningful changes among hundreds of unused settings.

Validation Before Reconciliation

Flux is an excellent reconciler, but the cluster should not be the first parser and policy engine to see a change. Local and continuous-integration checks can build Kustomize overlays, render Helm charts, validate YAML and Kubernetes schemas, scan images and manifests, and enforce policy before merge.

This mattered most for destructive changes. Renaming a resource may be a delete-and-create operation. Moving a manifest outside a reconciled path can cause pruning. Changing an immutable field can force replacement. Reviewing source text alone may not reveal the live impact, so I looked at rendered objects and the expected add, change, and delete set.

Validation also protected the bootstrap repository itself. A broken application should not prevent Flux system resources from reconciling. Separate paths and checks made it harder for one malformed overlay to block the entire cluster definition.

Pruning, Drift, and Emergency Changes

Pruning is central to GitOps because it makes absence declarative. Without it, deleting YAML leaves abandoned resources running indefinitely. With it, an incorrect path, branch, or refactor can remove healthy resources. I enabled pruning at boundaries where Git truly owned the full lifecycle and treated repository moves as production changes.

Drift correction created a new operational rule: manual edits are temporary unless they are committed. That is good for consistency, but it can surprise an operator responding to an incident. If an emergency edit is necessary, the team must either suspend the relevant reconciliation, commit the intended state immediately, or expect Flux to revert the change.

Suspension is a scalpel, not a maintenance mode for forgetting about automation. I recorded why a source or Kustomization was suspended and resumed it as part of the same incident workflow. A suspended reconciler silently converts Git from desired state into stale documentation.

Secret Decryption and Recovery

Encrypted secrets solved only one half of the problem. Flux also needed a decryption identity inside the cluster, and that identity needed a recovery process outside the cluster. I separated the encrypted payloads—which could safely live in Git—from the key material required to unlock them.

The recovery sequence had to be explicit: create the cluster, install or restore the decryption identity through a protected channel, bootstrap Flux, and allow the remaining secrets and workloads to reconcile. If that first secret depended on Flux to exist, the design contained a circular dependency.

Rotation was tested as well. New credentials must reach the consuming workloads without exposing plaintext in Git, logs, shell history, or rendered diagnostics. Old credentials should remain valid only for the overlap required to complete the rollout. Git history gives an audit trail for encrypted objects, but access logs and rotation records still belong in the operational process.

Image Automation Without Surrendering Change Control

Flux image automation can watch a registry, select a version according to semantic-version, alphabetical, or numerical policy, update marked fields, and commit the result. That is more accountable than a floating latest tag because the selected version becomes a Git change.

I treated automatic image updates as a separate maturity step. Registry credentials, tag conventions, policy ranges, update frequency, and rollback behavior all needed to be predictable first. Infrastructure controllers and stateful services usually deserved tighter version constraints than low-risk stateless tools.

A good policy answers not just “is there a newer image?” but “which versions are allowed to cross this boundary automatically?” Patch releases may be reasonable for one service, while another requires a reviewed chart and database migration. Automation should encode that risk difference rather than erase it.

Testing Rebuild, Not Just Reconciliation

The strongest GitOps test was not changing a replica count. It was asking how much of the environment could return from an empty cluster. The exercise exposed every hidden dependency: DNS records created by hand, storage paths prepared outside code, certificates with no renewal path, secret keys stored on one workstation, or an application backup that had never been restored.

I separated cluster reconstruction from data restoration. Flux can recreate namespaces, controllers, policies, Services, and application definitions. It cannot invent database contents or persistent files. Recovery documentation therefore linked each stateful workload to its backup source, restore procedure, integrity check, and point at which Flux should resume managing it.

The target was not a magical one-command disaster recovery story. It was a bounded, explainable sequence in which Git rebuilt the platform, protected secrets unlocked configuration, backups restored state, and health checks proved that services returned.

Lessons Learned

GitOps is not Git-backed copy and paste. Its value is continuous convergence, explicit dependency, reviewable history, and a recovery path that does not depend on one workstation.

Reconciliation also amplifies mistakes. A bad manifest is no longer a one-time command; it is an instruction the platform will keep trying to enforce. Validation, protected branches, scoped permissions, and small reconciliation units matter more after automation, not less.

The best test was rebuilding. Bootstrap Flux, restore its decryption capability, and watch infrastructure and applications return in order. Wherever that process required memory or a manual command, the platform was telling me exactly what still needed to become code.