The k3s Home Lab, Part 2: From Compose to Reconciliation

Once the k3s foundation was stable, the temptation was to move everything. That would have been the fastest way to turn a platform project into an outage.

The existing Docker and Traefik deployment was known, understandable, and easy to restore from Compose. The Kubernetes cluster was newer and had more moving parts. I treated migration as a series of service-level changes, starting with stateless, low-impact applications and leaving stateful or infrastructure-adjacent workloads until the storage, backup, and ingress paths had proved themselves.

Inventory Before Translation

A Compose file tells only part of a service's story. I inventoried images, ports, labels, environment variables, bind mounts, named volumes, health checks, restart behavior, DNS dependencies, scheduled tasks, outbound access, backup jobs, and the users who would notice a failure.

Each item then mapped to a Kubernetes concept. Containers became Deployments or StatefulSets; Docker networks became Services and policy; Traefik labels became Ingress or Traefik custom resources; environment files became ConfigMaps and Secrets; and volumes became PersistentVolumeClaims. The mapping exposed assumptions Compose had allowed me to leave implicit.

Health Checks Became Routing Decisions

In Compose, a container being “up” had often been good enough. In Kubernetes, liveness, readiness, and startup probes have different jobs. A weak liveness probe can restart a slow but healthy application. No readiness probe can send traffic to a process before its database migration or cache warm-up has finished.

I started with readiness, because removing an unhealthy pod from service is safer than restarting it. Startup probes protected slow initializations. Liveness checks were added only when there was a reliable signal that a restart would improve the situation.

Ingress Without a Big-Bang DNS Change

The old and new Traefik paths could coexist during migration. For each service, I deployed the Kubernetes workload under a temporary internal name, validated TLS, headers, redirects, uploads, and authentication, then moved the normal DNS record during a planned cutover.

Low DNS time-to-live values helped, but DNS was not the rollback plan by itself. The old container remained stopped but intact until the new workload had survived normal use and a node reschedule. If the new route failed, I could restore the previous record and start the Compose service without reconstructing it under pressure.

Stateful Services Were Data Migrations

A container image is portable; application state often is not. For stateful services I documented ownership, permissions, database version, quiescing steps, export format, destination storage, integrity checks, and rollback boundaries. Copying a live database directory into a PVC is not a migration strategy.

The safe pattern was application-aware: stop writes, take a backup or supported export, restore into the destination, verify records and permissions, then expose the new instance. Backups stayed outside Kubernetes. A PVC protects scheduling semantics; it is not a backup and it does not prove recoverability.

Security Improved, but Not Automatically

Moving away from the Docker provider removed Traefik's need for the Docker socket proxy. Kubernetes replaced it with API access controlled through service accounts and RBAC. That is a stronger model because permissions can be scoped to resources and namespaces, but default service accounts, overbroad roles, unencrypted Secrets, and unrestricted pod traffic can recreate the same trust problem in a different form.

I used non-root containers where images allowed it, explicit resource requests and limits, namespace boundaries, minimal service-account permissions, and deny-first network policy for workloads with clear flows. Those controls were introduced gradually so a security change and an application migration were not impossible to troubleshoot separately.

Choosing the Migration Order

I scored services by statefulness, user impact, dependency count, recovery confidence, and network complexity. A disposable web tool with no login and no volume is an excellent first candidate. A home-automation platform with hardware integrations, time-sensitive events, and a database is not.

This order made the platform earn trust. Every wave reused the tooling and lessons from the previous one. It also prevented the migration from becoming a single event with a single, enormous rollback decision.

Translating Configuration Without Copying Old Mistakes

A direct conversion tool can produce Kubernetes YAML from Compose, but syntactic conversion does not produce a good operational model. I reviewed every environment variable and mount instead. Static, non-sensitive settings belonged in ConfigMaps. Credentials belonged in Secrets. Application defaults that did not need overriding disappeared from the deployment entirely.

Bind-mounted configuration files needed special attention. Some applications rewrite their configuration at runtime, which makes a read-only ConfigMap mount inappropriate. Others expect ownership or permissions that differ from the container's runtime user. Init containers can prepare writable directories, but they also add ordering and failure modes. The right answer depended on whether the file was desired configuration or application-managed state.

I also stopped using container start order as a dependency mechanism. Kubernetes will schedule a web application and its database independently. The application must retry connections, expose readiness accurately, and recover when the database restarts later. That is a healthier distributed-system behavior than assuming one successful boot sequence will remain true forever.

Designing the Kubernetes Workload

A Deployment was the default for replaceable replicas. StatefulSets were reserved for workloads that actually required stable network identity or ordered, persistent replicas—not simply for anything with a volume. CronJobs replaced host-level scheduled container invocations where the task belonged to the application lifecycle.

Services received stable names and explicit ports. Selectors were kept narrow and consistent with the pod template. A surprising number of “ingress problems” are actually Services selecting no pods or forwarding to the wrong target port, so endpoint validation became part of every deployment check.

I added a pod disruption budget only where multiple healthy replicas could genuinely satisfy it. A budget does not create availability. Applied to a single-replica application, it may merely prevent a voluntary node drain. Replica counts, shared-nothing behavior, storage attachment, and dependency availability have to support the promise first.

Resource requests began conservatively and were adjusted from observed use. Missing requests allowed workloads to pack unpredictably; unrealistic limits produced throttling or OOM kills. The goal was not perfect capacity modeling, but enough information for the scheduler and enough guardrail to stop one service from exhausting a node.

A Repeatable Cutover Runbook

Every service used the same basic runbook, with application-specific data steps added where necessary:

1. Capture the existing image version, configuration, DNS record, volume location, and known-good backup.
2. Deploy the Kubernetes workload under a temporary hostname without changing the existing service.
3. Verify startup, readiness, authentication, uploads, redirects, scheduled jobs, logs, and outbound dependencies.
4. For stateful services, stop writes, export or back up data, restore it to the new storage, and verify integrity.
5. Reduce DNS TTL in advance, switch the production name, and test from every relevant VLAN or remote-access path.
6. Observe normal use, restart the pod, and drain its node to verify rescheduling behavior.
7. Retain the stopped Compose workload and original data through an agreed rollback window.
8. Only then remove the old runtime definition and age out obsolete backups according to retention policy.

The runbook turned migration into a routine operation. It also made the stop condition clear. If integrity checks, network tests, or rollback prerequisites failed, the service did not move that day.

Problems That Looked Like Kubernetes Problems

Several failures originated outside Kubernetes. Split DNS returned the old proxy address to one client network. A firewall rule permitted access from the trusted VLAN but not the remote Tailscale path. An application generated redirects using its internal scheme because forwarded headers were incomplete. A restored volume had the right files but the wrong numeric user ID.

The lesson was to test complete user journeys, not only cluster objects. A ready Deployment, populated endpoint list, and valid Ingress prove that controllers agree; they do not prove that a browser on an IoT or trusted network can authenticate and complete the intended task.

Inside the cluster, the most common issues were equally mundane: mismatched selectors, incorrect target ports, missing environment values, storage permissions, and probes that were too aggressive. Kubernetes made those failures visible, but it did not make application assumptions disappear.

When Not to Migrate

Kubernetes is not automatically the best destination for every home-lab workload. Services that require direct hardware access, unusual broadcast or multicast behavior, host networking, privileged operation, or extremely simple single-host lifecycle may be clearer and safer outside the cluster.

I treated “remains on a VM” as a valid architecture decision, not unfinished work. The goal was a more operable platform, not a Kubernetes logo attached to every process. A mixed environment can be easier to secure and recover when each workload runs where its dependencies and failure modes are best understood.

What I Would Do Again

Migrate the boring service first. A small stateless application exercises image pulls, DNS, ingress, certificates, logging, probes, and rollback without putting important data at risk.

Keep the source platform recoverable. Deleting old volumes after the first successful page load saves little and removes the cheapest rollback option.

Finally, measure success by operations, not deployment. The real milestone was not a green pod. It was surviving an upgrade, a node drain, a restore test, and a configuration change without manual reconstruction. That was the point at which the service had actually moved to Kubernetes.