The k3s Home Lab, Part 1: Small Cluster, Real Architecture

Docker Compose was a good home for individual services, but I wanted a platform that could reconcile desired state, spread workloads across nodes, expose a consistent API, and support GitOps cleanly. Full upstream Kubernetes would have worked; k3s fit the scale of the lab better.

k3s packages the Kubernetes control plane, containerd, networking, and common supporting components into a smaller operational footprint. That makes installation easier. It does not decide where the control plane belongs, how persistent data survives, which nodes receive ingress traffic, or what “high availability” really means when every node is a VM on the same physical hypervisor.

Start With Failure Domains

The cluster runs as virtual machines across the Proxmox environment. The important design step was mapping Kubernetes nodes to physical failure domains. Multiple control-plane VMs on one host may look redundant to Kubernetes while still failing together when that host reboots.

For an HA control plane with embedded etcd, k3s requires an odd number of server nodes and quorum. Three servers tolerate one unavailable member; adding a fourth does not increase that tolerance. Stable storage latency matters because etcd is a consensus database, not an ordinary application volume.

A fixed registration address also keeps node joins and administration independent of one server's address. Agents initially use that endpoint, then maintain connections to available servers. The virtual IP or load balancer in front of the API is therefore part of the control-plane design, not a cosmetic extra.

The Layers of the Cluster

Networking Was the Real Deployment

The lab already separated trusted, semi-trusted, IoT, guest, DMZ, and sandbox networks with OPNsense. Kubernetes added pod and service address spaces that also needed to be unique and routable only where intended. I treated the node VLAN, API access, ingress traffic, DNS, storage, and management paths as separate flows rather than opening broad node-to-node access and hoping the overlay would hide it.

k3s installs CoreDNS, Traefik, a network-policy controller, and ServiceLB by default. Those defaults are productive, but they have consequences. ServiceLB can claim host ports across eligible nodes, and the bundled Traefik service commonly consumes 80 and 443. Before installing applications, I decided which nodes should accept ingress and how internal DNS would resolve service names to those addresses.

NetworkPolicy was another lesson in layers. A policy object is only useful when the selected networking implementation enforces it. I tested both allowed and denied flows rather than treating the presence of YAML as proof of isolation.

State Changes Everything

Stateless services were easy to schedule. Persistent workloads forced harder questions: is the volume tied to one node, backed by shared storage, or replicated by the application? What happens after a node drain? Is the backup of the volume consistent with the database inside it? Can the service be restored without the cluster that created it?

I kept cluster state and application data as separate recovery problems. Etcd snapshots protect Kubernetes state. Application backups protect what users care about. Both feed the existing NAS and offsite replication strategy, and both require restore tests.

Building the Control Plane Deliberately

The easiest k3s installation is a single command. The more important work happens before that command: assigning stable addresses, deciding how names resolve, confirming time synchronization, documenting the cluster and service CIDRs, and making sure every server starts with compatible configuration.

Several k3s server flags must agree across the control plane, including network ranges, cluster DNS settings, disabled packaged components, and secrets-encryption behavior. A mismatch is not merely untidy configuration. It can prevent a server from joining or leave the cluster with behavior that depends on which control-plane node processed a request. I kept the server configuration in one repeatable definition instead of building each node from shell history.

The join token also deserves more care than a copied installation snippet suggests. It is a cluster credential and participates in protecting bootstrap data. I handled it as secret material, restricted its distribution, and included its recovery path in the control-plane backup plan. A cluster backup without the information required to restore it is an archive, not a recovery capability.

After initialization, I validated the control plane from failure outward. Could the API still be reached when one server was unavailable? Did agents reconnect through the stable endpoint? Did etcd retain quorum? Did a server return cleanly after maintenance? It is better to answer those questions during commissioning than during a hypervisor outage.

Scheduling Across Virtual and Physical Boundaries

Kubernetes schedules against nodes, while the actual failure domains live one layer lower in Proxmox. Without additional context, the scheduler does not know that two node VMs share a physical host, storage path, or power source. Labels and placement rules bridge part of that gap.

I used node labels to describe meaningful capabilities and locations rather than encoding them in application names. Workloads can then request the characteristic they require, such as ingress eligibility or access to a particular storage path. Taints are useful for the opposite direction: reserving control-plane or specialized nodes so ordinary workloads do not land there by accident.

Pod anti-affinity and topology spread constraints are valuable for replicated applications, but they need honest topology labels. Three replicas spread across three VMs on one Proxmox host still share a failure domain. The cluster view and the virtualization view must tell the same story.

Resource requests were another operational improvement. Without them, the scheduler has little evidence for placement, and one noisy workload can make an apparently healthy node unreliable. Requests represent the capacity an application needs for normal scheduling; limits are guardrails, not a substitute for measurement. Setting arbitrary low CPU or memory limits can create throttling and out-of-memory restarts that look like application defects.

Ingress, DNS, and the Path to a Pod

When an internal hostname resolves successfully, several systems still have to agree before the application loads. The client queries the local resolver, OPNsense permits the flow, traffic reaches an ingress address, ServiceLB or another load-balancer mechanism lands it on an eligible node, Traefik selects a router, a Kubernetes Service selects ready endpoints, and the pod accepts the request.

That path became my troubleshooting model. I worked from the outside inward instead of restarting random components:

This layered view shortened outages because each check eliminated an entire category of guesses. It also made monitoring more useful: an ingress probe and a pod health check answer different questions and both are necessary.

Upgrades Are an Architecture Test

A cluster that works only when untouched is not reliable. I treated upgrades as controlled failure exercises. Before changing versions, I reviewed k3s release notes, confirmed the supported Kubernetes version skew, captured datastore and application backups, and checked that the cluster was healthy.

Server nodes come first in a deliberate sequence, one at a time, while preserving etcd quorum. Agents follow after the control plane is stable. Draining a node before maintenance tests whether workloads can move and exposes local-storage or disruption-budget assumptions. Uncordoning only after the node and system pods are healthy prevents the scheduler from adding workload pressure during recovery.

The rollback question must be answered before upgrade day. Kubernetes data stores and custom resource definitions can move forward in ways that make binary rollback unsafe. “Reinstall the previous package” is not always a recovery plan. Snapshots, documented versions, configuration backups, and a tested restore process are the real boundary.

Security Boundaries Inside the Cluster

VLAN segmentation protects the cluster from other network zones, but it does not provide workload-level authorization inside Kubernetes. Namespaces organize resources and provide policy boundaries; they do not isolate traffic by themselves. NetworkPolicy, RBAC, service accounts, pod security settings, and secret handling do the enforcement work.

I avoided giving application pods the default service account token unless they needed the Kubernetes API. For controllers that did need access, permissions were scoped to the resources and namespaces they reconciled. Container settings such as running as a non-root user, dropping unnecessary Linux capabilities, using a read-only root filesystem where practical, and preventing privilege escalation reduced the impact of an application compromise.

Secrets required a similarly realistic view. A Kubernetes Secret is base64-encoded data stored through the API; it is not automatically an enterprise secret-management solution. Enabling datastore encryption, limiting RBAC access, avoiding secret values in logs and manifests, and designing a rotation workflow all matter. GitOps later gave that process a repeatable delivery path, but it did not remove the need to protect the underlying values.

Observability Before Applications

I wanted the platform to explain itself before it hosted important services. At minimum that meant node conditions, resource consumption, pod restarts, Kubernetes events, ingress health, certificate status, and datastore snapshot results. Logs alone are rarely enough: a pod may be perfectly healthy while the Service has no endpoints, or an application may be reachable while one control-plane member is degraded.

Events are particularly useful during initial deployment because they show scheduling failures, failed mounts, image-pull errors, probe failures, and controller decisions. They are also short-lived, so durable monitoring or log collection is needed if the goal is to investigate after the fact.

The result was a small operational baseline: know whether the API is reachable, whether all expected nodes are ready, whether system controllers are healthy, whether backups completed, and whether representative services work from the client networks that use them. That baseline became the gate for migrations.

Lessons Learned

First, lightweight Kubernetes is still Kubernetes. The distribution reduces component management, but operational safety still comes from documented addresses, repeatable configuration, backups, upgrades, and observability.

Second, high availability is end to end. A healthy etcd quorum cannot compensate for one storage target, one DNS resolver, one ingress address, or three VMs placed on one Proxmox host.

Third, defaults should be understood before they are replaced. The bundled Traefik and ServiceLB were useful because they produced a working baseline. Only after observing their behavior did it make sense to customize node selection and reconcile that configuration through Git.