What Kubernetes is, and why K3s
Quick grounding for anyone who hasn’t touched this before. Kubernetes runs containers across a group of machines and keeps them running the way you described, not the way they happen to be right now. You tell it “I want this app running, with this much memory, reachable on this port,” and a control plane continuously checks the real state against that description and corrects any gap: a crashed container gets restarted, a machine that goes away gets its work moved somewhere else. The machines are called nodes. The smallest thing Kubernetes schedules is a pod, one or more containers that share networking and storage. A cluster is the whole assembly: the API server and scheduler that make the decisions, plus every node that runs the actual work.
K3s is a real Kubernetes, not a scaled-down imitation of one: same API, same kubectl, same Helm charts you’d point at a full-size EKS or GKE cluster. What’s different is the packaging. Every control-plane component (API server, scheduler, controller-manager) runs out of a single binary instead of a dozen separate processes, and the default datastore is sqlite3 rather than etcd, though etcd is still available for exactly the case this cluster needs: a real multi-node HA control plane. It’s built for edge, homelab, IoT, and other places a full Kubernetes install would be overkill, and that’s the profile these VMs fit: modest CPU and RAM, sitting behind emulated network gear, on a fabric that’s slow and jittery on purpose.
The rule for this series has been that Nautobot is the source of truth and Git holds the intent behind it. The network devices already work that way through Golden Config. The servers didn’t, until now. By the end of this post the servers are provisioned by Ansible, and Ansible reads everything it needs, which hosts exist, how to reach them, what role each one plays, straight from the model.
One nuance this post makes concrete, because glossing over it is how “source of truth” turns into a lie: Git holding the intent and the running Nautobot reflecting it are two separate things. Some data (config contexts) Nautobot pulls from Git on every sync, so a commit is enough. Other data (anything Design Builder generates) is written once at build time and then lives only in the database, so a commit alone never touches it. When the two drift, you fix Git for the next build and you apply the change to the running instance as a deliberate second step. You will see both halves in this post.
The shape of the thing we’re building
Ten Ubuntu VMs, all already modeled in Nautobot as devices (Part 1 built them, Part 5 cabled the first one’s bond). Nine become a single K3s cluster. The tenth, DCA-DNS, becomes a BIND server.
The cluster is stretched, not sharded. One cluster spans DC-A, DC-B, and DC-C. That only works because VLAN 100 is one EVPN type-5 segment stretched across all three sites, so every node shares the 192.168.100.0/24 subnet no matter which datacenter it physically sits in. Three server nodes run embedded etcd and form the control plane, and all three live in DC-A. The other six nodes are agents, split evenly across DC-B and DC-C.
DCC-k3s-w6 came later than the other eight, added specifically so DC-C would have three workers instead of two. That matters once storage enters the picture below: three nodes per DC means every DC can hold a full 3-replica Longhorn volume without a single replica crossing the emulated core.
Keeping all three etcd members in one DC is not where this started. The first design put one member in each site, so the cluster would survive losing a whole datacenter. That’s the textbook layout, and on this fabric it does not work. The why turned out to be the most useful thing in this post, so we’ll come back to it once the cluster is standing.
