In Part 7, I got the routers and switches into Grafana, the dashboard I use to view the lab’s monitoring data. I could check interface traffic, errors and discards, and see the state of the routing sessions. That gave me a place to start when something went wrong.

The next thing I wanted was context for those graphs. If traffic between the datacenters increased, which hosts were responsible? What were they talking to?

Looking at network state raised another set of questions. BGP is the routing protocol these devices use to exchange reachability information. Seeing an established session tells me two devices are communicating, but I might still need to know whether a destination is in the routing table and where the device will send its packets next. I wanted to search that information across the Cisco routers and Arista switches without opening a console on each one.

Those questions led me to NetFlow, sFlow and SuzieQ. NetFlow summarizes the traffic a router observes. sFlow exports samples of traffic passing through a device. SuzieQ collects operational information from the devices and makes it available through a common query interface.

I already had an OpenTelemetry Collector receiving SNMP measurements. The Collector is a service that receives telemetry, processes it and sends it to storage. Adding flow collection would give me traffic details alongside the interface measurements. SuzieQ would give me a place to inspect routes, neighbors and interfaces across both platforms.

I’ll follow a file transfer between two datacenters, find its traffic in the collected data, and use Nautobot to put names beside the IP addresses. You don’t need prior experience with these tools to follow the example. Running it yourself assumes the earlier lab setup: devices, Nautobot inventory and the Kubernetes monitoring stack. The configurations below are from that lab, not a fresh-install guide.

Read more

After following packets through QEMU and the Arista leaves, I wanted to watch the network without opening a console on every device. I wanted to select CE1 in Grafana, find GigabitEthernet2, and see whether its traffic, errors or discards changed while Longhorn moved data between sites.

Getting a collector pod to start would not answer that question. Its requests had to reach the devices’ management routing tables, the devices had to permit them, and storage had to accept the measurements. Then the dashboard had to show an interface I could recognize from the router’s command-line interface (CLI).

I finished with fresh measurements from all 28 routers and switches and a network dashboard whose interface names match the CLI. I also added the cluster and storage measurements I needed beside those graphs, then organized Grafana around the systems I would investigate. I extended that collection to BGP peers, IS-IS neighbors and Cisco BFD sessions, checking the results against the CLI before trusting the dashboard. This post follows the path into Grafana, including the gaps I found in the devices’ SNMP tables and the event history I still need for brief routing failures.

Part 6 covers the original K3s installation and Nautobot-backed Ansible inventory. The free companion, The Interfaces Were Up. The Packets Had Stopped., explains the QEMU and Arista packet failures I repaired first. Here I build on those foundations to collect and display the network measurements.

K3s is the Kubernetes distribution running the applications. Longhorn stores their persistent volumes, VictoriaMetrics stores measurements over time, and Grafana displays them. Argo CD keeps the Kubernetes resources aligned with their declarations in Git. SNMP supplies the device measurements; OpenTelemetry (OTel) carries them to storage.

A place to run the monitoring system

The cluster has three masters in DC-A and six workers across DC-B and DC-C. The masters run the Kubernetes API and embedded etcd, the database holding cluster state. They remain together because the packet fixes did not establish consistently low latency under cross-site storage load. Loss of DC-A still means loss of the control plane.

The hosts use K3s v1.30.5+k3s1 and bond0 for Flannel traffic. I disabled the bundled Traefik and ServiceLB so the ingress components described later have their own Git-managed configuration. Flannel connects the pod networks over VXLAN. Its MTU is 8950 over the 9000-byte host data interfaces.

I needed each correction to survive the next deployment. Host configuration, device commands and Kubernetes resources have different owners, so I kept their sources separate:

Repository sourceWhat it controlsblog-sandbox/ansible/Ubuntu networking, K3s and standalone BINDblog-sandbox/golden-config/templates/Device configuration generated from Nautobotblog-sandbox/telemetry/Generated SNMP collector values and dashboardblog-sandbox-argo-cd/bootstrap/argocd-values.yamlArgo CD’s own Helm installationblog-sandbox-argo-cd/apps/Child Applications discovered by the rootblog-sandbox-argo-cd/values/Workload Helm configurationblog-sandbox-argo-cd/observability/Cluster and storage collection, health alerts and dashboard folders

The Ansible workflow obtains host addresses and roles from Nautobot. Neither a second static inventory nor manual edits to generated device configurations are needed.

Giving the recovered cluster a declared application state

Part 6 had already modeled host and K3s configuration in Git. For Kubernetes applications, I needed a controller to keep reconciling those declarations after the initial install. With GitOps, I put the desired resources in a repository and let the controller work toward that state. Reverting a commit can restore earlier resource declarations; it does not restore application data from a volume.

Argo CD represents each managed application with an Application custom resource: a repository, a path or chart, a revision, and a destination. Point one Application at a directory of other Application manifests and it becomes an app-of-apps. I apply the root once, then add workloads through files in Git.

Standing up Argo CD

Helm packages Kubernetes resource definitions into a chart, with a values file supplying configuration. I use the chart mirror prepared in Part 6 and keep Argo CD’s settings in bootstrap/argocd-values.yaml. The examples below assume both repositories are checked out on the automation host and kubectl is using the lab’s Kubernetes context.

I placed Argo CD’s components in DC-A, on the same three nodes that already hold etcd and the API server. The node selector below limits placement to that site. The toleration permits these pods to run on nodes reserved for the control plane:

global:
  nodeSelector:
    kubernetes.io/os: linux
    topology.kubernetes.io/zone: dc-a
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Equal
      value: "true"
      effect: NoSchedule

DC-A was reserved for the control plane in Part 6. Argo CD is a deliberate exception. The application controller talks continuously to the API, and keeping its repo-server and Redis in the same site avoids spreading that internal traffic across DC-A, DC-B and DC-C. I checked the available capacity before placing it there. The storage and application workloads run on the workers in DC-B and DC-C. The host-level monitoring collectors described later run on the nodes they monitor, including the masters.

The bootstrap values also select docker.io/library/redis, which uses the Docker Hub cache configured in Part 6. With the values in place, install Argo CD from the pinned chart:

cd /home/ubuntu/blog-sandbox-argo-cd
helm upgrade --install argocd \
  http://192.168.3.21:8888/helm/argo-cd-10.8.0.tgz \
  --namespace argocd --create-namespace \
  --values bootstrap/argocd-values.yaml \
  --wait --timeout 10m
kubectl -n argocd get pods

The chart version and mirror address belong to this lab. If you’re adapting the repository, check those values and the DC-A node labels against your cluster first. Before continuing, confirm the Argo CD pods are Ready.

Making the first sync wait for storage

Argo CD keeps comparing Git declarations with live Kubernetes resources. One root Application, applied by hand once, watches a directory in blog-sandbox-argo-cd and auto-discovers everything else:

spec:
  source:
    repoURL: https://github.com/byrn-baker/blog-sandbox-argo-cd.git
    targetRevision: main
    path: apps
    directory:
      recurse: false
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The first two children were Longhorn at sync-wave 1 and VictoriaMetrics/Grafana at wave 2. The monitoring deployment adds cluster-observability at wave 3, snmp-metrics at wave 4 and otel-snmp at wave 5. MetalLB and the ingress resources follow at waves 9 through 12. A sync wave is a numbered group applied before a higher-numbered group. Storage must be ready before the applications request volumes, and the stores should be ready before the collector starts exporting.

Argo CD needs Application health handling for the parent to use child health in this pattern. I installed the customization from bootstrap/argocd-values.yaml and checked it in the live argocd-cm.

The chart-based child Applications point repoURL at the mirror directory containing index.yaml. The chart and targetRevision fields select the package. For Longhorn, those fields are:

repoURL: http://192.168.3.21:8888/helm/
chart: longhorn
targetRevision: 1.12.1

A direct .tgz URL works in the Helm install command above, but it isn’t a Helm repository URL for an Argo Application.

Longhorn also needs this setting in values/longhorn-values.yaml:

preUpgradeChecker:
  jobEnabled: false

This disables an upgrade-check hook that otherwise runs before its service account exists during the initial Argo installation. The Longhorn issue explains the hook conflict.

Once Argo is Ready, apply the root and watch its children:

kubectl apply -f bootstrap/root-app.yaml
kubectl -n argocd get applications -w

Synced means the deployed resources match Git; Healthy means they meet Argo’s health checks. Confirm Longhorn and VictoriaMetrics have both statuses before moving on to collection. The collector also needs the credential Secret created in the generation step below. The Application health customization controls the initial ordering through the root. It doesn’t serialize every later update: existing children can auto-sync independently.

Argo’s own bootstrap values sit outside the root’s apps/ path. To change those settings later, rerun the Helm command with the updated values.

A running application still needed working storage

Longhorn keeps volume replicas, copies on separate workers, and attaches the volume to the application pod. A PersistentVolumeClaim (PVC) is the application’s request for storage. Kubernetes attachment state and Longhorn replica health answer different questions, so I checked both:

# From the automation host with its existing lab Kubernetes context
kubectl get nodes -o wide
kubectl get pvc -A
kubectl get volumeattachments
kubectl -n longhorn-system get volumes.longhorn.io
kubectl -n longhorn-system get pods

Before starting collection, check that the claims are Bound, the volumes are attached where needed, and Longhorn reports healthy replicas. Also check free space in both the Ubuntu guests and the hypervisor’s storage pool. Space inside a VM doesn’t tell you whether the underlying pool can accept more writes.

Grafana’s values use a Recreate deployment strategy, which stops the old pod before starting its replacement, and a longer startup allowance for database initialization. Those settings live in values/victoria-metrics-values.yaml. They let Kubernetes wait for Grafana to initialize its persistent database before treating a slow start as a failure.

Getting the devices to answer

SNMP, the Simple Network Management Protocol, lets a collector request measurements such as interface byte counters, errors and uptime. My network scope was 13 Cisco IOS-XE routers and 15 Arista EOS switches. Nautobot also held ten Ubuntu servers, but those were outside this device-polling rollout.

I already used Nautobot to generate device configuration, so I used the same model for SNMP access. A config context supplies structured values that templates turn into commands. The community, contact and allowed source network were shared across all seven network-device roles; location came from each device’s own record. One context was enough. The community below is a placeholder:

snmp:
  ro_community: "<snmp-read-only-community>"
  contact: "byrn-baker demo-lab"
  acl_source: "192.168.3.0/24"
  acl_source_wildcard: "0.0.0.255"

I used SNMPv2c for this lab’s first collection path, with read-only access and a source access control list (ACL), which permits requests from specified addresses. A community is the shared string the collector presents to an SNMPv2c agent. It travels without encrypted authentication. The ACL limits where requests may originate; it does not give v2c the security properties of SNMPv3. The choice kept this first deployment’s scope small, and it should remain an explicit lab tradeoff.

I permit the management VLAN because the collector can move between workers. On the captured polling path, requests left the worker’s eth0 interface using its management IP. When adapting the ACL, check the source address of actual SNMP requests; a pod’s internal address may be translated before the device sees it.

The ACL syntax differs between platforms:

{% raw %}

{# ios/snmp.j2 #}
ip access-list standard ACL-SNMP-RO
 permit {{ config_context.snmp.acl_source.split('/')[0] }} {{ config_context.snmp.acl_source_wildcard }}

{% endraw %}

{% raw %}

{# eos/snmp.j2 #}
ip access-list standard ACL-SNMP-RO
   10 permit {{ config_context.snmp.acl_source }}

{% endraw %}

The two templates look similar, but the devices expect different address syntax. EOS accepts CIDR, which writes a network and prefix length together, such as 192.168.3.0/24. IOS expects the network address followed by a wildcard mask, which marks the address bits the rule can ignore.

I validate the rendered commands before deploying them:

# From the blog-sandbox checkout
make ci
make ci-full

The first command checks template structure. The second adds Batfish configuration checks, which can catch invalid device syntax even when Jinja renders successfully. The active compliance definitions live in jobs/gc_compliance_setup/__init__.py.

EOS needed the management VRF and the ACL

A Virtual Routing and Forwarding instance (VRF) has its own routing table. The EOS management interfaces belong to MGMT-VRF, so SNMP also has to be enabled in that routing context to answer requests there. The template correction emits one snmp-server vrf <name> line per unique VRF on a modeled Management interface. The resulting device configuration includes:

ip access-list standard ACL-SNMP-RO
   10 permit 192.168.3.0/24
snmp-server community <snmp-read-only-community> ro ACL-SNMP-RO
snmp-server vrf MGMT-VRF

I deploy both the snmp and acl features. The SNMP feature covers snmp-server commands, but the access list is a separate configuration block. Deploying just the community’s ACL reference doesn’t install the restriction itself.

After syncing the source into Nautobot, I regenerate intended configuration, collect fresh backups, run compliance and generate Config Plans. I check one device per platform before expanding to the fleet, with Fail Job on Task Failure enabled. Live reads and successful polls confirmed the VRF binding and ACL on all 15 EOS devices.

Check that the settings will survive a restart

Running configuration is active now; startup configuration is what the device loads after reboot. I compare both, because polling can work even when the saved configuration is incomplete:

show running-config
show startup-config

Check the SNMP community, ACL and EOS management VRF in both outputs. If a save is needed, review the rest of the running configuration first so unrelated changes aren’t saved accidentally:

copy running-config startup-config

Then reconnect and read startup configuration again. Nautobot’s intended files, backups and per-device job logs help with that comparison, but a successful job status doesn’t replace the device readback.

Generate the collector from the same device model

The generator selects IOS-XE and EOS devices from Nautobot, using their roles, management addresses, sites and SNMP contexts. The ten Ubuntu servers are excluded from this 28-device scope. I use one collector with two export destinations, so writing to two stores does not double the polling load on devices. The completed configuration has 28 device receivers and three additional BGP-only receivers for the SERVERS VRF, covered later in the post.

The following commands are the repository’s generation workflow. Start in the parent directory of adjacent blog-sandbox and blog-sandbox-argo-cd checkouts. They require its Python dependencies, the existing Nautobot credentials in the environment, a Kubernetes context pointing at the lab, and the observability namespace and VictoriaMetrics operator already installed. --sync-secret writes the externally managed Kubernetes credential Secret; this is a deployment step, not a read-only preview.

cd blog-sandbox
python3 telemetry/generate_snmp.py \
  --canary CE1 DCA-Leaf01 \
  --output ../blog-sandbox-argo-cd/values/otel-snmp-values.yaml \
  --sync-secret
pytest telemetry/test_generate_snmp.py -q

The canary selects one Cisco router and one Arista leaf from the same model. After checking their measurements, the full-fleet generation uses the same command without --canary CE1 DCA-Leaf01. I review the generated non-secret diff, render it with the pinned chart and validate the collector configuration with the pinned binary before publishing the values to the Argo repository. The root Application discovers the declared children; I do not apply a separate collector Deployment by hand.

The generator runbook records dependencies, pinned artifacts and credential rotation. Credentials go into network-snmp-credentials in namespace observability, while Git holds the references. Synchronizing that Secret does not change the SNMP community on a router or switch.

The collector uses memory-backed export queues. A restart can lose queued samples and creates a polling gap; this deployment does not promise lossless delivery during outages. Freshness, polling errors and queue metrics make those gaps visible.

Following a measurement into storage

Once the canaries answered, I could follow their measurements through the rest of the path. The OpenTelemetry Collector polls every 60 seconds, with staggered starts so all devices do not answer at once. It forwards samples using OTLP, the OpenTelemetry Protocol, over HTTP to two VictoriaMetrics stores. The existing store holds cluster metrics; the second keeps a separate, longer history for SNMP. Grafana queries the stored measurements.

13 IOS-XE routers + 15 EOS switches
                 |
        SNMP over management VLAN
                 |
           OTel Collector
                 |
             OTLP/HTTP
            /         \
  cluster metrics     SNMP history
            \         /
               Grafana

The fleet rollout produced samples from all 28 devices in both stores. Each device had at least five uptime samples in the five-minute check, and the oldest latest sample was less than 60 seconds old. The collector’s export queues were empty, its refused-point counters were zero, and it reported no failed exports. That established more than a running pod: device replies had become stored measurements I could query.

The dedicated SNMP store is configured for 547 days of retention on a 20 GiB volume. I kept it separate because extending retention for every Kubernetes metric would have a different storage cost. The cluster store retains its existing one-month setting. A retention setting does not prove that the disk can hold that much history; I still need to measure growth. A time series is one metric with a particular set of labels, such as incoming bytes on one interface of one device. More series and more samples consume more space.

The interface name had to mean the same thing on the router

The first dashboard returned measurements, but its legends exposed the entire label set: device, address, interface index, abbreviated name and collector metadata. That was information the storage system needed, presented as though it were an interface name. I still had to work out which line belonged to the port I wanted to inspect.

I compared the collected names with show interfaces on all 28 devices. SNMP’s interface Management Information Base, or IF-MIB, gives each interface row an index and several descriptive fields. A MIB defines the objects a device exposes through SNMP. The index identifies a row for polling and joining counters; it is not a useful label for a graph.

The two name fields also differed. On CE1, ifName contained Gi2, while ifDescr contained GigabitEthernet2. On DCA-Leaf01, both contained Ethernet1. My collector already stored ifDescr in the interface label, so I could correct the presentation without changing the stored measurements.

I made the Device and Interface selectors use those full names. Traffic legends now pair the device with its interface, and the status table has three useful columns: Device, Interface and Interface status. Numeric states render as Up, Down or the corresponding named condition. Error and discard panels show rates in errors/s and discards/s. The internal labels remain available for diagnosis without taking over the dashboard.

Here I selected CE1 and GigabitEthernet2. The traffic panels follow that selection; the freshness counter still checks all 28 devices. The Arista view uses the same layout with DCA-Leaf01 / Ethernet1.

Why 372 SNMP rows became 318 displayed interfaces

The name comparison exposed another distinction: an SNMP interface row does not always represent another port. The stored data contained 372 rows. Of those, 28 described the Multiprotocol Label Switching (MPLS) layer on Cisco interfaces, and 26 were IOS Null0 or VoIP-Null0 sinks, internal destinations for discarded traffic. The remaining 318 names appeared in the devices’ show interfaces output.

MPLS forwards traffic using labels attached to packets. Its SNMP rows describe that protocol layer, so they could represent traffic already counted on the underlying interface. Plotting them as additional links would make the network appear busier than it was. They also lacked error and discard counters. On the inspected SP1 row, a direct request returned noSuchInstance: the requested object identifier, or OID, did not have a value there. The physical interface returned a real counter. Missing and zero meant different things.

I excluded the MPLS rows and IOS null sinks from the normal interface selectors and panels. I kept the raw measurements and a separate table identifying MPLS rows without error counters. That left a view I could compare with the CLI without throwing away diagnostic data.

There was one gap in the other direction. Each Arista leaf showed an internal Vlan4097 in its CLI output, but those nine interfaces were absent from the collected IF-MIB rows. I documented the exception rather than adding empty graphs that would imply collection was working for them.

Keeping dashboard changes in Git

I made these changes in telemetry/network_dashboard.py, the dashboard’s source. The generated JSON travels inside a ConfigMap in the Collector’s Helm values, then reaches Grafana through Argo CD. Editing the dashboard only in Grafana would leave the next deployment able to replace my changes.

The generator escapes Grafana’s legend placeholders so Helm preserves them when rendering the ConfigMap. I check the rendered chart as well as the dashboard JSON, because the dashboard passes through both formats before Grafana reads it.

For this presentation change, I used the generator’s dashboard-only mode from the blog-sandbox checkout:

python3 telemetry/generate_snmp.py --dashboard-only \
  --output ../blog-sandbox-argo-cd/values/otel-snmp-values.yaml

This updates the dashboard while preserving the receiver configuration and credential revision. Review and publish the values through the Argo repository, then check the result in Grafana with one Cisco interface and one Arista interface selected. Both should show full names, readable status values and explicit rate units.

My browser and query checks covered both platforms. The audit and screenshots record the CLI comparison and rendered result.

Giving each part of the lab a place in Grafana

Once I could recognize the network interfaces, I needed to make the rest of Grafana useful too. The chart had placed its dashboards together in General. Looking for a storage problem meant searching past cluster views, dashboards for the metrics stores and panels for operating systems I wasn’t running.

I organized them around where I would go to investigate a problem:

That left 39 dashboards and none in General. I removed the Windows, AIX, macOS, multi-cluster and Prometheus-server dashboards because those systems weren’t deployed here. The existing Linux and Kubernetes detail views stayed available when I needed to move beyond the overview.

A folder needed data behind it

The control-plane dashboards exposed a collection gap. K3s runs etcd, the scheduler and controller manager inside its server process. The scheduler chooses where pods run; the controller manager keeps resources aligned with their requested state. The chart’s normal discovery expected separate component pods, so those scrapes had no targets. A scrape is the collector’s request to a component’s metrics endpoint. An empty target list meant I had no measurement, not that the component had stopped.

I added a small vmagent collector on each master to read the existing loopback listeners: port 2381 for etcd, 10257 for the controller manager and 10259 for the scheduler. Loopback is the host’s local network connection. These collectors share the host’s network namespace so they can reach it without exposing the listeners elsewhere or restarting K3s. They send samples to the existing cluster metrics store every 30 seconds. The configuration is in observability/control-plane.yaml.

Longhorn’s endpoint was available too, but its NetworkPolicy, the Kubernetes rule controlling incoming connections, blocked the monitoring agent. I added a rule admitting that agent to the managers’ shared API and metrics port, then collected all six managers. I also added metrics from five Argo CD components, including its application controller and repository server. The collection manifests keep the endpoint selection and access rules together.

Now I can start with a container restart in the K3s folder and compare its time window with API latency and etcd disk writes. The Storage folder shows whether a volume was rebuilding or experiencing slow I/O. Network gives me the interface counters for the same window. That comparison helps narrow the next check; a host-wide retransmission counter still cannot identify which connection lost data.

This view shows a one-hour window. All nine nodes remain Ready and each control-plane component has three scrape targets, but the restart and API-latency panels still contain activity worth investigating. Readiness tells me the nodes are available; it doesn’t mean every request was fast or every container ran without interruption. The p99 latency estimates the value below which 99% of the measured requests fall.

Keeping the folders through the next deployment

I kept dashboard JSON under blog-sandbox-argo-cd/observability/dashboards/. The observability/kustomization.yaml file packages each dashboard into a ConfigMap with a grafana_folder annotation identifying the destination folder. The SNMP generator adds the same annotation for Network. Grafana’s sidecar, a helper container, copies those files into the corresponding directories.

Grafana’s file provider checks for dashboard updates every 30 seconds. The sidecar loads the files during startup and continues watching for changes. These settings live in values/victoria-metrics-values.yaml; a Git-managed migration hook removes the old chart-owned dashboards to avoid duplicate IDs.

After deployment, I checked the folders in Grafana and ran all 26 queries on the three new overview dashboards. Each returned data, and browser checks found no query errors or empty panels. The verification record and screenshots show the result.

Reaching Grafana by name

I use

https://grafana.sandbox.lab

for Grafana and

https://argocd.sandbox.lab

for Argo CD. Both names share an ingress address. MetalLB assigns that address to a Kubernetes LoadBalancer Service and announces it on the network; Traefik receives the web request and uses its hostname to select the application.

Browser -> DNS lookup -> ingress IP -> Traefik -> Grafana or Argo CD

The lab has two ingress addresses, one on each network:

The root Application deploys MetalLB, its address pools, Traefik and the application routes from the ingress manifests. The bundled K3s Traefik and ServiceLB remain disabled because I manage this Traefik installation through Argo CD and use MetalLB for address assignment. Two Traefik replicas run on workers in different sites. MetalLB announces the management address on eth0 and the lab address on bond0.

BIND supplies the DNS answers. Its views return the lab address for queries sourced from the lab’s 10.0.0.0/8 network and the management address for management clients. A view is a set of DNS answers selected by the query’s source address. If a resolver forwards the query, BIND sees that resolver’s address.

The names and addresses are declared in blog-sandbox/service_endpoints.yml. To publish them, run the BIND playbook with the existing Nautobot credential environment:

cd /home/ubuntu/blog-sandbox/ansible
.ansible/bin/ansible-playbook pb.dns.yml -e bind_validate_only=true
.ansible/bin/ansible-playbook pb.dns.yml

The first command validates staged DNS configuration without activating it. The second publishes it. My workstation uses pfSense for DNS, so I added a domain override under Services > DNS Resolver > Domain Overrides: sandbox.lab points to the BIND server at 192.168.3.71, with TLS Queries disabled. That destination is the DNS server, not the ingress address. The resolver must also be able to send queries through the management interface.

HTTPS uses a private lab certificate authority (CA), which signs the ingress certificate. After Argo creates the Traefik namespace, the certificate helper creates or reuses the local keys and installs the required Kubernetes Secrets:

cd /home/ubuntu/blog-sandbox-argo-cd
python3 tools/ingress_tls.py

Import the public ingress/lab-ingress-ca.crt into the workstation’s trusted certificate authorities. Private keys stay out of Git. Certificate renewal is a separate maintenance step covered in the ingress runbook.

From the workstation, check the DNS answer and then open Grafana:

nslookup grafana.sandbox.lab

The expected answer is 192.168.3.240. In Grafana, open the Network folder and select Network SNMP. Both Grafana and Argo CD still require their application logins. My ingress checks reached both applications through the lab address from all three datacenters and verified their HTTPS certificates.

Adding routing state to the same collection path

An interface can remain Up while the routing relationship using it fails. That mattered after the troubleshooting companion exposed brief BFD-triggered BGP resets under load. BFD, Bidirectional Forwarding Detection, checks whether a peer remains reachable. BGP, Border Gateway Protocol, exchanges routes and can use BFD to react quickly when a neighbor stops responding. Interface counters alone couldn’t tell me whether those sessions stayed established.

I started by asking the devices what they could return. BGP peer-state queries worked on Cisco and Arista. My first IS-IS query came back empty, but that wasn’t enough to conclude the router couldn’t expose it. IS-IS builds neighbor relationships called adjacencies, and Cisco placed the usable adjacency table under its own MIB branch. A Management Information Base, or MIB, defines the objects available through SNMP; each object has a numeric identifier in that tree. The standard branch and Cisco’s branch were different places to look.

The working Cisco state column was 1.3.6.1.4.1.9.10.118.1.6.1.1.2. I added it to telemetry/routing_metrics.py, alongside BGP state, time established and a counter of entries into Established. Nautobot’s device roles determine which receivers get each profile. The P routers don’t run BGP, so polling an empty BGP table on those four devices would add no useful measurement.

I checked CE1, DCA-Leaf01 and SP1 before expanding. Their existing SNMP access already allowed these reads through the management network. The change was in the collector configuration, deployed through Argo CD, and the samples followed the same path into both VictoriaMetrics stores.

Checking what the numbers actually represented

SP1 exposed five IS-IS adjacencies. To name their interfaces, I followed the circuit index to its IF-MIB index, then used the same full interface description as the interface dashboard. That produced GigabitEthernet2 rather than an unexplained circuit number. The mapping comes from each poll, so I didn’t need to maintain a separate list of circuit-to-port assignments.

BFD caught a problem with that approach. SP1 returned ten SNMP rows, while show bfd neighbors showed five sessions. Each session appeared under two application IDs, 9 and 15. The reported BFD interface indices also disagreed with IF-MIB. Joining those values would have put the wrong interface names beside otherwise plausible Up states.

I kept the BFD local discriminator instead. This is the identifier shown in the CLI’s LD/RD column: LD is the local discriminator and RD is the remote one. The dashboard retains the application rows and counts distinct local discriminators separately. Across the core, 56 application rows represented 28 CLI sessions. I didn’t turn those duplicate rows into extra links or invent protocol names for the application IDs.

The completed polling check looked like this in both metrics stores:

The default SNMP table omitted the SERVERS-VRF peer on each Leaf03 switch. On these EOS devices, polling with the existing community followed by @SERVERS returned the missing peer. I added a BGP-only receiver for each verified context, using the same credential Secret, and checked its state, remote AS, uptime and transition counter against direct SNMP reads. The peer address and Established state also matched show ip bgp summary vrf all.

The collection targets live in telemetry/bgp_vrf_profiles.yaml. The generator verifies that each configured VRF exists on that device’s modeled interfaces in Nautobot. Each measurement carries a vrf label so identical peer addresses in different routing tables remain separate. These metrics cover the verified peer transports; they don’t report how many prefixes each address family exchanges.

Reading the result in Grafana

I added Network Routing beside Network SNMP in the Network folder. BGP rows show the device, VRF, peer address, exact remote AS number and readable state. The BGP VRF selector switches between All, default and SERVERS without changing the IS-IS or BFD panels. Uptime and transition graphs include the VRF in their legends. IS-IS rows show the full local interface name. BFD rows use the local discriminator described above. The dashboard JSON keeps those choices in Git.

The screenshot selects SERVERS, so it shows three BGP peers. Selecting All shows 136. IS-IS and BFD retain their own counts because the VRF selector applies only to BGP. Remote AS values remain exact numbers, such as 65001, rather than abbreviated measurements.

For the transition graph, I use VictoriaMetrics’ increase_prometheus function to calculate changes between observed counter samples. Starting a new VRF time series must not turn its existing lifetime counter into an apparent burst of recent flaps. The newly labeled graphs begin with this collection rollout; they don’t reconstruct earlier VRF history.

The routing rules also check reported non-up states and row counts below the verified baseline for each device. That second check matters when a failed neighbor disappears from a table instead of remaining there with a Down value. Default-context coverage stays separate from the three SERVERS peers, which also have alerts for a missing device/VRF/peer combination. The baseline must change when I deliberately change the topology. I verified rule evaluation, not delivery to an external notification destination.

What a successful poll still could not tell me

A session can go down and recover between one-minute polls. BGP’s established-transition counter gives me another clue when that happens, but it doesn’t explain why BFD expired. I still need event history beside these graphs.

A poll starts with a request from the collector. An SNMP trap is a notification sent by the device when something happens. Syslog carries event messages, often including the peer, routing context and reason. These need receivers of their own; adding a trap destination to a switch wouldn’t make the OTel SNMP polling receiver accept it.

I checked the notification catalog on the Arista canaries with:

show snmp notification

EOS 4.34.6M listed BGP transitions, interface link-up/link-down and IS-IS adjacency notifications. It didn’t list a BFD notification. Its logs did contain %BFD-5-STATE_CHANGE messages with the peer, VRF and diagnostic reason, so syslog was the source I could follow for those BFD events.

The trap receiving side was still missing. None of the four inspected canaries had an snmp-server host destination, and the cluster had no trap listener or UDP 162 Service. I hadn’t demonstrated end-to-end trap delivery or deliberately interrupted a routing session during these checks.

That next collection path needs a destination reachable through the management VRF, a listener that decodes traps, and syslog ingestion for the BFD details already present on the switches. VictoriaLogs and its planned 14-day retention belong there. I’d start with a safe test notification from one device per platform and check that the stored event retains its device, peer, VRF, interface and reason before expanding.

Knowing whether collection is still working

A Ready collector means its process passed a health check. I also need to know whether a device answered recently. In Grafana’s Explore view, using the SNMP datasource, this query counts devices with an uptime sample less than three minutes old:

count(time() - timestamp(snmp_device_uptime_ticks{job="snmp"}) < 180)

For this fleet, I expect 28. A smaller count tells me to find the stale devices before interpreting their traffic graphs. For interface coverage, I use the same exclusions as the normal dashboard view:

count by (device) (
  snmp_interface_oper_status{
    job="snmp", if_type!="", if_type!="166",
    interface!~"(VoIP-)?Null0"
  }
)

The if_type value 166 identifies the MPLS protocol-layer rows. To inspect a particular port, I select its full name. This query converts the incoming byte counter on CE1’s GigabitEthernet2 into bits per second over a five-minute window:

rate(snmp_interface_in_octets_total{
  job="snmp", device="CE1", interface="GigabitEthernet2",
  if_type!="", if_type!="166"
}[5m]) * 8

The dashboard also shows sample age, polling errors and export queues. A missing sample must remain unknown, not turn into zero traffic. Reported interface speed needs the same care: the speed advertised by a virtual NIC is not a measurement of the virtual topology’s forwarding capacity.

The current verification and screenshots show all ten Argo Applications Synced and Healthy, all nine K3s nodes Ready, and all three Longhorn volumes attached and healthy with three read/write replicas each. Each control-plane component has three successful scrape targets, Longhorn has six and the selected Argo CD components have five. SNMP supplies fresh measurements from all 28 network devices, including 133 default-context and three SERVERS-VRF BGP peers.

I checked the dashboard queries and the pages they render. The three cluster, storage and Argo overview dashboards returned data for all 26 queries; the routing dashboard passed all 45 checks across its All, default and SERVERS selections. Coverage alerts check missing collection, unhealthy volumes and restarts of Longhorn’s storage-driver containers. Rule evaluation works, but delivery to an external notification destination remains untested.

I kept the first release small enough to follow all the way through: one device per platform, a stored measurement, a recognizable interface on a graph, then the rest of the fleet. The startup readback and missing-counter checks showed where a green status could conceal unfinished work. The browser check exposed a different problem: data could be present and still be awkward to use.

I can now select the same interface in Grafana that I would inspect at the CLI and watch its counters while the applications use the network. BGP and IS-IS state now sit beside those counters. Capturing routing events is the next collection step. Flow collection and SuzieQ remain outside this release. The free troubleshooting companion explains the failures that made those measurements necessary; this build gives me the first part of the record for the next investigation.

References

• Part 6: One Cluster, Three Datacenters, Zero Static Inventory

• Argo CD Application health

• Longhorn and Argo CD pre-upgrade hook issue

• SNMP generator, storage and validation runbook

• Interface naming, event-source audit and dashboard screenshots

• Cluster monitoring, dashboard folders and live verification

• VictoriaMetrics counter calculations

I set up Netclaw on Omarchy and connected it to my existing Cisco and Arista network lab. In this video, I walk through the setup, a health-check report across 28 devices, and a test of access to the telemetry I already collect through Grafana.

The interesting part is what those results actually mean. Loading an inventory, collecting device output, and confirming that an integration works are separate checkpoints. I show what worked and where the setup still needed attention.

The lab uses Nautobot for inventory, pyATS for device checks, and an observability stack that includes SNMP, NetFlow, sFlow, OpenTelemetry, and SuzieQ.

Music: “Molecule” by AGST, via Epidemic Sound.

Along the way, I separate a Deployment, ReplicaSet, Pod, and container; show how a Service gives replaceable Pods a stable address; and explain why Running, Ready, and healthy are different claims. I also show where embedded etcd fits in this K3s control plane and what K3s changes about packaging without changing the Kubernetes API model.

The live checks in this episode are read-only and point-in-time. Replacement and failure behavior use labeled illustrations. A Ready result is not presented as proof that every network, DNS, storage, or workload path is healthy.

Companion article: One Cluster, Three Datacenters, Zero Static Inventory.

I was working on getting SNMP data into Grafana, using the nine-node K3s cluster in my lab. K3s runs the containers, and Longhorn keeps copies of their storage volumes on different workers. Before I could finish the monitoring setup, I had to sort out why those workers kept losing traffic between datacenters.

I had already changed the MTUs and switched the Cisco routers to virtio interfaces. I thought I had checked the network after those changes. Then SPE3, one of the routers in my service-provider core, stopped receiving packets on both core interfaces. Both still showed up, and their transmit counters kept increasing.

Rebooting SPE3 brought IS-IS back up and the large pings started working again. IS-IS is the routing protocol my core routers use to discover neighbors and calculate paths. But the receive counters stopped a second time. Meanwhile, the K3s nodes could report Ready while Longhorn rebuilds stalled and TCP kept retransmitting.

So I started following the packets through the virtual topology. That took me into QEMU, the process emulating the Cisco router hardware, and then into the Linux networking inside the Arista switches. I found a receive stall in the first and malformed combined TCP segments in the second. Getting past the QEMU problem was what let me see the leaf problem clearly.

I’ll walk through the captures and checks below. There is still a storage-load issue to investigate, so I’ll also show where the successful tests stopped answering the question. The companion Part 7 implementation post covers the SNMP collection system I built on the recovered lab.

Following the retransmissions down to a virtual NIC

The C8000V routers were using virtio, a network interface designed for virtual guests, with data-port maximum transmission units (MTUs) of 9216 bytes. I kept that configuration while checking the receive failure. I wanted to change one thing at a time and see what it did to the same failing traffic.

The MTU sets the packet-size limit at an interface. My lab uses jumbo frames, larger than the usual 1500-byte Ethernet setting. TCP breaks a transfer into segments and sends bytes again if they are not acknowledged. Those retransmissions told me the transfer was having trouble, but I still needed to find where the packets were being lost or damaged.

Map the layers the connection crosses

Take a connection from a K3s node in DC-A to one in DC-B. Both Linux VMs run on Proxmox. Their data interfaces connect to Arista vEOS leaves in EVE-NG, while the Cisco C8000V core runs in Cisco Modeling Labs (CML). A leaf is the switch the server connects to.

There are VMs inside VMs here. Proxmox hosts CML, and CML hosts the emulated routers. That means “the host” depends on which part I am checking: the CML VM is a guest from Proxmox’s point of view, but it is the host running the router’s QEMU process.

In the diagram, follow the connection from the source VM into its leaf, across the datacenter switching fabric, and through the CML core toward the other site. The management connection is separate. I could still use it to log into the devices when this data path stopped working..

Figure 1. I followed the data connection across each virtualization boundary. This simplified view omits redundant links and individual core hops; the management network is a separate diagnostic path.

The routing state helps identify which part of that path is working. In the core, I checked IS-IS adjacencies, the neighbor relationships between routers. BGP distributes routes, and its EVPN extension carries information about endpoints in the datacenter overlay. VXLAN is the tunnel carrying the original Ethernet frame across the underlying IP network. I needed to account for that wrapper when checking packet sizes and captures.

Account for the headers between layers

The host’s data interface is set to 9000 bytes, but Flannel, which connects Kubernetes pod networks, is set to 8950. That leaves room for Flannel’s tunnel headers. The router data ports are set to 9216. Follow the packet through the diagram and the different sizes make sense: each layer needs room for whatever the previous layer added.

TCP has another size to consider: maximum segment size, or MSS. This controls the amount of data in each TCP segment. I used it to send small segments and jumbo segments over the same path and compare the results.

Figure 2. I compared the limit at each layer. The different values leave room for encapsulation; they are not conflicting measurements of one packet.

I planned to capture the same traffic at the K3s guests, Proxmox, EVE and the CML core. For each path, I attempted four transfers, alternating requested MSS values of 1200 and 8960. Each sent 256 KiB in each direction at 1 Mbit/s. KiB and MiB describe amounts of data in binary units; Mbit/s describes millions of bits per second. I limited the rate because I wanted to check delivery before testing capacity.

Neither the DC-C to DC-A connection nor an alternate DC-B to DC-A connection established TCP. I could not compare the transfer rates yet. I went back to the core router that had stopped receiving.

Locate the stop inside CML

On SPE3, both core interfaces still showed up/up and MTU 9216. Look at the counters, though: transmitted packets kept increasing while received packets did not. Management traffic and the CE3-facing interface were still receiving. The failure affected the two core interfaces.

Both IS-IS neighbors were gone, and the BGP sessions to the route reflectors were Idle. Route reflectors redistribute routing information among BGP peers. Losing those sessions along with the receive counters pointed me toward the core interfaces, even though they still showed up/up.

I captured the SP1-SPE3 and SP3-SPE3 links in CML. Both ends were sending IS-IS hellos, the messages used to discover and maintain neighbors. The captures included Ethernet frames of 9229 bytes. That length includes link-layer overhead, so it is not the same number as the IP MTU.

The hellos were there on the links. I then needed to check whether QEMU was delivering them into SPE3.

I then checked how CML was passing those packets into SPE3. These interfaces use QEMU UDP sockets over loopback, so packets move between processes on the CML host. They do not need to leave CML and cross the outer Proxmox-to-EVE connection for this hop.

The sockets feeding the two failed interfaces had growing receive queues, roughly 12 MB and 15 MB of kernel queue accounting in the samples. The sockets for management and the CE-facing interface had no backlog. Packets were reaching CML and waiting there while the router’s receive counters stayed still.

Figure 3. I narrowed the receive stall to the boundary between queued packets on the CML host and delivery into SPE3. The QEMU backport repaired this reproduced failure; it did not explain every later retransmission.

Here are the read-only commands for comparing the router’s counters with the CML host’s sockets and running executable. Run the IOS commands inside SPE3 and the Linux commands on the CML host. In the latter, <router-pid> is the process ID of the affected router’s QEMU process.

! Inside the SPE3 IOS console
show interfaces GigabitEthernet2
show interfaces GigabitEthernet3
show isis neighbors

# In the Linux shell of the CML host
sudo ss -u -a -n -p -m
sudo readlink /proc/<router-pid>/exe
sudo sha256sum /proc/<router-pid>/exe

In the ss output, I looked for the sockets owned by SPE3’s QEMU process and matched them to the affected NICs. A total for every socket on the host would not tell me which interface was stuck.

The /proc/<router-pid>/exe checks answer a different question: which executable is this router actually running? Replacing /usr/bin/qemu-system-x86_64 does not replace the code already loaded by a running process. SHA-256 gives me a fingerprint of the executable so I can compare the running copy with the one I tested.

The next place to look was QEMU’s receive ring. This is the set of buffers, regions of guest memory, that the router makes available for incoming packets. QEMU uses counters to track its position in the ring. These counters are 16 bits wide: after 65535, they return to zero.

The failed interfaces were all close to that rollover: three interfaces across two routers. Working interfaces kept advancing through their counters, so I compared the stuck ones more closely.

I read those counters through QMP, the QEMU Machine Protocol, using virsh qemu-monitor-command. virsh talks to libvirt, the VM-management layer. The QMP operation x-query-virtio-queue-status showed the consumed and cached available indices for queue 0 of each mapped NIC.

A name such as /machine/peripheral/net2/virtio-backend is a device path inside QEMU. It is not a Linux file, and net2 needs to be matched to the router’s actual interface. I saved the queue-status readings before inspecting individual entries with x-query-virtio-queue-element, because that deeper check also refreshes QEMU’s cached available index. Here are the original readings:

Both SPE3 interfaces had a full set of 256 buffers waiting in the guest. The head buffers were 2060 bytes each, and the guest and QEMU had negotiated mergeable receive buffers, allowing several buffers to hold one large packet. There should have been space for the jumbo frame.

QEMU also reported that the NICs were enabled and unbroken. I sampled them again: the consumed indices stayed frozen while working interfaces advanced. Inspecting the individual queue entries refreshed the cached index, but did not get reception moving.

Why a counter rollover looked like an empty queue

The guest had supplied receive buffers, so why was QEMU behaving as though there was nowhere to put the packet? I followed that question into virtqueue_num_heads() and virtqueue_split_get_avail_bytes() in QEMU 8.2.2. These functions help work out how many buffer entries are available and whether they provide enough space for the incoming packet.

The guest advertises its available buffers through a counter. QEMU keeps a cached copy so it does not have to reread guest memory for every check. “Cached” means a previously read value; it can fall behind when the guest supplies more buffers.

The scan has its own counter as it adds up the available space. This was the comparison I initially suspected: the guest’s counter wraps after 65535, but the scan uses a wider unsigned integer and can keep counting. “Unsigned” means it cannot be negative. Here, the important difference was what happened when one counter returned to zero and the other did not.

For SPE3 Gi2, the scan started at 65535 and the cached available counter was 1. In a 16-bit counter, those positions are only two steps apart:

Scan position using the wider integer:  65535 → 65536 → 65537
The same position expressed in 16 bits: 65535 →     0 →     1
Cached available counter:                                  1

Follow the two rows in that example. After two buffers, the wider counter reaches 65537 while the cached counter is still 1. Comparing 65537 == 1 returns false, so the scan does not take the branch that refreshes the cached value from guest memory.

The next calculation uses 16-bit arithmetic, though. In that calculation, 65537 becomes 1, and the number of remaining entries comes out as zero. My model stopped the scan at that point, even though the guest had supplied more buffers.

That explains the otherwise odd byte counts. The model counted two 2060-byte buffers for Gi2, totaling 4120 bytes. For Gi3 it counted three, totaling 6180 bytes. Neither total could hold the jumbo frame being tested, despite the replenished buffers waiting in guest memory.

I changed the model’s comparison to use only the lower 16 bits of the scan position. Now 65537 becomes 1 for that comparison, the cached value gets refreshed, and the scan finds enough space. A small packet also passed because it fit in the buffers counted before the problem appeared. A control test whose counters did not cross 65535 passed as well.

This was still a model of the arithmetic. I had not tested a running QEMU device, and I did not deploy that experimental comparison change. I used it to narrow the source review before testing an existing upstream correction.

The running CML binary reported QEMU 8.2.2 from Ubuntu package 1:8.2.2+ds-0ubuntu1.18. Its package patch directory had no changes to these functions.

I found upstream commit f937309, an existing virtio-net receive-stall correction. It changes the insufficient-buffer notification and retry path. I used that correction for the compiled comparison, keeping it separate from my experimental counter change.

Testing the correction inside CML

I cloned the CML VM and disconnected all 18 copied lab interfaces so the test would stay separate from the running network. In that clone, I built the same Ubuntu QEMU source package twice: once unchanged, then with upstream commit f937309 backported. A backport applies a fix to an older version without upgrading the whole application.

The installed Ubuntu binary did not include QEMU’s qtest accelerator. I enabled it in the source rebuild for the direct regression test. qtest is QEMU’s device-testing framework; it let me test the virtual NIC without booting an entire router for every attempt.

The compiled rx-jumbo-wrap test put an emulated receive queue into each of the three states I had observed. It then queued a jumbo frame, replenished the buffers and notified the device that more buffers were available.

The unchanged build stalled in all three cases. With the patch, the full frame arrived through three successive wraps in each case. I could now reproduce the failure without booting IOS, change the QEMU code, and run the same test again.

Next I booted the cloned SPE3 and SP1 on the patched executable, retaining their IOS XE configurations, virtio NICs, and router MTU 9216. Two isolated Linux endpoints sent traffic through both routers. Five 9000-byte DF pings passed. DF means “Don’t Fragment”: the packet must fit the path without being split into smaller IP fragments, making this a useful MTU check. A six-minute bidirectional TCP run delivered 360 MB in each direction at 8 Mbit/s. Counter sampling across the controls and load test recorded four wraps on each of the four traffic-facing NICs, with no receive stall and no IS-IS flap.

The long TCP test still recorded 15 retransmissions in one direction and 17 in the other. I looked at the live socket counters and found tiny 12-byte retransmissions with duplicate acknowledgments.

Notice the sizes: my test wrote 8960-byte blocks, but TCP negotiated an MSS of 8948. That left a 12-byte tail. I repeated the test with writes aligned to the negotiated MSS. It sustained the same rate for 30 seconds with zero retransmissions in either direction. The write pattern gave me a possible explanation for these short retransmissions, but it did not account for the earlier failures elsewhere in the lab.

The isolated router pair was my canary, a small deployment used to check a change before expanding it. The packet tests passed, so I could move on to live CML. This was still an experimental local backport, not a build supplied by Cisco.

Applying the backport to live CML

The routers were already on virtio, so there was no further NIC-model change to make. I needed them to use the patched QEMU executable on the CML host. An IOS reload would leave that QEMU process running. Each router needed a full CML stop/start.

I recorded the live QEMU package version, the executable hash, and the running domain definitions. The tested executable came from the isolated clone; I verified its SHA-256 again on the live host before installation.

The following is a record of the host change, not a standalone installation recipe. It depends on the exact tested binary, firmware checks, preserved configuration and rollback material. I preserved the packaged executable with a local diversion:

dpkg-divert --local --rename --add \
  --divert /usr/bin/qemu-system-x86_64.distrib \
  /usr/bin/qemu-system-x86_64

I put the patched binary at the normal executable path. Routers that were already running kept their original mapped executable until stopped. To roll back, the script restores the diverted vendor binary, and the affected routers need another stop/start. When a suitable vendor package replaces this local build, the diversion also needs to be removed.

The first patched start on live CML failed before IOS could boot:

could not load PC BIOS 'bios-256k.bin'

Moving the executable to /usr/bin had changed where it looked for firmware. The packaged BIOS was already under /usr/share/seabios; I added the missing firmware links under /usr/share/qemu and recorded them for rollback. After checking the startup path, I retried. The fix here was in the host installation, so there was no reason to change the router’s MTU or IOS configuration.

After correcting the firmware lookup, SPE3 booted on the patched executable. I verified the hash of /proc/<pid>/exe, compared its operational configuration with the saved pre-restart configuration, and checked its control plane. Both IS-IS neighbors and both route-reflector sessions were up. Five 9216-byte DF pings to its SP1 neighbor passed:

Sending 5, 9216-byte ICMP Echos to 10.0.0.16, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5)

SPE3 was now running the tested binary with its saved configuration and routing sessions intact. I restarted the remaining routers in waves, keeping alternate core and route-reflector paths available. Before moving to the next wave, I checked routing, saved configuration and the executable actually running.

The receive stall was gone, but TCP still failed

Once all 13 routers were running the tested QEMU build, the network started looking healthy again. Their saved configurations were intact, every expected IS-IS neighbor relationship was up, and BGP peers had re-established their sessions. I could reach all 28 network devices. Large packets were getting through too: each of the six directions between the three datacenters passed five 9000-byte pings with fragmentation disabled.

I also went back to the receive-ring counters. During the first live traffic window, I recorded 18 rollovers across ten interfaces, including two that had stalled before. This time reception continued through the rollovers, and the IS-IS logs showed no new neighbor changes. The observation window was short, but it included the condition that had stopped delivery.

The applications began recovering as connectivity returned. All nine K3s nodes reported Ready, and the Grafana and VictoriaMetrics storage volumes reattached automatically. Grafana’s volume recovered all three replicas. VictoriaMetrics was accessible again, but its volume was still degraded while Longhorn rebuilt the missing redundancy.

The pings passed, but the next TCP test still failed. A 1 MiB transfer using small segments timed out at only 1 Mbit/s, with hundreds of retransmissions.

I reduced the transfer to 256 KiB and compared small and jumbo segments. The small-segment replay still failed, with 117 retransmissions recorded at one endpoint and 95 at the other. The jumbo replay moved the same amount of data in both directions in about 2.2 seconds without a single retransmission. I had requested an MSS of 1200 for the small test and 8960 for the jumbo test; the latter negotiated to 8948 bytes.

The QEMU stall was no longer appearing, but small-segment transfers were still failing. I stopped the planned sustained throughput test and captured the smaller transfer as it crossed the leaves.

Finding the second fault at the leaf

I captured the failing transfer at both K3s guests, Proxmox and EVE. To follow those captures, take a look at what EVE creates when I draw a connection between two virtual devices.

For the QEMU-based nodes in this lab, EVE connects virtual Ethernet ports through host-side TAP interfaces and bridges. A TAP interface is a software Ethernet endpoint: QEMU reads and writes Ethernet frames through it on behalf of a guest NIC. A bridge joins those endpoints into a Layer 2 network, much like a small Ethernet switch. A simple internal connection has this shape once the nodes are running:

Figure 3a. The connection in the topology editor becomes a path through QEMU, TAP interfaces and a host bridge.

The vunl interfaces are the host-side endpoints, not the bridges themselves. An internal lab network uses a bridge named vnet... in this setup. An external Cloud connection attaches to a pnet... bridge, which also connects to an interface of the EVE host. That external path is how my K3s VMs on Proxmox reach the EVE-hosted leaves. The lab interconnect guide records that wiring.

Here is the mapping for DCA-Leaf01, EVE node ID 3. Its host interface vunl0_3_5 backs Ethernet5, and vunl0_3_2 backs Ethernet2. The final number is the interface index in this node’s mapping. Inside vEOS, the corresponding Linux NICs are vmnicet5 and vmnicet2.

These are names for the two sides of the same VM connection: vunl... on the EVE host and vmnicet... inside the switch. I needed the host-side names for my captures and the guest-side names when changing offload settings.

The saved EVE interface map shows Ethernet5’s TAP attached to pnet11, alongside the EVE host’s eth11. Ethernet2’s TAP belonged to vnet0_11, alongside vunl0_2_1, the peer spine’s TAP. For the traffic direction I was investigating, the path was:

Figure 3b. I captured traffic at two host-side TAPs around the same leaf. The labels map each capture to its guest port and bridge.

I was capturing two ports on DCA-Leaf01. At vunl0_3_5, I could see the packet arriving at the server-facing Ethernet5 port. At vunl0_3_2, I could see what left Ethernet2 toward the fabric. The outgoing packet had a VXLAN wrapper, so I compared the TCP packet inside it with the one that had entered Ethernet5.

On the EVE host, these read-only commands expose the relevant bridge membership:

bridge link show dev vunl0_3_5
bridge link show dev vunl0_3_2
ls /sys/class/net/pnet11/brif
ls /sys/class/net/vnet0_11/brif

Look for master in the bridge link output. That tells you which bridge owns the interface. The brif directory lists the other interfaces attached to that bridge. The names above belong to this lab instance, so check the mapping before using them as capture points in another lab.

Now look at the sequence numbers in the capture. A TCP sequence number identifies where a segment’s data starts in the stream. The second segment starts 1188 bytes after the first, exactly the length of the first payload.

After the leaf forwarded them, those two 1188-byte payloads had become one 2376-byte payload starting at the original sequence number. Combining segments is not automatically a fault. The problem was that the combined packet had an invalid checksum, the integrity check covering the TCP header and data.

I had used a known repeated byte for the test payload, which let me reconstruct the payload and verify its checksum against the captured headers. The invalid combined segment also appeared at the receiving host. About two seconds later, valid 1188-byte retransmissions arrived, and the receiver acknowledged those bytes.

That last check matters with offload captures. A checksum can look wrong in a sender-side capture before the NIC finishes processing it. Here, I could follow the malformed segment to the receiving host and then see the valid retransmissions arrive.

Both source leaf ingress NICs had generic-receive-offload: on: DCA-Leaf01 vmnicet5 and DCB-Leaf01 vmnicet4. Generic Receive Offload, or GRO, combines received segments to reduce processing work. Segmentation and checksum handling later in the path must still produce valid packets.

I had valid packets arriving at the leaf and an invalid combined packet leaving it. Disabling GRO on those two ingress NICs gave me a specific change to test against that capture.

Figure 4. I compared the packet before and after the leaf. The lower row previews the controlled test described next: preserving the path was essential to testing the workaround.

The packet captures and parsed results are available in the repository.

Testing GRO on the actual TCP path

I temporarily disabled GRO on DCA-Leaf01 vmnicet5 and DCB-Leaf01 vmnicet4, checking each setting before and after:

# On DCA-Leaf01
bash sudo ethtool -K vmnicet5 gro off
bash ethtool -k vmnicet5

# On DCB-Leaf01
bash sudo ethtool -K vmnicet4 gro off
bash ethtool -k vmnicet4

I ran the replay again and it still failed, with 136/140 retransmissions. Then I checked the captures. The new connection had entered Leaf02 at both sites. I had changed Leaf01.

The K3s hosts use bonds, which group multiple NICs into one logical interface. Their balance-xor mode uses a layer3+4 hash: IP addresses and transport ports determine which member carries a connection. Changing the TCP source port can therefore send a replay through another leaf even when its destination stays the same.

That is what happened here. The small-segment replay bypassed both modified ingress ports. Its paired jumbo replay passed with zero retransmissions, but neither result tested the change I had made on Leaf01.

Figure 5. I had changed Leaf01, but the first replay used Leaf02. Fixing the ports and checking the captures put the comparison back on the modified path. One end is shown; the same issue occurred at both sites.

I repeated the small-segment test with the original source port, 38869, and destination port 49492. Captures confirmed the original ingress path through DCA-Leaf01 and DCB-Leaf01. This time, 256 KiB transferred in each direction with matching SHA-256 hashes and zero retransmissions. Both directions completed within about 2.5 seconds.

I checked the captures again. Each source leaf received 221 valid data segments and emitted 221 valid inner TCP segments in VXLAN. The malformed combined segments from the baseline were gone at those capture points.

Turning GRO off worked on this path. I had not located the faulty code inside vEOS, and I still needed to test the other leaves and workloads.

After that temporary test, I put GRO back to on on both ports and checked the readback. The QEMU correction stayed deployed. I still needed to generate a persistent GRO configuration for the leaves and check the other paths.

Rolling the correction across the leaves

I needed the setting to survive beyond a shell command. I added a device context listing the ten modeled Ethernet data NICs on each of the nine leaves, then generated an EOS event handler for each NIC. An event handler is a saved action triggered by an event, in this case startup.

The source is golden-config/templates/eos/platform.j2, which reads the lab_gro_workaround context. I published the correction after 82 template/structure tests and 149 Batfish/config checks passed. Here is the handler for vmnicet5:

event-handler lab-gro-off-vmnicet5
   action bash sudo -n /sbin/ethtool -K vmnicet5 gro off
   trigger on-boot
   delay 60

EOS runs the on-boot handler after configuration as well as at boot. That gave me two things to inspect: ethtool -k for the current Linux offload state, and startup configuration for the saved instruction. I checked both during the rollout. I did not cold-boot the leaves.

DCA-Leaf01 passed first. I then deployed to the remaining leaves in two waves, comparing each plan with Nautobot’s fresh intended configuration. On each leaf I checked the current offload settings and the saved startup handlers.

On DCC-Leaf02, GRO was already off on all ten NICs, but only nine startup handlers had been saved. The switch was in the state I wanted at that moment; one interface was missing the instruction intended to restore that state after a reboot. Netmiko, the library sending the configuration commands, had reported Pattern not detected: 'end' in output. With Fail Job on Task Failure enabled, that incomplete deployment remained visible as a failure.

I read DCC-Leaf02 again and built a recovery plan containing only the missing handler. After deploying it, I read back all ten saved handlers. The fleet then had GRO off on all 90 data NICs, with a saved handler for each. An actual cold boot was still needed to test their startup execution.

I repeated the small-segment transfer and checked the leaf captures. The TCP checksums were valid at the observed boundaries, and the malformed combined segments were gone from that replay. Routing was up across the fabric. The transfer completed with matching data, zero retransmissions at one endpoint and one at the other.

I widened the test to include all nine K3s hosts and every site pair. All nine bidirectional transfers completed, but their endpoints recorded 41 retransmissions in total. A separate three-minute transfer also completed, with two retransmissions at one end and eight at the other. Packet delivery had improved enough to finish these transfers; it had not become consistently free of retransmissions. With Longhorn already rebuilding a volume, I held off on adding a higher-rate traffic test.

K3s showed all nine nodes and 65 pods Ready in the recorded samples. The first 23-minute comparison had no increase in container restarts. Later, four of Longhorn’s storage-integration containers restarted, so I went back to the volume that was rebuilding.

The clearest remaining failure came near the end of a VictoriaMetrics replica rebuild. It reached 97%, then stopped with a read/write timeout and connection resets. Longhorn automatically replaced the failed replica and tried again, but another attempt reset too. Grafana’s storage remained healthy. The GRO workaround had corrected the malformed packets in my captures, yet it had not made this storage transfer reliable.

CE2 and CE3 were carrying roughly 17–18 Mbit/s against a configured 20 Mbit/s throughput limit, so I checked whether the rebuild was running out of forwarding capacity. The sampled forwarding and queue counters did not show that limit causing drops. The endpoints did not show CPU or memory exhaustion either. I could not explain the resets from those checks, so I moved on to the connection Longhorn was losing.

Testing how quickly the rebuild connection gave up

The rebuild was getting almost to the end before its connection reset. Longhorn coordinates that work through remote procedure calls, or RPCs, between storage processes. I wanted to see how long the TCP connection carrying those calls would wait when packets stopped getting through.

I inspected the sockets owned by the live rebuild processes. They had TCP_USER_TIMEOUT set to 20 seconds, with TCP keepalive enabled after 15 seconds of inactivity and a 15-second interval between unanswered probes. I read those values from the running connections instead of assuming the chart defaults were in use.

TCP_USER_TIMEOUT limits how long TCP will tolerate data remaining unacknowledged, or buffered without being sent. It also affects when an idle connection with unanswered keepalive probes is closed. The interaction mattered here: a short interruption could miss a probe, and restoring the network would not necessarily rescue the connection before its timeout logic ran.

I tested that possibility in an isolated Linux network namespace, which gave me a separate network environment where I could deliberately drop packets. I introduced a five-second loss window covering the first keepalive probe, then let traffic through again. I ran the same experiment with a 20-second user timeout and a 60-second user timeout, changing only that setting.

By the check at about 35 seconds, the 20-second connection had reported ETIMEDOUT, Linux’s connection-timeout error. The 60-second connection was still open and could exchange data after the interruption. The loss window was the same; the longer timeout gave that test connection enough time to recover.

I had tested one connection under a controlled loss window. That did not reproduce a whole Longhorn rebuild or tell me what was dropping packets in the lab.

I raised the user timeout on specific live rebuild RPC sockets as a temporary mitigation. Those changes lasted only as long as the connections stayed open; they were not saved Longhorn settings.

Both application volumes eventually recovered three read/write replicas. I could confirm that they were healthy again, but I could not say the timeout change alone had done it. I still needed to find the cause of the interruptions and decide whether a permanent Longhorn timeout change was appropriate.

Could I trust the next successful test?

Once Longhorn finished rebuilding, I ran the small-segment and jumbo checks again. A reboot had already given me a temporary recovery earlier in this investigation, so I wanted to repeat the transfers that had exposed the failures.

Before starting, I checked the running QEMU binaries and the leaf offload settings. All 13 router processes matched the tested build, and all 90 leaf data NICs still had GRO off. Both CML socket samples had clear receive queues. Routing was up, both application volumes had three healthy replicas, and the cluster snapshots showed no pod replacements or increase in container restarts.

The small-segment transfers completed across nine host pairs, covering every K3s node and every site pair. The data matched at both ends, with zero retransmissions. Jumbo transfers across three site pairs also completed with matching data and zero retransmissions. Large pings passed in all six inter-site directions.

I then kept a jumbo transfer running for three minutes at 1 Mbit/s in each direction. It delivered just over 22.5 million bytes each way, with matching hashes. One endpoint recorded no retransmissions; the other recorded four. The original receive freeze had not returned, and the small-segment tests were working, but the longer run still gave me a reason to keep looking. None of these deliberately paced tests established the network’s maximum capacity or replaced a cold-boot check of the leaf handlers.

The remaining problem showed up under load

The short transfers were working. Then I looked at a cross-site Longhorn connection under storage load: roughly 3 MiB retransmitted out of 31 MiB sent, with a reported round-trip time around 734 ms. I still had a problem to investigate.

The routing logs supplied another clue. Before the repeat tests, BFD had triggered BGP session resets, including one between DCA-Spine01 and DCB-Spine02. BFD is a rapid failure detector: if its probes stop getting through in time, it can tell BGP to tear down a session. The most recent session recovered in roughly two seconds, but the message Cease/BFD down <Hard Reset> did not tell me why the probes had been missed. Packet loss and scheduling delays were possibilities to investigate, not conclusions from that log alone.

The QEMU and GRO changes stayed in place because their failure cases were now passing. I still needed to correlate Longhorn traffic with the BFD resets and run a sustained load test. The saved verification evidence contains the process checks and traffic samples for that work.

Where I would start next time

I would start with the failing connection and map its actual path. On CML, I have the router’s interface counters, the host socket queues and QEMU’s receive-ring state to compare. In EVE, I can capture the packet before and after the leaf and check that it leaves with the same valid TCP data.

I also need to keep the test addresses and ports fixed. Changing the source port sent the first GRO replay through another leaf. It looked like the setting had done nothing until I checked which interfaces the new flow was actually using.

The lab was working well enough to finish the short transfers and recover storage, but the load-related retransmissions and routing resets still needed attention. In From Network Devices to Grafana: Building the Lab’s SNMP Pipeline, I bring the network’s interface counters into Grafana so I can follow those measurements while the applications use the network. That post also covers the event-collection gap: polling counters will not capture every brief BFD or BGP transition.

References

• QEMU receive-stall correction, f937309

• Linux segmentation offloads and GRO

• Persistent leaf GRO source

• Linux TCP socket options

What Kubernetes is, and why K3s

Quick grounding for anyone who hasn’t touched this before. Kubernetes runs containers across a group of machines and keeps them running the way you described, not the way they happen to be right now. You tell it “I want this app running, with this much memory, reachable on this port,” and a control plane continuously checks the real state against that description and corrects any gap: a crashed container gets restarted, a machine that goes away gets its work moved somewhere else. The machines are called nodes. The smallest thing Kubernetes schedules is a pod, one or more containers that share networking and storage. A cluster is the whole assembly: the API server and scheduler that make the decisions, plus every node that runs the actual work.

K3s is a real Kubernetes, not a scaled-down imitation of one: same API, same kubectl, same Helm charts you’d point at a full-size EKS or GKE cluster. What’s different is the packaging. Every control-plane component (API server, scheduler, controller-manager) runs out of a single binary instead of a dozen separate processes, and the default datastore is sqlite3 rather than etcd, though etcd is still available for exactly the case this cluster needs: a real multi-node HA control plane. It’s built for edge, homelab, IoT, and other places a full Kubernetes install would be overkill, and that’s the profile these VMs fit: modest CPU and RAM, sitting behind emulated network gear, on a fabric that’s slow and jittery on purpose.

The rule for this series has been that Nautobot is the source of truth and Git holds the intent behind it. The network devices already work that way through Golden Config. The servers didn’t, until now. By the end of this post the servers are provisioned by Ansible, and Ansible reads everything it needs, which hosts exist, how to reach them, what role each one plays, straight from the model.

One nuance this post makes concrete, because glossing over it is how “source of truth” turns into a lie: Git holding the intent and the running Nautobot reflecting it are two separate things. Some data (config contexts) Nautobot pulls from Git on every sync, so a commit is enough. Other data (anything Design Builder generates) is written once at build time and then lives only in the database, so a commit alone never touches it. When the two drift, you fix Git for the next build and you apply the change to the running instance as a deliberate second step. You will see both halves in this post.

The shape of the thing we’re building

Ten Ubuntu VMs, all already modeled in Nautobot as devices (Part 1 built them, Part 5 cabled the first one’s bond). Nine become a single K3s cluster. The tenth, DCA-DNS, becomes a BIND server.

The cluster is stretched, not sharded. One cluster spans DC-A, DC-B, and DC-C. That only works because VLAN 100 is one EVPN type-5 segment stretched across all three sites, so every node shares the 192.168.100.0/24 subnet no matter which datacenter it physically sits in. Three server nodes run embedded etcd and form the control plane, and all three live in DC-A. The other six nodes are agents, split evenly across DC-B and DC-C.

DCC-k3s-w6 came later than the other eight, added specifically so DC-C would have three workers instead of two. That matters once storage enters the picture below: three nodes per DC means every DC can hold a full 3-replica Longhorn volume without a single replica crossing the emulated core.

Keeping all three etcd members in one DC is not where this started. The first design put one member in each site, so the cluster would survive losing a whole datacenter. That’s the textbook layout, and on this fabric it does not work. The why turned out to be the most useful thing in this post, so we’ll come back to it once the cluster is standing.

Read more

Part 4 built the datacenter fabric and proved EVPN worked across all three sites. The DCI overlay exchanged type-2 MAC routes between DC-A, DC-B, and DC-C, so a K3s node on a leaf in one datacenter could reach a node on a leaf in another, and a ping across the stretched VXLAN data plane came back with zero loss. What Part 4 did not build was a way for that SERVERS traffic to leave the fabric. The servers could reach each other across every DC, but they had no path to the internet.

This post builds that exit. By the end, a server in any of the three DCs can reach the internet through BORDER1, and the return traffic finds its way back.

Read more

That’s the datacenter fabric for this lab: 15 vEOS switches across three sites, eBGP underlay, EVPN overlay, all running inside EVE-NG on the same Proxmox host as the CML SP core. Getting them configured was the easy part (Nautobot templates handle that). Getting them to actually boot was the first real problem.

Each box gets a startup-config in EVE-NG so it comes up reachable. Production config is a Golden Config push from Nautobot.

If you want to reproduce this, you’ll need two VMs on a Proxmox host (or any KVM hypervisor). CML needs 12 vCPU, 80 GB RAM, and 80 GB disk for the 13 Cisco cat8000v nodes. EVE-NG needs 8 vCPU, 64 GB RAM, and 80 GB disk for the 15 Arista vEOS nodes. That’s roughly 144 GB total RAM for the network layer alone. You’ll need the EVE-NG VM running with the Blog-SandBox topology loaded.

Here’s the proof. Local leaves Established. Remote spines Active:

Read more

TLDR: Part 3 of 10. We configure Nautobot credentials, Nornir, compliance features, and Golden Config Plans, then deploy generated intent to 13 Cisco IOS-XE routers in CML. We finish by verifying IS-IS, MPLS LDP, VPNv4 and VPNv6 route reflection, and PE-to-CE BGP.

Where we left off

Part 2 rendered intended configurations for the full topology from Nautobot Source of Truth data. This walkthrough narrows the deployment to the 13 Cisco routers that make up the service-provider network:

• Four P routers running IS-IS and MPLS

• Two VPNv4 and VPNv6 route reflectors

• One border router

• Three service-provider edge routers

• Three customer edge routers

The intended files are stored in Git, but nothing should reach a router until we can answer four questions:

1. Can the worker authenticate to every device?

2. Does compliance extract the right configuration sections?

3. Will those sections be sent in a dependency-safe order?

4. Can we prove that running configuration matches intent after deployment?

This walkthrough sets up each layer in that order.

Deployment workflow

Golden Config deployment depends on fresh output from every earlier stage:

Credentials + primary IP + network reachability
    -> intended configuration
    -> running-config backup
    -> compliance rules
    -> compliance results
    -> Config Plan
    -> review and approval
    -> deployment
    -> new backup and compliance run

A Config Plan stores command text when it is created. If intent, backups, rules, or source-of-truth data change, generate a new plan.

Read more

TLDR: Part 3 deploys generated intent to 13 Cisco IOS-XE routers on Thursday. Before those configs reach a device, we add three validation gates: Jinja2 linting, strict render tests, and Batfish parsing. Together they catch template syntax errors, missing data, and invalid IOS-XE commands in under five seconds.

The problem we’re solving

Part 3 deploys generated intent to 13 Cisco IOS-XE routers this Thursday. While building it, I found a gap we needed to close first: a rendered configuration is not necessarily a valid configuration.

Parts 1 and 2 got us from Nautobot source-of-truth data to rendered config files. When I started working through the deployment, every template change followed the same pattern:

1. Edit the Jinja2 template

2. Regenerate intended configs

3. Build a Config Plan

4. Deploy to one device

5. Watch it fail with % Invalid input detected

6. Read the error, fix the template

7. Go back to step 2

That is the network version of testing in production. Each iteration costs 5 to 10 minutes while jobs run, configs render, and plans build. A missing | default() filter or a typo in a command can burn 30 minutes before you find it.

Software teams solved this problem decades ago. You don’t deploy code to production to find out if it compiles. You run the build locally, run the tests, and only ship what passes. We’re going to do the same thing for network configuration.

What we’re building

Three validation gates, each catching a different class of error:

Template change
    -> j2lint (Jinja syntax: unclosed blocks, bad delimiters)
    -> pytest render (undefined variables, None in output, empty config)
    -> Batfish parse (invalid CLI commands, malformed syntax)
    -> commit allowed
    -> push triggers GitHub Actions (same checks, fresh environment)
    -> only then: regenerate intent, compliance, deploy

The fast path (make ci) takes under 2 seconds. The full path with Batfish (make ci-full) adds about 3 more. Either way, it’s faster than a single Nautobot job run.

Layer 1: Jinja2 linting with j2lint

j2lint comes from Arista’s AVD team. It checks Jinja2 template syntax without needing any context data. Think of it as a compiler for your templates.

pip install j2lint

Run it against the templates directory:

j2lint golden-config/templates --extensions j2 \
  -i jinja-statements-indentation single-statement-per-line

We ignore two rules. The jinja-statements-indentation rule (S4) wants Jinja control blocks indented by 4 spaces per nesting level. In network config templates, that would mean {% if %} blocks indented 20+ spaces deep while the IOS commands they produce sit at 1-space indent. It makes the templates unreadable. The single-statement-per-line rule flags {% if list.append(x) %}{% endif %}, which is a standard Jinja idiom for building lists during iteration.

Everything else stays on. If you forget to close a {% for %} block or misspell a filter name, j2lint catches it instantly.

Layer 2: Render tests with strict variable checking

This is where the real value lives. We render every template against mock SoT data using Jinja2’s StrictUndefined mode. If the template references a variable that doesn’t exist in the context, the test explodes with the exact line and variable name.

The mock contexts

We need representative data for each device role and platform combination. I pulled these directly from the Nautobot GraphQL query that golden config uses:

tests/mock_contexts/
├── cisco_ios_route_reflector.yaml   # RR1: ISIS, MPLS, BGP with RR-client logic
├── cisco_ios_pe_router.yaml         # SPE1: VRF, PE-CE eBGP, full SP stack
├── arista_eos_leaf.yaml             # DCA-Leaf01: EVPN, VXLAN, SVIs, VRFs
└── arista_eos_spine.yaml            # DCA-Spine01: underlay + overlay redistribution

Each file mirrors the exact shape of what the sp_demo_lab_golden_config GraphQL query returns. Here’s a trimmed example for the PE router:

hostname: SPE1

config_context:
  isis:
    process_name: SP-ISIS
    metric_style: wide
    is_type: level-2-only
  mpls:
    ldp_router_id: Loopback0
    ldp_sync: true
    explicit_null: true
  bgp:
    timers:
      keepalive: 10
      hold: 30
    default_ipv4_unicast: false

interfaces:
  - name: GigabitEthernet3
    description: "to CE1 GigabitEthernet2"
    enabled: true
    vrf:
      name: CUSTOMER-A
      rd: "65000:100"
    ip_addresses:
      - address: "172.16.0.0/31"
        ip_version: 4
    # ... full structure continues

The test file

# tests/test_template_render.py
import jinja2
import pytest
import yaml

DEVICE_SCENARIOS = [
    ("cisco_ios_route_reflector.yaml", "cisco_ios"),
    ("cisco_ios_pe_router.yaml", "cisco_ios"),
    ("arista_eos_leaf.yaml", "arista_eos"),
    ("arista_eos_spine.yaml", "arista_eos"),
]

def build_jinja_env():
    return jinja2.Environment(
        loader=jinja2.FileSystemLoader(str(REPO_ROOT)),
        undefined=jinja2.StrictUndefined,  # THIS IS THE KEY
        trim_blocks=True,
        keep_trailing_newline=True,
    )

@pytest.mark.parametrize("context_file,platform", DEVICE_SCENARIOS)
def test_template_renders_without_error(context_file, platform):
    env = build_jinja_env()
    template = env.get_template(f"golden-config/templates/{platform}.j2")
    context = load_context(context_file)
    rendered = template.render(**context)

    assert len(rendered.strip()) > 50

The StrictUndefined setting is the entire point. Without it, Jinja2 silently renders missing variables as empty strings. With it, you get:

jinja2.exceptions.UndefinedError: 'config_context' is undefined
  File "golden-config/templates/ios/isis.j2", line 3

That error message tells you exactly what’s wrong and where. Compare that to deploying and getting % Invalid input detected at '^' marker from the device, which tells you nothing about the root cause.

What else the tests check

Beyond StrictUndefined, we validate four things per scenario:

1. The rendered output is longer than 50 characters (catches templates that render empty)

2. No raw {{ or {% tags leak into the output (catches templates that partially fail)

3. No Python None appears in the config lines (catches missing | default() filters)

4. The first meaningful line is hostname <expected> (catches structural problems)

Layer 3: Batfish vendor-aware config parsing

Batfish parses network configs the way a router does. It builds a vendor-specific model of your configuration and flags anything the device parser would reject.

Setting it up

docker run -d --name batfish -p 9997:9997 -p 9996:9996 batfish/batfish:latest
pip install pybatfish

What it validates

We render the templates, write them as .cfg files to a temp directory, and feed them to Batfish as a “snapshot.” Batfish then parses each file and reports:

• Parse status (PASSED, PARTIALLY_UNRECOGNIZED, or FAILED)

• Parse warnings (specific lines it couldn’t understand)

• Undefined references (route-maps, ACLs, or prefix-lists that are referenced but never defined)

For our IOS-XE devices, Batfish reported zero unexpected warnings. The only flagged line was ip ssh bulk-mode 131072, which is a newer IOS-XE 17.x command that Batfish’s grammar hasn’t added yet. We mark that as known-benign.

KNOWN_BENIGN_IOS = {
    "ip ssh bulk-mode",  # IOS-XE 17.x feature, Batfish grammar is behind
}

If you introduced a typo like routr bgp 65000 or ip addres 10.0.0.1 255.255.255.0, Batfish would catch it here. No device touched.

The EOS caveat

Batfish currently misidentifies Arista EOS configs as Cisco IOS, which produces many false-positive warnings for EOS-specific syntax (vrf instance, neighbor X peer group, VXLAN commands). We filter these out and rely primarily on the Jinja render tests for EOS validation. As Batfish improves its EOS parser detection, this will get better.

Cross-device BGP analysis

The more interesting Batfish capability is cross-device validation. When you load configs for multiple devices, Batfish can verify that BGP sessions have matching configurations on both sides:

def test_bgp_session_compatibility(self, batfish_full_results):
    bf = batfish_full_results
    bgp_edges = bf.q.bgpEdges().answer().frame()
    # If edges exist, the sessions are configured consistently

And it can find undefined references (a route-map referenced in a neighbor statement that doesn’t exist anywhere):

def test_undefined_references(self, batfish_full_results):
    bf = batfish_full_results
    undef = bf.q.undefinedReferences().answer().frame()
    # These would cause silent policy failures on the device

Running it locally

The Makefile gives you three entry points:

make ci        # j2lint + render tests (~2 seconds, no container needed)
make ci-full   # above + Batfish validation (~5 seconds, needs container)
make validate  # Batfish only

Here’s what a clean run looks like:

$ make ci-full
j2lint golden-config/templates --extensions j2 -i jinja-statements-indentation single-statement-per-line
pytest tests/test_template_render.py -v
tests/test_template_render.py::test_template_renders_without_error[cisco_ios_route_reflector.yaml-cisco_ios] PASSED
tests/test_template_render.py::test_template_renders_without_error[cisco_ios_pe_router.yaml-cisco_ios] PASSED
tests/test_template_render.py::test_template_renders_without_error[arista_eos_leaf.yaml-arista_eos] PASSED
tests/test_template_render.py::test_template_renders_without_error[arista_eos_spine.yaml-arista_eos] PASSED
...
16 passed in 0.93s

pytest tests/test_batfish_validate.py -v
tests/test_batfish_validate.py::TestBatfishIOSValidation::test_all_ios_configs_parsed PASSED
tests/test_batfish_validate.py::TestBatfishIOSValidation::test_no_unexpected_ios_parse_warnings PASSED
tests/test_batfish_validate.py::TestBatfishEOSValidation::test_all_eos_configs_parsed PASSED
tests/test_batfish_validate.py::TestBatfishEOSValidation::test_no_unexpected_eos_parse_warnings PASSED
tests/test_batfish_validate.py::TestBatfishBGPValidation::test_bgp_session_compatibility PASSED
tests/test_batfish_validate.py::TestBatfishBGPValidation::test_undefined_references PASSED
6 passed in 2.78s

22 tests, under 4 seconds, zero devices involved.

The pre-commit hook

We don’t want to rely on remembering to run make ci. A git pre-commit hook runs it automatically whenever you commit files that touch templates or config contexts:

#!/bin/bash
# .git/hooks/pre-commit

STAGED_TEMPLATES=$(git diff --cached --name-only | \
  grep -E '(golden-config/templates/|config_contexts/|tests/mock_contexts/)' || true)

if [ -z "$STAGED_TEMPLATES" ]; then
    exit 0
fi

echo "Template files staged for commit, running validation..."
make ci

If validation fails, the commit is rejected. You can bypass with --no-verify, but you shouldn’t.

GitHub Actions for CI/CD

The pre-commit hook is a local safety net. For the team-level guarantee, we add a GitHub Actions workflow that triggers on pushes and PRs touching template files:

# .github/workflows/validate-templates.yml
name: Validate Golden Config Templates

on:
  push:
    paths:
      - 'golden-config/templates/**'
      - 'config_contexts/**'
      - 'tests/**'
  pull_request:
    paths:
      - 'golden-config/templates/**'
      - 'config_contexts/**'
      - 'tests/**'

jobs:
  lint-and-render:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install jinja2 pyyaml pytest j2lint
      - run: make ci

  batfish-validate:
    runs-on: ubuntu-latest
    needs: lint-and-render
    services:
      batfish:
        image: batfish/batfish:latest
        ports:
          - 9997:9997
          - 9996:9996
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install jinja2 pyyaml pytest pybatfish
      - run: make validate

The pipeline is two stages. The fast lint-and-render job runs first. If it fails, Batfish never spins up. If it passes, the Batfish service container starts and runs the deeper validation. GitHub Actions provides the container as a service, so there’s no Docker-in-Docker complexity.

The maintenance trade-off

The mock contexts need updating when you add new variables to templates. That’s the cost. If you add config_context.new_feature.key to a template but don’t add it to the mock, pytest catches it immediately:

jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'new_feature'

This is a feature, not a bug. It forces you to answer “where does this data come from?” before the template consumes it. Is it in the config context YAML? Is it in the GraphQL query? If neither, the variable will fail in production too, not just in the test.

The full workflow now

1. Edit template or config context
2. git add + git commit
   └── pre-commit hook runs `make ci`
       ├── j2lint: Jinja syntax OK?
       └── pytest: renders clean with all variables?
3. git push
   └── GitHub Actions triggers
       ├── lint-and-render job (same as local)
       └── batfish-validate job (vendor grammar check)
4. CI passes → regenerate intended in Nautobot
5. Nautobot pushes intended configs to Git
   └── GitHub Actions triggers again
       ├── Sanity checks on all 28 device configs
       └── Batfish parses every .cfg for invalid syntax
6. CI passes → run compliance → build Config Plan
7. Deploy to one device per platform
8. Verify → expand in waves

Steps 1 through 3 catch template bugs. Step 5 catches data bugs (a device in Nautobot with missing or malformed SoT data that produces an invalid config). Both run automatically. Both block the pipeline if something is wrong.

The intended config validation is particularly useful because it checks real data. The mock contexts are representative, but they can’t cover every edge case across 28 devices. A VRF that’s missing a route-target, an interface with an unexpected prefix length, a BGP endpoint with no peering defined. Those only show up when you render against the real SoT, and Batfish catches them before they become a Config Plan.

Validating the actual intended configs (not just mock renders)

Everything above validates templates before they produce output. But there’s a second gate that matters just as much: validating what Nautobot actually generates.

When Nautobot’s intended job runs, it renders your templates against live SoT data for all 28 devices and pushes the results to golden-config/intended-configs/. That SoT data might have quirks the mocks don’t cover. A device someone added without a BGP routing instance. A VRF missing its route-targets. An interface with a /28 mask that the template only handles /24, /31, and /32.

We add a second test file that scans the actual generated configs:

# tests/test_intended_configs.py

INTENDED_CONFIGS = collect_intended_configs()  # finds all .cfg files

class TestIntendedConfigSanity:
    """Basic checks, no Batfish needed."""

    def test_config_not_empty(self, platform, cfg_path):
        content = cfg_path.read_text()
        assert len(content.strip()) > 200  # catch render failures

    def test_no_none_in_config(self, platform, cfg_path):
        # Python None leaking into config = missing | default() filter

    def test_no_jinja_artifacts(self, platform, cfg_path):
        # {{ or {% in output = partial render failure

    def test_starts_with_hostname(self, platform, cfg_path):
    def test_ends_with_end(self, platform, cfg_path):

class TestIntendedConfigsBatfish:
    """Feed all 28 configs to Batfish."""

    def test_no_failed_parses(self, batfish_results):
    def test_no_unexpected_parse_warnings(self, batfish_results):
    def test_no_undefined_references_ios(self, batfish_results):

Running it against our actual lab output:

$ pytest tests/test_intended_configs.py -v
...
143 passed in 2.37s

28 devices, 5 sanity checks each, plus 3 Batfish assertions covering all of them. Every single intended config parsed cleanly. The only warnings were the same ip ssh bulk-mode line on IOS-XE devices, which we already know is benign.

The GitHub Actions workflow triggers on pushes to golden-config/intended-configs/**. So the flow is: Nautobot runs the intended job, commits and pushes the rendered configs, GitHub Actions validates them, and you get a green or red check before you ever build a Config Plan.

What this means for network engineering

We just moved golden config development from “push and pray” to something that looks like a real software development lifecycle. The templates are code. They have tests. The tests run on every commit. Broken configs get rejected before they can reach a device.

This isn’t theoretical. In the SP demo lab, the BGP template alone is 140 lines of Jinja2 with 8 levels of nested logic handling route-reflector-client decisions, PE-CE VRF peering, VPNv4 and VPNv6 address families. A single missing variable or misplaced {% endif %} in that template would have required a full redeploy cycle to diagnose. Now it fails in under a second with a clear error message pointing at the exact line.

The tools are free. j2lint, pytest, and Batfish are all open source. The GitHub Actions minutes for this workload are negligible. The only ongoing cost is maintaining the mock contexts when templates evolve, and that cost is strictly less than the cost of a failed deployment.

What comes next

These checks answer one question: should this configuration be allowed into the deployment workflow? They don’t replace a reviewed Config Plan, a staged rollout, or protocol verification.

Part 3 publishes Thursday for paid subscribers. We configure Nautobot credentials and Nornir, deploy to the Cisco core in waves, then verify IS-IS, MPLS LDP, VPNv4, VPNv6, and PE-to-CE BGP.

This free companion is part of the SP Demo Lab series. All code is in the blog-sandbox repo.

The series is 10 parts. By the end, we'll have 28 devices running ISIS, MPLS, BGP, and EVPN. Configs generated from a source of truth. Monitoring that catches failures across layers. And an AI agent that receives alerts, pulls route tables and interface state, and writes up what broke without anyone logging into a router.

All of it runs on one Proxmox box. No cloud. No physical gear beyond a single server.

Here's what the topology looks like: an MPLS L3VPN core connecting 3 datacenter customers. Each customer site runs a leaf-spine EVPN/VXLAN fabric. 13 Cisco IOS-XE routers and 15 Arista EOS switches. The monitoring stack is Prometheus, Loki, Grafana, and OTel Collector, with device inventory pulled from Nautobot. The AI piece is NetClaw, an agent I built that receives alerts from Alertmanager, queries all those systems for context, and produces root cause analysis.

What I'm showing across the series:

1. SoT-driven infrastructure where every device, IP, cable, and BGP session lives in Nautobot and everything else queries it

2. Config generation from templates, not manual CLI

3. Failure detection across network, transport, and application layers

4. AI root cause analysis pulling state from before and after the alert

5. Config compliance and drift detection

6. The whole thing reproducible on one box

I believe the next generation of network engineers won't get the apprenticeship I had. Nobody's going to hand them an expensive network and say "learn by doing" for two years. So I'm building the lab, the walkthroughs, and eventually a rentable environment where you can follow along hands-on.

Free subscribers get post previews and occasional full unlocks. Paid subscribers get every walkthrough, every config, every template. Founding Members get all of that plus 20% off lab rentals when they launch, and priority on what I cover next.

If you learn by building, you're in the right place.

TLDR: Part 2 of 10. We wire Nautobot Golden Config to our blog-sandbox GitHub repo, write modular Jinja2 templates that pull ISIS/MPLS/BGP data via GraphQL plus config contexts for operational parameters, and generate intended configurations for all 28 devices - entirely from Source of Truth data. No spreadsheets, no manual config, no Ansible. One “Generate Intended Configs” button produces the complete SP core and DC fabric configurations.

Read more

TLDR: This is Part 1 of a 10-part series building a full service provider MPLS/EVPN lab (28 devices in CML) with AI-driven monitoring. In this post we deploy Nautobot 3.2.1 with Design Builder, BGP Models, IGP Models, and Golden Config, then populate the entire topology (devices, IPs, cabling, VRFs) in a single idempotent job run. Everything downstream, config generation, monitoring inventories, AI investigation, queries this one source of truth.

What we’re building and why

This series builds a complete service provider network from scratch and then monitors it the way a modern NOC would. An AI agent investigates alerts, correlates state history, and produces root cause analysis without a human touching a CLI.

The network: an MPLS L3VPN core with 3 datacenter customers, each running a leaf-spine EVPN/VXLAN fabric with K3s clusters and distributed applications. 28 devices total: 13 Cisco IOS-XE routers (CAT8000v) and 15 Arista EOS switches (vEOS), all running in CML on Proxmox.

The MPLS L3VPN core: P routers, PE routers, and route reflectors connecting three customer sites.

Each datacenter customer runs a leaf-spine EVPN/VXLAN fabric with K3s workloads.

The monitoring: a full observability stack (Prometheus, Loki, Grafana, OTel Collector) pulling inventory from a real Source of Truth (Nautobot), plus SuzieQ for historical state queries. When an alert fires, the system can ask “what did the BGP table look like 5 minutes before this happened?”

The AI NOC: NetClaw, an AI agent that receives alerts from Alertmanager, queries the monitoring stack and SuzieQ for context, and posts a root cause analysis to a diary, all without operator intervention for defined alert types.

What I hope to demonstrate by the end:

1. SoT-driven infrastructure: every device, IP, cable, and BGP session lives in Nautobot. Config templates, monitoring inventories, and investigations all query the same truth.

2. Failure detection across layers: a single link failure cascades through network (BGP/IGP reconvergence), transport (latency spike), and application (replication lag) layers, and the stack sees all three.

3. AI-assisted root cause analysis: when a PE-CE link drops, the agent doesn’t just report “BGP peer down.” It queries what the route table, LLDP neighbors, and interface state looked like before the alert, identifies the specific interface that failed, and reports whether traffic rerouted or the customer is isolated.

4. Configuration compliance: Nautobot Golden Config holds the intended state, detects drift, and the stack can remediate with assert-before-and-after validation via SuzieQ.

5. Reproducible by anyone: everything runs on a single Proxmox host with CML. No cloud accounts, no physical switches, no vendor licenses beyond the CML images.

The series is 10 parts. This is Part 1: we stand up Nautobot as the source of truth and populate it with the complete topology using a Design Builder job, so every subsequent part can query it programmatically instead of maintaining spreadsheets.

Why start with the Source of Truth

Every step that follows, configuring routers, deploying monitoring, wiring up AI investigation needs to know what devices exist, where they are, what IPs they have, and how they’re connected. Nautobot is the system that holds that data and makes it queryable by every tool in the stack.

Deploy Nautobot 3.2.1

1. Clone nautobot-docker-compose

cd ~
git clone https://github.com/nautobot/nautobot-docker-compose.git
cd nautobot-docker-compose

2. Pin Nautobot to 3.2.1

Edit pyproject.toml and set the Nautobot version:

[tool.poetry.dependencies]
nautobot = "3.2.1"

3. Add the apps

poetry add nautobot-design-builder
poetry add nautobot-bgp-models
poetry add nautobot-golden-config
poetry add nautobot-igp-models

This updates pyproject.toml and poetry.lock with all four apps and their dependencies resolved against Nautobot 3.2.1.

4. Enable the apps in nautobot_config.py

Edit config/nautobot_config.py:

PLUGINS = [
    "nautobot_design_builder",
    "nautobot_bgp_models",
    "nautobot_golden_config",
    "nautobot_igp_models",
]

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        "connection_options": {
            "napalm": {
                "extras": {
                    "optional_args": {
                        "global_delay_factor": 1,
                        "transport": "ssh"
                    },
                },
            },
            "netmiko": {
                "extras": {
                    "global_delay_factor": 1,
                    "fast_cli": False,
                    "read_timeout_override": 30,
                    "disabled_algorithms": {"pubkeys": ["rsa-sha2-256", "rsa-sha2-512"]},
                },
            },
        },
        "nornir_settings": {
            "credentials": "nautobot_plugin_nornir.plugins.credentials.nautobot_secrets.CredentialsNautobotSecrets",
            "runner": {
                "plugin": "threaded",
                "options": {
                    "num_workers": 20,
                },
            },
        },
    },
    "nautobot_golden_config": {
        "per_feature_bar_width": 0.15,
        "per_feature_width": 13,
        "per_feature_height": 4,
        "enable_backup": True,
        "enable_compliance": True,
        "enable_intended": True,
        "enable_sotagg": True,
        "enable_plan": True,
        "enable_deploy": True,
        "enable_postprocessing": True,
        "sot_agg_transposer": None,
        "postprocessing_callables": ['nautobot_golden_config.utilities.config_postprocessing.render_secrets'],
        "postprocessing_subscribed": [],
        "jinja_env": {
            "undefined": "jinja2.StrictUndefined",
            "trim_blocks": True,
            "lstrip_blocks": False,
            "extensions": ["jinja2.ext.do"],
        },
        "default_framework": {"all": "netmiko"},
        "get_config_framework": {"all": "netmiko"},
        "default_deploy_status": "Not Approved",
    }
}

5. Build and start

invoke build --no-cache
invoke start

Wait ~2 minutes for migrations (the BGP/IGP/Golden Config models add database tables on first start), then verify:

curl -s http://localhost:8080/health/

Log in at http://<your-vm-ip>:8080. You should see:

• Routing → BGP in the nav (BGP Models)

• Routing → IGP in the nav (IGP Models)

• Golden Config in the nav

• Design Builder under Extensibility or Jobs

Why these four apps

AppWhat it gives youUsed inDesign BuilderDeclarative bulk data population from YAML templatesPart 1: populate the full topology in one job runBGP ModelsASN, BGP Routing Instances, Peer Groups, Peerings, Address FamiliesPart 2–3: model all iBGP/eBGP sessionsIGP ModelsOSPF/IS-IS instances, areas, interface configsPart 2: model the IS-IS underlayGolden ConfigIntended config generation, backup, compliance checkingPart 10: drift detection and remediation

All four have Nautobot 3.0+ compatibility releases.

Load the SP Demo Lab design

The Design Builder job lives in the jobs/ folder of nautobot-docker-compose. Nautobot picks up any Python job modules in that folder on startup.

Copy the design into the jobs folder

# From wherever you have the design files
cp -r sp_demo_lab ~/nautobot-docker-compose/jobs/

The structure inside jobs/:

jobs/
  sp_demo_lab/
    __init__.py          # DesignJob class + register_jobs()
    context/
      __init__.py        # All 28 devices, links, VRFs from addressing plan
    designs/
      0001_foundations.yaml.j2  # Locations, roles, platforms, device types, VRFs
      0002_devices.yaml.j2      # 28 devices with interfaces + IPs
      0003_cabling.yaml.j2      # Cables + P2P IP assignments

Restart to pick up the new job

cd ~/nautobot-docker-compose
invoke stop start

Run the design

1. Jobs → SP Demo Lab - Full Topology → Run

2. The job renders the templates with the context data and creates all objects

3. Re-running is idempotent. Existing objects are updated, not duplicated

What the design creates

Object typeCountDetailsLocations5SP-Demo-Lab (region), SP-Core, DC-A, DC-B, DC-C (sites)Device Roles7P-Router, PE-Router, Route-Reflector, CE-Router, Border-Router, Spine, LeafPlatforms2cisco_iosxe, arista_eosDevice Types2CAT8000v, vEOSDevices2813 Cisco CAT8000v + 15 Arista vEOSInterfaces~250Management, Loopback0, data interfaces per deviceIP Addresses~100Management, loopbacks, P2P /31 assignmentsPrefixes~20Management, SP core, DC fabricsVRFs4MGMT-VRF, CUST-A, CUST-B, CUST-CCables~40All inter-device links

Verify

UI quick checks

• Devices → 28 total, filter by site to verify distribution

• IPAM → Prefixes → 10.0.0.0/24 has child /31s

• IPAM → IP Addresses → loopbacks and P2P links populated

• Cables → all inter-device connections present

• Routing → BGP → empty for now (Part 2 populates this)

Lifecycle management

• Update: edit context/__init__.py with new devices or IPs, restart Nautobot, re-run the job → objects updated in place

• Decommission: Design Builder → Deployments → select → Decommission → all objects from that run removed cleanly

• Add a device: add it to the context list, re-run. No need to touch 4 different places.

What you have now

LayerStateNautobot 3.2.1Running with BGP Models, IGP Models, Golden Config, Design BuilderSoT data28 devices, full IPAM, cabling, VRFs populatedNetwork devicesManagement IPs up, no production config yetMonitoringNot deployed yet (Part 6)NetClawNot deployed yet (Part 8)

Next: Part 2, SP Core: IS-IS, MPLS, and BGP

Reference

• nautobot-docker-compose: https://github.com/nautobot/nautobot-docker-compose

• Design Builder docs: https://docs.nautobot.com/projects/design-builder/

• BGP Models docs: https://docs.nautobot.com/projects/bgp-models/

• Golden Config docs: https://docs.nautobot.com/projects/golden-config/

• IGP Models: https://github.com/byrn-baker/nautobot-app-igp-models

• Blog Sandbox: https://github.com/byrn-baker/blog-sandbox/