Catch Broken Network Configs Before They Reach a Router

TLDR: Part 3 deploys generated intent to 13 Cisco IOS-XE routers on Thursday. Before those configs reach a device, we add three validation gates: Jinja2 linting, strict render tests, and Batfish parsing. Together they catch template syntax errors, missing data, and invalid IOS-XE commands in under five seconds.

The problem we’re solving

Part 3 deploys generated intent to 13 Cisco IOS-XE routers this Thursday. While building it, I found a gap we needed to close first: a rendered configuration is not necessarily a valid configuration.

Parts 1 and 2 got us from Nautobot source-of-truth data to rendered config files. When I started working through the deployment, every template change followed the same pattern:

1. Edit the Jinja2 template

2. Regenerate intended configs

3. Build a Config Plan

4. Deploy to one device

5. Watch it fail with % Invalid input detected

6. Read the error, fix the template

7. Go back to step 2

That is the network version of testing in production. Each iteration costs 5 to 10 minutes while jobs run, configs render, and plans build. A missing | default() filter or a typo in a command can burn 30 minutes before you find it.

Software teams solved this problem decades ago. You don’t deploy code to production to find out if it compiles. You run the build locally, run the tests, and only ship what passes. We’re going to do the same thing for network configuration.

What we’re building

Three validation gates, each catching a different class of error:

Template change
    -> j2lint (Jinja syntax: unclosed blocks, bad delimiters)
    -> pytest render (undefined variables, None in output, empty config)
    -> Batfish parse (invalid CLI commands, malformed syntax)
    -> commit allowed
    -> push triggers GitHub Actions (same checks, fresh environment)
    -> only then: regenerate intent, compliance, deploy

The fast path (make ci) takes under 2 seconds. The full path with Batfish (make ci-full) adds about 3 more. Either way, it’s faster than a single Nautobot job run.

Layer 1: Jinja2 linting with j2lint

j2lint comes from Arista’s AVD team. It checks Jinja2 template syntax without needing any context data. Think of it as a compiler for your templates.

pip install j2lint

Run it against the templates directory:

j2lint golden-config/templates --extensions j2 \
  -i jinja-statements-indentation single-statement-per-line

We ignore two rules. The jinja-statements-indentation rule (S4) wants Jinja control blocks indented by 4 spaces per nesting level. In network config templates, that would mean {% if %} blocks indented 20+ spaces deep while the IOS commands they produce sit at 1-space indent. It makes the templates unreadable. The single-statement-per-line rule flags {% if list.append(x) %}{% endif %}, which is a standard Jinja idiom for building lists during iteration.

Everything else stays on. If you forget to close a {% for %} block or misspell a filter name, j2lint catches it instantly.

Layer 2: Render tests with strict variable checking

This is where the real value lives. We render every template against mock SoT data using Jinja2’s StrictUndefined mode. If the template references a variable that doesn’t exist in the context, the test explodes with the exact line and variable name.

The mock contexts

We need representative data for each device role and platform combination. I pulled these directly from the Nautobot GraphQL query that golden config uses:

tests/mock_contexts/
├── cisco_ios_route_reflector.yaml   # RR1: ISIS, MPLS, BGP with RR-client logic
├── cisco_ios_pe_router.yaml         # SPE1: VRF, PE-CE eBGP, full SP stack
├── arista_eos_leaf.yaml             # DCA-Leaf01: EVPN, VXLAN, SVIs, VRFs
└── arista_eos_spine.yaml            # DCA-Spine01: underlay + overlay redistribution

Each file mirrors the exact shape of what the sp_demo_lab_golden_config GraphQL query returns. Here’s a trimmed example for the PE router:

hostname: SPE1

config_context:
  isis:
    process_name: SP-ISIS
    metric_style: wide
    is_type: level-2-only
  mpls:
    ldp_router_id: Loopback0
    ldp_sync: true
    explicit_null: true
  bgp:
    timers:
      keepalive: 10
      hold: 30
    default_ipv4_unicast: false

interfaces:
  - name: GigabitEthernet3
    description: "to CE1 GigabitEthernet2"
    enabled: true
    vrf:
      name: CUSTOMER-A
      rd: "65000:100"
    ip_addresses:
      - address: "172.16.0.0/31"
        ip_version: 4
    # ... full structure continues

The test file

# tests/test_template_render.py
import jinja2
import pytest
import yaml

DEVICE_SCENARIOS = [
    ("cisco_ios_route_reflector.yaml", "cisco_ios"),
    ("cisco_ios_pe_router.yaml", "cisco_ios"),
    ("arista_eos_leaf.yaml", "arista_eos"),
    ("arista_eos_spine.yaml", "arista_eos"),
]

def build_jinja_env():
    return jinja2.Environment(
        loader=jinja2.FileSystemLoader(str(REPO_ROOT)),
        undefined=jinja2.StrictUndefined,  # THIS IS THE KEY
        trim_blocks=True,
        keep_trailing_newline=True,
    )

@pytest.mark.parametrize("context_file,platform", DEVICE_SCENARIOS)
def test_template_renders_without_error(context_file, platform):
    env = build_jinja_env()
    template = env.get_template(f"golden-config/templates/{platform}.j2")
    context = load_context(context_file)
    rendered = template.render(**context)

    assert len(rendered.strip()) > 50

The StrictUndefined setting is the entire point. Without it, Jinja2 silently renders missing variables as empty strings. With it, you get:

jinja2.exceptions.UndefinedError: 'config_context' is undefined
  File "golden-config/templates/ios/isis.j2", line 3

That error message tells you exactly what’s wrong and where. Compare that to deploying and getting % Invalid input detected at '^' marker from the device, which tells you nothing about the root cause.

What else the tests check

Beyond StrictUndefined, we validate four things per scenario:

1. The rendered output is longer than 50 characters (catches templates that render empty)

2. No raw {{ or {% tags leak into the output (catches templates that partially fail)

3. No Python None appears in the config lines (catches missing | default() filters)

4. The first meaningful line is hostname <expected> (catches structural problems)

Layer 3: Batfish vendor-aware config parsing

Batfish parses network configs the way a router does. It builds a vendor-specific model of your configuration and flags anything the device parser would reject.

Setting it up

docker run -d --name batfish -p 9997:9997 -p 9996:9996 batfish/batfish:latest
pip install pybatfish

What it validates

We render the templates, write them as .cfg files to a temp directory, and feed them to Batfish as a “snapshot.” Batfish then parses each file and reports:

• Parse status (PASSED, PARTIALLY_UNRECOGNIZED, or FAILED)

• Parse warnings (specific lines it couldn’t understand)

• Undefined references (route-maps, ACLs, or prefix-lists that are referenced but never defined)

For our IOS-XE devices, Batfish reported zero unexpected warnings. The only flagged line was ip ssh bulk-mode 131072, which is a newer IOS-XE 17.x command that Batfish’s grammar hasn’t added yet. We mark that as known-benign.

KNOWN_BENIGN_IOS = {
    "ip ssh bulk-mode",  # IOS-XE 17.x feature, Batfish grammar is behind
}

If you introduced a typo like routr bgp 65000 or ip addres 10.0.0.1 255.255.255.0, Batfish would catch it here. No device touched.

The EOS caveat

Batfish currently misidentifies Arista EOS configs as Cisco IOS, which produces many false-positive warnings for EOS-specific syntax (vrf instance, neighbor X peer group, VXLAN commands). We filter these out and rely primarily on the Jinja render tests for EOS validation. As Batfish improves its EOS parser detection, this will get better.

Cross-device BGP analysis

The more interesting Batfish capability is cross-device validation. When you load configs for multiple devices, Batfish can verify that BGP sessions have matching configurations on both sides:

def test_bgp_session_compatibility(self, batfish_full_results):
    bf = batfish_full_results
    bgp_edges = bf.q.bgpEdges().answer().frame()
    # If edges exist, the sessions are configured consistently

And it can find undefined references (a route-map referenced in a neighbor statement that doesn’t exist anywhere):

def test_undefined_references(self, batfish_full_results):
    bf = batfish_full_results
    undef = bf.q.undefinedReferences().answer().frame()
    # These would cause silent policy failures on the device

Running it locally

The Makefile gives you three entry points:

make ci        # j2lint + render tests (~2 seconds, no container needed)
make ci-full   # above + Batfish validation (~5 seconds, needs container)
make validate  # Batfish only

Here’s what a clean run looks like:

$ make ci-full
j2lint golden-config/templates --extensions j2 -i jinja-statements-indentation single-statement-per-line
pytest tests/test_template_render.py -v
tests/test_template_render.py::test_template_renders_without_error[cisco_ios_route_reflector.yaml-cisco_ios] PASSED
tests/test_template_render.py::test_template_renders_without_error[cisco_ios_pe_router.yaml-cisco_ios] PASSED
tests/test_template_render.py::test_template_renders_without_error[arista_eos_leaf.yaml-arista_eos] PASSED
tests/test_template_render.py::test_template_renders_without_error[arista_eos_spine.yaml-arista_eos] PASSED
...
16 passed in 0.93s

pytest tests/test_batfish_validate.py -v
tests/test_batfish_validate.py::TestBatfishIOSValidation::test_all_ios_configs_parsed PASSED
tests/test_batfish_validate.py::TestBatfishIOSValidation::test_no_unexpected_ios_parse_warnings PASSED
tests/test_batfish_validate.py::TestBatfishEOSValidation::test_all_eos_configs_parsed PASSED
tests/test_batfish_validate.py::TestBatfishEOSValidation::test_no_unexpected_eos_parse_warnings PASSED
tests/test_batfish_validate.py::TestBatfishBGPValidation::test_bgp_session_compatibility PASSED
tests/test_batfish_validate.py::TestBatfishBGPValidation::test_undefined_references PASSED
6 passed in 2.78s

22 tests, under 4 seconds, zero devices involved.

The pre-commit hook

We don’t want to rely on remembering to run make ci. A git pre-commit hook runs it automatically whenever you commit files that touch templates or config contexts:

#!/bin/bash
# .git/hooks/pre-commit

STAGED_TEMPLATES=$(git diff --cached --name-only | \
  grep -E '(golden-config/templates/|config_contexts/|tests/mock_contexts/)' || true)

if [ -z "$STAGED_TEMPLATES" ]; then
    exit 0
fi

echo "Template files staged for commit, running validation..."
make ci

If validation fails, the commit is rejected. You can bypass with --no-verify, but you shouldn’t.

GitHub Actions for CI/CD

The pre-commit hook is a local safety net. For the team-level guarantee, we add a GitHub Actions workflow that triggers on pushes and PRs touching template files:

# .github/workflows/validate-templates.yml
name: Validate Golden Config Templates

on:
  push:
    paths:
      - 'golden-config/templates/**'
      - 'config_contexts/**'
      - 'tests/**'
  pull_request:
    paths:
      - 'golden-config/templates/**'
      - 'config_contexts/**'
      - 'tests/**'

jobs:
  lint-and-render:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install jinja2 pyyaml pytest j2lint
      - run: make ci

  batfish-validate:
    runs-on: ubuntu-latest
    needs: lint-and-render
    services:
      batfish:
        image: batfish/batfish:latest
        ports:
          - 9997:9997
          - 9996:9996
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install jinja2 pyyaml pytest pybatfish
      - run: make validate

The pipeline is two stages. The fast lint-and-render job runs first. If it fails, Batfish never spins up. If it passes, the Batfish service container starts and runs the deeper validation. GitHub Actions provides the container as a service, so there’s no Docker-in-Docker complexity.

The maintenance trade-off

The mock contexts need updating when you add new variables to templates. That’s the cost. If you add config_context.new_feature.key to a template but don’t add it to the mock, pytest catches it immediately:

jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'new_feature'

This is a feature, not a bug. It forces you to answer “where does this data come from?” before the template consumes it. Is it in the config context YAML? Is it in the GraphQL query? If neither, the variable will fail in production too, not just in the test.

The full workflow now

1. Edit template or config context
2. git add + git commit
   └── pre-commit hook runs `make ci`
       ├── j2lint: Jinja syntax OK?
       └── pytest: renders clean with all variables?
3. git push
   └── GitHub Actions triggers
       ├── lint-and-render job (same as local)
       └── batfish-validate job (vendor grammar check)
4. CI passes → regenerate intended in Nautobot
5. Nautobot pushes intended configs to Git
   └── GitHub Actions triggers again
       ├── Sanity checks on all 28 device configs
       └── Batfish parses every .cfg for invalid syntax
6. CI passes → run compliance → build Config Plan
7. Deploy to one device per platform
8. Verify → expand in waves

Steps 1 through 3 catch template bugs. Step 5 catches data bugs (a device in Nautobot with missing or malformed SoT data that produces an invalid config). Both run automatically. Both block the pipeline if something is wrong.

The intended config validation is particularly useful because it checks real data. The mock contexts are representative, but they can’t cover every edge case across 28 devices. A VRF that’s missing a route-target, an interface with an unexpected prefix length, a BGP endpoint with no peering defined. Those only show up when you render against the real SoT, and Batfish catches them before they become a Config Plan.

Validating the actual intended configs (not just mock renders)

Everything above validates templates before they produce output. But there’s a second gate that matters just as much: validating what Nautobot actually generates.

When Nautobot’s intended job runs, it renders your templates against live SoT data for all 28 devices and pushes the results to golden-config/intended-configs/. That SoT data might have quirks the mocks don’t cover. A device someone added without a BGP routing instance. A VRF missing its route-targets. An interface with a /28 mask that the template only handles /24, /31, and /32.

We add a second test file that scans the actual generated configs:

# tests/test_intended_configs.py

INTENDED_CONFIGS = collect_intended_configs()  # finds all .cfg files

class TestIntendedConfigSanity:
    """Basic checks, no Batfish needed."""

    def test_config_not_empty(self, platform, cfg_path):
        content = cfg_path.read_text()
        assert len(content.strip()) > 200  # catch render failures

    def test_no_none_in_config(self, platform, cfg_path):
        # Python None leaking into config = missing | default() filter

    def test_no_jinja_artifacts(self, platform, cfg_path):
        # {{ or {% in output = partial render failure

    def test_starts_with_hostname(self, platform, cfg_path):
    def test_ends_with_end(self, platform, cfg_path):

class TestIntendedConfigsBatfish:
    """Feed all 28 configs to Batfish."""

    def test_no_failed_parses(self, batfish_results):
    def test_no_unexpected_parse_warnings(self, batfish_results):
    def test_no_undefined_references_ios(self, batfish_results):

Running it against our actual lab output:

$ pytest tests/test_intended_configs.py -v
...
143 passed in 2.37s

28 devices, 5 sanity checks each, plus 3 Batfish assertions covering all of them. Every single intended config parsed cleanly. The only warnings were the same ip ssh bulk-mode line on IOS-XE devices, which we already know is benign.

The GitHub Actions workflow triggers on pushes to golden-config/intended-configs/**. So the flow is: Nautobot runs the intended job, commits and pushes the rendered configs, GitHub Actions validates them, and you get a green or red check before you ever build a Config Plan.

What this means for network engineering

We just moved golden config development from “push and pray” to something that looks like a real software development lifecycle. The templates are code. They have tests. The tests run on every commit. Broken configs get rejected before they can reach a device.

This isn’t theoretical. In the SP demo lab, the BGP template alone is 140 lines of Jinja2 with 8 levels of nested logic handling route-reflector-client decisions, PE-CE VRF peering, VPNv4 and VPNv6 address families. A single missing variable or misplaced {% endif %} in that template would have required a full redeploy cycle to diagnose. Now it fails in under a second with a clear error message pointing at the exact line.

The tools are free. j2lint, pytest, and Batfish are all open source. The GitHub Actions minutes for this workload are negligible. The only ongoing cost is maintaining the mock contexts when templates evolve, and that cost is strictly less than the cost of a failed deployment.

What comes next

These checks answer one question: should this configuration be allowed into the deployment workflow? They don’t replace a reviewed Config Plan, a staged rollout, or protocol verification.

Part 3 publishes Thursday for paid subscribers. We configure Nautobot credentials and Nornir, deploy to the Cisco core in waves, then verify IS-IS, MPLS LDP, VPNv4, VPNv6, and PE-to-CE BGP.

---

This free companion is part of the SP Demo Lab series. All code is in the blog-sandbox repo.