Three attempts to separate the planes: VLANs in a two-node Proxmox homelab

June 15, 2026

For most of the lab’s life, every device on the network shared a single /24: 192.168.2.0/24. All thirty-eight VMs. Both Proxmox hosts’ management interfaces. Two iDRACs. Unraid. The NAS’s IPMI. Kea DHCP. Prometheus scraping everything it could reach. All of it on the same wire, same broadcast domain, same network segment.

This is called a flat network, and it works fine — right up until it doesn’t. The failure mode isn’t a daily operational problem. It’s a quiet architectural one. When everything shares a layer, the compromised VM and the hypervisor console are neighbors. The rogue IoT bulb and the NAS are on the same broadcast domain. The Echo in the kitchen can send ARP probes to the storage cluster.

Fixing this means VLANs, and VLANs meant a cutover. It took three attempts.

What a flat network costs you

The standard answer is security: segmentation limits blast radius when something goes wrong. That’s true and worth stating, but it undersells the operational case.

In a Proxmox environment specifically, mixing management traffic with guest traffic means:

The hypervisor is reachable from its own guests. A virtual machine running on pve1 and a Proxmox web UI session on pve1 share the same physical uplink, the same switch port, and the same IP subnet. If a VM is compromised, the attacker is one subnet hop away from the console that controls every other VM on the host.

iDRAC and IPMI are on the guest network. Out-of-band management — the “lights-out” controllers that let you power cycle, mount ISOs, and access a serial console even when the OS is down — should only be reachable by administrators. On a flat network, every VM can reach them. If your iDRAC firmware has a CVE (and it will), the scope of that CVE is every device on your network.

Cluster traffic isn’t isolated. Proxmox corosync, live migration, and storage replication all run on whatever network you’ve given them. If that’s the same flat segment as guest traffic, your cluster heartbeat competes with whatever load your VMs generate.

None of this causes visible problems on a quiet day. But a well-designed network shouldn’t rely on everything behaving.

The architecture: three planes, one physical cable

The target design splits traffic into three logical planes, all running over the same physical 10G uplink between each host and the core switch:

Data plane (VLAN 1, native/untagged): VM guest traffic, NFS and SMB to storage, Prometheus scrapes. Everything that lives in 192.168.2.0/24. This VLAN rides untagged — existing VMs don’t need reconfiguration to keep working during the transition.

Admin plane (VLAN 3, 192.168.3.0/24): Proxmox web UI, SSH to the hypervisors, Ansible, Terraform, corosync heartbeat. Only machines that need to administer the cluster are here. The Proxmox hosts themselves are dual-homed — a LAB address for VM traffic and a MGMT address for admin access.

Out-of-band plane (VLAN 4, 192.168.4.0/24): iDRAC and IPMI only. Nothing else. The last octet mirrors the host’s MGMT address by convention — pve1’s iDRAC lives at 192.168.4.40, matching its MGMT address at 192.168.3.40. Makes the mapping obvious without looking it up.

Beyond those three, the full layout adds VLANs for home clients, guests, IoT, future cameras, and future sensors — eight segmented networks in total, each with its own firewall rules enforced by OPNsense.

The default stance is deny-between-VLANs. Exceptions are explicit and enumerated in OPNsense firewall rules — things like Prometheus on LAB being allowed to scrape MGMT hosts, or Home Assistant on LAB being allowed to reach IoT device ports.

The prerequisite: hardware first

The original plan called for a dedicated 1G management port on each host — one NIC for MGMT, a separate 10G for LAB. That required having a spare NIC slot and cable run for each host, and it created two classes of interface to track.

The NDC swap changed that. In early April, the old mix of onboard BCM5720 1G NICs and Intel X520 PCIe 10G cards came out of both pve1 and pve2. In went Dell 68M95 quad-port 10G SFP+ NDCs — mezzanine cards that slot into the server’s internal NIC bay rather than a PCIe slot. Both hosts now carry identical hardware: four 10G ports each, named uniformly eno1np0 through eno4np3.

The new setup uses eno1np0 as the trunk uplink to the core switch on both hosts, carrying all VLANs tagged over a single 10G cable. eno4np3 goes to a direct 10G DAC crosslink between pve1 and pve2 for corosync. The two middle ports are spares.

The old vmbr0 management bridge, which used to sit on the 1G NIC, is gone. MGMT traffic rides vmbr2 alongside guest traffic — separated by VLAN tag, not by physical wire. It’s one fewer cable run per host, and the interface inventory is uniform across both nodes.

Three attempts

Attempt one: looking at the wrong layer

The first cutover attempt collapsed during the switch configuration phase. The hosts were set up correctly, OPNsense had the VLAN interfaces defined — but traffic between VLANs wasn’t routing the way it should.

The instinct was to look at OPNsense: firewall rules, VLAN interface assignments, Kea DHCP scopes. A solid hour of cross-checking configuration that was, as it turned out, fine. The problem was one layer down, on the Dell switch. A trunk port wasn’t carrying the right VLAN tags — VLAN 3 wasn’t being passed to the uplink — so MGMT traffic was being dropped at the switch before it ever reached OPNsense.

Once that was found, the fix was a single switch configuration line. The hour chasing OPNsense was unnecessary. The lesson was the standard one: when traffic isn’t flowing, work the path from the physical layer up, not from the application layer down.

Attempt two: hardcoded interface names

The second attempt hit a different wall. The NDC swap had changed every interface name on both hosts — the old NIC identifiers were gone, replaced by eno1np0..eno4np3. The Proxmox network configuration in /etc/network/interfaces had the old names hardcoded. When the new card came up, the bridge assignments were wrong: the new 10G uplink mapped to vmbr3, the corosync crosslink bridge, instead of vmbr2, the LAB trunk.

The hypervisors came up thinking their lab uplink was a host-to-host link. Connections went to the wrong places. The fix required dropping to console, editing the network interfaces file to reference the correct eno1np0 for vmbr2, and restarting networking before the cutover could continue.

This one was a direct consequence of not auditing the interface-to-bridge mapping after swapping the hardware. The name change from the old scheme to enoNnpM is a documented behavior of these cards — it just wasn’t accounted for in the prep.

Attempt three: April 17

The third attempt went cleanly. The switch VLAN tags were verified on the bench before touching the hosts. The interface-to-bridge mapping was confirmed in the config files before any network changes. The order was: switch trunks → OPNsense VLAN interfaces → Kea scopes → dual-home the hosts on MGMT → verify → OOB cables → verify.

The corosync crosslink stayed on its dedicated DAC throughout. Live migration of a test VM between nodes confirmed the cluster was healthy before anything touched the management path. The cutover took about ninety minutes from start to verified-stable.

What the network looks like now

The Proxmox web UI and SSH are reachable on .3.40 (pve1) and .3.45 (pve2) from MGMT only. Ansible and Terraform reach the hosts on those addresses. If something on LAB tries to reach the Proxmox console, the firewall drops it.

The iDRACs are on .4.40 and .4.45, behind a strict OOB policy. The only inbound traffic allowed from OOB is return traffic to MGMT hosts — LDAP for authentication, syslog to the monitoring stack. iDRAC-to-internet is blocked entirely.

The LAB VLAN (192.168.2.0/24) operates the same as the old flat network, for the VMs. They have no visibility into the admin or OOB planes. Their gateway is OPNsense at 192.168.2.1, which handles routing and enforces the inter-VLAN rules.

The IoT devices — Echoes, Wyze bulbs, Shelly plugs, Fire TV, the Sleep Number hub, the printer — moved to VLAN 10. They can reach the internet (for now; that’ll be tightened). They can reach Home Assistant on LAB for automation commands. They cannot reach the NAS, the hypervisors, or anything on MGMT. An Echo on the kitchen counter and the Proxmox cluster are now genuinely isolated from each other.

One design rule held through all of this: zero cloud dependency for core services. Every *.solar-hawk.cc hostname resolves to a LAN IP via AdGuard — not a Cloudflare edge address. The Cloudflare tunnel is additive, for services that benefit from external exposure. But DNS, Authentik SSO, Gitea, Vault, and the monitoring stack all work with the WAN unplugged. The VLAN segmentation didn’t change that contract; it reinforced it by making the DNS and authentication infrastructure unreachable from the guest and IoT VLANs.

What it didn’t fix

A few things are still pending. The iDRAC cables are physically on a dedicated OOB switch port, but the iDRAC firmware updates — which require contacting Dell’s update servers — need a temporary internet path through OOB that I haven’t fully audited. The corosync redundant ring on MGMT (running corosync traffic on a separate interface from the admin management traffic) is planned but not yet configured. And the QFX5100-48S that will eventually replace the Dell N3224T-ON as the core switch means this VLAN config gets rebuilt when that hardware arrives.

The flat network is gone, though. The admin plane is separated from the data plane. The lights-out controllers are isolated. That was the goal.

The VLAN scheme, firewall rule matrix, and Kea reservation layout live in the Obsidian knowledge base and get committed to Gitea on change. The switch config is in the switch itself — Dell N3224T-ON, accessible only from MGMT.

← all posts