Skip to main content

06 · Physical network topology

Per-host cabling · 4 × 25G data ports + 1 × 1G BMC — same cabling on all 18 hosts. · interactive original ↗

spine-1100Gspine-2100G2 × 100G uplinksleaf-1a · ToRrack switch · 25G portsleaf-1b · ToRrack switch · 25G portsMLAG pair · peer linkOOB mgmt switchVLAN 90 · isolatedHV-XX · ANY HARVESTER NODE · SAME CABLING ON ALL 18 HOSTSnic-0 25Gnic-1 25Gnic-2 25Gnic-3 25Gbmc 1Gbond0 · LACP (nic-0 + nic-1)mgmt VLAN 100 · pod/CNI overlay · guest VLANs 300–399bond0 — one 25G to each ToRbond1 · LACP (nic-2 + nic-3)Longhorn 200 · Ceph 210 / 211 · MTU 9000bond1 — one 25G to each ToRbmc · IPMIlights-out · never bonded or routed

Bonds​

bond0 · LACP (nic-0 + nic-1)​

Application side: mgmt VLAN 100 · pod/CNI overlay · guest VLANs 300–399. One 25G port to each ToR.

bond1 · LACP (nic-2 + nic-3)​

Storage side: Longhorn VLAN 200 · Ceph public 210 · Ceph cluster 211 · MTU 9000. One 25G port to each ToR.

bmc · IPMI​

Lights-out management on the isolated OOB switch (VLAN 90) · never bonded or routed.

Terminology​

  • "2 NIC for application" → bond0: 2 × 25G ports in one LACP bond
  • "2 NIC for storage" → bond1: 2 × 25G ports in one LACP bond
  • LACP bond — two ports act as one link: 2× bandwidth, survives port/cable loss
  • MLAG — the bond's two cables go to two different ToR switches, so a whole switch can fail
  • VLAN tagging (802.1Q) — multiple networks share a bond, isolated by tags
  • ToR / leaf — top-of-rack switch; each rack has a redundant pair

Failure behavior​

  • Cable or port dies → bond keeps running on the other 25G link
  • ToR switch dies → both bonds keep running via the surviving leaf
  • Storage rebuild storm → saturates bond1 only; application traffic on bond0 unaffected