Platform Engineering · Lesson 64

Network Switching & Routing

A frame leaves a laptop on one floor and reaches a server in a data center. It crosses a switch, a VLAN tag, a trunk, a spanning tree check, a router, and a VXLAN tunnel. Nine hops. Each hop reads a different table, and each table can lie in its own way. I have debugged all of them at 2 AM. This lesson walks the frame through, hop by hop, and shows you which table to read when the frame never arrives.

Difficulty: Intermediate · Know this first: F14, F15, F16, Lesson 63

What you will build

By the end of this lesson you will have run three Python simulators on your own machine (a MAC table, an OSPF cost calculator, a frame-size checker), built a Packet Tracer lab with two switches, two VLANs, a trunk, spanning tree, and a router, and read a routing table the way you would read it on call. You will also know why a blocked spanning tree port is the design working, why a static route is a promise you keep by hand, and why a VXLAN tunnel adds 50 bytes to every frame.

What you need: Python 3 for the three simulators, and Packet Tracer (free with a Cisco Networking Academy account) or GNS3 for the lab. Nothing in this lesson runs against a live network from this page. The Python parts run on this machine and their outputs are shown exactly as they ran. The Packet Tracer parts run on your machine, and their outputs are marked as what you should see in Packet Tracer. No device output is faked. A local network-namespace lab was attempted in this VM and failed: the VM runs as uid 0 but ip netns returns Operation not permitted, and no mininet, GNS3, or Packet Tracer binary exists in the VM. The reader-side labs are the honest path.

The story: one frame, nine hops

A laptop sits on the third floor, VLAN 10, address 10.20.1.10. The file server it wants sits in the data center, VLAN 20, address 10.20.2.20. Between them: an access switch, a trunk, a core switch, a router, a VXLAN tunnel, and a second switch. The laptop knows the server is not on its own subnet, so it sends the frame to its default gateway. The frame travels nine hops. Each hop reads a table. The switch reads its MAC table. The trunk reads the VLAN tag. Spanning tree reads the port states. The router reads its routing table. The tunnel endpoint reads the VNI. When the frame never arrives, the fix is always the same move: find the hop where the table is wrong, and fix the table.

The CIDR and subnet math behind the two VLANs comes from F15 and F16. VLAN 10 is 10.20.1.0/24. VLAN 20 is 10.20.2.0/24. They do not overlap, and F16 taught you to check that before you build. Lesson 63 taught the packet journey in general. This lesson teaches the switching and routing that carries it.

HopWhat it readsWhat goes wrong
Access switchMAC tableUnknown MAC floods every port
VLAN tagPort assignmentWrong VLAN, wrong network
TrunkAllowed VLAN listDropped VLAN, silent failure
Spanning treePort statesTwo forwarding uplinks, a loop
RouterRouting tableWrong next hop, a detour
VXLAN tunnelVNI and MTUFrame too big, silent drop

Do this now, before you read the hops

Open Packet Tracer and build the topology from the lab. Two switches, one router, two laptops, one server. You will come back to it in every section below. If you have never opened Packet Tracer, the download link and the first steps are in the lab.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one frame from a laptop to a server. These four paths are the same idea drawn four ways. One frame, delivered. One frame, caught in a loop. One frame, measured in bytes. One network, at 3 AM. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Plug a cable into itself, and the orange dot stops at the spanning tree gate and never comes back.

Path 1: the clean delivery

Loop: skipped

One frame walks from the laptop to the server. Access switch, VLAN tag, trunk, spanning tree, router, VXLAN tunnel, far switch, server. Every hop answers, and the server answers back.

Laptop0 ms
Access switch1 ms
VLAN tag1 ms
Trunk2 ms
Spanning tree1 ms
Router2 ms
VXLAN tunnel3 ms
Dest switch2 ms
Server2 ms

Not run yet.

Path 2: the loop spanning tree stops

Server: skipped

Same frame, same floor, one extra cable. The frame enters the loop between the two switches. Spanning tree blocks the second uplink, and the frame stops at the blocked port. The server never sees it.

Laptop0 ms
Access switch1 ms
Trunk2 ms
Second switch2 ms
Blocked portblocked

Not run yet.

Path 3: the frame size trail

Routing: skipped

The same frame, measured in bytes at each hop. The VLAN tag adds 4 bytes. The VXLAN tunnel adds 50. The far side strips the tunnel back off, and the server gets the same 64 bytes the laptop sent.

Laptop64 B
Access switch64 B
VLAN tag68 B
Trunk68 B
Spanning tree68 B
Router68 B
VXLAN tunnel118 B
Dest switch68 B
Server64 B

Not run yet.

Path 4: the tables at 3 AM

Peak traffic: skipped

The same network at 3 AM. Each hop shows what its table holds when nobody is working. The tables are small, the links are quiet, and the whole network waits for morning.

Laptop1 host
Access switch48 MACs
VLAN tag4 VLANs
Trunk4 tagged
Spanning tree1 blocked
Router120 routes
VXLAN tunnel6 tunnels
Dest switch48 MACs
Server1 host

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one frame through the switching and routing path, not a measurement from a live network, but the shape is honest. The router and the tunnel do most of the work. The switch lookups are cheap because a MAC table lookup is one hash. At the simulator defaults the total is 14 ms. Move a simulator slider and watch the access, trunk, router, and tunnel hops move with it. Switch the scenario and watch the trunk hop or the router hop change instead.

Bigger network, slower hops: move the sliders and watch the tables grow

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the question every network grows into: what happens to the trip when the network gets bigger. You set how many hosts sit on the access switch, how many VLANs ride the trunk, how many routes sit in the routing table, and how many VXLAN tunnels leave this VTEP. Move a slider and watch the MAC table, the broadcast traffic, the spanning tree convergence, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: every table costs time, and the broadcast domain is the one that hurts first.

48 hosts

How many hosts plug into this switch. Each one is one MAC table entry, and each one ARPs. More hosts means a bigger table and more broadcast frames.

4 VLANs

How many VLANs ride the trunk to the core. Each VLAN is one spanning tree instance, and each instance takes about 2 seconds to converge.

120 routes

How many routes the router holds. The lookup is fast, but a table with 2,000 routes is a table a human cannot read in a hurry.

6 tunnels

How many tunnels leave this tunnel endpoint. Each one is a remote site or a remote rack, and each one adds encapsulation work.

Try this: (1) At the defaults, 48 hosts, 4 VLANs, 120 routes, 6 tunnels, the MAC table holds 48 entries. Broadcast traffic is 96 frames a second. Spanning tree converges in 8 seconds. The budget total is 14 ms. (2) Drop to 10 hosts, 1 VLAN, 10 routes, 1 tunnel. The MAC table holds 10 entries. Broadcast traffic is 20 frames a second. Spanning tree converges in 2 seconds. The budget total is 13 ms. One VLAN means the trunk hop costs 1 ms instead of 2. (3) Push to 500 hosts, 20 VLANs, 2000 routes, 50 tunnels. The MAC table holds 500 entries. Broadcast traffic is 1000 frames a second. The access hop costs 5 ms, the trunk 6 ms, the router 10 ms, the tunnel 9 ms. The budget total is 36 ms. The status turns bad: 500 hosts in one broadcast domain is a storm waiting for a loop.

Formulas, in plain terms. MAC table entries = hosts. Broadcast frames per second = hosts times 2. Spanning tree convergence seconds = VLANs times 2. Access switch hop = 1 plus hosts divided by 120, rounded down. Trunk hop = 1 plus VLANs divided by 4, rounded down. Router hop = 2 plus routes divided by 250, rounded down. VXLAN tunnel hop = 3 plus tunnels divided by 8, rounded down. Budget total = access plus trunk plus router plus tunnel plus the five fixed hops (laptop 0, VLAN tag 1, spanning tree 1, far switch 2, server 2). The numbers are a teaching model, not live measurements. The shape will not change when you measure a real network: every table costs time, and the broadcast domain is the one that hurts first.

Lab: run the network yourself

Do this now, on your own machine and in Packet Tracer. You will run a MAC table, compute OSPF costs, and measure what a VXLAN tunnel adds to a frame. The Python parts run on this machine and their outputs are shown exactly as they ran. The Packet Tracer parts run on your machine, and their outputs are marked as what you should see in Packet Tracer. Your device names and MAC addresses will differ. The shapes will not.

Step 0: run the MAC table on your own machine

Save this file as mac_table.py. It simulates one access switch. It learns MAC addresses by watching frames arrive, forwards known unicast out of one port, and floods unknown unicast out of every port except the one it came in on.

"""A switch MAC table, simulated. Standard library only.

A switch learns by watching. When a frame arrives on a port, the switch
writes down the sender's MAC address next to that port number. When a
frame leaves, the switch looks up the destination MAC in the table.
Known destination: forward out of one port. Unknown destination: flood
out of every port except the one it came in on. That flood is the cost
of not knowing.
"""


class Switch:
    def __init__(self, name):
        self.name = name
        self.table = {}  # mac address -> port number

    def frame(self, src, dst, in_port):
        self.table[src] = in_port
        if dst in self.table:
            out = self.table[dst]
            return "forward out of port %d (known unicast)" % out
        return "flood out of every port except %d (unknown unicast)" % in_port

    def show_table(self):
        print("MAC table on %s:" % self.name)
        for mac in sorted(self.table):
            print("  %s  ->  port %d" % (mac, self.table[mac]))


sw = Switch("access-1")
LAPTOP = "aa:bb:cc:00:00:01"
SERVER = "aa:bb:cc:00:00:02"
PRINTER = "aa:bb:cc:00:00:03"

print("1. Laptop on port 1 sends to the server. The table is empty.")
print("   " + sw.frame(LAPTOP, SERVER, 1))
print("2. The server on port 8 answers the laptop. The table learns twice.")
print("   " + sw.frame(SERVER, LAPTOP, 8))
print("3. The laptop sends to the server again. This time the switch knows.")
print("   " + sw.frame(LAPTOP, SERVER, 1))
print("4. The printer on port 5 sends to the laptop.")
print("   " + sw.frame(PRINTER, LAPTOP, 5))
print()
sw.show_table()
print()
print("A table with 3 entries. A real access switch holds thousands.")
print("The table ages out entries it has not seen in 300 seconds.")
python3 mac_table.py

Expected output, exactly as it ran on this machine:

1. Laptop on port 1 sends to the server. The table is empty.
   flood out of every port except 1 (unknown unicast)
2. The server on port 8 answers the laptop. The table learns twice.
   forward out of port 1 (known unicast)
3. The laptop sends to the server again. This time the switch knows.
   forward out of port 8 (known unicast)
4. The printer on port 5 sends to the laptop.
   forward out of port 1 (known unicast)

MAC table on access-1:
  aa:bb:cc:00:00:01  ->  port 1
  aa:bb:cc:00:00:02  ->  port 8
  aa:bb:cc:00:00:03  ->  port 5

A table with 3 entries. A real access switch holds thousands.
The table ages out entries it has not seen in 300 seconds.

Now break it on purpose. Send a fifth frame to a MAC the table has never seen. It floods. That flood is the cost of not knowing, and it is the same flood that fills the trunk in the loop scenario.

Step 1: compute the OSPF costs

Save this file as ospf_cost.py. It builds a five link topology, computes the OSPF cost of each link (100 divided by megabits per second, minimum 1), and runs Dijkstra to find the cheapest path from the access switch to the data center. Then it kills the best link and runs the math again.

"""OSPF cost and the shortest path, simulated. Standard library only.

OSPF gives every link a cost. Cost is 100 divided by the bandwidth in
megabits per second, with a minimum of 1. A 100 Mbps link costs 1. A
10 Mbps link costs 10. A 1 Gbps link costs 1 too, because the formula
floors at 1. Each router runs Dijkstra over the costs and picks the
cheapest path. Change a cost and the path changes. That is the whole
idea of dynamic routing: the table follows the costs, and the costs
follow the links that are actually up.
"""

import heapq

LINKS = [
    ("access-1", "core-1", 1000),   # 1 Gbps
    ("access-1", "core-2", 100),    # 100 Mbps
    ("core-1", "dc-1", 1000),       # 1 Gbps
    ("core-2", "dc-1", 10),         # 10 Mbps
    ("core-1", "core-2", 100),      # 100 Mbps cross-link
]


def ospf_cost(mbps):
    return max(1, 100 // mbps)


def shortest_path(links, start, end):
    graph = {}
    for a, b, mbps in links:
        c = ospf_cost(mbps)
        graph.setdefault(a, []).append((b, c))
        graph.setdefault(b, []).append((a, c))
    dist = {start: 0}
    prev = {}
    heap = [(0, start)]
    while heap:
        d, node = heapq.heappop(heap)
        if d != dist[node]:
            continue
        if node == end:
            break
        for nxt, c in graph[node]:
            nd = d + c
            if nd < dist.get(nxt, float("inf")):
                dist[nxt] = nd
                prev[nxt] = node
                heapq.heappush(heap, (nd, nxt))
    path = [end]
    while path[-1] != start:
        path.append(prev[path[-1]])
    path.reverse()
    return path, dist[end]


print("Link costs (OSPF cost = 100 / Mbps, minimum 1):")
for a, b, mbps in LINKS:
    print("  %s -- %s : %d Mbps -> cost %d" % (a, b, mbps, ospf_cost(mbps)))
print()
path, cost = shortest_path(LINKS, "access-1", "dc-1")
print("Cheapest path access-1 -> dc-1: %s, total cost %d"
      % (" -> ".join(path), cost))
print()
print("Now the 1 Gbps link access-1 -- core-1 goes down. Remove it.")
down = [l for l in LINKS if l != ("access-1", "core-1", 1000)]
path, cost = shortest_path(down, "access-1", "dc-1")
print("Cheapest path access-1 -> dc-1: %s, total cost %d"
      % (" -> ".join(path), cost))
print()
print("A static route would have pointed at the dead link and stayed")
print("there. OSPF re-ran the math and moved the traffic by itself.")
python3 ospf_cost.py

Expected output, exactly as it ran on this machine:

Link costs (OSPF cost = 100 / Mbps, minimum 1):
  access-1 -- core-1 : 1000 Mbps -> cost 1
  access-1 -- core-2 : 100 Mbps -> cost 1
  core-1 -- dc-1 : 1000 Mbps -> cost 1
  core-2 -- dc-1 : 10 Mbps -> cost 10
  core-1 -- core-2 : 100 Mbps -> cost 1

Cheapest path access-1 -> dc-1: access-1 -> core-1 -> dc-1, total cost 2

Now the 1 Gbps link access-1 -- core-1 goes down. Remove it.
Cheapest path access-1 -> dc-1: access-1 -> core-2 -> core-1 -> dc-1, total cost 3

A static route would have pointed at the dead link and stayed
there. OSPF re-ran the math and moved the traffic by itself.

Now change one cost. Set the 1 Gbps link access-1 to core-1 to cost 10 and run it again. The path moves. That is dynamic routing doing its job. A static route would have pointed at the dead link and stayed there.

Step 2: build the Packet Tracer lab

This step runs on your machine, in Packet Tracer or GNS3. Packet Tracer is free from the Cisco Networking Academy Resource Hub at netacad.com/resources/lab-downloads (free account required). GNS3 is the alternative, free from gns3.com/software/download. Both links were checked on November 21, 2026. GNS3 needs real device images for the full experience. Packet Tracer runs the lab as drawn, with no images to find.

Build this topology: two switches, one router, two laptops, one server. Laptop A in VLAN 10 (10.20.1.10/24, gateway 10.20.1.1). Laptop B in VLAN 20 (10.20.2.10/24, gateway 10.20.2.1). Server in VLAN 20 (10.20.2.20/24, gateway 10.20.2.1). Connect both switches with two links, so spanning tree has something to block.

Switch config. Paste this into each switch, changing the names to match your lab:

enable
configure terminal
vlan 10
 name users
vlan 20
 name servers
exit
interface fastethernet0/1
 switchport mode access
 switchport access vlan 10
exit
interface fastethernet0/2
 switchport mode access
 switchport access vlan 20
exit
interface gigabitethernet0/1
 switchport mode trunk
 switchport trunk allowed vlan 10,20
exit
spanning-tree mode rapid-pvst
spanning-tree vlan 10,20 priority 4096
end

Router config. This is router-on-a-stick: one physical interface, one subinterface per VLAN:

enable
configure terminal
interface gigabitethernet0/0
 no shutdown
interface gigabitethernet0/0.10
 encapsulation dot1q 10
 ip address 10.20.1.1 255.255.255.0
interface gigabitethernet0/0.20
 encapsulation dot1q 20
 ip address 10.20.2.1 255.255.255.0
router ospf 1
 network 10.20.1.0 0.0.0.255 area 0
 network 10.20.2.0 0.0.0.255 area 0
end

Then run the tests, in order. Ping laptop A to laptop B before the router is up. It must fail: different VLANs, no route yet. Ping laptop A to its gateway, 10.20.1.1. It must pass. Ping laptop A to the server, 10.20.2.20. It must pass after the router is up. Then unplug one uplink between the switches and watch the blocked port unblock. Then plug in a spare cable between two ports on the same switch with spanning tree off, watch the floor crawl, and turn spanning tree on.

What you should see in Packet Tracer when the lab is built right:

show vlan brief
VLAN 10 users, ports Fa0/1
VLAN 20 servers, ports Fa0/2

show interfaces trunk
Gi0/1 trunking, VLANs allowed 10,20

show spanning-tree vlan 10
One uplink forwarding, one uplink blocking

show ip route
10.20.1.0/24 connected, 10.20.2.0/24 connected
OSPF routes marked with O

Those are the outputs to look for in your Packet Tracer session. Your port numbers and device names will differ. The shapes will not.

Step 3: measure the frame

Save this file as frame_size.py. It measures what a VLAN tag and a VXLAN tunnel add to a frame, and checks the result against a 1500 MTU.

"""Frame sizes: what a VLAN tag and a VXLAN tunnel add. Standard library only.

A plain Ethernet frame carrying a full payload is 1500 bytes. An
802.1Q VLAN tag adds 4 bytes. A VXLAN tunnel adds 50 bytes: 14 bytes
of outer Ethernet, 20 bytes of outer IP, 8 bytes of UDP, 8 bytes of
VXLAN header. The frame that leaves the tunnel is bigger than the
frame that entered it. If the path MTU is 1500, the big frame drops,
silently. Size the MTU for the overhead, or clamp the TCP MSS so the
payload shrinks before the tunnel grows it.
"""

VLAN_TAG = 4
VXLAN_OVERHEAD = 14 + 20 + 8 + 8  # outer Ethernet + IP + UDP + VXLAN


def sizes(payload):
    plain = payload
    tagged = payload + VLAN_TAG
    tunneled = payload + VLAN_TAG + VXLAN_OVERHEAD
    return plain, tagged, tunneled


for payload in (64, 1500):
    plain, tagged, tunneled = sizes(payload)
    print("Payload %d bytes:" % payload)
    print("  plain frame:   %d bytes" % plain)
    print("  VLAN tagged:   %d bytes (+%d for the 802.1Q tag)"
          % (tagged, VLAN_TAG))
    print("  VXLAN tunnel:  %d bytes (+%d tag, +%d tunnel)"
          % (tunneled, VLAN_TAG, VXLAN_OVERHEAD))
print()
print("VXLAN overhead breakdown: 14 outer Ethernet + 20 outer IP")
print("  + 8 UDP + 8 VXLAN = %d bytes" % VXLAN_OVERHEAD)
print()
MTU = 1500
_, _, big = sizes(1500)
if big > MTU:
    print("A 1500-byte payload inside VXLAN is %d bytes. That is %d bytes"
          % (big, big - MTU))
    print("over a 1500 MTU. It drops unless the path MTU is %d or more,"
          % big)
    print("or the TCP MSS is clamped to %d." % (1500 - VXLAN_OVERHEAD))
else:
    print("Fits inside a %d MTU." % MTU)
python3 frame_size.py

Expected output, exactly as it ran on this machine:

Payload 64 bytes:
  plain frame:   64 bytes
  VLAN tagged:   68 bytes (+4 for the 802.1Q tag)
  VXLAN tunnel:  118 bytes (+4 tag, +50 tunnel)
Payload 1500 bytes:
  plain frame:   1500 bytes
  VLAN tagged:   1504 bytes (+4 for the 802.1Q tag)
  VXLAN tunnel:  1554 bytes (+4 tag, +50 tunnel)

VXLAN overhead breakdown: 14 outer Ethernet + 20 outer IP
  + 8 UDP + 8 VXLAN = 50 bytes

A 1500-byte payload inside VXLAN is 1554 bytes. That is 54 bytes
over a 1500 MTU. It drops unless the path MTU is 1554 or more,
or the TCP MSS is clamped to 1450.

Now set the MTU to 1500 on the tunnel path in your Packet Tracer lab and ping with a full size frame. Watch it fail. Raise the MTU to 1554 and watch it pass. That is the whole VXLAN lesson in one ping.

Check yourself

In Packet Tracer, move laptop A to a port in VLAN 20 without changing its address. Ping the server. It fails. Move it back. The address did not change. The VLAN did. That is the lesson in one move.

Clean up

Close Packet Tracer without saving, or save the file for the next lesson. The Python scripts live on your machine and run again any time. Nothing in this lab touched a live network.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Switching and routing is not one skill. It is four levels of the same job, and every network engineer I have worked with sits on one of them. The levels are not about how many protocols you can name. They are about whether you can find the broken table when the traffic stops. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · Read a MAC table

At the basic level, you read the one table that decides everything on a switch. You run mac_table.py and watch the table learn, forward, and flood. You learn the three moves: known unicast forwards out of one port, unknown unicast floods, broadcast floods. You can point at a MAC address and say which port it lives on. The failure that ends this level is the first flood you cannot explain. A table you cannot read is a network you cannot debug.

Types:

  • MAC address table (CAM table): which port each MAC address was last seen on.
  • Known unicast: forwarded out of exactly one port.
  • Unknown unicast and broadcast: flooded out of every port in the VLAN except the one it came in on.
  • Collision domain per port, broadcast domain per VLAN.

Uses: The first check when one host cannot reach another on the same network: is its MAC address in the table, and on the port you expect?

Scale and failure: MAC tables have a size limit. When they fill, from a very large flat network or a MAC-flooding attack, the switch floods unknown traffic out of every port, which looks like a slow network and leaks traffic to everyone on the segment.

Trade-off: A flat layer 2 network is simple and every host can reach every other, and one broadcast storm or loop takes all of it down at once.

Do this: run step 0 of the lab. Send the four frames in mac_table.py and write down what each one does. Then send a fifth frame to a MAC the table has never seen. Write down what happens. Flood is the answer, and flood is the cost of not knowing.

Try it: Basic

What does a switch do with a frame to a MAC it has never seen?

Now do it: I run step 0 of the lab on my lab machine: I send the four frames in mac_table.py, I write down what each one does, then I send a fifth frame to a MAC the table has never seen. The lab is not on this machine, so I run this on my lab machine.

Intermediate · VLANs, a trunk, and spanning tree

At the intermediate level, you build the lab and you break it on purpose. Two VLANs, one trunk, two uplinks, one blocked port. You move a host to the wrong VLAN and watch it lose the network. You unplug the forwarding uplink and watch the blocked port unblock. You add the spare cable and watch the storm, then you turn spanning tree on and watch the storm end. The failure that ends this level is the first loop you cause yourself. Causing it on purpose is how you learn to spot it for real.

Types:

  • VLANs (802.1Q): several isolated layer 2 networks on one set of switches.
  • Trunk ports carrying tagged traffic for many VLANs; access ports carrying one untagged VLAN.
  • Spanning Tree (STP, RSTP): blocks redundant links so there is no loop, and unblocks one when a link fails.
  • Inter-VLAN routing on a router or layer 3 switch.

Uses: Separating departments, environments, or device types on shared hardware, with redundant uplinks that do not create loops.

Scale and failure: Classic STP takes 30 to 50 seconds to converge after a link fails, which is an outage for anything waiting; RSTP brings that to seconds. A root bridge left to elect itself can end up on the weakest switch, forcing traffic through a desk switch.

Trade-off: More VLANs give better isolation and more trunk configuration to keep consistent. One mismatched native VLAN on a trunk can quietly leak traffic between networks.

Do this: build step 2 of the lab. Ping across VLANs before the router is up and write down the failure. Then bring the router up and ping again. Then plug in the spare cable with spanning tree off, watch the floor crawl, and turn spanning tree on. Three failures, three fixes, one lab.

Try it: Intermediate

What happens when I plug in the spare cable with spanning tree off?

Now do it: I build step 2 of the lab on my lab machine: I ping across VLANs before the router is up, I bring the router up and ping again, then I plug in the spare cable with spanning tree off. The lab is not on this machine, so I run this on my lab machine.

Advanced · OSPF, static routes, and the tunnel

At the advanced level, you stop typing routes by hand and let the routers talk. You run ospf_cost.py and watch the path move when a link dies. You learn the cost formula, 100 divided by megabits, minimum 1, and you learn why two 1 Gbps links cost the same even when one is twice as fast. You run frame_size.py and learn that the tunnel adds 50 bytes and the MTU must fit it. The failure that ends this level is the first static route that outlives the link it points at. A route that stays after its next hop dies is a route that blackholes traffic.

Types:

  • Static routes: simple, and they never notice that a next hop has died.
  • OSPF: a link-state protocol that computes shortest paths from interface costs.
  • OSPF cost: reference bandwidth divided by link bandwidth, minimum 1.
  • MTU and tunnel overhead: GRE adds 24 bytes, VXLAN about 50, IPsec more.

Uses: Networks with more than a couple of routers, where paths must move on their own when links fail.

Scale and failure: With the default reference bandwidth of 100 Mbps, every link of 100 Mbps or faster costs 1, so OSPF cannot tell a 1 Gbps link from a 100 Gbps one; set the reference bandwidth higher across all routers. MTU mismatches cause the classic "small pings work, large transfers hang" failure.

Trade-off: Dynamic routing heals itself and is harder to reason about and to secure; static routing is predictable and blackholes traffic silently when a link dies.

Do this: run steps 1 and 3 of the lab. Change one link cost in ospf_cost.py and write down the new path. Then set the MTU to 1500 on the tunnel path and ping with a full size frame. Watch it fail. Raise the MTU to 1554 and watch it pass.

Try it: Advanced

What is the OSPF cost of a 1 Gbps link?

Now do it: I ran python3 -c "print(max(1, 100 // 1000))" and it printed 1. I change one link cost in ospf_cost.py on my lab machine and I write down the new path.

Master · Design the floor

At the master level, you design the network before anyone plugs in a cable. You place the layer 2 and layer 3 boundary on purpose, not by accident. You put the spanning tree root on the core switch, not on the switch under someone's desk. You choose OSPF areas and you name the BGP edge. You size the broadcast domains so a loop hurts one floor, not the building. You write the MTU down in the design doc, not in a ticket after the first big frame drops. The failure that ends this level is the first design review where someone asks where the root bridge is and you do not know. Know before you build.

Types:

  • Hierarchical design: access, distribution, and core layers.
  • Layer 2 and layer 3 boundaries placed on purpose, usually at the distribution layer.
  • OSPF areas to contain the impact of link changes, and BGP at the edge.
  • Design documents that state the root bridge, MTU, addressing plan, and failure domains.

Uses: Campus and data-centre networks that must grow without redesign and fail without taking the building down.

Scale and failure: Large layer 2 domains turn every loop or broadcast storm into a site-wide outage; routing at the access or distribution layer keeps a failure to one floor or rack. Leaf-spine fabrics with ECMP replace spanning tree entirely in modern data centres.

Trade-off: Routing everywhere shrinks failure domains and makes moving a device between subnets harder. The design decides where mobility matters more than isolation.

Do this: do the design exercise below. Then draw the spanning tree for your design on paper, root bridge and all, and mark every blocked port. If you cannot mark the blocked ports, the design is not finished.

Try it: Master

Where do I put the spanning tree root?

Now do it: I do the design exercise below on my own machine, then I draw the spanning tree for my design on paper, root bridge and all, and I mark every blocked port.

Go deeper

Static routes are a promise you keep by hand

A static route is a line you typed. It stays until you remove it. That is its strength and its failure. It never flaps, it never reconverges, and it never notices the link died. OSPF notices. It hears the hello packets stop, it ages the neighbor out, it reruns the math, and the traffic moves. The price is chatter. OSPF routers talk all the time, and the table they build is only as good as the costs you give them. The rule I use: static routes for the edges that never move, the default route out of a small site, the one link that has no alternative. OSPF for everything with two ways to get there. RIP is the old answer to the same question. It counts hops, not bandwidth, and it caps at 15. You will meet it in old networks. You will not build a new network on it. EIGRP is Cisco's answer. It converges fast and it only runs on Cisco gear in practice. Know the name, know the trade, and build on OSPF unless you have a reason not to.

Spanning tree, RSTP, and why PortFast exists

Classic spanning tree takes 30 to 50 seconds to unblock a port. It listens, it learns, then it forwards. That is a long time for a laptop to wait. Rapid spanning tree, RSTP, cuts it to a few seconds by letting switches agree directly instead of waiting out timers. PortFast is the shortcut for a user port: it skips the wait and forwards at once, because a laptop cannot create a loop by itself. The danger is a switch plugged into a PortFast port. It can loop before spanning tree notices. BPDU guard is the fix. It shuts the port down the moment a spanning tree message arrives on it. The rule: PortFast on user ports, BPDU guard with it, never PortFast on a trunk, never PortFast on a port another switch can reach.

The 4094 VLAN ceiling, and why VXLAN exists

A VLAN tag is 12 bits. That is 4094 usable VLANs. A small office never notices. A data center with a tenant per VLAN hits the ceiling in a year. VXLAN fixes it with a 24 bit identifier, the VNI. Sixteen million networks, not four thousand. The price is the tunnel: 50 bytes of overhead, a VTEP on each end, and a control plane to map VNIs to VTEPs. EVPN is the control plane most teams use. It runs on BGP, and it carries the MAC addresses the same way BGP carries routes. That is the one line to remember about EVPN: BGP, but for MAC addresses. The VXLAN detail lives here, because the tunnel is the hop that carries the frame to the data center.

BGP at the edge, in one paragraph

BGP is the protocol the internet runs on. It does not care about link speed. It cares about policy: which autonomous system you trust, which path you prefer, which routes you accept. Your network has an AS number. Your provider has one. eBGP runs between them. iBGP runs inside yours. The Facebook outage of October 2021 is the story everyone in networking knows: a BGP change withdrew the routes to Facebook's own DNS servers, and the whole company vanished from the internet, including the tools they needed to fix it. Lesson 63 tells that story. The lesson for this hop is smaller: never withdraw your own routes by accident, and never make the path to the fix depend on the network you just broke.

What changes at 10x and 100x

At 10x the hosts, the broadcast domain is the first thing that breaks. Five hundred hosts in one VLAN is a storm waiting for a loop, and the fix is more VLANs, not a bigger switch. At 10x the routes, the routing table stops being readable by a human, and the fix is route summarization and areas, not a longer table. At 100x, the layer 2 network ends at the rack. Everything above it is layer 3, and VXLAN carries the VLANs that still need to stretch. The spanning tree that protected one floor becomes a liability across a data center, because one blocked link is one link wasted. The design changes with the scale. The tables do not. A MAC table is still a MAC table at 100x. It is just bigger, and it ages out faster.

The lab config, in full

The switch and router configs from the lab, kept here as the reference copy. Paste them into Packet Tracer, changing the names to match your lab.

enable
configure terminal
vlan 10
 name users
vlan 20
 name servers
exit
interface fastethernet0/1
 switchport mode access
 switchport access vlan 10
exit
interface fastethernet0/2
 switchport mode access
 switchport access vlan 20
exit
interface gigabitethernet0/1
 switchport mode trunk
 switchport trunk allowed vlan 10,20
exit
spanning-tree mode rapid-pvst
spanning-tree vlan 10,20 priority 4096
end
enable
configure terminal
interface gigabitethernet0/0
 no shutdown
interface gigabitethernet0/0.10
 encapsulation dot1q 10
 ip address 10.20.1.1 255.255.255.0
interface gigabitethernet0/0.20
 encapsulation dot1q 20
 ip address 10.20.2.1 255.255.255.0
router ospf 1
 network 10.20.1.0 0.0.0.255 area 0
 network 10.20.2.0 0.0.0.255 area 0
end

Further reading

Cisco Packet Tracer, official download. Free Networking Academy account required. GNS3, official download. Free account required. Both links were checked on November 21, 2026. The CIDR and overlap prerequisite lives in F15 and F16. The Facebook BGP incident is told in Lesson 63.

One time it broke

What broke

A floor went quiet on a Tuesday morning. Not down, quiet. Pings answered, but slowly. File transfers crawled. Video calls froze. The helpdesk tickets all said the same thing: the network is slow. The switches showed no errors. The router showed no drops. The firewall showed nothing. The floor was not broken. It was flooded.

Timeline

  • 08:00A move crew rearranges desks on the third floor. One spare patch cable is plugged into two ports on the same access switch, both ends, to keep it tidy.
  • 08:05The first broadcast frame enters the loop. It circles. Every lap, the switch copies it out of every port. The copies circle too.
  • 08:10The trunk to the core carries the storm to every VLAN on it. Four VLANs slow down at once.
  • 08:30The first tickets arrive. The network is slow. The monitoring shows high utilization on the trunk, which looks like a busy morning.
  • 09:15I check the access switch. The MAC table shows the same addresses flapping between ports. That is the tell. A MAC address does not move between ports every few seconds unless the frame is arriving on both.
  • 09:20I find the cable. Both ends in the same switch. I pull one end. The storm stops in seconds. The floor comes back.
  • 09:25I check spanning tree. It is off on that switch. It was never turned on after the switch was replaced the month before.

What changed after

Three things. First, spanning tree on every switch, no exceptions, checked by a script that reads the port states every week. Second, BPDU guard on every user port, so a switch plugged into a user port shuts the port down instead of looping the floor. Third, a rule for the move crew: no cable is plugged in without a ticket naming both ends. The cable was the trigger. The missing spanning tree was the cause. The tickets that said slow instead of down were the reason it took an hour to find.

If you remember nothing else

Design exercise

Design the network for a new office floor and its link to the data center. Two hundred hosts, six VLANs, two uplinks from each access switch to the core, a router for inter-VLAN traffic, and a VXLAN tunnel to the data center. The server VLAN must reach the data center. The user VLANs must not. One uplink will die on a bad day. A loop must never take the floor down. Write down the VLANs and subnets, the spanning tree design, the routing design, the tunnel MTU, and the first three things you check when a host cannot reach the server.

Clarifying questions to ask

  • Which VLANs need to reach the data center, and which must stay on the floor?
  • Where does layer 2 end and layer 3 begin, and why there?
  • Where is the spanning tree root, and which ports are blocked in the normal state?
  • What is the MTU on the tunnel path, and what happens to a full size frame?
  • When one uplink dies, what converges, and how long does it take?

Worked approach

Score the shape first. Six VLANs, 200 hosts. Users split across four VLANs, about 50 hosts each, so no broadcast domain is big enough to hurt. Servers in one VLAN. Printers and voice in the sixth. Subnets from F16: 10.20.10.0/24 through 10.20.15.0/24, no overlaps, checked before building. The router sits at the core, one subinterface per VLAN, and it is the only path between VLANs. The VXLAN tunnel leaves from a VTEP at the core, not from the access switches, so the tunnel count stays small.

Then the loop protection. Spanning tree root on the core switch, priority 4096, so the root never lands on a switch under a desk. RSTP, not classic STP, so convergence takes seconds, not 30. PortFast plus BPDU guard on every user port. No PortFast on any trunk. In the normal state each access switch has one uplink forwarding and one blocking. The blocked ports are marked on the diagram, because a blocked port nobody knows about is a port someone will unblock by accident.

Then the routing. OSPF area 0 across the core and the router, costs from the 100 divided by megabits rule. The data center link is the only static route, a default toward the tunnel, because it never moves. The user VLANs get no route to the data center at all: the router has no route, and the firewall has no rule. A missing route is a stronger wall than a blocked one, because a blocked rule can be edited by mistake.

Then the MTU. VXLAN adds 50 bytes, the VLAN tag adds 4. A 1500 byte payload is 1554 bytes in the tunnel. The path MTU is 1600, set on the core, the VTEPs, and the data center side, and the TCP MSS is clamped to 1450 as the backstop. The big ping is tested before opening day, not after the first file transfer fails.

Trade-offs to name. Layer 2 against layer 3 is the big one. More VLANs mean smaller broadcast domains and more router hops. Fewer VLANs mean fewer hops and bigger storms. Six VLANs is the middle. OSPF against static is the second. OSPF converges by itself and chatters all day. Static is silent and stays after its next hop dies. Cost against redundancy is the third. Two uplinks cost two ports and one of them is blocked in the normal state. That blocked port is not waste. It is the spare that saves the floor on a bad day.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series · Lesson 65
Wireless Networking: The Air Between the Laptop and the Switch

The frame leaves the cable behind. Same switching, same routing, new problems: air as the medium, channels that overlap, and clients that roam between access points while you are still debugging the last one.

Go to Lesson 65 →

Or browse the full course from the Lessons index.