BGP, from peering to policy
This is the skill the hiring manager named first. Not "BGP in AWS" — BGP you configure, filter, and rescue on-prem. Sessions, attributes, path selection, and the hardening moves that keep a DC network from being your pager's favorite topic.
The table stakes
What BGP is for
The only routing protocol that scales to the internet — and the default choice for datacenter fabrics and any network where policy matters more than shortest path. Path-vector, TCP port 179, never fast-converging by design: stability over speed.
eBGP vs iBGP
eBGP: different ASNs, directly connected (TTL 1) by default, strips local routes and prepends its AS to the AS_PATH. iBGP: same ASN, full mesh required (no AS_PATH change → no loop prevention), TTL 255, split-horizon rule: routes learned via iBGP are never re-advertised to iBGP peers.
Session state machine (know it cold)
| State | What it means | Stuck here usually means |
|---|---|---|
| Idle | Not trying yet | Config error, route to peer missing, admin down |
| Connect | TCP SYN in flight | ACL/firewall blocking 179, no route back |
| Active | TCP failed, retrying | Peer unreachable, peer's config wrong, TCP RST |
| OpenSent | OPEN sent, waiting | ASN mismatch, hold-time mismatch, MD5 password mismatch |
| OpenConfirm | OPEN agreed, waiting KEEPALIVE | Rarely stuck; watchdog for MTU/blocked packets |
| Established | Exchange UPDATEs | That's the goal — now watch prefix counts |
The attributes that decide everything
| Attribute | Class | What it does |
|---|---|---|
| ORIGIN | Well-known mandatory | IGP / EGP / incomplete — lower wins, rarely decisive in practice |
| AS_PATH | Well-known mandatory | List of ASNs traversed; loop prevention + shortest-path heuristic |
| NEXT_HOP | Well-known mandatory | Must be reachable via IGP before a route is usable — classic on-prem gotcha |
| LOCAL_PREF | Well-known discretionary | Inbound preference, stays inside the AS. Higher wins. The main traffic-engineering knob |
| MED | Optional non-transitive | Hint to a neighboring AS about your preferred entry point. Lower wins, compared only between same-AS paths |
| Communities | Optional transitive | Tags for policy: RTBH (blackhole), no-export, prepends, customer/peer/transit classes |
Best-path selection order (say it in your sleep)
Highest weight → highest local preference → locally originated → shortest AS path → lowest origin → lowest MED → eBGP over iBGP → lowest IGP metric to NEXT_HOP → oldest eBGP route → lowest router ID → lowest neighbor address.
Don't rattle off the Cisco order when the question is "a route flaps between two links every few minutes — walk me through why." That's a stability question: check max-prefix, dampening, MTU, BFD timers, and whether both peers are iBGP with no IGP metric difference. Answer the question asked, then show the algorithm.
FRR in the datacenter
MetalLB runs FRR under the hood — so FRR syntax is the right dialect to have at your fingertips for this interview. These are the patterns to be able to reproduce from memory.
/etc/frr/bgpd.conf — hardened eBGP session to a ToR switchrouter bgp 65001
bgp router-id 10.10.0.11
no bgp ebgp-requires-policy # FRR enforces inbound policy by default
bgp bestpath as-path multipath-relax
neighbor 10.10.0.1 remote-as 65000
neighbor 10.10.0.1 description TOR-1
neighbor 10.10.0.1 password s3cret-md5-pass # MD5 (RFC 2385)
neighbor 10.10.0.1 ttl-security hops 1 # GTSM (RFC 5082)
neighbor 10.10.0.1 timers 3 9
neighbor 10.10.0.1 maximum-prefix 5000 restart 60
neighbor 10.10.0.1 bfd # BFD for sub-second failover
!
address-family ipv4 unicast
network 10.20.0.0/16
neighbor 10.10.0.1 route-map TOR-IN in
neighbor 10.10.0.1 route-map TOR-OUT out
exit-address-family
!
ip prefix-list OURS seq 5 permit 10.20.0.0/16 le 24
ip prefix-list DEFAULT seq 5 deny 0.0.0.0/0
!
route-map TOR-IN deny 10
match ip address prefix-list DEFAULT
!
route-map TOR-OUT permit 10
match ip address prefix-list OURS
set community 65001:100
!
route-map TOR-OUT deny 20show bgp summary # peers, state, Up/Down, prefixes
show bgp ipv4 unicast neighbors 10.10.0.1 advertised-routes
show bgp ipv4 unicast neighbors 10.10.0.1 received-routes
show bgp ipv4 unicast 10.20.0.0/16 # how THIS router sees the prefix
vtysh -c 'show ip route 10.20.1.5' # did it make it into the RIB/FIB?Secure the session, then the routes
"Network hardening" in this JD is BGP-first. Four layers:
1 · Session security
- GTSM / TTL security — reject packets with wrong TTL; kills spoofed third-party sessions
- MD5 (RFC 2385) or TCP-AO (RFC 5925) — authenticate every TCP segment; AO replaces MD5 for modern boxes
- Loopback peering + multihop — session survives link flapping, not tied to one interface
- BFD (RFC 5880) — 300ms failover instead of 3× hold-timer
2 · Route security
- Inbound filters — only accept what the peer should announce; default-deny
- maximum-prefix — session tears down instead of melting your RIB (route leak insurance)
- RPKI / ROV — drop invalid-origin routes at the edge
- Outbound filters — only advertise your own blocks; never leak a full table
3 · Policy via communities
- Tag at the edge, act in the core: prepend, localpref, blackhole classes
- RTBH (RFC 5635) — drop a DDoS target at the edge with one community tag
- no-export / no-advertise to bound where routes travel
4 · Stability
- Route dampening for flap-prone peers
- Logging + netflow to catch announcement anomalies
- Change control: diff the routing table before/after every maintenance
In AWS/Azure, the provider owns the underlay: you get a BGP session into a virtual gateway and the provider handles ECMP, failover, and much of the policy. On-prem, you own MTU, next-hop reachability, switch ACLs, physical diversity, and every session timer. Say that difference explicitly — it's the distinction the hiring manager drew, and agreeing with it on his terms signals you actually run this stuff.
Where BGP fits in a datacenter fabric
Modern DC = leaf-spine. Every leaf and spine runs BGP (often BGP unnumbered — no per-link IPv4, uses IPv6 link-locals + RFC 5549 to carry v4). Spines are pure L3, no STP, ECMP across all parallel paths, and the fabric converges without any L2 topology to babysit.
Leaf-spine + ECMP
East-west traffic: 2–4 equal-cost paths everywhere. Bandwidth scales by adding spines — that's the whole point.
EVPN/VXLAN on top
When L2 adjacency must span racks, VXLAN tunnels carry it over the BGP underlay. Worth naming — it shows you know how far this rabbit hole goes.
Anycast + MetalLB
MetalLB BGP mode peers with the leaves and announces Service VIPs from every node — the switches see identical prefixes from N next-hops and ECMP-load-balance. That's the exact pattern this company runs. Details on the MetalLB page.
The five incidents he'll ask about
| Symptom | First three moves |
|---|---|
| Session stuck in Active/Connect | 1. Can I reach the peer IP (ping to port 179 — `nc -vz 10.10.0.1 179`)? 2. ACLs/firewall on both sides? 3. Peer's config: right ASN, right source IP? |
| Established but no routes | 1. Outbound filter blocking (route-map/prefix-list)? 2. Peer's inbound filter? 3. NEXT_HOP reachable in my IGP? |
| Flapping session | 1. Interface errors/MTU mismatch on the path? 2. max-prefix hit? 3. Hold timer too tight vs slow peer? Add BFD carefully. |
| Asymmetric paths | 1. LOCAL_PREF inconsistent across route reflectors? 2. MED from the other AS? 3. Stale route on one node — compare RIBs. |
| Traffic blackholes after maintenance | 1. Did a route-map change silently deny a prefix? 2. Route still in BGP but missing in FIB? 3. Roll back, diff tables, then investigate. |
Questions to answer out loud
Walk me through BGP best-path selection.
Weight → local preference → locally originated → shortest AS path → lowest origin → lowest MED → eBGP over iBGP → lowest IGP metric → oldest eBGP route → lowest router ID → lowest neighbor address. Then explain the two knobs you actually use day to day: local pref for outbound policy, AS prepends + MED for inbound influence.
Your peer announces 40,000 prefixes and your box is dying. What should already have been in place, and what do you do now?
Should have been: maximum-prefix with restart, inbound prefix filters, and a default-deny inbound policy. Now: confirm what's arriving (received-routes count), apply the filter + prefix limit, restore the session, then trace the leak upstream. If it's an internet edge, check RPKI status on the flood.
eBGP vs iBGP — why do the rules differ?
Loop prevention. eBGP uses AS_PATH (a router drops routes containing its own ASN). iBGP peers don't append to AS_PATH, so BGP needs the full-mesh rule + split horizon instead. That's also exactly why route reflectors exist: fix the iBGP full-mesh N² scaling problem while keeping the loop rule intact.
How does MetalLB actually use BGP?
The speaker pods run FRR, peer with your ToR/spine switches as neighbors, and announce the Service VIP's /32 from every node running the service. The switches ECMP across next-hops; kube-proxy (or OVN) then does the pod-level distribution. Failure = withdraw, not re-route. See the MetalLB page.