OpenShift Networking: OVN-Kubernetes + MetalLB BGP (the exact stack they run)

1. OpenShift architecture — 60-second version

  • Control plane: 3 masters (or compact 3-node for small sites) running kube-apiserver, controllers, scheduler, etcd (quorum — 3 minimum), plus OCP-specific operators.
  • Workers: RHCOS (immutable Red Hat CoreOS) or RHEL; configured via Ignition at install, then MachineConfig Operator (MCO). You don't hand-edit node config — you write MachineConfigs; nodes drain/reboot via MachineConfigPool. This is the #1 thing AWS-K8s admins don't know and on-prem admins must.
  • CRI-O runtime (not containerd), Podman (not Docker).
  • Operators everywhere (Cluster Version Operator drives upgrades via update channels/EUS; OLM manages app operators: subscription → install plan → CSV).
  • OpenShift specifics: Route objects (vs Ingress), SecurityContextConstraints (SCCs — restricted-v2 default), oc CLI, image streams, Routes backed by HAProxy router pods.
  • Upgrade discipline: etcd backup → upgrade control plane → MCP updates workers (serial, respecting PDBs/maxUnavailable). EUS (extended update support) gives 2-year support per even minor.

2. The CNI: OVN-Kubernetes (default since 4.12; OpenShift SDN deprecated 4.14)

  • OVN (Open Virtual Network) = SDN controller + OVS dataplane. OVN-Kubernetes wires it to K8s.
  • Overlay: Geneve tunnels between nodes. Each node runs ovnkube + OVS; br-ex bridges to the physical NIC; pod traffic is encapsulated except where it exits.
  • Distributed everything: logical switches per node, logical router (distributed gateway) — east-west pod traffic routes locally, no tromboning.
  • NetworkPolicies compile to OVN ACLs (stateful, on the flow tables — efficient). OpenShift adds AdminNetworkPolicy/BaselineAdminNetworkPolicy and EgressFirewall (stateful egress ACL per namespace).
  • Egress: pods SNAT through node by default. EgressIP pins a namespace's outbound traffic to specific node IPs (useful for firewall whitelisting of on-prem app traffic — very likely relevant for their Windows-app decommissioning). EgressRouter = dedicated pod doing NAT/redirect.
  • Services: OVN-K does distributed load balancing (ovn-k service LB), no kube-proxy in newer releases.
  • Key interview point: OVN-K does NOT run BGP. Pod-to-pod is overlay routing. BGP enters ONLY at the edge (MetalLB/FRR peering to physical routers). If the interviewer says "BGP in OVN," gently layer it: "OVN handles the overlay; the BGP piece is MetalLB's FRR speakers announcing VIPs northbound."

3. MetalLB — their load balancer (know this end to end)

Why it exists: on-prem K8s has no cloud provider to hand out LoadBalancer IPs. MetalLB implements the LoadBalancer Service type on bare metal. OCP: MetalLB Operator (GA since 4.10).

Core objects:

  • IPAddressPool: CIDR ranges of VIPs (e.g. 10.96.20.0/24 for services).
  • BGPAdvertisement: which pools to announce, aggregation length, communities, localPref.
  • BGP peers: ToR/spine IPs, ASN, holdtime, password, ebgp-multihop, BFD profiles.
  • FRR-K8s (OCP 4.14+): MetalLB now drives FRR running as a DaemonSet (FRRConfiguration CRDs). FRR = the same BGP stack as router vendors — you can vtysh into it on a node. Mention FRR by name; hiring managers who run this stack love it.

BGP mode — mechanics:

  1. Create LoadBalancer service → MetalLB controller allocates a VIP from the pool.
  2. Speaker (DaemonSet, one per node) announces the VIP as a /32 (per BGPAdvertisement) to configured peers — but only from nodes where the service has ready endpoints.
  3. ToR routers receive the same /32 via multiple nodes = ECMP next hops.
  4. Traffic from clients hashes across nodes (5-tuple) → hits a node → DNAT to pod (externalTrafficPolicy default = Cluster, SNAT'd).
  5. Node dies → its session drops → route withdrawn → ECMP set shrinks. With BFD, sub-second.
  6. Node drain: MetalLB signals speakers to withdraw (graceful) so no traffic lands on a drained node.

Why BGP mode over L2 mode (this question is coming):

  • L2 mode: one elected leader node answers ARP/NDP for the VIP. Simple, works on ANY switch (no BGP needed), but ALL traffic to a VIP flows through ONE node (bottleneck, hairpin for pod-on-other-node), and failover = leader election (memberlist, seconds).
  • BGP mode: true load sharing across nodes via ECMP, no single node bottleneck, standard fabric integration, graceful withdrawals. Cost: requires BGP-capable switches and competent BGP ops — which is literally the job.
  • Rule of thumb to say: "L2 mode for labs/small sites; BGP mode for production DC with proper fabrics."

externalTrafficPolicy (guaranteed question):

  • Cluster (default): SNAT at the node → pod sees node IP; works across all nodes with ECMP; loses client source IP (needed for X-Forwarded-For anyway).
  • Local: no SNAT, preserves source IP; only nodes with a ready pod advertise the VIP (else blackholed path), and traffic arriving at a node without a pod gets dropped (strict same-node return path). Use Local for apps that need real client IPs (or use proxy protocol).
  • Tie to F5: same tradeoff as SNAT automap vs transparency on BIG-IP — you've lived this.

4. Where BGP sits in the full path (draw this in your head, say it in 30s)

Client → DC edge/firewall → ToR (BGP-learned /32 VIP, ECMP) → worker node → (L4: OVN-K service DNAT) → OpenShift Router pod (HAProxy, L7, TLS term + host/path routing) → ClusterIP service → pod (Geneve overlay to the pod's node).

Two LBs, two layers: MetalLB = L4 VIP entry into the cluster; Router = L7 HTTP routing inside. Same mental model as F5: virtual server (L4 VIP) + pool with HTTP monitors + iRules (L7 logic).

5. OpenShift networking ops (day-2 questions)

  • Internal DNS: CoreDNS in-cluster; on-prem usually adds a DNS record for the VIP (or the API/cluster wildcard), often *.apps.<cluster> → router VIP.
  • Multus + NetworkAttachmentDefinition: attach pods/VMs to host bridges (VLAN trunking on the node NIC) — this is how migrated Windows VMs get real network legs (see 04).
  • SR-IOV for performance-critical pods/VMs.
  • NetworkPolicy micro-segmentation: default-deny in sensitive namespaces, allow by label selectors. OpenShift's "cluster network policy" for global rules.
  • Health checks: HAProxy router monitors via readiness endpoints; MetalLB doesn't health-check apps itself (K8s endpoints do) — BGP announces happen only when endpoints are ready.
  • Debugging toolkit: oc get endpoints, oc get svc, oc debug node/, ovn-trace (trace a packet through the logical topology!), ovnkube-trace, vtysh on FRR pods, tcpdump in pod (oc debug with privileged container).

6. Day-2 OCP admin (they'll probe breadth)

  • Patch RHEL nodes: dnf on RHEL workers / RHCOS via MCO; cluster upgrades via ClusterVersionOperator + EUS channels; operators via OLM.
  • Storage: PVCs on StorageClasses — NFS provisioner (fine for apps, never for etcd), iSCSI (LSO - Local Storage Operator, multipath), LVM Storage (OCP 4.14+, host LVM volumes) — matches the posting's LVM/NFS/iSCSI line.
  • RBAC: namespaces, roles/rolebindings, SCCs, service accounts; quotas + LimitRanges.
  • Monitoring: Prometheus/Grafana/Alertmanager built in; etcd and node alerts matter; audit logs via ClusterLogForwarder.
  • HA: 3 control planes, workers across failure domains (racks), PDBs for critical pods, anti-affinity.
  • App deployment: OLM/Helm; GitOps via Argo CD (OpenShift GitOps) — "a plus" in the posting, and you can talk Ansible honestly (your strength).

7. Quick quiz to self-test (answer out loud)

  • Where exactly does BGP terminate on an OCP node, and what does it announce?

(FRR speaker DaemonSet pod → eBGP session to ToR → announces service VIPs as /32s.)

  • A pod on node B is unreachable via the VIP but node A works. Where do you look?

(ECMP path → node B → endpoint readiness; check the service's endpoints, policy Cluster/Local, node's BGP session state — maybe the speaker on B isn't announcing or ToR still has stale route.)

  • How does traffic reach a pod on a different node after hitting node A's VIP? (SNAT + OVN-K service DNAT → Geneve tunnel to pod's node.)