One headless Service with zero Pods stopped every nginx router from starting (Monzo 2017), and why a normal Service wouldn't have
A detail I hadn't fully appreciated until I rebuilt Monzo's 2017 outage: a normal Service always resolves (to its ClusterIP) even with no endpoints, but a **headless** Service (`clusterIP: None`) resolves straight to its ready Pods, so with zero Pods its name has no records at all.
Put that name in an nginx `proxy_pass` and nginx won't start: it resolves every upstream once at startup and exits if any lookup fails. Running routers don't notice, because they loaded the config before the route was added. Restart them all at once and everything behind the router goes down:
nginx: [emerg] host not found in upstream "cards-v2" in /etc/nginx/conf.d/default.conf:6
`kubectl logs deploy/router --previous` is the command that finds it, since the crash-looping container has usually restarted by the time you look.
Guide (free): [https://learn.k8sight.in/go/nginx-upstream](https://learn.k8sight.in/go/nginx-upstream)
Also new: CPU throttling from limits set to the average (Buffer), and Pods Running but not Ready after a `dnsPolicy: None` change (Salesforce). They're all at [https://learn.k8sight.in/go/k8s-guides](https://learn.k8sight.in/go/k8s-guides) Disclosure: I built the platform; the hands-on incidents are paid, the guides aren't. Which outage would you want rebuilt next?