RedditPP Mascot

RedditPP

r/kubernetes • … • u/Fabulous-Confection6

A one-line gVisor bug left pods stuck in Terminating: random memory layout moved every region except one queue

Follow-up to [yesterday's gVisor intro](https://www.reddit.com/r/kubernetes/comments/1xbjawo/). This is the stub-memory bug I mentioned there, with the debugging steps.

gVisor's systrap platform runs app code in host "stub" processes and shifts their shared memory by a random, page-aligned gap at startup, so the addresses are hard to guess. In `stub_unsafe.go` three regions get `+= gap`. The spinning queue, sitting between them in the layout, did not.

With a big gap the queue sits alone in empty space. With a small one, the shifted per-thread stacks land on the queue's old address. The stub then spins in `spinning_queue_remove_first` waiting for a timestamp slot that stack bytes keep at 0. That check sits before the timeout, so it never gives up. The Sentry waits on a context no stub ever acknowledged, and my 30s fail-stop, which kills a stuck stub process, only arms once an acknowledged context was interrupted. The sandbox never exits.

On Kubernetes it looks like this: a pod stuck in Terminating, one stub process pinned at 100% of a core, and `runsc kill` hanging.

Back of the envelope from the code: the gap is uniform below a random room of up to 1 TiB, so the odds come out roughly 1 in 100,000 starts for a 16-CPU sandbox, and about 1 in 35,000 at 64. That is an upper bound, since a hang also needs the overlapped stack in use. It had been there since the spinning queue landed in March 2023.

Forcing a small gap in an arm64 VM: stock build hung 2 of 2, with the one-line fix 20 of 20 clean.

If you run gVisor with systrap (`release-20230417.0` through `release-20261005.0`): upgrade to a runsc build that contains `c7f458aa3`. Until then `--systrap-disable-fast-path` (`release-20260622.0` or later) stops the spinning, at some syscall latency cost. To clear a hung pod, kill its `runsc-sandbox` process on the node.

Fix: https://github.com/google/gvisor/pull/15573

Write-up with the debugging steps and diagrams: https://www.catchkill9.dev/posts/gvisor-spinning-queue-gap.html

— 0