gVisor, a kernel in Go: what sits between untrusted code and Linux

I write about gVisor a lot, because it's most of my day. My job is running a non-standard kernel in production, under code nobody on our side has read. When it breaks there's usually no answer to search for. You read the source, you talk to the maintainers, and you fix it upstream. I have 4 fixes merged in gVisor so far. It makes for a very interesting engineering job. This post is the map the other gVisor posts here build on.

In August 2026 I asked gVisor to kill its own kernel to free 1 stuck thread. Konstantin Bogomolov, a gVisor maintainer, read the pull request and called it "an over-reaction". He was right. A gVisor sandbox isn't one process. It's a Sentry, a gofer per container and a few stub processes that run the app's code, and I had proposed killing all of them to fix one. This post is the map he was reading from.

Why we run a second kernel

We run code we've never seen. Thousands of sites, each with its own backend code, written by people we'll never meet. Any of it could be hostile.

A normal container is a Linux process with walls around it: namespaces decide what it can see, cgroups decide how much it can use, seccomp decides which system calls it may make. Every call it is allowed still lands in the host kernel, and every container on the node shares that kernel. One kernel bug is enough to escape.

A virtual machine fixes that with a guest kernel and a hypervisor, and pays for it in boot time and memory. gVisor sits in the middle. A sandbox starts like a container, but the app's system calls never reach the host kernel.

The pieces

The core is the Sentry. gVisor's own intro calls it "a Go-based, from-scratch reimplementation of the Linux system call interface, memory management, filesystems, a network stack, process" management and more. When the app calls open, read or mmap, the Sentry answers. It keeps the file tables, the memory mappings, the signals and even the TCP connections, in one Go process that is locked down with seccomp and namespaces of its own.

flowchart LR
  Stub[Stub processes<br/>run the app's code] -->|every syscall| Sentry[Sentry<br/>Linux in Go]
  Sentry -->|few, filtered calls| Host[Host kernel]
  Gofer[Gofer<br/>one per container] -->|opens mount roots| Host
  Gofer -.hands over file descriptors.-> Sentry

One sandbox on the node. In August I thought the Sentry box was the whole sandbox.

The gofer is a separate, more trusted process. At startup it opens each mount's root on the host and hands those file descriptors to the Sentry. runsc is the OCI runtime that starts all of it, so containerd runs it where it would run runc, and Kubernetes picks it per pod with a RuntimeClass. One pod, one sandbox.

For the most common escape, a bug in the system-call interface, an attacker now needs two bugs: one to take over the Sentry, then one in the much smaller set of host calls the Sentry may make. gVisor describes them as two kernels "which do not share any code".

The stubs, where I got it wrong

On cloud VMs gVisor catches system calls with a platform called systrap, the default since mid-2023. With systrap the app's code doesn't run inside the Sentry. It runs in stub processes, plain host processes whose system calls the kernel turns into a SIGSYS signal. A small handler in the stub saves the thread's registers into a shared slot called a context, and wakes the Sentry. The Sentry handles the call in Go and puts the context back on a queue, where any idle stub thread can pick it up.

That was the part I hadn't pictured. Our hang was one app thread whose interrupt signal got lost, so its context never came back, and a pod that wouldn't die for days. My fix said: if a context stays stuck, kill the Sentry. In my head the Sentry was the sandbox.

Konstantin's review, on 26 August: "If there's one stuck subproc but the others are fine, we'd end up killing everything. Like in NotifyInterrupt, the first action should still be to kill the subprocess." gVisor already had a path that kills one stub process and lets the Sentry report that thread as dead. I just hadn't seen it. He also asked me to drop the comments that "redescribe the code in comment form". Fair.

Then the timeout. An engineer who had hit the same hang at another company worried that 30 seconds was too aggressive under host pressure, so I raised it to 3 minutes. Three days later Konstantin sent it back down. Once the kill only takes one stub, "30 seconds is still ages in terms of responding to a signal in time." The PR, gvisor#14201, merged on 31 August: if a context is still stuck after 30 seconds and the interrupt didn't free it, the Sentry kills that one stub process.

The stub's memory

The second lesson came in October. Every stub reserves one block at high addresses, shared with the Sentry: per-thread stacks, the context queue, a spinning queue where idle threads wait, and one slot per context. At startup systrap shifts most of that block by a random, page-aligned gap, so one sandbox's layout tells an attacker nothing about the next.

flowchart TB
  A[Stub code, handler<br/>and seccomp rules] --> G[Random gap]
  G --> B[Per-thread stacks<br/>partly shared with the Sentry]
  B --> C[Context queue<br/>shared]
  C --> D[Spinning queue<br/>private to the stub]
  D --> E[Context region<br/>one slot per context, shared]
  class D bug

Lower addresses first. Everything after the gap has to move together. The highlighted box didn't.

Since a 2023 change, the spinning queue stayed put while everything around it moved, so a small gap could land it on top of a thread's stack. The stub then spins forever and never picks up the context, so the 30-second kill from August never arms. I filed gvisor#15572 and sent a one-line fix with a layout test, #15573. Konstantin approved it the same day: "Thanks for the fix."

What you pay

Nothing here is free. A system call is now a trap, a trip to the Sentry and a trip back, so code that does many small reads and writes feels it (performance guide). Not every syscall and flag is there (compatibility list). And debugging moves: a hung app is now a hung Sentry, a stub process and a queue in shared memory, and runsc debug --stacks only answers if the Sentry still does.

For code we've never seen, we take that trade. Two pull requests, six weeks apart, same reviewer. The first one got "an over-reaction". The second got "thanks".

Where it bites

gVisor: one lost signal: the August hang and the fix, in full.
Next: a sandbox that hung with that fix already running.

Thanks to Konstantin Bogomolov for reviewing both fixes, and to the engineer at another company for the stack dump we never could get. Sources are linked inline.

Nahum Litvin, Lead Platform Engineer. Drafted with Claude Code.

References

PR · merged google/gvisor#14201: systrap: fail-stop sentry on stuck context
issue · closed google/gvisor#15572: systrap: spinning queue isn't shifted by the stub gap, aliases a sysmsg stack and wedges the sandbox
PR · merged google/gvisor#15573: systrap: shift the spinning queue by the stub gap