![header](card.jpg)

# The Go error recover() cannot catch

`fatal error: concurrent map iteration and map write` is not a panic. No recover() catches it. Since Go 1.6 the runtime watches for one goroutine writing a map while another reads it. When it sees that, it prints the stacks and exits the process with status 2. Every request in flight goes with it. The map that does this is often hiding inside something that is safe to share, like a `context.Context`.

## The note

A teammate once left a comment above one loop in our shared logging code. "TODO important: somehow there's a read/write race condition here", it said. And: "the stack trace points to this exact line."

He had the line right. He even had the error string. What he didn't have was the other goroutine. That's what "somehow" meant, and the comment stayed there for a year.

When I came to it, the question was the same as his: who else touches this map?

## Why Go stops the program

Go maps are not safe for concurrent use. The Go 1.6 release notes say it in two sentences: "if one goroutine is writing to a map, no other goroutine should be reading or writing the map concurrently. If the runtime detects this condition, it prints a diagnosis and crashes the program."

That crash is a fatal error, not a panic, so `defer recover()` never runs. A recover middleware that keeps an HTTP server alive through panics has a hole exactly here. The check is cheap and always on, but it only fires when the overlap actually happens. The same code can run clean for days and then stop twice in an hour.

On Kubernetes you can spot it without the logs. The container's last state shows exit code 2, and the log tail starts with `fatal error:` rather than `panic:`.

## Where the map hid

Every request carries a `context.Context`. A context is safe to share between goroutines by contract, so code shares it freely. One handler started a background goroutine and handed it the request's context. Inside that context sat a small struct of logging fields, and the fields were a plain map.

```mermaid
sequenceDiagram
  participant A as Request handler
  participant M as Shared ctx<br/>logging fields map
  participant B as Background goroutine<br/>(same ctx)
  B->>M: logger iterates the map
  A->>M: helper adds a field to the same map
  Note over M: write and iteration overlap
  M--xA: fatal error, process exits (status 2)
```

_The goroutine the note was missing: a background job holding the same context, logging while the handler added a field._

A simplified version of the shape:

```go
type LogContext struct{ Fields map[string]string }

func AddField(ctx context.Context, k, v string) context.Context {
    lc := ctx.Value(logKey).(*LogContext)
    lc.Fields[k] = v // writes the map every goroutine shares
    return ctx
}

// meanwhile, in a goroutine started with the same ctx:
for k, v := range ctx.Value(logKey).(*LogContext).Fields { // fatal
    fields = append(fields, k+"="+v)
}
```

The context was goroutine-safe. The map inside it wasn't, and nothing in the types tells you that.

## Answering the "somehow"

The explanation took one test. It builds one shared request context, starts some goroutines that add fields and others that log, and runs them together. Under `go test -race` the detector named both lines. One was the read in the logger loop under his note. The other was the write in the helper he never saw. Without `-race`, the same test can kill the test binary with the identical fatal error.

This kind of race hides from normal tests, because most unit tests run one request at a time. The race detector finds it because it watches every memory access, not just the ones that happen to collide. My pull request said: this PR answers the "somehow".

## The fix

1. Copy-on-write in the helper. Adding a field builds a new log context with a copied map, and returns a new context that carries it. The original map never changes after it's created, so anyone can iterate it without a lock. It's the idea behind `context.WithValue`: never change what's there, wrap it in something new. `WithValue` doesn't copy anything, so the copy of the map is ours.
2. Keep the test. It runs under `-race` in CI, so the next write to that shared map on a path the test covers fails the build.
3. Rebuild everything that links the helper. It lived in shared code, so merging fixed nothing by itself.

```go
func AddField(ctx context.Context, k, v string) context.Context {
    old := ctx.Value(logKey).(*LogContext)
    fields := make(map[string]string, len(old.Fields)+1)
    for key, val := range old.Fields {
        fields[key] = val
    }
    fields[k] = v
    return context.WithValue(ctx, logKey, &LogContext{Fields: fields})
}
```

The obvious alternative was a mutex or `sync.Map`. Both stop the crash. Both also put synchronization on the logging path, which runs on every log call, to protect a writer that almost never runs. Copy-on-write costs nothing when you read.

One behavior changes. A field added with the helper is now only visible through the context it returns. Callers that ignored the return value used to leak fields to sibling goroutines through the shared map. That is the unsafe behavior being removed. Check that every caller uses the returned context.

## The reviewer's catch

Step 3 almost had a hole. Our CI only rebuilds a service when something in its own folder changes. So the pull request touched a small file in each service that uses the shared code. The teammate reviewing it approved and added one line. One more service called the helper directly and wasn't in the list, so it wouldn't rebuild. Non-blocking, he said.

I added it. One more commit, he approved again, and it merged the same day.

## What I kept

recover() doesn't catch fatal errors, so a recover middleware has a hole exactly where data races live. A goroutine-safe container can carry unsafe contents. And a TODO that names the exact line is half a bug report. A test under `-race` supplied the other half.

## Credits and links

- [Go 1.6 release notes: detection of concurrent map misuse](https://go.dev/doc/go1.6#runtime)
- [Go: Data Race Detector](https://go.dev/doc/articles/race_detector)
- [Go blog: Go maps in action](https://go.dev/blog/maps)

_Nahum Litvin, Lead Platform Engineer._