Nahum LitvinKubernetes, containers and sandboxes, from the production side
Agent tooling6 min read

STPA-inspired questions tied a plain checklist in my coding agent. Their one win: Claude asking about timing.

A safety method from MIT, three replayed changes, 54 plans, two blind judges, and a headline I had to take back.

STPA is a safety method from MIT that asks when an automatic action becomes unsafe. I put its questions into my AI coding agent's rules, then replayed three real past changes with no rule, with a plain "check what this could break" rule, and with STPA. On average STPA did no better than the plain rule. It won clearly in one case, where the risks were about timing. Only with STPA did Claude ask whether a timeout was too long (3 of 6 grades, against 0) or when to stop waiting (4 of 6, against 0).

The test

My first version of this post said STPA helped. One question, whether a change breaks apps outside its target, had jumped from 1 in 6 to 5 in 6 in a single replay of case A. Two reviewers asked the obvious thing: would any rule that says "check what you might break" do the same? So I ran it properly. The two rules, word for word.

The STPA rule (62 words):

When a plan or design adds or changes something that acts automatically (scaler, quarantine, drain, hook, mutating script), list each control action and ask: unsafe if missing, if wrong context, if early/late, if it stops too soon or runs too long. Name the feedback it acts on and what happens when that feedback is stale or empty.

The plain checklist (35 words):

Before shipping any change, check what it could affect beyond its intended target, how you would notice if it goes wrong in production, and how you would roll it back quickly.
flowchart TB
  subgraph setup[Same code, same prompt]
    direction LR
    K["3 past changes"] --> R["3 arms<br/>no rule, plain list, STPA"] --> M["2 models<br/>3 runs each"]
  end
  subgraph grade[Graded blind]
    direction LR
    P["54 plans"] --> J["2 judges<br/>method words removed"] --> S["bugs found<br/>out of 8"]
  end
  setup --> grade

The bugs are the ones each change's real review or production run found. The judges agreed on 90% of calls.

The three changes, in generic form. A: a controller taking over a value another controller used to write. B: automatic recovery for a worker that stops responding. C: a dispatcher that picks workers from a cache that can be stale.

First, the headline from my first version. In case A, regraded with stricter blinding: no rule 4 of 6, plain list 6, STPA 6. The gap shrank, and the plain list closed it too.

xychart-beta
  title "Real bugs found per plan, out of 8"
  x-axis ["A none", "A list", "A STPA", "B none", "B list", "B STPA", "C none", "C list", "C STPA"]
  y-axis "bugs" 0 --> 8
  bar [6.3, 6.8, 0, 4.6, 5.1, 0, 6.3, 6.4, 0]
  bar [0, 0, 6.6, 0, 0, 6.1, 0, 0, 5.8]

Orange: STPA. Both models pooled. Across all three, the plain list and STPA tie at 6.1.

The plain list helped a little everywhere. STPA beat it only in B, and fell below no rule in C.

Case B, Claude: only the STPA rule made it ask whether the timeout was too long (3 of 6) and when to stop waiting (4 of 6). No rule and the plain list: 0.

STPA's "too late" and "applied too long". Out of 6 judge grades, 3 plans each graded twice. Without STPA, zero.

In B, the one thing the method added for Claude was questions about time. Codex asked about the timeout even with no rule, and more with either rule. But neither model asked when waiting stops helping, except Claude with STPA. One honest limit: the STPA rule names timing ("early/late", "runs too long") and the checklist doesn't, so this test can't tell STPA apart from a rule that just mentions time.

In C, STPA cost Codex the basics, even though Claude gained a little.

Case C, Codex: visible check errors, a metric, a proving test. No rule 4, 3 and 1 of 6. Plain list 6, 4, 3. STPA 0 on all three.

With STPA, Codex dropped all three. The plain list kept them.

Where the rule came from

My agent learns from a git repo of rules, and until recently almost every rule came from something that had already gone wrong. One says "never conclude from a zero". During an incident, a metrics query with a wrong label returned nothing, with no error. The graph looked calm while the thing it measured was on fire.

On 13 October I was at SREcon in Dublin, listening to John Thomas of MIT, co-author of the STPA Handbook. His talk: What Your SLOs Can't See: Applying STPA to a Distributed Software Service. My laptop was open.

The idea comes from Nancy Leveson at MIT. Her accident model, STAMP (System-Theoretic Accident Model and Processes, 2004), later expanded in her book Engineering a Safer World (2011), treats safety as a control problem.

flowchart LR
  C["Controller<br/>picture of the world<br/>decision logic"] -->|control action| P[Controlled process]
  P -->|feedback| C

STAMP's loop. Accidents come from inadequate control: a wrong picture, flawed logic, or missing feedback.

STPA (System-Theoretic Process Analysis) is the hazard analysis built on it. For every control action you ask when it becomes an unsafe control action (UCA): not provided, provided when it causes a hazard, too early, too late or out of order, stopped too soon or applied too long.

I asked my agent if any of it applied to me. It said: "You're doing half of it, and only after the fact." So I put the questions where the agent already decides:

flowchart TB
  subgraph before[Before it ships]
    direction LR
    A["Plan<br/>how could this act unsafely?<br/>what feedback does it trust?"] --> B["Review<br/>new automatic actor?"] --> C["Handover<br/>who else writes this field?"]
  end
  subgraph after[After something breaks]
    direction LR
    D["Post-mortem<br/>what did it believe, and why?"]
  end
  before --> after

The post-mortem step borrows from CAST, Leveson's accident analysis, also built on STAMP.

That is mostly STPA's third step. No losses, hazards or control structure written down. A real STPA starts there. Mine doesn't, yet.

A second model and a review bot then found my rules breaking their own method: a post-mortem line that only blamed bad feedback, a missing UCA type, and a writer check placed after the writers act. A check that runs too late, written during a talk on what monitoring misses. I fixed it and merged before the lecture ended.

What I kept

A plain checklist does most of the work. Across the three cases it matched STPA, 6.1 to 6.1, against 5.7 with no rule. What I keep is the timing questions, whether the credit goes to STPA or just to naming time: my checklist now adds "too early, too late, too long". For Claude nothing else raised them, 0 of 6 without and 3 and 4 with. And I now watch what a method pushes out. In C, Codex's three operational checks went to 0 of 6 with STPA.

Three changes is not a study, but it was enough to retract my first headline.

Nahum Litvin, Lead Platform Engineer. catchkill9.dev

Comments