Nahum Litvin

Mermaid charts in Claude Code's terminal: what a mod could do that my VS Code harness couldn't

Fri, 02 Oct 2026 00:00:00 GMT

One reply on the Dracula theme. Everything here is text the mod redrew.

Every day I watch two terminals. In one, Codex prints a table with a colored header, a rule under it, versions in green. In the other, Claude Code prints the same table as pipes and dashes. prismantis is the fix: a Claude Code mod that redraws every reply with colored tables, highlighted code and mermaid diagrams, right inside the terminal.

What did I try first?

What I wanted was small: mermaid charts and some basic HTML rendering in Claude's replies. About a year ago I built a harness around Claude Code as a VS Code extension to get them.

It was bad. I dropped it, and the tables stayed pipes and dashes.

What changed on 1 Oct?

Claude Code 2.1.287 shipped mods. A mod is a plugin made of TypeScript function hooks that run inside Claude Code. One hook is ui.render. Claude Code calls it for every piece of the transcript it is about to draw, hands you the props, and asks what to draw instead.

For an assistant reply, the props are the reply's markdown. You return a tree of Box and Text elements, and that is what lands on screen. No extension. No second window.

The next day prismantis went from 0.1.0 to 0.3.2. Four versions, all dated 2 Oct in the changelog. I read the announcement in the morning and had a working table by lunch.

The name is prism plus mantis. The mantis shrimp has 12 to 16 kinds of color receptors. Your terminal had 8.

Why did I delete the best-looking feature?

Version 0.2 drew real mermaid diagrams. It shelled out to mermaid-cli, rendered a PNG, and showed it in terminals that speak the kitty graphics protocol.

Mods run in-process, inside Claude Code, and they aren't sandboxed. A mod that spawns programs and writes image files is a mod you have to trust a lot more than one that only moves text around.

0.3 removed image mode and the mermaid-cli dependency. Diagrams draw as colored box art now. The mod runs no external programs, writes no files, reads no files and makes no network calls.

What did a second reviewer find?

I built it with Claude Code, in Claude Code. I described what I wanted, Claude read the API declarations the engine writes next to every mod, and I looked at screenshots and said "almost" a lot.

Then I asked a second model, Codex, to review it. It flagged 12 issues.

The worst one was quiet. The copy button on a table gave back my redrawn table instead of the markdown Claude wrote, so a pasted table came back reformatted. The other bad one: a fence of four backticks closed on the first three-backtick line inside it, which broke every code example nested in a code example.

I fixed most of them, documented one and skipped two as rare. The fixes came with 10 regression tests, and I ran them against the old code first. 7 failed there, which is the only way I trust a regression test.

Why didn't Claude draw any charts?

Once the charts rendered, I noticed something annoying. Claude almost never wrote one. It doesn't know the terminal can draw a mermaid block, so it writes a table or a paragraph instead.

The obvious fix is one line in the system prompt. Plugins can't do that. Claude Code's built-in sec-default policy keeps the system prompt for the organization, not for whatever you installed.

So 0.3.2 added diagramHints. The mod attaches a short note to each prompt you type, read by the model and never shown, saying tables, code, mermaid diagrams and xychart-beta charts render here. It costs about 80 tokens per prompt. It is off whenever mermaid is off, and skipped for headless claude -p runs.

After that, charts started showing up without anyone asking for them.

Drawing the chart was the easy part. Getting Claude to write one was the work.

What does it draw?

Tables get a colored header, rules, column alignment from :---: and colored numbers, sized to the terminal. Code gets a language header and Prism highlighting in 24 languages. Mermaid covers flowcharts, sequence, state, class and ER diagrams, plus bar and line charts, one color per box, participant and bar.

Every code block, table, list and quote has a copy button; diagrams have two, one for the mermaid source and one for the drawn art. Tool calls shrink to one line, Ran gh pr view 12, and collapsed groups to a summary like Ran 3 commands, read 2 files. 15 MIT themes ship, plus mono, and each of the 20 color slots can be overridden in /config.

What did the mod API teach me?

Hooks run without Node. No require, no package imports, no dynamic import(). Every library has to be bundled into one ESM file. I bundle two, beautiful-mermaid and Prism, 206 KB together, both MIT.
The engine handle can't cross a file import. claude plugin validate doesn't notice. claude plugin test does, with "hooks module did not load". All my code that touches $ lives in one file now.
The built-in Markdown element takes no colors. So a mod that themes replies has to parse markdown itself. Mine is regex, not CommonMark. It covers what Claude writes.
There is no font size, and the clipboard takes text only. Headings get bold, underline or a rule instead. A diagram can't be copied as a picture.

The whole thing is about 950 lines of TypeScript and 52 tests. CI runs them on Linux, macOS and Windows against the newest Claude Code, and on Linux against 2.1.287, the oldest with mods. It also type-checks, rebuilds the bundled libraries byte for byte, fails if a bundled package isn't MIT, and installs the plugin into a clean config.

What doesn't work yet?

The parser is not CommonMark, so nested quotes and HTML draw as plain text. CJK and emoji count as two columns in tables, and terminals disagree on a few emoji, so those can still be off by one. Pie charts don't render. I've hand-tested it in the macOS terminal only.

All of it is reversible. Press ctrl+o on any reply to see the original, or turn the mod off in /plugin and Claude Code's own renderer comes back.

How do I try it?

/plugin marketplace add NahumLitvin/prismantis
/plugin install prismantis@prismantis

Needs Claude Code 2.1.287 or later. Then ask Claude to print docs/demo.md from the repo verbatim as its whole reply. Every feature is in that one file. Pick a theme in /config. MIT licensed, source at https://github.com/NahumLitvin/prismantis.

Credits

Diagrams come from beautiful-mermaid, highlighting from Prism. The palettes belong to their authors: Catppuccin, Dracula, Nord, Tokyo Night, Gruvbox, Rosé Pine, Everforest, GitHub Primer, One Dark and Solarized, all credited in the repo's third-party notices.

Nahum Litvin, Lead Platform Engineer.

A zero timeout in Go is not no timeout

Tue, 29 Sep 2026 00:00:00 GMT

While working on snapshotter timeouts in containerd, I assumed a timeout of 0 meant "no limit". It doesn't.

ctx, cancel := context.WithTimeout(context.Background(), 0)
defer cancel()
fmt.Println(ctx.Err()) // context deadline exceeded

The deadline is now, so the context is expired before the first call. containerd's timeout.WithContext passes the configured value straight to context.WithTimeout, so a key set to 0 fails every call on the spot.

That's why the new default_timeout for proxy snapshotters is off when empty, and only wraps calls whose context has no deadline yet: https://github.com/containerd/containerd/pull/14187

A 12 second go run settled it. Cheaper than arguing about it.

One lost signal, five days stuck, 45,000 frozen threads: fixing a gVisor hang upstream

Tue, 29 Sep 2026 00:00:00 GMT

It looked like a deadlock in gVisor. It was a single missed wake-up, hiding behind a symptom teams have been reporting since 2021, and two strangers and a maintainer fixed it in about a week.

The hang in one picture: one signal, no retry, and a deadline check whose only action is a log line.

On a Tuesday morning our alerting claimed that 559 pods were stuck Terminating in production. The real number was one. The rule counts kubelet failed-kill events rather than pods, and a sandbox that never dies gets retried forever, so a single wedged pod read as a fleet on fire. That pod turned out to be the interesting part.

At Wix, our Velo grid runs users' backend JavaScript as untrusted code inside gVisor sandboxes on Amazon EKS. gVisor puts a userspace kernel between that code and the host kernel, so untrusted code never talks to the host kernel directly. Close to two thousand sandboxes per node group, short lifecycles, heavy churn. Deleting a pod should take seconds. This one had been Terminating since 07:11. kubelet's kill requests got DeadlineExceeded every 2 minutes and would have continued forever. We intervened by hand after about 90 minutes. A week earlier, eight pods on four nodes had sat like that for days.

Why nobody's timeout saved us

Before the forensics, the shape of the problem. Deleting a pod is a relay: each layer asks the next one, politely, to make something die. Every layer should hold two more things beyond the polite ask: a bound on how long it waits, and an escalation that works without the cooperation of the thing it is waiting on. In our case the relay runs from kubelet (the Kubernetes node agent) to containerd (the container runtime) to the gVisor shim (the per-sandbox process containerd talks to) to the sandbox itself. Here is that stack, graded on all three:

Three arrows per hop: blue attempts, amber detects, violet recovers. Blue failed once, at the bottom. Amber either did not exist, fired into a log line, or fired and could only re-send blue. Violet existed nowhere, so one lost message at the bottom of the stack became the whole stack's hang.

Look at the violet column. Every escalation asked the very component being escalated against to cooperate. containerd's dead-shim cleanup only fires when the shim disconnects. Ours kept answering Connect and State, which never touched the lock, while Kill and Stats blocked behind it. The connection stayed up, so cleanup never fired. Alive enough to prevent its own cleanup.

What a frozen sandbox looks like

That userspace kernel is called the sentry. On systrap, the platform we run, the app's threads run in stripped-down stub processes and the sentry coordinates them. Our wedged pod had a sentry that was alive but answering nothing. runsc kill, gVisor's own command for stopping a sandbox, hung. runsc debug --stacks, the tool that is supposed to tell you why something hangs, also hung. The panic log was 0 bytes.

The node told the rest of the story. Load average climbing about 3 per hour while actual CPU sat near 30%. Hundreds of sentry threads parked, waiting on shared memory. Stub processes accumulating. On the worst nodes, load kept climbing for days with the machine mostly idle. The only remediation that worked was SIGKILL to the whole sandbox process tree, by hand, over AWS SSM, a remote shell into the node. That became a runbook. Runbooks like that are a debt.

Stub processes pile up behind the frozen sentry: load average keeps climbing while the CPU does almost nothing. Threads stuck waiting, not working: the rising top panel against the flat bottom one is the bug.

So we filed gvisor#14408 with what we had: the outside view, two affected releases, and an honest admission that we could not get stacks because the debug tooling was wedged along with the sandbox. Our proposed fix, PR #14201, had been open since the week before.

Then a stranger showed up with the other half

The same day, another engineer filed gvisor#14405. Same bug at a different company, and they had the one thing we could not get: a full dump of the sentry's goroutines, Go's lightweight threads, from inside a frozen sentry. Their dump showed one of the sentry's internal worker threads sitting in the same wait loop for over five days, waiting for a stub thread that had missed a single wake-up signal.

In their shim, about 45,000 waiting threads were queued behind one lock held by a kill that would not finish, and their containerd and kubelet memory grew until nodes ran out.

The bug itself is painfully simple. The sentry sends the stub one interrupt signal. One. If that signal is lost, and nobody has yet pinned down why it sometimes is, the sentry keeps waiting for the answer with no retry and no escape. There is even a 30 second deadline in the code that notices the wait is too long. Here is the whole thing it did about it:

if time.Now().After(deadline) { log.Warningf("Systrap task goroutine has been waiting on " + "ThreadContext.State futex too long. ...") // no retry, no escalation }

Where did 45,000 waiting threads come from? cAdvisor kept polling every container's Stats. Each request added a goroutine that blocked behind the hung kill's lock. Its caller timed out, but a mutex wait cannot observe cancellation, so the goroutine stayed. Over five days, 45,000 accumulated: roughly one every 10 seconds, the scrape interval fossilized in a thread dump. The metrics system asking "how are you?" every 10 seconds had pushed the shim to about 600MB.

A timeout that only logs a warning is not a timeout. It is a diary.

And this exact wait sits on the teardown path. Killing a sandbox starts with a freeze-everything step that waits for every worker thread to park, so one stuck worker means the kill command can never answer, the shim call hangs, kubelet gets DeadlineExceeded, and each retry parks another goroutine behind the first one. The pod stays Terminating for days.

So, how did we fix it?

The merged fix escalates in stages, gentlest first:

1. Tap again. The interrupt is resent on each 5 second checkup wakeup instead of once. When the resend lands, a lost signal costs the workload a stall of a few seconds, then it carries on.

2. Photograph the scene. If the stub is still unresponsive after the 30 second deadline, the sentry dumps every internal stack trace to the log. A frozen sandbox can not be debugged from outside unless it was started with --panic-signal; ours were not, theirs were, which is where their dump came from. So this failure now documents itself. The next team to hit anything like this gets for free what took two companies a week to assemble.

3. Remove only the broken part. Then the sentry kills just the one stuck subprocess, through the same path it already uses when a stub dies naturally. The blocked task unwinds, teardown can finish, and the healthy subprocesses in the sandbox are untouched.

Update, 1 Oct 2026: a gVisor maintainer reported that step 3 also hits healthy subprocesses. A host page fault that takes more than 30 seconds, for example on a slow FUSE server, looks the same as a lost signal, because the stub cannot take the interrupt until the fault returns. gvisor#15182 keeps the resend and the stack dump, and kills only when the stub thread no longer exists, not after 30 seconds. It is in review.

My first version killed the entire sentry. gVisor maintainer Konstantin Bogomolov pushed for killing only the stuck subprocess, sparing healthy neighbors. He was right. The other reporter had raised the same concern earlier that day. Konstantin also caught a dropped continue, which led me to a race: a context that recovered during the wait could still get killed. Three review rounds in under a week took the PR from nobody having looked at it to merged on master.

The merged fix, staged from least to most disruptive. My first version jumped straight to killing the whole sandbox; review scoped it down.

The part that bothers me

After the merge, we found reports since 2021 from a Cloudflare engineer, a GKE user, Knative, Talos Linux, Arista and others on the containerd tracker. Roughly eight teams over five years. Causes differed, and some cases got fixes. But the pattern kept returning: pods stuck Terminating, debugging tools hanging, someone killing processes by hand, and more often than not the thread ends without a stack trace from inside the sandbox.

Two teams filing in the same week broke that pattern: our production evidence and proposed fix met the other team's internal stack dump.

Five years of the same symptom, with different root causes underneath. Most threads ended with the manual workaround, until two half-reports completed each other.

The relay diagram near the top of this post still isn't fully green. The merged fix repairs the layer where the hang started. We filed gvisor#14548 to bound shim kill waits, then SIGKILL the sandbox tree, and containerd#14081 to SIGKILL the shim after repeated kill RPC timeouts. In review of the shim PR, gvisor#14549, the same maintainer asked to narrow it to bounded waits on Kill, Stats and Status, because killing the sandbox from the shim would take out healthy containers too, the same over-reaction he flagged in the first fix. That revision merged, and escalation stays open. Either escalation, as originally filed, would have capped our incident at minutes.

Lessons learned

A watchdog that only barks is not a watchdog. If code detects a should-never-happen state, it must act: retry, self-report, fail. Grep your own codebase for deadline checks whose only body is a log line. This one surfaced because a sandbox sat stuck for five days behind it.
File the issue even with half the evidence. Our report had no stacks and said so plainly. Within a day a stranger's independent report supplied them, and their dump changed the fix. The half-report you are embarrassed to file is someone else's missing half.
Remediate the smallest thing that unblocks you. Kill the subprocess, not the sandbox. The instinct under pressure is the big hammer. Review cut the kill from the whole sentry to one subprocess. The same restraint belongs in your incident runbooks.
Audit your escalation paths for circular dependencies. Walk your stack with that diagram's two questions: bounded wait, and escalation that works without the target's cooperation. If the answer to the second bottoms out at "a human with SSH", write that down, because that is your actual design.

The fix shipped in gVisor release-20260831.0. Until our fleet is on it, the SSM kill runbook stays, but this one has an ending, and the next report of this hang will arrive with its stacks attached.

This post was written by Nahum Litvin.

containerd part 3: the fix got reverted

Fri, 25 Sep 2026 00:00:00 GMT

People share their victories. A month ago I shared mine: my containerd fix got merged after 40 days and 30 review rounds.

13 days later it got reverted. It never made it into a release.

My fix put a timeout on the one call that hung on my nodes. It worked for my incident, and it felt like a win.

So I opened a second PR to cover more of the calls that could hang the same way.

That's where it came apart. Working through it with the maintainers, we saw the real problem was one layer out. The thing hanging was an external plugin, and it can hang on any of its calls. Patching them one at a time would never end.

The right fix is a single timeout where containerd talks to the plugin, covering every call at once. That made my first fix redundant, so with a release about to be cut, Michael Brown reverted it. I agreed.

In Part 2 I said the review is the distance between a patch that's correct for your incident and one that's correct for everyone. Turns out it doesn't stop at merge.

A new PR is on its way. Round 31. Not giving up.

containerd part 2: 17 lines, 40 days, 30 reviews

Tue, 25 Aug 2026 00:00:00 GMT

The fix was 17 lines. Getting it into containerd took 40 days and 30 review rounds, and taught me more than writing it did.

Part 2 of the bricked-node story: the resilience patch got merged upstream this week. My biggest open source contribution to date.

Writing the code was the easy part. The review was where the real engineering happened.

The timeout key got renamed twice, because naming in a project with hundreds of config keys is an API commitment, not a variable name.

The default went from 30 minutes to 5 and back to 30. I pushed hard for 5, a maintainer pushed back with an argument I hadn't considered: the old behavior was unbounded, so on upgrade the least surprising default is the big one, and anyone running remote snapshotters can tune it down. He was thinking about thousands of clusters upgrading. I was thinking about mine.

My test got rewritten around Go's synctest, so it runs in milliseconds on a fake clock instead of mutating global state.

And when CI wedged in the merge queue, Michael Brown restarted the buckets, herded the reviewers, and queued it again until it landed. Maintainers do this invisible work daily, for free, for strangers.

A patch that's correct for your incident and a patch that's correct for everyone running containerd are two different things. The review is the distance between the two.

What's the hardest review you've been through, and what did it fix that the code didn't?

A node that reported Ready and started nothing: four parts, one containerd fix

Tue, 11 Aug 2026 00:00:00 GMT

What happened

In July one of our Kubernetes nodes stopped starting pods for 4.5 hours. It reported Ready the entire time. The scheduler kept placing pods on it, every new container hung in creation, and nothing on the node's health path looked wrong.

We run untrusted user code at Wix, one site per sandbox, so pods come and go all day. A node that silently stops starting them is worse than a node that dies: the scheduler keeps feeding it.

The easy answer was a file descriptor limit. The snapshotter logs were full of "too many open files". Raise the limit, restart, move on. I almost shipped that.

This is the whole story in one place: the outage, the two bugs, the fix that got reverted, and the one that stays.

Why it happens

Two bugs lined up.

The first was in containerd. During snapshot garbage collection, containerd calls the snapshotter's Remove while holding that snapshotter's write lock in containerd's metadata layer, with no deadline. It even drops the caller's cancellation. If the snapshotter never answers, the lock is held forever, and every snapshot operation on the node queues behind it, including the one that starts a new pod sandbox. Nothing on the readiness path takes that lock, which is why the node kept saying Ready.

Readiness is a separate path. Kubelet asks the runtime for its status, the runtime answers from memory, and nobody touches the snapshot metadata. So the node was healthy by every check we had, and useless by the only one that mattered: can it start a container.

The second was the trigger, in AWS's soci-snapshotter, the plugin that lazily loads container images. Every file verified on first access opened a reader from the span cache and never closed it. The sibling function about 45 lines up closes the exact same reader. One leaked file descriptor per file, on every node running lazy-loaded images. Under a burst of pod starts the snapshotter hit its limit of 65,535 and stopped answering. containerd was waiting on it, holding the lock.

A proxy snapshotter like soci runs in its own process, and containerd talks to it over gRPC. That boundary is where the real problem lived, though it took me two more PRs to see it.

The goroutine dumps told the story once we knew where to look. Hundreds of goroutines parked on that snapshotter lock, and one GC goroutine holding it, waiting for a gRPC header from the snapshotter that was never coming. The node was not broken. It was waiting, politely and forever.

What we got wrong first

I got the diagnosis wrong twice before I got it right. After the storm the file descriptor count looked low, 151 out of 65,535, so I concluded the limit theory was wrong. Then I flip-flopped back. The leaked descriptors had been reclaimed once Go's finalizers ran, so the steady state hid the peak.

I did most of that investigation with Claude, reading goroutine dumps from the wedged node and two unfamiliar codebases side by side. It didn't write the fix. It made the investigation cheap enough to do properly, including catching my own wrong conclusion.

The bigger mistake came later. My first containerd fix put a timeout on the one call that hung on my nodes, the GC's Remove. It was correct for my incident. It got merged after 40 days and 30 review rounds, and I wrote about it as a win.

Those 30 rounds were the real engineering. The timeout's config key got renamed twice, because in a project with hundreds of keys a name is an API commitment. The default went from 30 minutes to 5 and back to 30. I pushed hard for 5. A maintainer pushed back: the old behavior was unbounded, so on upgrade the least surprising default is the big one, and anyone running remote snapshotters can tune it down. He was thinking about thousands of clusters upgrading. I was thinking about mine.

My test got rewritten around Go's synctest, so it runs in milliseconds on a fake clock instead of sleeping and mutating global state. And when CI wedged in the merge queue, Michael Brown restarted the buckets, herded reviewers and queued it again until it landed.

Then I opened a second PR to cover another call that could hang the same way. Working through it with the maintainers, we saw that I was patching symptoms one call at a time. The thing that hangs is an external plugin, and it can hang on any of its calls. With a release about to be cut, Michael Brown reverted my first fix 13 days after it merged, so it never shipped. I agreed with the call.

The suggestion that changed the plan came in that second PR's thread: stop bounding components one by one, and put the bound where containerd talks to the plugin. Michael agreed. Reverting my first fix was the price of doing it that way, because two mechanisms bounding the same call would have been worse than one.

The fix

soci-snapshotter: close the reader in file Verify. One line, defer r.Close(), merged upstream in July.
containerd, first attempt: bound the GC path's Remove with a configurable timeout. Merged, then reverted before release, because it fixed one call out of many.
containerd, the fix that stays: a default_timeout setting on proxy plugins. When it is set, every call containerd makes to a proxy snapshotter gets that deadline if the incoming context has none. A caller that brings its own deadline keeps it. Leave it empty and nothing changes on upgrade.

The proxy snapshotter has 10 gRPC methods and none of them had a deadline unless the caller already set one. Putting the bound at the client covers all 10 at once, including the GC path my first fix targeted. It also covers paths I never hit in production, which is the point.

Streaming calls needed a decision. Walk gets one deadline for the whole stream, not one per message, so time spent in callbacks counts against it. The content store proxy has the same gap but a different shape: a writer outlives the call that created it, so a per-call deadline would cut an ingest that is still being written. That one needs its own setting and its own PR.

The review on the final PR was shorter, and it still taught me something. I had added an options argument to the exported constructor. Compatible, I thought. The maintainer pointed out it still breaks anyone who stores that function as a value, so the old constructor stays untouched and a new one takes the options.

The SOCI side was the quick part. The PR went in on July 15 and AWS merged it five days later. The containerd side took three PRs and two months. The difference is not the size of the change. A one-line fix to an obvious leak has one right answer. A timeout in a runtime that thousands of clusters run has many plausible answers, and the review is where you find out which one the project can live with.

It doesn't fix everything, and the PR says so. A GC pass over many slow snapshots can still add up while it holds the lock, because each call finishes just under the limit. That needs the lock itself reworked, and it is the next piece.

Four posts, three PRs, one revert. The fix that finally lands is smaller than the one that got reverted, and it covers more. That is usually how it goes when the review is doing its job.

Lessons

A patch that is correct for your incident and a patch that is correct for everyone running the project are different things. The review is the distance between them.
Review doesn't stop at merge. Mine got reverted, and the revert was the right engineering.
Put the bound at the boundary. One deadline where containerd talks to the plugin covers every call; one deadline per call never ends.
The steady state lies. Look at the peak, not the numbers after the storm.
Maintainers do invisible work daily, for free, for strangers. Say so when they do it for you.

Credits and links

Michael Brown and the containerd maintainers, for 30 review rounds, a revert I agreed with, and a better fix
containerd/containerd#13799
containerd/containerd#14187
awslabs/soci-snapshotter#2043

Krew is K8s Game Changer!

Thu, 11 Jul 2024 00:00:00 GMT

Kubernetes can be quite complex, but it doesn't have to slow you down. Here's how I simplified my K8s management with some awesome Krew plugins.

installation guide for Krew

list of all Krew plugins

alternate list of all plugins

Discovering Krew Plugins

When I first started using Kubernetes, I found it overwhelming to manage all the resources efficiently. That's when I stumbled upon Krew, a package manager for kubectl plugins, which transformed my workflow. Here are the plugins that made a huge difference for me:

foreach This plugin is a game-changer. Imagine being able to execute commands across multiple resources with just one line. It's perfect for bulk operations, saving me a ton of time when managing multiple pods or services.
get-all Getting a comprehensive view of all resources in a namespace used to be a hassle. Not like using kubectl get all With get-all, I can actually retrieve every resource type in one go.
ktop Monitoring cluster resources is crucial, and ktop makes it so much easier. It's like having a real-time dashboard for your cluster's health and performance. I check it regularly to ensure everything is running smoothly.
netshoot This one really changed my life. it allows me to easily debug any networking issues, attaching debug container to any pod with ease.
node-shell Sometimes, you just need direct access to the nodes. Node-shell allows me to get a shell on any node in the cluster. It's invaluable for troubleshooting and performing maintenance tasks directly on the node. and doesn't require ssh,ssm,ec2-connect or any of those. it is a very privileged container and might cause your security team to call you to ask WTF?
stern Debugging applications can be tedious, especially when you need to tail logs from multiple pods. Stern makes this process seamless by streaming logs from all relevant pods in real-time. It's a lifesaver when tracking down issues across pods.

Running kubectl commands on multiple clusters

Mon, 01 Jul 2024 00:00:00 GMT

Managing multiple Kubernetes clusters can be a real headache. I know firsthand how frustrating it can be to juggle multiple contexts, keep deployments consistent, and maintain stability across different environments.

so I found this awesome repo: https://github.com/ahmetb/kubectl-foreach

Run a kubectl command in one or more contexts (clusters) in parallel (similar to GNU parallel/xargs).

Usage

kubectl foreach [OPTIONS] [PATTERN]... -- [KUBECTL_ARGS...]

and created a small helper function to use it. It's a simple way to run kubectl commands across multiple clusters in one go, making life a bit easier.

Here's the function you can add to your bashrc or zshrc:

function kfe() {
  eval "kubectl foreach -q dc1 dc2 dc3 -- $*"
}

Just use kfe followed by your kubectl command, for example:

kfe get pods

and it will execute across dc1, dc2, and dc3 without any hassle. No more repetitive tasks or constant context switching, just a smoother workflow.

I hope this helps lighten the load for anyone dealing with the complexities of Kubernetes. We're all in this together, and sharing little tools like this can make a big difference.

Fixing EKS overprivileged cluster settings

Sun, 28 Jan 2024 00:00:00 GMT

Are you aware that your Amazon EKS cluster could be at risk due to a default permission setting? By default, the IAM role that creates an EKS cluster is granted the system:master RBAC role, providing broad administrative access. This could be a significant security concern.

In the past, rectifying this issue was a cumbersome process, often requiring the recreation of the entire cluster. But now, AWS has streamlined this with new cluster access management APIs. With a simple command:

aws eks delete-access-entry --cluster-name <CLUSTER_NAME> --principal-arn <IAM_PRINCIPAL_ARN>

you can adjust these permissions efficiently without the need to recreate the cluster.

This development is a game-changer in EKS cluster security management, aligning with the principle of least privilege access. It's crucial for EKS users to be aware of these changes and implement them to ensure their clusters are secure.

For more detailed steps and insights on this topic, make sure to read the full AWS blog post here.

My first open source PR was 3 lines

Mon, 04 Dec 2023 00:00:00 GMT

We ran the Security Profiles Operator to ship AppArmor profiles to our Kubernetes nodes. It worked. It also logged the same error over and over:

getting owner profile: the node status owner is of an unknown kind

Millions of lines of it. The profiles still loaded, so nothing was broken. Just loud.

The operator keeps a status object per node for every profile, and a reconciler looks up which profile owns it. That lookup was a switch on the owner's kind. Seccomp was in it. SELinux was in it, twice. AppArmor wasn't.

So on 30 November 2023 I opened issue #1993 and asked three honest questions: is this important, does it mean something is wrong, and can someone guide me to a fix. An hour later I sent the fix myself as PR #1994:

	case "AppArmorProfile":
		prof = &apparmorapi.AppArmorProfile{}

Plus the import. Three lines.

Writing it took minutes. Getting it merged took 4 days, and none of the delay was the fix itself:

The linter failed on import order. Then on a whitespace change I couldn't see. My actual comment on the PR was "I have no idea what fails, can any1 explain?"
The CLA bot blocked it. Corporate or individual? Nobody on our internal Slack knew which one I should sign, so the PR sat until someone decided.
The e2e tests needed a maintainer to type /ok-to-test and /retest for me, more than once.

Sascha Grunert did all of that: pasted the exact linter diff, retriggered the tests, approved it. It merged on 4 December.

The lesson I took from it: a first PR is mostly about learning the project's machinery, the linter, the CLA, the bots, the slash commands. The code is the small part.

Awesome K8s Using Bash

Mon, 27 Nov 2023 00:00:00 GMT

stop suffering when using kubectl

Introduction

Efficient Kubernetes management often entails repetitive kubectl commands. To streamline this, I've developed a set of Bash functions, collectively accessible via a gist.

All those functions will find a pod by partial name: partial_pod_name and run common k8s commands on the found pod.

we effectively scan through the pods in the current namespace and retrieves the name of the first pod that matches a given partial string. Those functions becomes particularly handy when dealing with dynamically generated pod names, where you typically know the deployment name but not the unique identifier of each pod.

The Functions and Their Usage

podNameSeeker [partial_pod_name] # find pod by partial name
shpod [partial_pod_name] # Access a pod (by partial name) using /bin/sh
bashpod [partial_pod_name] # Access a pod (by partial name) using /bin/bash
commandpod [partial_pod_name] [command] # Execute a command inside a pod (by partial name).
logpod [partial_pod_name] # Fetch logs from a pod (by partial name).
debugbashpod <partial_pod_name> #attach netshoot debug container to pod and bash into it"
debugcommandpod <partial_pod_name> <command> # attach netshoot debug container to pod and run command in it

The Debug Functions

netshoot is an awesome project , a container with all the network tools you will ever need. iperf , nc, curl, wget and more. you can use kubectl debug to attach and ephemeral container with it to your pod to debug it see Debugging with an ephemeral debug container

think about a container named "bob" trying to access "myserver.address.com" on port "myport" and falining but you have no networking tools installed on the pod

debugcommandpod bob nc -vz myserver.address.com myport

will attach netshoot to the bob container and run NC to myserver.address.com and check if myport is open

The Codehttps://medium.com/media/9dd483e9b2a7fb251de1a40bc598d401/href

Integrating into Your Shell

To add these functions to your shell environment, you can run the following command in your terminal:

curl -sL https://gist.githubusercontent.com/NahumLitvin/e8e67bb33a0fc0f8f61b28c64303aea1/raw/d35e95e1c408c382739258dd7048bb866277946e/podseeker.sh > ~/podseeker.sh && \
echo '[ -f ~/podseeker.sh ] && source ~/podseeker.sh' >> ~/.${SHELL:t}rc && \
source ~/.${SHELL:t}rc

This one-liner will append the functions to your .bashrc or .zshrc and reload the configuration.

Conclusion

These custom Bash functions are designed to make Kubernetes pod management more efficient and user-friendly. By incorporating them into your workflow, you can reduce time spent on routine commands and focus on more complex tasks.

Effortless Git: Introducing compush, a One-Stop Git Command

Sun, 19 Nov 2023 00:00:00 GMT

"One command to add them all, one command to commit them, One command to push them all, and in the remote branch compush them; In the Land of Bash, where the Git repos lie."

TL;DR

"One command to rule them all: Add, commit, and push in a single stroke. Simply integrate it into your bash configuration, and embark on your coding quest with ease."

What is compush?

compush is a clever shell function that transforms the usual multi-step process of staging, committing, and pushing changes into a single, streamlined operation. Its standout feature is its ability to work from any subdirectory within your Git repository, eliminating the hassle of navigating back to the root directory. This functionality is a real time-saver, allowing you to concentrate more on coding and less on command-line intricacies.

How compush Enhances Your Workflow

When you call compush into action, it kicks off a series of operations:

Finds the Git Repository's Root: It intelligently locates the top-level directory of your Git repository, or uses the current directory if it's not in a Git repo.
Stages Changes: Automatically adds all changes from the top-level directory to the staging area.
Commits Changes: Commits the staged changes, using either your provided commit message or an empty message if none is specified.
Pushes to Remote: Effortlessly pushes the commit to the origin remote of the current branch and sets up tracking.

Setting Up compush

To incorporate compush into your workflow, add this function to your shell configuration file (.zshrc for Zsh users or .bashrc for Bash users):https://medium.com/media/e6e7c5168b04ddaa25f18e4f9ffba6fe/href

Activating compushTo activate compush, source your configuration file:

For Zsh: source ~/.zshrc
For Bash: source ~/.bashrc

Using **compush **, within any directory of your Git repository, execute:

comush "Your insightful commit message here"

To a commit without a message:

compush

Git Takes Sha Instead Of Remote Branch

Thu, 26 Nov 2020 00:00:00 GMT

Was debugging a weird production issue. the version deploy was absolutely wrong.

Premise:

When our CI build the product it creates a branch. with the build number. in our CI it was a running integer but in the repo I created for the example it will be 2c4f227

by coincidence it was starting the same way as a random commit on the main branch.

on the deploy stage we clone the repo (on main branch by default)

git clone git@github.com:NahumLitvin/GitTakesShaInsteadOfRemoteBranch.git

git checkout 2c4f227

and thats the result I got:

Note: checking out '2c4f227'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

HEAD is now at 2c4f227 Initial commit

git checked out the commit _2c4f227 instead of the branch _2c4f227!

Solution:

git checkout origin/2c4f227

will checkout the correct branch

another more concise way is to use the new git switch command available from git 2.29.0

git switch

git switch 2c4f227 warning: refname '2c4f227' is ambiguous. Switched to branch '2c4f227' Your branch is up to date with 'origin/2c4f227'.

git switch will _'Switch to a specified branch. The working tree and the index are updated to match the branch. All new commits will be added to the tip of this branch.' when: ** git checkout _'Updates files in the working tree to match the version in the index or the specified tree. If no pathspec was given, _git checkout** will also update _HEAD to set the specified branch as the current branch.'

additional reading

see this stack overflow Q&A for a better explanation: https://stackoverflow.com/questions/57123031/git-checkout-commit-hash-vs-git-checkout-branch/