apk upgrade -U -a && rc-service sshd restart

On 2026-11-05, Alpine published fixed openssh packages for every branch from 3.16 to edge, and OpenSSH released 10.6p2. The bug is CVE-2026-123158. On an affected host, a person who can reach the SSH port can log in to an account that has no password, without a password. The cause, in one sentence: when sshd is built without PAM, an error value from musl's crypt() could be treated as a match for accounts that have no password.

This post tells you how to find out if a host or a container is affected, how to fix it, and what to look for in your logs. It does not show how to trigger the bug.

TLDR

• You are affected only if four conditions are all true. In our tests, a fresh Alpine container with openssh-server failed two of them.
• The fix is apk upgrade -U -a, then a restart of sshd. The upgrade alone does not replace the running daemon.
• If you cannot upgrade, set PasswordAuthentication no and restart sshd.
• Our read-only script checks the four conditions, the package version and the restart for a host or for every running container.

Prerequisites

• Root on the Alpine host, or access to docker exec for containers
• A second SSH session open before you change sshd settings

Who is affected

The Alpine advisory says you are affected only if all of these are true:

1. sshd is built without PAM. The openssh-server package is. The openssh-server-pam package is affected too if UsePAM is no, which is the OpenSSH default when the config does not set it.
2. PasswordAuthentication yes, which is the OpenSSH default.
3. An account has * in its password field. On Alpine base images root is root:*, but the default PermitRootLogin prohibit-password blocks password logins for root. Key-only users, the alpine user on tiny-cloud images, and directory users through musl-nscd (they have no shadow entry) also count.
4. At least one local account has a $5$ or $6$ hash. One user with a password is enough.

You are not affected if PasswordAuthentication is no, if you use openssh-server-pam with UsePAM yes, if no account has a $-style hash, or if your containers do not run sshd. Locked accounts (! in the password field) are not affected. Instances where cloud-init sets PasswordAuthentication no are not affected.

These are the fixed openssh versions from the advisory:

Check one host

We wrote a small POSIX sh script that reads these facts and changes nothing:

The-DevOps-Daily/alpine-sshd-check on GitHub

It reads the branch from /etc/alpine-release, the installed package version from apk, the effective settings from sshd -T, and whether the running sshd is still the sshd binary on disk. From /etc/shadow it reads only the type of each password field: it counts * fields and $5$/$6$ fields. It never prints a hash or a user name, and it does not connect to sshd. Because it runs as root, it executes only the packaged sshd binaries in /usr/sbin, and only if root owns them and no one else can write to them. If openssh can come from edge on a stable host (an untagged edge repository, or a pin such as openssh-server@edge), it compares against the edge fix, and it says "cannot tell" when it cannot find where the package came from. Exit codes are 0 not affected, 1 affected, 2 fixed but restart needed, 3 cannot tell. Add --json for machine-readable output.

sudo sh check.sh

We ran it in throwaway containers on Alpine 3.21, 3.22, 3.23, 3.24 and edge after apk add openssh-server. sshd ran with --network none, so no port was open to anyone. Every branch installed the fixed version, and every run exited 0. The run below is the closest one to an affected host: Alpine 3.24 with a key-only user set to * and one user with a random password, so all four conditions are true. The version is the only thing that keeps it safe:

check.sh in the Alpine 3.24 test container

# all four conditions true, fixed package installed
$ docker exec -i asc-users sh -s < check.sh
alpine-sshd-check: CVE-2026-123158
Alpine release:     3.24.4 (branch 3.24)
sshd binary:        /usr/sbin/sshd (package openssh-server)
Installed openssh:  10.3_p1-r2
Fixed in:           10.3_p1-r2 (installed is fixed: yes)
[1] non-PAM sshd:                  yes
[2] PasswordAuthentication yes:    yes
[3] account with '*' that can use a password login: yes (2 '*' accounts, 1 usable; root is '*': yes, PermitRootLogin prohibit-password)
[4] local $5$/$6$ hash:            yes (1 found)
sshd running:       yes, started 2026-11-05T21:49:52Z, binary on disk changed 2026-11-05T21:49:35Z, restart needed: no
note: root is '*' but PermitRootLogin is 'prohibit-password'; root counts only with PermitRootLogin yes (an account that can doas to root still matters)
Result:             NOT AFFECTED: fixed openssh installed (exit 0)

Two things we found while we ran it:

• On 3.24, adduser -D deploy wrote ! (locked), not *. A locked account is not affected, but it cannot log in with a key either. If you changed key-only users to * so that key login works, those are the accounts that matter.
• OpenSSH 10.6 prints sshd -T keywords in mixed case (PasswordAuthentication yes). Older versions print lower case. The first version of our script matched only lower case and reported "unknown" on edge. Use grep -i in your own checks.

Check containers

Containers that do not run sshd are not affected. To find the ones that do, scan-containers.sh lists the running containers, skips those that are not Alpine or have no sshd process, and pipes check.sh into the rest with docker exec -i ... sh -s. Nothing is copied into the container. Because it runs each container's own sh as root inside that container, scan only containers whose images you trust.

scan-containers.sh

$ sh scan-containers.sh
CONTAINER      IMAGE          ALPINE                INSTALLED    FIXED        EXIT  RESULT
asc-users      asc-demo:3.24  3.24.4                10.3_p1-r2   10.3_p1-r2   0     not affected
asc-pa-no      asc-demo:3.24  3.24.4                10.3_p1-r2   10.3_p1-r2   0     not affected
asc-edge       asc-demo:edge  3.25.0_alpha20260805  10.6_p2-r0   10.6_p2-r0   0     not affected
asc-3.24       asc-demo:3.24  3.24.4                10.3_p1-r2   10.3_p1-r2   0     not affected
asc-3.23       asc-demo:3.23  3.23.8                10.2_p1-r1   10.2_p1-r1   0     not affected
asc-3.22       asc-demo:3.22  3.22.8                10.0_p1-r11  10.0_p1-r11  0     not affected
asc-3.21       asc-demo:3.21  3.21.10               9.9_p2-r1    9.9_p2-r1    0     not affected

If you build images that install openssh-server, a rebuild can reuse a cached apk add layer with the old package. Rebuild with docker build --pull --no-cache, then replace the running containers. A restart of the old container does not change its package.

Fix it, and restart sshd

apk upgrade -U -a
rc-service sshd restart

The restart is not optional. The package upgrade replaces the files on disk, but it does not restart the sshd that is already running, and the advisory says to restart it after the upgrade. check.sh checks whether the running sshd is still the same file as the sshd binary on disk. An upgrade writes a new file, so a daemon that was not restarted is a different file, and the script exits 2. Run it again after the restart.

On edge, the fix comes with OpenSSH 10.6p2. If you jump from an older 10.x, read our post on what OpenSSH 10.6 changed first: it refuses some usernames and changes ssh -C.

Workaround: turn off password logins

If you cannot upgrade now, turn off password authentication. Either step closes the hole.

Warning

Make sure that key login works first. Keep your current session open, and test a new key login in a second session before you close it.

mkdir -p /etc/ssh/sshd_config.d
echo 'PasswordAuthentication no' > /etc/ssh/sshd_config.d/00-no-passwords.conf
sshd -t && rc-service sshd restart
sshd -T | grep -i passwordauthentication   # must print: passwordauthentication no

Alpine's default sshd_config reads /etc/ssh/sshd_config.d/ from 3.19 onward. On 3.18 and older, change the line in /etc/ssh/sshd_config itself; the advisory has a sed command for that. A Match block can turn password logins back on for some users or addresses, and sshd -T without -C does not show that, so read your Match blocks too. In our 3.24 test container, the drop-in changed condition 2 to no.

The tiny-cloud alpine user

The official tiny-cloud images (qcow2, VHD and AMI) from 3.18.11 to 3.24.3 have an alpine user with * in the password field and passwordless doas. They are safe on first boot. They become exposed as soon as any $6$ hash is on the machine, for example when an admin sets a console password or user-data adds a user with hashed_passwd. Then an attacker can get a shell as alpine and use doas to become root.

Images from 3.21.10, 3.22.8, 3.23.8 and 3.24.4 contain the fixed package. Instances that started from older images do not get the fix from a new image: upgrade openssh on each one, or turn off password logins.

What to look for in logs

Look for password logins to accounts that have no password. On a host with syslog, that is /var/log/messages. For a container that runs sshd -e, it is docker logs.

# accounts with '*' in the password field (names only)
awk -F: '$2 == "*" { print $1 }' /etc/shadow

# password logins; compare the user names with the list above
grep 'sshd.*Accepted password for' /var/log/messages
docker logs <container> 2>&1 | grep 'Accepted password for'

Any Accepted password for alpine, for root, or for a key-only user is a bad sign. On exposed machines, also review doas use, new users, new files, and changes to authorized_keys files. If you cannot rule out such a login, treat the machine as compromised and rebuild it.

What we could not check

• A vulnerable package or a real upgrade. Alpine replaced the packages on every branch the same day, so each fresh install got the fixed version. Exit codes 1 and 2 are simulations in the repo's test-exit-codes.sh: a copy of the script with a raised fixed version, and the same sshd binary renamed over /usr/sbin/sshd while sshd runs. Both gave the expected code.
• openssh-server-pam, musl-nscd users, tiny-cloud images and branches 3.16 to 3.20. We did not measure them.
• Other musl systems. The CVE record covers sshd before 10.6p2 on Linux with musl and without PAM, and names Alpine as the example. We did not check other musl-based distributions or firmware builds.

Sources: the Alpine advisory, the OpenSSH 10.6p2 release notes and the CVE record.

The binary adds a little to startup: contrib needed 110 to 160 ms more than the small build on the droplet, and 300 to 500 ms more on the Pi. The detector list in your config can add seconds. We started both builds with cloud detectors for clouds the host was not on, in four network setups. In every list without akamai, each start waited for the slowest detector. When that detector got no answer, it waited for the whole timeout, which is 5 s by default. That happened on a Raspberry Pi with no metadata service. With a list of 13 cloud detectors it also happened on a DigitalOcean droplet, because two of them ask addresses that DigitalOcean does not answer. The detectors run in parallel, so where nothing answered, the list with 3 cloud detectors and the list with 13 cost the same 5 s.

timeout: 1s cut the wait to about 1 s in most cases. Two detectors do not obey it. digitalocean ignores the deadline, so where nothing answers it waits its own 2 s. akamai makes its network call while the processor is created. In all four setups we tested, none of them on Akamai, the Collector did not start. It exited at once where the request got an answer or a fast error, and after 30 s where it got no answer.

On 29 October the OpenTelemetry blog showed how to compile detectors out with ocb and reported a Kubernetes distribution about 20% smaller. This post measures three things that page does not cover. What does the same filter do to a small build? What does the build do to memory at idle? And what does the detector list cost at startup on hosts that are not on the clouds the config names? Every number below comes from a file in the companion repo.

TL;DR

• Size. On amd64, contrib is 420.3 MB, the core otelcol release is 183.8 MB, our agent with every detector is 124.1 MB, and the same agent with only env and system is 64.3 MB. On this small build, the detector filter alone removed 48% of the binary and 64 Go modules: the AWS SDK, the Google Cloud clients, the Docker client, the Consul API and four more cloud metadata clients. arm64 gives the same picture.
• Memory. After 10 s idle: contrib 199.8 MiB RSS, core 98.4 MiB, our agent 61.5 MiB, the agent without cloud detectors 45.1 MiB. About four fifths of each number is file-backed memory (RssFile), which grows with the binary. The anonymous part (RssAnon) was 34.8 MiB for contrib and 8.0 MiB for the small build.
• Startup with no cloud detectors: about 190 to 220 ms for contrib and 45 to 85 ms for the small agents on the droplet, and 480 to 610 ms against 110 to 270 ms on the Pi.
• Startup with cloud detectors for other clouds: [env, system, ec2, gcp, azure] held startup for the full 5 s timeout where 169.254.169.254 got no answer. Where the connect failed at once, it held startup for 4.1 s, because the AWS SDK retries the connect. On the droplet those three cost under 4 ms each, but the 13-cloud list still took 5.1 s there, because alibaba_ecs and scaleway ask other addresses.
• The timeout setting bounds the wait: 1 s, 5 s or 10 s plus normal startup. The exceptions are digitalocean, which waits its own 2 s whatever you set when nothing answers, and akamai.
• No start at all. akamai in the list: exit 1 in all four setups, after 30 s in the two with no answer. fail_on_missing_metadata: true: exit 1 after the 5 s timeout. A detector that ocb compiled out but the config still names: validate passes, then a nil pointer panic at start. The processor's name for a detector in the ocb manifest (ec2 instead of aws_ec2): the build stops.
• What to do. List only the detectors for the platform the host runs on, and build with the same list in dist.resource.detection.detectors. If one config must cover several clouds, keep the list short, set timeout to 1 or 2 s, and leave out akamai.

Prerequisites

• An x86_64 Linux host for the builds, with about 3 GB of free disk. ocb cross-compiles the arm64 binaries there.
• curl, python3 and ip (iproute2). Root only for the two network namespaces.
• An arm64 Linux host if you want the second architecture. We used a Raspberry Pi with 4 cores and less than 4 GB of memory.
• About an hour for a full run. The cases that fail after 30 s take most of it.

What we built and where we ran it

We downloaded the official 0.164.0 releases of otelcol-contrib and otelcol for both architectures and checked each tarball against its published .sha256. Then ocb 0.164.0 built two versions of one small agent. It has the components a VM agent uses:

• receivers: OTLP, hostmetrics and filelog
• processors: batch, memory_limiter and resource_detection
• exporters: debug, OTLP and OTLP/HTTP
• extension: health_check

otelcol-agent uses the ocb default, so every detector is compiled in. otelcol-agent-envsys adds the ocb setting that compiles out every detector you do not list:

# manifests/agent-envsys.yaml (excerpt, comment on detectors added)
dist:
  name: otelcol-agent-envsys
  version: 0.164.0
  build_tags: 'grpcnotrace' # same tag as the contrib release manifest
  resource:
    detection:
      detectors: # ocb 0.163.1 and later: every detector not listed gets an omit_detector_<name> build tag
        - env:
        - system:

go version -m on the release binaries reports go1.26.10, so we pinned the same Go for our builds. go version -m and file show -trimpath=true, CGO_ENABLED=0 and stripped symbols for every binary (results/buildinfo/).

Every start ran the same pipeline: OTLP in, batch, debug out. The harness gave each start a clean environment with only PATH and an empty HOME. No OTEL_* or AWS_* variable and no ~/.aws file could change what a detector found. The Pi runs also set TZ=UTC, so their logs use UTC like the droplet's; the droplet runs used an earlier revision of the harness without it. Only the resource_detection block changed between configs. Each case ran 5 times in a shuffled order, after one warm-up start per binary that we left out of the numbers.

• DigitalOcean droplet 4 vCPU, 8 GB, amd64
  • host network metadata server answers
  • blackhole namespace packets dropped
  • noroute namespace connect fails at once
• Raspberry Pi 4 cores, under 4 GB, arm64
  • home network router drops 169.254.169.254

The four network setups matter more than the two architectures:

The-DevOps-Daily/otelcol-ocb-vs-contrib on GitHub

Size: what ocb removes

Binary size, linux/amd64, OpenTelemetry Collector 0.164.0

results/sizes.tsv. Stripped, -trimpath, CGO_ENABLED=0, go1.26.10. The two agents have the same 10 components; only the compiled-in detectors differ.

The contrib manifest lists 246 components and 9 config providers, and the binary links 1,088 Go modules. The core release is not small either. Its 30 components include the Kafka and Prometheus receivers and exporters, and it also ships resource_detection with every detector. Our agent with every detector links 289 modules, and the detector filter takes it to 225.

The filter saves more on a small build than on a big one. The OpenTelemetry blog measured the Kubernetes distribution and got 182 MB to 147 MB with 13 fewer dependencies. On our agent it removed 59.8 MB of 124.1 MB (48%) on amd64, 52.9 MB of 112.7 MB (47%) on arm64, and 64 modules. A big distribution needs most of those SDKs for other components, so removing the detectors does not remove the SDKs. In a small agent, nothing else needs them. results/deps/removed-by-detector-filter-amd64.txt lists the 64: 15 aws-sdk-go-v2 and smithy-go modules, the Google Cloud compute, auth and API clients, the Docker (moby) client, the Consul API with its HashiCorp libraries, and the Hetzner, Scaleway, Linode and DigitalOcean clients.

For containers, the official otel/opentelemetry-collector-contrib:0.164.0 image is 116.1 MB compressed on amd64 (results/images.tsv). We did not build an image of the small agent. Its binary compresses to 20.3 MB with gzip -6, so a FROM scratch image would be near that, but that is an estimate, not a measurement.

One build difference favours our side. The contrib manifest asks for the grpcnotrace build tag and our builds use it, but the release binaries record no build tags in go version -m. We built otelcol-agent-notag to put a size on it: 124.23 MB against 124.11 MB, so the tag is worth 0.1%.

Memory at idle

Resident memory after 10 s idle, droplet, amd64

Median of 5 starts, /proc//status, config without resource_detection, results/summary.md. RssFile is resident file-backed memory; RssAnon is resident anonymous memory (heap, stacks, runtime).

Resident memory follows the binary size, but not one to one. Contrib held 199.8 MiB, 4.4 times the 45.1 MiB of the small agent. About four fifths of every number is RssFile, resident memory backed by files. It grows with the binary: 165.2 MiB for contrib, 37.0 MiB for the small agent. That points at the binary's own pages, but we did not map them one by one. The kernel can drop clean file-backed pages under memory pressure and read them again later. RssAnon is anonymous memory, which it cannot drop without swap. There contrib held 34.8 MiB against 8.0 MiB. A Go program runs the init code of every package it links before main, so a likely part of that difference is setup work for code that this pipeline never uses. We did not profile the heap to prove it.

Detection with [env, system] added little: contrib went to 205.4 MiB and the small agent to 47.8 MiB. The Raspberry Pi gave the same order on arm64: contrib 183.8 MiB with 34.7 MiB anonymous, the small agent 40.6 MiB with 8.0 MiB.

We did not test what happens to RssFile under memory pressure, or how a cgroup memory limit counts it. Read the RssAnon column as the floor and VmRSS as what top shows you.

Startup: the slowest detector sets the time

Median time from exec to the log line "Everything is ready", 5 starts each, from results/summary.md. Each cell is the small agent with every detector, then contrib in brackets:

Without a network wait, contrib needs about 110 to 160 ms more than the small agent to start on the droplet, and about 300 to 500 ms more on the Pi. That is real but small, and it matters only if you start collectors very often, for example one per short CI job.

The wait comes from three things in the processor source (processor/resourcedetectionprocessor at v0.164.1):

1. Detectors run in parallel. detectResource starts one goroutine per detector and then reads their results in order (internal/resourcedetection.go, lines 203 to 222). The wait is the slowest detector, not the sum.
2. The whole detection has one deadline: timeout. The default is 5 s (factory.go, lines 153 to 157). With the default retry.max_elapsed_time: 0, Refresh puts that timeout on the context that every detector gets (internal/resourcedetection.go, lines 159 to 168).
3. Startup waits for it. The processor's Start runs the first detection and returns its error (resourcedetection_processor.go, lines 36 to 40). The Collector logs "Everything is ready" only after every processor has started.

The processor also records how long each detector took, in its own otelcol_resourcedetection_detector_duration histogram. We read it from the Collector's Prometheus endpoint after each start. This is the 13-cloud list in the blackhole namespace:

Time per detector, 13 VM cloud detectors, blackhole namespace, timeout 5s

Median of 5 starts of the small agent with all detectors, from otelcol_resourcedetection_detector_duration in results/runs/droplet-blackhole-extra.jsonl. Every detector reported outcome=success: "not on this cloud" is a success with no attributes.

The same histogram tells you which detector held each of the other setups:

How each detector probes decides how long it waits:

• azure sends one GET with a plain &http.Client{} that has no timeout (internal/metadataproviders/azure/metadata.go, lines 33 to 38). When the packets get no answer, it waits for the processor's deadline. Then it returns an empty resource and no error (internal/azure/azure.go, lines 55 to 63).
• ec2 uses the AWS SDK's IMDS client: a 250 ms dial timeout, a 500 ms response header timeout (aws-sdk-go-v2 feature/ec2/imds v1.20.1, api_client.go, lines 277 and 278), and up to 3 attempts per request (internal/aws/ec2/config.go, lines 16 to 19). A connect that fails at once is retried with backoff, which is why ec2 took 4.0 s in the noroute namespace. With no answer at all and a 10 s timeout, it gave up by itself after 5.5 s.
• gcp calls metadata.OnGCEWithContext, which races a DNS lookup of metadata.google.internal against an HTTP probe. On a host that does not look like a Google VM, it trusts the first answer (cloud.google.com/go/compute/metadata v0.10.0, metadata.go, lines 456 to 480). A fast NXDOMAIN or a fast DNS failure ends it.
• digitalocean ignores the context: its Detect takes _ context.Context (internal/digitalocean/digitalocean.go, line 51), and the DigitalOcean metadata client has a fixed 2 s timeout (digitalocean/go-metadata, client.go, lines 25 and 64).

We started from this claim: on a host that is not on EC2, Google Cloud or Azure, each cloud detector adds a wait at startup. Two results would prove it wrong: no wait at all, or the same wait for 3 detectors and for 13. We got the second, so it was half wrong. The detectors do not each add a wait. During detection they overlap, and the slowest one sets the time. But a host on a cloud with a metadata server is not safe either, because not every cloud uses 169.254.169.254.

The timeout setting

Start to "Everything is ready" with [env, system, ec2, gcp, azure]

Median of 5 starts per point, results/summary.md. The droplet host answers on 169.254.169.254; the blackhole namespace and the Pi do not. The noroute namespace fails the connect at once.

Where nothing answers, the wait is the timeout plus normal startup, on both architectures. In the noroute namespace it stops at 4.1 to 4.2 s, where ec2 gives up. timeout: 1s brought every three-cloud case that waited down to 1.1 to 1.2 s.

The 13-cloud list with timeout: 1s did not get there. In the blackhole namespace it started in 2,097 ms (contrib 2,234 ms), and on the Pi in 2,175 ms (contrib 2,410 ms), because digitalocean took its 2 s anyway.

A short timeout has a cost that we could not measure. On a real EC2, Google Cloud or Azure VM, a detector that needs more than 1 s gets cut off. It returns an empty resource with no error, and your telemetry has no cloud attributes. We did not run on those clouds, so we cannot say whether 1 s is enough there. Most examples in the processor README use timeout: 2s.

Four ways the Collector does not start

1. ocb build unknown detector name: stops
2. otelcol validate passes a compiled-out detector
3. create processor akamai fetches a token
4. Start: detection parallel, one deadline
5. Everything is ready

1. The akamai detector

Of the 14 VM cloud detectors we ran, akamai is the only one that waited or failed while the processor was created. Its NewDetector calls newAkamaiClient(context.Background()) (internal/akamai/akamai.go, lines 43 to 57), and the Linode metadata client fetches a token inside NewClient (linode/go-metadata v0.3.1, client.go, lines 88 to 91). This happens before detection, so timeout does not apply, and a failure there is a failure to build the pipeline.

On the droplet, DigitalOcean's metadata server answers the token request, but not with a token, and the Collector exits at once:

results/runs/droplet-host-extra.jsonl, otelcol-contrib, configs/akamai.yaml

$ bin/linux-amd64/otelcol-contrib --config configs/akamai.yaml
Error: failed to build pipelines: failed to create "resource_detection" processor, in pipeline "traces": failed creating detector type "akamai": failed to refresh metadata token: failed to generate metadata token: no token returned from API
# exit code 1, 0.2 s after exec

In the noroute namespace it exited at once with connect: network is unreachable. In the blackhole namespace and on the Pi, the token request first waited for a 30 s dial timeout. The Collector exited after 30.1 s in the namespace and 30.2 s on the Pi (contrib: 30.2 s and 30.5 s). With timeout: 1s it still took 30 s.

2. fail_on_missing_metadata: true

This setting makes a cloud detector return an error when its metadata service is missing, instead of an empty resource. Retry is on by default (factory.go, lines 142 to 151), so the processor retries the detector with backoff until timeout, and then Start returns the error. With [env, system, ec2, gcp, azure], the Collector exited with code 1 after 5.1 to 5.3 s in all three droplet setups. The config comment says the setting is for nodes that boot before their metadata service is ready (config.go, lines 63 to 70). It also turns "not on EC2" into "does not start".

3. A detector that ocb compiled out

This is the trap in the new ocb feature. otelcol-agent-envsys has only env and system. We gave it the three-cloud config:

results/break/amd64-validate.txt and results/break/amd64-start.txt

$ bin/linux-amd64/otelcol-agent-envsys validate --config configs/clouds3.yaml
exit code: 0
$ bin/linux-amd64/otelcol-agent-envsys --config configs/clouds3.yaml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x130a757]

goroutine 57 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor/internal.(*ResourceProvider).detectWithRetry.func2()
	github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor@v0.164.1/internal/resourcedetection.go:314 +0xb7
...
exit code: 2

In v0.164.1 each *_omit.go stub returns nil, nil from NewDetector, for example internal/aws/ec2/ec2_omit.go. The processor keeps the nil detector and calls it. This is issue #52519, open when we wrote this. On main, #52063 changes the stubs so they register no factory. It was merged after v0.164.1. With it, the processor returns detector "ec2" is not compiled into this binary with the list of compiled detectors. Until a release has that change, validate does not catch the mismatch, and the first sign is a crash.

4. The processor's names in the ocb manifest

ocb and the processor do not use the same detector names. The processor config says ec2, nova and aks. The ocb manifest wants the build tag names aws_ec2, openstack_nova and azure_aks (cmd/builder/internal/builder/config.go, lines 117 to 151). This one fails early, which is the good kind of failure:

results/build/agent-badname-amd64.log

$ ocb --config manifests/agent-badname.yaml
Error: invalid configuration: ec2: unknown resource detector

What to put in the manifest and the config

Make the two lists match, each in its own names:

# ocb manifest: build tag names
dist:
  resource:
    detection:
      detectors:
        - env:
        - system:
        - aws_ec2: # the processor calls this one "ec2"

# Collector config: processor names, the same detectors
processors:
  resource_detection:
    detectors: [env, system, ec2]
    timeout: 2s # the most a missing metadata service can add to startup

• Use one list per platform. A fleet on EC2 does not need azure, and a fleet on bare metal needs neither.
• If one config must cover several clouds, keep the list short and set timeout to 1 or 2 s. Expect digitalocean to add 2 s, whatever the timeout, on hosts where its request gets no answer.
• Leave akamai out unless the hosts run on Akamai Cloud.
• Keep fail_on_missing_metadata off unless "no metadata" should mean "do not start".
• Use the name resource_detection. In 0.164 the old name resourcedetection still works, with a deprecation warning (results/break/amd64-legacy-name.txt).
• Until the #52063 change is in a release, start each new custom build once with its real config before you ship it. validate is not enough.

What we could not conclude

• On a real AWS, Google Cloud or Azure VM. We had no instances there, so we did not measure how long a detector takes when it succeeds, or whether a 1 s timeout cuts it off.
• The container image of the small agent. We report the compressed size of the official image and the gzip size of our binary. We did not build or push an image.
• Memory under pressure. RssFile can be reclaimed. We did not measure how much the kernel takes back, or how a cgroup limit counts it.
• Small startup differences. Each case has 5 starts. The 110 to 160 ms between contrib and the small agent repeats in every droplet setup. The Pi also ran other work, and three of its starts took 4.6 to 22 s. In all three, the time went before the Collector wrote its first log line, which points at reading the binary from the SD card, not at detection. We did not confirm that. They move the medians little, but they are in the min and max columns of results/summary.md.
• Slow DNS. DNS failed at once in the namespaces and answered on the Pi, so gcp and tencent_cvm were fast everywhere. A host whose DNS times out would make them wait too. We did not test that.
• Other versions. Everything here is 0.164.0 and 0.164.1. The akamai and omit behaviour may change in the next release.

Related reading:

• Distributed tracing with OpenTelemetry runs a Collector end to end.
• OpenTelemetry graduated covers which vendor agents a Collector can replace.
• Trace context in environment variables follows a trace through a CI pipeline.

Summary

A custom Collector with only the components you use is 6.5 times smaller than contrib and holds under a quarter of its memory. On a small build, compiling out the unused detectors is half of the size saving. The binary costs tens to hundreds of milliseconds at startup. The detector list can cost seconds. Apart from akamai, the cloud detectors you list for clouds the host is not on share one wait: the slowest probe. Where that probe gets no answer, it usually runs to the full timeout, even on a host on another cloud. digitalocean waits its own 2 s instead, even past a shorter timeout. Keep the list to the platforms you run on, and build with the same list. Set a short timeout if the list must stay long. Start each new binary once with its real config. Leave akamai out unless you run on Akamai.

We ran the same pods on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. On the three older versions, the empty block moved the pod into the BestEffort cgroup slice. The kubelet set its oom_score_adj to 1000, the highest value the kernel accepts. Next to a busy neighbour, a pod that requested half a CPU got 0.06 to 0.08 cores instead of 0.57 to 0.58. A namespace quota that blocks BestEffort pods rejected it. On 1.37.1 the same pods kept their Burstable and Guaranteed classes.

One thing did not change, and the research we started from got it wrong. It said these pods are first in line for kubelet eviction under memory pressure. They are not. The kubelet's eviction ranking compares memory usage with the memory request. For an empty block, that request still comes from the containers. The QoS class is not part of the ranking, and the kubelet evicted our pods in the same order on every version.

TL;DR

• On k3s v1.34.12, v1.35.9 and v1.36.5, a pod with container requests and limits plus spec.resources: {} reported qosClass: BestEffort. The Guaranteed variant did too. On v1.37.1 they reported Burstable and Guaranteed. With the new PodLevelResourcesFixKubeletQOSClass feature gate turned off, 1.37.1 went back to BestEffort.
• The node settings followed the class. oom_score_adj went from 935 (Burstable) and -997 (Guaranteed) to 1000, and the pod cgroup moved to kubepods-besteffort.slice with cpu.weight 1 and no pod-level memory.max. The container-level limits stayed in place.
• A neighbour pod kept both vCPUs busy. Next to it, the BestEffort-classified pod got a median of 0.061 to 0.083 cores on the three affected versions. The same pod without the empty block got 0.573 to 0.580 (so 10.6% to 14.3%). On 1.37.1 the two were equal.
• A ResourceQuota with scope BestEffort and pods: "0" rejected the pod on 1.34 to 1.36 and admitted it on 1.37.
• The kubelet's memory-pressure eviction order was the same on all four versions, in all 12 repetitions: the true BestEffort pod first, then by memory usage relative to the container request. The empty block did not move a pod forward.
• If you run GitLab Runner on Kubernetes, upgrade to 19.5.0 or later. As of 2 November 2026 the latest Kubernetes patch releases for 1.34, 1.35 and 1.36 still have the bug.

Prerequisites

• A cluster on Kubernetes 1.34, 1.35 or 1.36 (kubectl version), or a machine where you can run the lab
• kubectl and jq
• If you run GitLab Runner with the Kubernetes executor: its version (gitlab-runner --version)
• To run the lab: a DigitalOcean token (referral link); one s-2vcpu-4gb droplet for about an hour costs a few cents

Where the empty block comes from

Pod-level resources (KEP-2837) let you set CPU and memory for the pod as a whole, not only per container. The feature gate PodLevelResources went alpha in 1.32 and beta, on by default, in 1.34. Our Kubernetes 1.36 post covers the in-place resize of those values.

The QoS calculation has a shortcut for them. This is ComputePodQOS at v1.34.0, v1.35.0 and v1.36.0, and still at the latest patches, v1.34.13, v1.35.10 and v1.36.6:

// pkg/apis/core/v1/helper/qos/qos.go, v1.36.6
// When pod-level resources are specified, we use them to determine QoS class.
if utilfeature.DefaultFeatureGate.Enabled(features.PodLevelResources) &&
    pod.Spec.Resources != nil {

An empty block is not nil. It has no requests and no limits, so the function returns BestEffort. The loop over the containers never runs. The issue that reported this, #135082 from November 2025, says that almost everywhere else Kubernetes does not differentiate between nil and empty pod-level resources. The helper for that, IsPodLevelResourcesSet, treats {} as not set.

The fix is PR #137150, merged on 15 July 2026 for 1.37. It adds the condition IsPodLevelResourcesSet(pod) behind a new gate, PodLevelResourcesFixKubeletQOSClass, beta and on by default from 1.37. It was not backported. The patch releases of 27 October 2026, v1.34.13, v1.35.10 and v1.36.6, still have the old condition. The rest of that release is in our Kubernetes 1.37 review; PodLevelResources itself stays beta in 1.37.

GitLab Runner added pod-level resources to its job pods in 18.10.0 (MR !5922). At every tag we read (18.10.0, 19.3.5 and 19.4.3), the helper that builds the block returns a non-nil object:

// executors/kubernetes/kubernetes.go, GitLab Runner v19.4.3
func (s *executor) podResourcesReference() *api.ResourceRequirements {
	resources := api.ResourceRequirements{}

	if s.configurationOverwrites != nil {
		resources.Limits = s.configurationOverwrites.podLimits
		resources.Requests = s.configurationOverwrites.podRequests
	}

	return &resources
}

If the runner config sets none of pod_cpu_request, pod_memory_request, pod_cpu_limit and pod_memory_limit, both maps are empty. The pod then goes to the API server with resources: {}. The MR that fixed it, !7346, says the same: job pods "were showing qosClass: BestEffort instead of the expected Burstable", and after the fix .spec.resources "should now be empty/absent (was {})". It merged on 1 October 2026 and shipped in 19.5.0. The 19.4.3 and 19.3.5 tags of 28 October 2026 still have the old function. The class matters only when the runner config gives the job containers requests or limits. Without them, the pods were BestEffort anyway.

The lab

The lab runs on one DigitalOcean droplet: s-2vcpu-4gb in fra1, Ubuntu 24.04, cgroup v2. A cloud firewall allows SSH from one address and nothing else. The Kubernetes API port is never exposed, and kubectl runs on the node over SSH. For each version, a script removes k3s, installs the pinned release, and runs five stages. The kubelet gets an explicit hard eviction threshold, memory.available<2Gi. It also gets a 30 second pressure transition period instead of the default 5 minutes, so repetitions do not wait for the node condition to clear.

1. pod spec resources: {} + container requests
2. ComputePodQOS sees a block: BestEffort
3. IsPodLevelResourcesSet sees {}: not set
4. oom_score_adj 1000
5. cgroup slice besteffort, cpu.weight 1
6. ResourceQuota scope BestEffort
7. eviction ranking container requests

Connections:

• pod spec -> ComputePodQOS
• pod spec -> IsPodLevelResourcesSet
• ComputePodQOS -> oom_score_adj
• ComputePodQOS -> cgroup slice
• ComputePodQOS -> ResourceQuota scope
• IsPodLevelResourcesSet -> eviction ranking

The pods:

plr-burstable is the control for the feature itself: a real pod-level block works as documented on every version. The empty block is the only problem.

We wrote down the claim and what would disprove it before the droplet existed, and committed it first (CLAIM.md). Every recorded file carries a hash of the manifests and node scripts, and the summary script refuses to mix files with different hashes.

The-DevOps-Daily/k8s-empty-pod-resources on GitHub

Finding 1: the class flips on 1.34 to 1.36

The API server stored the empty block exactly as written. On v1.34.12, v1.35.9 and v1.36.5 the kubelet then reported BestEffort for both emp-* pods, while their controls kept Burstable and Guaranteed. On v1.37.1 the emp-* pods got the same class as their controls:

results/demo-k3s-v1.36.5.txt and results/demo-k3s-v1.37.1.txt

# k3s v1.36.5+k3s1
# ... kubectl apply and wait omitted; the whole session is in the file
$ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'
NAME             POD-LEVEL   QOS
ctl-burstable    <none>      Burstable
ctl-guaranteed   <none>      Guaranteed
emp-burstable    map[]       BestEffort
emp-guaranteed   map[]       BestEffort
# k3s v1.37.1+k3s1, same manifests
# ... kubectl apply and wait omitted
$ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'
NAME             POD-LEVEL   QOS
ctl-burstable    <none>      Burstable
ctl-guaranteed   <none>      Guaranteed
emp-burstable    map[]       Burstable
emp-guaranteed   map[]       Guaranteed

The same result on every version, from results/summary.md:

plr-burstable, with real pod-level requests and limits, was Burstable everywhere. Pod-level resources work. Only the empty block breaks the class.

To check that the 1.37 gate is the fix and not something else in the release, we reinstalled v1.37.1 with PodLevelResourcesFixKubeletQOSClass=false on both the API server and the kubelet. Both emp-* pods went back to BestEffort, with oom_score_adj 1000. The API server and kubelet metrics reported the gate as BETA, on by default and off in that run.

Finding 2: the kernel sees a BestEffort pod

QoS is not only a label. The kubelet turns it into numbers that the kernel uses. We read them from /proc and /sys/fs/cgroup on the node, for the main process of each container:

oom_score_adj of the main container process

Read from /proc//oom_score_adj on the node. The 1.34.12, 1.35.9 and 1.36.5 values are identical. The kernel adds this value to a process's badness score when it picks one to kill. Source: results/summary.md.

The kernel adds oom_score_adj to each process's badness score before it picks a process to kill, and it accepts values from -1000 to +1000 (proc.rst). A Burstable container gets 1000 - 1000 * request / node memory, which is 935 for a 256Mi request on this node. A Guaranteed container gets -997. The empty block gave both 1000, the same as a pod with no resources at all. oom_score, the current OOM killer score the kernel shows for the process, went from 1290 (Burstable) and 4 (Guaranteed) to 1334.

The cgroup changed too. On v1.36.5:

The container cgroup still has its own limits, so emp-burstable cannot use more than 512 MiB or half a CPU. What it loses is everything the kubelet sets at pod and QoS level. The pod cgroup has no memory limit and no CPU quota of its own. It sits in kubepods-besteffort.slice, where the kubelet sets cpu.weight to 1 for the whole slice.

Finding 3: half a CPU requested, 0.06 to 0.08 cores received

cpu.weight 1 matters only when the CPU is busy. To see how much, we ran cpu-neighbour, a Burstable pod that requests 1 CPU and runs two busy processes, so both vCPUs are always taken. Next to it, cpu-ctl and cpu-emp took turns, three times each, alternating which went first. Both request 500m CPU with a 1 CPU limit and run one busy process; cpu-emp adds the empty block. Each pod reads its own cgroup's usage_usec over 20 seconds after a 5 second warm-up.

CPU used next to a busy neighbour, median of 3 runs

2 vCPU droplet. Both pods request 500m with a 1 CPU limit and run one busy process; the neighbour requests 1 CPU and runs two. Each pod measured its own cgroup usage_usec over 20 s. Source: results/summary.md.

On the three affected versions, cpu-emp got a median of 0.083, 0.070 and 0.061 cores, against 0.579, 0.580 and 0.573 for cpu-ctl. That is 14.3%, 12.1% and 10.6% of the control's CPU, or 7.0 to 9.4 times less. On v1.37.1 the two were the same: 0.576 and 0.577 cores.

The weights explain the direction. cpu-ctl's pod cgroup has cpu.weight 20 for its 500m request and sits in the burstable slice next to the neighbour. The kubelet sets the burstable slice's weight from the sum of the CPU requests of the Burstable pods in it, which gave 63 here. cpu-emp's pod cgroup has cpu.weight 1 and sits in the besteffort slice. The kubelet always sets that slice to the minimum, cpu.weight 1. The 500m request of cpu-emp still sets a weight on its container cgroup, but that only counts against other containers in the same pod. On a busy node like this one, a CI build in the BestEffort slice competes with cpu.weight 1 at that level and can get far less CPU than Burstable or Guaranteed pods. We did not run a build.

The runs were short and sequential. Each pod ran three times in a fixed alternating order, and we did not record the neighbour's own usage or other load on the host. On the affected versions, each of the nine cpu-emp runs got less CPU than every cpu-ctl run, including the runs where cpu-emp went first. The exact percentages still belong to this setup.

Finding 4: a "no BestEffort" quota rejects the pod

Some clusters forbid BestEffort pods with a quota like this one, so every workload must declare resources:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: no-besteffort
spec:
  hard:
    pods: '0'
  scopes: ['BestEffort']

The API server applies the scope at create time, before any kubelet sees the pod, and it calls the same GetPodQOS function. With no status yet, that function computes the class from the spec. On 1.34 to 1.36 it rejected emp-burstable, a pod whose container sets requests and limits:

quota.py on k3s v1.36.5

$ grep -A3 ' quota$' results/k3s-v1.36.5/run.log
2026-11-02T10:33:20Z quota
be              admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/be.yaml": pods "be" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0
ctl-burstable   admitted=True  pod/ctl-burstable created
emp-burstable   admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/emp-burstable.yaml": pods "emp-burstable" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0

On v1.37.1 the API server admitted emp-burstable, and still rejected be. We did not run a runner. But a job pod from 18.10.0 to 19.4.x carries the same empty block, so in a namespace with this quota it should fail the same way on 1.34 to 1.36. The fix discussion names the reverse risk: a pod "could theoretically bypass restricted BestEffort quotas by being classified as Guaranteed during admission, while still behaving as BestEffort at the node level". That can happen when the API server has the fix and the kubelet does not.

Finding 5: the eviction order did not change

The five eviction pods each allocate a fixed amount of memory and then sleep. They are sized so that the two possible rules predict different orders:

If the kubelet ranked by QoS class, emp and gua-emp would go out together with be, before ctl and gua-ctl. If it ranks by container requests, be goes first because any usage exceeds a request of zero, and the rest follow in order of usage minus request: ctl, gua-ctl, emp, gua-emp.

A host process outside every pod, hog.py, then holds MemAvailable at 1000 MiB. That is below the 2Gi threshold even after all five pods are gone, so the kubelet keeps evicting until nothing evictable is left. The kubelet cannot evict the hog. The pods differ in size on purpose, to separate the two models, so this is not a matched pair. CoreDNS also ran on the node. The kubelet ranked it last in every ranking it logged. In 5 of the 12 logs the window ran long enough to show the next step: the kubelet logged cannot evict a critical pod once only CoreDNS was left.

Every repetition on every version gave the request order. This is the kubelet's own ranking on v1.36.5, while it reported emp and gua-emp as BestEffort:

k3s v1.36.5, repetition 1

# the class the kubelet reported for each pod
$ jq -c 'select(.rep == 1) | .qos' results/k3s-v1.36.5/40-pressure.jsonl
{"be":"BestEffort","ctl":"Burstable","emp":"BestEffort","gua-ctl":"Guaranteed","gua-emp":"BestEffort"}
# the kubelet's ranking before each eviction; sed keeps the journald timestamp and the message, and drops the host name, the service PID, the severity letter, the kubelet's own timestamp, the second PID and the source file
$ grep -E 'ranked for eviction|evicted successfully' results/k3s-v1.36.5/40-kubelet-rep1.log | sed -E 's/^([^ ]+) .*eviction_manager.go:[0-9]+\] /\1 /'
2026-11-02T10:38:17.816980+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/be","evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:20.108268+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/be"
2026-11-02T10:38:21.127785+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:23.555889+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/ctl"
2026-11-02T10:38:24.579473+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:27.242735+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-ctl"
2026-11-02T10:38:28.259762+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:30.555840+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/emp"
2026-11-02T10:38:31.572048+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:34.078890+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-emp"

The source explains it. rankMemoryPressure sorts by three keys. First, whether usage exceeds the memory request. Then pod priority. Then usage minus the memory request. The first and third keys get the request from GetResourceRequestQuantity. For an empty block, that helper asks IsPodLevelResourcesSet, gets false, and uses the container requests. Priority does not read requests, and the QoS class is not an input at all. If the kubelet had used a request of zero for emp, emp would have been in the first group with be. It came fourth.

When each pod was evicted, 12 repetitions

Seconds after the host hog started, from the kubelet's log. 3 repetitions each on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. Every repetition evicted the pods in the same order. Source: results/summary.md.

In all 12 repetitions the kubelet started to reclaim 3.0 to 9.2 s after the hog started, and evicted the first pod 2.3 to 3.3 s later. After that it evicted one pod at a time. 41 of the 48 gaps between evictions were 4.0 s or less (median 3.6 s, longest 8.2 s). The last pod was gone 19.5 to 29.9 s after the hog started. It does not wait 10 seconds between evictions: after each one it waits for the pod to be cleaned up and then checks again. Its own available figure at the first eviction was 931 to 1212 MiB, well under the 2Gi threshold and far from empty. The kernel logged no OOM kill in any repetition, so the order is the kubelet's alone.

The two mechanisms are easy to mix up. The kubelet's eviction manager checks every 10 seconds by default when it has nothing to evict, and it ignores the class. The kernel OOM killer acts when memory runs out before the kubelet frees any. It adds oom_score_adj to each process's badness score, and the empty block gives the pod the same 1000 as a pod with no resources at all (Finding 2). We did not trigger a kernel OOM, so we did not watch the kernel make that choice.

What we got wrong before measuring

The research note we started from had the core right and two details wrong.

• Right: Kubernetes 1.34 to 1.36 compute BestEffort for an empty pod-level block. GitLab Runner 18.10 to 19.4 build their job pods with that block (from the source and the fix MR). On 1.34 to 1.36 the pods get oom_score_adj 1000. 1.37 fixes the class while its new gate is on, which is the default.
• Wrong: it called #137150 the upstream issue. That is the fix PR. The issue is #135082.
• Wrong, and it matters: it said these pods are first in line for kubelet eviction under memory pressure. Finding 5 shows they are not.
• Missing: the 1.37 fix is a beta feature gate, not a plain code change, and nothing was backported to the 1.34, 1.35 or 1.36 patch releases.

We got one prediction wrong ourselves. CLAIM.md said the BestEffort-classified pod would get under 10% of the control's CPU. It got 10.6% to 14.3%. Our falsification line was 50%, so the claim stands, but the number we wrote down was too pessimistic, and we report it as a miss.

We also kept a one-repetition smoke run on v1.36.5 (results/_smoke-k3s-v1.36.5/). It used the same code stamp and gave the same results, but the summary does not count it.

What to do

1. Find the pods. This query lists every pod with an empty pod-level block and the class it got:

   kubectl get pods -A -o json \
     | jq -r '.items[] | select(.spec.resources == {}) | [.metadata.namespace, .metadata.name, .status.qosClass] | @tsv'
   

2. GitLab Runner 18.10.0 to 19.4.x: upgrade to 19.5.0 or later. The job pods then go out without a pod-level block, and the class comes from the containers again.

3. If you cannot upgrade the runner yet, give the block real values. Set pod_cpu_request and pod_memory_request (and the _limit keys if you want limits) under [runners.kubernetes]. The QoS class then comes from those values, like plr-burstable above. This is a behaviour change, not only a workaround: the pod cgroup then takes its limits from the pod-level values too, as it did for plr-burstable.

4. Kubernetes 1.37 fixes it by default. The fix sits behind a beta gate, so check that nobody turned PodLevelResourcesFixKubeletQOSClass off.

5. Check your own templates. A Helm chart that renders a pod-level block from a value that defaults to {} produces the same bug. Render the key only when it has content:

   # wrong: renders "resources: {}" at pod level when the value is empty
   spec:
     resources: {{- toYaml .Values.podResources | nindent 4 }}
   
   # right: no key at all when the value is empty
   spec:
     {{- with .Values.podResources }}
     resources: {{- toYaml . | nindent 4 }}
     {{- end }}
   

An empty resources: {} inside a container is harmless. Only the pod-level one under spec changes the class.

What we could not conclude

• k3s is one distribution. We ran upstream Kubernetes code as k3s packages it. A managed service can turn the PodLevelResources gate off or carry its own patches. Run the jq query on your own cluster.
• No GitLab instance and no runner ran here. The runner part rests on its source at each tag and on the MR description. The emp-* pods reproduce the field the runner sends, not a full job pod with helper and service containers.
• We did not trigger a kernel OOM kill. The oom_score values are the numbers the kernel uses to choose a victim, read from /proc. We did not watch it choose.
• The eviction timing is ours. It depends on the 2Gi threshold, the hog's 1000 MiB target and how fast it allocates. The order is the finding, not the seconds.
• The CPU gap depends on the neighbours. One neighbour with a 1 CPU request on a 2 vCPU node gave the numbers above. More Burstable or Guaranteed load should make the BestEffort slice's share smaller. On an idle node the weights do not matter, because they only divide contended CPU time; we did not measure that case.
• One node, one admission check. We did not test scheduling across nodes, or any admission rule other than the one quota. The gate-off run on 1.37.1 covered only the class, the kernel-facing numbers and the quota, not CPU or eviction.
• Other consumers of the class were not tested. From the source, the kubelet also rejects new BestEffort pods while the node has the MemoryPressure condition. The static CPU manager gives exclusive CPUs only to Guaranteed pods. We did not measure either.

Summary

On Kubernetes 1.34, 1.35 and 1.36, with the PodLevelResources gate at its default (on), spec.resources: {} makes a pod BestEffort no matter what its containers request. On our node that meant oom_score_adj 1000 instead of 935 or -997, and a place in the BestEffort slice with cpu.weight 1 and no pod-level memory or CPU limit. Next to a busy neighbour the pod got about 7 to 9 times less CPU, and a quota that forbids BestEffort pods rejected it. The empty block did not change the kubelet's eviction order. The ranking uses memory requests, and for an empty block those still come from the containers: 12 of 12 repetitions evicted our pods in exactly the order the requests predict. Kubernetes 1.37 fixes the class behind a beta gate that is on by default, and GitLab Runner 19.5.0 stops sending the block. Run the jq query above on your clusters, and upgrade whichever side you can first.

We drew 100 dashboard panels with known answers and asked four questions about each: the peak value, the time of the peak, whether the data crossed the threshold, and which series is highest. Our claim, written before we measured: Nemotron 3.5 Super VL gets all four right on at least 90 of 100 panels, for under $0.001 per screenshot. It got all four right on 48 panels with reasoning off and 54 with reasoning on. The claim failed. The cost limit held only with reasoning off: $0.00026 per screenshot, against $0.0024 with reasoning on.

None of the four models we tested came close to 90. Qwen3.8-Max got 65 and Gemma 4 got 64, at $0.11 per 1,000 screenshots for Gemma. Log-scale axes broke every model.

So the numbers in an alert message should come from the alert payload, not the model. The more useful result is a small bot that lets the model describe the screenshot and lets rules decide whether to trust it. On Nemotron's 300 reads, the rules caught 63 percent of the wrong reads and blocked no correct read. With the wrong screenshot attached on purpose, the bot refused it 99 times in 100.

TL;DR

• All four facts right, of 100 Grafana-style panels: Qwen3.8-Max 65, Gemma 4 64, Nemotron 3.5 Super VL 54 with reasoning and 48 without, Llama 4 Maverick 21. Our bar was 90.
• Peak time and threshold were easy (86 to 99 percent for every model). Peak value and top series were hard. On log axes, the best model read all four right on 8 of 25 panels.
• Reasoning made Nemotron 13 times slower (19 seconds at the median, 82 at worst) and 9 times more expensive. The gain from 48 to 54 is inside the noise.
• At temperature 0, Nemotron gave the same four answers in all three runs on 13 of 100 panels. Qwen3.8-Max did on 76.
• A cross-check against the alert caught 63 percent of Nemotron's wrong reads (81 percent at a 5 percent tolerance). It cannot check the peak time, the series on a stacked panel, or the words in the summary.

Prerequisites

• A DigitalOcean account (referral link) with a Serverless Inference model access key
• Node.js 22 or later
• About $1 of inference if you rerun every model; the reasoning-mode run is most of it

The-DevOps-Daily/grafana-panel-vision on GitHub

The job and the claim

A bot that reads the attached panel could save the on-call engineer one click: "p99 went to 1.8 s at 13:55 UTC and stayed under the 2.5 s threshold". The question is whether you can trust that sentence during a night page.

Each model got the same prompt for every panel, with the panel as a PNG, and had to answer in JSON. This is a real answer from Nemotron 3.5 Super VL with reasoning off:

{
  "peak": { "value": 1.8, "unit": "s" },
  "peakTime": "13:45",
  "thresholdCrossed": "no",
  "topSeries": "search",
  "summary": "Upstream response time stays under 1 second until midday, then spikes sharply to about 1.8 seconds in the early afternoon while remaining below the 2.5 s warning threshold."
}

The truth for that panel is 1.85 s at 13:55 UTC, not crossed, search, so this read passes. The tolerances, set before the first run:

• Peak value: within 5 percent of the true peak, after unit conversion. A unit from the wrong family, such as MiB on a latency panel, is wrong. "MB" counts as "MiB", because the axis says MiB.
• Peak time: within 5 percent of the panel's time range. That is 3 minutes on a 1-hour panel, 18 on 6 hours and 72 on 24 hours.
• Threshold crossed: exact. The answer is "yes", "no" or "no threshold".
• Top series: the answer is matched to the nearest legend name, with at most 2 typing errors (1 for short names). A tie, or an answer that names two legend entries, counts as wrong. On a stacked panel, the top series is the one with the thickest band at the peak, not the band drawn on top, and the prompt says so.

A panel counts as read only when all four facts are right.

100 panels with known answers

Warning

These are Grafana-style panels drawn by our own renderer, not screenshots from Grafana. Real data has plateaus and near-equal peaks, which our generator rejects so that each question has one answer. Many real dashboards show a legend with Max or Last columns, which would turn the value question into text reading and help the models. A model trained on real Grafana screenshots could read real panels better or worse than ours. We do not know which.

The renderer writes SVG and converts it to PNG with @resvg/resvg-js. It copies the Grafana dark theme: panel background, Inter font, classic palette, grid, axis labels with units, a list legend, and thresholds as a dashed line, a shaded region or both. There are no tooltips and no legend values. A seeded generator (mulberry32) draws the data, and the truth comes from the same numbers.

In an earlier project, a bug in our own test data made strong models look wrong, so we checked the truth first. A second script reads every panel back from the SVG alone, the way a reader would: the peak from the tick labels, the time from the clock labels, and the top series from the line colours and the legend (on the 19 stacked panels, from the band heights at the peak). It agreed with the truth on all 100 panels, with a largest peak error of 0.011 percent. To prove the check can fail, a third script breaks the renderer in a copy: wrong byte labels gave 38 problems, lines drawn 3 percent too high gave 204, and stacked bands drawn in the wrong colours failed the band check on all 19 stacked panels. We also viewed 14 panels next to their truth records.

Which models we tested

A probe sent the same question to 15 models, with no image and then with the smallest and the largest panel. A model passed if it answered and its prompt token count rose with the image, which proves the image arrived. 11 passed. Mistral 3 14B returned "model does not support image input", and Kimi K2.6 rejected temperature 0. Our account gets HTTP 403, "not available for your subscription tier", for the GPT models, so this test has open-weight models only.

Several vision models reason before they answer, which is too slow for a page. On the two probe panels, Kimi K3 took 28 and 165 seconds, DeepSeek V4.1 Flash 45 and 13, and Qwen 3.5 397B 71 (with no answer) and 42. We kept Nemotron 3.5 Super VL (nemotron-3.5-super-vl), with reasoning off and on, and three models that answered in 1.5 to 3 seconds: Qwen3.8-Max (qwen3.8-max), Gemma 4 (gemma-4-31B-it) and Llama 4 Maverick (llama-4-maverick). Nemotron Nano 12B v2 VL was also fast; we left it out to keep the test to four models.

Every call in the comparison used temperature 0 and at most 1,024 output tokens. Reasoning mode got 16,384, because the reasoning counts against that limit. Without reasoning, each model read every panel three times. Reasoning calls took about 20 seconds each, so that mode read all 100 panels once and a fixed 34-panel subset two more times.

Nemotron reasons by default. Two request settings turn it off. The /no_think system prompt does not:

stage 0b: turning reasoning off on one 1000x500 panel

$ node scripts/00b-reasoning-switch.mjs nemotron-3.5-super-vl
nemotron-3.5-super-vl | default                                      | HTTP 200 | 22795 ms | completion 4096 | reasoning chars 13191 | $0.00291
nemotron-3.5-super-vl | reasoning_effort=low                         | HTTP 200 | 22271 ms | completion 4096 | reasoning chars 14191 | $0.00291
nemotron-3.5-super-vl | reasoning_effort=none                        | HTTP 200 | 1207 ms | completion 99 | reasoning chars 0 | $0.00031
nemotron-3.5-super-vl | chat_template_kwargs.enable_thinking=false   | HTTP 200 | 1268 ms | completion 92 | reasoning chars 0 | $0.00031
nemotron-3.5-super-vl | system prompt /no_think                      | HTTP 200 | 21921 ms | completion 4096 | reasoning chars 14294 | $0.00291

That probe had a 4,096-token limit, and the reasoning runs used all of it without an answer. In the main run, none of the 168 reasoning calls hit the 16,384 limit. The longest used 11,979 tokens.

Results

Accuracy and cost are from run 1. Latency covers every successful call in all runs, and it is the time of the attempt that succeeded. The parallel runs hit HTTP 429 often (retried attempts: 88 for Nemotron with reasoning off, 7 with reasoning on, 99 for Qwen3.8-Max, 75 for Gemma 4), and those waits are not in the latency figures.

Panels with all four facts right, by panel feature (run 1)

100 Grafana-style panels drawn by our renderer, 31 October 2026. Panels per row: 1 series 17, stacked 19, small 19, similar colours 27, 4-5 series 36, log y-axis 25. Rows this small give wide intervals; read them as directions, not rates.

What stands out:

• Time and threshold are easy. Every model got the peak time right on 89 to 99 panels and the threshold on 86 to 97.
• A simple panel is almost solved. With one series, Nemotron and Qwen3.8-Max both got 15 of 17 right. Every added feature cost accuracy. Similar colours cost Nemotron the most: 16 of its 25 wrong series names were on the 27 similar-colour panels.
• Qwen3.8-Max and Gemma 4 cannot be told apart. Each got 14 or 13 panels right that the other got wrong. Both beat Nemotron with reasoning off: Qwen3.8-Max on 20 panels against 3, Gemma 4 on 24 against 8 (sign test p = 0.007).
• The 5 percent bar hurt Gemma 4 most. It rounds: 20,000 req/s for a peak of 21,330, 300 ms for 331. Within 10 percent, Gemma read the value right on 93 panels, Qwen3.8-Max on 87 and Nemotron on 82.

Repeat runs at temperature 0 showed another difference. Qwen3.8-Max and Gemma 4 gave the same verdict in all three runs on 97 and 98 panels. Nemotron with reasoning off did on 71, and gave the same four answers on only 13. Llama 4 Maverick gave the same four answers on 92 panels and was right in all three runs on only 21. Consistent is not the same as correct.

Where the reads go wrong

Log axes. The models read a log axis as if it were linear, or round to a tick. Nemotron read the value within 5 percent on 1 of 25 log panels, Qwen3.8-Max on 4 and Gemma 4 on 10.

The spike peaks at 21.3K req/s at 14:28 UTC. Every model found the series and the time. Qwen3.8-Max and Gemma 4 said 20,000 every time, 6.2 percent low. Nemotron's three runs said 30 req/s, 45,000 req/s and 30K req/s.

The threshold label as the answer. When a spike crosses a dashed line, the label of that line is the easiest number on the panel to read.

The rise at the right edge goes to 246 ms, above the 200 ms line. Qwen3.8-Max, Gemma 4 and Llama 4 Maverick answered 200 ms in all three runs: the threshold, not the peak. Nemotron answered 500 ms, 0.3 s and 300 ms. Only Llama 4 Maverick named inventory in every run, and it said the threshold was not crossed.

The band on top is not the biggest. On a stacked panel, the series drawn last sits on top, even when its own values are small.

node-c3 is the thin band on top. Qwen3.8-Max and Llama 4 Maverick named it in all three runs and Nemotron in two. Gemma 4 named node-a1 every time. This panel also shows the hardest threshold style, a shaded region with no line. The models got a region-only threshold right on 12 to 15 of 17 panels, and a region with a line on 19 or 20 of 20.

Reasoning on: slower and more expensive, and not clearly better

With reasoning on, Nemotron got 54 panels right against 48. Only reasoning got 14 panels right, and only no-reasoning got 8. A sign test gives p = 0.29, so 100 panels cannot say whether reasoning helps. The cost is clear: 3,349 output tokens per call on average in run 1 (8,163 at the 95th percentile), and 19.0 seconds at the median over all 168 calls, 81.8 at worst.

• The usage block hides the reasoning. Every reasoning call returned reasoning_tokens: 0, while completion_tokens included the reasoning and the reasoning text came back in the reply. If you cap or alert on reasoning_tokens, you see nothing.
• Temperature 0 does not explain it. Reasoning models are often tuned for a higher temperature, so we ran the 34-panel subset again at 0.6. It got 19 right, against 20 at temperature 0 and 20 with reasoning off.
• It is not more repeatable. On the subset, reasoning gave the same four answers in all three runs on 3 panels, against 7 with reasoning off.

What a screenshot costs

DigitalOcean's pricing page ("Last verified 30 Oct 2026", read on 31 October) lists no separate image rate for these models. We priced each call from the token counts the API reported, at list input and output prices. That assumes the image tokens inside prompt_tokens are billed at the input rate. The closed October invoice agrees with this. For each of the four models, the change in billed input, cached input and output tokens is equal to the totals the API reported for our 1,522 recorded calls. Each rate on the invoice is the list price, and the invoice has no separate charge for images.

Image tokens are the prompt tokens with the panel minus those of the same question with no image. Gemma 4 uses about 260 at any size, Qwen3.8-Max scales with the pixels, and Llama 4 Maverick jumps from 437 to 1,017 above 600 pixels wide.

Our first cost numbers were wrong in our favour. Runs 2 and 3 send the same image again, and Qwen3.8-Max, Gemma 4 and Llama 4 Maverick served most of it from the prompt cache: Llama's cached tokens went from 271 in run 1 to 1,044. A bot sees each screenshot once, so we now price run 1 only. That raised Qwen3.8-Max from $1.11 to $1.47 per 1,000 and Gemma 4 from $0.097 to $0.113. Nemotron reported no cached tokens on any call, even for the prompt text that every call shares.

Rules decide, the model explains

The alert already knows the numbers. In our test, each alert comes from a rule on the panel's query, reduced with max over the panel's range, so the payload has the peak value, the series label (unless the rule is on a sum), the threshold and the firing state. If your rule reduces with last or a shorter window, compare against that value instead. The model reads the image blind, with no numbers from the alert, and three rules compare its read with the payload:

// lib/bot.mjs (shortened). tol is 10% by default.
export function crossCheck(alert, answer, { tol = 0.1 } = {}) {
  if (!answer) return { verdict: 'could not confirm', checks: [{ fact: 'read', ok: false }] };
  const checks = [];
  // 1. Peak value against the alert's reduced value.
  const p = readPeak(alert.unit, answer.peak);
  checks.push({
    fact: 'peak value',
    ok: p.ok && Math.abs(p.base - alert.values.B) / alert.values.B <= tol,
  });
  // 2. Series, when the alert is for one series. A rule on a sum has none.
  if (alert.labels.series) {
    checks.push({
      fact: 'series',
      ok: matchSeries(answer.topSeries, [alert.labels.series]) === alert.labels.series,
    });
  }
  // 3. Threshold state, when the model saw a threshold on the panel.
  const th = normThreshold(answer.thresholdCrossed);
  if (th === 'yes' || th === 'no') {
    checks.push({ fact: 'threshold', ok: th === (alert.status === 'firing' ? 'yes' : 'no') });
  }
  return { verdict: checks.every((c) => c.ok) ? 'confirmed' : 'could not confirm', checks };
}

The message always starts with the alert's own facts. The model's sentence is added only when its read agrees. Here is the bot on three panels, with Nemotron and reasoning off:

the bot on three alerts

$ node scripts/05b-bot-demo.mjs p034 p090 p003
--- p034 (data/panels/p034.png) -> confirmed, 2033 ms
Container RSS: firing, max 3.19 GiB over the last 6h, threshold 2.51 GiB.
Screenshot (checked against the alert): Container RSS memory usage spikes sharply around 08:15 UTC to over 3 GiB, driven primarily by worker-5c6b-h3t7, then settles back to roughly 1-2 GiB.
--- p090 (data/panels/p090.png) -> confirmed, 1380 ms
Checkout latency p95: firing, max 106.72 ms over the last 24h, threshold 73 ms, series search.
Screenshot (checked against the alert): Checkout latency p95 is stable around 30-40 ms until a spike to about 100 ms near 22:00 UTC, crossing the red 100 ms threshold.
--- p003 (data/panels/p003.png) -> could not confirm, 1311 ms
HTTP requests: firing, max 21.33K req/s over the last 6h, threshold 17K req/s, series POST /login.
Could not confirm the screenshot (the screenshot reads 25K req/s, the alert says 21.33K req/s). Open the dashboard before you trust the image.

The p090 message shows the limit. The value, series and threshold state agree, so the bot says "confirmed", but the summary says "the red 100 ms threshold" and the threshold is 73 ms. The rules check the structured answer, not every word. That is why the bot posts the alert's numbers itself.

Over all recorded reads:

"Correct reads blocked: 0" is by construction. Our alert value is the true peak and the bot's checks are looser than the scorer's, so a correct read cannot fail them. A real alert that reduces differently from the panel would add false flags. What the 10 percent check let through for Nemotron:

• 24 reads with only a value error between 5 and 10 percent, such as 300 ms for 331 ms
• 21 wrong series names, all on stacked panels, where the alert is on the sum and has no series to compare
• 6 threshold errors where the model said "no threshold" on a panel that shows one; the bot cannot tell that apart from a rule threshold that is not drawn
• 7 wrong peak times, alone or with a small value error; the payload has no peak time

Then we broke it on purpose: each read was checked against the alert of a different panel with the same unit, like a wrong image on the notification. The bot said "could not confirm" for 99 of 100 reads from Nemotron and from Qwen3.8-Max, and for 100 of 100 from Gemma 4 and Llama 4 Maverick. The pair that passed had a wrong alert on a sum (no series check) whose value was 8.6 percent from the read.

We expected the model to copy the alert's numbers if the prompt contained them, which is why the bot reads blind. For Nemotron that was wrong. With the wrong alert in the prompt, its peak matched the image on 81 of 100 panels and the alert on 1, and the cross-check still refused 98. We keep the blind read because it costs nothing, but this test does not show that it is necessary.

What 100 panels cannot tell you

• The claim failed by a wide margin. The 95 percent intervals for Nemotron end at 58 and 63 panels, far below 90.
• We cannot rank Qwen3.8-Max against Gemma 4, or Nemotron with reasoning against without. Both gaps are too small for 100 panels.
• Feature rows are directions, not rates. Each covers 17 to 36 panels. Gemma 4's 8 of 25 on log axes has a 95 percent interval of 17 to 52 percent.
• The panels are ours. Real Grafana screenshots are a different input, easier in some ways and harder in others.
• One prompt. We did not try a prompt that warns about log axes. We did not use a JSON response mode, and Qwen3.8-Max wrote invalid JSON twice in 300 calls ("thresholdCrossed": no).
• One day of one service. The models DigitalOcean served on 31 October 2026, at temperature 0. Prices are list prices times reported tokens. The closed October invoice billed the same token totals at the list prices.

Run it yourself

Each stage writes a file that the next one reads. Every recorded call is stamped with a hash of the runner code and the inputs. The scorer refuses unstamped, mixed or stale files, and records filed under the wrong configuration. The runner stops before its own count of spend passes a cap ($3.50 by default).

git clone https://github.com/The-DevOps-Daily/grafana-panel-vision
cd grafana-panel-vision
npm ci
cp .env.example .env                 # add your model access key

node scripts/01-render-panels.mjs    # 100 PNGs + data/truth.jsonl + data/alerts.jsonl
node scripts/01b-check-panels.mjs    # read every panel back from the SVG
mv results/raw results/raw-ours      # keep our run; yours goes into results/raw
node scripts/02-run-models.mjs --configs nemotron-3.5-super-vl,gemma-4-31B-it --runs 3
node scripts/03-score.mjs            # results/summary.md
node scripts/05-crosscheck.mjs       # how often the bot catches a wrong read

To test your own dashboards, replace the renderer with exports of your real panels and write the truth for each by hand. It is slower, and it is the only way to know how these models read your panels.

Summary

A vision model on DigitalOcean can describe a Grafana panel in about 1.5 seconds for a fraction of a cent. On our 100 panels, no model read the four key facts reliably enough to be the source of truth in a page. Log axes, stacked areas and similar colours caused most errors, and reasoning did not fix them.

What we would ship:

• Take the numbers from the alert, not from the image. The payload already has the value, the threshold and the series.
• Let the model narrate, and check it first. Compare its value, series and threshold state with the payload, and say "could not confirm" when they disagree.
• Use a fast model with reasoning off. Gemma 4 and Qwen3.8-Max were the most accurate, and Gemma 4 cost $0.11 per 1,000 screenshots.
• Make attached panels easy to read. Use a linear axis, distinct colours, few series, and a threshold drawn as a line and a region.

Most of them were in places nobody writes a rule for: the email copied onto each order at checkout, the JSON payloads of login and checkout events, the text of support tickets. The other 20,000 copies came from the masking run itself. The masking updates each row of users, so the audit trigger on that table fired 20,000 times and saved every original row into audit_log.

Anonymized branches mask the columns you list, and in our runs they did. This post measures what is left in the rest of a realistic schema, and shows the scanner we run as a CI gate before preview branches are made from the masked branch. Everything ran on a real Neon project through the Neon API, and the scripts are public.

TL;DR

• Masking the listed columns works. No masked email, name, phone, address or IP kept its original value.
• The copies are not masked. After round 1, 18,776 of the 20,000 original emails (93.9%) were still readable outside the audit table: in orders, event payloads, support tickets and invitations.
• The masking run fires your triggers. audit_log grew from 1,888 to 21,888 rows during masking, and each new row held one user's full original row. The branch refuses connections until the run ends, so you cannot switch the trigger off first.
• Free text and JSON need a decision. We found no rule that edits JSON in place, and regexp_replace removed emails and phone numbers from ticket bodies but not names. Only replacing free text and JSON passed the gate.
• Pick generators with care. anon.dummy_free_email() and anon.dummy_safe_email() made the run fail on a unique column at 20,000 rows, and random IP addresses and phone numbers can be equal to real ones.
• Previews from the passing template were ready in 2.3 to 2.5 seconds. The same rules failed the scan after a migration copied emails into a new table.

Prerequisites

• A Neon account and an API key (Account settings > API keys). Everything here ran on a Scale plan project. Anonymized branches are a beta feature.
• Node.js 22 or newer.
• Basic Postgres knowledge: triggers, jsonb, materialized views.

What an anonymized branch does

Neon's data anonymization creates a branch of a Lakebase Postgres database from a parent branch, then masks it with PostgreSQL Anonymizer (the anon extension, version 2.5.1 on our branches). The masking is static: the run rewrites the rows on the new branch, and the parent does not change. You send the rules with the create call:

curl -X POST "https://console.neon.tech/api/v2/projects/$PROJECT_ID/branch_anonymized" \
  -H "Authorization: Bearer $NEON_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "branch_create": { "branch": { "name": "template", "parent_id": "'"$MAIN_ID"'" } },
    "masking_rules": [
      { "database_name": "neondb", "schema_name": "public", "table_name": "users",
        "column_name": "full_name", "masking_function": "anon.dummy_name()" }
    ],
    "start_anonymization": true
  }'

Three details matter later:

1. The branch refuses connections until the run ends (Access to this endpoint is temporarily disabled). We saw the same error after each of our failed runs, so in our tests a failed run did not hand out unmasked data.
2. GET /branches/{id}/anonymized_status reports anonymizing, then anonymized or error.
3. Each rule names one column. Nothing in the API looks at the columns you did not name.

That is by design, and Neon's pg_redact post makes the same point: column rules cannot reach PII inside free text. We wanted to know how much PII that is in an ordinary schema, where it sits, and how to find it before someone gets a connection string.

The test database

The schema is a small invoicing SaaS. users holds the PII on purpose. Everything else holds it by accident, the way real apps do:

1. users email, full_name, phone, address, signup_ip
2. orders contact_email, shipping_*
3. support_tickets.body free text
4. events.payload jsonb
5. invitations invitee email, name, message
6. audit_log trigger: old and new row
7. customer_summary materialized view
8. users.search generated tsvector

Connections:

• users -> orders (copied)
• users -> support_tickets.body (quoted)
• users -> events.payload (logged)
• users -> invitations (named)
• users -> audit_log (every UPDATE)
• users -> customer_summary (refresh)
• users -> users.search (generated)

A seeded generator writes 20,000 users, 40,191 orders, 6,558 tickets, 59,738 events and 2,035 invitations (47.6 MB). Then it changes the email or the address of 1,888 users, so the audit trigger has history with former values. All names are made up, emails use the reserved .example and .test domains, and phone numbers use the fictional 555-01xx range. The generator records every PII value it writes into each row, so we know what each column holds, and it mixes in decoys such as invoice numbers, tracking numbers and the app's own support address.

The scanner

The-DevOps-Daily/neon-anonymized-branches on GitHub

scan.mjs takes two branches: the anonymized branch and its parent. It reads the PII columns you list in pii-sources.json on the parent (here five columns of users), then reads every text, JSON, tsvector, inet and xml column of the user tables and materialized views on the anonymized branch (the anon extension's own tables are left out). Per column it reports:

• known: rows that still hold a value from the parent, also inside free text and JSON. Phone numbers match on their last 10 digits, and a full name matches as two neighbouring words, so /team/tarek-stratkett in a URL matches "Tarek Stratkett".
• pattern: values shaped like an email, phone number, IP address or street address that are not known values, such as former emails and people who are not users.
• row counts of every table that changed during masking, and a fingerprint of the data on the branch (per table, the row count and a sha256 over the sorted md5 of each row).

It fails when a column holds a known value, when a column that no rule replaces holds PII-shaped values, when a rule changed no row, when the branch has a database it did not read, when the branch holds data it cannot read (such as a bytea column, large objects or foreign tables), or when the data changed during the scan. Two columns here are computed only from masked columns, a materialized view and a generated search column. They are on a review list: their PII-shaped values go to a person instead of failing the gate, and known values in them still fail.

A pass means the scan found no known value anywhere, and no PII-shaped value outside replaced and review columns. It does not mean the branch is clean. The scanner does not know phone numbers outside the US format, nicknames or encoded values, and it does not read numeric columns.

Against the unmasked parent, the rows it found matched the planted ground truth for all 33 combinations of column and kind. That is not an independent test of recall, because we planted the data in formats the scanner knows. It shows that the scanner counts what it should on this data. A second check, 08-predict.mjs, predicts each column from the parent's data and the rules, without Neon. Where the real scan differs from the prediction, the masking run did something on its own.

The first gate was too clever

The first version of the gate raised four false alarms on two templates. Three were values that the fake data generator made by chance and that belong to other users: 2 masked IP addresses and 1 phone number. The fourth was our bug in the tsvector name check. We first excused any known value that also appeared in a masked column. A review showed that this fails open: a real leak with the same value would pass too.

So the gate has no exception for chance matches. Every known value fails it, and a person reads the report. On real data this happens more often than in ours, because anon.dummy_name() returns real first and last names. A false alarm costs a few minutes. A leak costs much more. The old scans are in results/first-gate/.

Round 1: the columns everyone lists

The first rule set masks the PII columns of users, the way the docs and the GitHub Action examples do. For the unique email column we used the Console's Random Unique Email, pg_catalog.concat(anon.dummy_uuidv4(), '@example.com'), which Neon recommends for unique columns.

scan the round 1 template

$ node scripts/scan.mjs --target template-v1 --known main --evidence --out results/scan-template-v1.json
scanned 26 columns in 7 tables of template-v1 (68.8 s)
  audit_log.old_row                  rule=none      known: email 20938, name 21888, phone 18552, address 20950, ip 21888 pattern: email 950, address 938
  audit_log.new_row                  rule=none      known: email 1888, name 1888, phone 1585, address 1888, ip 1888 pattern: email 20000, phone 17807, address 954, ip 20000
  customer_summary.email             rule=review    known: -                                            pattern: email 20000, phone 227
  events.payload                     rule=none      known: email 19931, name 12816, ip 11888            pattern: ip 8017
  invitations.invitee_email          rule=none      known: email 585                                    pattern: email 1450
  invitations.invitee_name           rule=none      known: name 585                                     pattern: -
  invitations.message                rule=none      known: name 2035                                    pattern: -
  orders.contact_email               rule=none      known: email 40191                                  pattern: -
  orders.shipping_name               rule=none      known: name 36939                                   pattern: -
  orders.shipping_address            rule=none      known: phone 31371, address 36939                   pattern: phone 3252, address 3252
  support_tickets.body               rule=none      known: email 1324, name 1231, phone 1162, address 588 pattern: email 687, phone 622
  users.search                       rule=review    known: -                                            pattern: email 20000, phone 220
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
FAIL: 16 problems

These are the rows that still hold a user's original PII:

Counted as distinct values, this is how much is still readable on the branch:

The scan also flagged PII of people who are not users, such as 1,450 invitee emails and 8,017 login IPs that are not signup IPs. The prediction matched 21 of the 26 columns, and each difference points to something the masking run did. Two of them are good news: the generated search column and the materialized view customer_summary held no original value, because Postgres computes a generated column again on every UPDATE and Neon refreshes materialized views after the run. The audit trigger caused the others.

The masking run fires your triggers

The prediction for audit_log was 1,888 rows. The branch had 21,888. Static masking updates every row of each table that has a rule, so a row-level AFTER UPDATE trigger fires once for each masked row. Our trigger is a common audit pattern:

CREATE FUNCTION audit_users() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
  INSERT INTO audit_log (table_name, row_id, action, old_row, new_row)
  VALUES ('users', OLD.id, TG_OP, to_jsonb(OLD), CASE WHEN TG_OP = 'UPDATE' THEN to_jsonb(NEW) END);
  RETURN NULL;
END $$;

So the masking wrote 20,000 new audit rows, and each old_row holds one user's original email, name, phone, address and IP. A rule on users alone made the branch hold a second full copy of users.

There is no clean way to stop this on the branch:

• The branch refuses connections before the run starts. With start_anonymization: false our client was refused too, so you cannot run ALTER TABLE ... DISABLE TRIGGER first.
• The trigger cannot skip the masking run by role. All 21,888 audit rows on the masked branch record neondb_owner, the role the app uses too.

What worked was a rule on the audit table itself (old_row and new_row masked with NULL). In our runs Neon masked users first and audit_log after it, so the masking also cleared the rows that its own trigger had written. We put the audit rules first in one rules file (v2-audit-first), and the order did not change. We did not find this order documented, so do not depend on it. The 20,000 extra rows stay on the branch with NULL in both columns, which is why the scanner reports tables whose row count changed.

Round 2: mask the copies, edit the free text

The second rule set (v2-careful) adds a rule for every structured copy the first scan found: the order contact email and shipping name, the shipping address rebuilt with jsonb_build_object, the invitee email and name, and the two audit columns. For the ticket bodies it keeps the text and cuts out emails and phone numbers with the same patterns the scanner uses:

pg_catalog.regexp_replace(
  pg_catalog.regexp_replace(body, '<email pattern>', '[email]', 'g'),
  '<phone pattern>', '[phone]', 'g')

Three columns still failed:

The regexp_replace rule did its job. No email or phone number was left in any ticket body, and the JavaScript prediction matched the scan exactly. But regular expressions find shapes, and a name has no shape. "Contact Maeve Ashquist on [phone] before 5pm" is still personal data.

The events table has no rule because we could not write one that keeps the payload. The rule lab (03-rule-lab.mjs) tried 17 expressions, each on its own anonymized branch:

The rule parser accepts function calls from the anon schema and 36 pg_catalog functions that the extension labels as trusted, such as concat, regexp_replace, md5 and jsonb_build_object. jsonb_set is not one of them, and without casts a rule cannot turn jsonb into text and back. So in our tests a JSON column had two options: rebuild the whole document from fake values, which loses its shape when the shape changes from row to row (our events have four), or replace it with a constant.

Round 3: replace what you cannot edit

The third rule set (v3-strict) replaces what round 2 could only edit: ticket bodies become anon.lorem_ipsum(words := 20), invitation messages become anon.lorem_ipsum(words := 10), and event payloads become {}. It also takes IP addresses and phone numbers from ranges that no real value can be in (more on that below). That passed:

scan the round 3 template

$ node scripts/scan.mjs --target template-v3 --known main --evidence --out results/scan-template-v3.json
scanned 26 columns in 7 tables of template-v3 (41.7 s)
  customer_summary.email             rule=review    known: -                                            pattern: email 20000, phone 212
  users.search                       rule=review    known: -                                            pattern: email 20000, phone 207
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
PASS: no PII from the source columns and no PII-shaped values found outside replaced and review columns (2 to review)

It took 37 seconds to mask six tables. This is how much of the original PII each round left readable, counted as distinct values:

Original PII still readable on the masked branch

Distinct original values of the 20,000 users found anywhere on each template, as a share of all distinct values of that kind (20,000 emails, 17,396 full names, 16,967 phones, 20,000 addresses, 20,000 IPs). Round 1: 6 rules on users. Round 2: 14 rules, regexp_replace on ticket bodies. Round 3: 16 rules, free text and JSON replaced.

The pass has a price. The support inbox in a preview shows lorem ipsum, and the events table has no payloads, so code that reads payloads has nothing to test against. If those features need realistic data, you can generate synthetic tickets and events for the template, or keep the free text and give the branch to fewer people. The regex rule is not a middle ground, because it cannot find names.

The gate, and preview branches from the template

The workflow we tested is: create the anonymized branch, wait for the run, scan it, and only then make preview branches from it.

1. main production data
2. anonymized branch masking rules from git
3. wait state: anonymized
4. scan.mjs known values + patterns

Outcomes:

• pass: make preview branches
• fail: delete the branch

The wait matters if you use Neon's GitHub Action. neondatabase/create-branch-action v6 accepts masking_rules and returns the connection string right away, before the masking run ends. A job that connects next gets Access to this endpoint is temporarily disabled, and after our failed runs the branch stayed closed, so a job that only retries waits until it times out. scan.mjs --wait polls anonymized_status first and fails if the run did not finish. The repository has an example nightly workflow with these steps. We ran the steps through the scripts, not in GitHub Actions.

Two more things to know about the template branch:

• Do not give it an expiry. The API refused to create a child from an anonymized branch with expires_at: Branches with an expiration date cannot have child branches. Set expires_at on the previews instead.
• Check the data before you make previews. A scan report is only good for the data it read. 04-previews.mjs refuses to branch from the template unless its report is a pass, the branch has no database and no data that the scan did not read, and the data still has the scanned fingerprint. It then checks each preview against the same fingerprint.

From the passing template, three previews were ready in 2.3 to 2.5 seconds each, and each one matched the scanned fingerprint. To see the check refuse, the script makes a copy of the template, scans it, and writes one support ticket with a real user's email into it, the way someone might paste a production row into a shared branch:

previews behind the gate

$ node scripts/04-previews.mjs --template template-v3
gate: template-v3 passed its scan and still has the scanned data (0ff0ef425216dfe8)
preview-pr-101: ready in 2.4 s, checked after 6.1 s, same data as scanned: true
preview-pr-102: ready in 2.3 s, checked after 6.0 s, same data as scanned: true
preview-pr-103: ready in 2.5 s, checked after 6.1 s, same data as scanned: true
reset br-restless-paper-b9i5hgnz to the template in 3.9 s; same data as scanned again: true
stale gate: after the scan open; after one INSERT refused: data changed since the scan: public.support_tickets (rows 6558 -> 6559)

So treat the template as read-only once it passes: hand out preview branches, never the template's own connection string, and scan again after any change to it.

Then we broke it on purpose. 07-break.mjs runs a migration on a branch of main that adds a newsletter_signups table, a copy of 9,956 users' emails and names. The round 3 rules do not know that table, so we made a new template with them and scanned it:

scan after the migration

$ node scripts/scan.mjs --target template-drift --known main-drift --out results/scan-template-drift.json
scanned 29 columns in 8 tables of template-drift (39.1 s)
  customer_summary.email             rule=review    known: -                                            pattern: email 20000, phone 213
  newsletter_signups.email           rule=none      known: email 9956                                   pattern: -
  newsletter_signups.full_name       rule=none      known: name 9956                                    pattern: -
  users.search                       rule=review    known: -                                            pattern: email 20000, phone 207
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
FAIL: 2 problems

This is why the scan reads all the text and JSON columns, and not only the columns in the rules file. Rules describe the schema as it was when someone wrote them. The scan checks the schema as it is now. The new first_name column was not flagged, because a first name alone is not a full name.

What your tests will notice

Unique columns. Fake data generators repeat themselves. We generated each email function many times and counted the repeats:

On the unique users.email column, masking runs with dummy_free_email() and dummy_safe_email() both failed with duplicate key value violates unique constraint "users_email_key". In the final rule lab run, dummy_free_email() also failed on a 200-row table with a unique column, so a small table is no guarantee. The unmasked data stayed unreachable, but a nightly template job fails.

Where fake emails point. dummy_free_email() writes addresses at yahoo.com, hotmail.com and gmail.com, so a fake address can belong to a real person. If a preview app sends password resets or receipts, use a function that writes example.com addresses, or switch email off in previews.

Stability. Two templates made from the same parent with the round 1 rules shared no masked email, name, phone, address or IP, so a test that expects "user 42 is called X" fails on the next template. Values derived from the primary key do not move: with pg_catalog.concat('user-', id, '@example.com') and anon.pseudo_first_name(id), both templates matched in 20,000 of 20,000 rows. Only use such rules where the input is not PII itself, because anyone with the function can compute the output again.

Foreign keys. The docs say foreign key columns cannot be masked directly, and that the Console shows an alert instead. Through the API, a rule on orders.user_id was accepted and the run completed. Afterwards, all 40,191 orders belonged to a different user than on the parent, so joins between orders and users no longer match the parent.

Generators that can produce real values. With the first gate, 2 masked IP addresses from anon.dummy_ipv4() and 1 phone number from anon.dummy_phone_number() were equal to real values of other users. The round 3 rules take IP addresses from the 198.51.100.0/24 documentation range and give phone numbers the area code 000, which no real number has. Names are harder, because anon.dummy_name() returns real names.

anon.detect() suggests PII columns from their names. On our schema it missed full_name, street_address, signup_ip and every copy. It is a start for a rules file, not a check.

How long masking takes

The masking run rewrites every row of every table that has a rule. 06-timing.mjs masked three columns of one table at three sizes, three runs each, with the compute fixed at 1 CU:

The time grew much less than the row count: 100 times the rows took about 7 times as long. It also varied between runs, from 17 to 32 seconds at 1,000,000 rows, and three runs per size cannot show why. Neon reports the run's start and end in whole seconds. The time to a usable branch also includes creating the branch and starting its compute. For our 47.6 MB database it was 20.3 seconds for round 1 and 47.6 seconds for round 3, which rewrote six tables. We did not measure storage.

What we could not conclude

• One schema and data we wrote. The scanner found everything we planted, in the formats we chose. Real free text has misspellings, nicknames and names in other scripts, so its recall on real data will be lower, and we cannot say by how much.
• Names of people who are not users are invisible. In our data that left 5,985 names the scanner could not see by design: gift recipients, colleagues, accountants.
• The table order is observed, not documented. With a different schema Neon may mask the audit table first, and then the trigger's copies would stay unmasked.
• Small sizes and one region. One project in aws-eu-central-1, PostgreSQL 18, tables up to 1,000,000 rows, three runs per size.
• Static masking only. A Neon blog post says dynamic masking is planned. We tested what is available now.

Summary

Anonymized branches masked every column we named. The leaks were all in our schema: copies made at checkout, free text, JSON payloads, a trigger, a new table from a migration. A rules file cannot list what nobody knows is there, so treat the masked branch the way you treat a build: test it before anyone gets a connection string.

What to do:

1. Write rules for the obvious columns, then scan the masked branch to find the copies.
2. Mask audit and history tables in the same rule set, and check row counts after the run.
3. Replace JSON columns that hold PII. We found no rule that edits JSON in place.
4. Decide what to do with free text. regexp_replace removes emails and phone numbers, not names.
5. Use Random Unique Email (or a value derived from the primary key) for unique email columns, and keep fake emails on example.com.
6. Gate on the scan, wait for the masking run before you connect, and make previews only from a template that passed. Do not give the template an expiry.

Run it against your own project:

git clone https://github.com/The-DevOps-Daily/neon-anonymized-branches
cd neon-anonymized-branches && cp .env.example .env   # add a Neon API key
npm install && npm run setup && npm run seed
node scripts/02-template.mjs --rules rules/v1-obvious.json --name template-v1
npm run scan -- --target template-v1 --known main

We collected 34 trust stores from package files, published bundles and source trees, and verified the public test chains for the compromised keys against each one. Then we connected real clients to the public test hosts, and checked what Debian's fix does to the chains that 16,348 of the top 20,000 sites send.

Three results matter if you run anything outside a browser:

• Two roots, not one. Each compromised TLS key chains to emSign ECC Root CA - G3 and, through a Sectigo cross-sign, to AAA Certificate Services. Before the fix, Debian 11, 12 and 13, Ubuntu 24.04 as released, Alpine 3.18 and OpenJDK's cacerts held both. There, removing only the emSign root leaves the Sectigo path open.
• The root list predicts everything. In 560 OpenSSL verifications, the two roots decided every result. The fixed Go, Node.js, Bun and CPython releases block the keys in code. Node.js 24.22.1 and Go 1.26.10 rejected them even with the old Debian 12 bundle.
• Removing AAA broke one chain in our test. Of 16,125 served chains that verified before Debian's fix, one (jisc.ac.uk) failed after it with OpenSSL 3. GnuTLS and Go also failed on that site. 262 more fail only for a client that never looks for another path.

TL;DR

• Update ca-certificates and rebuild images from a fresh base. A digest pinned before 20 October keeps the old bundle.
• Debian 11, Alpine 3.18 and 3.19 and Node.js 20 get no fix. Their newest releases still trust the keys.
• On Debian and Ubuntu, delete /etc/ssl/certs/java/cacerts and run update-ca-certificates --fresh; the Java keystore is not rebuilt on its own.
• Upgrade to runtimes that block the keys in code: Go 1.26.10 or 1.27.3 (then rebuild your Go binaries), Node.js 22.23.4, 24.22.1 or 26.12.1, Bun 1.4.4, Python 3.11.18, 3.12.16, 3.13.18, 3.14.10 or 3.15.2.
• If you run a server, do not end your chain at a certificate issued by AAA Certificate Services.
• Run scan-tree.py on an unpacked image, a virtualenv or a JDK to find what still trusts the keys.

Prerequisites

• Container images, CI runners or servers that make outbound TLS connections
• Python 3.9 or later and the OpenSSL 3 command line tool, to run the companion scripts
• docker (or any tool that can export an image filesystem) if you want to scan images

What happened, in short

On 17 October, a researcher reported to the Apple, Chrome, Microsoft and Mozilla root programs that the private keys of eight eMudhra intermediate CAs could be computed from their public certificates. Mozilla confirmed it and had Firefox blocking the TLS intermediates through OneCRL by 22:59 UTC the same day. Six of the eight CAs issue TLS certificates; two are code signing CAs (Mozilla bug 2083449).

The keys are not theoretical. Between 20 and 21 October, people other than eMudhra logged 107,107 certificates signed with six of the keys to Certificate Transparency. 106,050 of them are proof-of-concept certificates for google.com (comment 44). Almost none of them carry a revocation URL.

Browsers get blocks through OneCRL, CRLSets and Apple's and Microsoft's trust updates. Almost everything else reads a file. We used the public chains and test hosts of the incident tracker, which also lists the patched versions.

One key, two certificates, two roots

Each of the six TLS keys appears in two intermediate certificates with the same subject and the same public key. One was issued by emSign Root TLS CA - G3, which is cross-signed by emSign ECC Root CA - G3. The other was cross-signed by Sectigo's AAA Certificate Services (the old Comodo root, called Comodo AAA Services root in Mozilla's data). A leaf signed with the key verifies through either one.

1. Forged leaf any hostname
2. EM DV TLS CA - G3B issued by eMudhra
3. EM DV TLS CA - G3B cross-signed by Sectigo
4. emSign Root TLS CA - G3 cross-signed
5. AAA Certificate Services Comodo, 2004
6. emSign ECC Root CA - G3 removed 20 Oct

Connections:

• Forged leaf -> EM DV TLS CA - G3B
• Forged leaf -> EM DV TLS CA - G3B
• EM DV TLS CA - G3B -> emSign Root TLS CA - G3
• EM DV TLS CA - G3B -> AAA Certificate Services
• emSign Root TLS CA - G3 -> emSign ECC Root CA - G3

Mozilla stopped trusting AAA Certificate Services for TLS in April 2025. Its root data (NSS builtins) marks the root as trusted for server authentication in version 2.74. Version 2.76, which came with NSS 3.111, does not (aaa-trust-history.txt reads it from the Firefox source). Debian's 20250419 package is built from 2.74, so Debian 11, 12 and 13 hold AAA, and so do Ubuntu 24.04's release-pocket package and Alpine 3.18. Before the fix, bundles built from later data held only the emSign root. These include Ubuntu's June 2026 update, Alpine 3.19, certifi, curl, Node.js, Bun, RHEL and Temurin. Debian's own August 2026 unstable package, built from 2.90, has no AAA either. So the cause is the age of the data, not Debian's packaging. That is why Debian removed two roots and the others removed one.

Finding 1: the root list decides it

Every script, recorded result and the scanner are in the companion repository. Each stage writes a file the next stage reads, and every number below comes from one of those files.

The-DevOps-Daily/emudhra-trust-store-check on GitHub

We took each trust store from a fixed file (a package, a published bundle or a source tree), so we know exactly which version we tested. The repository records the URL and SHA-256 of every download. For each store, a script first predicted the result: it accepts the emSign path if it holds emSign ECC Root CA - G3, and the Sectigo path if it holds AAA Certificate Services. Then openssl verify checked the 16 public test chains against each store, with the clock fixed at 30 October 00:00 UTC. There are eight chains per path, including leaves never logged to CT and chains through a rollover certificate.

The result: 560 verifications, 0 that differed from the prediction. A store accepts all eight chains of a path, or none.

Three details from the table are easy to miss:

• Temurin is not OpenJDK. OpenJDK's own cacerts holds both roots, and OpenJDK tells you to delete four of them by hand. Temurin builds its cacerts from Mozilla's data, so its 23 October release already holds neither. A JDK from another vendor can be either; check the file, not the version.
• certifi over-removed first. certifi 2026.10.18 removed every emSign root, including emSign Root CA - G1, which Mozilla still trusts. Users of the InCommon certificate service, which many US universities use, reported breakage in issue #439, and 2026.10.19 put the unaffected roots back. In our scan below, 47 of the served chains end at emSign Root CA - G1.
• Removing one root is not enough everywhere. We built one synthetic store, Debian 12's old bundle minus only emSign ECC Root CA - G3. It rejected all eight emSign-path chains and accepted all eight Sectigo-path chains. That is what you get if you apply the one-root workaround, which is right for Alpine 3.19 and later, to a Debian image.

the Sectigo-path chain against three Debian 12 bundles

# old bundle: holds both roots
$ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile debian-12-before.pem -untrusted aaa-chain.pem aaa-chain.pem
aaa-chain.pem: OK
# old bundle minus emSign ECC Root CA - G3 only
$ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile minus-emsign.pem -untrusted aaa-chain.pem aaa-chain.pem
aaa-chain.pem: OK
# 20250419~deb12u2: both roots removed
$ openssl verify -no-CApath -no-CAstore -attime 1793318400 -CAfile debian-12-fixed.pem -untrusted aaa-chain.pem aaa-chain.pem
C = NL, O = eMudhra BV, CN = EM DV TLS CA - G3B
error 20 at 1 depth lookup: unable to get local issuer certificate
error aaa-chain.pem: verification failed

-no-CApath -no-CAstore keeps the machine's own /etc/ssl/certs out of the test, and -attime makes the result repeatable. This command does not check revocation. The 16 intermediate certificates were revoked on 18 and 19 October. In the next section, our client commands did not turn on CRL or OCSP checking either, and with the old bundles they accepted the two public test chains.

Finding 2: a blocklist in the runtime beats a stale store

Four runtimes did not wait for trust stores. They are Go 1.26.10 and 1.27.3 (go.dev/issue/82345), Node.js 22.23.4, 24.22.1 and 26.12.1, Bun 1.4.4, and the October 26 CPython releases. Each one compiles a list of the eight compromised public keys into its TLS code. It rejects any chain that contains one of them. Bun's release notes mention only a root update; the list is in blocked_spkis.cpp. Our first stage downloads the four source files from the fix commits. All four hold the same eight SHA-256 hashes of the SubjectPublicKeyInfo. The lists match the key, not a certificate. So they cover both certificates of every key, and the rollover chains.

To see the difference with real clients, we connected curl, Python, Node.js and Go on a Debian 12 machine to two of the public test hosts: one serves the emSign path, the other the Sectigo path. Each client got a trust store from the matrix above. For Go, we built the same small HTTPS client with Go 1.26.9 and 1.26.10. All 24 results matched the prediction:

With the old Debian bundle, curl, Python, Node.js 24.22.0 and Go 1.26.9 accept a chain signed with a compromised key on both paths. Node.js 24.22.1 and Go 1.26.10 with the same bundle refuse both. Go 1.26.10 reports x509: no valid chains built, and Node.js reports CERT_REVOKED:

same stale bundle, two Node.js versions (30 October, 11:25 UTC)

# debian-12-before.pem is ca-certificates 20250419~deb12u1; get.js is in tools/
$ SSL_CERT_FILE=debian-12-before.pem SSL_CERT_DIR=/nonexistent ./node-24.22.0 --use-openssl-ca get.js dv-g3b-aaa-emudhra.we-have-a-problem.com
ACCEPT HTTP 200
$ SSL_CERT_FILE=debian-12-before.pem SSL_CERT_DIR=/nonexistent ./node-24.22.1 --use-openssl-ca get.js dv-g3b-aaa-emudhra.we-have-a-problem.com
reject CERT_REVOKED: certificate revoked

Debian 12's own Python (3.11.2-6+deb12u9 on our machine) has no denylist, and OpenJDK fixed Java only by removing roots from cacerts. Both are only as safe as the bundle they read. On macOS, Apple says that the system curl, git and python3 check /etc/ssl/cert.pem and accept these chains until a future update.

Finding 3: what removing AAA breaks

Removing AAA Certificate Services closes the Sectigo path, but other chains still use this 2004 root, and Apple and the incident tracker both warn that removing it can break sites. We measured how many.

We connected once to each of the top 20,000 domains of the Tranco list PYK6J, or to www. if the bare name failed. All connections came from one IPv4 address on 30 October between 11:11 and 11:36 UTC, and we saved the chain each server sent. 16,348 answered. Then a small C program on OpenSSL 3.0.22 verified every chain against Debian 12's bundle before and after the fix. Debian 13's bundles hold the same certificates. The program verified each chain in three ways:

• modern: the OpenSSL default since 1.1.0, which looks in the trust store first at each step
• nofirst: trust store first turned off, alternative chain search on
• strict: both turned off, so the verifier follows only the chain the server sent

Of the 16,348 chains, 16,125 verified before the fix in modern mode. 1,296 contain a Sectigo, Comodo, USERTrust or AAA certificate, and 283 include a certificate that AAA issued. After the fix:

No chain was anchored at emSign ECC Root CA - G3, so every break comes from AAA. Most of the 262 servers send an old copy of a current root, such as USERTrust RSA Certification Authority, cross-signed by AAA. A modern verifier finds the self-signed copy of that root in the store first and never reaches AAA. A verifier that follows only the served chain gets to AAA, does not find it, and fails:

Chains that break only for a client that follows the served chain

262 of the 16,111 chains that verified in strict mode before the fix, Tranco PYK6J top 20,000, scanned 30 October 2026. Grouped by the certificate issued by AAA Certificate Services that the server sends. In modern mode, OpenSSL 3.0 accepts all but one of them after the fix.

The one real break is jisc.ac.uk, and it is not a Sectigo certificate. The server sends its leaf and SSL.com TLS Issuing ECC CA R2, cross-signed by AAA, and nothing else. The same key also has a certificate from SSL.com TLS ECC Root CA 2022 (crt.sh 8505503579), which Debian holds. The server does not send it, and OpenSSL does not fetch missing intermediates. So on our Debian 12 machine after the fix, curl fails:

jisc.ac.uk before and after Debian removed AAA

# Debian 12, ca-certificates 20250419~deb12u2, 30 October 11:44 UTC
$ curl -sS -o /dev/null --max-time 20 -w '%{http_code}\n'  https://jisc.ac.uk/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
000
$ curl -sS -o /dev/null --max-time 20 -w '%{http_code}\n' --cacert stores/debian-12-before.pem --capath /nonexistent https://jisc.ac.uk/
301

To check this with a second TLS stack, we ran git ls-remote with both bundles. On Debian 12, git 2.39.5 uses GnuTLS 3.7.9. The targets were jisc.ac.uk and a seeded random sample of 20 of the other 261 strict-mode breaks. GnuTLS rejected jisc.ac.uk after the fix and accepted the 20 others. Our Go client, built with Go 1.26.9 and 1.26.10, also failed on jisc.ac.uk with the fixed bundle (x509: certificate signed by unknown authority).

So in our tests the AAA removal costs little: one chain in 16,125 for OpenSSL 3, and GnuTLS and Go agreed on the chains we gave them. The risk is in two places. The first is servers that send only an AAA cross-sign of an intermediate, like jisc.ac.uk; the fix is to send the intermediate that the current root issued. The second is clients that cannot search for another path, which we modelled but did not identify.

Where the old roots hide

Updating the host does not update what your services read. Check these first:

1. Images built before 20 October, and every image pinned by digest. The bundle is a file in a layer. Docker's own note puts it plainly: an image "keeps trusting them until it is rebuilt".
2. Bases and runtimes that will not get a fix. debian:bullseye and everything built on it, alpine:3.18 and alpine:3.19: their newest ca-certificates packages still trust both paths or the emSign path. Node.js 20, end of life since 30 April 2026, still bundles the emSign root and has no blocklist.
3. The Java keystore on Debian and Ubuntu. ca-certificates-java does not rebuild /etc/ssl/certs/java/cacerts when ca-certificates changes, so OpenJDK packages keep the old roots after the update (OpenJDK guidance).
4. Bundles copied by hand. A COPY --from=build /etc/ssl/certs/ca-certificates.crt into a scratch or distroless image, a certifi pin in a lockfile, or SSL_CERT_FILE, REQUESTS_CA_BUNDLE, PIP_CERT and NODE_EXTRA_CA_CERTS pointing at a file from last year.
5. Static Go binaries built with Go older than 1.26.10. The blocklist is compiled in, so only a rebuild adds it. go version -m ./binary prints the Go version that built it.
6. Runtimes with their own roots. Node.js before 22.23.4 or 24.22.1 and Bun before 1.4.4 use bundled roots by default, not the system store.

To find them, the companion repository has scan-tree.py. It reads a directory tree and reports bundles, keystores and binaries that hold either root, runtimes that carry the blocklist, and Go programs and CPython ssl modules that do not. This is its real output on a test tree with six files: the old Debian 12 bundle, Node.js 24.22.0 and 24.22.1, a fixed certifi bundle (correctly silent), the ssl module of Debian 12's Python, and Temurin 11's cacerts:

TRUSTS   fixture/etc/ssl/certs/ca-certificates.crt
         CA bundle with 150 certificates holds emSign ECC Root CA - G3 and AAA Certificate Services
TRUSTS   fixture/usr/local/bin/node
         embeds emSign ECC Root CA - G3 in its bundled roots
TRUSTS*  fixture/usr/lib/python3.11/lib-dynload/_ssl.cpython-311-aarch64-linux-gnu.so
         CPython ssl module without the eMudhra blocklist: it accepts whatever its trust store accepts
TRUSTS   fixture/usr/lib/jvm/temurin-11/lib/security/cacerts
         CA bundle with 118 certificates holds emSign ECC Root CA - G3
ok       fixture/opt/node-24.22.1/bin/node
         holds all 8 eMudhra blocklist keys (a byte match, not a runtime test)

4 finding(s) that still trust a chain through the compromised keys.
TRUSTS*: safe only if every trust store the program reads at run time is fixed.
No finding is not proof: only the bundle, keystore and binary formats listed in --help are checked.

TRUSTS* means the program has no blocklist of its own, so it is only as safe as the bundle it reads. An ok is a byte match for all eight keys, not a test of the running program. For a container image, export the filesystem and scan it:

mkdir rootfs && docker export "$(docker create IMAGE)" | tar -xf - -C rootfs
python3 scan-tree.py rootfs

Then fix what it finds:

# Debian and Ubuntu images: upgrade the bundle, then rebuild the Java keystore
apt-get update && apt-get install --only-upgrade -y ca-certificates
rm -f /etc/ssl/certs/java/cacerts && update-ca-certificates --fresh

# Alpine 3.20 and later (3.18 and 3.19 get no fix: move to a newer base)
apk upgrade --no-cache ca-certificates ca-certificates-bundle

# Check what an image actually has
dpkg-query -W ca-certificates   # fixed: 20250419~deb12u2, 20250419+deb13u1, 20260816ubuntu0.*
apk info -v ca-certificates-bundle   # fixed: 20261020-r0
rpm -q ca-certificates   # RHEL 9 fixed: 2026.2.92_v9.0.319-90.3.el9_8

What we could not conclude

• How many systems are still exposed. The matrix says what each version does, not how many copies of each version are running. We picked old versions on purpose.
• Other JDKs and platform verifiers. We did not check the cacerts of Amazon Corretto, Azul Zulu, Microsoft's build, Oracle JDK or distribution packages. We did not test Windows, macOS or Android, and relied on the vendors for those.
• RHEL's real extraction. We did not run trust extract; we parsed the p11-kit file and kept certificates trusted for server authentication. Our fixed RHEL 9 store has 118 certificates, the same count as the curl bundle of 20 October, which is a cross-check, not proof.
• Revocation. We did not test clients with CRL or OCSP checks turned on. The leaves on the test hosts have no revocation pointers. The eMudhra CRL endpoint also refused some User-Agents (comment 44), so we would not count on it.
• AAA beyond the top 20,000. One IPv4 address, one morning, the bare name or www. only. Internal services, other host names, mail servers and the long tail can differ, and a CDN can send a different chain to a different network.
• Which real clients behave like "strict" mode. We modelled them with two OpenSSL flags. OpenSSL 3.0 in modern mode handled 261 of the 262 chains, and GnuTLS 3.7.9 the 20 we sampled. We did not test Java, rustls, Windows, macOS or embedded TLS libraries on them, or Go beyond jisc.ac.uk.

Summary

• Each compromised key chains to two roots. Stores built from Mozilla data older than April 2025, including Debian 11, 12 and 13, need both emSign ECC Root CA - G3 and AAA Certificate Services removed.
• The roots in a store predicted all 560 of our verifications. Check the file, not the version string; scan-tree.py does that for a whole image.
• The fixed Go, Node.js, Bun and Python releases block the keys whatever the store says. Debian 11, Alpine 3.18 and 3.19 and Node.js 20 get no fix.
• In our OpenSSL test, removing AAA broke 1 of 16,125 served chains (jisc.ac.uk). GnuTLS and Go failed on it too, and the fix for it is on the server.

For a one-line source change, every cache worked. The rebuild went from 79.0 s with no cache to 19.2 to 26.3 s with any registry-backed cache, because BuildKit skipped the whole builder stage, npm install included. For this Dockerfile, the cheapest backends were the best ones. Inline and registry mode=min cost 0.0 and 1.2 s to export. Inline adds only metadata to the image, and mode=min stored a separate 47.7 MB cache with the same layers as the image. mode=max stored 259.9 MB, added 29.8 s of export to every cold build, and made the dependency bump slower, not faster: the cached compiler toolchain saved no time once its layers had to be pulled, and the export added about 20 s. The local directory cache, written fresh each run, was the slowest of all for the one-line change.

TL;DR

• Source change: all caches skip the builder stage. Median 19.2 to 26.3 s with inline or registry, against 79.0 s with no cache. In our three runs each, inline and mode=min did as well as mode=max, because BuildKit only loads the cache for the last step that matches.
• Export cost is mostly compression. mode=max spent 27.9 s in preparing build cache for export on a cold build, and about 2 s writing 259.9 MB to the registry. mode=min exported in 1.2 s, inline in 0.0 s.
• Dependency bump: mode=max did not help. It kept the compiler toolchain, but pulling and unpacking the cached base and toolchain layers took 15.0 to 20.6 s per build, about what mode=min spent pulling the base image and running apk add again (13.4 to 19.8 s). Its export then added 18.4 to 24.7 s. Median 132.4 s for mode=max against 110.8 s for mode=min.
• Local cache, fresh directory each run: 74.7 s for a one-line change, because it repeats the full mode=max export every time.
• What did not change: npm install ran on every dependency bump with every backend. A cache saves the steps before the change, never the step that changed.
• Our registry ran on the same machine, so these times leave out the network. The sizes carry over: 47.7 MB pulled per warm source build, 259.9 MB per mode=max cache.

Prerequisites

• Docker with Buildx (we used Docker 29.9.2, Buildx 0.38.1 and BuildKit 0.34.1).
• A registry you can push to. We ran registry:2.8.3 on the build machine; GHCR, ECR, Docker Hub or your own registry work the same way.
• Node.js and npm, only to create the dependency change (npm install --package-lock-only).
• If you want to rerun the whole matrix: a 4 vCPU Linux machine and about 80 minutes. We used a droplet on DigitalOcean (referral link).

Why a fresh builder changes everything

On a laptop, BuildKit keeps every layer it built in its own store, so the second docker build reuses them without any configuration. A CI runner that starts from a clean VM or container has no store. Unless you tell BuildKit where to import a cache from and where to export it to, every build is a cold build.

BuildKit can export its cache in several ways. inline writes cache metadata into the image you push. registry writes a separate cache manifest to a registry. local writes an OCI layout to a directory that your CI then saves and restores. gha and s3 write to GitHub's cache service and to S3. mode=min, the default, exports only the layers of the final image. mode=max also exports the layers of every intermediate stage, such as a builder stage.

The question for a CI pipeline is which of those pays for itself, for the changes that you actually push.

How we measured

The Dockerfile. We used src/paymentservice from Google's Online Boutique demo at commit c7cb6d5b, without changes. It is a real multi-stage Node service, and both base images are pinned by digest. This is the whole Dockerfile without its license header:

# Define a default value so it's not empty if the builder fails to provide it
ARG BUILDPLATFORM=linux/amd64

FROM --platform=$BUILDPLATFORM node:24.20.0-alpine@sha256:e67514e5d0f6c46656005e1b693b2ec9d52e80b641307de684d4a015ba7a4eaf AS builder

# Some packages (e.g. @google-cloud/profiler) require additional
# deps for post-install scripts
RUN apk add --update --no-cache \
    python3 \
    make \
    g++

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install --only=production

FROM alpine:3.24.1@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b

RUN apk add --no-cache nodejs

WORKDIR /usr/src/app

COPY --from=builder /usr/src/app/node_modules ./node_modules

COPY . .

EXPOSE 50051

ENTRYPOINT [ "node", "index.js" ]

The expensive work is in the builder stage: a compiler toolchain from Alpine's mirror, then npm install. The final stage installs Node from Alpine, copies node_modules, and copies the source last.

An ephemeral runner, every time. Each build got a new BuildKit builder (docker buildx create --driver docker-container --bootstrap), and we removed it after the build. Nothing survived in the builder's store between builds except what the cache backend exported. The base image mirror and the Docker host itself did persist. We recorded builder start-up separately; it is the same for every backend.

Three scenarios:

• Cold: an empty cache. The build exports its cache to the backend.
• Source change: one line in server.js (a log message). The build imports the cache that the last cold run wrote.
• Dependency change: @grpc/grpc-js from 1.14.4 to 1.14.7, made with npm install --package-lock-only --save-exact, the size of a typical Dependabot patch bump. The lockfile diff changes only that package (npm also re-sorted two keys in package.json). It imports the same cache.

Warm builds import from the cold cache and export to a separate scratch location, so every warm run starts from the same cache and still pays the export cost that a real pipeline pays.

The backends:

Every build pushed its image with --provenance=false --sbom=false, so attestations do not add noise.

Repeats. We planned five runs of every backend and scenario. The shared droplet ran slower than planned and we had a fixed budget, so we got 5 cold runs per backend, 3 source-change runs, and 3 (2 for the local cache) for the dependency bump. Runs went in round-robin order inside each scenario (run 1 of every backend, then run 2, and so on), which spreads slow periods on the shared host across the backends but does not remove order effects (see the caveats). The tables show the median and the number of runs (with 2 runs, the median is their mean). We dropped the page cache before each build and logged the CPU counters around each one, because the droplet's vCPUs are shared and steal time moves build times.

What we recorded. Wall time of docker buildx build, BuildKit's own step timings from --progress=plain (which steps were CACHED, how long the export took), the cache size from the registry manifest or the cache directory, and the bytes BuildKit pulled from the cache registry, from the registry's access log.

Where the cache lived. The registry for inline and type=registry was a registry:2.8.3 container on the same droplet, at 127.0.0.1:5000. We did not have a container registry we could use at no cost, and DigitalOcean's Spaces object storage has a monthly minimum, so there was no remote store that cost nothing. On loopback, writing the 259.9 MB mode=max cache took about 2 s, so our export and import times are close to the best case. On a real runner the same bytes cross a network to GHCR, ECR or your own registry, and the time grows with the size. That is why the tables show bytes next to every time: the bytes carry over to your setup, our seconds do not.

Base images. Every fresh builder resolved node:24.20.0-alpine and alpine:3.24.1, and downloaded their layers whenever a step needed them. We pointed BuildKit at a pull-through mirror (registry:2.8.3 with REGISTRY_PROXY_REMOTEURL, also on the droplet), so Docker Hub's rate limits and speed stay out of the numbers. A real ephemeral runner pulls them from Docker Hub or a mirror on every build that needs them, so add that time to the rows where the FROM step ran.

Package downloads were real. apk add fetched from Alpine's CDN and npm install fetched from the npm registry whenever those steps ran. That is the cost a cache saves, so we left it in.

Why there is no type=s3 or type=gha. type=s3 needs an S3 bucket. The only free option here was an S3 emulator on the same droplet, which would measure the emulator, not S3. type=gha works only inside GitHub Actions. We left both out.

Cold builds and what the export costs

Cold builds: median of 5 runs. Export is the exporting cache step in the BuildKit log; for inline, the preparing layers for inline cache line.

With no cache, a cold build took a median 79.0 s on a fresh builder. npm install was 45.3 s of that, the compiler toolchain (apk add python3 make g++) 6.5 s, and compressing and pushing the image about 9.5 s.

The totals in the table are noisier than the export column. npm install alone took between 29.2 and 70.9 s across the cold runs, because it downloads from the npm registry, and the droplet reported 8 to 27% CPU steal per build. Read the export column for what each backend costs:

• inline costs nothing measurable. BuildKit logged preparing layers for inline cache 0.0s done. The cache is metadata in the image config, and its layers are the image's own.
• registry, mode=min added 1.2 s for a 47.7 MB cache: the same 5 layers as the final image, stored under the cache tag.
• registry, mode=max added 29.8 s for 259.9 MB in 12 layers. 27.9 s of that was one line in the BuildKit log, preparing build cache for export: BuildKit compressing the builder stage's layers (the Node base image, the compiler toolchain and node_modules), which a mode=min export never touches. Writing the layers to the registry took about 2 s, on loopback.
• local, mode=max added 39.1 s for the same 259.9 MB, written as an OCI layout on disk. Preparing took 30.7 s here too.

So the expensive part of a mode=max export is compression, and it happens before a byte leaves the machine. On a real network, the upload of 259.9 MB comes on top.

A one-line source change

The none row is the cold build. Pulling cached layers is the time BuildKit reported on the cached step that had to be unpacked; pulled from registry is from the cache registry's access log.

Every backend skipped the whole builder stage. BuildKit marked 7 of the 8 build steps CACHED, including RUN npm install, and only COPY . . ran. That was true for inline and registry mode=min too, although those caches do not contain a single layer of the builder stage.

That works because of how BuildKit finds a cache hit. A step's cache key comes from its instruction and the keys of the steps before it, and a COPY from the build context adds a checksum of the files it copies. In this graph, that let BuildKit match the final stage's COPY --from=builder step without running the builder stage. Its solver documentation says it "will not require the cache to be loaded or exists for all the vertexes in the graph but will only load it for the final node that is determined to match cache". Here, that node is COPY --from=builder, which is a layer of the final image, and every backend exported it. The builder stage never ran, and none of its layers were downloaded: every registry build pulled the same 47.7 MB, the final image's layers, even from the 259.9 MB mode=max cache.

What a warm build still paid for was pulling and unpacking those cached layers to put COPY . . on top. BuildKit reports that time on the cached COPY --from step: a median of 12.4 to 16.9 s with the registry backends, out of 19.2 to 26.3 s for the whole build.

The local backend was the exception: 74.7 s. Importing from the directory was quick, but exporting to a fresh directory took 52.4 s, because every run wrote the full mode=max cache again. The registry backends exported to a new tag in the same repository, and that took 1.4 s or less, because the layers they referenced were already in the registry. We exported to a new directory on purpose, to stop the cache from growing; the section on the local cache below explains the trade.

A dependency bump

A changed lockfile invalidates COPY package*.json in the builder stage, so npm install ran in every build, with every backend. The backends differ only in the step before it, the compiler toolchain:

• inline and registry mode=min do not hold the builder stage, so BuildKit pulled the Node base image from the mirror (4.3 to 10.2 s) and ran apk add python3 make g++ again (a median 9.9 s). Only the final stage's RUN apk add --no-cache nodejs and WORKDIR came from the cache.
• registry mode=max and local held it, and BuildKit marked the toolchain step CACHED. But to run npm install on top of it, BuildKit first had to pull and unpack the cached layers, the Node base image and the toolchain, including one layer of 112.3 MB. With the registry, that took 15.0 to 20.6 s per build, about the same as the 13.4 to 19.8 s that mode=min spent on the base image and apk add. The cache hit saved nothing.

We expected mode=max to win this scenario, and wrote that down before the runs. It did not.

Then the export. With mode=max, BuildKit also had to compress the builder stage's new npm install layer, which the image does not contain, and preparing build cache for export took 19.0 s, against 0.7 s for the whole mode=min export.

Put together, the dependency bump took a median 132.4 s with mode=max and 110.8 s with mode=min, and mode=max was slower in each of the three rounds. The export accounts for most of the gap: mode=max took 18.4 to 24.7 s to export, against under 1 s for mode=min, while the toolchain stage cost about the same either way. mode=max also pulled 196.8 MB from the cache registry per build, against 26.0 MB for mode=min. The local cache, also mode=max, took 132.1 s (2 runs). With a remote registry the gap would grow, because those 196.8 MB and the new npm install layer would cross the network.

Build time on a fresh builder (median)

One new BuildKit builder per build; cache registry on the same droplet (loopback). Medians; the none bars are the cold build; run counts are in the tables. s-4vcpu-8gb, 2 November 2026.

The local cache: an expensive export

type=local writes the cache as an OCI layout in a directory, and your CI saves and restores that directory between runs, for example with a cache action. In our runs the import side was cheap: reading the cache manifest took 0.1 s or less, and the cached layers came from local disk.

The export side was the most expensive of all the backends. A cold build spent 39.1 s on it, and a one-line change spent 52.4 s, because each run wrote the full mode=max cache into a new, empty directory. The registry backends wrote their warm cache to a new tag in a registry that already held the layers, and finished in 1.4 s or less.

We wrote to a fresh directory on purpose. When you export to the same directory you import from, BuildKit by default keeps every old blob. Docker's GitHub Actions cache guide puts it plainly: exporting to a directory that already holds a cache "leaves the blobs of the previous export behind, so the directory grows with every run". BuildKit 0.34.1 documents an export option for this, reset=true, which removes "any blobs in the cache directory that are not referenced by the current manifests in index.json" and is "useful for keeping the local cache directory from growing indefinitely". We planned to measure both the growth and reset=true, and ran out of droplet time first, so we cannot tell you how much either one changes.

Which backend to use

For this Dockerfile, on a runner that starts empty:

• Use inline or registry mode=min. They gave the same warm builds as mode=max for a source change, cost 0 to 1.2 s to export, and stored no more than the image's own 47.7 MB of layers. Inline needs no extra tag at all, but it only works when you push the image, and its cache is only as old as the last image you pushed.
• Use mode=max only when a builder step is slow to run and cheap to pull. BuildKit pulls and unpacks a cached layer before it can build on top of it. Here the toolchain step took about 10 s to run, and pulling its cached layers with the base image took about as long as pulling the base image and running it again. A long compile that produces a small output layer is the opposite case, and that is where mode=max earns its export cost.
• For a slow Gradle or Bazel step, look past the layer cache. BuildKit reuses a RUN step whole or runs it again, so a source change before that step reruns the whole build. A tool's own remote build cache works per task and skips the tasks whose inputs did not change. Bitrise, for example, sells one for Gradle, Bazel, Xcode and React Native that it says works on any CI (checked on its public pages on 2 November 2026). We did not test one here.
• Price the export, not just the hit. mode=max added about 30 s of compression to every cold build and 19.0 s to every dependency bump, before any upload.
• Be careful with type=local. Writing to a fresh directory each run, the older way to keep the cache small, repeats the full mode=max export every time, which took 52.4 s on a one-line change. Docker's guide now reuses the directory with reset=true, which needs Buildx 0.35 or later and removes blobs that the new cache no longer references. We did not get to test it.

To check your own Dockerfile, build it twice on a fresh builder with --progress=plain and read three lines per step: whether it says CACHED, how long its DONE takes after that (the pull), and how long preparing build cache for export takes. Those three numbers decide which backend wins.

What we could not conclude

• The cache registry was on loopback. Our export and import times leave out the network. With a remote registry, add the transfer of 47.7 MB for a source change with any backend, and of the 259.9 MB mode=max cache on every cold export that writes new layers.
• One Dockerfile. In this one, the expensive work comes before COPY . ., so any cache skips it on a source change. In a Dockerfile whose expensive step comes after COPY . ., no backend can help with a source change. The Go frontend of the same demo is like that: its go build step comes after COPY . . and took 66.0 to 105.8 s in three uncached runs of it that we made on the same droplet for our CI build memory post.
• No no-cache rerun of the changes. With no cache, a source change or a dependency bump is a cold build, so we used the cold median (79.0 s) as the reference and did not rerun it with the changed files. The cold builds ran earlier in the session, when steal was lower (8 to 27%, against 15 to 39% for the warm builds), so if anything that reference is a little fast.
• Fewer warm runs than planned. Builds on the shared droplet ran slower than we planned, and we had a fixed budget. Cold builds have 5 runs per backend, source changes 3, and dependency bumps 3 (2 for the local cache). The direction of every comparison we draw held in every round; the exact medians would move with more runs.
• Fixed order inside each round. The backends ran in the same order in every round, so a backend's position could bias its time. The host-wide steal we logged per build does not show a pattern by position, but it rose over the session (from 8% to 39% per build). Dependency round 3 ran in the opposite order, mode=max first, and mode=max was still the slowest registry backend. We point at the step timings because they depend less on any of this.
• The local result is about a fresh destination. We exported each local warm build to a new, empty directory, while the registry backends exported next to layers that were already stored. That shows that a fresh local destination is expensive, not that type=local is slow in general, and we did not measure the cache action's save and restore of the directory.
• Untested: type=gha and type=s3, RUN --mount=type=cache cache mounts, two pipelines writing the same cache tag at once, and the reset=true option for local caches.

Run it yourself

Create a 4 vCPU Ubuntu machine with Docker, copy the files below to /root/bench/, and run:

bash /root/bench/registries.sh
STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh prep   # contexts for the source and dependency changes
STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh cold   # 5 rounds x 5 backends
STOP_AT_UTC=2099-01-01 bash /root/bench/p2.sh warm   # 5 source rounds, then dependency rounds while time allows
python3 summarize_p2.py /root/res/p2

The script below is the one we ran. Our run did not use warm to the end: to fit the budget we stopped it after three source rounds, ran two dependency rounds with warm2, and a third with dep3, which starts with mode=max. packnow (a tar and zstd test of the local cache) and grow (the same directory in and out, with and without reset=true) are in the script too; we ran out of time before them.

Each build writes its own directory with the BuildKit log, the registry access log for that build, and the sizes, so you can check any single number.

#!/bin/bash
# The cache registry and the two pull-through mirrors, all on the build machine.
docker pull -q moby/buildkit:v0.34.1
docker pull -q registry:2.8.3
docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3
docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3
docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3
git init -q /root/src/msd
git -C /root/src/msd remote add origin https://github.com/GoogleCloudPlatform/microservices-demo
git -C /root/src/msd fetch -q --depth 1 origin c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712
git -C /root/src/msd checkout -q FETCH_HEAD

[registry."docker.io"]
  mirrors = ["127.0.0.1:5001"]
[registry."gcr.io"]
  mirrors = ["127.0.0.1:5002"]
[registry."127.0.0.1:5000"]
  http = true
  insecure = true
[registry."127.0.0.1:5001"]
  http = true
  insecure = true
[registry."127.0.0.1:5002"]
  http = true
  insecure = true

#!/bin/bash
# Post 2: BuildKit cache backends on a fresh builder per build.
# usage: p2.sh prep | smoke | cold | pack | warm | grow | warmnone
set -u
export PATH=/usr/local/bin:$PATH
B=/root/bench R=/root/res/p2 C=/root/p2
SVC=/root/src/msd/src/paymentservice
# stop starting new builds after this time, to stay inside a machine budget (our last stages used 2026-11-02T18:50:00Z)
STOP_AT=$(date -u -d "${STOP_AT_UTC:-tomorrow}" +%s)
REG=127.0.0.1:5000 BK=moby/buildkit:v0.34.1
mkdir -p $R $C/cache
log() { echo "$(date -u +%FT%TZ) $*" | tee -a $R/progress.log; }

prep() {
  rm -rf $C/ctx-*
  cp -a $SVC $C/ctx-base
  cp -a $SVC $C/ctx-src
  sed -i 's/PaymentService gRPC server started on port/PaymentService gRPC server listening on port/' $C/ctx-src/server.js
  cp -a $SVC $C/ctx-dep
  ( cd $C/ctx-dep && npm install --package-lock-only --save-exact --ignore-scripts --no-audit --no-fund @grpc/grpc-js@1.14.7 )
  for i in 1 2 3 4 5; do
    cp -a $SVC $C/ctx-grow$i
    sed -i "s/PaymentService gRPC server started on port/PaymentService gRPC server (build $i) started on port/" $C/ctx-grow$i/server.js
  done
  diff -u $SVC/server.js $C/ctx-src/server.js > $R/src-change.diff
  diff -u $SVC/package.json $C/ctx-dep/package.json > $R/dep-change.diff
  diff -u $SVC/package-lock.json $C/ctx-dep/package-lock.json >> $R/dep-change.diff
  grep -c "listening on port" $C/ctx-src/server.js
  head -60 $R/dep-change.diff
}

# build CFG SCEN RUN [CTX]
build() {
  local cfg=$1 scen=$2 r=$3 ctx=${4:-}
  local out=$R/$scen/$cfg/$r name=p2-$cfg-$scen-$r
  if [ $(date +%s) -gt $STOP_AT ]; then log "$scen $cfg $r SKIPPED (time budget)"; return; fi
  rm -rf $out; mkdir -p $out
  [ -z "$ctx" ] && case $scen in cold) ctx=$C/ctx-base ;; src) ctx=$C/ctx-src ;; dep) ctx=$C/ctx-dep ;; esac
  local img=$REG/paymentservice:$cfg-$scen from=() to=()
  case $cfg in
    none) ;;
    inline)
      to=(--cache-to type=inline)
      if [ $scen = cold ]; then img=$REG/paymentservice:inline-base
      else from=(--cache-from type=registry,ref=$REG/paymentservice:inline-base); fi ;;
    regmin|regmax)
      local m=${cfg#reg}
      if [ $scen = cold ]; then to=(--cache-to type=registry,ref=$REG/cache:$m,mode=$m)
      else from=(--cache-from type=registry,ref=$REG/cache:$m); to=(--cache-to type=registry,ref=$REG/cache:$m-scratch,mode=$m); fi ;;
    local)
      if [ $scen = cold ]; then rm -rf $C/cache/local; to=(--cache-to type=local,dest=$C/cache/local,mode=max)
      else rm -rf $C/cache/scratch; from=(--cache-from type=local,src=$C/cache/local); to=(--cache-to type=local,dest=$C/cache/scratch,mode=max); fi ;;
    grow)
      from=(--cache-from type=local,src=$C/cache/grow); to=(--cache-to type=local,dest=$C/cache/grow,mode=max) ;;
    growr)
      from=(--cache-from type=local,src=$C/cache/growr); to=(--cache-to type=local,dest=$C/cache/growr,mode=max,reset=true) ;;
  esac
  sync; echo 3 > /proc/sys/vm/drop_caches
  local t0 t1 t2 t3 rc
  t0=$(date +%s.%N)
  docker buildx create --name $name --driver docker-container --driver-opt network=host \
    --driver-opt image=$BK --buildkitd-config $B/buildkitd.toml --bootstrap > $out/create.log 2>&1
  t1=$(date +%s.%N)
  local cpu0; cpu0=$(grep '^cpu ' /proc/stat)
  docker buildx build --builder $name --progress=plain --provenance=false --sbom=false \
    --metadata-file $out/meta.json "${from[@]}" "${to[@]}" \
    --output type=image,name=$img,push=true,registry.insecure=true $ctx > $out/build.log 2>&1
  rc=$?
  t2=$(date +%s.%N)
  local cpu1; cpu1=$(grep '^cpu ' /proc/stat)
  docker buildx rm $name > /dev/null 2>&1
  t3=$(date +%s.%N)
  docker logs --since $t1 --until $t2 reg-cache > $out/registry.log 2>&1
  { echo "rc=$rc"; echo "t0=$t0"; echo "t1=$t1"; echo "t2=$t2"; echo "t3=$t3"; echo "ctx=$ctx"; echo "cpu0=$cpu0"; echo "cpu1=$cpu1"; } > $out/result.env
  case $cfg in
    local) if [ $scen = cold ]; then d=$C/cache/local; else d=$C/cache/scratch; fi
           echo "cache_bytes=$(du -sb $d | cut -f1)" >> $out/result.env
           echo "cache_blobs=$(ls $d/blobs/sha256 | wc -l)" >> $out/result.env
           [ $scen = cold ] || rm -rf $C/cache/scratch ;;
    grow|growr) echo "cache_bytes=$(du -sb $C/cache/$cfg | cut -f1)" >> $out/result.env
           echo "cache_blobs=$(ls $C/cache/$cfg/blobs/sha256 | wc -l)" >> $out/result.env ;;
  esac
  case $cfg-$scen in
    regmin-cold) python3 $B/regsize.py $REG cache min > $out/cache-size.json ;;
    regmax-cold) python3 $B/regsize.py $REG cache max > $out/cache-size.json ;;
    regmin-*) python3 $B/regsize.py $REG cache min-scratch > $out/cache-size.json ;;
    regmax-*) python3 $B/regsize.py $REG cache max-scratch > $out/cache-size.json ;;
  esac
  python3 $B/regsize.py $REG paymentservice ${img##*:} > $out/image-size.json
  log "$scen $cfg $r rc=$rc build=$(python3 -c "print(round($t2-$t1,1))")s"
}

pack_test() {
    for r in r1 r2 r3 r4 r5; do
      sync; echo 3 > /proc/sys/vm/drop_caches
      t0=$(date +%s.%N); tar -C $C/cache/local -cf - . | zstd -T0 -q -f -o $C/local.tar.zst; sync; t1=$(date +%s.%N)
      rm -rf $C/unpack; mkdir -p $C/unpack; sync; echo 3 > /proc/sys/vm/drop_caches
      t2=$(date +%s.%N); zstd -dc -q $C/local.tar.zst | tar -C $C/unpack -xf -; sync; t3=$(date +%s.%N)
      echo "$r pack_s=$(python3 -c "print(round($t1-$t0,2))") unpack_s=$(python3 -c "print(round($t3-$t2,2))") dir_bytes=$(du -sb $C/cache/local | cut -f1) tar_zst_bytes=$(stat -c %s $C/local.tar.zst)" | tee -a $R/pack.txt
    done
    rm -rf $C/unpack $C/local.tar.zst; log PACK-DONE
}

case ${1:-} in
  prep) prep ;;
  smoke) for c in none inline regmin regmax local; do build $c cold s0; done; log SMOKE-DONE ;;
  cold) for r in r1 r2 r3 r4 r5; do for c in none inline regmin regmax local; do build $c cold $r; done; done; log COLD-DONE ;;
  warm) # all source-change rounds first, then whole dependency rounds while they fit before STOP_AT
    for r in r1 r2 r3 r4 r5; do for c in inline regmin regmax local; do build $c src $r; done; done
    last=400
    for r in r1 r2 r3 r4 r5; do
      t=$(date +%s)
      if [ $((t + last)) -gt $STOP_AT ]; then log "dep $r and later skipped: a round needs about ${last}s"; break; fi
      for c in inline regmin regmax local; do build $c dep $r; done
      last=$(( $(date +%s) - t ))
    done
    log WARM-DONE ;;
  warm2) # switched to after source round 3: dependency rounds first, then the last source rounds, while they fit
    last=400
    for r in r1 r2 r3 r4 r5; do
      t=$(date +%s)
      if [ $((t + last)) -gt $STOP_AT ]; then log "dep $r and later skipped: a round needs about ${last}s"; break; fi
      for c in inline regmin regmax local; do build $c dep $r; done
      last=$(( $(date +%s) - t ))
      [ $r = r3 ] && break
    done
    for r in r4 r5; do
      t=$(date +%s)
      if [ $((t + 150)) -gt $STOP_AT ]; then log "src $r skipped: time budget"; break; fi
      for c in inline regmin regmax local; do build $c src $r; done
    done
    log WARM-DONE ;;
  dep3) # third dependency round, started after round 2 (time budget)
    for c in regmax regmin inline local; do build $c dep r3; done
    log WARM-DONE ;;
  packnow) pack_test ;;
  warmnone) for r in r1 r2 r3 r4 r5; do for s in src dep; do build none $s $r; done; done; log WARMNONE-DONE ;;
  pack) log "pack deferred until after the warm builds" ;;
  grow) # local cache pack/unpack test, then the same-directory growth test, without and with reset=true
    if [ $(date +%s) -lt $((STOP_AT + 120)) ]; then pack_test; fi
    rm -rf $C/cache/grow $C/cache/growr; cp -a $C/cache/local $C/cache/grow; cp -a $C/cache/local $C/cache/growr
    echo "start cache_bytes=$(du -sb $C/cache/grow | cut -f1) blobs=$(ls $C/cache/grow/blobs/sha256 | wc -l)" > $R/grow-start.txt
    for i in 1 2 3 4 5; do build grow src g$i $C/ctx-grow$i; build growr src g$i $C/ctx-grow$i; done; log GROW-DONE ;;
  *) echo "usage"; exit 2 ;;
esac

#!/usr/bin/env python3
"""Sum the compressed layer bytes behind a registry tag (image or BuildKit cache)."""
import json, sys, urllib.request

reg, repo, tag = sys.argv[1:4]
ACCEPT = ", ".join([
    "application/vnd.oci.image.index.v1+json",
    "application/vnd.oci.image.manifest.v1+json",
    "application/vnd.docker.distribution.manifest.list.v2+json",
    "application/vnd.docker.distribution.manifest.v2+json",
])

def get(ref):
    req = urllib.request.Request(f"http://{reg}/v2/{repo}/manifests/{ref}", headers={"Accept": ACCEPT})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)

def walk(m, layers):
    for d in m.get("manifests", []):
        mt = d.get("mediaType", "")
        if "manifest" in mt or "index" in mt:
            walk(get(d["digest"]), layers)
        elif "layer" in mt:
            layers[d["digest"]] = d["size"]
    for d in m.get("layers", []):
        layers[d["digest"]] = d["size"]

try:
    top = get(tag)
except Exception as e:
    print(json.dumps({"error": str(e)})); sys.exit(0)
layers = {}
walk(top, layers)
print(json.dumps({"tag": f"{repo}:{tag}", "mediaType": top.get("mediaType"), "layers": len(layers), "bytes": sum(layers.values()), "sizes": sorted(layers.values(), reverse=True)}))

#!/usr/bin/env python3
"""Summarize post 2 runs: parse BuildKit plain progress, registry logs and sizes; medians per cell."""
import json, os, re, statistics, sys

root = sys.argv[1]
VERTEX = re.compile(r"^#(\d+) (.*)$")
DONE = re.compile(r"^DONE ([\d.]+)s$")
SUBDONE = re.compile(r"^(preparing layers for inline cache|preparing build cache for export|exporting layers|pushing layers|exporting manifest list|exporting config).*?([\d.]+)s done$")

def env(path):
    d = {}
    if os.path.exists(path):
        for line in open(path):
            if "=" in line:
                k, v = line.strip().split("=", 1)
                d[k] = v
    return d

def parse_build(path):
    v = {}
    order = []
    for line in open(path, errors="replace"):
        line = line.rstrip("\n")
        m = VERTEX.match(line)
        if not m:
            continue
        vid, rest = int(m.group(1)), m.group(2)
        if vid not in v:
            v[vid] = {"name": rest, "done": None, "cached": False, "error": False, "sub": {}}
            order.append(vid)
            continue
        if rest == "CACHED":
            v[vid]["cached"] = True
        elif rest.startswith("ERROR"):
            v[vid]["error"] = True
        else:
            d = DONE.match(rest)
            if d:
                v[vid]["done"] = float(d.group(1))
            s = SUBDONE.match(rest)
            if s:
                v[vid]["sub"][s.group(1)] = v[vid]["sub"].get(s.group(1), 0) + float(s.group(2))
    return [v[i] for i in order]

def registry_bytes(path):
    got = put = 0
    seen_logrus = False
    lines = open(path, errors="replace").read().splitlines() if os.path.exists(path) else []
    for line in lines:
        if 'msg="response completed"' in line:
            seen_logrus = True
            meth = re.search(r"http\.request\.method=(\w+)", line)
            uri = re.search(r'http\.request\.uri="([^"]+)"', line)
            w = re.search(r"http\.response\.written=(\d+)", line)
            if meth and uri and w and meth.group(1) == "GET" and "/blobs/" in uri.group(1):
                got += int(w.group(1))
    if not seen_logrus:
        for line in lines:
            m = re.search(r'"GET (/v2/[^ ]*/blobs/[^ ]*) HTTP/[\d.]+" (\d+) (\d+)', line)
            if m and m.group(2) == "200":
                got += int(m.group(3))
    return got

def step_time(vs, pattern):
    for x in vs:
        if re.search(pattern, x["name"]):
            return 0.0 if x["cached"] else x["done"]
    return None

def run(d):
    r = env(os.path.join(d, "result.env"))
    if not r:
        return None
    vs = parse_build(os.path.join(d, "build.log"))
    steps = [x for x in vs if re.match(r"^\[(builder|stage-\d+) \d+/\d+\]", x["name"])]
    exp = [x for x in vs if x["name"].startswith("exporting cache to")]
    img = [x for x in vs if x["name"] == "exporting to image"]
    imp = [x for x in vs if x["name"].startswith("importing cache manifest")]
    info = {
        "rc": int(r["rc"]),
        "bootstrap_s": round(float(r["t1"]) - float(r["t0"]), 2),
        "build_s": round(float(r["t2"]) - float(r["t1"]), 2),
        "steps": len(steps), "cached_steps": sum(1 for x in steps if x["cached"]),
        "cached_names": [x["name"] for x in steps if x["cached"]],
        "apk_builder_s": step_time(vs, r"RUN apk add --update"),
        "npm_install_s": step_time(vs, r"RUN npm install"),
        "apk_final_s": step_time(vs, r"RUN apk add --no-cache nodejs"),
        "copy_modules_s": step_time(vs, r"COPY --from=builder"),
        "export_cache_s": sum(x["done"] or 0 for x in exp) if exp else None,
        "export_image_s": img[0]["done"] if img else None,
        "inline_prep_s": img[0]["sub"].get("preparing layers for inline cache") if img else None,
        "prep_export_s": sum(x["sub"].get("preparing build cache for export", 0) for x in exp) if exp else None,
        "import_manifest_s": sum(x["done"] or 0 for x in imp) if imp else None,
        "pulled_from_cache_registry_bytes": registry_bytes(os.path.join(d, "registry.log")),
        "errors": [x["name"] for x in vs if x["error"]],
        # a CACHED step still reports DONE time when its layers must be pulled and unpacked for a later step
        "cached_pull_s": max([x["done"] or 0 for x in steps if x["cached"]] or [0]),
    }
    if "cpu0" in r and "cpu1" in r:
        a = [int(x) for x in r["cpu0"].split()[1:9]]
        b = [int(x) for x in r["cpu1"].split()[1:9]]
        dd = [y - x for x, y in zip(a, b)]
        info["steal_pct"] = round(100 * dd[7] / sum(dd), 1) if sum(dd) else None
    if "cache_bytes" in r:
        info["cache_bytes"] = int(r["cache_bytes"])
        info["cache_blobs"] = int(r["cache_blobs"])
    for f in ("cache-size.json", "image-size.json"):
        p = os.path.join(d, f)
        if os.path.exists(p):
            try:
                j = json.load(open(p))
                info[f.split(".")[0].replace("-", "_") + "_bytes"] = j.get("bytes")
                info[f.split(".")[0].replace("-", "_") + "_layers"] = j.get("layers")
            except Exception:
                pass
    return info

def med(xs):
    xs = [x for x in xs if x is not None]
    return round(statistics.median(xs), 2) if xs else None

out = {}
for scen in ("cold", "src", "dep"):
    for cfg in ("none", "inline", "regmin", "regmax", "local", "grow", "growr"):
        d0 = os.path.join(root, scen, cfg)
        if not os.path.isdir(d0):
            continue
        runs = {rn: run(os.path.join(d0, rn)) for rn in sorted(os.listdir(d0))}
        runs = {k: v for k, v in runs.items() if v}
        meas = [v for k, v in runs.items() if k.startswith("r")]
        keys = ["build_s", "bootstrap_s", "cached_steps", "apk_builder_s", "npm_install_s", "apk_final_s", "copy_modules_s", "export_cache_s", "export_image_s",
                "inline_prep_s", "prep_export_s", "import_manifest_s", "cached_pull_s", "pulled_from_cache_registry_bytes", "steal_pct", "cache_bytes", "cache_size_bytes", "image_size_bytes"]
        cell = {"n": len(meas), "rcs": [x["rc"] for x in meas]}
        for k in keys:
            cell[k] = med([x.get(k) for x in meas])
            vals = [x.get(k) for x in meas if x.get(k) is not None]
            if vals and k in ("build_s", "export_cache_s", "import_manifest_s"):
                cell[k + "_range"] = [min(vals), max(vals)]
        cell["runs"] = runs
        out[f"{scen}/{cfg}"] = cell
json.dump(out, open(os.path.join(root, "summary.json"), "w"), indent=1)
MB = 1e6
print(f"{'cell':14} {'n':>2} {'build':>7} {'range':>14} {'cached':>6} {'apkB':>6} {'npm':>6} {'expC':>6} {'expI':>6} {'impM':>5} {'pullMB':>7} {'cacheMB':>8} {'imgMB':>6}")
for k, c in out.items():
    cb = c.get("cache_size_bytes") or c.get("cache_bytes")
    print(f"{k:14} {c['n']:>2} {str(c['build_s']):>7} {str(c.get('build_s_range')):>14} {str(c['cached_steps']):>6} {str(c['apk_builder_s']):>6} {str(c['npm_install_s']):>6} {str(c['export_cache_s']):>6} {str(c['export_image_s']):>6} {str(c['import_manifest_s']):>5} {round((c['pulled_from_cache_registry_bytes'] or 0)/MB,1):>7} {round(cb/MB,1) if cb else '-':>8} {round(c['image_size_bytes']/MB,1) if c['image_size_bytes'] else '-':>6} rcs={c['rcs']}")

Summary

A fresh CI runner has no BuildKit cache, so the backend you pick decides what gets reused. For this multi-stage Node service, the cheapest backends did the most: inline and registry mode=min cut a one-line rebuild from 79.0 s to about 20 s and cost almost nothing to export, because BuildKit could skip the whole builder stage by matching cache keys alone. mode=max stored five times as much, added about 30 s of compression to every cold build, and did not make the dependency bump faster: pulling the cached toolchain took as long as installing it, and the export came on top. Before you switch on mode=max because a guide says so, read --progress=plain for your own build and compare the pull time of each cached step with the time it takes to run.

The number that most tools show you was the wrong one for the other four. cgroup v2's memory.peak said the GitHub CLI's Go build peaked at 1,775 MiB, but only 424 MiB of that was anonymous memory, the heaps and stacks of its processes. Most of the rest was page cache, which the kernel drops when it needs the room, plus some kernel memory. A Docker image build looked like 2,072 MiB for the same reason. And a TypeScript type check that held 1,139 MiB with the whole machine available finished inside 2 GB with 970 MiB, because Node sizes its heap limit from the cgroup limit it finds.

This post shows the numbers, the five traps behind them, and every script we used, so you can size a runner from your own builds instead of from a guess.

TL;DR

• Four of five builds fit in 2 GB. Only the Excalidraw Vite build needed more: 3,248 MiB of anonymous memory, killed at 2 GB, fine at 4 GB.
• memory.peak counts page cache. The Go build and the Docker build showed 1,775 and 2,072 MiB, but their anonymous memory peaked at 424 and 355 MiB. Size from anonymous memory plus a margin, then confirm by running the build inside that limit.
• Runtimes size their heaps from the cgroup limit. Inside 2 GB, Node's default heap limit was 1,120 MiB and the JVM's default maximum heap was 512 MiB. The type check used less memory at 2 GB than at 8 GB and still finished, about 24% slower.
• The Gradle daemon keeps its memory after the build. Five seconds after gradlew exited, the daemon still held 1,048 MiB of anonymous memory. With --no-daemon, that dropped to under 1 MiB, and in our one-command runs the daemon gave no repeatable speed-up.
• Limiting the docker CLI did not limit the build. The CLI's cgroup peaked at 40 MiB while the BuildKit container that did the work peaked at 2,072 MiB.
• Wall times on a shared-CPU droplet are noisy: the CPUs reported 23.5% steal time across the runs we tracked. Trust the memory columns more than the time columns.

Prerequisites

• A Linux machine with cgroup v2 and systemd (any recent Ubuntu, Debian or Fedora). Check with stat -fc %T /sys/fs/cgroup, which prints cgroup2fs.
• Root access, for systemd-run --scope and for dropping the page cache.
• The toolchains for the builds you want to test. Our setup script installs Go, Node, Yarn, a JDK, Rust and Docker.
• If you want to rerun our matrix as it is: a 4 vCPU, 8 GB machine and about two hours. We used a droplet on DigitalOcean (referral link).

How we measured

We used one DigitalOcean droplet, size s-4vcpu-8gb in fra1, with Ubuntu 24.04 and kernel 6.8. The kernel reports 7,941 MiB of memory and there is no swap. Every build ran in its own transient systemd scope:

systemd-run --scope --slice=bench.slice --unit=p1-<build>-<limit>-<run> \
  -p MemoryMax=2G -p MemorySwapMax=0 -p AllowedCPUs=0-3 -p OOMPolicy=continue \
  /root/bench/wrap.sh <outdir> <timeout> <build command>

These are cgroup limits on one droplet, not three real machine sizes. A real 2 GB runner has a smaller kernel footprint, a runner agent and often a Docker daemon in the same memory, so it has less room than our 2 GB scope. The 8 GB limit is above the 7.9 GiB that the droplet has, so the 8 GB column means "the whole machine".

The rest of the setup:

• CPU. Every scope was pinned to the same 4 vCPUs with AllowedCPUs=0-3, and nproc printed 4 inside each one. Compilers start about one job per CPU, so a 2 vCPU runner would use less memory and an 8 vCPU runner more. We did not measure other CPU counts.
• Cold build, warm dependencies. We downloaded dependencies once before the runs: yarn install, go mod download, cargo fetch, the Gradle distribution and dependencies, and the base images for the Docker build. Before every run we deleted the build output and the compiler caches (excalidraw-app/build, GOCACHE, target/, Gradle's build/ and .gradle/), stopped any Gradle daemon, and dropped the page cache. That is a CI job that restores a dependency cache but not a build cache.
• What we recorded. Wall time, the exit code, memory.peak at the end of the run, memory.events (how often the limit was hit, and OOM kills), and a sampler that read memory.stat every 100 ms for anonymous memory (anon) and page cache (file).
• Repeats. Runs went in round-robin order: run 1 of every cell, then run 2, then run 3, so a slow period on the shared host does not land on one build. Builds ran slower than we planned, so to stay inside our droplet budget the third round covered only the 8 GB and 2 GB limits. That gives 3 runs per cell at 8 GB and 2 GB, and 2 at 4 GB. The tables show the median of 3, or the mean of 2 at 4 GB. Each build also had one warm-up run at 8 GB that we did not count; it downloaded Gradle and filled the image mirror.

The builds, all pinned to exact commits:

Node was 24.22.2 with Yarn 1.22.22, Docker was 29.9.2 with Buildx 0.38.1.

The results

Times are medians of 3 runs at 8 GB and 2 GB, and the mean of 2 runs at 4 GB. Memory columns are from the 8 GB runs.

What the cgroup counter says vs what the build held (8 GB limit, median of 3)

memory.peak includes page cache; anonymous memory is the sampled peak of anon in memory.stat (100 ms). One s-4vcpu-8gb droplet, 4 vCPUs, 2 November 2026.

Three things stand out. Only one build reached the 2 GB limit at all. The max counter in memory.events, which counts how often a cgroup's usage was about to go over its limit, stayed at 0 for every other run. Memory use at 4 GB and at 8 GB was almost the same for every build, so the extra 4 GB bought nothing. And the gap between memory.peak and anonymous memory is large for exactly the builds that read and write many files: the Go compiler and BuildKit.

The single Rust build, ripgrep 15.2.0 with cargo build --release, fits the same pattern: 896 MiB memory.peak, 572 MiB anonymous, 61.1 s. We ran it once at 8 GB only, because the full matrix did not fit our time budget.

Trap 1: memory.peak counts page cache

memory.peak is the highest value of memory.current that the cgroup reached. memory.current charges the cgroup for everything it caused to be in memory, and that includes page cache: the source files the compiler read, the object files it wrote, the layers BuildKit unpacked. Page cache is reclaimable. When the cgroup gets close to its limit, the kernel drops clean cache pages and writes back dirty ones before it ever considers killing a process.

Anonymous memory is the part that cannot go anywhere without swap: heaps, stacks, the compiler's working data. memory.stat reports it as anon. That, plus a smaller amount of kernel memory (37 to 136 MiB in these builds), is what has to fit. Our sampler read it every 100 ms, so it can miss a shorter spike. memory.peak and the max counter cannot miss one, and they show that no build except Vite at 2 GB ever reached its limit. The safe test is still the one we ran: the build inside the limit you plan to buy.

The Go build of the GitHub CLI shows the difference well. Its peak was 1,775 MiB. Anonymous memory never went over 424 MiB; about 1,400 MiB was page cache from the module sources and the build cache that go build writes. At the 2 GB limit, the same build finished with the same peak, because 1,775 MiB is still under 2,048 MiB. With a smaller limit the kernel may reclaim that cache instead of failing the build, but we did not test below 2 GB, so test the lower limit before you rely on it.

So if a dashboard shows that your CI job "peaked at 1.8 GB", that does not mean a 2 GB runner is too small. Look at anon in memory.stat (or RSS in your container metrics), and then add a margin for the runner itself.

Trap 2: a production build that is also a type check

The Excalidraw build needed far more memory than anything else: 3,248 MiB of anonymous memory, against 1,139 MiB for a full type check of the same code. The reason is in its Vite config:

checker({
  typescript: true,
  eslint:
    envVars.VITE_APP_ENABLE_ESLINT === "false"
      ? undefined
      : { lintCommand: 'eslint "./**/*.{js,ts,tsx}"' },
  // ...
}),

vite-plugin-checker runs the TypeScript checker during vite build too: in version 0.7.2, which Excalidraw pins, enableBuild defaults to true. The production .env turns ESLint off but leaves the type check on, and the build also writes source maps. So one vite build is Rollup, source maps and a full tsc run at the same time.

At 2 GB the kernel killed it in all 3 runs, 69 to 122 seconds in, after the log said ✓ 1442 modules transformed. The kernel log names the cgroup and the process it chose:

oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=p1-vite-2G-r2.scope,mems_allowed=0,oom_memcg=/bench.slice/p1-vite-2G-r2.scope,task_memcg=/bench.slice/p1-vite-2G-r2.scope,task=MainThread,pid=77242,uid=0
Memory cgroup out of memory: Killed process 77242 (MainThread) total-vm:11451308kB, anon-rss:1138988kB, file-rss:4864kB, shmem-rss:0kB, UID:0 pgtables:29992kB oom_score_adj:0

Our wrapper's record of that run shows the kill. memory.events counted 39,825 times that the cgroup was about to go over its limit, and one OOM kill:

vite build, 2 GB limit, run 2

$ cat /root/res/p1/vite/2G/r2/result.env
rc=1
t0=1793637034.083018067
t1=1793637156.285208072
peak=2147483648
current_after=867889152
anon_after=801075200
file_after=38674432
$ cat /root/res/p1/vite/2G/r2/memory.events
low 0
high 0
max 39825
oom 1
oom_kill 1
oom_group_kill 0

The process list right after the kill shows the second half of the build. The checker runs tsc as its own child process, and it was still running after Vite had died:

  77265  1808 /bin/sh -c tsc --noEmit
  77266 799108 node /root/src/excalidraw/node_modules/.bin/tsc --noEmit

That tsc held 799,108 KiB (780 MiB) of resident memory under the 2 GB limit. Yarn reported Command failed with exit code 1, but the type check kept going until our harness stopped the scope. On a real runner it would keep using memory and CPU until the job ends.

We did not split the 3,248 MiB peak at 8 GB between Rollup and the checker, so we cannot say exactly how much the type check adds. What we can say: if your bundler runs a checker, your build step needs the memory of both, and moving the type check to its own CI job is the first thing to try before you buy a bigger runner.

Trap 3: runtimes size their heaps from the limit

Node and the JVM both read the cgroup limit when they start and size their default heap from it. Our harness asks each runtime what it would use inside each limit:

probe inside a 2 GB scope

# what Node 24 and JDK 21 pick inside MemoryMax=2G (from p1.sh probe)
$ cat /root/res/p1/probe-2G.txt
nproc=4
memory.max=2147483648
node_heap_size_limit=1174405120
      int ActiveProcessorCount                     = -1                                        {product} {default}
   size_t MaxHeapSize                              = 536870912                                 {product} {ergonomic}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}

All four places side by side:

Two consequences showed up in the runs.

Heap use follows the limit. tsc --extendedDiagnostics prints Memory used, which is Node's heapUsed at the end of the run, in units of 1,000 bytes. With the whole machine, the type check of 1,508 files reported Memory used: 1018378K (median of 3, about 971 MiB). Inside 2 GB it reported 886903K (about 846 MiB), anonymous memory fell from 1,139 to 970 MiB, and the check finished about 24% slower (median 72.7 s against 58.8 s). The 2 GB run was 9.5 to 14.2 seconds slower in each of the three rounds. In rounds 2 and 3, where we logged the CPU counters, its steal time was only 4 to 7 points higher, so steal explains only part of the gap. More frequent garbage collection under the smaller heap limit would explain the rest, but we did not record GC activity, so that is a hypothesis. A big heap number at a high limit says little about what the program needs. We expected this check to fail at 2 GB with a heap error, and it did not.

Node stopped at 2,240 MiB even with more memory. On our 8 GB machine, Node's default heap limit was 2,240 MiB with a 4 GB scope, an 8 GB scope and no scope at all. We did not test a bigger machine, so check what your runner gives you with node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)'. If a build reports JavaScript heap out of memory, that is Node's own limit, not the kernel's. Set NODE_OPTIONS=--max-old-space-size=<MiB> explicitly instead of only buying more memory.

The JVM takes 25% of the limit by default (MaxRAMPercentage=25), so a Maven build or a test JVM without -Xmx gets 512 MiB inside 2 GB. Gradle sets its own daemon heap, which brings us to the next trap.

Trap 4: the Gradle daemon outlives the build

PlantUML's gradle.properties sets org.gradle.jvmargs=-Xmx1g. That cap kept the build inside 2 GB: anonymous memory peaked at about 1,143 MiB with the daemon and 1,144 MiB without it, and neither came near the limit.

The difference is what happens after gradlew exits. With the daemon, which is the default, the JVM stays alive and keeps its heap. Five seconds after the build, the scope still held 1,378 MiB, 1,048 MiB of it anonymous, in one process:

37292 1105976 /usr/lib/jvm/java-21-openjdk-amd64/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED ...

With --no-daemon, Gradle printed To honour the JVM settings for this build a single-use Daemon process will be forked and Daemon will be stopped at the end of the build, and five seconds after exit the anonymous memory in the scope was under 1 MiB.

Anonymous memory around the end of the Gradle build (8 GB limit, run 1)

Max of 100 ms samples in each 5 s bin, labelled by the end of the bin; 0 s is the moment gradlew exits, +5 s is the 5 s after. One run each, PlantUML jar task, JDK 21, Gradle 9.3.1.

On a CI runner, that leftover daemon sits in the same memory as every later step of the job: the Docker build, the integration tests, the upload. At 8 GB the median time was 123.5 s with the daemon and 87.5 s without (112.7 s and 99.7 s at 2 GB), The ranges overlap and CPU steal was high, so we see no repeatable speed-up from the daemon for one Gradle command per job. The daemon is built to pay off across several Gradle commands in the same job, which we did not test. If a job runs one Gradle command, use --no-daemon (or org.gradle.daemon=false). If it runs several, consider keeping the daemon (we did not test that case), and run ./gradlew --stop before the steps that need the memory.

Trap 5: a limit on the docker CLI does not limit the build

docker build is a client. In our setup the work ran in a BuildKit container (Buildx's docker-container driver), which has its own cgroup. Other drivers and CI executors lay out cgroups differently, but the CLI is only a client in all of them. We wrapped the docker buildx build command in the same kind of scope as everything else, and at every limit that scope peaked at 39 to 42 MiB: the CLI, streaming logs.

To measure the build, we put the limit on the BuildKit container itself:

docker buildx create --name p1d8gr1 --driver docker-container \
  --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \
  --driver-opt memory=8g --driver-opt memory-swap=8g --driver-opt cpuset-cpus=0-3 \
  --buildkitd-config buildkitd.toml --bootstrap

and read memory.peak from /sys/fs/cgroup/system.slice/docker-<container id>.scope. That container peaked at 2,072 MiB, of which only 355 MiB was anonymous: the Go compiler inside the build. Most of the rest was page cache from the image layers and the build's output, plus 136 MiB of kernel memory.

This matters for runner sizing in two ways. Your CI's memory graph may not show the build at all, depending on whether it measures the job's process tree or the whole machine. And if a self-hosted runner limits only the job's process tree, a Docker build in that job can run outside the limit, so a "2 GB job" can push the host far past 2 GB. Check which cgroup your executor puts the Docker daemon in.

Two method bugs that cost us a run

The first 2 GB Vite run left no result at all. The kernel killed the Vite process, and then systemd stopped the whole scope, including our wrapper script, before it could write anything. The journal said so:

p1-vite-2G-r1.scope: A process of this unit has been killed by the OOM killer.
p1-vite-2G-r1.scope: Failed with result 'oom-kill'.

systemd 255 sets DefaultOOMPolicy=stop, and scopes follow it. We set DefaultOOMPolicy=continue before the next 2 GB run (the published p1.sh also passes -p OOMPolicy=continue), and checked with a 64 MiB scope that the wrapper now survives a kill and records oom_kill 1. Runs 2 and 3 at 2 GB have full records.

Then our rerun of run 1 failed too, for a different reason. systemd keeps a failed transient unit loaded, so starting a new scope with the same name was refused:

Failed to start transient scope unit: Unit p1-vite-2G-r1.scope was already loaded or has a fragment file.

By the time we saw that, the Docker cache tests for the next post were running on the same droplet, so we did not try again. Run 1 still counts as a kill: its kernel log line reads oom_memcg=/bench.slice/p1-vite-2G-r1.scope,...,task=MainThread,pid=36041, and our sampler, which runs outside the scope, saw the cgroup reach 2,048 MiB, with up to 1,985 MiB anonymous, before its last sample at 70.5 seconds. The published redo mode now runs systemctl reset-failed first.

If you measure builds this way, set OOMPolicy=continue and give each run a unique unit name, or the runs that you care about most, the ones that fail, will be the ones with no data.

A sizing table

From these runs, for a 4 vCPU runner. The "smallest limit that passed" column is measured. The last column is our rule of thumb, not a measurement: a real VM also holds the OS, the runner agent and any service containers, so we pick the smallest size that leaves at least 1 GB above the anonymous peak.

Hosted CI services list their machines by vCPU and memory, so you can match a build to a size. Bitrise, for example, lists every Linux machine type on its pricing page with its vCPU count and memory, such as 2 vCPU with 8 GB, and also sells a remote build cache for Gradle, Bazel and Xcode (checked on its public pages on 2 November 2026). Whatever you use, measure anonymous memory first, then pick the size.

What we could not conclude

• Three limits on one machine are not three machines. A real 2 GB runner has less page cache to spare, its own kernel, and a runner agent in the same memory. Leave a margin above the anonymous peak; our sizing table uses 1 GB.
• CPU count changes memory. Every run had 4 vCPUs. The Go compiler, Rollup's workers and Gradle's workers scale with CPUs, so these numbers are not valid for 2 or 16 vCPUs.
• Wall times are noisy. The droplet reported 23.5% CPU steal time across the 33 runs that our CPU counter log covered, and between 3.7% and 35.5% in single runs. The Go build took between 121.3 and 199.6 seconds at limits it never came close to. We do not read anything into a time difference between limits unless it repeats in every round, as the tsc gap did.
• Two or three runs per cell is enough to see a failure repeat and a peak hold steady, and that is all. The anonymous peaks varied by up to 21% for the Go build (353 to 441 MiB at 8 GB) and up to 15% for Gradle, and 6% or less for Vite, tsc and Docker between runs of the same cell.
• We did not split shared builds. We cannot say how much of the Vite build's memory is Rollup and how much is the checker, or how much of the Docker build's page cache is layer extraction.
• Rust ran once. One ripgrep build at 8 GB, so no repeats and no 2 GB or 4 GB data.

Run it yourself

Create a 4 vCPU, 8 GB Ubuntu 24.04 machine, copy the files below to /root/bench/, and run:

bash /root/bench/setup.sh
bash /root/bench/p1.sh warmup   # probes the runtimes, one unmeasured run per build
bash /root/bench/p1.sh matrix   # 3 rounds x 3 limits x each build (we stopped it after round 2)
bash /root/bench/p1.sh round3   # what we ran as round 3: 8 GB and 2 GB only
bash /root/bench/p1.sh redo     # reruns any cell that left no result
python3 summarize_p1.py /root/res/p1

Each run writes its own directory with result.env, memory.events, samples.json and the build log, so you can read any single run on its own.

#!/bin/bash
# Droplet setup: toolchains, pinned sources, dependency prefetch, local registries.
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive
mkdir -p /root/src /root/res
cloud-init status --wait || true   # first boot runs apt; wait for its lock

fetch_sha() { git init -q "$1"; git -C "$1" remote add origin "$2"; git -C "$1" fetch -q --depth 1 origin "$3"; git -C "$1" checkout -q FETCH_HEAD; }
fetch_sha /root/src/excalidraw https://github.com/excalidraw/excalidraw 8bcdca4526867433f99fdfcb55a063668db0c9dc
fetch_sha /root/src/plantuml https://github.com/plantuml/plantuml db06d70e18e71839dd5e330ac4086f3845371e3b
fetch_sha /root/src/cli https://github.com/cli/cli 9490d204a2ba95434153dedce99898ce9f281426
git clone -q --depth 1 --branch 15.2.0 https://github.com/BurntSushi/ripgrep /root/src/ripgrep
fetch_sha /root/src/msd https://github.com/GoogleCloudPlatform/microservices-demo c7cb6d5bfd6ea3aed768634ebbcb98cb7a523712

curl -fsSL https://go.dev/dl/go1.27.3.linux-amd64.tar.gz | tar -C /usr/local -xz
curl -fsSL https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.xz | tar -C /usr/local --strip-components=1 -xJ
curl -fsSL https://sh.rustup.rs | sh -s -- -y -q --profile minimal --default-toolchain stable
apt-get update -q
apt-get install -y -q openjdk-21-jdk-headless build-essential git jq zstd python3 ca-certificates curl
curl -fsSL https://get.docker.com | sh
export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local
npm install -g yarn@1.22.22

docker pull -q moby/buildkit:v0.34.1
docker pull -q registry:2.8.3
docker run -d --name reg-cache -p 127.0.0.1:5000:5000 registry:2.8.3
docker run -d --name mirror-hub -p 127.0.0.1:5001:5000 -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2.8.3
docker run -d --name mirror-gcr -p 127.0.0.1:5002:5000 -e REGISTRY_PROXY_REMOTEURL=https://gcr.io registry:2.8.3

# dependencies are downloaded once, outside the measured runs
(cd /root/src/excalidraw && yarn install --frozen-lockfile --network-timeout 600000)
(cd /root/src/cli && go mod download)
(cd /root/src/ripgrep && cargo fetch --locked)

# systemd stops a scope after an OOM kill by default; keep the wrapper alive to record the result
mkdir -p /etc/systemd/system.conf.d
printf '[Manager]\nDefaultOOMPolicy=continue\n' > /etc/systemd/system.conf.d/oom.conf
systemctl daemon-reexec

#!/bin/bash
# Runs inside the systemd scope. usage: wrap.sh OUTDIR TIMEOUT_S cmd...
out=$1; tmo=$2; shift 2
cg=/sys/fs/cgroup$(sed -n 's/^0:://p' /proc/self/cgroup)
echo "$cg" > "$out/cgroup"
cat "$cg/memory.max" > "$out/memory.max"
t0=$(date +%s.%N)
timeout -k 15 "$tmo" "$@" > "$out/build.log" 2>&1
rc=$?
t1=$(date +%s.%N)
{
  echo "rc=$rc"
  echo "t0=$t0"
  echo "t1=$t1"
  echo "peak=$(cat "$cg/memory.peak")"
  echo "current_after=$(cat "$cg/memory.current")"
  awk '$1=="anon"||$1=="file"{print $1"_after="$2}' "$cg/memory.stat"
} > "$out/result.env"
cp "$cg/memory.events" "$out/memory.events"
for p in $(cat "$cg/cgroup.procs"); do
  [ "$p" = "$$" ] && continue
  ps -o pid=,rss=,args= -p "$p" | cut -c1-400
done > "$out/leftover.txt" 2>/dev/null
if [ -n "${LEFTOVER_WAIT:-}" ]; then
  sleep "$LEFTOVER_WAIT"
  echo "current_later=$(cat "$cg/memory.current")" >> "$out/result.env"
  awk '$1=="anon"{print "anon_later="$2}' "$cg/memory.stat" >> "$out/result.env"
  for p in $(cat "$cg/cgroup.procs"); do
    [ "$p" = "$$" ] && continue
    ps -o pid=,rss=,args= -p "$p" | cut -c1-400
  done > "$out/leftover_later.txt" 2>/dev/null
fi
touch "$out/done"
exit $rc

#!/usr/bin/env python3
"""Poll one cgroup v2 directory every 100 ms until a done file appears.

usage: sample.py CGROUP_DIR OUT_JSON DONE_FILE
Records the maximum of anon, file, kernel and memory.current, plus a series.
"""
import json, os, sys, time

cg, out, done = sys.argv[1:4]
start = time.time()
while not os.path.isdir(cg):
    if time.time() - start > 60 or os.path.exists(done):
        json.dump({"error": "cgroup never appeared"}, open(out, "w"))
        sys.exit(0)
    time.sleep(0.02)

keys = ("anon", "file", "kernel", "shmem")
mx = {k: 0 for k in keys}
mx["current"] = 0
series = []
t0 = time.time()
n = 0
while not os.path.exists(done):
    try:
        with open(os.path.join(cg, "memory.stat")) as f:
            st = {}
            for line in f:
                k, v = line.split()
                if k in keys:
                    st[k] = int(v)
        with open(os.path.join(cg, "memory.current")) as f:
            cur = int(f.read())
    except OSError:
        break
    n += 1
    for k in keys:
        mx[k] = max(mx[k], st.get(k, 0))
    mx["current"] = max(mx["current"], cur)
    series.append([round(time.time() - t0, 2), st.get("anon", 0), st.get("file", 0), cur])
    time.sleep(0.1)

json.dump({"max": mx, "samples": n, "interval_s": 0.1, "series": series}, open(out, "w"))

#!/bin/bash
# Build definitions for post 1. prep_* runs outside the cgroup, cmd_* inside it.
export PATH=/usr/local/go/bin:/root/.cargo/bin:/usr/local/bin:$PATH GOTOOLCHAIN=local
EX=/root/src/excalidraw PU=/root/src/plantuml GH=/root/src/cli RG=/root/src/ripgrep
FRONTEND=/root/src/msd/src/frontend

stop_gradle() { pkill -f GradleDaemon || true; pkill -f GradleWrapperMain || true; pkill -f KotlinCompileDaemon || true; sleep 1; }

prep_vite() { rm -rf $EX/excalidraw-app/build $EX/node_modules/.vite $EX/excalidraw-app/node_modules/.vite; }
cmd_vite() { cd $EX && yarn build:app:docker; }

prep_tsc() { find $EX -maxdepth 2 -name '*.tsbuildinfo' -delete; }
cmd_tsc() { cd $EX && node_modules/.bin/tsc -p tsconfig.json --extendedDiagnostics; }

prep_gradle_daemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; }
cmd_gradle_daemon() { cd $PU && ./gradlew jar --console=plain --no-build-cache; }

prep_gradle_nodaemon() { stop_gradle; rm -rf $PU/build $PU/.gradle; }
cmd_gradle_nodaemon() { cd $PU && ./gradlew --no-daemon jar --console=plain --no-build-cache; }

prep_go() { rm -rf /root/gocache /root/out; mkdir -p /root/out; }
cmd_go() { cd $GH && GOCACHE=/root/gocache go build -o /root/out/gh ./cmd/gh; }

prep_rust() { rm -rf $RG/target; }
cmd_rust() { cd $RG && cargo build --release --locked; }

prep_docker() { docker buildx ls --format '{{.Name}}' 2>/dev/null | grep '^p1d' | xargs -r -n1 docker buildx rm >/dev/null 2>&1 || true; }

if [ "${BASH_SOURCE[0]}" = "$0" ]; then "$@"; fi

#!/bin/bash
# Post 1 matrix. usage: p1.sh warmup | p1.sh matrix
set -u
B=/root/bench R=/root/res/p1
mkdir -p $R
source $B/p1-cmds.sh
LIMITS=(8G 4G 2G)
BUILDS=(vite tsc gradle_daemon gradle_nodaemon go rust docker)
# rust ran once (warmup) only: the matrix did not fit the droplet time budget
MATRIX_BUILDS=(vite tsc gradle_daemon gradle_nodaemon go docker)
CPUS=0-3
log() { echo "$(date -u +%FT%TZ) $*" | tee -a $R/progress.log; }

scope_run() { # unit limit out tmo cmd...
  local unit=$1 lim=$2 out=$3 tmo=$4; shift 4
  python3 $B/sample.py /sys/fs/cgroup/bench.slice/$unit.scope $out/samples.json $out/done &
  local sp=$!
  systemd-run --scope --quiet --slice=bench.slice --unit=$unit \
    -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS -p OOMPolicy=continue \
    $B/wrap.sh $out $tmo "$@"
  wait $sp
  systemctl stop $unit.scope 2>/dev/null || true
}

docker_cell() { # lim out tmo tag
  local lim=$1 out=$2 tmo=$3 tag=$4
  local name=$(echo p1d$tag | tr A-Z a-z) m=$(echo $1 | tr G g)
  docker buildx create --name $name --driver docker-container \
    --driver-opt network=host --driver-opt image=moby/buildkit:v0.34.1 \
    --driver-opt memory=$m --driver-opt memory-swap=$m --driver-opt cpuset-cpus=$CPUS \
    --buildkitd-config $B/buildkitd.toml --bootstrap > $out/create.log 2>&1
  local cid; cid=$(docker inspect -f '{{.Id}}' buildx_buildkit_${name}0)
  local cg=/sys/fs/cgroup/system.slice/docker-$cid.scope
  cat $cg/memory.max > $out/container_memory.max
  mkdir -p $out/cli
  python3 $B/sample.py $cg $out/container_samples.json $out/cli/done &
  local sp=$!
  scope_run p1-docker-$tag $lim $out/cli $tmo \
    docker buildx build --builder $name --progress=plain --provenance=false --output type=cacheonly $FRONTEND
  wait $sp
  { echo "container_peak=$(cat $cg/memory.peak)"; cat $out/cli/result.env; } > $out/result.env
  cp $cg/memory.events $out/container_memory.events
  cp $out/cli/build.log $out/build.log
  docker buildx rm $name > /dev/null 2>&1
}

run_cell() { # build limit run tmo
  local b=$1 lim=$2 r=$3 tmo=$4
  local tag=$b-$lim-$r out=$R/$b/$lim/$r
  rm -rf $out; mkdir -p $out
  prep_$b
  sync; echo 3 > /proc/sys/vm/drop_caches; sleep 2
  if [ $b = docker ]; then
    docker_cell $lim $out $tmo ${lim}$r
  else
    case $b in gradle_*) export LEFTOVER_WAIT=5 ;; esac
    scope_run p1-$tag $lim $out $tmo bash $B/p1-cmds.sh cmd_$b
    unset LEFTOVER_WAIT
  fi
  [ $b = gradle_daemon ] || [ $b = gradle_nodaemon ] && stop_gradle
  log "$tag $(tr '\n' ' ' < $out/result.env)"
}

probe() { # record what the runtimes see inside each limit
  for lim in "${LIMITS[@]}"; do
    systemd-run --scope --quiet --slice=bench.slice --unit=probe-$lim -p MemoryMax=$lim -p MemorySwapMax=0 -p AllowedCPUs=$CPUS \
      bash -c 'echo "nproc=$(nproc)"; echo "memory.max=$(cat /sys/fs/cgroup$(sed -n "s/^0:://p" /proc/self/cgroup)/memory.max)";
        echo "node_heap_size_limit=$(node -e "console.log(require(\"v8\").getHeapStatistics().heap_size_limit)")";
        java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage|ActiveProcessorCount) "' > $R/probe-$lim.txt 2>&1
  done
  { echo "nproc=$(nproc)"; free -b | head -2; echo "node_heap_size_limit=$(node -e 'console.log(require("v8").getHeapStatistics().heap_size_limit)')";
    java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E " (MaxHeapSize|MaxRAMPercentage) "; } > $R/probe-host.txt 2>&1
}

case ${1:-} in
  warmup)
    probe
    for b in "${BUILDS[@]}"; do run_cell $b 8G r0 900; done
    log WARMUP-DONE ;;
  matrix)
    declare -A TMO
    for b in "${MATRIX_BUILDS[@]}"; do
      . <(grep -E '^t[01]=' $R/$b/8G/r0/result.env)
      w=$(python3 -c "print(int(float('$t1')-float('$t0')))")
      t=$((3 * w + 60)); [ $t -gt 600 ] && t=600; [ $t -lt 120 ] && t=120
      TMO[$b]=$t
    done
    log "timeouts: $(for b in "${MATRIX_BUILDS[@]}"; do printf '%s=%s ' $b ${TMO[$b]}; done)"
    for r in r1 r2 r3; do
      for b in "${MATRIX_BUILDS[@]}"; do
        for lim in "${LIMITS[@]}"; do run_cell $b $lim $r ${TMO[$b]}; done
      done
    done
    log MATRIX-DONE ;;
  round3) # third run at 8G and 2G only: the full round did not fit the droplet time budget
    declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351)
    log "round 3 at 8G and 2G only"
    for b in "${MATRIX_BUILDS[@]}"; do
      for lim in 8G 2G; do run_cell $b $lim r3 ${TMO[$b]}; done
    done
    log MATRIX-DONE ;;
  redo) # rerun cells that left no result (systemd stopped the scope after an OOM kill)
    declare -A TMO=([vite]=300 [tsc]=381 [gradle_daemon]=600 [gradle_nodaemon]=351 [go]=414 [docker]=351)
    for b in "${MATRIX_BUILDS[@]}"; do for lim in "${LIMITS[@]}"; do for r in r1 r2 r3; do
      d=$R/$b/$lim/$r
      if [ -d $d ] && ! grep -q '^rc=' $d/result.env 2>/dev/null; then
        rm -rf $d.lost; mv $d $d.lost; log "redo $b $lim $r"
        systemctl reset-failed p1-$b-$lim-$r.scope 2>/dev/null  # added after our run: a failed scope blocks its name
        run_cell $b $lim $r ${TMO[$b]}
      fi
    done; done; done
    log REDO-DONE ;;
  *) echo "usage: $0 warmup|matrix|round3|redo"; exit 2 ;;
esac

[registry."docker.io"]
  mirrors = ["127.0.0.1:5001"]
[registry."gcr.io"]
  mirrors = ["127.0.0.1:5002"]
[registry."127.0.0.1:5000"]
  http = true
  insecure = true
[registry."127.0.0.1:5001"]
  http = true
  insecure = true
[registry."127.0.0.1:5002"]
  http = true
  insecure = true

#!/usr/bin/env python3
"""Summarize post 1 runs: medians per build and limit over the measured runs (r1..r3)."""
import json, os, re, statistics, sys

root = sys.argv[1]
BUILDS = ["vite", "tsc", "gradle_daemon", "gradle_nodaemon", "go", "rust", "docker"]
LIMITS = ["8G", "4G", "2G"]
MiB = 2 ** 20

def env(path):
    d = {}
    if os.path.exists(path):
        for line in open(path):
            if "=" in line:
                k, v = line.strip().split("=", 1)
                d[k] = v
    return d

def events(path):
    d = {}
    if os.path.exists(path):
        for line in open(path):
            k, v = line.split()
            d[k] = int(v)
    return d

def run_info(d, build):
    r = env(os.path.join(d, "result.env"))
    if not r or "rc" not in r:
        return None
    log = open(os.path.join(d, "build.log"), errors="replace").read() if os.path.exists(os.path.join(d, "build.log")) else ""
    if build == "docker":
        if not r.get("container_peak"):
            return None  # warm-up run 0 recorded no container numbers (builder name case bug)
        ev = events(os.path.join(d, "container_memory.events"))
        smp = json.load(open(os.path.join(d, "container_samples.json")))
        peak = int(r["container_peak"])
        cli_peak = int(r["peak"])
    else:
        ev = events(os.path.join(d, "memory.events"))
        smp = json.load(open(os.path.join(d, "samples.json")))
        peak = int(r["peak"])
        cli_peak = None
    rc = int(r["rc"])
    wall = float(r["t1"]) - float(r["t0"])
    if rc == 0:
        outcome = "ok"
    elif ev.get("oom_kill", 0) > 0:
        outcome = "oom_kill"
    elif "heap out of memory" in log or "Reached heap limit" in log:
        outcome = "node_heap"
    elif "OutOfMemoryError" in log:
        outcome = "java_heap"
    elif rc in (124, 137):
        outcome = "timeout"
    else:
        outcome = f"fail_rc{rc}"
    mx = smp.get("max", {})
    return {
        "rc": rc, "outcome": outcome, "wall_s": round(wall, 1),
        "peak_mib": round(peak / MiB), "anon_max_mib": round(mx.get("anon", 0) / MiB),
        "file_max_mib": round(mx.get("file", 0) / MiB), "kernel_max_mib": round(mx.get("kernel", 0) / MiB),
        "samples": smp.get("samples"),
        "oom_kill": ev.get("oom_kill", 0), "max_events": ev.get("max", 0), "high_events": ev.get("high", 0),
        "after_mib": round(int(r.get("current_after", 0)) / MiB), "anon_after_mib": round(int(r.get("anon_after", 0)) / MiB),
        "cli_peak_mib": round(cli_peak / MiB) if cli_peak is not None else None,
        "later_mib": round(int(r["current_later"]) / MiB) if "current_later" in r else None,
        "anon_later_mib": round(int(r["anon_later"]) / MiB) if "anon_later" in r else None,
        "leftover_later": open(os.path.join(d, "leftover_later.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover_later.txt")) else None,
        "leftover": open(os.path.join(d, "leftover.txt")).read().strip() if os.path.exists(os.path.join(d, "leftover.txt")) else "",
    }

def med(xs):
    xs = [x for x in xs if x is not None]
    return round(statistics.median(xs), 1) if xs else None

out = {}
for b in BUILDS:
    for lim in LIMITS:
        runs = {}
        for rn in ("r0", "r1", "r2", "r3"):
            d = os.path.join(root, b, lim, rn)
            if os.path.isdir(d):
                info = run_info(d, b)
                if info:
                    runs[rn] = info
        meas = [runs[k] for k in ("r1", "r2", "r3") if k in runs]
        ok = [x for x in meas if x["outcome"] == "ok"]
        out[f"{b}/{lim}"] = {
            "runs": runs, "n": len(meas), "n_ok": len(ok),
            "outcomes": [x["outcome"] for x in meas],
            "wall_med_ok": med([x["wall_s"] for x in ok]),
            "wall_med_all": med([x["wall_s"] for x in meas]),
            "peak_med": med([x["peak_mib"] for x in meas]),
            "anon_med": med([x["anon_max_mib"] for x in meas]),
            "file_med": med([x["file_max_mib"] for x in meas]),
            "peak_range": [min(x["peak_mib"] for x in meas), max(x["peak_mib"] for x in meas)] if meas else None,
            "anon_range": [min(x["anon_max_mib"] for x in meas), max(x["anon_max_mib"] for x in meas)] if meas else None,
            "wall_range": [min(x["wall_s"] for x in meas), max(x["wall_s"] for x in meas)] if meas else None,
            "after_med": med([x["after_mib"] for x in meas]),
            "anon_after_med": med([x["anon_after_mib"] for x in meas]),
            "cli_peak_med": med([x["cli_peak_mib"] for x in meas]),
            "later_med": med([x["later_mib"] for x in meas]),
            "anon_later_med": med([x["anon_later_mib"] for x in meas]),
            "max_events_med": med([x["max_events"] for x in meas]),
        }
json.dump(out, open(os.path.join(root, "summary.json"), "w"), indent=1)
print(f"{'cell':26} {'n':>2} {'ok':>2} {'wall_ok':>8} {'peak':>6} {'anon':>6} {'file':>6} {'after':>6} outcomes / ranges")
for k, v in out.items():
    print(f"{k:26} {v['n']:>2} {v['n_ok']:>2} {str(v['wall_med_ok']):>8} {str(v['peak_med']):>6} {str(v['anon_med']):>6} {str(v['file_med']):>6} {str(v['after_med']):>6} {v['outcomes']} wall{v['wall_range']} anon{v['anon_range']} peak{v['peak_range']} maxev={v['max_events_med']} cli={v['cli_peak_med']} later={v['later_med']}/{v['anon_later_med']}")

Summary

Most CI builds need less memory than the cgroup counter suggests, because memory.peak includes page cache that the kernel gives back under pressure. Measure anonymous memory, run the build inside the limit you want to buy, and check memory.events for max and oom_kill. Then look for the traps: a bundler that runs a type checker, a runtime that sizes its heap from the limit, a Gradle daemon that stays after the build, and a Docker build that runs outside your cgroup. In our runs, fixing any of those was worth more than doubling the runner's memory.

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 From Working Group to SIG Architecture: spotlight on AI Conformance

In this SIG Architecture spotlight we talked with Janet Kuo (Google) and Yuan Tang (TensorScale AI) about the journey of AI Conformance in Kubernetes, from the initial Working Group to the integration

📅 Oct 29, 2026 • 📰 Kubernetes Blog

🔗 Read more

📄 Announcing the Workload Conformance Working Group

We are excited to announce that the Kubernetes community now has a Workload Conformance Working Group. The working group is writing a shared standard for how workloads behave at runtime on Kubernetes.

📅 Oct 27, 2026 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Keycloak's observability journey: a guide for project maintainers

As part of an ongoing series, the Developer Experience SIG interviews organizations about their real-world OpenTelemetry deployments to share practical lessons with the broader community. This post fe

📅 Nov 2, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 The Ecosystem is the Answer: Docker, the Blueprint Alliance, and the Importance of Collaboration in the Agentic Era

Docker is a founding member of the Blueprint Alliance, a 12-company reference architecture for securing AI agents, built around four governance questions.

📅 Nov 2, 2026 • 📰 Docker Blog

🔗 Read more

📄 Where should your code live in a Docker Sandbox?

Docker Sandboxes support three ways to organize an agent’s code: shared mounts, Git worktrees, and clone mode. Here’s how to choose between them.

📅 Nov 2, 2026 • 📰 Docker Blog

🔗 Read more

📄 The Linkerd Story: The people Behind The Service Mesh — A new documentary premieres at KubeCon + CloudNativeCon North America 2026

Join CNCF and Buoyant on November 9 in Salt Lake City for the premiere of a new documentary celebrating a decade of Linkerd, the evolution of service mesh technology, and the people who helped make it

📅 Nov 2, 2026 • 📰 CNCF Blog

🔗 Read more

📄 KubeCon + CloudNativeCon NA 2026: End User TAB recommendations and tips

KubeCon + CloudNativeCon NA 2026 in Salt Lake City will bring the cloud native community together with projects, vendors and end users in what is one of the largest open source events. In this post, w

📅 Oct 30, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Coding Agent Horror Stories: The CVE That Did Not Exist

An AI code review agent invented a CVE and recommended dangerous fixes. Learn how Docker Sandboxes contain agents and why their output needs verification.

📅 Oct 29, 2026 • 📰 Docker Blog

🔗 Read more

📄 Volcano v1.16 Released: NamespaceQueue, fairshare, and a broader HyperNode ecosystem

Volcano v1.16.0 is available . v1.16.0 introduces NamespaceQueue and the fairshare plugin. Tenants can use hierarchical queues within their own namespaces to organize quotas and queue hierarchies with

📅 Oct 28, 2026 • 📰 CNCF Blog

🔗 Read more

📄 Win the Agentic Era, Together: Introducing the Docker Navigator Partner Program

Docker is launching the Navigator Partner Program, with three tiers, co-selling benefits, and a new incentive for every registered AI governance deal.

📅 Oct 28, 2026 • 📰 Docker Blog

🔗 Read more

📄 Hardening Tetragon’s gRPC API with Tetragon

Tetragon versions before v1.7 expose an unauthenticated gRPC admin API on , and any pod running with on the same node can reach it. This…

📅 Oct 28, 2026 • 📰 Cilium Blog

🔗 Read more

🔄 CI/CD

📄 10 questions to ask any software vendor about sovereignty

Sovereignty is moving from buzzword to buying criterion. The European Commission’s proposed Cloud and AI Development Act (CADA) would set assurance levels for the cloud and AI services public authorit

📅 Nov 2, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Unified API Inventory: Accelerate Patching with Cross-Source

Cross-source API correlation connects runtime findings to their code, documentation, and fixes for improved defense against machine-speed attacks. | Blog

📅 Nov 2, 2026 • 📰 Harness Blog

🔗 Read more

📄 Game Off 2026: The theme is SHIFT

GitHub’s annual month-long game jam is back for its 14th year! This November, make a game inspired by the theme SHIFT. Change gears, swap perspectives, bend reality, or just press a certain key. The p

📅 Nov 1, 2026 • 📰 GitHub Blog

🔗 Read more

📄 What is release automation? From deploy pipelines to automated releases

Release automation moves code from build to production without manual steps. Learn how it works, where AI fits, and how to automate the release step.

📅 Oct 29, 2026 • 📰 LaunchDarkly Blog

🔗 Read more

📄 GitLab Orbit is GA: Higher agent accuracy, lower token cost

An agent’s true cost isn’t only the tokens it burns. It’s the review that calls your code “safe to merge” after seeing the diff, but without any context on the pipeline. We call that artificial confid

📅 Oct 29, 2026 • 📰 GitLab Blog

🔗 Read more

📄 GitLab Orbit: Find vulnerability blast radius without the manual hunt

A critical vulnerability needs a fix and a map. The fix is usually fast. Finding where the vulnerable component lives, who owns it, and whether it's exposed to the Internet is what costs you time. Mos

📅 Oct 29, 2026 • 📰 GitLab Blog

🔗 Read more

📄 GitHub Universe 2026: Evolving every layer of the stack

Today, we laid out our vision for GitHub as the most comprehensive, end-to-end software delivery system in the world. The post GitHub Universe 2026: Evolving every layer of the stack appeared first on

📅 Oct 28, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Octoverse: Commits, pushes, reach new highs in 2026 as developers build with agents

See how a record year of growth reached every industry, every field, and every kind of builder in the 2026 Octoverse report. The post Octoverse: Commits, pushes, reach new highs in 2026 as developers

📅 Oct 28, 2026 • 📰 GitHub Blog

🔗 Read more

📄 Measuring developer experience on an Amazon EKS developer platform

Most platform teams can't prove their internal developer platform is improving software delivery. This post shows how to automatically collect the four DORA metrics (deployment frequency, lead time fo

📅 Oct 26, 2026 • 📰 AWS Containers Blog

🔗 Read more

🏗️ IaC

📄 Amazon Connect Customer automatically saves in-progress emails as drafts

Amazon Connect Customer automatically saves in-progress emails as drafts, helping agents resume work without losing progress when connectivity is interrupted or when their browser is accidentally clos

📅 Oct 30, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Friday Five — October 30, 2026 | Red Hat

Fierce Network - Telcos can cash in on AI, not just pay for it: Red HatRed Hat's Joe Fernandes discusses how open models can help telcos move from renting AI capacity to owning it and selling inferenc

📅 Oct 30, 2026 • 📰 Red Hat Blog

🔗 Read more

📄 Tracing failures across Terraform state boundaries with AWS DevOps Agent

When a change in one Terraform stack silently breaks an application in another, the team seeing the symptoms often doesn't own the stack that caused it. This post shows how AWS DevOps Agent traces the

📅 Oct 26, 2026 • 📰 AWS DevOps Blog

🔗 Read more

📊 Observability

📄 Instrumentation Hub in Grafana Cloud: start easier, stay in control

It's no secret that getting started with observability can be much harder than it should be. That's why, at ObservabilityCon 2025, we introduced Instrumentation Hub: a guided, scalable way to roll out

📅 Nov 2, 2026 • 📰 Grafana Blog

🔗 Read more

📄 E-Maj 5.0.1 : Major Security Update.

We are very glad to announce the E-Maj 5.0.1 version. E-Maj is a PostgreSQL extension which enables fine-grained write logging and time travel on subsets of the database. This version mainly fixes a v

📅 Oct 31, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 What’s new with Google Cloud

Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not

📅 Oct 30, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 OpenTelemetry eBPF Instrumentation reaches v1.0.0

We’re excited to announce v1.0.0 of OpenTelemetry eBPF Instrumentation—OBI! OBI brings OpenTelemetry observability to running Linux applications without requiring changes to their source code. It uses

📅 Oct 30, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Announcing the 2026 Governance Committee Election Results

We are delighted to announce the results of the 2026 OpenTelemetry Governance Committee Election! The OpenTelemetry project continues to bring together an exceptionally vibrant community, who are shap

📅 Oct 29, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Only include resource detectors you need by building your own Collector

There are many reasons for building your own OpenTelemetry Collector distribution. The existing distributions are designed to be broadly useful, which means they include many more components than what

📅 Oct 29, 2026 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Grafana 13.3 release: manage variables across dashboards, share findings with Notebooks, a UI refresh, and more

Grafana 13.3 is here, bringing more consistency and control to how you work across Grafana. Cross-dashboard variables help you manage shared dashboard controls from one place, while Notebooks let you

📅 Oct 29, 2026 • 📰 Grafana Blog

🔗 Read more

📄 Turo + New Relic: Turn AI code volume into production reliability

See how Turo uses the New Relic MCP server to give AI agents live production context, slash incident triage to 60 seconds, and scale releases safely.

📅 Oct 29, 2026 • 📰 New Relic Blog

🔗 Read more

📄 Measuring soft navigation web vitals

The Sentry JavaScript SDK v11 reports LCP, CLS, and INP for soft navigations. Here's what that reveals about your SPA, with interactive demos.

📅 Oct 29, 2026 • 📰 Sentry Blog

🔗 Read more

📄 fluent-package v6.1.0 has been released

Hi users! We have released fluent-package v6.1.0 on 2026-10-29. Fluent Package is a stable distribution package of Fluentd. (successor of td-agent) This is a new release of the v6.1 series. Note that

📅 Oct 29, 2026 • 📰 Fluentd Blog

🔗 Read more

📄 DigitalOcean Signals: Observability for AI Agents and Inference

You ship an agent. It runs in production, sessions start and finish, and the dashboards stay green. But then a user mentions that they spent ten minutes retrying the same failing command with slightly

📅 Oct 28, 2026 • 📰 DigitalOcean Blog

🔗 Read more

📄 Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1

When evaluating AI models for observability, generating a plausible query or explanation isn't enough. Teams need to know whether an agent's conclusions are supported by telemetry, whether its changes

📅 Oct 28, 2026 • 📰 Grafana Blog

🔗 Read more

🔐 Security

📄 New Bug-Fix Releases for MPS Versions 2026.2 RC2, 2026.1.2, 2025.3.3, 2025.2.5, and 2025.1.5

We’ve released updates for multiple MPS versions that fix a critical security issue. MPS 2026.1.2 and 2026.2 RC2 additionally bring a substantial improvement to the Projectional Agent Toolkit plugin.

📅 Nov 2, 2026 • 📰 JetBrains Blog

🔗 Read more

📄 Gitea 28.3.0 is released

We are excited to announce the release of Gitea 28.3.0, the third maintenance release for the 28 series. It contains a large number of security fixes, mostly around token scopes, and bug fixes and

📅 Nov 1, 2026 • 📰 Gitea Blog

🔗 Read more

📄 Open-source security in the AI exploit era: Patch without changing a thing

Here’s how a vulnerability in a popular open-source library plays out in 2026: A maintainer merges a fix and publishes The post Open-source security in the AI exploit era: Patch without changing a thi

📅 Nov 1, 2026 • 📰 The New Stack

🔗 Read more

📄 pgAdmin 4 v9.19 Released

The pgAdmin Development Team is pleased to announce pgAdmin 4 version 9.19. This release of pgAdmin 4 includes 136 bug fixes and new features, including fixes for 15 security vulnerabilities (CVE-2026

📅 Oct 31, 2026 • 📰 PostgreSQL News

🔗 Read more

📄 Daybreak Red from OpenAI now available to eligible customers on Amazon Bedrock in AWS GovCloud (US-East)

Security teams need powerful tools to find vulnerabilities, test exploits, and build defenses. Daybreak Red from OpenAI is now available to eligible customers on Amazon Bedrock in AWS GovCloud (US-Eas

📅 Oct 30, 2026 • 📰 CloudFormation Updates

🔗 Read more

📄 Why embargoes matter for open source security

Open source is built on a foundation of transparency, collaboration, and speed. We share code, we share ideas, and together, we build better software. But when it comes to discovering and patching sec

📅 Oct 30, 2026 • 📰 Canonical Blog

🔗 Read more

📄 Blog: Falco 0.45.2: security updates and a look back at 0.45.1

Hey Falco community, Falco 0.45.2 is available! This patch brings certificate trust updates and newer security dependencies, alongside the reliability fixes delivered in 0.45.1. Since we did not publi

📅 Oct 30, 2026 • 📰 Falco Blog

🔗 Read more

📄 PHP2Shell Won't Be the Last. Here's Where the Next One Will Show Up First.

PHP2Shell showed how quickly an engine-level flaw can put internet-facing applications at risk. Learn why outside-in inventory, rapid testing, and repeat checks matter before the next disclosure.

📅 Oct 28, 2026 • 📰 Snyk Blog

🔗 Read more

📄 What happens when vulnerabilities are discovered faster than you can fix them?

We're entering a new era of application security. Frontier AI models are uncovering complex flaws that traditional tools overlooked, and attackers are using the same models to exploit them faster. Rem

📅 Oct 28, 2026 • 📰 GitLab Blog

🔗 Read more

📄 Announcing SOC 1, FedRAMP 20x Class A, PCI, and HIPAA

Explore Harness' latest compliance milestones: SOC 1, FedRAMP 20x Class A, PCI DSS, and HIPAA assessments across its platform and runtime security. | Blog

📅 Oct 28, 2026 • 📰 Harness Blog

🔗 Read more

📄 Snyk UNFILTERED: Inside Our First-Ever Customer Summit

At Snyk’s first customer summit, security leaders shared candid perspectives on AI risk, vulnerability remediation, and governing autonomous agents. Five themes reveal how teams are adapting security

📅 Oct 28, 2026 • 📰 Snyk Blog

🔗 Read more

📄 Azure DevOps Server RC (26H2) Is Now Available

Today we’re releasing the release candidate (RC) of the next Azure DevOps Server release. It brings new features previously available in our hosted version, along with security and quality improvement

📅 Oct 27, 2026 • 📰 Azure DevOps Blog

🔗 Read more

💾 Databases

📄 PLEASE_READ_ME: The Opportunistic Ransomware Devastating MySQL Servers

Guardicore Labs uncovers a Ransomware detection campaign targeting MySQL servers. Attackers use Double Extortion and publish data to pressure victims.

📅 Nov 2, 2026 • 📰 Linode Blog

🔗 Read more

📄 Beyond Migration: Using Database Migration Service for disk shrinking

When database engineers hear “database migration", they typically think of a one-time onboarding pathway—moving databases from on-premises environments to the cloud, or between different cloud provide

📅 Nov 2, 2026 • 📰 Google Cloud Blog

🔗 Read more

📄 How to Build a LlamaIndex Vector Store With TiDB

A LlamaIndex vector store is where your index keeps node embeddings, usually alongside the text chunks and metadata, so queries can find similar content. The default store lives in memory, so a restar

📅 Oct 30, 2026 • 📰 TiDB Blog

🔗 Read more

📄 Why Real-Time AI at Scale is So Hard

**📅 Oct 29, 2026** • **📰 ScyllaDB Blog**

[**🔗 Read more**](https://www.scylladb.com/2026/10/29/why-real-time-ai-at-scale-is-so-hard/)

### 📄 Connecting coding agents to the right documentation with a navigable context layer

Ask a question about Redis and the job seems simple: find the right documentation and explain it. But documentation questions come in many shapes. “What does XACK do?” is a precise command question. “

**📅 Oct 29, 2026** • **📰 Redis Blog**

[**🔗 Read more**](https://redis.io/blog/navigable-context-layer-for-coding-agents/)

### 📄 PostgreSQL@SCaLE24x CfP closing soon on Nov 1, 2027

Hi folks! We're hosting a PostgreSQL track at SCaLE again — come join us in sunny Pasadena! PostgreSQL@SCaLE is a two-day event on Thursday–Friday, April 1–2, 2027, at the Pasadena Convention Center, 

**📅 Oct 28, 2026** • **📰 PostgreSQL News**

[**🔗 Read more**](https://www.postgresql.org/about/news/postgresqlscale24x-cfp-closing-soon-on-nov-1-2027-3363/)

### 📄 Estuary Expands PostgreSQL CDC with Targeted Backfills and Scalable Discovery

Estuary has released several updates to its PostgreSQL change data capture (CDC) connector focused on backfill recovery, discovery at scale, and schema-change detection. Targeted Backfills PostgreSQL 

**📅 Oct 28, 2026** • **📰 PostgreSQL News**

[**🔗 Read more**](https://www.postgresql.org/about/news/estuary-expands-postgresql-cdc-with-targeted-backfills-and-scalable-discovery-3419/)

### 📄 P99 CONF 2026 Recap: A Tail of Two Days

Our Aussie host’s take on P99 CONF around the clock and (now) anytime on demand

**📅 Oct 27, 2026** • **📰 ScyllaDB Blog**

[**🔗 Read more**](https://www.scylladb.com/2026/10/27/p99-conf-2026-recap/)

### 📄 Introducing Semantic Operator: Governed Data Access for AI Agents

Semantic Operator is a new Kubeflow subproject that gives AI agents trusted answers from business data. It works in three steps: Define metrics. Data teams describe certified metrics once in an Apache

**📅 Oct 27, 2026** • **📰 Kubeflow Blog**

[**🔗 Read more**](https://blog.kubeflow.org/introducing-semantic-operator/)

### 📄 Three layers, all config: how DigitalOcean Managed Agents enforce NVIDIA OpenShell-aligned boundaries

A few weeks ago, we handed an AI agent one of the scariest jobs in infrastructure: moving huffl’s production backend from Supabase to DigitalOcean. huffl.ai is a real product. Underneath it sits 31 ta

**📅 Oct 26, 2026** • **📰 DigitalOcean Blog**

[**🔗 Read more**](https://www.digitalocean.com/blog/managed-agents-enforce-nvidia-openshell)

---

## 🌐 Platforms

### 📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

**📅 Nov 2, 2026** • **📰 Linode Blog**

[**🔗 Read more**](https://www.akamai.com/blog/security/the-oracle-of-delphi-steal-your-credentials)

### 📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

**📅 Nov 2, 2026** • **📰 Linode Blog**

[**🔗 Read more**](https://www.akamai.com/blog/security/the-nansh0u-campaign-hackers-arsenal-grows-stronger)

### 📄 The Rise of LLMJacking: How Threat Actors Monetize Compromised Cloud and Model Resources

Written by: Christopher Romano, Codi Starks, Garrett Mahoney, John Wolfram, Josh Murchie, K Singh, Mathew Potaczek, Mike Lombardi, Rik Dolfing Introduction Mandiant responded to multiple incidents in 

**📅 Nov 2, 2026** • **📰 Google Cloud Blog**

[**🔗 Read more**](https://cloud.google.com/blog/topics/threat-intelligence/llmjacking-monetizing-compromised-cloud-models/)

### 📄 Shutting the Door on the XCPE Botnet

Introduction Last week, Google and industry partners dismantled the XCPE backdoor which enrolled more than 1.3 million set-top boxes into a botnet. We assess with high confidence that the backdoor and

**📅 Nov 2, 2026** • **📰 Google Cloud Blog**

[**🔗 Read more**](https://cloud.google.com/blog/topics/threat-intelligence/shutting-the-door-on-the-xcpe-botnet/)

### 📄 How we halved DNS build delay and cut memory use by 85%

Zone setting changes used to rebuild entire DNS zones, even ones with tens of millions of records. Here is how we built a planner that picks the smallest safe build, cutting full builds by 40%, peak m

**📅 Nov 2, 2026** • **📰 Cloudflare Blog**

[**🔗 Read more**](https://blog.cloudflare.com/dns-targeted-zone-builds/)

### 📄 Amazon Bedrock expands API support and introduces Cross-Region Inferencing for OpenAI models in AWS GovCloud (US)

Amazon Bedrock now supports the OpenAI GPT-5.6 models (Terra and Luna) on the bedrock-runtime endpoint, with support for the Responses, Converse, and Chat Completions APIs in AWS GovCloud (US). It als

**📅 Oct 30, 2026** • **📰 CloudFormation Updates**

[**🔗 Read more**](https://aws.amazon.com/about-aws/whats-new/2026/10/bedrock-openai-runtime-govcloud/)

### 📄 Web Search now available on Kiro in AWS GovCloud (US)

Kiro web search gives the agent real-time access to current information on the internet, so its answers draw on up-to-date search results and page content. The agent uses web search automatically when

**📅 Oct 30, 2026** • **📰 CloudFormation Updates**

[**🔗 Read more**](https://aws.amazon.com/about-aws/whats-new/2026/10/kiro-web-search-aws-govcloud-us/)

### 📄 Frontier AI–Accelerated Attacks: Securing the Agentic Web and MCP Surface

**📅 Oct 30, 2026** • **📰 Linode Blog**

[**🔗 Read more**](https://www.akamai.com/blog/ai/2026/oct/frontier-ai-accelerated-attacks-agentic-web-mcp-surface)

### 📄 4 real? Tracking exploitation and patching of eMudhra’s weak keys

Four, just four: that’s the key to break into any unpatched connection on the Internet that is using eMudhra certificates. Here’s how we quickly patched Cloudflare, what evidence we can see of attempt

**📅 Oct 30, 2026** • **📰 Cloudflare Blog**

[**🔗 Read more**](https://blog.cloudflare.com/emudhra-patching/)

### 📄 Cloudflare services for private origins is now generally available

Public traffic to private origins, with WAF, bot management, caching, and Workers, is now available to every customer with Cloudflare One connectivity.

**📅 Oct 30, 2026** • **📰 Cloudflare Blog**

[**🔗 Read more**](https://blog.cloudflare.com/private-origins-public-to-private/)

### 📄 Direct VM networking in AWS: Introducing BGP cloud connector on Red Hat OpenShift Service on AWS

When migrating virtual machines (VMs) to the cloud, virtualization engineers hit a familiar wall: how do you route traffic directly to a Red Hat OpenShift Virtualization guest VM using that VM’s own I

**📅 Oct 30, 2026** • **📰 OpenShift Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/direct-vm-networking-aws-introducing-bgp-cloud-connector-red-hat-openshift-service-aws)

### 📄 The boundary problem in agentic AI is a platform problem

An autonomous agent will do exactly what its tools let it do, and no more. This sounds like a constraint. In practice it is the whole risk, because the set of things an agent’s tools let...

**📅 Oct 29, 2026** • **📰 CNCF Blog**

[**🔗 Read more**](https://www.cncf.io/blog/2026/10/29/the-boundary-problem-in-agentic-ai-is-a-platform-problem/)

---

## 📰 Misc

### 📄 Visual Studio Code 1.145 (Insiders)

Learn what is new in Visual Studio Code 1.145 (Insiders). Read the full article

**📅 Nov 4, 2026** • **📰 VS Code Blog**

[**🔗 Read more**](https://code.visualstudio.com/updates/v1_145)

### 📄 Governing Coding-Agent Sprawl with MLflow AI Gateway

A demonstration of how to use MLflow AI Gateway to manage coding agent sprawl.

**📅 Nov 3, 2026** • **📰 MLflow Blog**

[**🔗 Read more**](https://mlflow.org/blog/govern-coding-agent-sprawl/)

### 📄 OpenAI’s new shared usage limit pits ChatGPT against Codex

OpenAI has revealed plans to combine usage allowances across ChatGPT and Codex, as part of a broader effort to integrate The post OpenAI’s new shared usage limit pits ChatGPT against Codex appeared fi

**📅 Nov 2, 2026** • **📰 The New Stack**

[**🔗 Read more**](https://thenewstack.io/openai-chatgpt-codex-shared-limits/)

### 📄 JetBrains Earns 180 G2 Badges in Fall 2026, Recognized for Enterprise Leadership, Usability, and Developer Experience

Discover how JetBrains products are recognized for enterprise leadership, developer experience, and ease of adoption, and why this matters for engineering teams looking to improve productivity and sca

**📅 Nov 2, 2026** • **📰 JetBrains Blog**

[**🔗 Read more**](https://blog.jetbrains.com/blog/2026/11/02/jetbrains-180-g2-badges/)

### 📄 Livestream: Scaling Agentic Development With Air Governance

Join us on November 18, from 4:00–5:00 pm CET, for a livestream with Viktor Kiselev and Guillaume Garnier from the JetBrains Air product team. Your developers want to use the best AI agents for the jo

**📅 Nov 2, 2026** • **📰 JetBrains Blog**

[**🔗 Read more**](https://blog.jetbrains.com/ai/2026/11/livestream-scaling-agentic-development-with-air-governance/)

### 📄 What’s new in Red Hat OpenShift migration advisor: Clearer reports and smarter planning

Red Hat OpenShift migration advisor helps you assess your VMware inventory of virtual machines (VMs) and plan migrations to Red Hat OpenShift Virtualization. Over the past few months, we’ve focused on

**📅 Nov 2, 2026** • **📰 Red Hat Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/whats-new-red-hat-openshift-migration-advisor-clearer-reports-and-smarter-planning)

### 📄 Sovereignty in the AI era: Choice, control, and autonomy

For the past few years, AI felt like a high-stakes exhibition game. Teams across every industry were on the pitch experimenting with large language models, running rapid proofs-of-concept, and testing

**📅 Nov 2, 2026** • **📰 Red Hat Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/sovereignty-ai-era-choice-control-and-autonomy)

### 📄 A step-by-step guide for how to kill the line-by-line code review

Agents verify; people decide. Most teams running AI code review today have it the other way around. They point a The post A step-by-step guide for how to kill the line-by-line code review appeared fir

**📅 Nov 1, 2026** • **📰 The New Stack**

[**🔗 Read more**](https://thenewstack.io/ai-code-review-invariants/)

### 📄 The context problem: How Docusign built agents that understand the business behind contracts

Ask a general-purpose language model to summarize a contract, and it will probably do a reasonable job. Ask it to The post The context problem: How Docusign built agents that understand the business b

**📅 Oct 31, 2026** • **📰 The New Stack**

[**🔗 Read more**](https://thenewstack.io/docusign-contract-ai-agents/)

### 📄 IntelliJ IDEA 2026.2.4 Is Out!

IntelliJ IDEA 2026.2.4 is now available! You can update to this version from inside the IDE, using the Toolbox App, or using snaps if you are a Ubuntu user. You can also download it from our website. 

**📅 Oct 30, 2026** • **📰 JetBrains Blog**

[**🔗 Read more**](https://blog.jetbrains.com/idea/2026/10/intellij-idea-2026-2-4/)

### 📄 How parental leave helped 2 new parents find their rhythm

Becoming a parent brings so much excitement, and can have its fair share of challenges. At Red Hat, parental leave is designed so work isn't one of those worries. In 2025, we set an ambitious goal to 

**📅 Oct 30, 2026** • **📰 Red Hat Blog**

[**🔗 Read more**](https://www.redhat.com/en/blog/how-parental-leave-helped-2-new-parents-find-their-rhythm)

### 📄 Visual Studio Code 1.144

Learn what's new in Visual Studio Code 1.144 Read the full article

**📅 Oct 28, 2026** • **📰 VS Code Blog**

[**🔗 Read more**](https://code.visualstudio.com/updates/v1_144)

Then we deleted one record, changed the TTL of two others behind Terraform's back, and planned again. A normal plan found all three. With -minimal-refresh, Terraform made no API requests and reported no changes.

That trade is the whole story of this flag. It is a real speed-up, and it is also a switch that turns off drift detection for most of your state. This post shows both, with the numbers, so you can decide where it belongs in your pipeline.

TL;DR

• On 200 DNS records plus a zone and a CNAME (202 resources), a full refresh made 202 GET requests. -minimal-refresh made 0 when nothing changed in config and 1 when one record changed. Median plan time fell from 9.5 s to 4.3 s, about the same as -refresh=false (4.5 s).
• A second full plan within a minute of the first took 60 s instead of 9.5 s, because two full plans went over DigitalOcean's limit of 250 API requests per minute. The provider's request log showed only 200s; the 429s appeared only in the HTTP client's retry messages (retrying in 51s).
• After we deleted one record and changed the TTL of two others out of band, a full plan found all three and -minimal-refresh found none. With a config change on r[0], it refreshed only r[0] and still hid the drift on the CNAME that depends on it.
• A destroy plan with -minimal-refresh skips the refresh of managed resources, so it plans to delete resources that are already gone. The DigitalOcean provider treats the 404 on delete as already deleted, so our destroy still succeeded; a provider that returns an error there would fail.
• Use it for fast merge request plans on large states, refresh fully before you apply, and keep a scheduled terraform plan -refresh-only (or a normal plan) so drift still has somewhere to show up.

Prerequisites

• Terraform 1.17.0 or later (terraform version)
• A DigitalOcean API token with domain read, create and delete scopes, if you want to run the lab
• A zone name under a domain you control; the lab never delegates it, so no real DNS changes

What the flag does

A refresh reads each resource in the state from the provider's API and records what it finds. What Does Terraform Refresh Really Do? covers the basics. -minimal-refresh changes which resources get that read.

The plan command docs describe it in one line: it "reduces the number of remote API requests by refreshing only resource instances that Terraform plans to change". The implementation is in PR #38953, which spells out the details:

• For each resource instance, Terraform first plans against the state it already has, without reading the remote object.
• If that plan has no change, the instance is not refreshed at all.
• If it has a change, Terraform throws that plan away, refreshes the instance, and plans it again.
• Each instance decides on its own. A refresh of one resource does not automatically refresh the resources that depend on it.
• Destroy plans, and deposed or orphaned instances, skip the refresh with this flag.

1. resource in state
2. plan against state no API read

Outcomes:

• no change: skip the refresh
• change: refresh, plan again

The docs also say what that costs: the option "causes Terraform to ignore external changes to resources that have no planned changes". The rest of this post measures both sides of that sentence.

The lab

The companion repository creates 202 resources on DigitalOcean: one DNS zone, 200 TXT records and one CNAME. DigitalOcean DNS costs nothing and needs no droplets, and a record refresh is one GET request, so counting requests is simple.

resource "digitalocean_record" "r" {
  count  = var.records
  domain = digitalocean_domain.lab.id
  type   = "TXT"
  name   = format("r%03d", count.index)
  value  = count.index == var.changed ? "v2" : "v1" # -var changed=7 changes one record in config
  ttl    = 1800
}

# Depends on r[0], so a config change to r[0] touches it in the graph.
resource "digitalocean_record" "pointer" {
  domain = digitalocean_domain.lab.id
  type   = "CNAME"
  name   = "pointer"
  value  = "${digitalocean_record.r[0].fqdn}."
  ttl    = 1800
}

Each plan in the clean runs uses TF_LOG=DEBUG, because the DigitalOcean provider prints its API requests only at that level, and plan.sh counts the logged request lines and reads the planned changes from terraform show -json. The plans ran on a Raspberry Pi 3 B+ with 1 GB of RAM, so the absolute times are slower than on a laptop. The request counts do not depend on the machine. They are the requests the provider's request log records; a retry below that logger (Finding 2) is not counted again. We wrote down what we expected before each measured run and committed it to the repository first. SOURCES.md in the repository keeps the exact excerpts from the docs, the PR and the provider source that this post quotes.

The-DevOps-Daily/terraform-minimal-refresh-check on GitHub

Finding 1: 202 requests down to 0, and half the plan time

With nothing changed in the configuration, a full plan read every resource. -minimal-refresh read none. With one record changed in config, it read exactly that one record:

bench2.sh, round 1, then three full plans back to back

$ cut -f1-5 results/bench2-20261029T114834Z.tsv | grep -E '^(label|quiet/[a-z]+/[a-z-]+/1|burst)'
label	seconds	gets	requests	429s
quiet/full/no-change/1	9.5	202	202	0
quiet/minimal/no-change/1	4.3	0	0	0
quiet/minimal/one-change/1	4.9	1	1	0
quiet/norefresh/no-change/1	4.5	0	0	0
burst/full/no-change/1	9.4	202	202	0
burst/full/no-change/2	60.4	202	202	0
burst/full/no-change/3	60.6	202	202	0

Across three rounds, the median times were:

Median plan time, 202 DigitalOcean resources

Three runs each, every full-refresh plan after a 65 s pause; no 429 was logged. Raspberry Pi 3 B+, Terraform 1.17.0, provider digitalocean 2.106.0. Source: results/bench2-20261029T114834Z.tsv.

Two things stand out. First, -minimal-refresh is as fast as -refresh=false here, because with nothing changed it does the same work. Second, a plan without any refresh still took about 4.3 s on this machine. Refreshing all 202 resources added about 5 s on top, roughly 25 ms of wall-clock time per resource with Terraform's default parallelism of 10. We did not split that into network time and Terraform or provider work, so expect both numbers to be different on your hardware.

Finding 2: a second full plan within a minute took six times longer

Our first benchmark looked wrong. Identical full plans took anything from 9 to 62 seconds. The cause was our own script: it ran every plan twice in a row, once to time it and once with logging, so each measurement meant about 404 requests across two consecutive full plans. DigitalOcean allows 5,000 requests per hour and 250 per minute per token. We marked that run as confounded in the repository and measured the effect on purpose instead: three full plans back to back.

Three full plans back to back

Each plan sends 202 GET requests; DigitalOcean allows 250 per minute. Source: results/bench2-20261029T114834Z.tsv.

The odd part was the log. Every slow plan still showed exactly 202 GET requests and no 429 status line. A rerun of two back-to-back full plans that kept the whole log (burst-log.sh) found the 429s somewhere else:

burst-log.sh, second of two full plans back to back

$ sed -n '/^burst\/full\/2/,/^# ended/p' results/burst-log-20261029T132653Z.txt | grep -E '^burst|status lines|retry:'
burst/full/2	59.7	202	202	0	none	none
  all status lines: {'200': 202}; retry messages: 10; longest silent gap in the log: 50.0 s
  retry: digitalocean.com/v2/domains/tf-refresh-lab.devops-daily.com/records/1836867618 (status: 429): retrying in 51s (4 left)
  retry: digitalocean.com/v2/domains/tf-refresh-lab.devops-daily.com/records/1836867807 (status: 429): retrying in 51s (4 left)
  retry: digitalocean.com/v2/domains/tf-refresh-lab.devops-daily.com/records/1836867744 (status: 429): retrying in 51s (4 left)

In that rerun, every HTTP status line in the request log was a 200, but the HTTP client printed ten retry messages. The three the script kept each show a 429 Too Many Requests and a retry scheduled 51 seconds later. The log was silent for 50 seconds, and the second plan took 59.7 s.

The 429s never show up as status lines because of how the provider builds its client (config.go). It creates an HTTP client with retries (4 by default, 1 to 30 s apart; a 51 s wait is longer than that maximum, which suggests it came from DigitalOcean's retry-after header) and then wraps its request logging around that client. The request log sees one request going in and the final 200 coming out. The 429 exists only in the retry library's own message. If a plan suddenly takes a minute longer, search the TF_LOG output for retrying in or status: 429; the HTTP status lines alone hide the retries.

This is where -minimal-refresh earns its keep. A CI system that runs a plan for every push to every open pull request can easily send several full refreshes a minute against the same token. With the flag, an unchanged resource costs no requests at all.

Finding 3: the drift it stops seeing

drift.sh changes three records through the DigitalOcean API, behind Terraform's back:

• r[42] is deleted
• r[43] gets a TTL of 60 instead of 1800
• pointer, the CNAME that depends on r[0], gets a TTL of 60

Then it plans in several ways. Each full-refresh plan waits a minute first, and no 429 was logged in any of them.

Source: results/drift-20261029T124036Z.tsv. Full-refresh plans had a 65-second pause before them; the -minimal-refresh plans ran right after.

The second row is the one to remember. One record is gone and two have the wrong TTL, and the plan says there is nothing to do. It will keep saying that on every pull request until someone changes those resources in code or runs a plan that refreshes them.

The fourth row shows the rule from the PR in practice: each resource decides on its own whether to refresh. pointer depends on r[0], and r[0] was refreshed because its config changed, but pointer had no planned change of its own, so its TTL drift stayed hidden. If you expected a change to one resource to pull its dependents into the refresh, it does not do that automatically: a dependent is refreshed only when it has a planned change of its own.

-refresh-only caught all three, which is what it is for. The docs say you cannot combine it with -minimal-refresh, so it cannot be made cheap the same way.

Finding 4: destroy plans skip the refresh completely

With -minimal-refresh, a destroy plan skips the refresh of managed resources, as the PR describes. After r[42] was deleted out of band, a normal destroy plan refreshed, noticed it was gone, and planned 201 deletions. The -minimal-refresh destroy plan made no API requests and planned 202, including the record that no longer existed.

We then destroyed the lab with terraform apply -destroy -minimal-refresh. It exited 0 and reported 202 resources destroyed, one of which no longer existed (results/teardown-20261029T124712Z.txt). That worked only because the DigitalOcean provider's record delete treats a 404 as already gone. A provider that returns an error on a missing object would fail, possibly after some deletions have already completed. For destroy, let Terraform refresh.

What went wrong while we measured

Three mistakes, all caught before writing, all kept in the repository:

• The first benchmark ran every plan twice back to back, so each full plan meant about 404 requests in quick succession, more than the 250 a minute DigitalOcean allows. Its request counts are right; its timings mix in rate-limit waits. The file says so on its second line, and bench2.sh has the clean timings.
• Our first drift run asked DigitalOcean to change the CNAME's TTL with a PATCH that sent only the TTL. The API answered 422: Data needs to end with a dot, even though it stores the value without one, so the dependent-resource case did not run. The fixed script sends the value with the trailing dot.
• plan.sh first read only resource_changes from the plan JSON. A refresh-only plan reports what it found under resource_drift, so it looked like it found nothing. The script now records both.

A fourth check failed in a harmless way: bench2.sh tried to measure the cost of debug logging by unsetting TF_LOG, but the log was still written, so both rows were logged runs. So the cost of debug logging is unmeasured. All the clean timings in this post come from logged runs, so they compare fairly with each other.

When to use it

-minimal-refresh fits the fast plan that runs on every merge request, if your state is large or your provider's rate limit is tight. The resources you changed are still refreshed, so their part of the plan starts from the real remote state. Treat it as early feedback, though: resources without a planned change of their own are planned from state, and a refresh of one resource does not automatically refresh the ones that depend on it, as Finding 3 shows. Run a fully refreshed plan before you apply.

It is the wrong tool anywhere you rely on plans to surface drift. Keep at least one plan that refreshes everything:

# Assumes terraform init, the backend and provider credentials are set up
# in before_script or the image, and a pipeline schedule exists for drift-check.

# Fast plan for every merge request: only changed resources are read.
mr-plan:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script:
    - terraform plan -minimal-refresh

# Nightly drift check: reads every resource and fails when anything moved.
drift-check:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
  script:
    - terraform plan -refresh-only -detailed-exitcode # exit code 2 means drift

The docs add two more rules: do not use it, or -refresh=false, when you upgrade provider versions, and you cannot combine it with -refresh-only or -refresh=false. From our results we would add two more: do not use it for destroy plans you want to be accurate, and do not apply a plan made with it without a full refresh first.

What we could not conclude

• One provider, two resource types, all DNS. A zone and its records are cheap to read. Providers with slow reads (databases, Kubernetes objects, large IAM policies) should save more time per skipped refresh; we did not measure them.
• One slow machine. The 4.3 s without a refresh was measured on a Raspberry Pi 3 B+. On other hardware both the baseline and the cost of the refresh will be different.
• Other providers' destroy behaviour. The DigitalOcean provider treats a 404 on delete as already deleted. A provider that returns an error there would fail the destroy instead.
• Large states. We stopped at 202 resources to stay inside DigitalOcean's 5,000 requests an hour, and our repeated runs still took the token down to 4 remaining requests once.

Summary

-minimal-refresh does what it says. On 202 DigitalOcean resources it cut API requests from 202 to 0 or 1 and plan time from 9.5 s to 4.3 s, about the same as -refresh=false, while still refreshing the resources you change. It also stays clear of the slowdown we saw when a second full plan ran within a minute of the first (60 s), when requests hit DigitalOcean's per-minute limit and were retried 51 s later. The price is drift: with no config change it reported none of the three records we changed behind its back, and a change to one resource did not refresh the resource that depends on it. Use it for fast merge request plans, refresh fully before you apply, keep a scheduled full or refresh-only plan for drift, and do not use it for destroy.

This post shows how base64 works, why its output is a third bigger, and what it does not protect. Then it walks through the bugs that keep coming back. Every command and output below comes from a real run.

TLDR

• Base64 maps every 3 bytes to 4 characters from a 64-character alphabet. Anyone can reverse it with one command. It is an encoding, not encryption.
• A Kubernetes Secret's data field is plain base64. kubectl get secret -o yaml plus base64 -d shows the value to anyone with read access.
• The output is 33.3% bigger than the input. GNU base64 also adds a newline every 76 characters, which makes it 35.1%. Gzip wins most of it back on the wire.
• The common bugs: a trailing newline from echo, wrapped output pasted into YAML or an env var, double encoding with --from-literal or stringData, and the URL-safe alphabet without padding in JWTs.

Prerequisites

• A Linux shell with GNU coreutils for base64 and basenc. We used coreutils 9.1 on Debian 12.
• openssl and python3 for the JWT and comparison examples.
• kubectl for the Secret examples. Every kubectl command here uses --dry-run=client, so you do not need a cluster. We used kubectl v1.37.1.

How base64 turns 3 bytes into 4 characters

Base64 uses 64 symbols, and 64 is 2 to the power of 6, so each output character carries exactly 6 bits. Three bytes are 24 bits, and 24 bits split cleanly into four 6-bit groups. That is the whole algorithm: take 3 bytes, cut them into four 6-bit numbers, and look each number up in the alphabet.

This script prints each step for the string k8s:

import base64

data = b"k8s"
bits = "".join(f"{b:08b}" for b in data)
groups = [bits[i:i + 6] for i in range(0, len(bits), 6)]

print("bytes   ", " ".join(f"{b:08b}" for b in data))
print("6 bits  ", " ".join(groups))
print("values  ", " ".join(str(int(g, 2)) for g in groups))
print("base64  ", base64.b64encode(data).decode())

bytes    01101011 00111000 01110011
6 bits   011010 110011 100001 110011
values   26 51 33 51
base64   azhz

The standard alphabet from RFC 4648 is:

 0-25   A B C ... Z
26-51   a b c ... z
52-61   0 1 2 ... 9
62      +
63      /

So 26 is a, 51 is z and 33 is h, and k8s becomes azhz. Decoding runs the same steps backwards. There is no key and nothing to guess, which is why base64 is not a way to hide anything.

Padding: what the = signs mean

When the input length is not a multiple of 3, the last group is short. Base64 fills the missing bits with zeros and adds = so the output length stays a multiple of 4:

printf 'k' | base64
printf 'k8' | base64
printf 'k8s' | base64

aw==
azg=
azhz

One leftover byte gives 2 characters plus ==. Two leftover bytes give 3 characters plus =. Padding carries no data, which is why some formats, JWT among them, drop it. That saves a few bytes and causes a bug we will get to below.

Because the mapping is fixed, common inputs have recognizable prefixes. JSON that starts with {" followed by a letter encodes to a string starting with eyJ, so eyJ... is almost always JSON, often a JWT. Base64-encoded PEM text (-----BEGIN ...) starts with LS0tLS1CRUdJTi.

The 33% overhead, measured

Every 3 input bytes become 4 output characters, so the encoded size is 4 × ceil(n / 3). For large inputs that is 4/3 of the original, an overhead of 33.3%. GNU base64 also wraps its output at 76 characters by default, which adds one newline for every 76 characters.

We encoded 1 MiB of random data to check the math:

head -c 1048576 /dev/urandom > blob.bin
wc -c < blob.bin
base64 blob.bin | wc -c        # default: wrapped at 76 characters
base64 -w0 blob.bin | wc -c    # -w0: no wrapping
base64 blob.bin | wc -l

1048576
1416501
1398104
18397

The formula gives the same numbers: 4 × ceil(1048576 / 3) is 1,398,104 characters, and ceil(1398104 / 76) is 18,397 lines, each ending in a newline, for 1,416,501 bytes in total. That is 1.3333 times the input without wrapping and 1.3509 times with it.

1 MiB of random data, before and after base64

Measured with wc -c on Debian 12, GNU coreutils 9.1 and gzip -9. The base64 overhead is 33.3% (35.1% with line wrapping); after gzip it is 0.9% on the wire.

On the wire, compression gets most of it back. Base64 only uses 6 of every 8 bits, and gzip notices:

gzip -9 -c blob.bin | wc -c
base64 -w0 blob.bin | gzip -9 | wc -c

1048763
1058004

For this sample, gzip cut the overhead from 33.3% to 0.9%. The overhead still counts wherever the encoded size is what matters: memory, environment variables, cookies and request headers. A JWT is sent in a header on every call, and header size limits apply to its encoded form: each claim costs 4/3 of its size, and nginx's default large_client_header_buffers allows 8 KB per header line.

Kubernetes Secrets are base64, not encryption

A Secret's data field holds base64 strings. Create one locally to see what the API server would store:

kubectl create secret generic db-creds \
  --from-literal=username=app \
  --from-literal=password='hunter2-prod' \
  --dry-run=client -o yaml

apiVersion: v1
data:
  password: aHVudGVyMi1wcm9k
  username: YXBw
kind: Secret
metadata:
  name: db-creds

Getting the value back takes one pipe. On a cluster you would use kubectl get secret db-creds -o jsonpath='{.data.password}'; here we read the same field from the dry run:

kubectl create secret generic db-creds --from-literal=password='hunter2-prod' \
  --dry-run=client -o jsonpath='{.data.password}' | base64 -d; echo

hunter2-prod

Base64 is there so binary data such as keystores and TLS keys fits inside JSON and YAML. It does nothing for secrecy.

Warning

Anyone who can get or list Secrets in a namespace can read every value in it, and so can anyone who can create a pod there that mounts them. A Secret manifest committed to Git is the password in plain sight. What protects Secrets is RBAC, encryption at rest for etcd, and keeping the values out of Git with a tool such as Sealed Secrets, SOPS or an external secret store.

For a longer walkthrough, see How to Decode a Kubernetes Secret and the ConfigMaps and Secrets part of our Kubernetes guide.

Bug 1: the newline from echo

echo adds a newline, and base64 encodes it like any other byte:

echo 'hunter2-prod' | base64
printf '%s' 'hunter2-prod' | base64

aHVudGVyMi1wcm9kCg==
aHVudGVyMi1wcm9k

The first value ends in Cg==, which is the encoded \n. Decode it and look at the bytes:

echo 'aHVudGVyMi1wcm9kCg==' | base64 -d | od -c

0000000   h   u   n   t   e   r   2   -   p   r   o   d  \n
0000015

Put that value in a Secret and the password is now hunter2-prod plus a newline. The app sends 13 bytes, the database expects 12, and you get an authentication error for a password that looks right in every log line.

The same thing happens with files. A file written with echo ends in a newline, and --from-file keeps it:

echo 'hunter2-prod' > password.txt
kubectl create secret generic db-creds --from-file=password=password.txt \
  --dry-run=client -o jsonpath='{.data.password}'; echo

printf '%s' 'hunter2-prod' > password.txt
kubectl create secret generic db-creds --from-file=password=password.txt \
  --dry-run=client -o jsonpath='{.data.password}'; echo

aHVudGVyMi1wcm9kCg==
aHVudGVyMi1wcm9k

Use printf '%s' whenever the exact bytes matter, and pipe a suspect value through base64 -d | od -c to see the last byte.

Bug 2: wrapped output in YAML and env vars

GNU base64 breaks its output into 76-character lines. A database URL is long enough to hit that:

DB_URL='postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require'
printf '%s' "$DB_URL" | wc -c
printf '%s' "$DB_URL" | base64

69
cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9vcmRlcnM/c3Ns
bW9kZT1yZXF1aXJl

Now put it into a manifest the way many scripts do, with a heredoc:

cat <<EOF > secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-url
data:
  url: $(printf '%s' "$DB_URL" | base64)
EOF
kubectl create --dry-run=client -f secret.yaml -o yaml

error: error parsing secret.yaml: error converting YAML to JSON: yaml: line 8: could not find expected ':'

The second line of the value landed at column 0, and YAML read it as a new key. In an env var or a CI variable the newline does not fail this loudly: it ends up inside the value. Turn wrapping off with -w0:

cat <<EOF > secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-url
data:
  url: $(printf '%s' "$DB_URL" | base64 -w0)
EOF
sed -n 's/^  url: //p' secret.yaml | base64 -d

postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require

Even simpler: let kubectl create secret generic db-url --from-literal=url="$DB_URL" --dry-run=client -o yaml write the manifest, and the encoding is never your problem.

macOS and openssl

macOS ships a different base64. According to its man page, it does not wrap by default (-b sets a line length, and -w is accepted for GNU compatibility). Older macOS releases decode only with -D, and on them -d is a debug flag. A script that has to run on both should encode with a pipe that works everywhere:

printf '%s' "$DB_URL" | base64 | tr -d '\n'

cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9vcmRlcnM/c3NsbW9kZT1yZXF1aXJl

openssl base64 is another option, but it wraps at 64 characters and has a sharper edge when decoding:

printf '%s' "$DB_URL" | openssl base64
printf '%s' "$DB_URL" | base64 -w0 | openssl base64 -d | wc -c; echo "rc=${PIPESTATUS[2]}"
printf '%s' "$DB_URL" | base64 -w0 | openssl base64 -d -A; echo

cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9v
cmRlcnM/c3NsbW9kZT1yZXF1aXJl
0
rc=0
postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require

Without -A, OpenSSL 3.0 decoded our 92-character single line to zero bytes and exit code 0. No error, just an empty value. If you use openssl base64 for single-line data, pass -A both ways.

Bug 3: double encoding

--from-literal, --from-file and stringData all take plain text and encode it for you. Give them a value that is already encoded and it gets encoded again:

kubectl create secret generic db-creds \
  --from-literal=password="$(printf '%s' 'hunter2-prod' | base64)" \
  --dry-run=client -o jsonpath='{.data.password}'; echo

kubectl create secret generic db-creds \
  --from-literal=password="$(printf '%s' 'hunter2-prod' | base64)" \
  --dry-run=client -o jsonpath='{.data.password}' | base64 -d; echo

YUhWdWRHVnlNaTF3Y205aw==
aHVudGVyMi1wcm9k

The pod now gets aHVudGVyMi1wcm9k as its password. The giveaway is a first decode that looks like base64 instead of the value you stored. Decode it again to confirm:

echo 'YUhWdWRHVnlNaTF3Y205aw==' | base64 -d | base64 -d; echo

hunter2-prod

The rule: data takes base64, and stringData, --from-literal and --from-file take the raw value. In a Helm chart, b64enc belongs in data:, never in stringData:.

Not every double encoding is a bug. PEM is itself base64 with header lines, so a TLS Secret's tls.crt field is encoded twice on purpose:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout tls.key -out tls.crt -days 30 -subj '/CN=orders.internal' 2>/dev/null
kubectl create secret tls orders-tls --cert=tls.crt --key=tls.key \
  --dry-run=client -o jsonpath='{.data.tls\.crt}' | cut -c1-40
kubectl create secret tls orders-tls --cert=tls.crt --key=tls.key \
  --dry-run=client -o jsonpath='{.data.tls\.crt}' | base64 -d | head -2

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J
-----BEGIN CERTIFICATE-----
MIIBijCCAS+gAwIBAgIUazfeROKqRj7HLheO41+kH88CZagwCgYIKoZIzj0EAwIw

That LS0tLS1CRUdJTi prefix is the -----BEGIN from earlier. The certificate-authority-data field in a kubeconfig works the same way.

base64url and JWTs

Two characters of the standard alphabet cause trouble outside of files. + means a space in a form-encoded query string, and / is a path separator. RFC 4648 defines a second alphabet, base64url, that uses - and _ instead:

printf '\xfb\xff\xbf' | base64
printf '\xfb\xff\xbf' | basenc --base64url

+/+/
-_-_

This is not only about odd bytes. The database URL from earlier encodes to ...cnM/c3Ns... in standard base64 and ...cnM_c3Ns... in base64url. In a URL path, that / splits the value in two.

JWTs use base64url and also drop the = padding (RFC 7515). This script, saved as make-jwt.sh, builds a real HS256 token with shell tools:

#!/usr/bin/env bash
set -euo pipefail

b64url() { basenc --base64url -w0 | tr -d '='; }

header=$(printf '%s' '{"alg":"HS256","typ":"JWT"}' | b64url)
payload=$(printf '%s' '{"sub":"deploy-bot","role":"admin","exp":1793318400}' | b64url)
signature=$(printf '%s' "$header.$payload" \
  | openssl dgst -sha256 -hmac 'not-a-real-key' -binary | b64url)

echo "$header.$payload.$signature"

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkZXBsb3ktYm90Iiwicm9sZSI6ImFkbWluIiwiZXhwIjoxNzkzMzE4NDAwfQ.mirpAmeTzGcS59Fo6JImIz4IBPSFSqpDJy1qltzzzuA

Both JSON parts start with eyJ. Now try to read the payload the obvious way:

chmod +x make-jwt.sh
TOKEN=$(./make-jwt.sh)
echo "$TOKEN" | cut -d. -f2 | base64 -d; echo " rc=$?"

{"sub":"deploy-bot","role":"admin","exp":1793318400}base64: invalid input
 rc=1

The payload part is 70 characters, not a multiple of 4, so GNU base64 prints what it could decode and then exits with 1. The JSON looks complete, which makes this easy to miss: under set -e the script stops here, and without it the script carries on with output that may be cut short. basenc --base64url -d fails the same way, because it also expects padding. Translate the alphabet and put the padding back first:

b64url_decode() {
  local s
  s=$(tr '_-' '/+')               # back to the standard alphabet
  case $(( ${#s} % 4 )) in        # put the stripped padding back
    2) s="$s==" ;;
    3) s="$s=" ;;
  esac
  printf '%s' "$s" | base64 -d
}

echo "$TOKEN" | cut -d. -f2 | b64url_decode; echo
echo "$TOKEN" | cut -d. -f1 | b64url_decode; echo

{"sub":"deploy-bot","role":"admin","exp":1793318400}
{"alg":"HS256","typ":"JWT"}

In Python, base64.urlsafe_b64decode(part + "=" * (-len(part) % 4)) does the same in one line.

Anyone holding a signed JWT like this one can read its claims, because they are only base64url. The signature stops people from changing them, not from reading them, so never put a secret in a JWT payload. (Encrypted JWTs, JWE, are a different format with five parts.) For a quick look at a token, our JWT decoder runs in the browser, and the base64 tool does the same for plain values.

Data URIs

A data URI puts a whole file inside a URL: data:<media type>;base64,<data>. Browsers then skip the extra request. We built one from this site's 180x180 Apple touch icon, saved as icon.png:

wc -c < icon.png
printf 'data:image/png;base64,%s' "$(base64 -w0 icon.png)" | wc -c
printf 'data:image/png;base64,%s' "$(base64 -w0 icon.png)" | cut -c1-60

1167
1578
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAA

1,167 bytes became 1,578 characters: the 33% plus the 22-character prefix. Use -w0 again so the URI stays on one line in HTML or CSS. The image also loses its own cache entry and is downloaded again with every page or stylesheet that inlines it. That is fine for a few small icons and bad for anything large or shared across pages.

For SVG, which is already text, base64 is not always the best choice. Our 110-byte SVG made a 174-character base64 data URI and a 143-character URL-encoded one (data:image/svg+xml,%3Csvg ..., with single quotes inside the SVG).

Summary

Base64 is a fixed mapping from bytes to 64 safe characters. It makes binary data fit in text formats, costs a third more space, and hides nothing. The bugs come from a short list of mistakes:

• Base64 does not protect Secrets. RBAC, encryption at rest and keeping values out of Git do.
• echo adds a newline. Use printf '%s' and check suspect values with base64 -d | od -c.
• GNU base64 wraps at 76 characters. Use -w0, or | tr -d '\n' where macOS is involved. With openssl base64, pass -A both ways.
• data takes base64; stringData, --from-literal and --from-file take raw values. If the first decode still looks like base64, check for double encoding.
• JWTs use base64url without padding. Swap -_ back to +/ and restore the = before decoding.
• Check the exit code of base64 -d. GNU prints partial output before it fails, and the Rust coreutils in Ubuntu 26.10 behave differently on bad input.