VM steps lose files of /src mid-job: stale file handles, failed bulk transfers and files 'not found' in tilo's image chain (most runs on 11-13) #65

New issue

Open

opened 2026-11-13 22:28:55 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-13 22:28:55 +00:00

Owner

VM steps of tilo's OS image chain fail reading the job's /src (the checkout as the guest sees it over virtiofs). Since 11-11 this ends most runs that need a fresh image before their own check starts; on 11-13 it is nearly every run. All on t-worker-ci-x86-1, all in isolation = "vm" steps, most in buildah/podman build COPY from /src.

Three shapes, the files differ every time and all exist in the commit the run used:

"error during bulk transfer for copier.request" on a COPY from /src

r9f6e32413c: os-tools-x86_64 (COPY os/external-assets/doubletake/) and os-layer-system-x86_64 (COPY os/etc/systemd/), two in one run
r68a322f58c: os-layer-system-x86_64 (COPY os/usr/lib/firewalld/)
ra3d481a041: os-layer-image-x86_64 (/src/crates)
r8050e08b88, r2c7e9a03e3: os-layer-image-x86_64 (/src/.cargo, /src/apps)

"Stale file handle" (ESTALE) on a path under /src

r8067f4786b and r127c920f83: os-layer-trust-x86_64, twice in a row at the same line: OSError: [Errno 116] Stale file handle: '/src/tools/ci/os' when python lists that directory (os/image/build-layer.sh:253-262); earlier reads of /src in the same step work
r3a8d95831b, radec7aab3f: os-assembler-x86_64 (os/dev/image-epoch.sh)
r97e6733298: os-dev-image-lean-x86_64 (os/dev/_common.sh)

"no such file or directory" for a file that is in the tree

r928c73a5e7: os-dev-image-lean-x86_64, COPY os/usr/libexec/tilo-session
r983b54178f: same step, COPY os/usr/lib/systemd/user/tilo-ush.service

Related, seen 11-08 (tilo 1a69c31c14c works around it): a file descriptor a VM step held open on its /target share started returning EBADF mid-job while new opens of the same file kept working.

What it is not: the checks declare no input filter that would leave these files out, a rerun on the same commit passes the step that failed and fails another, and none of the runs touched the files named.

Each failure is also kept as the step's answer for those inputs, so the next run on the same inputs repeats it without running (the asker has to --force or move to a new commit).

Cost: the e2e checks on real tilo VMs (e2e-branch, e2e-ws-branch, e2e-remote, e2e-agents) need the image chain; about two of three attempts since 11-12 ended here after 1 to 4 hours of image builds.

Logs: t log --run <run> <check> for each run above (session claude:26a57165-a7c7-4223-839d-7e1eaacff314, repository tilo).

VM steps of tilo's OS image chain fail reading the job's `/src` (the checkout as the guest sees it over virtiofs). Since 11-11 this ends most runs that need a fresh image before their own check starts; on 11-13 it is nearly every run. All on t-worker-ci-x86-1, all in `isolation = "vm"` steps, most in `buildah`/`podman build` `COPY` from `/src`. Three shapes, the files differ every time and all exist in the commit the run used: **"error during bulk transfer for copier.request" on a COPY from /src** - r9f6e32413c: `os-tools-x86_64` (`COPY os/external-assets/doubletake/`) and `os-layer-system-x86_64` (`COPY os/etc/systemd/`), two in one run - r68a322f58c: `os-layer-system-x86_64` (`COPY os/usr/lib/firewalld/`) - ra3d481a041: `os-layer-image-x86_64` (`/src/crates`) - r8050e08b88, r2c7e9a03e3: `os-layer-image-x86_64` (`/src/.cargo`, `/src/apps`) **"Stale file handle" (ESTALE) on a path under /src** - r8067f4786b and r127c920f83: `os-layer-trust-x86_64`, twice in a row at the same line: `OSError: [Errno 116] Stale file handle: '/src/tools/ci/os'` when python lists that directory (`os/image/build-layer.sh:253-262`); earlier reads of `/src` in the same step work - r3a8d95831b, radec7aab3f: `os-assembler-x86_64` (`os/dev/image-epoch.sh`) - r97e6733298: `os-dev-image-lean-x86_64` (`os/dev/_common.sh`) **"no such file or directory" for a file that is in the tree** - r928c73a5e7: `os-dev-image-lean-x86_64`, `COPY os/usr/libexec/tilo-session` - r983b54178f: same step, `COPY os/usr/lib/systemd/user/tilo-ush.service` Related, seen 11-08 (tilo 1a69c31c14c works around it): a file descriptor a VM step held open on its `/target` share started returning `EBADF` mid-job while new opens of the same file kept working. What it is not: the checks declare no input filter that would leave these files out, a rerun on the same commit passes the step that failed and fails another, and none of the runs touched the files named. Each failure is also kept as the step's answer for those inputs, so the next run on the same inputs repeats it without running (the asker has to `--force` or move to a new commit). Cost: the e2e checks on real tilo VMs (`e2e-branch`, `e2e-ws-branch`, `e2e-remote`, `e2e-agents`) need the image chain; about two of three attempts since 11-12 ended here after 1 to 4 hours of image builds. Logs: `t log --run <run> <check>` for each run above (session `claude:26a57165-a7c7-4223-839d-7e1eaacff314`, repository tilo).

xemmerling referenced this issue

2026-11-13 23:09:35 +00:00

VM steps lose /src (#65): virtiofsd addresses the worker's source filesystem by file handle, which it cannot honour once the host evicts the inode #66

xemmerling referenced this issue

2026-11-13 23:09:41 +00:00

A unit's recorded peak above its node's ceiling is refused for good: t's test checks error on testing since 21:11Z 11-13 #62

xemmerling referenced this issue from a commit

2026-11-13 23:39:07 +00:00

node/observe: a handle to a source inode opens again after the host forgot it