The forge workflow's test job has failed on every push to main since at least 11-05: eight tests fail on its runner without naming what they lack #67

New issue

Open

opened 2026-11-14 00:21:46 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-14 00:21:46 +00:00

Owner

The forge's own workflow for t (.forgejo/workflows/ci.yml) has not had a passing test job on main in any run I sampled back to 2026-11-05. fmt and clippy pass. Nobody reads it, me included until tonight, so it no longer tells a regression from its standing failures. Read from the job logs with fgj run view <run> --log-failed.

What fails, by run

Test	1125 (11-05)	1334 (11-10)	1449 (11-13, `e5835fde`)	1450 (11-13, `6ea02834`)
`node::observe::tests::every_lookup_once_units_begin_is_recorded_per_process`	fail	fail	fail	fail
`node::observe::tests::a_build_script_reading_git_head_is_keyed_by_it`	fail	fail	fail	fail
`node::observe::tests::what_a_linker_and_the_programs_it_runs_read_is_the_links`			fail	fail
`node::observe::tests::a_handle_opens_again_after_the_host_forgot_its_inode` (mine, new in `6ea02834`)				fail
`node::pasta_flows::linux::tests::a_destroyed_socket_resets_its_peer`		fail	fail	fail
`node::sandbox::tests::a_job_s_own_process_is_told_from_what_its_launch_started_for_it`			fail	fail
`node::sandbox::linux::tests::starts_from_the_template_copy_none_of_the_running_jobs_mounts`				fail
`node::cgroup::tests::a_job_past_its_memory_high_is_throttled_never_killed_and_its_node_is_told_at_once`			timeout 180 s	timeout 180 s
`ci::tests::a_check_failing_across_a_range_of_heads_is_probed_log2_times`, `ci::tests::checks_failing_across_a_range_of_heads_share_log2_probe_runs`			timeout	pass

Run 1125 also failed four tests that pass today (layers::every_module_imports_only_what_its_layer_allows, proto::tests::every_route_is_classified, harness::tests::a_dropped_harness_kills_its_group, jobs::flight::tests::evaluations_run_wide_within_the_sets_in_hand).

What each says

The four mounted tests of the source server: the first read through the mount answers ECONNREFUSED (once ENOTCONN), e.g. src/node/observe.rs:1842:38: called Result::unwrap() on an Err value: Os { code: 111, kind: ConnectionRefused }. The mount itself succeeds. That errno is what the kernel gives once the server's answer to INIT was refused, and Sources::init answers with an error when add_capabilities(KERNEL_WANTS) finds an ability missing (passthrough needs Linux 6.9, idmapped FUSE 6.12). Not confirmed: the runner's kernel version is not in the log, and the server's own error is never printed because the test panics before it joins the server thread.
My test: it fails there the same way, at its first stat, before it reaches open_by_handle_at. So defef173's need (handles, for a refusal with EPERM) is not what that runner hit; it may still be right for another machine.
a_destroyed_socket_resets_its_peer: src/node/pasta_flows.rs:432:27: destroying the held socket: No such file or directory (os error 2).
a_job_s_own_process_is_told_from_what_its_launch_started_for_it: src/node/sandbox.rs:2575:9: pasta and a helper alone.
starts_from_the_template_copy_none_of_the_running_jobs_mounts: src/node/sandbox.rs:1955:13: a start from the worker's copies all its 274 mounts: [273, 273, …].
The cgroup test runs past nextest's 180 s and is killed; its log says nothing else.

Why it matters

The workflow's own comment says a test that lacks what it needs "fails and says so, so everything below here is a decision about this machine". These eight fail without naming a need, and the workflow declares only volume,unprivileged.
t-testing's test check declares root,fuse,score,volume missing, so the mounted tests of the source server return early there. Together: those tests pass on no machine that runs on every push. The fix for #65/#66 (6ea02834) is proven by the reproduction on testing and by production, not by its mounted test. scripts/privileged-tests.sh is the one place they can run, by hand.

What I am doing

One developer is taking this: each of these tests either passes on the workflow's runner or fails naming the exact need it lacks (src/testing.rs), and the workflow declares each such need with its reason, as it does for volume and unprivileged. Proof is the workflow's own run on a pull request. No change to what t does on a node. If a test turns out to show a fault in t rather than in the machine, it gets an issue of its own instead.

00:30Z 11-14: main's head `c2d52b34`, run 1452

The test job failed again: 3,323 run, 6 failed, 1 timed out. The same tests as in run 1450 except the mount-count test, which passed this time (so it is not steady). My mounted test still fails at its first read through the mount with ECONNREFUSED (src/node/observe.rs:1842:38), with defef173's Need::Handles in: that change does not reach what this runner hits.

00:57Z 11-14: first trial on the runner (branch `dev/forge-ci`, commit `ec41199e`, run 1454)

The commit makes each failing test name what the machine lacks or print the fact that decides it, and corrects two tests that assumed too much. fmt and clippy passed. Test: 3,323 run, 6 failed, none timed out.

Test	What the run showed	Outcome
The four mounted tests of the source server	`Linux 6.12.47-kvm-podesta refused the mount's first request (Connection refused), and the source server said: the FUSE handshake: the kernel lacks InitFlags(FUSE_PASSTHROUGH)`	Confirmed: the runner's kernel is built without FUSE passthrough. New need `fusekernel`.
`a_destroyed_socket_resets_its_peer`	`Linux 6.12.47-kvm-podesta names no socket for the four-tuple of one /proc/net/tcp lists (No such file or directory)`	Confirmed: no TCP socket diagnostics in that kernel. New need `sockdiag`.
`a_job_s_own_process_is_told_from_what_its_launch_started_for_it`	`the cgroup holds ["30266 pasta.avx2 NSpid 30266 1", "30268 tilo-job-init NSpid 30268", "30305 sleep NSpid 30305"]`	The Fedora pasta the workflow installs puts itself in a PID namespace of its own, and `runs_own` takes it for a process of the job. Production's worker runs t's own build (`pasta 2026_10_02.cba3570-t1`). Whether that is the machine's lack or a fault in `runs_own` is being decided from the code.
`a_job_past_its_memory_high_is_throttled_never_killed_and_its_node_is_told_at_once`	Passed	The test kept a `memory.high` of 64 MiB under its 128 MiB `memory.max`; the half that is killed at its limit now has no `memory.high`.
`starts_from_the_template_copy_none_of_the_running_jobs_mounts`	Passed	It now counts the running jobs' mounts, not every mount of the namespace.
The two range-of-heads probe tests	Passed	Unchanged; each step now prints when it ended.

Next: the workflow declares fusekernel and sockdiag with their reasons, and the pasta test gets its decision.

01:19Z 11-14: landed as `86e00d08` and `3b8072d8`

Second trial on the runner (run 1455, with fusekernel and sockdiag declared in the workflow): fmt and clippy pass; test 3,323 run, 3,322 passed, 1 failed. The four mounted tests and the socket test now skip and the workflow's last step prints them by reason, with the kernel's release.
t's own check set on testing passed for the landing candidate (run r8a39552ebb, 8 of 8).
On main: 86e00d08 (the tests say what their machine lacked; two tests corrected) and 3b8072d8 (the workflow declares the runner's two limits, with what would lift them: a guest kernel with CONFIG_FUSE_PASSTHROUGH=y, and with CONFIG_INET_DIAG, CONFIG_INET_TCP_DIAG and CONFIG_INET_DIAG_DESTROY).
The one failure left is a_job_s_own_process_is_told_from_what_its_launch_started_for_it. It is not the runner's lack: it shows a fault in t, filed as #68, whose fix is on trial now. The test job turns green with it.

Still true after this: the source server's mounted tests run on no machine that runs on every push. They need a runner kernel with FUSE passthrough, or scripts/privileged-tests.sh by hand.

01:54Z 11-14: the workflow's `test` job is green on main

Run 1459, for 95935ccd (the fix for #68 on top of the two commits above): fmt, clippy and test all succeeded. It is the first green test job on main in every run I sampled back to 11-05. The trial before it on the runner (run 1456): 3,326 tests run, 3,326 passed, 42 skipped, with the skips printed by reason.

What stays open here:

The runner's kernel (6.12.47-kvm-podesta) has no FUSE passthrough and no TCP socket diagnostics, so the four mounted tests of the source server and the socket-destroy test skip there, declared as fusekernel and sockdiag. With t-testing declaring fuse and root missing, those tests still run on no machine on every push. A guest kernel with CONFIG_FUSE_PASSTHROUGH=y, CONFIG_INET_DIAG, CONFIG_INET_TCP_DIAG and CONFIG_INET_DIAG_DESTROY would close that; then drop the two names from the workflow.
The two range-of-heads probe tests run close to nextest's 180 s there (165 to 171 s when they pass). Each step now prints when it ended, so a timeout will name the slow step.
From now on a red test job on main means something, so it should be read after every push.

The forge's own workflow for t (`.forgejo/workflows/ci.yml`) has not had a passing `test` job on main in any run I sampled back to 2026-11-05. `fmt` and `clippy` pass. Nobody reads it, me included until tonight, so it no longer tells a regression from its standing failures. Read from the job logs with `fgj run view <run> --log-failed`. ## What fails, by run | Test | 1125 (11-05) | 1334 (11-10) | 1449 (11-13, e5835fde) | 1450 (11-13, 6ea02834) | |---|---|---|---|---| | `node::observe::tests::every_lookup_once_units_begin_is_recorded_per_process` | fail | fail | fail | fail | | `node::observe::tests::a_build_script_reading_git_head_is_keyed_by_it` | fail | fail | fail | fail | | `node::observe::tests::what_a_linker_and_the_programs_it_runs_read_is_the_links` | | | fail | fail | | `node::observe::tests::a_handle_opens_again_after_the_host_forgot_its_inode` (mine, new in 6ea02834) | | | | fail | | `node::pasta_flows::linux::tests::a_destroyed_socket_resets_its_peer` | | fail | fail | fail | | `node::sandbox::tests::a_job_s_own_process_is_told_from_what_its_launch_started_for_it` | | | fail | fail | | `node::sandbox::linux::tests::starts_from_the_template_copy_none_of_the_running_jobs_mounts` | | | | fail | | `node::cgroup::tests::a_job_past_its_memory_high_is_throttled_never_killed_and_its_node_is_told_at_once` | | | timeout 180 s | timeout 180 s | | `ci::tests::a_check_failing_across_a_range_of_heads_is_probed_log2_times`, `ci::tests::checks_failing_across_a_range_of_heads_share_log2_probe_runs` | | | timeout | pass | Run 1125 also failed four tests that pass today (`layers::every_module_imports_only_what_its_layer_allows`, `proto::tests::every_route_is_classified`, `harness::tests::a_dropped_harness_kills_its_group`, `jobs::flight::tests::evaluations_run_wide_within_the_sets_in_hand`). ## What each says - **The four mounted tests of the source server**: the first read through the mount answers `ECONNREFUSED` (once `ENOTCONN`), e.g. `src/node/observe.rs:1842:38: called Result::unwrap() on an Err value: Os { code: 111, kind: ConnectionRefused }`. The mount itself succeeds. That errno is what the kernel gives once the server's answer to INIT was refused, and `Sources::init` answers with an error when `add_capabilities(KERNEL_WANTS)` finds an ability missing (passthrough needs Linux 6.9, idmapped FUSE 6.12). Not confirmed: the runner's kernel version is not in the log, and the server's own error is never printed because the test panics before it joins the server thread. - **My test**: it fails there the same way, at its first `stat`, before it reaches `open_by_handle_at`. So defef173's need (`handles`, for a refusal with `EPERM`) is not what that runner hit; it may still be right for another machine. - `a_destroyed_socket_resets_its_peer`: `src/node/pasta_flows.rs:432:27: destroying the held socket: No such file or directory (os error 2)`. - `a_job_s_own_process_is_told_from_what_its_launch_started_for_it`: `src/node/sandbox.rs:2575:9: pasta and a helper alone`. - `starts_from_the_template_copy_none_of_the_running_jobs_mounts`: `src/node/sandbox.rs:1955:13: a start from the worker's copies all its 274 mounts: [273, 273, …]`. - The cgroup test runs past nextest's 180 s and is killed; its log says nothing else. ## Why it matters - The workflow's own comment says a test that lacks what it needs "fails and says so, so everything below here is a decision about this machine". These eight fail without naming a need, and the workflow declares only `volume,unprivileged`. - t-testing's `test` check declares `root,fuse,score,volume` missing, so the mounted tests of the source server return early there. Together: those tests pass on no machine that runs on every push. The fix for #65/#66 (6ea02834) is proven by the reproduction on testing and by production, not by its mounted test. `scripts/privileged-tests.sh` is the one place they can run, by hand. ## What I am doing One developer is taking this: each of these tests either passes on the workflow's runner or fails naming the exact need it lacks (`src/testing.rs`), and the workflow declares each such need with its reason, as it does for `volume` and `unprivileged`. Proof is the workflow's own run on a pull request. No change to what t does on a node. If a test turns out to show a fault in t rather than in the machine, it gets an issue of its own instead. ## 00:30Z 11-14: main's head c2d52b34, run 1452 The `test` job failed again: 3,323 run, 6 failed, 1 timed out. The same tests as in run 1450 except the mount-count test, which passed this time (so it is not steady). My mounted test still fails at its first read through the mount with `ECONNREFUSED` (`src/node/observe.rs:1842:38`), with defef173's `Need::Handles` in: that change does not reach what this runner hits. ## 00:57Z 11-14: first trial on the runner (branch `dev/forge-ci`, commit ec41199e, run 1454) The commit makes each failing test name what the machine lacks or print the fact that decides it, and corrects two tests that assumed too much. fmt and clippy passed. Test: 3,323 run, 6 failed, none timed out. | Test | What the run showed | Outcome | |---|---|---| | The four mounted tests of the source server | `Linux 6.12.47-kvm-podesta refused the mount's first request (Connection refused), and the source server said: the FUSE handshake: the kernel lacks InitFlags(FUSE_PASSTHROUGH)` | Confirmed: the runner's kernel is built without FUSE passthrough. New need `fusekernel`. | | `a_destroyed_socket_resets_its_peer` | `Linux 6.12.47-kvm-podesta names no socket for the four-tuple of one /proc/net/tcp lists (No such file or directory)` | Confirmed: no TCP socket diagnostics in that kernel. New need `sockdiag`. | | `a_job_s_own_process_is_told_from_what_its_launch_started_for_it` | `the cgroup holds ["30266 pasta.avx2 NSpid 30266 1", "30268 tilo-job-init NSpid 30268", "30305 sleep NSpid 30305"]` | The Fedora pasta the workflow installs puts itself in a PID namespace of its own, and `runs_own` takes it for a process of the job. Production's worker runs t's own build (`pasta 2026_10_02.cba3570-t1`). Whether that is the machine's lack or a fault in `runs_own` is being decided from the code. | | `a_job_past_its_memory_high_is_throttled_never_killed_and_its_node_is_told_at_once` | Passed | The test kept a `memory.high` of 64 MiB under its 128 MiB `memory.max`; the half that is killed at its limit now has no `memory.high`. | | `starts_from_the_template_copy_none_of_the_running_jobs_mounts` | Passed | It now counts the running jobs' mounts, not every mount of the namespace. | | The two range-of-heads probe tests | Passed | Unchanged; each step now prints when it ended. | Next: the workflow declares `fusekernel` and `sockdiag` with their reasons, and the pasta test gets its decision. ## 01:19Z 11-14: landed as 86e00d08 and 3b8072d8 - Second trial on the runner (run 1455, with `fusekernel` and `sockdiag` declared in the workflow): fmt and clippy pass; test 3,323 run, 3,322 passed, 1 failed. The four mounted tests and the socket test now skip and the workflow's last step prints them by reason, with the kernel's release. - t's own check set on testing passed for the landing candidate (run r8a39552ebb, 8 of 8). - On main: 86e00d08 (the tests say what their machine lacked; two tests corrected) and 3b8072d8 (the workflow declares the runner's two limits, with what would lift them: a guest kernel with CONFIG_FUSE_PASSTHROUGH=y, and with CONFIG_INET_DIAG, CONFIG_INET_TCP_DIAG and CONFIG_INET_DIAG_DESTROY). - The one failure left is `a_job_s_own_process_is_told_from_what_its_launch_started_for_it`. It is not the runner's lack: it shows a fault in t, filed as #68, whose fix is on trial now. The `test` job turns green with it. Still true after this: the source server's mounted tests run on no machine that runs on every push. They need a runner kernel with FUSE passthrough, or `scripts/privileged-tests.sh` by hand. ## 01:54Z 11-14: the workflow's `test` job is green on main Run 1459, for 95935ccd (the fix for #68 on top of the two commits above): fmt, clippy and test all succeeded. It is the first green `test` job on main in every run I sampled back to 11-05. The trial before it on the runner (run 1456): 3,326 tests run, 3,326 passed, 42 skipped, with the skips printed by reason. What stays open here: - The runner's kernel (6.12.47-kvm-podesta) has no FUSE passthrough and no TCP socket diagnostics, so the four mounted tests of the source server and the socket-destroy test skip there, declared as `fusekernel` and `sockdiag`. With t-testing declaring `fuse` and `root` missing, those tests still run on no machine on every push. A guest kernel with `CONFIG_FUSE_PASSTHROUGH=y`, `CONFIG_INET_DIAG`, `CONFIG_INET_TCP_DIAG` and `CONFIG_INET_DIAG_DESTROY` would close that; then drop the two names from the workflow. - The two range-of-heads probe tests run close to nextest's 180 s there (165 to 171 s when they pass). Each step now prints when it ended, so a timeout will name the slow step. - From now on a red `test` job on main means something, so it should be read after every push.

xemmerling referenced this issue from a commit

2026-11-14 00:41:20 +00:00

testing, node, ci: a test the forge's runner fails says what its machine lacked or what it saw there

xemmerling referenced this issue

2026-11-14 01:06:34 +00:00

node: a cut launch with a network is still followed as running: pasta lives in a PID namespace of its own, so runs_own takes it for the job's process #68

xemmerling referenced this issue from a commit

2026-11-14 01:19:10 +00:00

testing, node, ci: a test the forge's runner fails says what its machine lacked or what it saw there