drivers/cargo: the tilo-sized wave test fails on the forge runner two runs in five: CI's run was handed 443 shards for 442 packages (a binary run twice) #72

New issue

Open

opened 2026-11-14 05:32:14 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-14 05:32:14 +00:00

Owner

drivers::cargo::tests_multi_tilo::a_tilos_sized_wave_runs_across_nodes_as_it_does_on_one made the forge workflow red twice tonight, on commits that do not touch what it tests. The first failure says a test binary's run was handed out twice.

Where it failed and where it passed

Run	Commit	Result
forge 1463	`af8b5477`	passed
forge 1464	a branch one commit past `af8b5477` (database steps only)	failed at `tests_multi_tilo.rs:372`
forge 1466	`618f31ea`	passed
forge 1468	`ade63356`, main's head then (a deploy script only)	failed at `tests_multi_tilo.rs:307`
forge 1469	`ade63356` again, started by hand	passed
testing, alone, forced, 8 times	`ade63356`	8 passed, 23 to 39 s each

On testing it was among test/t's failed tests twice in six days (11-10 and 11-14), of 655 pass or fail verdicts of test/t.

Run 1464: 443 shards for 442 packages

442 checks of CI's wave and 442 of a person's across 3 nodes, mid-wave: 23.2s
panicked at src/drivers/cargo/tests_multi_tilo.rs:372:5:
assertion `left == right` failed: CI's run: a shard per package
  left: 443
 right: 442

The asserts before it had passed: every check of both runs has its one-node verdict, and no job was handed twice (the ids are distinct).
So CI's run was handed 443 distinct jobs with an archive for 442 packages: one binary's run went out twice, under two job ids. The comment above the assert says what should hold: "a binary is never run twice".

Run 1468: the wave never read as handed out

panicked at src/drivers/cargo/tests_multi_tilo.rs:307:5:
CI's wave is handed out

wait_for(&h, || shards(&h) == packages) did not hold in its 60 s. The test ran 94 s.
The condition is an equality. If a 443rd shard is handed out it can never hold, so this may be the same thing seen earlier in the test, before the node restarts. The log does not say how many shards there were.

Why I file it

If a binary's run can be handed out twice in one wave, production does that work twice. I have not found where it happens.
It keeps main's forge status red on commits that are fine: ade63356's push run is red, its second run is green.

A message with the count in the line-307 assert would tell the two cases apart the next time it fails.

2026-11-14 10:26Z: again on `58b70474`, now with the count

Forge run 1482 on 58b70474 failed in the same test after 42 s, at a later assert that says the count (tests_multi_tilo.rs:372):

442 checks of CI's wave and 442 of a person's across 3 nodes, mid-wave: 17.9s
assertion `left == right` failed: CI's run: a shard per package
  left: 446
 right: 442

So it is not a wait that was too short: CI's run was handed 446 shard jobs for 442 packages, four binaries' runs twice. "No job was handed twice" held, so they are different jobs.
The same test passed on runs 1476, 1479 and 1481 in 40 to 47 s, and t's own run of 58b70474 on testing passed.

`drivers::cargo::tests_multi_tilo::a_tilos_sized_wave_runs_across_nodes_as_it_does_on_one` made the forge workflow red twice tonight, on commits that do not touch what it tests. The first failure says a test binary's run was handed out twice. ## Where it failed and where it passed | Run | Commit | Result | |---|---|---| | forge 1463 | af8b5477 | passed | | forge 1464 | a branch one commit past af8b5477 (database steps only) | failed at `tests_multi_tilo.rs:372` | | forge 1466 | 618f31ea | passed | | forge 1468 | ade63356, main's head then (a deploy script only) | failed at `tests_multi_tilo.rs:307` | | forge 1469 | ade63356 again, started by hand | passed | | testing, alone, forced, 8 times | ade63356 | 8 passed, 23 to 39 s each | On testing it was among `test/t`'s failed tests twice in six days (11-10 and 11-14), of 655 pass or fail verdicts of `test/t`. ## Run 1464: 443 shards for 442 packages ``` 442 checks of CI's wave and 442 of a person's across 3 nodes, mid-wave: 23.2s panicked at src/drivers/cargo/tests_multi_tilo.rs:372:5: assertion `left == right` failed: CI's run: a shard per package left: 443 right: 442 ``` - The asserts before it had passed: every check of both runs has its one-node verdict, and `no job was handed twice` (the ids are distinct). - So CI's run was handed 443 distinct jobs with an archive for 442 packages: one binary's run went out twice, under two job ids. The comment above the assert says what should hold: "a binary is never run twice". ## Run 1468: the wave never read as handed out ``` panicked at src/drivers/cargo/tests_multi_tilo.rs:307:5: CI's wave is handed out ``` - `wait_for(&h, || shards(&h) == packages)` did not hold in its 60 s. The test ran 94 s. - The condition is an equality. If a 443rd shard is handed out it can never hold, so this may be the same thing seen earlier in the test, before the node restarts. The log does not say how many shards there were. ## Why I file it - If a binary's run can be handed out twice in one wave, production does that work twice. I have not found where it happens. - It keeps main's forge status red on commits that are fine: ade63356's push run is red, its second run is green. A message with the count in the line-307 assert would tell the two cases apart the next time it fails. ## 2026-11-14 10:26Z: again on 58b70474, now with the count Forge run 1482 on 58b70474 failed in the same test after 42 s, at a later assert that says the count (`tests_multi_tilo.rs:372`): ``` 442 checks of CI's wave and 442 of a person's across 3 nodes, mid-wave: 17.9s assertion `left == right` failed: CI's run: a shard per package left: 446 right: 442 ``` - So it is not a wait that was too short: CI's run was handed 446 shard jobs for 442 packages, four binaries' runs twice. "No job was handed twice" held, so they are different jobs. - The same test passed on runs 1476, 1479 and 1481 in 40 to 47 s, and t's own run of 58b70474 on testing passed.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

tilo/t#72

No description provided.

Rows
Columns