A unit's recorded peak above its node's ceiling is refused for good: t's test checks error on testing since 21:11Z 11-13 #62

New issue

Open

opened 2026-11-13 21:35:15 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-13 21:35:15 +00:00

Owner

Seen on testing under b0dec1c3 and under 32527a94. Read-only finding: I changed nothing in testing's learned costs, so the case is still live there.

What happens. Every run that has to compile t's test binary (rustc --crate-name t ... src/main.rs, bin "t" test) ends its test driver twice with cache_unreachable and tells its test checks error:

t-wrapper: the compiler cache refused: a compile booked at 60.8 GiB can never fit on this node, which could offer 44.0 GiB at most

Runs: main's own r8011e0413f for 32527a94 at 21:11Z 11-13 (jobs j369-53, j369-54; errors: 1, test/t) and my forced r6ebf0642cd at 21:27Z (j371-6, j371-8; errors: 3, test/t, test/t-cache, test/t-wrapper). The run before, r31167eb59f for 98af034e at 20:16Z, had its test checks cached and did not compile.

Where the booking comes from. unit_costs key 29fa58706163… (family false), written 19:06:27Z: peaks 12.5, 57.0, 12.8 and 12.6 GiB over 202 runs. Its four kept courses and its four work courses never pass 13.4 GiB. So one peak sample of 57.0 GiB stands beside courses of 13; the booking follows the sample, the node's ceiling is 44.0 GiB, and the compile queue refuses it as never fitting (src/node/slots.rs:508).

When the sample was taken. Between 19:03:55 and 19:06:27Z. Your run r6bcc7805c5's test driver j364-26 (started 19:00:16) ended node_memory at 19:03:55, and its rerun j369-20 passed at 19:06:27, the second the row was written. That driver ran through my rollback proof of issue #61: testing's worker was replaced in place twice between 19:00 and 19:02Z (9faf6c61, then b0dec1c3 again). So the sample may come from a compile measured across a worker replaced under it, or from the attempt the node ended for memory. I did not find which.

Two things wrong, as I read it.

A peak of 57 GiB was recorded for a compile whose courses say 13 GiB, on a node that offers 44 GiB at most.
Once a unit's booking passes its node's ceiling there is no way out. It is refused before it runs, so it is never measured again and the sample never leaves its last four; its checks end error, as t's own fault, on every run until the row ages out or is removed. Nothing bounds a booking by the ceiling (the unit alone at the ceiling) or drops a sample no node could have held.

Consequence now. t's test checks cannot run on testing for any head that compiles t's test binary, and production does not run t's checks: 32527a94 has no test verdict from CI. On production I found no sign of the same: 0 jobs ended cache_unreachable since 16:32Z, and of 30,514 unit costs with peaks 28 hold a peak at least three times their smallest, one of them above 20 GiB (from 10-31). Its ceiling is 552 GB, so point 2 cannot bite there, but a sample like this one would book a unit several times too high until four newer runs displace it.

Evidence kept on my side. The row as JSON, the failed check's log and the worker's lines of 21:27Z.

21:44Z 11-13, again on the next head. Main's own run r4367f6aa27 for 9dedec3b ended errors: 1 the same way (6 cached, 14 passed), so that head has no test verdict either. Nothing in testing's learned costs has changed: the row still holds the 57.0 GiB peak.

22:48Z 11-13, still so, three heads on. Main's runs for 6ebaa239 (r645daa7738, 21:55Z) ended errors: 1 the same way, and the run for e5835fde (r504838c695, 22:47Z) has already had its test driver end cache_unreachable twice. The row still holds the 57.0 GiB peak (peaks: 13434385384, 61222353200, 13734233176, 13540253288). 6ebaa239 went to production at 22:08Z without a test verdict from CI.

23:08Z 11-13: I reset the one row on testing; the mechanism is untouched and this issue stays open. I need t's tests to run on testing before I land a fix for #65, and this refusal blocked them. In testing's unit_costs row of key 29fa58706163… I removed the 61,222,353,200-byte sample from peaks (left: 13434385384, 13734233176, 13540253288) and set peak.mean to the mean of those, then restarted testing's server so it reads the row again. The row as it was is saved on my side. To have the case back, write a peak above the node's ceiling into any compile's row: both points above still hold in the code.

23:15Z: with the row reset, the tests run. My forced run r58f25295a0 on testing (e5835fde) ended errors: 0, passed: 3 after 286 s, 19 jobs all passing; the test driver that had ended cache_unreachable within 7 s took 135 s. The row learned a fresh peak of 13.76 GB beside the three left. So the refusal was that one sample and nothing else.

23:45Z: main's own run passes again. r81bc98e0f9, testing's run for main's head 6ea02834, ended errors: 0, failed: 0, passed: 15, cached: 6: the first of main's runs tonight with its test checks run and green. The reset row is the only difference from the three before it.

Seen on testing under b0dec1c3 and under 32527a94. Read-only finding: I changed nothing in testing's learned costs, so the case is still live there. **What happens.** Every run that has to compile `t`'s test binary (`rustc --crate-name t ... src/main.rs`, bin "t" test) ends its `test` driver twice with `cache_unreachable` and tells its test checks `error`: ``` t-wrapper: the compiler cache refused: a compile booked at 60.8 GiB can never fit on this node, which could offer 44.0 GiB at most ``` Runs: main's own r8011e0413f for 32527a94 at 21:11Z 11-13 (jobs j369-53, j369-54; `errors: 1`, test/t) and my forced r6ebf0642cd at 21:27Z (j371-6, j371-8; `errors: 3`, test/t, test/t-cache, test/t-wrapper). The run before, r31167eb59f for 98af034e at 20:16Z, had its test checks cached and did not compile. **Where the booking comes from.** `unit_costs` key `29fa58706163…` (family false), written 19:06:27Z: `peaks` 12.5, 57.0, 12.8 and 12.6 GiB over 202 runs. Its four kept courses and its four work courses never pass 13.4 GiB. So one peak sample of 57.0 GiB stands beside courses of 13; the booking follows the sample, the node's ceiling is 44.0 GiB, and the compile queue refuses it as never fitting (src/node/slots.rs:508). **When the sample was taken.** Between 19:03:55 and 19:06:27Z. Your run r6bcc7805c5's `test` driver j364-26 (started 19:00:16) ended `node_memory` at 19:03:55, and its rerun j369-20 passed at 19:06:27, the second the row was written. That driver ran through my rollback proof of issue #61: testing's worker was replaced in place twice between 19:00 and 19:02Z (9faf6c61, then b0dec1c3 again). So the sample may come from a compile measured across a worker replaced under it, or from the attempt the node ended for memory. I did not find which. **Two things wrong, as I read it.** 1. A peak of 57 GiB was recorded for a compile whose courses say 13 GiB, on a node that offers 44 GiB at most. 2. Once a unit's booking passes its node's ceiling there is no way out. It is refused before it runs, so it is never measured again and the sample never leaves its last four; its checks end `error`, as t's own fault, on every run until the row ages out or is removed. Nothing bounds a booking by the ceiling (the unit alone at the ceiling) or drops a sample no node could have held. **Consequence now.** t's test checks cannot run on testing for any head that compiles `t`'s test binary, and production does not run t's checks: 32527a94 has no test verdict from CI. On production I found no sign of the same: 0 jobs ended `cache_unreachable` since 16:32Z, and of 30,514 unit costs with peaks 28 hold a peak at least three times their smallest, one of them above 20 GiB (from 10-31). Its ceiling is 552 GB, so point 2 cannot bite there, but a sample like this one would book a unit several times too high until four newer runs displace it. **Evidence kept on my side.** The row as JSON, the failed check's log and the worker's lines of 21:27Z. **21:44Z 11-13, again on the next head.** Main's own run r4367f6aa27 for 9dedec3b ended `errors: 1` the same way (6 cached, 14 passed), so that head has no test verdict either. Nothing in testing's learned costs has changed: the row still holds the 57.0 GiB peak. **22:48Z 11-13, still so, three heads on.** Main's runs for 6ebaa239 (r645daa7738, 21:55Z) ended `errors: 1` the same way, and the run for e5835fde (r504838c695, 22:47Z) has already had its `test` driver end `cache_unreachable` twice. The row still holds the 57.0 GiB peak (`peaks`: 13434385384, 61222353200, 13734233176, 13540253288). 6ebaa239 went to production at 22:08Z without a test verdict from CI. **23:08Z 11-13: I reset the one row on testing; the mechanism is untouched and this issue stays open.** I need t's tests to run on testing before I land a fix for #65, and this refusal blocked them. In testing's `unit_costs` row of key `29fa58706163…` I removed the 61,222,353,200-byte sample from `peaks` (left: 13434385384, 13734233176, 13540253288) and set `peak.mean` to the mean of those, then restarted testing's server so it reads the row again. The row as it was is saved on my side. To have the case back, write a peak above the node's ceiling into any compile's row: both points above still hold in the code. **23:15Z: with the row reset, the tests run.** My forced run r58f25295a0 on testing (e5835fde) ended `errors: 0, passed: 3` after 286 s, 19 jobs all passing; the `test` driver that had ended `cache_unreachable` within 7 s took 135 s. The row learned a fresh peak of 13.76 GB beside the three left. So the refusal was that one sample and nothing else. **23:45Z: main's own run passes again.** r81bc98e0f9, testing's run for main's head 6ea02834, ended `errors: 0, failed: 0, passed: 15, cached: 6`: the first of main's runs tonight with its test checks run and green. The reset row is the only difference from the three before it.

xemmerling referenced this issue

2026-11-13 21:51:13 +00:00

b0dec1c3 (declared keys through the cache): a rollback to 9faf6c61 cannot start, and five more findings to settle before a promote #61

xemmerling referenced this issue

2026-11-13 22:20:47 +00:00

The compile cache handed four jobs a crate file their dependents were not built against (E0463, E0460), 21:54-21:58Z 11-13 #63