tilo/t

Watch

Fork 0

node/admit: `7ef99e3f` held CI out of the node while people used half of it on 11-12 (CI job starts 148 to 1.2 a minute); did not repeat on `eeb2996a` on 11-13, see the update #52

New issue

Open

opened 2026-11-13 02:19:27 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-13 02:19:27 +00:00

Owner

Update, 2026-11-13 03:45Z, by the author of this issue. Main is being promoted again: its head went to production as eeb2996a at 02:42Z and as d26adb93 at 03:40Z (infra bd5adfd3 and 4ce769e0). What is described below did not repeat on eeb2996a:

	`86fa55c0`, 02:15 to 02:42Z	`eeb2996a`, 03:13 to 03:40Z
Cores used by CI's job cgroups, mean of one sample a minute	96	91 (lowest 49)
Cores used by people's job cgroups, mean	64	65 (highest 110)
Both	160	155
Node minutes, mean cores used	153.5	about 147
People's jobs, wait to start	0.5 to 1.4 s median	1.0 s median, 3.0 s p95

With people at 84 to 110 cores (03:14 to 03:20Z and 03:34 to 03:36Z) CI kept 49 to 76 cores. With people at 29 to 35 cores (03:26 to 03:29Z) CI took 118 to 126 within a minute.
CPU pressure "some" avg60 read 23% at 03:12Z and 21% at 03:43Z (28 to 29% in the table below); the quota was throttled in 119 and 171 of 201 periods.
No job ended with a cause and no error of t's own was told in the hour; 52 jobs ran across the restart at 02:42Z and 28 across the one at 03:40Z.
Not seen yet on this commit: people asking for well over half the node for ten minutes or more, the case of the table below.
Between ac720615 and eeb2996a these commits touch src/node/admit.rs: 7c9c656b. I have not measured which of them made the difference.
Still open, and the owner's decision: what share of the cores CI keeps when people ask for all of them.

Second update, 2026-11-13 04:07Z. The case not seen yet came twenty minutes later, on d26adb93: people asking for the whole node.

Five minutes from	03:40Z	03:45Z	03:50Z	03:55Z	04:00Z	04:05Z
Cores used by CI's job cgroups	63	54	19	23	18	2
Cores used by people's job cgroups	86	79	138	136	143	159

From 03:59Z no CI job started in any minute. At 04:05:54Z CI's 17 running jobs (9 clippy, 7 clippy-x86_64, one web-desktop-wasm) used 0 cores between them: placed and holding their memory, with none of their units entering.
People's side at 04:05Z: one principal's 9 sessions with 54 jobs, 435 GB booked and 647 tokens (two jobs 163 wide), another's 2 sessions with 4 jobs. People's waits stayed under 3 s.
The count read 166.8 against the budget of 163 (sampled 159.2); all 46,435 queued asks waited for room.
This is the case of the paragraph "For fairness" below, where 86fa55c0 held CI out as well. So it is not a regression of the admission change; it is the open decision on what share CI keeps, now with CI's placed jobs standing still on memory they hold.

Third update, 2026-11-13 04:43Z. The fairest comparison I have: CI's cores at equal people's load, from one cgroup sample a minute. Old is 86fa55c0 from 02:15 to 02:42Z (30 samples); new is eeb2996a and d26adb93 from 02:57 to 04:42Z without the minutes around the 03:40Z restart (111 samples).

People's cores	Old: samples	Old: CI cores	Old: both	New: samples	New: CI cores	New: both
under 40	2	129	160	7	118	152
40 to 69	19	101	161	33	96	151
70 to 99	8	80	158	40	66	153
100 to 129	1	53	159	10	42	156
130 and more	0			21	16	160

At equal people's load the node runs 3 to 10 cores lower and CI 5 to 14 cores lower since the change. A moderate cost on few old samples, not the collapse of 2026-11-12.
Not measured: whether CI's work per core rose with the lower pressure, which was the change's aim.
By the hour, CI's job-hours ended were 72.0, 97.2 and 75.4 in the three hours to 03:00Z and 34.5 in the hour after, with people's at 23.1, 32.0, 30.9 and 31.4. That hour holds two restarts, so I do not put the difference on the change.

Fourth update, 2026-11-13 05:45Z. Where the cores go, from Prometheus (production only): the count admission holds (t_node_busy_cores, budget 163) against the cores the node's jobs were sampled using (t_node_sampled_cores).

Half hour from	01:00Z	01:30Z	02:00Z	03:00Z	03:30Z	04:00Z	04:30Z	05:00Z	05:30Z
Code	`86fa55c0`	`86fa55c0`	`86fa55c0`	`eeb2996a`	`eeb2996a`	`d26adb93`	`d26adb93`	`166e9649`	`166e9649`
Count over cores sampled	108%	109%	104%	115%	118%	116%	111%	122%	118%
Cores sampled, mean	150	148	154	148	143	139	149	147	148

Two-minute samples from 05:30Z to 05:36Z: the count read 164, 164, 164, 165 while the jobs used 137, 141, 123, 112 cores; of the count, 24, 20, 39, 50 cores were for jobs not yet sampled (t_node_unsampled_cores). CI was starting 100 to 165 short jobs a minute then, and 55,000 asks waited for room.

So the count now runs 11 to 22% over what the jobs use, against 4 to 9% before the change, and it holds the node at its budget while 20 to 50 cores stand idle. The largest part in these minutes is the cores counted for jobs that have started and not been sampled yet: a stream of short CI jobs keeps that term high.
That matches the third update: CI 5 to 14 cores lower and the node 3 to 10 lower at equal people's load.

The original report follows unchanged.

Seen on t.tilo.so on 2026-11-12. The commit 7ef99e3f ("node/admit, sched/knee: a unit is counted at what it asks of its node's cores, what it kept busy and the share of its second its tasks waited for a core") was on production as ac720615 from 20:43Z to 21:05Z and was taken off again (production back on 86fa55c0). Main still carries it, so main has not been promoted since; fourteen commits wait behind it, fixes for issues #49, #50 and #51 among them.

What happened while it was live:

	`86fa55c0`, 20:00 to 20:42Z	`ac720615`, 20:45 to 21:04Z
CI job starts a minute	148 (one minute of 42 without a start)	1.2 (20:54 to 21:03Z: 0, 0, 0, 0, 4, 1, 0, 6, 0, 3)
Cores used by CI's job cgroups	86 to 91	12 to 24
Cores used by people's job cgroups	74 to 100	137 to 149
Quota periods throttled (20 s samples)	131 of 201, 169 of 215	197 of 200
CPU pressure "some", avg60	28 to 29%	33 to 38%
People's median job run time	58 to 60 s	48 s

From 20:48 to 20:52Z CI started nothing while people used only 70 to 102 cores, kept memory was under 82% of the offer and no entry was refused for memory. Every one of CI's 46,223 waiting entries was short of cores, none of memory.
The count (t_node_busy_cores) read 237 to 274 against the budget of 163, of it 50 to 113 the waited term. People's units enter on memory alone and keep tasks waiting, so the count did not come back under the budget and CI's work did not enter. CI held only what its running jobs had: 201 running entries fell to 9 jobs in twenty minutes.
No job was lost and no error was told; people's waits stayed at 1 to 3 s.
The replay that argued for the change showed makespan level. It has no class that enters past the count.

For fairness, what this does not show: later that evening, from about 21:15Z to 22:15Z on 86fa55c0, CI was held out as well (0 to 20 cores) while one principal's 14 sessions asked for the whole node (1,084 GB booked, 150 to 160 cores). On the old rule too, people's units enter on memory alone and take the cores first when they ask for all of them. The measured difference between the two rules is the case where people ask for about half the node: there 86fa55c0 gave CI 86 to 91 cores and this change gave it none.

Wanted, one of: a fix of the count under which CI's work still enters while people's units wait for a core (what share CI keeps when people ask for all the cores is the owner's decision and is open), or a revert of 7ef99e3f on main until that fix is ready. Either unblocks the promote of everything behind it.

This was sent to the admission change's author by relay message on the evening itself; the relay inbox it went to has been offline since 2026-11-12 10:31Z, so it is filed here.

2026-11-13 evening: it repeated in another form (read 00:33Z 11-14)

No build of ours held CI out this time. People's work took the node by the rule that people go first, and CI's head verdicts stopped. From t.job_history by priority, two-hour buckets by end time:

Ended in	CI jobs	CI job-hours	People's jobs	People's job-hours	People's share of job-hours
16-18Z	15,451	193	1,606	43	18%
18-20Z	8,383	85	4,034	109	56%
20-22Z	2,323	38	4,378	144	79%
22-24Z	505	26	3,526	182	87%

People's job-hours 20-24Z by principal: xemmerling 175 (4,910 jobs), quinten 150 (2,994 jobs).
The node was busy throughout: 172 to 176 of 192 threads in each hour.
CI heads finished per hour: 15Z 6, 16Z 9, 17Z 11, 18Z 8, 19Z 3, 20Z 1, 21Z 1, 22Z 0, 23Z 1, 00Z 0 until 00:36Z.
Open CI heads at 00:36Z: 369, the oldest created 17:00Z 11-12. The queue is served oldest first as designed: the two oldest heads had only their last tests running, the third had 18 jobs running and 228 queued, every later one was wholly queued for room, short by one core.
On top of it, pushes to tilo's main tripled at 23:00Z: 16, 14 and 14 head runs per half hour spanning 69, 52 and 56 commits, against 3 to 8 heads before. That is about 30 heads an hour.

Nothing in t is stuck, so there is nothing here for me to patch. It is the same question as on 11-12: whether CI keeps a share of the node while people's sessions run, or a principal has a limit. That decision is with the user.

**Update, 2026-11-13 03:45Z, by the author of this issue.** Main is being promoted again: its head went to production as eeb2996a at 02:42Z and as d26adb93 at 03:40Z (infra bd5adfd3 and 4ce769e0). What is described below did not repeat on eeb2996a: | | 86fa55c0, 02:15 to 02:42Z | eeb2996a, 03:13 to 03:40Z | |---|---|---| | Cores used by CI's job cgroups, mean of one sample a minute | 96 | 91 (lowest 49) | | Cores used by people's job cgroups, mean | 64 | 65 (highest 110) | | Both | 160 | 155 | | Node minutes, mean cores used | 153.5 | about 147 | | People's jobs, wait to start | 0.5 to 1.4 s median | 1.0 s median, 3.0 s p95 | - With people at 84 to 110 cores (03:14 to 03:20Z and 03:34 to 03:36Z) CI kept 49 to 76 cores. With people at 29 to 35 cores (03:26 to 03:29Z) CI took 118 to 126 within a minute. - CPU pressure "some" avg60 read 23% at 03:12Z and 21% at 03:43Z (28 to 29% in the table below); the quota was throttled in 119 and 171 of 201 periods. - No job ended with a cause and no error of t's own was told in the hour; 52 jobs ran across the restart at 02:42Z and 28 across the one at 03:40Z. - Not seen yet on this commit: people asking for well over half the node for ten minutes or more, the case of the table below. - Between ac720615 and eeb2996a these commits touch src/node/admit.rs: 7c9c656b. I have not measured which of them made the difference. - Still open, and the owner's decision: what share of the cores CI keeps when people ask for all of them. **Second update, 2026-11-13 04:07Z.** The case not seen yet came twenty minutes later, on d26adb93: people asking for the whole node. | Five minutes from | 03:40Z | 03:45Z | 03:50Z | 03:55Z | 04:00Z | 04:05Z | |---|---|---|---|---|---|---| | Cores used by CI's job cgroups | 63 | 54 | 19 | 23 | 18 | 2 | | Cores used by people's job cgroups | 86 | 79 | 138 | 136 | 143 | 159 | - From 03:59Z no CI job started in any minute. At 04:05:54Z CI's 17 running jobs (9 clippy, 7 clippy-x86_64, one web-desktop-wasm) used 0 cores between them: placed and holding their memory, with none of their units entering. - People's side at 04:05Z: one principal's 9 sessions with 54 jobs, 435 GB booked and 647 tokens (two jobs 163 wide), another's 2 sessions with 4 jobs. People's waits stayed under 3 s. - The count read 166.8 against the budget of 163 (sampled 159.2); all 46,435 queued asks waited for room. - This is the case of the paragraph "For fairness" below, where 86fa55c0 held CI out as well. So it is not a regression of the admission change; it is the open decision on what share CI keeps, now with CI's placed jobs standing still on memory they hold. **Third update, 2026-11-13 04:43Z.** The fairest comparison I have: CI's cores at equal people's load, from one cgroup sample a minute. Old is 86fa55c0 from 02:15 to 02:42Z (30 samples); new is eeb2996a and d26adb93 from 02:57 to 04:42Z without the minutes around the 03:40Z restart (111 samples). | People's cores | Old: samples | Old: CI cores | Old: both | New: samples | New: CI cores | New: both | |---|---|---|---|---|---|---| | under 40 | 2 | 129 | 160 | 7 | 118 | 152 | | 40 to 69 | 19 | 101 | 161 | 33 | 96 | 151 | | 70 to 99 | 8 | 80 | 158 | 40 | 66 | 153 | | 100 to 129 | 1 | 53 | 159 | 10 | 42 | 156 | | 130 and more | 0 | | | 21 | 16 | 160 | - At equal people's load the node runs 3 to 10 cores lower and CI 5 to 14 cores lower since the change. A moderate cost on few old samples, not the collapse of 2026-11-12. - Not measured: whether CI's work per core rose with the lower pressure, which was the change's aim. - By the hour, CI's job-hours ended were 72.0, 97.2 and 75.4 in the three hours to 03:00Z and 34.5 in the hour after, with people's at 23.1, 32.0, 30.9 and 31.4. That hour holds two restarts, so I do not put the difference on the change. **Fourth update, 2026-11-13 05:45Z.** Where the cores go, from Prometheus (production only): the count admission holds (`t_node_busy_cores`, budget 163) against the cores the node's jobs were sampled using (`t_node_sampled_cores`). | Half hour from | 01:00Z | 01:30Z | 02:00Z | 03:00Z | 03:30Z | 04:00Z | 04:30Z | 05:00Z | 05:30Z | |---|---|---|---|---|---|---|---|---|---| | Code | 86fa55c0 | 86fa55c0 | 86fa55c0 | eeb2996a | eeb2996a | d26adb93 | d26adb93 | 166e9649 | 166e9649 | | Count over cores sampled | 108% | 109% | 104% | 115% | 118% | 116% | 111% | 122% | 118% | | Cores sampled, mean | 150 | 148 | 154 | 148 | 143 | 139 | 149 | 147 | 148 | Two-minute samples from 05:30Z to 05:36Z: the count read 164, 164, 164, 165 while the jobs used 137, 141, 123, 112 cores; of the count, 24, 20, 39, 50 cores were for jobs not yet sampled (`t_node_unsampled_cores`). CI was starting 100 to 165 short jobs a minute then, and 55,000 asks waited for room. - So the count now runs 11 to 22% over what the jobs use, against 4 to 9% before the change, and it holds the node at its budget while 20 to 50 cores stand idle. The largest part in these minutes is the cores counted for jobs that have started and not been sampled yet: a stream of short CI jobs keeps that term high. - That matches the third update: CI 5 to 14 cores lower and the node 3 to 10 lower at equal people's load. The original report follows unchanged. --- Seen on t.tilo.so on 2026-11-12. The commit 7ef99e3f ("node/admit, sched/knee: a unit is counted at what it asks of its node's cores, what it kept busy and the share of its second its tasks waited for a core") was on production as ac720615 from 20:43Z to 21:05Z and was taken off again (production back on 86fa55c0). Main still carries it, so main has not been promoted since; fourteen commits wait behind it, fixes for issues #49, #50 and #51 among them. What happened while it was live: | | 86fa55c0, 20:00 to 20:42Z | ac720615, 20:45 to 21:04Z | |---|---|---| | CI job starts a minute | 148 (one minute of 42 without a start) | 1.2 (20:54 to 21:03Z: 0, 0, 0, 0, 4, 1, 0, 6, 0, 3) | | Cores used by CI's job cgroups | 86 to 91 | 12 to 24 | | Cores used by people's job cgroups | 74 to 100 | 137 to 149 | | Quota periods throttled (20 s samples) | 131 of 201, 169 of 215 | 197 of 200 | | CPU pressure "some", avg60 | 28 to 29% | 33 to 38% | | People's median job run time | 58 to 60 s | 48 s | - From 20:48 to 20:52Z CI started nothing while people used only 70 to 102 cores, kept memory was under 82% of the offer and no entry was refused for memory. Every one of CI's 46,223 waiting entries was short of cores, none of memory. - The count (`t_node_busy_cores`) read 237 to 274 against the budget of 163, of it 50 to 113 the waited term. People's units enter on memory alone and keep tasks waiting, so the count did not come back under the budget and CI's work did not enter. CI held only what its running jobs had: 201 running entries fell to 9 jobs in twenty minutes. - No job was lost and no error was told; people's waits stayed at 1 to 3 s. - The replay that argued for the change showed makespan level. It has no class that enters past the count. For fairness, what this does not show: later that evening, from about 21:15Z to 22:15Z on 86fa55c0, CI was held out as well (0 to 20 cores) while one principal's 14 sessions asked for the whole node (1,084 GB booked, 150 to 160 cores). On the old rule too, people's units enter on memory alone and take the cores first when they ask for all of them. The measured difference between the two rules is the case where people ask for about half the node: there 86fa55c0 gave CI 86 to 91 cores and this change gave it none. Wanted, one of: a fix of the count under which CI's work still enters while people's units wait for a core (what share CI keeps when people ask for all the cores is the owner's decision and is open), or a revert of 7ef99e3f on main until that fix is ready. Either unblocks the promote of everything behind it. This was sent to the admission change's author by relay message on the evening itself; the relay inbox it went to has been offline since 2026-11-12 10:31Z, so it is filed here. ## 2026-11-13 evening: it repeated in another form (read 00:33Z 11-14) No build of ours held CI out this time. People's work took the node by the rule that people go first, and CI's head verdicts stopped. From `t.job_history` by `priority`, two-hour buckets by end time: | Ended in | CI jobs | CI job-hours | People's jobs | People's job-hours | People's share of job-hours | |---|---|---|---|---|---| | 16-18Z | 15,451 | 193 | 1,606 | 43 | 18% | | 18-20Z | 8,383 | 85 | 4,034 | 109 | 56% | | 20-22Z | 2,323 | 38 | 4,378 | 144 | 79% | | 22-24Z | 505 | 26 | 3,526 | 182 | 87% | - People's job-hours 20-24Z by principal: xemmerling 175 (4,910 jobs), quinten 150 (2,994 jobs). - The node was busy throughout: 172 to 176 of 192 threads in each hour. - CI heads finished per hour: 15Z 6, 16Z 9, 17Z 11, 18Z 8, 19Z 3, 20Z 1, 21Z 1, 22Z 0, 23Z 1, 00Z 0 until 00:36Z. - Open CI heads at 00:36Z: 369, the oldest created 17:00Z 11-12. The queue is served oldest first as designed: the two oldest heads had only their last tests running, the third had 18 jobs running and 228 queued, every later one was wholly queued for room, short by one core. - On top of it, pushes to tilo's main tripled at 23:00Z: 16, 14 and 14 head runs per half hour spanning 69, 52 and 56 commits, against 3 to 8 heads before. That is about 30 heads an hour. Nothing in t is stuck, so there is nothing here for me to patch. It is the same question as on 11-12: whether CI keeps a share of the node while people's sessions run, or a principal has a limit. That decision is with the user.

xemmerling changed title from ~~node/admit: 7ef99e3f held CI out of the node while people used half of it (CI job starts 148 to 1.2 a minute); taken off production, main not promotable since~~ to node/admit: 7ef99e3f held CI out of the node while people used half of it on 11-12 (CI job starts 148 to 1.2 a minute); did not repeat on eeb2996a on 11-13, see the update

2026-11-13 03:45:32 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

tilo/t#52

No description provided.

Rows
Columns

node/admit: 7ef99e3f held CI out of the node while people used half of it on 11-12 (CI job starts 148 to 1.2 a minute); did not repeat on eeb2996a on 11-13, see the update #52

2026-11-13 evening: it repeated in another form (read 00:33Z 11-14)

node/admit: `7ef99e3f` held CI out of the node while people used half of it on 11-12 (CI job starts 148 to 1.2 a minute); did not repeat on `eeb2996a` on 11-13, see the update #52