tilo/t

Watch

Fork 0

node/admit, sched: after `d59240c7` booked cores hold at the offer between waves, but a wave of starts still reads 294 and 326 of 159 and the node counts 412 busy while the machine has 170 (05:08 to 05:30Z 11-14) #71

New issue

Open

opened 2026-11-14 05:31:30 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-14 05:31:30 +00:00

Owner

Data for the planner, read on production after the promote of 618f31ea (04:39Z 11-14), which carries d59240c7. Nothing here is a request to change admission by hand.

Between waves the fix holds

t status, 30 samples 20 s apart, 05:15 to 05:26Z, beside the node's own busy count from Prometheus:

	Before `d59240c7` (25 readings, 11-13)	After (30 samples)
Over the offer of 159 by more than two cores	13	11
Mean excess of those	34	11
Largest booked	256	186
Median booked		159
Mean booked / used by the server's count / busy by the node's count		164 / 157 / 156

At the start of a wave booked still doubles

Two readings of /v1/views/jobs fell on waves: 294/159 at 05:12Z and 326/159 at 05:27Z. Minute by minute around the first (Prometheus; the operating system's figures are node-exporter's for the node):

Minute	Jobs started	Running	Node's busy cores	Unsampled	Awaiting	OS busy cores	CPU pressure
05:10	25	49	243	27	10	178	0.40
05:11	215	167	127	26	0	141	0.11
05:12	507	421	146	24	0	169	0.23
05:13	20	233	412	1	0	170	0.50
05:14	73	137	137	0	12	170	0.19
05:27	152	97	132	2	57	169	0.23
05:28	12	112	185	3	68	181	0.49

507 jobs started in the minute of 05:12Z. The server's booked cores read 294, and a minute later the node's own count read 412 busy cores, with 1 unsampled.
The machine did not follow either count: 169 and 170 cores busy by the operating system, CPU pressure 0.50 for that one minute against 0.2 to 0.3 around it.
The minute after each wave few jobs start (20 at 05:13Z, 12 at 05:28Z) while the operating system's busy cores stay at 170 to 181.
No job ended for memory or for a cause of t's own in these half hours. Kept memory stayed under the offer.

Also seen at the takeover

For about ten minutes after the server's start at 04:39Z: 337 jobs running at 04:44Z, booked memory 639 GB and then 649 GB against 554 GB offered, the node's busy count 411, load 640, CPU pressure 0.47 then 0.70. No job ended for memory.

The sampler is ~/.cache/t-factory/booked-vs-busy.sh on my side; its log holds each sample with the checks running and queued.

On `67588367` (promoted 06:21Z), the same at 07:18Z

369 jobs started in the minute of 07:18Z and 355 were running.
/v1/views/jobs read 474 cores used on the node, with 159 of 159 booked. The node's own busy count read 238 (153 sampled), the operating system 178, CPU pressure 0.57.
Between waves in the minutes before: the node's busy count 166 to 224, the operating system 175 to 180.

So the views' used overshoots at a wave's start as booked did, and by more.

Six minutes with no job started, 09:54 to 09:59Z, on `c6dca088`

Minute	Jobs started	Running	Node's busy cores	Sampled	Unsampled	OS busy cores	CPU pressure	Load	Kept memory	Offered
09:53	73	96	143	136	7	174	0.32	386	238 GB	560 GB
09:54	1	84	167	156	1	176	0.34	399	238 GB	561 GB
09:55	1	73	160	155	0	178	0.33	329	238 GB	558 GB
09:56	0	63	837	157	0	177	0.84	1,066	681 GB	547 GB
09:57	0	54	549	158	0	176	0.86	1,262	691 GB	558 GB
09:58	0	47	453	157	0	176	0.78	1,242	676 GB	558 GB
09:59	0	42	127	125	0	166	0.47	948	681 GB	553 GB
10:00	236	151	165	134	28	147	0.14	597	238 GB	555 GB

About 72,000 asks were queued throughout. No job ended for memory and none was lost.
The node's busy count read 837, 549 and 453 cores while its sampled cores stayed at 157 and its unsampled at 0. So the busy count held something that is neither.
Kept memory read 676 to 691 GB, 120 to 135 GB over the offer, for four minutes, and 238 GB in the minutes before and after.
The machine was saturated at the time: 176 cores busy, CPU pressure 0.78 to 0.86, load over 1,000.
People's wide jobs were out: chromium-host-repeat at width 159 (09:47:57 to 10:01:46Z), tilo-sh-programs at width 65 with 36 GB booked (09:50:33 to 10:01:44Z), a cargo build --profile lounge-gate at width 42 and a test driver at width 34 (both 09:47Z to about 10:00Z). Starts came back in the minute the first of them ended.

So here the counts did more than read high: they closed the node to every start for six minutes.

How often: over the 24 hours to 10:04Z there were three stretches of three or more minutes with no job started while asks were queued, 12 minutes in all: 22:43Z 11-13 (3 minutes), 01:40Z 11-14 (5 minutes) and this one. Kept memory was over the offer in 10 minutes of that day. Only this stretch had both, and only this one had the busy count far above the node.

Data for the planner, read on production after the promote of 618f31ea (04:39Z 11-14), which carries d59240c7. Nothing here is a request to change admission by hand. ## Between waves the fix holds `t status`, 30 samples 20 s apart, 05:15 to 05:26Z, beside the node's own busy count from Prometheus: | | Before d59240c7 (25 readings, 11-13) | After (30 samples) | |---|---|---| | Over the offer of 159 by more than two cores | 13 | 11 | | Mean excess of those | 34 | 11 | | Largest booked | 256 | 186 | | Median booked | | 159 | | Mean booked / used by the server's count / busy by the node's count | | 164 / 157 / 156 | ## At the start of a wave booked still doubles Two readings of `/v1/views/jobs` fell on waves: **294/159 at 05:12Z and 326/159 at 05:27Z.** Minute by minute around the first (Prometheus; the operating system's figures are node-exporter's for the node): | Minute | Jobs started | Running | Node's busy cores | Unsampled | Awaiting | OS busy cores | CPU pressure | |---|---|---|---|---|---|---|---| | 05:10 | 25 | 49 | 243 | 27 | 10 | 178 | 0.40 | | 05:11 | 215 | 167 | 127 | 26 | 0 | 141 | 0.11 | | 05:12 | 507 | 421 | 146 | 24 | 0 | 169 | 0.23 | | 05:13 | 20 | 233 | 412 | 1 | 0 | 170 | 0.50 | | 05:14 | 73 | 137 | 137 | 0 | 12 | 170 | 0.19 | | 05:27 | 152 | 97 | 132 | 2 | 57 | 169 | 0.23 | | 05:28 | 12 | 112 | 185 | 3 | 68 | 181 | 0.49 | - 507 jobs started in the minute of 05:12Z. The server's booked cores read 294, and a minute later the node's own count read 412 busy cores, with 1 unsampled. - The machine did not follow either count: 169 and 170 cores busy by the operating system, CPU pressure 0.50 for that one minute against 0.2 to 0.3 around it. - The minute after each wave few jobs start (20 at 05:13Z, 12 at 05:28Z) while the operating system's busy cores stay at 170 to 181. - No job ended for memory or for a cause of t's own in these half hours. Kept memory stayed under the offer. ## Also seen at the takeover For about ten minutes after the server's start at 04:39Z: 337 jobs running at 04:44Z, booked memory 639 GB and then 649 GB against 554 GB offered, the node's busy count 411, load 640, CPU pressure 0.47 then 0.70. No job ended for memory. The sampler is `~/.cache/t-factory/booked-vs-busy.sh` on my side; its log holds each sample with the checks running and queued. ## On 67588367 (promoted 06:21Z), the same at 07:18Z - 369 jobs started in the minute of 07:18Z and 355 were running. - `/v1/views/jobs` read **474 cores used** on the node, with 159 of 159 booked. The node's own busy count read 238 (153 sampled), the operating system 178, CPU pressure 0.57. - Between waves in the minutes before: the node's busy count 166 to 224, the operating system 175 to 180. So the views' `used` overshoots at a wave's start as `booked` did, and by more. ## Six minutes with no job started, 09:54 to 09:59Z, on c6dca088 | Minute | Jobs started | Running | Node's busy cores | Sampled | Unsampled | OS busy cores | CPU pressure | Load | Kept memory | Offered | |---|---|---|---|---|---|---|---|---|---|---| | 09:53 | 73 | 96 | 143 | 136 | 7 | 174 | 0.32 | 386 | 238 GB | 560 GB | | 09:54 | 1 | 84 | 167 | 156 | 1 | 176 | 0.34 | 399 | 238 GB | 561 GB | | 09:55 | 1 | 73 | 160 | 155 | 0 | 178 | 0.33 | 329 | 238 GB | 558 GB | | 09:56 | 0 | 63 | 837 | 157 | 0 | 177 | 0.84 | 1,066 | 681 GB | 547 GB | | 09:57 | 0 | 54 | 549 | 158 | 0 | 176 | 0.86 | 1,262 | 691 GB | 558 GB | | 09:58 | 0 | 47 | 453 | 157 | 0 | 176 | 0.78 | 1,242 | 676 GB | 558 GB | | 09:59 | 0 | 42 | 127 | 125 | 0 | 166 | 0.47 | 948 | 681 GB | 553 GB | | 10:00 | 236 | 151 | 165 | 134 | 28 | 147 | 0.14 | 597 | 238 GB | 555 GB | - About 72,000 asks were queued throughout. No job ended for memory and none was lost. - The node's busy count read 837, 549 and 453 cores while its sampled cores stayed at 157 and its unsampled at 0. So the busy count held something that is neither. - Kept memory read 676 to 691 GB, 120 to 135 GB over the offer, for four minutes, and 238 GB in the minutes before and after. - The machine was saturated at the time: 176 cores busy, CPU pressure 0.78 to 0.86, load over 1,000. - People's wide jobs were out: `chromium-host-repeat` at width 159 (09:47:57 to 10:01:46Z), `tilo-sh-programs` at width 65 with 36 GB booked (09:50:33 to 10:01:44Z), a `cargo build --profile lounge-gate` at width 42 and a `test` driver at width 34 (both 09:47Z to about 10:00Z). Starts came back in the minute the first of them ended. So here the counts did more than read high: they closed the node to every start for six minutes. How often: over the 24 hours to 10:04Z there were three stretches of three or more minutes with no job started while asks were queued, 12 minutes in all: 22:43Z 11-13 (3 minutes), 01:40Z 11-14 (5 minutes) and this one. Kept memory was over the offer in 10 minutes of that day. Only this stretch had both, and only this one had the busy count far above the node.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

tilo/t#71

No description provided.

Rows
Columns

node/admit, sched: after d59240c7 booked cores hold at the offer between waves, but a wave of starts still reads 294 and 326 of 159 and the node counts 412 busy while the machine has 170 (05:08 to 05:30Z 11-14) #71