Jobs nothing measured get one jobserver token, run single-threaded, measure parallelism 1, and stay at one core forever while nodes idle #30

New issue

Open

opened 2026-10-29 21:38:49 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-10-29 21:38:49 +00:00

Owner

What happened in prod (t.tilo.so, 2026-10-29, 21:40 UTC)

14 jobs running, cores 20/175 booked, 12.6 used; ci-x86-1 cores 8/163 booked, 6.9 used. Twelve vm-guest-x86_64 jobs (a kernel build, make -j"$(nproc)") ran 16–25 min each in parallel. On ci-x86-1 every running job's cgroup had cpu.weight 100 (one core), cpu.max max, cpuset.cpus.effective 0-187 — no CPU cap — yet the node used ~7 cores. CI on main fell ~45 min behind.

Root cause

src/sched/book.rs, parallelism/tokens: a job's tokens are its key's measured parallelism (CPU-seconds over wall time), else its check's, else 1 (2271d3b0). After that deploy, and after the timing tables were reset (21:30), nothing is measured, so every job gets 1 token.
A job's tokens are its jobserver pipe (node::jobserver, MAKEFLAGS/CARGO_MAKEFLAGS): make and cargo start only as many processes as tokens, whatever -j/nproc says. So the job runs single-threaded on an idle 163-core node.
Its run then measures a parallelism of ~1, which is what it is booked at next time: a job held to one token can never learn it could use more. The loop is closed; unmeasured or once-capped keys stay at one core forever.

Impact

CI uses a few percent of the cores it has; long builds (kernel, release binary, tests) take many times longer than the hardware allows. The user's priority #2: "nothing waits while a single core is idle".

## What happened in prod (t.tilo.so, 2026-10-29, 21:40 UTC) 14 jobs running, `cores 20/175 booked, 12.6 used`; ci-x86-1 `cores 8/163 booked, 6.9 used`. Twelve `vm-guest-x86_64` jobs (a kernel build, `make -j"$(nproc)"`) ran 16–25 min each in parallel. On ci-x86-1 every running job's cgroup had `cpu.weight 100` (one core), `cpu.max max`, `cpuset.cpus.effective 0-187` — no CPU cap — yet the node used ~7 cores. CI on main fell ~45 min behind. ## Root cause - `src/sched/book.rs`, `parallelism`/`tokens`: a job's tokens are its key's measured parallelism (CPU-seconds over wall time), else its check's, else **1** (2271d3b0). After that deploy, and after the timing tables were reset (21:30), nothing is measured, so every job gets 1 token. - A job's tokens are its jobserver pipe (`node::jobserver`, MAKEFLAGS/CARGO_MAKEFLAGS): `make` and `cargo` start only as many processes as tokens, whatever `-j`/`nproc` says. So the job runs single-threaded on an idle 163-core node. - Its run then measures a parallelism of ~1, which is what it is booked at next time: a job held to one token can never learn it could use more. The loop is closed; unmeasured or once-capped keys stay at one core forever. ## Impact CI uses a few percent of the cores it has; long builds (kernel, release binary, tests) take many times longer than the hardware allows. The user's priority #2: "nothing waits while a single core is idle".

xemmerling referenced this issue from a commit

2026-10-29 22:17:41 +00:00

fix(sched): an unmeasured job gets cores to spare, and a job held to its tokens learns it needs more

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

tilo/t#30

No description provided.

Rows
Columns