claims: runs from 10-31 hold 155k claims with no step until they finish #48

New issue

Open

opened 2026-11-03 00:03:40 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-03 00:03:40 +00:00

Owner

Follow-up note (not urgent; don't fix now — coordinator's call).

On t-staging-aws at 2026-11-02 ~23:20 UTC, t.job_claims held 155,215 claims, every one held by a run id (none by a job):

659 holder ids; the largest are runs created on 10-31 (07:00–17:30 UTC), ~3,000–4,000 claims each, e.g. rd8f4356795 4,034, r0d0d8ad39c 3,999, r686fd1aa14 3,787 (the run of the stuck cloud-rystic-build job j309-351).
155,157 of them have no step (step IS NULL).

src/jobs/claims.rs says a run holds the keys the engine makes itself (checks' verdicts, builds and test runs) under its id until the run finishes, with no claim deadline. The oldest runs from 10-31 are still unfinished, so their claims have been held for two days, and any other run's job asking for one of those keys waits on a holder that makes no progress (only the node's 3-minute HOLDER_PATIENCE, ce63eb65, ends that for a node compile).

To look at later: whether a run's claims should be bounded by its progress (or by what its jobs are actually making) rather than held until it finishes; and whether job_claims rows of runs are being restored at every takeover.

Not the cause of the 11-02 cloud-rystic-build timeouts: that was the node's 180 GB replica being read after each worker start (compiles waited for it).

Follow-up note (not urgent; don't fix now — coordinator's call). On t-staging-aws at 2026-11-02 ~23:20 UTC, `t.job_claims` held **155,215 claims**, every one held by a **run** id (none by a job): - 659 holder ids; the largest are runs created on **10-31** (07:00–17:30 UTC), ~3,000–4,000 claims each, e.g. `rd8f4356795` 4,034, `r0d0d8ad39c` 3,999, `r686fd1aa14` 3,787 (the run of the stuck cloud-rystic-build job j309-351). - 155,157 of them have no step (`step IS NULL`). `src/jobs/claims.rs` says a run holds the keys the engine makes itself (checks' verdicts, builds and test runs) under its id until the run finishes, with no claim deadline. The oldest runs from 10-31 are still unfinished, so their claims have been held for two days, and any other run's job asking for one of those keys waits on a holder that makes no progress (only the node's 3-minute HOLDER_PATIENCE, ce63eb65, ends that for a node compile). To look at later: whether a run's claims should be bounded by its progress (or by what its jobs are actually making) rather than held until it finishes; and whether `job_claims` rows of runs are being restored at every takeover. Not the cause of the 11-02 cloud-rystic-build timeouts: that was the node's 180 GB replica being read after each worker start (compiles waited for it).

xemmerling referenced this issue

2026-11-14 03:17:11 +00:00

server: memory follows the open runs (about 74 MB a run, 3 GB an hour): the container is at its 64 GiB limit and anonymous memory reaches it in about eight hours #69

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

tilo/t#48

No description provided.

Rows
Columns