declared keys: two runs keeping the same builds deadlock on declared_builds, their batches lock the same rows in different orders (one in three hours, 2026-11-13 05:35Z) #55

New issue

Open

opened 2026-11-13 05:43:51 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-13 05:43:51 +00:00

Owner

Update, 2026-11-13 06:05Z. On main as 9c3a8fb4 ("declared keys: rows are kept in the order of their keys, so two runs keeping the same builds never deadlock"). jobs::declared::record orders its rows by key before it makes its batches; nothing else changes. Its test runs four recorders of the same 199 keys at once, two of them in reverse, for 20 rounds: without the ordering 44 to 48 of 80 deadlocked in each of three runs, with it none. The whole check set ran green on testing (run re2e4de487d). Not on production yet; after it is, deadlock detected in the database's log is the count to watch.

Seen on t.tilo.so on 2026-11-13 at 05:35:48Z (166e9649). One deadlock in the database's log since 02:42Z, and not the one #51 was about (result_latest, quiet since its fix went live).

What the database said

deadlock detected
Process 389414 waits for ShareLock on transaction 63941924; blocked by process 389393.
Process 389393 waits for ShareLock on transaction 63941913; blocked by process 389414.
Process 389414: INSERT INTO declared_builds (declared, outputs, outside, trust, entry, reads_id, at_ms)
  SELECT k, o, n::jsonb, t, e, r, $7 FROM unnest(...) AS u (k, o, n, t, e, r)
  ON CONFLICT (declared) DO UPDATE SET ...
context: while locking tuple (359,61) in relation "declared_builds"

The server logged it once, as a warning: t::drivers::cargo::declared: declared keys: keeping: keeping declared keys: db error: ERROR: deadlock detected.

Cause

jobs::declared::record (src/jobs/declared.rs) writes its rows a batch at a time, each batch one INSERT ... ON CONFLICT (declared) DO UPDATE over unnest of the rows in the order its caller gave them. Two runs keeping builds of the same packages give overlapping keys in different orders, so the two statements lock the same rows in opposite orders. It became likely once many heads build side by side and every row is keyed anew (2d9a8fdb): 25 test builds of 25 heads ran at once that night, each listing a median of 317 builds, most of them shared.

What it costs

The loser's statement fails, record returns at that batch, and that batch and the ones after it are not kept. No verdict is wrong and no check is told an error: a key not kept is a miss of its lookup later. One in three hours so far.

Fix (a developer of mine is on it)

Write the rows in the order of their keys. Two statements that lock overlapping rows in one order wait for each other and never deadlock. Nothing else changes.

**Update, 2026-11-13 06:05Z.** On main as 9c3a8fb4 ("declared keys: rows are kept in the order of their keys, so two runs keeping the same builds never deadlock"). `jobs::declared::record` orders its rows by key before it makes its batches; nothing else changes. Its test runs four recorders of the same 199 keys at once, two of them in reverse, for 20 rounds: without the ordering 44 to 48 of 80 deadlocked in each of three runs, with it none. The whole check set ran green on testing (run re2e4de487d). Not on production yet; after it is, `deadlock detected` in the database's log is the count to watch. --- Seen on t.tilo.so on 2026-11-13 at 05:35:48Z (166e9649). One deadlock in the database's log since 02:42Z, and not the one #51 was about (`result_latest`, quiet since its fix went live). **What the database said** ``` deadlock detected Process 389414 waits for ShareLock on transaction 63941924; blocked by process 389393. Process 389393 waits for ShareLock on transaction 63941913; blocked by process 389414. Process 389414: INSERT INTO declared_builds (declared, outputs, outside, trust, entry, reads_id, at_ms) SELECT k, o, n::jsonb, t, e, r, $7 FROM unnest(...) AS u (k, o, n, t, e, r) ON CONFLICT (declared) DO UPDATE SET ... context: while locking tuple (359,61) in relation "declared_builds" ``` The server logged it once, as a warning: `t::drivers::cargo::declared: declared keys: keeping: keeping declared keys: db error: ERROR: deadlock detected`. **Cause** `jobs::declared::record` (src/jobs/declared.rs) writes its rows a batch at a time, each batch one `INSERT ... ON CONFLICT (declared) DO UPDATE` over `unnest` of the rows in the order its caller gave them. Two runs keeping builds of the same packages give overlapping keys in different orders, so the two statements lock the same rows in opposite orders. It became likely once many heads build side by side and every row is keyed anew (2d9a8fdb): 25 test builds of 25 heads ran at once that night, each listing a median of 317 builds, most of them shared. **What it costs** The loser's statement fails, `record` returns at that batch, and that batch and the ones after it are not kept. No verdict is wrong and no check is told an error: a key not kept is a miss of its lookup later. One in three hours so far. **Fix (a developer of mine is on it)** Write the rows in the order of their keys. Two statements that lock overlapping rows in one order wait for each other and never deadlock. Nothing else changes.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

tilo/t#55

No description provided.

Rows
Columns