tilo/t

Watch

Fork 0

declared_builds takes 0.86 GB an hour since `f06bfdc3`: each row repeats its closure's units' reads, and the database volume has about a day and a half left #59

New issue

Open

opened 2026-11-13 13:50:58 +00:00 by xemmerling · 0 comments

xemmerling commented

2026-11-13 13:50:58 +00:00

Owner

Update 15:17Z 2026-11-13: 123aaf58 is on production since 15:14:52Z (my promote). It also carries 34ce569d and b96c6335.

Proof on testing first: the schema step (34 to 35) with the old rows intact; the forced whole check set 8 of 8 under the new server; a state with new keys left rows named by ids, each with of equal to its outputs and outside NULL, their units' reads reused across runs; and one build served by its declared key from such a row (test/t-cache asked again after a change only the test run reads: its tests ran in 4 s with no build job, t_declared_served_total{by="declared"} 0 to 1 on testing).

The restart on production
Schema steps to leading	under 0.1 s (15:14:46.50 to 15:14:46.58Z)
Lead shown	8.0 s
Runs driven again	424 in 16 s, no give-up or timeout
Worker	re-attached to 54 jobs in 7.1 s
Database log since the promote	no error, no lock wait
At 15:16Z	1 row named by ids, 233 units' reads kept, 92,203 rows still by value

Two properties of the new form to know. A build whose row the trust rule refuses (a person's build over a row main wrote) still leaves its units' reads, named by no row, until they age out after four days; they are a few KB each and shared by content. And what is left of a row is its id map, about 25 KB in hex; a bytea form would about halve it.

I read the table's growth, the declared counters and admission at 5, 15 and 60 minutes and put the hour's table here.

Update 14:52Z 2026-11-13: fixed on main as 123aaf58; not on production yet. jobs/declared: a unit's outside reads are kept once, and a declared build's row names them by id. It is the change described below with the rollback rule added: outside_ids is {"of": <outputs>, "ids": {unit: id}}; a reader takes outside when it is set (the row's last writer said it), else the ids only when of is the row's outputs, else the reads are unknown. A test replays the statement of the build before over new rows in the four cases (known or unknown reads, the same or other outputs): no row is read through ids left behind.

Forced whole check set on testing: 8 of 8 on the commit (run ra0aa006676, PostgreSQL 18), its nine new tests among them. Production-size test with the extra field: first head 7.7 times smaller, a second head 9.3 times, 8.4 after two.

Next from my side: main's head on testing (the schema step, the old rows still read, a forced set writing rows in the new form, a declared hit from one), then the promote, which also carries your 34ce569d and b96c6335. I read b96c6335 for the upgrade: no protocol, contract or job file changes. After it I read the table's growth, the declared counters against 13:23 to 13:53Z (served by declared key 1,184 per half hour), the database's memory against 4.9 GB, and admission against the half hour before. If you would rather promote b96c6335 yourselves first, say so here soon; the volume had 28.1 GB free at 14:43Z.

Update 14:36Z 2026-11-13: the fix is written and passes on testing; I am holding it for one more rule before it lands. A unit's reads are kept once in outside_reads under the hash of their JSON text and a row names them by unit (outside_ids, outside NULL for rows written from now on). The schema step only creates the table and adds the column. Rows written before stay readable until they age out. Kept, the upsert's rule and what a hit is held to are unchanged.

Production-size test (431 packages, 172 units a row, 74,012 units' reads, 1,200 distinct)	Held by value	Kept once
First head	102.3 MB	13.3 MB	7.7 times
A second head, every package re-keyed, a twentieth of the reads changed	102.3 MB	10.9 MB	9.4 times

What is left is mostly the rows' id maps (about 25 KB a row in hex); a bytea form would about halve that and is a later step. Largest statement 2.55 MB of reads and 1.51 MB of rows against the 4 MiB bound; a read of 32 keys named 5,501 units' reads and fetched 1,187. The forced whole check set passed 8 of 8 on testing (run r3489aa9296, PostgreSQL 18), the eight new tests among them.

Why I hold it: a rollback. The build before this one writes rows with its own statement, which does not know outside_ids and leaves it. After a roll-forward a row it rewrote with unknown reads and other outputs would be read through the ids of the build before, as known, and a declared hit could be served against another build's reads. So an id map will name the outputs it was written for, and a reader takes outside when it is there, else the ids only when their outputs are the row's, else unknown. That is being added with tests that replay the old statement over a new row; the amended commit gets its own check set. The volume had 28.0 GB free at 14:31Z.

Update 14:01Z 2026-11-13: the table's own size over a full hour, which corrects my rate. From 12:59:44 to 13:59:56Z declared_builds grew 882 MB (+88, +551, +243 MB in the three 20-minute steps), the database 979 MB, the volume's use 875 MB; reads did not grow and commit_keys took 16 MB. That is 0.86 GB an hour, not the 1.1 I wrote from 70 minutes of rows last written, and nine tenths of the volume's growth. With 28.0 GB free the room lasts about 33 hours at the hour's rate, 18 to 24 at the rate of its bursts. The fix is being written; nothing of it is on main yet.

What is happening, 2026-11-13 on t.tilo.so. Since f06bfdc3 went live (12:32Z) declared_builds takes about 1.1 GB an hour, three times yesterday's average, and the database's volume has about a day of room left. A full t-pg-1 stops production (it did on 11-02).

Reading (production, read-only)	Value
Volume (100 GB): used and free at 13:39Z	73.4 GB and 29.0 GB
Volume used, 06Z 11-12 to 13Z 11-13	64.7 to 76.4 GB, 0.38 GB an hour
`declared_builds` at 13:48Z	21.5 GB: 29 MB heap, 21,489 MB TOAST; 128,000 rows, 86,538 with `outside`
`declared_builds`, 12:59 to 13:19Z and 13:19 to 13:39Z	+88 MB and +551 MB
`outside` stored by rows last written, per ten minutes, 11:10 to 12:20Z	1 to 35 MB
The same from 12:40Z	254, 362, 42, 1.5, 175, 291, 164 MB (bursts with the heads' clippy builds)
A row's `outside` now and before	250 to 380 KB and 90 to 150 KB
Units in a row (200 rows)	173 on average, up to 458
`outside` stored by day of last write	11-11 6.8 GB, 11-12 8.8 GB

It is not bloat: autovacuum ran on the table at 13:39Z and on its TOAST table at 13:46Z, with no dead rows left.

Why. f06bfdc3 did what it was for: builds held to unit fell from 29% to 1.3%, so the clippy family's builds are now kept with what their units read, where their outside was NULL before. A row keeps the reads of every unit of its package's closure (closure, outside_of), by value. So a unit's reads are written again in the row of every package that builds on it, and again for every head that re-keys those packages. With three days kept, 1.1 GB an hour wants about 80 GB.

How much repeats (names and counts only read)	Unit values	Distinct	Text	Text of the distinct
40 rows written together (13:45Z)	7,835	608	56 MB	4.5 MB
40 rows spread over the table by key (11-10 to 11-13)	7,205	2,485	38 MB	21 MB

Proposal. Keep each unit's reads once. A table outside_reads (id BYTEA PRIMARY KEY, reads JSONB NOT NULL, at_ms BIGINT NOT NULL), id the hash of the unit's Outside as written; a row of declared_builds names its units' ids (a new column, outside left NULL for new rows) and is read by fetching the ids its chunk names once. Rows written before stay readable as they are until their three days pass. What a lookup is given (Kept) is the same value as today for the same build; a row naming an id that is gone is a row whose reads are unknown, which is never served. This also cuts what kept carries for a chunk of keys (up to 42 MB of text today), since units shared by the chunk's rows come once.

I am commissioning this now, with a test at production's size (stored bytes, a statement's size, and the same Kept back), because the room is about a day. Say here if you would rather have it another way; I build on declared.rs as it is and change nothing of what a declared hit is held to.

One hour on production (123aaf58, 15:14:52Z to 16:15Z 11-13): kept.

	the hour before	the hour after
`declared_builds`	+0.86 GB an hour	22,577 MB at 15:20, 15:40, 16:00 and 16:15Z: no growth
database volume used	77.5 to 78.6 GB	78.6 to 78.7 GB
rows written holding reads by value	every row	0 of 1,559 (1,522 by ids, 37 unknown)
`outside_reads`	none	4,127 unit values, 10 MB
database memory, CPU throttled	3.5 to 5 GB	4.5 GB (peak 4.6), 9% of periods
lock waits, database errors		0, 0
people's wait, median and p95	1.5 s, 4.0 s	1.3 s, 4.0 s
head runs finished	3	8

Builds held to outside in the hour: 1,962 (37 to unit, 1 undeclared). No check was told error. The one job that ended with a cause is #60's store_io, not this change. Still to read: the table after 12:30Z 11-14, when the rows that hold reads by value begin to pass their three days.

**Update 15:17Z 2026-11-13: 123aaf58 is on production since 15:14:52Z (my promote).** It also carries 34ce569d and b96c6335. Proof on testing first: the schema step (34 to 35) with the old rows intact; the forced whole check set 8 of 8 under the new server; a state with new keys left rows named by ids, each with `of` equal to its outputs and `outside` NULL, their units' reads reused across runs; and one build served by its declared key from such a row (test/t-cache asked again after a change only the test run reads: its tests ran in 4 s with no build job, `t_declared_served_total{by="declared"}` 0 to 1 on testing). | The restart on production | | |---|---| | Schema steps to leading | under 0.1 s (15:14:46.50 to 15:14:46.58Z) | | Lead shown | 8.0 s | | Runs driven again | 424 in 16 s, no give-up or timeout | | Worker | re-attached to 54 jobs in 7.1 s | | Database log since the promote | no error, no lock wait | | At 15:16Z | 1 row named by ids, 233 units' reads kept, 92,203 rows still by value | Two properties of the new form to know. A build whose row the trust rule refuses (a person's build over a row main wrote) still leaves its units' reads, named by no row, until they age out after four days; they are a few KB each and shared by content. And what is left of a row is its id map, about 25 KB in hex; a `bytea` form would about halve it. I read the table's growth, the declared counters and admission at 5, 15 and 60 minutes and put the hour's table here. **Update 14:52Z 2026-11-13: fixed on main as 123aaf58; not on production yet.** `jobs/declared: a unit's outside reads are kept once, and a declared build's row names them by id.` It is the change described below with the rollback rule added: `outside_ids` is `{"of": <outputs>, "ids": {unit: id}}`; a reader takes `outside` when it is set (the row's last writer said it), else the ids only when `of` is the row's outputs, else the reads are unknown. A test replays the statement of the build before over new rows in the four cases (known or unknown reads, the same or other outputs): no row is read through ids left behind. Forced whole check set on testing: 8 of 8 on the commit (run ra0aa006676, PostgreSQL 18), its nine new tests among them. Production-size test with the extra field: first head 7.7 times smaller, a second head 9.3 times, 8.4 after two. Next from my side: main's head on testing (the schema step, the old rows still read, a forced set writing rows in the new form, a declared hit from one), then the promote, which also carries your 34ce569d and b96c6335. I read b96c6335 for the upgrade: no protocol, contract or job file changes. After it I read the table's growth, the declared counters against 13:23 to 13:53Z (served by declared key 1,184 per half hour), the database's memory against 4.9 GB, and admission against the half hour before. If you would rather promote b96c6335 yourselves first, say so here soon; the volume had 28.1 GB free at 14:43Z. **Update 14:36Z 2026-11-13: the fix is written and passes on testing; I am holding it for one more rule before it lands.** A unit's reads are kept once in `outside_reads` under the hash of their JSON text and a row names them by unit (`outside_ids`, `outside` NULL for rows written from now on). The schema step only creates the table and adds the column. Rows written before stay readable until they age out. `Kept`, the upsert's rule and what a hit is held to are unchanged. | Production-size test (431 packages, 172 units a row, 74,012 units' reads, 1,200 distinct) | Held by value | Kept once | | |---|---|---|---| | First head | 102.3 MB | 13.3 MB | 7.7 times | | A second head, every package re-keyed, a twentieth of the reads changed | 102.3 MB | 10.9 MB | 9.4 times | What is left is mostly the rows' id maps (about 25 KB a row in hex); a `bytea` form would about halve that and is a later step. Largest statement 2.55 MB of reads and 1.51 MB of rows against the 4 MiB bound; a read of 32 keys named 5,501 units' reads and fetched 1,187. The forced whole check set passed 8 of 8 on testing (run r3489aa9296, PostgreSQL 18), the eight new tests among them. **Why I hold it:** a rollback. The build before this one writes rows with its own statement, which does not know `outside_ids` and leaves it. After a roll-forward a row it rewrote with unknown reads and other outputs would be read through the ids of the build before, as known, and a declared hit could be served against another build's reads. So an id map will name the outputs it was written for, and a reader takes `outside` when it is there, else the ids only when their outputs are the row's, else unknown. That is being added with tests that replay the old statement over a new row; the amended commit gets its own check set. The volume had 28.0 GB free at 14:31Z. **Update 14:01Z 2026-11-13: the table's own size over a full hour, which corrects my rate.** From 12:59:44 to 13:59:56Z `declared_builds` grew 882 MB (+88, +551, +243 MB in the three 20-minute steps), the database 979 MB, the volume's use 875 MB; `reads` did not grow and `commit_keys` took 16 MB. That is 0.86 GB an hour, not the 1.1 I wrote from 70 minutes of rows last written, and nine tenths of the volume's growth. With 28.0 GB free the room lasts about 33 hours at the hour's rate, 18 to 24 at the rate of its bursts. The fix is being written; nothing of it is on main yet. **What is happening, 2026-11-13 on t.tilo.so.** Since f06bfdc3 went live (12:32Z) `declared_builds` takes about 1.1 GB an hour, three times yesterday's average, and the database's volume has about a day of room left. A full t-pg-1 stops production (it did on 11-02). | Reading (production, read-only) | Value | |---|---| | Volume (100 GB): used and free at 13:39Z | 73.4 GB and 29.0 GB | | Volume used, 06Z 11-12 to 13Z 11-13 | 64.7 to 76.4 GB, 0.38 GB an hour | | `declared_builds` at 13:48Z | 21.5 GB: 29 MB heap, 21,489 MB TOAST; 128,000 rows, 86,538 with `outside` | | `declared_builds`, 12:59 to 13:19Z and 13:19 to 13:39Z | +88 MB and +551 MB | | `outside` stored by rows last written, per ten minutes, 11:10 to 12:20Z | 1 to 35 MB | | The same from 12:40Z | 254, 362, 42, 1.5, 175, 291, 164 MB (bursts with the heads' clippy builds) | | A row's `outside` now and before | 250 to 380 KB and 90 to 150 KB | | Units in a row (200 rows) | 173 on average, up to 458 | | `outside` stored by day of last write | 11-11 6.8 GB, 11-12 8.8 GB | It is not bloat: autovacuum ran on the table at 13:39Z and on its TOAST table at 13:46Z, with no dead rows left. **Why.** f06bfdc3 did what it was for: builds held to `unit` fell from 29% to 1.3%, so the clippy family's builds are now kept with what their units read, where their `outside` was NULL before. A row keeps the reads of every unit of its package's closure (`closure`, `outside_of`), by value. So a unit's reads are written again in the row of every package that builds on it, and again for every head that re-keys those packages. With three days kept, 1.1 GB an hour wants about 80 GB. | How much repeats (names and counts only read) | Unit values | Distinct | Text | Text of the distinct | |---|---|---|---|---| | 40 rows written together (13:45Z) | 7,835 | 608 | 56 MB | 4.5 MB | | 40 rows spread over the table by key (11-10 to 11-13) | 7,205 | 2,485 | 38 MB | 21 MB | **Proposal.** Keep each unit's reads once. A table `outside_reads (id BYTEA PRIMARY KEY, reads JSONB NOT NULL, at_ms BIGINT NOT NULL)`, `id` the hash of the unit's `Outside` as written; a row of `declared_builds` names its units' ids (a new column, `outside` left NULL for new rows) and is read by fetching the ids its chunk names once. Rows written before stay readable as they are until their three days pass. What a lookup is given (`Kept`) is the same value as today for the same build; a row naming an id that is gone is a row whose reads are unknown, which is never served. This also cuts what `kept` carries for a chunk of keys (up to 42 MB of text today), since units shared by the chunk's rows come once. I am commissioning this now, with a test at production's size (stored bytes, a statement's size, and the same `Kept` back), because the room is about a day. Say here if you would rather have it another way; I build on declared.rs as it is and change nothing of what a declared hit is held to. **One hour on production (123aaf58, 15:14:52Z to 16:15Z 11-13): kept.** | | the hour before | the hour after | |---|---|---| | `declared_builds` | +0.86 GB an hour | 22,577 MB at 15:20, 15:40, 16:00 and 16:15Z: no growth | | database volume used | 77.5 to 78.6 GB | 78.6 to 78.7 GB | | rows written holding reads by value | every row | 0 of 1,559 (1,522 by ids, 37 unknown) | | `outside_reads` | none | 4,127 unit values, 10 MB | | database memory, CPU throttled | 3.5 to 5 GB | 4.5 GB (peak 4.6), 9% of periods | | lock waits, database errors | | 0, 0 | | people's wait, median and p95 | 1.5 s, 4.0 s | 1.3 s, 4.0 s | | head runs finished | 3 | 8 | Builds held to `outside` in the hour: 1,962 (37 to `unit`, 1 undeclared). No check was told `error`. The one job that ended with a cause is #60's `store_io`, not this change. Still to read: the table after 12:30Z 11-14, when the rows that hold reads by value begin to pass their three days.

xemmerling changed title from ~~declared_builds takes 1.1 GB an hour since f06bfdc3: each row repeats its closure's units' reads, and the database volume has about a day left~~ to declared_builds takes 0.86 GB an hour since f06bfdc3: each row repeats its closure's units' reads, and the database volume has about a day and a half left

2026-11-13 14:47:39 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

tilo/t#59

No description provided.

Rows
Columns

declared_builds takes 0.86 GB an hour since f06bfdc3: each row repeats its closure's units' reads, and the database volume has about a day and a half left #59

declared_builds takes 0.86 GB an hour since `f06bfdc3`: each row repeats its closure's units' reads, and the database volume has about a day and a half left #59