t

t is a build and test runner with remote execution. A project declares its checks (tests, lints, builds, any shell command) and t runs them on a t server, either one on your own machine or one shared by a team. Each result is cached by the exact files it depends on, so unchanged checks return at once. Build directories are kept between runs, so cargo builds are incremental even when many people and agents work on different versions of the same code.

It works from the command line and as an MCP server, so coding agents use it the same way people do. On a server that follows the project's repository, failures are compared with the main branch, so you can tell whether you broke something or it was already broken.

t is developed for and used by Tilo, and is meant to be useful to any project.

What it does

• Runs checks and caches the results. A check's result is keyed by its command, its environment and the contents of the files it reads. Running it again on the same inputs returns the stored result, whoever ran it first, marked as a replay of the run it came from and how long ago that was. A compiler failure is stored as rustc's own diagnostics, so the excerpt a reader sees is rendered for that reader's width and colour, not frozen at one terminal's.
• Runs cargo remotely. t cargo, t build and t run send only the files that changed, run cargo on the server in a build directory that already holds a build of a nearby version, stream cargo's output back and exit like cargo. Programs that were built are copied into your ./target, built on a node of your machine's architecture and OS, so they run where they land.
• Runs what you name. t test <selection> runs exactly the checks, families, packages or tests named; t test alone runs what failed for you last time and what your change touches (Testing a change). Main stays ungated: you push to it as usual, and main's CI informs.
• Tells your failures from existing ones. When the server follows the repository ([[repo]] in its config), every failure is checked against the main branch at your base commit. The exit code is 3 when everything that failed also fails on main. Test suites are compared test by test (nextest and libtest output are understood): a new failing test in a package that already fails on main is still reported as possibly yours.
• Keeps a board of what fails on main. t main lists the failing tests and checks of the repository you are in (the server's repository one of its remotes points to), grouped by the commit they started failing on, with who has claimed each one and the t log … command that shows the failing run's log from any session. Packages that one failed build kept from being tested are one item for their family (test@1a2b3c4d5e), and a failure's excerpt is what explains it: the compiler's errors with their spans, a panic, or the last lines a passing run doesn't print too. A run that fails because the machine ran out of disk space is an error, not a failure of the code.
• Tells whoever broke main. When a new failure on main is narrowed down to one commit, the session behind it hears about it on its next t command, in t inbox, or through an agent hook (Hearing about breakages).
• Spots flaky tests. A test that passed and failed on the same inputs is flaky: its failures are marked as such and never blamed on a change, and t never retries it silently. t flaky lists them, with the runs and logs they failed in. A run that failed most of a check's tests at once, or a test check that failed in no test (its build, a timeout), is no flake of anything.
• Fixes what a command can fix. A check can declare a fix such as cargo fmt. t fix applies it to your files, and the server can apply it on main by itself: when such a check fails there, t commits the fix and pushes it (docs/fixes.md).
• Tracks sessions. Each agent or shell is a session with a name, a permission preset and a history of its runs, failures and claims.
• Scales out. Other machines join as workers. Each job runs on the node with the most useful build directory for it, then the least loaded one.

Quick start on your own machine

You need Linux or macOS and a Rust toolchain. On Linux every check runs in Tilo's job sandbox, which t has built in, and checks with a container image are unpacked by t itself from their registry. A Linux node runs as root and starts only where a trial job in the sandbox runs: there is no weaker sandbox to fall back on. On macOS, which has none, checks run directly on the machine.

cargo install --locked --git https://code.tilo.so/tilo/t
t serve            # in one terminal: a server for this machine, data in ~/.local/share/t
cd your-project
t                  # in another: run the default checks on this directory

The client talks to http://127.0.0.1:7777 unless told otherwise, and a server without tokens in its config lets every local caller do everything. A cargo workspace with no t.toml gets one test/<package> check per package.

Commands

t --help lists them by what they are for; t <command> --help has examples.

# Everyday
t                    # what failed for you last time and what your change touches, uncommitted
                     # changes included (the same as `t test`)
t test all           # every default check (cached ones return at once)
t test clippy        # a family, a package, a check id, a glob, or `all`
t test test/mylib --test parser::tests::empty   # one test, a module of one, or a test binary
                     # (`mylib::parser`), never cached
t cargo clippy -p mylib --tests   # any compiling cargo subcommand, output streamed
t build -p myapp     # build on the server; programs land in ./target
t run -p myapp -- --help          # build on a node like this machine, run here
t log test/mylib     # that check's log from your latest run here that has it: its failing
                     # tests first, without cargo's and nextest's JSON (--verbose keeps it)
t log 9f2c41d0a7     # a log by the start of its hash, from any run
t log --run r1a2b3c4d5 test/mylib   # that check's log from one run (`t session` lists them)
t log test/accounts  # not in your runs: the log of its failure on main, as `t main` shows it
t log test/mylib --test parser::empty   # what one test printed (or `t log test/mylib#parser::empty`)
t log -f test/mylib  # still running (or not started yet): what it printed so far, then what it
                     # prints until it ends; without -f, only what it printed so far
t failed             # run again what failed for you last time (`t test` also runs what you touch)
t suggest            # what to run next: what failed for you, then checks with no result yet,
                     # then the opt-in checks your change touches (their key differs from your
                     # base commit's), each with the command that runs it; --all lists the rest
t scope              # what your change touches: the checks to name (--names for scripts)
t fix                # apply the `fix` of failing checks (e.g. `cargo fmt`) to your files
t test --rev HEAD fmt          # a commit's files as committed: no uncommitted or untracked files
t fix --rev <commit> --patch   # the fix of a commit as a patch on stdout (--patch=FILE for a file)
t sync               # fetch main and rebase this branch onto the newest head of it t has built

# The main board
t main               # what fails on main, test by test, since when, who is on it
t main --repo <name> # another repository's main; --all for every one the server follows
t main --coverage    # main's last 20 commits: what each passes, fails, runs, or has no result for
                     # (read-only; `stable` is the stable branch's tip, `qualified` the newest
                     # commit with every check passing)
t claim <item>       # tell others you are fixing it; `t release <item>` to undo
t claims             # who holds which claim, and which claimed items pass again
t inbox              # main failures traced to your commits, until claimed or fixed
t send <who> '<text>' # message another session (id, prefix or name), pushed into its agent
t install claude|pi  # set this machine's agents up to take messages and notices as they come
t flaky              # this repository's tests that passed and failed on the same inputs lately (--repo)

# Activity
t jobs               # every running and queued check (`t ps`); --mine, --watch
t status             # the server at a glance: nodes, cores and memory, check counts,
                     # and each repository's CI: last fetch, what it does, its runs under way
t disk               # each node's disk: free, being freed, booked by jobs
t session            # this session: its runs, failures, claims and compute
t session <id|name>  # another one; `t session name <name>` names this one
t sessions           # sessions active lately
t limit <who> --cpus 32 --memory 64G   # cap a session (or --principal) at once; admins
t show <run>         # a run by its id: results, followed to its end if it still runs (`t follow`)
t follow             # this directory's latest run (the running one first), e.g. after its client
                     # was stopped: stopping a client never stops its run (`t attach` too)
t cancel <run>       # stop a run
t restart <run> <check>  # stop a check's running job and run it again in its place in the run

# Publishing
t outputs image/app  # download the files a check produced (its `outputs`)
t log publish/app    # the log of its latest publication; `t main` lists what each one published
t dispatch publish/app --ref stable --input apps=browser --publish   # start a dispatchable
                     # check; one that publishes needs --publish

# Server
t serve | t worker | t token new | t mcp | t completions <shell>

t jobs shows how long each job has run (or waited), its node (or place in the queue), the cores and memory it is booked at (with what it uses now before them, 2.5/4, where the node measures it), priority, who runs it (the session's name, else a short id like claude:c441a605; the server's own CI of a followed repository under the session ci:<repo>, also in t session ci:<repo>), what it runs and its run id. Queued jobs say what they wait for.

Exit codes: 0 passed, 1 something failed that may be yours, 3 no failure is yours (each fails on main too, or is a flaky test), 2 t could not do what was asked (no server, a bad token, a check that doesn't exist) or a check could not run. They are the same in text and JSON.

A check that could not run says nothing about the code, and t tells it from a failure: its node went away, it ran out of memory again after a retry with more, the machine ran out of disk space, the sandbox or a tool it needs could not start, cargo could not download its dependencies, or something besides the build changed its build directory (an archiver, linker or compiler driver missing an object the build had just made, as when another check's compiler server compiled it). Such a result is shown as could not run, exits 2 (unless something else may be yours), and is never cached: running it again runs it. t runs it again once by itself first, on another node where one fits (with more memory after running out of it). A failure stored before t told such a log apart is not served either: t drops it when it reads it.

When main was not checked at your base yet, a failure is compared with main's latest result of the same check, test by test: a test failing there too is not yours.

Completion

t completes its commands and options, and the checks, families and packages of the project you are in. Add one line to your shell's startup file:

source <(t completions bash)     # ~/.bashrc
source <(t completions zsh)      # ~/.zshrc
t completions fish | source      # ~/.config/fish/config.fish

For agents

Agents use t as people do, from the shell or through t mcp. The loop:

t test clippy mycrate              # what your change needs, uncommitted changes included
t log test/core                    # a failure's full log (each failure prints its own `t log …`)
t test test/core --test parser::empty   # one failing test again, never cached
t                                  # after the fix: what failed and what you touched, again; what
                                   # didn't change is cached
git commit -am '…' && git push    # main is ungated: its CI runs what the push affects

• t test is narrow on purpose. An edit to a core crate reaches most of the tree, and checking all of it on every iteration is what drowns a shared server. It says what it covered and how far the change reaches ("your change reaches 140 more, because ui-core changed and 140 packages build on it (not run, 120 not cached, ~8m)"); t test all or named checks widen it when you want that.

• Whose failure is it. Every failure carries one label, so a developer reads at a glance which are theirs: [yours] (caused by the change: it passes on your base), [flaky] (the same inputs passed before), [inherited] (main fails it too, with who claimed it), or [caused by t] (a check that could not run: a node or its machine, never the code's, never cached). A failure the base hasn't been compared with yet says [not compared yet]: take it for yours, and the next run compares again. The summary lists the change's failures first, whole, then names the rest with their labels and their t log and counts them by label. Exit codes say the same: 0 passed, 1 something failed that may be yours, 3 no failure is yours (main fails it too, or it is flaky), 2 t could not do what was asked (no server, a bad token, a check that doesn't exist) or a check could not run (could not run: t's fault, a node or its machine, not the code's; never cached, and t status shows the node's problem).

• Compile remotely. t cargo check -p x, t build, t run instead of cargo: the warm build directories are on the server, the programs land in ./target. A job builds in its own build directory, so a --target-dir of your own is refused before any job starts, as your mistake: leave it out, and the programs come back under the checkout's own target. So is a compiler wrapper of your own (--config build.rustc-wrapper=… or build.rustc-workspace-wrapper=…): every compile runs through t's own wrapper, its compiler cache.

• Fixing main. t main lists what fails on main by item id; t claim <item> before working on one, push the fix, and the item leaves the board when main passes it. t inbox says when one of your commits broke main (Hearing about breakages).

• Runs by id. Every run has an id (r5d1c9e0a2b). t show <run> prints its results, waiting for it if it still runs (t follow is the same); t log --run <run> <check> a log of it, or of a check still running what it printed so far (-f follows it to its end, so a stuck job shows where it stopped); t cancel <run> stops it. t restart <run> <check> stops one check's running job and runs it again in its place, the run's other checks going on (a job booked under older rules, such as a VM step that booted on fewer vCPUs than it asked for, comes back booked by today's); the check's page has a Restart button for the same. t test --detach starts one without waiting.

• Ctrl-C cancels; any other stop leaves the run going. Ctrl-C at a terminal cancels the run t started: its jobs stop, except those another run follows (the same inputs, deduplicated), which go on for that run, and t says how many of each. A second Ctrl-C exits at once. A t stopped any other way (SIGTERM or SIGINT from a program such as an agent's tool timeout, a closed terminal, a lost connection, a reboot) only stops following, so no half-finished work is thrown away: it says which run goes on, and t follow (or t attach) with no id, in the same directory, follows that directory's latest run again. Ctrl-C in t follow only stops following too. A client also follows its run across a server restart by itself.

• Long waits say why. A job that waits for a node says so within seconds, with what holds it back and the numbers that decide it (t: waiting 3.0s for 8 cores, 16.0 GiB of memory on a node, next to start: disk on t-worker-x86-1: 42.0 GiB short), again when that changes or a minute on, and once it starts, how long it waited and where it runs. A cargo job says how many of its crates it has left as it builds (t: building on t-worker-x86-2: 120 of 480 crates left), and as it ends where its time went (t: time: waited 2m14s for a node, prepared 14s on t-worker-x86-2 (fetch 10s), ran 3m10s; 480 crates: 250 from the cache, 30 compiled, 200 fresh; that last line is shown with t cargo and --verbose). A run quiet for ten seconds prints what all its checks wait for or how long they have run, every thirty seconds.

• Name the session after its task (t session name fix-flaky-fs) so others can tell it apart in t jobs, t sessions and on the board.

Four workflows

A feature with failing tests. Write the tests first: t runs the checks of the packages you touched, and the new tests fail as new. Iterate with t test <check> --test <name> and t log; t again runs what failed for you last time, then only what you touch, fastest first. When it is green, commit and push to main.

A big refactor. An edit to a core crate reaches most of the tree. t still checks only the crate you touched, and says how far the change reaches ("your change reaches 140 more, because ui-core changed and 140 packages build on it (not run, 120 not cached, ~8m)"). To check the breadth, name it: t test test (a family) or t test all. A failing check that declares a fix (cargo fmt, clippy --fix) says <check>: fixable, t fix applies it.

Fixing main. t main lists what fails on main, test by test, with who claimed it. t claim <item>, reproduce with t test <check> --test <name> (it fails as not yours: fails on main too), fix it and push: the item leaves the board, and its claim with it, once main's run of the fix passes it; t release <item> lets go of it before. t inbox says when one of your own commits broke main.

Stabilising. The stabiliser (a session with the integrator preset) makes the stable branch: it picks candidates from main, carries exclusions and fixes as commits on stable, and moves it; t never does. t stable shows how far stable lags main (main's commits since their merge-base that stable doesn't carry), what each of them passed and failed, and the runs the stabiliser asked for: t stable verify <commit> runs a candidate's checks main has no verdict for (or those named) as a trusted run. t main --coverage shows what main's newest commits pass and fail; stable's head runs the checks declared on = ["stable"], and t dispatch runs a dispatchable check (a release) on it.

Output is text for everyone, agents too: it carries the same facts as JSON in a fraction of the tokens, and every failure ends with the commands that act on it (repro, log, fix). --json (or T_OUTPUT=json) is for scripts, --human (or T_OUTPUT=human) forces text. Colour only on a terminal (--color auto|always|never, off under NO_COLOR).

t mcp serves the commands as tools (test, run_test, failed, show, run_status, why, failures, cancel, cargo, fix, suggest, log, board, claim, release, inbox, send, flaky, session, sessions, status, jobs, disk, dispatch, outputs, limit, stable); each answers with the text in content and the JSON below in structuredContent. Only the server's own commands (serve, worker, token) are not tools. A run that would keep an agent waiting can be started with test and detach, and asked about without waiting: run_status gives its counts, its failures so far and each unfinished check with where it stands (running where, for how long, with how many crates left; or waiting for a node, how long and why); why the whole reason one check waits; failures its failures with their excerpts and repro commands, no logs. A call whose agent sends a progressToken is told, as MCP progress notifications, what its jobs wait for and which checks finished.

The log tool of a check still running answers with its last lines so far, running: true, its run and next; asked again with offset: next (and that run), it answers with what the check printed since, until running is false and the answer is the finished log.

An agent that connects out

t mcp runs on the agent's own machine, because its tools run commands there. An agent that needs nothing but what the server knows connects the other way: the server serves /mcp over MCP's streamable HTTP transport (revision 2025-06-18), so nothing listens on the agent's machine. It authenticates as any client does, with its t token as a bearer, and names the t session it acts as with x-t-session (a subagent adds x-t-parent-session) or, in initialize, the session and parent params; the session is recorded like any other, under the token's person. It offers the tools whose work is the server's: inbox (what t told this session, and the ids to ack as read), send (a message to another of the person's sessions) and sessions (the person's sessions active lately, with what each runs).

Each client names a session of its own: two clients naming one id are one session, and t's delivery to it reaches only one of them.

claude mcp add --transport http t https://t.example/mcp \
  --header "Authorization: Bearer $T_TOKEN" \
  --header "x-t-session: claude:my-agent" \
  --header "x-t-session-name: my-agent"

Signing in by OAuth

A person's local agent can sign in through the provider the pages use instead of holding a t token. t is the protected resource (MCP 2025-06-18, "Authorization") and the provider is the authorization server: a login with mcp_audience set is the one that serves MCP clients, and the audience their access tokens must name. Without it, /mcp takes t's tokens alone.

On the provider's side, make a public OAuth2/OpenID client for MCP clients (Authentik: Applications → Create with Provider → OAuth2/OpenID Provider, client type Public, no client secret), and let it use PKCE. Its redirect URIs are the local callbacks MCP clients listen on: http://localhost:* and http://127.0.0.1:* (Claude Code takes a free port on each sign-in), and the client's own scheme where it has one. Give it the scopes the login's scopes name, which must bring the user and groups claims. Its client id is the login's mcp_audience:

[[server.logins]]
issuer = "https://auth.example/application/o/t/"
client_id = "t-web"          # the pages' client
mcp_audience = "t-mcp"       # the MCP clients' public client: its client id
# …

A provider that cannot register clients dynamically needs the client's id given to the client: Claude Code takes it as the server's OAuth client id, with PKCE:

claude mcp add --transport http t https://t.example/mcp \
  --header "x-t-session: claude:my-agent" \
  --header "x-t-session-name: my-agent"

and, in ~/.claude.json (or the project's .mcp.json), under that server, "oauth": {"clientId": "t-mcp"}. On first use the client reads https://t.example/.well-known/oauth-protected-resource/mcp, signs in at the provider, and sends the access token as its bearer; t checks it against the issuer's published keys, that it is for t-mcp and not expired, and maps the person's groups to a preset exactly as it does on the pages. The person is then the same principal, under the same preset, in both.

While the client's event stream (a GET /mcp) is open, the session's messages and breakage notices are pushed on it as notifications/message. A pushed notice is not a read one: it stays in t inbox and t's hooks until the agent acknowledges it with inbox, and a client watching does not take the session's delivery from them.

JSON

One-shot commands (status, jobs, main, session, sessions, suggest, scope, log, claim, release, cancel, inbox, flaky, disk) print one JSON document; suggest, sessions, inbox and flaky wrap their lists as {"checks": [...]}, {"sessions": [...]}, {"notices": [...]} and {"flaky": [...]}, acknowledgements are {"ok": true, "action": "claim", "target": "..."}. Commands that stream (test, failed, show, dispatch, cargo, build, run, jobs --watch, log --follow) print one object per line, each with a type: planned, started, output (started and output of checks only with --verbose), result, artifact, exec, jobs, job (where a check's job stands, for everyone, --verbose or not: state is {"kind": "waiting", "waited_ms", "reason", "need", "ahead"}, {"kind": "placed", "waited_ms", "node"}, {"kind": "building", "node", "units": {"total", "left", "running", "fresh", "cached", "compiled"}} or {"kind": "took", "node", "waited_ms", "prepared_ms", "steps", "command_ms", "units"}, and line says it in words), waiting (every thirty seconds while a run is quiet: its running jobs (with units while they build), its queued ones with how long and what each waits for, and in waiting its checks without a job yet, each with its stage: needs (checks it waits for) or shared (another run's job of the same inputs, whose result it takes)), and finally done with the exit code. When t cannot do what was asked it prints {"type": "error", "message": "...", "hint": "..."} and exits with 2.

t log --json of a check still running is {"check", "run", "running": true, "log", "next"} with what it printed so far; t log --follow --json streams its output lines, then its result and done.

t test --json with a failure:

{"type":"planned","run":"r5d1c9e0a2b","follow":"t follow r5d1c9e0a2b","checks":["test/core","clippy/core"],"cached":1,"estimate_ms":42000,"queued_ahead":0}
{"type":"result","check":"clippy/core","outcome":"pass","cached":true,"duration_ms":0,"ran_by":"pi:01a138ab-…"}
{"type":"result","check":"test/core","outcome":"fail","cached":false,"duration_ms":1200,"blame":{"kind":"new"},"tests":{"passed":41,"failed":2,"ignored":0},"failures":[{"test":"parser::empty","excerpt":"assertion failed","repro":"t test test/core --test 'parser::empty'","blame":{"kind":"new"}},{"test":"lexer::eof","excerpt":"assertion failed","repro":"t test test/core --test 'lexer::eof'","blame":{"kind":"inherited","claimed_by":"pi:01a138ab-…"}}],"log":"9f2c…","log_command":"t log 9f2c41d0a7"}
{"type":"done","exit_code":1,"run":"r5d1c9e0a2b","passed":1,"failed":1,"errors":0,"cached":1,"duration_ms":1300,"only_failures_main_has_too":false}

blame is new, inherited (with claimed_by), flaky or unknown. For a test suite each failing test has its own blame, by that test's result on your base, and the check's is new if any test's is. tests counts the suite's tests, when its output names them. queued_ahead counts the queued jobs of its priority or higher competing for the same nodes, not held back by a limit; estimate_ms is how long the run's checks typically take once they run, not counting time in the queue. In a plain t, unaffected says how many of the default checks the change can't affect were left to main's runs of the commit it branched from (base): their keys are the ones they have there. Their cached results still come back.

t scope --json (--rev adds the commit as rev; estimate_ms is the slowest check's time once running; reaches counts the default checks more whose keys differ from the base's):

{"rev":"4e1d…","state":"b7c2…","base":"9a0f…","checks":["fmt","clippy/core","test/core"],"estimate_ms":90000,"reaches":140}

t fix --rev <commit> --patch=FILE --json ends with {"type":"fixed","changed":[...],"skipped":[...],"patch":"FILE","exit_code":0}; with the patch on stdout, that line goes to stderr.

t cargo (build, run) prints cargo's own output as cargo would, on stderr, also with --json: grep error: works on it as on cargo's. JSON goes to stdout: a started line with the run's id once the server names it (t follow <run> picks a stopped command up again; in text, a t: run … line on stderr when that isn't a terminal), job lines on where its job stands (as above), and the summary at the end. t cargo build -p demo --json:

{"type":"started","check":"cargo build -p demo","run":"r5d1c9e0a2b"}
{"type":"result","check":"cargo build -p demo","outcome":"pass","cached":false,"duration_ms":4200}
{"type":"artifact","path":"target/debug/demo","size":1024}
{"type":"done","exit_code":0,"passed":1,"failed":0,"errors":0,"cached":0,"duration_ms":4200,"only_failures_main_has_too":false}

When the server runs a newer t than the client, t says so on stderr, at most once a day. The client protocol changes rarely: a server that only adds to it (a route, an optional field) serves older clients as ever, and changes between nodes and the server never touch it. When the server speaks another client protocol, a command puts the server's build where t is installed and goes on in it (one another session already put there is run as it is). t self-update replaces the client with the server's own build, downloaded from the server (it checks the download runs here first): the server's program on a machine of its OS and architecture, and on Linux machines of the image's other architecture (an aarch64 laptop of an x86_64 server) the build of the same commit that t's image carries for it (/usr/local/lib/t/clients/linux-<arch>/t, deploy/oci.py). For others it prints the cargo install command that builds the server's version. A client that can't reach its server says so within seconds (five for one on this machine, a minute for one elsewhere); once it has reached it, it waits out a restart.

Hearing about breakages

When the server follows the repository, every new head of main runs its checks, and a new failure is narrowed down to the commit that introduced it, test by test. t then tells whoever pushed that commit onto main, once, so the failure is fixed by the one who knows the change. Who pushed is recorded where t learns of the push:

1. a forge's push webhook names its pusher: every session of the principal whose authors list the pusher's email hears of it;
2. a session that tells t of its own push (POST /v1/repos/<name>/push with its token and session, from a git hook or an agent) is the pusher itself.

Failures that are flaky, already claimed, or being fixed by t itself (docs/fixes.md) are not announced, nor commits no push named: those already on main when t began following it. The notice names the commit, the failing tests, the command that reproduces them and how to claim them:

t: the commit you pushed, cb6dfe6e65 (Break a), broke unit on main of myproject: a fails; `t test --rev cb6dfe6e65 unit --test a` reproduces it, `t claim unit#a@cb6dfe6e65` if you fix it, `t main --repo myproject` has the rest

It arrives on stderr with the session's next t command, in structuredContent.notices of its next MCP answer, and in the agent's hooks, and stays in t inbox until one of its items is claimed or passes on main again. A subagent has usually ended by the time main fails, so the session that started it hears its notices too. A session that ended never asks again: its open breakage notices stay in the t inbox of every session of the same principal, marked with the session they were for, until claimed or fixed; they don't reach the hooks of those other sessions, which would each be told what every other one broke.

t inbox --hook prints only what the session itself hears and wasn't shown yet, prints nothing when there is nothing, and exits 0 even when the server cannot be reached within two seconds, so an agent can call it on every turn.

Claude Code, in ~/.claude/settings.json (t install claude puts them there, and keeps them current): with --hook=claude, t reads the hook's input for the session and answers with additionalContext, which reaches the model. When the session starts, before each prompt and after each shell command (where pushes happen; after a command only in a project with a t.toml, so that elsewhere no command waits for the server). When the session starts it also gives the session's commands its id (below, Sessions). A subagent's hooks show nothing: its commands are its parent's session's, whose own hooks show what it hears.

{
  "hooks": {
    "SessionStart": [{ "hooks": [{ "type": "command", "command": "t inbox --hook=claude", "timeout": 10 }] }],
    "UserPromptSubmit": [{ "hooks": [{ "type": "command", "command": "t inbox --hook=claude", "timeout": 10 }] }],
    "PostToolUse": [{ "matcher": "Bash", "hooks": [{ "type": "command", "command": "t inbox --hook=claude", "timeout": 10 }] }]
  }
}

t hook claude, which older t put among the hooks, does nothing now: it says to remove it, where Claude Code shows its hooks' errors, without holding commands up (t install claude removes it).

pi, as an extension in ~/.pi/agent/extensions/t-inbox.ts: before each prompt and after each tool call, the session's new notices are added for the model.

import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";

export default function (pi: ExtensionAPI) {
	const inbox = async (ctx: ExtensionContext) => {
		const session = `PI_SESSION_ID=${ctx.sessionManager.getSessionId()}`;
		const r = await pi.exec("env", [session, "t", "inbox", "--hook"], { timeout: 10_000 });
		return r.code === 0 ? r.stdout.trim() : "";
	};
	pi.on("before_agent_start", async (_event, ctx) => {
		const text = await inbox(ctx);
		return text ? { message: { customType: "t-inbox", content: text, display: true } } : undefined;
	});
	pi.on("tool_result", async (event, ctx) => {
		const text = await inbox(ctx);
		if (!text) return undefined;
		return { content: [...event.content, { type: "text", text }], structuredContent: event.structuredContent };
	});
}

Other agents can run t inbox --hook (plain lines) wherever they take extra context.

Messages between sessions, pushed

Sessions message each other through t: t send <session> '<text>' (or the MCP tool send), with the session named as t names it: by its name (t session name), <name>@<person> where several sessions share a name, its id, or 8 or more characters of the id after its prefix, among the sessions active within the last day (t sessions lists them, and t session <who> finds one as t send does). A name several sessions have is refused, with each of them listed to pick from. A message is a turn in the receiving agent, so it is sent as a session does anything else in t (run) to the sender's own person's sessions, and to another person's only with manage_sessions; a subagent takes none (it ends with its task: message its parent). t pushes the message, and t's breakage notices for that session, into the agent's conversation the moment they are left: an idle agent starts a turn on one, a busy one reads it when it next can. t's other notices (main passes again, t fixed a commit) are worth no turn of their own, and come on the pull paths. A message reads as one line saying whose it is, session <name> (<person>), then the text, where a line of the text that would read as such a line (or as t's own, t: ) is indented; a session's name and id can't hold parentheses or @. One session sends another at most 5 messages a minute and 40 an hour: enough for a long exchange over a task, and two agents answering each other's every message stop within minutes.

t send deploy-fix 'I took the scheduler tests, leave them to me'
sent to claude:7da3557a-… (quinten): its delivery is connected, and hands it to the agent as soon as the agent takes it

Set the agents on a machine up once; each recipe ends by reaching the server once and saying as which session and person, or why not:

• t install claude adds t mcp to Claude Code for the user (claude mcp add -s user t -- <this t> mcp) and puts t's hooks (above) into the user's settings.json, keeping a dated copy of each file it changes, and every other key in its order and value as written. Claude Code starts t mcp as a child of every session, with the session's inbox socket in its environment; on Linux, t writes each message there, as Claude Code's own messages between sessions arrive. /clear starts a new session in the same process: the moment Claude Code's record of the process names it, the new session takes the old one's place at the server, proving with a token only the old one's delivery was given that it holds that delivery: the old session's name (unless the new one has one already) and everything still on its way to it move to the new one (so t send <name> reaches the conversation that goes on), and delivery moves with them; the server logs each succession, and each one it refuses. Claude Code says nothing back on that socket, not even when it refuses what was written (it only closes it, as it does once it read every line), so a message counts as delivered once it closed the socket: push into Claude Code is at most once, and a message Claude Code drops after reading it is not written again. One it doesn't close within 5 seconds counts as not taken: writing it again would only wait as long again, so push into that session ends there, and its messages and notices (that one included) wait in t inbox and t's hooks. A newer delivery for the session taking over (another process delivering to it) ends push too, and t says so in the conversation, once. t connects a session's delivery only once it can deliver (Claude Code's record of the process names this socket and session) and not where Claude Code would hold or drop it (crossSessionInbound other than accept in any settings file, or a socket that isn't the session's own); those sessions read their messages in t inbox and t's hooks. None of this is an interface Claude Code documents: its records of its processes, its inbox socket and the variables it gives its MCP servers are what its own sessions use between themselves, and may change with any release. t session says whether a delivery is connected, and if not, why, as far as it can tell (no t mcp, a setting that holds messages, Claude Code's records not as t reads them), and scripts/live-messaging.sh checks the whole way through a real Claude Code, failing at the first step that changed. On other systems than Linux, t pushes nothing into Claude Code: its sessions read their messages in t inbox and t's hooks, and t session says so.
• t install pi writes ~/.pi/agent/extensions/t-messages.ts, which runs t inbox follow --json (this t, by its path) for each session, hands what arrives to pi (pi.sendMessage with triggerTurn) and writes each notice's id back once pi has it. A line it can't hand to pi ends that delivery (t inbox follow waits for each line's acknowledgement). It starts the delivery again when it ends, unless it failed at once (no t there, one that speaks another protocol than the server, or a line it can't read) or a newer delivery for the session took over, which it says instead.

Anything else can run t inbox follow: it holds one connection open and prints each notice as it arrives. As text it counts each delivered once printed; with --json only once its reader writes {"ack": <id>} back on its stdin. A session has one delivery at a time: a newer one takes over, and the older one is told and closed (t inbox follow then exits 3); t session says whether one is connected. Notices t leaves for every session of a person (main broken by a commit no session of theirs is recorded for) are pushed into no one agent of theirs: they ride on answers and show in each session's hooks and t inbox, delivery connected or not. A session's messages and breakage notices go to its delivery while one is connected, and then the hooks and answers leave them out, so nothing shows twice; without one, answers carry t's own notices (a few KiB at most) and say how many messages wait, and messages wait in t inbox and the hooks. A message stays a day, and is deleted then. Delivery to pi and t inbox follow is at least once: an acknowledgement lost on the way, or a server handing over to the next one, can deliver a notice twice. Only the leading server hands notices out, on answers and to hooks alike: it alone knows which deliveries are connected.

Heavy checks on main

Main's every commit runs the default checks, so each failure is told to the commit that caused it. Heavy checks (an app's size budget, a browser SDK build, end-to-end tests in a VM) are too slow for that: declare them on = ["idle"], which leaves them out of the default checks, and main keeps them honest anyway:

[[check]]
name = "sdk-web"
run = "tools/sdk-web.sh"
on = ["idle"]

• Run on main's newest head. Each idle check runs on main's newest head unless it has a verdict there, already ran there, or a run of it is under way on any head: one run per check at a time; when that run ends, the next goes to the then newest head. Results are cached by key, so a head whose push didn't touch its inputs costs nothing.
• Queued behind main's heads. Its jobs go in the spare tier, as bisection's probes do: they wait behind every job of a run a session waits on and behind main's heads, and take the room those leave. A job of one that a head's or a person's check joined is that run's work, and stands at its tier until the check is answered. Like every CI job they get CI's CPU weight and are the out-of-memory killer's first choice.
• Keyed on every commit. Keys come from inputs, so recording them for every commit of main costs no runs. When the check fails on a head, bisection finds the commit that broke it with the budget default checks get, and t tells whoever pushed that commit, as above. t main, t claims, the pages and the notice mark it as heavy.
• Named before a push. t suggest lists the opt-in checks a change touches (their key at the working state differs from their key at the base commit), with the command that runs each.

Without idle in on, a check that isn't a default one runs only where its on says and when named: nothing bisects it on main.

Declaring checks

Checks live in t.toml at the root of the project:

# One check per workspace package: test/<package>.
[[cargo]]
name = "test"
tool = "test"           # or "nextest", "clippy", "check"
args = ["--locked"]

[[cargo]]
name = "clippy"
tool = "clippy"
args = ["--locked", "--all-targets"]
trailing = ["-D", "warnings"]

# Any shell command. Its result is cached under what it read as it ran.
[[check]]
name = "fmt"
run = "cargo fmt --check"
fix = "cargo fmt"       # what fixes a failure: `t fix fmt`, or the server by itself on main

[[check]]
name = "no-todo"
run = "! grep -rn TODO src"
default = false         # only when asked for: `t test no-todo`

No declaration says what a check reads: t records what each job reads as it runs and keys its result by that (docs/work.md, "What a check reads").

A check's name may have parts, nightly/test and nightly/linux/build: t test nightly selects them all, t test nightly/linux the second. A name can't start with a [[cargo]] family's name and a /, which names that family's per-package checks.

Checks also take env, timeout_secs, devices and requires. A cargo family takes wrapper (a command prefix such as a script that sets up the test environment), packages and exclude; a test family takes starts, the programs of other packages its packages' tests start (starts = { session = ["osk:osk"] }), which its build builds and its shards are told of (docs/work.md, "Programs a family's tests start"). heavy = true marks a check, or a family's checks, for a stabiliser's own tooling that reads t.toml; t itself acts on it nowhere. examples/tilo.t.toml is a larger, real configuration.

Matrices

One [[check]] can stand for a check per combination of values: per architecture, per app, per both. Its matrix names the axes, and {variable} in any of its strings (and table keys) is replaced by the combination's value:

[[check]]
name = "app-size/{app}-{arch}"
run = "tools/app-size.sh {arch} {app}"
requires = ["arch={node}"]
default = false

[check.matrix]
# A value can bring other variables along: aarch64 nodes are labelled arm64.
arch = [{ arch = "x86_64", node = "x86_64" }, { arch = "aarch64", node = "arm64" }]
# What `*` stands for in the tree's paths: every directory under apps/ with an app.toml.
app = { from = "apps/*/app.toml" }
# Combinations to leave out.
exclude = [{ app = "legacy", arch = "aarch64" }]

# Fields that differ for some combinations replace the template's there.
[[check.matrix.set]]
when = { app = "mail" }
run = "tools/app-size.sh {arch} mail-app"

The matrix is expanded where t.toml is read, before anything else looks at it, into the [[check]]s one would otherwise write out: same names, same declarations, so the same keys, and a project that switches to a matrix keeps its cached results. They are listed in the order name uses the axes (above, each app's architectures together). Only the matrix's own variables are replaced: {commit} in a publish tag, ${HOME} in a command and {a,b} in a glob stay as they are. needs = ["build-{arch}"] makes each combination need its own; where {arch} is no variable of the check that needs it, a need naming a template as it is written stands for every check of that template (needs = ["vm-acceptance/{shard}"] needs every shard). A from axis reads which paths exist, never what files say, and nothing runs to make the list: a state's plan stays a function of its files, so a new directory under apps/ is a new check in the commit that adds it.

An axis can also take its values from a TOML file of the tree, each with fields of its own:

[[check]]
name = "policy/{step}"
run = "{step.run}"
matrix = { step = { from_toml = "tools/policy/steps.toml", key = "step" } }

Each [[step]] table of the file is a value, named by its name, and {step.<field>} is its field (a string, number or boolean, as written), in any string as {step} is; a field an entry lacks is refused, naming the entry. The file is read as t.toml is, from the state, so an edit to it changes the plan, and nothing runs to make the list. A check holds the fields it uses as text, so an edit to its entry changes its key and an edit to another entry leaves it as it was. A list that only a program knows is better written into a file by the generator that knows it.

A [[cargo]] family takes a matrix too, and stands for a family per value, the variables filled in its name, packages, exclude, requires, env and every other string:

[[cargo]]
name = "test-{arch}"
tool = "nextest"
packages = ["crates/*", "os/{arch}-*"]
requires = ["arch={node}"]
matrix = { arch = [{ arch = "x86_64", node = "x86_64" }, { arch = "aarch64", node = "arm64" }] }

t dev checks <repo> [--commit C] [--project t.toml] [--only app-size/] prints the checks a commit declares as t reads them, matrices expanded, each under its key (--json for scripts): diff the output of a hand-written t.toml with that of its matrix to see that every check, and its key, stayed the same. A server older than matrices ignores matrix and says so, and plans the template as one check: update the server before a project uses them.

Testing a change

t test <selection> runs exactly what it names: check ids, families (clippy), packages, globs, single tests (--test), with what they need. t test alone runs what failed for this session last time first, then the checks this change touches (against the main commit it branched from, as t scope says), fastest first, and says how many it picked and why; when nothing failed and nothing is touched it runs nothing and says so. all is every default check, and needs --confirm, which the server enforces. A check with default = false runs only when its id or family is named (miri, miri/*): a package or a glob such as */my-crate selects only default checks. Checks whose inputs didn't change return their cached result at once.

t scope says what a change touches, compared with the main commit it branched from: the default cargo checks of the packages whose files it changes, and the default command checks whose inputs it changes (fastest first, with an estimate, and how many more checks it reaches). It runs nothing: a script names what it prints, t test $(t scope --names) fmt. With --json it carries the base and the estimate too. It exits 2 when it can't tell (no base on main, or a server that doesn't know it) rather than print an empty list a script would take for "nothing".

--rev <commit> makes t test, t scope and t fix take a commit of the repository instead of the working tree: its files as git's objects have them, never uncommitted changes or untracked files, and its own position (where it branched from main, its commits since). It needn't be pushed: it travels as what differs from the pushed commit it is based on, like a working tree. t fix --rev writes into the working tree only when the commit is the one checked out (and, as always, not into files edited meanwhile); for another commit, t fix --rev <commit> --patch prints what the fixes change as a patch on top of it (--patch=FILE writes it to a file; git apply takes it). Nothing is ever committed or amended. A pre-push hook needs no checkout of its own:

pushed=$local_sha     # from the hook's input
t test --rev "$pushed" $(t scope --rev "$pushed" --names) fmt lockfile
t fix --rev "$pushed" fmt              # when the push is HEAD; else --patch=fix.patch

Main is ungated: changes are pushed to it directly, and its CI informs (who broke what, and where it began). A fixable failure says <check>: fixable, t fix applies it; t session autofix on has t test apply such fixes to the session's working tree by itself (off by default). On main, t runs a failing check's fix and pushes it (fix = ["main"]).

Test binaries built once, and only the tests a change can affect

A nextest family keeps what its builds make, so the same checks never compile twice:

• Built once, run many times. A job that builds the family's test binaries also leaves a cargo nextest archive of them in t's store. Running exactly those checks again on the same inputs (--force, a retry, a job handed to another node) runs the archive instead of building: no workspace build, no build directory needed. If running from the archive fails before any test reports, the result is an error (run again, not cached) and the archive is forgotten.
• Only the binaries a change can affect. After a build, the node reads each test binary's dep-info (every source file rustc read for it). When a package's check passes, t keeps which binaries read which files of the package's own directory. Next time, if nothing outside that directory changed (dependencies, lockfile, toolchain, image, environment) and every changed file is Rust source read only by integration-test binaries, only those binaries run. A change to the library or a binary target, to anything that isn't Rust source, to a file no binary read (a fixture, Cargo.toml, build.rs, a new test), or a package with tests seen flaky in the last 14 days runs the whole package. Only passing runs are kept as a base.

A family's wrapper sees what t does: a run from an archive takes no -p, so the packages a job tests are in T_NEXTEST_PACKAGES as well, and a selection is in T_NEXTEST_FILTER (a filterset of binaries) for a wrapper that adds filters of its own to combine with (-E "(<its own>) and ($T_NEXTEST_FILTER)"). A wrapper that ignores it runs more tests than needed, never fewer.

What cargo checks read

A [[cargo]] family's check of a package is keyed, like any check, by what its jobs read as they ran (chapter 4 of docs/spec.md): its package's build by its share of the family's driver, and its test binaries' runs by each binary and what the run read. So editing a crate reruns the checks of the packages whose builds read it, and nothing else. A first run of a key, and a job nothing recorded the reads of, is keyed by every file of its state.

A check writes into its files as it likes (npm ci making node_modules, a generator writing next to its sources): every job sees the tree through a copy-on-write view of its own, over the read-only layers checks of the same version share. What it writes there is gone when it ends, and never reaches the tree other checks use.

A check sees no Git metadata unless it declares git = true. It then gets a minimal .git in its sources: HEAD is the commit being checked, detached, and main (and origin/main) is the commit it branched from. For a followed branch that is the commit itself. For your working tree it is your HEAD, with your uncommitted changes showing in git status and git diff, and main is its merge base with main. The history comes from the server's mirror of the repository it follows ([[repo]]), borrowed read-only as an alternate object store, so nothing of your .git is uploaded. A HEAD you haven't pushed isn't there: HEAD is then the merge base, your commits show as uncommitted changes, and the log says so. The commits are part of the check's key, so its result is kept per commit, while checks without git stay cached across commits with the same files. Workers fetch the repository themselves (they need git and access to its URL).

fix (on a [[check]] or a [[cargo]] family) is a command that resolves the check's failures by changing files the check reads. t fix runs it on the server against your files and writes the changes back; a server can also run it on a followed branch, commit the result and push it. docs/fixes.md describes both, and what keeps the automatic fixes safe.

A [[check]] can also publish: build and push images, sign, release. These fields are described in docs/pipelines.md:

• on: when the server runs it by itself: on new heads of a followed branch ("stable", or a glob such as "release/*"), new tags ("tag:v*"), a schedule ("schedule:0 3 * * *", UTC), "dispatch" (t dispatch), or "idle": on main's newest head where it has no verdict yet, with its key recorded for every commit of main, so a failure is bisected to the commit that broke it and told to whoever pushed that commit, as a default check's is (see "Heavy checks on main" below). A check that names an on runs only there and when asked, unless it says default = true, which runs it on every change too; one without on is a default check unless it says default = false. on also works on a [[cargo]] family, for each of its checks;

• needs: checks whose outputs it consumes; they run first, and one that fails skips it;

• optional_needs: needs that run only on some references (signing on stable): where their on applies they are ordinary needs, elsewhere the check runs without them;

• needs_any_outcome: checks it runs after whatever their outcome, like Forgejo's if: always(), for cleanup and reports; T_NEEDS_FAILED names those that did not pass;

• outputs: globs of files it produces under the source root or /target/, kept with its result and downloaded with t outputs;

• id_token: audiences it may ask the server for short-lived identity tokens for (OpenID Connect, as Forgejo and GitHub Actions jobs do), only in trusted runs on the refs each allows: a registry or cloud that trusts t exchanges them for credentials, so nothing stored is needed;

• secrets: secrets from the server's config, given only to trusted runs on their refs, for what can't take an identity token;

• publish: an image the server pushes itself, instead of running the check's command: publish = { layout = "/target/publish/image/oci", repository = "registry.example/team/app", tags = ["sha-{commit}", "main"] }. layout is an OCI image layout among the outputs of the checks it needs, and the registry must be one of its id_token audiences. In a trusted run the server logs in with the run's identity token for that registry, uploads only the blobs the registry lacks, straight from its store, puts the manifests and then the index under each tag, and checks the registry serves that digest; its last line is <repository>@<digest>. No job runs and no micro-VM boots, so pushing costs what the upload costs. Servers that don't know publish run run instead, so keep a command that does the same;

• pointers, reads_pointers: named values (the current app collection) a check publishes from a trusted run for its reference and later checks read as T_POINTER_<NAME>;

• isolation: the check's isolation level, "strict" unless it says otherwise:

  • "strict": Tilo's job sandbox, as every check by default: no capabilities, no new namespaces or mounts, seccomp and Landlock;
  • "nested": the check starts sandboxes of its own (tests that run bubblewrap or unshare). It may then create user, mount and network namespaces and mount in them; the filter still refuses nftables and XFRM netlink, bpf, io_uring, the keyrings, userfaultfd and perf, and Landlock is off (it refuses every mount). Such a check runs as uid 1000 inside rather than root, so bubblewrap makes its namespaces as for any unprivileged user, and its /tmp is a plain directory, where every other check's is an overlay (overlayfs takes no overlay as the upper layer of the overlays such sandboxes make there);
  • "vm": the check needs real root (rootful podman or buildah, loop devices, mounts, VMs of its own) and runs as root in a micro-VM step: its image is the root of a guest kernel of its own, while the VM itself is an ordinary job in the sandbox. Nothing runs as root on the node. See "Micro-VM steps" below. A check with secrets always runs as "vm": a stored secret reaches only a guest kernel of its own, so no node is set aside for them. A check with only id_token keeps the isolation it declares: an identity token names its run and expires in minutes.

  A [[cargo]] family takes one level for all its packages (isolation = "nested") or packages by level (isolation = { nested = ["tilo-shell", "ush"] }); its checks can't be "vm". The older spellings (nested_sandboxes, vm = true, privileged = true) are refused, naming isolation;

• cache = false: run every time, never from a stored result;

• [check.dispatch_inputs]: the inputs t dispatch --input name=value may set, with defaults, passed as T_INPUT_<NAME>.

Micro-VM steps

A check with isolation = "vm" runs as root in a guest kernel of its own, so it may run rootful podman and buildah, attach loop devices, mount filesystems and boot VMs of its own, while the virtual machine itself is QEMU in the job sandbox, unprivileged, with /dev/kvm as its only device. It boots in about a quarter of a second.

[[check]]
name = "image-disk"
isolation = "vm"
run = "podman build -t disk . && bash make-disk.sh"
outputs = ["/target/disk/disk.qcow2"]

It sees what a check in its image sees, as root: the image as / (writable, copy-on-write, dropped at the end), its files at /src (copy-on-write), /target (see below), a .git with git = true, the network through passt (outbound connections and DNS, nothing inwards, no way to the node's loopback but the compiler cache's port), the cores and memory it was leased, and a /dev/kvm of its own if it requires /dev/kvm. /tmp, /var/tmp and /var/lib/containers are on a sparse disk of its own. Its outputs come back like any check's.

Its userspace is its image's, and nothing else: the guest adds only a kernel (t's) and an init (busybox) that mounts /proc, /sys, cgroup2 at /sys/fs/cgroup, /dev/pts and /dev/shm, brings up the network, then runs the command (after t's mounts of its inputs) as pid 1's only child. A program the step runs must be in the image. No systemd runs either, so systemctl can't start services even where the image has it: a step starts the daemons it needs itself (podman system service &, dockerd &). What it doesn't get: cargo's shared downloads, the jobserver and fix. A step gets the compiler cache as any check in its image does (t's wrapper, and the node's endpoint, the one port of the node it reaches). Its sources and /target are the very layers a check outside a VM is given (where the node prepares jobs from snapshots, see "Disk space", the build snapshot it was leased under the outputs of the checks it needs), shared with the guest read-only over virtio-fs and copy-on-write in it: nothing is copied before the command starts, what a step writes stays in its guest, no snapshot is ever changed and no step's build directory becomes one. Its guest gets the tokens and memory its job is booked at, less room for QEMU and its helpers (see "How jobs share a node"): an 8 GiB guest until a run of it was measured. A guest can't grow; memory it runs out of is reported in its log, and the step runs again with twice as much.

A node runs micro-VM steps (micro-VMs in t status) when its checks run in the job sandbox and it has a usable /dev/kvm, QEMU (qemu-system-x86_64 or qemu-system-aarch64), mkfs.ext4, passt and virtiofsd. The image's root becomes a read-only disk once per image and node, next to the unpacked image. The guest's kernel and busybox are the outputs of t's own vm-guest-<arch> checks (deploy/vm-guest), built reproducibly and pinned by digest in t; the server keeps them in its store and nodes fetch them like any outputs. t test vm-guest-x86_64 in t's repository puts them in a server's store. T_VM_GUEST, a directory holding kernel and busybox, makes a node boot another guest.

Unknown and removed fields

A t.toml field the server doesn't know is refused, naming the known field it most resembles (did you mean \timeout_secs`?`): ignored, it could change what a check runs without anyone noticing. So a repository updates the servers it runs on before it uses a new field. A field t removed is refused, saying what replaced it. The server's own config is strict the same way, with the same suggestion.

Where checks run

Without an image, checks run with the tools installed on the server's machine, in a copy of your files: in the job sandbox they see the machine's /usr and /etc read-only, the directories on t's PATH and rustup's toolchains, their files at the same paths as on the machine, and HOME in their own /tmp. That is the simplest setup on your own machine, but results depend on what is installed.

A check with image = false runs that way too, whatever the project's image: for a check that needs what the node has installed. It needs no label.

On nodes running t's own image (deploy/Containerfile.base), what the node has installed is a minimal Fedora: bash, coreutils, git, tar, podman, bubblewrap, passt, QEMU and e2fsprogs. There is no python3, no compiler and no systemd (so no systemctl). A check that needs more runs in an image that has it. A check without an image that exits with status 127, the shell's "command not found", gets a line in its log saying where its tools come from.

A [[check]] can also run in an image another check of the same run built: image = { check = "image-builder" } takes that check's output archive (OCI, or what docker save writes) as its image, so a candidate image is tested before it is published (docs/pipelines.md).

With image = "…" at the top of t.toml (or per check), every check runs in that container image, read-only, with your files at /src, its build directory at /target, a private /tmp and network access. /var is writable, copy-on-write, as in a container: what a check writes to /var/tmp, /var/lib or /var/cache stays its own and is dropped when it ends (on nodes that can't mount overlays, only /var/tmp is, as a private directory like /tmp). Results then mean the same on every server. The image must contain every tool the checks use: the official rust images, for example, have no clippy or rustfmt, so build your own image on top of one, and pin it by digest. Cargo's downloads are shared between checks.

Images don't go through podman on the way: the server fetches an image's manifest, config and layers from its registry (anonymously, or with the token the registry's token service hands out) into its own store, once for the whole cluster, and each node fetches the layers from the server like any other blob and unpacks them itself, applying whiteouts. When a followed branch's new head names an image no node has (a builder digest bump), the server fetches it at once and every node unpacks it in the background, one image at a time and at low priority, so the jobs that need it don't pay for it. An image the server can't fetch that way (a registry that wants credentials) is pulled with podman as before.

Each check has a network namespace of its own when the node has pasta (Fedora's and Debian's passt package): it connects to the outside and resolves names as the node does, but its loopback is its own. A server one check starts on 127.0.0.1 (sccache's, which CMake uses whenever sccache is installed) is out of reach of every other check and of the node's own services, and ports don't collide. A job whose spec names an egress allowlist reaches only what it names: the node writes an nftables filter into the job's namespace before pasta connects it, admitting the hosts and address blocks on the ports the spec names and refusing everything else, private and cluster addresses included. Such a job gets no DNS — a query would be a way out of the sandbox — so a name on the list is resolved on the node when the job starts, and the job's /etc/hosts carries the addresses the filter admits. A node that can't enforce a list refuses the job rather than run it with a wider network. Without such a list a check reaches whatever pasta connects it to, as before. Without pasta, checks share the node's network, and t only points sccache at a socket in the check's /tmp (SCCACHE_SERVER_UDS) and at a port no other check gets (SCCACHE_SERVER_PORT, for versions of sccache without sockets): a server another check started would compile in that check's sandbox, into its build directory. t serve warns about it when it starts. A node that has pasta tries it when it starts and, where pasta can't connect a namespace, shares its network as without it. Should pasta fail for one check, that check errors rather than run without the network it would have had: an error is never cached and runs again, while a build without network would fail as if its commit were at fault. Under bubblewrap, checks without an image share the machine's network.

In the job sandbox a check may start sandboxes of its own (bubblewrap, unshare) only with isolation = "nested"; every other check is refused new namespaces and mounts. Containers that need root (podman, buildah) run in micro-VM steps (isolation = "vm"). Under bubblewrap a check may, such as tests that run a program under bubblewrap with --unshare-all and --proc /proc. bwrap running as root (as t does in a privileged container) covers parts of the /proc it mounts with read-only binds, and the kernel refuses a new procfs in a nested user namespace while those covers hide parts of the only one there is. t removes the covers before the check starts: such a check runs as root with root's capabilities anyway and could remove them itself.

In a container, run t under an init that reaps orphaned processes, as the image does with tini: bwrap running as root and pasta leave a process per check to the container's init, and t as pid 1 would let them pile up as zombies until the container can't start processes.

A shared server

A team or a fleet of agents shares one server, so a result one of them produced serves all, and a build directory one of them warmed up speeds up the next.

t serve --config server.toml

[server]
listen = "0.0.0.0:7777"
metrics_listen = "0.0.0.0:9777" # optional: Prometheus /metrics without a token; keep it private
data = "/srv/t"                 # run spools, mirrors, the content store
ticket_key = "tickets"          # the secret node credentials are signed with: a [[secret]] without `refs`

[server.database]               # see "The database"
url = "postgres://t:...@db:5432/t"
schema = "t"

[[server.tokens]]
name = "agents"                 # the principal its sessions are
token_sha256 = "…"              # `t token new`, then `t token hash`
preset = "default"

[[server.tokens]]
name = "release"
token_sha256 = "…"
preset = "integrator"

[[server.tokens]]
name = "release-workers"
token_sha256 = "…"
preset = "worker"
node_labels = ["release"]       # optional: labels its workers may offer; `release` needs a token listing it

[server.presets.reviewer]       # presets beyond the provided ones (see "Sessions and presets"),
capabilities = ["run", "claim", "take_over"]  # or `presets = {}` under [server] for none
description = "Triage without moving the stable branch"   # optional

[[server.logins]]               # a provider people log in with; `logins = []` under [server] for none
issuer = "https://auth.example/application/o/t/"  # exactly as its tokens name it (`iss`)
client_id = "t-web"             # the web UI's client, the ID tokens' `aud`
name = "Example login"          # optional: what the login button calls it
user_claim = "preferred_username"  # names the user, who becomes the principal
groups_claim = "groups"
scopes = ["openid", "profile", "groups"]  # must bring the user and groups claims
preset = "default"              # optional: for a user in none of the groups; without it they are refused
mcp_audience = "t-mcp"          # optional: this login's public OAuth client, for MCP clients signing in to /mcp
[[server.logins.group]]         # presets by group: the first listed group the user is in decides
name = "t-admins"
preset = "admin"

# What jobs need of the server: the key development VMs' attach tokens are signed with, by the
# name of a [[secret]] without `refs`, and where jobs reach the server. Without `public_url` jobs
# get no attach token.
[execution]
attach_key = "attach"
public_url = "https://t.example" # optional

# A secret. One without `refs` is the server's own (`ticket_key`, `attach_key`, a `push_credential`)
# and never reaches a check. One with `refs` reaches only trusted runs on these refs (of these
# followed repositories, if `repos` names any), only on nodes with these labels (`release`, which
# takes trusted work alone, when it names none), and is masked in every log.
[[secret]]
name = "tickets"
value = "…"                     # any value derives the key: long and random, so credentials are the server's alone

[[secret]]
name = "attach"
value = "…"

[[secret]]
name = "push-token"
value = "…"

[[secret]]
name = "SOME_API_KEY"
value = "…"
refs = ["stable", "tag:v*"]
repos = ["myproject"]           # optional
nodes = ["release"]             # optional: `release` alone when it names none

[[principal]]
name = "agents"
authors = ["dev@example.com"]   # commit authors whose breakages this principal's sessions hear of

# Optional: follow a repository's main branch, so failures can be compared with it. Each
# `[[repo]]` has a history, board and claims of its own, even where branch names are the same.
[[repo]]
name = "myproject"
url = "https://example.com/me/myproject.git"
main = "main"                   # optional, the default
stable = "stable"               # optional
follow = ["release/*"]          # optional: more branches whose new heads run their checks (names or globs)
fix = ["main"]                  # optional: fix failing checks that declare a `fix` on these branches
push_credential = "push-token"  # optional: the [[secret]] without `refs` t pushes fixes with

# Optional: identity tokens for publishing checks instead of stored credentials
# (docs/pipelines.md): t is an OpenID Connect issuer, and a check listing an audience in
# `id_token` may ask for tokens for it in trusted runs on these refs, on nodes with these labels.
# Discovery and keys are at <issuer_url>/.well-known/openid-configuration and
# <issuer_url>/v1/oidc/jwks, without a token.
[oidc]
issuer_url = "https://t.example" # the server's public base URL, as registries and clouds reach it
request_url = "http://t:7777/v1/oidc/token"  # optional: where jobs ask, as a job reaches the server
signing_key_pem = "…"           # PKCS#8 PEM: RSA of 2048 bits or more, or EC on P-256
request_key = "…"               # at least 32 characters
[[oidc.audiences]]
name = "registry.example"
refs = ["main", "stable", "tag:v*"]
[[oidc.audiences]]
name = "sts.amazonaws.com"
refs = ["stable", "tag:v*"]
nodes = ["release"]             # optional

Clients read ~/.config/t/config.toml (server, token) or T_SERVER and T_TOKEN.

The compiler cache

Served by t from its own store: each crate compiled once, fetched everywhere else. Every job gets t's own compiler wrapper (RUSTC_WRAPPER, a copy of t in the job's /tmp) and a socket to its node's cache daemon, served by the job's trust, which forwards to the server and keeps a copy of what it read and wrote. A daemon that is away (its worker restarting, seconds) is waited for, never compiled around: a compile outside it would skip the cache and the node's admission alike, and a node that never comes back is lost, its jobs ended by the server. A check can't set RUSTC_WRAPPER itself: t.toml refuses it, and t cargo refuses a --config naming a wrapper. The cache is part of no key.

The wrapper caches every compile it can tell the outputs of: libraries (external crates and the workspace's, as cargo doesn't compile them incrementally under CARGO_INCREMENTAL=0), proc macros, build scripts' programs, binaries and test harnesses, not incremental compilations. The cache helps where a build directory is cold: a new node, a state no build snapshot is near, and every job of a node without overlays.

Entries are kept per key and trust: stable for stable's runs and release tags, which read only stable's; main for main's other trusted runs, which read main's and stable's; and the shared pool for every untrusted run, which reads all three. What an untrusted run writes reaches neither main nor a release. /metrics has t_cache_find_total, t_cache_misses_total, t_cache_puts_total, t_cache_placed_total and t_cache_trimmed_bytes_total. Every hit is counted on its entry, a node's placements and the server's runs' answers alike. Collection drops entries by worth alone, never by age: while the store's volume is short of room, those that yield the fewest hits per byte of the store (each blob's bytes split among the entries naming it, times the time since the entry was last used, per hit), until what they free covers what the volume lacks; the store's collector takes the blobs no entry holds.

Each entry keeps the kinds of its key's fields: each run of neighbouring fields of one kind (the toolchain, its keyed arguments, its keyed variables, its dependencies and each file the compile read, the first 2000 of them) with a digest of the run and how many fields it holds, a few hundred bytes, and the names of the files and variables it read, by the id of its read set. A miss keeps the fields its job keyed, so t why-miss <package> shows the stored entry of the package nearest it (the one holding the most fields alike), how long ago that entry was stored, the kind of field that differs first with both sides' digests, and the files and variables one of the two read and the other didn't. The explanation is worked out after the miss is answered, so it never holds a compile back. /metrics groups the misses by the kind of field that differed from that entry: t_cache_miss_causes_total{cause="argument"|"variable"|"dependency"|"file"|"toolchain"|"none", node,platform}, none where the package has no stored entry or its nearest differs in nothing the explanation covers.

Pushes start checks

The server learns of a push only from a notification: POST /v1/repos/<name>/push, where <name> is the [[repo]] name, from a hook in the repository or a forge's webhook, carrying a t token whose preset has the push capability (pusher, or admin) as a bearer token. It fetches every repository once when it starts; after that, a push nobody notifies it of runs nothing. The request only wakes that repository's CI, which fetches the followed branches, starts the new heads' runs and then fetches the rest. The body is trusted for the branch, the old and new head it names (ref, before, after) and who pushed: a forge's pusher.email, else the session of the token that sent it. The heads only say what to fetch; who pushed decides who hears of a breakage among the commits between the two heads, so a token that may push can steer those notices, and no more. Bursts of pushes are coalesced into the fetches they wake.

On Forgejo (Settings → Webhooks → Add webhook → Forgejo): target URL http://<t server>/v1/repos/myproject/push, method POST, content type application/json, trigger on push events, and "Authorization header" Bearer <token>, a t token whose preset may push (pusher). A webhook without that token, or with a token that may not push, is refused (401 or 403) and starts nothing; the forge's list of recent deliveries shows it. GitHub's (Settings → Webhooks) can't send such a header: notify t from a workflow or a hook instead.

A server that doesn't lead at that moment (a restart, a rollout, its database connection lost) answers 503 and did nothing, for the sender to send again. Forgejo's webhook doesn't retry by itself (redeliver from its list of recent deliveries), so a server fetches every repository whole each time it starts leading (a server that loses its database connection loses the lead, exits and is started again). A push whose notification only met 503s while no server led still runs. A post-receive hook in the bare repository retries itself:

#!/bin/sh
# hooks/post-receive: T_TOKEN is a token of a principal with preset "pusher"; each pushed branch
# is told with its old and new head, the pusher being that token's session
while read -r before after ref; do
  curl -fsS -m 5 --retry 10 --retry-delay 1 -X POST -H "Authorization: Bearer $T_TOKEN" \
    -H 'Content-Type: application/json' \
    -d "{\"ref\":\"$ref\",\"before\":\"$before\",\"after\":\"$after\"}" \
    "$T_SERVER/v1/repos/myproject/push" >/dev/null || true
done

A missed notification's push is found by the next notification's fetch, which fetches every followed branch. Every commit a push brings to main is recorded and imported, however many came at once, so bisection has them all.

How your files get to the server

Every command that runs something takes the directory as it is, uncommitted changes included. A server that follows the repository ([[repo]]) already has almost all of it: the client finds the commit its work is based on (where HEAD meets its upstream branch, else the main branch of origin), and sends that commit's id and only the files that differ from it: edited, added and deleted files, and those of commits not pushed yet. The server takes the rest from its own copy of the repository, fetching once with plain git if the commit was pushed after it last looked. A fresh checkout of a large repository costs a few hundred bytes instead of an upload of the whole tree, and so does every run after it.

Files are compared with the commit byte for byte (by git blob id, computed from the file on disk), so a file git converts on checkout (line endings, filters) is sent as a change, and the server checks that what it built is exactly the client's tree before it uses it. Whatever it can't take from the repository (a repository it doesn't follow, no remote, a server too old for this) is uploaded, and only the files the server has never seen from anyone. The client keeps every file's hashes by size, modification time and inode, and where HEAD branched by the commits involved, so a run reads only files that changed since the last one: in a checkout of Tilo (8,700 files) a run's own work before it reaches the server takes under a tenth of a second, most of it git ls-files.

With --rev <commit> the state is that commit's files as git stores them, read from its objects (before any checkout filter): what a scan of the working tree hashed, and the blobs of the last commit asked about, are known by their blob ids, so only blobs neither saw are read from git.

What git ignores stays home: target/, editor files, anything in .gitignore.

Sessions and presets

Every request names a session. Agents are recognized by PI_SESSION_ID, CLAUDE_CODE_SESSION_ID or CODEX_THREAD_ID; T_SESSION sets one explicitly, and a shell is human:<user>.<host>. A session belongs to the token that first used it. Name it with T_SESSION_NAME, t session name <name>, or the MCP session tool. Neither a name nor an id can hold parentheses or @ (a message's header names its sender as session <name> (<person>), and t send finds sessions as <name>@<person>): the server refuses a request that says one, and names stored before lose those characters when a server starts leading. t sessions lists the sessions active lately as their records have them, and t session <name or id> shows one, found as t send finds it.

In Claude Code, the t inbox --hook=claude hook that runs when a session starts leaves export T_CLAUDE_SESSION=claude:<id> T_CLAUDE_FOR=<process> in the file Claude Code names in CLAUDE_ENV_FILE, which it runs ahead of each of the session's shell commands: they run as the session Claude Code holds, the new one after /clear too (CLAUDE_CODE_SESSION_ID in their environment stays the one the process started with). <process> is that id, followed by :pi:<id> when the process inherited a PI_SESSION_ID from a pi it was started in. The pair counts only where the process is still <process>, and then comes before PI_SESSION_ID: a pi or a claude -p started from the session's commands has an id of its own and is that session, not the one that started it, and a Claude Code started from pi's commands is its own session, not pi's. A T_SESSION set where Claude Code runs is left as it is, and comes first. A subagent's commands are its parent's session's: Claude Code runs them in the parent's environment, and a hook could only tell them apart by rewriting each command, which Claude Code's permission rules then match against too. So a subagent's runs, claims and name are its parent's session's.

A subagent of another agent can still be a session of its own, linked to the session that started it: T_SESSION names it (by convention its parent's id with its own after a /, pi:01a138ab/aa91869a), T_PARENT_SESSION its parent and T_AGENT its kind. Unless it is named it is named after its parent and its kind (deploy and metrics › Explore); t session shows its parent. A parent that acts only after its subagent is linked when the subagent's record is next written, within a minute.

A token's preset decides what its sessions may do:

capability	allows
run	running checks and builds, reading results, logs, the board and sessions; messaging the principal's own sessions (t send)
claim	claiming board items
take_over	releasing another session's claim
attest	recording results a stable branch may advance on
cancel_any	cancelling any session's runs
manage_sessions	renaming any session; messaging any principal's sessions
limit	capping a session's or principal's cores and memory (t limit); leaving a session a notice
work	taking jobs as a worker node
publish	starting dispatchable checks on the server's repository (t dispatch)
push	telling the server a followed repository was pushed to (POST /v1/repos/<name>/push)

The provided presets are default (run, claim), integrator (adds take_over, attest, publish), admin (everything but work), worker (work) and pusher (push). A config can redefine them or add its own:

[server.presets.reviewer]
capabilities = ["run", "claim", "take_over"]
description = "Triage without moving the stable branch"

T_PRESET lets a session act under a narrower preset than its token's, for example an agent started from a person's admin token as T_PRESET=default.

Workers

Jobs run on workers: machines that join with a token whose preset has work. The server runs no jobs itself: it holds the database, every token and the secrets, and a check on its machine would be one sandbox escape away from all of them (see docs/security.md).

T_SERVER=https://t.example T_TOKEN=… t worker --data /srv/t-worker

A worker fetches only the files it lacks, keeps its own build directories and streams output back. It keeps its disk as "Disk space" says and reports it with every poll, so t disk and the server's /metrics cover every node. A worker that stops responding for 90 seconds (three of its polls in a row unanswered, proto::worker::LOST_AFTER) gets no new jobs, and the jobs it had run once more, on another node if one has room; a job whose node fails under it does too. Only if the second attempt can't run either is it reported as "could not run", and that is never cached. A result that took a second attempt says so (retried in JSON, "ran again" in text).

--label <label> gives a worker labels checks can requires. Two are special: privileged nodes are kept for checks that require that label, and release nodes take only the isolated jobs of trusted runs (see docs/pipelines.md). Both are kept for the jobs that need them: a job that requires neither label waits for another node rather than taking one, and uses them only when no other node could ever run it, so a one-job privileged VM isn't held for an hour by a lint. t worker --once takes one job, finishes it and exits 0; under a VM-isolating container runtime every such job gets a fresh machine.

A macOS worker runs checks on the host only, as the user it runs as, and says so (no images in t status): checks with an image never go there. A check gets there by requiring os=macos (any architecture, unless it names one too); it then runs without the project's image, which is a Linux root filesystem:

[[check]]
name = "runtime-macos"
run = "cargo check -p tilo-runtime --all-targets --locked"
requires = ["os=macos", "arch=arm64"]

Such a check needs no isolation level: on macOS every check runs on the host, so isolation (or privileged) asks for nothing more there, and older configurations that set it keep working.

A check that names no architecture runs on the server's own: its result is cached under a declaration that doesn't say, so it must not come from another architecture. A check whose result is the same on any (a formatter, a lint of the sources, a script that only reads them) says so with requires = ["arch=any"], and runs on whichever node has room, arm64 or x86_64, its image permitting; that changes its key once.

t cargo, t build and t run run on a node of the client's own architecture and OS: from a Mac they build on a macOS worker of its architecture (on the host, without the project's image), from an arm64 Linux machine on an arm64 Linux node, because the programs they bring back must run there. --on <requirement> (before the cargo subcommand, repeatable) asks for something else, --on any leaves it to the server, and a build with its own --target needs no particular node. Without such a node connected the command fails at once, naming the nodes there are (no macos/aarch64 node connected; nodes: local (x86_64 linux, …); …).

t run -p app                                     # from a Mac: built on a macOS arm64 worker
t build --on arch=x86_64 --on os=linux -p tool   # a Linux binary, from a Mac

Checks from t.toml keep their declared placement: on the server's own architecture and OS unless they requires another.

How jobs share a node

One scheduler, on one thread of the leading server, decides where and when every job runs, its tokens, and what it is booked at. t.toml declares no memory, cores or disk: a requirement, cost, cpus, memory or [defaults] in it is refused, saying so.

Booking. A job is booked at what its runs measured, and the booking is its limit: what it may hold that the kernel can't take back, written to its cgroup's memory.low, with memory.max its limit plus room for the page cache of its files; for disk, its job directory's quota (with room for one refused write).

job	memory booked
cargo driver	the most its key's runs held: cargo's own, its compiles booked apart as they start
test shard	the most its binary's runs held
opaque command, setup, VM step	the most its key's runs held; else its check's
none of its keys measured	its kind's default: 1 GiB, a VM step an 8 GiB guest with QEMU and its helpers

A measured booking has room above the peak: a fifteenth of it, so a run holding it again stays below where its node raises it (a job has no memory.high), and at least 32 MiB; disk likewise, at least 1 GiB. While a job runs, its node raises its limit where its own room lets it (node::admit); only its disk is asked of the server (/v1/grow, granted where the node's free space has room), and no limit is ever lowered. A job at its limit with no room to grow ends grown_out and runs again once, booked at what it held. A VM step's guest can't grow: one whose kernel killed a process for memory ends guest_memory, and runs again with twice the memory, as does one that failed starved of it without a kill (its tasks stalled on memory, or an allocation failed). The bookings on a node never pass what it offers: the memory its host can give, page cache not counted as used, less what t itself holds.

Tokens. Each node has a CPU budget, its knee: the most compiles at once at which its throughput still grows, measured from what its compiles report, and its cores before that. The tokens granted on a node stay within it, and the node hands them to every unit by one rule (node::admit::Pool::enter_beside). A job gets its key's measured parallelism, else its check's, else book::FIRST_WIDTH (a driver nothing measured: what its compiles are expected to keep busy, else one), as many as are free. A driver holds none for its compiles: each compile is a unit of its own, and its node hands it tokens as it starts. The server keeps no count of cores: what a node says is free is what placement reads.

Order. Jobs are looked at in three tiers, then by when they were asked for: first those of runs a session waits on, then everything else (main's heads), then the spare tier (bisection's probes and main's idle checks), which takes what room the heads leave. The job another run waits on — a check of it joined that run's work — stands where that run does, at its tier and its session's hold too; the rest of its own run's jobs keep their run's place. A job takes the eligible node where it starts soonest with least to fetch (a session's driver the one holding its kept build, a driver one where the largest of its expected compiles fits), where its booking fits and a token is free. One that fits nowhere claims its booking on the node it is least short of, and the jobs behind it may not take that room; claims hold no time and are worked out again on every pass, so nothing waits on a forecast.

No limits apply to sessions unless an admin sets them: t limit <session> --cpus 32 --memory 64G, or t limit --principal <name> ... for every session of a token together. Jobs past a limit wait (saying why in t jobs), and others go ahead. t limit <who> --clear removes one, t limit --clear all of them, and t limit lists them; t jobs, t status and t sessions show them too. Setting limits takes the limit capability, which the admin preset has. A managing agent does the same with the MCP limit tool.

Every job is measured. On Linux, a node that may manage its own cgroup (cgroup v2, delegated: a privileged container, or systemd-run --scope -p Delegate=yes t serve) runs each job in a cgroup of its own, and records its CPU time, the most cores it kept busy, the most memory it held that the kernel can't reclaim (anonymous and shared memory, kernel stacks, page tables, sockets and unreclaimable slab; page cache aside), the bytes it wrote and whether the kernel killed it for memory. Whatever a job leaves running is ended with it. In a container, t moves itself and the rest of the container (an init such as tini, say) into a leaf t-server of the container's cgroup, and the jobs into t-jobs/<job> beside it; that also works when the pod sees the host's whole cgroup tree. The last line of a job's log says what it used:

t: used 11.3 cores on average (42 at most), 6.2 GiB of memory at most, in 3m 10s

The server keeps the last 30 measurements of each check (of a cargo family, of a t cargo command line). Without cgroups (macOS, or a cgroup t may not manage, which it says once at startup) only the CPU time of jobs that ran alone on their node is known.

Disk space

Builds write tens of GB, and cargo never deletes what it built for earlier commits, profiles and targets, so left alone a node's volume fills and jobs die of "No space left on device". A node keeps its volume's free space for its jobs: disk is booked like memory, and everything else t keeps there (builds, sources, images, crates, blobs) is a cache that gives way to them.

• Measured, not walked. On Linux a node's data directory is an XFS volume of its own, mounted with prjquota; the node refuses to start on anything else (store::trees::VolumeRefused). Made with reflink=1 its copies are clones and move no bytes, which is what a store wants: on prod's volume without it, a worker wrote 27.8 TB in 16 hours copying. Every job directory and every layer is an XFS project of its own, so what it holds is the filesystem's count, read in one call however many files it has, live while a job writes. macOS has no project quotas: a layer is walked once as it becomes one, and a job's directory at its end.
• Bookings. A job is booked the most a whole run of its key wrote, with a fifteenth above it and at least 1 GiB (a cargo driver a quarter above, at least 4 GiB: its kept build directory grows between runs), and nothing before it was measured. Its directory's hard quota is its booking plus 1 GiB, the one write XFS may refuse whole; a job booked nothing has no quota. Within the headroom its node asks for more as it nears its quota (node::run::grow::near_disk_quota: what it wrote since the last sample, an eighth of the quota, or 16 GiB), granted where its node's free space has room and never lowered (grow::disk_grown). A job that still ends at its quota is t's failure (disk_full, its log says t: ended at its disk limit of … bytes, having written …) and runs again once, booked at twice its booking plus 1 GiB.
• Admission. A node offers its volume's free space. The scheduler places a job only where its booking fits that, less what the node's running jobs booked and haven't written yet and what jobs ahead of it claimed there. A job refused for disk claims the node with the least it is short of and waits there, and the node frees room for it. A node never offers more than its ceiling: its free space, what it is deleting, and everything it could free. A job whose booking is more than every node's ceiling waits for a node that could hold it, and no node's cache is emptied for it.
• One freer, one order. Each node keeps its volume's free space above a line: what the server says its jobs need (their bookings not yet written, and the booking of a job waiting for disk there). Below it, one freer frees in one batch what is worth least per byte — the seconds it would take to have an item again times how likely it is needed — until the batch covers the gap. Above the line nothing goes: no fixed size, no expiry by age. Nothing anyone holds is offered. The order, the line and the floor are docs/storage.md's.
• Sources and builds. A job's sources are one layer over the nearest state the node has, shown read-only, and its target/ is an overlay of a build's layers, a touched layer and an upper of its own, which is kept as a new layer on the build it used (a rename, nothing copied) when the job passed or failed with cargo itself failing. It builds on a build of its own state, else on the one leaving the fewest seconds to build. Which builds are kept, how deep a stack goes and what a stack is on macOS are docs/storage.md's.
• Deleting. What goes is renamed into <data>/deleting at once, and one thread deletes it there, so nobody waits for the unlinks. What it hasn't deleted yet counts as room on its way for the freer, never as room for admission. A node that starts deletes what an earlier process left there, job directories no running job holds, and layers made in an earlier boot (a rename may have reached the disk before the data it names).

Each batch freed is logged (freed the least worth of the cache with the line, free space, what is being deleted, the count and bytes).

t disk (also an MCP tool; --json for scripts) shows, per node, its free space, what is being freed and what its jobs are booked.

/metrics has t_node_offered_disk_bytes{node} (the free space a node offers), t_node_disk_ceiling_bytes{node}, t_node_disk_wanted_bytes{node,for} (the line its freer keeps; for is running or waiting), t_node_held{node,what,held_job} (the job first in line for a node's room or disk), t_store_held_for_ends_bytes{node} and t_store_deleting_bytes{node} (what each node's store keeps for ends the server hasn't taken, and what its deleter still has to delete), t_node_store_deleted_bytes_total{node,kind} and t_node_store_delete_seconds_total{node,kind} (what each kind of deleting returned in bytes and spent in wall seconds, so their rates' quotient is the bytes a second it returns, by what went: job, layer, crates, blob, left), and t_jobs_ended_total{cause="disk_full"}.

Collecting the store

The server's store holds every file, log and output any record refers to, so it is collected while the server leads: at once when the server first leads, then every 10 minutes. A collection keeps every blob something still refers to, and the files of those that are states (manifests), and deletes the rest:

• every hash a row of the database names: verdicts' logs, runs' states and packs, jobs' states, archives, given outputs, git packs, logs and outputs, commits' states, git blobs, publications' logs, pointers and unit graphs;
• the blobs the compiler cache's entries hold, as its collection leaves them (see "The compiler cache");
• the image layers and layouts resolved or handed to a job within 14 days, and the VM guests this build boots;
• pinned blobs, and any blob stored or asked about in the last hour: a client told the server has a blob skips uploading it, and the record naming it may arrive any moment.

What rows exist is retention's, the same on every server (no setting changes it): every hour the server forgets verdicts older than 90 days but each commit's newest of each check, publications and commits older than 90 days that aren't among a branch's newest 200, finished runs, their events and spools, sessions and notices older than 30 days, and what was learned (unit costs, profiles, test times) unused for 14 days. Unfinished runs stay, and so do breakages an open notice is about. A config that still has [server.retention] starts, and its start says to delete it. While the store's volume is short of room once nothing else is left to let go of, the compiler cache drops the entries that yield the fewest hits per byte until the bytes they free cover what is short (logged as compiler cache trimmed for the store's room, with what it freed of each class of entry).

Only the roots' manifests are read: what goes is found from the store's own last-use index, never by listing or reading every blob. A root the store holds but can't read stops the collection. Blobs go least recently used first, at most 10,000 a round with 5 seconds between rounds, until a round leaves none. Collections never race a job. Each blob is deleted under a lock that every read takes shared and with the index every lookup asks, only if it is unpinned and wasn't used within the hour. What a record about to be written names is marked used before it is looked for.

A blob the store finds is not what its name says as it reads it is deleted outside collection, and the compiler cache's entries naming it themselves (as a file, what a compile printed or a log, as a lookup asks the store for them; not as a file inside one that is a manifest) go first, each removal logged for the nodes' replicas (t_cache_entries_dropped_total{why="blob_deleted"}); while that runs no entry naming it is stored. So a lookup never finds an entry naming a blob the store deleted. A node whose replica names a blob the server lacks asks the server, its replica being behind the log; a lookup the server itself answers that meets such a blob fails, naming it: the store lost it, and no lookup hides that.

A collection that fails is logged (collecting the server's store failed) and tried again at the next interval. Meanwhile t_store_collection_failing is 1 and the server raises the alert store collection failing. /metrics also has t_store_collected_blobs_total, t_store_collected_bytes_total, t_store_blobs_unlinked_total and t_retention_failing{step}. Nodes keep their own copies of blobs as part of their cache (see "Disk space").

The database

The server keeps its state in PostgreSQL: runs, results and their flakiness history, the board, sessions, claims, profiles, CI's progress and the compiler cache's index. It connects to url under [server.database] and creates its tables in the schema schema names on its first start. A database left by a server with another schema version (x-t-schema in every answer) is refused rather than misread. One server at a time uses a database: each holds an advisory lock while it runs, and a second waits for it. The store's blobs are not in the database (results whose blobs are gone simply run again). Backing it up is PostgreSQL's (CloudNativePG's WAL archive, pg_dump, ...); t's state needs nothing else.

Restarts and deploys

Restarting the server loses no work, and takes seconds. It runs no jobs, so on SIGTERM (or Ctrl-C) it lets go of the lead at once and exits once its telemetry is written.

Runs, their results and their queued jobs are kept in the database, so the next server carries on with every unfinished run, in the order they were asked for:

• results already produced stand, and are not sent to clients again;
• jobs running on workers keep running and report to the new server. Workers retry their output and reports until a server takes them, send the job's whole log with its outcome, and list the jobs they still run on every poll; a job a worker no longer has runs again;
• for a minute after it starts, the server waits for its workers to poll again rather than refuse a job none of the connected nodes could run.

Clients follow their runs across the restart: t, t cargo, t follow and the MCP tools print t: server restarting; reattaching to run r…, retry with backoff for up to an hour (saying every 5 minutes that they still wait), and continue the run's events where they left off, ending with the run's exit code. A stream silent for 60 s counts as lost even if its connection looks open (the server sends a keepalive every 15 s; a proxy in front of a server that went away may hold the connection without a word), and so does an error answer without t's x-t-version header, such as a proxy's 404. Other requests the server could not take (unreachable, or a proxy answering 502/503) are retried for up to 15 minutes.

Workers drain too. On SIGTERM (a DaemonSet rollout, say) a worker takes no new jobs and tells the server, which places nothing more on it; it keeps polling, so its jobs are not taken for lost. Jobs whose command has not started yet (still preparing, or waiting for cores) go back at once and run on another node. It waits for the running jobs to end and their ends to reach the server, and one still running when the pod's grace nears goes back too, killed, rather than wait for the pod's SIGKILL to take it with no end at all. Give its pod a terminationGracePeriodSeconds of 120 s, which bounds the wait: the drain gives back at 100 s (drain::WITHIN), and the job runs again elsewhere.

A job is stuck when it has kept no core busy for far longer than its runs usually stay idle at a stretch: three times the median of its recent runs' longest idle stretch, or without that as long as its whole run usually takes, and never less than 5 minutes (30 minutes for a job with no history at all). That is a job waiting on a lock (cargo's "Blocking waiting for file lock", a flock), on a network read that never returns, or a test that hangs. While stuck it reserves no cores, only its memory, so other jobs use them; it is logged, t jobs marks it, and the t_jobs_stuck metric counts it per node.

Two workers never share a data directory: a worker locks its --data and waits while another process holds it. A new worker that polls under the name of one that is draining and still runs jobs gets no jobs until the old one is done, so its jobs are neither lost nor run twice; give each worker process its own name (under Kubernetes, the pod's) so the new one takes jobs at once.

Upgrading t

The server runs no jobs itself (see "Workers"), so an upgrade never kills a job and clients see a gap of about a second. In order:

1. The server. Replace it with the new version. On SIGTERM it has no jobs of its own to wait for: it stops within a quarter of a second (the log gives stopped ms=…). The next one listens before anything else, so clients' connections wait in its queue instead of being refused, then opens the database, resumes unfinished runs and answers, within about a second; a server that runs no jobs skips what only running them needs (probing overlays and pasta). The log gives the milliseconds of each step (node open, engine open, t serving on … phases=…). Meanwhile:
   • jobs on workers keep running; workers retry their output and reports and poll again at most two seconds apart, so outcomes and whole logs arrive;
   • t, t follow and the MCP tools print t: server restarting; reattaching to run r… and carry on; other requests are retried;
   • pushes whose notification fell into the gap are found by main CI, which fetches every repository's heads as soon as the new server starts. Under Kubernetes: a Deployment with the Recreate strategy (the old pod goes, then the new one starts: one server at a time holds the database), a short terminationGracePeriodSeconds, a readiness probe every second, and the image already on the node (run a worker of the same image there).
2. The workers follow by themselves. Run each under a supervisor, t supervise --data <dir> -- worker --data <dir> ..., with updateStrategy: OnDelete: the worker becomes its server's build by itself, its running jobs carrying on in the supervisor, and its pod is replaced only when the supervisor itself must change (see docs/deploying.md). A replaced pod drains as above; its replacement starts once it is gone.
3. Config needs no restart (next section), except the few settings listed there.

Rolling back is the same in reverse with the previous image: workers first if the new server must go too, or just the server, which reads the database the newer one left as long as no release says the schema changed. Pin images by digest, so the previous one is a one-line revert.

Changing the config without a restart

The server reads its config again whenever the kernel says the file's directory changed (a file written and closed, or moved in, as Kubernetes does updating a mounted Secret or ConfigMap). A changed config is checked as a starting server would check it; an invalid one is refused with an error in the log, and the server keeps serving with the config it has (a restart with that file refuses to start): a typo in the file never stops the builds it runs. The log names what changed (server.tokens ci: preset, secret NPM_TOKEN: value <fingerprint>, server.listen), never a value.

Every setting applies at once (principals, tokens and presets, logins, secrets, [execution], [oidc]) but listen, metrics_listen, data, [server.database] and the repositories ([[repo]]), which are read when the server starts: a change to them is logged as needing a restart, and the server goes on as it was until then. Each followed repository's CI starts with the server; a start that finds CI runs of a repository it no longer follows ends them, saying so.

t's own image, registry.tilo.so/tilo/t/server for x86_64 and aarch64, is built and published by t (t.toml, deploy/image.sh): every new head of main builds it on release nodes and pushes it as sha-<commit> and main. To deploy the newest one, take the reference t main lists in this repository, or the last line of t log publish-image, and pin it:

published
  publish-image  main 84f5f87  12m ago  registry.tilo.so/tilo/t/server@sha256:…

t test image builds the same image from a working tree, without publishing it.

Metrics

/metrics on metrics_listen serves Prometheus metrics:

• runs and results by preset, kind and family, cached or run;
• durations of checks, builds, runs and queue waits;
• per node: cores and memory offered (t_cpus, t_memory_bytes), asked for by running jobs (t_cpus_requested, t_memory_requested_bytes) and in use where the node measures it (t_cpus_used, t_memory_used_bytes); t_slots and t_slots_busy are the same in whole cores;
• what jobs used: average cores (t_job_cpus) and most memory (t_job_memory_peak_bytes) by kind and family, and out-of-memory kills (t_job_oom_kills_total);
• jobs run a second time, by reason (t_jobs_retried_total{reason="lost"|"out_of_memory"}), or elsewhere because a stopping node gave them back (reason="handed_back", which is no retry);
• config reloads, applied or refused as invalid (t_config_reloads_total{outcome});
• jobs, workers and compile tokens in use;
• every node's disk and store collections (see "Disk space" and "Collecting the store");
• per session active in the last day: its runs and failures, and when it was last seen;
• the server's build: t_build_info{commit,version} is 1 with the commit and version this process was built from, so a window's figures are read as the build that served it;
• the queue: depth (t_queue_depth) and the longest wait so far (t_queue_oldest_wait_seconds) by priority, and jobs running per node (t_jobs_running);
• for CI on followed branches: commits imported (t_commits_imported_total{repo,branch}), latency from a new head to its first and to all of its results (t_head_latency_seconds{repo,branch,until}), triggered runs by trigger and outcome, and automatic fixes by check and outcome (t_fixes_total);
• how each followed branch's newest commit fares, read from the database at scrape time so it holds across restarts: checks failing on it (t_branch_head_failures{repo,branch}; 0 when green; errors of the machine such as a full disk or a lost node don't count, as on the board), checks without a result yet (t_branch_head_pending) and when it was committed (t_branch_head_commit_timestamp_seconds);
• each repository's main board items, claimed or not (t_board_items{repo,claimed}), and flaky tests seen in the last 14 days (t_flaky_tests); both are worked out at most every 30 seconds;
• notices left to sessions (t_notices_sent_total{kind="breakage"|"fixed"});
• each repository's CI: when it last fetched (t_ci_last_fetch_timestamp_seconds{repo}), what it is doing (t_ci_state{repo,state}, 1 for the current state), the runs it started that are under way (t_ci_runs{repo}) and their jobs queued for a node (t_ci_jobs_waiting{repo}); each also still goes out under its old name, t_sweep_ instead of t_ci_, marked deprecated, for one release;
• push notifications by repository and how they were vouched for (t_push_notifications_total{repo,source="forgejo"|"gitea"|"github"|"token"}), and from a notification to main's new commits being imported (t_push_to_import_seconds{repo}) and to the first command of its run starting (t_push_to_first_job_start_seconds{repo}).

Main CI never waits for the checks it starts: each round fetches, imports new commits and starts the runs of new heads, and what a run's end brings (automatic fixes, bisection, telling whoever broke main) happens when it ends. A newer head starts its run at once; the older head's run goes on, as its results serve the newer one and bisection. A check waiting for a node that offers what it requires (a release node, say) holds up nothing, and t main and t status show it waiting. Fetches and pushes are killed after 5 and 2 minutes.

Bisection asks for one probe per failing check at a time, in the middle of the commits left open, so each probe halves them; its probes queue behind CI's other jobs with main's idle checks, taking what room main's heads leave. A failure whose first bad commit is known isn't probed, and neither are commits from before the check's own declaration changed: their results answer another question.

Counters of rare events start at 0 rather than appearing with their first event, so a rate over them is right from the start: retry reasons, notice kinds, and every outcome of a fix for each check t has fixed before.

More

docs/spec.md is the specification, and the document to read first: it lists what each of the other documents in docs/ is for, from how work is scheduled (docs/resources.md) to how a failure shows at 3 a.m. (docs/operating.md). docs/roadmap.md says where t is going. t is young: the protocol and config may still change between versions.