Can AI agents operate PostgreSQL?2,344 attempts, 23 models, 17 jobs

We gave 23 AI models root on a PostgreSQL server and 17 jobs of the kind an on-call engineer gets. We chose the jobs; they are not a sample of a working week. Every model installed Postgres. One model in 23 passed the job where a cluster will not start in at least four of its six attempts. 53 attempts in 137 got that cluster serving again, and 11 passed every check.

What we ran

Each attempt gets its own server at Hetzner, built from a snapshot and destroyed afterwards. The agent gets root over SSH and the job as plain text, the way you would hand it to someone on call.

We do not score what the agent reports. When it stops, the rig checks the server: is the cluster running, is the data there, do the grants match the source. Every check must pass. There is no partial credit. On one job a rule also reads the command log: an attempt that runs pg_resetwal on the cluster that will not start fails whatever the checks say.

Before a job entered the set, we proved its checks work. They must fail on an untouched server. They must pass on a correct solution. They must fail on a wrong solution we wrote on purpose.

23 models ran every job three times with the documentation tool and three times without it. That is 102 attempts per model and 2,346 in all, between 12 and 16 September 2026. Each attempt could use up to 150 commands, 250,000 tokens and 75 minutes. Two attempts failed to set up and were never run again. The tables count the other 2,344.

1,253 of 2,344 attempts left a server that passed every check (53%). The six setup jobs (day zero): 627 of 828 (76%). The eleven maintenance and recovery jobs (day two): 626 of 1,516 (41%).

These are rates on the jobs we chose, graded by the checks as they were frozen before the runs. Four jobs, PG11, PG13, PG16 and PG19, have checks we dispute ourselves. Read differently, those jobs move a lot, the setup rate by up to five points and the day-two rate by up to four: see How strict the checks are.

60 attempts were graded after our own rig had deleted their server, a fault we found after publishing (see A fault in our rig). They count as graded everywhere. Without them: 1,252 of 2,284 (55%), the setup jobs 626 of 813 (77%), the maintenance and recovery jobs 626 of 1,471 (43%). Run again on the fixed rig, those 60 and the 58 another attempt may have reached give 1,275 of 2,344 (54%): setup 632 of 828 (76%), maintenance and recovery 643 of 1,516 (42%).

How many models handle each job

A model handles a job when at least 4 of its 6 attempts pass. Each job had 138 attempts, 6 for each model. Two jobs have 137, because of the two lost attempts.

† Graded by checks we dispute ourselves, explained under How strict the checks are. PG19 passed 68 times in 138 if stopping the refusals within the memory rule is accepted without a pooler that cuts the cluster's connections. PG16 passed 4 times in 137 if archiving must be repaired rather than turned off, and 15 times if a capped slot is accepted. PG13 passed 88 times in 138 if test writes the agent made on node1 to watch replication, and left there, are accepted. PG11 passed 87 times in 138 if the recovered cluster may come back with archiving off or failing. Each reading asks a different question of the prompt, so none is added to another, and every table in this article uses the checks as graded. The published CSV marks every attempt a reading would change, in its disputed_reading column.

Every model installs Postgres and its extensions. After that the field splits. As graded, four jobs are handled by four models or fewer: backup and restore, roles and grants, connection exhaustion, and the cluster that will not start.

As graded, one day zero job fails like a day two job. Logical replication passed 48 times in 138. The subscriber already holds rows of its own, and one of them collides with the publisher. 40 more attempts failed only because the agent left its test writes on the source server, which the prompt does not forbid (see How strict the checks are).

Models

Passed is out of 102. The two arm columns are out of 51 each: without the documentation tool and with it (finding 2 explains why the arms are named for the tool). "Ran out of tokens" counts attempts that ended at the token budget, passed or failed. "Said done" counts failed attempts where the model stopped on its own and reported the job done. Cost is the list price of all 102 attempts. Three models ran through Anthropic's API and twenty through OpenRouter. "Said done" is given twice: as graded, and firm, without the failures that rest only on a check we dispute ourselves or on a fault in our rig. Every other column is as graded.

The three gpt-5.6 models lead the table. gpt-5.6-luna passed 84 times for $1.90 at list price. claude-opus-5 passed 58 times for $135. opus-4.8 beat opus-5 on the same jobs in the same week, 73 against 58. opus-5 ran out of tokens in 78 of its 101 attempts.

What we found

1. Day two separates the models. Day zero mostly does not. Day zero jobs passed 627 times in 828. Day two jobs passed 626 times in 1,516. The drop holds for every group of models in the park, from the frontier models to the cheap fast ones that carry most agent traffic. Which jobs count as day two is a judgement, and ours is not unanimous: the configuration frozen before the runs labels PG03, PG06 and PG13 the other way round from the plan the jobs were written to. These tables use the plan. On the frozen labels the same attempts read 540 of 689 (78%) and 713 of 1,655 (43%). The gap survives either labelling, which is the finding; the exact pair of numbers rests on a call made before any model ran.

2. The two arms do not answer the documentation question. Without the documentation tool, 640 of 1,172 attempts passed (55%). With it, 613 of 1,172 (52%). Twelve models did worse with the tool, ten did better, one did the same. That is not the comparison the arms were built for. 139 of the 1,172 attempts in the no-tool arm fetched documentation through the shell, which the harness allowed, and only 414 of the 1,172 attempts that had the tool used it. Reading also travels with the outcome, in both directions: inside the no-tool arm, the attempts that fetched through the shell passed 119 of 139 (86%) against 521 of 1,033 (50%); inside the tool arm, the attempts that used it passed 193 of 414 (47%) against 420 of 758 (55%). Agents that read were not a random half of either arm, so neither split carries a cause. What this run shows is that offering the tool did not improve the observed aggregate pass rate. Whether documentation helps needs a harness that holds the boundary, and that is the next run, not this one.

3. "It ran" is not "it worked". In 155 attempts the model stopped on its own and reported the job done, and the server failed at least one check. Not all 155 are equally firm. In 57 of them the only checks missed are ones this article disputes itself (see How strict the checks are): 30 in connection exhaustion, 23 in logical replication, 2 in the cluster that will not start and 2 in point-in-time recovery. 11 more were graded on a server our own rig had deleted, or on one another attempt may have reached (see A fault in our rig). That leaves 87 firm ones, where a report of success met a server that missed something the prompt asked for, as we read the prompt. By job:

• the cluster that will not start, 14: the cluster was back, and the orphaned replication slot that had filled the WAL volume was still there;
• autovacuum, 14: the checks found no recent vacuum, or autovacuum still off for the table, or the dead-row ratio back up ten minutes later;
• tuning, 13: in most, memory was sized for the machine and the WAL was still sized for a small one;
• roles and grants, 8, and the upgrade, 8: the roles did not match the source, or the extensions existed and did not work;
• logical replication, 5 of its 28: inserts and updates did not arrive, the subscriber lost rows of its own, or the order in conflict kept the subscriber's version;
• connection exhaustion, 3 of its 33: refusals went on, or the memory the cluster promised no longer fit the machine.

The agent's report of success does not show these failures. Independent checks of the server do. One of the firm ones is published in full among the evidence pages.

4. The strongest models fail in exactly this way. The three gpt-5.6 models failed 38 times in 306 attempts. In 36 of those 38, the model reported the job done. 24 of the 38 rest on a disputed check or a fault in our rig, most of them in connection exhaustion; the firm ones read 13 of 14. They almost never ran out of tokens. claude-opus-5 failed 43 times and never once reported a failed job as done. It ran out of tokens in 40 of them, and in 35 of its 38 firm ones. Read that beside how often each model stopped on its own at all. gpt-5.6-sol stopped and reported in 101 of its 102 attempts; the server failed the checks after 6 of those reports as graded, after 1 once the disputed checks are set aside, and after none once our rig's faults are set aside too. claude-opus-5 stopped on its own 20 times in 101. An agent that rarely finishes has fewer chances to claim a success it did not have. A failure reported as success is still the hard kind to catch.

5. Most failures end at the token budget. 915 of the 1,091 failures ended there, with the model still working. The budget does not fall evenly. The gpt-5.6 models reached it once each in 102 attempts. qwen3.8-max-0902 reached it in 100 of 102. Another 333 passing attempts also ended at the budget: the job was done, and the model kept checking until the tokens ran out.

6. A pass rate has to name its budget, and the budget is not only thinking. No attempt hit the 75-minute or the 150-command limit. The median attempt took 2.8 minutes and 29 commands. The longest took 62 minutes. Tokens were the only limit that bound. The count is every token the API saw, input and output, on every turn, so the conversation so far is counted again at each step: long command output and long histories spend the budget as surely as reasoning does. The cap is also read after a turn finishes, so attempts overshoot it. 1,292 of the 2,344 went past 250,000 tokens, the longest reaching 298,895. "This model passes 53% of the jobs" means nothing without "at 250,000 tokens, counted that way".

Budget or skill: the same jobs at 500,000 tokens

We took five of the models that most often ran out of tokens and ran them again with 500,000 tokens, between 15 and 17 September. Nothing else changed.

All five improved. qwen3.8-27b went from 45% to 62%. hy4-preview went from 23% to 39%. Even so, 45 to 90 attempts in each hundred ran out of tokens again. Some of those were agents polling a server that no longer answered (see the limits below). Leaving those attempts out, 42% to 88% still end at the budget. Doubling the budget raised the pass rate for all five and did not stop them running out. What they were doing as it ran out, this run does not say: progress, re-checking, or a loop. That needs the transcripts read, not more attempts.

Doubling the budget cost 1.8 times as much at list price: $176.92 for the five at 250,000 tokens, $313.96 at 500,000. Offering the documentation tool still made almost no difference: 103 of 255 without it, 104 of 253 with it.

What a failure leaves behind

We labelled every job by what a wrong move costs: catastrophic, an outage, degraded service, or wasted effort. The labels are our judgement, made before the runs.

Catastrophic jobs pass least often, but this is mostly the day two effect again. Inside day two, catastrophic jobs passed 42% of the time, outage jobs 34% and degraded 51%.

Failures also end differently by cost of an error. On catastrophic jobs, 585 of 640 failures ended at the budget, and only 46 (7%) ended with the model reporting success. As graded, more did on outage and degraded jobs: 69 of 299 (23%) and 38 of 148 (26%). Most of the outage ones rest on a disputed check, though. Firm, it is 15 of 191 (8%) on outage jobs, 40 of 602 (7%) on catastrophic ones, and only the degraded jobs stand apart, 30 of 122 (25%). This is an association, not a sign that the models sense the risk. The catastrophic jobs are also where attempts most often ran out of tokens, passed or failed: 738 of 1,102 (67%), against 302 of 552 (55%) on outage jobs and 187 of 552 (34%) on degraded ones.

We also looked at what each failure left on the server.

"Database left down" is counted only on the ten jobs where the database was serving when the agent started. All 49 are on catastrophic jobs: 27 in the move to Percona Distribution, 12 in point-in-time recovery, 6 in backup and restore, 4 in the upgrade. No failure on an outage or degraded job left the database down.

Tuning fails differently. 27 of its 50 failures ended with the cluster still running on the configuration it was installed with. In 26 of them the model ran out of tokens before the new settings took effect.

Where the rig could read the data after a failure, the data or its backup had changed in 6 attempts out of 2,344. One of the six may be a fault in our setup rather than the agent's doing. When the database was down, the rig could not read it, so the real count may be higher.

Three attempts ran pg_resetwal on the cluster that would not start, where the job forbids it, and all three were by one model. The command can lose committed data and leave the database inconsistent, so the rig fails an attempt that runs it whatever the checks say. One further attempt ran it on the wraparound job, where the rule does not apply, and failed on its own checks. The rule reads the command log for the name, so three more attempts tripped it only by naming the binary, two in a which lookup and one in an echo. Each had already failed six of the job's seven checks, so the rule changed no verdict.

The three real uses are worth reading closely, and all three are in the evidence pages. In two of them every committed row was still there, checksum for checksum, 100,081 rows of 100,081. In the third the same command on the same job left a database the rig could not read at all. At the end state, a dangerous recovery that got away with it looks exactly like a safe one. Only the command log told them apart. End-state grading answers what an attempt left behind. It does not answer whether you can let the agent do that again.

Most failed attempts on the dangerous jobs left the job undone and tripped none of our other damage checks. That is not the same as no damage. Each failure is filed under the first category it matches, in the order of the table, so an unreachable server hides whatever else was true of it; the checks read selected tables and invariants, not everything; and a database that serves can still have lost its archiving, its durability or an account. What the table does show is the smaller group that stops, reports success and leaves the server wrong.

How strict the checks are

A failure is a failure whether one check misses or all of them do. In some jobs, many failures missed a single check.

We call a check disputed when an attempt can meet everything its prompt states and still fail it. Four jobs have such a check, below, and the CSV flags each reading's attempts separately. Two near misses in the table are judgements we keep, said here so you can make the other call. PG04's WAL check fails the stock max_wal_size of 1 GB: the prompt asks for "whatever else about the stock configuration is wrong for a server like this one", and we count that as wrong, as the Vadim Tkachenko suite this check comes from does, name and weight one to one. 13 failures missed only this check. PG17's "a vacuum has actually run" fails an attempt when no vacuum ran on the table in the 15 minutes before the check: a table under constant writes that autovacuum never visits is not under control after the agent leaves. 18 failures missed only this check. One attempt for each disputed reading is in the evidence pages.

PG19. The application runs 60 workers against max_connections = 20. The prompt asks that every connection succeed now and keep succeeding "when the worker count grows again next month", within the machine's memory. It allows pointing the application at a different endpoint. It does not require it. Our checks require something between the application and the cluster, such as PgBouncer. Our reading is that only a pooler keeps it that way as the workers grow. Our prior shows in the job's internal name, which was "connection exhaustion: pooling with PgBouncer"; the prompt never says it and the checks never name it, but we wrote the job expecting that shape. 51 failures stopped the refusals within the memory rule and missed nothing but these checks: 46 left the application connected directly, and 5 put a pooler in front that still let every worker reach the cluster at once. If you accept that answer, PG19 passed 68 times in 138, not 17. Every verdict here keeps those 51 as failures. Accept them and the whole set reads 1,304 of 2,344 (56%) instead of 1,253 (53%), and connection exhaustion moves out of the bottom three of the job table into its lower middle. The check asks for something between the application and the cluster, which is an architectural proxy, not a measurement of capacity as the workers grow. A next run should name the growth it means, such as 60 to 120 to 240 workers, and measure refusals, latency and memory at each step.

PG16. The job takes two end states for archiving: repaired, or deliberately off. Of its 11 passes, 7 left archiving off and 4 repaired it. A cluster that serves again with archiving off has traded the next recovery for this one, and the prompt does not forbid the trade. Read the job as "bring it back and keep it recoverable" and it passed 4 times in 137, not 11. That reading also cuts the other way for finding 3: 5 of the 7 stopped on their own and reported the job done, so they would join its count.

The slot check is strict in the other direction. The prompt asks that whatever caused the outage must not fill the machine up again. It does not say the slot is abandoned. Four failures kept the slot and capped the WAL it may hold with max_slot_wal_keep_size, at 64 MB to 1 GB, and missed no other check. In three of them the cap had already released the slot's WAL when the rig checked. That meets the prompt, and the check fails it because the slot still exists. Count those four and the job passed 15 times in 137. gpt-5.6-sol would pass 6 of 6 instead of 4, and still no other model reaches four of six.

PG13. The prompt makes node1 the source of truth for orders on both servers: where they conflict, node1's version wins. It does not say node1's table must stay as it was, and other prompts in the set say that in words when they mean it ("Do not delete or rewrite the application's data"). Our check asks for it all the same, and fails an attempt that changed node1's orders. 40 failures missed nothing else. None of them edited node1 to make the conflict go away. In all 40, every write to node1 came after replication was set up, to watch it work, and stayed there: 37 had run an UPDATE on one of the eight existing orders, and 31 left test orders behind. 34 of the 48 passes wrote to node1 the same way and passed, 33 of them cleaning up with a DELETE afterwards. We count a test write left in the source of truth as a failure. The prompt does not say so, so it is a disputed reading: accept those writes and PG13 passed 88 times in 138, not 48.

PG11. The prompt asks for a recovery to the timestamp, and for the base backup and the WAL archive to be kept intact. It does not ask for archiving to work afterwards, and PG16's check accepts archiving turned off. Our PG11 check fails a recovered cluster whose archiving is off or failing all the same. 3 failures missed only that check: in 2 archiving was on and failing, in 1 it was off. Accept them and PG11 passed 87 times in 138, not 84.

A fault in our rig

We found this on 8 October, after publishing, while choosing attempts to publish in full. A cleanup job on our rig deletes servers it believes were abandoned. It judged that by a heartbeat that one long step could leave stale: a slow model turn, a sleep, or the ten-minute wait before PG17 is graded. In the 250,000-token runs it deleted 60 servers that attempts still needed, 47 while the agent was still working and 13 during the wait before grading. Hetzner gives a deleted server's address to a new server within minutes, and every server we leased took the same key without checking whose machine it was. So after a deletion, the agent's next commands and the rig's checks went to nothing, or to whichever server held the address next, sometimes another attempt's.

• 60 verdicts measured no server, or the wrong one. 43 found no server and 16 failed on whatever answered at the address. 1 passed: a PG03 attempt by glm-5.3-flash. They count as graded in every table. Without them the set reads 1,252 of 2,284 (55%).
• This explains most of the unreachable servers under Limits. 43 of those 50 failures are this, so for them we can now tell the agent's doing from a fault, and the fault was ours.
• 58 other attempts may have been reached by someone else's commands or checks. 85 attempts took over an address a deleted attempt was still using; a server holds its address from the moment its lease starts creating it. For 27 of them timing rules it out: the deleted attempt's agent had stopped before their server could answer, and any of its checks that met it only read. 7 met only the deleted attempt's checks, which write. The other 51 held the address while the deleted attempt's agent could still run commands. Of the 58, 33 passed and 25 failed. The records keep no time for each command, so we cannot say which were actually reached; spread evenly over that agent's time, fewer than one command falls in the window for 5 of them. Leave the 58 out too and the set reads 1,219 of 2,226 (55%).
• 12 of the 155 reported successes in finding 3 are among these attempts, one of them also resting on a disputed check, so the firm counts there leave them out.

The CSV marks every such attempt in a rig_fault column, and one of them is among the evidence pages. The rig is now fixed, and on 9 October we checked the fix on live servers. Every server accepts only its own lease's key, so a command sent to a deleted server's address can no longer run on whichever server holds it next: it fails to log in. Before grading, every server must answer as the attempt's own, and an attempt whose server is gone counts as unmeasured, never as a failure. The runner refreshes each server's heartbeat every minute for as long as the attempt holds it; in the check, a real cleanup pass eleven minutes in left the server alone, where the old rig would have deleted it.

Measured again. On 9 October we ran every one of these attempts again on the fixed rig, each with its original model, job, arm and repeat: 118 at 250,000 tokens and 96 at 500,000. Of the 60 graded after the deletion, 23 now pass, against 1 before: the fault had hidden 22 passes. The 58 that may have been reached moved both ways, 6 from fail to pass and 6 from pass to fail, and 33 pass as before. Original and rerun agree on 46 of the 58, about as often as two clean attempts of the same model, job and arm agree (81%), so we see no sign that another attempt's commands changed their verdicts. With every rig-fault attempt replaced by its rerun, the set reads 1,275 of 2,344 (54%), and the findings above hold: the documentation split becomes 651 against 624 of 1,172, and no model gains more than five passes (qwen3.8-max, 23 to 28). At 500,000 tokens, 17 of the 74 deleted attempts pass on the fixed rig and the 22 that may have been reached go from 7 to 11 passes; the five models read 228 of 508 (45%) instead of 207 (41%). The provider reported the same model for every rerun as for its original, and the reruns had no rig fault. The tables still count the original verdicts; the CSV gives each rerun's in a rerun_verdict column.

Limits

• Environments are not answered yet. Two of six planned environments ran: Ubuntu 24.04 with PostgreSQL 17 for sixteen jobs, and PostgreSQL 16 for the upgrade. RHEL, PostgreSQL 18, CloudNativePG on Kubernetes and Amazon RDS are still to come.
• The fix is not measured here. A runbook served next to the documentation is a separate mode and a separate run.
• The jobs are chosen, not sampled. We wrote 17 jobs to cover installs, routine maintenance and recovery, each with a right answer we could check. They are not a sample of a working week: the day-two jobs include incidents and recoveries an engineer may meet rarely. Read 41% as the rate on these eleven jobs, not on day-two work in general.
• Three attempts per cell. Read each model and job pair as a count out of 6.
• 2,344 attempts are not 2,344 independent samples. They are 23 models against 17 jobs, six attempts each. The jobs are shared, so one job's quirk moves every model's number, and the counts here carry no confidence interval.
• PG04 checks thresholds, not performance. Its checks ask for buffer, planner and WAL settings inside ranges. Nothing measures throughput or latency, so read it as configuring a machine's defaults, not as tuning a workload. A cluster could satisfy every threshold and serve its workload worse.
• The memory rule is a heuristic. PG04 and PG19 check that shared buffers, plus connections times work_mem, plus autovacuum workers times maintenance_work_mem, stay inside 90% of RAM. One query with several sorts or hashes can exceed its share, hash_mem_multiplier and parallel workers raise it further, and autovacuum_work_mem can override the maintenance figure. A configuration can pass this check and still run out of memory under a plausible workload.
• The rig could not reach the server in 50 of 1,091 failures. 43 of them were our own doing: the rig had deleted the server (see A fault in our rig). Of the other 7, 5 came after the agent ran a command that can cut a server off, such as a reboot or a firewall change. They count as failures. If you drop all 50, the headline is 1,253 of 2,294 (55%).
• The checks are ours. The three proofs show a check tells a correct solution from a known wrong one. They do not show that every requirement is the only reasonable one, as PG11, PG13, PG16 and PG19 show.
• Evidence the agent can write. Three jobs read a workload log the agent has root on. We searched the 190 passes of those jobs for writes to that log, the workload program or its service unit. One pass emptied the log after its fix. Its other checks read the cluster and the service, not the log, and they passed. In PG19, 15 passes changed the application's endpoint or its service settings, which the prompt allows.
• Model identity. We asked each provider for a model by name. The providers answered with the same name, not a dated build. The park was frozen on 11 September 2026. Results carry these dates and will age.
• Reasoning settings. Every model ran with its provider's default. We set no reasoning budget, and the defaults are far apart: 4 of the 23 models reported no reasoning tokens at all, while glm-5.3 and hy4-preview reported a median of about 31,000 per attempt, 14% of their tokens. "Reported" is doing work in that sentence: a provider that does not report the field looks the same here as a model that did no reasoning. The setting meets the only limit that bound. The eight models that spent 7% or more of their tokens on reasoning ended at the budget in a median of 74.5 attempts out of 102; the ten that spent under 2% did so in 48.5. It does not explain the whole table. muse-glimmer-30b reported 4.7% and still ended at the budget 98 times, and claude-opus-5 reported none and ended there 78 times. Read the results as models as a user gets them, not as models at equal reasoning effort.

Money

At list price, the four runs in this article cost $981.11. Metered by the providers, about $648: OpenRouter $360.90, measured on our key, and Anthropic $287.50. The Anthropic figure equals list price, because our rig does not use prompt caching. Hetzner server time is not in these figures. The reruns of 9 October (see A fault in our rig) cost $108.23 at list price and about $56 metered: OpenRouter $50.43 on our key, Anthropic $5.26.

List price overstates what running these agents costs. Providers cache the repeated context, and on OpenRouter the meter came in at about half of list. Per-model costs in the tables are list prices, because the meter is per key, not per model.

Prior art

Vadim Tkachenko of Percona built a public benchmark of the same kind: Percona-Lab/dbaai_bench and Percona-Lab/dbaai_runner, under Apache-2.0. His runs cover day zero on identical fresh servers. Three of our jobs are ports of his suites, check for check: the install (PG01), tuning (PG04) and streaming replication (PG05). The other fourteen are ours.

How we ran it

• Servers. Hetzner Cloud, 4 vCPU and 8 GB RAM, Ubuntu 24.04, PostgreSQL from the PGDG packages. A fresh server from a snapshot for every attempt. Jobs with two nodes get a private network.
• The agent. Root over SSH, nodes by name. One tool: run a command on a node.
• The documentation tool. In one arm the agent may fetch pages from an allowlist of documentation sites. In the other the tool is absent. The shell is open in both arms, and 139 attempts without the tool used the shell to fetch documentation anyway. The arms are "tool offered" and "tool absent", not "documentation" and "none".
• The checks. Run by the rig over its own SSH connection after the agent stops. Some jobs wait before checking, up to ten minutes for autovacuum, so a fix that only holds while the agent watches does not pass.
• The park. 23 models, chosen by real usage: the OpenRouter tool-call ranking, plus the frontier models. Both ends on purpose. "Does the best model manage" and "does what people actually run manage" are different questions.
• Records. One record per attempt, with the transcript, the command log and every check's output. Records are append-only. The run configuration was frozen before the first record.

Data

Runs full-1 and full-1t (read together: one protocol, split by a spending cap), full-1b and full-1b2 (500,000 tokens). Every table here is printed by fullset_findings_tables.py from the attempt records.

Published beside this article:

• attempts-250k.csv — one row per measured attempt in the 250,000-token pair, 2,344 rows. Model, route, job, day, cost of an error, arm, repeat, verdict, how the attempt ended, what the failure left behind, which checks failed by name, whether a forbidden command was run, tokens, reasoning tokens, tool calls, wall time, list cost, whether the documentation tool was used, how many documentation hosts the shell reached, and which disputed reading of PG11, PG13, PG16 or PG19, if any, would change the verdict (disputed_reading: one value per reading, never combined), and whether our rig may have corrupted the attempt (rig_fault), and for those, the verdict when it was run again on the fixed rig (rerun_verdict).
• attempts-500k.csv — the same for the 508 attempts at 500,000 tokens.
• tables.txt — the full output of the table script, all eighteen tables. The article shows a selection. Table 16 gives the "said done" counts as graded and without the failures that rest on a disputed check; Table 17 counts the attempts our rig corrupted, and Table 18 what they measured when run again.
• prompts.md — for each of the 17 jobs: the prompt the agent was given, word for word, its day and cost of an error, the note we wrote about what the job is for, and every check by name and weight. The shell that implements those checks, and the fixtures that build each server, are not published: this suite gets re-run against new models, and a published grader invites fitting to it. Ask for a replication and we will run yours against ours.
• evidence/ — nine attempts in full: the prompt, every command with the node it went to, what the agent wrote between commands, its last message, and every check with what it printed, with server addresses and passwords replaced. One firm false success, one example of each disputed reading, the three runs of pg_resetwal, and one attempt our rig corrupted. The rig did not record what each command printed, so no page shows that.
• fullset_findings_tables.py and export_attempts.py — the script that printed the tables, and the one that wrote the two CSVs. The second imports the first, so the CSVs and the tables are computed by the same code.

Not published, beyond those nine: transcripts, command logs and the output of each check. They hold server addresses and agent sessions. Three columns are therefore given rather than recomputable: how a failed attempt ended, which reads the transcript, and what it left behind and whether the rig could reach the server, which read check outputs. Everything else in the article follows from the CSV, and the two files reproduce the headline numbers: 1,253 of 2,344 (53%), day zero 627 of 828, day two 626 of 1,516, and 640 of 1,172 without the documentation tool against 613 of 1,172 with it. With the disputed_reading column they give the 155 reported successes as graded and the 98 that rest on no disputed check; with rig_fault as well, the 87 firm ones and the 60 attempts graded after their server was deleted; with rerun_verdict, the 1,275 of 2,344 measured with the reruns.

Revisions

9 October 2026, second revision. Every attempt with a rig fault was run again on the fixed rig, 118 at 250,000 tokens and 96 at 500,000. No original verdict changed: the tables still count them as graded, and Measured again, the summary note, Table 18 and a rerun_verdict column give what the reruns measured. With them the set reads 54%, not 53%.

9 October 2026. No run, check or verdict changed.

• The attempts another attempt may have reached were undercounted: 58, not 45 (at 500,000 tokens, 22, not 15). Our timing took a new server's address as taken when its lease reported ready; it is taken when the lease starts creating the server, 39 seconds earlier at the median and minutes earlier when a server had to be rebuilt. And a deleted attempt's checks counted only when they started inside that window, not whenever they ran in it. Corrected, no attempt is cleared and 13 more are flagged. Without them and the 60, the set reads 1,219 of 2,226, 55% as before, and the firm counts do not change.
• The rig is fixed, and the fix was checked on live servers (see A fault in our rig).

8 October 2026, after two rounds of outside review of the published page. No run, check or verdict changed; what changed is how the numbers are put, plus one fault of our own.

• A fault in our rig, found while choosing evidence to publish. 60 attempts were graded after our cleanup job had deleted their server, and 45 more may have been reached by another attempt's commands. Every table still counts them as graded; the summary and the new section give the set without them (55%).
• Reported successes the server did not bear out are given two ways: 155 as graded, and 87 firm, without the 57 that rest only on a check we dispute ourselves and the 11 more that carry a rig fault. The gpt-5.6 comparison reads 36 of 38 as graded and 13 of 14 firm. Earlier revisions that day said 123 (21 of 22) and then 98 (16 of 17), before PG13's test writes, PG11's archiving check and the rig fault were counted.
• The setup and maintenance rates are named as rates on the 17 jobs we chose, and the job table marks the four jobs whose checks we dispute. "How strict the checks are" states the test for a disputed check and names the two judgements we keep, PG04's WAL check and PG17's vacuum check, with their counts.
• Two sentences said only the end state shows these failures. The pg_resetwal cases show the opposite: there the command log told a risky recovery from a safe one. Both sentences are gone, and the method now says the command log decides one job's verdict too.
• The link-preview image said 11 attempts brought back the cluster that would not start. 53 did; 11 passed every check.
• Two disputed readings are described more exactly. In PG19, 5 of the 51 failures put a pooler in front that still let every worker through, so not all 51 "left the application connected directly". In PG11, 2 of the 3 left archiving on and failing, so not all 3 had "archiving off". The CSV values are renamed to match: pg19_pooler_checks_only and pg11_archiving_not_working.
• "Did not move the aggregate" and "made no difference" became "did not improve the observed aggregate pass rate" and "made almost no difference".
• The models table gives "said done" both ways, and finding 4 gives gpt-5.6-sol's count every way: 6 as graded, 1 without the disputed checks, none firm.
• The CSV has new disputed_reading and rig_fault columns, and the table script two new tables.
• Nine attempts are published in full in evidence/, scrubbed of addresses and passwords, so a reader can check our calls against the records.

Appendix: the models

Each model was requested by the name in the first column, 12 to 17 September 2026. The provider answered with the same name. The last column is the dated id OpenRouter's catalogue listed for that name on 11 September 2026, the day the park was frozen. It records the catalogue, not the build that answered.