# Full set at 250,000 tokens: full-1 + full-1t

Planned 2346 · definitive 2344 (full-1 1986, full-1t 358) · passed 1253 = 53% · never measured 2
  - vm-ubuntu2404-pg16:pg06_pg_upgrade_16_17:v1:claude-opus-5:doc:r1 (full-1t: error, LeaseError: lease.py create --cell failed (1): …ithout-ipv6 --ssh-key pgeval --u)
  - vm-ubuntu2404-pg17:pg16_cluster_will_not_start:v1:google/gemini-3.1-pro-preview:none:r1 (full-1t: error, FixtureError: fixture step apply.sh on node1 exited 1: [fx 08:33:52] conf: archi)

## Table 1. Models (17 jobs × 2 arms × 3 repeats = 102 attempts each)

| Model | Route | Tier | Passed | % | No docs | With docs | Token cap | Said done, server disagreed | ... firm (no disputed check, no rig fault) | Median tokens | $ list |
|---|---|---|---|---|---|---|---|---|---|---|---|
| openai/gpt-5.6-sol | openrouter | spine | **95/102** | 93% | 47/51 | 48/51 | 1 | 6 | 0 | 65,267 | 20.08 |
| openai/gpt-5.6-terra | openrouter | anchor | **89/102** | 87% | 44/51 | 45/51 | 1 | 13 | 5 | 74,990 | 25.22 |
| openai/gpt-5.6-luna | openrouter | core | **84/102** | 82% | 40/51 | 44/51 | 1 | 17 | 8 | 55,987 | 1.90 |
| google/gemini-3.1-pro-preview | openrouter | spine | **81/101** | 80% | 41/50 | 40/51 | 49 | 2 | 1 | 250,148 | 42.05 |
| claude-opus-4-8 | anthropic_direct | anchor | **73/102** | 72% | 37/51 | 36/51 | 33 | 8 | 3 | 182,503 | 101.95 |
| z-ai/glm-5.2 | openrouter | core | **72/102** | 71% | 38/51 | 34/51 | 39 | 8 | 4 | 215,527 | 12.34 |
| moonshotai/kimi-k3 | openrouter | open_tail | **68/102** | 67% | 32/51 | 36/51 | 38 | 14 | 6 | 183,446 | 42.68 |
| tencent/hy3 | openrouter | core | **65/102** | 64% | 34/51 | 31/51 | 35 | 8 | 1 | 187,426 | 2.14 |
| google/gemini-3.7-flash | openrouter | core | **64/102** | 63% | 31/51 | 33/51 | 61 | 0 | 0 | 251,751 | 16.85 |
| xiaomi/mimo-v2.5 | openrouter | core | **64/102** | 63% | 34/51 | 30/51 | 48 | 7 | 3 | 250,822 | 3.10 |
| claude-sonnet-5 | anthropic_direct | spine | **63/102** | 62% | 34/51 | 29/51 | 58 | 3 | 0 | 254,030 | 50.44 |
| claude-opus-5 | anthropic_direct | spine | **58/101** | 57% | 33/51 | 25/50 | 78 | 0 | 0 | 257,908 | 135.12 |
| z-ai/glm-5.3-flash | openrouter | core | **55/102** | 54% | 28/51 | 27/51 | 67 | 0 | 0 | 255,946 | 4.51 |
| google/gemma-4-31b-it | openrouter | core | **51/102** | 50% | 25/51 | 26/51 | 20 | 30 | 29 | 76,662 | 1.05 |
| qwen/qwen3.8-27b | openrouter | open_tail | **46/102** | 45% | 22/51 | 24/51 | 65 | 4 | 1 | 256,570 | 16.60 |
| minimax/minimax-m3 | openrouter | open_tail | **43/102** | 42% | 20/51 | 23/51 | 65 | 7 | 3 | 253,405 | 7.35 |
| deepseek/deepseek-v4-pro-0813 | openrouter | spine | **39/102** | 38% | 24/51 | 15/51 | 80 | 4 | 2 | 257,195 | 16.07 |
| qwen/qwen3-coder-plus | openrouter | open_tail | **38/102** | 37% | 20/51 | 18/51 | 41 | 21 | 19 | 198,905 | 12.47 |
| deepseek/deepseek-v4-flash-0731 | openrouter | core | **26/102** | 25% | 13/51 | 13/51 | 82 | 3 | 2 | 259,715 | 1.87 |
| qwen/qwen3.8-max-0902 | openrouter | spine | **23/102** | 23% | 13/51 | 10/51 | 100 | 0 | 0 | 263,184 | 66.74 |
| tencent/hy4-preview | openrouter | core | **23/102** | 23% | 10/51 | 13/51 | 91 | 0 | 0 | 258,249 | 28.50 |
| z-ai/glm-5.3 | openrouter | open_tail | **18/102** | 18% | 8/51 | 10/51 | 97 | 0 | 0 | 261,375 | 49.02 |
| meta/muse-glimmer-30b | openrouter | open_tail | **15/102** | 15% | 12/51 | 3/51 | 98 | 0 | 0 | 255,070 | 8.81 |

## Table 2. Jobs (23 models × 2 arms × 3 repeats = 138 attempts each)

A model handles a job when at least 4 of its 6 attempts pass.

| Job | Day | Cost of an error | Structure | Passed | % | Models that handle it | 6 of 6 | 0 of 6 | Token cap | Said done, server disagreed |
|---|---|---|---|---|---|---|---|---|---|---|
| PG01 Install Postgres from the vendor apt repo + app db + scoped  | zero | degraded | single-function | **132/138** | 96% | 23 of 23 | 18 | 0 | 31 | 0 |
| PG02 Install and manage extensions (which are bundled vs external | zero | wasted-effort | single-function | **134/138** | 97% | 23 of 23 | 19 | 0 | 21 | 2 |
| PG03 Configure pg_stat_monitor (not pg_stat_statements) | zero | degraded | multi-step | **114/138** | 83% | 19 of 23 | 14 | 1 | 42 | 3 |
| PG04 Tune a fresh cluster for its machine (shared_buffers, work_m | zero | degraded | multi-step | **88/138** | 64% | 15 of 23 | 6 | 3 | 46 | 13 |
| PG05 Streaming replication across two servers over a private netw | zero | outage | multi-step | **111/138** | 80% | 19 of 23 | 12 | 2 | 62 | 0 |
| PG06 Major version upgrade with pg_upgrade (16 -> 17), extensions | two | catastrophic | version-trap | **75/137** | 55% | 12 of 23 | 7 | 7 | 98 | 8 |
| PG08 Back up and restore with pgBackRest (repo + stanza + full/in | two | catastrophic | multi-step | **35/138** | 25% | 4 of 23 | 3 | 13 | 115 | 0 |
| PG10 Migrate community PostgreSQL to Percona Distribution (packag | two | catastrophic | multi-step | **51/138** | 37% | 7 of 23 | 2 | 8 | 109 | 2 |
| PG11 Point-in-time recovery to a timestamp from a WAL archive | two | catastrophic | multi-step | **84/138** | 61% | 15 of 23 | 6 | 5 | 75 | 5 |
| PG12 Migrate roles, passwords and grants between clusters | two | catastrophic | multi-step | **28/138** | 20% | 2 of 23 | 1 | 12 | 112 | 10 |
| PG13 Logical replication: publication/subscription, slot lifecycl | zero | outage | orchestration | **48/138** | 35% | 5 of 23 | 0 | 6 | 75 | 28 |
| PG14 Diagnose and repair broken replication: slot bloat, WAL rete | two | outage | diagnostic | **77/138** | 56% | 11 of 23 | 7 | 5 | 73 | 8 |
| PG15 Online schema change without holding ACCESS EXCLUSIVE (CONCU | two | catastrophic | multi-step | **103/138** | 75% | 17 of 23 | 11 | 2 | 39 | 1 |
| PG16 Cluster will not start: corrupt pg_control, bad permissions, | two | catastrophic | diagnostic | **11/137** | 8% | 1 of 23 | 0 | 16 | 114 | 16 |
| PG17 Autovacuum is not keeping up: a bloating table under sustain | two | degraded | multi-step | **70/138** | 51% | 9 of 23 | 5 | 4 | 68 | 22 |
| PG18 Transaction ID wraparound: cluster approaching a forced shut | two | catastrophic | diagnostic | **75/138** | 54% | 11 of 23 | 8 | 7 | 76 | 4 |
| PG19 Connection exhaustion: pooling with PgBouncer | two | outage | multi-step | **17/138** | 12% | 2 of 23 | 1 | 18 | 92 | 33 |

## Table 3. Day zero against day two (labels from the 2026-09-01 plan)

| Tier | Day zero | Day two |
|---|---|---|
| anchor | 65/72 (90%) | 97/132 (73%) |
| spine | 171/216 (79%) | 188/394 (48%) |
| core | 257/324 (79%) | 247/594 (42%) |
| open_tail | 134/216 (62%) | 94/396 (24%) |
| **whole park** | 627/828 (76%) | 626/1516 (41%) |

For the record, with the labels as frozen in config/full-1.json (PG03=two, PG06=zero, PG13=two): day zero 540/689 (78%), day two 713/1655 (43%).

## Table 4. How the failures ended

| Ending | Attempts | Share of failures |
|---|---|---|
| token cap | 915 | 84% |
| said the job was done | 155 | 14% |
| stopped mid-work | 15 | 1% |
| turn_cap | 4 | 0% |
| said it could not go on | 2 | 0% |
| **all failures** | **1091** | |

All 2344 attempts by stop reason: token_cap 1248, completed 1092, turn_cap 4

## Table 5. Arms: the documentation tool absent against the tool offered (the shell is open in both)

| Arm | Passed | Attempts | % | Token cap |
|---|---|---|---|---|
| tool absent | 640 | 1172 | 55% | 621 |
| tool offered | 613 | 1172 | 52% | 627 |

Per model: tool offered lower for 12, higher for 10, equal for 1 of 23.
Tool absent: 139 of 1172 attempts fetched documentation through the shell and passed 119/139 (86%); the other 1033 passed 521/1033 (50%).
Tool offered: 414 of 1172 used the tool and passed 193/414 (47%); the other 758 passed 420/758 (55%). Counting the shell too, 462 of 1172 read documentation.

## Table 6. RQ3 — cost of an error

| Cost of an error | Jobs | Passed | % | Failures | Token cap | Said done, server disagreed | Share of failures |
|---|---|---|---|---|---|---|---|
| catastrophic | 8 (PG06 PG08 PG10 PG11 PG12 PG15 PG16 PG18) | 462/1102 | 42% | 640 | 585 | 46 | 7% |
| outage | 4 (PG05 PG13 PG14 PG19) | 253/552 | 46% | 299 | 223 | 69 | 23% |
| degraded | 4 (PG01 PG03 PG04 PG17) | 404/552 | 73% | 148 | 105 | 38 | 26% |
| wasted-effort | 1 (PG02) | 134/138 | 97% | 4 | 2 | 2 | 50% |

Day two only, so the comparison is not just day zero against day two in disguise:

| Cost of an error (day two) | Jobs | Passed | % | Said done, server disagreed, of failures |
|---|---|---|---|---|
| catastrophic | PG06 PG08 PG10 PG11 PG12 PG15 PG16 PG18 | 462/1102 | 42% | 46/640 (7%) |
| outage | PG14 PG19 | 94/276 | 34% | 41/182 (23%) |
| degraded | PG17 | 70/138 | 51% | 22/68 (32%) |

All attempts, passed or failed, that ended at the token budget: catastrophic 738/1102 (67%), outage 302/552 (55%), degraded 187/552 (34%), wasted-effort 21/138 (15%).

By structure:

| Structure | Jobs | Passed | % |
|---|---|---|---|
| single-function | PG01 PG02 | 266/276 | 96% |
| multi-step | PG03 PG04 PG05 PG08 PG10 PG11 PG12 PG15 PG17 PG19 | 701/1380 | 51% |
| diagnostic | PG14 PG16 PG18 | 163/413 | 39% |
| orchestration | PG13 | 48/138 | 35% |
| version-trap | PG06 | 75/137 | 55% |

## Table 7. Failures that said the job was done: the first check that failed

| Job | First failed check | Attempts |
|---|---|---|
| PG19 | the application no longer connects straight to the cluster | 27 |
| PG13 | the publisher's table was not edited to make the conflict go away | 23 |
| PG16 | the slot that held the WAL is gone | 16 |
| PG04 | the write-ahead log has room between checkpoints | 11 |
| PG17 | a vacuum has actually run on app.events recently | 9 |
| PG06 | extensions work, not just exist | 8 |
| PG14 | no replication slot retains WAL without a consumer | 6 |
| PG17 | autovacuum will keep running on app.events | 6 |
| PG12 | the application roles exist on node2 with the same attributes as on node1 | 4 |
| PG12 | the privilege matrix on node2 equals node1's | 4 |
| PG17 | the cluster answers | 4 |
| PG18 | the protection is on again | 4 |
| PG19 | the cluster is not carrying every worker at once | 4 |
| PG11 | app.payments is the table as it stood at the timestamp | 3 |
| PG13 | an insert on the publisher arrives at the subscriber | 3 |
| PG17 | dead-tuple ratio on app.events is under control ten minutes later | 3 |

(rows with fewer than 3 attempts omitted; 155 attempts in all)

## Table 8. 500,000 tokens: full-1b + full-1b2, beside the 250k pair

| Model | 250k passed | 250k % | 500k passed | 500k % | Still at the cap at 500k | $ list 250k | $ list 500k |
|---|---|---|---|---|---|---|---|
| qwen/qwen3.8-27b | 46/102 | 45% | **63/102** | 62% | 45 of 102 | 16.60 | 24.45 |
| deepseek/deepseek-v4-pro-0813 | 39/102 | 38% | **46/101** | 46% | 77 of 101 | 16.07 | 28.29 |
| tencent/hy4-preview | 23/102 | 23% | **39/101** | 39% | 80 of 101 | 28.50 | 49.31 |
| qwen/qwen3.8-max-0902 | 23/102 | 23% | **30/102** | 29% | 89 of 102 | 66.74 | 122.37 |
| z-ai/glm-5.3 | 18/102 | 18% | **29/102** | 28% | 90 of 102 | 49.02 | 89.54 |

500k, arm none: 103/255 (40%)
500k, arm doc: 104/253 (41%)

The five together: 508 attempts at 500k; $176.92 at list at 250k, $313.96 at 500k (1.8 times).
At 500k, ended at the token budget once failures the rig could not reach are left out: deepseek/deepseek-v4-pro-0813 74/98 (76%), qwen/qwen3.8-27b 39/93 (42%), z-ai/glm-5.3 85/97 (88%), tencent/hy4-preview 67/88 (76%), qwen/qwen3.8-max-0902 65/78 (83%).

## Table 9. Environment cells

| Cell | Jobs | Passed | Attempts |
|---|---|---|---|
| vm-ubuntu2404-pg16 | PG06 | 75 | 137 |
| vm-ubuntu2404-pg17 | PG01 PG02 PG03 PG04 PG05 PG08 PG10 PG11 PG12 PG13 PG14 PG15 PG16 PG17 PG18 PG19 | 1178 | 2207 |

## Table 10. Provenance: what each route answered, and when

| Model asked for | Route | Answered as (attempts) |
|---|---|---|
| claude-opus-4-8 | anthropic_direct | claude-opus-4-8 (102) |
| claude-opus-5 | anthropic_direct | claude-opus-5 (101) |
| claude-sonnet-5 | anthropic_direct | claude-sonnet-5 (102) |
| deepseek/deepseek-v4-flash-0731 | openrouter | deepseek/deepseek-v4-flash-0731 (102) |
| deepseek/deepseek-v4-pro-0813 | openrouter | deepseek/deepseek-v4-pro-0813 (203) |
| google/gemini-3.1-pro-preview | openrouter | google/gemini-3.1-pro-preview (101) |
| google/gemini-3.7-flash | openrouter | google/gemini-3.7-flash (102) |
| google/gemma-4-31b-it | openrouter | google/gemma-4-31b-it (102) |
| meta/muse-glimmer-30b | openrouter | meta/muse-glimmer-30b (102) |
| minimax/minimax-m3 | openrouter | minimax/minimax-m3 (102) |
| moonshotai/kimi-k3 | openrouter | moonshotai/kimi-k3 (102) |
| openai/gpt-5.6-luna | openrouter | openai/gpt-5.6-luna (102) |
| openai/gpt-5.6-sol | openrouter | openai/gpt-5.6-sol (102) |
| openai/gpt-5.6-terra | openrouter | openai/gpt-5.6-terra (102) |
| qwen/qwen3-coder-plus | openrouter | qwen/qwen3-coder-plus (102) |
| qwen/qwen3.8-27b | openrouter | qwen/qwen3.8-27b (204) |
| qwen/qwen3.8-max-0902 | openrouter | qwen/qwen3.8-max-0902 (204) |
| tencent/hy3 | openrouter | tencent/hy3 (102) |
| tencent/hy4-preview | openrouter | tencent/hy4-preview (203) |
| xiaomi/mimo-v2.5 | openrouter | xiaomi/mimo-v2.5 (102) |
| z-ai/glm-5.2 | openrouter | z-ai/glm-5.2 (102) |
| z-ai/glm-5.3 | openrouter | z-ai/glm-5.3 (204) |
| z-ai/glm-5.3-flash | openrouter | z-ai/glm-5.3-flash (102) |

full-1: 2013 attempts, 2026-09-12 to 2026-09-16, $413.48 at list
full-1t: 360 attempts, 2026-09-16 to 2026-09-16, $253.35 at list
full-1b: 306 attempts, 2026-09-15 to 2026-09-16, $142.28 at list
full-1b2: 204 attempts, 2026-09-15 to 2026-09-17, $172.00 at list

## Table 11. Failures that said the job was done, by job

PG19 33, PG13 28, PG17 22, PG16 16, PG04 13, PG12 10, PG14 8, PG06 8, PG11 5, PG18 4, PG03 3, PG02 2, PG10 2, PG15 1 · 155 in all

## Table 12. RQ3 — what a failure left behind

Availability is judged only on jobs whose database was serving at handover (PG06 PG08 PG10 PG11 PG14 PG15 PG17 PG18 PG19). A data change counts only when the check read a different value, not when it could not read.

| Cost of an error | Failures | job not done, nothing else detected | database left down, read-only or not restarting | data changed | server unreachable when checked |
|---|---|---|---|---|---|
| catastrophic | 640 | 569 (89%) | 49 (8%) | 4 (1%) | 18 (3%) |
| outage | 299 | 286 (96%) | 0 (0%) | 1 (0%) | 12 (4%) |
| degraded | 148 | 127 (86%) | 0 (0%) | 1 (1%) | 20 (14%) |
| wasted-effort | 4 | 4 (100%) | 0 (0%) | 0 (0%) | 0 (0%) |
| **all** | **1091** | 986 | 49 | 6 | 50 |

By job:

| Job | Failures | job not done, nothing else detected | database left down, read-only or not restarting | data changed | server unreachable when checked |
|---|---|---|---|---|---|
| PG01 | 6 | 5 | 0 | 0 | 1 |
| PG02 | 4 | 4 | 0 | 0 | 0 |
| PG03 | 24 | 22 | 0 | 0 | 2 |
| PG04 | 50 | 45 | 0 | 0 | 5 |
| PG05 | 27 | 24 | 0 | 0 | 3 |
| PG06 | 62 | 57 | 4 | 0 | 1 |
| PG08 | 103 | 94 | 6 | 1 | 2 |
| PG10 | 87 | 57 | 27 | 0 | 3 |
| PG11 | 54 | 41 | 12 | 1 | 0 |
| PG12 | 110 | 109 | 0 | 0 | 1 |
| PG13 | 90 | 84 | 0 | 1 | 5 |
| PG14 | 61 | 58 | 0 | 0 | 3 |
| PG15 | 35 | 29 | 0 | 1 | 5 |
| PG16 | 126 | 120 | 0 | 1 | 5 |
| PG17 | 68 | 55 | 0 | 1 | 12 |
| PG18 | 63 | 62 | 0 | 0 | 1 |
| PG19 | 121 | 120 | 0 | 0 | 1 |

Measured data changes: PG13 google/gemma-4-31b-it doc r1; PG11 z-ai/glm-5.3-flash doc r3; PG16 tencent/hy3 doc r2; PG17 z-ai/glm-5.2 doc r1; PG08 tencent/hy4-preview doc r2; PG15 moonshotai/kimi-k3 none r1
PG16: attempts that ran pg_resetwal as a command: 3 (qwen/qwen3-coder-plus); attempts that only named it: 3.
  ran: qwen/qwen3-coder-plus none r1 fail · every committed row is still there: payments_checksum=243fb6bf783168e341e128a6365579f4 expected=243fb6bf783168e341e128a6365579f4 rows=100081 expected_rows=100081
  ran: qwen/qwen3-coder-plus none r3 fail · every committed row is still there: payments_checksum=243fb6bf783168e341e128a6365579f4 expected=243fb6bf783168e341e128a6365579f4 rows=100081 expected_rows=100081
  ran: qwen/qwen3-coder-plus doc r3 fail · every committed row is still there: payments_checksum=none expected=243fb6bf783168e341e128a6365579f4 rows=none expected_rows=100081
  named: qwen/qwen3.8-max-0902 none r2 failed 6 of 7 checks · `which pg_filedump pg_filedump17 psql pg_ctl pg_resetwal initdb 2>&1`
  named: qwen/qwen3.8-max-0902 none r3 failed 6 of 7 checks · `which pg_controldata pg_resetwal rsync tar resize2fs truncate 2>&1`
  named: tencent/hy4-preview doc r2 failed 6 of 7 checks · `echo "=== check for pg_resetwal leftovers / backup_label ==="`
PG18: attempts that ran pg_resetwal as a command: 1 (minimax/minimax-m3); attempts that only named it: 0.
  ran: minimax/minimax-m3 doc r1 fail · every committed row is still there: (no such check on this job)

## Table 13. Near misses: failures on exactly one check

| Job | Failures | On one check | The check (count) |
|---|---|---|---|
| PG01 | 6 | 0 |  |
| PG02 | 4 | 1 | the other databases were left alone (1) |
| PG03 | 24 | 2 | the buckets are the size and the number that were asked for (2) |
| PG04 | 50 | 17 | the write-ahead log has room between checkpoints (13); the app database and its role came through it (2) |
| PG05 | 27 | 0 |  |
| PG06 | 62 | 1 | writable, and the application still gets in (1) |
| PG08 | 103 | 59 | app.payments is back to its pre-incident state (50); the drill ran and its record survived (9) |
| PG10 | 87 | 3 | Percona's own extension is available to create (2); the community server package is gone (1) |
| PG11 | 54 | 4 | archiving is still on for the next incident (3); app.payments is the table as it stood at the timestamp (1) |
| PG12 | 110 | 4 | default privileges identical to node1's (2); the privilege matrix on node2 equals node1's (1) |
| PG13 | 90 | 41 | the publisher's table was not edited to make the conflict go away (40); the conflicting row was resolved toward the publisher (1) |
| PG14 | 61 | 3 | the credential rotation on the primary stands (1); the standby serves reads (1) |
| PG15 | 35 | 6 | foreign key events.account_id -> accounts.id exists and is validated (4); the application saw no errors and no latency spike in the window (2) |
| PG16 | 126 | 27 | the slot that held the WAL is gone (24); every committed row is still there (1) |
| PG17 | 68 | 27 | a vacuum has actually run on app.events recently (18); autovacuum will keep running on app.events (9) |
| PG18 | 63 | 20 | the protection is on again (19); the cluster is out of wraparound danger (1) |
| PG19 | 121 | 51 | the application no longer connects straight to the cluster (46); the cluster is not carrying every worker at once (5) |

PG19: 51 failures missed only the pooler-shaped checks (no refusals, memory rule kept): 46 with the application still connected directly, 5 through a pooler that still let every worker reach the cluster at once. Read the prompt as not requiring a pooler and PG19 is 68/138, not 17/138.
PG13: 40 failures missed only 'the publisher's table was not edited to make the conflict go away': 31 added or removed rows in the source table, 9 changed values in it.
PG13: writes to node1's orders table in those 40: verify 40 (verify = every write after the subscription was created; dodge = rows deleted or truncated, or order 7 changed before it). Of the 48 passes, 34 also wrote to node1 after the subscription and passed, 33 of them deleting rows afterwards. Accept writes made to watch replication and left on node1, and PG13 is 88/138, not 48/138.
PG13: 37 of those 40 ran an UPDATE on node1 naming one of the eight seeded orders by id (22 of the 23 that said the job was done). Of the 31 with a different row count, 31 had more rows than the seed's 8.
Whole set under the PG19 reading: 1304/2344 (56%), not 1253/2344 (53%).
PG16: of its 11 passes, 7 left archiving off and 4 repaired it. 24 failures missed only 'the slot that held the WAL is gone'; 4 of them kept the slot and capped it with max_slot_wal_keep_size (3 with its WAL already released when checked). Count those and PG16 is 15/137, not 11/137; models at 4 of 6 or more: openai/gpt-5.6-sol 6.
PG11: 3 failures missed only 'archiving is still on for the next incident' (2 of them said done): archiving on and failing in 2, off in 1. Accept a recovered cluster whose archiving does not work and PG11 is 87/138, not 84/138.
PG04, kept as graded: 13 failures missed only 'the write-ahead log has room between checkpoints' (10 said done).
PG17, kept as graded: 18 failures missed only 'a vacuum has actually run on app.events recently' (9 said done).
PG16: 53 of 137 attempts left the cluster running, starting at boot and taking writes, 52 of them with every committed row; 11 passed. Of the 42 back in service that failed, 38 still had the slot. The other 84 never brought it back; 84 of them ended at the token budget.

## Table 14. Attempts the rig could not reach when it checked the server

250k pair: 50 of 1091 failures (10 of them after a command that can cut a server off: reboot, firewall, network or ssh change). Counted as failures in every table above.
Counted as unmeasured instead: 1253/2294 (55%) against 1253/2344 (53%).

| Model | Unreachable failures | Passed if those are left out |
|---|---|---|
| tencent/hy4-preview | 8 | 23/94 (24%) |
| z-ai/glm-5.3-flash | 7 | 55/95 (58%) |
| qwen/qwen3.8-max-0902 | 7 | 23/95 (24%) |
| xiaomi/mimo-v2.5 | 6 | 64/96 (67%) |
| deepseek/deepseek-v4-flash-0731 | 5 | 26/97 (27%) |
| moonshotai/kimi-k3 | 5 | 68/97 (70%) |
| qwen/qwen3.8-27b | 4 | 46/98 (47%) |
| deepseek/deepseek-v4-pro-0813 | 2 | 39/100 (39%) |
| z-ai/glm-5.3 | 2 | 18/100 (18%) |
| google/gemma-4-31b-it | 1 | 51/101 (50%) |
| tencent/hy3 | 1 | 65/101 (64%) |
| qwen/qwen3-coder-plus | 1 | 38/101 (38%) |
| openai/gpt-5.6-terra | 1 | 89/101 (88%) |

Every wave, for the record (counted here only, never merged):
  wave-1: 12 of 1188 definitive attempts (2 after a lockout-shaped command)
  wave-1c: 29 of 540 definitive attempts (4 after a lockout-shaped command)
  wave-1b: 39 of 648 definitive attempts (6 after a lockout-shaped command)
  full-1: 49 of 1986 definitive attempts (10 after a lockout-shaped command)
  full-1t: 1 of 358 definitive attempts (0 after a lockout-shaped command)
  full-1b: 17 of 305 definitive attempts (3 after a lockout-shaped command)
  full-1b2: 37 of 203 definitive attempts (12 after a lockout-shaped command)
500k, per model: deepseek/deepseek-v4-pro-0813 3, qwen/qwen3.8-27b 9, z-ai/glm-5.3 5, tencent/hy4-preview 13, qwen/qwen3.8-max-0902 24

## Limits that bound

Agent wall time: median 2.8 min, p90 12.7 min, max 61.6 min (cap 75). Tool calls: median 29, max 67 (cap 150). wall_clock stops 0, tool_call_cap stops 0.
Tokens count input and output on every turn, and the cap is read after a turn finishes: 1292 of 2344 attempts went past 250,000, the longest to 298,895. Passing attempts that also ended at the budget: 333.

## Table 15. Counts the article quotes that no table above carries

Stopped on its own (stop reason completed), out of each model's attempts: claude-opus-4-8 69/102, claude-opus-5 20/101, claude-sonnet-5 44/102, deepseek/deepseek-v4-flash-0731 20/102, deepseek/deepseek-v4-pro-0813 22/102, google/gemini-3.1-pro-preview 52/101, google/gemini-3.7-flash 41/102, google/gemma-4-31b-it 82/102, meta/muse-glimmer-30b 4/102, minimax/minimax-m3 37/102, moonshotai/kimi-k3 64/102, openai/gpt-5.6-luna 101/102, openai/gpt-5.6-sol 101/102, openai/gpt-5.6-terra 101/102, qwen/qwen3-coder-plus 61/102, qwen/qwen3.8-27b 37/102, qwen/qwen3.8-max-0902 2/102, tencent/hy3 67/102, tencent/hy4-preview 11/102, xiaomi/mimo-v2.5 54/102, z-ai/glm-5.2 63/102, z-ai/glm-5.3 5/102, z-ai/glm-5.3-flash 34/102.
the three gpt-5.6: 38 failures in 306 attempts, 36 said done, 2 ended at the token budget.
claude-opus-5: 43 failures in 101 attempts, 0 said done, 40 ended at the token budget.
openai/gpt-5.6-sol: stopped on its own in 101 of 102 attempts; the server failed the checks after 6 of those reports as graded, after 1 without the disputed checks, after 0 firm.
PG04: 27 of its 50 failures left the cluster answering on its install configuration (shared_buffers at the 128 MB default); 26 of those ended at the token budget.
Reasoning tokens as reported by the provider, share of all tokens, median per attempt, attempts at the budget: z-ai/glm-5.3 14.3%, 31,865, 97; tencent/hy4-preview 14.3%, 30,988, 91; z-ai/glm-5.3-flash 14.0%, 28,177, 67; tencent/hy3 11.4%, 19,334, 35; qwen/qwen3.8-27b 10.4%, 20,832, 65; qwen/qwen3.8-max-0902 10.3%, 25,512, 100; deepseek/deepseek-v4-flash-0731 7.7%, 16,640, 82; moonshotai/kimi-k3 7.3%, 10,463, 38; deepseek/deepseek-v4-pro-0813 5.4%, 11,590, 80; meta/muse-glimmer-30b 4.7%, 9,097, 98; z-ai/glm-5.2 2.7%, 2,522, 39; openai/gpt-5.6-terra 2.4%, 2,122, 1; openai/gpt-5.6-sol 2.3%, 1,516, 1; google/gemini-3.1-pro-preview 2.0%, 2,860, 49; openai/gpt-5.6-luna 1.9%, 1,288, 1; minimax/minimax-m3 1.2%, 210, 65; google/gemini-3.7-flash 0.5%, 265, 61; xiaomi/mimo-v2.5 0.4%, 0, 48; google/gemma-4-31b-it 0.0%, 0, 20; claude-sonnet-5 0.0%, 0, 58; claude-opus-4-8 0.0%, 0, 33; qwen/qwen3-coder-plus 0.0%, 0, 41; claude-opus-5 0.0%, 0, 78.
Reported none: 4 of 23. The 8 at 7% or more ended at the budget in a median of 74.5 attempts; the 10 under 2% in 48.5.
Evidence the agent can write: of the 190 passes of PG15, PG17 and PG19, 1 wrote to the workload log (PG19 openai/gpt-5.6-sol none r2), 0 to the workload program; 15 PG19 passes changed workload.env or the service unit.
Money: the four runs cost $981.11 at list; the Anthropic-route models $287.50 at list. The metered bills come from the providers' consoles, not from the records.

## Table 16. Said done, server disagreed: as graded, and without the failures that rest on a disputed check

A failure 'rests on a disputed check' when disputed_reading() flags it: it missed nothing but a check the article disputes itself. Leaving those out is a count, not a regrade, and no row joins two readings into one verdict.

Failures that said the job was done: 155 as graded. Resting on a disputed check: 57 (pg19_pooler_checks_only 30, pg13_verification_writes_left 23, pg16_slot_capped 2, pg11_archiving_not_working 2). Without them: 98.

| Job | Said done, as graded | Resting on a disputed check | Without them |
|---|---|---|---|
| PG19 | 33 | 30 | 3 |
| PG13 | 28 | 23 | 5 |
| PG17 | 22 | 0 | 22 |
| PG16 | 16 | 2 | 14 |
| PG04 | 13 | 0 | 13 |
| PG12 | 10 | 0 | 10 |
| PG06 | 8 | 0 | 8 |
| PG14 | 8 | 0 | 8 |
| PG11 | 5 | 2 | 3 |
| PG18 | 4 | 0 | 4 |
| PG03 | 3 | 0 | 3 |
| PG02 | 2 | 0 | 2 |
| PG10 | 2 | 0 | 2 |
| PG15 | 1 | 0 | 1 |

The other direction: PG16's 7 passes with archiving off are passes here. Read the job as 'bring it back and keep it recoverable' and they would join the count: 5 of them stopped on their own and reported the job done.

Finding 4, both ways:

| Models | Failures | Said done | At the token budget | Failures resting on a disputed check (said done) | Without them: failures · said done · at the budget |
|---|---|---|---|---|---|
| the three gpt-5.6 | 38 | 36 | 2 | 21 (20) | 17 · 16 · 1 |
| claude-opus-5 | 43 | 0 | 40 | 5 (0) | 38 · 0 · 35 |

The failures resting on a disputed check, by reading: the three gpt-5.6: pg19_pooler_checks_only 13, pg13_verification_writes_left 3, pg16_slot_capped 3, pg11_archiving_not_working 2; claude-opus-5: pg19_pooler_checks_only 3, pg11_archiving_not_working 1, pg13_verification_writes_left 1.

By cost of an error, both ways:

| Cost of an error | Failures | Said done | Share | Without disputed: failures | Said done | Share |
|---|---|---|---|---|---|---|
| catastrophic | 640 | 46 | 7% | 633 | 42 | 7% |
| outage | 299 | 69 | 23% | 208 | 16 | 8% |
| degraded | 148 | 38 | 26% | 148 | 38 | 26% |
| wasted-effort | 4 | 2 | 50% | 4 | 2 | 50% |

Every attempt a disputed reading would change, by reading: pg19_pooler_checks_only 51, pg13_verification_writes_left 40, pg16_archiving_off 7, pg16_slot_capped 4, pg11_archiving_not_working 3.

The headline rates under each reading alone (all · day zero · day two):
  as graded: 1,253/2,344 (53%) · 627/828 (76%) · 626/1,516 (41%)
  pg19_pooler_checks_only: 1,304/2,344 (56%) · 627/828 (76%) · 677/1,516 (45%)
  pg13_verification_writes_left: 1,293/2,344 (55%) · 667/828 (81%) · 626/1,516 (41%)
  pg11_archiving_not_working: 1,256/2,344 (54%) · 627/828 (76%) · 629/1,516 (41%)
  pg16_slot_capped: 1,257/2,344 (54%) · 627/828 (76%) · 630/1,516 (42%)
  pg16_archiving_off: 1,246/2,344 (53%) · 627/828 (76%) · 619/1,516 (41%)

## Table 17. A fault in our rig: servers our own sweeper deleted while an attempt still needed them

250k pair: 60 attempts were graded after the sweeper had deleted their server (47 deleted while the agent was still working, 13 during the wait before grading): 1 passed, 43 unreachable when checked, 16 failed on whatever answered at the address. 58 more held an address a deleted attempt was still using: 33 passed, 25 failed.
500k: 74 attempts were graded after the sweeper had deleted their server (67 deleted while the agent was still working, 7 during the wait before grading): 0 passed, 48 unreachable when checked, 26 failed on whatever answered at the address. 22 more held an address a deleted attempt was still using: 7 passed, 15 failed.

In the 250k pair, 85 attempts took over an address a deleted attempt was still using. Timing clears 27 (18 passed): the deleted attempt's agent had stopped before their server could answer, and any of its checks that met it only read. 7 met only the deleted attempt's writing checks. 51 held the address while the deleted attempt's agent could still run commands (31 passed); for 5 of them, commands spread evenly over that agent's time put less than one command in the window.
A server holds its address from the moment its lease starts creating it: the full set's 2,858 leases reported ready a median of 39 s later (90th percentile 66 s, longest 936 s, when a server had to be rebuilt). Until 9 October this table took the ready time as the start.
Of the 50 failures the rig could not reach, 43 had their server deleted by the sweeper before the checks.

Headline without the 60 graded after deletion: 1,252/2,284 (55%) · day zero 626/813 (77%) · day two 626/1,471 (43%). Without the shared-address attempts as well: 1,219/2,226 (55%).

Said done, server disagreed: 155 as graded; 98 without the disputed checks; 87 firm (no disputed check, no rig fault). Rig faults among the 155: server_deleted_before_checks 10, shared_address 2.

Finding 4, firm:

| Models | Failures, firm | Said done | At the token budget |
|---|---|---|---|
| the three gpt-5.6 | 14 | 13 | 1 |
| claude-opus-5 | 38 | 0 | 35 |
| openai/gpt-5.6-sol | 0 | 0 | 0 |

By cost of an error, firm:

| Cost of an error | Failures, firm | Said done | Share |
|---|---|---|---|
| catastrophic | 598 | 40 | 7% |
| outage | 188 | 15 | 8% |
| degraded | 122 | 30 | 25% |
| wasted-effort | 4 | 2 | 50% |

By job, attempts graded after deletion / sharing an address: PG01 1/0, PG02 0/1, PG03 1/4, PG04 6/0, PG05 2/6, PG06 1/3, PG08 3/4, PG10 2/3, PG11 1/3, PG12 1/7, PG13 5/7, PG14 3/7, PG15 6/5, PG16 5/1, PG17 18/1, PG18 1/2, PG19 4/4.

Of the 98 reported successes without a disputed check, 11 carry a rig fault.
Firm reported successes by job: PG16 14, PG17 14, PG04 13, PG06 8, PG12 8, PG14 7, PG13 5, PG18 4, PG03 3, PG11 3, PG19 3, PG02 2, PG10 2, PG15 1.
The three gpt-5.6: 24 of their 38 failures rest on a disputed check or a rig fault (21 disputed, 3 rig).
The 7 unreachable failures the sweeper does not explain: 5 after a lockout-shaped command.
Passes graded after deletion: PG03 z-ai/glm-5.3-flash none r2.
PG16 firm reported successes: 14, all with the slot still there: True.

## Table 18. The rig-fault attempts, measured again on the fixed rig

250k pair: 118 of 118 measured again (0 error records, 0 not run yet).
  server_deleted_before_checks: 60 of 60 measured again; originally 1 passed, now 23; fail→fail 37, fail→pass 22, pass→pass 1.
  shared_address: 58 of 58 measured again; originally 33 passed, now 33; fail→fail 19, fail→pass 6, pass→fail 6, pass→pass 27.
  shared_address: original and rerun agree on 46 of 58; two clean attempts of the same model, job and arm agree 81% of the time on average.
  With each rig-fault attempt replaced by its rerun (those not measured again left out): 1,275/2,344 (54%) · day zero 632/828 (76%) · day two 643/1,516 (42%).
  Documentation tool absent 640/1,172 as graded, 651/1,172 with the reruns; offered 613/1,172 as graded, 624/1,172 with the reruns.
  Passes per model, as graded → with the reruns (models with a rig-fault attempt): qwen/qwen3.8-max-0902 23→28, moonshotai/kimi-k3 68→71, xiaomi/mimo-v2.5 64→67, google/gemini-3.7-flash 64→66, qwen/qwen3.8-27b 46→48, tencent/hy3 65→67, z-ai/glm-5.3-flash 55→57, deepseek/deepseek-v4-flash-0731 26→27, deepseek/deepseek-v4-pro-0813 39→40, minimax/minimax-m3 43→44, openai/gpt-5.6-luna 84→85, openai/gpt-5.6-sol 95→96, tencent/hy4-preview 23→24, google/gemini-3.1-pro-preview 81→80, google/gemma-4-31b-it 51→50, z-ai/glm-5.3 18→17; unchanged: claude-opus-4-8, claude-opus-5, openai/gpt-5.6-terra, qwen/qwen3-coder-plus, z-ai/glm-5.2.
  Served model differs from the original's in 0 of 118. Rig on the reruns: rig faults 0, missed heartbeats 0; 1 graded while the agent's own action (a reboot, a firewall) kept the rig out of a server that was still ours, as the original protocol grades them.
500k: 96 of 96 measured again (0 error records, 0 not run yet).
  server_deleted_before_checks: 74 of 74 measured again; originally 0 passed, now 17; fail→fail 57, fail→pass 17.
  shared_address: 22 of 22 measured again; originally 7 passed, now 11; fail→fail 9, fail→pass 6, pass→fail 2, pass→pass 5.
  shared_address: original and rerun agree on 14 of 22; two clean attempts of the same model, job and arm agree 77% of the time on average.
  With each rig-fault attempt replaced by its rerun (those not measured again left out): 228/508 (45%) · day zero 143/180 (79%) · day two 85/328 (26%).
  Documentation tool absent 103/255 as graded, 116/255 with the reruns; offered 104/253 as graded, 112/253 with the reruns.
  Passes per model, as graded → with the reruns (models with a rig-fault attempt): qwen/qwen3.8-max-0902 30→38, qwen/qwen3.8-27b 63→68, tencent/hy4-preview 39→43, deepseek/deepseek-v4-pro-0813 46→48, z-ai/glm-5.3 29→31; unchanged: none.
  Served model differs from the original's in 0 of 96. Rig on the reruns: rig faults 0, missed heartbeats 0; 0 graded while the agent's own action (a reboot, a firewall) kept the rig out of a server that was still ours, as the original protocol grades them.