Docka LabCan Agents Run It?
Can Agents Run It? · Issue 3 · Evidence

PG17 · gpt-5.6-sol · no doc tool · repeat 3

A fault in our rig. As graded: fail.

A fault in our rig. Our sweeper deleted this attempt's server during the ten-minute wait before grading. The checks then read whatever answered at the address: there, the workload service had never started and the table this job creates did not exist, while the agent's own commands show both in place. The verdict says nothing about the agent. It counts as graded everywhere, and the CSV marks it server_deleted_before_checks.

Job
PG17 · Autovacuum is not keeping up: a bloating table under sustained write load
Model
openai/gpt-5.6-sol
Documentation tool
absent
Repeat
3
Verdict as graded
fail
How it ended
said the job was done
Disputed reading
none
Rig fault
server_deleted_before_checks
Tokens
83,577
Commands
21
Agent time
3.0 min

The prompt

You are the on-call engineer for a PostgreSQL 17 server on this machine (node1, root over SSH). The application service `pgeval-workload` updates the table `app.events` (about 5 million rows) continuously and must keep doing so at its current rate.
Over the last hour the table has grown far faster than its row count and its queries are slowing down; the dashboard shows dead tuples piling up. Get the bloat under control, and make sure it stays under control after you leave.
Constraints: the application must not see errors, and no application statement may wait on a lock for more than 2 seconds. Do not stop, pause or restart the `pgeval-workload` service. Do not delete rows.
The `postgres` OS account administers the cluster (peer authentication). The application connects as `app_rw`; its connection settings are in /etc/pgeval/workload.env.

What the agent did

Every command in the order sent, with the node it went to, and what the agent wrote between them. The rig did not record what each command printed, so this page cannot show it.

1.

node1
hostname; date; systemctl is-active pgeval-workload; systemctl status pgeval-workload --no-pager -l | head -30
node1
cat /etc/pgeval/workload.env
node1
sudo -u postgres psql -X -Atqc "select version(); show data_directory; show config_file; show autovacuum; show track_counts;"

2.

node1
sudo -u postgres psql -X -d app -P pager=off -c "SELECT c.oid::regclass, pg_size_pretty(pg_total_relation_size(c.oid)) total, pg_size_pretty(pg_relation_size(c.oid)) heap, pg_size_pretty(pg_indexes_size(c.oid)) indexes, s.n_live_tup,s.n_dead_tup,s.n_tup_upd,s.n_tup_hot_upd,s.last_vacuum,s.last_autovacuum,s.vacuum_count,s.autovacuum_count FROM pg_class c JOIN pg_stat_user_tables s ON s.relid=c.oid WHERE c.oid='app.events'::regclass;" -c "SELECT indexrelname,pg_size_pretty(pg_relation_size(indexrelid)),idx_scan FROM pg_stat_user_indexes WHERE relid='app.events'::regclass ORDER BY pg_relation_size(indexrelid) DESC;" -c "SELECT reloptions FROM pg_class WHERE oid='app.events'::regclass;"
node1
sudo -u postgres psql -X -d app -P pager=off -c "SELECT pid,usename,application_name,state,now()-xact_start xact_age,now()-query_start query_age,wait_event_type,wait_event,left(query,120) query FROM pg_stat_activity WHERE datname='app' ORDER BY xact_start NULLS LAST;" -c "SELECT slot_name,slot_type,active,xmin,catalog_xmin,restart_lsn FROM pg_replication_slots;"
node1
sudo -u postgres psql -X -d app -P pager=off -c "SHOW autovacuum_max_workers; SHOW autovacuum_naptime; SHOW autovacuum_vacuum_threshold; SHOW autovacuum_vacuum_scale_factor; SHOW autovacuum_vacuum_cost_delay; SHOW autovacuum_vacuum_cost_limit; SHOW maintenance_work_mem;" -c "SELECT a.attname,a.attstorage,a.attcompression FROM pg_attribute a WHERE attrelid='app.events'::regclass AND attnum>0 AND NOT attisdropped ORDER BY attnum;" -c "\d+ app.events"
node1
tail -50 /var/log/pgeval/workload.log; journalctl -u postgresql@17-main --since '1 hour ago' --no-pager | tail -80

3.

node1
sudo -u postgres psql -X -v ON_ERROR_STOP=1 -d app <<'SQL'
SET lock_timeout = '1500ms';
ALTER TABLE app.events SET (
  autovacuum_enabled = true,
  autovacuum_vacuum_threshold = 5000,
  autovacuum_vacuum_scale_factor = 0.002,
  autovacuum_vacuum_cost_delay = 0,
  autovacuum_vacuum_cost_limit = 10000,
  autovacuum_analyze_threshold = 5000,
  autovacuum_analyze_scale_factor = 0.01,
  fillfactor = 80
);
SQL
sudo -u postgres psql -X -d app -Atqc "select reloptions from pg_class where oid='app.events'::regclass;"

4.

node1
free -h; df -h /var/lib/postgresql/17/main
node1
sudo -u postgres psql -X -d app -Atqc "select pg_size_pretty(pg_database_size('app')), pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),'0/0'));"; grep -E '^(shared_buffers|maintenance_work_mem|autovacuum_work_mem|max_wal_size)' /etc/postgresql/17/main/postgresql.conf

5.

node1
sudo -u postgres psql -X -v ON_ERROR_STOP=1 -d app <<'SQL'
SET lock_timeout = '1500ms';
ALTER TABLE app.events SET (
  autovacuum_work_mem = 524288,
  autovacuum_vacuum_max_threshold = 20000,
  autovacuum_vacuum_insert_threshold = 5000,
  autovacuum_vacuum_insert_scale_factor = 0.002,
  autovacuum_vacuum_truncate = false
);
SQL
cat > /tmp/pgeval_vacuum.sql <<'SQL'
SET maintenance_work_mem='1GB';
SET vacuum_cost_delay=0;
SET lock_timeout='1500ms';
VACUUM (VERBOSE, ANALYZE, TRUNCATE FALSE, PARALLEL 0) app.events;
SQL
sudo -u postgres psql -X -v ON_ERROR_STOP=1 -d app -f /tmp/pgeval_vacuum.sql > /tmp/pgeval_vacuum.out 2>&1 & echo $! > /tmp/pgeval_vacuum.pid
echo "vacuum shell pid $(cat /tmp/pgeval_vacuum.pid)"

6.

node1
sleep 3; sudo -u postgres psql -X -d app -P pager=off -c "SELECT pid,phase,heap_blks_total,heap_blks_scanned,heap_blks_vacuumed,index_vacuum_count,num_dead_tuples,max_dead_tuples FROM pg_stat_progress_vacuum WHERE relid='app.events'::regclass;"; tail -20 /tmp/pgeval_vacuum.out
node1
tail -500 /var/log/pgeval/workload.log | awk '{x[$NF]++} END{for(k in x)print k,x[k]}'

7.

node1
sudo -u postgres psql -X -v ON_ERROR_STOP=1 -d app <<'SQL'
SET lock_timeout = '1500ms';
ALTER TABLE app.events SET (
  autovacuum_vacuum_max_threshold = 20000,
  autovacuum_vacuum_insert_threshold = 5000,
  autovacuum_vacuum_insert_scale_factor = 0.002,
  autovacuum_vacuum_truncate = false
);
SQL
sudo -u postgres psql -X -d app -P pager=off -c "SELECT pg_size_pretty(pg_total_relation_size(relid)) total,pg_size_pretty(pg_relation_size(relid)) heap,n_live_tup,n_dead_tup,n_tup_upd,n_tup_hot_upd,last_vacuum,last_autovacuum,vacuum_count,autovacuum_count FROM pg_stat_user_tables WHERE relid='app.events'::regclass;" -c "SELECT reloptions FROM pg_class WHERE oid='app.events'::regclass;"

8.

node1
sudo -u postgres psql -X -v ON_ERROR_STOP=1 -d app <<'SQL'
SET lock_timeout = '1500ms';
ALTER TABLE app.events SET (
  autovacuum_vacuum_insert_threshold = 5000,
  autovacuum_vacuum_insert_scale_factor = 0.002,
  autovacuum_vacuum_truncate = false
);
ALTER SYSTEM SET autovacuum_naptime = '10s';
SELECT pg_reload_conf();
SQL
sleep 1
sudo -u postgres psql -X -d app -Atqc "show autovacuum_naptime; select reloptions from pg_class where oid='app.events'::regclass;"

9.

node1
sudo -u postgres psql -X -v ON_ERROR_STOP=1 -d app <<'SQL'
SET lock_timeout = '1500ms';
ALTER TABLE app.events SET (
  autovacuum_vacuum_insert_threshold = 5000,
  autovacuum_vacuum_insert_scale_factor = 0.002
);
ALTER SYSTEM SET autovacuum_naptime = '10s';
SELECT pg_reload_conf();
SQL
sleep 2
sudo -u postgres psql -X -d app -P pager=off -c "SHOW autovacuum_naptime;" -c "SELECT reloptions FROM pg_class WHERE oid='app.events'::regclass;" -c "SELECT n_live_tup,n_dead_tup,n_tup_upd,n_tup_hot_upd,round(100.0*n_tup_hot_upd/nullif(n_tup_upd,0),1) hot_pct,last_vacuum,last_autovacuum FROM pg_stat_user_tables WHERE relid='app.events'::regclass;"

10.

node1
sleep 15
sudo -u postgres psql -X -d app -P pager=off -c "SELECT now(),n_live_tup,n_dead_tup,n_tup_upd,n_tup_hot_upd,last_vacuum,last_autovacuum,vacuum_count,autovacuum_count FROM pg_stat_user_tables WHERE relid='app.events'::regclass;" -c "SELECT pid,phase,heap_blks_total,heap_blks_scanned,heap_blks_vacuumed,index_vacuum_count FROM pg_stat_progress_vacuum WHERE relid='app.events'::regclass;"
tail -1000 /var/log/pgeval/workload.log | awk '$NF!="ok"{print}' | tail -10; echo non_ok=$(tail -1000 /var/log/pgeval/workload.log | awk '$NF!="ok"{n++} END{print n+0}')

11.

node1
sleep 20
sudo -u postgres psql -X -d app -P pager=off -c "SELECT now(),n_dead_tup,n_tup_upd,n_tup_hot_upd,last_autovacuum,autovacuum_count FROM pg_stat_user_tables WHERE relid='app.events'::regclass;" -c "SELECT pid,phase,heap_blks_total,heap_blks_scanned,heap_blks_vacuumed,index_vacuum_count FROM pg_stat_progress_vacuum WHERE relid='app.events'::regclass;" -c "SELECT pid,backend_type,state,wait_event_type,wait_event,left(query,100) FROM pg_stat_activity WHERE backend_type like 'autovacuum%';"
tail -1000 /var/log/pgeval/workload.log | awk '$NF!="ok"{n++} END{print "non_ok=" n+0}'

12.

node1
sudo -u postgres psql -X -v ON_ERROR_STOP=1 -d app <<'SQL'
SET lock_timeout='1500ms';
ALTER TABLE app.events SET (autovacuum_vacuum_threshold=1000, autovacuum_vacuum_scale_factor=0.001);
SQL
sleep 15
sudo -u postgres psql -X -d app -P pager=off -c "SELECT now(),n_dead_tup,n_tup_upd,n_tup_hot_upd,last_autovacuum,autovacuum_count FROM pg_stat_user_tables WHERE relid='app.events'::regclass;" -c "SELECT pid,phase,heap_blks_total,heap_blks_scanned,heap_blks_vacuumed,index_vacuum_count FROM pg_stat_progress_vacuum WHERE relid='app.events'::regclass;"
tail -1500 /var/log/pgeval/workload.log | awk '$NF!="ok"{n++} END{print "non_ok=" n+0}'

13.

node1
sleep 20; sudo -u postgres psql -X -d app -P pager=off -c "SELECT now(),pg_size_pretty(pg_total_relation_size(relid)) total,n_live_tup,n_dead_tup,n_tup_upd,n_tup_hot_upd,last_vacuum,last_autovacuum,vacuum_count,autovacuum_count FROM pg_stat_user_tables WHERE relid='app.events'::regclass;" -c "SELECT name,setting,unit,source,sourcefile FROM pg_settings WHERE name='autovacuum_naptime';"
node1
systemctl is-active pgeval-workload; tail -3000 /var/log/pgeval/workload.log | awk '{tot++; if($NF!="ok") bad++; if($(NF-1)+0>2000) over++} END{print "samples="tot,"non_ok="bad+0,"over_2s="over+0}'

14. Completed on `node1` without stopping or restarting the workload. - Re-enabled autovacuum on `app.events`. - Ran an online `VACUUM (ANALYZE)` with a 1.5-second lock timeout. - Reduced dead tuples from roughly **1.5 million** to a few thousand. - Applied persistent table-specific tuning: - Aggressive vacuum/analyze thresholds - Higher vacuum cost limit and no cost delay - `fillfactor=80` to encourage HOT updates - Reduced cluster autovacuum polling interval from 60s to 10s via reload; no restart required. - Verified autovacuum subsequently ran on the table. - Confirmed `pgeval-workload` remains active. - Checked 3,000 recent workload operations: **0 errors and 0 operations over 2 seconds**.

Its last message

Completed on `node1` without stopping or restarting the workload.

- Re-enabled autovacuum on `app.events`.
- Ran an online `VACUUM (ANALYZE)` with a 1.5-second lock timeout.
- Reduced dead tuples from roughly **1.5 million** to a few thousand.
- Applied persistent table-specific tuning:
  - Aggressive vacuum/analyze thresholds
  - Higher vacuum cost limit and no cost delay
  - `fillfactor=80` to encourage HOT updates
- Reduced cluster autovacuum polling interval from 60s to 10s via reload; no restart required.
- Verified autovacuum subsequently ran on the table.
- Confirmed `pgeval-workload` remains active.
- Checked 3,000 recent workload operations: **0 errors and 0 operations over 2 seconds**.

The checks

CheckNodeResultWhat it printed
the cluster answersnode1pass[node1] answer=1
dead-tuple ratio on app.events is under control ten minutes laternode1fail[node1] n_dead_tup=none n_live_tup=none ratio_before=0.3032
autovacuum will keep running on app.eventsnode1fail[node1] reloptions=none autovacuum=on pending_restart=none cluster_scale_factor=0.2 table_scale_factor=unset
a vacuum has actually run on app.events recentlynode1fail[node1] last_autovacuum= last_vacuum= seconds_since_last=
the application service kept running (active, not restarted, no stall)node1fail[node1] active=inactive active_enter=1789171200 t0=1789238994 restart_lines=0 lines_in_window= max_gap_s= stderr: Traceback (most recent call last): File "<stdin>", line 3, in <module> FileNotFoundError: [Errno 2] No such file or directory: '/var/log/pgeval/workload.log'
the application saw no errors and no latency spike in the windownode1fail[node1] stderr: Traceback (most recent call last): File "<stdin>", line 4, in <module> FileNotFoundError: [Errno 2] No such file or directory: '/var/log/pgeval/workload.log'
no exclusive-lock rewrite under load (VACUUM FULL signature)node1fail[node1] relfilenode=none relfilenode_before=16429 errors_in_window=none stderr: Traceback (most recent call last): File "<stdin>", line 3, in <module> FileNotFoundError: [Errno 2] No such file or directory: '/var/log/pgeval/workload.log'
data intact: immutable columns unchanged, ids unchanged, no rows lost or addednode1fail[node1] cold_sum=none expected=890790bbc6c0a4f3cfa1c450f5ed19c5 ids_sum=none ids_expected=3096864166c3c18fb4062ff10653a467 rows=none rows_expected=5000000