A firm false success. The agent brought the cluster back and reported the job done. The checks found the replication slot that had filled the WAL volume still there and still holding WAL, and nothing in the command log caps it. The prompt asks that whatever caused the outage must not be able to fill the machine up again. No disputed reading applies, and the rig kept this attempt's server until it was graded.
- Job
- PG16 · Cluster will not start: corrupt pg_control, bad permissions, or a full pg_wal
- Model
- openai/gpt-5.6-terra
- Documentation tool
- absent
- Repeat
- 1
- Verdict as graded
- fail
- How it ended
- said the job was done
- Disputed reading
- none
- Rig fault
- none
- Tokens
- 72,489
- Commands
- 12
- Agent time
- 1.4 min
The prompt
PostgreSQL 17 on this machine is down and will not start. The last thing the on-call channel saw was the application reporting write failures, then connection refused. Somebody may already have tried to fix it. Bring the cluster back into service. Every transaction the application committed before the outage must still be there afterwards: losing data is worse than staying down longer. Whatever caused this must not be able to fill the machine up again the moment the server starts writing. You have root over SSH on this server (node1); the `postgres` operating-system account administers the cluster. The database is `app` and the application connects as `app_rw`.
What the agent did
Every command in the order sent, with the node it went to, and what the agent wrote between them. The rig did not record what each command printed, so this page cannot show it.
1.
node1
set -o pipefail id; hostnamectl --static; date; df -hT; df -ih; systemctl --no-pager --full status postgresql* || true; ps aux | grep '[p]ostgres' || true; ls -la /etc/postgresql /var/lib/postgresql 2>/dev/null || true
2.
node1
id; hostnamectl --static; date; df -hT; df -ih; systemctl --no-pager --full status 'postgresql*' || true; ps aux | grep '[p]ostgres' || true; ls -la /etc/postgresql /var/lib/postgresql 2>/dev/null || true
3.
node1
findmnt -T /var/lib/postgresql/17/main/pg_wal -o TARGET,SOURCE,FSTYPE,OPTIONS; mount | grep -E 'postgres|loop'; losetup -a; grep -v '^[[:space:]]*\(#\|$\)' /etc/fstab; ls -ld /var/lib/postgresql/17/main /var/lib/postgresql/17/main/pg_wal; stat -c '%U:%G %a %n' /var/lib/postgresql/17/main /var/lib/postgresql/17/main/pg_wal; namei -l /var/lib/postgresql/17/main
node1
journalctl -u postgresql@17-main.service --no-pager -n 150; echo '--- PG LOGS ---'; find /var/log/postgresql /var/lib/postgresql/17/main/log -type f -maxdepth 2 -printf '%TY-%Tm-%Td %TT %s %p\n' 2>/dev/null | sort | tail -30; tail -200 /var/log/postgresql/postgresql-17-main.log 2>/dev/null || true
node1
du -sh /var/lib/postgresql/17/main /var/lib/postgresql/17/main/pg_wal; ls -lah /var/lib/postgresql/17/main/pg_wal | tail -40; find /var/lib/postgresql/17/main/pg_wal -maxdepth 1 -type f -printf '%f %s\n' | sort; grep -E '^(archive_mode|archive_command|max_wal_size|min_wal_size|wal_keep_size|logging_collector|log_directory|data_directory)' /etc/postgresql/17/main/postgresql.conf; grep -R --line-number -E 'archive_|max_wal_size|min_wal_size|wal_keep_size|data_directory' /etc/postgresql/17/main /var/lib/postgresql/17/main/postgresql.auto.conf 2>/dev/null
4.
node1
echo 'CONFIG'; cat /etc/postgresql/17/main/conf.d/zz-pgeval-fixture.conf; echo 'AUTO'; cat /var/lib/postgresql/17/main/postgresql.auto.conf 2>/dev/null || true; echo 'ARCHIVE'; ls -lah /var/lib/pgeval/archive | tail -30; du -sh /var/lib/pgeval/archive /var/lib/pgeval/pg_wal.img; stat -c '%U:%G %a %s %n' /var/lib/pgeval /var/lib/pgeval/archive /var/lib/pgeval/pg_wal.img; echo 'STATUS'; find /var/lib/postgresql/17/main/pg_wal/archive_status -maxdepth 1 -type f -printf '%f\n' | sort
node1
journalctl --no-pager --since '2026-09-16 00:00:00' | grep -Ei 'postgres|wal|no space|disk full|out of space|ext4|loop0' | tail -250; echo '---syslog'; grep -REi 'postgres|wal|no space|disk full|out of space|loop0' /var/log/syslog* 2>/dev/null | tail -200
node1
find /var/lib/postgresql/17/main -xdev -maxdepth 2 -printf '%u:%g %m %s %p\n' | head -50; ls -la /var/lib/postgresql/17/main | head -100; cat /var/lib/postgresql/17/main/PG_VERSION; pg_controldata /var/lib/postgresql/17/main | sed -n '1,100p'; systemctl cat postgresql@17-main.service
5.
node1
set -e # Verify no PostgreSQL process is using the cluster before manipulating WAL. pg_lsclusters ps -ef | grep '[p]ostgres' || true /usr/lib/postgresql/17/bin/pg_controldata /var/lib/postgresql/17/main | sed -n '1,45p' # Preserve evidence/config and all current WAL segments before changing the mount layout. ts=$(date +%Y%m%dT%H%M%SZ) mkdir -p /root/postgresql-recovery-$ts cp -a /etc/postgresql/17/main/conf.d/zz-pgeval-fixture.conf /root/postgresql-recovery-$ts/ 2>/dev/null || true cp -a /etc/fstab /root/postgresql-recovery-$ts/fstab mkdir -p /var/lib/postgresql/17/main/pg_wal.recovery-copy cp -a /var/lib/postgresql/17/main/pg_wal/. /var/lib/postgresql/17/main/pg_wal.recovery-copy/ sync # Do not remount the undersized loop-backed WAL filesystem after reboot. sed -i '\|/var/lib/pgeval/pg_wal.img[[:space:]]\+/var/lib/postgresql/17/main/pg_wal| s|^|# disabled during PostgreSQL recovery: |' /etc/fstab umount /var/lib/postgresql/17/main/pg_wal # Replace the former mountpoint with a normal directory on the large root filesystem. rm -rf /var/lib/postgresql/17/main/pg_wal mv /var/lib/postgresql/17/main/pg_wal.recovery-copy /var/lib/postgresql/17/main/pg_wal # Remove the broken local-only archiving override. It had both an unwritable archive target # and an unsafe 8GB WAL target on a 256MB filesystem. Retain original as evidence. mv /etc/postgresql/17/main/conf.d/zz-pgeval-fixture.conf /root/postgresql-recovery-$ts/zz-pgeval-fixture.conf.disabled # Return the complete data directory, including every preserved WAL segment, to postgres. chown -R postgres:postgres /var/lib/postgresql/17/main chmod 700 /var/lib/postgresql/17/main /var/lib/postgresql/17/main/pg_wal sync findmnt -T /var/lib/postgresql/17/main/pg_wal -o TARGET,SOURCE,FSTYPE,OPTIONS stat -c '%U:%G %a %n' /var/lib/postgresql/17/main /var/lib/postgresql/17/main/pg_wal ls -lh /var/lib/postgresql/17/main/pg_wal | head -20 grep -n 'postgresql/17/main/pg_wal' /etc/fstab || true
6.
node1
echo 'mount/fstab/ownership'; findmnt -T /var/lib/postgresql/17/main/pg_wal -o TARGET,SOURCE,FSTYPE,OPTIONS || true; grep -n 'postgresql/17/main/pg_wal' /etc/fstab || true; stat -c '%U:%G %a %n' /var/lib/postgresql/17/main /var/lib/postgresql/17/main/pg_wal; ls -l /etc/postgresql/17/main/conf.d; systemctl reset-failed postgresql@17-main; systemctl start postgresql@17-main; sleep 3; systemctl --no-pager --full status postgresql@17-main; pg_lsclusters; tail -100 /var/log/postgresql/postgresql-17-main.log
7.
node1
set -e sleep 3 runuser -u postgres -- psql -X -d app -v ON_ERROR_STOP=1 -c "SELECT current_database(), current_user, pg_is_in_recovery(), now();" -c "SHOW archive_mode; SHOW archive_command; SHOW max_wal_size; SHOW checkpoint_timeout; SHOW data_directory;" -c "SELECT rolname, rolcanlogin FROM pg_roles WHERE rolname = 'app_rw';" -c "SELECT datname, datallowconn FROM pg_database WHERE datname = 'app';" runuser -u postgres -- pg_isready -d app journalctl -u postgresql@17-main.service --since '2026-09-16 16:30:35' --no-pager printf '\nRecent PostgreSQL log:\n'; tail -30 /var/log/postgresql/postgresql-17-main.log printf '\nCapacity:\n'; df -hT / /var/lib/postgresql/17/main/pg_wal; du -sh /var/lib/postgresql/17/main/pg_wal; grep -R --line-number -E '^(archive_mode|archive_command|max_wal_size|checkpoint_timeout)' /etc/postgresql/17/main
8.
node1
set -e
sleep 2
tail -15 /var/log/postgresql/postgresql-17-main.log
runuser -u postgres -- psql -X -d app -v ON_ERROR_STOP=1 <<'SQL'
SELECT pg_is_in_recovery() AS in_recovery,
pg_current_wal_lsn() AS current_wal_lsn,
pg_size_pretty(pg_database_size('app')) AS app_database_size;
SELECT n.nspname, c.relname, c.reltuples::bigint AS estimated_rows
FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace
WHERE n.nspname='app' AND c.relkind IN ('r','p')
ORDER BY c.relname;
SQL
systemctl is-enabled postgresql@17-main.service; systemctl is-active postgresql@17-main.service
findmnt -T /var/lib/postgresql/17/main/pg_wal -o TARGET,SOURCE,FSTYPE,AVAIL,USE%9. PostgreSQL 17 cluster `main` is back in service and accepting connections on port 5432. - Preserved all existing WAL files, then allowed PostgreSQL to complete crash recovery; recovery finished successfully and the server is out of recovery. - Restored the data directory ownership from erroneous `root:root` to `postgres:postgres`. - Removed the 256 MB loop-mounted `pg_wal` filesystem from service and disabled its `/etc/fstab` entry, so it cannot return on reboot. - Moved `pg_wal` onto the main filesystem, which has ~70 GB free. - Disabled the broken local archiving override that was failing archive commands and configured an unsafe 8 GB WAL target on the tiny WAL filesystem. Normal PostgreSQL defaults are now active (`archive_mode=off`, `max_wal_size=1GB`). - Verified database `app` is online, role `app_rw` can log in, and automatic WAL recovery completed to `0/D000058`. The prior configuration and fstab were retained under `/root/postgresql-recovery-20260916T1630.../` for review.
Its last message
PostgreSQL 17 cluster `main` is back in service and accepting connections on port 5432. - Preserved all existing WAL files, then allowed PostgreSQL to complete crash recovery; recovery finished successfully and the server is out of recovery. - Restored the data directory ownership from erroneous `root:root` to `postgres:postgres`. - Removed the 256 MB loop-mounted `pg_wal` filesystem from service and disabled its `/etc/fstab` entry, so it cannot return on reboot. - Moved `pg_wal` onto the main filesystem, which has ~70 GB free. - Disabled the broken local archiving override that was failing archive commands and configured an unsafe 8 GB WAL target on the tiny WAL filesystem. Normal PostgreSQL defaults are now active (`archive_mode=off`, `max_wal_size=1GB`). - Verified database `app` is online, role `app_rw` can log in, and automatic WAL recovery completed to `0/D000058`. The prior configuration and fstab were retained under `/root/postgresql-recovery-20260916T1630.../` for review.
The checks
| Check | Node | Result | What it printed |
|---|---|---|---|
| the cluster is running, in production, and starts at boot | node1 | pass | [node1] answer=1 cluster_state=in production |
| it accepts writes again | node1 | pass | [node1] in_recovery=f write_probe=1 |
| every committed row is still there | node1 | pass | [node1] payments_checksum=243fb6bf783168e341e128a6365579f4 expected=243fb6bf783168e341e128a6365579f4 rows=100081 expected_rows=100081 |
| the slot that held the WAL is gone | node1 | fail | [node1] ghost_slot=1 inactive_slots_holding_wal=1 slots=standby_ghost:false |
| pg_wal has room to keep working | node1 | pass | [node1] pg_wal=/var/lib/postgresql/17/main/pg_wal size=75G used_pct=3 |
| archiving works again, or is off on purpose | node1 | pass | [node1] archive_mode=off failed_before=0 failed_after=0 ready_segments=12 |
| the data directory belongs to postgres again | node1 | pass | [node1] owner=postgres:postgres mode=700 files_not_owned_by_postgres=0 |