Can Agents Run It? · evidence — Issue 1: Supabase

claude-opus-5 · d2_pooling · rep 1

fail

model

claude-opus-5

task

d2_pooling

rep

day

steps

tokens

86,460

est. cost

$1.038

agent time

154s

wave

s4b

started

2026-07-25T06:19:05Z

The task as the agent received it

You have operator access to a hosted Supabase project. Credentials are in environment variables: SUPABASE_URL, SUPABASE_PROJECT_REF, SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_DB_URL (a session-mode Postgres connection string), SUPABASE_DB_PASSWORD. Our serverless workers keep exhausting database connections. Write an executable script named pool_smoke.sh in the current working directory that proves the right pooling setup: - it must connect through the project's TRANSACTION-MODE connection pooler (not the session-mode port, not a direct database connection); - it must run 20 concurrent one-shot queries (each e.g. SELECT pg_backend_pid()), wait for all of them, and print the single line "POOL OK 20/20" only if all 20 succeeded, then exit 0; - it must take credentials from the environment variables — never hardcode the database password. Run it yourself to confirm it passes before finishing.

What it ran — 18 commands

#1exit 0

echo "URL: $SUPABASE_URL"; echo "REF: $SUPABASE_PROJECT_REF"; echo "DB_URL: $SUPABASE_DB_URL" | sed 's/:[^:@]*@/:***@/'; pwd; ls -la

URL: https://<ref>.supabase.co
REF: <ref>
DB_URL: postgresql://postgres.<ref>:***@aws-0-us-east-1.pooler.supabase.com:5432/postgres
/work
total 8
drwx------ 2 1000 1000 4096 Jul 25 06:19 .
drwxr-xr-x 1 root root 4096 Jul 25 06:19 ..

#2exit 0

which psql pg_isready python3 nc dig getent 2>&1; psql --version 2>&1

/usr/bin/psql
/usr/bin/pg_isready
/usr/bin/python3
/usr/bin/nc
/usr/bin/dig
/usr/bin/getent
psql (PostgreSQL) 18.4 (Debian 18.4-1.pgdg12+1)

#3exit 0

getent hosts aws-0-us-east-1.pooler.supabase.com; for p in 5432 6543; do timeout 8 nc -vz aws-0-us-east-1.pooler.supabase.com $p 2>&1 | tail -1; done

44.216.29.125   pool-tcp-us-east-1-e9f793b-79951640844772aa.elb.us-east-1.amazonaws.com aws-0-us-east-1.pooler.supabase.com
44.208.221.186  pool-tcp-us-east-1-e9f793b-79951640844772aa.elb.us-east-1.amazonaws.com aws-0-us-east-1.pooler.supabase.com
52.45.94.125    pool-tcp-us-east-1-e9f793b-79951640844772aa.elb.us-east-1.amazonaws.com aws-0-us-east-1.pooler.supabase.com
Connection to aws-0-us-east-1.pooler.supabase.com (52.45.94.125) 5432 port [tcp/postgresql] succeeded!
Connection to aws-0-us-east-1.pooler.supabase.com (44.216.29.125) 6543 port [tcp/*] succeeded!

#4exit 0

getent hosts db.<ref>.supabase.co 2>&1 || echo "no direct DNS (IPv4)"

2600:1f18:4850:7c01:c70c:3b85:612c:200e db.<ref>.supabase.co

#5exit 0

cd /work
TXN_URL=$(printf '%s' "$SUPABASE_DB_URL" | sed -E 's#:5432/#:6543/#')
PGCONNECT_TIMEOUT=15 psql "$TXN_URL" -Atc "select 'connected', current_setting('server_version');" 2>&1 | head -5

connected|17.6

#6exit 0

cd /work
TXN_URL=$(printf '%s' "$SUPABASE_DB_URL" | sed -E 's#:5432/#:6543/#')
echo "--- port 6543 (expect app_name to NOT persist => transaction mode) ---"
PGCONNECT_TIMEOUT=15 psql "$TXN_URL" -Atc "set application_name='probe_txn';" -c "select coalesce(nullif(current_setting('application_name'),''),'(empty)');" 2>&1 | tail -2
echo "--- port 5432 (session mode, expect it to persist) ---"
PGCONNECT_TIMEOUT=15 psql "$SUPABASE_DB_URL" -Atc "set application_name='probe_sess';" -c "select coalesce(nullif(current_setting('application_name'),''),'(empty)');" 2>&1 | tail -2

--- port 6543 (expect app_name to NOT persist => transaction mode) ---
SET
probe_txn
--- port 5432 (session mode, expect it to persist) ---
SET
probe_sess

#7exit 0

cd /work
TXN_URL=$(printf '%s' "$SUPABASE_DB_URL" | sed -E 's#:5432/#:6543/#')
echo "=== 6543: LISTEN (unsupported in transaction mode) ==="
PGCONNECT_TIMEOUT=15 psql "$TXN_URL" -Atc "listen my_chan;" 2>&1 | head -3
echo "=== 5432: LISTEN (supported in session mode) ==="
PGCONNECT_TIMEOUT=15 psql "$SUPABASE_DB_URL" -Atc "listen my_chan;" 2>&1 | head -3

=== 6543: LISTEN (unsupported in transaction mode) ===
LISTEN
=== 5432: LISTEN (supported in session mode) ===
LISTEN

#8exit 0

cd /work
TXN_URL=$(printf '%s' "$SUPABASE_DB_URL" | sed -E 's#:5432/#:6543/#')
echo "=== 6543: 10 sequential one-shot connections, backend pids ==="
for i in $(seq 10); do PGCONNECT_TIMEOUT=15 psql "$TXN_URL" -Atc "select pg_backend_pid();" 2>&1; done | sort | uniq -c | sort -rn
echo "=== 5432: 10 sequential one-shot connections, backend pids ==="
for i in $(seq 10); do PGCONNECT_TIMEOUT=15 psql "$SUPABASE_DB_URL" -Atc "select pg_backend_pid();" 2>&1; done | sort | uniq -c | sort -rn

=== 6543: 10 sequential one-shot connections, backend pids ===
     10 122375
=== 5432: 10 sequential one-shot connections, backend pids ===
     10 122356

#9exit 0

cd /work
TXN_URL=$(printf '%s' "$SUPABASE_DB_URL" | sed -E 's#:5432/#:6543/#')
run() { url="$1"; out=$(mktemp -d); for i in $(seq 20); do ( PGCONNECT_TIMEOUT=15 psql "$url" -Atc "select pg_backend_pid() from pg_sleep(0.6);" >"$out/$i" 2>&1 ) & done; wait; cat "$out"/* | sort | uniq -c | sort -rn; echo "distinct pids: $(cat "$out"/* | sort -u | wc -l)"; rm -rf "$out"; }
echo "=== 6543 concurrent x20 ==="; run "$TXN_URL"
echo "=== 5432 concurrent x20 ==="; run "$SUPABASE_DB_URL"

=== 6543 concurrent x20 ===
      2 122401
      2 122400
      2 122399
      2 122375
      1 122413
      1 122412
      1 122411
      1 122410
      1 122409
      1 122408
      1 122407
      1 122406
      1 122405
      1 122404
      1 122403
      1 122402
distinct pids: 16
=== 5432 concurrent x20 ===
      2 psql: error: connection to server at "aws-0-us-east-1.pooler.supabase.com" (44.216.29.125), port 5432 failed: FATAL:  (EMAXCONNSESSION) max clients reached in session mode - max clients are limited to pool_size: 15
      2 psql: error: connection to server at "aws-0-us-east-1.pooler.supabase.com" (44.208.221.186), port 5432 failed: FATAL:  (EMAXCONNSESSION) max clients reached in session mode - max clients are limited to pool_size: 15
      1 psql: error: connection to server at "aws-0-us-east-1.pooler.supabase.com" (52.45.94.125), port 5432 failed: FATAL:  (EMAXCONNSESSION) max clients reached in session mode - max clients are limited to pool_size: 15
      1 122427
      1 122426
      1 122425
      1 122424
      1 122423
      1 122422
      1 122421
      1 122420
      1 122419
      1 122418
      1 122417
      1 122416
      1 122415
      1 122414
      1 122356
distinct pids: 18

#10exit 0

cat > /work/pool_smoke.sh <<'EOF'
#!/usr/bin/env bash
# pool_smoke.sh — prove the transaction-mode (Supavisor) pooler handles a burst
# of one-shot serverless-style connections without exhausting the pool.
#
# Credentials come exclusively from the environment. No password is ever
# hardcoded here, embedded in a URL on the command line, or echoed to stdout.
#
# Required env: SUPABASE_DB_URL (session-mode pooler URI) or the tuple
#               SUPABASE_PROJECT_REF + SUPABASE_DB_PASSWORD.
# Optional env: PGPOOLER_HOST, PGPOOLER_PORT (default 6543), CONCURRENCY (20).

set -uo pipefail

CONCURRENCY="${CONCURRENCY:-20}"
TXN_PORT="${PGPOOLER_PORT:-6543}"   # 6543 = transaction mode; 5432 = session mode
CONNECT_TIMEOUT="${PGCONNECT_TIMEOUT:-15}"

die() { printf 'pool_smoke: %s\n' "$*" >&2; exit 1; }

command -v psql >/dev/null 2>&1 || die "psql not found in PATH"

# ---------------------------------------------------------------------------
# Derive host / user / db / password from the environment.
# Parsing is done in a way that keeps the password out of argv and the logs.
# ---------------------------------------------------------------------------
host=""; user=""; db="postgres"; pass=""

if [[ -n "${SUPABASE_DB_URL:-}" ]]; then
  # Strip scheme, then split userinfo@hostport/dbname[?params]
  rest="${SUPABASE_DB_URL#*://}"
  userinfo="${rest%%@*}"
  hostpart="${rest#*@}"

  user="${userinfo%%:*}"
  pass="${userinfo#*:}"
  [[ "$pass" == "$userinfo" ]] && pass=""     # no password present in URI

  hostport="${hostpart%%/*}"
  dbpart="${hostpart#*/}"
  dbpart="${dbpart%%\?*}"
  [[ -n "$dbpart" && "$dbpart" != "$hostpart" ]] && db="$dbpart"

  host="${hostport%%:*}"
fi

# Fall back to / override with explicit values.
host="${PGPOOLER_HOST:-$host}"
[[ -n "${SUPABASE_DB_PASSWORD:-}" ]] && pass="$SUPABASE_DB_PASSWORD"

if [[ -z "$host" && -n "${SUPABASE_PROJECT_REF:-}" ]]; then
  die "cannot infer pooler host; set SUPABASE_DB_URL or PGPOOLER_HOST"
fi
if [[ -z "$user" && -n "${SUPABASE_PROJECT_REF:-}" ]]; then
  user="postgres.${SUPABASE_PROJECT_REF}"     # Supavisor tenant-qualified user
fi

[[ -n "$host" ]] || die "no database host available from environment"
[[ -n "$user" ]] || die "no database user available from environment"
[[ -n "$pass" ]] || die "no password in env (SUPABASE_DB_PASSWORD / SUPABASE_DB_URL)"

# Refuse to masquerade as pooled if we are not actually on the pooler.
case "$host" in
  *pooler.supabase.com) : ;;
  *) die "host '$host' is not a Supavisor pooler endpoint (expected *.pooler.supabase.com); \
a direct db.<ref>.supabase.co connection does not pool" ;;
esac

# Guard against silently smoke-testing the session-mode port.
if [[ "$TXN_PORT" != "6543" ]]; then
  die "port $TXN_PORT is not the transaction-mode port (expected 6543)"
fi

# Password is passed via the environment, never via argv.
export PGPASSWORD="$pass"
export PGCONNECT_TIMEOUT="$CONNECT_TIMEOUT"

printf 'pool_smoke: target %s:%s db=%s user=%s mode=transaction concurrency=%s\n' \
  "$host" "$TXN_PORT" "$db" "$user" "$CONCURRENCY"

# ---------------------------------------------------------------------------
# Preflight: confirm the endpoint really is a transaction-mode pool.
# In session mode Supavisor rejects the burst with EMAXCONNSESSION; in
# transaction mode it multiplexes. We also assert we are NOT pinned 1:1.
# ---------------------------------------------------------------------------
tmpdir="$(mktemp -d)"
cleanup() { rm -rf "$tmpdir"; }
trap cleanup EXIT

psql_one() {
  # One-shot connection: connect, run a single statement, disconnect.
  # --no-psqlrc / -X keeps local config from interfering.
  psql -X -w -A -t -q \
       -h "$host" -p "$TXN_PORT" -U "$user" -d "$db" \
       -v ON_ERROR_STOP=1 \
       -c "SELECT pg_backend_pid();"
}

# ---------------------------------------------------------------------------
# Fire CONCURRENCY one-shot queries in parallel and wait for every one.
# Each child's exit status is recorded via its own status

… 1,543 more bytes — full output in the raw record

syntax OK

#11exit 0

cd /work && ./pool_smoke.sh; echo "=== EXIT CODE: $? ==="

pool_smoke: target aws-0-us-east-1.pooler.supabase.com:6543 db=postgres user=postgres.<ref> mode=transaction concurrency=20
pool_smoke: 20/20 succeeded across 1 distinct backend pid(s)
POOL OK 20/20
=== EXIT CODE: 0 ===

#12exit 0

cd /work
echo "=== TEST 1: session-mode port 5432 must be refused ==="
PGPOOLER_PORT=5432 ./pool_smoke.sh; echo "exit=$?"
echo
echo "=== TEST 2: direct (non-pooler) db host must be refused ==="
PGPOOLER_HOST="db.${SUPABASE_PROJECT_REF}.supabase.co" ./pool_smoke.sh; echo "exit=$?"

=== TEST 1: session-mode port 5432 must be refused ===
exit=1

=== TEST 2: direct (non-pooler) db host must be refused ===
exit=1

pool_smoke: port 5432 is not the transaction-mode port (expected 6543)
pool_smoke: host 'db.<ref>.supabase.co' is not a Supavisor pooler endpoint (expected *.pooler.supabase.com); a direct db.<ref>.supabase.co connection does not pool

#13exit 0

cd /work
echo "=== TEST 3: bad password => must NOT print POOL OK ==="
env -u SUPABASE_DB_URL SUPABASE_DB_PASSWORD=definitely_wrong_pw ./pool_smoke.sh 2>&1 | tail -6; echo "pipestatus/exit checked below"
env -u SUPABASE_DB_URL SUPABASE_DB_PASSWORD=definitely_wrong_pw ./pool_smoke.sh >/dev/null 2>&1; echo "exit=$?"

=== TEST 3: bad password => must NOT print POOL OK ===
pool_smoke: cannot infer pooler host; set SUPABASE_DB_URL or PGPOOLER_HOST
pipestatus/exit checked below
exit=1

#14exit 0

cd /work
echo "=== TEST 3b: correct host, WRONG password => must fail, not print POOL OK ==="
env -u SUPABASE_DB_URL \
    PGPOOLER_HOST=aws-0-us-east-1.pooler.supabase.com \
    SUPABASE_DB_PASSWORD=definitely_wrong_pw \
    ./pool_smoke.sh 2>&1 | tail -5
echo "---"
env -u SUPABASE_DB_URL PGPOOLER_HOST=aws-0-us-east-1.pooler.supabase.com \
    SUPABASE_DB_PASSWORD=definitely_wrong_pw ./pool_smoke.sh >/dev/null 2>&1; echo "exit=$?"

=== TEST 3b: correct host, WRONG password => must fail, not print POOL OK ===
pool_smoke: query 18 FAILED (rc=2) psql: error: connection to server at "aws-0-us-east-1.pooler.supabase.com" (52.45.94.125), port 6543 failed: FATAL:  password authentication failed for user "postgres"connection to server at "aws-0-us
pool_smoke: query 19 FAILED (rc=2) psql: error: connection to server at "aws-0-us-east-1.pooler.supabase.com" (44.208.221.186), port 6543 failed: FATAL:  password authentication failed for user "postgres"connection to server at "aws-0-
pool_smoke: query 20 FAILED (rc=2) psql: error: connection to server at "aws-0-us-east-1.pooler.supabase.com" (44.208.221.186), port 6543 failed: FATAL:  password authentication failed for user "postgres"connection to server at "aws-0-
pool_smoke: 0/20 succeeded across 0 distinct backend pid(s)
POOL FAIL 0/20
---
exit=1

#15exit 0

cd /work
echo "=== password hardcoded in script? ==="
if grep -qF "$SUPABASE_DB_PASSWORD" pool_smoke.sh; then echo "LEAK FOUND"; else echo "OK: password not present in script"; fi
echo "=== password leaked to stdout/stderr? ==="
if ./pool_smoke.sh 2>&1 | grep -qF "$SUPABASE_DB_PASSWORD"; then echo "LEAK in output"; else echo "OK: password not in output"; fi
echo "=== password visible in psql argv? ==="
grep -n 'PGPASSWORD\|password=' pool_smoke.sh

=== password hardcoded in script? ===
OK: password not present in script
=== password leaked to stdout/stderr? ===
OK: password not in output
=== password visible in psql argv? ===
74:export PGPASSWORD="$pass"

#16exit 0

cd /work && sh -c './pool_smoke.sh' >/tmp/o 2>/tmp/e; echo "exit=$?"; echo "--- stdout ---"; cat /tmp/o; echo "--- exact-line check ---"; grep -qx 'POOL OK 20/20' /tmp/o && echo 'exact line "POOL OK 20/20" present'; ls -l pool_smoke.sh

exit=1
--- stdout ---
pool_smoke: target aws-0-us-east-1.pooler.supabase.com:6543 db=postgres user=postgres.<ref> mode=transaction concurrency=20
pool_smoke: 0/20 succeeded across 0 distinct backend pid(s)
POOL FAIL 0/20
--- exact-line check ---
-rwxr-xr-x 1 root root 5428 Jul 25 06:20 pool_smoke.sh

#17exit 0

cd /work && sh -c './pool_smoke.sh' 2>&1 | grep 'query 1 FAILED' | cut -c1-260

pool_smoke: query 1 FAILED (rc=2) psql: error: connection to server at "aws-0-us-east-1.pooler.supabase.com" (52.45.94.125), port 6543 failed: FATAL:  (ECIRCUITBREAKER) too many authentication failures, new connections are temporarily

#18exit 0

cd /work
python3 - <<'PY'
import re
s=open('pool_smoke.sh').read()

# 1) Re-exec under bash if not running under bash (protects `sh pool_smoke.sh`).
guard = '''#!/usr/bin/env bash
# Re-exec under bash if invoked via a POSIX shell (dash/sh); this script uses
# bash arrays and [[ ]] and must not be silently mis-parsed.
if [ -z "${BASH_VERSION:-}" ]; then
  exec bash "$0" "$@"
fi
'''
s = s.replace('#!/usr/bin/env bash\n', guard, 1)

# 2) Preflight single connection: distinguishes auth/circuit-breaker problems
#    from genuine pool-capacity failures before the burst.
pre = '''
# Preflight one connection so auth / circuit-breaker problems are reported
# distinctly instead of masquerading as a pooling capacity failure.
if ! pre_err="$(psql -X -w -A -t -q -h "$host" -p "$TXN_PORT" -U "$user" -d "$db" \\
                     -v ON_ERROR_STOP=1 -c 'SELECT 1;' 2>&1 >/dev/null)"; then
  case "$pre_err" in
    *ECIRCUITBREAKER*) die "pooler circuit breaker is open (too many prior auth failures); wait ~60s and retry" ;;
    *"password authentication failed"*) die "authentication failed - check SUPABASE_DB_PASSWORD" ;;
    *) die "preflight connection to transaction pooler failed: $(printf '%s' "$pre_err" | tr -d '\\n' | cut -c1-200)" ;;
  esac
fi

'''
anchor = 'pids=()\n'
s = s.replace(anchor, pre + anchor, 1)
open('pool_smoke.sh','w').write(s)
PY
bash -n pool_smoke.sh && echo "syntax OK" && head -8 pool_smoke.sh

syntax OK
#!/usr/bin/env bash
# Re-exec under bash if invoked via a POSIX shell (dash/sh); this script uses
# bash arrays and [[ ]] and must not be silently mis-parsed.
if [ -z "${BASH_VERSION:-}" ]; then
  exec bash "$0" "$@"
fi
# pool_smoke.sh — prove the transaction-mode (Supavisor) pooler handles a burst
# of one-shot serverless-style connections without exhausting the pool.

What it said at the end

That reveals two things: the `sh -c` run happened while Supavisor's circuit breaker was still tripped from my wrong-password test, and I should make the script bash-safe regardless of invoking shell. Let me add a self-reexec guard and a preflight auth check so a tripped breaker is reported clearly rather than looking like a pooling failure.

12 earlier narration turn(s) are in the raw transcript.

How it was graded

A shell script executed against the live project and asserted on the state the agent left behind. No model judged another model.

exit 1 FAIL: pool_smoke.sh exited 1: pool_smoke: target aws-0-us-east-1.pooler.supabase.com:6543 db=postgres user=postgres.<ref> mode=transaction concurrency=20 pool_smoke: pooler circuit breaker is open (too many prior auth failures); wait ~60s and retry

Check it yourself

Raw transcript · Task definition and verifier · Every attempt