Can Agents Run It? · evidence — Issue 1: Supabase

openai/gpt-5.6-luna · d2_pooling · rep 1

pass
model
openai/gpt-5.6-luna
task
d2_pooling
rep
1
day
2
steps
2
tokens
4,030
est. cost
$0.011
agent time
15s
wave
s4b
started
2026-07-25T05:51:05Z

The task as the agent received it

You have operator access to a hosted Supabase project. Credentials are in environment variables: SUPABASE_URL, SUPABASE_PROJECT_REF, SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_DB_URL (a session-mode Postgres connection string), SUPABASE_DB_PASSWORD. Our serverless workers keep exhausting database connections. Write an executable script named pool_smoke.sh in the current working directory that proves the right pooling setup: - it must connect through the project's TRANSACTION-MODE connection pooler (not the session-mode port, not a direct database connection); - it must run 20 concurrent one-shot queries (each e.g. SELECT pg_backend_pid()), wait for all of them, and print the single line "POOL OK 20/20" only if all 20 succeeded, then exit 0; - it must take credentials from the environment variables — never hardcode the database password. Run it yourself to confirm it passes before finishing.

What it ran — 2 commands

#1exit 0
printf 'psql='; command -v psql || true; printf 'python='; command -v python3 || true; printf 'dburl_present='; [ -n "$SUPABASE_DB_URL" ] && echo yes || echo no; printf 'url_host='; python3 - <<'PY'
import os, urllib.parse
u=os.environ.get('SUPABASE_DB_URL','')
try:
 p=urllib.parse.urlsplit(u); print(p.hostname or '')
except: print('')
PY
printf 'url_port='; python3 - <<'PY'
import os, urllib.parse
u=os.environ.get('SUPABASE_DB_URL','')
try: print(urllib.parse.urlsplit(u).port or '')
except: print('')
PY
psql=/usr/bin/psql
python=/usr/bin/python3
dburl_present=yes
url_host=aws-0-us-east-1.pooler.supabase.com
url_port=5432
#2exit 0
cat > pool_smoke.sh <<'EOF'
#!/usr/bin/env bash
# Verify that the Supabase transaction-mode pooler can handle concurrent one-shot queries.
set -u

: "${SUPABASE_DB_URL:?SUPABASE_DB_URL is required}"
: "${SUPABASE_DB_PASSWORD:?SUPABASE_DB_PASSWORD is required}"

# Read only connection metadata from the supplied session-mode URL.  The password is
# deliberately not read from or placed in the command line; psql gets it via PGPASSWORD.
if ! IFS=$'\t' read -r DB_HOST DB_USER DB_NAME < <(python3 - "$SUPABASE_DB_URL" <<'PY'
import sys
from urllib.parse import unquote, urlsplit

p = urlsplit(sys.argv[1])
if p.scheme not in ("postgres", "postgresql") or not p.hostname or not p.username:
    raise SystemExit("SUPABASE_DB_URL is not a PostgreSQL URL")
if not p.path or p.path == "/":
    raise SystemExit("SUPABASE_DB_URL has no database name")
host = p.hostname.lower()
# A pooler hostname plus the transaction port is required.  In particular, this
# prevents accidentally testing the direct/session database endpoint.
if not host.endswith(".pooler.supabase.com"):
    raise SystemExit("SUPABASE_DB_URL does not name a Supabase pooler host")
print("\t".join((host, unquote(p.username), unquote(p.path[1:]))))
PY
); then
  echo "could not parse SUPABASE_DB_URL" >&2
  exit 1
fi

if [[ -z "$DB_HOST" || -z "$DB_USER" || -z "$DB_NAME" ]]; then
  echo "incomplete database connection details" >&2
  exit 1
fi

# 6543 is Supabase's transaction-mode pooler port (5432 is session mode).
POOL_PORT=6543
WORKDIR=$(mktemp -d "${TMPDIR:-/tmp}/pool-smoke.XXXXXX")
trap 'rm -rf "$WORKDIR"' EXIT

pids=()
for i in $(seq 1 20); do
  PGPASSWORD=$SUPABASE_DB_PASSWORD psql \
    "host=$DB_HOST port=$POOL_PORT user=$DB_USER dbname=$DB_NAME sslmode=require connect_timeout=10" \
    --no-psqlrc --quiet --tuples-only --no-align \
    -c 'SELECT pg_backend_pid();' \
    >"$WORKDIR/$i.out" 2>"$WORKDIR/$i.err" &
  pids+=("$!")
done

failed=0
for pid in "${pids[@]}"; do
  if ! wait "$pid"; then
    failed=1
  fi
done

if (( failed == 0 )); then
  echo 'POOL OK 20/20'
  exit 0
fi

echo "POOL FAILED: one or more transaction-pooler queries failed" >&2
exit 1
EOF
chmod +x pool_smoke.sh
./pool_smoke.sh; rc=$?; echo "exit=$rc"; exit $rc
POOL OK 20/20
exit=0

What it said at the end

Created and executable `pool_smoke.sh`. It: - Uses the Supabase pooler hostname from `SUPABASE_DB_URL`. - Forces transaction-mode port `6543`, not session-mode `5432`. - Runs 20 concurrent one-shot `pg_backend_pid()` queries. - Waits for every query and prints `POOL OK 20/20` only on complete success. - Uses `SUPABASE_DB_PASSWORD` via `PGPASSWORD` without hardcoding credentials. Confirmed by running it successfully: ```text POOL OK 20/20 ```

How it was graded

A shell script executed against the live project and asserted on the state the agent left behind. No model judged another model.

exit 0 OK

Check it yourself

Raw transcript · Task definition and verifier · Every attempt