Can Agents Run It? · evidence — Issue 1: Supabase

openai/gpt-5.6-luna · d2_pooling · rep 2

pass
model
openai/gpt-5.6-luna
task
d2_pooling
rep
2
day
2
steps
3
tokens
5,078
est. cost
$0.014
agent time
21s
wave
s4b
started
2026-07-25T05:51:30Z

The task as the agent received it

You have operator access to a hosted Supabase project. Credentials are in environment variables: SUPABASE_URL, SUPABASE_PROJECT_REF, SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_DB_URL (a session-mode Postgres connection string), SUPABASE_DB_PASSWORD. Our serverless workers keep exhausting database connections. Write an executable script named pool_smoke.sh in the current working directory that proves the right pooling setup: - it must connect through the project's TRANSACTION-MODE connection pooler (not the session-mode port, not a direct database connection); - it must run 20 concurrent one-shot queries (each e.g. SELECT pg_backend_pid()), wait for all of them, and print the single line "POOL OK 20/20" only if all 20 succeeded, then exit 0; - it must take credentials from the environment variables — never hardcode the database password. Run it yourself to confirm it passes before finishing.

What it ran — 3 commands

#1exit 0
env | grep '^SUPABASE_' | sed -E 's/=.*/=<set>/' && printf '%s\n' "$SUPABASE_DB_URL" | sed -E 's#(://[^:]+:)[^@]+@#\1***@#'
SUPABASE_DB_PASSWORD=<set>
SUPABASE_DB_URL=<set>
SUPABASE_URL=<set>
SUPABASE_SERVICE_ROLE_KEY=<set>
SUPABASE_PROJECT_REF=<set>
SUPABASE_ANON_KEY=<set>
postgresql://postgres.<ref>:***@aws-0-us-east-1.pooler.supabase.com:5432/postgres
#2exit 0
command -v psql || true; python3 - <<'PY'
import os
from urllib.parse import urlsplit
u=urlsplit(os.environ['SUPABASE_DB_URL'])
print(u.hostname,u.port,u.username,u.path)
PY
/usr/bin/psql
aws-0-us-east-1.pooler.supabase.com 5432 postgres.<ref> /postgres
#3exit 0
cat > pool_smoke.sh <<'EOF'
#!/usr/bin/env bash
# Verify that the Supabase transaction-mode pooler can serve 20 one-shot queries.
set -u

: "${SUPABASE_DB_URL:?SUPABASE_DB_URL is required}"
: "${SUPABASE_DB_PASSWORD:?SUPABASE_DB_PASSWORD is required}"
: "${SUPABASE_PROJECT_REF:?SUPABASE_PROJECT_REF is required}"

# Extract only non-secret connection components from the supplied session URL.
# The port is deliberately ignored and replaced with Supabase's transaction port.
read -r db_host db_user db_name < <(python3 - "$SUPABASE_DB_URL" <<'PY'
import sys
from urllib.parse import urlsplit, unquote

u = urlsplit(sys.argv[1])
if u.scheme not in ("postgres", "postgresql") or not u.hostname or not u.username:
    raise SystemExit("invalid SUPABASE_DB_URL")
print(u.hostname, unquote(u.username), unquote(u.path.lstrip("/")) or "postgres")
PY
)

# Do not accidentally turn a session/direct URL into this test's endpoint.
case "$db_host" in
  *.pooler.supabase.com) ;;
  *) echo "SUPABASE_DB_URL is not a Supabase pooler URL" >&2; exit 1 ;;
esac
if [[ "$db_user" != "postgres.${SUPABASE_PROJECT_REF}" ]]; then
  echo "SUPABASE_DB_URL is for a different Supabase project" >&2
  exit 1
fi

workdir=$(mktemp -d)
trap 'rm -rf "$workdir"' EXIT
pids=()

for i in $(seq 1 20); do
  (
    PGPASSWORD="$SUPABASE_DB_PASSWORD" \
      psql -X -qAt \
        -h "$db_host" -p 6543 \
        -U "$db_user" -d "$db_name" \
        "sslmode=require" \
        -c 'SELECT pg_backend_pid();' \
        >"$workdir/$i.out" 2>"$workdir/$i.err"
  ) &
  pids+=("$!")
done

failed=0
for pid in "${pids[@]}"; do
  if ! wait "$pid"; then
    failed=1
  fi
done

if (( failed == 0 )); then
  echo 'POOL OK 20/20'
  exit 0
fi

echo 'POOL FAILED (one or more transaction-pooler queries failed)' >&2
exit 1
EOF
chmod +x pool_smoke.sh
./pool_smoke.sh; rc=$?; echo "exit=$rc"; sed -n '1,8p' pool_smoke.sh
POOL OK 20/20
exit=0
#!/usr/bin/env bash
# Verify that the Supabase transaction-mode pooler can serve 20 one-shot queries.
set -u

: "${SUPABASE_DB_URL:?SUPABASE_DB_URL is required}"
: "${SUPABASE_DB_PASSWORD:?SUPABASE_DB_PASSWORD is required}"
: "${SUPABASE_PROJECT_REF:?SUPABASE_PROJECT_REF is required}"

What it said at the end

Created executable `pool_smoke.sh` in the current directory. It: - Uses environment-provided credentials only. - Connects to the Supabase transaction-mode pooler on port `6543`. - Runs 20 concurrent one-shot `SELECT pg_backend_pid()` queries. - Waits for all queries and reports success only when all pass. Confirmed by running it successfully: ```text POOL OK 20/20 ```

How it was graded

A shell script executed against the live project and asserted on the state the agent left behind. No model judged another model.

exit 0 OK

Check it yourself

Raw transcript · Task definition and verifier · Every attempt