Can Agents Run It? · evidence — Issue 1: Supabase

moonshotai/kimi-k3 · d2_pooling · rep 1

fail
model
moonshotai/kimi-k3
task
d2_pooling
rep
1
day
2
steps
9
tokens
19,433
est. cost
$0.146
agent time
226s
wave
s4
started
2026-07-24T06:12:47Z

The task as the agent received it

You have operator access to a hosted Supabase project. Credentials are in environment variables: SUPABASE_URL, SUPABASE_PROJECT_REF, SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_DB_URL (a session-mode Postgres connection string), SUPABASE_DB_PASSWORD. Our serverless workers keep exhausting database connections. Write an executable script named pool_smoke.sh in the current working directory that proves the right pooling setup: - it must connect through the project's TRANSACTION-MODE connection pooler (not the session-mode port, not a direct database connection); - it must run 20 concurrent one-shot queries (each e.g. SELECT pg_backend_pid()), wait for all of them, and print the single line "POOL OK 20/20" only if all 20 succeeded, then exit 0; - it must take credentials from the environment variables — never hardcode the database password. Run it yourself to confirm it passes before finishing.

What it ran — 9 commands

#1exit 0
echo "SUPABASE_URL=$SUPABASE_URL"; echo "SUPABASE_PROJECT_REF=$SUPABASE_PROJECT_REF"; echo "SUPABASE_DB_URL=$SUPABASE_DB_URL" | sed -E 's#(://[^:]+:)[^@]+@#\1****@#'; echo "DB_PASSWORD set: $([ -n "$SUPABASE_DB_PASSWORD" ] && echo yes || echo no)"; echo "ANON set: $([ -n "$SUPABASE_ANON_KEY" ] && echo yes || echo no)"; echo "SERVICE set: $([ -n "$SUPABASE_SERVICE_ROLE_KEY" ] && echo yes || echo no)"
SUPABASE_URL=https://<ref>.supabase.co
SUPABASE_PROJECT_REF=<ref>
SUPABASE_DB_URL=postgresql://postgres.<ref>:<redacted>@aws-0-us-east-1.pooler.supabase.com:5432/postgres
DB_PASSWORD set: yes
ANON set: yes
SERVICE set: yes
#2exit 0
which psql pg_isready python3 jq curl 2>/dev/null; psql --version 2>/dev/null; echo "---"; cat /etc/os-release 2>/dev/null | head -3
/usr/bin/psql
/usr/bin/pg_isready
/usr/bin/python3
/usr/bin/jq
/usr/bin/curl
psql (PostgreSQL) 18.4 (Debian 18.4-1.pgdg12+1)
---
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
#3exit 0
timeout 5 bash -c '</dev/tcp/aws-0-us-east-1.pooler.supabase.com/6543' && echo "6543 reachable" || echo "6543 NOT reachable"; timeout 5 bash -c '</dev/tcp/aws-0-us-east-1.pooler.supabase.com/5432' && echo "5432 reachable" || echo "5432 NOT reachable"
6543 reachable
5432 reachable
#4exit 0
PGPASSWORD="$SUPABASE_DB_PASSWORD" psql "host=aws-0-us-east-1.pooler.supabase.com port=6543 dbname=postgres user=postgres.$SUPABASE_PROJECT_REF sslmode=require" -tAc "SELECT pg_backend_pid(), 'connected via transaction pooler'" 2>&1
14255|connected via transaction pooler
#5exit 0
cat > pool_smoke.sh <<'EOF'
#!/usr/bin/env bash
#
# pool_smoke.sh — prove the Supabase TRANSACTION-mode connection pooler
# (PgBouncer/Supavisor, port 6543) serves 20 concurrent one-shot queries.
#
# It deliberately uses ONLY the transaction-mode pooler:
#   - never the session-mode pooler port (5432)
#   - never a direct database connection (db.<ref>.supabase.co)
#
# Credentials come from environment variables; nothing is hardcoded:
#   SUPABASE_DB_URL        used to derive pooler host / user / dbname
#                          (its port is overridden to 6543 = transaction mode)
#   SUPABASE_DB_PASSWORD   database password (never echoed, never stored)
#   SUPABASE_PROJECT_REF   fallback for building the pooler username
#   SUPABASE_POOLER_HOST   optional override for the pooler hostname
#
# On full success prints exactly one line on stdout:  POOL OK 20/20
# and exits 0. Any failure exits non-zero and never prints that line.

set -euo pipefail

CONCURRENCY=20
POOLER_PORT=6543   # transaction-mode pooler port. Session mode is 5432 — do NOT use.

die() { echo "pool_smoke: ERROR: $*" >&2; exit 1; }

command -v psql >/dev/null 2>&1 || die "psql not found in PATH"

# --- credentials from environment --------------------------------------------
[ -n "${SUPABASE_DB_URL:-}" ]      || die "SUPABASE_DB_URL is not set"
[ -n "${SUPABASE_DB_PASSWORD:-}" ] || die "SUPABASE_DB_PASSWORD is not set"

# --- derive pooler host / user / dbname from SUPABASE_DB_URL ------------------
# e.g. postgresql://postgres.<ref>:<pw>@aws-0-<region>.pooler.supabase.com:5432/postgres
url_no_scheme="${SUPABASE_DB_URL#*://}"
userinfo="${url_no_scheme%%@*}"
hostport_db="${url_no_scheme#*@}"
hostport="${hostport_db%%/*}"
DBNAME="${hostport_db#*/}"; DBNAME="${DBNAME%%\?*}"; DBNAME="${DBNAME:-postgres}"
POOLER_HOST="${SUPABASE_POOLER_HOST:-${hostport%%:*}}"
DBUSER="${userinfo%%:*}"   # pooler requires the postgres.<project-ref> form

# Fallback: build the pooler-style username from the project ref if needed.
if [[ "$DBUSER" != *.* && -n "${SUPABASE_PROJECT_REF:-}" ]]; then
  DBUSER="postgres.${SUPABASE_PROJECT_REF}"
fi

# --- safety rails: prove this is the transaction pooler, nothing else ---------
[[ "$POOLER_HOST" == *.pooler.supabase.com ]] \
  || die "host '$POOLER_HOST' is not the Supabase connection pooler (*.pooler.supabase.com); refusing direct/session database connection"
[ "$POOLER_PORT" = "6543" ] \
  || die "port $POOLER_PORT is not the transaction-mode pooler port (6543)"

export PGPASSWORD="$SUPABASE_DB_PASSWORD"   # from env; never hardcoded
export PGCONNECT_TIMEOUT=10

run_one() {
  psql "host=$POOLER_HOST port=$POOLER_PORT dbname=$DBNAME user=$DBUSER sslmode=require" \
       -X -q -t -A -c "SELECT pg_backend_pid()" >/dev/null 2>&1
}

echo "pool_smoke: $CONCURRENCY concurrent one-shot queries via transaction-mode pooler $POOLER_HOST:$POOLER_PORT ..." >&2

pids=()
for _ in $(seq 1 "$CONCURRENCY"); do
  run_one &
  pids+=($!)
done

ok=0
for p in "${pids[@]}"; do
  if wait "$p"; then ok=$((ok + 1)); fi
done

if [ "$ok" -eq "$CONCURRENCY" ]; then
  echo "POOL OK ${ok}/${CONCURRENCY}"
  exit 0
fi

echo "pool_smoke: FAILURE — only $ok/$CONCURRENCY queries succeeded" >&2
exit 1
EOF
chmod +x pool_smoke.sh
ls -l pool_smoke.sh && bash -n pool_smoke.sh && echo "syntax OK"
-rwxr-xr-x 1 root root 3186 Jul 24 06:15 pool_smoke.sh
syntax OK
#6exit 0
./pool_smoke.sh; echo "exit_code=$?"
POOL OK 20/20
exit_code=0
pool_smoke: 20 concurrent one-shot queries via transaction-mode pooler aws-0-us-east-1.pooler.supabase.com:6543 ...
#7exit 0
for i in 1 2 3; do out=$(./pool_smoke.sh 2>/dev/null); rc=$?; echo "run$i: rc=$rc stdout=[$out]"; done
run1: rc=0 stdout=[POOL OK 20/20]
run2: rc=0 stdout=[POOL OK 20/20]
run3: rc=0 stdout=[POOL OK 20/20]
#8exit 0
SUPABASE_DB_PASSWORD="definitely-wrong" ./pool_smoke.sh >/tmp/out.txt 2>/tmp/err.txt; echo "rc=$?"; echo "stdout: $(cat /tmp/out.txt)"; echo "stderr tail: $(tail -1 /tmp/err.txt)"
rc=1
stdout: 
stderr tail: pool_smoke: FAILURE — only 0/20 queries succeeded
#9exit 0
grep -qF "$SUPABASE_DB_PASSWORD" pool_smoke.sh && echo "PASSWORD LEAKED IN SCRIPT" || echo "no password in script"; grep -c "6543" pool_smoke.sh; grep -c "pg_backend_pid" pool_smoke.sh
no password in script
5
1

What it said at the end

Done. The script `pool_smoke.sh` is in place, executable, and verified against the live project. **What it does:** - **Connects only via the transaction-mode pooler** — it derives the host (`aws-0-us-east-1.pooler.supabase.com`), user (`postgres.<project-ref>`), and dbname from `SUPABASE_DB_URL`, but overrides the port to **6543** (transaction mode; 5432 = session mode is never used). It also hard-refuses any host that isn't `*.pooler.supabase.com`, so a direct connection to `db.<ref>.supabase.co` is impossible. - **Runs 20 concurrent one-shot queries** — 20 background `psql -c "SELECT pg_backend_pid()"` processes are launched in parallel, `wait`ed on individually, and each exit code is counted. Only when all 20 succeed does it print the single line `POOL OK 20/20` and exit 0. - **Never hardcodes credentials** — the password comes from `SUPABASE_DB_PASSWORD` (passed via `PGPASSWORD`), everything else from `SUPABASE_DB_URL` / `SUPABASE_PROJECT_REF` / optional `SUPABASE_POOLER_HOST` override. Verified the password string appears nowhere in the script. **Verification performed:** - 3 consecutive runs: all printed exactly `POOL OK 20/20` with exit code 0. - Negative test with a wrong password: exits 1, no success line printed.

6 earlier narration turn(s) are in the raw transcript.

How it was graded

A shell script executed against the live project and asserted on the state the agent left behind. No model judged another model.

exit 1 FAIL: pool_smoke.sh exited 1: pool_smoke: 20 concurrent one-shot queries via transaction-mode pooler aws-0-us-east-1.pooler.supabase.com:6543 ... pool_smoke: FAILURE — only 0/20 queries succeeded

Check it yourself

Raw transcript · Task definition and verifier · Every attempt