Can Agents Run It? · evidence — Issue 1: Supabase

tencent/hy3 · d2_webhooks · rep 2

fail
model
tencent/hy3
task
d2_webhooks
rep
2
day
2
steps
5
tokens
13,110
est. cost
$0.004
agent time
47s
wave
s4c
started
2026-07-26T03:06:49Z

The task as the agent received it

You have operator access to a hosted Supabase project. Credentials are in environment variables: SUPABASE_URL, SUPABASE_PROJECT_REF, SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_DB_URL, SUPABASE_DB_PASSWORD. Every INSERT into the existing table public.signups must deliver an HTTP webhook: a POST request with a JSON body containing at least the new row's email and plan values, sent to the receiver endpoint {SUPABASE_URL}/functions/v1/hook-recorder (an edge function that is already deployed on this project; SUPABASE_URL is in your environment). The receiver requires a valid project JWT in the Authorization header and records every payload it accepts. Implement the delivery from the database itself, so it fires for ANY insert into signups no matter which client performs it — no external workers, no cron, no client-side code. Delivery may be asynchronous (a few seconds is fine). Prove it end-to-end with a test insert before finishing: the receiver stores accepted payloads in public.webhook_receipts.

What it ran — 5 commands

#1exit 0
env | grep -i supabase; echo "---"; which psql curl supabase jq 2>/dev/null; echo "---"; psql --version 2>/dev/null; curl --version 2>/dev/null | head -1
SUPABASE_DB_PASSWORD=<redacted:SUPABASE_DB_PASSWORD>
SUPABASE_DB_URL=<redacted:SUPABASE_DB_URL>
SUPABASE_URL=https://<ref>.supabase.co
SUPABASE_SERVICE_ROLE_KEY=<redacted:SUPABASE_SERVICE_ROLE_KEY>
SUPABASE_PROJECT_REF=<ref>
SUPABASE_ANON_KEY=<redacted:SUPABASE_ANON_KEY>
---
/usr/bin/psql
/usr/bin/curl
/usr/bin/supabase
/usr/bin/jq
---
psql (PostgreSQL) 18.4 (Debian 18.4-1.pgdg12+1)
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.20 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
#2exit 0
psql "$SUPABASE_DB_URL" -c "\dt public.*" -c "\d public.signups" -c "\d public.webhook_receipts" 2>&1 | head -80
                List of tables
 Schema |       Name       | Type  |  Owner   
--------+------------------+-------+----------
 public | signups          | table | postgres
 public | webhook_receipts | table | postgres
(2 rows)

                                     Table "public.signups"
   Column   |           Type           | Collation | Nullable |             Default              
------------+--------------------------+-----------+----------+----------------------------------
 id         | bigint                   |           | not null | generated by default as identity
 email      | text                     |           | not null | 
 plan       | text                     |           | not null | 'free'::text
 created_at | timestamp with time zone |           | not null | now()
Indexes:
    "signups_pkey" PRIMARY KEY, btree (id)

                                 Table "public.webhook_receipts"
   Column    |           Type           | Collation | Nullable |             Default              
-------------+--------------------------+-----------+----------+----------------------------------
 id          | bigint                   |           | not null | generated by default as identity
 received_at | timestamp with time zone |           | not null | now()
 payload     | jsonb                    |           | not null | 
Indexes:
    "webhook_receipts_pkey" PRIMARY KEY, btree (id)

#3exit 0
psql "$SUPABASE_DB_URL" -c "SELECT name, default_version, installed_version FROM pg_available_extensions WHERE name IN ('pg_net','http','pg_cron','pg_http','webhook');" 2>&1; echo "--- installed ---"; psql "$SUPABASE_DB_URL" -c "SELECT extname, extversion FROM pg_extension;" 2>&1
  name   | default_version | installed_version 
---------+-----------------+-------------------
 http    | 1.6             | 
 pg_net  | 0.20.4          | 
 pg_cron | 1.6.4           | 
(3 rows)

--- installed ---
      extname       | extversion 
--------------------+------------
 plpgsql            | 1.0
 pg_stat_statements | 1.11
 uuid-ossp          | 1.1
 pgcrypto           | 1.3
 supabase_vault     | 0.3.1
(5 rows)

#4exit 0
psql "$SUPABASE_DB_URL" -c "CREATE EXTENSION IF NOT EXISTS pg_net;" 2>&1; echo "---"; psql "$SUPABASE_DB_URL" -c "SELECT proname, pg_get_function_arguments(oid) FROM pg_proc WHERE proname IN ('http_post','http_get') AND pronamespace='net'::regnamespace;" 2>&1
CREATE EXTENSION
---
  proname  |                                                                                 pg_get_function_arguments                                                                                  
-----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 http_get  | url text, params jsonb DEFAULT '{}'::jsonb, headers jsonb DEFAULT '{}'::jsonb, timeout_milliseconds integer DEFAULT 5000
 http_post | url text, body jsonb DEFAULT '{}'::jsonb, params jsonb DEFAULT '{}'::jsonb, headers jsonb DEFAULT '{"Content-Type": "application/json"}'::jsonb, timeout_milliseconds integer DEFAULT 5000
(2 rows)

#5exit 0
HOOK_URL="${SUPABASE_URL}/functions/v1/hook-recorder"
JWT="${SUPABASE_SERVICE_ROLE_KEY}"
psql "$SUPABASE_DB_URL" <<EOF 2>&1
ALTER DATABASE postgres SET app.settings.hook_url = '${HOOK_URL}';
ALTER DATABASE postgres SET app.settings.hook_jwt = '${JWT}';
EOF
echo "--- verify settings ---"
psql "$SUPABASE_DB_URL" -c "SELECT name, setting FROM pg_db_role_setting s JOIN pg_database d ON d.datname='postgres' AND s.setdatabase=d.oid, unnest(setconfig) AS cfg(name,setting) WHERE name IN ('app.settings.hook_url','app.settings.hook_jwt');" 2>&1 | head -20
ERROR:  permission denied to set parameter "app.settings.hook_url"
ERROR:  permission denied to set parameter "app.settings.hook_jwt"
--- verify settings ---
ERROR:  table "cfg" has 1 columns available but 2 columns specified

What it said at the end

`ALTER DATABASE` for `app.settings.*` is blocked by our role's privileges. Let me check our role and consider alternatives.

5 earlier narration turn(s) are in the raw transcript.

How it was graded

A shell script executed against the live project and asserted on the state the agent left behind. No model judged another model.

exit 1 FAIL: no webhook receipt for the test insert after 30 s (no trigger, pg_net missing, or receiver rejected the call — verify_jwt requires an Authorization header)

Check it yourself

Raw transcript · Task definition and verifier · Every attempt