Can Agents Run It? · evidence — Issue 1: Supabase

openai/gpt-5.6-sol · d2_webhooks · rep 2

pass
model
openai/gpt-5.6-sol
task
d2_webhooks
rep
2
day
2
steps
8
tokens
25,616
est. cost
$0.359
agent time
72s
wave
s4
started
2026-07-24T05:57:03Z

The task as the agent received it

You have operator access to a hosted Supabase project. Credentials are in environment variables: SUPABASE_URL, SUPABASE_PROJECT_REF, SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_DB_URL, SUPABASE_DB_PASSWORD. Every INSERT into the existing table public.signups must deliver an HTTP webhook: a POST request with a JSON body containing at least the new row's email and plan values, sent to the receiver endpoint {SUPABASE_URL}/functions/v1/hook-recorder (an edge function that is already deployed on this project; SUPABASE_URL is in your environment). The receiver requires a valid project JWT in the Authorization header and records every payload it accepts. Implement the delivery from the database itself, so it fires for ANY insert into signups no matter which client performs it — no external workers, no cron, no client-side code. Delivery may be asynchronous (a few seconds is fine). Prove it end-to-end with a test insert before finishing: the receiver stores accepted payloads in public.webhook_receipts.

What it ran — 8 commands

#1exit 0
set -euo pipefail
command -v psql
printf 'URL=%s\nREF=%s\nDBURL_SET=%s\n' "$SUPABASE_URL" "$SUPABASE_PROJECT_REF" "${SUPABASE_DB_URL:+yes}"
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X -P pager=off -c "select version();"
/usr/bin/psql
URL=https://<ref>.supabase.co
REF=<ref>
DBURL_SET=yes
                                   version                                    
------------------------------------------------------------------------------
 PostgreSQL 17.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 15.2.0, 64-bit
(1 row)
#2exit 0
set -euo pipefail
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X -P pager=off <<'SQL'
select e.extname, n.nspname as schema, e.extversion from pg_extension e join pg_namespace n on n.oid=e.extnamespace where e.extname in ('pg_net','supabase_vault');
\d+ public.signups
\d+ public.webhook_receipts
select tgname, pg_get_triggerdef(oid) from pg_trigger where tgrelid='public.signups'::regclass and not tgisinternal;
select routine_schema,routine_name from information_schema.routines where routine_name ilike '%http%' or routine_name ilike '%hook%';
SQL
    extname     | schema | extversion 
----------------+--------+------------
 supabase_vault | vault  | 0.3.1
(1 row)

                                                                Table "public.signups"
   Column   |           Type           | Collation | Nullable |             Default              | Storage  | Compression | Stats target | Description 
------------+--------------------------+-----------+----------+----------------------------------+----------+-------------+--------------+-------------
 id         | bigint                   |           | not null | generated by default as identity | plain    |             |              | 
 email      | text                     |           | not null |                                  | extended |             |              | 
 plan       | text                     |           | not null | 'free'::text                     | extended |             |              | 
 created_at | timestamp with time zone |           | not null | now()                            | plain    |             |              | 
Indexes:
    "signups_pkey" PRIMARY KEY, btree (id)
Access method: heap

                                                            Table "public.webhook_receipts"
   Column    |           Type           | Collation | Nullable |             Default              | Storage  | Compression | Stats target | Description 
-------------+--------------------------+-----------+----------+----------------------------------+----------+-------------+--------------+-------------
 id          | bigint                   |           | not null | generated by default as identity | plain    |             |              | 
 received_at | timestamp with time zone |           | not null | now()                            | plain    |             |              | 
 payload     | jsonb                    |           | not null |                                  | extended |             |              | 
Indexes:
    "webhook_receipts_pkey" PRIMARY KEY, btree (id)
Access method: heap

 tgname | pg_get_triggerdef 
--------+-------------------
(0 rows)

 routine_schema | routine_name 
----------------+--------------
(0 rows)
#3exit 0
set -euo pipefail
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X -P pager=off <<'SQL'
select name, default_version, installed_version from pg_available_extensions where name='pg_net';
\df vault.*
\d vault.decrypted_secrets
SQL
  name  | default_version | installed_version 
--------+-----------------+-------------------
 pg_net | 0.20.4          | 
(1 row)

                                                                                                        List of functions
 Schema |           Name            | Result data type |                                                                        Argument data types                                                                        | Type 
--------+---------------------------+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+------
 vault  | _crypto_aead_det_decrypt  | bytea            | message bytea, additional bytea, key_id bigint, context bytea DEFAULT '\x7067736f6469756d'::bytea, nonce bytea DEFAULT NULL::bytea                                | func
 vault  | _crypto_aead_det_encrypt  | bytea            | message bytea, additional bytea, key_id bigint, context bytea DEFAULT '\x7067736f6469756d'::bytea, nonce bytea DEFAULT NULL::bytea                                | func
 vault  | _crypto_aead_det_noncegen | bytea            |                                                                                                                                                                   | func
 vault  | create_secret             | uuid             | new_secret text, new_name text DEFAULT NULL::text, new_description text DEFAULT ''::text, new_key_id uuid DEFAULT NULL::uuid                                      | func
 vault  | update_secret             | void             | secret_id uuid, new_secret text DEFAULT NULL::text, new_name text DEFAULT NULL::text, new_description text DEFAULT NULL::text, new_key_id uuid DEFAULT NULL::uuid | func
(5 rows)

                        View "vault.decrypted_secrets"
      Column      |           Type           | Collation | Nullable | Default 
------------------+--------------------------+-----------+----------+---------
 id               | uuid                     |           |          | 
 name             | text                     |           |          | 
 description      | text                     |           |          | 
 secret           | text                     |           |          | 
 decrypted_secret | text                     | C         |          | 
 key_id           | uuid                     |           |          | 
 nonce            | bytea                    |           |          | 
 created_at       | timestamp with time zone |           |          | 
 updated_at       | timestamp with time zone |           |          |
#4exit 0
set -euo pipefail
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X -P pager=off <<'SQL'
create extension if not exists pg_net with schema extensions;
\df net.http_post
\df net.*
SQL
CREATE EXTENSION
                                                                                                             List of functions
 Schema |   Name    | Result data type |                                                                                    Argument data types                                                                                     | Type 
--------+-----------+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------
 net    | http_post | bigint           | url text, body jsonb DEFAULT '{}'::jsonb, params jsonb DEFAULT '{}'::jsonb, headers jsonb DEFAULT '{"Content-Type": "application/json"}'::jsonb, timeout_milliseconds integer DEFAULT 5000 | func
(1 row)

                                                                                                                           List of functions
 Schema |             Name              |     Result data type     |                                                                                    Argument data types                                                                                     | Type 
--------+-------------------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------
 net    | _await_response               | boolean                  | request_id bigint                                                                                                                                                                          | func
 net    | _encode_url_with_params_array | text                     | url text, params_array text[]                                                                                                                                                              | func
 net    | _http_collect_response        | net.http_response_result | request_id bigint, async boolean DEFAULT true                                                                                                                                              | func
 net    | _urlencode_string             | text                     | string character varying                                                                                                                                                                   | func
 net    | check_worker_is_up            | void                     |                                                                                                                                                                                            | func
 net    | http_collect_response         | net.http_response_result | request_id bigint, async boolean DEFAULT true                                                                                                                                              | func
 net    | http_delete                   | bigint                   | url text, params jsonb DEFAULT '{}'::jsonb, headers jsonb DEFAULT '{}'::jsonb, timeout_milliseconds integer DEFAULT 5000, body jsonb DEFAULT NULL::jsonb                                   | func
 net    | http_get                      | bigint                   | url text, params jsonb DEFAULT '{}'::jsonb, headers jsonb DEFAULT '{}'::jsonb, timeout_milliseconds integer DEFAULT 5000                                                                   | func
 net    | http_post                     | bigint                   | url text, body jsonb DEFAULT '{}'::jsonb, params jsonb DEFAULT '{}'::jsonb, headers jsonb DEFAULT '{"Content-Type": "application/json"}'::jsonb, timeout_milliseconds integer DEFAULT 5000 | func
 net    | wait_until_running            | void                     |
… 696 more bytes — full output in the raw record
#5exit 3
set -euo pipefail
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X --set=hook_url="$SUPABASE_URL/functions/v1/hook-recorder" --set=hook_jwt="$SUPABASE_ANON_KEY" <<'SQL'
DO $do$
DECLARE
  v_id uuid;
BEGIN
  SELECT id INTO v_id FROM vault.decrypted_secrets WHERE name = 'signups_webhook_url' ORDER BY created_at DESC LIMIT 1;
  IF v_id IS NULL THEN
    PERFORM vault.create_secret(:'hook_url', 'signups_webhook_url', 'Endpoint for public.signups insert webhook');
  ELSE
    PERFORM vault.update_secret(v_id, :'hook_url', 'signups_webhook_url', 'Endpoint for public.signups insert webhook');
  END IF;

  SELECT id INTO v_id FROM vault.decrypted_secrets WHERE name = 'signups_webhook_jwt' ORDER BY created_at DESC LIMIT 1;
  IF v_id IS NULL THEN
    PERFORM vault.create_secret(:'hook_jwt', 'signups_webhook_jwt', 'JWT for public.signups insert webhook');
  ELSE
    PERFORM vault.update_secret(v_id, :'hook_jwt', 'signups_webhook_jwt', 'JWT for public.signups insert webhook');
  END IF;
END
$do$;

CREATE SCHEMA IF NOT EXISTS private;
REVOKE ALL ON SCHEMA private FROM PUBLIC;

CREATE OR REPLACE FUNCTION private.deliver_signup_webhook()
RETURNS trigger
LANGUAGE plpgsql
SECURITY DEFINER
SET search_path = ''
AS $function$
DECLARE
  v_url text;
  v_jwt text;
  v_request_id bigint;
BEGIN
  SELECT decrypted_secret INTO STRICT v_url
  FROM vault.decrypted_secrets
  WHERE name = 'signups_webhook_url'
  ORDER BY created_at DESC
  LIMIT 1;

  SELECT decrypted_secret INTO STRICT v_jwt
  FROM vault.decrypted_secrets
  WHERE name = 'signups_webhook_jwt'
  ORDER BY created_at DESC
  LIMIT 1;

  SELECT net.http_post(
    url := v_url,
    body := jsonb_build_object(
      'id', NEW.id,
      'email', NEW.email,
      'plan', NEW.plan,
      'created_at', NEW.created_at
    ),
    headers := jsonb_build_object(
      'Content-Type', 'application/json',
      'Authorization', 'Bearer ' || v_jwt
    ),
    timeout_milliseconds := 5000
  ) INTO v_request_id;

  RETURN NEW;
END
$function$;

REVOKE ALL ON FUNCTION private.deliver_signup_webhook() FROM PUBLIC;

DROP TRIGGER IF EXISTS signups_deliver_webhook_after_insert ON public.signups;
CREATE TRIGGER signups_deliver_webhook_after_insert
AFTER INSERT ON public.signups
FOR EACH ROW
EXECUTE FUNCTION private.deliver_signup_webhook();
SQL

psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X -P pager=off -c "select tgname, pg_get_triggerdef(oid) from pg_trigger where tgrelid='public.signups'::regclass and not tgisinternal;"
ERROR:  syntax error at or near ":"
LINE 7:     PERFORM vault.create_secret(:'hook_url', 'signups_webhoo...
                                        ^
#6exit 0
set -euo pipefail
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X --set=hook_url="$SUPABASE_URL/functions/v1/hook-recorder" --set=hook_jwt="$SUPABASE_ANON_KEY" <<'SQL'
SELECT vault.update_secret(id, :'hook_url', 'signups_webhook_url', 'Endpoint for public.signups insert webhook')
FROM vault.decrypted_secrets WHERE name='signups_webhook_url';
SELECT vault.create_secret(:'hook_url', 'signups_webhook_url', 'Endpoint for public.signups insert webhook')
WHERE NOT EXISTS (SELECT 1 FROM vault.decrypted_secrets WHERE name='signups_webhook_url');
SELECT vault.update_secret(id, :'hook_jwt', 'signups_webhook_jwt', 'JWT for public.signups insert webhook')
FROM vault.decrypted_secrets WHERE name='signups_webhook_jwt';
SELECT vault.create_secret(:'hook_jwt', 'signups_webhook_jwt', 'JWT for public.signups insert webhook')
WHERE NOT EXISTS (SELECT 1 FROM vault.decrypted_secrets WHERE name='signups_webhook_jwt');

CREATE SCHEMA IF NOT EXISTS private;
REVOKE ALL ON SCHEMA private FROM PUBLIC;

CREATE OR REPLACE FUNCTION private.deliver_signup_webhook()
RETURNS trigger
LANGUAGE plpgsql
SECURITY DEFINER
SET search_path = ''
AS $function$
DECLARE
  v_url text;
  v_jwt text;
  v_request_id bigint;
BEGIN
  SELECT decrypted_secret INTO STRICT v_url
  FROM vault.decrypted_secrets
  WHERE name = 'signups_webhook_url'
  ORDER BY created_at DESC LIMIT 1;

  SELECT decrypted_secret INTO STRICT v_jwt
  FROM vault.decrypted_secrets
  WHERE name = 'signups_webhook_jwt'
  ORDER BY created_at DESC LIMIT 1;

  SELECT net.http_post(
    url := v_url,
    body := jsonb_build_object('id', NEW.id, 'email', NEW.email, 'plan', NEW.plan, 'created_at', NEW.created_at),
    headers := jsonb_build_object('Content-Type', 'application/json', 'Authorization', 'Bearer ' || v_jwt),
    timeout_milliseconds := 5000
  ) INTO v_request_id;
  RETURN NEW;
END
$function$;

REVOKE ALL ON FUNCTION private.deliver_signup_webhook() FROM PUBLIC;
DROP TRIGGER IF EXISTS signups_deliver_webhook_after_insert ON public.signups;
CREATE TRIGGER signups_deliver_webhook_after_insert
AFTER INSERT ON public.signups
FOR EACH ROW EXECUTE FUNCTION private.deliver_signup_webhook();
SQL
 update_secret 
---------------
(0 rows)

            create_secret             
--------------------------------------
 60da4e83-bd68-4837-bd61-cc7939242a92
(1 row)

 update_secret 
---------------
(0 rows)

            create_secret             
--------------------------------------
 f78a9e02-f85d-4252-b826-a6272365f880
(1 row)

CREATE SCHEMA
REVOKE
CREATE FUNCTION
REVOKE
DROP TRIGGER
CREATE TRIGGER

NOTICE:  schema "private" already exists, skipping
NOTICE:  trigger "signups_deliver_webhook_after_insert" for relation "public.signups" does not exist, skipping
#7exit 1
set -euo pipefail
email="webhook-e2e-$(date +%s)-$RANDOM@example.com"
plan="webhook-test"
printf 'Testing email: %s\n' "$email"
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X --set=test_email="$email" --set=test_plan="$plan" -P pager=off <<'SQL'
INSERT INTO public.signups(email, plan) VALUES (:'test_email', :'test_plan') RETURNING id,email,plan,created_at;
SQL

found=0
for i in $(seq 1 20); do
  result=$(psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X -At --set=test_email="$email" --set=test_plan="$plan" -c "SELECT id || E'\t' || received_at || E'\t' || payload::text FROM public.webhook_receipts WHERE payload->>'email'=:'test_email' AND payload->>'plan'=:'test_plan' ORDER BY id DESC LIMIT 1;")
  if [ -n "$result" ]; then
    printf 'Receipt verified: %s\n' "$result"
    found=1
    break
  fi
  sleep 1
done
if [ "$found" -ne 1 ]; then
  echo 'Webhook receipt not found; diagnostics:' >&2
  psql "$SUPABASE_DB_URL" -X -P pager=off -c "select * from net._http_response order by id desc limit 5;" >&2 || true
  exit 1
fi
psql "$SUPABASE_DB_URL" -X -P pager=off -c "select tgname, pg_get_triggerdef(oid) as definition from pg_trigger where tgrelid='public.signups'::regclass and not tgisinternal;"
Testing email: webhook-e2e-1784872696-6837@example.com
 id |                  email                  |     plan     |          created_at           
----+-----------------------------------------+--------------+-------------------------------
  1 | webhook-e2e-1784872696-6837@example.com | webhook-test | 2026-07-24 05:58:16.963008+00
(1 row)

INSERT 0 1

ERROR:  syntax error at or near ":"
LINE 1: ...M public.webhook_receipts WHERE payload->>'email'=:'test_ema...
                                                             ^
#8exit 0
set -euo pipefail
email='webhook-e2e-1784872696-6837@example.com'
found=0
for i in $(seq 1 20); do
  result=$(printf "SELECT id || E'\\t' || received_at || E'\\t' || payload::text FROM public.webhook_receipts WHERE payload->>'email' = '%s' AND payload->>'plan' = 'webhook-test' ORDER BY id DESC LIMIT 1;\n" "$email" | psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 -X -At)
  if [ -n "$result" ]; then printf 'Receipt verified: %s\n' "$result"; found=1; break; fi
  sleep 1
done
if [ "$found" -ne 1 ]; then
 echo 'Not found; diagnostics:' >&2
 psql "$SUPABASE_DB_URL" -X -P pager=off <<'SQL'
 select * from net._http_response order by id desc limit 5;
 select * from net.http_request_queue order by id desc limit 5;
SQL
 exit 1
fi
psql "$SUPABASE_DB_URL" -X -P pager=off -c "select tgname, pg_get_triggerdef(oid) as definition from pg_trigger where tgrelid='public.signups'::regclass and not tgisinternal;"
Receipt verified: 2	2026-07-24 05:58:17.118133+00	{"id": 1, "plan": "webhook-test", "email": "webhook-e2e-1784872696-6837@example.com", "created_at": "2026-07-24T05:58:16.963008+00:00"}
                tgname                |                                                                    definition                                                                     
--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------
 signups_deliver_webhook_after_insert | CREATE TRIGGER signups_deliver_webhook_after_insert AFTER INSERT ON public.signups FOR EACH ROW EXECUTE FUNCTION private.deliver_signup_webhook()
(1 row)

What it said at the end

Implemented and verified database-level webhook delivery: - Enabled the `pg_net` extension for asynchronous HTTP requests. - Stored the endpoint URL and project JWT securely in Supabase Vault. - Created a `SECURITY DEFINER` trigger function in the private schema. - Added an `AFTER INSERT FOR EACH ROW` trigger on `public.signups`. - Payload includes `id`, `email`, `plan`, and `created_at`. - Requests include `Authorization: Bearer <project JWT>`. End-to-end test succeeded: - Inserted `webhook-e2e-1784872696-6837@example.com` - Plan: `webhook-test` - Matching payload was recorded in `public.webhook_receipts` with receipt ID `2`.

How it was graded

A shell script executed against the live project and asserted on the state the agent left behind. No model judged another model.

exit 0 OK

Check it yourself

Raw transcript · Task definition and verifier · Every attempt