Can Agents Run It? · evidence — Issue 1: Supabase

deepseek/deepseek-v4-pro · d1_rls_owner · rep 2

fail
model
deepseek/deepseek-v4-pro
task
d1_rls_owner
rep
2
day
1
steps
4
tokens
5,646
est. cost
$0.003
agent time
23s
wave
s4
started
2026-07-24T05:43:04Z

The task as the agent received it

You have operator access to a hosted Supabase project. Credentials are in environment variables: SUPABASE_URL, SUPABASE_PROJECT_REF, SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, SUPABASE_DB_URL, SUPABASE_DB_PASSWORD. The project has a public.notes table (id, user_id references auth.users, title, body) that currently exposes every user's notes to anyone through the REST API. Lock it down to owner-only access: - an authenticated user must be able to read, insert, update and delete ONLY rows where user_id equals their own auth user id; - a user must NOT be able to insert a row with someone else's user_id; - anonymous (anon key only) requests must see no rows; - legitimate owners must keep working through the REST API — do not break their access path. Two existing users for your own testing: alice@cari.dev / Passw0rd-alice-42 and bob@cari.dev / Passw0rd-bob-58. Each currently owns 2 notes. Do not modify the data.

What it ran — 4 commands

#1exit 0
echo "SUPABASE_URL=$SUPABASE_URL" && echo "SUPABASE_PROJECT_REF=$SUPABASE_PROJECT_REF" && echo "SUPABASE_DB_URL=$SUPABASE_DB_URL"
SUPABASE_URL=https://<ref>.supabase.co
SUPABASE_PROJECT_REF=<ref>
SUPABASE_DB_URL=<redacted:SUPABASE_DB_URL>
#2exit 0
psql "$SUPABASE_DB_URL" -c "
-- Check current RLS on notes
SELECT tablename, rowsecurity FROM pg_tables WHERE schemaname='public' AND tablename='notes';

-- Check existing policies
SELECT schemaname, tablename, policyname, permissive, roles, cmd, qual, with_check
FROM pg_policies
WHERE schemaname='public' AND tablename='notes';
"
 tablename | rowsecurity 
-----------+-------------
 notes     | f
(1 row)

 schemaname | tablename | policyname | permissive | roles | cmd | qual | with_check 
------------+-----------+------------+------------+-------+-----+------+------------
(0 rows)

#3exit 1
psql "$SUPABASE_DB_URL" -c "
-- Look at notes schema
\d public.notes
"
ERROR:  syntax error at or near "\"
LINE 3: \d public.notes
        ^
#4exit 0
psql "$SUPABASE_DB_URL" -c "
SELECT column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_schema='public' AND table_name='notes'
ORDER BY ordinal_position;
"
 column_name | data_type | is_nullable | column_default 
-------------+-----------+-------------+----------------
 id          | bigint    | NO          | 
 user_id     | uuid      | NO          | 
 title       | text      | NO          | 
 body        | text      | NO          | ''::text
(4 rows)

What it said at the end

Let me start by exploring the environment and understanding the current state of the database

How it was graded

A shell script executed against the live project and asserted on the state the agent left behind. No model judged another model.

exit 1 FAIL: RLS is not enabled on public.notes (relrowsecurity=f)

Check it yourself

Raw transcript · Task definition and verifier · Every attempt