feat: add net.http_request callback interface (on_success/on_error) from #62

[riderx]

What kind of change does this PR introduce?

Feature: a first implementation of the callback interface proposed by @steve-chavez in #62, added alongside the existing API (which is left completely untouched).

Note: this PR was reworked after the maintainer feedback. The initial version added a store_response parameter to the existing http_* functions; per this comment that approach was dropped in favor of a new callback-based interface.

What is the new behavior?

net.http_request(
    method net.http_method,
    url text,
    params jsonb default '{}',
    headers jsonb default '{}',
    body jsonb default null,
    timeout_milliseconds int default 5000,
    on_success text default null,  -- $1 = status_code int, $2 = headers jsonb, $3 = body text
    on_error text default null     -- $1 = error message text, $2 = timed_out bool
) returns void

• Requests made with net.http_request never touch net._http_response: the response is handed to on_success, or discarded when no callback is given.
• Fire-and-forget (the use case from the discussion): leave on_success empty. The response body is then not even buffered in memory by the worker (a discarding libcurl write callback is used).
• Every HTTP response — including 4xx/5xx — goes through on_success with the status code as $1. Only failures without a response (timeout, connection error) go through on_error, with the error message and a timed_out flag so users can implement their own retry decisions (e.g. re-enqueue via net.http_request again). A failure with no on_error callback is reported in the logs at LOG level.
• http_get/http_post/http_delete and their table-based behavior are unchanged — existing users and observability workflows are unaffected.

Callback execution semantics

• Callbacks are executed by the background worker via SPI_execute_with_args, inside a subtransaction, as the role that enqueued the request, with SET ROLE/SET SESSION AUTHORIZATION blocked (SECURITY_LOCAL_USERID_CHANGE | SECURITY_RESTRICTED_OPERATION). A callback can't do anything the enqueuing role couldn't — tested.
• A callback that raises is rolled back and reported as a WARNING without aborting the batch or crashing the worker — tested.

Upgrade safety

The shared library is typically replaced before ALTER EXTENSION pg_net UPDATE runs. The worker checks (via get_attnum) whether the new queue columns exist and falls back to the legacy query when they don't, so a newer binary keeps processing requests normally against a not-yet-updated extension. Verified end-to-end locally (v0.20.5 binary → new binary with extension still at 0.20.4 → ALTER EXTENSION pg_net UPDATE).

Benchmark

5,000 async GET requests against a local HTTP server returning a 10 KB body, default pg_net.batch_size=200, measured on the background worker process (PostgreSQL 17.10, Apple M-series, macOS; single run, ps CPU-time granularity ~10 ms — treat as directional):

	net.http_get (stored)	net.http_request (fire-and-forget)
worker CPU, request processing	0.53 s	0.29 s (−45%)
worker CPU, TTL expiration of the 5,000 rows	0.04 s	—
net._http_response tuple writes (ins + del)	5,000 + 5,000	0
net._http_response size after run	1,912 kB	32 kB (empty)

Wall-clock is identical in both modes (dominated by the worker's fixed 1 s sleep between batches). The savings are the response-table inserts, their later TTL deletes, the resulting autovacuum pressure (not captured above), and the response-body buffering.

Additional context

• New tests in test/test_http_request_callbacks.py: fire-and-forget stores nothing; on_success receives status/headers/body; non-2xx statuses go through on_success; POST with body; on_error receives the error message; a raising callback doesn't kill the worker or the batch; callbacks run as the enqueuing role; a low-privilege enqueuer can't write where its role can't; legacy functions still store responses; missing-column fallback.
• This intentionally implements only the interface semantics from Change table queue to in-memory queue and add callbacks #62 on top of the existing queue table — the in-memory queue is orthogonal and can replace the storage underneath later without changing the interface.
• Use case (as context for the discussion): Capgo's queue dispatcher wakes an edge function up to ~60×/min via net.http_post and never reads responses; the response-row insert + TTL delete churn is pure overhead at that frequency. Happy to also open a dedicated issue elaborating this if still useful.
• Version bumped to 0.21.0 with migration sql/pg_net--0.20.4--0.21.0.sql (columns added to net.http_request_queue; no existing objects modified).

🤖 Generated with Claude Code

[riderx]

hey @steve-chavez i would love your imput in this, i have been strugglying with this for 2 years now.
and i think this will make a huge change in my usage of Supabase in Capgo

[riderx]

@olirice as well a review would be amazing, i can provide more test or more details if needed

[steve-chavez]

@riderx Discussed this on #62 (comment), I don't think we should complicate the current interface more and instead add a new one.

[riderx]

@steve-chavez thanks for the direction — I reworked the PR based on your comment in #62 (comment).

The store_response parameter is gone. The existing http_get/http_post/http_delete functions are now completely untouched. Instead there's a single new function implementing the interface you described in #62:

select net.http_request(
  'POST', 'https://example.com/webhook',
  body := '{"key": "value"}'::jsonb,
  on_success := 'insert into results values (, , )',  -- status, headers, body
  on_error   := 'insert into failures values (, )'      -- error msg, timed_out
);

• Empty on_success = fire-and-forget (my use case): nothing is written to net._http_response, and the response body isn't even buffered.
• All HTTP statuses (incl. 4xx/5xx) go through on_success with the status as $1; on_error only fires for timeouts/connection errors, with the message and a timed_out flag — so retry policy stays in user hands (a callback can re-enqueue via net.http_request again).
• Callbacks run in a subtransaction as the enqueuing role with SET ROLE blocked, so they can't escalate; a raising callback is rolled back and logged as WARNING without aborting the batch.
• It's implemented on top of the existing queue table, so the in-memory queue from Change table queue to in-memory queue and add callbacks #62 can later replace the storage underneath without changing the interface.

Full test coverage in test/test_http_request_callbacks.py (callback args, non-2xx, empty body, failing callback, role security, legacy behavior untouched, binary-newer-than-extension fallback) — CI matrix is green on my fork across pg 12–19 + macOS.

Could you take a look and tell me if this matches the idea in #62 better? Happy to adjust the semantics (callback arg shapes, error handling, naming) to whatever you prefer, and to open the use-case issue you asked for if that's still the right process.