Skip to content

drpcstream: byte-account the per-stream receive queue - #93

Open
suj-krishnan wants to merge 1 commit into
sujatha/flow-control-overdraftfrom
sujatha/flow-control-recv-bytes
Open

drpcstream: byte-account the per-stream receive queue#93
suj-krishnan wants to merge 1 commit into
sujatha/flow-control-overdraftfrom
sujatha/flow-control-recv-bytes

Conversation

@suj-krishnan

@suj-krishnan suj-krishnan commented Aug 3, 2026

Copy link
Copy Markdown

Stacked on #92 (base: sujatha/flow-control-overdraft).

Problem

The receive queue (ringBuffer) bounded the producer by message-slot count — a fixed 256-slot ring blocking on count == len(buf). Slot-based bounding makes the flow-control memory bound a fiction: a slow consumer can buffer 256 arbitrarily large messages, so the "receiver holds ≤ StreamWindow + MaxMessageSize" guarantee doesn't hold. Grants are already byte-accurate (recvw.consumed(len)); the queue that holds those bytes was not.

Change

Bound and account the queue in bytes when flow control is on:

  • Budget = StreamWindow + MaxMessageSize — the per-stream receive peak (a window of un-granted bytes plus one message overdrafting to finish).
  • The budget is denominated in payload bytes — the same unit the sender debits and grants return — so a conforming sender within its wire credit can never fill the queue and block the shared connection reader (manageReader).
  • Admission (admits) folds in the incoming message's length with an empty-queue exception, so the queue never overshoots by a message and an oversized message is still deliverable.
  • The pooled copy is made after admission, so a blocked Enqueue holds no buffer outside the accounted queue.
  • Under a budget the ring grows past its initial capacity, so message count never blocks the producer. Flow control off keeps the legacy 256-slot bound unchanged.

Deferred

  • Ownership transfer (drop the Enqueue copy and the assembler's retained backing array) — closes the accounted-vs-retained gap; a follow-up.
  • Zero/tiny-credit (empty) message floods — a payload-aligned queue cannot cap these without breaking liveness, because an empty message consumes no wire credit. The liveness-safe fix is a per-message minimum charged symmetrically in send credit, grants, and the queue.

Testing

go test ./drpcstream/ ./drpcmanager/ ./drpcwire/ -race (and the full suite) pass; gofmt clean. New ring-buffer tests cover byte-budget blocking/unblocking, incoming-length admission, byte release on dispatch, oversized-on-empty
acceptance, and ring growth; a stream-level test asserts the budget is StreamWindow + MaxMessageSize (and unset when flow control is off).

Part of CRDB-65748.

The receive queue (ringBuffer) bounded the producer by message-slot count: a
fixed 256-slot ring blocking on count == len(buf). Slot-based bounding makes the
flow-control memory bound a fiction -- a slow consumer can buffer 256
arbitrarily large messages -- so the "receiver holds <= StreamWindow +
MaxMessageSize" guarantee does not hold. Grants are already byte-accurate
(recvw.consumed(len)); the queue that holds those bytes was not.

Bound and account the queue in bytes when flow control is enabled. The budget is
StreamWindow + MaxMessageSize -- the per-stream receive peak (a window of
un-granted bytes plus one message overdrafting to finish) -- and, crucially, is
denominated in payload bytes: the same unit the sender debits and grants return.
Admission (admits) includes the incoming message's length with an empty-queue
exception, so the queue never overshoots the budget by a message and an oversized
message is still deliverable. The pooled copy is made only after admission, so a
blocked Enqueue holds no buffer outside the accounted queue. Under a budget the
ring grows past its initial capacity, so message count never blocks the producer;
flow control off keeps the legacy slot bound unchanged.

Keeping the budget in payload bytes (rather than charging a per-entry overhead)
preserves flow-control liveness: a conforming sender within its wire credit
cannot fill the queue and block the shared connection reader. Bounding the
per-message memory overhead and zero-credit (empty) message floods is left to
follow-ups -- ownership transfer to drop the copy, and, for empty messages, a
per-message minimum charged symmetrically in send credit, grants, and the queue.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant