Skip to content

drpcstream: gate flow control at the message boundary and overdraft - #92

Open
suj-krishnan wants to merge 1 commit into
sujatha/flow-control-consume-enablefrom
sujatha/flow-control-overdraft
Open

drpcstream: gate flow control at the message boundary and overdraft#92
suj-krishnan wants to merge 1 commit into
sujatha/flow-control-consume-enablefrom
sujatha/flow-control-overdraft

Conversation

@suj-krishnan

Copy link
Copy Markdown

Stacked on #89 (base = sujatha/flow-control-consume-enable). Makes grant-on-consume flow control deadlock-free for multi-frame and larger-than-window messages via a message-boundary gate with overdraft, and bounds a single message with MaxMessageSize.

The deadlock

Per-frame credit gating deadlocks under grant-on-consume: credit is returned only when a complete message is consumed, so a multi-frame message that runs out of credit mid-way strands the receiver with an incomplete message it cannot consume — and no grant is ever issued. Even a message that fits the window can deadlock once coalesced credit is withheld below the threshold.

Message-boundary gate + overdraft

The first frame of a message acquires credit (parking until it arrives, or the window closes on termination); once committed, later frames overdraft — debit without parking, letting the balance go negative — so a started message always completes on the wire. The receiver then consumes the whole message and returns the credit (applyGrant repays the deficit first). Backpressure is preserved at message granularity: a new message parks on its first frame until the overdraft is repaid.

Bounded by MaxMessageSize (new FlowControl field)

  • a send larger than the bound fails fast, and the receiver's PacketAssembler rejects an assembling message that exceeds it before buffering it whole;
  • oversized surfaces as a dedicated MessageSizeErrorcodes.ResourceExhausted (matching gRPC) and fails only that streamHandleFrame sends the peer an abortive terminal error so a credit-gated send there wakes instead of hanging, then returns without tearing down the multiplexed connection (framing faults, which do tear it down, rely on the transport close to notify the peer);
  • MaxMessageSize may be smaller than StreamWindow (a stricter, safer bound); an unset bound defaults to 64 MiB independently. Validation uses frame = min(SplitSize, MaxMessageSize), since only the first frame blocks-acquires and a message is at most MaxMessageSize.

Because a message may overdraft past the window up to MaxMessageSize, peak per-stream memory is roughly StreamWindow + MaxMessageSize rather than StreamWindow.

End to end

drpcmanager tests over a real net.Pipe: a 128 KiB message equal to the whole window completes via overdraft after a sub-threshold consume withholds its credit; context cancellation wakes a sender parked on credit through the production manageStream path; and the default-config tripwire (with a message past the grant threshold) confirms no KindWindowUpdate reaches the wire when flow control is off.

Per-frame credit gating deadlocks under grant-on-consume: credit is
returned only when a complete message is consumed, so a multi-frame
message that runs out of credit mid-way strands the receiver with an
incomplete message it cannot consume, and no grant is ever issued. Even
a message that fits the window can deadlock once coalesced credit is
withheld below the grant threshold.

Gate at the message boundary instead. The first frame of a message
acquires credit (parking until it arrives, or the window closes on
termination); once committed, later frames overdraft -- debit without
parking, letting the balance go negative -- so a started message always
completes on the wire. The receiver then consumes the whole message and
returns the credit, and applyGrant repays the deficit before any credit
accrues. Backpressure is preserved at message granularity: a new message
parks on its first frame until the overdraft is repaid.

The overdraft is bounded by MaxMessageSize, a new FlowControl field. A
send larger than the bound fails fast, and the receiver's PacketAssembler
rejects an assembling message that exceeds it before buffering it whole.
Only the first frame need fit the window, so MaxMessageSize may be
smaller than StreamWindow (a stricter, safer bound); an unset bound
defaults to 64 MiB independently, so omitting it does not force the
window and threshold to their defaults. Because a message may overdraft
past the window up to MaxMessageSize, peak per-stream memory is roughly
StreamWindow + MaxMessageSize rather than StreamWindow.

An oversized message surfaces as a dedicated MessageSizeError class, not
a protocol fault: ToRPCErr maps it to codes.ResourceExhausted, as gRPC
does. An oversized receive fails only its own stream -- HandleFrame sends
the peer an abortive terminal error so a credit-gated send there wakes
instead of hanging, then returns without tearing down the multiplexed
connection (the framing faults that do tear it down rely on the
transport close to notify the peer).

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant