drpcstream: gate flow control at the message boundary and overdraft - #92
Open
suj-krishnan wants to merge 1 commit into
Open
drpcstream: gate flow control at the message boundary and overdraft#92suj-krishnan wants to merge 1 commit into
suj-krishnan wants to merge 1 commit into
Conversation
Per-frame credit gating deadlocks under grant-on-consume: credit is returned only when a complete message is consumed, so a multi-frame message that runs out of credit mid-way strands the receiver with an incomplete message it cannot consume, and no grant is ever issued. Even a message that fits the window can deadlock once coalesced credit is withheld below the grant threshold. Gate at the message boundary instead. The first frame of a message acquires credit (parking until it arrives, or the window closes on termination); once committed, later frames overdraft -- debit without parking, letting the balance go negative -- so a started message always completes on the wire. The receiver then consumes the whole message and returns the credit, and applyGrant repays the deficit before any credit accrues. Backpressure is preserved at message granularity: a new message parks on its first frame until the overdraft is repaid. The overdraft is bounded by MaxMessageSize, a new FlowControl field. A send larger than the bound fails fast, and the receiver's PacketAssembler rejects an assembling message that exceeds it before buffering it whole. Only the first frame need fit the window, so MaxMessageSize may be smaller than StreamWindow (a stricter, safer bound); an unset bound defaults to 64 MiB independently, so omitting it does not force the window and threshold to their defaults. Because a message may overdraft past the window up to MaxMessageSize, peak per-stream memory is roughly StreamWindow + MaxMessageSize rather than StreamWindow. An oversized message surfaces as a dedicated MessageSizeError class, not a protocol fault: ToRPCErr maps it to codes.ResourceExhausted, as gRPC does. An oversized receive fails only its own stream -- HandleFrame sends the peer an abortive terminal error so a credit-gated send there wakes instead of hanging, then returns without tearing down the multiplexed connection (the framing faults that do tear it down rely on the transport close to notify the peer). Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
suj-krishnan
requested review from
5hubh4m,
Nukitt,
cthumuluru-crdb and
shubhamdhama
and removed request for
5hubh4m
August 3, 2026 05:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #89 (base =
sujatha/flow-control-consume-enable). Makes grant-on-consume flow control deadlock-free for multi-frame and larger-than-window messages via a message-boundary gate with overdraft, and bounds a single message withMaxMessageSize.The deadlock
Per-frame credit gating deadlocks under grant-on-consume: credit is returned only when a complete message is consumed, so a multi-frame message that runs out of credit mid-way strands the receiver with an incomplete message it cannot consume — and no grant is ever issued. Even a message that fits the window can deadlock once coalesced credit is withheld below the threshold.
Message-boundary gate + overdraft
The first frame of a message acquires credit (parking until it arrives, or the window closes on termination); once committed, later frames overdraft — debit without parking, letting the balance go negative — so a started message always completes on the wire. The receiver then consumes the whole message and returns the credit (
applyGrantrepays the deficit first). Backpressure is preserved at message granularity: a new message parks on its first frame until the overdraft is repaid.Bounded by
MaxMessageSize(newFlowControlfield)PacketAssemblerrejects an assembling message that exceeds it before buffering it whole;MessageSizeError→codes.ResourceExhausted(matching gRPC) and fails only that stream —HandleFramesends the peer an abortive terminal error so a credit-gated send there wakes instead of hanging, then returns without tearing down the multiplexed connection (framing faults, which do tear it down, rely on the transport close to notify the peer);MaxMessageSizemay be smaller thanStreamWindow(a stricter, safer bound); an unset bound defaults to 64 MiB independently. Validation usesframe = min(SplitSize, MaxMessageSize), since only the first frame blocks-acquires and a message is at mostMaxMessageSize.Because a message may overdraft past the window up to
MaxMessageSize, peak per-stream memory is roughlyStreamWindow + MaxMessageSizerather thanStreamWindow.End to end
drpcmanagertests over a realnet.Pipe: a 128 KiB message equal to the whole window completes via overdraft after a sub-threshold consume withholds its credit; context cancellation wakes a sender parked on credit through the productionmanageStreampath; and the default-config tripwire (with a message past the grant threshold) confirms noKindWindowUpdatereaches the wire when flow control is off.