Skip to content

RATIS-2632. Apply backpressure on install-snapshot chunk loop - #1540

Open
spacemonkd wants to merge 1 commit into
apache:masterfrom
spacemonkd:RATIS-2632
Open

RATIS-2632. Apply backpressure on install-snapshot chunk loop#1540
spacemonkd wants to merge 1 commit into
apache:masterfrom
spacemonkd:RATIS-2632

Conversation

@spacemonkd

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

RATIS-2632. Apply backpressure on install-snapshot chunk loop

When the leader streams a snapshot to a lagging follower, the chunk send loop in GrpcLogAppender.installSnapshot() calls snapshotRequestObserver.onNext(request) for every chunk without ever checking CallStreamObserver.isReady().

The append-entries path already checks isReady(), but the snapshot path does not, so it is possible that chunks are enqueued faster than the network can drain them.

While on the default path INSTALL_SNAPSHOT_REQUEST_ELEMENT_LIMIT_DEFAULT is set to 8 so StreamObserverWithTimeout applies a request-response window, but since we allow setting the limit to 0, the semaphore can be null and cause:

  • Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue when the follower is slow.
  • Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which can crash the leader and impact the whole group.

This is handled by having an explicit isReady() check, which causes minimal overhead on the default path (element-limit=8) but handles the case when it might be set to 0.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2632

How was this patch tested?

Patch was tested via unit tests.

@spacemonkd

Copy link
Copy Markdown
Contributor Author

@szetszwo @ss77892 could you take a look at this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant