Hand out the checkpoint when the source is what died - #141
Merged
tonytonycoder11 merged 2 commits intoAug 11, 2026
Conversation
An ingest whose source threw left the batches that were still in flight to be cancelled where they stood, because the exception propagated straight out of the coroutineScope. Whether the run reported any checkpoint at all then came down to which request happened to come back first, and a source that died early enough could leave the caller with no token whatsoever: the case a resume token exists for was the one case it was missing. The source's failure is now recorded and thrown after the workers have drained, which is what a batch failure already did and for the same reason. A batch cancelled after the server accepted it is a point the collection holds and no token counts, and re-sending an acknowledged point is free while skipping an unacknowledged one is silent data loss. This is what failed "an ingest killed partway resumes from its token" on the JVM REST run in CI: the assertion that a checkpoint was handed out at all. The linuxX64 run of the same suite failed in the same build with its case names truncated out of the console log, and the gRPC run of the same case passed, which is the shape a timing race has. The new unit test holds a batch in flight past the source's death, so the old behaviour fails it every time rather than now and then on a loaded runner.
The hosted runner intermittently fails the TLS handshake against the release host with "self signed certificate", on a certificate that verifies from everywhere else and verifies from the runner on the next attempt. There was no retry, so a hiccup lasting one second red-lined the whole job in nine. Three attempts with a growing pause, and the failure message names what could not be downloaded. Verification stays on: -k would trade a flake for a job that runs whatever answers the name.
|
This pull request adds no capability the code did not already have. What I could not follow3 calls resolve to a name whose type is written nowhere, so what they reach is not established here, and 2 more are in tests and benchmarks, which this report does not judge. Across everything it read that is 5 of 204, 2%.
If one of those reaches the network or starts a process, this report does not show it. 4 files read · 3131 ms · against |
tonytonycoder11
deleted the
fix/a-dying-source-still-hands-out-its-token
branch
August 11, 2026 22:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CI run on
mainafter #140 went red in three jobs. Two of them are the same defect, the third isthe runner.
The ingest loses the token in the one case the token is for
Build (JDK 21)andClient contract (Linux native)both failed the shared contract case an ingestkilled partway resumes from its token, on
assertNotNull(token, "no checkpoint was ever handed out").The source is a flow that dies partway on purpose. When it threw, the exception propagated straight out
of
ingest'scoroutineScope, which cancelled the batches that were still in flight. If the firstbatch happened to be one of them, nothing had advanced the acknowledged prefix, and the run reported no
checkpoint at all — so the caller of a two-hundred-million-point load that was killed at four hundred
thousand starts again from zero. Which batches were in flight is a matter of which request came back
first, which is why the same commit passed on the pull request the day before and why the gRPC run of
the same case passed in the failing build.
ingestalready refuses to throw a batch failure where it happens, and the comment onsendBatchgives the reason: cancelling a sibling mid-request turns a point the server accepted into a point no
token counts. A source failure had never been given the same treatment. It is now recorded, the queue
is closed, the workers drain, and it is thrown once the last
onCheckpointhas run.The new unit test holds the first batch in flight past the source's death, so the old behaviour fails
it deterministically rather than once in a while under load — 11 tests in
IngestTest, green with thefix and red without it.
The macOS runner's TLS hiccup
Client contract (macOS native)failed in nine seconds oncurl: (60) SSL certificate problem: self signed certificate, downloading the Qdrant release. The certificate is fine; the runner's handshake isnot, intermittently. There was no retry, so a hiccup lasting a second costs the job. Three attempts
with a growing pause, and the failure names what could not be downloaded.
-kis not the fix: it wouldtrade a flake for a job that runs whatever answers the name.
Checked locally
:kdrant-core:jvmTest,ktlintCheck,detektand amacosArm64compile of the changed common code.The rest is on CI here.