CNDB-14624: do not fail user read when speculative retry handling throws #1875
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Checklist before you submit for review
|
jakubzytka
force-pushed
the
cndb-14624-ignore-exceptions-during-spec-retry
branch
from
7824335 to
a166649
Compare
….from instead of from() like in id(includeReal)
sbtourist
reviewed
Jul 22, 2025
|
✔️ Build ds-cassandra-pr-gate/PR-1875 approved by ButlerApproved by Butler |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the issue
When the speculative retries code throws, the whole user read fails. This is suboptimal. In particular, we do not want to fail reads that would otherwise complete from the initial replicas, even if the latency is in the upper percentiles.
Another motivation is that we would like to be able to throw an exception if an internode connection tries to connect to an unknown CNDB service to prevent #14624 from happening.
Currently, throwing such an exception would break user reads, and this is not acceptable.
What does this PR fix and why was it fixed
This change introduces a specific
UnknownEndpointExceptionthat may be thrown in CNDB's snitch implementation when the internode tries to connect to an unknown service.Additionally, we catch all the exceptions thrown during speculative read issuance.
UnknownEndpointExceptionis treated as a common case (due to races between service removal and internode communication), whereas other exceptions emit a warning but do not fail the read either.That way, allow user reads to complete even if the speculative retry fails, and we leave a path open for CNDB to prevent opening a new connection by throwing
UnknownEndpointExceptionwhen necessary.