Fix MaxScoreBulkScorer window-bounds bug (prevent TermScorer docID >= maxDoc EOF) #15345
+96
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benwtrent
reviewed
Oct 21, 2025
Comment on lines
202
to
207
|
|
||
| * GITHUB#15343: Ensure that `AcceptDocs#cost()` only ever calls `BitSets#cardinality()` | ||
| once per instance to avoid redundant computation. (Ben Trent) | ||
|
|
There was a problem hiding this comment.
woops, seems like a bad delete?
|
@kdt523 did you verify your test failed and that the code change actually addresses the issue? I ran your new test against Lucene main and it always passed. We should have a repeatable test that reproduces the bad behavior to confirm the fix. |
benwtrent
reviewed
Oct 21, 2025
| public class TestMaxScoreBulkScorerFilterBounds extends LuceneTestCase { | ||
|
|
||
| public void testFilteredDisjunctionDoesNotScorePastMaxDoc() throws Exception { | ||
| Directory dir = new RAMDirectory(); |
There was a problem hiding this comment.
Did you actually run this? RAMDirectory hasn't existed since Lucene 9....
|
This has that AI smell to it |
…lkScorerFilterBounds
… GITHUB#14963 entries
|
FYI, this isn't the cause of the bug, we found the real cause |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Summary
This fixes a bug where MaxScoreBulkScorer could set its inner/outer window upper bounds past the current leaf maxDoc, allowing TermScorer to be called with docIDs >= maxDoc and causing EOF or ArrayIndex errors when accessing norms.
Motivation
Under certain disjunction + filter workloads the bulk scorer's windowing logic could loop with upper bounds computed without clamping to the leaf's maxDoc. This could result in TermScorer attempting to read norms past the leaf end (NO_MORE_DOCS), leading to intermittent EOF/ArrayIndex errors in production and tests.
What changed
lucene/core/src/java/org/apache/lucene/search/MaxScoreBulkScorer.java
Clamp outer and inner window upper bounds to the leaf maxDoc to ensure scoring loops never iterate past the leaf boundary.
lucene/core/tests/src/test/org/apache/lucene/search/TestMaxScoreBulkScorerFilterBounds.java
Add regression test exercising the disjunction + restrictive filter code path that previously triggered the failure.
lucene/CHANGES.txt
Add an entry referencing this change (GITHUB#15324) and the contributor.
How to test
Unit tests:
The newly-added test in lucene/core/tests reproduces the scenario; run it with:
./gradlew :lucene:core:test --tests org.apache.lucene.search.TestMaxScoreBulkScorerFilterBounds
Manual validation:
Build core and run the relevant test suite:
./gradlew :lucene:core:compileJava :lucene:core:test
Note: Some generate/format tasks may require python3 available on PATH; if you see checksum mismatches, run:
./gradlew :lucene:core:generateForUtil --no-daemon --info
after ensuring python3 is callable by Gradle.
CHANGES entry
GITHUB#15324: Fix MaxScoreBulkScorer could call TermScorer with docID >= maxDoc, causing EOFException on norms access (contributor: kdt523)
Checklist (from CONTRIBUTING.md)
[x] My PR title is short and descriptive.
[x] I have included a summary of the change and why it was needed.
[x] I added tests that reproduce the issue and prove the fix.
[x] I updated CHANGES.txt with an entry referencing the PR/issue.
[x] I ran ./gradlew tidy and applied code formatting.
I ran ./gradlew check locally and all checks pass.
If the change affects generated sources, I ran the generator tasks and included updated files.
task done are marked with [x]
fix: #15324