backfill: add a separate control for number of vectors merged per batch #155284
+17
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add the 'bulkio.index_backfill.vector_merge_batch_size', which controls the number of rows to merge into vector indexes per transaction while the index is being created. This is analogous to the 'bulkio.index_backfill.merge_batch_size' setting, but it only applies when the target index is a vector index. By default, 3 vectors will be merged per transaction to reduce contention with fixup tasks. Fixes: cockroachdb#155283 Release note (sql change): Added the bulkio.index_backfill.vector_merge_batch_size cluster setting to control how many vectors to merge into a vector index per transaction during create operations. By default, this defaults to 3.
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=95b789b1d594ac6b74f377aa7555d8d98d63240d&v=4){kind=avatar}
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
This change is |
DrewKimball
approved these changes
Oct 11, 2025
There was a problem hiding this comment.
@DrewKimball reviewed 1 of 1 files at r1, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @fqazi)
|
bors r+ |
craig bot
pushed a commit
that referenced
this pull request
Oct 15, 2025
155192: backfill: split merge memory accounting mutex acquisitions r=mw5h a=mw5h Previously, constructMergeBatch() would get the mutex on memory accounting once per batch to minimize the number of atomics needed to perform memory accounting. With the addition of vector indexing, we now need to perform KV operations in order to construct batches, which can lead to deadlocks between KV and the memory accounting mutex. This patch splits the mutex acquisition into two acquisitions, releasing the mutex in between. Memory used is accumulated as merged entries are built and then accounting is notified for the entire batch. This has the effect of making the accounting lag behind allocation a bit, but that seems preferable than paying the cost of a mutex acquisition for every row merged. Fixes: #155190 Release note (bug fix): A potential deadlock during vector index creation has been corrected. 155284: backfill: add a separate control for number of vectors merged per batch r=mw5h a=mw5h Add the 'bulkio.index_backfill.vector_merge_batch_size', which controls the number of rows to merge into vector indexes per transaction while the index is being created. This is analogous to the 'bulkio.index_backfill.merge_batch_size' setting, but it only applies when the target index is a vector index. By default, 3 vectors will be merged per transaction to reduce contention with fixup tasks. Fixes: #155283 Release note (sql change): Added the bulkio.index_backfill.vector_merge_batch_size cluster setting to control how many vectors to merge into a vector index per transaction during create operations. By default, this defaults to 3. 155412: kvserver: introduce test harness for replica lifecycle events r=pav-kv a=arulajmani First two commits from #155408. This patch introduces the scaffolding for a harness to test various replica lifecycle events in a data-driven manner. The harness works from th context of a single node/store -- n1,s1. The intention is to print all engine interactions of the replica on this store on various replica lifecycle events. To kick things off, we add the ability to create a replica -- both initialized and uninitialized. Epic: none Release note: None Co-authored-by: Matt White <[email protected]> Co-authored-by: Arul Ajmani <[email protected]>
|
Build failed (retrying...):
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add the 'bulkio.index_backfill.vector_merge_batch_size', which controls the number of rows to merge into vector indexes per transaction while the index is being created. This is analogous to the 'bulkio.index_backfill.merge_batch_size' setting, but it only applies when the target index is a vector index. By default, 3 vectors will be merged per transaction to reduce contention with fixup tasks.
Fixes: #155283
Release note (sql change): Added the
bulkio.index_backfill.vector_merge_batch_size cluster setting to control how many vectors to merge into a vector index per transaction during create operations. By default, this defaults to 3.