[BUG] Sparse vector index incorrect blockfile path

[Sicheng-Pan]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • After collection fork, the sparse vector blockfile are put under the child collection prefix, which is inconsistent to the existing behavior where new blockfiles are put under the root collection prefix. Although the long term fix is to move all new blockfiles after fork under child collection prefix, as a temporary workaround we just put the sparse vector blockfiles under the root collection as well.
• New functionality
  • N/A

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?

Observability plan

What is the plan to instrument and monitor this change?

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[Sicheng-Pan]

• [BUG] Sparse vector index incorrect blockfile path #5418 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[propel-code-bot]

Fix Sparse Vector Index Blockfile Path Consistency After Collection Fork

This PR addresses a bug where, after a collection fork, newly created sparse vector index blockfiles were being placed under the child collection's prefix, which was inconsistent with other blockfiles that remained under the root collection's prefix. As an interim measure, the code is updated so that sparse vector blockfiles are also created under the original (root) collection's prefix, keeping all metadata segment blockfiles consistent post-fork. The change uses more idiomatic Rust (leveraging flattening of file paths) and improves related error handling. An extensive new asynchronous test is added to ensure correctness in scenarios involving collection forking and legacy segments.

Key Changes

• Modified logic in MetadataSegmentWriter::from_segment to, when present, place new sparse vector blockfiles under the same prefix as existing blockfiles, maintaining storage consistency post-collection fork.
• Refactored file path lookup logic using .values().flatten().next() instead of the prior less idiomatic approach.
• Improved error handling to retain error context when UUID parsing fails on file paths.
• Added a comprehensive async test (test_sparse_index_recreated_with_existing_prefix) that simulates multiple cycles of blockfile creation, forking, legacy handling, and sparse index recreation to verify the correct prefix is always used across scenarios.

Affected Areas

• rust/segment/src/blockfile_metadata.rs (MetadataSegmentWriter::from_segment logic and related tests)

This summary was automatically generated by @propel-code-bot

[HammadB]

would be good to add a test for the other case we discussed offline

[Sicheng-Pan]

Extended existing test