Add support for using ListView arrays and types through FFI #8822
+204
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AdamGS
commented
Nov 11, 2025
| ], | ||
| can_contain_null_mask: true, | ||
| variadic: true, | ||
| variadic: false, |
Contributor
Author
There was a problem hiding this comment.
This is a bug, see the spec, list views aren't represented by a variable number of buffers.
AdamGS
commented
Nov 11, 2025
|
|
||
| // returns the number of bits that buffer `i` (in the C data interface) is expected to have. | ||
| // This is set by the Arrow specification | ||
| /// returns the number of bits that buffer `i` (in the C data interface) is expected to have. |
Contributor
Author
There was a problem hiding this comment.
Just nicer when working with IDEs.
AdamGS
changed the title
AdamGS
force-pushed
the
adamg/ffi-list-view
branch
from
247eca5 to
7375a94
Compare
AdamGS
commented
Nov 11, 2025
|
|
||
| _unsupported_pyarrow_types = [ | ||
| ] | ||
| if PYARROW_MAJOR_VER >= 16: |
Contributor
Author
There was a problem hiding this comment.
Did this to get everything green, but I think that it might be a good opportunity to only test with pyarrow 16 and above, which supports all relevant features here - stringview, bianryview and listview and the pycapsule API.
By the time this change (if accepted) will be released, pyarrow 15 will be almost two years old, which seems like enough time to drop dedicated testing for it. This will allow the CI to focus on more relevant releases that also include all the APIs arrow-rs supports.
AdamGS
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Adds support for using list view arrays through FFI
What changes are included in this PR?
Support for the ListView types and arrays
Are these changes tested?
I've added additional tests that verify round trips through FFI, and a pyarrow integration test.
Are there any user-facing changes?
Extended support for ListView, doesn't effect existing functionality.