Add memory resources to I/O modules #20136
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updated text.pyx and text.pxd to support DeviceMemoryResource for multibyte_split function: - Added DeviceMemoryResource parameter to function signature in .pxd file - Updated function to accept and process DeviceMemoryResource parameter - Pass memory resource to Column.from_libcudf() for proper memory management
Updated timezone.pyx and timezone.pxd to support DeviceMemoryResource for make_timezone_transition_table function: - Added DeviceMemoryResource parameter to function signature in .pxd file - Updated function to accept and process DeviceMemoryResource parameter - Pass memory resource to Table.from_libcudf() for proper memory management - Note: C++ libcudf function does not support memory resource parameter, so we accept it for consistency but only pass it to Table.from_libcudf()
- Add DeviceMemoryResource parameter to read_parquet() function - Add DeviceMemoryResource parameter to ChunkedParquetReader.read_chunk() method - Update TableWithMetadata.from_libcudf calls to pass memory resource - Enable memory-controlled parquet data loading operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add DeviceMemoryResource parameter to read_csv() function - Update TableWithMetadata.from_libcudf calls to pass memory resource - Enable memory-controlled CSV data loading operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add DeviceMemoryResource parameter to read_orc() function - Update TableWithMetadata.from_libcudf calls to pass memory resource - Enable memory-controlled ORC data loading operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add DeviceMemoryResource parameter to read_avro() function - Update TableWithMetadata.from_libcudf calls to pass memory resource - Enable memory-controlled Avro data loading operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This commit ensures comprehensive DeviceMemoryResource consistency between C++ headers and Cython bindings across the entire pylibcudf.io subpackage. Changes include: - Add memory resource parameter to JSON read_json function - Fix memory resource forwarding in ORC, Avro, and Text modules - Update all .pxd files with missing device_memory_resource parameters - Fix .pyx implementations to forward memory resources to C++ calls - Update .pyi type annotations for complete API consistency - Fix ChunkedParquetReader constructor to accept memory resource - Resolve TableWithMetadata.from_libcudf argument count issues All IO operations now properly support user-specified memory resources, enabling fine-grained control over GPU memory allocation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
The chunked_read_json function was missing DeviceMemoryResource support, causing it to always use the default memory resource instead of allowing users to specify their own. Fixed: - Add DeviceMemoryResource parameter to function signature in .pxd file - Update function definition in .pyx file to accept the parameter - Forward memory resource to C++ call using mr.get_mr() - Update type annotation in .pyi file This completes 100% DeviceMemoryResource consistency across all pylibcudf I/O modules that perform actual data operations. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
github-actions
bot
added
Python
mroeschke
approved these changes
Sep 29, 2025
Contributor
Author
|
/merge |
rapids-bot
bot
merged commit 2fd5141
into
rapidsai:branch-25.12
237 of 239 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Contributes to #15170
Checklist