Skip to content

Conversation

bdice
Copy link
Contributor

@bdice bdice commented Sep 26, 2025

Description

The PDS-H queries don't obey --rmm-async unless using the distributed scheduler. This adds a note to the docs.

Perhaps we should solve this differently, so that --rmm-async actually works for all schedulers?

A workaround is setting POLARS_GPU_ENABLE_CUDA_MANAGED_MEMORY=0.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@bdice bdice requested a review from a team as a code owner September 26, 2025 23:55
@github-actions github-actions bot added Python Affects Python cuDF API. cudf-polars Issues specific to cudf-polars labels Sep 26, 2025
@GPUtester GPUtester moved this to In Progress in cuDF Python Sep 26, 2025
@TomAugspurger
Copy link
Contributor

Sorry for missing this earlier.

Perhaps we should solve this differently, so that --rmm-async actually works for all schedulers?

That'd be done at

engine = pl.GPUEngine(
raise_on_fail=True,
executor=run_config.executor,
executor_options=executor_options,
)
. It'd be a bit of logic to parse out the rmm-related options and construct the memory_resource argument to GPUEngine.

#20042 is also a bit related here, which adds another (slightly more flexible) way to configure RMM MRs.

I think that your change here is a good improvement and we can merge it as is. I can handle integrating these RMM options into that as part of #20042.

@bdice bdice added doc Documentation non-breaking Non-breaking change labels Sep 30, 2025
@bdice
Copy link
Contributor Author

bdice commented Sep 30, 2025

/merge

@rapids-bot rapids-bot bot merged commit 706164d into rapidsai:branch-25.12 Sep 30, 2025
140 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in cuDF Python Sep 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf-polars Issues specific to cudf-polars doc Documentation non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants