[RELEASE] cudf v25.10 #20108
Open
[RELEASE] cudf v25.10 #20108
+103,890
−83,110
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…benchmarks (#19674) Follows up #19631. I didn't set the set `query_set` (ie. the benchmark name `pdsh` or `pdsds`) arg when running the PDS-DS benchmarks in validation mode (ie. `--engine validate`) or the DuckDB benchmarks (ie. `--engine duckdb`). Therefore we'd get this error ``` $ python python/cudf_polars/cudf_polars/experimental/benchmarks/pdsds.py --engine duckdb 12 --root tpcds_parquet --scale 1.0 Traceback (most recent call last): File "/home/coder/cudf/python/cudf_polars/cudf_polars/experimental/benchmarks/pdsds.py", line 218, in <module> run_duckdb(PDSDSDuckDBQueries, extra_args) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/coder/cudf/python/cudf_polars/cudf_polars/experimental/benchmarks/pdsds.py", line 109, in run_duckdb run_config = RunConfig.from_args(args) File "/home/coder/cudf/python/cudf_polars/cudf_polars/experimental/benchmarks/utils.py", line 216, in from_args name = args.query_set ^^^^^^^^^^^^^^ AttributeError: 'Namespace' object has no attribute 'query_set' ``` This PR fixes this error. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: #19674
Updates to `numba-cuda >=0.18.0,<0.19.0`. Drops dependency on `pynvjitlink`. Updates cuda-python pinning to `>=12.9.1,<13.0.0a0` to get cuda-bindings support and fix for a segfault with Python 3.13. Authors: - Bradley Dice (https://github.com/bdice) - https://github.com/brandon-b-miller Approvers: - https://github.com/brandon-b-miller - Graham Markall (https://github.com/gmarkall) - Vyas Ramasubramani (https://github.com/vyasr) URL: #19604
Contributes to #15163 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #19671
Contributes to #15163 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #19672
…19541) This pull-request restructures the AST headers to enable them to be used in online-compiled (JIT-compiled) CUDA. It also adds an operator-to-string utility necessary for CUDA codegen. It also fixes the `std::` namespace references in the floating_conversion header to use `cuda::std::`. Precedes #19467 Authors: - Basit Ayantunde (https://github.com/lamarrr) Approvers: - Tianyu Liu (https://github.com/kingcrimsontianyu) - David Wendt (https://github.com/davidwendt) - Yunsong Wang (https://github.com/PointKernel) URL: #19541
This implements the hash-based workflow for `M2` groupby aggregation. A benchmark is also added, showing that the hash-based approach can improve performance over sort-based by up to 40 percent. Since the output is generated using hash table, the order of keys/values will be undefined thus may not be the same as in the input. --- Benchmark: ``` ## [0] Quadro RTX 6000 | T | value_key_ratio | num_rows | null_probability | Ref Time | Ref Noise | Cmp Time | Cmp Noise | Diff | %Diff | Status | |-----|-------------------|------------|--------------------|------------|-------------|------------|-------------|--------------|---------|----------| | I32 | 10 | 10000 | 0 | 215.940 us | 9.98% | 156.946 us | 8.62% | -58.993 us | -27.32% | FAST | | I32 | 30 | 10000 | 0 | 198.239 us | 1.32% | 149.199 us | 5.59% | -49.039 us | -24.74% | FAST | | I32 | 100 | 10000 | 0 | 196.804 us | 3.81% | 145.584 us | 4.77% | -51.220 us | -26.03% | FAST | | I32 | 10 | 1000000 | 0 | 626.309 us | 1.10% | 635.264 us | 1.85% | 8.955 us | 1.43% | SLOW | | I32 | 30 | 1000000 | 0 | 617.688 us | 1.15% | 477.760 us | 2.14% | -139.929 us | -22.65% | FAST | | I32 | 100 | 1000000 | 0 | 612.476 us | 1.23% | 381.148 us | 1.72% | -231.328 us | -37.77% | FAST | | I32 | 10 | 10000000 | 0 | 7.335 ms | 0.78% | 8.007 ms | 0.64% | 671.021 us | 9.15% | SLOW | | I32 | 30 | 10000000 | 0 | 7.140 ms | 0.78% | 7.845 ms | 0.59% | 704.838 us | 9.87% | SLOW | | I32 | 100 | 10000000 | 0 | 6.975 ms | 0.78% | 6.285 ms | 0.77% | -690.236 us | -9.90% | FAST | | I32 | 10 | 10000 | 0.1 | 288.683 us | 1.84% | 179.761 us | 4.03% | -108.923 us | -37.73% | FAST | | I32 | 30 | 10000 | 0.1 | 287.724 us | 2.13% | 177.806 us | 2.02% | -109.918 us | -38.20% | FAST | | I32 | 100 | 10000 | 0.1 | 285.990 us | 2.53% | 174.958 us | 3.17% | -111.033 us | -38.82% | FAST | | I32 | 10 | 1000000 | 0.1 | 768.166 us | 1.03% | 666.330 us | 1.69% | -101.836 us | -13.26% | FAST | | I32 | 30 | 1000000 | 0.1 | 753.631 us | 1.69% | 514.416 us | 2.36% | -239.214 us | -31.74% | FAST | | I32 | 100 | 1000000 | 0.1 | 747.192 us | 1.01% | 417.477 us | 1.86% | -329.715 us | -44.13% | FAST | | I32 | 10 | 10000000 | 0.1 | 7.948 ms | 0.80% | 7.958 ms | 0.55% | 9.830 us | 0.12% | SAME | | I32 | 30 | 10000000 | 0.1 | 7.704 ms | 0.74% | 7.616 ms | 0.51% | -87.225 us | -1.13% | FAST | | I32 | 100 | 10000000 | 0.1 | 7.538 ms | 0.80% | 6.068 ms | 0.78% | -1469.687 us | -19.50% | FAST | | I32 | 10 | 10000 | 0.9 | 289.933 us | 3.40% | 172.943 us | 2.03% | -116.989 us | -40.35% | FAST | | I32 | 30 | 10000 | 0.9 | 289.697 us | 3.85% | 171.887 us | 2.65% | -117.811 us | -40.67% | FAST | | I32 | 100 | 10000 | 0.9 | 286.871 us | 2.91% | 168.887 us | 1.81% | -117.984 us | -41.13% | FAST | | I32 | 10 | 1000000 | 0.9 | 759.703 us | 1.84% | 469.550 us | 2.02% | -290.153 us | -38.19% | FAST | | I32 | 30 | 1000000 | 0.9 | 749.905 us | 4.00% | 405.214 us | 1.81% | -344.691 us | -45.96% | FAST | | I32 | 100 | 1000000 | 0.9 | 743.373 us | 2.09% | 370.548 us | 1.76% | -372.825 us | -50.15% | FAST | | I32 | 10 | 10000000 | 0.9 | 7.901 ms | 0.69% | 4.686 ms | 1.03% | -3214.663 us | -40.69% | FAST | | I32 | 30 | 10000000 | 0.9 | 7.663 ms | 0.79% | 4.038 ms | 1.30% | -3624.460 us | -47.30% | FAST | | I32 | 100 | 10000000 | 0.9 | 7.483 ms | 0.82% | 3.183 ms | 1.46% | -4299.710 us | -57.46% | FAST | | F64 | 10 | 10000 | 0 | 206.186 us | 1.98% | 152.118 us | 2.07% | -54.068 us | -26.22% | FAST | | F64 | 30 | 10000 | 0 | 206.396 us | 3.77% | 151.780 us | 3.92% | -54.616 us | -26.46% | FAST | | F64 | 100 | 10000 | 0 | 202.879 us | 1.54% | 148.957 us | 3.89% | -53.922 us | -26.58% | FAST | | F64 | 10 | 1000000 | 0 | 678.915 us | 1.96% | 662.225 us | 1.68% | -16.690 us | -2.46% | FAST | | F64 | 30 | 1000000 | 0 | 665.775 us | 1.27% | 515.827 us | 1.72% | -149.948 us | -22.52% | FAST | | F64 | 100 | 1000000 | 0 | 660.599 us | 1.73% | 424.674 us | 1.79% | -235.925 us | -35.71% | FAST | | F64 | 10 | 10000000 | 0 | 7.677 ms | 0.76% | 8.056 ms | 0.59% | 378.792 us | 4.93% | SLOW | | F64 | 30 | 10000000 | 0 | 7.466 ms | 1.05% | 7.851 ms | 0.50% | 385.055 us | 5.16% | SLOW | | F64 | 100 | 10000000 | 0 | 7.285 ms | 0.92% | 6.317 ms | 0.71% | -968.127 us | -13.29% | FAST | | F64 | 10 | 10000 | 0.1 | 299.562 us | 3.55% | 183.848 us | 5.22% | -115.714 us | -38.63% | FAST | | F64 | 30 | 10000 | 0.1 | 299.092 us | 3.72% | 182.238 us | 2.39% | -116.854 us | -39.07% | FAST | | F64 | 100 | 10000 | 0.1 | 296.364 us | 3.12% | 179.645 us | 2.52% | -116.719 us | -39.38% | FAST | | F64 | 10 | 1000000 | 0.1 | 811.874 us | 1.19% | 698.105 us | 1.64% | -113.769 us | -14.01% | FAST | | F64 | 30 | 1000000 | 0.1 | 795.446 us | 1.93% | 553.580 us | 2.28% | -241.867 us | -30.41% | FAST | | F64 | 100 | 1000000 | 0.1 | 801.054 us | 4.61% | 456.706 us | 1.58% | -344.348 us | -42.99% | FAST | | F64 | 10 | 10000000 | 0.1 | 8.174 ms | 0.88% | 8.025 ms | 0.50% | -149.418 us | -1.83% | FAST | | F64 | 30 | 10000000 | 0.1 | 7.929 ms | 0.80% | 7.693 ms | 0.58% | -235.359 us | -2.97% | FAST | | F64 | 100 | 10000000 | 0.1 | 7.736 ms | 0.80% | 6.178 ms | 0.73% | -1557.934 us | -20.14% | FAST | | F64 | 10 | 10000 | 0.9 | 294.503 us | 1.49% | 176.414 us | 9.54% | -118.089 us | -40.10% | FAST | | F64 | 30 | 10000 | 0.9 | 298.363 us | 6.18% | 175.130 us | 2.80% | -123.233 us | -41.30% | FAST | | F64 | 100 | 10000 | 0.9 | 292.613 us | 1.48% | 171.707 us | 2.15% | -120.906 us | -41.32% | FAST | | F64 | 10 | 1000000 | 0.9 | 792.239 us | 1.35% | 482.531 us | 1.84% | -309.708 us | -39.09% | FAST | | F64 | 30 | 1000000 | 0.9 | 776.355 us | 1.14% | 422.448 us | 1.78% | -353.907 us | -45.59% | FAST | | F64 | 100 | 1000000 | 0.9 | 771.947 us | 1.58% | 391.754 us | 2.64% | -380.193 us | -49.25% | FAST | | F64 | 10 | 10000000 | 0.9 | 8.051 ms | 0.76% | 4.735 ms | 1.10% | -3315.632 us | -41.18% | FAST | | F64 | 30 | 10000000 | 0.9 | 7.822 ms | 0.75% | 4.110 ms | 1.41% | -3711.202 us | -47.45% | FAST | | F64 | 100 | 10000000 | 0.9 | 7.645 ms | 0.80% | 3.254 ms | 1.47% | -4390.196 us | -57.43% | FAST | ``` Authors: - Nghia Truong (https://github.com/ttnghia) Approvers: - Alessandro Bellina (https://github.com/abellina) - Yunsong Wang (https://github.com/PointKernel) - Shruti Shivakumar (https://github.com/shrshi) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #19569
Contributes to #15163 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: #19683
…ucture (#19637) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19637
…cture (#19640) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19640
Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19677
This PR changes cudf CI to run pylibcudf tests twice, once with optionals installed and once without. I'd love to be able to test without pyarrow for our interop bits, but I don't think it's realistic given how heavily we rely on pyarrow in the test suite. Resolves #18201 Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #19657
…19267) Adds support to return the aggregation result for MEDIAN and QUANTILE as specified in the `output_type` parameter in the `cudf::reduce` API. The underlying code calls `cudf::detail::quantile` which has an `exact` parameter that returns a `double` type if set to true. Otherwise, it will return a value using the input type. The code change here is to set `exact` to true only if the `output_type` for `cudf::reduce` is set to `FLOAT64` This means the current behavior can be achieved by always setting the `output_type` to `FLOAT64`. Authors: - David Wendt (https://github.com/davidwendt) - Matthew Murray (https://github.com/Matt711) Approvers: - Lawrence Mitchell (https://github.com/wence-) - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) - Nghia Truong (https://github.com/ttnghia) URL: #19267
…strings (#19558) This case was falling back when _sometimes_ this case can be faithfully represented in cuDF. The exception to _sometimes_ is when the data has missing values. pandas strings, pre- and post- pandas 3.0, can have a varied amount of missing value sentinels (`None`, `np.nan`, `pd.NA`) depending if the pandas string type is `object` or `pd.StringDtype`. I don't anticipate it being practical to preserve which missing value sentinel(s) were specified in pandas while round-tripping though cuDF. Currently cuDF always `to_pandas` string data as `object` type with `None` as the missing value sentinel which isn't always correct (per the additions to `conftest-patch.py`). Discussed offline, it probably isn't worth falling back if strings with missing value could never be accelerated. Additionally, defines `DataFrame.__iter__` on the proxy object to short circuit to returning `pandas.DataFrame.__iter__` to avoid the fallback logic Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: #19558
…reshape._merge_sorted (#19614) Towards #9999 Towards #15723 Also removed `reshape._merge_sorted` and its associated tests which were very slow. It appears this was once used by `dask_cudf`, xref #10713, but I cannot find usage of this anymore (possibly removed with migration to dask expressions?) Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19614
…cture (#19639) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19639
…ucture (#19668) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19668
…sic test directories (#19599) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19599
…ty) (#19680) Closes #19679. Also unblocks #19242 Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19680
Contributes to NVIDIA/spark-rapids#12982 Different GPU should generate different UUID sequences. I plan to use GPU card UUID to participate in the calculation of initial seed, so post this PR to expose the Gpu UUID. The seed will be calculated on Java side, the following informations will parqicipate in the intial seed: - Gpu card UUID - Current timestamp - Java process ID - Java process start time - an static incremental sequence ID on Java side. ...... Maybe more to make sure no same UUID sequences are generated. Authors: - Chong Gao (https://github.com/res-life) Approvers: - Liangcai Li (https://github.com/firestarman) URL: #19646
…tory (#19681) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19681
Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19688
We have two new failures in the pandas test suite that I started seeing yesterday. I strongly suspect that these are due to [the new xarray release](https://pypi.org/project/xarray/). Adding to the xfail list for now to unblock our CI. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: #19705
Towards #18863 To support decimal types in cudf_polars, we'll need to preserve/assign a resulting precision when exporting `to_polars` based on the input decimal precision from Polars. This PR allows us to write out the intended precision to the Arrow schema when utilizing `column_metadata` to interop between cuDF and Polars via Arrow. Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - David Wendt (https://github.com/davidwendt) URL: #19587
…ing directory (#19675) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19675
#19687) Towards #9999 Towards #15723 Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19687
The default value for the `--executor` pytest parameter is still `"in-memory"`. This means we are currently running the in-memory tests twice in CI. The good news is that the `--blocksize-mode small` tests have still been hitting the `"streaming"` engine, so I'm pretty sure there is no need to worry about 25.08. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) - Matthew Murray (https://github.com/Matt711) - Vyas Ramasubramani (https://github.com/vyasr) URL: #19703
Moves the libcudf quantiles benchmark from googlebench to nvbench. Authors: - David Wendt (https://github.com/davidwendt) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Nghia Truong (https://github.com/ttnghia) - Vyas Ramasubramani (https://github.com/vyasr) URL: #19692
This is a follow-up to #19657 to remove hard pyarrow and numpy requirements from the pylibcudf conda recipe. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #19706
…ucture (#19626) Towards #9999 Towards #15723 Includes some reduction themed test in test_string.py and test_timedelta.py Some test were consolidated or removed as similar to existing test Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #19626
This merge request switches CUDF's JITIFY2 upstream from the RAPIDSAI fork (https://github.com/rapidsai/jitify/tree/jitify2) to the NVIDIA fork (https://github.com/NVIDIA/jitify/tree/jitify2). The NVIDIA fork has newer features & improvements, including NVTX ranges, which are needed to properly benchmark each compilation/cache step of a specific JIT workload. This was previously impossible to track accurately and concisely. Follows up: #18023 Authors: - Basit Ayantunde (https://github.com/lamarrr) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Muhammad Haseeb (https://github.com/mhaseeb123) - Vyas Ramasubramani (https://github.com/vyasr) URL: #19561
This pull request updates cuDF to support building with CCCL 3.1.0. This only makes it _possible_ to build with CCCL 3.1 while retaining compatibility with CCCL 3.0. The actual CCCL 3.1 update is targeting the 25.12 release. Authors: - Basit Ayantunde (https://github.com/lamarrr) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) - David Wendt (https://github.com/davidwendt) URL: #19886
Closes #19976. Contributes to rapidsai/velox#16 This PR implements filtering parquet row groups using byte offset bounds such that only the row groups that **start** (and not necessarily end) within the specified byte range via `skip_bytes` and `num_bytes` options. Note that byte range based row group filtering is only supported for single-source parquet reads. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Devavret Makkar (https://github.com/devavret) URL: #19991
## Description Fixes some subtle edge cases found in #20079 That PR will be re-targeted to 25.12. However, it **may** make sense to include these fixes in 25.10 if there is still time. ## Checklist - [ ] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes.
Fix `numexpr` related failures in that started happening with latest version of `numexpr`: https://github.com/rapidsai/cudf/actions/runs/17979446647/job/51141280437 https://pypi.org/project/numexpr/#history Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: #20092
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
github-actions
bot
added
libcudf
## Description This PR fixes CI failures seen with the release of `pandas-2.3.3`(released yesterday). ## Checklist - [x] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-25.10and v25.10 releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-25.10until release (merging of this PR).What is the purpose of this PR?
branch-25.10intomainfor the release