Release DataFusion `51.0.0` (Nov 2025)

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Previous release will be #16799 (Sep, 2025) so next major release would be approximately late Nov 2025

Steps:

• Create a branch-51 branch:
• Create blog post
• Update version and changelog:
• Test with DataFusion Python
• Test with DataFusion Comet
• Test with delta.rs:
• Test vortex:
• Test with iceberg-rust:
• Test with LakeSail:
• Test with parquet viewer
• Test with datafusion-materialized-views
• Voting Thread:
• Publish to crates.io
• Create ticket for next release:

TODOs

Prior release tickets:

• Release DataFusion 50.0.0 (Aug/Sep 2025) #16799

Related

• Boston USA Nov 2025 meetup: DISCUSSION: DataFusion Meetup in Boston, USA - Nov 12, 2025 #16703

Features to mention in the blog (if they make it)

• Support Decimal32/64 types #17501 from @AdamGS
• Support WHERE, ORDER BY, LIMIT, SELECT, EXTEND pipe operators #17278 (comment)
• Reduce number of object store requests when reading parquet files by default (set metadata_size_hint) #18118
• [EPIC] A collection of items to improve CASE performance #18075
• feat: change Expr OuterReferenceColumn and Alias to Box type for reducing expr struct size #16771
• Examples of extending SQL syntax #17824 from the @theirix

Issuses needed

• Default time_zone config to None  #18204

Bugs that need to be fixed

• Restore window sort optimizations without exponential planning time #17624
• Exponential planning time when window function is partitioned by multiple columns #17401
• Non-constant DATE_TRUNC expression regression for values before epoch #18334
• Optimizer 'common_sub_expression_eliminate' fails in a window function (SQLStorm) #17770 / fix: optimizer common_sub_expression_eliminate fails in a window function #17852
• CI is failing on main / current_date() = cast(now() as date); #18062
• NowFunc::new() seeds the timezone with "+00" while the canonical UTC offset remains "+00:00" in ConfigOptions::default(). #18219
• to_timestamp(double) gives different results depending on scalar/vectorized call context #16678
• Regression: sql_planner benchmark panic'ing on main #17801
• Respect execution timezone in to_timestamp and related functions #18025

Community Wishlist

• Streaming Aggregate operator not being used in deduplication of pre-sorted Parquet files #16919
• Regression: DataFrameWriteOptions::with_single_file_output produces a directory #13323
• fix(parquet): converting parquet schema with backward compatible repeated struct/primitive with provided arrow schema arrow-rs#8496

[mjgarton]

Should the title of this issue say 51.0.0 instead of 50.0.0?

[xudong963]

Should the title of this issue say 51.0.0 instead of 50.0.0?

yes, thank you, i changed it.

[findepi]

I removed #16842 from "Changes to add to upgrade guide". I think it was completed in 50.0.0. If not, let's add it back.

[xudong963]

I think we could start working on the release next week?

[rluvaton]

If we could get the following arrow fix merged and released as a patch and upgrade to that patch before the upcoming release it would be sweet:

• fix(parquet): converting parquet schema with backward compatible repeated struct/primitive with provided arrow schema arrow-rs#8496

[alamb]

If we could get the following arrow fix merged and released as a patch and upgrade to that patch before the upcoming release it would be sweet:

• fix(parquet): converting parquet schema with backward compatible repeated struct/primitive with provided arrow schema arrow-rs#8496

I think that one will be released in arrow 57.1.0 (a minor update that won't require a datafusion upgrade).

So once we have released datafusion 51 we'll be able to do a minor update to arrow 57.1.0

(in other words, I don't think the release timeline of DataFusion will impact the flow of that PR)

[xudong963]

FYI, I plan to cut the 51.0.0 in the next two days. Do we have some high-priority PRs waiting for review?

[pepijnve]

FYI, I plan to cut the 51.0.0 in the next two days. Do we have some high-priority PRs waiting for review?

#18444 and #18183 are the remaining two for the CASE epic. Would be nice if we could still get those in.

[alamb]

@Omega359 do we have additional work needed for the default timezone fixes?

• E.g. follow ups from Change default time_zone to None (was "+00:00")  #18359