[SPARK-53348] [SQL] Always persist ANSI value when creating a view or assume it when querying if not stored #52092
+215
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mihailoale-db
force-pushed
the
ansiview
branch
3 times, most recently
from
b1a5ff1 to
d86b348
Compare
|
@cloud-fan PTAL when you have time. Thanks! |
cloud-fan
reviewed
Aug 22, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
Outdated
Show resolved
Hide resolved
cloud-fan
reviewed
Aug 22, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
Outdated
Show resolved
Hide resolved
cloud-fan
reviewed
Aug 22, 2025
| */ | ||
| def sqlConfigsToProps(conf: SQLConf): Map[String, String] = { | ||
| val modifiedConfs = ViewHelper.getModifiedConf(conf) | ||
| modifiedConfs.map { case (key, value) => s"$SQL_CONFIG_PREFIX$key" -> value } | ||
|
|
||
| val alwaysCaptured = Seq(SQLConf.ANSI_ENABLED) |
There was a problem hiding this comment.
We should unify it with the view code. We can do it later in a followup.
There was a problem hiding this comment.
Wanted to do that in a followup. We also don't store SESSION_LOCAL_TIMEZONE for UDFs which is a gap.
cloud-fan
approved these changes
Aug 22, 2025
mihailoale-db
force-pushed
the
ansiview
branch
from
d86b348 to
181900d
Compare
|
thanks, merging to master/4.0! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
I propose that we always store ANSI value when creating a view because otherwise users can be affected by unwanted behavior. For example if user creates a view on version that has ANSI = false by default he expects this not to fail.
But if user queries the view on the version which has ANSI = true by default, above query is going to fail (because when we don't store the value, and we store it only if explicitly set, we use the default one). Number of this and similar use cases is huge, because ANSI impact area is huge and thus I propose that we always store the value.
If the value is not stored, I propose that we use createVersion field to determine whether the ANSI value should be true (Spark 4.0.0 and above) or false (lower than Spark 4.0.0). If the createVersion field wasn't stored during view creation, I propose that we assume that the ANSI = false because number of those views is incomparable larger than the ones expecting ANSI = true
Why are the changes needed?
To improve user experience.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added suite.
Was this patch authored or co-authored using generative AI tooling?
No.