[Spark] Check isDataFrameWriterV1 option for V1/V2 differentiation #5560
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add support for detecting DataFrameWriter V1 calls via the explicit isDataFrameWriterV1 option provided by Spark 4.1, while maintaining backward compatibility with stack trace inspection for older Spark versions. This allows Delta to differentiate between DataFrameWriter V1 and V2 API calls in Spark Connect mode where stack trace inspection no longer works.
Contributor
Author
|
Superseeded by #5579 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
Add support for detecting DataFrameWriter V1 calls via the explicit isDataFrameWriterV1 option provided by Spark 4.1 in apache/spark#53173, while maintaining backward compatibility with stack trace inspection for older Spark versions.
This allows Delta to differentiate between DataFrameWriter V1 and V2 API calls in Spark Connect mode where stack trace inspection no longer works.
This has been an existing horrible hack in Delta. Having Spark pass it as an option is a leaky solution. However, at this point this is an emergency fix, as releasing Spark 4.1 as is would cause data corruption issues with Delta in DataFrameWriter saveAsTable in overwrite mode, as it would not be correctly interpreting it's overwriteSchema mode.
How was this patch tested?
It has been tested with tests that are not part of the PR. To properly test in connect mode, changes are needed on both Spark and Delta side and integrating it will be done as followup work.
Does this PR introduce any user-facing changes?
No.