Skip to content

Commit f0e8999

Browse files
asl3zhengruifeng
authored andcommitted
[SPARK-53265][PYTHON][DOCS] Add Arrow Python UDF Type Coercion Tables in Arrow Python UDF Docs
### What changes were proposed in this pull request? Link Arrow Python UDF Type Coercion Tables in Arrow Python UDF Docs, from #51225. This PR replaces #52004 due to docs build failure. ### Why are the changes needed? Improve documentation of behavior change ### Does this PR introduce _any_ user-facing change? Yes, updates docs ### How was this patch tested? Docs build ### Was this patch authored or co-authored using generative AI tooling? No Closes #52025 from asl3/arrowpandasudf-typecoerciontabledoc2. Authored-by: Amanda Liu <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
1 parent 8530444 commit f0e8999

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

python/docs/source/tutorial/sql/arrow_pandas.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -375,6 +375,12 @@ fallback for type mismatches, leading to potential ambiguity and data loss. Addi
375375
and tuples to strings can yield ambiguous results. Arrow Python UDFs, on the other hand, leverage Arrow's
376376
capabilities to standardize type coercion and address these issues effectively.
377377

378+
A note on Arrow Python UDF type coercion: In Spark 4.1, unnecessary conversion to pandas instances is removed in the serializer
379+
when ``spark.sql.execution.pythonUDF.arrow.enabled`` is enabled. As a result, the type coercion changes
380+
when the produced output has a schema different from the specified schema. To restore the previous behavior,
381+
enable ``spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled``.
382+
The behavior difference is summarized in the tables `here <https://github.com/apache/spark/pull/51225>`__.
383+
378384
Usage Notes
379385
-----------
380386

0 commit comments

Comments
 (0)