[SPARK-52982][PYTHON] Disallow lateral join with Arrow Python UDTFs #52048
Conversation
| """ | ||
| SELECT t.id, f.x, f.result | ||
| FROM test_table t, LATERAL simple_arrow_udtf(t.id) f | ||
| """ |
There was a problem hiding this comment.
nit: one more indent?
| """ | |
| SELECT t.id, f.x, f.result | |
| FROM test_table t, LATERAL simple_arrow_udtf(t.id) f | |
| """ | |
| """ | |
| SELECT t.id, f.x, f.result | |
| FROM test_table t, LATERAL simple_arrow_udtf(t.id) f | |
| """ |
|
|
||
|
|
| LateralSubquery(SubqueryAlias(alias, tvfWithTableColumnIndexes)), Inner, None) | ||
|
|
||
| // Set the tag so that it can be used to differentiate lateral join added by | ||
| // TABLE argument vs added by user. |
There was a problem hiding this comment.
why arrow UDTF is fine for LATERAL JOIN generated by TABLE arguement?
There was a problem hiding this comment.
This lateral join is only used for analyzer to resolve the plan and it will be removed during the optimization stage and become:
EvalPythonUDTFArrow
+- SubqueryAlias
+- Table Subquery
| @@ -4086,6 +4086,13 @@ | |||
| ], | |||
| "sqlState" : "42K0L" | |||
| }, | |||
| "LATERAL_JOIN_WITH_ARROW_UDTF_UNSUPPORTED" : { | |||
| "message" : [ | |||
| "LATERAL JOIN with Arrow-optimized user-defined table functions (UDTFs) is not supported. Arrow UDTFs cannot be used on the right-hand side of a lateral join.", | |||
There was a problem hiding this comment.
why arrow UDTF cannot support lateral join?
is it because the row id is missing? Just wondering can we provide a arrow column containing the row id, so that the JVM side can join input rows and output rows again
There was a problem hiding this comment.
Yes that's one option to support lateral join with arrow UDTFs, but I expect most of the use cases will be via TABLE arguments.
allisonwang-db
force-pushed
the
spark-52982-disallow-lateral-join
branch
from
b245030 to
4d88390
Compare
|
Thanks for the review! Merge to master. |
What changes were proposed in this pull request?
Arrow-optimized User-Defined Table Functions (UDTFs) have arbitrary output cardinality, meaning they can produce different numbers of rows for the same batch input rows. This fundamental characteristic is incompatible with lateral join semantics which combines each row from LHS with all rows from RHS for that particular input row. This PR implements a restriction that blocks Arrow Python UDTFs on the right hand side of lateral join:
SELECT * FROM table, LATERAL arrow_udtf(...).It is always recommended to use table argument with arrow UDTFs:
SELECT * FROM arrow_udtf(table(...))Note Regular UDTFs with lateral joins are allowed (unchanged behavior).
Why are the changes needed?
Prevents runtime errors and inconsistent results while maintaining full Arrow UDTF functionality for supported patterns.
Does this PR introduce any user-facing change?
Yes
How was this patch tested?
New unit tests.
Was this patch authored or co-authored using generative AI tooling?
No