Skip to content

Commit dc9c3cd

Browse files
authored
[Variant] Implement row builders for cast_to_variant (#8299)
Note to reviewers: This PR includes 1600+ LoC of new unit tests. The actual changes are half that big. # Which issue does this PR close? We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. - Closes #8310 # Rationale for this change The original `cast_to_variant` code did columnar conversions of types to variant. For primitive types this worked ok, but for deeply nested types it means repeatedly creating new variants (and variant metadata), only to re-code them by copying the variant values to new arrays (with new metadata and field ids). Very expensive. # What changes are included in this PR? Follow the example of #8280, and introduce a row builder concept that takes individual array values and writes them to an `impl VariantBuilderExt`. Row builders for complex types instantiate the appropriate list or object builder to pass to their children. # Are these changes tested? Existing unit tests continue to pass. Extensive new unit tests added as well. # Are there any user-facing changes? * `VariantBuilderExt` has a new `append_null` method. * `ObjectFieldBuilder` moved to `builder.rs` and made public
1 parent f751a45 commit dc9c3cd

File tree

7 files changed

+2151
-737
lines changed

7 files changed

+2151
-737
lines changed

parquet-variant-compute/src/arrow_to_variant.rs

Lines changed: 1995 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)