[VARIANT] Path-based Field Extraction for VariantArray

[carpecodeum]

Which issue does this PR close?

This PR implements efficient path-based field extraction and manipulation capabilities for VariantArray, enabling direct access to nested fields without expensive unshredding operations.
Follow-up on #7919

• Closes [Variant] Support variant_get kernel for shredded variants #7941

Rationale for this change

This work builds directly on the path navigation concepts introduced in #7919, sharing the fundamental VariantPathElement design with Field and Index variants. While PR #7919 provided a compute kernel approach with a variant_get function, this PR provides instance-based methods directly on VariantArray with a builder API using owned strings rather than PR #7919 vector-based approach.

This is a draft still, as the changes for #7919 got merged today, I still have to incorporate those changes, and looking forward to reviews and suggestions.

This PR is complementary to #7921, which implements schema-driven shredding during array construction. This PR provides runtime path-based access to both shredded and unshredded data, creating a complete solution for both efficient construction and efficient access of variant data.

Big Thanks to @mprammer @PinkCrow007 for their continued support throughout my Variant exploration

What changes are included in this PR?

Field removal operations through methods like remove_field and remove_fields enable removal of specific fields from variant data, crucial for shredding operations where temporary or debug fields need to be stripped. field_operations.rs provides direct binary manipulation through functions like get_path_bytes, extract_field_bytes, and remove_field_bytes that operate on raw binary format without constructing intermediate objects. variant_parser.rs supports all variant types with parsers for 17 different primitive types, providing the foundation for efficient binary navigation.

The performance-critical byte operations could serve as the underlying implementation for PR #7919's compute kernel, potentially providing better performance for batch operations by avoiding object construction overhead. The field removal capabilities could extend PR #7919's functionality beyond extraction to comprehensive field manipulation. The instance-based approach provides different ergonomics that complement PR #7919's compute kernel approach.

This PR focuses on runtime access and manipulation rather than construction-time optimization, leaving build-time schema-driven shredding to PR #7921. Future work is integration with PR #7919's compute kernel approach, potentially using this PR's byte-level operations as the underlying implementation.

Are these changes tested?

Yes, tests are added

Are there any user-facing changes?

Not yet

[carpecodeum]

CC - @alamb @Samyak2 @friendlymatthew @scovich

[alamb]

Thank you for this PR @carpecodeum

This is very cool

I think there is already a variant_get implementation in https://github.com/apache/arrow-rs/blob/d809f19bc0fe2c3c1968f5111b6afa785d2e8bcd/parquet-variant-compute/src/variant_get.rs#L35-L34 contributed by @Samyak2

To take the next steps and implement shredding I think we will need two things:

1. A way to create shredded variants
2. A way to represent shredded variants

The idea of removing fields from Variants is interesting, though I wonder if that is an operation we would ever want to do on single Variant instance -- it seems like removing fields for shredding will require copying the underlying bytes anyways, so I was thinking we might just want to create an output variant array entirely

Something like

fn variant_shred(input: VariantArray, output: VariantArray, schema: SchemaRef)

Maybe it is worth looking at how the java or go implementations work

[alamb]

i think if you merge up from main this code will no longer be required

[carpecodeum]

i agree, a lot of this becomes redundant but get_field_bytes is able to get field bytes from an object at the byte level, will this wont be required too?

[alamb]

I am not quite sure what you are asking.

I think it might help to move shredded variant forward by writing the tests / examples of how variant_get should work with shredded arrays

I tried to work up a simple example here:

• [Variant] WIP Tests for variant_get of shredded variants #7965

[carpecodeum]

I saw your PR it looks great, im myself trying to work up a few examples

[alamb]

One idea would to be to try and create the other examples from https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/ in code

[carpecodeum]

Thank you for this PR @carpecodeum

This is very cool

I think there is already a variant_get implementation in https://github.com/apache/arrow-rs/blob/d809f19bc0fe2c3c1968f5111b6afa785d2e8bcd/parquet-variant-compute/src/variant_get.rs#L35-L34 contributed by @Samyak2

To take the next steps and implement shredding I think we will need two things:

1. A way to create shredded variants
2. A way to represent shredded variants

The idea of removing fields from Variants is interesting, though I wonder if that is an operation we would ever want to do on single Variant instance -- it seems like removing fields for shredding will require copying the underlying bytes anyways, so I was thinking we might just want to create an output variant array entirely

Something like

fn variant_shred(input: VariantArray, output: VariantArray, schema: SchemaRef)

Maybe it is worth looking at how the java or go implementations work

Is there any issue for implementing this? I would love to work on it

[alamb]

Is there any issue for implementing this? I would love to work on it

I think we are discussing reading shredded variants on

• [Variant] Support variant_get kernel for shredded variants #7941

We are discussing creating shredded variants on

• [Variant] API to construct Shredded Variant Arrays #7895

I don't think we have enough of an idea of how this will work to break them down into finer grained tasks yet.

[scovich]

Partial review.

The overriding takeaway for me (which got in the way of reviewing the actual PR functionality) is that this PR has a dangerously high level of code duplication vs. existing code in the parquet-variant crate. Dangerous because it includes many bugs and inefficiencies that the other code already solved (which is ~inevitable with such a high amount of redundancy).

Can we please take the time to harmonize with existing code so reviewers can focus on the really important and exciting net new functionality instead of debugging a reinvented wheel?

[scovich]

Seems like this should be an unwrap, since the only way to get None here is by an out of bounds index?

[carpecodeum]

Partial review.

The overriding takeaway for me (which got in the way of reviewing the actual PR functionality) is that this PR has a dangerously high level of code duplication vs. existing code in the parquet-variant crate. Dangerous because it includes many bugs and inefficiencies that the other code already solved (which is ~inevitable with such a high amount of redundancy).

Can we please take the time to harmonize with existing code so reviewers can focus on the really important and exciting net new functionality instead of debugging a reinvented wheel?

Hi @scovich yes, currently this is still in progress, which essentially means that I am still working on removing the redundancy, its just taking a bit more time than I thought for me because I had to focus on some other stuff the past few days, I still have it in draft as to just make sure people dont confuse it as a ready-to-go PR, but thanks for this review its very helpful.

[scovich]

Partial review.
The overriding takeaway for me (which got in the way of reviewing the actual PR functionality) is that this PR has a dangerously high level of code duplication vs. existing code in the parquet-variant crate. Dangerous because it includes many bugs and inefficiencies that the other code already solved (which is ~inevitable with such a high amount of redundancy).
Can we please take the time to harmonize with existing code so reviewers can focus on the really important and exciting net new functionality instead of debugging a reinvented wheel?

Hi @scovich yes, currently this is still in progress, which essentially means that I am still working on removing the redundancy, its just taking a bit more time than I thought for me because I had to focus on some other stuff the past few days, I still have it in draft as to just make sure people dont confuse it as a ready-to-go PR, but thanks for this review its very helpful.

Ah, sorry if I was over-eager to review this PR. LMK if there are specific things that would be helpful to review, or if it's better to just wait for it to finish baking?

[carpecodeum]

After taking another look at both this PR and other PRs that have been opened or merged over the week, it seems like the only non-duplicate portions of this PR are the "get_field_names" and the "remove_field(s)" functionality. I've deduplicated this PR and cleaned things up to reflect this current status.
@scovitch has mentioned (#7935) that efficient, in-place Variant mutators are a piece that's still missing. Field removal naturally fits into a PR focused on that functionality.
I can take "add_field(s)" if we'd like both append and removal functionality as part of the initial implementation. Separately, convenience functions to streamline interacting with Variants seem to be absent as well. Is there an interest in implementing traits for Variants like Index/IndexMut? I'm happy to take either of these efforts.

Thanks @scovitch and @alamb for reviewing. Do we think this PR is both substantial and useful enough on its own? Or, do we want to close this PR and open one that's more targeted? I'm happy to follow through with either, favoring whatever the reviewers think would keep things clean.

[scovich]

convenience functions to streamline interacting with Variants seem to be absent as well. Is there an interest in implementing traits for Variants like Index/IndexMut? I'm happy to take either of these efforts.

Problem is, those traits return references (which the compiler then transparently dereferences). AFAIK, it's impossible to implement those traits if the return value is "manufactured" by the function call itself, because that would require returning a reference to a temporary object that goes out of scope as soon as the function returns.

... which is why we have Index for VariantMetadata (which does return a reference to an underlying string entry), but not for Variant

[alamb]

👋 Hi @carpecodeum -- I was checking in on this PR and seeing if it is ready for another round of review

I am not sure if you have seen, but @rdblue gave us an early holiday 🎁 in the form of example shredded parquet variant files in parquet-testing: apache/parquet-testing#90

[alamb]

@scovitch has mentioned (#7935) that efficient, in-place Variant mutators are a piece that's still missing. Field removal naturally fits into a PR focused on that functionality.

I don't really understand why in-place mutations are needed to implement get for VariantArray. I think that would only be necessary (potentially) for writing shredded values 🤔

It seems to me that we already have

1. Variant::get_path to get some arbitrary path from a single Variant:

   arrow-rs/parquet-variant/src/variant.rs

   Lines 1076 to 1103 in 16794ab

   	/// # Example
   	/// ```
   	/// # use parquet_variant::{Variant, VariantBuilder, VariantObject, VariantPath};
   	/// # let mut builder = VariantBuilder::new();
   	/// # let mut obj = builder.new_object();
   	/// # let mut list = obj.new_list("foo");
   	/// # list.append_value("bar");
   	/// # list.append_value("baz");
   	/// # list.finish();
   	/// # obj.finish().unwrap();
   	/// # let (metadata, value) = builder.finish();
   	/// // given a variant like `{"foo": ["bar", "baz"]}`
   	/// let variant = Variant::new(&metadata, &value);
   	/// // Accessing a non existent path returns None
   	/// assert_eq!(variant.get_path(&VariantPath::from("non_existent")), None);
   	/// // Access obj["foo"]
   	/// let path = VariantPath::from("foo");
   	/// let foo = variant.get_path(&path).expect("field `foo` should exist");
   	/// assert!(foo.as_list().is_some(), "field `foo` should be a list");
   	/// // Access foo[0]
   	/// let path = VariantPath::from(0);
   	/// let bar = foo.get_path(&path).expect("element 0 should exist");
   	/// // bar is a string
   	/// assert_eq!(bar.as_string(), Some("bar"));
   	/// // You can also access nested paths
   	/// let path = VariantPath::from("foo").join(0);
   	/// assert_eq!(variant.get_path(&path).unwrap(), bar);
   	/// ```

2. The variant_get compute kernel to extract a column of variants from a VariantArray :

   arrow-rs/parquet-variant-compute/src/variant_get.rs

   Line 35 in 16794ab

   pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {

What we are missing from the read side from what I can tell is the ability to read shredded variants (aka VariantArrays where the underlying array has a typed_value field)

What would you think about trying to implement the code to make the tests in #7965 work?