[Variant] [Shredding] Support typed_access for `Struct`

[alamb]

NOTE: This is likely a pretty tricky one to implement so it is not a good choice if you are looking for something straightforward to help with)

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• Part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736

We are implementing Variant support in arrow-rs (🎉 ), and we are currently implementing "shredding" - the ability to read Variants when they have been partially extracted into a type value column.

This ticket tracks adding support for accessing typed values of the type listed in the description of this ticket as a Variant value

Describe the solution you'd like

Fix the corresponding test cases in the shredded_variant test suite.

After #8325 is merged, you can run that suite using:

cargo test --all-features --test variant_integration

Describe alternatives you've considered

1. Extend typed_value_to_variant for the type (source link)
2. Add an appropriate test case, following the models of tests such as

arrow-rs/parquet-variant-compute/src/variant_get/mod.rs

Lines 442 to 445 in fb7d02e

	#[test]
	fn get_variant_partially_shredded_uint8_as_variant() {
	numeric_partially_shredded_test!(u8, partially_shredded_uint8_variant_array);
	}

Additional context

[scovich]

While doing a bit of pathfinding for this issue, I realized that both struct (this issue) and list (#8337) support will hit a fundamental limitation/flaw in VariantArray::value and the Variant class it returns:

• Variant is a borrowed type, when it comes to complex variant values (objects and arrays)
• But unshredding an object or array requires building a new variant value, whose bytes are now owned by the function that created them.
• We currently have no way to return owned variant value bytes
• Primitive variant values only worked "by accident" because they parse the input bytes to values instead of retaining a reference to those bytes.

Meanwhile, VariantArray::value has a huge number of call sites, so any change we make here will seriously disrupt the user surface. Things I tried so far, none of which seemed satisfactory:

1. VariantArray::value returns a new enum VariantArrayValue, with an Borrowed variant (a wrapper aroundVariant) and an Owned variant (which contains a metadata byte slice and a value byte vec).
   • PRO: Expedient. A relatively local change that makes clear what's going on.
   • CON: It's very annoying API surface, because the same borrowing rules that prevent VariantArray::value from returning a Variant directly also prevent any member method of VariantArrayValue from doing so (unless we start delving into unsafe cell magic).
2. VariantObject and VariantList take Cow<'v, [u8]> instead of &v [u8].
   • PRO: It should hide COW issues from the vast majority of users of Variant API.
   • CON: Several internal methods rely on slice manipulation that is not friendly to an owned byte slice
   • CON: Bloats those classes from 64B to 72B
3. Some kind of split API in VariantArray that replaces the value method, e.g. primitive_value or complex_value
   • PRO: Only call sites that expect complex owned values would have to deal with them, and in theory no such call sites exist yet because this is currently an unsupported operation.
   • CON: s/value/primitive_value/g will still be a hugely disruptive operation, and it's also not obvious that "primitive" vs. "complex" is really the right place to split it. The real issue is owned vs. borrowed, and there's no obvious way for a reader to efficiently determine whether a given row will produce an owned or borrowed value, without doing a lot of up-front work.

CC @alamb @codephage2020 @klion26 @liamzwbao @friendlymatthew who have worked on various bits of this code over the past few months.

[scovich]

A smaller issue I encountered is that unshredding a partially shredded object is a fallible operation for various reasons, but VariantArray::value and the helper methods it relies on are all infallible. I suspect the same issue will also apply to shredded variant arrays.

Today we work around some existing failure cases by returning Variant::Null (along with a failing debug_assert!), but it would be much nicer to surface a fallible API that callers can choose to use if they don't like to risk panics.

[alamb]

But unshredding an object or array requires building a new variant value, whose bytes are now owned by the function that created them.

Maybe this is the core problem 🤔 Can we update what "unshredding" means so it is also a borrowing operation?

For example, if we unshred a field that is a String but was shredded into a StringArray then it could return a Variant::String that borrows from the underlying string array

Variant::Object with some shredded and some unshredded fields is probably the trickiest thing to get working

[scovich]

Can we update what "unshredding" means so it is also a borrowing operation?

No. Partially shredded objects and arrays have to be rebuilt from scratch, allocation-wise. Some of the bytes come from the value column, and others are newly created from the typed_value column. And all this happens recursively. I don't know any way to represent it as a borrowed byte slice.

[alamb]

Can we update what "unshredding" means so it is also a borrowing operation?

No. Partially shredded objects and arrays have to be rebuilt from scratch, allocation-wise. Some of the bytes come from the value column, and others are newly created from the typed_value column. And all this happens recursively. I don't know any way to represent it as a borrowed byte slice.

I was thinking of adding an enum Variant::ShreddedObject that could hold whatever info was necessary

I know I have mentioned this before but haven't yet produced any code to see how the idea could actually work in practice 🤔

[scovich]

Can we update what "unshredding" means so it is also a borrowing operation?

No. Partially shredded objects and arrays have to be rebuilt from scratch, allocation-wise. Some of the bytes come from the value column, and others are newly created from the typed_value column. And all this happens recursively. I don't know any way to represent it as a borrowed byte slice.

I was thinking of adding an enum Variant::ShreddedObject that could hold whatever info was necessary

I know I have mentioned this before but haven't yet produced any code to see how the idea could actually work in practice 🤔

Given how ugly some of the other alternatives are proving to be, maybe it's worth throwing that on the pile of things to try out. It would be a lot of small byte slices and other bits of information to worry about tho, and would ~double the cost of unshredding (one pass collects all the stuff and a second pass emits the new bytes).