[Variant] Support Shredded Objects in `variant_get`

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736

Note this is likely one of the most complex parts of implementing Shredded Variants , so it is not a good first task

We introduced the basic variant_get kernel in and now we need to expand its feature set

• [Variant] Add variant_get and Shredded VariantArray #8021

In particular, Shredded Objects

• Representing Variant In Arrow Proposal: "Shredding an Object"
• Variant Shredding::Objects

Describe the solution you'd like
I would like variant_get to support shredded objects, along with being able to extract elements from them.

So roughly that means

// get the named field of variant object as a Variant
variant_get(array, "$.field_name")
// get the named field of variant object as a typed field 
variant_get(array, "$.field_name", DataType::Int)

This should work for:

1. Variants where the field_name is in a typed_value
2. Variants where the field_name is not in the typed value

Describe alternatives you've considered

1. Add a test that manually constructs a shredded variant array (follow the example in the arrow proposal)
2. Add a test that calls variant_get appropriately
3. Implement the code

I suggest getting this working for non-nested obejcts first, and then working on nesting / pathing as a second pR

Additional context
Reference

• Variant Spec
• Variant Shredding Spec
• Representing Variant In Arrow Proposal

[alamb]

Here are some good comments from @scovich related to this

#8021 (comment)

Ah, I think I see -- for a strongly-typed builder, the caller could do all the pathing and just pass in the correct leaf column here. But this is variant, so we'd actually need to extract the variant leaf value (using a path) on a per-row basis. We'd either have to do all the pathing here (recursively), or caller would have to to extract the struct/array once and recurse on its fields. And the latter would require a row-oriented builder where this builder seems to be column-oriented.

#8021 (comment)

Currently, the output builder seems to be fully column-oriented -- it assumes that all values for each leaf column are extracted in a tight loop. This can work for primitive builders, but nested builders will quickly run into pathing and efficiency problems.

I think we'll need to do something similar to the JSON builder, with a row-oriented approach where each level of a nested builder receives an already-constructed Variant for the current row and does a field extract for each child builder; child builders can then cast the result directly or recurse further as needed (based on their own type). And then the top-level builder call would construct a Variant for each row to kick-start the process.

But see the other comment -- to the extend that the shredding aligns nicely, we can hoist a subset of this per-row pathing of the append method up to columnar pathing of the builder's constructor and finalizer.

[alamb]

@scovich since you have thought a lot about this and seem to be the most expert on shredding, perhaps you are interested in taking on this task? I would be very happy to review PRs too

[scovich]

There's definitely interest but I'm less certain about available time, with unrelated things taking a lot of my attention right now. I'll see what I can do, tho. Will check back here before starting so I don't duplicate effort like has happened once or twice in the past.

[alamb]

Thank you

Will check back here before starting so I don't duplicate effort like has happened once or twice in the past.

Yes, this is unfortunate -- I am sorry about that. I am trying to avoid this via tickets, etc but as you see it is not perfect. Checking the existing PRs is probably a good idea too

[scovich]

Thank you

Will check back here before starting so I don't duplicate effort like has happened once or twice in the past.

Yes, this is unfortunate -- I am sorry about that. I am trying to avoid this via tickets, etc but as you see it is not perfect. Checking the existing PRs is probably a good idea too

Last time I spent a couple days working on a ticket but had forgotten to take it... and then pushed my PR just an hour or two after somebody else did take the ticket. It was 100% my bad.

[alamb]

Last time I spent a couple days working on a ticket but had forgotten to take it... and then pushed my PR just an hour or two after somebody else did take the ticket. It was 100% my bad.

Thank you for clarifying -- I know that feeling and it is not great. Thank you again for all the help so far

[carpecodeum]

@alamb @scovich I am up for taking this! Have you had time to think about this?

[scovich]

I have been trying to do some pathfinding over the last few days and almost have something resembling a sketch now. I'm hoping a couple more hours of work will get it to a place where it makes any sense to others.

Note: This is nothing even remotely like merge-ready code. It's more like a back of the envelope sketch that happens to compile. I'll try to post it today as a conversation starter in case it's helpful to moving things along?

[alamb]

Yes that would be super helpful -- I think this ticket is likely one of the trickiest as it needs the right abstractions implemented

I will wait with bated breath for your pathfinding and in the meantime will try and review / merge as much code as possible

[scovich]

Pathfinding PR posted:

• [WIP][Variant] Very rough pathfinding for variant get/shredding #8122

Hopefully the commentary is helpful.