Consider using upstream arrow-avro reader

[alamb]

Is your feature request related to a problem or challenge?

Currently DataFusion has its own avro --> arrow implementation in: https://github.com/apache/datafusion/blob/54a5d3fd3f98997f048c2e3e7ba67d7bfe2f124d/datafusion/core/src/datasource/avro_to_arrow/mod.rs

This was added way back in the day by @Igosuki and @jorgecarleitao in

• Avro Table Provider #910

Recently there is work upstream in arrow-rs to add additional avro support:

• Add Avro Support arrow-rs#4886

For example, the recent PR from @jecsand838

• Avro codec enhancements arrow-rs#6965

Describe the solution you'd like

One arrow-avro is more mature, it would be great to consider removing the DataFusion specific implementation and using the upstream arrow-avro one

This would allow DataFusion to take advantage of the upstream work and keep our codebase a bit simpler

Describe alternatives you've considered

No response

Additional context

No response

[getChan]

take

[getChan]

I'm waiting for the arrow-avro PR below as it might include changes to the public API.
apache/arrow-rs#6965
I will resume related work once this PR is merged and arrow-avro is released.

[jecsand838]

@getChan @alamb

I had a quick update I wanted to give.

Once apache/arrow-rs#8293, apache/arrow-rs#8348, and apache/arrow-rs#8349 are reviewed and merged in, The arrow-avro Reader will be functionally ready for use in DataFusion. We are getting extremely close.

Also on the arrow-avro Writer side, we have about 4-5 PRs left as well.

[getChan]

@getChan @alamb

I had a quick update I wanted to give.

Once apache/arrow-rs#8293, apache/arrow-rs#8348, and apache/arrow-rs#8349 are reviewed and merged in, The arrow-avro Reader will be functionally ready for use in DataFusion. We are getting extremely close.

Also on the arrow-avro Writer side, we have about 4-5 PRs left as well.

@jecsand838
Thank you for the update. I'll get started on the work soon. If you'd like to work on this, you can take this issue.

[getChan]

waiting arrow-avro 56.0.0 release.
for use brand-new reader APIs. like ReaderBuilder

[alamb]

FYI @getChan I don't think we are going to release the arrow-avro crate for a while

You can start testing immediately using crates.io patch in Cargo.toml

An example of how to do so is here

• WIP: Update to arrow/parquet 54.2.0 #14628

Something like

[patch.crates-io]
arrow-avro = { git = "https://github.com/apache/arrow-rs.git", rev = "..." }