Support AVRO Format for Write Queries

[devinjdangelo]

Is your feature request related to a problem or challenge?

Writing Avro files is currently not supported in DataFusion, such as via COPYor INSERT INTO queries. This would be a nice feature to have and recently came up in the discord channel.

Describe the solution you'd like

• Implement create_write_physical_plan for AvroFormat
• Implement FileSink on a AvroSink struct
• Implement file writing options / string parsing in FileTypeWriterOptions for avro

I expect some of the common functions in https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/file_format/write.rs can be reused in the avro implementation.

Describe alternatives you've considered

Don't support writing avro files

Additional context

No response

[Veeupup]

@alamb Hi, I can help with this ticket : )

[alamb]

@alamb Hi, I can help with this ticket : )

Thanks @Veeupup -- for this one the first thing we need to do is get an avro writer and the second part is hooking it into DataFusion. There is some work upstream in arrow-rs to make an avro reader/writer in apache/arrow-rs#4886 but I think that project is stilled at the moment

[jecsand838]

@alamb @devinjdangelo

Since apache/arrow-rs#4886 is now closed and arrow-avro is set for release with arrow-rs v57.0.0 this month, we could probably resume progress here.

I wouldn't mind taking this and having it ready for the DataFusion v51 release.

[alamb]

In case you want to start early, I have a PR that integrates with the latest arrow main that you could use until we actually release arrow 57 (likely will be late next week)

• Upgrade DataFusion to arrow/parquet 57.0.0 #17888

[jecsand838]

take