rfc: Making Storage a Trait #1885

CTTY · 2025-11-25T00:05:33Z

Which issue does this PR close?

Related to
- Make FileIO a Trait #1314
- Consider Using object_store as IO Abstraction #172

What changes are included in this PR?

Are these changes tested?

CTTY · 2025-11-25T00:09:01Z

docs/rfcs/0002_storage_trait.md

+
+    // Deletion operations
+    async fn delete(&self, path: &str) -> Result<()>;
+    async fn remove_dir_all(&self, path: &str) -> Result<()>;


Migrating comments from @Fokko

This name feels very much file-system like, while Iceberg is designed to work against object stores. How about delete_prefix?

CTTY · 2025-11-25T00:10:40Z

docs/rfcs/0002_storage_trait.md

+#[async_trait]
+pub trait Storage: Debug + Send + Sync {
+    // File existence and metadata
+    async fn exists(&self, path: &str) -> Result<bool>;


Migrating comments from @c-thiel

I know that we use these results everywhere, but I think introducing more specific error types that we can match on for storage operations makes sense. They can implement Into of course.
For example, a RateLimited error that we got from the storage service should be treated differently from NotFound or CredentialsExpired.
With Lakekeeper we are currently using our own trait based IO due to many limitations in iceberg-rust, mainly due to unsupported signing mechanisms, missing refresh mechanisms, intransparent errors and missing extendability.
I would gladly switch to iceberg-rust if we get these solved.
Maybe this can serve as some inspiration: https://github.com/lakekeeper/lakekeeper/blob/b8fcf54c627d48a547ef0baf6863949b68579388/crates/io/src/error.rs#L291

To address @c-thiel 's comments, we have several approaches:

Introduce another set of errors for storage.

Extend current ErrorKind for storage errors.

Extend current ErrorKind, but with another enum, for example

pub enum IoErrorKind { FileNotFound, CredentialExpired, } pub enum ErrorKind { // Existing variants ... Io(IoErrorKind) }

CTTY · 2025-11-25T00:11:48Z

docs/rfcs/0002_storage_trait.md

+    // File object creation
+    fn new_input(&self, path: &str) -> Result<InputFile>;
+    fn new_output(&self, path: &str) -> Result<OutputFile>;
+}


Migrating comments from @c-thiel

Many object stores have a good way of running batch deletions, for example the DeleteObjects API in AWS S3. How would you feel about including a delete_batch method too?

liurenjie1024

Thanks @CTTY for this pr, generally LGTM! One missing point is, I want the StorageBuilderRegistry to have some built in StorageBuilder registered when user creating a new catalog instance. I currenlty don't have a good solution, one approach would be to have a standalone crate, which loads built in StorageBuilders when StorageBuilderRegistry is initiated. And then we could have catalog crates to depend on it.

liurenjie1024 · 2025-11-25T11:06:42Z

docs/rfcs/0002_storage_trait.md

+#[async_trait]
+pub trait Storage: Debug + Send + Sync {
+    // File existence and metadata
+    async fn exists(&self, path: &str) -> Result<bool>;


To address @c-thiel 's comments, we have several approaches:

Introduce another set of errors for storage.

Extend current ErrorKind for storage errors.

Extend current ErrorKind, but with another enum, for example

pub enum IoErrorKind { FileNotFound, CredentialExpired, } pub enum ErrorKind { // Existing variants ... Io(IoErrorKind) }

liurenjie1024 · 2025-11-25T11:07:10Z

docs/rfcs/0002_storage_trait.md

+#[async_trait]
+pub trait Storage: Debug + Send + Sync {
+    // File existence and metadata
+    async fn exists(&self, path: &str) -> Result<bool>;


Suggested change

async fn exists(&self, path: &str) -> Result<bool>;

async fn exists(&self, path: AsRef<str>) -> Result<bool>;

A little more rusty.

I think this breaks object dyn compatibility

liurenjie1024 · 2025-11-25T11:07:44Z

docs/rfcs/0002_storage_trait.md

+
+    // Deletion operations
+    async fn delete(&self, path: &str) -> Result<()>;
+    async fn remove_dir_all(&self, path: &str) -> Result<()>;


liurenjie1024 · 2025-11-25T11:09:12Z

docs/rfcs/0002_storage_trait.md

+The `StorageBuilder` trait defines how storage backends are constructed:
+
+```rust
+pub trait StorageBuilder: Debug + Send + Sync {


Suggested change

pub trait StorageBuilder: Debug + Send + Sync {

pub trait StorageFactory: Debug + Send + Sync {

nit: Factory sounds a litte better.

It should also be Serializable

+1, I haven't figured out how to have both Serializable and dyn Trait yet, and I'm planning to explore more with typetag. Serializability is a huge pain and I'll add more details once I have more clarity

liurenjie1024 · 2025-11-25T11:10:25Z

docs/rfcs/0002_storage_trait.md

+    pub fn new() -> Self { /* ... */ }
+    pub fn register(&mut self, scheme: impl Into<String>, builder: Arc<dyn StorageBuilder>);
+    pub fn get_builder(&self, scheme: &str) -> Result<Arc<dyn StorageBuilder>>;
+    pub fn supported_types(&self) -> Vec<String>;


Suggested change

pub fn supported_types(&self) -> Vec<String>;

pub fn supported_types(&self) -> impl Iterator<Item=&str>>;

liurenjie1024 · 2025-11-25T11:12:29Z

docs/rfcs/0002_storage_trait.md

+use iceberg::io::FileIOBuilder;
+
+// Basic usage (same as the existing code)
+let file_io = FileIOBuilder::new("s3")


We no longer need FileIOBuilder?

I agree that FileIOBuilder seems excessive. I'm keeping FileIOBuilder here for now mainly because I'm uncertain where to keep Extensions right now mainly due to serde-related concerns

After StorageBuilder becomes serializable, we should be able to remove it.

liurenjie1024 · 2025-11-25T11:14:45Z

docs/rfcs/0002_storage_trait.md

+    ↓
+FileIOBuilder::build()
+    ↓
+StorageBuilderRegistry::new()


I think we should remove FileIOBuilder since then?

The StorageBuilderRegistery should be an instance in catalog instance?

Co-authored-by: Renjie Liu <[email protected]>

This reverts commit e7d67d8.

CTTY added 2 commits November 24, 2025 16:03

rfc: Making Storage a trait

0506c1e

add license header

16bea19

CTTY commented Nov 25, 2025

View reviewed changes

CTTY changed the title ~~rfc: Making Storage a trait~~ rfc: Making Storage a Trait Nov 25, 2025

liurenjie1024 reviewed Nov 25, 2025

View reviewed changes

CTTY and others added 3 commits November 25, 2025 10:02

Update docs/rfcs/0002_storage_trait.md

e7d67d8

Co-authored-by: Renjie Liu <[email protected]>

Merge branch 'main' into ctty/storage-rfc

8253d4f

Revert "Update docs/rfcs/0002_storage_trait.md"

fe59361

This reverts commit e7d67d8.

	async fn exists(&self, path: &str) -> Result<bool>;
	async fn exists(&self, path: AsRef<str>) -> Result<bool>;

	pub trait StorageBuilder: Debug + Send + Sync {
	pub trait StorageFactory: Debug + Send + Sync {

	pub fn supported_types(&self) -> Vec<String>;
	pub fn supported_types(&self) -> impl Iterator<Item=&str>>;

rfc: Making Storage a Trait #1885

Are you sure you want to change the base?

rfc: Making Storage a Trait #1885

Uh oh!

Conversation

CTTY commented Nov 25, 2025

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CTTY Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CTTY Nov 25, 2025 •

edited

Loading