[Spec] Linking Schema ID to Data & Delete Files

[manirajv06]

Proposed Change

Schema evolve over time and data files could have different columns at different point of time. It is quite natural that data files created at T1 with Schema S1 could have columns C1 to C5, data files created at T2 with Schema S1 could have columns C4 to C10 and so on..

Linking Schema ID with data files would be handy to extract any Schema details easily. For an instance, Files could be filtered based on whether column exists or not using its field id by comparing with file's max field id. Max field id of the file is nothing but the max field id of the linked schema. Schema's Max field id is already available and can be used straight away. C5 is the max field id for all files linked to S1. C10 is the max field id for all files linked to S2. Another instance, to know whether Parquet files has UnknownType type or not, all files needs to be opened as there is no statistics or other way to know it. Linking schema's to these files could pull those info very easily. Similarly, other schema info can be used based on the requirements.

I would like to propose that linking the schema id with files would be useful in carrying out files and schema related operations going forward.

Proposal document

No response

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other

[manirajv06]

@Fokko @amogh-jahagirdar @RussellSpitzer @rdblue Can you please take a look and share your views?

[RussellSpitzer]

I probably would not link schema ID because that alone would not indicate the presence of a field (Optional Fields) but we probably should have some way in the metrics of determining the difference between a "missing metric" and a missing field. Maybe we should be storing "columns written" as a metric for file so that the planner can avoid a file. This would require adding a set to each datafile entry but it should compress very well since it should be mostly identical between all data files in a manifest.

If the goal is determining which files have which columns we should probably start by with that problem statement before jumping to linking to schema.

[manirajv06]

@RussellSpitzer Thanks for sharing your views.

Yes, problem statement is to know the list of columns in each file. I can also understand the problems with optional fields. In case of null values for all records in data file, we might get mis leaded if we go by linked schema id. However, max field id of linked schema id could help us in making lenient (with few % error rate) decisions which might not be correct for the places where decisions have to be made strictly.

"columns written" as a metric for file helps in making accurate decisions and suits for all places. It can be added easily as we must be having all the info in our hand but comes up with a cost of keeping the list for all files. If we think this cost is not really a overhead to worry about it, then we can go with this approach.

@Fokko @amogh-jahagirdar @rdblue Please share your views as well so that we can move head with making changes.

[amogh-jahagirdar]

I think I agree with @RussellSpitzer . The core of the issue is a lack of file level information on which field IDs are actually written to a particular file. We'll also need to make sure from an implementation perspective whatever optimizations we do with this work with any default values that may be defined. If we take Parquet manifests, I do think storing a list of field IDs per file, should compress pretty well since you'd have repeated runs of the same list of field IDs in a manifest (I think in the spec we'd have to say that they are sorted)

cc @singhpk234 because I remember talking to him about this a while back.

@manirajv06 There are ongoing discussions around some aspects of the Iceberg V4 table format and since what's being proposed here is essentially a spec change that would only be able to be applied in the next table format version, I encourage also sending an email on the Iceberg dev list.

[manirajv06]

@RussellSpitzer @amogh-jahagirdar Sent an email to the dev mailing list to discuss the same. In the meantime, Can I work start working on POC to demonstrate the changes?

[emkornfield]

Could we dive into these use-cases and how often they ocurr. For UnknownType columns why do we care about these specifically at the metadata level (we really just care that they are all null?)

What use-cases have high enough schema churn that we expect to get a large performance boost here?

I probably would not link schema ID because that alone would not indicate the presence of a field (Optional Fields) but we probably should have some way in the metrics of determining the difference between a "missing metric" and a missing field.

In terms of approach, I think linking schema ID probably solves 90 + % of the use case defined. If a file is written with an older schema we can definitely say the new column is not present. But given writers are expected to write all columns the only case where a column is missing would be for direct file imports (in this case we'd have to at least open the file to collect the metrics which might not be cheap). If presence/absence is very important to certain writers then ensure we collect Value Count/Null count as statistics is an alternative that already exists for the filtering.

If we want to optimize the last 10% of the use case, I'd say it would like be better to link two columns:

1. Schema ID.
2. Files not present in the schema in the file.

Overall, I expect 2 to be empty most of the time, and either way scales with the exception case, rather then schema size.

[singhpk234]

I think this would be a good addition, thanks for brining this up !

can check it out, there had been previous work on this area

• [CORE] Support file filtering based on schema #4842
• API, Core: Add schema_id to ContentFile #4898

[manirajv06]

Thanks @emkornfield

What use-cases have high enough schema churn that we expect to get a large performance boost here?

Even if schema churn is less or schema changes doesn't happen very often for the most of the use cases, Skipping significant number of files from being processed for the tables having huge number of records boost the performance significantly. Isn't it?

I think linking schema ID probably solves 90 + % of the use case defined.

As explained earlier, it might mislead us when using optional fields. Referring earlier response

"However, max field id of linked schema id could help us in making lenient (with few % error rate) decisions which might not be correct for the places where decisions have to be made strictly."

May be, Am I missing anything here?

If we want to optimize the last 10% of the use case, I'd say it would like be better to link two columns:

I didn't get this two way linking. Can you please explain this?

can check it out, there had been previous work on this area

Thanks @singhpk234 , Will check it out.

[emkornfield]

As explained earlier, it might mislead us when using optional fields. Referring earlier response

Can you explain how this would be misleading when using optional fields?

"However, max field id of linked schema id could help us in making lenient (with few % error rate) decisions which might not be correct for the places where decisions have to be made strictly."

I'm not sure where places need to be made strictly?

If we want to optimize the last 10% of the use case, I'd say it would like be better to link two columns:

I didn't get this two way linking. Can you please explain this?

This was in response to a suggestion above about tracking "fields written". I don't think it is a good idea to pursue this. Instead I think starting with Schema ID, and then if necessary, we can add a separate field of field "fields not written" if necessary to solve the use-case. However, "field written" and fields not written", already has an overlaps with stats tracked so if it is important we might want to consider alternatives that use existing stats.

In general, I think there are enough threads on in that it might pay to create a google doc to summarize the proposal they can all be tracked in one place.

[RussellSpitzer]

My summary thoughts,

Currently metrics are not good for this because we have almost no way to determine the difference between, didn't store metrics for a column and column wasn't written. In general we assume that if we don't see metrics, that column exists.

Linking Schema has a similar issue, If I don't have metrics for an optional column it could be missing or It could have values, so I can't make the call. It would let me avoid columns that don't exist entirely in the leaked schema so there is a benefit there. I just worry that the more common case is schemas with optional columns (possibly many many optional columns) where we aren't storing metrics.

[emkornfield]

Currently metrics are not good for this because we have almost no way to determine the difference between, didn't store metrics for a column and column wasn't written. In general we assume that if we don't see metrics, that column exists.

I agree with this.

Linking Schema has a similar issue, If I don't have metrics for an optional column it could be missing or It could have values, so I can't make the call.

IIUC, the proposal is not meant to address this. There are two cases:

1. The column did not exist at the time or writing (and therefore we couldn't possibly write statistics for that file) so we need some additional metadata to record this fact (e.g. schema ID).
2. Some writers are not writing out optional columns even if they are in the schema or we want to identify all null values for a column that existed in the schema.

(1) seems reasonable. (2) seems like it can already covered (see below).

I just worry that the more common case is schemas with optional columns (possibly many many optional columns) where we aren't storing metrics.

This is an implementation issue though? It seems it can be mostly solved by adapting metadata writers to write out null-counts when null-count = value count? (It might be a little awkward in V4). But we have a current PR reiterating that all columns must be written even if values are null. Or is the argument here solely that the cost to store the statistics in the current data-structures too expensive?

[manirajv06]

@emkornfield Are we guaranteeing that all optional fields even when it is null would be written in the file going forward based on the spec changes you mentioned?

In case there is no such guarantee, how do we know whether the optional field has been written or not?