Spec: Forbid writing new equality deletes in v4 - #17783
Conversation
github-actions
Bot
added
the
Specification
|
|
||
| Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row. | ||
|
|
||
| Equality delete files must not be added to v4 tables. Equality deletes carried over from v2 or v3 remain in the delete manifests that already track them and readers must still apply them. Converting them to [deletion vectors](#deletion-vectors) is a separate, optional maintenance action. |
There was a problem hiding this comment.
| Equality delete files must not be added to v4 tables. Equality deletes carried over from v2 or v3 remain in the delete manifests that already track them and readers must still apply them. Converting them to [deletion vectors](#deletion-vectors) is a separate, optional maintenance action. | |
| Equality delete files must not be added to v4 tables. Equality deletes files referenced by metadata upgraded to v4 must still be applied. Moving entries covered by an equality delete to a v4 manifest requires applying the deletes and writing a new data file entry. |
I don't think we want to reference/rely on maintenance actions as the spec should have a full path to operate without their use. We don't reference maintenance anywhere else in the spec.
We should include something what's required to relocate entries.
There was a problem hiding this comment.
Applied suggested changes. Thanks.
| * Readers must continue to apply equality deletes for v2 and v3 tables and for equality deletes carried over into upgraded v4 tables | ||
| * Upgrading a v2 or v3 table to v4 is metadata-only and does not rewrite data or delete files | ||
| * Equality deletes remain in the carried-over v2 or v3 delete manifests that already track them | ||
| * Converting equality deletes to DVs is a separate, optional maintenance action |
There was a problem hiding this comment.
I don't think we need these two sub-bullets. The first restates the main bullet and I don't think we want to reference the maintenance actions from the sepc.
| * Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap | ||
| * Position delete files identify deleted rows by file location and row position (**deprecated** in v3) | ||
| * Equality delete files identify deleted rows by the value of one or more columns | ||
| * Equality delete files identify deleted rows by the value of one or more columns (**deprecated** in v4) |
There was a problem hiding this comment.
| * Equality delete files identify deleted rows by the value of one or more columns (**deprecated** in v4) | |
| * Equality delete files identify deleted rows by the value of one or more columns (**prohibited** in v4) |
A bit of a nit (and fully aware it's used directly above in relation to row-level deletes): deprecated means "discouraged" and in this context we're actually prohibiting use. This is a common misuse, but I rather be explicit.
|
We talk a little bit about V2/V3 equality deletes here but I think we should probably be a bit more explicit. "Equality deletes referenced in V2 or V3 delete manifests still apply in a V4 Table. New V3 and V2 manifests may not be written into a V4 Table and V4 Manifests cannot refer to equality deletes." Maybe it's just me though? |
4 participants
This PR updates the table spec for v4 equality deletes. V4 writers must not write new equality deletes, but readers must still apply existing ones from v2/v3 tables.