[Ankitraj-builds]

Part of #4156

Rationale for this change

The format version check for v3-only types only ran in create_table. On a v1 or v2 table, update_schema().add_column(...) accepted timestamp_ns and unknown columns and a non-null default_value, and committed them.

This runs check_format_version_compatibility when a schema is added to the table metadata, and extends it to reject a non-null initial-default before v3, matching Java's Schema.checkCompatibility. Dropping a v3-only column still works, so an invalid table can be repaired (rule 2 of the issue).

This PR only covers schema changes from rule 1. Properties, encryption and the write-side rules are left for follow-up PRs.

Are these changes tested?

Yes. Two new tests in tests/catalog/test_catalog_behaviors.py, run for format versions 1 and 2 on every catalog:

• test_update_schema_rejects_v3_only_types: adding timestamp_ns or unknown raises, and the table schema is unchanged.
• test_update_schema_rejects_initial_default_before_v3: adding a column with a non-null default raises.

All existing tests in tests/table, tests/test_schema.py and tests/catalog/test_catalog_behaviors.py pass.

Are there any user-facing changes?

Yes. Schema updates on v1/v2 tables that add a v3-only type or a non-null initial default now raise ValueError instead of being committed.

[rambleraptor]

A side effect of most new validations is that they'll cause existing tests to fail in unexpected ways.

Could you take a look at the integration test failures?

[Ankitraj-builds]

Hi @rambleraptor , Thanks for checking. The failure was test_initial_default in test_reads.py. Spark upgrades the table to v3, but the test kept using the old tbl object, which still had v2 metadata, so the new check rejected the default value. I added tbl.refresh() after the Spark upgrade and the integration tests pass now.

[rambleraptor]

One more quick change just to make sure that we're doing this check in all of the proper places. Thanks so much for the contribution!

[rambleraptor]

This only does this check for AddSchemaUpdate. I think it's worth adding it to UpdateSchema._commit as well.

[Ankitraj-builds]

Done @rambleraptor, added the check to UpdateSchema._commit as well, so it fails before the updates are sent. I put it inside the “schema changed” branch so a commit with no schema change still works. Also added test_update_schema_commit_checks_format_version, which calls _commit() directly. Thanks for the review!