Compatibility with SQLA2.1

[Dev-iL]

This PR is intended to ensure compatibility with SQLA2.1.

The plan is as follows:

• Force installing 2.1 (prerelease if necessary) in the context of the Latest SQLAlchemy test: core and Latest SQLAlchemy test: providers tasks.
• Fix any incompatibilities / raise bug reports in the appropriate repo(s).
  • Workaround for importing ScalarAttributeImpl on SQLA2.1 kvesteri/sqlalchemy-utils#804
  • Use native SQLA UUID/Uuid instead of sqlalchemy-utils #61532
  • fix: Remove ez_setup for compatibility with setuptools v82 cassandra-python-driver#1268
  • Robustify SQLA version determination logic snowflakedb/snowflake-sqlalchemy#647
• Revert forcing the latest version and keep the compatibility fixes

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[Dev-iL]

According to an old discussion in the SQLA repo,

It seems that sqlalchemy_utils is using an internal api in an unsupported way.

In SQLA2.1 this api was finally marked as private and sqlalchemy-utils broke . I believe these are the options we have:

1. Move away from sqlalchmy-utils.
2. Get sqlalchmy-utils fixed.
3. Restrict the SQLA version to <2.1 until one of the above happens.

Thoughts?

[potiuk]

Thanks for runnig it @Dev-iL -> really cool you are doing it !

Move away from sqlalchmy-utils.

We only seem to use sqlalchmy_utils for UUIDtype

from sqlalchemy_utils import UUIDType

I guess in sqlalchemy 2.+ there should be already an alternative type we can use ?

https://docs.sqlalchemy.org/en/21/core/custom_types.html#backend-agnostic-guid-type

Since version 2.0 the built-in Uuid type that behaves similarly should be preferred. This example is presented just as an example of a type decorator that receives and returns python objects.

Not sure about compatibility, but I would imagine it should be quite easy to replace it

[Dev-iL]

@potiuk Thanks for the tip! I think we might be able to remove both of these dependencies: sqlalchemy-jsonfield and sqlalchemy-utils.

[Dev-iL]

@jscheffl Do you see any issue with modifying the old migrations so we can move to new ORM type hints? The DB should stay the same (if I did everything right).

The SQLA docs mention that:

To have the Uuid datatype work with string-based Uuids (e.g. 32 character hexadecimal strings), pass the Uuid.as_uuid parameter with the value False.

So here's what I think:

• Step 1: Whenever a uuid is treated as a string we should have sa.Uuid(as_uuid=False) in old migrations.
• Step 2: Create a new migration that converts all of these fields into "native" Uuids (i.e. sa.Uuid(as_uuid=True) and update the ORM accordingly.

[jscheffl]

@jscheffl Do you see any issue with modifying the old migrations so we can move to new ORM type hints? The DB should stay the same (if I did everything right).

I do not see an issue in this except (1) testing and (2) review. But all looks like simple subsitutes. So OK for me.

[Dev-iL]

I do not see an issue in this except (1) testing and (2) review.

How to test this besides adding the full tests needed tag?

[jscheffl]

I do not see an issue in this except (1) testing and (2) review.

How to test this besides adding the full tests needed tag?

Mhm, yeah taking a Airflow 2.7.0 setup and run the migration :-) Such integrative tests are not in CI in my view.

[Dev-iL]

Mhm, yeah taking a Airflow 2.7.0 setup and run the migration :-) Such integrative tests are not in CI in my view.

Isn't that what the migration tests (e.g. this) do?

[jscheffl]

Mhm, yeah taking a Airflow 2.7.0 setup and run the migration :-) Such integrative tests are not in CI in my view.

Isn't that what the migration tests (e.g. this) do?

Ah... yeah ... 🤦

[Dev-iL]

This addresses the issue with the snowflake provider.