Use trusted publishing

[kohtala]

PyPi releases were not those built in this repository. For better supply chain traceability, use trusted publishing that creates the attestations required.

This also moves to use GitHub hosted runners for ubuntu ARM64 and MacOS intel and M1 builds.

Fixes #1191.

This requires setup of

• an environment for each project on PyPi on GitHub and
• a Trusted Publisher configuration for each project on PyPi to accept the release.yml workflow with the environment to publish on the project

[EpicWink]

Duplicate of #1169, plus other changes (added workflows, new Dependabot config, updated Action versions, pinned container image hashes, newer/different job runners)

[kohtala]

Duplicate of #1169, plus other changes (added workflows, new Dependabot config, updated Action versions, pinned container image hashes, newer/different job runners)

It's a draft for a purpose. Can't be merged and is not asking to. Feel free to take any ideas for improvements, that is what I made it available here for.

I am solving a bit different problem from yours. My problem comes from the #1191 issue of having wrong openssl in the published wheels and the failure to attest the release artifacts. Solution to that needs to pin the container images for us to know what version openssl is included and works towards being able to tell where it came from. The runners also are for that same purpose since we don't know what is on those self-hosted runners. They were most likely created at a time when GitHub didn't yet offer their runners, so now could be a good time to move forward from them.

The Trusted Publishing creates the attestation link from artifacts on PyPi back to the source and build.

This PR is still missing the build of the manylinux images. The pinning should perhaps change into verifying attestations on the manylinux images used in the build. I might also consider dropping the self-built manylinux images altogether and rather have workflows for each of the dependencies and instead cache them for faster builds of the wheels.

But this PR is really to tell what are my ideas for how to better secure the supply chain to prove what we have on PyPi. For someone like you working on contributing to the project

The upgraded deprecated action versions, dependabot.yml as well as some workflow syntax updates are cleanup that I rather take on my own fork for testing. You are right it is not related to the problem being solved and I'd clean it up for a PR that I'd offer for a merge. At least I should make it a separate commit for clarity. Depending on open-source project I contribute to, some are happy to receive additional cleanup in the same PR rather than go through managing additional PR for trivial things. It also saves my time.

Compared to #1265 that combined all builds and releases into one large workflow, I think it's more readable and maintainable to have separate workflows for builds and releases. The suggested approach on this draft PR is to consolidate only the releases in separate workflow from the builds and checks. The release and scheduled workflows that publish on PyPi are on this single workflow to reduce the amount of needed trusted publishing configuration (problem mentioned in #1169 (comment)). Artifact naming is changed such that a single job can download only necessary artifacts and can attest and upload all files for a version in a PyPi project in a single action.

The hierarchy of reused workflows also looks nice on the GitHub build UI.