Add qgis-deb.sh to mirror QGIS apt repositories via apt-sync.py

[yaoge123]

Summary

Add qgis-deb.sh to mirror the QGIS apt repositories (https://qgis.org/{debian,debian-ltr,ubuntugis,ubuntugis-ltr}, with ubuntu/ubuntu-ltr served as symlinks) using the existing apt-sync.py helper — which this PR leaves byte-identical to master.

Why

QGIS publishes sibling apt trees under qgis.org. Hand-rolling one job per repo with bespoke options is fragile when codenames change (Debian/Ubuntu rotates ~yearly). This single shell driver loops over the trees with shared codename/architecture lists.

How it works

• sync_repo() runs apt-sync.py --delete per tree. Since apt-sync.py always exits 0 (it only logs failures), the wrapper detects failures itself: it tees the output and fails the run when the Failed APT repos log line appears or the process exits nonzero.
• DEB_CODENAMES deliberately mixes Debian and Ubuntu codenames: the qgis.org S3 backend serves the union of both families under each tree (verified live 2026-09-25: /debian/dists/jammy and /ubuntu/dists/bookworm both return 200), and apt-sync.py --delete removes any on-disk .deb not referenced by the synced codenames — splitting the list per tree would delete still-published content. sid/unstable are included; stale/EOL codenames upstream still serves (buster, kinetic, lunar, mantic, oracular) are intentionally not mirrored (documented in the script comment).
• Retired distributions are probed explicitly: before each repo sync, every codename's Release file is HEAD-probed (the S3 backend answers 403/404 for missing keys). A gone codename is skipped; if it was mirrored before, its local dists tree is removed first so --delete can garbage-collect its packages and the mirror never serves a half-retired distribution. Probe errors other than 403/404 keep the codename (transient failures cannot remove content); if no codename is available at all, the script refuses to sync an empty tree.
• ubuntu → debian and ubuntu-ltr → debian-ltr are symlinks: the upstream trees are byte-identical (Release sha256 matched for every codename, verified 2026-09-25), avoiding a second ~50G copy. ln -sfnT treats the destination as the link itself, and the script refuses to replace a pre-existing real directory.
• REPO_SIZE_FILE comes from mktemp; helpers/size-sum.sh --rm is the single cleanup point (no EXIT trap) and stays non-fatal with a logged WARNING — a failed size report must not fail the sync.

Testing

Verified on the host against the live upstream: all 13 debian + 5 ubuntugis production codenames probe 200; the ubuntu/debian Release files are sha256-identical per codename.

Deployment note (one-time migration)

On a deployment that already has a real ubuntu/ (or ubuntu-ltr/) directory from an older mirror layout, the next sync stops with an actionable error (remove it before creating the symlink) instead of nesting into it. This is expected: remove the duplicate ubuntu copy tree manually once, and the symlink is created on the following run. Fresh deployments are unaffected.

Notes

• Codename / arch lists are two top-of-file variables (DEB_CODENAMES/UBUNTUGIS_CODENAMES, arches alongside) and can be updated in one place.
• Calls existing helpers/size-sum.sh for tunasync size reporting.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new Bash script to sync QGIS APT repositories (Debian/Ubuntu + ubuntugis) via apt-sync.py, including a symlink for ubuntu-ltr and repository size summarization.

Changes:

• Introduces qgis-deb.sh to mirror multiple QGIS repo trees with configured codename/arch lists.
• Adds a ubuntu-ltr -> debian-ltr symlink step.
• Generates a repo size summary file and attempts cleanup at the end.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yaoge123]

Copilot review addressed:

1. _here unused — fixed by using it for the apt-sync.py and helpers/size-sum.sh paths, matching chef.sh/cvmfs.sh/bazel-apt.sh convention (a1c2452).
2. Unset TUNASYNC_WORKING_DIR → root-level paths — fixed: the script now fails fast with a clear error when it is unset/empty (af84c33).
3. Fixed /tmp REPO_SIZE_FILE — fixed: mktemp + trap cleanup (e0e0dcc).
4. DEB_CODENAMES mixing Debian and Ubuntu codenames — intentionally not split. Verified 2026-09-25 against the live upstream: the qgis.org backend serves the union of both codename families under each tree (/debian/dists/jammy/Release and /ubuntu/dists/bookworm/Release both return 200), and apt-sync.py --delete removes every on-disk .deb not referenced by the codenames synced in that run — splitting per tree would delete content that is still published. Codenames absent upstream are skipped with a warning. An explanatory comment is now in the script (af84c33).
5. Hard-coded helper paths / || true — paths are now _here-relative; the size aggregation stays non-fatal by design (a failed size report must not fail an otherwise successful sync) but now logs a warning instead of failing silently (af84c33).

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved failures in sync status, stale distributions, symlink handling, and size reporting can produce incomplete or misleading mirror results.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 1

Open (2)

• Mirror failures are masked by a zero exit status · New
• Symlink creation nests inside an existing destination directory · New

Resolved since last review (11)

• If TUNASYNC_WORKING_DIR is unset/empty, the destination paths become \"/debian\",…
• If TUNASYNC_WORKING_DIR is unset/empty, the destination paths become \"/debian\",…
• If TUNASYNC_WORKING_DIR is unset/empty, the destination paths become \"/debian\",…
• With set -e, || true suppresses all failures from size-sum.sh, which can hide real problems…
• Creating a predictable-ish filename in /tmp can allow symlink/hardlink attacks or accidental…
• Creating a predictable-ish filename in /tmp can allow symlink/hardlink attacks or accidental…
• DEB_CODENAMES includes both Debian (e.g., bullseye/bookworm/trixie) and Ubuntu (e.g.,…
• DEB_CODENAMES includes both Debian (e.g., bullseye/bookworm/trixie) and Ubuntu (e.g.,…
• The script relies on hard-coded absolute paths for apt-sync.py and size-sum.sh, reducing…
• The script relies on hard-coded absolute paths for apt-sync.py and size-sum.sh, reducing…
• _here is defined but not used anywhere in this script. Please remove it, or use it to locate…

[yaoge123]

Resolved differently in f07b2c6: apt-sync.py is reverted to master (its exit-code change affected all 23 callers), and the wrapper now detects failures itself by teeing the apt-sync output and failing on the 'Failed APT repos' log line or a nonzero exit.

[happyaron]

Thanks for the thorough follow-up on the codename question — I re-checked it against the live upstream and the union claim holds: every codename in DEB_CODENAMES returns 200 under all of debian, ubuntu, debian-ltr and ubuntu-ltr. The ubuntu-ltr → debian-ltr symlink is also justified (identical Release for bookworm/trixie/jammy/noble/resolute).

Inline comments below. One non-line note: several earlier thread replies no longer match the final code (e.g. the TUNASYNC_WORKING_DIR replies say the check was not added, and the size-sum reply says the fallback was reverted, but both are now present). Could you update or resolve those threads so they don't mislead later readers?

[happyaron]

This changes behaviour for every caller of apt-sync.py, not just QGIS. 23 shell scripts in the repo call it under set -e, and adoptium.py runs it with check=True. Today a partial failure exits 0 and those scripts continue; after this change they abort at the first failed repo (e.g. proxmox.sh would skip the ISO/images sync whenever one apt repo fails).

Reporting failures to tunasync is probably the right direction, but since it is a repo-wide behaviour change, could it be split into its own PR so maintainers can evaluate it separately from adding the QGIS mirror?

[yaoge123]

Reverted in f07b2c6: apt-sync.py is byte-identical to master again and keeps exiting 0, so none of the 23 callers (or adoptium.py's check=True) change behaviour. qgis-deb.sh now detects failures itself: it tees the apt-sync.py output and fails the run if the 'Failed APT repos' log line appears (or the process exits nonzero).

[happyaron]

debian and ubuntu appear to be identical trees upstream, just like debian-ltr/ubuntu-ltr. I compared https://qgis.org/{debian,ubuntu}/dists/<c>/Release for bookworm, noble, sid, unstable, oracular, mantic and buster — the sha256 matched in every case.

If that holds, ubuntu could be a symlink to debian too (same treatment as ubuntu-ltr below), which would avoid storing a second full copy.

[yaoge123]

Done in f07b2c6: ubuntu is now a symlink to debian with the same treatment as ubuntu-ltr (ln -sfnT, and the script refuses to replace a pre-existing real directory). Note this change does not delete the already-mirrored ubuntu tree anywhere: on a host that still has a real ubuntu/ directory the job stops with an actionable error until the operator removes the duplicate tree.

[happyaron]

Upstream publishes more codenames than this list covers. sid/unstable are actively updated (Release dated 2026-08-29), and oracular, mantic, lunar, kinetic and buster also return 200 (stale, though). By the same reasoning as the comment above ("splitting would lose coverage"), these are not mirrored.

Could you either add at least sid, or note in the comment that the omissions are deliberate?

[yaoge123]

Added sid and unstable in f07b2c6 (both verified live, Release 200 and recently updated). The comment now also records that the remaining codenames upstream still serves but are stale/EOL (buster, kinetic, lunar, mantic, oracular) are intentionally not mirrored.

[happyaron]

Combined with the apt-sync.py exit-code change and set -e, one failed repo aborts the remaining ones. This will also become permanent once upstream drops a codename we have already synced: apt_mirror only ignores a missing Release if it never existed locally (apt-sync.py:174-178), otherwise it returns 1 on every run. Several listed codenames are already EOL (xenial, bionic, focal, bullseye, plucky).

Consider running all five syncs and exiting non-zero at the end if any failed, so a single broken repo doesn't stall the rest.

[yaoge123]

Fixed in f07b2c6 without touching apt-sync.py: before each repo sync, qgis-deb.sh probes every codename's Release (the S3 backend answers 403/404 for missing keys). Absent codenames are skipped; if one was mirrored before, its local dists tree is removed up front so --delete can garbage-collect its packages and the mirror never serves a half-retired distribution. Probe errors other than 403/404 keep the codename in the list, so a transient failure cannot remove content.

[happyaron]

Nit: size-sum.sh ... --rm already deletes this file on the success path, so the trap and --rm overlap. Harmless — keeping one of them would be enough.

[yaoge123]

Fixed in f07b2c6: the EXIT trap is dropped; size-sum.sh --rm is now the single cleanup point (on failure paths the container's /tmp is ephemeral anyway).

[Copilot]

Copilot review overview

🟡 Changes recommended

Two unresolved moderate issues affect failure reporting and handling of retired distributions.

Review effort: Lite
Findings: 1 · 1

Open (2)

• Mirror failures are masked by a zero exit status
• Retired distributions abort sync instead of being handled · New

Resolved since last review (1)

• Symlink creation nests inside an existing destination directory

[yaoge123]

Fixed in f07b2c6: retired distributions are now handled explicitly inside qgis-deb.sh (apt-sync.py is unchanged). Each codename's Release is probed before syncing; a 403/404 codename is skipped, and if it exists locally its dists tree is removed first so --delete garbage-collects its packages consistently. A codename whose probe fails for any other reason stays in the sync list, so transient errors cannot delete content.

[yaoge123]

Branch cleanup note: this branch has been squashed to a single commit, 2bc1f64. All commit SHAs referenced earlier in the review threads (up to f07b2c6) are now orphaned commits — the links still open, but please rely on the current diff, whose tree is byte-identical to the previous head f07b2c6.

Review items → how they were addressed (all included in the current diff)

Maintainer review (@happyaron):

• apt-sync.py exit-code change affects all 23 callers (+ adoptium.py's check=True) → reverted: apt-sync.py is byte-identical to master and keeps exiting 0. qgis-deb.sh detects failures itself instead: it tees the apt-sync.py output and fails the run when the Failed APT repos log line appears or the process exits nonzero. (If repo-wide failure propagation is wanted, it can be evaluated as its own PR.)
• ubuntu could be a symlink to debian too → done: both ubuntu → debian and ubuntu-ltr → debian-ltr are now symlinks (ln -sfnT), justified by byte-identical upstream trees (Release sha256 matched for every codename). Deployment note: this does not delete any already-mirrored ubuntu tree — on a host that still has a real ubuntu/ directory the job stops with an actionable error until the operator removes the duplicate tree; the symlink is then created on the next run. Expected one-time migration.
• sid/unstable actively updated but not mirrored; EOL codenames undocumented → sid and unstable added (verified live: 200 and recently updated); the script comment now records that the remaining upstream-served but stale/EOL codenames (buster, kinetic, lunar, mantic, oracular) are intentionally not mirrored.
• One failed repo would abort the remaining ones, permanently once upstream drops a synced codename → fixed without touching apt-sync.py: every codename's Release is probed before syncing (S3 answers 403/404 for missing keys); a gone codename is skipped, and if mirrored before, its local dists tree is removed up front so --delete garbage-collects its packages and the mirror never serves a half-retired distribution. Other probe errors keep the codename, so transient failures cannot remove content.
• Trap and --rm overlap (nit) → the EXIT trap is dropped; size-sum.sh --rm is the single cleanup point.

Copilot rounds:

• Round 1: unused _here → now used for the apt-sync.py/size-sum.sh paths; unset TUNASYNC_WORKING_DIR → root-level paths → fail-fast check added; fixed /tmp REPO_SIZE_FILE → mktemp; mixed codename lists → intentionally kept (union verified live; splitting would delete published content), documented in the script; || true on size-sum → non-fatal by design, now logs a WARNING; hard-coded helper paths → _here-relative.
• Round 2: relying on apt-sync.py's exit code for failure detection → wrapper-side tee+grep detection (above); ln -sfn nesting into a pre-existing directory → -sfnT plus an explicit refusal to replace a real directory.
• Round 3: "skip absent upstream" unsafe for already-mirrored distributions → explicit probe-and-remove handling (above).

The stale thread replies you flagged (the TUNASYNC_WORKING_DIR check and the size-sum fallback) have been corrected in place in those threads.

Some earlier in-thread replies described intermediate states of the branch; please rely on the current diff and this summary.