Add php-sync.py: sync PHP releases from php.net JSON API by yaoge123 · Pull Request #214 · tuna/tunasync-scripts

yaoge123 · 2026-09-23T15:44:03Z

Summary

Add php-sync.py: sync PHP source releases directly from the official php.net JSON API. Stdlib-only Python 3.

Why

The old php rsync upstream (rsync.php.net) was decommissioned, and the second-hand rsync sources mirrors fell back to have disappeared one by one, so PHP release tarballs have been stuck (TUNA's php mirror has been failing too).

How it works

The file list and sha256 hashes come from www.php.net/releases/index.php?json. Every API request carries a unique &_=<epoch> cache-buster: BunnyCDN caches the API for up to 30 days keyed on the full query string, so without it a run can read a stale release list and still report success.
Each supported line is fetched with max=10, so earlier patch releases are mirrored too, not just the latest one per line.
Filenames from the API must be plain basenames — a malformed/compromised response cannot escape TUNASYNC_WORKING_DIR.
Tarballs (tar.gz / tar.bz2 / tar.xz) are verified against the published sha256, downloaded to .tmp and renamed into place.
Existing files are re-hashed before being skipped, so truncated/corrupt leftovers are re-downloaded; when a repair download fails permanently, the known-corrupt local file is removed rather than kept served.
An empty file list from the API fails the run loudly (sys.exit) instead of reporting success on a stale mirror.
.asc signatures (not in the JSON API) are fetched best-effort and never fail the run.
Downloaded files keep the upstream Last-Modified time as mtime, so listings show when a file was released, not when it was synced.
Already-downloaded files are skipped, so reruns are cheap.

Environment variables

Variable	Default	Meaning
`TUNASYNC_WORKING_DIR`	(required)	mirror data directory
`PHP_SYNC_PREFER_IPV6`	off	`1` tries IPv6 first (php.net's BunnyCDN IPv4 is unreachable or slow from some networks); off by default because silently-dropped IPv6 would cost a full timeout per connection
`PHP_SYNC_UA`	`tunasync php-sync`	User-Agent for API and download requests

Testing

Verified against the live API: a remote_filelist() run discovers 120 tarballs across the 8.2/8.3/8.4/8.5 lines. Running in production on mirror.nju.edu.cn, where the php-8.4.26 tarballs and signatures appeared on release day.

Deployment

Stdlib only, runs in the standard tunathu/tunasync-scripts image (needs TUNASYNC_WORKING_DIR). Example job config:

[[mirrors]]
name = "php"
provider = "command"
upstream = "https://www.php.net/distributions/"
command = "/home/tunasync-scripts/php-sync.py"
docker_image = "tunathu/tunasync-scripts:latest"

Copilot

Copilot review overview

🟡 Changes recommended

Existing tarballs are not revalidated, and the API request omits max=1, preventing files from being discovered.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

Validate existing tarball checksum before skipping download · New

What changed in this PR

Adds a standard-library PHP release synchronization script using php.net’s JSON API, SHA-256 verification, retries, IPv6 preference, and best-effort signatures.

Changes:

Discovers supported PHP release lines and archives.
Downloads and atomically verifies tarballs.
Fetches optional .asc signatures.

Reviewed findings include a critical checksum-validation issue for existing files and an API query issue preventing release discovery.

File	Summary
`php-sync.py`	Implements PHP release discovery and synchronization; requires fixes for existing-file validation and API release selection.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

yaoge123 · 2026-09-25T14:30:03Z

Verified: the Copilot review item is already resolved in 9a7da5f — local_ok() hashes existing non-empty tarballs and compares the digest against the sha256 published by the releases API, re-downloading on mismatch. No further changes needed.

Copilot

Copilot review overview

🟡 Changes recommended

Critical issues remain with unsafe path handling and incorrect JSON response iteration, which prevents downloads.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

Iterate version-keyed release response to download source tarballs · New
Reject unsafe filenames before joining with mirror root · New

Resolved since last review (1)

Validate existing tarball checksum before skipping download

+        rel = get_json(f'{API_URL}?json&version={line}')
+        for src in rel.get('source', []):
+            if 'sha256' in src:
+                files[src['filename']] = src['sha256']


yaoge123 · 2026-09-25T16:10:48Z

Round-2 review dispositions:

"Response structure wrong, nothing downloads" — false positive. Re-verified against the live API today:

https://www.php.net/releases/index.php?json&max=1 returns a version-keyed map: top-level keys 3,4,5,7,8, each with supported_versions (e.g. 8 → ['8.2','8.3','8.4','8.5']). Iterating top.values() is correct — rel here is a per-major-version record, not a release.
?json&version=8.4 returns a flat single release record (keys: announcement, date, source, supported_versions, tags, version) whose source list currently contains php-8.4.26.tar.{gz,bz2,xz} with sha256 values. rel.get('source') is correct.
End-to-end proof: php 8.4.26 was released on 24 Sep 2026 and all three tarballs (+ .asc) are present on mirror.nju.edu.cn's storage (/data/mirrors/php/php-8.4.26.tar.*, synced 25 Sep), i.e. the script downloads as intended.

"Path traversal via src['filename']" — valid, fixed in d035fc4. Filenames from the API are now accepted only if they are plain basenames; any name containing / or .. is rejected with a warning before it can be used to build local paths under WORKING_DIR.

happyaron

Thanks for picking this up. The php mirror has needed a replacement for rsync.php.net. Two main issues: the releases API is cached by BunnyCDN for 30 days, so new releases can be missed while runs still report success (comment 1), and without max only the latest patch release of each line is mirrored (comment 2). The rest are smaller. Checked against the live API today; commands are included inline.

happyaron · 2026-09-25T16:40:24Z

+
+def get_json(url):
+    req = urllib.request.Request(url, headers={'User-Agent': UA})
+    with urllib.request.urlopen(req, timeout=TIMEOUT) as r:


The releases API is served through BunnyCDN with cache-control: public, max-age=2592000 (30 days), so the script can read a stale release list. Checked today (Sep 25) from the FR edge:

$ curl -sI 'https://www.php.net/releases/index.php?json&version=8.4' | grep -i cdn cdn-cachedat: 09/24/2026 13:47:44 cdn-cache: HIT $ curl -s '...?json&version=8.4' | jq -r .version # 8.4.25 $ curl -s '...?json&version=8.4&_=1' | jq -r .version # 8.4.26

?json&max=1 is cached the same way. Whether a new release is picked up depends on which edge answers and when it last cached the URL. The lag can be up to 30 days, and the run still reports success. The NJU mirror getting 8.4.26 means its edge happened to be fresh, not that this is reliable. Adding a cache-busting parameter to every API request fixes it, e.g. &_={int(time.time())}.

Done in 4bb19f2 — every API request now gets a unique &_= parameter. Verified from our edge today: cache-control: public, max-age=2592000 with cdn-cache: HIT, and your own test above shows BunnyCDN keys its cache on the full query string (&_=1 returned fresh 8.4.26 while the plain URL served cached 8.4.25), so the buster reliably defeats the 30-day cache. For the record: our production run did see php-8.4.26 on release day, which just means our edge happened to be fresh — as you said, edge behavior varies and the run would still have reported success, so the buster is the right fix regardless.

happyaron · 2026-09-25T16:40:24Z

+    for rel in top.values():
+        lines.update(rel.get('supported_versions') or [])
+    for line in sorted(lines):
+        rel = get_json(f'{API_URL}?json&version={line}')


Without max, ?json&version=X returns only the latest release of that line. Each run therefore sees one patch release per line. A fresh mirror would hold only e.g. 8.4.26 and none of the earlier 8.4.x releases. If syncs fail across two consecutive point releases, the earlier one is never fetched. With max, the response is a map keyed by version:

$ curl -s '...?json&version=8.4&max=3' | jq 'keys' ["8.4.24", "8.4.25", "8.4.26"]

Suggest ?json&version={line}&max=<N>&_=<ts> and iterating over .values(). A large N would cover all releases in the line.

Done in 4bb19f2 — each line is now fetched with max=10 (MAX_PER_LINE) and the response iterated via .values(). Verified empirically: ?json&version=8.4&max=10 returns a version-keyed map (keys 8.4.17..8.4.26, each with its source list), and the same shape holds for 8.2/8.3/8.5 (10 releases, 30 source entries each). A live run of remote_filelist() against the real API now discovers 120 tarballs. (This is also the map shape Copilot's earlier finding referred to — with max it really is keyed by full version.)

happyaron · 2026-09-25T16:40:24Z

+
+
+def main():
+    files = remote_filelist()


If the API response ever changes shape, files is empty and the run exits 0 with downloaded: 0, skipped: 0, failed: 0. Tunasync would then report success while the mirror goes stale, which is the problem this script is meant to fix. Could this fail loudly instead, e.g. if not files: sys.exit('no files from releases API')?

Done in 4bb19f2 — main() now calls sys.exit('ERROR: no files from the releases API') when the file list is empty, so a response-shape change fails the tunasync job loudly instead of reporting success on a stale mirror.

happyaron · 2026-09-25T16:40:24Z

+    return res
+
+
+socket.getaddrinfo = _ipv6_first


This makes IPv6 the default for every deployment. Many Docker setups have no working IPv6. When the network immediately reports "unreachable", create_connection moves on to IPv4 and nothing is lost. When IPv6 packets are silently dropped, though, each connection (download, retry, .asc) waits the full TIMEOUT=120 before falling back. Could this be opt-in via an env var (e.g. PHP_SYNC_PREFER_IPV6=1), so it fits NJU's network without changing the default for others?

Done in 4bb19f2 — IPv6-first is now opt-in via PHP_SYNC_PREFER_IPV6=1, default off, exactly for the silently-dropped-IPv6 scenario you describe. NJU sets the env var in its job config, keeping our current behavior.

happyaron · 2026-09-25T16:40:24Z

+                    h.update(chunk)
+            if h.hexdigest() != sha256:
+                raise Exception(f'sha256 mismatch: got {h.hexdigest()}')
+            os.rename(tmp, dst)


Nit: downloaded files keep the sync time as their mtime, so directory listings show when a file was synced rather than when it was released. github-release.py sets os.utime from the upstream timestamp. The same could be done here with the response's Last-Modified header (and for the .asc at line 99).

Done in 4bb19f2 — both tarballs and .asc files now get os.utime from the response Last-Modified header (via parsedate_to_datetime, unparseable/missing values tolerated), so listings show the upstream release time.

happyaron · 2026-09-25T16:40:24Z

+            else:
+                print(f'ERROR: giving up on {filename}', flush=True)
+                failed += 1
+        if failed == 0 or dst.is_file():


Nit: failed counts failures across all files, so this condition is effectively just dst.is_file(). The failed == 0 part reads as a check on this file only, but it isn't one. if dst.is_file(): says what the code actually does.

Done in 4bb19f2 — the condition is now simply if dst.is_file():, which is what the code actually meant.

happyaron · 2026-09-25T16:40:24Z

+            filename = src['filename']
+            # Only accept plain basenames: a name containing '/' or '..'
+            # could escape WORKING_DIR when used to build local paths.
+            if '/' in filename or '..' in filename:


Nit: Path(filename).name == filename (plus rejecting ''/'.'/'..') checks for a plain basename directly. '..' in filename matches the characters anywhere in the name, not just a .. path component.

Adopted in 4bb19f2 — the check is now a strict basename test: reject ''/'.'/'..' and require Path(filename).name == filename. Entries without sha256 are still skipped silently as before.

Copilot

Copilot review overview

🟡 Changes recommended

API error responses must be validated, and failed repairs must not leave corrupt files exposed.

Review effort: Lite
Findings: 2

Open (2)

Remove corrupt destination after failed hash-mismatch repair · New
Iterate version-keyed release response to download source tarballs

Resolved since last review (1)

Reject unsafe filenames before joining with mirror root

yaoge123 · 2026-09-26T02:18:30Z

+                print(f'ERROR: giving up on {filename}', flush=True)
+                failed += 1


Fixed in 4bb19f2: when a repair download fails permanently, a pre-existing (known-corrupt) dst is now removed — dst.unlink() with an explicit log line — so bad content is never left served, and the next run starts clean.

The old php rsync upstream (rsync.php.net) was decommissioned and the second-hand rsync sources that mirrors fell back to have disappeared one by one, so PHP release tarballs have been stuck (TUNA's php mirror has been failing too). This script mirrors all currently supported PHP release lines directly from the official site, stdlib only: - The file list and sha256 hashes come from www.php.net/releases/index.php?json. Every API request carries a unique &_=<epoch> parameter because BunnyCDN caches the API for up to 30 days keyed on the full query string; without it a run can read a stale release list and still report success. - Each supported line is fetched with max=10 so earlier patch releases are mirrored too, not just the latest one per line. - Filenames from the API must be plain basenames, so a malformed or compromised response cannot escape TUNASYNC_WORKING_DIR. - Tarballs are verified against the published sha256, downloaded to .tmp and renamed into place. Existing files are re-hashed before being skipped, so truncated/corrupt leftovers are re-downloaded; when a repair download fails permanently, the known-corrupt local file is removed rather than kept served. - An empty file list from the API fails the run loudly instead of reporting success on a stale mirror. - .asc signatures (not in the JSON API) are fetched best-effort and never fail the run. Downloaded files keep the upstream Last-Modified time as mtime, so listings show when a file was released. - PHP_SYNC_PREFER_IPV6=1 tries IPv6 first (php.net's BunnyCDN IPv4 is unreachable or slow from some networks); off by default because silently-dropped IPv6 would cost a full timeout per connection. Verified against the live API (a remote_filelist() run discovers 120 tarballs across the 8.2/8.3/8.4/8.5 lines). Running in production on mirror.nju.edu.cn, where php-8.4.26 tarballs and signatures appeared on release day.

yaoge123 · 2026-09-26T03:02:05Z

Branch cleanup note: this branch has been squashed to a single commit, 2f5d750. All commit SHAs referenced earlier in the review threads (fa237ad, 9a7da5f, d035fc4, 4bb19f2) are now orphaned commits — the links still open, but please rely on the current diff, whose tree is byte-identical to the previous head 4bb19f2.

Review items → how they were addressed (all included in the current diff)

Maintainer review (@happyaron):

BunnyCDN caches the releases API for up to 30 days → every API request now carries a unique &_=<epoch> parameter; BunnyCDN keys its cache on the full query string, so the buster reliably defeats the 30-day cache.
Without max, only the latest patch release per line is mirrored → each line is fetched with max=10 (MAX_PER_LINE) and the version-keyed response is iterated via .values(); a live remote_filelist() run discovers 120 tarballs across 8.2/8.3/8.4/8.5.
Empty file list exits 0 while the mirror goes stale → main() now does sys.exit('ERROR: no files from the releases API'), failing the tunasync job loudly.
IPv6-first as a default is risky where IPv6 is silently dropped → now opt-in via PHP_SYNC_PREFER_IPV6=1, default off.
Keep upstream release time as mtime → both tarballs and .asc files get os.utime from the response Last-Modified header.
failed == 0 condition read as per-file but wasn't → simplified to if dst.is_file():.
Basename check nit → strict test: reject ''/'.'/'..' and require Path(filename).name == filename.

Copilot rounds:

Existing tarballs skipped without checksum validation → local_ok() re-hashes existing non-empty files against the API's sha256 and re-downloads on mismatch.
Path traversal via src['filename'] → plain-basename rejection (above).
Failed repair left the known-corrupt file served → a permanently failed repair now removes the corrupt destination (dst.unlink() with a log line), so the next run starts clean.
"Response shape means nothing downloads" → false positive for the original no-max request shape (a flat single-release record, verified against the live API); the switch to max=10 + .values() made the iteration shape explicit anyway.

Some earlier in-thread replies described intermediate states of the branch; please rely on the current diff and this summary.

Copilot AI lite review requested due to automatic review settings September 23, 2026 15:44

Copilot started reviewing on behalf of yaoge123 September 23, 2026 15:44 View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

Comment thread php-sync.py Outdated

yaoge123 requested a lite review from Copilot September 25, 2026 15:31

Copilot started reviewing on behalf of yaoge123 September 25, 2026 15:32 View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

Comment thread php-sync.py Outdated

Comment on lines +47 to +50

rel = get_json(f'{API_URL}?json&version={line}')

for src in rel.get('source', []):

if 'sha256' in src:

files[src['filename']] = src['sha256']

Comment thread php-sync.py Outdated

happyaron reviewed Sep 25, 2026

View reviewed changes

yaoge123 requested a lite review from Copilot September 26, 2026 01:51

Copilot started reviewing on behalf of yaoge123 September 26, 2026 01:51 View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

yaoge123 force-pushed the php-sync-upstream branch from 4bb19f2 to 2f5d750 Compare September 26, 2026 02:52

		print(f'ERROR: giving up on {filename}', flush=True)
		failed += 1

Conversation

yaoge123 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

How it works

Environment variables

Testing

Deployment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

yaoge123 commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

yaoge123 commented Sep 25, 2026

Uh oh!

happyaron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaoge123 commented Sep 26, 2026

Review items → how they were addressed (all included in the current diff)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yaoge123 commented Sep 23, 2026 •

edited

Loading