Add buildroot.py to mirror sources.buildroot.net via .mk parsing by yaoge123 · Pull Request #204 · tuna/tunasync-scripts

yaoge123 · 2026-05-24T09:21:16Z

Summary

Add buildroot.py, a self-contained incremental sync script for sources.buildroot.net.

Why

sources.buildroot.net (the official Buildroot source backup) has permanently disabled directory listing (Cloudflare 403 on every path that isn't a known file) and offers no rsync module:

Tool	Result
rsync	no rsync module offered
tsumugu / wget --mirror	cannot enumerate; every directory request 403
recursive HTML scraping	no HTML index exists

The Buildroot maintainers have stated on the mailing list that they will not enable rsync, so a mirror has to derive the file list from somewhere else. This script derives it from the buildroot git tree.

How it works

Shallow-clone the buildroot master branch
Parse boot/, linux/, package/, toolchain/, utils/ .mk files for VERSION/SITE/SOURCE
Expand version variables (including $(subst)), Kconfig mirror defaults (BR2_GNU_MIRROR, BR2_KERNEL_MIRROR, BR2_LUAROCKS_MIRROR, BR2_CPAN_MIRROR), the github/gitlab/sourceforge macros, and pkg-download.mk's VCS filename suffixes (-git4/-svn5/-cargo6/-go2)
HEAD-compare each candidate URL against the local mirror by Content-Length before skipping
Download new/changed files atomically (.tmp + rename), trying the backup site (TUNASYNC_UPSTREAM_URL) first — it is canonical for this mirror — and the package's own site as fallback
Optionally clean up stale files (opt-in, capped)

Existing local data (478G at NJU) is preserved: the script only adds/updates files unless cleanup is explicitly enabled.

Environment variables

Variable	Default	Meaning
`TUNASYNC_WORKING_DIR`	(required)	mirror data directory
`TUNASYNC_UPSTREAM_URL`	`http://sources.buildroot.net/`	backup site, tried first
`TUNASYNC_BUILDROOT_GIT`	`https://github.com/buildroot/buildroot.git`	metadata source
`TUNASYNC_BUILDROOT_BRANCH`	`master`	branch to parse
`TUNASYNC_BUILDROOT_JOBS`	`1`	parallel downloads (serial by default)
`TUNASYNC_BUILDROOT_CLEANUP`	off	`1` enables stale-file deletion
`TUNASYNC_BUILDROOT_MAXDELETE`	`10000`	safety cap for cleanup
`TUNASYNC_BUILDROOT_DRYRUN`	off	log only, write nothing
`TUNASYNC_BUILDROOT_WGET_TIMEOUT`	`3600`	per-file download cap (s)
`https_proxy` / `http_proxy`		honored for downloads

Safety properties

wget replaces the destination only on success; a failed download never deletes a known-good mirrored file
the state file advances only after a fully successful non-dry run; an unchanged HEAD still runs a full verification pass, so files deleted or corrupted since the last run self-heal
an extraction yielding 0 packages aborts before touching state or local files; cleanup refuses to run on an empty extraction and is skipped whenever any download failed
Kconfig-driven packages (linux/gcc/uboot via $(call qstrip,$(BR2_...))) are counted as unresolvable, not failures — full resolution would need make show-info and is out of scope; mirroring covers master only (per-branch/LTS tracking is a deliberate limitation, and cleanup stays opt-in because of it)

Testing

Smoke-tested the macro emit helpers against buildroot master (aichat/z3/leafnode2 archive refs, wf111 conditional sources, zip's $(subst)), archive-extension detection, and the zero-package cleanup fuse, plus a live extraction pass over the full tree. Running in production at mirror.nju.edu.cn against the existing 478G mirror.

Deployment

Stdlib Python 3 + wget + git, all already present in the standard tunathu/tunasync-scripts image — no image change needed.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a self-contained Python sync script to mirror Buildroot package sources without relying on upstream directory listing, by extracting download URLs from the Buildroot git tree and incrementally downloading missing artifacts.

Changes:

Introduces buildroot.py to clone/update Buildroot git, parse .mk metadata, and download package sources incrementally.
Adds basic logging + state tracking to skip runs when git HEAD hasn’t changed.
Implements atomic downloads via .tmp → rename with fallback to sources.buildroot.net.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

yaoge123 · 2026-09-23T16:11:04Z

Update from production (mirror.nju.edu.cn): packages whose .mk leaves a variable unexpanded (e.g. $(FOO_VERSION) in the computed filename) used to create garbage files on disk. 5b5cbb3 now detects $(/${ in the local filename, logs it, counts the package as failed, and skips the download. The .tmp cleanup part of that production fix was already covered here by the finally-block in wget().

yaoge123 · 2026-09-25T14:37:06Z

Copilot review addressed (fixes spread across 7bc1b5a, fb0bc70, 5b5cbb3, 511d86f):

Full tarball URLs re-appended with a filename — fixed: sync_package() detects URLs that already end in a filename (url_has_filename(), e.g. github/gitlab archive URLs) and uses them as-is; ...tar.gz/<file> URLs are no longer produced.
stats['total'] under-counting — fixed: total is bumped at the top of sync_package(), before any skip return, so progress logging and the summary cover every processed package.
Leftover .tmp on download failure — fixed: .tmp is removed via a finally block on every code path, and the wget stderr tail (and subprocess exceptions) are now logged on failure (511d86f).
**:=' assignments missed** — fixed: the assignment regex now accepts [:?+]?=`.
JOBS/MAXDELETE printed but ineffective — resolved by making the knobs real instead of removing them: downloads parallelize via ThreadPoolExecutor when TUNASYNC_BUILDROOT_JOBS>1 (default 1 = serial), and cleanup is functional but opt-in via TUNASYNC_BUILDROOT_CLEANUP, bounded by TUNASYNC_BUILDROOT_MAXDELETE.
git fetch/reset return codes ignored — fixed: rc is checked and stderr logged for fetch/reset/clone, and the run aborts rc=1 when no git tree is available instead of syncing an empty package list. The 180s timeout in run() is unchanged.

Copilot

Copilot review overview

🟡 Changes recommended

One critical cleanup-safety issue and multiple moderate correctness issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 5

Open (6)

Refuse cleanup when no packages are extracted · New
Generate the standard GitHub archive URL · New
Add explicit SourceForge macro support · New
Compare remote Content-Length before skipping files · New
Honor the configurable upstream URL · New
Advance state only after successful non-dry runs · New

Resolved since last review (12)

For GitHub/GitLab $(call ...) sites, emit_* returns what appears to be a full tarball URL, but…
For GitHub/GitLab $(call ...) sites, emit_* returns what appears to be a full tarball URL, but…
The git fetch/reset calls ignore return codes; if either fails (network hiccup, lock contention,…
The git fetch/reset calls ignore return codes; if either fails (network hiccup, lock contention,…
JOBS and MAXDELETE are configurable and printed, but neither affects behavior: downloads are…
JOBS and MAXDELETE are configurable and printed, but neither affects behavior: downloads are…
JOBS and MAXDELETE are configurable and printed, but neither affects behavior: downloads are…
The variable assignment regex does not match Makefile := assignments (used in Buildroot in some…
On download failure, the .tmp file can be left behind (the caller deletes dest, not tmp).…
On download failure, the .tmp file can be left behind (the caller deletes dest, not tmp).…
stats['total'] is not incremented for skipped packages, so progress logging and the final summary…
stats['total'] is not incremented for skipped packages, so progress logging and the final summary…

happyaron

I ran extract_packages() / sync_package() from this PR against a fresh shallow clone of buildroot master. Several issues are reproducible (details inline). The three most serious are: HTML pages being saved as tarballs, the GitHub tag being ignored, and wrong filenames for git/svn/cargo/go packages.

Design concerns:

Master only: LTS branches (e.g. 2025.02.x) are never mirrored, and with cleanup enabled, every older version still used by LTS users would be deleted.
Sources whose version comes from Kconfig are never mirrored: linux, uboot, gcc, arm-trusted-firmware, etc. use $(call qstrip,$(BR2_...)), so they are skipped as "unexpanded". Sites based on $(BR2_GNU_MIRROR) / $(BR2_KERNEL_MIRROR) are silently dropped, and variables defined in another file (e.g. QT6BASE_VERSION = $(QT6_VERSION)) are not resolved, so qt6base is missing entirely. As a side effect the script almost always exits with 2.
Parsing Make by regex is fragile (+= overwrites instead of appending, and inside ifeq blocks the last assignment wins). Consider letting Buildroot produce the list itself (make <defconfig> show-info / source gives the exact filenames and URIs), e.g. across all defconfigs. It is slower, but correct.
Copilot's second round is still unaddressed in the code: cleanup runs when the extracted package list is empty, there is no sourceforge macro support, there is no Content-Length check (both are promised in the description), and UPSTREAM is unused.

Minor: .buildroot-sync.log lives in the published directory and grows without bound; import threading sits in the middle of the file; TimeoutExpired from run() is not caught.

happyaron · 2026-09-25T17:38:49Z

+            f"/-/archive/{version}/{repo_name}-{version}.tar.gz")
+
+
+def url_has_filename(url: str) -> bool:


Saves HTML pages as tarballs. Any last path segment containing a . is treated as a filename. That matches versioned directories (https://python.org/ftp/python/3.14.7, .../releases/download/16.1) and .git repository URLs. On current master, 515 of 2903 candidates are affected. wget then downloads the directory listing or the repo page and stores it under the tarball name:

wd/python3/Python-3.14.7.tar.xz: HTML document, ASCII text wd/aer-inject/aer-inject-9bd5e2c7...tar.gz: HTML document, Unicode text

Because of the existence-only check at L306, these files are never replaced, and Buildroot users who fall back to the mirror get a hash mismatch. It is worth checking the existing mirror with file(1). The distinction should come from the source (macro vs. plain SITE), not from a heuristic on the URL. Verifying against the package's .hash file (sha256_file is already there, just unused) would also catch this.

Fixed in 1e11a0a: filename detection uses an archive-extension whitelist (since round 2), and with the backup site now tried first, the Content-Length comparison automatically re-downloads HTML error pages previously saved under a tarball name. .hash-based verification is left as a follow-up.

happyaron · 2026-09-25T17:38:49Z

+        if m:
+            packages.append({
+                "name": name, "version": version,
+                "url": emit_github(m.group("org").strip(),


The tag from $(call github,...) is ignored. The regex captures version, but emit_github is passed the bare *_VERSION. Any prefix is lost: AICHAT_SITE = $(call github,sigoden,aichat,v$(AICHAT_VERSION)) becomes https://github.com/sigoden/aichat/archive/0.30.0/aichat-0.30.0.tar.gz, which returns 404. About 377 packages build the tag like this. Use m.group("version") (the same applies to gitlab).

Fixed in 1e11a0a: emit_github now receives the third macro argument verbatim, so prefixed refs resolve correctly -- verified against buildroot master: aichat -> archive/v0.30.0, z3 -> archive/z3-4.16.0.

happyaron · 2026-09-25T17:38:49Z

+                break
+
+        if not source and name and version:
+            source = f"{name}-{version}.tar.gz"


Wrong default filename for git/svn/cargo/go packages. Buildroot uses $(RAWNAME)-$(VERSION)$(pkg_source_ext), and pkg_source_ext adds -git4, -svn5, -cargo6 or -go2 depending on SITE_METHOD / DOWNLOAD_POST_PROCESS (see package/pkg-utils.mk). Example:

https://sources.buildroot.net/aichat/aichat-0.30.0-cargo6.tar.gz -> 200 https://sources.buildroot.net/aichat/aichat-0.30.0.tar.gz -> 404

About 148 packages are affected, and these are exactly the files that exist only on the backup site.

Fixed in 1e11a0a: the default SOURCE now appends pkg-download.mk's suffixes (-git4/-svn5 from SITE_METHOD, -go2/-cargo6 from the download post-process, including $(eval $(cargo-package))/$(golang-package)). 159 packages get the suffixed name, and VCS-method packages whose SITE is a repo URL now fall back to the backup site only.

happyaron · 2026-09-25T17:38:49Z

+            })
+            continue
+
+        cleaned = re.sub(r'\$\([^)]+\)', '', site).rstrip('/')


Stripping $(...) here silently drops every package whose SITE starts with $(BR2_GNU_MIRROR) / $(BR2_KERNEL_MIRROR). The result doesn't start with http, so the package does not even reach the fallback to sources.buildroot.net. If a variable in the middle of the URL is stripped, the result is a nonsense URL.

Fixed in 1e11a0a: BR2_GNU_MIRROR, BR2_KERNEL_MIRROR, BR2_LUAROCKS_MIRROR and BR2_CPAN_MIRROR are seeded from their Config.in defaults, and a two-pass scan resolves cross-file references -- 93 gnu/kernel-mirror packages recovered (verified on master).

happyaron · 2026-09-25T17:38:49Z

+            urls_to_try.append(url)
+        else:
+            urls_to_try.append(url.rstrip("/") + "/" + local_file)
+    urls_to_try.append(f"{_BR_PRIMARY_SITE}/{name}/{local_file}")


sources.buildroot.net is only the fallback, after the original upstream. For a mirror of the backup site the order should be reversed: the backup copy is the canonical file (e.g. for non-reproducible GitHub tarballs), and reversing it also reduces load on upstream projects. UPSTREAM (L46) should be used here instead of the hard-coded _BR_PRIMARY_SITE.

Fixed in 1e11a0a: the backup site (TUNASYNC_UPSTREAM_URL) is tried first and the package's own upstream site only as a fallback.

happyaron · 2026-09-25T17:38:49Z

+
+    clean_stale_files(packages)
+
+    STATE_FILE.write_text(head)


The state is written even after failed downloads and in DRYRUN mode. A dry run therefore permanently suppresses the next real sync, and failed files are not retried until master moves. Write it only after a real run with no failures. Relatedly, L422 returns 0 when there are no new commits even though the previous run ended with 2, so the tunasync status flips back and forth.

State is only written after a non-dry run with zero failures (since round 2); in 1e11a0a an unchanged HEAD also no longer returns early -- it runs a full verification pass so an interrupted run self-heals.

happyaron · 2026-09-25T17:38:49Z

+        cmd[1:1] = ["-e", "use_proxy=on", "-e", f"https_proxy={HTTP_PROXY}"]
+    cmd.append(url)
+    try:
+        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=300)


A 300 s hard timeout kills large downloads (linux, gcc, qt) on slower links, and they then fail again on every run. Rely on wget's --timeout / --read-timeout, or make this configurable.

Fixed in 1e11a0a: the hard timeout is now configurable via TUNASYNC_BUILDROOT_WGET_TIMEOUT (default 3600 s); wget's own --timeout=60 network timeout is unchanged.

happyaron · 2026-09-25T17:38:49Z

+    Only runs when TUNASYNC_BUILDROOT_CLEANUP is enabled, and refuses to
+    remove more than MAXDELETE files in a single run.
+    """
+    if not CLEANUP:


Cleanup compares against master only, so with CLEANUP=1 it would delete every older version that users of LTS branches still download. It is currently prevented only because the real file count exceeds MAXDELETE=10000. It also still runs when packages is empty (Copilot's comment).

The empty-extraction fuse is in place (cleanup refuses to run on 0 packages). On LTS: cleanup stays opt-in (off by default) and capped by TUNASYNC_BUILDROOT_MAXDELETE, but mirroring master only is a deliberate limitation -- per-branch tracking would need per-branch state and is out of scope for this PR.

yaoge123 · 2026-09-26T00:19:46Z

Round-2 review findings addressed in 575e3c6:

Cleanup fuse: clean_stale_files() refuses to run when extraction yielded 0 packages, so a parser/tree failure can no longer make every local file look stale.
GitHub archive URL: emit_github()/emit_gitlab() now return the bare archive base URL exactly like Buildroot's github/gitlab helpers (https://github.com/<org>/<repo>/archive/<ref>), and sync_package() appends the actual source filename — so packages overriding <pkg>_SOURCE get the correct URL. (For the default source name the resulting URL is unchanged, matching what Buildroot itself downloads.)
SourceForge macro: explicit $(call sourceforge,<project>,<file>[,<version>]) expansion to downloads.sourceforge.net was added before the generic URL handling; such packages are no longer silently omitted.
Content-Length check: an existing non-empty file is no longer skipped blindly — the remote is HEADed and the file is re-downloaded on size mismatch (unknown remote size keeps the local copy, so HEAD rejection never causes re-download storms).
UPSTREAM honored: the fallback download path now uses TUNASYNC_UPSTREAM_URL (default http://sources.buildroot.net/) instead of the hard-coded site.
State file: only written after a non-dry run with zero failures; a transient failure no longer records the commit as complete, and dry runs never suppress the first real sync.

Smoke-tested: emit helpers, archive-extension detection, and the 0-package cleanup fuse.

Copilot

Copilot review overview

🟡 Changes recommended

Critical parsing, URL generation, failure handling, and state-management issues remain unresolved.

Review effort: Lite
Findings: 12

Open (12)

Make subst expressions remain unexpanded · New
Unexpanded qstrip expression prevents kernel mirroring · New
GitHub helper emits an invalid archive URL · New
Parser discards conditional source assignments · New
Parser misses generated package variable assignments · New
GitHub URL uses package version instead of archive ref · New
GitLab URL uses package version instead of archive ref · New
Failed downloads delete valid existing files · New
Unchanged HEAD prevents mirror reconciliation · New
Empty extraction incorrectly marks commit complete · New
Worker failures do not increment failed count · New
Cleanup deletes files after incomplete downloads · New

Resolved since last review (6)

Refuse cleanup when no packages are extracted
Advance state only after successful non-dry runs
Honor the configurable upstream URL
Compare remote Content-Length before skipping files
Add explicit SourceForge macro support
Generate the standard GitHub archive URL

Previously missed (1)

In code that hasn't changed since last review

VERSION_OVERRIDES is never applied

buildroot.py:255

VERSION_OVERRIDES is declared as the manual escape hatch described in the module documentation, but it is never read while selecting version. Adding an entry to this dict currently has no effect, so packages requiring an override remain unhandled; apply the dict during version selection.

yaoge123 · 2026-09-26T02:30:50Z

+def expand_variable(val: str, variables: dict[str, str]) -> str:
+    """Recursively expand $(VAR) references in val."""
+    if not val or "$(" not in val:
+        return val
+    changed = True
+    max_iter = 20
+    while changed and max_iter > 0:
+        changed = False
+        max_iter -= 1
+        for var, vval in sorted(variables.items(), key=lambda x: -len(x[0])):
+            needle = f"$({var})"
+            if needle in val:
+                val = val.replace(needle, vval)
+                changed = True
+    return val


Fixed in 1e11a0a: expand_variable now iteratively resolves innermost $(...) and supports $(subst ...) -- verified: zip -> zip30.tar.gz.

yaoge123 · 2026-09-26T02:30:52Z

+def expand_variable(val: str, variables: dict[str, str]) -> str:
+    """Recursively expand $(VAR) references in val."""
+    if not val or "$(" not in val:
+        return val
+    changed = True
+    max_iter = 20
+    while changed and max_iter > 0:
+        changed = False
+        max_iter -= 1
+        for var, vval in sorted(variables.items(), key=lambda x: -len(x[0])):
+            needle = f"$({var})"
+            if needle in val:
+                val = val.replace(needle, vval)
+                changed = True
+    return val


In 1e11a0a these Kconfig-driven packages (linux/gcc/uboot...) are counted as 'unresolvable' instead of failed, so they no longer fail the run. Fully resolving them needs make show-info / Kconfig evaluation, out of scope for this parser.

yaoge123 · 2026-09-26T02:30:54Z

+    <site>/<source>; keep the same split so sync_package() appends the actual
+    source filename.
+    """
+    return f"https://github.com/{repo_org}/{repo_name}/archive/{version}"


This one is a false positive: per package/pkg-download.mk the github helper is the base https://github.com/$(1)/$(2)/archive/$(3) and Buildroot downloads /. sync_package appends the source filename to directory-style sites in exactly that way (verified: .../archive/v0.30.0/aichat-0.30.0-cargo6.tar.gz downloads fine).

yaoge123 · 2026-09-26T02:30:56Z

+        for line in text.splitlines():
+            stripped = line.strip()
+            if not stripped or stripped.startswith("#"):
+                continue
+            m = _ASSIGN_RE.match(stripped)


Fixed in 1e11a0a: conditional *_SOURCE reassignments (ifeq arch blocks) are collected as extra entries, so all three wf111 variants are mirrored (verified on master).

yaoge123 · 2026-09-26T02:30:58Z

+            m = _ASSIGN_RE.match(stripped)
+            if m:
+                variables[m.group(1)] = m.group(2).strip()


Acknowledged as a limitation: inner-package generated definitions need Make macro expansion that a regex parser cannot do; they are skipped rather than mis-parsed.

yaoge123 · 2026-09-26T02:31:04Z

+        if wget(try_url, dest):
+            bump("downloaded")
+            return True
+        dest.unlink(missing_ok=True)


Fixed in 1e11a0a: wget only replaces dest on success and the failure path no longer unlinks the existing file, so the previously mirrored copy stays in place.

yaoge123 · 2026-09-26T02:31:06Z

+    if last_head == head:
+        log(f"No new commits since {head}, skipping")
+        return 0


Fixed in 1e11a0a: an unchanged HEAD now runs a full verification pass instead of returning early, so files deleted or corrupted since the last run are reconciled.

yaoge123 · 2026-09-26T02:31:08Z

+    packages = extract_packages(BR_GIT_DIR)
+    log(f"Found {len(packages)} packages to process")


Fixed in 1e11a0a: main() aborts immediately on zero extracted packages without touching the state file or local files.

yaoge123 · 2026-09-26T02:31:11Z

+                    fut.result()
+                except Exception as e:
+                    log(f"  unexpected error: {e}")


Fixed in 1e11a0a: a crashed worker now bumps stats['failed'], so the state file is not advanced and the next run retries.

yaoge123 · 2026-09-26T02:31:13Z

+                    f"downloaded={stats['downloaded']} "
+                    f"failed={stats['failed']}")
+
+    clean_stale_files(packages)


Fixed in 1e11a0a: cleanup is skipped whenever any download failed (or on dry runs), so a failed re-download never deletes previously mirrored files.

sources.buildroot.net (the official Buildroot source backup site) has permanently disabled directory listing (Cloudflare 403) and offers no rsync module, so neither tsumugu nor wget -m can discover which files exist. This script derives the file list from the buildroot git tree instead: it parses boot/linux/package/toolchain .mk files for VERSION/SITE/SOURCE, expands version variables (including $(subst)), Kconfig mirror defaults (BR2_GNU_MIRROR & friends) and the github/gitlab/sourceforge macros, applies pkg-download.mk's VCS filename suffixes (-git4/-svn5/-cargo6/-go2), and downloads only new or changed files. Key design decisions (several shaped by review): - The backup site (TUNASYNC_UPSTREAM_URL) is canonical for this mirror and is tried FIRST; the package's own site is only a fallback, which also reduces load on upstream projects. - Existing files are HEAD-checked (Content-Length) before being skipped, so republished/corrected same-name files are re-downloaded; an unknown remote size always keeps the local copy. - Downloads are atomic (.tmp + rename); wget replaces the destination only on success, so a failed run never deletes known-good mirrored data. - A state file records the last fully synced commit, but an unchanged HEAD still runs a full verification pass, so files deleted or corrupted since the last run self-heal. State advances only after a non-dry run with zero failures; an extraction that yields zero packages aborts without touching state or local files. - Stale-file cleanup is opt-in (TUNASYNC_BUILDROOT_CLEANUP), capped by TUNASYNC_BUILDROOT_MAXDELETE, refuses to run on an empty extraction, and is skipped entirely when any download failed. - Packages whose version comes from Kconfig (linux/gcc/uboot via $(call qstrip,$(BR2_...))) cannot be resolved by regex parsing; they are counted as "unresolvable", not failures, so the run stays green. Full resolution would need `make show-info` and is out of scope. Mirroring covers master only; per-branch (LTS) tracking is a deliberate limitation, and cleanup stays opt-in because of it. - JOBS defaults to 1 (serial); the per-file download cap is TUNASYNC_BUILDROOT_WGET_TIMEOUT (default 3600 s). Smoke-tested: macro emit helpers (aichat/z3/leafnode2 archive refs, wf111 conditional sources, zip's $(subst)), archive-extension detection, and the zero-package cleanup fuse, plus a live extraction pass against buildroot master. Existing local data (478G) is preserved; running in production at mirror.nju.edu.cn.

yaoge123 · 2026-09-26T02:57:39Z

Branch cleanup note: this branch has been squashed to a single commit, 08ecd52. All commit SHAs referenced earlier in the review threads (up to 1e11a0a) are now orphaned commits — the links still open, but please rely on the current diff, whose tree is byte-identical to the previous head 1e11a0a.

Review items → how they were addressed (all included in the current diff)

Maintainer review (reproduced against buildroot master):

HTML pages saved as tarballs (515/2903 candidates) → filename detection uses an archive-extension whitelist; with the backup site tried first, the Content-Length check re-downloads HTML error pages previously saved under tarball names. .hash-based verification is left as a follow-up.
Tag in $(call github,org,repo,TAG) ignored → the third macro argument is now passed verbatim (verified: aichat → archive/v0.30.0, z3 → archive/z3-4.16.0); same fix for gitlab.
Wrong default filename for git/svn/cargo/go packages → pkg-download.mk suffixes (-git4/-svn5/-cargo6/-go2) are applied; 159 packages get the suffixed name, and VCS-method packages fall back to the backup site.
$(BR2_GNU_MIRROR)/$(BR2_KERNEL_MIRROR) sites silently dropped → these (plus luarocks/CPAN) are seeded from their Config.in defaults with a two-pass scan; 93 gnu/kernel-mirror packages recovered.
Backup site should be tried first, UPSTREAM unused → TUNASYNC_UPSTREAM_URL is tried first; the package's own site is the fallback.
State written after failed/dry runs, status flapping → state advances only after a non-dry run with zero failures, and an unchanged HEAD now runs a full verification pass instead of exiting early.
300 s hard timeout kills large downloads → configurable TUNASYNC_BUILDROOT_WGET_TIMEOUT (default 3600 s); wget's own network timeout unchanged.
Cleanup vs. LTS branches → cleanup stays opt-in (off by default) and MAXDELETE-capped; mirroring master only is a documented, deliberate limitation.
"Let Buildroot produce the list (make show-info)" → acknowledged as the correct-but-heavy approach; out of scope for this regex parser, documented in the docstring.

Copilot rounds:

Round 1: double-appended filenames, stats['total'] under-count, leftover .tmp, := assignments, ineffective JOBS/MAXDELETE knobs, unchecked git return codes → all fixed.
Round 2: cleanup fuse on empty extraction, GitHub archive URL shape, sourceforge macro support, Content-Length check before skipping, UPSTREAM honored, state-file timing → all fixed.
Round 3: $(subst) support, Kconfig-driven packages counted as unresolvable instead of failed, conditional *_SOURCE reassignments collected (wf111), destination preserved on failed re-download, zero-extraction abort, crashed workers counted as failures, cleanup skipped on incomplete runs → all fixed; inner-*-package generated definitions are an acknowledged parser limitation (skipped, not mis-parsed). One round-3 finding (GitHub helper URL shape) was a false positive — per package/pkg-download.mk the helper is the base .../archive/<ref> and Buildroot downloads <site>/<source>, which is exactly how sync_package() uses it.

Some earlier in-thread replies described intermediate states of the branch; please rely on the current diff and this summary.

Copilot AI review requested due to automatic review settings May 24, 2026 09:21

Copilot AI reviewed May 24, 2026

View reviewed changes

yaoge123 force-pushed the add-buildroot-py branch from 8f2e0cb to ff1cbf5 Compare May 24, 2026 09:25

yaoge123 requested a lite review from Copilot September 25, 2026 15:30

Copilot started reviewing on behalf of yaoge123 September 25, 2026 15:31 View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

Comment thread buildroot.py

Comment thread buildroot.py Outdated

Comment thread buildroot.py Outdated

Comment thread buildroot.py Outdated

Comment thread buildroot.py Outdated

Comment thread buildroot.py Outdated

happyaron requested changes Sep 25, 2026

View reviewed changes

yaoge123 requested a lite review from Copilot September 26, 2026 01:50

Copilot started reviewing on behalf of yaoge123 September 26, 2026 01:50 View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

yaoge123 force-pushed the add-buildroot-py branch from 1e11a0a to 08ecd52 Compare September 26, 2026 02:49

		f"/-/archive/{version}/{repo_name}-{version}.tar.gz")


		def url_has_filename(url: str) -> bool:

		packages = extract_packages(BR_GIT_DIR)
		log(f"Found {len(packages)} packages to process")

Conversation

yaoge123 commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

How it works

Environment variables

Safety properties

Testing

Deployment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yaoge123 commented Sep 23, 2026

Uh oh!

yaoge123 commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

happyaron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaoge123 commented Sep 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

yaoge123 commented May 24, 2026 •

edited

Loading