Add mirror-clone.sh wrapper for sjtug/mirror-clone

[yaoge123]

Summary

Add a generic shell wrapper for the sjtug/mirror-clone binary so multiple tunasync jobs can share one script, plus dockerfiles/mirror-clone/Dockerfile to build the image.

Why

mirror-clone supports many source plugins (ghcup, crates.io, GitHub releases, …). Today each tunasync job using mirror-clone has to write its own short shell file, which is mostly boilerplate (resolve TUNASYNC_WORKING_DIR, set --target-type file, choose source). This wrapper centralises that boilerplate so each tunasync .conf only sets a few env vars.

Interface

Environment variables:

Variable	Required	Purpose
TUNASYNC_MIRRORCLONE_SOURCE	Yes	source plugin name (e.g. ghcup, crates-io)
TUNASYNC_MIRRORCLONE_OPTIONS	No	mirror-clone options that come before the source plugin name
TUNASYNC_MIRRORCLONE_ARGS	No	plugin-specific arguments after the source plugin
TUNASYNC_MIRRORCLONE_BUFFER	No	file-backend buffer dir (default: $TUNASYNC_WORKING_DIR/.mirror-clone-buffer)
MIRRORCLONE_BIN	No	path/name of the binary (default: mirror-clone, looked up in PATH, like tsumugu.sh)

TUNASYNC_WORKING_DIR is validated with an actionable error and resolved to an absolute path before use. The pass-through option variables are intentionally word-split (same interface as tsumugu.sh); set -f keeps them from also being glob-expanded against the mirror dir.

Buffer directory

mirror-clone's file backend moves finished downloads into place with rename(2) (src/file_backend.rs), which cannot cross filesystems, and tunasync's docker provider only bind-mounts the working dir — so the buffer defaults to a hidden dir inside the working dir (same filesystem by construction). Upstream recommends keeping the buffer outside the base path (the backend scans every file under the base as mirror content), so deployments that mount an extra volume can set TUNASYNC_MIRRORCLONE_BUFFER to a path outside the working dir (on the same filesystem) to keep it out of the served tree.

Image

dockerfiles/mirror-clone/Dockerfile builds the binary from a pinned commit (GIT_REF=02134d5…, overridable via --build-arg GIT_REF=<sha/tag/branch>; cargo install --rev accepts all three) and the runtime image installs rsync (mirror-clone's rsync source shells out to it, src/rsync.rs) — git is not needed at runtime and not installed. The image is deliberately not wired into docker-images.yml yet; that can follow once the approach is approved.

Testing

The image builds with docker build from the pinned commit; the wrapper is running in production at mirror.nju.edu.cn.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a mirror-clone.sh wrapper script to run mirror-clone in a tunasync working directory, driven by environment variables.

Changes:

• Introduces a bash entrypoint that validates TUNASYNC_MIRRORCLONE_SOURCE and runs mirror-clone via exec
• Uses TUNASYNC_WORKING_DIR as the working directory and configures file buffer/base paths under it

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Harry-Chen]

如果要增加外部工具，最好也提供一份 dockerfile 安装此工具（如 rustup-mirror）

[yaoge123]

Done — added dockerfiles/mirror-clone/Dockerfile (c9fca11), following the rustup-mirror/tsumugu image pattern. Since mirror-clone is not published on crates.io, the image builds it from the sjtug/mirror-clone git repo with a GIT_REF build-arg (default master).

One caveat: wiring the new image into .github/workflows/docker-images.yml is not included in the push because my token lacks the workflow scope. Three one-line spots need it:

1. all_images='["ftpsync","mirror-clone",...]'
2. dispatch enum: ftpsync|mirror-clone|nix-channels|...
3. path filter: dockerfiles/mirror-clone/*) selected="$selected mirror-clone" ;;

Happy to add that as a separate commit if a maintainer confirms, or feel free to apply it directly.

[yaoge123]

Copilot review addressed:

1. TUNASYNC_WORKING_DIR unbound under set -u — fixed: explicit validation with an actionable error message (e07451e).
2. Missing pipefail — fixed: set -euo pipefail (c06edcb).
3. Unquoted $TUNASYNC_MIRRORCLONE_OPTIONS / $TUNASYNC_MIRRORCLONE_ARGS — intentional: word-splitting is exactly how per-mirror option lists are passed through these variables; switching to arrays would change the interface and break existing tunasync mirror configs. Keeping as-is.
4. Hard-coded binary path — fixed: ${MIRRORCLONE_BIN:-${_here}/mirror-clone} allows an environment override (c06edcb).

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical wrapper defects and moderate Docker integration issues block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 1

Open (3)

• Default binary path conflicts with documented deployment mount · New
• File buffer path is missing and nested inside the target base · New
• Mirror-clone Dockerfile is excluded from image workflows · New

Resolved since last review (4)

• TUNASYNC_WORKING_DIR is referenced without being initialized/validated, and set -u will cause…
• The binary path is hard-coded to /home/mirror-clone, which reduces portability and can fail…
• Unquoted $TUNASYNC_MIRRORCLONE_OPTIONS / $TUNASYNC_MIRRORCLONE_ARGS will undergo word-splitting…
• Consider set -euo pipefail (or set -eu plus set -o pipefail) to avoid pipelines masking…

[yaoge123]

Noted, and called out in the PR description: the Dockerfile is deliberately not wired into docker-images.yml yet — the workflow entry can follow once the image contents are approved, so a merge does not immediately start publishing tunathu/mirror-clone.

[happyaron]

Thanks for putting this together — a shared wrapper is a nice cleanup, and the recent fixes (TUNASYNC_WORKING_DIR check, pipefail) look good. I checked it against current mirror-clone (89fcc52) and tunasync's docker provider, and found a couple of things that I think will stop it from working in practice:

1. Buffer dir on a different filesystem → rename fails

mirror-clone's file backend moves finished downloads into place with tokio::fs::rename (src/file_backend.rs:97), which can't cross filesystems. tunasync's docker provider only bind-mounts the working dir plus configured volumes (worker/docker.go:62), so <workdir>.mirror-clone-buffer ends up in the container's overlay fs and I'd expect every file-backed transfer to fail with EXDEV (Invalid cross-device link). The same can happen on bare metal when each mirror is its own ZFS dataset / btrfs subvolume. As a side effect, it also leaves a foo.mirror-clone-buffer dir next to the mirrors in the served root.

Maybe make the buffer path configurable (e.g. TUNASYNC_MIRRORCLONE_BUFFER) and document that it must be on the same filesystem as the working dir and mounted into the container?

2. Wrapper can't find the binary in the new image

The Dockerfile installs /usr/local/bin/mirror-clone, but the wrapper only looks at $MIRRORCLONE_BIN, /home/mirror-clone, and ${_here}/mirror-clone — never PATH. So with the image from this PR it won't find the binary unless overridden. Following tsumugu.sh, something like mirror_clone=${MIRRORCLONE_BIN:-mirror-clone} would work, and the /home/mirror-clone bind-mount case could probably go (the PR description would then need a small update too).

Smaller things (non-blocking)

• GIT_REF=master makes image builds non-reproducible compared to the pinned CRATE_VERSION images; also --branch "$GIT_REF" won't accept a tag or commit SHA — --rev handles all of them.
• The runtime image installs git, which mirror-clone doesn't use, but not rsync, which the rsync source shells out to (src/rsync.rs:79).
• Totally fine to keep $TUNASYNC_MIRRORCLONE_OPTIONS / $TUNASYNC_MIRRORCLONE_ARGS unquoted for word-splitting (same as tsumugu.sh), but they'll also be glob-expanded against the mirror dir; a set -f before the exec would avoid surprises with * in args.
• As you noted, the image still needs wiring into docker-images.yml before it gets built.

Thanks again!

[yaoge123]

Round-2 review findings addressed in 7ac3aeb (also deployed to NJU production):

• MIRRORCLONE_BIN default: now prefers the documented deployment mount /home/mirror-clone, falling back to the wrapper-adjacent path, so the default works with the documented container layout.

• File buffer path: the buffer is now a separate sibling temp directory outside the target base (created before invoking the binary), satisfying the upstream file-backend contract that the buffer must not live inside the base path.

• Dockerfile workflow wiring: the token available to this account lacks the workflow scope, so I cannot push changes to .github/workflows/docker-images.yml myself. Here is the 3-spot patch that wires mirror-clone into all_images, the manual-dispatch allowlist, and the changed-path mapping — could a maintainer apply it (or I can push it once given a workflow-scoped credential)?

diff --git a/.github/workflows/docker-images.yml b/.github/workflows/docker-images.yml
index bc442d0..cb09679 100644
--- a/.github/workflows/docker-images.yml
+++ b/.github/workflows/docker-images.yml
@@ -47,7 +47,7 @@ jobs:
         run: |
           set -euo pipefail
 
-          all_images='["ftpsync","nix-channels","rubygems-mirror","rustup-mirror","tsumugu","tunasync-scripts"]'
+          all_images='["ftpsync","mirror-clone","nix-channels","rubygems-mirror","rustup-mirror","tsumugu","tunasync-scripts"]'
           if [ "$EVENT_NAME" = schedule ]; then
             images="$all_images"
           elif [ "$EVENT_NAME" = workflow_dispatch ]; then
@@ -59,7 +59,7 @@ jobs:
               IFS=','
               for image in $requested; do
                 case "$image" in
-                  ftpsync|nix-channels|rubygems-mirror|rustup-mirror|tsumugu|tunasync-scripts)
+                  ftpsync|mirror-clone|nix-channels|rubygems-mirror|rustup-mirror|tsumugu|tunasync-scripts)
                     selected="$selected $image"
                     ;;
                   *)
@@ -90,6 +90,7 @@ jobs:
                   selected="$selected tunasync-scripts"
                   ;;
                 dockerfiles/ftpsync/*) selected="$selected ftpsync" ;;
+                dockerfiles/mirror-clone/*) selected="$selected mirror-clone" ;;
                 dockerfiles/nix-channels/*) selected="$selected nix-channels" ;;
                 dockerfiles/rubygems-mirror/*) selected="$selected rubygems-mirror" ;;
                 dockerfiles/rustup-mirror/*) selected="$selected rustup-mirror" ;;

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate issues must be addressed before approval.

Review effort: Lite
Findings: 1 · 1

Open (2)

• Relative buffer directory is created inside the backend base · New
• Mirror-clone Dockerfile is excluded from image workflows

Resolved since last review (2)

• File buffer path is missing and nested inside the target base
• Default binary path conflicts with documented deployment mount

[yaoge123]

Fixed in 65496d2: TUNASYNC_WORKING_DIR is resolved to an absolute path ($(cd ... && pwd)) before the cd, and the buffer default is derived from that absolute path, so a relative working dir can no longer relocate the buffer inside the base.

[yaoge123]

Branch cleanup note: this branch has been squashed to a single commit, 0ba82f9. All commit SHAs referenced earlier in the review threads (up to 65496d2) are now orphaned commits — the links still open, but please rely on the current diff, whose tree is byte-identical to the previous head 65496d2.

Review items → how they were addressed (all included in the current diff)

Maintainer review (@happyaron, from the review body — no inline anchors, so answering here by name):

1. Buffer dir on a different filesystem → rename fails (EXDEV) → TUNASYNC_MIRRORCLONE_BUFFER is now configurable and defaults to a hidden dir inside the working dir ($TUNASYNC_WORKING_DIR/.mirror-clone-buffer). Rationale, documented in the script comment: the file backend moves finished downloads into place with rename(2) (src/file_backend.rs), which cannot cross filesystems, and tunasync's docker provider only bind-mounts the working dir — so the in-workdir default shares a filesystem with the base by construction. Upstream's recommendation to keep the buffer outside the base path is honored as an option: deployments mounting an extra volume can point the variable at an outside path on the same filesystem to keep the buffer out of the served tree.
2. Wrapper can't find the binary in the new image → the wrapper now follows tsumugu.sh exactly: mirror_clone=${MIRRORCLONE_BIN:-mirror-clone} with a plain PATH lookup. The /home/mirror-clone and beside-wrapper fallbacks are gone; the image installs the binary to /usr/local/bin/mirror-clone, so it is found by default.

Smaller items from the same review:

• GIT_REF=master non-reproducible / --branch rejects tags and SHAs → pinned to commit 02134d5 and built with cargo install --rev, which accepts sha/tag/branch.
• Runtime image had git but not rsync → now installs rsync (the rsync source shells out to it, src/rsync.rs), no git.
• Word-split pass-through variables also glob-expand → set -f before exec.
• docker-images.yml wiring → deliberately not included; the 3-spot patch is posted in the thread above and can be applied once the image contents are approved.

Copilot rounds:

• Round 1: unvalidated TUNASYNC_WORKING_DIR → explicit check with actionable error (later also resolved to an absolute path before cd); missing pipefail → set -euo pipefail; hard-coded binary path → MIRRORCLONE_BIN override (now PATH-based); unquoted pass-through variables → intentional word-splitting, kept (same interface as tsumugu.sh).
• Round 2: default binary path not matching the deployment mount → superseded by the PATH lookup above; buffer as a child of the base, never created → buffer is now created before exec and its location contract documented; Dockerfile not reachable from the image workflow → called out above and in the PR description.
• Round 3: relative TUNASYNC_WORKING_DIR relocating the buffer inside the base → TUNASYNC_WORKING_DIR is resolved to an absolute path first.

Two stale intermediate replies in the round-1 threads were already corrected in place (the TUNASYNC_WORKING_DIR check is present now; the binary lookup is PATH-based now).

Some earlier in-thread replies described intermediate states of the branch; please rely on the current diff and this summary.