feat(car): volume-sync provider/consumer for cross-node state replication

[toderian]

Summary

Adds volume-sync to the Container App Runner — a provider/consumer mechanism that lets one CAR instance snapshot its app's persistent state and have other CAR instances apply that snapshot, enabling cross-node state replication for containerized apps. The feature is implemented as a self-contained sync/ subpackage and a _SyncMixin woven into ContainerAppRunnerPlugin.

Companion PR (UI side, deeploy-dapp): Ratio1/deeploy-dapp#96.

What it does

When an app deployed via CAR opts in, the runner orchestrates state replication around the container lifecycle:

Provider (the source-of-truth node)

1. App writes /r1en_system/volume-sync/request.json listing the container paths it wants snapshot-published.
2. CAR polls (SYNC.POLL_INTERVAL, default 10s), claims the request atomically, stops the container, archives the requested paths (resolving container-absolute paths to host bind-mount paths via self.volumes), uploads the bundle to R1FS, and publishes {cid, version, timestamp} to ChainStore under SYNC.KEY.
3. Container is restarted, app sees the response in volume-sync/response.json and the published metadata persisted under <plugin_data>/sync_history/sent/.

Consumer (a node that wants to mirror provider state)

1. CAR polls ChainStore (same cadence) for the same SYNC.KEY.
2. On a new CID (identity is CID, not version — see "design notes" below), CAR stops the container, fetches the bundle from R1FS, extracts the archive members to the matching host mount paths, writes a volume-sync/last_apply.json describing the applied state, restarts the container.
3. App sees the freshly populated paths on next boot.

Both sides record a full audit trail per instance under <plugin_data>/sync_history/{sent,received}/<version>__<short_cid>.json so operators can inspect any past replication event.

Always-on system volume

Independent of the SYNC role, every CAR instance now mounts a 10M ext4 loopback at /r1en_system (provisioned via the same FIXED_SIZE_VOLUMES machinery as user-declared fixed-size volumes). The mount carries the control-plane files (volume-sync/request.json, response.json, etc.). Apps that don't use SYNC simply don't write into it; the volume costs ~10MB per instance and never sees data traffic.

Env vars exported to the container:

• R1_SYSTEM_VOLUME — always
• R1_VOLUME_SYNC_DIR — always
• R1_SYNC_REQUEST_FILE — always (so apps can write the request unconditionally; CAR just won't act on it without SYNC.ENABLED)
• R1_SYNC_TYPE, R1_SYNC_KEY — only when SYNC is enabled, for apps that want to branch on role

CAR config block

Added under ContainerAppRunnerPlugin._CONFIG:

"SYNC": {
    "ENABLED": False,         # master switch — default off, no SYNC traffic
    "KEY": None,              # shared UUID across the sync set (provider + consumer)
    "TYPE": None,             # "provider" | "consumer"
    "POLL_INTERVAL": 10,      # seconds between sync ticks
}

Default-off, wire-compatible with existing pipelines.

Code layout

The whole feature lives in extensions/business/container_apps/sync/:

• constants.py — file names, ChainStore hkey, manifest schema version, failure-stage labels. No code, just data — cat sync/constants.py gives the entire data-plane vocabulary in one place.
• manager.py — SyncManager class. Pure I/O orchestration: claim → archive → upload → publish (provider), fetch → validate → extract → record (consumer). Delegates network/storage to owner.r1fs / owner.chainstore_* so it stays testable. ~860 lines.
• mixin.py — _SyncMixin. Plugin-class integration: provisions the system volume in on_init, recovers any orphaned request.json.processing from a prior crash, drives provider/consumer ticks from _perform_periodic_monitoring. Both ticks call stop_container → manager_work → start_container inline (must NOT route through _restart_container, which unmounts the system volume before the sync slice can read from it).
• __init__.py — re-exports the public surface so callers can from extensions.business.container_apps.sync import SyncManager.

Tests: extensions/business/container_apps/tests/test_sync_manager.py (866 lines) and test_sync_mixin.py (390 lines) cover happy paths, every documented failure stage, and the crash-recovery flow.

Design notes worth flagging for review

1. Identity is the CID, not the version. The consumer compares ChainStore's cid against the last applied CID. version is informational (timestamp-derived, kept for filename ordering and human-readable logs). Comparing CIDs eliminates a whole class of clock-skew failure modes — a provider with a wonky timestamp can never make a consumer permanently ignore a corrected snapshot — and makes multi-provider sync sets coherent without ordering assumptions. (Final commit 75e48a7 made this switch; earlier commits compared on version.)

2. Sync slices bypass _restart_container. _restart_container calls _cleanup_fixed_size_volumes, which would unmount the /r1en_system loopback before the sync code can read or write into it. The mixin ticks inline stop_container() → work → start_container() and never set a StopReason that would route through the restart path. This is documented at the top of mixin.py.

3. The system volume is always-on, even with SYNC disabled. The mount is cheap (10M ext4 loopback) and lets us add future CAR↔app control-plane features without re-architecting volume provisioning. The volume-sync/ subdirectory naming convention reserves space for those future features.

4. INITIAL_SYNC_TIMEOUT was removed. Earlier iterations blocked the consumer's first boot until a snapshot arrived. Final design (0882dd4) starts consumers immediately on an empty volume and lets the next tick apply whatever ChainStore has — apps that strictly require state at startup must implement their own poll-and-retry in their entrypoint. This matches how every other CAR feature degrades.

5. Permissions: /r1en_system is chmod 0o777. The system volume is per-CAR-instance and the app already owns the rest of its container; restricting the control-plane subdir to root would just mean every non-root app silently can't write the request file. See bcf3193 / 4201532 for context.

Test plan

Unit tests cover the full SyncManager and _SyncMixin surfaces (over 1250 lines of tests across the two files). For integration:

• Deploy a CAR app with SYNC.ENABLED=False (or no SYNC block) → confirm /r1en_system mounts and CAR runs to completion with no SYNC log lines
• Deploy a provider with SYNC.ENABLED=True, TYPE='provider', KEY=<uuid> → write a request.json from inside the container, observe stop → publish → start log sequence, confirm record appears in ChainStore under the KEY and in sync_history/sent/
• Deploy a consumer on a different node with the same KEY → confirm it picks up the latest CID, stops → applies → starts, and sync_history/received/ reflects the apply
• Kill a provider mid-publish (during the upload step) → confirm next tick recovers via the orphan .processing rename and republishes successfully
• Roll multiple provider snapshots back-to-back → confirm consumer applies only the newest CID and skips the intermediate ones
• Publish a snapshot containing paths that don't match the consumer's bind-mounts → confirm consumer fails gracefully with a stage-labeled error (no container thrash)
• Restart a node mid-apply → confirm the consumer's next tick re-applies cleanly

Commits

a5bc2bd skeleton → 9cc09a4 path resolver + atomic JSON + history I/O → cc2df3e claim_request + archive helpers → 2fe9ffe orchestrators → d9948b3 mixin → ff9e73b plugin wiring → 4201532 + bcf3193 permissions → 01e7aa9 cleanup-mount fix → 75e48a7 CID identity → 4aed83f collapse into sync/ subpackage → 0882dd4 remove INITIAL_SYNC_TIMEOUT → bbe20a4 version bump.