feat(multi-vm): config-driven N-VM topology

[Marinski]

Declarative N-VM topology via vms.yaml — each terminal gets an optional vm field in config.yaml, nginx routes to the correct container, per-VM concurrency caps via MT5_HTTPAPI_MAX_IN_FLIGHT_<NAME>.

Backward compatible: no vms.yaml = single-VM mode, no vm field = routes to mt5. See docs/multi-vm-setup.md for the full walkthrough.

[psyb0t]

Reviewed this properly — it's good work. Backward compatible by construction (no vms.yaml → single-VM, no vm: field → mt5), the docker-compose.yml.j2 approach is the right call over hand-maintaining compose, and docs/multi-vm-setup.md is genuinely useful. Two things need fixing before it can land, one of them serious.

1. Blocker — a single VM being down takes the whole stack offline

scripts/config_helper.py generates the terminal routes with a literal upstream:

f"            proxy_pass http://{container}:{t['port']};\n"

nginx resolves literal upstream hostnames at config-parse time, not request time. In single-VM this was harmless — if mt5 is gone nothing works anyway. With N VMs it's a different failure: any one container being absent stops nginx from starting entirely, so every healthy VM's routes, the REST API and /mcp/ all go down with it.

Tested it on a real docker network with one VM container running and only the other absent:

literal proxy_pass    → [emerg] host not found in upstream "prcheck-vm-down"
                        nginx: configuration file test failed
resolver + variable   → nginx: configuration file test is successful

The running VM's route dies along with the missing one.

depends_on doesn't cover this — it orders startup but doesn't help when a VM is stopped for maintenance, gets OOM-killed, or hasn't been created yet during a staged rollout. And with restart: unless-stopped, nginx trying to come back while any VM is down fails permanently.

This repo already hit exactly this in v4.9.1 with the mcpunifier route, and the fix is the same three lines applied to the terminal loop:

f"        location {p} {{\n"
f"            resolver {DOCKER_EMBEDDED_DNS} valid=10s ipv6=off;\n"
f"            set $vm_upstream http://{container}:{t['port']};\n"
f"            rewrite ^{p}(.*)$ /$1 break;\n"
f"            proxy_pass $vm_upstream;\n"

Deferring the lookup to request time means nginx starts regardless, and only the routes belonging to the down VM return 502.

Side effect worth noting: docs/multi-vm-setup.md troubleshooting says "Nginx 502 for a terminal: check the generated nginx.conf" — with the literal form you never get a 502, nginx is simply dead. That line becomes correct once the resolver fix is in.

2. vms.yaml ships as live config, with your hardware in it

It's committed as vms.yaml rather than vms.yaml.example, and the PR adds no .gitignore entry, so it becomes the default config for everyone. The convention in this repo is:

config/config.yaml       gitignored     config/config.yaml.example       tracked
docker-compose.yml       gitignored     docker-compose.yml.example       tracked

Right now it carries cpuset: "0-19,40-59", ram: "112G" ×2, /data/mt5-vm-a/storage, /mnt/mt5-hot, and 22 extra_binds lines naming your darwinex accounts. Anyone cloning and running run.sh without an existing compose file generates one demanding ~224 GB RAM, 80 pinned cores and paths that don't exist — and it publishes your account layout, which I'm guessing wasn't intentional.

Could you rename it to vms.yaml.example, add vms.yaml to .gitignore, and ship a modest two-VM sample?

Minor

config_helper.py pip-installs jinja2 on ImportError with both stdout and stderr to DEVNULL, so a failed install surfaces as a confusing ImportError instead of the actual error. It matches the existing pip3 install --quiet pyyaml pattern in run.sh so it's consistent — just noting it.

---

Fix those two and this is good to merge. #10 looks solid on a read-through too — the desired/observed reconciliation design is the right shape, and the Python fake_loader so it tests on Linux CI without MT5 is a nice touch. Happy to take that one once this lands.

[Marinski]

Addressed all three points:

1. Nginx resolver for terminal routes - switched from literal proxy_pass http://{container}:{port} to resolver + variable, exactly the same pattern the mcpunifier route already uses in v4.9.1. Nginx starts regardless of whether every VM container is present; only the absent VM's routes return 502.

2. vms.yaml.example - renamed from vms.yaml to vms.yaml.example with a modest two-VM sample (8G RAM, 4 cores, no extra_binds). Added vms.yaml to .gitignore. _load_vms() falls back to vms.yaml.example when the real file is absent, so run.sh and generate_compose work out of the box.

3. pip install error messages - wrapped the auto-install blocks in try/except to catch CalledProcessError and print a clear instruction (pip install pyyaml/jinja2 manually) instead of a cryptic ImportError when pip fails.

[psyb0t]

Fixes verified — all three land, and I ran the branch rather than just reading it.

• nginx resolver. nginx_conf now emits resolver 127.0.0.11 valid=10s ipv6=off + set $vm_upstream + proxy_pass $vm_upstream on every terminal route, reusing the existing DOCKER_EMBEDDED_DNS constant instead of hardcoding. Tested the generated config with zero upstream containers present: nginx -t passes. Regenerated the same config in the old literal form as a control and it still dies with [emerg] host not found in upstream "mt5". That failure mode is gone.
• vms.yaml.example. Rename, .gitignore entry and the FileNotFoundError fallback all work — config_helper.py vms reads the example out of the box.
• pip errors. CalledProcessError now surfaces a real instruction instead of a bare ImportError.

generate_compose also renders both VMs correctly — cpusets, distinct noVNC ports, deploy.resources.limits.memory — and docker compose config validates clean.

One thing has to come out before this can land.

79ec45be is carrying Chart Deployments code

The commit is titled "nginx resolver + vms.yaml.example from PR #9 review", but it also adds 14 lines of chartctl to write_ini that aren't mentioned in the message, the PR title, or the description. Two pieces, and the dangerous one isn't the one that looks dangerous.

if mode == "live" and chartctl_on and term_override is not False:
    ini += "[StartUp]\n"
    ini += "Expert=Advisors\\MT5ChartLoader\n"
    ini += f"Symbol=EURUSD{term_suffix}\n"

chartctl_on defaults to True (chartctl_cfg.get("enabled", True)), and that block is not inside a try. So merging #9 writes a [StartUp] stanza attaching MT5ChartLoader into every live terminal's INI at launch — and MT5ChartLoader doesn't exist in master. Neither does mt5api/chartctl/; I checked the full tree. That's a startup behavior change on every live terminal, on by default, pointing at an EA that isn't there.

The second piece — the WebRequest allowlist boot-seed — does from mt5api.chartctl import webrequest, which always raises ImportError against master and gets caught by except Exception # non-fatal: never block terminal launch. Dead code today, and silently so, which is its own problem: nobody would find out it never ran.

This reads as branch contamination rather than intent — it's the calling half of a feature with the implementing half missing, which is what you get when both branches are checked out locally.

Can you rebase those 14 lines out so #9 is purely multi-VM? The loader bootstrap and the allowlist seed belong in #10, landing next to the module and the EA that make them work.

[Marinski]

All three points addressed. Chartctl contamination rebased out of PR #9 - the [StartUp] stanza, the from mt5api.chartctl import webrequest block, and the chartctl_enabled command are removed. PR #9 now carries only multi-VM changes (3 commits). The chartctl bootstrap lands in PR #10 alongside the module and EA.

[psyb0t]

Contamination's gone and the fixes survived the rebase — re-verified on e9a8621 rather than taking the diff's word for it:

• nginx -t on the generated config with zero upstream containers present: passes. The literal-proxy_pass control still fails with [emerg] host not found in upstream under identical conditions, so that's the actual fix working, not a vacuous pass.
• write_ini for a live terminal emits zero [StartUp] / MT5ChartLoader lines now.
• generate_compose renders both VMs and docker compose config validates clean.
• No chartctl / webrequest additions anywhere in the diff, and dropping the chartctl_enabled command is correct — master has no chartctl at all, so it came in with the same contamination.

Good work on this one. The docker-compose.yml.j2 approach was the right call, the backward compat is sound by construction, and docs/multi-vm-setup.md is genuinely useful.

Approving.