feat(chartctl): Chart Deployments — remote EA deployment over HTTP - #10
Draft
Marinski wants to merge 19 commits into
Draft
feat(chartctl): Chart Deployments — remote EA deployment over HTTP#10Marinski wants to merge 19 commits into
Marinski wants to merge 19 commits into
Conversation
Stage .ex5/.set artifacts, declare deployments as desired state, and a resident loader EA reconciles the terminal's charts to it (attach EAs with set files, no RDP, no restart). API owns desired state; loader owns observed truth; success = observed converges on desired. - mt5api/chartctl/: paths, setparse, tpl_builder, registry, command - mt5api/handlers/chartctl.py: lock-free REST surface, gated on config - assets/experts/MT5ChartLoader.mq5 + include/ChartControl.mqh - Chart Control Protocol v1 (docs/chart-control-protocol.md) - 38 tests + Python fake_loader for Linux CI; config + README + CHANGELOG
- scripts/compile-chartctl-loader.bat: compile MT5ChartLoader in every broker base via MetaEditor64 on boot, propagate .ex5 to existing terminal instances (new ones inherit from base) - config_helper.py write_ini: emit [StartUp] Expert=Advisors\MT5ChartLoader (+Symbol with symbol_suffix, Period) gated on live mode + chartctl enabled + per-terminal override; new chartctl_enabled query cmd - start.bat: hook the compile before terminal launch; pass instance+mode through to write_ini - ChartControl.mqh: Init(close_own_chart_on_duplicate) — standalone duplicates self-close so re-fired [StartUp] never accumulates charts; embedded hosts keep the safe passive default - run.sh: sync the new script; docs/README/CHANGELOG updated
- ChartApplyTemplate never searches the GUI templates\ directory and resolves BOM-less relative paths against the calling EX5's folder; write .tpl files into MQL5\Files\chartctl\ and reference them as \Files\chartctl\<id>.tpl (leading backslash = <data>\MQL5 root, the only host-EA-independent search root). Fixes err 5019 on apply. - setparse: stop blind-trying utf-16 first — it "succeeds" on any even-length ASCII .set by pairing bytes into CJK garbage, parsing to zero inputs so a deployment silently runs on EA defaults. Decode by BOM, then NUL-sniff for BOM-less UTF-16, then utf-8. Regression tests for both. Verified end-to-end on the live rig 2026-07-12 (deployment reached running, screenshot confirmed).
- start.bat/install.bat: stamp the start.running/install.running lock
with the boot session id (Win32_OperatingSystem.LastBootUpTime in a
sibling .bootid file, keeping the lock dir empty so plain rmdir
release still works). A lock stamped by any other boot — or none —
is stale and removed: hard reboots (MT5AutoReboot, docker restart,
power loss) give the holder no cleanup window, and install.bat's
staged reboots only get a 5s shutdown window that TCG emulation can
miss.
- start.bat: read chartctl_enabled via tempfile instead of
for /f ('command') — cmd's quote-stripping mangles the subshell when
both python.exe and the script path are quoted and silently yields
nothing (same failure mode the api_token block documents).
- normalize 4 stray LF lines to the file's CRLF endings.
Two bugs found while verifying live deployments on build 5836: - AttachDeployment verified with `CHART_EXPERT_NAME != ""`, but MQL5 returns NULL (not "") for a chart with no expert, and NULL != "" is true — so the very first verify iteration passed and the loader reported a deployment "running" (and stamped the attribution comment) even when the expert never loaded. Test StringLen(en) > 0 instead. This is what masked the indexing issue below as success. - WriteObserved did FileDelete + FileMove(...,0); the delete fails 5020 whenever the API side has observed.json open for a read, then the move onto the still-present file also fails. Use FileMove with FILE_REWRITE and drop the pre-delete. Bumped CHARTCTL_VERSION to 1.0.1 so `GET /loader` reports which build is attached. Takes effect on the next terminal boot (compile-chartctl- loader recompiles the .mqh at startup). Note: separately confirmed MT5 does NOT index an .ex5 dropped into the data folder after the terminal launched (no FS-change notification over the shared mount; a CLI MetaEditor recompile doesn't trigger it either) — a freshly uploaded EA is only deployable after the next terminal restart. Documented for follow-up; not fixed here.
Set a terminal's WebRequest() allowed-URL list via the API instead of the
Options dialog. The list lives in Config/common.ini as an encrypted,
machine-independent blob; scripts/webrequest_allowlist_codec.py encodes/
decodes it (format reverse-engineered from terminal64.exe, verified
byte-identical against 18 real broker blobs).
Dedicated GET/PUT /webrequest call (not a deployment field) since URLs are
rarely needed and MT5 only reads the list at terminal startup:
- PUT persists the desired list per terminal, then restarts the terminal.
- restart_terminal re-emits common.ini while the terminal is down (MT5
rewrites it on exit, so the write must land after kill / before launch).
- config_helper.py write_ini re-emits it on every boot (start.bat deletes
common.ini), so the allowlist survives the periodic auto-restart.
First use migrates from the terminal's existing list, preserving
manually-configured URLs. Live-mode chartctl terminals only; no
start.bat/install.bat changes. UTF-16LE common.ini read/merge preserves all
other sections. Tests in tests/test_webrequest.py; full suite 218 passing.
run.sh copies the mt5api package but the standalone codec is a separate file under scripts/, and mt5api/chartctl/webrequest.py imports it by path from <shared>/scripts/. Without this copy the in-VM API's /webrequest handler fails to import the codec. Verified live: after syncing, PUT /webrequest encodes https://forexsb.com into the terminal's common.ini (decodes back correctly) and the allowlist persists in webrequest.json.
On the dockur-VM terminal build the WebRequest allowlist is not stored in common.ini — it lives in the machine-bound MQL5\experts.dat and MT5 drops it on every restart, so file injection can't provision it there. Set it the way a user would instead: a bundled portable AutoIt interpreter drives Tools -> Options -> Expert Advisors and types the URLs in. This takes effect immediately in-session (verified: a probe EA's WebRequest() returns HTTP 200 right after), and since MT5 forgets it on restart the API re-applies the persisted list ~25s after each terminal (re)start, so it survives the periodic auto-reboot. On bare metal where common.ini IS the store, it falls back to encoding the blob + restarting. - assets/autoit/: portable AutoIt3_x64.exe + set_webrequest.au3, plus inspect_options.au3 / selftest.au3 GUI-automation diagnostics. - mt5api/chartctl/autoit_webrequest.py: the applier. GUI applies are serialized host-wide by a named Windows kernel mutex (crash-safe via WAIT_ABANDONED) since desktop focus is a shared resource across terminals. - handlers/webrequest.py: PUT applies via AutoIt when available, else the common.ini + restart fallback; POST /webrequest/apply re-apply hook. - main.py: automatic, port-staggered boot re-apply. - README, CHANGELOG, tests.
…hart The chartctl:<id> chart-comment stamp does not survive MT5's profile save/restore, so after every terminal restart (including the periodic auto-reboot) the loader failed to recognize its own charts and opened a fresh duplicate per deployment — accumulating until the terminal's ~100-chart cap, where ChartOpen err=4105 froze all deployments at 'pending'. Observed live: 99 charts, 65 duplicates of one deployment. Loader v1.0.2 (ChartControl.mqh): - Reconcile first ADOPTS an unowned chart already running the exact expert + symbol + timeframe before opening a new one. Adoption — not the comment — is now the cross-restart attribution mechanism. - Comment stamps verified by read-back with retries (ChartSetString is asynchronous); a failed stamp aborts the attach. - A failed attach closes the chart it opened (previously an expert-less chart leaked per attempt) and backs off 60 s before retrying. - Errors tracked per deployment instead of one shared slot, so one deployment's failure no longer masks another's state. New cleanup primitive: POST /charts/<chart_id>/close -> loader close_chart command. Closes any chart by id, including charts the loader cannot attribute; the loader refuses to close its own chart (CLOSE_REFUSED). Used live to clear 92 leaked duplicates. Verified on the VM: both deployments converge via adoption after a full restart with no new charts opened; a further reboot no longer leaks. Docs, CHANGELOG, fake-loader + endpoint tests updated (225 passing).
Mid-session PUT /webrequest failed with options_not_found while boot re-applies worked — and worse, an apply could silently type URLs into the WRONG terminal. Root cause: the .au3 scripts located the MT5 main window by title substring (the login) over WinList(), but WinList() also returns hidden windows, and cloned terminals of the same account (instance routes) have byte-identical window titles, so the match was ambiguous. - autoit_webrequest.py resolves this terminal's terminal64.exe PID by executable path (WMI via PowerShell — reads elevated processes' paths, same pattern as restart_terminal's kill) and passes it as argv[2] to every script. - set_webrequest/inspect_options/selftest.au3 match only VISIBLE windows owned by that PID (title as tie-break), verify the Options dialog belongs to the same PID before driving it, and retry activation (WinWaitActive + 3x Ctrl+O). Legacy 3-arg invocation still works with title-fallback, so mixed old-API/new-script states stay functional. Verified live on all 4 terminals (2 same-login clone pairs): boot re-apply OK with requested pid == driven window pid after both a manual container restart and the scheduled 30-min auto-reboot, and the previously failing mid-session PUT now applies cleanly. Also: gitignore /terminals/ — running mt5api outside the VM scaffolds a terminal data tree at the repo root (config.py BROKERS_DIR); a real one must never be committed.
…eature/chart-deployments Brings in upstream changes v4.4.1 through v4.8.2: - MCP interface (v4.7.0, v4.8.0): typed tools + streamable-HTTP MCP server - Boot-lock hardening (v4.5.0, v4.6.0): acquire_lock.ps1, reboot.bat, make lint/format - CI/badges (v4.4.1, v4.8.1, v4.8.2): ClawHub publish, pipeline/badges - Keep chartctl additions: WebRequest auto-reapply, loader compile, write_ini instance/mode params, [Unreleased] changelog section
…cs, example script - README: proper route table, field reference for POST /deployments, GET /loader response, GET /charts response, video placeholder - CHANGELOG: fix double-separator from merge - SKILL.md: add Chart Deployments section with endpoint table + curl workflow - examples/chartctl/deploy.sh: full workflow script (stage, deploy, poll, screenshot)
Integrate upstream changes: - feat(claude-plugin): Claude Code plugin support (.agents/.claude-plugin/) - feat(codex-plugin): Codex plugin manifest (.agents/.codex-plugin/) - v4.9.x: unified MCP endpoint (mcpunifier/ service) - v4.8.x: docs, Agent integrations section, install fixes
Upstream v4.10.0+v4.10.1: multi-VM topology, handler contract tests, CI improvements, nginx resolver fixes, lazy jinja2 import. Resolved: - CHANGELOG.md: kept both unreleased (chartctl) and v4.10.0/v4.10.1 sections - test_config_generation.py: adapted startup-expert test for chartctl All other files auto-merged cleanly.
Integrates upstream v4.10.0/v4.10.1 (multi-VM topology, handler contract tests, CI improvements) into the chart-deployments branch. Resolved conflicts: - test_config_generation.py: adapted startup-expert test for chartctl's default [StartUp] section; added test for disabled-chartctl case All 281 tests pass.
Upstream v4.11.0: complete MCP range parity, Go client tests, docs restructure (concise README split into docs/*.md guides), CI updates. Resolved: - CHANGELOG.md: kept both unreleased (chartctl) and v4.11.0 - README.md: accepted upstream concise version, added chartctl content (What it ships bullet, docs table link, API at a glance section) - test_config_generation.py auto-merged cleanly
Marinski
force-pushed
the
feature/chart-deployments
branch
from
July 30, 2026 21:27
8a4b7da to
90905af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chart Deployments (chartctl)
Chart Deployments (chartctl) lets you attach Expert Advisors to charts with
.setfiles over the HTTP API—no RDP, no terminal restart, and no manual chart work.The API stores the desired state, while a resident Loader EA inside the terminal reconciles charts to match that state and reports the observed state back. A deployment only reaches the
runningstate after the Loader confirms the Expert Advisor is actually live on a chart.New Endpoints
POST/GET/DELETE/experts,/experts/<hash>.ex5filesPOST/GET/sets,/sets/<name>.setfiles (returns parsed inputs)POST/GET/deploymentsGET/PATCH/DELETE/deployments/<id>POST/deployments/reconcileGET/chartsGET/loaderPOST/charts/<id>/screenshotPOST/charts/<id>/closeGET/PUT/webrequestPOST/webrequest/applyWhat's Included
mt5api/chartctl/— paths, setparse, tpl_builder, registry, commandmt5api/handlers/chartctl.py— lock-free REST surface, gated byCHARTCTL_ENABLED(config-driven; routes are omitted when disabled)mt5api/handlers/webrequest.py— WebRequest allowlist provisioningmt5api/chartctl/autoit_webrequest.py— AutoIt GUI automationassets/experts/MT5ChartLoader.mq5— reference Loader EA (~60 lines)assets/experts/include/ChartControl.mqh— portable protocol includeassets/autoit/— AutoIt interpreter and GUI automation scriptsscripts/compile-chartctl-loader.bat— zero-touch bootstrap compilerscripts/config_helper.py—write_iniemits[StartUp] Expert=and provides thechartctl_enabledquery commanddocs/chart-control-protocol.md— complete protocol specificationtests/test_chartctl_units.py+tests/test_chartctl_endpoints.py— 43 tests plus a Pythonfake_loaderfor Linux CI (no MT5 dependency)tests/test_webrequest.py— 35 tests covering the allowlist codec and endpointsexamples/chartctl/deploy.sh— runnable deployment exampleZero-Touch Bootstrap
No RDP step is required.
On VM boot:
compile-chartctl-loader.batcompiles the Loader in every broker base using MetaEditor64..ex5is propagated to every terminal instance.config_helper.py write_iniadds a[StartUp]section tomt5start.ini.The Loader automatically attaches when the terminal starts.
To disable this behavior for a specific terminal, set:
How It Works
The Loader polls
desired.json, reconciles the desired state with actual charts, writesobserved.jsonso the API knows what is actually running. All handler endpoints are lock-free and perform only file I/O.WebRequest Allowlist
Two application methods selected automatically at runtime: AutoIt GUI automation (Windows VM) or
common.iniencoding (Bare Metal).Changelog
See the
CHANGELOG.mdUnreleased section for the complete list of changes.Testing
Note: This PR was recreated after a fork deletion closed the original (#7). Depends on #9 (multi-VM) being merged first.