Wire up control-plane ↔ data-plane communication in the Docker stack

[Lang-Akshay]

Summary

This PR extends the local Docker Compose stack so the Rust data-plane (contextforge-gateway-rs) runs alongside the Python control-plane (mcp-context-forge), backed by Postgres/PgBouncer, and fronted by a single nginx entrypoint that routes between them.

Closes IBM/mcp-context-forge#5803

PRD

Problem
Iterating on the data-plane locally required manual image builds and piecemeal docker compose invocations, with no easy way to bring up the dependent control-plane stack (Postgres, PgBouncer, Redis, control-plane, nginx) end-to-end.

Goal
One command builds the data-plane image, one command brings up the whole stack, one command tears it down — so the dev/test loop for the data-plane is fast and repeatable.

Requirements (from the issue)

1. make docker-prod builds the data-plane production image (dataplane:latest) from docker/Dockerfile.
2. make testing-up brings up the full local stack via docker/docker-compose.yml, and fails fast with a clear message if dataplane:latest isn't built yet.
3. make testing-down stops the stack.
4. docker-compose.yml/nginx.conf updated so the stack is actually testable end-to-end:
   • Consistent network naming (mcp-gateway-net → contextforge-gateway-net).
   • data-plane service builds from the local image instead of a ghcr.io tag.
   • control-plane + one-shot migration + postgres + pgbouncer added so the data-plane has a real control-plane to talk to.
   • Healthchecks + service_healthy dependency gating so services don't race each other on startup.
   • nginx routes split correctly between data-plane and control-plane container names.

Services make testing-up brings up

Service	Role
nginx	single entrypoint, routes /contextforge-rs/* → data-plane, everything else → control-plane
data-plane	the Rust gateway under test, built from local dataplane:latest
control-plane	Python mcp-context-forge control-plane — config, policy, identity owner
migration	one-shot Alembic upgrade/bootstrap, runs before control-plane (implicit dependency, not listed explicitly on the Make target)
postgres	control-plane's database
pgbouncer	connection pooler in front of postgres
redis	shared cache/session store used by both planes
fast_time_server	test MCP server used to exercise the stack end-to-end
register_fast_time	one-shot job that registers fast_time_server with control-plane over its admin API

Resource limits (make testing-up)

Service	CPU Limit	Mem Limit
nginx	0.5	256 M
control-plane	2	2 G
redis	1	1 G
postgres	2	2 G
pgbouncer	0.5	256 M
data-plane	2	2 G
fast_time_server	1	512 M
Total	9 cores	8 G

Not part of this target: contextforge-gateway-one/two and mcp-counter-tool-one/two remain separate, pre-existing demo services untouched by testing-up.

Stack topology (make testing-up)

flowchart LR
    client(["client"]) --> nginx["nginx"]
    nginx --> dataplane["data-plane"]
    nginx --> controlplane["control-plane"]
    dataplane --> redis["redis"]
    controlplane --> redis
    controlplane --> postgres["postgres\n(via pgbouncer)"]
    dataplane --> fastts["fast_time_server"]

Loading

Out of scope

• Production deployment/orchestration (this is a local dev/test stack only).
• CI wiring for the new Make targets.

Testing (from the issue)

• make docker-prod builds image successfully.
• make testing-up brings up all services healthy.
• make testing-down stops stack cleanly.

Changes

docker/docker-compose.yml

• Renamed network mcp-gateway-net → contextforge-gateway-net and applied it consistently across all services.
• Renamed gateway service contextforge-gateway-rs → data-plane, now built from the local dataplane:latest image (${IMAGE_LOCAL:-dataplane:latest}) instead of a pre-built ghcr.io image.
• Added new environment variables to the data-plane services: CONTEXTFORGE_GATEWAY_RS_REDIS_CONNECTION_MODE, CONTEXTFORGE_GATEWAY_RS_UPSTREAM_CONNECTION_MODE, and renamed the token-verification key env vars to the CONTEXTFORGE_GATEWAY_RS_* prefix.
• data-plane now waits on redis and control-plane via service_healthy (previously just service_started on redis).
• Added container_name to every service for easier docker ps / log inspection.
• Added new services to bring up the control-plane stack:
  • postgres — Postgres 18 with tuned connection/memory settings, healthcheck, and a named pgdata volume.
  • pgbouncer — connection pooler in front of Postgres (transaction pooling, sized pool limits).
  • migration — one-shot init container that runs Alembic migrations + bootstrap directly against Postgres (bypasses PgBouncer since advisory locks are session-scoped) before control-plane starts.
  • control-plane — the Python mcp-context-forge control-plane image, wired to Postgres (via PgBouncer), Redis, and JWT keys; gated on migration completing successfully and pgbouncer/redis being healthy.
  • fast_time_server — test MCP server used for exercising the SSO/monitoring/testing profiles.
  • register_fast_time — one-shot job that mints an admin JWT and registers fast_time_server with the control-plane over its admin API.
• Added a healthcheck to redis and switched its network to contextforge-gateway-net.

docker/nginx.conf

• Removed the old round-robin upstream contextforge-gateway-rs block.
• Added a map for $contextforge_control_backend_url pointing at control-plane:4444, alongside the existing data-plane backend map (now pointing at data-plane:4445 instead of contextforge-gateway-rs:4445).
• Added a catch-all location / block that proxies everything not matched by /contextforge-rs, /health, or /nginx_status to the control-plane, with the same rate-limiting/timeout/proxy-header conventions used elsewhere in the file.

docker/Dockerfile

• Build the release binary with --features contextforge-gateway-rs-lib/with_tools so the local image includes the tools feature set used by the compose stack.

Makefile (new)

• make docker-prod — builds the dataplane:latest image from docker/Dockerfile.
• make testing-up — brings up the full testing stack (nginx, control-plane, redis, postgres, pgbouncer, data-plane, fast_time_server), refusing to start if dataplane:latest hasn't been built yet.
• make testing-down — stops the stack.
• make help — self-documenting target list.

Test plan

• make docker-prod builds dataplane:latest successfully.
• make testing-up brings up the full stack and all healthchecks pass (postgres, pgbouncer, redis, control-plane, data-plane).
• migration container runs to completion before control-plane starts.
• nginx routes /contextforge-rs/* to data-plane and everything else to control-plane.
• register_fast_time successfully registers fast_time_server against the control-plane admin API.

[gandhipratik203]

Nice work on this. The docs (topology diagram, resource table, troubleshooting matrix) make it easy to follow, and the dependency gating / RS256 wiring all check out. Two medium items worth a look before merge:

🟡 Medium

M1 · AUTH_ENCRYPTION_SECRET defaults diverge — migration and control-plane fall back to different values. Harmless today (bootstrap_db never encrypts with it, just hashes the admin password and seeds roles), but it breaks the moment anything is encrypted at rest. Worth aligning the defaults, or dropping the var from migration since it doesn't use it.

M2 · PgBouncer transaction pooling + psycopg3 prepared statements — if the control-plane image doesn't disable server-side prepared statements, transaction mode can throw intermittent "prepared statement already exists" errors under concurrency. The register flow is too low-volume to trip it, so no action needed for the local loop, but worth confirming prepare_threshold=None (or equivalent) before driving real load through this stack.

[lucarlig]

Requesting changes based on clean, isolated Docker/Make validation:

1. P1 — Build the checked-out source. make docker-prod passes the workspace as build context, but the Dockerfile clones remote main and only copies .cargo/config.toml. The PR build therefore compiled the base branch instead of the PR SHA, and publication is not pinned to the workflow checkout. Please copy and build the local workspace.

2. P2 — Fix the nginx health route. /health is forwarded unchanged and returns 404; the dataplane health endpoint is /contextforge-rs/health. Please rewrite or proxy to that path.

All Make targets otherwise work: the full stack became healthy, Fast Time registered 7 tools, MCP initialize/tools/list passed, and testing-down stopped the stack. The latest image publication also succeeded. Confirmed the runtime path is dataplane → Fast Time.

[Lang-Akshay]

Thanks allot for request @gandhipratik203 and @lucarlig
All the recommend fix are in place

• M1 · AUTH_ENCRYPTION_SECRET defaults diverge ✅
• M2 · PgBouncer transaction pooling + psycopg3 prepared statements ✅
• P1 — Build the checked-out source ✅
• P2 — Fix the nginx health route ✅

[gandhipratik203]

Nice work, this looks good to me.

[lucarlig]

LGTM