diff --git a/apps/README.md b/apps/README.md index 728ea30d..d49c5828 100644 --- a/apps/README.md +++ b/apps/README.md @@ -12,3 +12,4 @@ Javascript and Python apps and demos showcasing how to use MongoDB in GenAI appl | Voice OpenAI Rentals Agent | MongoDB Atlas, Python, Flask, OpenAI | Voice interaction agent for real estate listings using vector search and real-time voice processing | [![View Code](https://img.shields.io/badge/view-code-blue?logo=github)](voice-openai-mongo-rentals-agent) | | Video Intelligence App | MongoDB Atlas, FastAPI, React, OpenAI, VoyageAI | AI-powered video search system allowing users to upload videos, extract insights, and search content using natural language queries | [![View Code](https://img.shields.io/badge/view-code-blue?logo=github)](video-intelligence) | | Voice AI Agents with Memory | MongoDB Atlas, VoyageAI, Gemini Live | Voice AI Agents powerd with MongoDB Hybrid searchable memory| [![View Code](https://img.shields.io/badge/view-code-blue?logo=github)](voice-memory-demo) | +| Remote MCP Performance Triage | MongoDB Atlas, Python, MongoDB Remote MCP, ChatGPT Workspace Agents | A ChatGPT Workspace Agent triages a simulated PagerDuty incident end-to-end over Remote MCP — finding the slow query, consulting Performance Advisor, and proposing the missing index for human approval | [![View Code](https://img.shields.io/badge/view-code-blue?logo=github)](remote-mcp-perf-triage) | diff --git a/apps/remote-mcp-perf-triage/.gitignore b/apps/remote-mcp-perf-triage/.gitignore new file mode 100644 index 00000000..ce38d252 --- /dev/null +++ b/apps/remote-mcp-perf-triage/.gitignore @@ -0,0 +1,6 @@ +__pycache__/ +*.py[cod] +.env +.venv/ +load.log +checkout.log diff --git a/apps/remote-mcp-perf-triage/README.md b/apps/remote-mcp-perf-triage/README.md new file mode 100644 index 00000000..fdba1c07 --- /dev/null +++ b/apps/remote-mcp-perf-triage/README.md @@ -0,0 +1,282 @@ +# Remote MCP — Performance Triage Demo + +A live demo of a ChatGPT Workspace Agent triaging a production performance incident +end-to-end through the **MongoDB Remote MCP** plugin. A simulated PagerDuty incident +triggers the agent through the Workspace Agents API. The agent connects to Atlas, +pinpoints the offending collection and query, consults the Atlas **Performance +Advisor**, proposes the missing index, and verifies the approved fix. + +## The story + +**Setup (world before the incident):** An e-commerce app uses MongoDB Atlas as the +system of record for payments. After a user clicks *Pay*, the checkout page polls +MongoDB until the payment status flips to `completed`: + +```js +db.payments.findOne({ session_id: "sess_...", status: "completed" }) +``` + +The `payments` collection has grown to hundreds of thousands of documents, and there is **no +index on `session_id`**. Every poll is a full collection scan. + +**Incident:** Checkout hangs on "Processing your payment…" and times out. A Datadog +monitor observes an elevated payment-confirmation timeout rate, causing +PagerDuty to open a SEV-2 incident. The incident contains application symptoms but +does *not* identify MongoDB, a collection, a query, or an index. Pinpointing the +database cause is the agent's job. + +**Triage (live, via MCP):** +1. Agent inspects the slow query and runs `explain()` → `COLLSCAN`, every document examined (~5 s). +2. Agent consults the **Performance Advisor** → confirms a missing index on `session_id`. +3. Agent proposes the index `{ session_id: 1, status: 1 }` and waits for approval. +4. After approval, the agent creates the index and re-runs `explain()` → `IXSCAN`, + ~1 doc examined, sub-millisecond. Checkout recovers. + +**Takeaway:** a conversational agent traversed from a *business symptom* (failed +checkouts in PagerDuty) to a *database root cause* and fix — cross-layer triage that +would normally require a developer who knows exactly where to look. + +## What's real vs. staged + +- **Real:** the seeded collection, the slow query, the `explain()` output, the + Performance Advisor recommendation, and the index creation. The agent does + genuine work against a live M10 cluster over MCP. The checkout page's hang is also + real — it runs the same unindexed poll query and genuinely exceeds its timeout. +- **Staged:** PagerDuty itself. `trigger_chatgpt.py` acts as the adapter that sends a + PagerDuty incident resource to the Workspace Agents API. The incident contains + application symptoms, but no database namespace, query shape, root cause, or index + recommendation. Also staged: the payment processor's confirmation webhook, which + `checkout_app.py` simulates with a ~2 s delay, and PagerDuty's on-call resolution — + the assignee is a fixed name, not resolved from a real schedule. + +## Files + +| File | Purpose | +|------|---------| +| `checkout_app.py` | The Leafy Electronics checkout page. Really hangs on the slow poll, then fires the incident to ChatGPT. | +| `seed_payments.py` | Seeds a large, realistic `payments` collection (no index on `session_id`); `--drop-index` resets the demo. | +| `generate_load.py` | Runs the checkout status-poll query repeatedly to feed Performance Advisor. | +| `trigger_chatgpt.py` | Sends a realistic PagerDuty-style incident to a published ChatGPT Workspace Agent and prints the conversation URL. | +| `trigger_slack.py` | Posts that *same* incident to a Slack channel, on demand — shows the fan-out to a second surface. | +| `requirements.txt` | `pymongo`, `python-dotenv`, and FastAPI/uvicorn for the checkout page. | + +## Prerequisites + +- An Atlas **M10** cluster (dedicated tier — required for the full Performance Advisor). +- A **read-write database user** + connection string for the seed/load scripts. + This is a direct MongoDB connection, separate from the Remote MCP OAuth grant. +- The **Remote MCP** plugin connected to the same Atlas org/project, with MCP access + enabled for the org (Org Admin → App Authorizations). +- A published ChatGPT Workspace Agent with an API channel and the MongoDB MCP plugin. +- A Workspace Agent access token. An admin must enable Workspace Agents and + **Allow users to create personal access tokens** under Admin > Permissions & roles. +- Optional: a **Slack incoming webhook** URL if you also want the alert shown in Slack. + +## Setup + +```bash +pip install -r requirements.txt +``` + +Add the direct database connection used by the seed and load scripts to `.env`: + +```dotenv +MONGODB_URI="mongodb+srv://:@cluster0.<...>.mongodb.net/" +``` + +This URI must point to the same Atlas cluster exposed to the Workspace Agent through +MongoDB MCP. + +### 1. Seed the data + +```bash +python seed_payments.py # ~300,000 docs, ~1.6 KB gateway payload each +``` + +**Sizing (measured on M10, 2 GB RAM):** 300k docs of ~2 KB is the sweet spot. A +COLLSCAN of the poll query runs **~9 s cold** and settles to **~5 s warm** — clearly +slow and dramatic in `explain()`, yet safely under the MongoDB MCP server's **60 s +`maxTimeMS` cap**. Do **not** seed millions: a scan that large can exceed the 60 s +cap, which makes the agent's `explain()`/`find()` **error out** during the demo +instead of returning stats. Bigger is worse, not better. + +**The scan is only slow if the collection outgrows the WiredTiger cache.** That cache +is ~50% of host RAM, so a 2 GB M10 gives ~537 MB against this collection's ~0.63 GB — +scans hit disk and take seconds. On a larger tier the whole collection fits in cache +and the same query returns in **~200 ms**, which quietly kills the demo's drama (it +was measured at 222 ms on a 4 GB host). Check `hostInfo.memSizeMB` if the scan comes +back suspiciously fast; the fix is a smaller tier, not more documents. + +Document size lives in a realistic `gateway_response` field (an opaque base64 +payload — screenshot-safe, and high-entropy so WiredTiger's compression can't shrink +the collection and let it be served entirely from cache). + +### 2. Keep the slow-query pattern fresh (trickle load) + +Performance Advisor builds its recommendation from slow queries in a rolling ~24h +window. The recommendation only persists while slow queries **keep recurring** AND +the index is **still absent** — so keep a gentle trickle running until showtime, and +**do not create the index before the demo**. + +```bash +python generate_load.py # default trickle: 3 queries every 5 min, until Ctrl+C +``` + +Recency matters more than volume, and the M10 has a burstable CPU, so a light trickle +is enough (and kinder to the cluster) — no need to hammer it. To run unattended: + +```bash +nohup python generate_load.py > load.log 2>&1 & # background; tail -f load.log to watch +``` + +Notes: +- Give Performance Advisor ~15–30 min of traffic to first surface the recommendation. +- Warm the cache before demoing (let the trickle run a bit) so scans are ~5 s, not ~9 s. +- Confirm readiness with `atlas-get-performance-advisor` (expect a suggested index on + `{ session_id: 1, status: 1 }` for `ecommerce.payments`). + +## Running the demo + +### The checkout page (recommended opening) + +`checkout_app.py` gives the demo a visual first act with no terminal on screen. It +serves a Leafy Electronics checkout that runs the **real** status-poll query against the **real** +unindexed collection — so the hang isn't staged, it's the bug: + +```bash +python checkout_app.py # http://127.0.0.1:8000 +python checkout_app.py --no-incident # rehearse without paging the agent +``` + +Click **Submit payment**. A real `pending` payment is inserted, a simulated processor +confirms it ~2 s later, and the page polls for the confirmation. Pre-index each poll is +a ~6 s COLLSCAN, so the page blows its 10 s budget and fails — then fires the PagerDuty +incident to the Workspace Agent automatically. The status panel shows each poll's +latency, so the audience sees *why* it hung. + +After the agent's index is approved, click **Submit payment** again: polls drop to +~20 ms and checkout confirms in ~2.5 s. That's the demo's closing beat. + +The incident fires **once**, so a rehearsal or a double-click doesn't spend the demo or +litter your workspace with conversations. The **Re-arm** control in the status panel +resets it. The ChatGPT token stays server-side; the browser never sees it. + +Each poll passes its *remaining* budget as `maxTimeMS`, so a slow scan can't outlive the +checkout timeout. Without that a 6 s poll starting at t=9.9 s would finish at t=15.9 s — +after the gateway confirmed the payment — and a checkout that should fail would succeed. + +Because the page is a server, it survives laptop sleep: start it before you leave, wake +the machine on stage with the tab already open, and click. Nothing to type. + +### Triggering from the CLI instead + +1. In ChatGPT, publish the incident agent to an API channel and copy its stable + `agtch_...` trigger identifier. Connect the MongoDB MCP plugin to the agent. Keep + its instructions generic: investigate incoming MongoDB incidents, gather evidence, + and require human approval before changes. Do not give it this demo's root cause. +2. Create a ChatGPT access token under Admin > Access tokens with the **Workspace + Agents** scope, then add both values to the git-ignored `.env` file: + ```dotenv + AGENT_ACCESS_TOKEN="..." + WORKSPACE_AGENT_TRIGGER_ID="agtch_..." + ``` +3. Trigger a new incident conversation: + ```bash + python trigger_chatgpt.py --wait + ``` + Open the printed `https://chatgpt.com/c/...` URL. The agent receives only the + simulated alert fields and must discover the database issue through MongoDB MCP. +4. Approve the proposed database action in that conversation, then let the agent + re-run `explain()` and summarize the result. + +Preview the complete PagerDuty incident resource and Workspace Agents request without +credentials or a network call: + +```bash +python trigger_chatgpt.py --dry-run +``` + +### Showing the Slack fan-out (optional) + +The same incident can land in Slack as well as ChatGPT — one webhook event, two +surfaces. This never happens automatically; run it when you want the beat: + +```bash +export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ" +python trigger_slack.py --dry-run # preview, sends nothing +python trigger_slack.py # post it +``` + +To make it obviously *one* incident on screen, pass the same ID to both and link the +Slack message back to the agent's conversation: + +```bash +python trigger_chatgpt.py --incident-id PY1Z69L +python trigger_slack.py --incident-id PY1Z69L \ + --conversation-url "https://chatgpt.com/c/..." # adds an "Open agent triage" button +``` + +The incident comes from `trigger_chatgpt.build_pagerduty_incident()`, so the two +channels cannot drift apart — only the rendering differs. Note that nothing reads the +Slack message back: it shows the incident *reaching* Slack, not a conversation with +the agent there. A real deployment would need a Slack app wired to the agent for that. + +The Workspace Agents API accepts a caller-defined `conversation_key`. The simulator +uses the PagerDuty incident ID for that key, so future webhook events for the same +incident can continue in one conversation. The separate event ID becomes the +`Idempotency-Key`, preventing a retried webhook from creating a duplicate task: + +```bash +python trigger_chatgpt.py \ + --incident-id QDEMO123456789 \ + --event-id 01DEMO123456789 \ + --wait +``` + +## The fix + +```js +db.payments.createIndex({ session_id: 1, status: 1 }) +``` + +`session_id` is the selective field; the compound index also covers the `status` +predicate. After creation the poll query uses `IXSCAN`, examines ~1 document, and +completes sub-millisecond — well under any checkout timeout. + +## Reset (to re-run the demo) + +### Light reset (usual case) + +If the collection is intact and you just created the index during the demo, drop the +index so the slow condition returns: + +```bash +python seed_payments.py --drop-index # drops the index, keeps the 300k docs +``` + +This exits before the insert loop, so your data is untouched. (Equivalent by hand: +`db.payments.dropIndex("session_id_1_status_1")`.) + +Then make sure the trickle is running again ahead of the next run: + +```bash +nohup python generate_load.py > load.log 2>&1 & # background; tail -f load.log to watch +``` + +### Full reseed ritual (fresh data) + +If you need to rebuild the collection from scratch: + +```bash +python seed_payments.py --drop # drop, then seed a fresh 300k +nohup python generate_load.py > load.log 2>&1 & # warm cache + rebuild Advisor recommendation +``` + +**Always use `--drop` when reseeding.** Running `seed_payments.py` without it *appends* +another 300k (→ 600k total), which roughly doubles scan time and pushes the cold case +into the MCP server's 60 s `maxTimeMS` cap — the failure mode 300k is sized to avoid. + +After any reseed, remember: +- The cache is cold again (first scans ~9 s, settling to ~5 s) — let the trickle warm it. +- The Performance Advisor recommendation resets with the collection — give the trickle + ~15–30 min to rebuild it, then confirm with `atlas-get-performance-advisor` before going live. diff --git a/apps/remote-mcp-perf-triage/checkout_app.py b/apps/remote-mcp-perf-triage/checkout_app.py new file mode 100644 index 00000000..76d3b59b --- /dev/null +++ b/apps/remote-mcp-perf-triage/checkout_app.py @@ -0,0 +1,535 @@ +"""Leafy Electronics checkout page — the demo's opening act, and a real reproduction +of the bug. + +Nothing here is simulated except the payment processor. The page runs the SAME +status-poll query the demo is about, against the SAME unindexed collection: + + db.payments.findOne({ session_id: ..., status: "completed" }) + +Pre-index that poll is a ~6 s COLLSCAN, so the page genuinely blows its client-side +budget and genuinely fails. Post-index it returns in ~1 ms and the page succeeds. +The hang is not staged; it is the bug. + +Flow when you click "Submit payment": + 1. POST /api/pay inserts a real payment doc with status="pending", and + schedules a background task to flip it to "completed" + after GATEWAY_DELAY_S (stands in for the processor webhook). + 2. GET /api/status runs the real poll query. The browser calls this in a loop + and shows each attempt's latency in the status panel. + 3. On timeout, the browser calls POST /api/incident, which sends the PagerDuty + incident to the ChatGPT Workspace Agent — ONCE. A "Re-arm" + control resets it so a rehearsal doesn't spend the demo. + +The ChatGPT access token stays server-side; the browser never sees it. + +Usage: + python checkout_app.py # http://127.0.0.1:8000 + python checkout_app.py --port 9000 + python checkout_app.py --no-incident # rehearse without paging anyone +""" + +import argparse +import asyncio +import os +import secrets +import sys +import time +from datetime import datetime, timezone +from pathlib import Path + +try: + import uvicorn + from dotenv import load_dotenv + from fastapi import FastAPI + from fastapi.responses import HTMLResponse, JSONResponse + from fastapi.staticfiles import StaticFiles + from pymongo import MongoClient + from pymongo.errors import ExecutionTimeout +except ImportError: + sys.exit("Missing deps. Run: pip install -r requirements.txt") + +# Reuse the incident builder and API client rather than duplicating the payload. +import trigger_chatgpt + +load_dotenv() + +DB_NAME = "ecommerce" +COLLECTION_NAME = "payments" + +# How long the fake processor takes to confirm. Realistic (real gateways take +# 1-3 s) and well under CLIENT_TIMEOUT_S, so post-index the page succeeds fast. +GATEWAY_DELAY_S = 3.0 + +# The user-facing budget: how long the shopper watches the spinner before the page +# gives up. Real checkouts often allow 30 s, but that is a long silence to narrate +# on stage — 12 s reads as clearly broken while staying brisk. Correctness does not +# depend on this value (see POLL_DEADLINE_MS), so it is safe to tune for pacing. +# Must stay comfortably above GATEWAY_DELAY_S or the post-index SUCCESS case breaks. +CLIENT_TIMEOUT_S = 12.0 + +# Gap between polls. Real checkout pages poll every 2-3 s rather than hammering. +# With a 12 s budget this yields 3 visible poll lines, ~5 s apart. +POLL_INTERVAL_MS = 2_500 + +# Per-request deadline for ONE poll, applied as maxTimeMS. Real services set a +# per-call deadline (API gateway, service SLO) far below the overall user budget, +# so having one is normal — but this VALUE is calibrated deliberately: +# +# Measured COLLSCANs on this cluster range 4.0-9.0 s. 2.5 s sits below the +# fastest of them, so EVERY pre-index poll is killed server-side before it can +# complete. No poll ever observes the gateway's confirmation, so checkout +# ALWAYS fails — regardless of cluster load on the day. +# +# Without this, the outcome depends on scan time vs. the overall budget: a poll +# that completes after the gateway confirms would find the record and checkout +# would SUCCEED, silently killing the demo. A 4.0 s burst was measured, so that +# edge is real, not theoretical. Post-index polls take ~20 ms and are unaffected. +POLL_DEADLINE_MS = 2_500 + +app = FastAPI(title="Leafy Electronics Checkout") + +# The MongoDB leaf, copied from the Leafy Roasters inventory demo so both apps use +# the identical asset. Resolved relative to this file, not the working directory, +# so the app can be started from anywhere. +STATIC_DIR = Path(__file__).parent / "static" +app.mount("/static", StaticFiles(directory=STATIC_DIR), name="static") + +# Module state. Single-process, single-presenter demo; no locking needed. +state = { + "incident_armed": True, + "incident_enabled": True, + "last_incident": None, +} + +client: MongoClient | None = None + +# Strong references to in-flight "processor confirms the payment" tasks. Without +# this asyncio can garbage-collect them mid-sleep (see RUF006). +_gateway_tasks: set[asyncio.Task] = set() + + +def coll(): + return client[DB_NAME][COLLECTION_NAME] + + +ORDER = [ + {"sku": "SKU-1001", "name": "Wireless Headphones", "qty": 1, "unit_price": 14900}, + {"sku": "SKU-1002", "name": "USB-C Cable", "qty": 1, "unit_price": 1200}, +] +ORDER_TOTAL = sum(i["qty"] * i["unit_price"] for i in ORDER) + + +async def confirm_payment_later(session_id: str): + """Stand in for the payment processor's confirmation webhook.""" + await asyncio.sleep(GATEWAY_DELAY_S) + await asyncio.to_thread( + coll().update_one, + {"session_id": session_id}, + {"$set": {"status": "completed", "updated_at": datetime.now(timezone.utc)}}, + ) + + +@app.post("/api/pay") +async def pay(): + """Create a real pending payment, then let the 'processor' confirm it.""" + session_id = f"sess_{secrets.token_hex(12)}" + now = datetime.now(timezone.utc) + doc = { + "session_id": session_id, + "order_id": f"ord_{secrets.randbelow(900000) + 100000}", + "user_id": "usr_408122", + "amount": ORDER_TOTAL, + "currency": "USD", + "status": "pending", + "payment_method": "card", + "card": { + "brand": "visa", + "last4": "4242", + "exp_month": 11, + "exp_year": 2029, + "fingerprint": secrets.token_hex(8), + }, + "billing_address": { + "city": "Seattle", + "state": "WA", + "country": "US", + "postal_code": "98104", + }, + "line_items": ORDER, + "gateway": "stripe", + "risk_score": 4, + "created_at": now, + "updated_at": now, + } + await asyncio.to_thread(coll().insert_one, doc) + # Keep a strong reference: asyncio only holds a weak one, so an unreferenced + # task can be garbage-collected before it runs — which here would mean the + # payment never gets confirmed and even the post-index checkout fails. + task = asyncio.create_task(confirm_payment_later(session_id)) + _gateway_tasks.add(task) + task.add_done_callback(_gateway_tasks.discard) + return {"session_id": session_id, "amount": ORDER_TOTAL} + + +@app.get("/api/status") +async def status(session_id: str, budget_ms: int = POLL_DEADLINE_MS): + """The checkout poll. THIS is the slow query the whole demo is about. + + The query is bounded by whichever is SMALLER: this poll's own deadline + (POLL_DEADLINE_MS) or the checkout's remaining budget passed in as budget_ms. + Two different limits, both real: + + * the per-poll deadline is the service's own request SLO, and it's what + makes the pre-index failure deterministic (see POLL_DEADLINE_MS); + * the remaining-budget cap stops the last poll of a run from overrunning + the user-facing timeout, which would let a scan finish AFTER the gateway + confirms and turn a should-fail checkout into a success. + """ + budget_ms = max(1, min(budget_ms, POLL_DEADLINE_MS, 60_000)) + t0 = time.perf_counter() + try: + found = await asyncio.to_thread( + coll().find_one, + {"session_id": session_id, "status": "completed"}, + max_time_ms=budget_ms, + ) + killed = False + except ExecutionTimeout: + found, killed = None, True + elapsed_ms = (time.perf_counter() - t0) * 1000 + return { + "confirmed": found is not None, + "timed_out": killed, + "elapsed_ms": round(elapsed_ms, 1), + "order_id": found.get("order_id") if found else None, + } + + +@app.post("/api/incident") +async def incident(): + """Page the on-call via the ChatGPT Workspace Agent — once per arming.""" + if not state["incident_enabled"]: + return JSONResponse({"fired": False, "reason": "disabled"}, status_code=200) + if not state["incident_armed"]: + return JSONResponse( + {"fired": False, "reason": "already_fired"}, status_code=200 + ) + + token = os.environ.get("AGENT_ACCESS_TOKEN") + trigger_id = os.environ.get("WORKSPACE_AGENT_TRIGGER_ID") + if not token or not trigger_id: + return JSONResponse( + {"fired": False, "reason": "missing_credentials"}, status_code=200 + ) + + # Build the same PagerDuty resource the CLI sends, with the same defaults. + args = trigger_chatgpt.main_defaults() + incident_id = trigger_chatgpt.pagerduty_id() + event_id = secrets.token_hex(16) + payload = trigger_chatgpt.build_pagerduty_incident(args, incident_id) + + try: + response = await asyncio.to_thread( + trigger_chatgpt.trigger_agent, + trigger_id, + token, + payload, + f"pagerduty-{incident_id}", + event_id, + ) + except (RuntimeError, KeyError) as exc: + return JSONResponse({"fired": False, "reason": str(exc)}, status_code=200) + + state["incident_armed"] = False + state["last_incident"] = { + "incident_id": incident_id, + "conversation_url": response.get("conversation_url"), + } + return { + "fired": True, + "incident_id": incident_id, + "conversation_url": response.get("conversation_url"), + } + + +@app.post("/api/rearm") +async def rearm(): + state["incident_armed"] = True + return {"armed": True} + + +@app.get("/api/config") +async def config(): + return { + "client_timeout_s": CLIENT_TIMEOUT_S, + "poll_interval_ms": POLL_INTERVAL_MS, + "incident_enabled": state["incident_enabled"], + "incident_armed": state["incident_armed"], + } + + +@app.get("/", response_class=HTMLResponse) +async def index(): + # Loading the checkout page re-arms the incident, so a browser refresh is the + # reset — no demo control on the shopper's screen. Deliberately scoped to this + # HTML route: /api/* calls never re-arm, so the polling loop can't rearm + # mid-run and fire a second incident. + state["incident_armed"] = True + return PAGE + + +PAGE = """ + + + +Leafy Electronics — Checkout + + + +
+
+ +
+ Leafy Electronics + Checkout +
+
+
+
+
+

Order summary

+
Wireless Headphones$149.00
+
USB-C Cable$12.00
+
Total$161.00
+
+ +
•••• •••• •••• 4242
+
+ +
+
Processing your payment…
+
Please don't close this window.
+
+
+ +
+

Checkout status poll

+
idle — awaiting payment
+
+
+ + + +""" + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=8000) + parser.add_argument( + "--no-incident", + action="store_true", + help="rehearse the checkout without paging the agent", + ) + args = parser.parse_args() + + uri = os.environ.get("MONGODB_URI") + if not uri: + sys.exit("ERROR: set MONGODB_URI in .env") + + global client + client = MongoClient(uri, appname="perf-triage-demo-checkout") + client.admin.command("ping") # fail fast, before the stage + + state["incident_enabled"] = not args.no_incident + if args.no_incident: + print("Incident dispatch DISABLED (--no-incident): checkout will fail quietly.") + elif not os.environ.get("AGENT_ACCESS_TOKEN"): + print("WARNING: AGENT_ACCESS_TOKEN unset — checkout will fail but page no one.") + + print(f"Checkout page: http://{args.host}:{args.port}") + uvicorn.run(app, host=args.host, port=args.port, log_level="warning") + + +if __name__ == "__main__": + main() diff --git a/apps/remote-mcp-perf-triage/generate_load.py b/apps/remote-mcp-perf-triage/generate_load.py new file mode 100644 index 00000000..898342cd --- /dev/null +++ b/apps/remote-mcp-perf-triage/generate_load.py @@ -0,0 +1,189 @@ +""" +Generate checkout status-poll traffic against the unindexed `payments` collection. + +This stands in for real production traffic. It runs the checkout page's status-poll +query on an interval so that (a) Atlas Performance Advisor observes the slow query +pattern and surfaces an index recommendation, and (b) the pattern stays fresh in the +Advisor's rolling ~24h window right up to demo time. + +The query mirrors what a checkout page runs while waiting for a payment to confirm: + db.payments.findOne({ session_id: , status: "completed" }) + +It queries session_ids with no completed record yet (still "pending") — realistic, +since the page polls before the processor webhook flips the status. With no index on +session_id, every poll is a full COLLSCAN. + +No maxTimeMS here: we WANT each query to complete slowly so it lands in the slow-query +log (>100 ms) that Performance Advisor analyzes. The "checkout timing out" symptom is +demonstrated for real by checkout_app.py, which runs this same query and genuinely +exceeds its timeout. + +TRICKLE MODE (default): recency matters more than volume for Performance Advisor, and +the M10 has a burstable CPU, so by default this runs a small BURST of queries every +INTERVAL seconds rather than hammering continuously. That keeps the recommendation +alive and the cache warm while being gentle on the cluster. + +Usage: + export MONGODB_URI="mongodb+srv://:@host/" + + # Default trickle: 3 queries every 5 minutes, forever (Ctrl+C to stop). + python generate_load.py + + # Custom trickle cadence. + python generate_load.py --burst 5 --interval 900 + + # Continuous (hammer) mode: back-to-back queries, no sleep between bursts. + python generate_load.py --interval 0 + + # Run unattended in the background, logging to a file: + nohup python generate_load.py > load.log 2>&1 & + # ...check on it later: tail -f load.log ; stop it: kill %1 (or the PID) + + # Or via cron — a burst every 15 minutes (no long-running process): + # */15 * * * * cd /path/to/apps/remote-mcp-perf-triage && \ + # MONGODB_URI="mongodb+srv://..." python generate_load.py --burst 3 --interval 0 --duration 30 +""" + +import argparse +import os +import secrets +import sys +import time +from datetime import datetime + +try: + from dotenv import load_dotenv + from pymongo import MongoClient + from pymongo.errors import PyMongoError +except ImportError: + sys.exit("pymongo not installed. Run: pip install -r requirements.txt") + +load_dotenv() + +DB_NAME = "ecommerce" +COLLECTION_NAME = "payments" + +POLL_FILTER_STATUS = "completed" + + +def run_query(coll): + """Run one checkout status-poll query; return its latency in ms. + + A random session_id that has no 'completed' record forces a full COLLSCAN. + Raises PyMongoError on connection trouble; the caller decides whether to + keep going. + """ + session_id = f"sess_{secrets.token_hex(12)}" + t0 = time.perf_counter() + coll.find_one({"session_id": session_id, "status": POLL_FILTER_STATUS}) + return (time.perf_counter() - t0) * 1000 + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--burst", type=int, default=3, help="number of queries to run per cycle" + ) + parser.add_argument( + "--interval", + type=float, + default=300, + help="seconds between the start of each burst " + "(0 = continuous / no sleep between bursts)", + ) + parser.add_argument( + "--duration", + type=float, + default=0, + help="total seconds to run (0 = run until Ctrl+C)", + ) + args = parser.parse_args() + + uri = os.environ.get("MONGODB_URI") + if not uri: + sys.exit( + 'ERROR: set MONGODB_URI, e.g.\n export MONGODB_URI="mongodb+srv://user:pass@host/"' + ) + + client = MongoClient(uri, appname="perf-triage-demo-load") + coll = client[DB_NAME][COLLECTION_NAME] + # Fail loudly at launch on a bad URI or firewall — that's a config error to fix, + # not something to retry. Mid-run failures are handled in the loop below. + try: + client.admin.command("ping") + except PyMongoError as exc: + sys.exit(f"ERROR: cannot reach MongoDB — {type(exc).__name__}: {exc}") + + mode = ( + "continuous" + if args.interval == 0 + else f"trickle ({args.burst} queries / {args.interval:.0f}s)" + ) + limit = "until Ctrl+C" if args.duration == 0 else f"{args.duration:.0f}s" + print( + f"Polling {DB_NAME}.{COLLECTION_NAME} — {mode}, {limit}. " + "Each query is a full COLLSCAN (no index on session_id)." + ) + + deadline = time.time() + args.duration if args.duration > 0 else None + total = 0 + errors = 0 + try: + while deadline is None or time.time() < deadline: + cycle_start = time.time() + ts = datetime.now().strftime("%H:%M:%S") + + # Survive transient trouble instead of dying. This process is meant to + # run unattended for hours before a demo, across laptop sleep, wifi + # handoffs and Atlas blips — any of which raises PyMongoError. Losing + # the trickle means Performance Advisor's recommendation goes stale, + # so a failed burst is logged and skipped, never fatal. pymongo + # reconnects on its own; we just have to keep asking. + latencies = [] + failures = [] + for _ in range(args.burst): + try: + latencies.append(run_query(coll)) + except PyMongoError as exc: + failures.append(type(exc).__name__) + + total += len(latencies) + errors += len(failures) + + if latencies: + avg = sum(latencies) / len(latencies) + slow = 100 * sum(1 for x in latencies if x > 100) / len(latencies) + suffix = f" | {len(failures)} failed" if failures else "" + print( + f" {ts} burst of {len(latencies)}: avg {avg:7.1f} ms | " + f"{slow:3.0f}% over 100 ms | {total:,} total{suffix}", + flush=True, + ) + else: + # Whole burst failed — almost always a dropped connection. + print( + f" {ts} burst FAILED ({', '.join(sorted(set(failures)))}) — " + f"will retry next cycle | {total:,} total, {errors} errors", + flush=True, + ) + + if args.interval > 0: + sleep_for = args.interval - (time.time() - cycle_start) + if sleep_for > 0 and ( + deadline is None or time.time() + sleep_for < deadline + ): + time.sleep(sleep_for) + elif deadline is not None: + break # not enough time left for another cycle + elif not latencies: + # Continuous mode with a dead connection would spin as fast as the + # driver can fail. Back off so the log stays readable. + time.sleep(5) + except KeyboardInterrupt: + pass + tail = f", {errors} errors" if errors else "" + print(f"\nStopped after {total:,} queries{tail}.") + + +if __name__ == "__main__": + main() diff --git a/apps/remote-mcp-perf-triage/requirements.txt b/apps/remote-mcp-perf-triage/requirements.txt new file mode 100644 index 00000000..62831438 --- /dev/null +++ b/apps/remote-mcp-perf-triage/requirements.txt @@ -0,0 +1,4 @@ +pymongo>=4.6 +python-dotenv>=1.0 +fastapi>=0.110 +uvicorn>=0.27 diff --git a/apps/remote-mcp-perf-triage/seed_payments.py b/apps/remote-mcp-perf-triage/seed_payments.py new file mode 100644 index 00000000..85b583c5 --- /dev/null +++ b/apps/remote-mcp-perf-triage/seed_payments.py @@ -0,0 +1,269 @@ +""" +Seed a large, realistic `payments` collection on an Atlas M10 for the +performance-triage demo. + +Goal: make a full collection scan genuinely slow so that (a) the checkout +status-poll query trips the slow-query log that feeds Performance Advisor, and +(b) a live explain() shows a real COLLSCAN examining every document. + +Two design constraints, satisfied together: + +1. The scan must be reliably slow. Empirically on an M10 with 2 GB RAM (measured), + 300,000 documents of ~2 KB is the sweet spot: a COLLSCAN runs ~9 s cold and + settles to ~5 s once the cache is warm. That is clearly slow and dramatic in + explain(), yet stays safely under the MongoDB MCP server's 60 s maxTimeMS cap. + NOTE: do NOT seed millions here — a scan of that size can exceed the 60 s cap, + making the agent's explain()/find() ERROR out during the demo instead of + returning stats. Bigger is worse, not better. + + What makes the scan slow is that the collection (~0.63 GB) OUTGROWS the + WiredTiger cache (~50% of host RAM, so ~537 MB on a 2 GB M10), forcing reads + from disk. On a bigger tier the collection fits entirely in cache and the same + query returns in ~200 ms — measured at 222 ms on a 4 GB host — which kills the + demo. If scans come back suspiciously fast, check hostInfo.memSizeMB: the fix + is a smaller tier, NOT more documents. + + The document bytes live in a realistic `gateway_response` field (an opaque + base64 payload — exactly what payment processors return and apps store for + reconciliation). It is high-entropy so WiredTiger's snappy compression can't + shrink the collection, AND screenshot-safe: if a document appears on screen + during the demo, it reads as a real payment record, not obvious junk padding. + +2. There is deliberately NO index on `session_id` (the field the checkout poll + filters on). Only the default _id index exists. + +Tip: after seeding, warm the cache (run generate_load.py or a few scans) so the +demo sees the ~4 s warm time rather than the ~30 s cold time. + +Usage: + export MONGODB_URI="mongodb+srv://:@cluster0.z7basj.mongodb-dev.net/" + python seed_payments.py # defaults: 300,000 docs, ~1.6 KB gateway blob + python seed_payments.py --docs 300000 --blob-bytes 1600 + python seed_payments.py --drop # drop the collection first (fresh reseed) + python seed_payments.py --drop-index # light reset: drop the demo index, keep the data +""" + +import argparse +import base64 +import os +import random +import secrets +import sys +import time +from datetime import datetime, timedelta, timezone + +try: + from dotenv import load_dotenv + from pymongo import InsertOne, MongoClient +except ImportError: + sys.exit("pymongo not installed. Run: pip install -r requirements.txt") + +load_dotenv() + +DB_NAME = "ecommerce" +COLLECTION_NAME = "payments" + +# The index the agent creates during the demo, and that --drop-index removes to +# restore the slow condition. Name is MongoDB's default for { session_id: 1, status: 1 }. +DEMO_INDEX_NAME = "session_id_1_status_1" + +# Realistic reference data. These enum-ish fields are a small fraction of each +# document, so their compressibility doesn't matter — the bulk is the random blob. +STATUS_WEIGHTS = [("completed", 0.92), ("pending", 0.05), ("failed", 0.03)] +PAYMENT_METHODS = ["card", "paypal", "apple_pay", "google_pay", "bank_transfer"] +CURRENCIES = ["USD", "EUR", "GBP", "CAD"] +CARD_BRANDS = ["visa", "mastercard", "amex", "discover"] +GATEWAYS = ["stripe", "adyen", "braintree"] +CITIES = [ + ("Austin", "TX", "US"), + ("Seattle", "WA", "US"), + ("Denver", "CO", "US"), + ("London", "", "GB"), + ("Toronto", "ON", "CA"), + ("Berlin", "", "DE"), + ("Dublin", "", "IE"), + ("New York", "NY", "US"), +] +PRODUCTS = [ + ("SKU-1001", "Wireless Headphones"), + ("SKU-1002", "USB-C Cable"), + ("SKU-1003", "Laptop Stand"), + ("SKU-1004", "Mechanical Keyboard"), + ("SKU-1005", "4K Monitor"), + ("SKU-1006", "Webcam"), + ("SKU-1007", "Desk Lamp"), + ("SKU-1008", "Notebook"), +] + + +def weighted_status(): + r = random.random() + cumulative = 0.0 + for status, weight in STATUS_WEIGHTS: + cumulative += weight + if r <= cumulative: + return status + return "completed" + + +def gateway_blob(blob_bytes): + """An opaque, high-entropy base64 payload — realistic AND incompressible. + + base64 of random bytes has no repeated structure, so WiredTiger's snappy + compression can't shrink it: the collection stays large on disk. + """ + raw = os.urandom(max(1, blob_bytes * 3 // 4)) # base64 expands ~4/3 + return base64.b64encode(raw).decode("ascii") + + +def make_doc(blob_bytes, base_time): + """Build one realistic payment document.""" + city, state, country = random.choice(CITIES) + n_items = random.randint(1, 3) + line_items = [] + for _ in range(n_items): + sku, name = random.choice(PRODUCTS) + line_items.append( + { + "sku": sku, + "name": name, + "qty": random.randint(1, 4), + "unit_price": random.randint(500, 20_000), + } + ) + amount = sum(i["qty"] * i["unit_price"] for i in line_items) + created = base_time - timedelta(seconds=random.randint(0, 90 * 24 * 3600)) + return { + # High-cardinality session id — the checkout poll filters on this (unindexed). + "session_id": f"sess_{secrets.token_hex(12)}", + "order_id": f"ord_{random.randint(100000, 999999)}", + "user_id": f"usr_{random.randint(1, 500_000)}", + "amount": amount, + "currency": random.choice(CURRENCIES), + "status": weighted_status(), + "payment_method": random.choice(PAYMENT_METHODS), + "card": { + "brand": random.choice(CARD_BRANDS), + "last4": f"{random.randint(0, 9999):04d}", + "exp_month": random.randint(1, 12), + "exp_year": random.randint(2026, 2031), + "fingerprint": secrets.token_hex(8), + }, + "billing_address": { + "city": city, + "state": state, + "country": country, + "postal_code": f"{random.randint(10000, 99999)}", + }, + "line_items": line_items, + "gateway": random.choice(GATEWAYS), + # The bulk of the document size: an opaque processor response payload. + "gateway_response": gateway_blob(blob_bytes), + "risk_score": random.randint(0, 99), + "created_at": created, + "updated_at": created + timedelta(seconds=random.randint(1, 30)), + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--docs", type=int, default=300_000, help="number of documents to insert" + ) + parser.add_argument( + "--blob-bytes", + type=int, + default=1600, + help="approx size of the gateway_response payload per doc", + ) + parser.add_argument("--batch", type=int, default=5000, help="insert batch size") + parser.add_argument( + "--drop", action="store_true", help="drop the collection before seeding" + ) + parser.add_argument( + "--drop-index", + action="store_true", + help="drop the demo index and exit WITHOUT reseeding " + "(the light reset after a demo has created it)", + ) + args = parser.parse_args() + + uri = os.environ.get("MONGODB_URI") + if not uri: + sys.exit( + 'ERROR: set MONGODB_URI, e.g.\n export MONGODB_URI="mongodb+srv://user:pass@host/"' + ) + + client = MongoClient(uri, appname="perf-triage-demo-seed") + coll = client[DB_NAME][COLLECTION_NAME] + + # Fail fast on bad credentials / network before the long insert loop. + client.admin.command("ping") + + if args.drop_index: + # Light reset: restore the slow condition on an intact collection. Exits + # before the insert loop, so the existing 300k documents are untouched. + if args.drop: + sys.exit( + "ERROR: --drop-index and --drop are mutually exclusive. " + "--drop-index keeps the data; --drop destroys it." + ) + existing = list(coll.index_information().keys()) + if DEMO_INDEX_NAME in existing: + print( + f"Dropping index {DEMO_INDEX_NAME} from {DB_NAME}.{COLLECTION_NAME} ..." + ) + coll.drop_index(DEMO_INDEX_NAME) + else: + print(f"Index {DEMO_INDEX_NAME} not present — nothing to drop.") + print( + f"Indexes now: {list(coll.index_information().keys())} (expect only _id_)" + ) + print(f"Documents kept: {coll.estimated_document_count():,}") + print( + "Next: make sure generate_load.py is running so Performance Advisor " + "rebuilds its recommendation before the next demo." + ) + return + + if args.drop: + print(f"Dropping {DB_NAME}.{COLLECTION_NAME} ...") + coll.drop() + + print( + f"Seeding {args.docs:,} docs (~{args.blob_bytes}B gateway payload each) into " + f"{DB_NAME}.{COLLECTION_NAME} — NO index on session_id." + ) + base_time = datetime.now(timezone.utc) + start = time.time() + inserted = 0 + + while inserted < args.docs: + n = min(args.batch, args.docs - inserted) + ops = [InsertOne(make_doc(args.blob_bytes, base_time)) for _ in range(n)] + coll.bulk_write(ops, ordered=False) + inserted += n + if inserted % 100_000 == 0 or inserted == args.docs: + elapsed = time.time() - start + rate = inserted / elapsed if elapsed else 0 + print( + f" {inserted:,}/{args.docs:,} ({rate:,.0f} docs/s, {elapsed:,.0f}s elapsed)" + ) + + stats = client[DB_NAME].command("collstats", COLLECTION_NAME) + size_gb = stats.get("size", 0) / 1e9 + storage_gb = stats.get("storageSize", 0) / 1e9 + print( + f"\nDone. Logical size ~{size_gb:.2f} GB, on-disk storage ~{storage_gb:.2f} GB." + ) + print( + f"Indexes present: {list(coll.index_information().keys())} (expect only _id_)" + ) + print( + "Next: run generate_load.py to warm the cache and feed Performance Advisor, " + "then confirm scan time with an explain() (expect ~9 s cold, ~5 s warm on a 2 GB M10)." + ) + + +if __name__ == "__main__": + main() diff --git a/apps/remote-mcp-perf-triage/static/mongodb-logo.png b/apps/remote-mcp-perf-triage/static/mongodb-logo.png new file mode 100644 index 00000000..c87c2a79 Binary files /dev/null and b/apps/remote-mcp-perf-triage/static/mongodb-logo.png differ diff --git a/apps/remote-mcp-perf-triage/trigger_chatgpt.py b/apps/remote-mcp-perf-triage/trigger_chatgpt.py new file mode 100644 index 00000000..d864188f --- /dev/null +++ b/apps/remote-mcp-perf-triage/trigger_chatgpt.py @@ -0,0 +1,332 @@ +"""Send a simulated PagerDuty incident to a ChatGPT Workspace Agent. + +The input mirrors a PagerDuty incident resource created from a Datadog monitor. It +includes only application symptoms, never the database namespace, query shape, +root cause, or remediation. + +Usage: + export AGENT_ACCESS_TOKEN="..." + export WORKSPACE_AGENT_TRIGGER_ID="agtch_..." + python trigger_chatgpt.py +""" + +import argparse +import hashlib +import json +import os +import secrets +import string +import sys +import time +import urllib.error +import urllib.request +from datetime import datetime, timezone + +try: + from dotenv import load_dotenv +except ImportError: + sys.exit("python-dotenv not installed. Run: pip install -r requirements.txt") + +load_dotenv() + +API_ROOT = "https://api.chatgpt.com/v1/workspace_agents" +BETA_HEADER = "workspace_agent_runs=v1" +WAIT_STOP_STATUSES = {"completed", "failed", "suspended"} + +# The trigger POST has been measured at ~5.5 s, but tail latency is much worse: a +# live run once died on a 30 s read timeout, then succeeded three times in a row. +# Generous timeout + retries, because failing this call on stage is the one error +# the audience actually sees. +REQUEST_TIMEOUT_S = 90 +MAX_ATTEMPTS = 3 +RETRY_BACKOFF_S = 2.0 + +# Retrying a POST is only safe because we always send Idempotency-Key: the server +# dedupes, so a retry after a timeout resumes the original request rather than +# creating a second conversation. Never retry a mutating call without that key. +RETRY_STATUS = {408, 429, 500, 502, 503, 504} + + +def request_json(url, token, method="GET", payload=None, idempotency_key=None): + data = json.dumps(payload).encode("utf-8") if payload is not None else None + headers = { + "Authorization": f"Bearer {token}", + "Content-Type": "application/json", + "OpenAI-Beta": BETA_HEADER, + } + if idempotency_key: + headers["Idempotency-Key"] = idempotency_key + + request = urllib.request.Request( + url, + data=data, + headers=headers, + method=method, + ) + + last_error = None + for attempt in range(1, MAX_ATTEMPTS + 1): + try: + with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_S) as response: + body = response.read().decode("utf-8") + return response.status, json.loads(body) + except urllib.error.HTTPError as exc: + body = exc.read().decode("utf-8", errors="replace") + # 4xx other than 408/429 is our bug (bad token, bad trigger id) — + # retrying just delays the error message. + if exc.code not in RETRY_STATUS: + raise RuntimeError( + f"ChatGPT API returned HTTP {exc.code}: {body}" + ) from exc + last_error = RuntimeError(f"ChatGPT API returned HTTP {exc.code}: {body}") + except (urllib.error.URLError, TimeoutError, OSError) as exc: + reason = getattr(exc, "reason", exc) + last_error = RuntimeError(f"Could not reach the ChatGPT API: {reason}") + + if attempt < MAX_ATTEMPTS: + delay = RETRY_BACKOFF_S * attempt + print( + f" request failed ({last_error}); retrying in {delay:.0f}s " + f"[attempt {attempt + 1}/{MAX_ATTEMPTS}]", + file=sys.stderr, + flush=True, + ) + time.sleep(delay) + + raise last_error + + +def pagerduty_id(prefix="P", length=7): + alphabet = string.ascii_uppercase + string.digits + return prefix + "".join(secrets.choice(alphabet) for _ in range(length - 1)) + + +def build_pagerduty_incident(args, incident_id): + created_at = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") + account_url = f"https://{args.account_subdomain}.pagerduty.com" + + return { + "id": incident_id, + "type": "incident", + "self": f"https://api.pagerduty.com/incidents/{incident_id}", + "html_url": f"{account_url}/incidents/{incident_id}", + "number": args.incident_number, + "status": "triggered", + "incident_key": hashlib.sha256(incident_id.encode("utf-8")).hexdigest()[:32], + "created_at": created_at, + "reopened_at": None, + "title": ( + f"[Datadog] {args.service} payment confirmation timeout rate is " + f"{args.timeout_rate:g}% in {args.environment}/{args.region}" + ), + "incident_type": {"name": "major"}, + "service": { + "html_url": f"{account_url}/services/{args.service_id}", + "id": args.service_id, + "self": f"https://api.pagerduty.com/services/{args.service_id}", + "summary": args.service, + "type": "service_reference", + }, + "assignees": [ + { + "html_url": f"{account_url}/users/{args.assignee_id}", + "id": args.assignee_id, + "self": f"https://api.pagerduty.com/users/{args.assignee_id}", + "summary": args.assignee, + "type": "user_reference", + } + ], + "escalation_policy": { + "html_url": f"{account_url}/escalation_policies/{args.policy_id}", + "id": args.policy_id, + "self": f"https://api.pagerduty.com/escalation_policies/{args.policy_id}", + "summary": args.escalation_policy, + "type": "escalation_policy_reference", + }, + "teams": [ + { + "html_url": f"{account_url}/teams/{args.team_id}", + "id": args.team_id, + "self": f"https://api.pagerduty.com/teams/{args.team_id}", + "summary": args.team, + "type": "team_reference", + } + ], + "priority": { + "html_url": f"{account_url}/account/incident_priorities", + "id": args.priority_id, + "self": f"https://api.pagerduty.com/priorities/{args.priority_id}", + "summary": "P2", + "type": "priority_reference", + }, + "urgency": "high", + "conference_bridge": None, + "resolve_reason": None, + } + + +def build_trigger_payload(incident, conversation_key): + return { + "conversation_key": conversation_key, + "input": "PagerDuty incident:\n```json\n" + + json.dumps(incident, indent=2) + + "\n```", + } + + +def trigger_agent(trigger_id, token, incident, conversation_key, event_id): + payload = build_trigger_payload(incident, conversation_key) + url = f"{API_ROOT}/{trigger_id}/trigger" + status, response = request_json( + url, + token, + method="POST", + payload=payload, + idempotency_key=event_id, + ) + if status != 202: + raise RuntimeError(f"Expected HTTP 202, received {status}: {response}") + return response + + +def wait_for_run(trigger_id, run_id, token, poll_interval, timeout): + url = f"{API_ROOT}/{trigger_id}/runs/{run_id}" + deadline = time.monotonic() + timeout + last_status = None + + while time.monotonic() < deadline: + _, run = request_json(url, token) + status = run.get("status", "unknown") + if status != last_status: + print(f"Agent run: {status}") + last_status = status + if status in WAIT_STOP_STATUSES: + return run + time.sleep(poll_interval) + + raise RuntimeError(f"Agent run did not settle within {timeout:.0f} seconds") + + +def build_parser(): + parser = argparse.ArgumentParser(description=__doc__) + # Leafy Electronics is an invented company (MongoDB's leaf, and a sibling to the + # Leafy Roasters demo), so a public repo never puts a real organization's name on + # a fabricated incident. + parser.add_argument("--account-subdomain", default="leafyelectronics") + parser.add_argument("--service", default="Checkout API") + parser.add_argument("--service-id", default="PF9KMXH") + parser.add_argument("--environment", default="production") + parser.add_argument("--region", default="us-west-1") + parser.add_argument("--timeout-rate", type=float, default=18.6) + parser.add_argument("--assignee", default="Dana Whitfield") + parser.add_argument("--team", default="Payments Platform") + parser.add_argument( + "--escalation-policy", + default="Checkout — Production", + help="PagerDuty escalation policy name shown on the incident", + ) + parser.add_argument("--assignee-id", default="PTUXL6G") + parser.add_argument("--policy-id", default="PUS0KTE") + parser.add_argument("--team-id", default="PFCVPS0") + parser.add_argument("--priority-id", default="PSO75BM") + parser.add_argument("--incident-number", type=int, default=4821) + parser.add_argument( + "--incident-id", + help="PagerDuty-style P... ID; shared by all events for one incident", + ) + parser.add_argument( + "--event-id", + help="unique webhook event ID; reuse only when retrying the same event", + ) + parser.add_argument( + "--wait", + action="store_true", + help="poll until the run completes, fails, or is suspended for human action", + ) + parser.add_argument("--poll-interval", type=float, default=2.0) + parser.add_argument("--timeout", type=float, default=180.0) + parser.add_argument( + "--dry-run", + action="store_true", + help="print the Workspace Agents API request without sending it", + ) + return parser + + +def main_defaults(): + """The incident defaults as a namespace, with no CLI parsing. + + Lets checkout_app.py build the exact same PagerDuty resource this CLI sends, + so the two entry points can never drift apart. + """ + return build_parser().parse_args([]) + + +def main(): + parser = build_parser() + args = parser.parse_args() + + incident_id = args.incident_id or pagerduty_id() + event_id = args.event_id or secrets.token_hex(16) + conversation_key = f"pagerduty-{incident_id}" + incident = build_pagerduty_incident(args, incident_id) + + if args.dry_run: + preview = { + "method": "POST", + "url": f"{API_ROOT}/agtch_.../trigger", + "headers": { + "Authorization": "Bearer ", + "Content-Type": "application/json", + "OpenAI-Beta": BETA_HEADER, + "Idempotency-Key": event_id, + }, + "body": build_trigger_payload(incident, conversation_key), + } + print(json.dumps(preview, indent=2)) + return 0 + + token = os.environ.get("AGENT_ACCESS_TOKEN") + trigger_id = os.environ.get("WORKSPACE_AGENT_TRIGGER_ID") + if not token: + sys.exit("ERROR: set AGENT_ACCESS_TOKEN") + if not trigger_id: + sys.exit( + "ERROR: set WORKSPACE_AGENT_TRIGGER_ID to the agtch_... API channel ID" + ) + if not trigger_id.startswith("agtch_"): + sys.exit("ERROR: WORKSPACE_AGENT_TRIGGER_ID must begin with agtch_") + + try: + response = trigger_agent( + trigger_id, + token, + incident, + conversation_key, + event_id, + ) + print(f"PagerDuty incident: {incident_id}") + print(f"Webhook event: {event_id}") + print(f"Conversation: {response['conversation_url']}") + + run_id = response.get("agent_trigger_run_id") + if run_id: + print(f"Run ID: {run_id}") + if args.wait and run_id: + run = wait_for_run( + trigger_id, + run_id, + token, + args.poll_interval, + args.timeout, + ) + if run.get("error"): + print(f"Run error: {json.dumps(run['error'])}") + return 1 if run.get("status") == "failed" else 0 + return 0 + except (RuntimeError, KeyError) as exc: + sys.exit(f"ERROR: {exc}") + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/apps/remote-mcp-perf-triage/trigger_slack.py b/apps/remote-mcp-perf-triage/trigger_slack.py new file mode 100644 index 00000000..2d571459 --- /dev/null +++ b/apps/remote-mcp-perf-triage/trigger_slack.py @@ -0,0 +1,220 @@ +"""Post the SAME PagerDuty incident that goes to ChatGPT into a Slack channel. + +The point is the fan-out: one incident, two surfaces. It shows the agent isn't +bolted to one chat client — the same webhook event that opens a ChatGPT +conversation can just as easily land in Slack, where the on-call engineer already +lives. Pair it with trigger_chatgpt.py in the demo: + + python trigger_chatgpt.py --incident-id PY1Z69L # opens the agent conversation + python trigger_slack.py --incident-id PY1Z69L # same incident, in Slack + +Passing the same --incident-id to both makes the two views obviously the same +incident on screen. Omit it and each call invents its own. + +The incident payload comes from trigger_chatgpt.build_pagerduty_incident(), so +the two channels can never drift apart: same title, service, assignee, team, +escalation policy, priority. Only the rendering differs. + +This is the staged part of the demo. A real deployment would have PagerDuty POST +its webhook to an adapter that fans out to both the Workspace Agents API and +Slack; this script stands in for that adapter's Slack leg. Nothing here reads the +Slack message back — it's a delivery surface, not a second trigger path. + +Like the ChatGPT payload, this carries application symptoms only — no database +namespace, query shape, root cause, or index recommendation. The agent still has to +discover those itself. + +Usage: + # Add to the git-ignored .env (or export it): + # SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ" + + python trigger_slack.py # post the incident + python trigger_slack.py --dry-run # print the payload, send nothing + python trigger_slack.py --incident-id PY1Z69L # match a ChatGPT trigger + python trigger_slack.py --conversation-url https://chatgpt.com/c/... +""" + +import argparse +import json +import os +import sys +import urllib.error +import urllib.request +from datetime import datetime + +try: + from dotenv import load_dotenv +except ImportError: + sys.exit("python-dotenv not installed. Run: pip install -r requirements.txt") + +# Reuse the incident builder so Slack and ChatGPT get identical incidents. +import trigger_chatgpt + +load_dotenv() + +# PagerDuty's accent red for a triggered, high-urgency incident. +PAGERDUTY_ALERT_COLOR = "#CD3B48" + +# Rendered as the Slack sender avatar so the message reads as "from PagerDuty". +PAGERDUTY_ICON_URL = "https://avatars.slack-edge.com/2019-11-19/822144368room_72.png" + + +def build_slack_payload(incident, conversation_url=None): + """Render a PagerDuty incident resource as a Slack message. + + Mirrors the shape of PagerDuty's own Slack integration: title as the header, + a field grid of the routing metadata, and a button through to the incident. + """ + # created_at is ISO-8601 with microseconds; trim to whole seconds for display. + created = datetime.fromisoformat( + incident["created_at"].replace("Z", "+00:00") + ).strftime("%Y-%m-%d %H:%M:%S UTC") + service = incident["service"]["summary"] + assignee = incident["assignees"][0]["summary"] + team = incident["teams"][0]["summary"] + policy = incident["escalation_policy"]["summary"] + priority = incident["priority"]["summary"] + + blocks = [ + { + "type": "header", + "text": { + "type": "plain_text", + "text": f"🚨 Triggered: {priority} incident", + "emoji": True, + }, + }, + { + "type": "section", + "text": {"type": "mrkdwn", "text": f"*{incident['title']}*"}, + }, + { + "type": "section", + "fields": [ + { + "type": "mrkdwn", + "text": f"*Incident*\n#{incident['number']} · {incident['id']}", + }, + {"type": "mrkdwn", "text": f"*Service*\n{service}"}, + {"type": "mrkdwn", "text": f"*Assigned to*\n{assignee}"}, + {"type": "mrkdwn", "text": f"*Team*\n{team}"}, + {"type": "mrkdwn", "text": f"*Escalation policy*\n{policy}"}, + {"type": "mrkdwn", "text": f"*Urgency*\n{incident['urgency'].title()}"}, + ], + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": f"Triggered {created} · status *{incident['status']}*", + } + ], + }, + ] + + elements = [ + { + "type": "button", + "text": {"type": "plain_text", "text": "View in PagerDuty"}, + "url": incident["html_url"], + "style": "danger", + } + ] + # When the ChatGPT trigger has already run, link the two surfaces together so + # the audience can see it is one incident rather than two. + if conversation_url: + elements.append( + { + "type": "button", + "text": {"type": "plain_text", "text": "Open agent triage"}, + "url": conversation_url, + } + ) + blocks.append({"type": "actions", "elements": elements}) + + return { + "username": "PagerDuty", + "icon_url": PAGERDUTY_ICON_URL, + "attachments": [ + { + "color": PAGERDUTY_ALERT_COLOR, + "blocks": blocks, + "fallback": f"[{priority}] {incident['title']} ({incident['id']})", + } + ], + } + + +def post(webhook_url, payload): + data = json.dumps(payload).encode("utf-8") + req = urllib.request.Request( + webhook_url, data=data, headers={"Content-Type": "application/json"} + ) + try: + with urllib.request.urlopen(req, timeout=30) as resp: + return resp.status, resp.read().decode("utf-8") + except urllib.error.HTTPError as exc: + body = exc.read().decode("utf-8", errors="replace") + raise RuntimeError(f"Slack returned HTTP {exc.code}: {body}") from exc + except urllib.error.URLError as exc: + raise RuntimeError(f"Could not reach Slack: {exc.reason}") from exc + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--incident-id", + help="PagerDuty-style incident ID. Pass the same value you gave " + "trigger_chatgpt.py so both surfaces show one incident.", + ) + parser.add_argument( + "--conversation-url", + help="ChatGPT conversation URL from trigger_chatgpt.py; adds an " + "'Open agent triage' button linking the two surfaces.", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="print the Slack payload without sending it", + ) + args, _ = parser.parse_known_args() + + # Same defaults as the ChatGPT trigger: Leafy Electronics, Checkout API, + # Dana Whitfield. + incident_args = trigger_chatgpt.main_defaults() + incident_id = args.incident_id or trigger_chatgpt.pagerduty_id() + incident = trigger_chatgpt.build_pagerduty_incident(incident_args, incident_id) + payload = build_slack_payload(incident, args.conversation_url) + + if args.dry_run: + print(json.dumps(payload, indent=2)) + return 0 + + webhook_url = os.environ.get("SLACK_WEBHOOK_URL") + if not webhook_url: + sys.exit( + "ERROR: SLACK_WEBHOOK_URL is not set. Add it to .env (git-ignored):\n" + ' SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXX/YYY/ZZZ"\n' + "Create one at https://api.slack.com/apps → your app → Incoming Webhooks." + ) + if not webhook_url.startswith("https://hooks.slack.com/services/"): + sys.exit( + f"ERROR: SLACK_WEBHOOK_URL does not look like an incoming webhook:\n" + f" {webhook_url[:40]}...\n" + "Expected https://hooks.slack.com/services/T.../B.../..." + ) + + try: + status, body = post(webhook_url, payload) + except RuntimeError as exc: + sys.exit(f"ERROR: {exc}") + + if status == 200: + print(f"Incident {incident_id} posted to Slack.") + return 0 + sys.exit(f"Slack returned {status}: {body}") + + +if __name__ == "__main__": + raise SystemExit(main())