Skip to content

Commit 7902198

Browse files
Copilotchefgs
Copilot
and
chefgs
committed
Add MCP test automation: wire-protocol tests and setup script smoke test
Agent-Logs-Url: https://github.com/cloudengine-labs/devops_os/sessions/bcbbfe9c-443e-4778-b9fa-0d9a5d00fc75 Co-authored-by: chefgs <7605658+chefgs@users.noreply.github.com>
1 parent 76e6f3d commit 7902198

3 files changed

Lines changed: 701 additions & 1 deletion

File tree

‎.github/workflows/mcp-setup-smoke.yml‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
name: MCP Setup Smoke Test
2+
3+
# What this workflow proves
4+
# ─────────────────────────
5+
# The setup_devops_os_mcp.sh script is the primary on-boarding path for users
6+
# who want to connect DevOps-OS to Claude Code. This workflow proves the
7+
# full installation flow works end-to-end on a clean Ubuntu machine:
8+
#
9+
# 1. setup_devops_os_mcp.sh --local creates a Python venv at .venv/
10+
# 2. All MCP server dependencies are installed (importable from the venv)
11+
# 3. The MCP server process starts successfully under the venv interpreter
12+
# 4. The server completes the MCP initialize handshake
13+
# 5. tools/list returns all 8 expected DevOps-OS tools
14+
# 6. The claude mcp add / claude mcp list commands are exercised via a
15+
# stub so the registration code path is validated even without a real
16+
# Claude CLI binary or API key
17+
18+
on:
19+
push:
20+
branches: [main, "copilot/**"]
21+
paths:
22+
- "mcp_server/**"
23+
- ".github/workflows/mcp-setup-smoke.yml"
24+
pull_request:
25+
branches: [main]
26+
paths:
27+
- "mcp_server/**"
28+
- ".github/workflows/mcp-setup-smoke.yml"
29+
30+
permissions:
31+
contents: read
32+
33+
jobs:
34+
mcp-setup-smoke:
35+
name: MCP Setup Smoke Test
36+
runs-on: ubuntu-latest
37+
38+
steps:
39+
- uses: actions/checkout@v4
40+
41+
- name: Set up Python 3.11
42+
uses: actions/setup-python@v5
43+
with:
44+
python-version: "3.11"
45+
46+
# ── Step 1: stub the 'claude' binary ────────────────────────────────────
47+
# The setup script calls `claude mcp list` and `claude mcp add`.
48+
# We provide a minimal stub that:
49+
# • Returns a non-zero exit code for `mcp list` (so the script skips
50+
# the "remove existing entry" path) — mirrors a fresh install.
51+
# • Exits 0 for `mcp add` and logs the invocation so we can assert
52+
# the correct arguments were passed.
53+
# This validates the registration code path without a real Claude binary.
54+
- name: Install claude CLI stub
55+
run: |
56+
mkdir -p "$HOME/.local/bin"
57+
cat > "$HOME/.local/bin/claude" << 'EOF'
58+
#!/usr/bin/env bash
59+
# Claude CLI stub for CI smoke testing.
60+
# Logs every invocation to $HOME/claude-stub.log.
61+
echo "claude stub called: $*" >> "$HOME/claude-stub.log"
62+
if [[ "$1 $2" == "mcp list" ]]; then
63+
# Return 1 so the 'grep -q "^devops-os"' check fails ─ simulates
64+
# a clean install with no prior registration.
65+
exit 1
66+
fi
67+
# All other sub-commands (mcp add, mcp remove) succeed silently.
68+
exit 0
69+
EOF
70+
chmod +x "$HOME/.local/bin/claude"
71+
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
72+
73+
# ── Step 2: run the setup script in local mode ──────────────────────────
74+
- name: Run setup_devops_os_mcp.sh --local
75+
run: bash mcp_server/setup_devops_os_mcp.sh --local
76+
77+
# ── Step 3: verify .venv was created ────────────────────────────────────
78+
- name: Verify .venv was created
79+
run: |
80+
echo "Checking for .venv directory..."
81+
test -d .venv || { echo "ERROR: .venv was not created by setup script"; exit 1; }
82+
test -f .venv/bin/python || { echo "ERROR: .venv/bin/python not found"; exit 1; }
83+
echo "✓ .venv exists: $(.venv/bin/python --version)"
84+
85+
# ── Step 4: verify MCP server dependencies are installed ────────────────
86+
- name: Verify MCP server dependencies installed in venv
87+
run: |
88+
echo "Checking that 'mcp' package is importable from .venv..."
89+
.venv/bin/python -c "import mcp; print('✓ mcp version:', mcp.__version__)" \
90+
|| { echo "ERROR: mcp package not installed in .venv"; exit 1; }
91+
92+
echo "Checking that 'yaml' package is importable from .venv..."
93+
.venv/bin/python -c "import yaml; print('✓ pyyaml imported')" \
94+
|| { echo "ERROR: pyyaml not installed in .venv"; exit 1; }
95+
96+
# ── Step 5: verify claude stub was called with the right arguments ───────
97+
- name: Verify claude mcp add was invoked with correct arguments
98+
run: |
99+
echo "Contents of claude-stub.log:"
100+
cat "$HOME/claude-stub.log"
101+
102+
# Assert 'claude mcp add' was called
103+
grep -q "mcp add" "$HOME/claude-stub.log" \
104+
|| { echo "ERROR: 'claude mcp add' was never called by the setup script"; exit 1; }
105+
106+
# Assert --transport stdio was passed
107+
grep -q "stdio" "$HOME/claude-stub.log" \
108+
|| { echo "ERROR: '--transport stdio' was not passed to 'claude mcp add'"; exit 1; }
109+
110+
# Assert the server name 'devops-os' was registered
111+
grep -q "devops-os" "$HOME/claude-stub.log" \
112+
|| { echo "ERROR: 'devops-os' server name was not passed to 'claude mcp add'"; exit 1; }
113+
114+
# Assert mcp_server.server module was referenced
115+
grep -q "mcp_server.server" "$HOME/claude-stub.log" \
116+
|| { echo "ERROR: 'mcp_server.server' module not referenced in 'claude mcp add' call"; exit 1; }
117+
118+
echo "✓ claude mcp add was called with all expected arguments"
119+
120+
# ── Step 6: verify the MCP server starts and responds via JSON-RPC ───────
121+
- name: Verify MCP server starts and responds to tools/list
122+
run: |
123+
.venv/bin/python - << 'PYEOF'
124+
import json, os, subprocess, sys
125+
126+
repo_root = os.getcwd()
127+
env = {**os.environ, "PYTHONPATH": repo_root}
128+
venv_python = os.path.join(repo_root, ".venv", "bin", "python")
129+
130+
print("Starting MCP server via venv python:", venv_python)
131+
proc = subprocess.Popen(
132+
[venv_python, "-m", "mcp_server.server"],
133+
stdin=subprocess.PIPE,
134+
stdout=subprocess.PIPE,
135+
stderr=subprocess.PIPE,
136+
text=True,
137+
env=env,
138+
cwd=repo_root,
139+
)
140+
141+
def send(method, params=None, req_id=None):
142+
msg = {"jsonrpc": "2.0", "method": method}
143+
if req_id is not None:
144+
msg["id"] = req_id
145+
if params:
146+
msg["params"] = params
147+
proc.stdin.write(json.dumps(msg) + "\n")
148+
proc.stdin.flush()
149+
if req_id is not None:
150+
return json.loads(proc.stdout.readline())
151+
152+
# MCP initialize handshake
153+
init_resp = send("initialize", {
154+
"protocolVersion": "2024-11-05",
155+
"capabilities": {},
156+
"clientInfo": {"name": "smoke-test", "version": "1.0"},
157+
}, req_id=1)
158+
159+
if "error" in init_resp:
160+
proc.terminate()
161+
print("ERROR: initialize failed:", init_resp["error"])
162+
sys.exit(1)
163+
164+
proto_ver = init_resp["result"]["protocolVersion"]
165+
server_name = init_resp["result"]["serverInfo"]["name"]
166+
print(f"✓ Initialize handshake OK — server: {server_name!r}, protocol: {proto_ver}")
167+
168+
# Send notifications/initialized (required before tool calls)
169+
send("notifications/initialized")
170+
171+
# tools/list
172+
list_resp = send("tools/list", {}, req_id=2)
173+
if "error" in list_resp:
174+
proc.terminate()
175+
print("ERROR: tools/list failed:", list_resp["error"])
176+
sys.exit(1)
177+
178+
tools = list_resp["result"]["tools"]
179+
names = {t["name"] for t in tools}
180+
expected = {
181+
"generate_github_actions_workflow",
182+
"generate_gitlab_ci_pipeline",
183+
"generate_jenkins_pipeline",
184+
"generate_k8s_config",
185+
"generate_argocd_config",
186+
"generate_sre_configs",
187+
"scaffold_devcontainer",
188+
"generate_unittest_config",
189+
}
190+
missing = expected - names
191+
if missing:
192+
proc.terminate()
193+
print("ERROR: missing tools:", missing)
194+
sys.exit(1)
195+
196+
print(f"✓ tools/list returned {len(tools)} tools, all 8 DevOps-OS tools present")
197+
for t in sorted(tools, key=lambda x: x["name"]):
198+
print(f" • {t['name']}")
199+
200+
# Quick tools/call sanity check: generate a GHA workflow
201+
call_resp = send("tools/call", {
202+
"name": "generate_github_actions_workflow",
203+
"arguments": {"name": "smoke-test-app", "languages": "python"},
204+
}, req_id=3)
205+
206+
if "error" in call_resp:
207+
proc.terminate()
208+
print("ERROR: tools/call failed:", call_resp["error"])
209+
sys.exit(1)
210+
211+
content_text = call_resp["result"]["content"][0]["text"]
212+
assert "smoke-test-app" in content_text, "app name not in GHA output"
213+
assert "runs-on:" in content_text, "not a valid GHA YAML"
214+
print("✓ tools/call generate_github_actions_workflow returned valid GHA YAML")
215+
216+
proc.terminate()
217+
proc.wait(timeout=5)
218+
print("\nAll smoke checks passed ✓")
219+
PYEOF

‎.github/workflows/sanity.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,11 @@ jobs:
7070
- name: "Scenario: MCP server — Dev container tool"
7171
run: pytest tests/test_comprehensive.py::TestMCPServerDevcontainer -v
7272

73+
# ── MCP wire-protocol tests ───────────────────────────────────────────
74+
75+
- name: "Scenario: MCP wire protocol — handshake, tool discovery, invocation"
76+
run: pytest tests/test_mcp_protocol.py -v
77+
7378
# ── AI skills definitions ──────────────────────────────────────────────
7479

7580
- name: "Scenario: AI skills definitions (OpenAI & Claude)"
@@ -88,7 +93,7 @@ jobs:
8893
- name: Generate combined HTML report
8994
if: always()
9095
run: |
91-
pytest cli/test_cli.py mcp_server/test_server.py tests/test_comprehensive.py \
96+
pytest cli/test_cli.py mcp_server/test_server.py tests/test_comprehensive.py tests/test_mcp_protocol.py \
9297
--html=sanity-report.html --self-contained-html -q
9398
9499
- name: Upload sanity test report

0 commit comments

Comments
 (0)