Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,4 @@ src/**/.agents/

## Generated reports
Reports/
__pycache__/
44 changes: 44 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ the command. `--verbose` is stripped from argv like `--tenant` and prints to std
- Per-tenant: `~/.config/timepro-cli/tenants/{id}.json`
- Repo mappings: `~/.config/timepro-cli/repo-mappings.json`
- Feature packs: `config.json` stores `features.<name>.enabled` and `features.<name>.version` so skills and MCP can share one persistent setting.
- `TIMEPRO_CLI_CONFIG_DIR` relocates the whole config root (`~`, relative and absolute paths all
work). It is read once by `ConfigPaths`, so it also moves the tenants directory, repo mappings and
the local command log. It exists so the MCP regression harness can launch a real `tp mcp` child
process against an isolated config instead of the developer's own tenants and feature flags; use
it for any test or script that must not read or write the real config.

### Project Files
- `Directory.Build.props` centralizes shared .NET defaults (`net10.0`, implicit usings, nullable)
Expand Down Expand Up @@ -91,6 +96,18 @@ resolving an iteration by name or ID) and `TimesheetAcceptService` owns accept;
`ts accept` commands and the `UpdateTimesheet` / `AcceptSuggestedTimesheet` MCP tools are adapters
over them. Never build a `TimesheetRequest` for an edit anywhere else.

`TimesheetCreateService` is the same prepare/apply pair for new entries and owns every resolution a
create needs: sell price from the client rate for the billable type, category from the repo mapping
then the last fortnight's entries, location from the WFH defaults, deducted minutes to hours, and the
read-back that turns an empty write response into the saved row. `ts create` and the
`CreateTimesheet` MCP tool are adapters over it; never build a `TimesheetRequest` for a create
elsewhere either.

The API cannot price a row for a client with no active rate, so `PrepareAsync` stops and reports it
rather than resolving it. Creating a rate stays with the caller: `ts create` keeps its interactive
prompt, and MCP returns the `tp rate create` recovery command. Neither the service nor MCP ever
writes a rate.

`ts update` and `ts delete` refuse suggested entries locally (`tp ts accept <id>` first) rather than
letting the API answer a bare 400; the MCP delete tool shares that check. Accept fails before the API
call when the project uses iterations and none can be resolved, listing the available ones.
Expand Down Expand Up @@ -219,6 +236,29 @@ separate DTOs:
property no DTO property binds. Use it on new endpoint tests — it is what catches shape drift
before it shows up as a zeroed field.

## MCP Regression Harness

`tests/SSW.TimePro.Cli.Integration/Mcp/` holds the contract tripwire for the CLI/MCP unification
work, with its snapshots under `Goldens/Mcp/`. Adding a tool means adding to the tables — several
tests fail until you do.

- `NorthwindApi` is the single fake TimePro instance. Response bodies are serialised from the real
DTOs, never hand-written, so a fixture that stops binding fails instead of producing a golden
full of nulls. Per-case overrides go in at `OverridePriority`.
- `McpToolCatalog` declares one populated case per tool plus the generated `empty` and `apiError`
variants; `Goldens/Mcp/Tools/*.json` is the raw text each tool returned. An API failure escaping
a tool as a protocol error rather than an `isError` payload is part of what is snapshotted.
- `McpStdioClient` launches the real `tp mcp` with `TIMEPRO_CLI_CONFIG_DIR` pointing at a throwaway
config. `Goldens/Mcp/Discovery/` holds the `tools/list` snapshots with accounting off (18 tools)
and on (47); `Goldens/Mcp/Calls/` holds `tools/call` envelopes.
- `McpCliParityTable` pairs every tool with its CLI command. Differences are declared per case as
JSON paths — there is no generic normalisation — and `ExpectParity` flips to true as each slice
lands. `ToolsWithoutCliMirror` may only shrink.
- Goldens are read from and written to the source tree. Regenerate deliberately with
`UPDATE_MCP_GOLDENS=1 dotnet test tests/SSW.TimePro.Cli.Integration/` and review the diff.
- The one declared normalisation is `WeekTokens`: `WeekCoverageService` derives its window from the
machine clock and has no clock seam, so the current week's five dates become tokens.

## Testing

```bash
Expand All @@ -231,6 +271,10 @@ dotnet test tests/SSW.TimePro.Cli.Integration/
# E2E (requires staging credentials)
./scripts/e2e/run-all.sh

# Staging MCP stdio gate only (run the candidate artifact, never production)
TIMEPRO_MCP_SMOKE_TP="dotnet src/SSW.TimePro.Cli/bin/Release/net10.0/SSW.TimePro.Cli.dll" \
scripts/e2e/test-mcp-smoke.sh

# NuGet package safety audit (also used by the optional Git pre-push hook)
scripts/security/nuget-audit.sh
```
Expand Down
315 changes: 315 additions & 0 deletions scripts/e2e/mcp_smoke.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,315 @@
#!/usr/bin/env python3
"""Staging MCP stdio smoke test: discovery, Northwind reads, one write, and cleanup.

Standard library only. Driven by scripts/e2e/test-mcp-smoke.sh; see that script for the
environment variables it accepts.
"""

import datetime
import json
import os
import pathlib
import subprocess
import sys
import threading
import uuid

PROTOCOL_VERSION = "2024-11-05"
READ_TIMEOUT_SECONDS = 90

CLIENT_ID = "NWIND"
PROJECT_ID = os.environ.get("TIMEPRO_MCP_SMOKE_PROJECT", "8W52M2")
TENANT = os.environ.get("TIMEPRO_MCP_SMOKE_TENANT", "ssw-staging")
CATEGORY_ID = os.environ.get("TIMEPRO_MCP_SMOKE_CATEGORY", "WEBDEV")

REPO_ROOT = pathlib.Path(__file__).resolve().parents[2]
DISCOVERY_GOLDEN = (
REPO_ROOT
/ "tests"
/ "SSW.TimePro.Cli.Integration"
/ "Goldens"
/ "Mcp"
/ "Discovery"
/ "tools-list.default.json"
)


class SmokeFailure(Exception):
pass


class McpProcess:
"""Newline-delimited JSON-RPC over the child's stdio, with stderr drained separately."""

def __init__(self, command):
self._process = subprocess.Popen(
command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
bufsize=1,
)
self._next_id = 1
self._stderr = []
self._stderr_thread = threading.Thread(target=self._drain_stderr, daemon=True)
self._stderr_thread.start()

def _drain_stderr(self):
for line in self._process.stderr:
self._stderr.append(line.rstrip())

@property
def stderr(self):
return "\n".join(self._stderr)

def request(self, method, params=None):
request_id = self._next_id
self._next_id += 1
frame = {"jsonrpc": "2.0", "id": request_id, "method": method}
if params is not None:
frame["params"] = params
self._write(frame)
return self._read_response(request_id)

def notify(self, method, params=None):
frame = {"jsonrpc": "2.0", "method": method}
if params is not None:
frame["params"] = params
self._write(frame)

def _write(self, frame):
self._process.stdin.write(json.dumps(frame) + "\n")
self._process.stdin.flush()

def _read_response(self, request_id):
deadline = threading.Event()
timer = threading.Timer(READ_TIMEOUT_SECONDS, deadline.set)
timer.start()
try:
while True:
if deadline.is_set():
raise SmokeFailure(f"timed out waiting for response {request_id}")

line = self._process.stdout.readline()
if line == "":
raise SmokeFailure(
f"MCP host exited before answering {request_id}\nstderr:\n{self.stderr}"
)

line = line.strip()
if not line:
continue

try:
frame = json.loads(line)
except json.JSONDecodeError:
raise SmokeFailure(f"non-protocol output on stdout: {line}")

if frame.get("id") != request_id:
continue # notification or out-of-order response

if "error" in frame:
raise SmokeFailure(f"{request_id} failed: {json.dumps(frame['error'])}")

return frame["result"]
finally:
timer.cancel()

def close(self):
try:
self._process.stdin.close()
self._process.wait(timeout=10)
except Exception:
self._process.kill()


def initialize(process):
result = process.request(
"initialize",
{
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {"name": "tp-mcp-smoke", "version": "1.0.0"},
},
)
if "protocolVersion" not in result or "capabilities" not in result:
raise SmokeFailure(f"initialize did not negotiate a session: {json.dumps(result)}")
if "tools" not in result["capabilities"]:
raise SmokeFailure("server did not advertise the tools capability")
process.notify("notifications/initialized")
return result


def list_tools(process):
names = []
cursor = None
while True:
params = {"cursor": cursor} if cursor else {}
result = process.request("tools/list", params)
names.extend(tool["name"] for tool in result["tools"])
cursor = result.get("nextCursor")
if not cursor:
return names


def call_tool(process, name, arguments):
result = process.request("tools/call", {"name": name, "arguments": arguments})

if result.get("isError"):
raise SmokeFailure(f"{name} returned isError: {json.dumps(result)}")

text = next(
(block["text"] for block in result.get("content", []) if block.get("type") == "text"),
None,
)
if text is None:
raise SmokeFailure(f"{name} returned no text content: {json.dumps(result)}")

payload = json.loads(text)
# Tools still answer some failures with a legacy embedded error payload.
if isinstance(payload, dict) and "error" in payload:
raise SmokeFailure(f"{name} returned an embedded error: {payload['error']}")

return payload


def assert_non_production(tp_command):
info = subprocess.run(
tp_command + ["tenant", "info", "--tenant", TENANT, "--json"],
capture_output=True,
text=True,
check=True,
)
tenant = json.loads(info.stdout)
api_url = tenant.get("apiUrl", "")
if tenant.get("isProduction", True):
raise SmokeFailure(f"refusing to run: tenant {TENANT} resolves to production ({api_url})")
print(f" tenant {TENANT} resolves to a non-production host: {api_url}")
return api_url


def check_discovery(names):
if not DISCOVERY_GOLDEN.exists():
raise SmokeFailure(f"discovery golden missing: {DISCOVERY_GOLDEN}")

expected = {tool["name"] for tool in json.loads(DISCOVERY_GOLDEN.read_text())["tools"]}
missing = sorted(expected - set(names))
if missing:
raise SmokeFailure(f"tools/list is missing golden tools: {missing}")

print(f" tools/list returned {len(names)} tools, all {len(expected)} default tools present")


def most_recent_weekday():
day = datetime.date.today()
while day.weekday() > 4:
day -= datetime.timedelta(days=1)
return day.isoformat()


def main():
tp_command = json.loads(os.environ["TIMEPRO_MCP_SMOKE_TP"])
api_url = assert_non_production(tp_command)

token = uuid.uuid4().hex[:8]
note = f"MCP smoke {token}, safe to delete"
date = most_recent_weekday()
created_id = None

process = McpProcess(tp_command + ["mcp", "--tenant", TENANT])
try:
session = initialize(process)
print(f" initialize negotiated protocol {session['protocolVersion']}")

check_discovery(list_tools(process))

projects = call_tool(process, "get_projects_for_client", {"clientId": CLIENT_ID})
project = next((p for p in projects if p.get("value") == PROJECT_ID), None)
if project is None:
raise SmokeFailure(f"project {PROJECT_ID} not found for client {CLIENT_ID}")
print(f" project {PROJECT_ID} found: {project.get('displayText')}")

iterations = call_tool(process, "list_iterations", {"projectId": PROJECT_ID})
if not iterations:
raise SmokeFailure(f"project {PROJECT_ID} returned no iterations")
iteration_id = iterations[0]["iterationId"]
print(f" using iteration {iteration_id} ({iterations[0].get('iterationName')})")

rate = call_tool(process, "get_client_rate", {"clientId": CLIENT_ID, "date": date})
if rate is None:
raise SmokeFailure(f"no client rate for {CLIENT_ID} on {date}")
print(f" client rate present for {CLIENT_ID}")

before = call_tool(process, "get_timesheets", {"date": date})
print(f" {len(before)} existing entries on {date}")

# The write goes through MCP on purpose: create_timesheet and `tp ts create` share one
# service, so this is the gate on that service against a real server.
create_result = call_tool(
process,
"create_timesheet",
{
"clientId": CLIENT_ID,
"projectId": PROJECT_ID,
"date": date,
"startTime": "04:00",
"endTime": "04:15",
"description": note,
"categoryId": CATEGORY_ID,
"iterationId": iteration_id,
},
)

after = call_tool(process, "get_timesheets", {"date": date})
matches = [entry for entry in after if entry.get("notes") == note]
if len(matches) != 1:
raise SmokeFailure(
f"expected exactly one entry noted '{note}' on {date}, found {len(matches)}"
)

created = matches[0]
created_id = created["timeId"]
reported_id = create_result.get("timesheetId")
if reported_id not in (None, created_id):
raise SmokeFailure(
f"create_timesheet reported id {reported_id} but the row read back is {created_id}"
)
for field, expected in (
("clientId", CLIENT_ID),
("projectId", PROJECT_ID),
("date", date),
):
if created.get(field) != expected:
raise SmokeFailure(
f"created entry {created_id} has {field}={created.get(field)!r}, expected {expected!r}"
)
print(f" created and read back entry {created_id} on project {PROJECT_ID}")

call_tool(process, "delete_timesheet", {"timesheetId": created_id, "date": date})

remaining = call_tool(process, "get_timesheets", {"date": date})
if any(entry.get("timeId") == created_id for entry in remaining):
raise SmokeFailure(f"entry {created_id} still present after delete")
created_id = None
print(" deleted the smoke entry and verified its absence")

print(f"MCP smoke passed against {api_url}")
return 0
except SmokeFailure as failure:
print(f"MCP smoke FAILED: {failure}", file=sys.stderr)
if created_id is not None:
print(
f"LEFTOVER TEST DATA: timesheet {created_id} on {date} "
f"(note '{note}') was not deleted",
file=sys.stderr,
)
if process.stderr:
print(f"server stderr:\n{process.stderr}", file=sys.stderr)
return 1
finally:
process.close()


if __name__ == "__main__":
sys.exit(main())
Loading