fix: keep interrupted ingest out of execution slot

[adity982]

Summary

• open observe-only CLI ingest campaigns as created instead of running
• normalize legacy ingest clients that still send running, so interrupted uploads cannot occupy the supervisor slot
• report health as degraded when the configured single execution slot is full
• cover legacy payload normalization, interrupted-ingest/demo-start behavior, and the health threshold

Closes #59.

Validation

• uv run --extra dev pytest tests/unit -q --tb=short — 483 passed
• uv run --extra dev pytest tests/integration -q --tb=short — 152 passed
• PYTHONUTF8=1 uv run --extra dev pytest tests/reliability -q --tb=short — 2 passed
• scoped Ruff syntax/import checks, Python compilation, and git diff --check passed

The first Windows reliability run reached the smoke script but CP1252 could not print its ≥ character; rerunning unchanged under UTF-8 passed. No pending-file retry or stale-row reaper is included.

This PR was created with Codex assistance.

[vercel]

@adity982 is attempting to deploy a commit to the priyanshu's projects Team on Vercel.

A member of the Team first needs to authorize it.

[CodewithJha]

Thanks @adity982 — this is close, and the core idea (ingest rows never occupy the execution slot, health degrades at >= max) is exactly what #59 asks for. I checked it out and tested it; one gap needs closing before merge.

What is working

• ✓ CLI opens ingest as created, and Hosted normalizes legacy running to created server-side, so old clients are covered too. Normalizing on the server is the right root-cause location.
• ✓ Health degrades when running_campaigns >= max_concurrent_campaigns.
• ✓ Open (legacy running) → demo create/start succeeds → batch/complete still works is covered end to end.
• ✓ 637 tests pass locally. Mutation check: reverting ingest.py, hosted_sync.py or the health change individually each fails a test. No new ruff findings.

What still needs to change

Ingest rows are now startable through POST /api/campaigns/{id}/start. Supervisor.start_campaign only checks status == "created", so after this PR an observe-only ingest row passes that gate. On main this returns 409 (cannot start campaign in status=running). On this branch, with a probe against TestClient:

Ingest config	main	this PR
CLI-shaped (target: openai_agents)	409	410 hosted_customer_execution_removed
no target (defaults to the harness)	409	200, worker crashes with KeyError: 'population_size', row left running

The second case recreates the wedge #59 is fixing: an ingest row stuck in running that blocks demo starts and degrades health. With a fuller config it would instead run the demo harness on top of a local-CLI record and mix harness events into ingested data. It's behind auth, so this is an integrity problem, not an auth bypass, but it's a new path this change opens.

Important

The fix belongs in the shared gate, not in ingest: start_campaign should refuse ingest rows (e.g. campaign["config"].get("execution_mode") == "local_cli", which _build_campaign_config always sets) with the existing RuntimeError → 409 campaign_conflict, before the concurrency check. Please add one test: open an ingest row (legacy running body), POST .../start → 409, row still created, health still ok.

Suggested direction

• ~3 lines in supervisor.py plus one integration test; no other changes needed.
• #97 just merged and also touches tests/integration/test_api.py right after test_health, so please rebase on main first. The demo-start body in your new test uses population_size: 2 with the default elite_count: 2, which #97 now rejects, so it needs "elite_count": 1. You can also drop use_boundary_seeds from that body, since that flag is gone.

Happy to re-review as soon as it's pushed.

[adity982]

Rebased onto current main and addressed the start-path gap at exact head cceb2a1:

• CampaignSupervisor.start_campaign now rejects execution_mode=local_cli rows before the concurrency check, returning the existing 409 campaign_conflict response.
• Added regression coverage proving a legacy running ingest payload is normalized to created, cannot be started, remains created, and leaves health ok with zero running campaigns.
• Updated the demo fixture for fix(core,cli,api): reject stalled elite configs and remove unused seed flag #97's elite-count validation and removed the retired boundary-seed option.

Validation on the exact rebased head:

• uv run --locked pytest tests/unit -q — 500 passed
• uv run --extra dev pytest tests/integration -q — 156 passed
• uv run --locked pytest tests/reliability -q — 2 passed
• uv run --locked ruff check — passed
• Python compileall and git diff --check — passed

[CodewithJha]

Thanks @adity982, this is exactly the fix, in exactly the right place. Approving and merging.

Re-review of cceb2a1 (tested merged onto current main, 7e13e60)

• ✓ start_campaign now refuses execution_mode == "local_cli" rows with the existing 409 campaign_conflict, before the concurrency check. Every start goes through that one shared gate, so no per-route guards are needed.
• ✓ I probed three ingest config shapes via POST /api/campaigns/{id}/start: no target, CLI-shaped openai_agents, and a harness-looking in_process_demo. All three return 409, the row stays created, and health stays ok with running_campaigns == 0. Nothing gets stuck in running.
• ✓ Mutation check: removing the new guard makes test_ingest_campaign_cannot_be_started fail.
• ✓ The rebase is clean, and the demo-start body now uses elite_count: 1.
• ✓ Full suite: 658 passed. Ruff is clean, the offline CLI sample run works, and all GitHub Actions checks are green. The Vercel fork-deploy check is an auth-only failure and doesn't affect this.

All #59 acceptance criteria are met. Interrupted ingests no longer hold the execution slot, health degrades at >= max, and open-then-complete still works. Stale-row reaping stays out of scope, as you noted.

If you'd like to keep going

• #55 (MUT-001/MUT-010): the Hosted UI injects the API token, and FastAPI /docs is public. It's the same Hosted API surface you just worked in.
• #60 (MUT-007): the Policies page still calls retired 410 APIs. It's a smaller, self-contained fix.