Instrument and provision API v1 observability

[anth-volk]

Fixes #3847

Summary

• configure one API-owned policyengine-observability 3.x runtime for Flask and ASGI request handling
• require a 1.0 trace sampling ratio in the runtime even if the surrounding environment supplies a lower value
• add economy request, cache decision, submission, polling, result, and persistence spans and events
• create one durable observability_id at the report boundary and use X-PolicyEngine-Observability-Id as its only HTTP transport
• read the identifier from simulation response headers; request and response JSON contain no observability identifier
• retain submission_claim_id as separate execution metadata and propagate W3C trace context through the simulation client
• emit structured JSON to standard output for Cloud Run collection while keeping log and OpenTelemetry destinations independently configurable
• move the API v1 collector, log routing, IAM, dashboard, alert, verification, and operations definitions into this repository
• read collector and trace-correlation settings from GitHub Actions variables during deployment

Deployment dependencies

1. policyengine-observability 3.0.0 must remain available on PyPI; the lock file contains its published artifact hashes.
2. Add Stage 12 observability identifier schema #3852 must be merged and its nullable Stage 12 observability_id database migration must be applied.
3. Configure and instrument simulation observability policyengine-sim-api#686 must be deployed. It accepts requests from the currently deployed API while treating the HTTP header as the only observability identifier source.
4. This PR can then be deployed. Roll back this PR before rolling back Configure and instrument simulation observability policyengine-sim-api#686.

Verification

• uv lock --check passed
• Ruff formatting and lint checks passed for all changed Python files
• focused simulation client and economy service suite: 180 tests passed
• focused observability runtime and migration-context suite: 23 tests passed
• the earlier full unit suite passed with 2,120 tests and 45 skipped
• focused deployment asset and runtime tests passed with 8 tests
• focused full-sampling runtime and Cloud Run deployment tests passed with 2 tests
• GitHub Actions workflow YAML parsed successfully

The Household API and UK Chat are outside this change.

[codecov]

Codecov Report

❌ Patch coverage is 89.17647% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.50%. Comparing base (3069d37) to head (9d2a186).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
policyengine_api/services/budget_window_cache.py	66.66%	18 Missing and 14 partials ⚠️
policyengine_api/services/economy_service.py	94.93%	1 Missing and 3 partials ⚠️
policyengine_api/asgi_factory.py	95.45%	1 Missing and 2 partials ⚠️
policyengine_api/observability/identifiers.py	84.61%	2 Missing ⚠️
policyengine_api/observability/runtime.py	90.47%	2 Missing ⚠️
policyengine_api/asgi.py	50.00%	1 Missing ⚠️
policyengine_api/gcp_logging.py	92.85%	0 Missing and 1 partial ⚠️
policyengine_api/migration_logging.py	96.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3850       +/-   ##
===========================================
+ Coverage   46.37%   87.50%   +41.12%     
===========================================
  Files         178      197       +19     
  Lines       10421    11795     +1374     
  Branches     1759     2065      +306     
===========================================
+ Hits         4833    10321     +5488     
+ Misses       5199      894     -4305     
- Partials      389      580      +191     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[anth-volk]

This PR now documents its rollout dependencies explicitly:

1. Merge Add Stage 12 observability identifier schema #3852 and apply its nullable Stage 12 observability_id migration.
2. Deploy Configure and instrument simulation observability policyengine-sim-api#686, which now accepts both the currently deployed API identifiers and the new observability identifiers.
3. Deploy this PR.

The schema changes currently present in this branch overlap with #3852. After #3852 merges, this branch should be updated from master; the overlapping schema diff will then disappear while this PR retains the API instrumentation that uses the new nullable field.

[anth-volk]

Rebased onto master at 3069d37a after #3852 merged. The duplicate Stage 12 migration, ORM field, and shared revision edits are now supplied only by master; this PR consumes the merged nullable observability_id field while retaining identifier propagation, persistence, the stage registry, and telemetry behavior. Local checks passed: repository formatting (473 files unchanged), Ruff lint, migration contract validation, and 433 focused tests.

[anth-volk]

Commit f5884d6e makes X-PolicyEngine-Observability-Id the sole transport for observability_id between API v1 and the simulation API.

• API v1 no longer puts observability_id in _telemetry.
• It reads the identifier from simulation response headers for submission and polling.
• submission_claim_id remains separate execution metadata.
• Focused simulation client and economy service tests pass: 180 tests.

Deployment order: database precursor #3852, simulation API #686, then this PR. Rollback order is the reverse for the two application PRs.

[anth-volk]

Implemented the latest review fixes in da8d7511:

• Native FastAPI routes now start and finish the shared observability request lifecycle, attach request_id and observability_id, emit response trace headers, and retain exception isolation around every observability call.
• Flask fallback routes remain under the existing Flask adapter and do not receive a duplicate ASGI lifecycle.
• The Modal Workload Identity condition, workload inventory, and _Default log exclusion now include exact policyengine-simulation-v2-py<major>-<minor>-<patch> Stage 12 application names.
• Applied the corresponding provider condition and _Default exclusion to the live policyengine-observability project, then verified the exact resources and complete provider/sink lists.

Verification:

• 2229 passed, 45 skipped across the API v1 test selection used by make test.
• Focused Ruff checks passed.
• API v2 mypy checks passed for all 97 configured source files.
• Migration contract quality checks passed.

[anth-volk]

Implemented the follow-up identifier durability fix in 9d2a1869.

• Budget-window cache state is now one versioned document for starting, submitted, completed, or failed status.
• The document stores the observability_id with the ownership claim, batch job identifier, completed result, or failure. Success and failure records therefore retain the identifier for the same lifetime as the cached outcome.
• Requests that encounter existing work adopt the identifier from that state before returning or polling. A request that loses the initial submission claim also rereads the state and adopts the winning request's identifier.
• Deterministic submission failures and worker failures are retained with the same identifier instead of deleting the batch identifier and losing correlation state.

Focused verification:

• 144 budget-window cache, economy service, and typed worker polling tests passed.
• Ruff passed for all changed modules and tests.
• The new cache module passed mypy independently.

I stopped the broader API test run at the user's request and did not use it as verification for this update.