Configure and instrument simulation observability

[anth-volk]

Fixes #685

Summary

• replace Logfire initialization and legacy global observability calls with explicitly owned policyengine-observability 3.x runtimes
• require a 1.0 trace sampling ratio in every simulation runtime even if the surrounding environment supplies a lower value
• keep simulation entry, gateway, executor, dispatch, and calculation instrumentation in this repository
• configure Cloud Run standard-output logging, Modal direct Cloud Logging, and OTLP export independently
• pass the Google Cloud project, log name, collector endpoint, service account, and workload identity provider from GitHub Actions variables only during deployment
• remove the unused executor settings and repository variables from the previous observability implementation
• use X-PolicyEngine-Observability-Id as the only HTTP source and response transport for observability_id
• persist the identifier with asynchronous job metadata so every polling response restores the report's original value
• pass validated request, trace, job, observability, and simulation context to Modal functions through the typed _observability_context object
• keep submission_claim_id as separate execution metadata in _telemetry
• accept and discard legacy JSON identity fields during the deployment and rollback window; they never override the HTTP header or enter worker context
• remove observability identifiers from request and response JSON schemas
• preserve application responses and simulation results when observability setup, recording, credentials, or export fail
• deploy the executor before the gateway so each receiving Modal function understands _observability_context before a caller sends it
• remove the Logfire secret and dependency configuration
• include the packaging production dependency required when the Modal image imports the simulation contract

Deployment dependencies

1. policyengine-observability 3.0.0 must remain available on PyPI; all six affected lock files contain its published artifact hashes.
2. Add Stage 12 observability identifier schema policyengine-api#3852 must be merged and its nullable Stage 12 observability_id database migration must be applied.
3. Deploy this PR before Instrument and provision API v1 observability policyengine-api#3850. Within this repository, deploy the executor before the gateway.
4. Deploy Instrument and provision API v1 observability policyengine-api#3850 after this PR.

The receiver permits the currently deployed API's run_id, process_id, request_id, traceparent, and body observability_id keys during this sequence only so requests remain valid. It discards those keys. A valid X-PolicyEngine-Observability-Id header is used when present; otherwise the simulation request boundary generates a UUID. Roll back PolicyEngine/policyengine-api#3850 before rolling back this PR.

Verification

• uv lock --check passed for all six lock files
• Ruff formatting and lint checks passed for all changed Python files
• make test passed across the repository after the canonical transport change
• after the final legacy-field compatibility adjustment: 11 simulation observability tests, 49 request contract tests, and 69 gateway endpoint tests passed
• focused policyengine-fastapi observability tests passed with the required coverage threshold
• the production-only gateway environment imported the simulation contract and parsed version 3.0.0, confirming that packaging is installed without development dependencies
• the simulation observability and Cloud Run simulation-entry Pyright checks passed with no errors
• GitHub Actions workflow YAML and deployment shell syntax checks passed

The Household API and UK Chat are outside this change.

[anth-volk]

The rollout dependencies are now explicit:

1. Merge Add Stage 12 observability identifier schema policyengine-api#3852 and apply its nullable Stage 12 observability_id migration.
2. Deploy this PR.
3. Deploy Instrument and provision API v1 observability policyengine-api#3850.

Commit 192cdca adds compatibility with the identifiers emitted by the currently deployed API, rejects conflicting old and new identifiers, and requires the executor deployment to finish before the gateway deployment begins. It also fixes the stale executor tests that failed in the previous CI run. The focused regression checks pass locally: 7 observability, 45 request contract, 19 deployment definition, and 53 executor tests.

[anth-volk]

Rebased this PR onto current main (6884241) after PolicyEngine/policyengine-api#3852 merged.

• Preserved the new Stage 12 deployment completion checks added on main.
• No additional simulation contract change was required for #3852: the PR still accepts run_id as observability_id and process_id as submission_claim_id during the deployment and rollback period, while emitting the new response fields.
• Confirmed the Stage 12 persistence model still stores the nullable 36-character observability_id expected by the merged API schema.

Checks run locally:

• Ruff formatting: 123 source files passed.
• Lock validation: all seven affected package lock files passed.
• Observability library: 23 passed.
• Simulation contract compatibility: 53 passed.
• Stage 12 persistence: 6 passed.
• Simulation entry: 96 passed.
• Simulation gateway: 195 passed.
• Simulation executor: 219 passed.
• Shared FastAPI observability: 11 passed.

The rebased head is adf5b89.

[anth-volk]

Commit 3e5114e establishes one canonical observability_id path.

• HTTP requests and responses use X-PolicyEngine-Observability-Id.
• The gateway persists that identifier with asynchronous job metadata and restores it on poll responses.
• Modal calls receive the identifier and trace data only through validated _observability_context.
• _telemetry retains submission_claim_id and other simulation metadata but contains no observability identifier.
• Legacy body identity fields are accepted and discarded during deployment compatibility; they cannot replace the header value.
• Public request and response JSON schemas no longer expose observability_id.

Deployment order: database precursor API #3852, this PR, then API v1 #3850. Within this deployment, update the executor before the gateway. Roll back API v1 #3850 before rolling this PR back.

The full repository test command passed. After the final compatibility adjustment, the focused observability, contract, and gateway suites also passed with 11, 49, and 69 tests respectively.

[anth-volk]

Follow-up commit 9113e1b removes the remaining Stage 12 fallback path:

• the HTTP request middleware's validated identifier is now required by the Stage 12 backend
• Stage 12 removes any identifier found in captured runtime context before inserting the explicit request identifier
• the coordinator no longer reapplies the identifier from the persisted parent record

The database field remains available for durable lookup and diagnostics. Runtime propagation into Modal now occurs only through _observability_context.

Verification: 67 simulation-entry tests and 29 Stage 12 executor tests passed together; the focused backend suite passed again with 13 tests, and Ruff formatting and lint checks passed.

[anth-volk]

Implemented the latest review fixes in 8940fc4:

• The shared FastAPI setup now installs an inner correlation layer after request tracing begins, so the canonical observability_id is attached to the active request span and request.completed record.
• This fixes both the Cloud Run simulation entry service and the Modal routing service, which use the same middleware ordering.
• Backend identity context is now attached while the entry service request runtime is active.
• Added regression tests that read the active runtime context from inside a request.

Verification:

• Full repository unit suite passed: 583 executor tests, 145 entry-service tests, 212 routing-service tests, 102 contract tests, 28 shared-observability tests, and 10 Stage 12 persistence tests (plus documented skips/deselections).
• Focused Ruff checks for all changed files passed.

The repository-wide make check command still reports existing lint findings in untouched modules and cannot find pyright after the test-only dependency sync. The changed files pass their focused Ruff checks.

[anth-volk]

Implemented temporary input compatibility in c402b59.

• Incoming telemetry now maps legacy process_id to canonical submission_claim_id.
• A canonical submission_claim_id takes precedence when both fields are present.
• run_id remains ignored, and the HTTP header remains the only source of observability_id.
• The compatibility code contains an explicit comment requiring removal after the API v1 deployment and rollback period.

Verification:

• The observability, contract, and gateway focused tests passed (12, 49, and 69 tests).
• Ruff formatting and lint checks passed for the changed files.