Skip to content

Add model discovery integration tests without managed config - #662

Open
andy-xu-db wants to merge 1 commit into
andy/model-discovery-integration-testsfrom
andy/model-discovery-integration-unmanaged
Open

andy-xu-db wants to merge 1 commit into
andy/model-discovery-integration-testsfrom
andy/model-discovery-integration-unmanaged

Conversation

@andy-xu-db

@andy-xu-db andy-xu-db commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add Tests-table Cases 13–24, where managed config does not exist, stacked on Add model discovery integration tests with managed config #655.
  • Cover 12 named tests and 20 configured/fresh executions across Claude Code and Codex.
  • Add only the remaining unmanaged discovery inputs, fixtures, and Claude-owned cache inspection support.
  • Validate Claude discovery only after the TUI starts. Unity Gateway never primes, refreshes, writes, or locks Claude's gateway-models.json before launch.

Cases

Cases Scope Executions
13–16 saved location and discovery-disabled native fallback 4
17–24 provider/location discovery and disabled fallback, configured + fresh 16

Each test function includes its Tests-table case number in its name (test_case_13_... through test_case_24_...). Together with #655, the stack has 44 numbered discovery executions.

Live result

The fresh restacked run 35144561045 exercised Claude Code 2.1.268 and Codex 0.154.0 against the existing integration workspace.

  • Cases 15, 16, and 21–24 pass in all 10 executions: disabling scoped discovery preserves each agent's native picker/catalog and creates no UG-owned scoped catalog.
  • Claude Cases 13 and 19 open the native picker and create no gateway-models.json. Both Case 17 executions create the correctly scoped Claude-owned cache, but the picker still renders native models.
  • Codex Cases 14, 18, and 20 return the native Codex catalog in all five executions because the workspace backend reports /ai-gateway/codex/v1/models unavailable.
  • These remain hard assertions: there are no skips, xfails, retries, or weaker expectations.

Claude Code currently does not forward the scoped custom routing header on its native /v1/models request. The Codex path needs the workspace backend catalog endpoint enabled for provider- and location-scoped requests. The inherited managed signal jobs also stop before pytest on the same M2M HTTP 403 described in #655.

Stack

Verification

  • Head: 89817938da804e588b10c8848a0328d3d21cb555.
  • tests/test_integration_contract.py: 5 passed.
  • This PR adds 20 numbered executions; cumulative discovery collection is 44.
  • Ruff, formatting, and diff checks passed.
  • No local Databricks profile was selected.

@andy-xu-db andy-xu-db changed the title Add unmanaged model discovery integration tests Add model discovery integration tests without managed config Sep 16, 2026
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 5204d14 to 9751ad8 Compare September 16, 2026 15:02
@andy-xu-db
andy-xu-db changed the base branch from main to codex/model-discovery-test-stack September 16, 2026 15:02
@andy-xu-db
andy-xu-db force-pushed the codex/model-discovery-test-stack branch from d85847c to 2c0b3df Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 9751ad8 to 88b4d6f Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db changed the base branch from codex/model-discovery-test-stack to andy/model-discovery-integration-tests September 16, 2026 20:06
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch 2 times, most recently from 8981793 to e907761 Compare September 17, 2026 19:56
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from e907761 to e4a7d5c Compare September 17, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant