openai · seratch · Jul 24, 2026 · Jul 24, 2026 · Jul 24, 2026 · Jul 24, 2026
diff --git a/.agents/skills/integration-tests/SKILL.md b/.agents/skills/integration-tests/SKILL.md
@@ -0,0 +1,64 @@
+---
+name: integration-tests
+description: Run the packaged OpenAI Agents Python SDK integration tests from clean wheel and source-distribution environments. Use for release readiness, live OpenAI regression checks, package import compatibility, optional-extra validation, or when asked to run integration tests after examples-auto-run.
+---
+
+# Integration Tests
+
+## Overview
+
+Run the release-oriented integration suite against the exact wheel and source distribution produced by `uv build`. The runner installs both artifacts into isolated environments and validates supported imports, optional extras, OpenAI model adapters, hosted tools, Realtime, and voice workflows.
+
+## Execution requirements
+
+- Fresh isolated environments download optional dependencies from PyPI and connect to the configured API providers.
+- When the execution environment requires approval for package downloads or configured provider connections, request elevated command execution (`sandbox_permissions=require_escalated`). Retry with the required network permissions before classifying a connectivity failure as an SDK regression.
+
+## Release workflow
+
+Run this command from the repository root:
+
+```bash
+env UV_DEFAULT_INDEX=https://pypi.org/simple \
+  OPENAI_AGENTS_INTEGRATION_EXTERNAL_PROVIDERS=1 \
+  OPENAI_AGENTS_INTEGRATION_DIRECT_PROVIDERS=0 \
+  make integration-tests-release
+```
+
+- Use the release profile as the default whenever `$integration-tests` is invoked without a narrower request.
+- Use OpenRouter as the standard multi-provider gateway. Add provider-specific direct connections only when the user explicitly requests that additional credential matrix.
+- Use existing `OPENAI_API_KEY` and `OPENROUTER_API_KEY` values without printing them. Missing optional service configuration may skip capability-specific tests unless strict mode was explicitly requested.
+- The command rebuilds the wheel and source distribution, creates isolated virtual environments, checks public imports and optional dependencies, and runs the release-oriented live suites.
+- Do not run watch mode, modify source files, create a branch, commit, push, or open a pull request as part of this skill.
+
+## Paired release validation
+
+When the user requests both pre-release checks, run `$examples-auto-run` first and follow that skill's required per-example behavioral validation. Then run the command above and report the examples and integration outcomes separately. Invoking `$integration-tests` alone does not implicitly start the examples suite.
+
+## Focused commands
+
+Use a focused target only when the user specifically asks to narrow the run:
+
+```bash
+env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-packaging
+env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-core
+env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-providers
+env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-hosted
+env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-realtime
+env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-voice
+env UV_DEFAULT_INDEX=https://pypi.org/simple make integration-tests-extras
+```
+
+For the minimum supported Python package boundary, use:
+
+```bash
+env UV_DEFAULT_INDEX=https://pypi.org/simple \
+  OPENAI_AGENTS_INTEGRATION_PYTHON=3.10 \
+  make integration-tests-packaging
+```
+
+Nightly and manual profiles include additional capability-specific or higher-cost checks. Run them only when explicitly requested; use the configured OpenRouter matrix by default and include direct providers only when explicitly selected.
+
+## Reporting
+
+Report the final pass, fail, skip, and deselection counts for each isolated environment. If a command fails, identify the exact profile, package environment, failing test, and actionable error. Separate product regressions from missing credentials, unsupported hosted features, dependency installation failures, and execution-environment restrictions.
diff --git a/.agents/skills/integration-tests/agents/openai.yaml b/.agents/skills/integration-tests/agents/openai.yaml
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Integration Tests"
+  short_description: "Run packaged Python SDK integration tests"
+  default_prompt: "Use $integration-tests to run the packaged Python SDK integration suite."
diff --git a/.github/scripts/run_integration_tests.py b/.github/scripts/run_integration_tests.py
@@ -0,0 +1,234 @@
+from __future__ import annotations
+
+import argparse
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+WORKSPACE = ROOT / ".tmp" / "integration-tests"
+DIST = WORKSPACE / "dist"
+TESTS = ROOT / "integration_tests"
+EXTRAS = "any-llm,litellm,realtime,voice"
+OPTIONAL_EXTRAS = (
+    "any-llm",
+    "litellm",
+    "realtime",
+    "voice",
+    "sqlalchemy",
+    "encrypt",
+    "redis",
+    "viz",
+    "s3",
+)
+PROFILES = (
+    "packaging",
+    "core",
+    "providers",
+    "realtime",
+    "voice",
+    "hosted",
+    "extras",
+    "full",
+    "release",
+    "nightly",
+    "manual",
+)
+
+
+def run(command: list[str], *, env: dict[str, str] | None = None) -> None:
+    print(f"[integration] {' '.join(command)}", flush=True)
+    subprocess.run(command, cwd=ROOT, env=env, check=True)
+
+
+def build_distributions() -> tuple[Path, Path]:
+    DIST.mkdir(parents=True, exist_ok=True)
+    run(["uv", "build", "--out-dir", str(DIST)])
+    wheels = sorted(DIST.glob("openai_agents-*.whl"), key=lambda path: path.stat().st_mtime)
+    sdists = sorted(DIST.glob("openai_agents-*.tar.gz"), key=lambda path: path.stat().st_mtime)
+    if not wheels or not sdists:
+        raise RuntimeError("uv build did not produce both an openai-agents wheel and sdist.")
+    return wheels[-1], sdists[-1]
+
+
+def _any_llm_provider_extras(
+    *, external_providers_enabled: bool, direct_providers_enabled: bool
+) -> list[str]:
+    provider_extras: set[str] = set()
+    configured_models = os.environ.get("OPENAI_AGENTS_INTEGRATION_ANY_LLM_MODELS", "")
+    for model in configured_models.split(","):
+        provider = model.strip().partition("/")[0]
+        if provider in {"anthropic", "openrouter"}:
+            provider_extras.add(provider)
+        elif provider in {"gemini", "google"}:
+            provider_extras.add("gemini")
+
+    if external_providers_enabled:
+        if direct_providers_enabled and os.environ.get("ANTHROPIC_API_KEY"):
+            provider_extras.add("anthropic")
+        if direct_providers_enabled and (
+            os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
+        ):
+            provider_extras.add("gemini")
+        if os.environ.get("OPENROUTER_API_KEY"):
+            provider_extras.add("openrouter")
+
+    return sorted(provider_extras)
+
+
+def create_environment(
+    name: str, distribution: Path, *, extras: bool = False, optional_extra: str | None = None
+) -> Path:
+    environment = WORKSPACE / name
+    venv_command = ["uv", "venv", "--clear", str(environment)]
+    if python_version := os.environ.get("OPENAI_AGENTS_INTEGRATION_PYTHON"):
+        venv_command.extend(["--python", python_version])
+    run(venv_command)
+    python = environment / ("Scripts/python.exe" if sys.platform == "win32" else "bin/python")
+    selected_extra = EXTRAS if extras else optional_extra
+    requirement = f"{distribution}[{selected_extra}]" if selected_extra else str(distribution)
+    requirements = [requirement, "pytest", "pytest-asyncio", "pytest-timeout"]
+    external_providers_enabled = os.environ.get(
+        "OPENAI_AGENTS_INTEGRATION_EXTERNAL_PROVIDERS", ""
+    ).lower() in {"1", "true", "yes"}
+    direct_providers_enabled = os.environ.get(
+        "OPENAI_AGENTS_INTEGRATION_DIRECT_PROVIDERS", ""
+    ).lower() in {"1", "true", "yes"}
+    if extras:
+        any_llm_extras = _any_llm_provider_extras(
+            external_providers_enabled=external_providers_enabled,
+            direct_providers_enabled=direct_providers_enabled,
+        )
+        if any_llm_extras:
+            requirements.append(f"any-llm-sdk[{','.join(any_llm_extras)}]")
+    proxy_values = [
+        os.environ.get(name, "")
+        for name in (
+            "ALL_PROXY",
+            "HTTP_PROXY",
+            "HTTPS_PROXY",
+            "all_proxy",
+            "http_proxy",
+            "https_proxy",
+        )
+    ]
+    if any(value.lower().startswith("socks") for value in proxy_values):
+        requirements.append("httpx[socks]")
+    run(["uv", "pip", "install", "--python", str(python), *requirements])
+    return python
+
+
+def run_suite(
+    python: Path,
+    wheel: Path,
+    sdist: Path,
+    *,
+    selection: str,
+    environment_kind: str,
+) -> None:
+    child_env = dict(os.environ)
+    child_env.pop("PYTHONPATH", None)
+    if child_env.get("OPENAI_AGENTS_INTEGRATION_DISABLE_PROXY", "").lower() in {
+        "1",
+        "true",
+        "yes",
+    }:
+        for variable in (
+            "ALL_PROXY",
+            "HTTP_PROXY",
+            "HTTPS_PROXY",
+            "all_proxy",
+            "http_proxy",
+            "https_proxy",
+        ):
+            child_env.pop(variable, None)
+    child_env["PYTHONNOUSERSITE"] = "1"
+    child_env["OPENAI_AGENTS_INTEGRATION_WHEEL"] = str(wheel)
+    child_env["OPENAI_AGENTS_INTEGRATION_SDIST"] = str(sdist)
+    child_env["OPENAI_AGENTS_INTEGRATION_ENVIRONMENT"] = environment_kind
+    if environment_kind.startswith("extra-"):
+        child_env["OPENAI_AGENTS_INTEGRATION_EXTRA"] = environment_kind.removeprefix("extra-")
+    if not os.environ.get("OPENAI_AGENTS_INTEGRATION_ENABLE_TRACING"):
+        child_env["OPENAI_AGENTS_DISABLE_TRACING"] = "1"
+    command = [
+        str(python),
+        "-I",
+        "-m",
+        "pytest",
+        "-c",
+        str(TESTS / "pytest.ini"),
+        str(TESTS),
+        "-v",
+        "--tb=short",
+        "-m",
+        selection,
+    ]
+    run(command, env=child_env)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Run packaged openai-agents integration tests.")
+    parser.add_argument("--profile", choices=PROFILES, default="full")
+    parser.add_argument(
+        "--all",
+        action="store_true",
+        help="Include configured direct Anthropic and Gemini providers alongside OpenRouter.",
+    )
+    args = parser.parse_args()
+    if args.all:
+        os.environ["OPENAI_AGENTS_INTEGRATION_EXTERNAL_PROVIDERS"] = "1"
+        os.environ["OPENAI_AGENTS_INTEGRATION_DIRECT_PROVIDERS"] = "1"
+    wheel, sdist = build_distributions()
+    print(f"[integration] wheel={wheel.name} sdist={sdist.name} profile={args.profile}")
+
+    if args.profile in {"packaging", "core", "hosted", "full", "release", "nightly", "manual"}:
+        python = create_environment("core", wheel)
+        selections = {
+            "packaging": "packaging",
+            "core": "packaging or core",
+            "hosted": "packaging or hosted",
+            "full": "packaging or ((core or hosted) and not nightly and not manual)",
+            "release": "packaging or ((core or hosted) and not nightly and not manual)",
+            "nightly": "packaging or ((core or hosted) and not manual)",
+            "manual": "packaging or core or hosted",
+        }
+        run_suite(
+            python,
+            wheel,
+            sdist,
+            selection=selections[args.profile],
+            environment_kind="core",
+        )
+
+    if args.profile in {"providers", "realtime", "voice", "full", "release", "nightly", "manual"}:
+        python = create_environment("extended", wheel, extras=True)
+        if args.profile in {"full", "release"}:
+            selection = "(providers or realtime or voice) and not nightly and not manual"
+        elif args.profile == "nightly":
+            selection = "(providers or realtime or voice) and not manual"
+        elif args.profile == "manual":
+            selection = "providers or realtime or voice"
+        else:
+            selection = args.profile
+        run_suite(
+            python,
+            wheel,
+            sdist,
+            selection=selection,
+            environment_kind="extended",
+        )
+
+    if args.profile in {"packaging", "full", "release", "nightly", "manual"}:
+        python = create_environment("sdist", sdist)
+        run_suite(python, wheel, sdist, selection="packaging", environment_kind="sdist")
+
+    if args.profile in {"extras", "full", "release", "nightly", "manual"}:
+        for optional_extra in OPTIONAL_EXTRAS:
+            environment_kind = f"extra-{optional_extra}"
+            python = create_environment(environment_kind, wheel, optional_extra=optional_extra)
+            run_suite(python, wheel, sdist, selection="extras", environment_kind=environment_kind)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/Makefile b/Makefile
@@ -55,6 +55,62 @@ tests-parallel:
 tests-serial:
 	uv run pytest -m serial
 
+.PHONY: integration-tests
+integration-tests:
+	uv run python .github/scripts/run_integration_tests.py --profile full $(filter --all,$(MAKECMDGOALS))
+
+.PHONY: integration-tests-release
+integration-tests-release:
+	uv run python .github/scripts/run_integration_tests.py --profile release $(filter --all,$(MAKECMDGOALS))
+
+.PHONY: integration-tests-nightly
+integration-tests-nightly:
+	uv run python .github/scripts/run_integration_tests.py --profile nightly $(filter --all,$(MAKECMDGOALS))
+
+.PHONY: integration-tests-manual
+integration-tests-manual:
+	uv run python .github/scripts/run_integration_tests.py --profile manual $(filter --all,$(MAKECMDGOALS))
+
+.PHONY: integration-tests-packaging
+integration-tests-packaging:
+	uv run python .github/scripts/run_integration_tests.py --profile packaging
+
+.PHONY: integration-tests-core
+integration-tests-core:
+	uv run python .github/scripts/run_integration_tests.py --profile core
+
+.PHONY: integration-tests-providers
+integration-tests-providers:
+	uv run python .github/scripts/run_integration_tests.py --profile providers $(filter --all,$(MAKECMDGOALS))
+
+.PHONY: integration-tests-providers-external
+integration-tests-providers-external:
+	OPENAI_AGENTS_INTEGRATION_EXTERNAL_PROVIDERS=1 uv run python .github/scripts/run_integration_tests.py --profile providers $(filter --all,$(MAKECMDGOALS))
+
+.PHONY: integration-tests-providers-all
+integration-tests-providers-all:
+	uv run python .github/scripts/run_integration_tests.py --profile providers --all
+
+.PHONY: --all
+--all:
+	@:
+
+.PHONY: integration-tests-realtime
+integration-tests-realtime:
+	uv run python .github/scripts/run_integration_tests.py --profile realtime
+
+.PHONY: integration-tests-voice
+integration-tests-voice:
+	uv run python .github/scripts/run_integration_tests.py --profile voice
+
+.PHONY: integration-tests-hosted
+integration-tests-hosted:
+	uv run python .github/scripts/run_integration_tests.py --profile hosted
+
+.PHONY: integration-tests-extras
+integration-tests-extras:
+	uv run python .github/scripts/run_integration_tests.py --profile extras
+
 .PHONY: coverage
 coverage:
 

diff --git a/integration_tests/README.md b/integration_tests/README.md
@@ -0,0 +1,28 @@
+# Packaged live integration tests
+
+These tests exercise the exact wheel produced by `uv build` after installing it into clean virtual environments. The `integration_tests/` directory, repository automation metadata, and local dependency/type-checking caches are excluded from published distributions.
+
+Run the complete release-oriented matrix with:
+
+    export UV_DEFAULT_INDEX=https://pypi.org/simple
+    make integration-tests
+
+`make integration-tests-release` runs the same release-safe matrix explicitly. `make integration-tests-nightly` also includes extended capability and transport checks, while `make integration-tests-manual` includes checks reserved for an intentionally configured manual run. Focused entry points are `make integration-tests-packaging`, `make integration-tests-core`, `make integration-tests-providers`, `make integration-tests-providers-external`, `make integration-tests-providers-all`, `make integration-tests-realtime`, `make integration-tests-voice`, `make integration-tests-hosted`, and `make integration-tests-extras`.
+
+Invoke the repository-local `$integration-tests` skill to run the release profile with configured OpenRouter-backed provider checks. OpenRouter provides a single configured gateway for the standard multi-provider matrix; provider-specific direct connections are optional extensions selected explicitly. When a release review also requires runnable examples, run `$examples-auto-run` first and then `$integration-tests`.
+
+Set `OPENAI_API_KEY` for live OpenAI calls. Override `OPENAI_AGENTS_INTEGRATION_MODEL`, `OPENAI_AGENTS_INTEGRATION_REALTIME_MODEL`, `OPENAI_AGENTS_INTEGRATION_ANY_LLM_MODELS`, and `OPENAI_AGENTS_INTEGRATION_LITELLM_MODELS` when testing different models or configured providers. Provider model lists contain comma-separated adapter model names and require the credentials matching each selected provider. Set `OPENAI_AGENTS_INTEGRATION_MCP_SERVER_URL` to use another trusted DeepWiki-compatible hosted MCP server that exposes the `ask_question` tool and can answer questions about the `openai/openai-agents-python` repository.
+
+Run `make integration-tests-providers-external` with `OPENROUTER_API_KEY` to exercise current OpenAI, Anthropic, and Google models through one provider gateway. To extend the matrix with separately configured direct-provider credentials, use `make integration-tests-providers-external -- --all`, `make integration-tests-providers-all`, or `uv run python .github/scripts/run_integration_tests.py --profile providers --all`. Set `ANTHROPIC_API_KEY` and `GEMINI_API_KEY` or `GOOGLE_API_KEY` for the direct providers you want to include. Override `OPENAI_AGENTS_INTEGRATION_ANTHROPIC_MODEL`, `OPENAI_AGENTS_INTEGRATION_GEMINI_MODEL`, or the comma-separated `OPENAI_AGENTS_INTEGRATION_OPENROUTER_MODELS` to select provider models.
+
+The default general model is `gpt-5.6`, while LiteLLM function-tool cases use the Chat Completions-native `openai/gpt-4.1-mini`. This avoids LiteLLM's separate Responses API bridge and keeps the adapter regression focused on its actual Chat Completions contract.
+
+When the host requires a SOCKS proxy, the runner installs `httpx[socks]` as a test-harness dependency without changing the SDK's published requirements. Set `OPENAI_AGENTS_INTEGRATION_DISABLE_PROXY=1` when the selected environment should connect without inherited proxy settings.
+
+Set `OPENAI_AGENTS_INTEGRATION_STRICT=1` to fail rather than skip when a requested live feature is not configured. Integration tests never run as part of ordinary `make tests`.
+
+Each live test has a 75-second timeout so a stalled provider connection cannot block a release review indefinitely.
+
+Set `OPENAI_AGENTS_INTEGRATION_PYTHON` to choose the Python interpreter used for isolated environments. For example, `OPENAI_AGENTS_INTEGRATION_PYTHON=3.10 make integration-tests-packaging` verifies the minimum supported Python package and import boundary; use Python 3.11 or newer for the full adapter matrix because the AnyLLM extra requires Python 3.11.
+
+The release suite also covers canonical and supported legacy public-import identity, client-side handoffs, nested agents as tools, custom and shell tools, namespaced tool search, approval/rejection plus serialized `RunState` resume, durable SQLite sessions, explicit and server-managed conversation continuation, controlled retries, input/output and tool guardrails, explicit prompt caching, structured streaming output, provider token logprobs, hosted web search/MCP approval, hosted multi-agent streaming, programmatic-tool streaming/handoffs, multi-turn Realtime history, usage, handoffs, agent updates, voice failure propagation, and independent installation of each selected optional dependency group. The nightly profile adds extended approval matrices, parallel tool concurrency, stateless reasoning replay, reusable Responses WebSocket sessions, collected trace trees, streamed provider tool calls, Realtime audio/guardrails, and streamed-input voice pipelines.