fix(stlc): restore the custom-code tree a stale seal anchor reverted

A build on 2026-09-21 (567abff "Build SDK") removed 5,224 lines across
105 files from this trunk -- 14 source files and 10 test files deleted
outright, including lib/utils/metadata_filters.py,
lib/core/observability/sgp_obs_setup.py, lib/core/temporal/logging.py and
lib/core/adapters/llm/_genai_metrics.py. 95 of the 96 changed paths were
under the hand-written src/agentex/lib/ and tests/ trees.

An stlc seal replays the content diff between its `base` and `integrated`
anchors, so anything the trunk gained past `integrated` is overwritten
rather than merely skipped. The tracking file in effect named an
`integrated` commit dated six days earlier that lived on a side branch,
not on this trunk -- and the back-sync that had just brought the custom
code here was not an ancestor of it. The replay therefore restored a tree
predating the custom code entirely, and every command reported success.

This restores the affected paths from the production trunk, which kept
all 24 files and is the authoritative copy. Verified: the diff against
production for src/agentex/lib/, tests/, adk/README.md and
adk/pyproject.toml is now empty.

Deliberately NOT restored, because this trunk is correct and production
is stale or the file does not apply:

  src/agentex/_client.py   production still defaults to localhost; this
                           trunk carries https://agentex.sgp.scale.com,
                           which is what stainless.yml configures
  .stats.yml               the SaaS-only spec/config hashes are gone by
                           design under self-hosted codegen
  .github/**               this trunk's own CI work
  release-please-config.json

Nothing shipped from the damage: both agentex-sdk-v0.28.1 and
agentex-client-v0.28.1 resolve to the production trunk with the custom
tree intact. It surfaced only because the promote gate refused to carry
the deletion into production.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: aringuyen3

adk/README.md

@@ -27,6 +27,26 @@ This automatically pulls in [`agentex-client`](../) (the slim Stainless-generate
 
 The two packages contribute disjoint files to the `agentex.*` namespace — `agentex/lib/*` ships only from `agentex-sdk`.
 
+## Workflow logging
+
+Use the workflow logger in Temporal workflow code:
+
+```python
+from agentex.lib.core.temporal.logging import make_workflow_logger
+
+logger = make_workflow_logger(__name__)
+```
+
+It suppresses logs while Temporal replays recorded history and adds top-level
+`workflow_id` and `run_id` fields during workflow execution. It preserves the
+message, caller fields, and exception details. Outside workflows, including in
+activities, it behaves like the ordinary SDK logger.
+
+New Temporal templates use this helper. Existing agents must replace their own
+workflow loggers to get the same behavior. This does not create trace context or
+add trace IDs to workflows that lack it. Temporal's worker diagnostics still report
+replay failures.
+
 ## Repo layout
 
 This package is hand-authored and lives at `adk/` inside [scaleapi/scale-agentex-python](https://github.com/scaleapi/scale-agentex-python). Stainless codegen never touches `adk/**` — it's outside the generated surface. The sibling `agentex-client` package lives at the repo root and IS Stainless-generated.

adk/pyproject.toml

@@ -65,6 +65,7 @@ dependencies = [
 # agentex/lib/* uses `from typing import override` (3.12+) in 19 files.
 # The slim agentex-client keeps 3.11 support.
 requires-python = ">= 3.12,<4"
+
 classifiers = [
   "Typing :: Typed",
   "Intended Audience :: Developers",
@@ -76,6 +77,23 @@ classifiers = [
   "License :: OSI Approved :: Apache Software License",
 ]
 
+# No `obs` extra, deliberately — do not add one for sgp-obs.
+#
+# sgp-obs is not on public PyPI (it is served from Scale's curated CodeArtifact
+# mirror), and declaring it in [project.optional-dependencies] makes THIS repo's uv
+# workspace unresolvable: `uv sync` re-locks, locking must resolve every declared
+# optional dependency of every workspace member, and there is no way to exempt one.
+# Measured: `uv lock --check`, `uv sync --all-extras`, plain `uv sync` with no extras,
+# and `uv sync --all-extras --no-extra obs` all fail (`--no-extra` filters what is
+# installed, not what is resolved); `uv lock` has no `--no-extra`; and
+# `[tool.uv] override-dependencies` does not exempt it either. Only `--frozen` works,
+# which would leave nobody able to re-lock this repo again.
+#
+# So the dependency is the AGENT's to declare — `sgp-obs[genai-auto,http,otlp]`
+# against the mirror — and the SDK wires it when it is importable. See
+# agentex/lib/core/observability/sgp_obs_setup.py; nothing imports sgp_obs outside a
+# try, so a plain `pip install agentex-sdk` is unaffected either way.
+
 [project.urls]
 Homepage = "https://github.com/scaleapi/scale-agentex-python"
 Repository = "https://github.com/scaleapi/scale-agentex-python"

src/agentex/lib/adk/utils/_modules/client.py

@@ -1,3 +1,4 @@
+import os
 from typing import override
 
 import httpx
@@ -26,7 +27,50 @@ def auth_flow(self, request):
         yield request
 
 
+# HTTP timeouts for the AgentEx client, in seconds. Defaults match the SDK's
+# DEFAULT_TIMEOUT, so leaving these unset changes nothing.
+_TIMEOUT_ENV_DEFAULTS = {
+    "connect": ("AGENTEX_CLIENT_CONNECT_TIMEOUT_SECONDS", 5.0),
+    "read": ("AGENTEX_CLIENT_READ_TIMEOUT_SECONDS", 300.0),
+    "write": ("AGENTEX_CLIENT_WRITE_TIMEOUT_SECONDS", 300.0),
+    "pool": ("AGENTEX_CLIENT_POOL_TIMEOUT_SECONDS", 300.0),
+}
+
+
+def _timeout_from_env() -> httpx.Timeout:
+    """Build the client timeout from environment variables.
+
+    Read from ``os.environ`` rather than from ``EnvironmentVariables``. That model
+    is loaded by worker startup and by ``EnvAuth.auth_flow`` on every request, and
+    ``agentex.lib.adk.utils`` builds a client at import time, so a field added
+    there would make a malformed timeout break all three. Reading here keeps the
+    blast radius to the one value that is actually wrong.
+
+    The connect timeout is the one worth raising: an AgentEx backend accepts
+    connections serially, so connect latency grows with the number of callers and
+    the 5s default is reached when a few hundred are in flight.
+    """
+    values = {}
+    for field, (env_var, default) in _TIMEOUT_ENV_DEFAULTS.items():
+        raw = os.environ.get(env_var)
+        if raw is None or raw.strip() == "":
+            values[field] = default
+            continue
+        try:
+            values[field] = float(raw)
+        except ValueError as exc:
+            raise ValueError(f"{env_var} must be a number in seconds, got {raw!r}") from exc
+    return httpx.Timeout(**values)
+
+
 def create_async_agentex_client(**kwargs) -> AsyncAgentex:
+    """Create an AsyncAgentex client.
+
+    An explicit ``timeout=`` always wins; otherwise the timeout comes from the
+    AGENTEX_CLIENT_*_TIMEOUT_SECONDS environment variables.
+    """
+    if "timeout" not in kwargs:
+        kwargs["timeout"] = _timeout_from_env()
     client = AsyncAgentex(**kwargs)
     client._client.auth = EnvAuth()
     return client

src/agentex/lib/cli/debug/debug_handlers.py

@@ -16,6 +16,7 @@
     pass
 
 from agentex.lib.utils.logging import make_logger
+from agentex.lib.cli.utils.cli_utils import SUBPROCESS_STREAM_LIMIT
 
 from .debug_config import DebugConfig, resolve_debug_port
 
@@ -66,6 +67,7 @@ async def start_temporal_worker_debug(
         env=debug_env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 
@@ -119,6 +121,7 @@ async def start_acp_server_debug(
         env=debug_env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 

src/agentex/lib/cli/handlers/deploy_handlers.py

@@ -389,6 +389,8 @@ def merge_deployment_configs(
             _deep_merge(helm_values, agent_env_config.helm_overrides)
         logger.info(f"After-merge helm values: {helm_values}")
 
+    _stamp_agent_version(helm_values, set(all_env_vars) | {var["name"] for var in secret_env_vars})
+
     # Set final environment variables
     # Environment variable precedence: manifest -> environments.yaml -> secrets (highest)
     if all_env_vars:
@@ -430,6 +432,14 @@ def _deep_merge(base_dict: dict[str, Any], override_dict: dict[str, Any]) -> Non
             base_dict[key] = value
 
 
+def _stamp_agent_version(helm_values: dict[str, Any], declared_env_names: set[str]) -> None:
+    """Set global.agent.version from the merged image tag unless the deployment declares AGENT_VERSION itself."""
+    if EnvVarKeys.AGENT_VERSION.value in declared_env_names:
+        # Chart >=0.6.0 renders global.agent.version as a second AGENT_VERSION env entry.
+        return
+    helm_values["global"]["agent"].setdefault("version", helm_values["global"]["image"]["tag"])
+
+
 def create_helm_values_file(helm_values: dict[str, Any]) -> str:
     """Create a temporary helm values file"""
     with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f:

src/agentex/lib/cli/handlers/run_handlers.py

@@ -12,6 +12,7 @@
 from agentex.lib.cli.debug import DebugConfig, start_acp_server_debug, start_temporal_worker_debug
 from agentex.lib.utils.logging import make_logger
 from agentex.config.agent_manifest import AgentManifest
+from agentex.lib.cli.utils.cli_utils import SUBPROCESS_STREAM_LIMIT
 from agentex.lib.cli.utils.path_utils import (
     get_file_paths,
     calculate_uvicorn_target_for_local,
@@ -23,6 +24,11 @@
 logger = make_logger(__name__)
 console = Console()
 
+# How many consecutive unreadable lines to skip before giving up on the stream.
+# Skipping is only known-safe for the limit-overrun case; this bounds the damage
+# if some other error repeats without consuming anything.
+MAX_CONSECUTIVE_READ_ERRORS = 100
+
 
 class RunError(Exception):
     """An error occurred during agent run"""
@@ -215,6 +221,7 @@ async def start_acp_server(
         env=env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 
@@ -234,23 +241,68 @@ async def start_temporal_worker(
         env=env,
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
+        limit=SUBPROCESS_STREAM_LIMIT,
     )
 
 
 async def stream_process_output(process: asyncio.subprocess.Process, prefix: str):
-    """Stream process output with prefix"""
+    """Stream process output with prefix.
+
+    This loop is the only reader of the child's stdout pipe. If it ever stops
+    reading, the pipe fills and the child blocks forever inside ``write()``,
+    which presents as a silent freeze: 0% CPU, no further logs, no traceback.
+    So a single unreadable line must never end the loop.
+    """
     try:
         if process.stdout is None:
             return
+        consecutive_read_errors = 0
         while True:
-            line = await process.stdout.readline()
+            try:
+                line = await process.stdout.readline()
+            except ValueError as e:
+                # readline() raises ValueError when a line exceeds the stream limit.
+                # In *that* case it has already discarded the line and resumed the
+                # transport, so skipping it makes guaranteed progress. Any other
+                # ValueError carries no such guarantee, and retrying it forever would
+                # spin without draining. We cannot tell the two apart (readline
+                # flattens LimitOverrunError into a bare ValueError), so bound the
+                # retries and let the outer handler report the hang risk.
+                consecutive_read_errors += 1
+                if consecutive_read_errors > MAX_CONSECUTIVE_READ_ERRORS:
+                    raise
+                logger.warning(
+                    f"Skipping an unreadable line from {prefix}: {e!r} "
+                    f"(consecutive failure {consecutive_read_errors}/{MAX_CONSECUTIVE_READ_ERRORS}). "
+                    f"If this says the chunk exceeded the limit, raise limit= on this "
+                    f"process's create_subprocess_exec."
+                )
+                continue
+
+            consecutive_read_errors = 0
+
             if not line:
                 break
-            decoded_line = line.decode("utf-8").rstrip()
+
+            try:
+                decoded_line = line.decode("utf-8").rstrip()
+            except UnicodeDecodeError as e:
+                logger.warning(f"Dropped an undecodable log line from {prefix} ({e}).")
+                continue
+
             if decoded_line:  # Only print non-empty lines
                 console.print(f"[dim]{prefix}:[/dim] {decoded_line}")
     except Exception as e:
-        logger.debug(f"Output streaming ended for {prefix}: {e}")
+        # The escalation path, including for the re-raise above. Anything reaching
+        # here ends the loop, so the child is now at risk of blocking on a full pipe.
+        # Warning rather than debug: this used to be a debug() that make_logger could
+        # never emit, which is why three freezes produced no clue.
+        # CancelledError derives from BaseException, so the auto-reload path that
+        # cancels these tasks passes straight through and is unaffected.
+        logger.warning(
+            f"Output streaming for {prefix} stopped on {e!r}. "
+            f"Nothing is draining its stdout now, so {prefix} will hang once the pipe fills."
+        )
 
 
 async def run_agent(manifest_path: str, debug_config: "DebugConfig | None" = None):

src/agentex/lib/cli/templates/PRIVATE_INDEX.md

@@ -0,0 +1,74 @@
+# The private package index in scaffold Dockerfiles
+
+Every scaffold Dockerfile mounts a build secret named `codeartifact-pip-conf`. It lets an agent
+install Scale-internal packages — `sgp-obs`, for instance — that are not on public PyPI, without the
+build holding any registry credential of its own. The control-plane broker mints a short-lived
+CodeArtifact token per build and injects it as that secret.
+
+- Design: [Private Package Access for Customer Agents (PRD)](https://app.notion.com/p/Private-Package-Access-for-Customer-Agents-PRD-3ad904d6e6cb802cb091df1c25e230bc)
+- Tracking: [SGPINF-1568](https://linear.app/scale-epd/issue/SGPINF-1568/provide-scale-internal-packages-to-agentex-agents-in-customer)
+
+## It is inert by default
+
+The mount is `required=false` and guarded by `[ -s ... ]`, so with no secret injected the build is
+byte-identical to one without any of this. That covers every local build, every CI build, and every
+agent that never opts in. An empty secret file is skipped too.
+
+## Opting in
+
+Add the index to the agent's `pyproject.toml`:
+
+```toml
+[[tool.uv.index]]
+name = "scale-pypi"
+url = "<the scale-customer-pypi URL>"
+```
+
+**No `default = true`, deliberately.** An earlier revision of this snippet had it, which was
+misleading in both directions. It would not survive the build — the Dockerfiles export
+`UV_INDEX`, which binds the mirror as a *named* index ahead of public PyPI rather than
+replacing it as the default, and a name rebound that way does not carry the project entry's
+default flag. And it is not the behaviour we want anyway: the mirror exists to supply the
+Scale-internal packages that are not on public PyPI, not to become the sole source for every
+dependency.
+
+So resolution is **mirror first, public PyPI as fallback**. `sgp-obs` can only come from the
+mirror, because it exists nowhere else. An ordinary dependency the mirror happens not to carry
+still resolves from PyPI instead of failing the build, which is what keeps a scaffolded agent
+building when the mirror is incomplete or unreachable.
+
+The name must be exactly `scale-pypi`. uv applies `UV_INDEX_SCALE_PYPI_USERNAME` /
+`UV_INDEX_SCALE_PYPI_PASSWORD` to the index of that name, so renaming it makes the credentials
+silently stop applying. Setting `UV_INDEX_URL` instead does not authenticate a *named* index at
+all, and the resolve fails with a 401.
+
+## Three things that are easy to get wrong
+
+**The token arrives percent-encoded.** The buildspec URL-encodes it to embed it in the pip config's
+URL userinfo, so a token containing `+`, `/` or `=` arrives as `%2B`, `%2F`, `%3D`. The `uv sync`
+templates decode it before exporting it as a password. Passing it through still-encoded sends a
+different string and the resolve 401s.
+
+**The credential must not follow project-controlled configuration.** uv binds credentials by index
+*name*, and the name-to-URL mapping would otherwise come from the agent's own `pyproject.toml` — so a
+project that pointed `scale-pypi` at another host would receive the token. Verified against a local
+server: the rogue host receives `Authorization: Basic aws:<token>` and the real index is never
+contacted. The templates therefore export `UV_INDEX` to re-bind the name to the URL the *broker*
+supplied, which overrides whatever the project declared. With that in place the rogue host is never
+contacted. The pinned URL carries no userinfo; the token still travels only in
+`UV_INDEX_SCALE_PYPI_PASSWORD`.
+
+The case this defends is not a malicious agent author — they also write the Dockerfile and could read
+the mounted secret directly. It is a *contributed* change to a project file, where a one-line URL edit
+is far less conspicuous in review than an exfiltration command in a Dockerfile.
+
+**The two template variants work differently, deliberately.**
+
+| Template | Install step | How the credential is supplied |
+| --- | --- | --- |
+| `Dockerfile-uv.j2` | `uv sync` against the agent's `pyproject.toml` | Named index `scale-pypi`, pinned via `UV_INDEX`, token decoded into `UV_INDEX_SCALE_PYPI_PASSWORD` |
+| `Dockerfile.j2` | `uv pip install -r requirements.txt` | No pyproject is present, so there is no named index to bind to. The credentialed URL is used directly via `UV_DEFAULT_INDEX` |
+
+The `requirements.txt` variant does **not** decode the token, and that is the point: it stays inside
+the URL, already encoded for exactly that use. Decoding it there would corrupt it. It is also not
+exposed to the redirection problem above, because the URL comes wholly from the injected secret.

src/agentex/lib/cli/templates/default-claude-code/Dockerfile-uv.j2

@@ -34,14 +34,34 @@ WORKDIR /app/{{ project_path_from_build_root }}
 COPY {{ project_path_from_build_root }}/pyproject.toml ./
 
 # Install dependencies (without project itself, for layer caching)
+# Optional private index for Scale-internal packages such as sgp-obs, injected by the
+# control-plane broker (SGPINF-1568). Inert unless the secret is present, so local
+# builds, CI builds, and agents that never opt in are unaffected.
+#
+# To opt in, and for why UV_INDEX is pinned to the broker's URL rather than trusting
+# the project's, see PRIVATE_INDEX.md in the agentex-sdk CLI templates directory.
 RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=secret,id=codeartifact-pip-conf,required=false \
+    if [ -s /run/secrets/codeartifact-pip-conf ]; then \
+      export UV_INDEX="scale-pypi=$(sed -n 's#.*index-url = https://aws:[^@]*@\(.*\)#https://\1#p' /run/secrets/codeartifact-pip-conf | head -1)"; \
+      export UV_INDEX_SCALE_PYPI_USERNAME=aws; \
+      export UV_INDEX_SCALE_PYPI_PASSWORD="$(sed -n 's#.*index-url = https://aws:\([^@]*\)@.*#\1#p' /run/secrets/codeartifact-pip-conf \
+        | python3 -c 'import sys,urllib.parse;print(urllib.parse.unquote(sys.stdin.read().strip()))')"; \
+    fi; \
     uv sync --no-install-project --no-dev
 
 # Copy the project code
 COPY {{ project_path_from_build_root }}/project ./project
 
 # Install the project
 RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=secret,id=codeartifact-pip-conf,required=false \
+    if [ -s /run/secrets/codeartifact-pip-conf ]; then \
+      export UV_INDEX="scale-pypi=$(sed -n 's#.*index-url = https://aws:[^@]*@\(.*\)#https://\1#p' /run/secrets/codeartifact-pip-conf | head -1)"; \
+      export UV_INDEX_SCALE_PYPI_USERNAME=aws; \
+      export UV_INDEX_SCALE_PYPI_PASSWORD="$(sed -n 's#.*index-url = https://aws:\([^@]*\)@.*#\1#p' /run/secrets/codeartifact-pip-conf \
+        | python3 -c 'import sys,urllib.parse;print(urllib.parse.unquote(sys.stdin.read().strip()))')"; \
+    fi; \
     uv sync --no-dev
 
 ENV PATH="/app/{{ project_path_from_build_root }}/.venv/bin:$PATH"

src/agentex/lib/cli/templates/default-claude-code/Dockerfile.j2

@@ -33,8 +33,19 @@ COPY {{ project_path_from_build_root }}/requirements.txt /app/{{ project_path_fr
 
 WORKDIR /app/{{ project_path_from_build_root }}
 
+# Optional private index for Scale-internal packages such as sgp-obs, injected by the
+# control-plane broker (SGPINF-1568). Inert unless the secret is present.
+#
+# This variant installs from requirements.txt, so there is no pyproject.toml for uv to
+# read a named index out of; the credentialed URL is used directly and is deliberately
+# NOT decoded. See PRIVATE_INDEX.md in the agentex-sdk CLI templates directory.
+#
 # Install the required Python packages
-RUN uv pip install --system -r requirements.txt
+RUN --mount=type=secret,id=codeartifact-pip-conf,required=false \
+    if [ -s /run/secrets/codeartifact-pip-conf ]; then \
+      export UV_DEFAULT_INDEX="$(sed -n 's#^[[:space:]]*index-url[[:space:]]*=[[:space:]]*##p' /run/secrets/codeartifact-pip-conf | head -1)"; \
+    fi; \
+    uv pip install --system -r requirements.txt
 
 # Copy the project code
 COPY {{ project_path_from_build_root }}/project /app/{{ project_path_from_build_root }}/project