diff --git a/.gitignore b/.gitignore index 93ecfccb..e11875c4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ # Python-generated files __pycache__/ +.pytest_cache/ *.py[oc] build/ dist/ @@ -9,3 +10,11 @@ wheels/ # Virtual environments .venv .env* + +# Run secrets loaded via `vero harbor run --env-file` (keep only *.example) +secrets.env +*.secrets.env +!*.example + +# Local benchmark scoping notes (paper planning; not tracked) +harness-engineering-bench/benchmark-scoping.md diff --git a/README.md b/README.md index f366e418..31b39cbb 100644 --- a/README.md +++ b/README.md @@ -1,122 +1,86 @@ -# VeRO: Versioning Rewards and Observations +# VeRO: a harness for agents to optimize programs, text, and agents +> The code for the original VeRO paper is archived in [`legacy/`](legacy/). [![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) - -VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control. - -> **Paper**: [VeRO: An Evaluation Harness for Agents to Optimize Agents](https://arxiv.org/abs/2602.22480) - -## Repository Structure - -``` -vero/ -├── vero/ # Core library (scale-vero) -├── vero-agents/ # Agent implementations (benchmarking targets) -├── vero-benchmarking/ # Benchmarking scripts and analysis -└── LICENSE -``` - -### [vero/](vero/) - -The core optimization framework. Provides: - -- **Policy** — orchestrates the optimization loop (agent + evaluator + git) -- **Agents** — VeroAgent (OpenAI Agents SDK) and ClaudeCodeAgent (Claude Agent SDK) -- **Evaluator** — runs task evaluations in isolated subprocess environments -- **Tools** — MCP-based tools for agents (bash, file I/O, experiment runner, dataset viewer, etc.) -- **Traces** — session analysis and LLM-based trace interpretation - -```bash -cd vero && uv sync --extra optimize +[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/) + +VeRO gives a coding agent something to edit, an evaluation boundary, and durable +memory of every candidate it tried. The target is anything you can put under Git +and score — a **program** (a single function up to a whole codebase), **text** +(a prompt, spec, or config), or an **agent** (its scaffold, tools, and prompts). +Agents are programs, but not everyone reads "program" that way, so VeRO names +them explicitly: it was introduced to optimize agents and generalizes the same +version / evaluate / select loop to any Git-versioned artifact. + +```mermaid +flowchart LR + S["Strategy
proposes ideas"] --> P["Candidate producers
edit isolated workspaces"] + P --> E["Evaluation backends
score versioned reports"] + E --> Sel["Selection keeps the
best feasible candidate"] + Sel -->|"next round"| S ``` -See [vero/README.md](vero/README.md) for full documentation. - -### [vero-agents/](vero-agents/) +The target and evaluator do not need to be Python. External evaluators and +candidate producers connect through command protocols; Python benchmarks can +use the optional, optimizer-independent `scale-vero-tasks` package. -Agent implementations used as optimization targets: +Targets may live locally or in an isolated sandbox. VeRO keeps optimization +state and experiment tracking on the host while running Git worktrees, producer +commands, builds, and evaluation commands in the target sandbox. The core guide +includes a no-bind-mount `DockerSandbox` example. -| Agent | Description | -|-------|-------------| -| **generic-agent** | General-purpose agent for MATH, GPQA, GAIA, GSM8K, etc. | -| **web_search_agent** | Web search agent for SimpleQA, Facts Search | -| **KIRA** | Terminal task agent for Terminal Bench 2.0 | -| **tau-bench** | Customer service tool-use agent | -| **pharma_summarizer** | Document summarization agent | +## Repository layout -See [vero-agents/README.md](vero-agents/README.md) for details. +| Directory | Purpose | +| --- | --- | +| [`vero/`](vero/) | The `scale-vero` optimization kernel, runtime, CLI, and coding-agent adapters | +| [`vero-tasks/`](vero-tasks/) | Narrow Python task types and schema-v1 evaluation runner | +| [`harness-engineering-bench/`](harness-engineering-bench/) | Harbor-native target programs and end-to-end optimization benchmarks | -### [vero-benchmarking/](vero-benchmarking/) +Start with the [generic C matrix-multiplication quickstart](vero/examples/c-matmul/), +try the [26-circle packing benchmark](vero/examples/circle-packing/), or read the +[core guide](vero/README.md). The C example demonstrates the language-neutral +command protocol without model credentials. Circle packing is a substantive +coding-agent benchmark with exact geometry checks and inspectable search +artifacts. -Scripts and infrastructure for running optimization experiments: +For a full agent-optimization example, the [GAIA benchmark](harness-engineering-bench/gaia/) +pairs a tool-using GPT-5.4 mini target with Harbor's canonical GAIA verifier and +an immutable 20% / 40% / 40% development, validation, and test split. ```bash -cd vero-benchmarking && uv sync --all-extras - -# Run an optimization experiment -uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math - -# Build datasets -./scripts/build_datasets.sh +cd vero +uv sync --all-extras +uv run vero --help ``` -See [vero-benchmarking/README.md](vero-benchmarking/README.md) for full documentation. - -## Quick Start +## Paper reproduction -### Prerequisites +VeRO was introduced in [*VeRO: A Harness for Agents to Optimize +Agents*](https://arxiv.org/abs/2602.22480), accepted at ICML 2026. The current +library generalizes that version/evaluate/select loop from agents to programs. -- Python 3.11+ -- [uv](https://docs.astral.sh/uv/getting-started/installation/) -- Git -- Access to an LLM provider (via LiteLLM, OpenAI, Anthropic, etc.) - -### Install +The exact paper implementation is frozen separately from the v0.5 redesign: ```bash -git clone && cd vero - -# Install core library -cd vero && uv sync --extra optimize - -# Install benchmarking tools -cd ../vero-benchmarking && uv sync --all-extras +git checkout paper-v1 ``` -### Run Your First Optimization - -```python -from agents import Agent as OAIAgent -from vero.policy import Policy -from vero.agents.vero import VeroAgent - -policy = Policy( - project_path="/path/to/my-agent", - dataset="/path/to/my-dataset", - agent=VeroAgent( - oai_agent=OAIAgent(name="VeroAgent", model="anthropic/claude-sonnet-4-5-20250929"), - ), - task="main", - train_budget=10, - max_turns=200, -) - -best = await policy.run() -print(f"Best commit: {best.commit}, score: {best.score}") -``` +Use the `paper/v1` branch or `paper-v1` tag for reproduction. Development of +the generic program optimizer continues on the `v0.5` branch. The frozen ref +also preserves the paper-era `vero-agents` and `vero-benchmarking` directories; +their Harbor-native replacement on `v0.5` is `harness-engineering-bench`. ## Citation ```bibtex @article{ursekar2026vero, - title={VeRO: An Evaluation Harness for Agents to Optimize Agents}, + title={VeRO: A Harness for Agents to Optimize Agents}, author={Ursekar, Varun and Shanker, Apaar and Chatrath, Veronica and Xue, Yuan (Emily) and Denton, Sam}, journal={arXiv preprint arXiv:2602.22480}, year={2026} } ``` -## License - -[MIT](LICENSE) +VeRO is licensed under the [MIT License](LICENSE). diff --git a/runs/.gitignore b/runs/.gitignore new file mode 100644 index 00000000..45ed60a1 --- /dev/null +++ b/runs/.gitignore @@ -0,0 +1,3 @@ +# Local GAIA/harbor run outputs — inspectable, not committed. +* +!.gitignore diff --git a/vero-tasks/README.md b/vero-tasks/README.md new file mode 100644 index 00000000..be26b0f0 --- /dev/null +++ b/vero-tasks/README.md @@ -0,0 +1,35 @@ +# scale-vero-tasks + +`scale-vero-tasks` is the optional Python task protocol used by VeRO benchmark +targets. It contains no optimizer, workspace, session, dataset-store, or +experiment-database code. + +```python +from vero_tasks import TaskOutput, TaskResult, create_task + +task = create_task("exact_match") + +@task.inference() +async def infer(case, context): + return TaskOutput(output=case["question"].upper()) + +@task.evaluation() +async def evaluate(case, output, context): + return TaskResult.from_task_output( + output, + score=float(output.output == case["answer"]), + ) +``` + +The runner accepts a VeRO command-evaluation request and an external JSON/JSONL +case file, imports a task module, and writes a schema-v1 evaluation report. The +standard VeRO adapter runs this package in a trusted evaluator project while +overlaying the candidate package as an editable dependency. This keeps Python +benchmark ergonomics separate from both the target program and VeRO's +language-neutral evaluation kernel. + +The runner applies VeRO's retry policy independently to each non-batch +inference and evaluation call. Provider rate limits, configured HTTP status +codes or messages, and timeouts can be retried with bounded exponential +backoff. Earlier failed attempts are retained in the canonical case error list; +they are non-terminal when a later attempt succeeds. diff --git a/vero-tasks/examples/matmul-eval/pyproject.toml b/vero-tasks/examples/matmul-eval/pyproject.toml new file mode 100644 index 00000000..17ba96ef --- /dev/null +++ b/vero-tasks/examples/matmul-eval/pyproject.toml @@ -0,0 +1,16 @@ +[project] +name = "matmul-eval" +version = "0.1.0" +description = "Evaluation task for the matmul kernel — measures correctness and speed" +requires-python = ">=3.11" +dependencies = ["scale-vero-tasks>=0.2.0"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/matmul_eval"] + +[tool.uv.sources] +scale-vero-tasks = { path = "../..", editable = true } diff --git a/vero-tasks/examples/matmul-eval/src/matmul_eval/__init__.py b/vero-tasks/examples/matmul-eval/src/matmul_eval/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/vero-tasks/examples/matmul-eval/src/matmul_eval/matmul_task.py b/vero-tasks/examples/matmul-eval/src/matmul_eval/matmul_task.py new file mode 100644 index 00000000..4a6fdb88 --- /dev/null +++ b/vero-tasks/examples/matmul-eval/src/matmul_eval/matmul_task.py @@ -0,0 +1,70 @@ +"""Matrix multiply evaluation task. + +Measures correctness and execution speed of a matmul kernel. +Score = average time per multiply (lower is better). +Incorrect results get a penalty score of 999999.0. +""" + +import time + +from vero_tasks import ( + TaskContext, + TaskOutput, + TaskParameters, + TaskResult, + create_task, +) + + +class MatmulParameters(TaskParameters): + n_repeats: int = 100 + + +matmul_task = create_task("matmul", task_parameters_type=MatmulParameters) + + +@matmul_task.inference() +async def run_inference(task: dict, context: TaskContext) -> TaskOutput: + from matmul_kernel import multiply + + a = task["matrix_a"] + b = task["matrix_b"] + + parameters = context.parse_task_params(MatmulParameters) + + # Warmup + multiply(a, b) + + # Timed runs (like timeit) + start = time.perf_counter() + for _ in range(parameters.n_repeats): + result = multiply(a, b) + elapsed_ms = (time.perf_counter() - start) / parameters.n_repeats * 1000 + + return TaskOutput(output={"result": result, "time_ms": elapsed_ms}) + + +@matmul_task.evaluation() +async def run_evaluation( + task: dict, output: TaskOutput, context: TaskContext +) -> TaskResult: + expected = task["expected"] + actual = output.output["result"] + time_ms = output.output["time_ms"] + + # Check correctness (within floating point tolerance) + correct = all( + abs(a_val - e_val) < 1e-6 + for a_row, e_row in zip(actual, expected) + for a_val, e_val in zip(a_row, e_row) + ) + + # Score = time if correct, penalty if wrong (lower is better) + score = time_ms if correct else 999999.0 + + return TaskResult.from_task_output( + output, + score=score, + metrics={"time_ms": time_ms, "correct": 1.0 if correct else 0.0}, + feedback=f"{'Correct' if correct else 'Wrong'}. Time: {time_ms:.2f}ms", + ) diff --git a/vero-tasks/examples/matmul-eval/uv.lock b/vero-tasks/examples/matmul-eval/uv.lock new file mode 100644 index 00000000..5aeb6ab9 --- /dev/null +++ b/vero-tasks/examples/matmul-eval/uv.lock @@ -0,0 +1,181 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] + +[[package]] +name = "annotated-types" +version = "0.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" }, +] + +[[package]] +name = "matmul-eval" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "scale-vero-tasks" }, +] + +[package.metadata] +requires-dist = [{ name = "scale-vero-tasks", editable = "../../../vero-tasks" }] + +[[package]] +name = "pydantic" +version = "2.12.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/69/44/36f1a6e523abc58ae5f928898e4aca2e0ea509b5aa6f6f392a5d882be928/pydantic-2.12.5.tar.gz", hash = "sha256:4d351024c75c0f085a9febbb665ce8c0c6ec5d30e903bdb6394b7ede26aebb49", size = 821591, upload-time = "2025-11-26T15:11:46.471Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/87/b70ad306ebb6f9b585f114d0ac2137d792b48be34d732d60e597c2f8465a/pydantic-2.12.5-py3-none-any.whl", hash = "sha256:e561593fccf61e8a20fc46dfc2dfe075b8be7d0188df33f221ad1f0139180f9d", size = 463580, upload-time = "2025-11-26T15:11:44.605Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.41.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/71/70/23b021c950c2addd24ec408e9ab05d59b035b39d97cdc1130e1bce647bb6/pydantic_core-2.41.5.tar.gz", hash = "sha256:08daa51ea16ad373ffd5e7606252cc32f07bc72b28284b6bc9c6df804816476e", size = 460952, upload-time = "2025-11-04T13:43:49.098Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e8/72/74a989dd9f2084b3d9530b0915fdda64ac48831c30dbf7c72a41a5232db8/pydantic_core-2.41.5-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:a3a52f6156e73e7ccb0f8cced536adccb7042be67cb45f9562e12b319c119da6", size = 2105873, upload-time = "2025-11-04T13:39:31.373Z" }, + { url = "https://files.pythonhosted.org/packages/12/44/37e403fd9455708b3b942949e1d7febc02167662bf1a7da5b78ee1ea2842/pydantic_core-2.41.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7f3bf998340c6d4b0c9a2f02d6a400e51f123b59565d74dc60d252ce888c260b", size = 1899826, upload-time = "2025-11-04T13:39:32.897Z" }, + { url = "https://files.pythonhosted.org/packages/33/7f/1d5cab3ccf44c1935a359d51a8a2a9e1a654b744b5e7f80d41b88d501eec/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:378bec5c66998815d224c9ca994f1e14c0c21cb95d2f52b6021cc0b2a58f2a5a", size = 1917869, upload-time = "2025-11-04T13:39:34.469Z" }, + { url = "https://files.pythonhosted.org/packages/6e/6a/30d94a9674a7fe4f4744052ed6c5e083424510be1e93da5bc47569d11810/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e7b576130c69225432866fe2f4a469a85a54ade141d96fd396dffcf607b558f8", size = 2063890, upload-time = "2025-11-04T13:39:36.053Z" }, + { url = "https://files.pythonhosted.org/packages/50/be/76e5d46203fcb2750e542f32e6c371ffa9b8ad17364cf94bb0818dbfb50c/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6cb58b9c66f7e4179a2d5e0f849c48eff5c1fca560994d6eb6543abf955a149e", size = 2229740, upload-time = "2025-11-04T13:39:37.753Z" }, + { url = "https://files.pythonhosted.org/packages/d3/ee/fed784df0144793489f87db310a6bbf8118d7b630ed07aa180d6067e653a/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:88942d3a3dff3afc8288c21e565e476fc278902ae4d6d134f1eeda118cc830b1", size = 2350021, upload-time = "2025-11-04T13:39:40.94Z" }, + { url = "https://files.pythonhosted.org/packages/c8/be/8fed28dd0a180dca19e72c233cbf58efa36df055e5b9d90d64fd1740b828/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f31d95a179f8d64d90f6831d71fa93290893a33148d890ba15de25642c5d075b", size = 2066378, upload-time = "2025-11-04T13:39:42.523Z" }, + { url = "https://files.pythonhosted.org/packages/b0/3b/698cf8ae1d536a010e05121b4958b1257f0b5522085e335360e53a6b1c8b/pydantic_core-2.41.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c1df3d34aced70add6f867a8cf413e299177e0c22660cc767218373d0779487b", size = 2175761, upload-time = "2025-11-04T13:39:44.553Z" }, + { url = "https://files.pythonhosted.org/packages/b8/ba/15d537423939553116dea94ce02f9c31be0fa9d0b806d427e0308ec17145/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:4009935984bd36bd2c774e13f9a09563ce8de4abaa7226f5108262fa3e637284", size = 2146303, upload-time = "2025-11-04T13:39:46.238Z" }, + { url = "https://files.pythonhosted.org/packages/58/7f/0de669bf37d206723795f9c90c82966726a2ab06c336deba4735b55af431/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_armv7l.whl", hash = "sha256:34a64bc3441dc1213096a20fe27e8e128bd3ff89921706e83c0b1ac971276594", size = 2340355, upload-time = "2025-11-04T13:39:48.002Z" }, + { url = "https://files.pythonhosted.org/packages/e5/de/e7482c435b83d7e3c3ee5ee4451f6e8973cff0eb6007d2872ce6383f6398/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:c9e19dd6e28fdcaa5a1de679aec4141f691023916427ef9bae8584f9c2fb3b0e", size = 2319875, upload-time = "2025-11-04T13:39:49.705Z" }, + { url = "https://files.pythonhosted.org/packages/fe/e6/8c9e81bb6dd7560e33b9053351c29f30c8194b72f2d6932888581f503482/pydantic_core-2.41.5-cp311-cp311-win32.whl", hash = "sha256:2c010c6ded393148374c0f6f0bf89d206bf3217f201faa0635dcd56bd1520f6b", size = 1987549, upload-time = "2025-11-04T13:39:51.842Z" }, + { url = "https://files.pythonhosted.org/packages/11/66/f14d1d978ea94d1bc21fc98fcf570f9542fe55bfcc40269d4e1a21c19bf7/pydantic_core-2.41.5-cp311-cp311-win_amd64.whl", hash = "sha256:76ee27c6e9c7f16f47db7a94157112a2f3a00e958bc626e2f4ee8bec5c328fbe", size = 2011305, upload-time = "2025-11-04T13:39:53.485Z" }, + { url = "https://files.pythonhosted.org/packages/56/d8/0e271434e8efd03186c5386671328154ee349ff0354d83c74f5caaf096ed/pydantic_core-2.41.5-cp311-cp311-win_arm64.whl", hash = "sha256:4bc36bbc0b7584de96561184ad7f012478987882ebf9f9c389b23f432ea3d90f", size = 1972902, upload-time = "2025-11-04T13:39:56.488Z" }, + { url = "https://files.pythonhosted.org/packages/5f/5d/5f6c63eebb5afee93bcaae4ce9a898f3373ca23df3ccaef086d0233a35a7/pydantic_core-2.41.5-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:f41a7489d32336dbf2199c8c0a215390a751c5b014c2c1c5366e817202e9cdf7", size = 2110990, upload-time = "2025-11-04T13:39:58.079Z" }, + { url = "https://files.pythonhosted.org/packages/aa/32/9c2e8ccb57c01111e0fd091f236c7b371c1bccea0fa85247ac55b1e2b6b6/pydantic_core-2.41.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:070259a8818988b9a84a449a2a7337c7f430a22acc0859c6b110aa7212a6d9c0", size = 1896003, upload-time = "2025-11-04T13:39:59.956Z" }, + { url = "https://files.pythonhosted.org/packages/68/b8/a01b53cb0e59139fbc9e4fda3e9724ede8de279097179be4ff31f1abb65a/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e96cea19e34778f8d59fe40775a7a574d95816eb150850a85a7a4c8f4b94ac69", size = 1919200, upload-time = "2025-11-04T13:40:02.241Z" }, + { url = "https://files.pythonhosted.org/packages/38/de/8c36b5198a29bdaade07b5985e80a233a5ac27137846f3bc2d3b40a47360/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ed2e99c456e3fadd05c991f8f437ef902e00eedf34320ba2b0842bd1c3ca3a75", size = 2052578, upload-time = "2025-11-04T13:40:04.401Z" }, + { url = "https://files.pythonhosted.org/packages/00/b5/0e8e4b5b081eac6cb3dbb7e60a65907549a1ce035a724368c330112adfdd/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:65840751b72fbfd82c3c640cff9284545342a4f1eb1586ad0636955b261b0b05", size = 2208504, upload-time = "2025-11-04T13:40:06.072Z" }, + { url = "https://files.pythonhosted.org/packages/77/56/87a61aad59c7c5b9dc8caad5a41a5545cba3810c3e828708b3d7404f6cef/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e536c98a7626a98feb2d3eaf75944ef6f3dbee447e1f841eae16f2f0a72d8ddc", size = 2335816, upload-time = "2025-11-04T13:40:07.835Z" }, + { url = "https://files.pythonhosted.org/packages/0d/76/941cc9f73529988688a665a5c0ecff1112b3d95ab48f81db5f7606f522d3/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eceb81a8d74f9267ef4081e246ffd6d129da5d87e37a77c9bde550cb04870c1c", size = 2075366, upload-time = "2025-11-04T13:40:09.804Z" }, + { url = "https://files.pythonhosted.org/packages/d3/43/ebef01f69baa07a482844faaa0a591bad1ef129253ffd0cdaa9d8a7f72d3/pydantic_core-2.41.5-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d38548150c39b74aeeb0ce8ee1d8e82696f4a4e16ddc6de7b1d8823f7de4b9b5", size = 2171698, upload-time = "2025-11-04T13:40:12.004Z" }, + { url = "https://files.pythonhosted.org/packages/b1/87/41f3202e4193e3bacfc2c065fab7706ebe81af46a83d3e27605029c1f5a6/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:c23e27686783f60290e36827f9c626e63154b82b116d7fe9adba1fda36da706c", size = 2132603, upload-time = "2025-11-04T13:40:13.868Z" }, + { url = "https://files.pythonhosted.org/packages/49/7d/4c00df99cb12070b6bccdef4a195255e6020a550d572768d92cc54dba91a/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:482c982f814460eabe1d3bb0adfdc583387bd4691ef00b90575ca0d2b6fe2294", size = 2329591, upload-time = "2025-11-04T13:40:15.672Z" }, + { url = "https://files.pythonhosted.org/packages/cc/6a/ebf4b1d65d458f3cda6a7335d141305dfa19bdc61140a884d165a8a1bbc7/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:bfea2a5f0b4d8d43adf9d7b8bf019fb46fdd10a2e5cde477fbcb9d1fa08c68e1", size = 2319068, upload-time = "2025-11-04T13:40:17.532Z" }, + { url = "https://files.pythonhosted.org/packages/49/3b/774f2b5cd4192d5ab75870ce4381fd89cf218af999515baf07e7206753f0/pydantic_core-2.41.5-cp312-cp312-win32.whl", hash = "sha256:b74557b16e390ec12dca509bce9264c3bbd128f8a2c376eaa68003d7f327276d", size = 1985908, upload-time = "2025-11-04T13:40:19.309Z" }, + { url = "https://files.pythonhosted.org/packages/86/45/00173a033c801cacf67c190fef088789394feaf88a98a7035b0e40d53dc9/pydantic_core-2.41.5-cp312-cp312-win_amd64.whl", hash = "sha256:1962293292865bca8e54702b08a4f26da73adc83dd1fcf26fbc875b35d81c815", size = 2020145, upload-time = "2025-11-04T13:40:21.548Z" }, + { url = "https://files.pythonhosted.org/packages/f9/22/91fbc821fa6d261b376a3f73809f907cec5ca6025642c463d3488aad22fb/pydantic_core-2.41.5-cp312-cp312-win_arm64.whl", hash = "sha256:1746d4a3d9a794cacae06a5eaaccb4b8643a131d45fbc9af23e353dc0a5ba5c3", size = 1976179, upload-time = "2025-11-04T13:40:23.393Z" }, + { url = "https://files.pythonhosted.org/packages/87/06/8806241ff1f70d9939f9af039c6c35f2360cf16e93c2ca76f184e76b1564/pydantic_core-2.41.5-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:941103c9be18ac8daf7b7adca8228f8ed6bb7a1849020f643b3a14d15b1924d9", size = 2120403, upload-time = "2025-11-04T13:40:25.248Z" }, + { url = "https://files.pythonhosted.org/packages/94/02/abfa0e0bda67faa65fef1c84971c7e45928e108fe24333c81f3bfe35d5f5/pydantic_core-2.41.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:112e305c3314f40c93998e567879e887a3160bb8689ef3d2c04b6cc62c33ac34", size = 1896206, upload-time = "2025-11-04T13:40:27.099Z" }, + { url = "https://files.pythonhosted.org/packages/15/df/a4c740c0943e93e6500f9eb23f4ca7ec9bf71b19e608ae5b579678c8d02f/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0cbaad15cb0c90aa221d43c00e77bb33c93e8d36e0bf74760cd00e732d10a6a0", size = 1919307, upload-time = "2025-11-04T13:40:29.806Z" }, + { url = "https://files.pythonhosted.org/packages/9a/e3/6324802931ae1d123528988e0e86587c2072ac2e5394b4bc2bc34b61ff6e/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:03ca43e12fab6023fc79d28ca6b39b05f794ad08ec2feccc59a339b02f2b3d33", size = 2063258, upload-time = "2025-11-04T13:40:33.544Z" }, + { url = "https://files.pythonhosted.org/packages/c9/d4/2230d7151d4957dd79c3044ea26346c148c98fbf0ee6ebd41056f2d62ab5/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:dc799088c08fa04e43144b164feb0c13f9a0bc40503f8df3e9fde58a3c0c101e", size = 2214917, upload-time = "2025-11-04T13:40:35.479Z" }, + { url = "https://files.pythonhosted.org/packages/e6/9f/eaac5df17a3672fef0081b6c1bb0b82b33ee89aa5cec0d7b05f52fd4a1fa/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:97aeba56665b4c3235a0e52b2c2f5ae9cd071b8a8310ad27bddb3f7fb30e9aa2", size = 2332186, upload-time = "2025-11-04T13:40:37.436Z" }, + { url = "https://files.pythonhosted.org/packages/cf/4e/35a80cae583a37cf15604b44240e45c05e04e86f9cfd766623149297e971/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:406bf18d345822d6c21366031003612b9c77b3e29ffdb0f612367352aab7d586", size = 2073164, upload-time = "2025-11-04T13:40:40.289Z" }, + { url = "https://files.pythonhosted.org/packages/bf/e3/f6e262673c6140dd3305d144d032f7bd5f7497d3871c1428521f19f9efa2/pydantic_core-2.41.5-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:b93590ae81f7010dbe380cdeab6f515902ebcbefe0b9327cc4804d74e93ae69d", size = 2179146, upload-time = "2025-11-04T13:40:42.809Z" }, + { url = "https://files.pythonhosted.org/packages/75/c7/20bd7fc05f0c6ea2056a4565c6f36f8968c0924f19b7d97bbfea55780e73/pydantic_core-2.41.5-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:01a3d0ab748ee531f4ea6c3e48ad9dac84ddba4b0d82291f87248f2f9de8d740", size = 2137788, upload-time = "2025-11-04T13:40:44.752Z" }, + { url = "https://files.pythonhosted.org/packages/3a/8d/34318ef985c45196e004bc46c6eab2eda437e744c124ef0dbe1ff2c9d06b/pydantic_core-2.41.5-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:6561e94ba9dacc9c61bce40e2d6bdc3bfaa0259d3ff36ace3b1e6901936d2e3e", size = 2340133, upload-time = "2025-11-04T13:40:46.66Z" }, + { url = "https://files.pythonhosted.org/packages/9c/59/013626bf8c78a5a5d9350d12e7697d3d4de951a75565496abd40ccd46bee/pydantic_core-2.41.5-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:915c3d10f81bec3a74fbd4faebe8391013ba61e5a1a8d48c4455b923bdda7858", size = 2324852, upload-time = "2025-11-04T13:40:48.575Z" }, + { url = "https://files.pythonhosted.org/packages/1a/d9/c248c103856f807ef70c18a4f986693a46a8ffe1602e5d361485da502d20/pydantic_core-2.41.5-cp313-cp313-win32.whl", hash = "sha256:650ae77860b45cfa6e2cdafc42618ceafab3a2d9a3811fcfbd3bbf8ac3c40d36", size = 1994679, upload-time = "2025-11-04T13:40:50.619Z" }, + { url = "https://files.pythonhosted.org/packages/9e/8b/341991b158ddab181cff136acd2552c9f35bd30380422a639c0671e99a91/pydantic_core-2.41.5-cp313-cp313-win_amd64.whl", hash = "sha256:79ec52ec461e99e13791ec6508c722742ad745571f234ea6255bed38c6480f11", size = 2019766, upload-time = "2025-11-04T13:40:52.631Z" }, + { url = "https://files.pythonhosted.org/packages/73/7d/f2f9db34af103bea3e09735bb40b021788a5e834c81eedb541991badf8f5/pydantic_core-2.41.5-cp313-cp313-win_arm64.whl", hash = "sha256:3f84d5c1b4ab906093bdc1ff10484838aca54ef08de4afa9de0f5f14d69639cd", size = 1981005, upload-time = "2025-11-04T13:40:54.734Z" }, + { url = "https://files.pythonhosted.org/packages/ea/28/46b7c5c9635ae96ea0fbb779e271a38129df2550f763937659ee6c5dbc65/pydantic_core-2.41.5-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:3f37a19d7ebcdd20b96485056ba9e8b304e27d9904d233d7b1015db320e51f0a", size = 2119622, upload-time = "2025-11-04T13:40:56.68Z" }, + { url = "https://files.pythonhosted.org/packages/74/1a/145646e5687e8d9a1e8d09acb278c8535ebe9e972e1f162ed338a622f193/pydantic_core-2.41.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1d1d9764366c73f996edd17abb6d9d7649a7eb690006ab6adbda117717099b14", size = 1891725, upload-time = "2025-11-04T13:40:58.807Z" }, + { url = "https://files.pythonhosted.org/packages/23/04/e89c29e267b8060b40dca97bfc64a19b2a3cf99018167ea1677d96368273/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:25e1c2af0fce638d5f1988b686f3b3ea8cd7de5f244ca147c777769e798a9cd1", size = 1915040, upload-time = "2025-11-04T13:41:00.853Z" }, + { url = "https://files.pythonhosted.org/packages/84/a3/15a82ac7bd97992a82257f777b3583d3e84bdb06ba6858f745daa2ec8a85/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:506d766a8727beef16b7adaeb8ee6217c64fc813646b424d0804d67c16eddb66", size = 2063691, upload-time = "2025-11-04T13:41:03.504Z" }, + { url = "https://files.pythonhosted.org/packages/74/9b/0046701313c6ef08c0c1cf0e028c67c770a4e1275ca73131563c5f2a310a/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4819fa52133c9aa3c387b3328f25c1facc356491e6135b459f1de698ff64d869", size = 2213897, upload-time = "2025-11-04T13:41:05.804Z" }, + { url = "https://files.pythonhosted.org/packages/8a/cd/6bac76ecd1b27e75a95ca3a9a559c643b3afcd2dd62086d4b7a32a18b169/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2b761d210c9ea91feda40d25b4efe82a1707da2ef62901466a42492c028553a2", size = 2333302, upload-time = "2025-11-04T13:41:07.809Z" }, + { url = "https://files.pythonhosted.org/packages/4c/d2/ef2074dc020dd6e109611a8be4449b98cd25e1b9b8a303c2f0fca2f2bcf7/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:22f0fb8c1c583a3b6f24df2470833b40207e907b90c928cc8d3594b76f874375", size = 2064877, upload-time = "2025-11-04T13:41:09.827Z" }, + { url = "https://files.pythonhosted.org/packages/18/66/e9db17a9a763d72f03de903883c057b2592c09509ccfe468187f2a2eef29/pydantic_core-2.41.5-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2782c870e99878c634505236d81e5443092fba820f0373997ff75f90f68cd553", size = 2180680, upload-time = "2025-11-04T13:41:12.379Z" }, + { url = "https://files.pythonhosted.org/packages/d3/9e/3ce66cebb929f3ced22be85d4c2399b8e85b622db77dad36b73c5387f8f8/pydantic_core-2.41.5-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:0177272f88ab8312479336e1d777f6b124537d47f2123f89cb37e0accea97f90", size = 2138960, upload-time = "2025-11-04T13:41:14.627Z" }, + { url = "https://files.pythonhosted.org/packages/a6/62/205a998f4327d2079326b01abee48e502ea739d174f0a89295c481a2272e/pydantic_core-2.41.5-cp314-cp314-musllinux_1_1_armv7l.whl", hash = "sha256:63510af5e38f8955b8ee5687740d6ebf7c2a0886d15a6d65c32814613681bc07", size = 2339102, upload-time = "2025-11-04T13:41:16.868Z" }, + { url = "https://files.pythonhosted.org/packages/3c/0d/f05e79471e889d74d3d88f5bd20d0ed189ad94c2423d81ff8d0000aab4ff/pydantic_core-2.41.5-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:e56ba91f47764cc14f1daacd723e3e82d1a89d783f0f5afe9c364b8bb491ccdb", size = 2326039, upload-time = "2025-11-04T13:41:18.934Z" }, + { url = "https://files.pythonhosted.org/packages/ec/e1/e08a6208bb100da7e0c4b288eed624a703f4d129bde2da475721a80cab32/pydantic_core-2.41.5-cp314-cp314-win32.whl", hash = "sha256:aec5cf2fd867b4ff45b9959f8b20ea3993fc93e63c7363fe6851424c8a7e7c23", size = 1995126, upload-time = "2025-11-04T13:41:21.418Z" }, + { url = "https://files.pythonhosted.org/packages/48/5d/56ba7b24e9557f99c9237e29f5c09913c81eeb2f3217e40e922353668092/pydantic_core-2.41.5-cp314-cp314-win_amd64.whl", hash = "sha256:8e7c86f27c585ef37c35e56a96363ab8de4e549a95512445b85c96d3e2f7c1bf", size = 2015489, upload-time = "2025-11-04T13:41:24.076Z" }, + { url = "https://files.pythonhosted.org/packages/4e/bb/f7a190991ec9e3e0ba22e4993d8755bbc4a32925c0b5b42775c03e8148f9/pydantic_core-2.41.5-cp314-cp314-win_arm64.whl", hash = "sha256:e672ba74fbc2dc8eea59fb6d4aed6845e6905fc2a8afe93175d94a83ba2a01a0", size = 1977288, upload-time = "2025-11-04T13:41:26.33Z" }, + { url = "https://files.pythonhosted.org/packages/92/ed/77542d0c51538e32e15afe7899d79efce4b81eee631d99850edc2f5e9349/pydantic_core-2.41.5-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8566def80554c3faa0e65ac30ab0932b9e3a5cd7f8323764303d468e5c37595a", size = 2120255, upload-time = "2025-11-04T13:41:28.569Z" }, + { url = "https://files.pythonhosted.org/packages/bb/3d/6913dde84d5be21e284439676168b28d8bbba5600d838b9dca99de0fad71/pydantic_core-2.41.5-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:b80aa5095cd3109962a298ce14110ae16b8c1aece8b72f9dafe81cf597ad80b3", size = 1863760, upload-time = "2025-11-04T13:41:31.055Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f0/e5e6b99d4191da102f2b0eb9687aaa7f5bea5d9964071a84effc3e40f997/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3006c3dd9ba34b0c094c544c6006cc79e87d8612999f1a5d43b769b89181f23c", size = 1878092, upload-time = "2025-11-04T13:41:33.21Z" }, + { url = "https://files.pythonhosted.org/packages/71/48/36fb760642d568925953bcc8116455513d6e34c4beaa37544118c36aba6d/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:72f6c8b11857a856bcfa48c86f5368439f74453563f951e473514579d44aa612", size = 2053385, upload-time = "2025-11-04T13:41:35.508Z" }, + { url = "https://files.pythonhosted.org/packages/20/25/92dc684dd8eb75a234bc1c764b4210cf2646479d54b47bf46061657292a8/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5cb1b2f9742240e4bb26b652a5aeb840aa4b417c7748b6f8387927bc6e45e40d", size = 2218832, upload-time = "2025-11-04T13:41:37.732Z" }, + { url = "https://files.pythonhosted.org/packages/e2/09/f53e0b05023d3e30357d82eb35835d0f6340ca344720a4599cd663dca599/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:bd3d54f38609ff308209bd43acea66061494157703364ae40c951f83ba99a1a9", size = 2327585, upload-time = "2025-11-04T13:41:40Z" }, + { url = "https://files.pythonhosted.org/packages/aa/4e/2ae1aa85d6af35a39b236b1b1641de73f5a6ac4d5a7509f77b814885760c/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2ff4321e56e879ee8d2a879501c8e469414d948f4aba74a2d4593184eb326660", size = 2041078, upload-time = "2025-11-04T13:41:42.323Z" }, + { url = "https://files.pythonhosted.org/packages/cd/13/2e215f17f0ef326fc72afe94776edb77525142c693767fc347ed6288728d/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d0d2568a8c11bf8225044aa94409e21da0cb09dcdafe9ecd10250b2baad531a9", size = 2173914, upload-time = "2025-11-04T13:41:45.221Z" }, + { url = "https://files.pythonhosted.org/packages/02/7a/f999a6dcbcd0e5660bc348a3991c8915ce6599f4f2c6ac22f01d7a10816c/pydantic_core-2.41.5-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:a39455728aabd58ceabb03c90e12f71fd30fa69615760a075b9fec596456ccc3", size = 2129560, upload-time = "2025-11-04T13:41:47.474Z" }, + { url = "https://files.pythonhosted.org/packages/3a/b1/6c990ac65e3b4c079a4fb9f5b05f5b013afa0f4ed6780a3dd236d2cbdc64/pydantic_core-2.41.5-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:239edca560d05757817c13dc17c50766136d21f7cd0fac50295499ae24f90fdf", size = 2329244, upload-time = "2025-11-04T13:41:49.992Z" }, + { url = "https://files.pythonhosted.org/packages/d9/02/3c562f3a51afd4d88fff8dffb1771b30cfdfd79befd9883ee094f5b6c0d8/pydantic_core-2.41.5-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:2a5e06546e19f24c6a96a129142a75cee553cc018ffee48a460059b1185f4470", size = 2331955, upload-time = "2025-11-04T13:41:54.079Z" }, + { url = "https://files.pythonhosted.org/packages/5c/96/5fb7d8c3c17bc8c62fdb031c47d77a1af698f1d7a406b0f79aaa1338f9ad/pydantic_core-2.41.5-cp314-cp314t-win32.whl", hash = "sha256:b4ececa40ac28afa90871c2cc2b9ffd2ff0bf749380fbdf57d165fd23da353aa", size = 1988906, upload-time = "2025-11-04T13:41:56.606Z" }, + { url = "https://files.pythonhosted.org/packages/22/ed/182129d83032702912c2e2d8bbe33c036f342cc735737064668585dac28f/pydantic_core-2.41.5-cp314-cp314t-win_amd64.whl", hash = "sha256:80aa89cad80b32a912a65332f64a4450ed00966111b6615ca6816153d3585a8c", size = 1981607, upload-time = "2025-11-04T13:41:58.889Z" }, + { url = "https://files.pythonhosted.org/packages/9f/ed/068e41660b832bb0b1aa5b58011dea2a3fe0ba7861ff38c4d4904c1c1a99/pydantic_core-2.41.5-cp314-cp314t-win_arm64.whl", hash = "sha256:35b44f37a3199f771c3eaa53051bc8a70cd7b54f333531c59e29fd4db5d15008", size = 1974769, upload-time = "2025-11-04T13:42:01.186Z" }, + { url = "https://files.pythonhosted.org/packages/11/72/90fda5ee3b97e51c494938a4a44c3a35a9c96c19bba12372fb9c634d6f57/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-macosx_10_12_x86_64.whl", hash = "sha256:b96d5f26b05d03cc60f11a7761a5ded1741da411e7fe0909e27a5e6a0cb7b034", size = 2115441, upload-time = "2025-11-04T13:42:39.557Z" }, + { url = "https://files.pythonhosted.org/packages/1f/53/8942f884fa33f50794f119012dc6a1a02ac43a56407adaac20463df8e98f/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-macosx_11_0_arm64.whl", hash = "sha256:634e8609e89ceecea15e2d61bc9ac3718caaaa71963717bf3c8f38bfde64242c", size = 1930291, upload-time = "2025-11-04T13:42:42.169Z" }, + { url = "https://files.pythonhosted.org/packages/79/c8/ecb9ed9cd942bce09fc888ee960b52654fbdbede4ba6c2d6e0d3b1d8b49c/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:93e8740d7503eb008aa2df04d3b9735f845d43ae845e6dcd2be0b55a2da43cd2", size = 1948632, upload-time = "2025-11-04T13:42:44.564Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1b/687711069de7efa6af934e74f601e2a4307365e8fdc404703afc453eab26/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f15489ba13d61f670dcc96772e733aad1a6f9c429cc27574c6cdaed82d0146ad", size = 2138905, upload-time = "2025-11-04T13:42:47.156Z" }, + { url = "https://files.pythonhosted.org/packages/09/32/59b0c7e63e277fa7911c2fc70ccfb45ce4b98991e7ef37110663437005af/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:7da7087d756b19037bc2c06edc6c170eeef3c3bafcb8f532ff17d64dc427adfd", size = 2110495, upload-time = "2025-11-04T13:42:49.689Z" }, + { url = "https://files.pythonhosted.org/packages/aa/81/05e400037eaf55ad400bcd318c05bb345b57e708887f07ddb2d20e3f0e98/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:aabf5777b5c8ca26f7824cb4a120a740c9588ed58df9b2d196ce92fba42ff8dc", size = 1915388, upload-time = "2025-11-04T13:42:52.215Z" }, + { url = "https://files.pythonhosted.org/packages/6e/0d/e3549b2399f71d56476b77dbf3cf8937cec5cd70536bdc0e374a421d0599/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c007fe8a43d43b3969e8469004e9845944f1a80e6acd47c150856bb87f230c56", size = 1942879, upload-time = "2025-11-04T13:42:56.483Z" }, + { url = "https://files.pythonhosted.org/packages/f7/07/34573da085946b6a313d7c42f82f16e8920bfd730665de2d11c0c37a74b5/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:76d0819de158cd855d1cbb8fcafdf6f5cf1eb8e470abe056d5d161106e38062b", size = 2139017, upload-time = "2025-11-04T13:42:59.471Z" }, + { url = "https://files.pythonhosted.org/packages/5f/9b/1b3f0e9f9305839d7e84912f9e8bfbd191ed1b1ef48083609f0dabde978c/pydantic_core-2.41.5-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:b2379fa7ed44ddecb5bfe4e48577d752db9fc10be00a6b7446e9663ba143de26", size = 2101980, upload-time = "2025-11-04T13:43:25.97Z" }, + { url = "https://files.pythonhosted.org/packages/a4/ed/d71fefcb4263df0da6a85b5d8a7508360f2f2e9b3bf5814be9c8bccdccc1/pydantic_core-2.41.5-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:266fb4cbf5e3cbd0b53669a6d1b039c45e3ce651fd5442eff4d07c2cc8d66808", size = 1923865, upload-time = "2025-11-04T13:43:28.763Z" }, + { url = "https://files.pythonhosted.org/packages/ce/3a/626b38db460d675f873e4444b4bb030453bbe7b4ba55df821d026a0493c4/pydantic_core-2.41.5-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58133647260ea01e4d0500089a8c4f07bd7aa6ce109682b1426394988d8aaacc", size = 2134256, upload-time = "2025-11-04T13:43:31.71Z" }, + { url = "https://files.pythonhosted.org/packages/83/d9/8412d7f06f616bbc053d30cb4e5f76786af3221462ad5eee1f202021eb4e/pydantic_core-2.41.5-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:287dad91cfb551c363dc62899a80e9e14da1f0e2b6ebde82c806612ca2a13ef1", size = 2174762, upload-time = "2025-11-04T13:43:34.744Z" }, + { url = "https://files.pythonhosted.org/packages/55/4c/162d906b8e3ba3a99354e20faa1b49a85206c47de97a639510a0e673f5da/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_aarch64.whl", hash = "sha256:03b77d184b9eb40240ae9fd676ca364ce1085f203e1b1256f8ab9984dca80a84", size = 2143141, upload-time = "2025-11-04T13:43:37.701Z" }, + { url = "https://files.pythonhosted.org/packages/1f/f2/f11dd73284122713f5f89fc940f370d035fa8e1e078d446b3313955157fe/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_armv7l.whl", hash = "sha256:a668ce24de96165bb239160b3d854943128f4334822900534f2fe947930e5770", size = 2330317, upload-time = "2025-11-04T13:43:40.406Z" }, + { url = "https://files.pythonhosted.org/packages/88/9d/b06ca6acfe4abb296110fb1273a4d848a0bfb2ff65f3ee92127b3244e16b/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_x86_64.whl", hash = "sha256:f14f8f046c14563f8eb3f45f499cc658ab8d10072961e07225e507adb700e93f", size = 2316992, upload-time = "2025-11-04T13:43:43.602Z" }, + { url = "https://files.pythonhosted.org/packages/36/c7/cfc8e811f061c841d7990b0201912c3556bfeb99cdcb7ed24adc8d6f8704/pydantic_core-2.41.5-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:56121965f7a4dc965bff783d70b907ddf3d57f6eba29b6d2e5dabfaf07799c51", size = 2145302, upload-time = "2025-11-04T13:43:46.64Z" }, +] + +[[package]] +name = "scale-vero-tasks" +version = "0.2.0" +source = { editable = "../../../vero-tasks" } +dependencies = [ + { name = "pydantic" }, +] + +[package.metadata] +requires-dist = [{ name = "pydantic", specifier = ">=2.11.7" }] + +[package.metadata.requires-dev] +dev = [ + { name = "pytest", specifier = ">=9.0.2" }, + { name = "pytest-asyncio", specifier = ">=1.3.0" }, +] + +[[package]] +name = "typing-extensions" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, +] diff --git a/vero-tasks/examples/matmul-kernel/README.md b/vero-tasks/examples/matmul-kernel/README.md new file mode 100644 index 00000000..7a5b3f95 --- /dev/null +++ b/vero-tasks/examples/matmul-kernel/README.md @@ -0,0 +1,30 @@ +# Matrix multiplication program optimization + +This is the smallest end-to-end demonstration of VeRO's v0.5 framing: the +target is an ordinary Python matrix multiplication function, not an agent. + +- `matmul-kernel` is the editable target and has no dependency on VeRO. +- `matmul-eval` is a trusted external evaluation harness built with + `scale-vero-tasks`. +- `run.py` connects the target, evaluator, objective, session, and optional + coding agent. + +From the `vero/` package directory: + +```bash +# Exercise the complete evaluation pipeline without credentials. +uv run python examples/matmul-kernel/run.py --eval-only + +# Let a coding agent optimize the function. +uv run python examples/matmul-kernel/run.py --agent vero +uv run python examples/matmul-kernel/run.py --agent claude +``` + +`--max-proposals` limits completed optimization attempts. Agent-requested +checkpoints are evaluations too, so use the independent `--max-evaluations` +option when you also want hard agent and system evaluation budgets. Those +budgets cover checkpoints and automatic candidate evaluations independently. + +Every candidate is edited and evaluated in an isolated Git worktree. The +original target template is unchanged, while reports, agent state, events, and +the best candidate identity are preserved in the printed session directory. diff --git a/vero-tasks/examples/matmul-kernel/pyproject.toml b/vero-tasks/examples/matmul-kernel/pyproject.toml new file mode 100644 index 00000000..fb2f31d9 --- /dev/null +++ b/vero-tasks/examples/matmul-kernel/pyproject.toml @@ -0,0 +1,13 @@ +[project] +name = "matmul-kernel" +version = "0.1.0" +description = "A naive matrix multiply kernel to optimize with VeRO" +requires-python = ">=3.11" +dependencies = [] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/matmul_kernel"] diff --git a/vero-tasks/examples/matmul-kernel/run.py b/vero-tasks/examples/matmul-kernel/run.py new file mode 100644 index 00000000..488c9f66 --- /dev/null +++ b/vero-tasks/examples/matmul-kernel/run.py @@ -0,0 +1,271 @@ +"""Run VeRO's matrix-multiplication program optimization example. + +Run this file from the scale-vero project environment: + + uv run python examples/matmul-kernel/run.py --eval-only + uv run python examples/matmul-kernel/run.py --agent vero +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import random +import shutil +import subprocess +import tempfile +from pathlib import Path + +from vero.evaluation import ( + EvaluationBudget, + EvaluationDefinition, + EvaluationPlan, + EvaluationPrincipal, + EvaluationSet, + MetricAggregation, + MetricSelector, + ObjectiveSpec, + PythonTaskBackend, + PythonTaskBackendConfig, + PythonTaskEvaluationConfig, +) +from vero.optimization import SequentialStrategy +from vero.runtime import create_local_optimization_session + +SCRIPT_DIR = Path(__file__).resolve().parent +EVALUATOR_DIR = SCRIPT_DIR.parent / "matmul-eval" + +INSTRUCTION = """You are optimizing a matrix multiplication function for speed. + +The target is src/matmul_kernel/__init__.py. Preserve the public multiply(a, b) +signature and numerical correctness. You may change the implementation and add +target dependencies. Use the evaluate tool with evaluation="matmul" when +measurement would help; the +objective is mean score in milliseconds, so lower is better. +""" + + +def _multiply(a: list[list[float]], b: list[list[float]]) -> list[list[float]]: + return [ + [ + sum(a_value * b[p][j] for p, a_value in enumerate(row)) + for j in range(len(b[0])) + ] + for row in a + ] + + +def create_cases(path: Path) -> Path: + def matrix(rows: int, columns: int, seed: int) -> list[list[float]]: + generator = random.Random(seed) + return [ + [generator.uniform(-10, 10) for _ in range(columns)] + for _ in range(rows) + ] + + inputs = [ + ([[1, 2], [3, 4]], [[5, 6], [7, 8]]), + ([[1, 0], [0, 1]], [[9, 10], [11, 12]]), + (matrix(8, 8, 42), matrix(8, 8, 52)), + (matrix(10, 10, 43), matrix(10, 10, 53)), + (matrix(12, 12, 44), matrix(12, 12, 54)), + ] + cases = [ + { + "id": f"matrix-{index}", + "matrix_a": a, + "matrix_b": b, + "expected": _multiply(a, b), + } + for index, (a, b) in enumerate(inputs) + ] + path.write_text(json.dumps(cases), encoding="utf-8") + return path + + +def create_target(work_dir: Path) -> Path: + target = work_dir / "matmul-kernel" + target.mkdir() + shutil.copytree(SCRIPT_DIR / "src", target / "src") + shutil.copy2(SCRIPT_DIR / "pyproject.toml", target / "pyproject.toml") + (target / ".gitignore").write_text( + ".venv/\n__pycache__/\n*.pyc\n*.egg-info/\n", + encoding="utf-8", + ) + subprocess.run( + ["git", "init", "-b", "main"], + cwd=target, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=target, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=target, + check=True, + capture_output=True, + ) + return target + + +def create_backend(cases: Path) -> PythonTaskBackend: + return PythonTaskBackend( + PythonTaskBackendConfig( + harness_root=str(EVALUATOR_DIR), + module="matmul_eval.matmul_task", + task="matmul", + evaluations=[ + PythonTaskEvaluationConfig(name="matmul", cases_path=str(cases)) + ], + passthrough_environment=["UV_CACHE_DIR"], + ) + ) + + +async def run_example( + *, + work_dir: Path, + agent_name: str | None, + max_proposals: int, + max_evaluations: int | None, +) -> None: + target = create_target(work_dir) + cases = create_cases(work_dir / "cases.json") + evaluation_set = EvaluationSet(name="matmul") + agent_budget = ( + EvaluationBudget( + backend_id="python-task", + evaluation_set_key=evaluation_set.budget_key("python-task"), + principal=EvaluationPrincipal.AGENT, + total_runs=max_evaluations, + ) + if max_evaluations is not None + else None + ) + system_budget = ( + EvaluationBudget( + backend_id="python-task", + evaluation_set_key=evaluation_set.budget_key("python-task"), + principal=EvaluationPrincipal.SYSTEM, + total_runs=max_evaluations, + ) + if max_evaluations is not None + else None + ) + evaluation_plan = EvaluationPlan( + evaluations=[ + EvaluationDefinition( + evaluation_set=evaluation_set, + agent_budget=agent_budget, + system_budget=system_budget, + ) + ], + selection_evaluation="matmul", + ) + producers = {} + if agent_name is not None: + from vero.agents import AgentCandidateProducer + + if agent_name == "claude": + from vero.agents import ClaudeCodeAgent + + coding_agent = ClaudeCodeAgent() + else: + from vero.agents import VeroAgent + + coding_agent = VeroAgent() + producers["default"] = AgentCandidateProducer( + coding_agent, + prompt=INSTRUCTION, + max_turns=100, + ) + + session = await create_local_optimization_session( + project_path=target, + session_dir=work_dir / "session", + backend_id="python-task", + backend=create_backend(cases), + objective=ObjectiveSpec( + selector=MetricSelector( + metric="score", + aggregation=MetricAggregation.MEAN, + case_failure_value=1.0e12, + ), + direction="minimize", + failure_value=1.0e12, + ), + evaluation_plan=evaluation_plan, + strategy=SequentialStrategy(instruction=INSTRUCTION), + producers=producers, + parameters={"n_repeats": 100}, + max_proposals=max_proposals, + ) + result = await session.run() + print(f"Session: {session.session_dir}") + print(f"Baseline score: {result.baseline.objective.value:.6f} ms") + if result.best is not None: + print(f"Best score: {result.best.objective.value:.6f} ms") + print(f"Best version: {result.best.request.candidate.version}") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--eval-only", + action="store_true", + help="Evaluate the baseline without starting a coding agent.", + ) + parser.add_argument( + "--agent", + choices=["vero", "claude"], + default="vero", + help="Coding-agent adapter used for optimization.", + ) + parser.add_argument("--max-proposals", type=int, default=5) + parser.add_argument( + "--max-evaluations", + type=int, + help=( + "Optional evaluation-run budget, including the baseline, agent " + "checkpoints, and completed candidates. By default evaluations are " + "not separately capped." + ), + ) + parser.add_argument("--work-dir", type=Path) + arguments = parser.parse_args() + if arguments.max_proposals < 0: + parser.error("--max-proposals must be non-negative") + if arguments.max_evaluations is not None and arguments.max_evaluations < 1: + parser.error("--max-evaluations must be positive") + + work_dir = arguments.work_dir or Path(tempfile.mkdtemp(prefix="vero-matmul-")) + work_dir = work_dir.expanduser().resolve() + work_dir.mkdir(parents=True, exist_ok=True) + print(f"Working directory: {work_dir}") + asyncio.run( + run_example( + work_dir=work_dir, + agent_name=None if arguments.eval_only else arguments.agent, + max_proposals=0 if arguments.eval_only else arguments.max_proposals, + max_evaluations=arguments.max_evaluations, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/vero-tasks/examples/matmul-kernel/src/matmul_kernel/__init__.py b/vero-tasks/examples/matmul-kernel/src/matmul_kernel/__init__.py new file mode 100644 index 00000000..072bf205 --- /dev/null +++ b/vero-tasks/examples/matmul-kernel/src/matmul_kernel/__init__.py @@ -0,0 +1,22 @@ +"""Naive matrix multiply kernel. Optimize this program.""" + + +def multiply(a: list[list[float]], b: list[list[float]]) -> list[list[float]]: + """Multiply two matrices using the naive O(n^3) algorithm. + + Args: + a: Matrix of shape (n, k) + b: Matrix of shape (k, m) + + Returns: + Result matrix of shape (n, m) + """ + n = len(a) + m = len(b[0]) + k = len(b) + result = [[0.0] * m for _ in range(n)] + for i in range(n): + for j in range(m): + for p in range(k): + result[i][j] += a[i][p] * b[p][j] + return result diff --git a/vero-tasks/examples/matmul-kernel/uv.lock b/vero-tasks/examples/matmul-kernel/uv.lock new file mode 100644 index 00000000..d055e471 --- /dev/null +++ b/vero-tasks/examples/matmul-kernel/uv.lock @@ -0,0 +1,8 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" + +[[package]] +name = "matmul-kernel" +version = "0.1.0" +source = { editable = "." } diff --git a/vero-tasks/pyproject.toml b/vero-tasks/pyproject.toml new file mode 100644 index 00000000..0dddbb3e --- /dev/null +++ b/vero-tasks/pyproject.toml @@ -0,0 +1,29 @@ +[project] +name = "scale-vero-tasks" +version = "0.2.0" +description = "Narrow Python task protocol and runner for VeRO evaluations." +readme = "README.md" +authors = [ + { name = "Varun Ursekar", email = "oss@scale.com" } +] +license = { text = "MIT" } +requires-python = ">=3.11" +dependencies = [ + "pydantic>=2.11.7", +] + +[project.scripts] +vero-task = "vero_tasks.runner:main" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/vero_tasks"] + +[dependency-groups] +dev = [ + "pytest>=9.0.2", + "pytest-asyncio>=1.3.0", +] diff --git a/vero-tasks/src/vero_tasks/__init__.py b/vero-tasks/src/vero_tasks/__init__.py new file mode 100644 index 00000000..611c466c --- /dev/null +++ b/vero-tasks/src/vero_tasks/__init__.py @@ -0,0 +1,24 @@ +"""Narrow Python task protocol for VeRO evaluation harnesses.""" + +from vero_tasks.models import ( + RetryPolicy, + TaskAttemptError, + TaskContext, + TaskOutput, + TaskParameters, + TaskResult, + TaskT, +) +from vero_tasks.task import TaskDefinition, create_task + +__all__ = [ + "RetryPolicy", + "TaskAttemptError", + "TaskContext", + "TaskDefinition", + "TaskOutput", + "TaskParameters", + "TaskResult", + "TaskT", + "create_task", +] diff --git a/vero-tasks/src/vero_tasks/models.py b/vero-tasks/src/vero_tasks/models.py new file mode 100644 index 00000000..9472de5e --- /dev/null +++ b/vero-tasks/src/vero_tasks/models.py @@ -0,0 +1,178 @@ +"""Provider-neutral task inputs, outputs, and execution context.""" + +from __future__ import annotations + +import asyncio +import re +import traceback +from dataclasses import dataclass, field +from typing import Any, Literal, Sequence, TypeVar + +from pydantic import BaseModel, ConfigDict, Field, JsonValue, model_validator + +TaskT = TypeVar("TaskT") +ParametersT = TypeVar("ParametersT", bound="TaskParameters") + + +class TaskParameters(BaseModel): + """Strict base class for typed task-specific parameters.""" + + model_config = ConfigDict(extra="forbid") + + +class RetryPolicy(BaseModel): + """Per-case retry policy from VeRO's backend-neutral request.""" + + model_config = ConfigDict(extra="forbid") + + max_attempts: int = Field(default=3, ge=1) + initial_delay_seconds: float = Field(default=4.0, ge=0.0) + maximum_delay_seconds: float = Field(default=120.0, ge=0.0) + multiplier: float = Field(default=2.0, ge=1.0) + retry_on_timeout: bool = True + retry_exception_names: list[str] = Field( + default_factory=lambda: [ + "openai.RateLimitError", + "anthropic.RateLimitError", + ] + ) + retry_status_codes: list[int] = Field(default_factory=lambda: [429, 503, 529]) + retry_message_patterns: list[str] = Field( + default_factory=lambda: ["rate limit", "too many requests"] + ) + + @model_validator(mode="after") + def validate_policy(self) -> RetryPolicy: + if self.maximum_delay_seconds < self.initial_delay_seconds: + raise ValueError("maximum retry delay cannot be less than initial delay") + if any(not value.strip() for value in self.retry_exception_names): + raise ValueError("retry exception names must not be empty") + if len(set(self.retry_exception_names)) != len(self.retry_exception_names): + raise ValueError("retry exception names must be unique") + if any(value < 100 or value > 599 for value in self.retry_status_codes): + raise ValueError("retry status codes must be between 100 and 599") + if len(set(self.retry_status_codes)) != len(self.retry_status_codes): + raise ValueError("retry status codes must be unique") + for pattern in self.retry_message_patterns: + if not pattern.strip(): + raise ValueError("retry message patterns must not be empty") + try: + re.compile(pattern) + except re.error as error: + raise ValueError( + f"invalid retry message pattern {pattern!r}: {error}" + ) from error + return self + + def should_retry(self, error: Exception) -> bool: + if self.retry_on_timeout and isinstance( + error, (TimeoutError, asyncio.TimeoutError) + ): + return True + exception_names = { + name + for error_type in type(error).__mro__ + for name in ( + error_type.__name__, + f"{error_type.__module__}.{error_type.__qualname__}", + ) + } + if any(name in exception_names for name in self.retry_exception_names): + return True + status_code = getattr(error, "status_code", getattr(error, "status", None)) + if status_code in self.retry_status_codes: + return True + message = str(error) + return any( + re.search(pattern, message, flags=re.IGNORECASE) + for pattern in self.retry_message_patterns + ) + + def delay_after(self, attempt: int) -> float: + return min( + self.maximum_delay_seconds, + self.initial_delay_seconds * self.multiplier ** (attempt - 1), + ) + + +class TaskAttemptError(BaseModel): + """One failed inference or evaluation attempt.""" + + model_config = ConfigDict(extra="forbid") + + message: str + phase: Literal["inference", "evaluation"] + attempt: int = Field(ge=1) + retryable: bool + terminal: bool + traceback: str | None = None + + +class TaskContext(BaseModel): + """Evaluation context visible to task inference and scoring functions.""" + + model_config = ConfigDict(extra="forbid") + + parameters: dict[str, JsonValue] = Field(default_factory=dict) + max_concurrency: int = Field(default=100, ge=1) + case_timeout_seconds: float = Field(default=180.0, gt=0.0) + retry: RetryPolicy = Field(default_factory=RetryPolicy) + seed: int | None = None + + @property + def task_params(self) -> dict[str, JsonValue]: + return self.parameters + + def parse_task_params(self, model: type[ParametersT]) -> ParametersT: + return model.model_validate(self.parameters) + + +@dataclass +class TaskOutput: + """In-process output of inference for one evaluation case.""" + + output: Any = None + error: Exception | None = None + execution_trace: Sequence[Any] | None = None + attempt_errors: list[TaskAttemptError] = field(default_factory=list) + + +class TaskResult(BaseModel): + """Serializable scoring result for one evaluation case.""" + + model_config = ConfigDict(extra="forbid", arbitrary_types_allowed=True) + + output: Any = None + error: str | None = None + execution_trace: Sequence[Any] | None = None + score: float | None = None + feedback: str | None = None + metrics: dict[str, float] = Field(default_factory=dict) + eval_error: str | None = None + evaluation_trace: Sequence[Any] | None = None + error_traceback: str | None = None + evaluation_error_traceback: str | None = None + attempt_errors: list[TaskAttemptError] = Field(default_factory=list) + + @classmethod + def from_task_output( + cls, + task_output: TaskOutput, + **values: Any, + ) -> TaskResult: + if task_output.error is not None: + values["error"] = str(task_output.error) + values["error_traceback"] = "".join( + traceback.format_exception( + type(task_output.error), + task_output.error, + task_output.error.__traceback__, + ) + ) + values["output"] = task_output.output + values["execution_trace"] = task_output.execution_trace + values.setdefault("attempt_errors", list(task_output.attempt_errors)) + return cls(**values) + + def is_error(self) -> bool: + return self.error is not None or self.eval_error is not None diff --git a/vero-tasks/src/vero_tasks/runner.py b/vero-tasks/src/vero_tasks/runner.py new file mode 100644 index 00000000..0666d75a --- /dev/null +++ b/vero-tasks/src/vero_tasks/runner.py @@ -0,0 +1,220 @@ +"""Run a registered Python task through VeRO's schema-v1 command contract.""" + +from __future__ import annotations + +import argparse +import asyncio +import importlib +import json +from collections import defaultdict +from pathlib import Path +from typing import Any + +from vero_tasks.models import RetryPolicy, TaskContext, TaskResult +from vero_tasks.task import TaskDefinition + + +def _json_value(value: Any) -> Any: + return json.loads(json.dumps(value, default=str)) + + +def _load_cases(path: Path) -> list[Any]: + content = path.read_text(encoding="utf-8") + if path.suffix == ".jsonl": + return [json.loads(line) for line in content.splitlines() if line.strip()] + try: + value = json.loads(content) + except json.JSONDecodeError as error: + lines = [line for line in content.splitlines() if line.strip()] + if len(lines) < 2: + raise error + value = [json.loads(line) for line in lines] + if isinstance(value, dict) and "cases" in value: + value = value["cases"] + if not isinstance(value, list): + raise ValueError( + "case file must contain a JSON list or an object with a cases list" + ) + return value + + +def _case_id(case: Any, index: int) -> str: + if isinstance(case, dict) and case.get("id") is not None: + return str(case["id"]) + return str(index) + + +def _select(cases: list[Any], selection: dict[str, Any]) -> list[tuple[str, Any]]: + indexed = [(_case_id(case, index), case) for index, case in enumerate(cases)] + case_ids = [case_id for case_id, _ in indexed] + if len(case_ids) != len(set(case_ids)): + raise ValueError("case IDs must be unique") + kind = selection.get("kind", "all") + if kind == "all": + return indexed + if kind == "range": + return indexed[selection.get("start", 0) : selection["stop"]] + if kind == "ids": + by_id = dict(indexed) + missing = [case_id for case_id in selection["ids"] if case_id not in by_id] + if missing: + raise ValueError(f"unknown case IDs: {missing}") + return [(case_id, by_id[case_id]) for case_id in selection["ids"]] + raise ValueError(f"unknown case selection kind: {kind!r}") + + +def _errors(result: TaskResult) -> list[dict[str, Any]]: + errors = [ + { + "message": error.message, + "code": f"task_{error.phase}_error", + "phase": error.phase, + "attempt": error.attempt, + "retryable": error.retryable, + "terminal": error.terminal, + "metadata": ( + {"traceback": error.traceback} if error.traceback is not None else {} + ), + } + for error in result.attempt_errors + ] + terminal_phases = {error.phase for error in result.attempt_errors if error.terminal} + if result.error is not None and "inference" not in terminal_phases: + metadata = ( + {"traceback": result.error_traceback} + if result.error_traceback is not None + else {} + ) + errors.append( + { + "message": result.error, + "code": "task_inference_error", + "phase": "inference", + "terminal": True, + "metadata": metadata, + } + ) + if result.eval_error is not None and "evaluation" not in terminal_phases: + metadata = ( + {"traceback": result.evaluation_error_traceback} + if result.evaluation_error_traceback is not None + else {} + ) + errors.append( + { + "message": result.eval_error, + "code": "task_evaluation_error", + "phase": "evaluation", + "terminal": True, + "metadata": metadata, + } + ) + return errors + + +def _report( + selected: list[tuple[str, Any]], + results: list[TaskResult], +) -> dict[str, Any]: + cases = [] + totals: dict[str, list[float]] = defaultdict(list) + error_count = 0 + for (case_id, case), result in zip(selected, results): + metrics = dict(result.metrics) + if result.score is not None: + metrics.setdefault("score", result.score) + errors = _errors(result) + terminal_errors = [error for error in errors if error["terminal"]] + if terminal_errors: + error_count += 1 + else: + for name, value in metrics.items(): + totals[name].append(value) + cases.append( + { + "case_id": case_id, + "status": "error" if terminal_errors else "success", + "metrics": metrics, + "input": _json_value(case), + "output": _json_value(result.output), + "feedback": result.feedback, + "errors": errors, + "execution_trace": ( + _json_value(list(result.execution_trace)) + if result.execution_trace is not None + else None + ), + "evaluation_trace": ( + _json_value(list(result.evaluation_trace)) + if result.evaluation_trace is not None + else None + ), + } + ) + metrics = { + name: sum(values) / len(values) for name, values in totals.items() if values + } + metrics["error_rate"] = error_count / len(results) if results else 0.0 + return { + "schema_version": 1, + "status": "failed" if results and error_count == len(results) else "success", + "metrics": metrics, + "cases": cases, + } + + +async def run_task( + *, + module: str, + task_name: str, + cases_path: Path, + request_path: Path, + report_path: Path, +) -> None: + request_envelope = json.loads(request_path.read_text(encoding="utf-8")) + if request_envelope.get("schema_version") != 1: + raise ValueError("unsupported command evaluation request schema") + request = request_envelope["request"] + importlib.import_module(module) + task = TaskDefinition.resolve(task_name) + selected = _select( + _load_cases(cases_path), + request["evaluation_set"]["selection"], + ) + limits = request["limits"] + context = TaskContext( + parameters=request.get("parameters", {}), + max_concurrency=limits["max_concurrency"], + case_timeout_seconds=limits["case_timeout_seconds"], + retry=RetryPolicy.model_validate(limits.get("retry", {})), + seed=request.get("seed"), + ) + results = await task.run([case for _, case in selected], context) + report_path.parent.mkdir(parents=True, exist_ok=True) + report_path.write_text( + json.dumps(_report(selected, results), ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--module", required=True) + parser.add_argument("--task", required=True) + parser.add_argument("--cases", type=Path, required=True) + parser.add_argument("--request", type=Path, required=True) + parser.add_argument("--report", type=Path, required=True) + arguments = parser.parse_args() + asyncio.run( + run_task( + module=arguments.module, + task_name=arguments.task, + cases_path=arguments.cases, + request_path=arguments.request, + report_path=arguments.report, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/vero-tasks/src/vero_tasks/task.py b/vero-tasks/src/vero_tasks/task.py new file mode 100644 index 00000000..df9cfbaf --- /dev/null +++ b/vero-tasks/src/vero_tasks/task.py @@ -0,0 +1,334 @@ +"""Decorator-based task definition without evaluation persistence concerns.""" + +from __future__ import annotations + +import asyncio +import inspect +import os +import traceback +from collections.abc import Awaitable, Callable, Sequence +from dataclasses import dataclass +from typing import Any + +from vero_tasks.models import ( + TaskAttemptError, + TaskContext, + TaskOutput, + TaskResult, + TaskT, +) + + +async def _resolve(value: Any) -> Any: + return await value if inspect.isawaitable(value) else value + + +@dataclass +class _AttemptOutcome: + value: Any + errors: list[TaskAttemptError] + + +def _merge_attempt_errors( + *groups: Sequence[TaskAttemptError], +) -> list[TaskAttemptError]: + merged: list[TaskAttemptError] = [] + for group in groups: + for error in group: + if error not in merged: + merged.append(error) + return merged + + +class TaskDefinition: + """Inference and evaluation functions registered under one task name.""" + + _registry: dict[str, TaskDefinition] = {} + + def __init__( + self, + name: str, + *, + register: bool = True, + task_parameters_type: type | None = None, + required_env_vars: Sequence[str] | None = None, + ): + if not name.strip(): + raise ValueError("task name must not be empty") + if register and name in self._registry: + raise ValueError(f"task {name!r} is already registered") + self.name = name + self.task_parameters_type = task_parameters_type + self.required_env_vars = tuple(required_env_vars or ()) + self._single: dict[str, Callable[..., Any]] = {} + self._batch: dict[str, Callable[..., Any]] = {} + if register: + self._registry[name] = self + + def _decorator(self, kind: str, *, batch: bool) -> Callable: + expected = { + ("inference", False): 2, + ("inference", True): 2, + ("evaluation", False): 3, + ("evaluation", True): 3, + ("load_data", False): 1, + }[(kind, batch)] + + def register(function: Callable[..., Any]) -> Callable[..., Any]: + parameters = inspect.signature(function).parameters + if len(parameters) != expected: + raise TypeError( + f"{kind} function {function.__name__!r} must accept " + f"{expected} parameters, got {len(parameters)}" + ) + functions = self._batch if batch else self._single + if kind in functions: + raise ValueError(f"{kind} function is already registered") + functions[kind] = function + return function + + return register + + def inference(self, *, batch: bool = False) -> Callable: + return self._decorator("inference", batch=batch) + + def evaluation(self, *, batch: bool = False) -> Callable: + return self._decorator("evaluation", batch=batch) + + def load_data(self) -> Callable: + return self._decorator("load_data", batch=False) + + def __call__(self, name: str, *, batch: bool = False) -> Callable: + aliases = { + "run_inference": "inference", + "run_evaluation": "evaluation", + "load_task_data": "load_data", + "create_task": "load_data", + } + try: + kind = aliases[name] + except KeyError as error: + raise ValueError(f"unknown task function kind: {name!r}") from error + return self._decorator(kind, batch=batch) + + def get(self, kind: str, *, batch: bool = False) -> Callable[..., Any] | None: + return (self._batch if batch else self._single).get(kind) + + @classmethod + def resolve(cls, name: str) -> TaskDefinition: + try: + return cls._registry[name] + except KeyError as error: + raise KeyError( + f"task {name!r} is not registered; available: {sorted(cls._registry)}" + ) from error + + @classmethod + def clear_registry(cls) -> None: + cls._registry.clear() + + def _validate(self, context: TaskContext) -> None: + missing = [name for name in self.required_env_vars if not os.environ.get(name)] + if missing: + raise ValueError( + "missing required task environment variables: " + ", ".join(missing) + ) + if self.task_parameters_type is not None: + context.parse_task_params(self.task_parameters_type) + if self.get("inference") is None and self.get("inference", batch=True) is None: + raise RuntimeError("task has no inference function") + if ( + self.get("evaluation") is None + and self.get("evaluation", batch=True) is None + ): + raise RuntimeError("task has no evaluation function") + + @staticmethod + def _output( + value: Any, + attempt_errors: Sequence[TaskAttemptError] = (), + ) -> TaskOutput: + if isinstance(value, TaskOutput): + errors = _merge_attempt_errors(attempt_errors, value.attempt_errors) + if errors == value.attempt_errors: + return value + return TaskOutput( + output=value.output, + error=value.error, + execution_trace=value.execution_trace, + attempt_errors=errors, + ) + if isinstance(value, BaseException): + error = value if isinstance(value, Exception) else Exception(str(value)) + return TaskOutput(error=error, attempt_errors=list(attempt_errors)) + return TaskOutput(output=value, attempt_errors=list(attempt_errors)) + + @staticmethod + def _result( + output: TaskOutput, + value: Any, + attempt_errors: Sequence[TaskAttemptError] = (), + ) -> TaskResult: + if isinstance(value, TaskResult): + updates: dict[str, Any] = {} + if output.error is not None and value.error is None: + updates["error"] = str(output.error) + updates["error_traceback"] = "".join( + traceback.format_exception( + type(output.error), + output.error, + output.error.__traceback__, + ) + ) + if output.execution_trace is not None and value.execution_trace is None: + updates["execution_trace"] = output.execution_trace + errors = _merge_attempt_errors( + output.attempt_errors, + value.attempt_errors, + attempt_errors, + ) + if errors != value.attempt_errors: + updates["attempt_errors"] = errors + return value.model_copy(update=updates) if updates else value + if isinstance(value, BaseException): + return TaskResult.from_task_output( + output, + eval_error=str(value) or type(value).__name__, + evaluation_error_traceback="".join( + traceback.format_exception(type(value), value, value.__traceback__) + ), + attempt_errors=_merge_attempt_errors( + output.attempt_errors, + attempt_errors, + ), + ) + raise TypeError( + f"evaluation returned {type(value).__name__}, expected TaskResult" + ) + + async def _map( + self, + factories: Sequence[Callable[[], Awaitable[Any] | Any]], + context: TaskContext, + *, + phase: str, + ) -> list[_AttemptOutcome]: + semaphore = asyncio.Semaphore(context.max_concurrency) + + async def run( + factory: Callable[[], Awaitable[Any] | Any], + ) -> _AttemptOutcome: + errors: list[TaskAttemptError] = [] + for attempt in range(1, context.retry.max_attempts + 1): + try: + async with semaphore: + async with asyncio.timeout(context.case_timeout_seconds): + value = await _resolve(factory()) + return _AttemptOutcome(value=value, errors=errors) + except Exception as error: + retryable = context.retry.should_retry(error) + terminal = not retryable or attempt == context.retry.max_attempts + errors.append( + TaskAttemptError( + message=str(error) or type(error).__name__, + phase=phase, + attempt=attempt, + retryable=retryable, + terminal=terminal, + traceback="".join( + traceback.format_exception( + type(error), error, error.__traceback__ + ) + ), + ) + ) + if terminal: + return _AttemptOutcome(value=error, errors=errors) + delay = context.retry.delay_after(attempt) + if delay: + await asyncio.sleep(delay) + raise AssertionError("retry loop completed without an outcome") + + return list(await asyncio.gather(*(run(factory) for factory in factories))) + + async def run( + self, + cases: Sequence[TaskT] | None, + context: TaskContext, + ) -> list[TaskResult]: + self._validate(context) + if cases is None: + loader = self.get("load_data") + if loader is None: + raise ValueError( + "cases are required when the task has no load_data function" + ) + cases = list(await _resolve(loader(context))) + else: + cases = list(cases) + + batch_inference = self.get("inference", batch=True) + if batch_inference is not None: + raw_outputs = list(await _resolve(batch_inference(cases, context))) + else: + inference = self.get("inference") + assert inference is not None + inference_outcomes = await self._map( + [lambda case=case: inference(case, context) for case in cases], + context, + phase="inference", + ) + raw_outputs = [outcome.value for outcome in inference_outcomes] + if len(raw_outputs) != len(cases): + raise ValueError("inference result count does not match case count") + if batch_inference is not None: + outputs = [self._output(value) for value in raw_outputs] + else: + outputs = [ + self._output(outcome.value, outcome.errors) + for outcome in inference_outcomes + ] + + batch_evaluation = self.get("evaluation", batch=True) + if batch_evaluation is not None: + raw_results = list( + await _resolve(batch_evaluation(cases, outputs, context)) + ) + else: + evaluation = self.get("evaluation") + assert evaluation is not None + evaluation_outcomes = await self._map( + [ + lambda case=case, output=output: evaluation(case, output, context) + for case, output in zip(cases, outputs) + ], + context, + phase="evaluation", + ) + raw_results = [outcome.value for outcome in evaluation_outcomes] + if len(raw_results) != len(cases): + raise ValueError("evaluation result count does not match case count") + if batch_evaluation is not None: + return [ + self._result(output, result) + for output, result in zip(outputs, raw_results) + ] + return [ + self._result(output, outcome.value, outcome.errors) + for output, outcome in zip(outputs, evaluation_outcomes) + ] + + +def create_task( + name: str, + *, + register: bool = True, + task_parameters_type: type | None = None, + required_env_vars: Sequence[str] | None = None, +) -> TaskDefinition: + return TaskDefinition( + name, + register=register, + task_parameters_type=task_parameters_type, + required_env_vars=required_env_vars, + ) diff --git a/vero-tasks/tests/test_runner.py b/vero-tasks/tests/test_runner.py new file mode 100644 index 00000000..b9efc16f --- /dev/null +++ b/vero-tasks/tests/test_runner.py @@ -0,0 +1,200 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from vero_tasks import TaskAttemptError, TaskResult +from vero_tasks.runner import _errors, _load_cases, run_task +from vero_tasks.task import TaskDefinition + + +def test_load_cases_detects_staged_jsonl_without_a_suffix(tmp_path: Path): + cases_path = tmp_path / "cases" + cases_path.write_text('{"id": "a"}\n{"id": "b"}\n', encoding="utf-8") + + assert _load_cases(cases_path) == [{"id": "a"}, {"id": "b"}] + + +def test_errors_preserve_transient_history_and_explicit_terminal_errors(): + errors = _errors( + TaskResult( + eval_error="invalid score", + attempt_errors=[ + TaskAttemptError( + message="rate limit", + phase="inference", + attempt=1, + retryable=True, + terminal=False, + ) + ], + ) + ) + + assert [(error["phase"], error["terminal"]) for error in errors] == [ + ("inference", False), + ("evaluation", True), + ] + + +@pytest.mark.asyncio +async def test_runner_writes_canonical_report(tmp_path: Path, monkeypatch): + module = tmp_path / "example_tasks.py" + module.write_text( + """ +from vero_tasks import TaskOutput, TaskResult, create_task + +task = create_task("double") + +@task.inference() +async def infer(case, context): + return TaskOutput(output=case["value"] * 2) + +@task.evaluation() +async def evaluate(case, output, context): + return TaskResult.from_task_output( + output, + score=float(output.output == case["expected"]), + metrics={"distance": abs(output.output - case["expected"])}, + ) +""", + encoding="utf-8", + ) + monkeypatch.syspath_prepend(str(tmp_path)) + cases_path = tmp_path / "cases.json" + cases_path.write_text( + json.dumps( + [ + {"id": "a", "value": 2, "expected": 4}, + {"id": "b", "value": 3, "expected": 7}, + {"id": "c", "value": 4, "expected": 8}, + ] + ), + encoding="utf-8", + ) + request_path = tmp_path / "request.json" + request_path.write_text( + json.dumps( + { + "schema_version": 1, + "request": { + "evaluation_set": { + "name": "test", + "partition": None, + "selection": {"kind": "ids", "ids": ["c", "a"]}, + }, + "parameters": {}, + "limits": { + "max_concurrency": 2, + "case_timeout_seconds": 1.0, + }, + "seed": 3, + }, + } + ), + encoding="utf-8", + ) + report_path = tmp_path / "report.json" + + await run_task( + module="example_tasks", + task_name="double", + cases_path=cases_path, + request_path=request_path, + report_path=report_path, + ) + + report = json.loads(report_path.read_text(encoding="utf-8")) + assert report["schema_version"] == 1 + assert report["status"] == "success" + assert report["metrics"] == { + "distance": 0.0, + "score": 1.0, + "error_rate": 0.0, + } + assert [case["case_id"] for case in report["cases"]] == ["c", "a"] + TaskDefinition.clear_registry() + + +@pytest.mark.asyncio +async def test_runner_applies_retry_policy_and_reports_attempts( + tmp_path: Path, monkeypatch +): + module = tmp_path / "retry_tasks.py" + module.write_text( + """ +from vero_tasks import TaskOutput, TaskResult, create_task + +task = create_task("retry") +attempts = 0 + +@task.inference() +async def infer(case, context): + global attempts + attempts += 1 + if attempts == 1: + raise RuntimeError("rate limit") + return TaskOutput(output=case["value"]) + +@task.evaluation() +async def evaluate(case, output, context): + return TaskResult.from_task_output(output, score=1.0) +""", + encoding="utf-8", + ) + monkeypatch.syspath_prepend(str(tmp_path)) + cases_path = tmp_path / "cases.json" + cases_path.write_text(json.dumps([{"id": "a", "value": 2}]), encoding="utf-8") + request_path = tmp_path / "request.json" + request_path.write_text( + json.dumps( + { + "schema_version": 1, + "request": { + "evaluation_set": { + "name": "test", + "partition": None, + "selection": {"kind": "all"}, + }, + "parameters": {}, + "limits": { + "max_concurrency": 1, + "case_timeout_seconds": 1.0, + "retry": { + "max_attempts": 2, + "initial_delay_seconds": 0, + }, + }, + }, + } + ), + encoding="utf-8", + ) + report_path = tmp_path / "report.json" + + await run_task( + module="retry_tasks", + task_name="retry", + cases_path=cases_path, + request_path=request_path, + report_path=report_path, + ) + + report = json.loads(report_path.read_text(encoding="utf-8")) + assert report["status"] == "success" + assert report["metrics"]["error_rate"] == 0.0 + assert report["cases"][0]["status"] == "success" + assert report["cases"][0]["errors"][0] == { + "message": "rate limit", + "code": "task_inference_error", + "phase": "inference", + "attempt": 1, + "retryable": True, + "terminal": False, + "metadata": { + "traceback": report["cases"][0]["errors"][0]["metadata"]["traceback"] + }, + } + TaskDefinition.clear_registry() diff --git a/vero-tasks/tests/test_task.py b/vero-tasks/tests/test_task.py new file mode 100644 index 00000000..1a952680 --- /dev/null +++ b/vero-tasks/tests/test_task.py @@ -0,0 +1,202 @@ +from __future__ import annotations + +import asyncio + +import pytest + +from vero_tasks import ( + RetryPolicy, + TaskContext, + TaskOutput, + TaskParameters, + TaskResult, + create_task, +) + + +class Parameters(TaskParameters): + multiplier: float = 1.0 + + +@pytest.mark.asyncio +async def test_task_runs_cases_with_typed_parameters_and_concurrency(): + task = create_task("score", register=False, task_parameters_type=Parameters) + active = 0 + maximum_active = 0 + + @task.inference() + async def infer(case, context): + nonlocal active, maximum_active + active += 1 + maximum_active = max(maximum_active, active) + await asyncio.sleep(0) + active -= 1 + return TaskOutput(output=case["value"] * 2) + + @task.evaluation() + async def evaluate(case, output, context): + parameters = context.parse_task_params(Parameters) + return TaskResult.from_task_output( + output, + score=output.output * parameters.multiplier, + metrics={"raw": output.output}, + ) + + results = await task.run( + [{"value": 1}, {"value": 2}, {"value": 3}], + TaskContext(parameters={"multiplier": 0.5}, max_concurrency=2), + ) + + assert [result.score for result in results] == [1.0, 2.0, 3.0] + assert maximum_active == 2 + + +@pytest.mark.asyncio +async def test_task_captures_inference_and_evaluation_errors(): + task = create_task("errors", register=False) + + @task("run_inference") + async def infer(case, context): + if case == "inference": + raise RuntimeError("inference failed") + return TaskOutput(output=case) + + @task("run_evaluation") + async def evaluate(case, output, context): + if case == "evaluation": + raise RuntimeError("evaluation failed") + return TaskResult.from_task_output(output, score=1.0) + + results = await task.run( + ["inference", "evaluation", "ok"], + TaskContext(), + ) + + assert results[0].error == "inference failed" + assert results[0].error_traceback is not None + assert results[1].eval_error == "evaluation failed" + assert results[1].evaluation_error_traceback is not None + assert results[2].score == 1.0 + + +@pytest.mark.asyncio +async def test_batch_evaluation_preserves_inference_errors(): + task = create_task("batch-errors", register=False) + + @task.inference() + async def infer(case, context): + if case == "broken": + raise RuntimeError("inference failed") + return TaskOutput(output=case) + + @task.evaluation(batch=True) + async def evaluate(cases, outputs, context): + return [TaskResult(score=1.0) for _ in cases] + + results = await task.run(["broken", "ok"], TaskContext()) + + assert results[0].error == "inference failed" + assert results[1].error is None + + +@pytest.mark.asyncio +async def test_task_retries_transient_inference_errors_and_preserves_history(): + task = create_task("retry-inference", register=False) + attempts = 0 + + @task.inference() + async def infer(case, context): + nonlocal attempts + attempts += 1 + if attempts < 3: + raise RuntimeError("rate limit from provider") + return TaskOutput(output=case) + + @task.evaluation() + async def evaluate(case, output, context): + return TaskResult.from_task_output(output, score=1.0) + + results = await task.run( + ["ok"], + TaskContext( + retry=RetryPolicy( + max_attempts=3, + initial_delay_seconds=0, + ) + ), + ) + + assert attempts == 3 + assert results[0].score == 1.0 + assert results[0].is_error() is False + assert [error.attempt for error in results[0].attempt_errors] == [1, 2] + assert all(not error.terminal for error in results[0].attempt_errors) + + +@pytest.mark.asyncio +async def test_task_retries_timeouts_and_evaluation_errors(): + task = create_task("retry-timeout-and-evaluation", register=False) + inference_attempts = 0 + evaluation_attempts = 0 + + @task.inference() + async def infer(case, context): + nonlocal inference_attempts + inference_attempts += 1 + if inference_attempts == 1: + await asyncio.sleep(0.02) + return TaskOutput(output=case) + + @task.evaluation() + async def evaluate(case, output, context): + nonlocal evaluation_attempts + evaluation_attempts += 1 + if evaluation_attempts == 1: + raise RuntimeError("too many requests") + return TaskResult.from_task_output(output, score=1.0) + + results = await task.run( + ["ok"], + TaskContext( + case_timeout_seconds=0.01, + retry=RetryPolicy(max_attempts=2, initial_delay_seconds=0), + ), + ) + + assert inference_attempts == 2 + assert evaluation_attempts == 2 + assert results[0].score == 1.0 + assert [ + (error.phase, error.attempt, error.terminal) + for error in results[0].attempt_errors + ] == [ + ("inference", 1, False), + ("evaluation", 1, False), + ] + + +@pytest.mark.asyncio +async def test_task_does_not_retry_non_transient_errors(): + task = create_task("no-retry", register=False) + attempts = 0 + + @task.inference() + async def infer(case, context): + nonlocal attempts + attempts += 1 + raise ValueError("invalid candidate") + + @task.evaluation() + async def evaluate(case, output, context): + return TaskResult.from_task_output(output) + + results = await task.run( + ["broken"], + TaskContext(retry=RetryPolicy(initial_delay_seconds=0)), + ) + + assert attempts == 1 + assert results[0].error == "invalid candidate" + assert len(results[0].attempt_errors) == 1 + assert results[0].attempt_errors[0].retryable is False + assert results[0].attempt_errors[0].terminal is True diff --git a/vero/.gitignore b/vero/.gitignore new file mode 100644 index 00000000..4b6c6a55 --- /dev/null +++ b/vero/.gitignore @@ -0,0 +1,40 @@ +# Environment +.env +.env.* +.venv/ +venv/ + +# Python +__pycache__/ +*.pyc +*.pyo +*.egg-info/ +dist/ +build/ +!src/vero/harbor/build/ +!src/vero/harbor/build/*.py +!src/vero/harbor/build/templates/ +!src/vero/harbor/build/templates/* + +#harbor +jobs/ + +# Testing +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +htmlcov/ +.coverage + +# IDE +.idea/ +.vscode/ +*.swp +*.swo + +# Jupyter +**/.ipynb_checkpoints + +# OS +.DS_Store +Thumbs.db diff --git a/vero/README.md b/vero/README.md new file mode 100644 index 00000000..7a501ab7 --- /dev/null +++ b/vero/README.md @@ -0,0 +1,850 @@ +# VeRO: a harness for agents to optimize programs, text, and agents + +[![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) +[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/) + +VeRO gives an optimizer something to edit, a controlled way to evaluate it, and +durable memory of everything it tried. The target is anything you can put under +Git and score — a **program** (one function to a whole repo), **text** (a prompt, +spec, or config), or an **agent** (its scaffold, tools, and prompts). + +VeRO runs the same **version → evaluate → select** loop over all of them. *Where* +each candidate is produced and contained is a swappable backend — and +**[Harbor](#optimize-an-agent-with-harbor) is the recommended one**: it runs the +whole coding agent inside a reproducible, credential-isolated container and scores +it against a trusted evaluation sidecar. That's the right default for optimizing +agents and for any untrusted or reproducibility-critical run. Lighter local +backends (a language-neutral command harness, Python tasks) are there for trusted +work that doesn't need containment. + +```mermaid +flowchart LR + S["Strategy
proposes ideas"] --> P["Producers edit
isolated candidate
workspaces"] + P --> E["Evaluation backend
scores each version"] + E --> Sel["Selection keeps the
best feasible candidate"] + Sel -->|"next round"| S +``` + +## Highlights + +| | | +| --- | --- | +| **Harbor-first containment** | run the whole agent in a reproducible container against a trusted eval sidecar, behind a credential-isolating inference gateway — the recommended backend | +| **Any target** | a program (one function up to a whole repo), text (prompts, specs, configs), or an agent | +| **Any producer** | a coding agent (any provider via LiteLLM), an external command, or a custom strategy | +| **Durable & inspectable** | every candidate is versioned and re-selectable; tool calls and evaluations stream to an event log | +| **Population search** | `EvolutionaryStrategy` fans out N offspring per round with tournament selection | + +## Backends + +VeRO separates *what* it optimizes from *how* candidates are produced and scored. +Start with Harbor; drop to a lighter backend when you don't need containment. + +| Backend | Best for | Entry point | +| --- | --- | --- | +| **[Harbor](#optimize-an-agent-with-harbor)** — recommended | optimizing agents; untrusted or reproducibility-critical runs. Runs the whole agent in a container against a trusted eval sidecar, with a credential-isolating inference gateway. | `vero harbor run` | +| [Command harness](#optimize-a-program-with-a-command-harness) | any language; a trusted local evaluator you drive over versioned JSON | `vero run` | +| [Python tasks](#python-benchmark-tasks) | Python evaluators via `scale-vero-tasks`, no JSON contract to write | `PythonTaskBackend` | +| [Native in-process](#python-api) | fast trusted local runs; a coding agent whose edits execute in a host-bound sandbox | `vero optimize` | + +**Harbor in one command** — compile a build file into a contained task and run an +agent against it, with secrets kept off the command line: + +```bash +vero harbor run \ + --config build.yaml \ + --agent claude-code --model claude-sonnet-4-6 \ + --env-file secrets.env +``` + +See [Optimize an agent with Harbor](#optimize-an-agent-with-harbor) for the build +file, the inference gateway, and how disclosure and budgets are enforced. + +> **Just want to kick the tires first?** The [Quickstart](#quickstart) C-matmul +> example is deterministic and needs no model credentials. + +## Quickstart + +Install VeRO, then try the checked-in C matrix multiplication example. Its +editable target contains only C; a trusted external harness compiles it, checks +correctness, and measures latency. + +```bash +# Install VeRO from this checkout. Do NOT `pip install scale-vero` from public +# PyPI — that name is currently an unrelated placeholder, not VeRO. +uv sync --extra optimize + +cd examples/c-matmul/target +git init -b main +git add . +git -c user.name=vero -c user.email=vero@localhost commit -m baseline +cd .. + +uv run vero evaluate --config vero.toml +uv run vero run --config vero.toml +``` + +The example is deterministic and needs no model credentials. VeRO evaluates the +baseline, gives an isolated worktree to the configured producer, evaluates its +commit, selects the faster feasible result, and leaves the original target +untouched. See [`examples/c-matmul`](examples/c-matmul/) for the complete target, +harness, optimizer, and config. + +For a more demanding coding-agent run, use the +[`examples/circle-packing`](examples/circle-packing/) benchmark adapted from +ShinkaEvolve. It asks an agent to improve a 26-circle packing, exposes exact +geometric diagnostics and layout artifacts after each authorized evaluation, +and re-evaluates the selected candidate through a hidden final evaluation. + +## Optimize an agent with Harbor + +Harbor is VeRO's recommended backend for optimizing agents. Each candidate runs +as a **contained agent**, scored by a **trusted sidecar**, with provider +credentials held by a **separate gateway** the agent never sees: + +```mermaid +flowchart LR + A["Optimizer container
agent edits the candidate
+ calls the CLI"] + S["Eval sidecar (trusted)
owns cases, scoring,
budgets, finalization"] + G["Inference gateway (trusted)
holds the upstream key,
issues scoped tokens"] + A -->|"evals run"| S + A -->|"scoped model calls"| G + G --> U["Provider / litellm proxy"] +``` + +Two steps: **build** a contained task from a YAML file, then **run** an agent +against it. + +### 1. The build file + +`build.yaml` declares the target repo, the task source and case partitions, what +the agent may evaluate, and the inference gateway: + +```yaml +name: example/optimize-agent +agent_repo: ../my-program # editable target (a Git repo) +task_source: example/terminal-benchmark@1.0 # pinned registry tasks +agent_import_path: my_program.agent:Agent +harbor_requirement: harbor[modal]==0.20.0 +environment_name: modal # where each nested eval runs +secrets: [MODAL_TOKEN_ID, MODAL_TOKEN_SECRET] # sidecar-only; stripped from the agent + +inference_gateway: + upstream_api_key_env: OPENAI_API_KEY + upstream_base_url_env: OPENAI_BASE_URL + producer: # the optimizer's scope + allowed_models: [gpt-5] + evaluation: # the target's scope + allowed_models: [gpt-5-mini] + max_requests: 5000 + max_tokens: 20000000 + +partitions: + validation: [example/task-a, example/task-b, example/task-c] + test: [example/task-hidden] + +agent_access: # what the agent may evaluate, and how much it sees + - partition: validation + disclosure: aggregate # full | aggregate | none + total_runs: 10 + total_cases: 50 + +selection_partition: validation # candidates are ranked here +targets: + - partition: test # held-out; trusted verifier scores it after selection + reward_key: reward +``` + +| Knob | What it controls | +| --- | --- | +| `agent_access[].disclosure` | how much of an evaluation the agent sees: `full` (per-case traces), `aggregate` (scores only), `none` | +| `agent_access[].total_runs` / `total_cases` | the agent's **cumulative** eval budget — it may spend it all in one run | +| `selection_partition` | the fixed set every candidate is ranked on | +| `targets` | held-out partitions scored after selection; never enter agent context | +| `inference_gateway.{producer,evaluation}.allowed_models` | the models each scope may call (optimizer vs. target) | + +### 2. Run it + +`vero harbor run` compiles the build file and launches the agent, keeping secrets +off the command line via `--env-file`: + +```bash +vero harbor run \ + --config build.yaml \ + --agent claude-code --model claude-sonnet-4-6 \ + --environment modal \ + --env-file secrets.env +``` + +`--agent` picks the coding agent (`claude-code`, `codex`, …) and `--model` sets +both its model and its producer-scope allow-list. `vero harbor build --config +build.yaml --output task` compiles without running, for inspection. + +> **Use `vero harbor run`, not `harbor run` directly.** For a gateway-enabled +> build the VeRO launcher renames provider credentials for the gateway before +> Harbor constructs the agent; a raw `harbor run` would let the agent adapter +> read the upstream key from its own host process first. + +Inside the container the agent evaluates candidates with `evals run +--detach`, then `evals status JOB` / `evals result JOB` / `evals status` (via `VERO_EVAL_URL`). +Detached evaluations are **durable jobs** — the candidate version is captured +before the command returns, so ending the agent process can't lose or race a +running measurement. + +### How the boundaries hold + +- **The sidecar is the only evaluator.** It owns the cases, scoring, budget + ledger, and candidate selection. The optimizer container gets only the editable + baseline, the agent CLI, and approved result projections; the `test` partition + and task source live only in the sidecar image. +- **The gateway holds the provider key.** A third trusted service alone receives + the upstream credential, then forwards any inference endpoint to your upstream + proxy while restricting each scope to its configured models and metering usage. + The optimizer gets a producer-scoped token (uncapped by default); each + candidate evaluation gets an independently budgeted token on a URL attributed + to its evaluation ID. +- **Disclosure is graded.** `full` exposes per-case traces for the agent to + inspect, `aggregate` returns scores only, and held-out `targets` are + unreachable from agent context. Aggregate validation is optimization data, not + a privacy guarantee — arbitrary subsets are allowed; the final target is not. + +### Artifacts + +The verifier shares the environment, so it exports the full session before +teardown: a successful run leaves `session.tar.gz` (+ `.sha256`), +`experiment.html`, `status.json`, and `finalization.json` in the verifier +artifacts. The archive holds the bare candidate Git repo, canonical evaluation +records, budget state, the finalization result, and the producer trajectory — +re-render it any time with `vero report`. Export failure fails the run rather +than discarding the only durable copy. + +> **Security boundary.** The inference gateway protects *provider credentials*, +> not the OS process. The pinned Harbor overlay and sidecar keep budget and +> scoring trusted, but candidate code still runs inside the nested Harbor +> process, and a Modal controller still needs Modal auth. When target programs +> are **adversarial**, run them in a separate sandbox that can't reach verifier +> data or credentials. + +
+Operational details — timeouts, finalization draining, budgets + +- **Timeouts.** `case_timeout_seconds` is VeRO's absolute limit for the agent + phase. Because Harbor applies task timeouts as a multiplier, set + `task_agent_timeout_seconds` to the pinned task's declared agent timeout; the + compiler passes their ratio (e.g. `180 / 600 = 0.3`) as Harbor's multiplier, + leaving verifier and setup timeouts unchanged. +- **Finalization** closes the agent's eval entrance and waits for every + already-accepted request (including ones launched from a background shell) + before selecting, so ending the optimizer can't race a running validation. + After `evaluation_drain_timeout_seconds` (default `timeout_seconds`) it cancels + and refunds the unfinished eval. Trusted verifier evals remain available after + the entrance closes. +- **Budgets** are cumulative and split between agent- and system-triggered evals. + Cancelled runs and execution failures are refunded; completed failure reports + stay charged as measurements. Whole-run infrastructure failures are retried and + surfaced separately, so an outage fails the session rather than becoming a + candidate regression. +- **Request vs. token limits.** Request limits are exact; a token limit stops the + *next* request after reported usage crosses it, so in-flight concurrent + responses can overshoot slightly. Omit either to record usage without a ceiling. +- **Admin credential.** The shared-container topology protects the admin token + with Unix ownership/permissions and assumes candidate code can't gain root; + higher-assurance setups keep finalization credentials out of the agent + workbench entirely. + +
+ +
+Advanced — Harbor as a plain EvaluationBackend + +Harbor is also a normal backend you can drive from Python. Map each +`EvaluationSet` case to one Harbor task and pin the orchestrating package: + +```json +{"id": "task-1", "task_name": "org/terminal-task-1"} +{"id": "task-2", "task_name": "org/terminal-task-2"} +``` + +```python +from vero.harbor import HarborBackend, HarborBackendConfig + +backend = HarborBackend(HarborBackendConfig( + task_source="org/terminal-benchmark@1.0", + agent_import_path="my_program.agent:Agent", + cases_path=str(Path("../harbor-cases.jsonl").resolve()), + harbor_requirement="harbor[modal]==0.20.0", + evaluation_set_name="terminal-benchmark", + partition="test", + passthrough_environment=["ANTHROPIC_API_KEY"], +)) +``` + +VeRO invokes `harbor run` without importing Harbor into the core library, +collates verifier rewards into schema-v1 case results, zero-fills dead attempts, +and preserves Harbor output as artifacts. + +For optimization-as-a-Harbor-task, `EvaluationSidecar` exposes the same engine +across a process boundary — install the `harbor` extra (`uv sync --extra harbor`) +and serve a trusted factory: + +```bash +vero harbor serve \ + --factory trusted_deployment:build_components \ + --config /etc/vero/sidecar.json \ + --admin-token /shared/admin-token +``` + +`SidecarEvaluationPolicy` maps partitions to full/aggregate/none disclosure, +`GitCandidateTransport` imports agent commits under trusted refs, and +`CanonicalVerifier` re-scores the selected candidate. `EvaluationBackend`, +`CandidateProducer`, `OptimizationStrategy`, and `SelectionPolicy` are all +protocols — implement them for a remote evaluator, a non-Git store, or a custom +search strategy. + +
+ +## Optimize a program with a command harness + +When you don't need containment — a trusted local evaluator, in any language — +the **command backend** is the lightest path: VeRO drives your evaluator over +versioned JSON, and `vero.toml` is the shortest way to configure it. The comments +below cover most of what you need: + +```toml +[target] +root = "./my-program" # a clean Git repo — the editable target +ref = "HEAD" + +[backend] +id = "command" +kind = "command" +harness_root = "../my-evaluator" # trusted; must live outside the target +command = ["python3", "evaluate.py", "{workspace}", "{request}", "{report}"] + +[backend.staged_inputs] # trusted files, referenced via {input:NAME} +train_cases = "../my-evaluator/train.jsonl" +validation_cases = "../my-evaluator/validation.jsonl" +test_cases = "../my-evaluator/test.jsonl" + +[backend.agent_context_inputs] # per-evaluation allowlist of what the agent may see +train = ["train_cases"] + +[[evaluations]] # what the agent may run, and how much it sees +name = "train" +partition = "train" +agent_can_evaluate = true +agent_visible = true +disclosure = "full" # full | aggregate | none +expose_case_resources = true + +[[evaluations]] +name = "validation" +partition = "validation" +agent_can_evaluate = true +agent_visible = true +disclosure = "aggregate" + +[evaluations.agent_budget] # cumulative; the agent decides how to spend it +total_runs = 50 +total_cases = 5000 + +[[evaluations]] +name = "test" # held-out — agent can neither see nor run it +partition = "test" +agent_can_evaluate = false +agent_visible = false +disclosure = "none" + +[protocol] +selection_evaluation = "validation" # every candidate is ranked here +final_evaluation = "test" # scored by the trusted runtime after selection +max_proposals = 5 +error_rate_threshold = 0.1 # an eval fails if >=10% of its cases error + +[objective] +metric = "latency_ms" +direction = "minimize" + +[[objective.constraints]] +metric = "correct" +operator = "==" +value = 1.0 + +[optimizer] +kind = "claude" +model = "claude-sonnet-4-5-20250929" +instruction = "Make the program faster without changing its output" + +[session] +directory = "../runs/my-program" +``` + +Run `vero evaluate` to measure only the baseline, or `vero run` to produce and +evaluate candidates. `vero init` writes this starter profile and `vero check` +validates it before an expensive run. Paths resolve relative to the config; the +session directory, harness, and producer must all live outside the target. + +The evaluator gets an isolated candidate workspace and writes a versioned JSON +report: + +```python +# ../my-evaluator/evaluate.py +import json, sys +from pathlib import Path + +workspace = Path(sys.argv[1]) +report_path = Path(sys.argv[2]) + +latency_ms = measure(workspace) # build, run, benchmark, call a service, ... + +report_path.write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"latency_ms": latency_ms}, +})) +``` + +
+Retries, error thresholds, and case aggregation + +- **Retries** wrap each case's inference/scoring call — by default provider rate + limits, HTTP 429/503/529, and timeouts, up to 3 attempts with bounded backoff. + A successful retry stays a success; earlier failed attempts are kept in the + case's structured error history. Set `max_attempts = 1` to disable them. +- **Error threshold.** An otherwise-successful evaluation becomes *failed* once + ≥10% of its selected cases error (`protocol.error_rate_threshold`). For + mean-aggregated objectives, set `aggregation = "mean"` and `case_failure_value` + so a candidate can't improve its score by failing hard cases. +- **Selection is fixed.** Candidates are ranked on the full `validation` + selection regardless of the cheaper subsets the agent explored; the agent sees + only aggregate validation feedback, and `test` never enters agent context. + +
+ +Then choose how candidates are changed. + +### Optimize with a coding agent + +```bash +vero optimize ./my-program \ + --harness-root ../my-evaluator \ + --evaluate 'python3 evaluate.py {workspace} {report}' \ + --agent claude \ + --instruction 'Make the program faster without changing its output' \ + --metric latency_ms \ + --direction minimize \ + --max-proposals 5 +``` + +Use `--agent vero` for VeRO's OpenAI Agents SDK implementation. It runs on any +provider via LiteLLM; its harness runs on the host while its shell and file +edits execute inside a sandbox bound to the candidate checkout (see [Safety +boundaries](#safety-boundaries)). In `vero.toml`, `optimizer.model` selects an +explicit model identifier; for the `vero` agent you can also set it via the +`VERO_OPTIMIZER_MODEL` environment variable, and omitting both preserves the +adapter's default. Provider-specific dependencies and credentials are required +for either built-in coding agent. For a contained/untrusted producer, run the +agent through Harbor (see [`examples/harbor-circle-packing`](examples/harbor-circle-packing/)). + +### Optimize with any external producer + +An external producer receives an isolated workspace and edits it in place: + +```bash +vero optimize ./my-program \ + --harness-root ../my-evaluator \ + --evaluate 'python3 evaluate.py {workspace} {report}' \ + --producer-root ../my-optimizer \ + --produce 'python3 improve.py {workspace}' \ + --metric latency_ms \ + --direction minimize +``` + +Commands are parsed into argument vectors, not executed through a shell. Use +absolute executable paths when the executable is not on the standard system +`PATH`. Available evaluation placeholders are `{workspace}`, `{request}`, +`{report}`, `{artifacts}`, and `{harness}`. External producers additionally get +`{producer}` and `{context}`; `VERO_CONTEXT_PATH` contains the same context path. +The harness and producer roots resolve to staged sandbox paths when the target +is not host-visible. + +`staged_inputs` are trusted evaluator inputs available to the evaluation +command through `{input:NAME}` placeholders. They remain hidden from candidate +producers unless their names are also explicitly listed in +`agent_context_inputs` for a specific named evaluation, as in the example +above. This per-evaluation allowlist prevents exposing a test input merely +because train and test share one command backend. + +The flag-based `vero optimize` command exposes the same objective constraints, +case selection, target ref, timeouts, environments, and concurrency controls; +run `vero optimize --help` for the full surface. + +## Track runs with Weights & Biases + +Install the optional integration (the `wandb` extra) and add a section to +`vero.toml`: + +```bash +uv sync --extra wandb +``` + +```toml +[wandb] +project = "program-optimization" +entity = "my-team" # optional +name = "matmul-v1" # optional +mode = "online" # online, offline, or disabled +tags = ["c", "latency"] +``` + +Each VeRO session maps to a stable W&B run. It logs canonical report metrics, +objective value and feasibility, case counts, candidate and evaluation IDs, and +the final baseline/best summary. Resuming the VeRO session resumes the same W&B +run. The direct CLI equivalent is `--wandb-project`, with optional entity, name, +and mode flags. + +## Python API + +The same pipeline can be assembled from backend-neutral interfaces: + +```python +from pathlib import Path + +from vero.evaluation import ( + CommandBackend, + CommandBackendConfig, + EvaluationPlan, + EvaluationSet, + MetricSelector, + ObjectiveSpec, +) +from vero.optimization import ( + CommandCandidateProducer, + CommandCandidateProducerConfig, +) +from vero.runtime import create_local_optimization_session +from vero.sandbox import LocalSandbox + +backend = CommandBackend(CommandBackendConfig( + harness_root=str(Path("../my-evaluator").resolve()), + command=["python3", "evaluate.py", "{workspace}", "{report}"], +)) +producer = CommandCandidateProducer(CommandCandidateProducerConfig( + root=str(Path("../my-optimizer").resolve()), + command=["python3", "improve.py", "{workspace}"], +)) + +session = await create_local_optimization_session( + project_path="./my-program", + session_dir="~/.vero/sessions/my-run", + backend_id="command", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector(metric="latency_ms"), + direction="minimize", + ), + evaluation_plan=EvaluationPlan.single(EvaluationSet(name="performance")), + producers={"default": producer}, + max_proposals=5, +) +result = await session.run() +print(result.best.request.candidate.version, result.best.objective.value) + +# Every candidate remains available after its producer workspace is gone. +inspection_sandbox = await LocalSandbox.create() +for candidate in session.candidate_repository.list(): + async with session.candidate_repository.checkout( + candidate, + sandbox=inspection_sandbox, + name=f"inspect-{candidate.id}", + ) as candidate_workspace: + print(candidate.id, candidate_workspace.project_path) +``` + +`vero session inspect SESSION_DIR` includes the same durable candidate records +alongside the manifest and evaluation summaries. + +### Run the target in a remote sandbox + +The local factory above is a convenience wrapper. For containers, remote VMs, +or another execution environment, provision a `Workspace` in that sandbox and +pass it to the generic factory: + +```python +from vero.runtime import create_optimization_session +from vero.candidate_repository import GitCandidateRepository +from vero.sandbox import DockerSandbox +from vero.workspace import GitWorkspace + +sandbox = await DockerSandbox.create(image="gcc:14-bookworm") +try: + # The repository is copied into the container; it is not bind-mounted. + await sandbox.upload("./my-program", "/workspace/my-program") + workspace = await GitWorkspace.from_path( + sandbox, + "/workspace/my-program", + ) + session_dir = Path("~/.vero/sessions/remote-run").expanduser() + candidate_repository = await GitCandidateRepository.create( + session_dir / "candidates", + workspace=workspace, + ) + + backend = CommandBackend(CommandBackendConfig( + harness_root=str(Path("../my-evaluator").resolve()), + command=[ + "sh", + "{harness}/evaluate.sh", + "{workspace}", + "{report}", + "{artifacts}", + ], + )) + producer = CommandCandidateProducer(CommandCandidateProducerConfig( + root=str(Path("../my-optimizer").resolve()), + command=["sh", "{producer}/improve.sh", "{workspace}"], + )) + session = await create_optimization_session( + workspace=workspace, + candidate_repository=candidate_repository, + session_dir=session_dir, + backend_id="command", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector(metric="latency_ms"), + direction="minimize", + ), + producers={"default": producer}, + ) + result = await session.run() +finally: + await sandbox.close() +``` + +Session manifests, databases, budgets, W&B logging, artifacts, and the bare Git +candidate repository stay on the host. Candidate commands, compilation, and +evaluation run in isolated checkouts inside the sandbox. VeRO transfers Git +bundles between remote checkouts and the durable repository, then removes each +temporary checkout after use. + +Remote command harnesses and producer directories must be self-contained. An +executable named in a command must either be installed in the sandbox or live +under `{harness}` or `{producer}`. `ClaudeCodeAgent` requires a host-visible +workspace because its SDK takes a local `cwd`; `VeroAgent` and custom agents +whose tools operate through `Sandbox` can work without one. Incompatible agents +are rejected when the session is created. + +## Python benchmark tasks + +Python targets can use the optional `scale-vero-tasks` package instead of +writing the JSON command contract directly. It provides only task definition +and execution types; target programs do not depend on the VeRO optimizer. + +```python +# ../my-evaluator/benchmark.py +from vero_tasks import TaskOutput, TaskResult, create_task +from my_program import run_program + +task = create_task("quality") + +@task.inference() +async def run(case, context): + return TaskOutput(output=run_program(case["input"])) + +@task.evaluation() +async def score(case, output, context): + return TaskResult.from_task_output( + output, + score=float(output.output == case["expected"]), + ) +``` + +Connect it with `PythonTaskBackend`. Keep the task module and cases in a trusted +external uv project; VeRO overlays each isolated candidate with +`uv --with-editable` so the harness imports the exact program version being +measured without making evaluator code editable. + +```python +from vero.evaluation import ( + PythonTaskBackend, + PythonTaskBackendConfig, + PythonTaskEvaluationConfig, +) + +backend = PythonTaskBackend(PythonTaskBackendConfig( + harness_root=str(Path("../evaluation-state").resolve()), + module="benchmark", + task="quality", + evaluations=[ + PythonTaskEvaluationConfig( + name="train", + partition="train", + cases_path=str(Path("../train.jsonl").resolve()), + ), + PythonTaskEvaluationConfig( + name="validation", + partition="validation", + cases_path=str(Path("../validation.jsonl").resolve()), + ), + ], +)) +``` + +## Core concepts + +Production is a swappable `GenerationBackend`: the native in-process producer (a +coding agent whose harness runs on the host while its edits run in a sandbox), or +a Harbor run (the whole agent contained). Either way the orchestrator evaluates, +selects, and remembers — separately from the feedback the producer saw. + +```mermaid +flowchart TB + O["Optimizer loop
(strategy · selection · durable session)"] --> G{"GenerationBackend"} + G -->|native| N["VeroAgent harness (host)
shell / read_file / write_file"] + N --> SB["Sandbox
candidate checkout"] + G -->|contained| H["Harbor run
agent in a container"] + SB --> EV["Evaluator
budget · disclosure · scoring"] + H --> EV + EV --> O +``` + +| Concept | Meaning | +| --- | --- | +| `Candidate` | A target identity (a program, text, or agent) plus an opaque workspace version and lineage | +| `EvaluationSet` | A backend-owned collection or selection of evaluation cases | +| `EvaluationPlan` | Named evaluations, agent access, independent budgets, canonical selection, and optional hidden final evaluation | +| `EvaluationRecord` | The durable request, report, provenance, and objective result | +| `EvaluationBackend` | Measures a candidate without assuming its language or framework | +| `CandidateProducer` | Edits one isolated workspace to realize a proposed idea | +| `GenerationBackend` | Produces candidates plus their generation-time feedback for a proposal — the swappable unit that is the in-process native producer by default or a Harbor run | +| `OptimizationStrategy` | Chooses parents, ideas, and producers for the next batch (e.g. `SequentialStrategy`, or `EvolutionaryStrategy` for population/tournament search) | +| `SelectionPolicy` | Chooses the best feasible evaluation for the configured objective | +| `OptimizationSession` | Owns lifecycle, events, artifacts, budgets, and durable state | + +Coding agents receive a scoped `AgentContext`. They can edit only their supplied +workspace and call `evaluate(evaluation=..., selection=..., candidate_id=...)`. +The current workspace is saved as a candidate when `candidate_id` is omitted; +supplying an existing candidate ID re-evaluates that durable version. The tool +returns a compact receipt with the evaluation ID, status, approved summary, and +path to the filesystem result. Large case records, traces, and artifacts stay +out of the tool response. Intermediate checkpoints are real candidates and +remain eligible for selection, even if the agent later makes the program worse. + +Each producer workspace contains a generated, read-only `.evals/` directory: + +```text +.evals/ +├── README.md +├── manifest.json +├── plan.json # available evaluations, selections, disclosure, budgets +├── tasks/ # only backend-approved task resources +├── candidates/ # metadata, parent patches, and repository-native refs +└── results/ # authorized summaries or full case/trace/artifact trees +``` + +The agent inspects this with ordinary filesystem and Git commands. Disclosure +governs what lands there: + +- **full** — per-case traces split into separate files (and, for Harbor, the + complete downloaded trial record per case: failure results, tracebacks, trial + logs, target-agent artifacts); +- **aggregate** — metrics and counts only, no case records or trial artifacts; +- **none** — status only. + +Candidate history uses durable Git refs, so siblings need not be ancestors of the +current checkout: a proposal sees the candidates from the start of its generation +plus its own evaluated checkpoints, and parallel siblings appear the next round. + +The `.evals/` view is disposable and read-only — Git-excluded, permission- +protected, and rejected if a candidate force-adds it; the durable candidate and +evaluation stores stay the trusted source. In Harbor the sidecar owns the +writable volume and the agent mounts it read-only, and case resources (for +datasets, the complete pinned task directories) are exported only when the +evaluation's authorization permits it. + +Strategies can propose a batch of candidates and route each proposal to a named +producer. Set `max_concurrency` to produce and evaluate independent candidates in +parallel. This supports sequential hill climbing, evolutionary search, and +orchestrator/sub-agent designs without changing the evaluation model. + +## Durable sessions + +Session state is stored outside the target repository: + +```text +sessions// +├── manifest.json +├── database.json +├── budgets.json # when evaluation is metered +├── events.jsonl +├── artifacts/ +└── evaluations/ + └── / + ├── evaluation.json + ├── cases/ + └── artifacts/ +``` + +The evaluation directories are the source of truth; the database can be rebuilt +from them. Reusing a session directory resumes its compatible baseline, +candidate lineage, evaluation history, completed rounds, and supported coding +agent state. VeRO rejects a resume if its backend configuration, evaluation +plan, run protocol, parameters, limits, seed, objective, or baseline is +incompatible with the schema-v3 manifest. + +```bash +vero session list +vero session inspect ~/.vero/sessions/ +vero report ~/.vero/sessions/ --output experiment.html +vero session fork OLD_SESSION NEW_SESSION --max-proposals 20 --reset-budgets +vero session export ~/.vero/sessions/ +vero session clear ~/.vero/sessions/ --yes +``` + +`vero report` creates a self-contained, read-only HTML view of the whole run: +the score trajectory, candidate lineage and parent diffs, evaluation artifacts, +producer traces, and runtime event timeline. The report embeds those materials, +so treat the resulting file as sensitive experiment data. + +## Safety boundaries + +- Candidate changes happen in isolated Git worktrees; the original target is not + edited. +- The target must be clean so its baseline has an unambiguous version. +- Session state, evaluation harnesses, and external producers must live outside + the editable target repository. +- Command execution uses argument vectors and an explicit environment. +- Evaluation secrets are passed through backend configuration and redacted from + diagnostics; they cannot be embedded in evaluation parameters. +- Agent-visible cases, histories, and evaluation details are projected into a + generated `.evals/` view according to the same authorization boundary. +- Budgets are reserved atomically before backend execution. +- A built-in coding agent's shell and file actions execute inside a sandbox + bound to its candidate checkout, so containment is the sandbox boundary rather + than in-process checks. The local sandbox is for fast, trusted runs; for a + contained or untrusted producer run the agent through Harbor, which isolates + the agent (and, in separate-verifier mode, the scorer) in its own environment. + +## Paper and reproduction + +VeRO was introduced in [*VeRO: A Harness for Agents to Optimize +Agents*](https://arxiv.org/abs/2602.22480), accepted at ICML 2026. The paper +studies agent-harness optimization; the current library generalizes the same +version/evaluate/select loop to programs more broadly. + +The frozen code for reproducing the paper is preserved on the `paper/v1` branch +and at the `paper-v1` tag: + +```bash +git checkout paper-v1 +``` + +## Development + +```bash +uv sync --all-extras +uv run pytest tests/test_v05_*.py +``` + +VeRO is licensed under the MIT License. diff --git a/vero/docs/agent-setup-guide.md b/vero/docs/agent-setup-guide.md new file mode 100644 index 00000000..4f4561be --- /dev/null +++ b/vero/docs/agent-setup-guide.md @@ -0,0 +1,133 @@ +# VeRO setup guide (for coding agents) + +This guide helps a coding agent (Claude, Cursor, Copilot, …) set up and run a +VeRO optimization for a user. VeRO optimizes a **versioned program** against an +**evaluation**: a producer (usually a coding agent) proposes candidate edits, a +trusted evaluator scores them, and the best candidate is selected. + +You do two things: (1) author an **evaluation harness**, and (2) run the +**optimizer**. There is no `.veroaccess`, `Policy`, or resource/namespace setup — +those were removed. Containment now comes from the sandbox the producer runs in, +and disclosure is controlled per evaluation set. + +## Before you start + +Understand the user's program (what it does, how it's invoked, what "better" +means) and read a matching example: + +- `examples/c-matmul/` — a language-neutral **command harness** + `vero.toml`. +- `examples/circle-packing/` — a Python program scored by a command harness. +- `../vero-tasks/examples/matmul-{kernel,eval}/` — a **Python task** using the + `scale-vero-tasks` protocol. +- `examples/harbor-circle-packing/` — a **Harbor** outer loop (contained agent) + with a simple command inner loop. + +## 1. Author the evaluation + +Pick whichever fits the user's program. + +### Option A — command harness (any language) + +Write a script that reads a request JSON and writes a report JSON. VeRO's +`CommandBackend` invokes it with placeholders it substitutes at run time: + +```bash +python evaluate.py --workspace {workspace} --request {request} \ + --report {report} --artifacts {artifacts} +``` + +- `{workspace}` — the candidate checkout to score. +- `{request}` — JSON: `{"schema_version": 1, "request": {"seed": , ...}}`. +- `{report}` — write JSON: `{"schema_version": 1, "status": "success", + "metrics": {"": , ...}}` (a **dict of metrics** — one becomes the + objective; others can be constraints or diagnostics). + +See `examples/circle-packing/harness/` for a complete scorer. + +### Option B — Python task (`scale-vero-tasks`) + +For Python benchmarks, use the task protocol (`../vero-tasks`): + +```python +from vero_tasks import TaskContext, TaskOutput, TaskResult, create_task + +task = create_task("main", required_env_vars=["OPENAI_API_KEY"]) + +@task.inference() +async def infer(case: dict, context: TaskContext) -> TaskOutput: + from my_agent import run_agent + return TaskOutput(output=await run_agent(case["question"])) + +@task.evaluation() +async def evaluate(case: dict, output: TaskOutput, context: TaskContext) -> TaskResult: + return TaskResult.from_task_output( + output, score=float(output.output == case["answer"]) + ) +``` + +The `scale-vero-tasks` runner adapts this to the same command-harness contract. +See `../vero-tasks/README.md` and `../vero-tasks/examples/matmul-eval/`. + +**Rules:** functions are `async`; let exceptions propagate (VeRO records them as +errored cases); keep heavy imports inside functions; declare API keys via +`required_env_vars`; the score is a float (define whether higher or lower is +better via the objective's `direction`). + +## 2. Run the optimizer + +The target must be a git repo whose working tree is clean. + +### Config-driven + +```bash +vero init ./run # scaffolds run/vero.toml (+ target/, harness/) +# edit run/vero.toml +vero check --config run/vero.toml # validate paths, git, eval refs +vero evaluate --config run/vero.toml # score the baseline only +vero run --config run/vero.toml # optimize +``` + +`vero.toml` sections: `[target]` (root, ref), `[backend]` (kind = `command`, +`harness_root`, `command`), one or more `[[evaluations]]` (name, partition, +`agent_can_evaluate`, `disclosure`, optional `agent_budget`), `[protocol]` +(`selection_evaluation`, `final_evaluation`, `max_proposals`), `[objective]` +(metric, direction), and `[optimizer]` (`kind = "vero"`, optional `model`, +`instruction`). + +### One-shot (flags) + +```bash +vero optimize ./target \ + --harness-root ./harness \ + --evaluate "python3 {harness}/evaluate.py --workspace {workspace} --request {request} --report {report} --artifacts {artifacts}" \ + --agent vero \ + --instruction "Improve the program without changing its intended behavior." \ + --metric score --direction maximize \ + --evaluation-set default --max-proposals 4 --max-turns 60 +``` + +## 3. The optimizer agent (`--agent vero`) + +`VeroAgent` runs on the OpenAI Agents SDK. Its harness runs on the host; its +shell/file effects run inside a sandbox (bound to the candidate checkout) via +function tools (`shell`, `read_file`, `write_file`), plus an `evaluate` tool for +mid-run self-scoring. There are no custom bash/grep/git tools to configure and no +in-process ACLs — the sandbox is the boundary. + +- **Model:** any provider via LiteLLM. Set `VERO_OPTIMIZER_MODEL` + (e.g. `openai/gpt-5.4`, `anthropic/claude-sonnet-4-5-...`) or `[optimizer] model`. +- **Containment:** the sandbox client is the seam — local for fast/trusted runs; + for a contained/untrusted agent, run it through **Harbor** (see + `examples/harbor-circle-packing/`). +- **Strategy:** the default proposes one candidate per round; for + population/evolutionary search use `EvolutionaryStrategy` + (`vero.optimization.EvolutionaryStrategy`). + +## Don't + +- Don't let the harness/task code be edited during optimization — it scores the + candidate and must stay fixed (keep it in `--harness-root`, outside the target). +- Don't hardcode absolute paths in harness/task code; use the `{...}` placeholders + and request parameters. +- Don't suppress exceptions in inference/evaluation — VeRO records them. +- Don't put secrets in code; declare them as env vars. diff --git a/vero/docs/harbor-architecture.md b/vero/docs/harbor-architecture.md new file mode 100644 index 00000000..6f2d569f --- /dev/null +++ b/vero/docs/harbor-architecture.md @@ -0,0 +1,196 @@ +# Harbor benchmark architecture + +**Audience:** an engineer picking this up cold, or reviewing the PRs. +**Scope:** the *harbor* deployment path — how VeRO turns an untrusted optimizer +agent into a trustworthy benchmark number. (The in-process +`OptimizationSession`/`optimizer.py` path is a separate, simpler runner and is +not used by the harbor deployment.) + +## The core idea + +VeRO benchmarks an **optimizer** (an LLM coding agent such as codex/claude) at +*improving a candidate program*. The optimizer edits code and asks a **trusted +evaluator** to score it; the evaluator holds the held-out answers and never lets +the optimizer see them. The whole system exists to get a *trustworthy* number +out of an *untrusted* agent editing code against data it must not peek at. + +Everything below hangs off one organizing idea: the **trust boundary** between +the optimizer and the evaluator. + +## Topology: three containers + +A run is a Docker Compose stack of three services (on Modal, or local docker): + +``` + UNTRUSTED │ TRUSTED + │ + ┌─────────────┐ │ ┌────────────────────┐ ┌──────────────────┐ + │ main │──┼─▶│ eval-sidecar │ │ inference-gateway│ + │ (optimizer) │ │ │ (FastAPI, root) │ │ (metered LLM │ + │ edits │ │ │ held-out data + │ │ proxy) │ + │ /work/agent │ │ │ scores + budgets │ │ │ + └─────┬───────┘ │ └─────────┬──────────┘ └────────┬─────────┘ + │ │ │ runs inner `harbor run` │ + │ LLM │ │ as unprivileged `harness`│ + └──────────┼────────────┴──── scoped token ───────┘ + │ +``` + +- **`main`** — runs the optimizer agent. *Untrusted*: it may write anything into + `/work/agent` (the candidate repo). It reaches the sidecar only over HTTP and + the LLM only through the gateway. Real secrets are scrubbed to `""` here. +- **`eval-sidecar`** — the *trusted control plane*. Owns the held-out task + lists, the scores database, the budget ledger, and the admin token. Serves + agent-facing endpoints (budgeted, disclosure-gated) and token-gated admin + endpoints. +- **`inference-gateway`** — a credential-isolating, budgeted proxy in front of + the real LLM upstream. Issues *scoped tokens* (producer / evaluation / + finalization), each with its own model allow-list and budget, so the optimizer + never sees the raw key and its LLM usage is metered. + +Named volumes carry state between services: `agent_repo`, `agent_context`, +`admin_state`, `token_state`, `inference_state`. + +## A run, end to end + +1. **`vero harbor run --config gaia/build.yaml --agent codex --model gpt-5.5`.** +2. **Compile** (`harbor/build/compiler.py`): the `build.yaml` + the target repo + + partitions → a self-contained task dir — three Dockerfiles, + `docker-compose.yaml`, `serve.json` (the sidecar's deployment config), the + case lists, `instruction.md` (what the agent reads), and `test.sh` (the + verifier phase). +3. It shells out to **inner `harbor run -e modal`**, which builds the images, + brings up the stack, and injects the optimizer into `main`. +4. **Optimizer loop**: the agent reads its `.evals` context, edits the candidate + in `/work/agent`, and calls the sidecar `/eval` to score candidates — on + **development** (full per-case feedback) and **validation** (aggregate score + only). It iterates, and may `/submit` a final pick. +5. **Finalization** (`test.sh`, admin-token'd): `verifier.finalize()` selects a + candidate, admin-re-scores it on the held-out **test** target, writes + `reward.json` + `finalization.json`; `export-session` archives the whole + session. +6. Harbor collects `/logs` back to **`jobs///`** on disk and + reports the reward. + +## The evaluation core + +- **`EvaluationEngine`** (`evaluation/engine.py`) runs every evaluation through a + **backend** and records an **`EvaluationRecord`** (`evaluation/models.py`): the + request (candidate, evaluation_set, limits) + a report (per-case results, + metrics, diagnostics, artifacts) + an objective value + a **principal**. It + enforces the **`BudgetLedger`** (`evaluation/store/budget.py`) and fires **listeners** + (where the W&B sink hooks in). +- **`HarborBackend`** (`harbor/backend.py`) is how a candidate is actually + scored: it runs a *nested* `harbor run` over the tasks (`uv run` the + candidate's package, execute the agent per task, collect trial records), + aggregating attempts (`mean` by default, zero-filling dead attempts and + recording `n_dead_infra`/`n_clean` so an outage-diluted mean is distinguishable + from a clean low one). +- **Principals** are the key access concept: + - **`agent`** — the optimizer's search evals: metered, disclosure-limited. + - **`admin`** — trusted finalization / re-scores: unmetered, full disclosure. + This is the authoritative number. + +## Disclosure & the anti-cheating contract + +Each partition has a **disclosure level** (`sidecar/sidecar.py`): +- **development → FULL**: per-case results shown (the disclosed set to iterate + on). +- **validation → AGGREGATE**: only an *average over ≥ k cases* is returned (the + k-anonymity floor, default 5), so individual held-out labels can't be read off + one case at a time. +- **test → held-out**: not agent-evaluable; only admin scores it, at + finalization. + +## Selection / finalization (what ships) + +`CanonicalVerifier.finalize()` (`sidecar/verifier.py`) decides what to ship and +computes the reward: +- **Chain**: the agent's explicit `submission` → else `auto_best` (rank + candidates by *coverage-qualified* validation evals, admin-re-score the top-K + on the full set) → else **pick-last** (the current candidate). It ships nothing + only if there is no candidate at all. +- **`baseline_floor`** (opt-in, off by default): ship only if the candidate + beats the seed. Because the seed is fixed, its score can be **pinned** + (`baseline_reward` / `baseline_selection_score`) — or produced reproducibly via + `vero harbor score-baseline --replicates N`. +- Result → `VerificationResult` (shipped, rewards, baseline_rewards) → + `reward.json` (what harbor consumes) + the durable session archive. + +## The security model (five mechanisms) + +1. **Credential isolation** — the gateway's scoped tokens; the raw upstream key + lives only in the sidecar. (`gateway/inference.py`, compose template.) +2. **Harness isolation** — the inner `harbor run` (arbitrary candidate code) runs + as unprivileged `harness`; the session dir is `0700 root`, case lists `700`, + `serve.json` `600` — so the candidate cannot read the answer key it is graded + on. (`harbor/backend.py` `run_as`, `Dockerfile.sidecar.j2`, + `harbor/deployment.py`.) +3. **k-anonymity floor** — aggregate subset evals must cover ≥ 5 cases. + (`sidecar/sidecar.py` `_enforce_aggregate_floor`.) +4. **Budget metering** — per-scope gateway budgets + a case/run ledger, with a + *reserved finalization scope* so the mandatory admin re-score can't be starved + by the agent's search. (`evaluation/store/budget.py`, gateway config.) +5. **Fail-safe finalization** — never ship an unverified candidate on an infra + blip; a session export never fails wholesale. (`sidecar/verifier.py`, + `sidecar/session.py`.) + +## The build pipeline + +`build.yaml` → `load_harbor_build_config` (`harbor/build/loader.py`) → +`HarborBuildConfig` (`harbor/build/config.py`, composed from the leaf models in +`harbor/build/specs.py`) → `compile_harbor_task` (`harbor/build/compiler.py`) +renders the whole deployable +task (Dockerfiles + compose + `serve.json` + cases + instructions). `serve.json` +deserializes into `HarborDeploymentConfig` (`harbor/deployment.py`), which +`build_harbor_components` turns into the live sidecar + verifier. So a benchmark +is *fully specified by one YAML* (see `harness-engineering-bench/gaia/baseline/`). + +## Observability + +`SidecarWandbSink` (`runtime/wandb.py`) subscribes to the engine's listeners and +streams every evaluation to Weights & Biases from the trusted side: scoped +metrics per `partition/principal`, `num_cases`, remaining budget, optional full +trace artifacts, and a run summary at finalize. Each `vero harbor run` gets its +own W&B run. + +--- + +## Suggested review order + +Read outside-in — from *what a benchmark declares* down to *how it runs and stays +honest*. Paths are under `vero/src/vero/` unless noted. + +1. **What a benchmark is** — `harness-engineering-bench/gaia/baseline/build.yaml` + and `harbor/build/specs.py` (`AgentAccessSpec`, `VerificationTargetSpec`) plus + `harbor/build/config.py` (`HarborBuildConfig` and the field groups it is + composed from). This is the declarative surface; everything else serves it. +2. **Compile → deployable task** — `harbor/build/compiler.py` + (`compile_harbor_task`) and the templates in `harbor/build/templates/` + (`docker-compose.yaml.j2`, `Dockerfile.sidecar.j2`, `instruction.md.j2`, + `test.sh.j2`). See how one YAML becomes the three-container stack + `serve.json`. +3. **The topology at runtime** — `harbor/deployment.py` + (`HarborDeploymentConfig`, `build_harbor_components`): how `serve.json` becomes + a live engine + sidecar + verifier. +4. **The evaluation core** — `evaluation/models.py` (`EvaluationRecord`, + `EvaluationPrincipal`, `DisclosureLevel`, `EvaluationSet`), then + `evaluation/engine.py` and `evaluation/store/budget.py`. This is the vocabulary the + rest of the system speaks. +5. **How a candidate is scored** — `harbor/backend.py` (`HarborBackend`): the + nested `harbor run`, attempt aggregation, and the infra-vs-candidate taxonomy. + The most intricate file; skim `_case_result`, `_command`, `_environment`. +6. **The sidecar** — `sidecar/sidecar.py` (access policies, disclosure floor, + tracked eval jobs, submission) and `sidecar/app.py` (the HTTP surface: agent + `/eval` vs admin `/finalize` `/score/baseline` `/session/export`). +7. **Selection & finalization** — `sidecar/verifier.py` (`CanonicalVerifier`): the + submit → auto_best → pick-last chain, coverage-qualified selection, the + baseline floor + pinning, `measure_baseline`. This decides the shipped number. +8. **Security specifics** — the five mechanisms: `run_as`/`harness_user` in + `harbor/backend.py` + `Dockerfile.sidecar.j2`; the gateway in + `gateway/inference.py`; the session archive in `sidecar/session.py`. +9. **Observability** — `runtime/wandb.py` (`SidecarWandbSink`). +10. **The CLI glue** — `harbor/cli.py` (`vero harbor run`, `finalize`, + `export-session`, `score-baseline`). + +Tests mirror this order (`tests/test_v05_harbor_*.py`) and are a good +executable spec for each layer. diff --git a/vero/examples/c-matmul/.gitignore b/vero/examples/c-matmul/.gitignore new file mode 100644 index 00000000..6af2388b --- /dev/null +++ b/vero/examples/c-matmul/.gitignore @@ -0,0 +1,2 @@ +.vero/ +.evals/ diff --git a/vero/examples/c-matmul/README.md b/vero/examples/c-matmul/README.md new file mode 100644 index 00000000..fb620abb --- /dev/null +++ b/vero/examples/c-matmul/README.md @@ -0,0 +1,24 @@ +# Generic C program optimization + +This example is the concrete proof that VeRO optimizes programs rather than +only Python agents. The editable target is a C repository with no Python +package or VeRO dependency. A trusted harness outside the target compiles it, +checks numerical correctness, and reports latency. + +Initialize the target as its own versioned program, then evaluate and optimize +it through the checked-in `vero.toml`: + +```bash +cd examples/c-matmul/target +git init -b main +git add . +git -c user.name=vero -c user.email=vero@localhost commit -m baseline +cd .. + +vero evaluate --config vero.toml +vero run --config vero.toml +``` + +The deterministic producer makes the example suitable for CI and requires no +model credentials. Replace `[optimizer]` with `kind = "vero"` or +`kind = "claude"` plus an `instruction` to use a built-in coding agent. diff --git a/vero/examples/c-matmul/harness/benchmark.c b/vero/examples/c-matmul/harness/benchmark.c new file mode 100644 index 00000000..0de850dc --- /dev/null +++ b/vero/examples/c-matmul/harness/benchmark.c @@ -0,0 +1,69 @@ +#define _POSIX_C_SOURCE 200809L + +#include "matmul.h" + +#include +#include +#include +#include + +static double elapsed_ms(struct timespec start, struct timespec end) { + return (double)(end.tv_sec - start.tv_sec) * 1000.0 + + (double)(end.tv_nsec - start.tv_nsec) / 1000000.0; +} + +static void reference_matmul( + const double *a, + const double *b, + double *c, + size_t n +) { + for (size_t i = 0; i < n; ++i) { + for (size_t j = 0; j < n; ++j) { + double sum = 0.0; + for (size_t k = 0; k < n; ++k) { + sum += a[i * n + k] * b[k * n + j]; + } + c[i * n + j] = sum; + } + } +} + +int main(void) { + const size_t n = 128; + const size_t elements = n * n; + double *a = malloc(elements * sizeof(double)); + double *b = malloc(elements * sizeof(double)); + double *actual = calloc(elements, sizeof(double)); + double *expected = calloc(elements, sizeof(double)); + if (a == NULL || b == NULL || actual == NULL || expected == NULL) { + return 2; + } + + for (size_t index = 0; index < elements; ++index) { + a[index] = (double)((int)(index % 13) - 6) / 7.0; + b[index] = (double)((int)(index % 11) - 5) / 5.0; + } + + struct timespec start; + struct timespec end; + clock_gettime(CLOCK_MONOTONIC, &start); + matmul(a, b, actual, n); + clock_gettime(CLOCK_MONOTONIC, &end); + reference_matmul(a, b, expected, n); + + int correct = 1; + for (size_t index = 0; index < elements; ++index) { + if (fabs(actual[index] - expected[index]) > 1e-9) { + correct = 0; + break; + } + } + + printf("%d %.9f\n", correct, elapsed_ms(start, end)); + free(a); + free(b); + free(actual); + free(expected); + return correct ? 0 : 3; +} diff --git a/vero/examples/c-matmul/harness/evaluate.py b/vero/examples/c-matmul/harness/evaluate.py new file mode 100644 index 00000000..6a99254a --- /dev/null +++ b/vero/examples/c-matmul/harness/evaluate.py @@ -0,0 +1,111 @@ +"""Trusted C build, correctness, and performance harness.""" + +from __future__ import annotations + +import argparse +import json +import subprocess +from pathlib import Path + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--workspace", type=Path, required=True) + parser.add_argument("--request", type=Path, required=True) + parser.add_argument("--report", type=Path, required=True) + parser.add_argument("--artifacts", type=Path, required=True) + args = parser.parse_args() + + request = json.loads(args.request.read_text(encoding="utf-8")) + if request.get("schema_version") != 1: + raise ValueError("unsupported command input schema") + + artifact_dir = args.artifacts / "c-matmul" + artifact_dir.mkdir(parents=True, exist_ok=True) + binary = artifact_dir / "benchmark" + benchmark_source = Path(__file__).with_name("benchmark.c") + compile_result = subprocess.run( + [ + "cc", + "-O2", + "-std=c11", + "-I", + str(args.workspace), + str(args.workspace / "matmul.c"), + str(benchmark_source), + "-lm", + "-o", + str(binary), + ], + capture_output=True, + text=True, + ) + (artifact_dir / "compiler.log").write_text( + compile_result.stdout + compile_result.stderr, + encoding="utf-8", + ) + if compile_result.returncode != 0: + args.report.write_text( + json.dumps( + { + "schema_version": 1, + "status": "failed", + "diagnostics": [ + { + "code": "compile_failed", + "message": "C candidate did not compile", + "severity": "error", + "phase": "compile", + } + ], + "artifacts": [ + { + "path": "c-matmul/compiler.log", + "media_type": "text/plain", + } + ], + } + ), + encoding="utf-8", + ) + return + + measurements: list[float] = [] + correct = True + for _ in range(3): + run = subprocess.run([str(binary)], capture_output=True, text=True) + if run.returncode != 0: + correct = False + break + correct_value, latency_value = run.stdout.strip().split() + correct = correct and correct_value == "1" + measurements.append(float(latency_value)) + + latency_ms = min(measurements) if measurements else 1.0e12 + args.report.write_text( + json.dumps( + { + "schema_version": 1, + "status": "success", + "metrics": { + "latency_ms": latency_ms, + "correct": 1.0 if correct else 0.0, + }, + "artifacts": [ + { + "path": "c-matmul/compiler.log", + "media_type": "text/plain", + }, + { + "path": "c-matmul/benchmark", + "media_type": "application/octet-stream", + }, + ], + } + ), + encoding="utf-8", + ) + + +if __name__ == "__main__": + main() diff --git a/vero/examples/c-matmul/optimizer/optimize.py b/vero/examples/c-matmul/optimizer/optimize.py new file mode 100644 index 00000000..6d941a6b --- /dev/null +++ b/vero/examples/c-matmul/optimizer/optimize.py @@ -0,0 +1,18 @@ +"""Deterministic candidate producer used by CI.""" + +from __future__ import annotations + +import argparse +import shutil +from pathlib import Path + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--workspace", type=Path, required=True) + args = parser.parse_args() + shutil.copy2(Path(__file__).with_name("optimized.c"), args.workspace / "matmul.c") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/c-matmul/optimizer/optimized.c b/vero/examples/c-matmul/optimizer/optimized.c new file mode 100644 index 00000000..e93806df --- /dev/null +++ b/vero/examples/c-matmul/optimizer/optimized.c @@ -0,0 +1,16 @@ +#include "matmul.h" + +#include + +/* Cache-friendly loop order with no redundant scalar delay. */ +void matmul(const double *a, const double *b, double *c, size_t n) { + memset(c, 0, n * n * sizeof(double)); + for (size_t i = 0; i < n; ++i) { + for (size_t k = 0; k < n; ++k) { + const double a_ik = a[i * n + k]; + for (size_t j = 0; j < n; ++j) { + c[i * n + j] += a_ik * b[k * n + j]; + } + } + } +} diff --git a/vero/examples/c-matmul/target/.gitignore b/vero/examples/c-matmul/target/.gitignore new file mode 100644 index 00000000..54985fc2 --- /dev/null +++ b/vero/examples/c-matmul/target/.gitignore @@ -0,0 +1,3 @@ +.DS_Store +*.o +benchmark diff --git a/vero/examples/c-matmul/target/matmul.c b/vero/examples/c-matmul/target/matmul.c new file mode 100644 index 00000000..aac353db --- /dev/null +++ b/vero/examples/c-matmul/target/matmul.c @@ -0,0 +1,16 @@ +#include "matmul.h" + +/* Correct but deliberately slow baseline. */ +void matmul(const double *a, const double *b, double *c, size_t n) { + for (size_t i = 0; i < n; ++i) { + for (size_t j = 0; j < n; ++j) { + double sum = 0.0; + for (size_t k = 0; k < n; ++k) { + sum += a[i * n + k] * b[k * n + j]; + } + c[i * n + j] = sum; + for (volatile size_t delay = 0; delay < 500; ++delay) { + } + } + } +} diff --git a/vero/examples/c-matmul/target/matmul.h b/vero/examples/c-matmul/target/matmul.h new file mode 100644 index 00000000..22cb7c08 --- /dev/null +++ b/vero/examples/c-matmul/target/matmul.h @@ -0,0 +1,8 @@ +#ifndef VERO_EXAMPLE_MATMUL_H +#define VERO_EXAMPLE_MATMUL_H + +#include + +void matmul(const double *a, const double *b, double *c, size_t n); + +#endif diff --git a/vero/examples/c-matmul/vero.toml b/vero/examples/c-matmul/vero.toml new file mode 100644 index 00000000..75d36550 --- /dev/null +++ b/vero/examples/c-matmul/vero.toml @@ -0,0 +1,50 @@ +[target] +root = "./target" +ref = "HEAD" + +[backend] +id = "command" +kind = "command" +harness_root = "./harness" +command = [ + "python3", + "evaluate.py", + "--workspace", "{workspace}", + "--request", "{request}", + "--report", "{report}", + "--artifacts", "{artifacts}", +] + +[[evaluations]] +name = "performance" +agent_can_evaluate = true +agent_visible = true +agent_selection = "arbitrary" +disclosure = "full" + +[protocol] +selection_evaluation = "performance" +timeout_seconds = 120 +max_proposals = 1 + +[protocol.retry] +max_attempts = 1 + +[objective] +metric = "latency_ms" +direction = "minimize" + +[[objective.constraints]] +metric = "correct" +operator = "==" +value = 1.0 + +[optimizer] +kind = "command" +root = "./optimizer" +command = ["python3", "optimize.py", "--workspace", "{workspace}"] +description = "Use a cache-friendly matrix kernel" + +[session] +id = "c-matmul-example" +directory = "./.vero/session" diff --git a/vero/examples/circle-packing/.gitignore b/vero/examples/circle-packing/.gitignore new file mode 100644 index 00000000..6af2388b --- /dev/null +++ b/vero/examples/circle-packing/.gitignore @@ -0,0 +1,2 @@ +.vero/ +.evals/ diff --git a/vero/examples/circle-packing/README.md b/vero/examples/circle-packing/README.md new file mode 100644 index 00000000..9530908b --- /dev/null +++ b/vero/examples/circle-packing/README.md @@ -0,0 +1,53 @@ +# Circle packing program optimization + +This is a non-trivial VeRO benchmark adapted from +[ShinkaEvolve's circle-packing example](https://github.com/SakanaAI/ShinkaEvolve/tree/main/examples/circle_packing). +The editable program places 26 circles in a unit square and maximizes the sum +of their radii. A trusted external harness checks the exact geometry and emits +the score, detailed validation measurements, a JSON layout, and an SVG rendering. + +The baseline is deliberately simple. The search space includes better initial +layouts, numerical optimization of radii and centers, stochastic global search, +local refinement, restarts, and hybrid algorithms. Unlike ShinkaEvolve's marked +code block, VeRO gives the coding agent an ordinary versioned repository and +allows it to change the complete implementation. + +## Run it + +The candidate must first be initialized as its own Git repository: + +```bash +cd examples/circle-packing/target +git init -b main +git add . +git -c user.name=vero -c user.email=vero@localhost commit -m baseline +cd .. + +vero check --config vero.toml +vero evaluate --config vero.toml +vero run --config vero.toml +``` + +`vero evaluate` is credential-free and records the baseline score. `vero run` +uses the built-in VeRO coding agent with the configured LiteLLM model identifier +and permits up to 30 agent-requested development evaluations in one proposal. +Change `optimizer.model` to any model available through your provider. The best +nominated candidate is then re-evaluated through the hidden final evaluation. +Candidate versions remain in the session candidate repository, while evaluation +artifacts are available in `.vero/session/evaluations/` and to the agent through +its read-only `.evals` context. + +Evaluation launches through the candidate's locked uv environment. The seed +program has no third-party dependencies; an optimizer can add reproducible +scientific-computing dependencies with `uv add`. The evaluator applies the +configured protocol seed to Python and NumPy RNGs and fixes `PYTHONHASHSEED` so +stochastic candidates are reproducible when they use those standard sources. +The same value is available to candidates as `VERO_EVALUATION_SEED`. + +## Attribution + +The starting algorithm is adapted from ShinkaEvolve, Copyright 2025 Sakana AI, +under the Apache License 2.0. The original used NumPy and restricted evolution +to a marked block; this version uses the standard library and is structured as +a complete VeRO target repository. See [UPSTREAM_LICENSE](UPSTREAM_LICENSE) for +the upstream license. diff --git a/vero/examples/circle-packing/UPSTREAM_LICENSE b/vero/examples/circle-packing/UPSTREAM_LICENSE new file mode 100644 index 00000000..e3a6e3db --- /dev/null +++ b/vero/examples/circle-packing/UPSTREAM_LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2025 Sakana AI + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/vero/examples/circle-packing/harness/evaluate.py b/vero/examples/circle-packing/harness/evaluate.py new file mode 100644 index 00000000..a80f94b0 --- /dev/null +++ b/vero/examples/circle-packing/harness/evaluate.py @@ -0,0 +1,315 @@ +"""Trusted validator and scorer for the 26-circle packing benchmark.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import math +import os +import random +import sys +import time +import traceback +from dataclasses import asdict, dataclass +from pathlib import Path +from types import ModuleType +from typing import Any, Sequence + + +@dataclass(frozen=True) +class Validation: + valid: bool + message: str + computed_sum: float + reported_sum: float + sum_error: float + minimum_boundary_clearance: float + minimum_pair_clearance: float + + +def _load_candidate(path: Path) -> ModuleType: + spec = importlib.util.spec_from_file_location("candidate_packing", path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load candidate program: {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _seed_runtime(seed: int | None) -> None: + """Seed common in-process RNGs before calling the candidate entry point.""" + + if seed is None: + return + os.environ["VERO_EVALUATION_SEED"] = str(seed) + random.seed(seed) + numpy = sys.modules.get("numpy") + numpy_random = getattr(numpy, "random", None) + numpy_seed = getattr(numpy_random, "seed", None) + if callable(numpy_seed): + numpy_seed(seed) + + +def _sequence(value: Any, *, name: str) -> list[Any]: + if isinstance(value, (str, bytes)): + raise TypeError(f"{name} must be a sequence, not text") + try: + return list(value) + except TypeError as error: + raise TypeError(f"{name} must be a sequence") from error + + +def _normalize_output( + output: Any, +) -> tuple[list[list[float]], list[float], float]: + values = _sequence(output, name="run_packing output") + if len(values) != 3: + raise ValueError("run_packing must return (centers, radii, reported_sum)") + + raw_centers = _sequence(values[0], name="centers") + raw_radii = _sequence(values[1], name="radii") + centers: list[list[float]] = [] + for index, center in enumerate(raw_centers): + coordinates = _sequence(center, name=f"centers[{index}]") + if len(coordinates) != 2: + raise ValueError(f"centers[{index}] must have exactly two coordinates") + centers.append([float(coordinates[0]), float(coordinates[1])]) + radii = [float(radius) for radius in raw_radii] + return centers, radii, float(values[2]) + + +def validate_packing( + centers: Sequence[Sequence[float]], + radii: Sequence[float], + reported_sum: float, +) -> Validation: + if len(centers) != 26: + raise ValueError(f"expected 26 centers, received {len(centers)}") + if len(radii) != 26: + raise ValueError(f"expected 26 radii, received {len(radii)}") + + flattened = [coordinate for center in centers for coordinate in center] + if not all(math.isfinite(value) for value in [*flattened, *radii, reported_sum]): + raise ValueError("centers, radii, and reported_sum must all be finite") + if any(radius < 0.0 for radius in radii): + raise ValueError("radii must be non-negative") + + computed_sum = math.fsum(radii) + sum_error = abs(computed_sum - reported_sum) + sum_matches = math.isclose(computed_sum, reported_sum, rel_tol=1e-9, abs_tol=1e-12) + boundary_clearances = [ + min(x - radius, y - radius, 1.0 - x - radius, 1.0 - y - radius) + for (x, y), radius in zip(centers, radii, strict=True) + ] + pair_clearances = [ + math.dist(centers[left], centers[right]) - radii[left] - radii[right] + for left in range(26) + for right in range(left + 1, 26) + ] + minimum_boundary_clearance = min(boundary_clearances) + minimum_pair_clearance = min(pair_clearances) + + failures: list[str] = [] + if not sum_matches: + failures.append( + f"reported sum differs from the computed sum by {sum_error:.6g}" + ) + if minimum_boundary_clearance < 0.0: + failures.append( + "at least one circle crosses the square boundary " + f"by {-minimum_boundary_clearance:.6g}" + ) + if minimum_pair_clearance < 0.0: + failures.append( + f"at least two circles overlap by {-minimum_pair_clearance:.6g}" + ) + + return Validation( + valid=not failures, + message="; ".join(failures) if failures else "packing is geometrically valid", + computed_sum=computed_sum, + reported_sum=reported_sum, + sum_error=sum_error, + minimum_boundary_clearance=minimum_boundary_clearance, + minimum_pair_clearance=minimum_pair_clearance, + ) + + +def _write_layout_json( + path: Path, + centers: Sequence[Sequence[float]], + radii: Sequence[float], + validation: Validation, +) -> None: + path.write_text( + json.dumps( + { + "schema_version": 1, + "circles": [ + {"index": index, "x": x, "y": y, "radius": radius} + for index, ((x, y), radius) in enumerate( + zip(centers, radii, strict=True) + ) + ], + "validation": asdict(validation), + }, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def _write_layout_svg( + path: Path, + centers: Sequence[Sequence[float]], + radii: Sequence[float], + validation: Validation, +) -> None: + size = 800 + circles = "\n".join( + ( + f' ' + ) + for index, ((x, y), radius) in enumerate(zip(centers, radii, strict=True)) + ) + label = ( + f"sum={validation.computed_sum:.12f}; " + f"valid={'yes' if validation.valid else 'no'}" + ) + path.write_text( + "\n".join( + [ + '', + ' ', + circles, + f' {label}', + "", + "", + ] + ), + encoding="utf-8", + ) + + +def _artifact(path: str, media_type: str, description: str) -> dict[str, str]: + return {"path": path, "media_type": media_type, "description": description} + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--workspace", type=Path, required=True) + parser.add_argument("--request", type=Path, required=True) + parser.add_argument("--report", type=Path, required=True) + parser.add_argument("--artifacts", type=Path, required=True) + args = parser.parse_args() + + command_input = json.loads(args.request.read_text(encoding="utf-8")) + if command_input.get("schema_version") != 1: + raise ValueError("unsupported command input schema") + request = command_input.get("request") + if not isinstance(request, dict): + raise ValueError("command input must contain an evaluation request") + seed = request.get("seed") + if seed is not None and not isinstance(seed, int): + raise ValueError("evaluation seed must be an integer or null") + + artifact_dir = args.artifacts / "circle-packing" + artifact_dir.mkdir(parents=True, exist_ok=True) + started = time.perf_counter() + try: + _seed_runtime(seed) + candidate = _load_candidate(args.workspace / "packing.py") + run_packing = getattr(candidate, "run_packing", None) + if not callable(run_packing): + raise AttributeError("packing.py must define a callable run_packing()") + # Reset after import-time work and seed NumPy if the candidate imported it. + _seed_runtime(seed) + centers, radii, reported_sum = _normalize_output(run_packing()) + runtime_ms = (time.perf_counter() - started) * 1000.0 + validation = validate_packing(centers, radii, reported_sum) + + _write_layout_json( + artifact_dir / "layout.json", centers, radii, validation + ) + _write_layout_svg(artifact_dir / "layout.svg", centers, radii, validation) + (artifact_dir / "validation.json").write_text( + json.dumps(asdict(validation), indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + artifacts = [ + _artifact( + "circle-packing/layout.json", + "application/json", + "Circle centers, radii, and validation measurements", + ), + _artifact( + "circle-packing/layout.svg", + "image/svg+xml", + "Rendered circle packing", + ), + _artifact( + "circle-packing/validation.json", + "application/json", + "Geometric feasibility diagnostics", + ), + ] + report: dict[str, Any] = { + "schema_version": 1, + "status": "success", + "metrics": { + "sum_radii": validation.computed_sum, + "valid": 1.0 if validation.valid else 0.0, + "minimum_boundary_clearance": validation.minimum_boundary_clearance, + "minimum_pair_clearance": validation.minimum_pair_clearance, + "runtime_ms": runtime_ms, + }, + "artifacts": artifacts, + } + if not validation.valid: + report["diagnostics"] = [ + { + "code": "invalid_packing", + "message": validation.message, + "severity": "warning", + "phase": "validation", + } + ] + except BaseException as error: + runtime_ms = (time.perf_counter() - started) * 1000.0 + (artifact_dir / "failure.log").write_text( + "".join(traceback.format_exception(error)), + encoding="utf-8", + ) + report = { + "schema_version": 1, + "status": "failed", + "metrics": {"runtime_ms": runtime_ms}, + "diagnostics": [ + { + "code": "candidate_execution_failed", + "message": f"{type(error).__name__}: {error}", + "severity": "error", + "phase": "candidate", + } + ], + "artifacts": [ + _artifact( + "circle-packing/failure.log", + "text/plain", + "Candidate exception and traceback", + ) + ], + } + + args.report.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/circle-packing/target/.gitignore b/vero/examples/circle-packing/target/.gitignore new file mode 100644 index 00000000..507cc660 --- /dev/null +++ b/vero/examples/circle-packing/target/.gitignore @@ -0,0 +1,5 @@ +.vero/ +.venv/ +__pycache__/ +*.py[oc] +.evals/ diff --git a/vero/examples/circle-packing/target/packing.py b/vero/examples/circle-packing/target/packing.py new file mode 100644 index 00000000..88bf3d10 --- /dev/null +++ b/vero/examples/circle-packing/target/packing.py @@ -0,0 +1,62 @@ +"""Constructor-based circle packing for 26 circles in a unit square. + +Adapted from ShinkaEvolve's circle-packing ``initial.py`` baseline: +https://github.com/SakanaAI/ShinkaEvolve/tree/main/examples/circle_packing + +Copyright 2025 Sakana AI. Licensed under the Apache License, Version 2.0. +This VeRO adaptation replaces NumPy with the Python standard library and lets +the coding agent edit the complete program instead of a marked evolve block. +""" + +from __future__ import annotations + +import math + + +def construct_packing() -> tuple[list[list[float]], list[float]]: + """Return centers and non-overlapping radii for 26 circles.""" + + centers = [[0.0, 0.0] for _ in range(26)] + centers[0] = [0.5, 0.5] + + for index in range(8): + angle = 2.0 * math.pi * index / 8 + centers[index + 1] = [ + 0.5 + 0.3 * math.cos(angle), + 0.5 + 0.3 * math.sin(angle), + ] + + for index in range(16): + angle = 2.0 * math.pi * index / 16 + centers[index + 9] = [ + 0.5 + 0.7 * math.cos(angle), + 0.5 + 0.7 * math.sin(angle), + ] + + centers = [ + [min(0.99, max(0.01, x)), min(0.99, max(0.01, y))] + for x, y in centers + ] + return centers, compute_max_radii(centers) + + +def compute_max_radii(centers: list[list[float]]) -> list[float]: + """Greedily shrink initially maximal radii until every pair is feasible.""" + + radii = [min(x, y, 1.0 - x, 1.0 - y) for x, y in centers] + for left in range(len(centers)): + for right in range(left + 1, len(centers)): + distance = math.dist(centers[left], centers[right]) + radius_sum = radii[left] + radii[right] + if radius_sum > distance: + scale = distance / radius_sum + radii[left] *= scale + radii[right] *= scale + return [max(radius, 0.0) for radius in radii] + + +def run_packing() -> tuple[list[list[float]], list[float], float]: + """Entry point called by the trusted evaluator.""" + + centers, radii = construct_packing() + return centers, radii, sum(radii) diff --git a/vero/examples/circle-packing/target/pyproject.toml b/vero/examples/circle-packing/target/pyproject.toml new file mode 100644 index 00000000..a3256b26 --- /dev/null +++ b/vero/examples/circle-packing/target/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "vero-circle-packing-candidate" +version = "0.1.0" +description = "An editable 26-circle packing program" +requires-python = ">=3.11" +dependencies = [] + +[tool.uv] +package = false diff --git a/vero/examples/circle-packing/target/uv.lock b/vero/examples/circle-packing/target/uv.lock new file mode 100644 index 00000000..519f3d03 --- /dev/null +++ b/vero/examples/circle-packing/target/uv.lock @@ -0,0 +1,8 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" + +[[package]] +name = "vero-circle-packing-candidate" +version = "0.1.0" +source = { virtual = "." } diff --git a/vero/examples/circle-packing/vero.toml b/vero/examples/circle-packing/vero.toml new file mode 100644 index 00000000..6e190164 --- /dev/null +++ b/vero/examples/circle-packing/vero.toml @@ -0,0 +1,86 @@ +[target] +root = "./target" +ref = "HEAD" + +[backend] +id = "circle-packing" +kind = "command" +harness_root = "./harness" +environment = { PYTHONHASHSEED = "0" } +passthrough_environment = ["PATH", "UV_CACHE_DIR"] +command = [ + "uv", + "run", + "--frozen", + "--project", + "{workspace}", + "python", + "{harness}/evaluate.py", + "--workspace", "{workspace}", + "--request", "{request}", + "--report", "{report}", + "--artifacts", "{artifacts}", +] + +[[evaluations]] +name = "development" +partition = "development" +agent_can_evaluate = true +agent_visible = true +agent_selection = "fixed" +disclosure = "full" + +[evaluations.agent_budget] +total_runs = 30 + +[[evaluations]] +name = "final" +partition = "final" +agent_can_evaluate = false +agent_visible = false +agent_selection = "fixed" +disclosure = "none" + +[protocol] +selection_evaluation = "development" +final_evaluation = "final" +evaluate_final_baseline = true +timeout_seconds = 300 +max_proposals = 1 +max_rounds = 1 +max_concurrency = 1 +seed = 1337 + +[protocol.retry] +max_attempts = 1 + +[objective] +metric = "sum_radii" +direction = "maximize" +failure_value = 0.0 + +[[objective.constraints]] +metric = "valid" +operator = "==" +value = 1.0 + +[optimizer] +kind = "vero" +model = "openai/gpt-5" +max_turns = 200 +instruction = """ +Improve the program in packing.py to maximize the sum of the radii of exactly +26 non-overlapping circles inside the unit square. The best published result is +approximately 2.635, but make measurable progress from the supplied baseline +before pursuing sophisticated refinements. + +Use the development evaluation repeatedly and inspect its read-only JSON and +SVG artifacts in .evals. Preserve run_packing() and its return contract. You may +rewrite the algorithm completely. If you need third-party Python packages, add +and lock them with `uv add`; evaluation uses the checked-in uv.lock with +`--frozen`. Never weaken, bypass, or reproduce the trusted evaluator. +""" + +[session] +id = "circle-packing-example" +directory = "./.vero/session" diff --git a/vero/examples/harbor-circle-packing/.gitignore b/vero/examples/harbor-circle-packing/.gitignore new file mode 100644 index 00000000..68ea40f6 --- /dev/null +++ b/vero/examples/harbor-circle-packing/.gitignore @@ -0,0 +1,3 @@ +# Vendored VeRO package copied into the build context before `harbor run` +# (VeRO is not on public PyPI). Do not commit the copy. +environment/vero/ diff --git a/vero/examples/harbor-circle-packing/README.md b/vero/examples/harbor-circle-packing/README.md new file mode 100644 index 00000000..5d5ac4a1 --- /dev/null +++ b/vero/examples/harbor-circle-packing/README.md @@ -0,0 +1,72 @@ +# Harbor circle-packing example + +A **Harbor outer loop with a simple inner loop**: Harbor drives a coding agent +that edits `packing.py` to pack 26 non-overlapping circles in the unit square +(maximize the sum of radii), and each candidate is scored by a **plain +`CommandBackend`** — a fast Python scorer — **not** a nested `harbor run`. + +``` + harbor run ──► main service (coding agent edits /work/agent/packing.py) + │ evals run (self-score during the run) + ▼ + eval-sidecar ──► CommandBackend → harness/evaluate.py + (trusted: scoring, budget, disclosure, final selection) +``` + +## Why a custom factory (and not `vero harbor build`) + +`vero harbor build` compiles a task whose inner evaluation is itself a nested +`harbor run` (it hardcodes `HarborBackend` and requires a `task_source`). That +is the right tool when the inner task is another Harbor benchmark, but it is +heavier than needed here. For a **simple** inner loop you supply a custom sidecar +factory (`sidecar/circle_factory.py`) that wires a `CommandBackend` and run it +with `vero harbor serve --factory circle_factory:build` (see +`environment/docker-compose.yaml`). + +## Layout + +| Path | Role | +|---|---| +| `task.toml` | Harbor task definition | +| `instruction.md` | What the coding agent is told to do | +| `environment/Dockerfile` | Main service image (target + VeRO CLI) | +| `environment/main/seed.sh` | Seeds `/work/agent` as a git repo, then idles | +| `environment/agent-seed/` | Baseline `packing.py` the agent starts from | +| `environment/sidecar/Dockerfile` | Trusted sidecar image | +| `environment/sidecar/circle_factory.py` | Wires the `CommandBackend`, sidecar policies, objective, verifier | +| `environment/sidecar/harness/evaluate.py` | The scorer (`sum_radii`, `valid`, clearances) | +| `environment/agent-baseline/` | Trusted baseline the sidecar scores against | +| `solution/solve.sh` | Reference: self-score via `evals run` | +| `tests/test.sh` | Verifier: `vero harbor finalize` → `/logs/verifier/reward.json` | + +The objective is `sum_radii` (maximize) subject to `valid == 1`; the baseline +scores ~0.96 and the best known result is ~2.635. + +## Run it + +Requires Docker and a coding agent supported by Harbor (e.g. `codex`), plus a +model endpoint. From this directory: + +```bash +export OPENAI_API_KEY=... # or your gateway key +export OPENAI_BASE_URL=... # e.g. a LiteLLM proxy exposing /v1 + +harbor run -e docker -a codex -m openai/gpt-5.4 --yes +``` + +Harbor builds the two images, starts the sidecar, runs the agent against +`instruction.md`, then runs `tests/test.sh` to emit the final reward. + +## Notes + +- **Install:** VeRO is not published on public PyPI (the `scale-vero` name there + is an unrelated placeholder), so both images vendor the package. Before + `harbor run`, copy the VeRO package into this build context: + `cp -r /vero environment/vero` (it is git-ignored). The Dockerfiles then + `COPY vero /opt/vero && uv pip install "/opt/vero[harbor]"`. +- Multi-metric rewards: the verifier emits `reward.json` (a dict), and the + sidecar can run in a separate verifier environment for stronger isolation from + an untrusted agent — see the Harbor task docs. +- This mirrors a pattern validated end-to-end (codex improved the baseline from + 0.96 toward the ~2.635 optimum); it needs Docker + a coding agent + model + access to run and is not exercised by the unit test suite. diff --git a/vero/examples/harbor-circle-packing/build.yaml b/vero/examples/harbor-circle-packing/build.yaml new file mode 100644 index 00000000..9ac8f69b --- /dev/null +++ b/vero/examples/harbor-circle-packing/build.yaml @@ -0,0 +1,112 @@ +# The same circle-packing problem as the hand-written task in this directory, +# but compiled by `vero harbor build` instead of wired by circle_factory.py. +# +# Two things this demonstrates that the custom-factory path does not: +# +# - `evaluation_backend: command` — a non-agent target scored by a program, +# built through the compiler. There is nothing for a nested `harbor run` to +# drive, so none of the Harbor-only knobs apply. +# - An inference gateway. The optimizer reaches its model only through a +# scoped, allow-listed, metered token; the raw upstream credential stays in +# the gateway container. The hand-written task hands the agent the real +# OPENAI_* credentials instead, which is convenient but unmetered. +# +# Build and run: +# cp -r /vero environment/vero # VeRO is not on public PyPI +# vero harbor run --config build.yaml --agent codex --model gpt-5.4 \ +# --environment docker +# +# Either optimizer family works. `vero harbor run` points both surfaces at the +# same producer scope: OPENAI_* lands in the compose environment for codex, and +# ANTHROPIC_* is set on the host, where harbor's claude-code agent reads it and +# injects it into the agent (with /v1 stripped, which the Anthropic SDK +# re-appends). Swap in `--agent claude-code --model claude-sonnet-4-5` and the +# producer allow-list follows via ${optimizer_model}. +# +# Note the target and harness are shared with the hand-written task, so the two +# wirings score identically and can be compared directly. + +name: vero/circle-packing-metered +description: >- + Optimize packing.py to maximize the sum of 26 non-overlapping circle radii in + the unit square, with the optimizer metered through the inference gateway. + +# The editable target: a plain Python program, not an agent. +agent_repo: environment/agent-seed +harbor_requirement: harbor==0.20.0 + +# The harness ignores case identity — one deterministic score per run — so each +# partition holds a single nominal case. Held-out still means held-out: the +# agent may reach development and validation, never test. +partitions: + development: [dev] + validation: [val] + test: [holdout] + +agent_access: + - partition: development + disclosure: full + expose_case_resources: true + min_aggregate_cases: 1 + - partition: validation + disclosure: aggregate + min_aggregate_cases: 1 + total_runs: 10 + +selection_partition: validation + +targets: + - partition: test + reward_key: sum_radii + +objective: + selector: + metric: sum_radii + direction: maximize + failure_value: 0.0 + constraints: + - selector: + metric: valid + operator: "==" + value: 1.0 + +# A program scores the candidate; there is no target agent to run. +evaluation_backend: command +command_backend: + harness_source: environment/sidecar/harness + # CommandBackend runs the harness with a deliberately minimal environment + # (PATH is os.defpath, i.e. /bin:/usr/bin) so the sidecar's own environment -- + # which holds secrets -- is not handed to it wholesale. The base image keeps + # python in /usr/local/bin, so PATH has to be forwarded explicitly or the + # harness dies with "No such file or directory: 'python'". + passthrough_environment: [PATH, HOME, UV_CACHE_DIR] + command: + - python + - "{harness}/evaluate.py" + - --workspace + - "{workspace}" + - --request + - "{request}" + - --report + - "{report}" + - --artifacts + - "{artifacts}" + +# The optimizer's only route to a model. Its token is minted at build time and +# never leaves the gateway config as anything but a digest; the raw upstream key +# is read from OPENAI_API_KEY on the build host and stays in the gateway. +# +# `evaluation` is required by the schema but unused here: a command harness gets +# no gateway token, so this target makes no model calls of its own. +inference_gateway: + producer: + allowed_models: ["${optimizer_model:-gpt-5.4}"] + max_requests: 500 + max_concurrency: 4 + evaluation: + allowed_models: ["${optimizer_model:-gpt-5.4}"] + max_requests: 1 + +timeout_seconds: 900 +case_timeout_seconds: 300 +verifier_timeout_seconds: 1800 diff --git a/vero/examples/harbor-circle-packing/environment/Dockerfile b/vero/examples/harbor-circle-packing/environment/Dockerfile new file mode 100644 index 00000000..d9739763 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/Dockerfile @@ -0,0 +1,20 @@ +# Main service: the optimizer workbench a coding agent works in. +# Holds the editable target program plus the VeRO CLI so the agent can call +# `evals run` / `evals status` against the trusted sidecar. +FROM ghcr.io/astral-sh/uv:python3.12-bookworm + +RUN apt-get update \ + && apt-get install -y --no-install-recommends git ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +# Install VeRO's harbor client. VeRO is NOT on public PyPI (the `scale-vero` +# name there is an unrelated placeholder), so vendor the package into this build +# context first — e.g. `cp -r /vero environment/vero` — then install it: +COPY vero /opt/vero +RUN uv pip install --system "/opt/vero[harbor]" + +COPY agent-seed /opt/agent-seed +COPY main/seed.sh /opt/seed.sh +RUN chmod +x /opt/seed.sh && useradd -m -u 1001 agent + +WORKDIR /work/agent diff --git a/vero/examples/harbor-circle-packing/environment/agent-baseline/packing.py b/vero/examples/harbor-circle-packing/environment/agent-baseline/packing.py new file mode 100644 index 00000000..88bf3d10 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/agent-baseline/packing.py @@ -0,0 +1,62 @@ +"""Constructor-based circle packing for 26 circles in a unit square. + +Adapted from ShinkaEvolve's circle-packing ``initial.py`` baseline: +https://github.com/SakanaAI/ShinkaEvolve/tree/main/examples/circle_packing + +Copyright 2025 Sakana AI. Licensed under the Apache License, Version 2.0. +This VeRO adaptation replaces NumPy with the Python standard library and lets +the coding agent edit the complete program instead of a marked evolve block. +""" + +from __future__ import annotations + +import math + + +def construct_packing() -> tuple[list[list[float]], list[float]]: + """Return centers and non-overlapping radii for 26 circles.""" + + centers = [[0.0, 0.0] for _ in range(26)] + centers[0] = [0.5, 0.5] + + for index in range(8): + angle = 2.0 * math.pi * index / 8 + centers[index + 1] = [ + 0.5 + 0.3 * math.cos(angle), + 0.5 + 0.3 * math.sin(angle), + ] + + for index in range(16): + angle = 2.0 * math.pi * index / 16 + centers[index + 9] = [ + 0.5 + 0.7 * math.cos(angle), + 0.5 + 0.7 * math.sin(angle), + ] + + centers = [ + [min(0.99, max(0.01, x)), min(0.99, max(0.01, y))] + for x, y in centers + ] + return centers, compute_max_radii(centers) + + +def compute_max_radii(centers: list[list[float]]) -> list[float]: + """Greedily shrink initially maximal radii until every pair is feasible.""" + + radii = [min(x, y, 1.0 - x, 1.0 - y) for x, y in centers] + for left in range(len(centers)): + for right in range(left + 1, len(centers)): + distance = math.dist(centers[left], centers[right]) + radius_sum = radii[left] + radii[right] + if radius_sum > distance: + scale = distance / radius_sum + radii[left] *= scale + radii[right] *= scale + return [max(radius, 0.0) for radius in radii] + + +def run_packing() -> tuple[list[list[float]], list[float], float]: + """Entry point called by the trusted evaluator.""" + + centers, radii = construct_packing() + return centers, radii, sum(radii) diff --git a/vero/examples/harbor-circle-packing/environment/agent-baseline/pyproject.toml b/vero/examples/harbor-circle-packing/environment/agent-baseline/pyproject.toml new file mode 100644 index 00000000..a3256b26 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/agent-baseline/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "vero-circle-packing-candidate" +version = "0.1.0" +description = "An editable 26-circle packing program" +requires-python = ">=3.11" +dependencies = [] + +[tool.uv] +package = false diff --git a/vero/examples/harbor-circle-packing/environment/agent-seed/packing.py b/vero/examples/harbor-circle-packing/environment/agent-seed/packing.py new file mode 100644 index 00000000..88bf3d10 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/agent-seed/packing.py @@ -0,0 +1,62 @@ +"""Constructor-based circle packing for 26 circles in a unit square. + +Adapted from ShinkaEvolve's circle-packing ``initial.py`` baseline: +https://github.com/SakanaAI/ShinkaEvolve/tree/main/examples/circle_packing + +Copyright 2025 Sakana AI. Licensed under the Apache License, Version 2.0. +This VeRO adaptation replaces NumPy with the Python standard library and lets +the coding agent edit the complete program instead of a marked evolve block. +""" + +from __future__ import annotations + +import math + + +def construct_packing() -> tuple[list[list[float]], list[float]]: + """Return centers and non-overlapping radii for 26 circles.""" + + centers = [[0.0, 0.0] for _ in range(26)] + centers[0] = [0.5, 0.5] + + for index in range(8): + angle = 2.0 * math.pi * index / 8 + centers[index + 1] = [ + 0.5 + 0.3 * math.cos(angle), + 0.5 + 0.3 * math.sin(angle), + ] + + for index in range(16): + angle = 2.0 * math.pi * index / 16 + centers[index + 9] = [ + 0.5 + 0.7 * math.cos(angle), + 0.5 + 0.7 * math.sin(angle), + ] + + centers = [ + [min(0.99, max(0.01, x)), min(0.99, max(0.01, y))] + for x, y in centers + ] + return centers, compute_max_radii(centers) + + +def compute_max_radii(centers: list[list[float]]) -> list[float]: + """Greedily shrink initially maximal radii until every pair is feasible.""" + + radii = [min(x, y, 1.0 - x, 1.0 - y) for x, y in centers] + for left in range(len(centers)): + for right in range(left + 1, len(centers)): + distance = math.dist(centers[left], centers[right]) + radius_sum = radii[left] + radii[right] + if radius_sum > distance: + scale = distance / radius_sum + radii[left] *= scale + radii[right] *= scale + return [max(radius, 0.0) for radius in radii] + + +def run_packing() -> tuple[list[list[float]], list[float], float]: + """Entry point called by the trusted evaluator.""" + + centers, radii = construct_packing() + return centers, radii, sum(radii) diff --git a/vero/examples/harbor-circle-packing/environment/agent-seed/pyproject.toml b/vero/examples/harbor-circle-packing/environment/agent-seed/pyproject.toml new file mode 100644 index 00000000..a3256b26 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/agent-seed/pyproject.toml @@ -0,0 +1,9 @@ +[project] +name = "vero-circle-packing-candidate" +version = "0.1.0" +description = "An editable 26-circle packing program" +requires-python = ">=3.11" +dependencies = [] + +[tool.uv] +package = false diff --git a/vero/examples/harbor-circle-packing/environment/docker-compose.yaml b/vero/examples/harbor-circle-packing/environment/docker-compose.yaml new file mode 100644 index 00000000..c5d742e6 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/docker-compose.yaml @@ -0,0 +1,44 @@ +# Harbor merges this after its generated main-service configuration. +services: + main: + command: ["/opt/seed.sh"] + environment: + VERO_EVAL_URL: "http://eval-sidecar:8000" + OPENAI_API_KEY: "${OPENAI_API_KEY:?OPENAI_API_KEY must be set}" + OPENAI_BASE_URL: "${OPENAI_BASE_URL:?OPENAI_BASE_URL must be set}" + volumes: + - agent_repo:/work/agent + - token_state:/state/token:ro + depends_on: + eval-sidecar: + condition: service_healthy + + eval-sidecar: + build: + context: . + dockerfile: sidecar/Dockerfile + command: + - "vero" + - "harbor" + - "serve" + - "--factory" + - "circle_factory:build" + - "--config" + - "/opt/serve.json" + - "--admin-token" + - "/state/token/admin.token" + volumes: + - agent_repo:/work/agent:ro + - admin_state:/state/admin + - token_state:/state/token + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s + +volumes: + agent_repo: + admin_state: + token_state: diff --git a/vero/examples/harbor-circle-packing/environment/main/seed.sh b/vero/examples/harbor-circle-packing/environment/main/seed.sh new file mode 100755 index 00000000..4f85e066 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/main/seed.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# Seed the agent's editable working copy as a git repo on first boot, then idle +# so the coding agent (driven by `harbor run`) can work in it. +set -eu +if [ ! -d /work/agent/.git ]; then + cp -a /opt/agent-seed/. /work/agent/ + cd /work/agent + git init -q + git add -A + git -c user.email=seed@vero.test -c user.name=seed commit -qm "baseline" +fi +find /work/agent -exec chown agent:agent {} + +git config --system --add safe.directory /work/agent +exec sleep infinity diff --git a/vero/examples/harbor-circle-packing/environment/sidecar/Dockerfile b/vero/examples/harbor-circle-packing/environment/sidecar/Dockerfile new file mode 100644 index 00000000..d0522acb --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/sidecar/Dockerfile @@ -0,0 +1,32 @@ +# Trusted evaluation sidecar. It owns scoring, budget, disclosure, and final +# candidate selection. The inner loop is a plain CommandBackend (circle-packing's +# evaluate.py) — a SIMPLE inner loop, NOT a nested `harbor run`. +FROM ghcr.io/astral-sh/uv:python3.12-bookworm + +RUN apt-get update \ + && apt-get install -y --no-install-recommends git ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +# VeRO is NOT on public PyPI (the `scale-vero` name there is an unrelated +# placeholder), so vendor the package into this build context first — e.g. +# `cp -r /vero environment/vero` — then install it: +COPY vero /opt/vero +RUN uv pip install --system "/opt/vero[harbor]" + +COPY agent-baseline /opt/agent-baseline +COPY sidecar/circle_factory.py /opt/circle_factory.py +COPY sidecar/harness /opt/harness +COPY sidecar/serve.json /opt/serve.json + +# The sidecar holds the trusted baseline as a git repo (source of the baseline +# candidate; also the transport root for importing the agent's commits). +RUN cd /opt/agent-baseline \ + && git init -q \ + && git add -A \ + && git -c user.email=baseline@vero.test -c user.name=baseline commit -qm "baseline" \ + && git config --system --add safe.directory /opt/agent-baseline \ + && git config --system --add safe.directory /work/agent \ + && git config --system --add safe.directory /work/agent/.git + +ENV PYTHONPATH=/opt +WORKDIR /opt diff --git a/vero/examples/harbor-circle-packing/environment/sidecar/circle_factory.py b/vero/examples/harbor-circle-packing/environment/sidecar/circle_factory.py new file mode 100644 index 00000000..b03db979 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/sidecar/circle_factory.py @@ -0,0 +1,167 @@ +"""Custom Harbor sidecar factory for circle-packing. + +Harbor drives the OUTER loop (a coding agent edits packing.py); the target is +scored by a plain CommandBackend (circle-packing's evaluate.py) — NOT a nested +harbor run. Same shape as the proven harbor-live demo, swapped to the +sum_radii / valid objective. +""" + +from __future__ import annotations + +from pathlib import Path + +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import ( + BackendRegistry, + BudgetLedger, + CaseRange, + ConstraintOperator, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationBudget, + EvaluationDatabase, + EvaluationSet, + Evaluator, + MetricConstraint, + MetricSelector, + ObjectiveSpec, +) +from vero.evaluation.backends.command import CommandBackend, CommandBackendConfig +from vero.evaluation.engine import EvaluationEngine +from vero.sandbox import LocalSandbox +from vero.sidecar.serve import SidecarComponents +from vero.sidecar.sidecar import EvaluationSidecar, SidecarEvaluationPolicy +from vero.sidecar.transport import GitCandidateTransport +from vero.sidecar.verifier import ( + CanonicalVerifier, + VerificationSelection, + VerificationTarget, +) +from vero.workspace import GitWorkspace + +SET = "circle-packing" +OBJ = ObjectiveSpec( + selector=MetricSelector(metric="sum_radii"), + direction="maximize", + failure_value=0.0, + constraints=[ + MetricConstraint( + selector=MetricSelector(metric="valid"), + operator=ConstraintOperator.EQ, + value=1.0, + ) + ], +) + + +def _backend(harness_root: str) -> CommandBackend: + return CommandBackend( + CommandBackendConfig( + harness_root=harness_root, + command=[ + "uv", "run", "--python", "3.12", "python", "{harness}/evaluate.py", + "--workspace", "{workspace}", + "--request", "{request}", + "--report", "{report}", + "--artifacts", "{artifacts}", + ], + environment={"PYTHONHASHSEED": "0"}, + passthrough_environment=["PATH", "HOME", "UV_CACHE_DIR"], + ) + ) + + +async def build(config: dict) -> SidecarComponents: + repo_path = config["repo_path"] + agent_repo_path = config["agent_repo_path"] + session_dir = Path(config["session_dir"]) + harness_root = config["harness_root"] + session_dir.mkdir(parents=True, exist_ok=True) + + sandbox = await LocalSandbox.create(root=Path(repo_path).parent) + workspace = await GitWorkspace.from_path(sandbox, repo_path) + candidate_repository = await GitCandidateRepository.create( + session_dir / "candidates", workspace=workspace + ) + database = EvaluationDatabase.load_reconciled( + database_path=session_dir / "database.json", + evaluations_dir=session_dir / "evaluations", + database_id="circle", + ) + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="cmd", + evaluation_set_key=f"cmd:{SET}:validation", + total_runs=10, + ) + ], + path=session_dir / "budgets.json", + ) + ledger.save() + + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=session_dir, + session_id="circle", + ), + backends=BackendRegistry({"cmd": _backend(harness_root)}), + database=database, + database_path=session_dir / "database.json", + budget_ledger=ledger, + ) + transport = GitCandidateTransport( + workspace=workspace, + candidate_repository=candidate_repository, + agent_repo_path=agent_repo_path, + ) + baseline = await transport.trusted_candidate("HEAD") + + selection = VerificationSelection( + mode="auto_best", + backend_id="cmd", + evaluation_set=EvaluationSet( + name=SET, partition="validation", selection=CaseRange(start=0, stop=1) + ), + objective=OBJ, + baseline_candidate=baseline, + ) + sidecar = EvaluationSidecar( + engine=engine, + candidate_transport=transport, + access_policies=[ + SidecarEvaluationPolicy( + backend_id="cmd", evaluation_set_name=SET, partition="development", + objective=OBJ, + access=EvaluationAccessPolicy( + disclosure=DisclosureLevel.FULL, expose_case_resources=True + ), + ), + SidecarEvaluationPolicy( + backend_id="cmd", evaluation_set_name=SET, partition="validation", + objective=OBJ, + access=EvaluationAccessPolicy( + disclosure=DisclosureLevel.AGGREGATE, min_aggregate_cases=1 + ), + ), + ], + admin_volume=session_dir / "admin", + ) + await sidecar.initialize_context() + + verifier = CanonicalVerifier( + engine=engine, + selection=selection, + targets=[ + VerificationTarget( + reward_key="sum_radii", backend_id="cmd", + evaluation_set=EvaluationSet(name=SET, partition="test"), + objective=OBJ, max_attempts=1, + ) + ], + admin_volume=session_dir / "admin", + score_baseline=True, + ) + return SidecarComponents(sidecar=sidecar, verifier=verifier) diff --git a/vero/examples/harbor-circle-packing/environment/sidecar/harness/evaluate.py b/vero/examples/harbor-circle-packing/environment/sidecar/harness/evaluate.py new file mode 100644 index 00000000..a80f94b0 --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/sidecar/harness/evaluate.py @@ -0,0 +1,315 @@ +"""Trusted validator and scorer for the 26-circle packing benchmark.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import math +import os +import random +import sys +import time +import traceback +from dataclasses import asdict, dataclass +from pathlib import Path +from types import ModuleType +from typing import Any, Sequence + + +@dataclass(frozen=True) +class Validation: + valid: bool + message: str + computed_sum: float + reported_sum: float + sum_error: float + minimum_boundary_clearance: float + minimum_pair_clearance: float + + +def _load_candidate(path: Path) -> ModuleType: + spec = importlib.util.spec_from_file_location("candidate_packing", path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load candidate program: {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _seed_runtime(seed: int | None) -> None: + """Seed common in-process RNGs before calling the candidate entry point.""" + + if seed is None: + return + os.environ["VERO_EVALUATION_SEED"] = str(seed) + random.seed(seed) + numpy = sys.modules.get("numpy") + numpy_random = getattr(numpy, "random", None) + numpy_seed = getattr(numpy_random, "seed", None) + if callable(numpy_seed): + numpy_seed(seed) + + +def _sequence(value: Any, *, name: str) -> list[Any]: + if isinstance(value, (str, bytes)): + raise TypeError(f"{name} must be a sequence, not text") + try: + return list(value) + except TypeError as error: + raise TypeError(f"{name} must be a sequence") from error + + +def _normalize_output( + output: Any, +) -> tuple[list[list[float]], list[float], float]: + values = _sequence(output, name="run_packing output") + if len(values) != 3: + raise ValueError("run_packing must return (centers, radii, reported_sum)") + + raw_centers = _sequence(values[0], name="centers") + raw_radii = _sequence(values[1], name="radii") + centers: list[list[float]] = [] + for index, center in enumerate(raw_centers): + coordinates = _sequence(center, name=f"centers[{index}]") + if len(coordinates) != 2: + raise ValueError(f"centers[{index}] must have exactly two coordinates") + centers.append([float(coordinates[0]), float(coordinates[1])]) + radii = [float(radius) for radius in raw_radii] + return centers, radii, float(values[2]) + + +def validate_packing( + centers: Sequence[Sequence[float]], + radii: Sequence[float], + reported_sum: float, +) -> Validation: + if len(centers) != 26: + raise ValueError(f"expected 26 centers, received {len(centers)}") + if len(radii) != 26: + raise ValueError(f"expected 26 radii, received {len(radii)}") + + flattened = [coordinate for center in centers for coordinate in center] + if not all(math.isfinite(value) for value in [*flattened, *radii, reported_sum]): + raise ValueError("centers, radii, and reported_sum must all be finite") + if any(radius < 0.0 for radius in radii): + raise ValueError("radii must be non-negative") + + computed_sum = math.fsum(radii) + sum_error = abs(computed_sum - reported_sum) + sum_matches = math.isclose(computed_sum, reported_sum, rel_tol=1e-9, abs_tol=1e-12) + boundary_clearances = [ + min(x - radius, y - radius, 1.0 - x - radius, 1.0 - y - radius) + for (x, y), radius in zip(centers, radii, strict=True) + ] + pair_clearances = [ + math.dist(centers[left], centers[right]) - radii[left] - radii[right] + for left in range(26) + for right in range(left + 1, 26) + ] + minimum_boundary_clearance = min(boundary_clearances) + minimum_pair_clearance = min(pair_clearances) + + failures: list[str] = [] + if not sum_matches: + failures.append( + f"reported sum differs from the computed sum by {sum_error:.6g}" + ) + if minimum_boundary_clearance < 0.0: + failures.append( + "at least one circle crosses the square boundary " + f"by {-minimum_boundary_clearance:.6g}" + ) + if minimum_pair_clearance < 0.0: + failures.append( + f"at least two circles overlap by {-minimum_pair_clearance:.6g}" + ) + + return Validation( + valid=not failures, + message="; ".join(failures) if failures else "packing is geometrically valid", + computed_sum=computed_sum, + reported_sum=reported_sum, + sum_error=sum_error, + minimum_boundary_clearance=minimum_boundary_clearance, + minimum_pair_clearance=minimum_pair_clearance, + ) + + +def _write_layout_json( + path: Path, + centers: Sequence[Sequence[float]], + radii: Sequence[float], + validation: Validation, +) -> None: + path.write_text( + json.dumps( + { + "schema_version": 1, + "circles": [ + {"index": index, "x": x, "y": y, "radius": radius} + for index, ((x, y), radius) in enumerate( + zip(centers, radii, strict=True) + ) + ], + "validation": asdict(validation), + }, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def _write_layout_svg( + path: Path, + centers: Sequence[Sequence[float]], + radii: Sequence[float], + validation: Validation, +) -> None: + size = 800 + circles = "\n".join( + ( + f' ' + ) + for index, ((x, y), radius) in enumerate(zip(centers, radii, strict=True)) + ) + label = ( + f"sum={validation.computed_sum:.12f}; " + f"valid={'yes' if validation.valid else 'no'}" + ) + path.write_text( + "\n".join( + [ + '', + ' ', + circles, + f' {label}', + "", + "", + ] + ), + encoding="utf-8", + ) + + +def _artifact(path: str, media_type: str, description: str) -> dict[str, str]: + return {"path": path, "media_type": media_type, "description": description} + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--workspace", type=Path, required=True) + parser.add_argument("--request", type=Path, required=True) + parser.add_argument("--report", type=Path, required=True) + parser.add_argument("--artifacts", type=Path, required=True) + args = parser.parse_args() + + command_input = json.loads(args.request.read_text(encoding="utf-8")) + if command_input.get("schema_version") != 1: + raise ValueError("unsupported command input schema") + request = command_input.get("request") + if not isinstance(request, dict): + raise ValueError("command input must contain an evaluation request") + seed = request.get("seed") + if seed is not None and not isinstance(seed, int): + raise ValueError("evaluation seed must be an integer or null") + + artifact_dir = args.artifacts / "circle-packing" + artifact_dir.mkdir(parents=True, exist_ok=True) + started = time.perf_counter() + try: + _seed_runtime(seed) + candidate = _load_candidate(args.workspace / "packing.py") + run_packing = getattr(candidate, "run_packing", None) + if not callable(run_packing): + raise AttributeError("packing.py must define a callable run_packing()") + # Reset after import-time work and seed NumPy if the candidate imported it. + _seed_runtime(seed) + centers, radii, reported_sum = _normalize_output(run_packing()) + runtime_ms = (time.perf_counter() - started) * 1000.0 + validation = validate_packing(centers, radii, reported_sum) + + _write_layout_json( + artifact_dir / "layout.json", centers, radii, validation + ) + _write_layout_svg(artifact_dir / "layout.svg", centers, radii, validation) + (artifact_dir / "validation.json").write_text( + json.dumps(asdict(validation), indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + artifacts = [ + _artifact( + "circle-packing/layout.json", + "application/json", + "Circle centers, radii, and validation measurements", + ), + _artifact( + "circle-packing/layout.svg", + "image/svg+xml", + "Rendered circle packing", + ), + _artifact( + "circle-packing/validation.json", + "application/json", + "Geometric feasibility diagnostics", + ), + ] + report: dict[str, Any] = { + "schema_version": 1, + "status": "success", + "metrics": { + "sum_radii": validation.computed_sum, + "valid": 1.0 if validation.valid else 0.0, + "minimum_boundary_clearance": validation.minimum_boundary_clearance, + "minimum_pair_clearance": validation.minimum_pair_clearance, + "runtime_ms": runtime_ms, + }, + "artifacts": artifacts, + } + if not validation.valid: + report["diagnostics"] = [ + { + "code": "invalid_packing", + "message": validation.message, + "severity": "warning", + "phase": "validation", + } + ] + except BaseException as error: + runtime_ms = (time.perf_counter() - started) * 1000.0 + (artifact_dir / "failure.log").write_text( + "".join(traceback.format_exception(error)), + encoding="utf-8", + ) + report = { + "schema_version": 1, + "status": "failed", + "metrics": {"runtime_ms": runtime_ms}, + "diagnostics": [ + { + "code": "candidate_execution_failed", + "message": f"{type(error).__name__}: {error}", + "severity": "error", + "phase": "candidate", + } + ], + "artifacts": [ + _artifact( + "circle-packing/failure.log", + "text/plain", + "Candidate exception and traceback", + ) + ], + } + + args.report.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/harbor-circle-packing/environment/sidecar/serve.json b/vero/examples/harbor-circle-packing/environment/sidecar/serve.json new file mode 100644 index 00000000..06ae9f4a --- /dev/null +++ b/vero/examples/harbor-circle-packing/environment/sidecar/serve.json @@ -0,0 +1 @@ +{ "repo_path": "/opt/agent-baseline", "agent_repo_path": "/work/agent", "session_dir": "/state/admin/session", "harness_root": "/opt/harness" } diff --git a/vero/examples/harbor-circle-packing/instruction.md b/vero/examples/harbor-circle-packing/instruction.md new file mode 100644 index 00000000..8f2b6a98 --- /dev/null +++ b/vero/examples/harbor-circle-packing/instruction.md @@ -0,0 +1,27 @@ +# Optimize the circle packing + +Improve `packing.py` in `/work/agent` so the program packs **26 non-overlapping +circles inside the unit square** with the **largest possible sum of radii**. + +`packing.py` must keep a callable `run_packing()` that returns +`(centers, radii, reported_sum)`, where `centers` is 26 `[x, y]` pairs, `radii` +is 26 non-negative floats, and `reported_sum` equals `sum(radii)`. Circles must +stay fully inside the unit square and must not overlap. The current baseline +scores about 0.96; the best known result is ~2.635. + +## Workflow + +1. Edit `/work/agent/packing.py`. +2. Commit your change with Git. +3. Score the current commit on the validation set: + + ```bash + evals run --backend cmd --evaluation-set circle-packing \ + --partition validation --start 0 --stop 1 + ``` + +4. Use `evals status` to see remaining evaluation budget. + +The trusted sidecar owns scoring, budget, and final candidate selection. Only +`sum_radii` (with a `valid == 1` constraint) counts. Iterate: try an +arrangement, measure it, keep what improves the validated sum. diff --git a/vero/examples/harbor-circle-packing/solution/solve.sh b/vero/examples/harbor-circle-packing/solution/solve.sh new file mode 100755 index 00000000..22463fd0 --- /dev/null +++ b/vero/examples/harbor-circle-packing/solution/solve.sh @@ -0,0 +1,6 @@ +#!/bin/sh +set -eu +cd /work/agent +git config user.name optimizer; git config user.email o@v.test +evals run --backend cmd --evaluation-set circle-packing --partition validation --start 0 --stop 1 +evals status diff --git a/vero/examples/harbor-circle-packing/task.toml b/vero/examples/harbor-circle-packing/task.toml new file mode 100644 index 00000000..4deb8212 --- /dev/null +++ b/vero/examples/harbor-circle-packing/task.toml @@ -0,0 +1,15 @@ +schema_version = "1.3" + +[task] +name = "vero/circle-packing" +description = "Optimize packing.py to maximize the sum of 26 non-overlapping circle radii in the unit square." + +[agent] +user = "agent" + +[verifier] +environment_mode = "shared" +timeout_sec = 600 + +[environment] +build_timeout_sec = 1800 diff --git a/vero/examples/harbor-circle-packing/tests/test.sh b/vero/examples/harbor-circle-packing/tests/test.sh new file mode 100755 index 00000000..ba6f13b8 --- /dev/null +++ b/vero/examples/harbor-circle-packing/tests/test.sh @@ -0,0 +1,5 @@ +#!/bin/sh +set -eu +mkdir -p /logs/verifier +vero harbor finalize --token-file /state/token/admin.token --output /logs/verifier/reward.json +cat /logs/verifier/reward.json diff --git a/vero/examples/harness-conformance/README.md b/vero/examples/harness-conformance/README.md new file mode 100644 index 00000000..d7b88fa9 --- /dev/null +++ b/vero/examples/harness-conformance/README.md @@ -0,0 +1,109 @@ +# harness-conformance + +A conformance check for the *stack*, not the model. Run it before spending a real +benchmark on a new optimizer harness or a new model, and you find out in minutes +whether the pieces an optimizer depends on actually work. + +It is deliberately the **same path** as `harness-engineering-bench`: the harbor +evaluation backend, a nested `harbor run` per case, a target agent metered through +the inference gateway's evaluation scope, and an optimizer metered through the +producer scope. Only the work is trivial — six arithmetic tasks, two per +partition, seconds each. A `command`-backend smoke would be cheaper still, but it +exercises different code and would not tell you what you need to know. + +## Run + +```bash +cd /vero +uv run vero harbor run \ + --config examples/harness-conformance/build.yaml \ + --env-file secrets.env \ + --environment docker \ + --agent claude-code --model claude-sonnet-5 \ + --yes -o /tmp/conformance +``` + +Swap `--agent` / `--model` for whatever is under test. Useful parameters: + +| parameter | default | why | +|---|---|---| +| `--param inner_env=...` | `modal` | **docker does not work** — the inner evaluation runs `harbor run -e docker` inside the sidecar, which has no docker CLI or socket, so every case fails with `Docker is not installed or not on PATH` and the eval 502s | +| `--param target_model=...` | `fireworks_ai/deepseek-v4-flash` | check a target model's gateway wiring | +| `--param optimizer_model=...` | follows `--model` | when the adapter requests a different string than it is launched with | +| `--param wandb_mode=online` | `disabled` | a smoke test should not need W&B | + +## What it proves + +Nine checks, in the order the optimizer meets them. The instruction is a literal +step-through — see `description` in `build.yaml`. + +| step | proves | +|---|---| +| 1 | the optimizer's `OPENAI_BASE_URL` is the compose-internal gateway, not a public endpoint — i.e. it is metered and cannot see upstream | +| 2 | `evals plan` exposes partitions, **case counts**, disclosure levels, and remaining budget | +| 3 | development task resources are mounted under `.evals/tasks/`; validation's are not | +| 4 | an evaluation runs, blocks in the foreground, and returns a score | +| 5 | **persistence** — the same result is recoverable from `.evals/results/` via `evals list` / `show` / `cases`, so truncated output is never lost | +| 5b | **traces** — `evals cases` reports `trace: true` and `evals trace ID CASE` returns per-phase spans with plausible durations | +| 6 | **disclosure is enforced** — `evals cases` refuses on an aggregate-only partition | +| 7 | budget accounting decrements by what was actually spent | +| 8 | the edit → commit → re-evaluate loop moves the score, and `evals diff` attributes it | +| 9 | `evals submit` accepts a nomination | + +## Reading the result + +Two independent signals, and they should agree: + +1. **`conformance-report.json`**, committed at the target repo root by the + optimizer. One entry per step plus a `summary`. This is also where the + optimizer records anything that *would* handicap it — a confusing message, a + missing number, a command that behaved oddly. +2. **The reward.** The seed target answers addition and abandons multiplication, + so a healthy stack scores **~0.5 for the seed and 1.0 after the fix**. Scoring + 0.0 throughout usually means the target could not reach the gateway; the + arithmetic is easy enough that no plausible model gets it wrong. + +If the report says every step passed but the reward is 0.0, trust the reward — +and treat the disagreement itself as a finding. + +## Why the seed is broken on purpose + +`target/src/conformance_agent/agent.py` returns early on `*`, leaving those +tasks unanswered. That gap is what makes step 8 meaningful: without it the seed +would already score 1.0 and there would be no way to see whether the +edit → evaluate → submit loop actually works. The fix is one early return. + +## Extending it + +Adding a check usually means adding a step to `description` — no code. Add a +*task* only if you need a new failure mode in the target: `tasks/` entries are +generated from a template (`task.toml`, `instruction.md`, +`environment/Dockerfile`, `tests/{test.sh,verify.py}`, `solution/solve.sh`), and +`partitions/*.json` list them by directory name. Keep every partition at two +cases so the whole run stays a few minutes. + +Timeouts follow the same rules as the benchmark suite (see +`harness-engineering-bench/CONFIGURATION.md`): `case_timeout_seconds` equals the +tasks' declared `[agent] timeout_sec` so harbor's agent-timeout multiplier is +exactly 1.0, and each ceiling sits above +`ceil(trials / max_concurrency) × case_timeout`. + +## Known-good and known-bad results + +The first run of this example found three real problems, which is roughly what it +is for: + +- `inner_env=docker` cannot work (see the parameter table). Fixed by defaulting to + `modal`. +- The checklist originally asked the optimizer to print `$OPENAI_API_KEY`. It + declined, correctly, and said so in its report. Now it only checks the variable + is non-empty. +- The optimizer committed its report *after* `evals submit`, so the submitted + candidate did not contain it and the report never reached the operator. The + checklist now fixes the order and ends with `cat conformance-report.json` so the + findings survive in the transcript regardless. + +It also confirmed the tooling is adequate for diagnosis: the agent-facing error was +a bare `502 evaluation failed`, but `evals show ID` surfaced harbor's verbatim +stderr, and the optimizer root-caused the Docker fault unaided and retried 7 times +to rule out transience. diff --git a/vero/examples/harness-conformance/SKILL.md b/vero/examples/harness-conformance/SKILL.md new file mode 100644 index 00000000..983637d5 --- /dev/null +++ b/vero/examples/harness-conformance/SKILL.md @@ -0,0 +1,188 @@ +--- +name: conformance-check +description: >- + Verify a new optimizer harness or model against the real VeRO stack in ~10 + minutes before spending a benchmark on it — gateway scoping and metering, the + evals CLI, result persistence, disclosure enforcement, budget accounting, and + the edit → evaluate → submit loop. Use when introducing an agent harness or + model, after changing gateway/sidecar/timeout configuration, or when a real run + fails and you need to know whether the stack or the benchmark is at fault. +--- + +# Conformance-checking a harness or model + +`vero/examples/harness-conformance` runs the **same path a real experiment takes** +— harbor evaluation backend, a nested `harbor run` per case, a target agent +metered through the gateway's evaluation scope, an optimizer through the producer +scope — over six arithmetic tasks that finish in seconds. It exists so that a +harness problem costs ten minutes rather than a benchmark. + +Read `README.md` in that directory for why it is built the way it is. This file is +how to run it and how to diagnose it when it fails. + +## When to run it + +- Before the first real run with a **new `--agent`** or a **new `--model`**. +- After changing anything shared: gateway budgets or allow-lists, timeouts, + `agent_env`, sidecar or harbor versions. +- When a benchmark run fails early and you cannot tell whether the fault is the + stack or that benchmark. + +## Run it + +```bash +cd /vero +uv run vero harbor run \ + --config examples/harness-conformance/build.yaml \ + --env-file \ + --environment docker \ + --agent --model \ + --yes -o /jobs +``` + +Parameters worth knowing: + +| parameter | default | why | +|---|---|---| +| `--param optimizer_model=` | follows `--model` | set it when the harness **requests** a different model string than it is launched with; it controls the producer allow-list independently | +| `--param target_model=` | a cheap chat model | the model the *target* harness calls; also the evaluation-scope allow-list | +| `--param inner_env=` | `modal` | leave it. `docker` cannot work — the inner evaluation runs `harbor run -e docker` inside the sidecar, which has no docker client or socket | +| `--param wandb_mode=online` | `disabled` | a smoke test should not need telemetry | + +Inner evaluations run on Modal, so the Modal credentials must be listed under +`secrets:` in the build — omit them and every case fails +`Modal requires authentication` with 0 trial groups. + +## Reading the result + +Two independent signals. **Check both, and if they disagree trust the reward.** + +1. **`conformance-report.json`**, committed by the optimizer at the target repo + root, one entry per step plus a `summary`. Recover it from the job's session + archive: + + ```bash + tar xzf /*/task__*/verifier/session.tar.gz -C + cd /session/candidates/repository.git + git log --oneline --all + git show :conformance-report.json + ``` + + The optimizer is asked to commit the report **before** submitting, because a + commit made after `evals submit` is not part of the submitted candidate and + never reaches you. It also `cat`s the report as its final action, so the + findings survive in the agent transcript even if the commit does not. + +2. **The reward.** The seed target answers addition and abandons multiplication, + so a healthy stack reads **~0.5 for the seed and 1.0 after the fix**. + `/*/task__*/verifier/reward.json` holds the final number. + +Interpreting the reward: + +| reward | meaning | +|---|---| +| `1.0` | the loop works: the optimizer measured, fixed the gap, and submitted | +| `~0.5` | evaluation works but the optimizer never fixed the seed — a harness-behaviour problem, not infrastructure | +| `0.0` | usually the target could not reach the gateway, or evaluations never ran. The arithmetic is trivial, so no plausible model gets it wrong | + +## What each step proves + +| step | proves | +|---|---| +| 1 | the optimizer's base URL is the compose-internal gateway, not a public endpoint — i.e. metered, and it cannot see upstream | +| 2 | `evals plan` exposes partitions, **case counts**, disclosure levels, remaining budget | +| 3 | development task resources are mounted; validation's are not | +| 4 | an evaluation runs, blocks in the foreground, returns a score | +| 5 | **persistence** — the same result is recoverable from `.evals/results/`, so truncated output is never lost | +| 5b | **traces** — `evals cases` reports `trace: true` and `evals trace ID CASE` returns per-phase spans with plausible durations | +| 6 | **disclosure is enforced** — `evals cases` refuses on an aggregate-only partition | +| 7 | budget decrements by what was actually spent | +| 8 | the edit → commit → re-evaluate loop moves the score, and `evals diff` attributes it | +| 9 | `evals submit` accepts a nomination | + +## When it fails: diagnose in this order + +**1. The evaluation record's diagnostics — the single most useful artifact.** +An agent-facing `502 evaluation failed` is deliberately terse, but the record +carries the backend's verbatim stderr: + +```bash +python3 -c " +import json,glob +for p in glob.glob('/session/evaluations/*/evaluation.json'): + d=json.load(open(p)); r=d.get('report') or {} + if r.get('status')!='success': + print(json.dumps(r.get('diagnostics'), indent=1)[:1200]) +" +``` + +This is where "Docker is not installed", "Modal requires authentication", and +"Can't instantiate abstract class" all surfaced immediately. + +**2. The gateway request log — which model, which endpoint, what status.** +Definitive for anything credential- or routing-shaped: + +```bash +python3 -c " +import json,glob,collections +c=collections.Counter() +for p in glob.glob('/session/artifacts/inference/requests/*.jsonl'): + for l in open(p,errors='replace'): + r=json.loads(l) + if r.get('scope')=='producer': + c[(r.get('status'), r.get('model'), r.get('endpoint'))]+=1 +for k,v in c.items(): print(k,'->',v) +" +``` + +Read it as: + +- **zero producer requests** → the harness bypassed the gateway entirely. It is + calling a provider's public endpoint with a scoped token, so it fails closed + with `401`; nothing leaks, but nothing works either. +- **`403 model_denied`** → the requested model string is not in the producer + allow-list. Note the string it *actually requested*, which is often not the one + you launched with, and set `--param optimizer_model=` to that. +- **`200` on an unexpected `endpoint`** → the harness is using an API surface you + did not intend (e.g. `responses` rather than `messages`), which matters both for + compatibility and for token attribution. + +**3. The harness's own trajectory**, under +`/*/task__*/agent/` — usually a `.txt` stream and/or `trajectory.json`. +Filter to errors rather than reading it whole: + +```bash +tr -d '\000' < /*/task__*/agent/.txt | grep -o '"type":"error".\{0,240\}' | tail -3 +``` + +**4. The trial phase timings**, via `evals trace ID CASE_ID` or the per-case +`execution_trace`. A near-zero `agent_execution` span means the harness never +called a model on that case. + +## Harness gotchas found this way + +Each of these was discovered by this check rather than by a benchmark run, and +each is the kind of thing that would otherwise read as "the model is bad": + +- **A secondary small model.** Several harnesses issue an auxiliary + summarisation/title call with a small model of the *same provider family*. The + producer allow-list usually holds one entry, so that call `403`s silently — it + is invisible outside the gateway log. Allow a second model. +- **Provider-prefixed model names.** Some harnesses require `provider/model` and + then request only the bare `model`, so the allow-list needs the bare form while + the launch flag needs the prefixed one. That is what `--param optimizer_model=` + is for. +- **Which API surface the harness drives.** Crossing model families through a + proxy can produce responses a harness cannot parse (mixed id namespaces in one + stream). Prefer the provider's native surface; vero supplies a gateway base URL + for non-`openai` opencode providers so Anthropic models stay on Messages. +- **A seed harness that will not instantiate.** Missing abstract methods surface + as `infrastructure_failure` on every case, not as a low score. + +## Extending it + +Adding a check is usually a step in the build's `description` — no code. Add a +*task* only to introduce a new failure mode in the target; keep every partition at +two cases so a run stays a few minutes. Timeouts follow the same rule as the +benchmark suite: `case_timeout_seconds` equals the tasks' declared +`[agent] timeout_sec` so harbor's agent-timeout multiplier is exactly 1.0. diff --git a/vero/examples/harness-conformance/build.yaml b/vero/examples/harness-conformance/build.yaml new file mode 100644 index 00000000..8a2c7af1 --- /dev/null +++ b/vero/examples/harness-conformance/build.yaml @@ -0,0 +1,222 @@ +# Harness/model conformance check: the cheapest complete exercise of the path a +# real experiment takes. Same shape as harness-engineering-bench -- harbor +# evaluation backend, nested `harbor run` per case, a target agent metered through +# the inference gateway's evaluation scope, an optimizer metered through the +# producer scope -- but with six arithmetic tasks that finish in seconds. +# +# Run it whenever you introduce a new optimizer harness or model, before spending +# a real benchmark on it: +# +# cd /vero +# uv run vero harbor run --config examples/harness-conformance/build.yaml \ +# --env-file secrets.env --environment docker \ +# --agent claude-code --model claude-sonnet-5 --yes -o /tmp/conformance +# +# Swap --agent/--model for the harness under test. `${optimizer_model}` follows +# --model into the producer allow-list; note some adapters request a different +# string than they are launched with (see README). +# +# Read the outcome from the agent's conformance-report.json (see description), and +# cross-check the reward: the seed answers addition only, so a working stack +# scores ~0.5 for the seed and 1.0 for a fixed candidate. +name: vero/harness-conformance +description: >- + This is a CONFORMANCE CHECK, not a research task. Your job is to work through + the checklist below in order and record what you observe, so an operator can + tell whether this harness is wired up correctly. Being thorough and literal + matters more than being clever. + + + Write your findings to `conformance-report.json` at the repository root as you + go, so partial progress survives. Use one entry per step: + `{"step_3_persistence": {"ok": true, "detail": "..."}}`. Commit it. + + + STEP 1 — gateway is scoped. Print `$OPENAI_BASE_URL` and confirm it points at + the compose-internal gateway, not a public provider endpoint. Check that + `$OPENAI_API_KEY` is non-empty WITHOUT printing or recording its value. + Record the host and path you see. If the URL is a public endpoint, stop and + report that loudly: it means the optimizer is unmetered. + + + STEP 2 — plan is legible. Run `evals plan`. Confirm you can see: two + partitions you may evaluate, a case count for each, the disclosure level of + each, and how much run/case budget remains. Record all of it. If the case + count is missing you cannot size a subset, which is a bug worth reporting. + + + STEP 3 — exposed cases. List `.evals/tasks/`. Confirm the development + partition's task resources are readable and that validation's are NOT + present. Record which partitions are exposed. + + + STEP 4 — baseline evaluation. Run the full development partition in the + FOREGROUND and record the score, the per-case wall time, and how long the call + blocked. Expect roughly 0.5: the seed harness answers addition tasks and + abandons multiplication ones. + + + STEP 5 — persistence. Without re-running anything, recover that same result + from disk: `evals list`, then `evals show ID`, then `evals cases ID`. Confirm + the score you read back matches the score that was printed, and that per-case + scores are visible. Record both numbers. This is the check that proves you + never need to re-run an evaluation to recover output you truncated. + + + STEP 5b — traces. `evals cases` prints a `trace` column; confirm it is true, + then run `evals trace ID CASE_ID` on the case that scored WORST and record the + span count and each span's phase and duration. Read one span with + `--span N`. Confirm the timings are plausible for what the harness actually + did — a case that never called a model should show a near-zero + `agent_execution` span. If the column says false, or the trace is empty, say so: + it means per-case debugging information is not reaching the optimizer. + + + STEP 6 — disclosure is enforced. Evaluate the validation partition, then try + `evals cases` on that result. It MUST refuse, because validation is + aggregate-only. Record the refusal. A success here is a disclosure leak and + the most important thing on this list to report. + + + STEP 7 — budget accounting. Run `evals plan` again and compare with STEP 2. + Confirm remaining runs and cases decreased by the amount you actually spent. + Record before and after. + + + STEP 8 — the edit loop works. Fix the target harness so it answers + multiplication tasks as well as addition ones (find the early return that + gives up), commit, and re-evaluate the SAME development selection. Confirm the + score rises to 1.0. Record before and after, and confirm `evals diff` between + the two evaluations attributes the gain to the multiplication cases. + + + STEP 9 — submit. Write the `summary` entry described below, commit the report, + and ONLY THEN nominate your candidate with `evals submit`. Order matters: a + report committed after submitting is not part of the submitted candidate and + an operator will never see it. Confirm the submission was accepted and record + the commit. + + + The `summary` entry: which steps passed, which failed, and any error text + verbatim. Anything you noticed that would handicap an optimizer -- a confusing + message, a missing number, a command that behaved unexpectedly -- belongs in + it. Then `cat conformance-report.json` as your last action, so the findings + survive in the transcript even if the commit does not. + +agent_repo: target +task_source: tasks +task_manifest: partitions/manifest.json +agent_import_path: conformance_agent.agent:ConformanceAgent +harbor_requirement: harbor[modal]==0.20.0 + +partition_files: + development: partitions/development.json + validation: partitions/validation.json + test: partitions/test.json + +# Deliberately mirrors the benchmark suite: development full-disclosure with task +# resources exposed, validation aggregate-only. STEP 6 depends on that asymmetry, +# so min_aggregate_cases is 2 (the whole partition) rather than the usual 5. +agent_access: + - partition: development + disclosure: full + expose_case_resources: true + total_runs: 20 + total_cases: 40 + - partition: validation + disclosure: aggregate + expose_case_resources: false + min_aggregate_cases: 2 + total_runs: 20 + total_cases: 40 + +selection_partition: validation +targets: + - partition: test + reward_key: reward + # A correct stack lands on 0.5 for the seed and 1.0 for a fixed candidate, so + # there is no measurement noise to average away and nothing to pin. + failure_value: 0.0 + max_attempts: 1 + +evaluation_set_name: conformance +objective: + selector: + metric: score + direction: maximize +reward_mode: submit +baseline_floor: false +score_baseline: false +rescore_top_k: 1 +rescore_attempts: 1 + +model: ${target_model:-fireworks_ai/deepseek-v4-flash} +# modal, matching the benchmarks. `inner_env=docker` does NOT work: the inner +# evaluation shells out to `harbor run -e docker` from inside the sidecar +# container, which has no docker CLI or socket, and every case fails with +# "Docker is not installed or not on PATH" -> 0 trial groups -> 502. +environment_name: ${inner_env:-modal} +harbor_python_version: "3.12" +n_attempts: 1 +max_retries: 1 +infrastructure_max_attempts: 2 +infrastructure_retry_delay_seconds: 5 +aggregate_attempts: best +feedback_transcripts: true +feedback_max_bytes: 8000 +expose_attempt_detail: false +# Every clock is derived the same way as the benchmark suite (see +# harness-engineering-bench/CONFIGURATION.md), just small: case_timeout equals the +# tasks' declared [agent] timeout_sec so harbor's multiplier is exactly 1.0, and +# each ceiling sits above ceil(trials / max_concurrency) x case_timeout. +timeout_seconds: 900 +case_timeout_seconds: 120 +task_agent_timeout_seconds: 120 +max_concurrency: 6 +error_rate_threshold: 0.5 +verifier_timeout_seconds: 1800 +# Passed through to the evaluation backend. The Modal tokens are REQUIRED, not +# optional: inner evals run on Modal (see environment_name), and without them +# harbor exits with "Modal requires authentication", produces 0 trial groups, and +# every evaluation 502s. +secrets: + - MODAL_TOKEN_ID + - MODAL_TOKEN_SECRET + - WANDB_API_KEY + - WANDB_BASE_URL + +harness_user: harness + +# Same loop protections as the benchmarks: a single blocking eval must fit in one +# Bash call, or the optimizer detaches and a headless run dies waiting. +agent_env: + BASH_MAX_TIMEOUT_MS: "1800000" + BASH_DEFAULT_TIMEOUT_MS: "1800000" + ENABLE_BACKGROUND_TASKS: "0" + FORCE_AUTO_BACKGROUND_TASKS: "0" + # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) default + # to symlinking their entry point into /usr/local/bin, which the unprivileged + # optimizer user cannot write: "Failed to install executable ... Permission + # denied". npm/nvm-based harnesses (claude-code, opencode) are unaffected. + UV_TOOL_BIN_DIR: "/home/agent/.local/bin" + +wandb: + project: vero-conformance + group: conformance + name: ${wandb_run:-conformance} + mode: ${wandb_mode:-disabled} # off by default; a smoke test should not need W&B + +inference_gateway: + upstream_api_key_env: OPENAI_API_KEY + upstream_base_url_env: OPENAI_BASE_URL + request_log_attribution: true + producer: + allowed_models: ["${optimizer_model:-openai/gpt-5.4}"] + max_concurrency: 4 + evaluation: + allowed_models: ["${target_model:-fireworks_ai/deepseek-v4-flash}"] + max_requests: 2000 + max_tokens: 20000000 + max_concurrency: 8 +instruct_multifidelity: false +instruct_exhaust_budget: false diff --git a/vero/examples/harness-conformance/partitions/development.json b/vero/examples/harness-conformance/partitions/development.json new file mode 100644 index 00000000..3861d171 --- /dev/null +++ b/vero/examples/harness-conformance/partitions/development.json @@ -0,0 +1,4 @@ +[ + "conformance-01", + "conformance-02" +] diff --git a/vero/examples/harness-conformance/partitions/manifest.json b/vero/examples/harness-conformance/partitions/manifest.json new file mode 100644 index 00000000..ccb2dede --- /dev/null +++ b/vero/examples/harness-conformance/partitions/manifest.json @@ -0,0 +1,19 @@ +{ + "schema_version": 1, + "task_source": "../tasks", + "dataset_name": "harness-conformance", + "seed": "vero-conformance-v1", + "partition_counts": { + "development": 2, + "validation": 2, + "test": 2 + }, + "tasks": [ + "conformance-01", + "conformance-02", + "conformance-03", + "conformance-04", + "conformance-05", + "conformance-06" + ] +} diff --git a/vero/examples/harness-conformance/partitions/test.json b/vero/examples/harness-conformance/partitions/test.json new file mode 100644 index 00000000..0eafdd61 --- /dev/null +++ b/vero/examples/harness-conformance/partitions/test.json @@ -0,0 +1,4 @@ +[ + "conformance-05", + "conformance-06" +] diff --git a/vero/examples/harness-conformance/partitions/validation.json b/vero/examples/harness-conformance/partitions/validation.json new file mode 100644 index 00000000..b94e9e30 --- /dev/null +++ b/vero/examples/harness-conformance/partitions/validation.json @@ -0,0 +1,4 @@ +[ + "conformance-03", + "conformance-04" +] diff --git a/vero/examples/harness-conformance/target/pyproject.toml b/vero/examples/harness-conformance/target/pyproject.toml new file mode 100644 index 00000000..1c904056 --- /dev/null +++ b/vero/examples/harness-conformance/target/pyproject.toml @@ -0,0 +1,16 @@ +[project] +name = "vero-conformance-agent" +version = "0.1.0" +description = "Trivial editable target harness for the VeRO conformance example" +requires-python = ">=3.12" +dependencies = [ + "harbor==0.20.0", + "openai==2.46.0", +] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/conformance_agent"] diff --git a/vero/examples/harness-conformance/target/src/conformance_agent/__init__.py b/vero/examples/harness-conformance/target/src/conformance_agent/__init__.py new file mode 100644 index 00000000..204ea14a --- /dev/null +++ b/vero/examples/harness-conformance/target/src/conformance_agent/__init__.py @@ -0,0 +1 @@ +"""Seed target harness for the VeRO conformance example.""" diff --git a/vero/examples/harness-conformance/target/src/conformance_agent/agent.py b/vero/examples/harness-conformance/target/src/conformance_agent/agent.py new file mode 100644 index 00000000..239f5286 --- /dev/null +++ b/vero/examples/harness-conformance/target/src/conformance_agent/agent.py @@ -0,0 +1,95 @@ +"""Seed target harness for the conformance example. + +Deliberately incomplete: it answers addition tasks and gives up on multiplication +ones, so the seed scores about 0.5 and a one-line fix takes it to 1.0. That gap is +the point -- it makes the optimizer's edit -> evaluate -> submit loop observable in +the score rather than something we have to infer. + +The model call is not optional decoration either: the answer has to come back from +the model, so an unreachable or mis-allow-listed inference gateway shows up as a +zero score instead of passing silently. +""" + +from __future__ import annotations + +import re +import shlex +from typing import Any, override + +from harbor.agents.base import BaseAgent +from harbor.environments.base import BaseEnvironment +from harbor.models.agent.context import AgentContext +from openai import AsyncOpenAI + +ANSWER_PATH = "/app/answer.txt" + +INSTRUCTIONS = """You answer a single arithmetic question. +Reply with the numeric result and nothing else: no words, units, or punctuation. +""" + + +# A model told to answer with a bare number may still reason in prose first, so +# take the last number in the completion rather than trusting the raw text. +NUMBER_RE = re.compile(r"-?\d+(?:\.\d+)?") + + +class ConformanceAgent(BaseAgent): + """One model call, one file written. Nothing else.""" + + # BaseAgent declares these abstract; omitting them fails at instantiation with + # "Can't instantiate abstract class" and every case reports an infrastructure + # failure, not a low score. + @staticmethod + @override + def name() -> str: + return "conformance-agent" + + @override + def version(self) -> str | None: + return "0.1.0" + + def __init__(self, *args: Any, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + if self.model_name is None: + raise ValueError("conformance agent requires a model name") + # The gateway matches the requested model as an exact string, and the + # evaluation scope is allow-listed with the unprefixed name. + self._api_model = self.model_name.removeprefix("openai/") + # base_url and api_key come from OPENAI_BASE_URL / OPENAI_API_KEY, which + # vero points at the metered evaluation scope of the inference gateway. + self._client = AsyncOpenAI(max_retries=4) + + @override + async def setup(self, environment: BaseEnvironment) -> None: + await environment.exec("mkdir -p /app", timeout_sec=30) + + async def _write_answer(self, environment: BaseEnvironment, answer: str) -> None: + quoted = shlex.quote(answer.strip()) + await environment.exec( + f"printf '%s' {quoted} > {ANSWER_PATH}", cwd="/app", timeout_sec=30 + ) + + @override + async def run( + self, + instruction: str, + environment: BaseEnvironment, + context: AgentContext, + ) -> None: + # THE DELIBERATE GAP: multiplication tasks are abandoned unanswered. + # Removing this early return is the fix the optimizer is meant to find. + if "*" in instruction: + await self._write_answer(environment, "") + return + + completion = await self._client.chat.completions.create( + model=self._api_model, + messages=[ + {"role": "system", "content": INSTRUCTIONS}, + {"role": "user", "content": instruction}, + ], + max_tokens=256, + ) + content = (completion.choices[0].message.content or "").strip() + numbers = NUMBER_RE.findall(content) + await self._write_answer(environment, numbers[-1] if numbers else content) diff --git a/vero/examples/harness-conformance/tasks/conformance-01/environment/Dockerfile b/vero/examples/harness-conformance/tasks/conformance-01/environment/Dockerfile new file mode 100644 index 00000000..484807c6 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-01/environment/Dockerfile @@ -0,0 +1,6 @@ +# Smallest thing that can run the verifier. The target harness needs no +# packages of its own -- it runs outside the task container and reaches this +# environment through harbor's exec interface. +FROM python:3.12-slim +WORKDIR /app +RUN mkdir -p /app /logs/verifier diff --git a/vero/examples/harness-conformance/tasks/conformance-01/instruction.md b/vero/examples/harness-conformance/tasks/conformance-01/instruction.md new file mode 100644 index 00000000..d73e0f4a --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-01/instruction.md @@ -0,0 +1,3 @@ +Compute 17 + 25 and write only the numeric result to /app/answer.txt. + +Write nothing else to the file: no words, units, sign, or trailing text. diff --git a/vero/examples/harness-conformance/tasks/conformance-01/solution/solve.sh b/vero/examples/harness-conformance/tasks/conformance-01/solution/solve.sh new file mode 100755 index 00000000..a00a4509 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-01/solution/solve.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Reference solution: what a fully working harness produces. +set -Eeuo pipefail +printf '%s' '42' > /app/answer.txt diff --git a/vero/examples/harness-conformance/tasks/conformance-01/task.toml b/vero/examples/harness-conformance/tasks/conformance-01/task.toml new file mode 100644 index 00000000..9ba60b3e --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-01/task.toml @@ -0,0 +1,29 @@ +version = "1.0" + +[task] +name = "conformance/conformance-01" + +[metadata] +author_name = "VeRO" +benchmark = "harness-conformance" +source = "vero-examples" +source_id = "conformance-01" +difficulty = "trivial" +category = "smoke" +tags = ["conformance", "smoke"] + +# Deliberately tight: this task is seconds of work, so a hang is obvious rather +# than something that quietly eats an hour. vero sets case_timeout_seconds to +# match [agent] exactly, making harbor's agent-timeout multiplier 1.0. +[verifier] +timeout_sec = 60.0 + +[agent] +timeout_sec = 120.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 1024 +storage_mb = 2048 +allow_internet = true diff --git a/vero/examples/harness-conformance/tasks/conformance-01/tests/test.sh b/vero/examples/harness-conformance/tasks/conformance-01/tests/test.sh new file mode 100755 index 00000000..919ca088 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-01/tests/test.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Harbor reads the score from /logs/verifier/reward.txt. Always leave a value +# there and always exit 0, so a verifier crash reads as 0.0 rather than as +# infrastructure error -- an unanswered task is a legitimate zero. +set -Eeuo pipefail + +mkdir -p /logs/verifier +python3 /tests/verify.py || echo 0 > /logs/verifier/reward.txt +cat /logs/verifier/reward.txt +exit 0 diff --git a/vero/examples/harness-conformance/tasks/conformance-01/tests/verify.py b/vero/examples/harness-conformance/tasks/conformance-01/tests/verify.py new file mode 100644 index 00000000..e815ae5c --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-01/tests/verify.py @@ -0,0 +1,22 @@ +"""Exact-match verifier: strips whitespace, compares as a number.""" + +from pathlib import Path + +EXPECTED = "42" +ANSWER = Path("/app/answer.txt") +REWARD = Path("/logs/verifier/reward.txt") + + +def main() -> None: + REWARD.parent.mkdir(parents=True, exist_ok=True) + text = ANSWER.read_text(encoding="utf-8").strip() if ANSWER.is_file() else "" + try: + matched = abs(float(text) - float(EXPECTED)) < 1e-9 + except ValueError: + matched = False + REWARD.write_text("1\n" if matched else "0\n", encoding="utf-8") + print(f"expected={EXPECTED} got={text!r} reward={int(matched)}") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/harness-conformance/tasks/conformance-02/environment/Dockerfile b/vero/examples/harness-conformance/tasks/conformance-02/environment/Dockerfile new file mode 100644 index 00000000..484807c6 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-02/environment/Dockerfile @@ -0,0 +1,6 @@ +# Smallest thing that can run the verifier. The target harness needs no +# packages of its own -- it runs outside the task container and reaches this +# environment through harbor's exec interface. +FROM python:3.12-slim +WORKDIR /app +RUN mkdir -p /app /logs/verifier diff --git a/vero/examples/harness-conformance/tasks/conformance-02/instruction.md b/vero/examples/harness-conformance/tasks/conformance-02/instruction.md new file mode 100644 index 00000000..7cc9999a --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-02/instruction.md @@ -0,0 +1,3 @@ +Compute 6 * 7 and write only the numeric result to /app/answer.txt. + +Write nothing else to the file: no words, units, sign, or trailing text. diff --git a/vero/examples/harness-conformance/tasks/conformance-02/solution/solve.sh b/vero/examples/harness-conformance/tasks/conformance-02/solution/solve.sh new file mode 100755 index 00000000..a00a4509 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-02/solution/solve.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Reference solution: what a fully working harness produces. +set -Eeuo pipefail +printf '%s' '42' > /app/answer.txt diff --git a/vero/examples/harness-conformance/tasks/conformance-02/task.toml b/vero/examples/harness-conformance/tasks/conformance-02/task.toml new file mode 100644 index 00000000..e457484a --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-02/task.toml @@ -0,0 +1,29 @@ +version = "1.0" + +[task] +name = "conformance/conformance-02" + +[metadata] +author_name = "VeRO" +benchmark = "harness-conformance" +source = "vero-examples" +source_id = "conformance-02" +difficulty = "trivial" +category = "smoke" +tags = ["conformance", "smoke"] + +# Deliberately tight: this task is seconds of work, so a hang is obvious rather +# than something that quietly eats an hour. vero sets case_timeout_seconds to +# match [agent] exactly, making harbor's agent-timeout multiplier 1.0. +[verifier] +timeout_sec = 60.0 + +[agent] +timeout_sec = 120.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 1024 +storage_mb = 2048 +allow_internet = true diff --git a/vero/examples/harness-conformance/tasks/conformance-02/tests/test.sh b/vero/examples/harness-conformance/tasks/conformance-02/tests/test.sh new file mode 100755 index 00000000..919ca088 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-02/tests/test.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Harbor reads the score from /logs/verifier/reward.txt. Always leave a value +# there and always exit 0, so a verifier crash reads as 0.0 rather than as +# infrastructure error -- an unanswered task is a legitimate zero. +set -Eeuo pipefail + +mkdir -p /logs/verifier +python3 /tests/verify.py || echo 0 > /logs/verifier/reward.txt +cat /logs/verifier/reward.txt +exit 0 diff --git a/vero/examples/harness-conformance/tasks/conformance-02/tests/verify.py b/vero/examples/harness-conformance/tasks/conformance-02/tests/verify.py new file mode 100644 index 00000000..e815ae5c --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-02/tests/verify.py @@ -0,0 +1,22 @@ +"""Exact-match verifier: strips whitespace, compares as a number.""" + +from pathlib import Path + +EXPECTED = "42" +ANSWER = Path("/app/answer.txt") +REWARD = Path("/logs/verifier/reward.txt") + + +def main() -> None: + REWARD.parent.mkdir(parents=True, exist_ok=True) + text = ANSWER.read_text(encoding="utf-8").strip() if ANSWER.is_file() else "" + try: + matched = abs(float(text) - float(EXPECTED)) < 1e-9 + except ValueError: + matched = False + REWARD.write_text("1\n" if matched else "0\n", encoding="utf-8") + print(f"expected={EXPECTED} got={text!r} reward={int(matched)}") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/harness-conformance/tasks/conformance-03/environment/Dockerfile b/vero/examples/harness-conformance/tasks/conformance-03/environment/Dockerfile new file mode 100644 index 00000000..484807c6 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-03/environment/Dockerfile @@ -0,0 +1,6 @@ +# Smallest thing that can run the verifier. The target harness needs no +# packages of its own -- it runs outside the task container and reaches this +# environment through harbor's exec interface. +FROM python:3.12-slim +WORKDIR /app +RUN mkdir -p /app /logs/verifier diff --git a/vero/examples/harness-conformance/tasks/conformance-03/instruction.md b/vero/examples/harness-conformance/tasks/conformance-03/instruction.md new file mode 100644 index 00000000..4e40e068 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-03/instruction.md @@ -0,0 +1,3 @@ +Compute 128 + 47 and write only the numeric result to /app/answer.txt. + +Write nothing else to the file: no words, units, sign, or trailing text. diff --git a/vero/examples/harness-conformance/tasks/conformance-03/solution/solve.sh b/vero/examples/harness-conformance/tasks/conformance-03/solution/solve.sh new file mode 100755 index 00000000..9fee7ecf --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-03/solution/solve.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Reference solution: what a fully working harness produces. +set -Eeuo pipefail +printf '%s' '175' > /app/answer.txt diff --git a/vero/examples/harness-conformance/tasks/conformance-03/task.toml b/vero/examples/harness-conformance/tasks/conformance-03/task.toml new file mode 100644 index 00000000..be89c1c1 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-03/task.toml @@ -0,0 +1,29 @@ +version = "1.0" + +[task] +name = "conformance/conformance-03" + +[metadata] +author_name = "VeRO" +benchmark = "harness-conformance" +source = "vero-examples" +source_id = "conformance-03" +difficulty = "trivial" +category = "smoke" +tags = ["conformance", "smoke"] + +# Deliberately tight: this task is seconds of work, so a hang is obvious rather +# than something that quietly eats an hour. vero sets case_timeout_seconds to +# match [agent] exactly, making harbor's agent-timeout multiplier 1.0. +[verifier] +timeout_sec = 60.0 + +[agent] +timeout_sec = 120.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 1024 +storage_mb = 2048 +allow_internet = true diff --git a/vero/examples/harness-conformance/tasks/conformance-03/tests/test.sh b/vero/examples/harness-conformance/tasks/conformance-03/tests/test.sh new file mode 100755 index 00000000..919ca088 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-03/tests/test.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Harbor reads the score from /logs/verifier/reward.txt. Always leave a value +# there and always exit 0, so a verifier crash reads as 0.0 rather than as +# infrastructure error -- an unanswered task is a legitimate zero. +set -Eeuo pipefail + +mkdir -p /logs/verifier +python3 /tests/verify.py || echo 0 > /logs/verifier/reward.txt +cat /logs/verifier/reward.txt +exit 0 diff --git a/vero/examples/harness-conformance/tasks/conformance-03/tests/verify.py b/vero/examples/harness-conformance/tasks/conformance-03/tests/verify.py new file mode 100644 index 00000000..1e0a19b5 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-03/tests/verify.py @@ -0,0 +1,22 @@ +"""Exact-match verifier: strips whitespace, compares as a number.""" + +from pathlib import Path + +EXPECTED = "175" +ANSWER = Path("/app/answer.txt") +REWARD = Path("/logs/verifier/reward.txt") + + +def main() -> None: + REWARD.parent.mkdir(parents=True, exist_ok=True) + text = ANSWER.read_text(encoding="utf-8").strip() if ANSWER.is_file() else "" + try: + matched = abs(float(text) - float(EXPECTED)) < 1e-9 + except ValueError: + matched = False + REWARD.write_text("1\n" if matched else "0\n", encoding="utf-8") + print(f"expected={EXPECTED} got={text!r} reward={int(matched)}") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/harness-conformance/tasks/conformance-04/environment/Dockerfile b/vero/examples/harness-conformance/tasks/conformance-04/environment/Dockerfile new file mode 100644 index 00000000..484807c6 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-04/environment/Dockerfile @@ -0,0 +1,6 @@ +# Smallest thing that can run the verifier. The target harness needs no +# packages of its own -- it runs outside the task container and reaches this +# environment through harbor's exec interface. +FROM python:3.12-slim +WORKDIR /app +RUN mkdir -p /app /logs/verifier diff --git a/vero/examples/harness-conformance/tasks/conformance-04/instruction.md b/vero/examples/harness-conformance/tasks/conformance-04/instruction.md new file mode 100644 index 00000000..2e5ea5cc --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-04/instruction.md @@ -0,0 +1,3 @@ +Compute 12 * 12 and write only the numeric result to /app/answer.txt. + +Write nothing else to the file: no words, units, sign, or trailing text. diff --git a/vero/examples/harness-conformance/tasks/conformance-04/solution/solve.sh b/vero/examples/harness-conformance/tasks/conformance-04/solution/solve.sh new file mode 100755 index 00000000..8faecbb7 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-04/solution/solve.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Reference solution: what a fully working harness produces. +set -Eeuo pipefail +printf '%s' '144' > /app/answer.txt diff --git a/vero/examples/harness-conformance/tasks/conformance-04/task.toml b/vero/examples/harness-conformance/tasks/conformance-04/task.toml new file mode 100644 index 00000000..38bd962c --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-04/task.toml @@ -0,0 +1,29 @@ +version = "1.0" + +[task] +name = "conformance/conformance-04" + +[metadata] +author_name = "VeRO" +benchmark = "harness-conformance" +source = "vero-examples" +source_id = "conformance-04" +difficulty = "trivial" +category = "smoke" +tags = ["conformance", "smoke"] + +# Deliberately tight: this task is seconds of work, so a hang is obvious rather +# than something that quietly eats an hour. vero sets case_timeout_seconds to +# match [agent] exactly, making harbor's agent-timeout multiplier 1.0. +[verifier] +timeout_sec = 60.0 + +[agent] +timeout_sec = 120.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 1024 +storage_mb = 2048 +allow_internet = true diff --git a/vero/examples/harness-conformance/tasks/conformance-04/tests/test.sh b/vero/examples/harness-conformance/tasks/conformance-04/tests/test.sh new file mode 100755 index 00000000..919ca088 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-04/tests/test.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Harbor reads the score from /logs/verifier/reward.txt. Always leave a value +# there and always exit 0, so a verifier crash reads as 0.0 rather than as +# infrastructure error -- an unanswered task is a legitimate zero. +set -Eeuo pipefail + +mkdir -p /logs/verifier +python3 /tests/verify.py || echo 0 > /logs/verifier/reward.txt +cat /logs/verifier/reward.txt +exit 0 diff --git a/vero/examples/harness-conformance/tasks/conformance-04/tests/verify.py b/vero/examples/harness-conformance/tasks/conformance-04/tests/verify.py new file mode 100644 index 00000000..12f73819 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-04/tests/verify.py @@ -0,0 +1,22 @@ +"""Exact-match verifier: strips whitespace, compares as a number.""" + +from pathlib import Path + +EXPECTED = "144" +ANSWER = Path("/app/answer.txt") +REWARD = Path("/logs/verifier/reward.txt") + + +def main() -> None: + REWARD.parent.mkdir(parents=True, exist_ok=True) + text = ANSWER.read_text(encoding="utf-8").strip() if ANSWER.is_file() else "" + try: + matched = abs(float(text) - float(EXPECTED)) < 1e-9 + except ValueError: + matched = False + REWARD.write_text("1\n" if matched else "0\n", encoding="utf-8") + print(f"expected={EXPECTED} got={text!r} reward={int(matched)}") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/harness-conformance/tasks/conformance-05/environment/Dockerfile b/vero/examples/harness-conformance/tasks/conformance-05/environment/Dockerfile new file mode 100644 index 00000000..484807c6 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-05/environment/Dockerfile @@ -0,0 +1,6 @@ +# Smallest thing that can run the verifier. The target harness needs no +# packages of its own -- it runs outside the task container and reaches this +# environment through harbor's exec interface. +FROM python:3.12-slim +WORKDIR /app +RUN mkdir -p /app /logs/verifier diff --git a/vero/examples/harness-conformance/tasks/conformance-05/instruction.md b/vero/examples/harness-conformance/tasks/conformance-05/instruction.md new file mode 100644 index 00000000..3c2ea4ee --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-05/instruction.md @@ -0,0 +1,3 @@ +Compute 301 + 99 and write only the numeric result to /app/answer.txt. + +Write nothing else to the file: no words, units, sign, or trailing text. diff --git a/vero/examples/harness-conformance/tasks/conformance-05/solution/solve.sh b/vero/examples/harness-conformance/tasks/conformance-05/solution/solve.sh new file mode 100755 index 00000000..87c0b682 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-05/solution/solve.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Reference solution: what a fully working harness produces. +set -Eeuo pipefail +printf '%s' '400' > /app/answer.txt diff --git a/vero/examples/harness-conformance/tasks/conformance-05/task.toml b/vero/examples/harness-conformance/tasks/conformance-05/task.toml new file mode 100644 index 00000000..ec5eec1f --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-05/task.toml @@ -0,0 +1,29 @@ +version = "1.0" + +[task] +name = "conformance/conformance-05" + +[metadata] +author_name = "VeRO" +benchmark = "harness-conformance" +source = "vero-examples" +source_id = "conformance-05" +difficulty = "trivial" +category = "smoke" +tags = ["conformance", "smoke"] + +# Deliberately tight: this task is seconds of work, so a hang is obvious rather +# than something that quietly eats an hour. vero sets case_timeout_seconds to +# match [agent] exactly, making harbor's agent-timeout multiplier 1.0. +[verifier] +timeout_sec = 60.0 + +[agent] +timeout_sec = 120.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 1024 +storage_mb = 2048 +allow_internet = true diff --git a/vero/examples/harness-conformance/tasks/conformance-05/tests/test.sh b/vero/examples/harness-conformance/tasks/conformance-05/tests/test.sh new file mode 100755 index 00000000..919ca088 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-05/tests/test.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Harbor reads the score from /logs/verifier/reward.txt. Always leave a value +# there and always exit 0, so a verifier crash reads as 0.0 rather than as +# infrastructure error -- an unanswered task is a legitimate zero. +set -Eeuo pipefail + +mkdir -p /logs/verifier +python3 /tests/verify.py || echo 0 > /logs/verifier/reward.txt +cat /logs/verifier/reward.txt +exit 0 diff --git a/vero/examples/harness-conformance/tasks/conformance-05/tests/verify.py b/vero/examples/harness-conformance/tasks/conformance-05/tests/verify.py new file mode 100644 index 00000000..b6f98fb5 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-05/tests/verify.py @@ -0,0 +1,22 @@ +"""Exact-match verifier: strips whitespace, compares as a number.""" + +from pathlib import Path + +EXPECTED = "400" +ANSWER = Path("/app/answer.txt") +REWARD = Path("/logs/verifier/reward.txt") + + +def main() -> None: + REWARD.parent.mkdir(parents=True, exist_ok=True) + text = ANSWER.read_text(encoding="utf-8").strip() if ANSWER.is_file() else "" + try: + matched = abs(float(text) - float(EXPECTED)) < 1e-9 + except ValueError: + matched = False + REWARD.write_text("1\n" if matched else "0\n", encoding="utf-8") + print(f"expected={EXPECTED} got={text!r} reward={int(matched)}") + + +if __name__ == "__main__": + main() diff --git a/vero/examples/harness-conformance/tasks/conformance-06/environment/Dockerfile b/vero/examples/harness-conformance/tasks/conformance-06/environment/Dockerfile new file mode 100644 index 00000000..484807c6 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-06/environment/Dockerfile @@ -0,0 +1,6 @@ +# Smallest thing that can run the verifier. The target harness needs no +# packages of its own -- it runs outside the task container and reaches this +# environment through harbor's exec interface. +FROM python:3.12-slim +WORKDIR /app +RUN mkdir -p /app /logs/verifier diff --git a/vero/examples/harness-conformance/tasks/conformance-06/instruction.md b/vero/examples/harness-conformance/tasks/conformance-06/instruction.md new file mode 100644 index 00000000..c9724347 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-06/instruction.md @@ -0,0 +1,3 @@ +Compute 25 * 16 and write only the numeric result to /app/answer.txt. + +Write nothing else to the file: no words, units, sign, or trailing text. diff --git a/vero/examples/harness-conformance/tasks/conformance-06/solution/solve.sh b/vero/examples/harness-conformance/tasks/conformance-06/solution/solve.sh new file mode 100755 index 00000000..87c0b682 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-06/solution/solve.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Reference solution: what a fully working harness produces. +set -Eeuo pipefail +printf '%s' '400' > /app/answer.txt diff --git a/vero/examples/harness-conformance/tasks/conformance-06/task.toml b/vero/examples/harness-conformance/tasks/conformance-06/task.toml new file mode 100644 index 00000000..6c44564a --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-06/task.toml @@ -0,0 +1,29 @@ +version = "1.0" + +[task] +name = "conformance/conformance-06" + +[metadata] +author_name = "VeRO" +benchmark = "harness-conformance" +source = "vero-examples" +source_id = "conformance-06" +difficulty = "trivial" +category = "smoke" +tags = ["conformance", "smoke"] + +# Deliberately tight: this task is seconds of work, so a hang is obvious rather +# than something that quietly eats an hour. vero sets case_timeout_seconds to +# match [agent] exactly, making harbor's agent-timeout multiplier 1.0. +[verifier] +timeout_sec = 60.0 + +[agent] +timeout_sec = 120.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 1024 +storage_mb = 2048 +allow_internet = true diff --git a/vero/examples/harness-conformance/tasks/conformance-06/tests/test.sh b/vero/examples/harness-conformance/tasks/conformance-06/tests/test.sh new file mode 100755 index 00000000..919ca088 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-06/tests/test.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Harbor reads the score from /logs/verifier/reward.txt. Always leave a value +# there and always exit 0, so a verifier crash reads as 0.0 rather than as +# infrastructure error -- an unanswered task is a legitimate zero. +set -Eeuo pipefail + +mkdir -p /logs/verifier +python3 /tests/verify.py || echo 0 > /logs/verifier/reward.txt +cat /logs/verifier/reward.txt +exit 0 diff --git a/vero/examples/harness-conformance/tasks/conformance-06/tests/verify.py b/vero/examples/harness-conformance/tasks/conformance-06/tests/verify.py new file mode 100644 index 00000000..b6f98fb5 --- /dev/null +++ b/vero/examples/harness-conformance/tasks/conformance-06/tests/verify.py @@ -0,0 +1,22 @@ +"""Exact-match verifier: strips whitespace, compares as a number.""" + +from pathlib import Path + +EXPECTED = "400" +ANSWER = Path("/app/answer.txt") +REWARD = Path("/logs/verifier/reward.txt") + + +def main() -> None: + REWARD.parent.mkdir(parents=True, exist_ok=True) + text = ANSWER.read_text(encoding="utf-8").strip() if ANSWER.is_file() else "" + try: + matched = abs(float(text) - float(EXPECTED)) < 1e-9 + except ValueError: + matched = False + REWARD.write_text("1\n" if matched else "0\n", encoding="utf-8") + print(f"expected={EXPECTED} got={text!r} reward={int(matched)}") + + +if __name__ == "__main__": + main() diff --git a/vero/pyproject.toml b/vero/pyproject.toml new file mode 100644 index 00000000..fb743449 --- /dev/null +++ b/vero/pyproject.toml @@ -0,0 +1,78 @@ +[project] +name = "scale-vero" +version = "0.5.0" +description = "A harness for agents to optimize programs." +readme = "README.md" +authors = [ + { name = "Varun Ursekar", email = "oss@scale.com" } +] +license = { text = "MIT" } +requires-python = ">=3.11,<3.14" +dependencies = [ + "click>=8.0.0", + "pydantic>=2.11.7", + "wcmatch>=10.1", +] + +[project.scripts] +vero = "vero.cli:main" +evals = "vero.evals_cli:main" + +[project.optional-dependencies] +harbor = [ + "fastapi>=0.110", + "httpx>=0.28.1", + "jinja2>=3.1.6", + "pyyaml>=6.0.2", + "uvicorn>=0.27", +] +claude = [ + "claude-agent-sdk>=0.1.56", +] +optimize = [ + "async-lru>=2.0.5", + "beautifulsoup4>=4.14.2", + "httpx>=0.28.1", + "lxml>=6.0.2", + "openai-agents[litellm]>=0.18.3,<0.19", + "orjson>=3.10", + "pypdf>=6.2.0", + "trafilatura>=2.0.0", +] +wandb = [ + "wandb>=0.19.10", +] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/vero"] + +[[tool.uv.index]] +url = "https://pypi.org/simple" +name = "pypi" + +[tool.uv.sources] +vero = { workspace = true } + +[tool.ruff] +line-length = 88 +target-version = "py311" + +[tool.ruff.lint] +select = ["E4", "E7", "E9", "F", "I"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +pythonpath = ["tests"] + +[dependency-groups] +dev = [ + "gitpython>=3.1.46", + "pytest>=9.0.2", + "pytest-asyncio>=1.3.0", + "pytest-json-report>=1.5.0", + "scale-vero[optimize,harbor,claude]", +] diff --git a/vero/src/vero/__init__.py b/vero/src/vero/__init__.py new file mode 100644 index 00000000..8dc2472d --- /dev/null +++ b/vero/src/vero/__init__.py @@ -0,0 +1,34 @@ +"""VeRO's public program-optimization API.""" + +from vero.candidate import Candidate +from vero.candidate_repository import CandidateRepository, GitCandidateRepository +from vero.evaluation import ( + EvaluationBackend, + EvaluationRecord, + EvaluationReport, + EvaluationSet, + ObjectiveSpec, +) +from vero.optimization import CandidateProducer, OptimizationStrategy, Optimizer +from vero.runtime import ( + OptimizationSession, + create_local_optimization_session, + create_optimization_session, +) + +__all__ = [ + "Candidate", + "CandidateRepository", + "CandidateProducer", + "EvaluationBackend", + "EvaluationRecord", + "EvaluationReport", + "EvaluationSet", + "GitCandidateRepository", + "ObjectiveSpec", + "OptimizationSession", + "OptimizationStrategy", + "Optimizer", + "create_optimization_session", + "create_local_optimization_session", +] diff --git a/vero/src/vero/agents/__init__.py b/vero/src/vero/agents/__init__.py new file mode 100644 index 00000000..7f39c8ed --- /dev/null +++ b/vero/src/vero/agents/__init__.py @@ -0,0 +1,24 @@ +from .producer import AgentCandidateProducer +from .protocol import AgentContext, AgentRequirements, AgentRunResult, CodingAgent + +__all__ = [ + "AgentCandidateProducer", + "AgentContext", + "AgentRequirements", + "AgentRunResult", + "ClaudeCodeAgent", + "CodingAgent", + "VeroAgent", +] + + +def __getattr__(name: str): + if name == "ClaudeCodeAgent": + from .claude_code import ClaudeCodeAgent + + return ClaudeCodeAgent + if name == "VeroAgent": + from .vero import VeroAgent + + return VeroAgent + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/vero/src/vero/agents/claude_code.py b/vero/src/vero/agents/claude_code.py new file mode 100644 index 00000000..3934d494 --- /dev/null +++ b/vero/src/vero/agents/claude_code.py @@ -0,0 +1,364 @@ +"""ClaudeCodeAgent - Agent backend using Claude Agent SDK for optimization.""" + +from __future__ import annotations + +import inspect +import logging +import typing +from dataclasses import asdict, dataclass, field +from typing import Any, Callable + +from claude_agent_sdk import ( + ClaudeAgentOptions, + ClaudeSDKClient, + McpSdkServerConfig, + ResultMessage, + SdkMcpTool, + create_sdk_mcp_server, +) +from claude_agent_sdk.types import ( + Message, + SystemPromptPreset, +) +from pydantic import BaseModel, create_model + +from vero.agents.events import AgentEvent +from vero.agents.protocol import AgentContext, AgentRequirements, AgentRunResult +from vero.tools.base import ToolSet +from vero.tools.evaluation import EvaluationTools +from vero.tools.utils import get_tools_from_class +from vero.utils.general import recursively_serialize + +logger = logging.getLogger(__name__) + + +def default_tool_sets() -> list: + """Default tool sets for ClaudeCodeAgent.""" + return [EvaluationTools()] + + +def _default_claude_options() -> ClaudeAgentOptions: + return ClaudeAgentOptions( + model="claude-sonnet-4-5-20250929", + permission_mode="bypassPermissions", + allowed_tools=["WebSearch", "WebFetch", "Task", "Bash"], + ) + + +@dataclass +class ClaudeCodeAgent: + """Claude Agent SDK coding agent for a scoped optimization proposal.""" + + requirements = AgentRequirements(host_visible_workspace=True) + + options: ClaudeAgentOptions = field(default_factory=_default_claude_options) + tool_sets: list[ToolSet] = field(default_factory=default_tool_sets) + output_format: type[BaseModel] | None = None + trace: list[Message] = field(default_factory=list, repr=False) + state: dict[str, str] | None = field(default=None, repr=False) + + _context: AgentContext | None = field(default=None, repr=False) + _tools: dict[str, McpSdkServerConfig] = field(default_factory=dict, repr=False) + _allowed_tools: list[str] = field(default_factory=list, repr=False) + + async def run( + self, + *, + context: AgentContext, + prompt: str | None, + max_turns: int, + on_event: Callable[[Any], Any] | None = None, + ) -> AgentRunResult: + """Run Claude Code in the candidate workspace.""" + + self._context = context + self._tools, self._allowed_tools = self._create_tools() + input = prompt or "Improve the program, using evaluation feedback when useful." + + async with self._create_client(max_turns=max_turns) as client: + await client.query(input) + async for msg in client.receive_response(): + self.trace.append(msg) + if on_event is not None: + event_result = on_event(msg) + if inspect.isawaitable(event_result): + await event_result + + # Update state with session ID for resumption + result = self.latest_result + if result is not None and hasattr(result, "session_id"): + self.state = {"session_id": result.session_id} + + metadata = {"usage": self.usage()} + model = self.options.model + if model is not None: + metadata["model"] = str(model) + return AgentRunResult( + description="Apply Claude coding-agent changes", + state=recursively_serialize(self.serialize_state()), + trace=recursively_serialize(self.serialize_trace()), + metadata=metadata, + ) + + def serialize_event(self, event: Any) -> AgentEvent | None: + """Convert a Claude Agent SDK Message to a normalized AgentEvent.""" + if isinstance(event, ResultMessage): + text = "" + if hasattr(event, "result") and event.result: + text = str(event.result) + return {"kind": "result", "text": text} + + if not hasattr(event, "content"): + return None + + # Claude SDK Message has .content (str) and .role + content = getattr(event, "content", "") + role = getattr(event, "role", "") + + if role == "system": + return {"kind": "system", "text": str(content)[:200]} + + if role == "assistant": + return {"kind": "message", "text": str(content) if content else ""} + + if role == "tool": + name = getattr(event, "tool_name", "") + return { + "kind": "tool_result", + "name": name, + "output": str(content)[:500] if content else "", + "is_error": bool(getattr(event, "is_error", False)), + } + + # Fallback + return {"kind": "message", "text": str(content) if content else ""} + + def serialize_trace(self) -> Any: + """Serialize the execution trace (event log).""" + if not self.trace: + return None + return [asdict(msg) for msg in self.trace] + + def serialize_state(self) -> dict[str, str] | None: + """Serialize state for resumption — the Claude SDK session ID.""" + return self.state + + def deserialize_state(self, state: dict[str, str]) -> None: + """Restore state by setting the session ID for resume.""" + self.state = state + + def usage(self) -> dict: + """Return usage statistics from the latest result.""" + result = self.latest_result + if result is None: + return {} + usage = recursively_serialize(result.usage) + if not usage: + return {} + return usage + + def summary(self) -> dict: + """Return structured output for wandb summary.""" + structured_output = None + if self.latest_structured_output is not None: + try: + structured_output = self.latest_structured_output.model_dump() + except AttributeError: + structured_output = self.latest_structured_output + return {"structured_output": structured_output} + + def dict(self) -> dict[str, Any]: + """Return serializable Claude Code configuration.""" + return { + "claude_agent_options": recursively_serialize( + { + "allowed_tools": self.options.allowed_tools, + "permission_mode": self.options.permission_mode, + "model": self.options.model, + "cwd": str(self.options.cwd) if self.options.cwd else None, + } # type: ignore + ), + } + + @property + def latest_result(self) -> ResultMessage | None: + """Gets the latest result from the run result.""" + if not self.trace: + return None + for msg in reversed(self.trace): + if isinstance(msg, ResultMessage): + return msg + return None + + @property + def latest_structured_output(self) -> BaseModel | None: + """Gets the latest structured output, parsed into the output_format model.""" + result = self.latest_result + + if result is None or result.structured_output is None: + return None + + if self.output_format is None: + return result.structured_output + + try: + if isinstance(result.structured_output, dict): + return self.output_format.model_validate(result.structured_output) + elif isinstance(result.structured_output, str): + return self.output_format.model_validate_json(result.structured_output) + else: + return result.structured_output + except Exception as e: + logger.warning(f"Failed to parse structured output: {e}") + return result.structured_output + + # ------------------------------------------------------------------------- + # Internal helpers + # ------------------------------------------------------------------------- + + def _create_client(self, max_turns: int | None = None) -> ClaudeSDKClient: + """Creates the ClaudeSDKClient for this step.""" + from copy import copy + + options = copy(self.options) + + if self._context is not None: + options.cwd = self._context.project_path + + # System prompt + options.system_prompt = self._build_system_prompt() + + # MCP servers + options.mcp_servers = self._tools + options.allowed_tools = list(set(self._allowed_tools + options.allowed_tools)) + + # Resume from saved state + if self.state and isinstance(self.state, dict) and "session_id" in self.state: + options.resume = self.state["session_id"] + options.continue_conversation = True + logger.info(f"Resuming Claude Code session: {self.state['session_id']}") + + # Max turns + if max_turns is not None: + options.max_turns = max_turns + + # Output format + if self.output_format is not None: + options.output_format = { + "type": "json_schema", + "schema": self.output_format.model_json_schema(), + } + + return ClaudeSDKClient(options=options) + + def _build_system_prompt(self) -> str | SystemPromptPreset: + """Builds the system prompt from instructions and/or options.system_prompt.""" + + assert self._context is not None, "Agent context is not set" + + instructions = self._context.instructions + + # If options already has a system_prompt (e.g. SystemPromptPreset), merge with instructions + if ( + isinstance(self.options.system_prompt, dict) + and "type" in self.options.system_prompt + ): + preset = self.options.system_prompt + if instructions: + current_append = preset.append or "" + new_append = ( + current_append + "\n\n" + instructions + if current_append + else instructions + ) + return SystemPromptPreset( + type=preset.type, + preset=preset.preset, + append=new_append, + ) + return preset + + configured = self.options.system_prompt + if isinstance(configured, str) and configured: + return ( + f"{configured}\n\n{instructions}" + if instructions + else configured + ) + return instructions or "" + + def _create_tools( + self, + ) -> tuple[dict[str, McpSdkServerConfig], list[str]]: + """Creates the tool set instances based on tool_sets.""" + + assert self._context is not None, "Agent context is not set" + tools: dict[str, McpSdkServerConfig] = {} + internal_allowed_tools: list[str] = [] + + for tool_set in self.tool_sets: + if hasattr(tool_set, "bind"): + tool_set.bind(self._context) + + tool_names, mcp_config = self._to_claude_sdk_server_config(tool_set) + key = type(tool_set).__name__ + tools[key] = mcp_config + for name in tool_names: + internal_allowed_tools.append(f"mcp__{key}__{name}") + + return tools, internal_allowed_tools + + @staticmethod + def _to_claude_sdk_server_config( + instance: object, + ) -> tuple[list[str], McpSdkServerConfig]: + """Convert a tool instance to a Claude Agent SDK server config.""" + import inspect + + tool_methods = get_tools_from_class(instance) + sdk_mcp_tools: list[SdkMcpTool] = [] + tool_names: list[str] = [] + + for method in tool_methods: + name = method.__name__ + description = method.__doc__ or "" + # Use get_type_hints() to resolve string annotations (from __future__ import annotations) + try: + resolved_hints = typing.get_type_hints(method, include_extras=True) + except Exception: + resolved_hints = {} + params = { + param.name: (resolved_hints.get(param.name, param.annotation), ...) + for param in inspect.signature(method).parameters.values() + if param.name != "self" + } + input_schema = create_model(name, **params).model_json_schema() + + def make_handler(m: Callable) -> Callable: + def format_content(content: str | Exception) -> list[dict]: + return [{"type": "text", "text": str(content)}] + + async def handler(args: dict) -> dict: + try: + content = m(**args) + if inspect.isawaitable(content): + content = await content + return {"content": format_content(content)} + except Exception as e: + return {"content": format_content(e), "is_error": True} + + return handler + + sdk_mcp_tools.append( + SdkMcpTool( + name=name, + description=description, + input_schema=input_schema, + handler=make_handler(method), + ) + ) + tool_names.append(name) + + return tool_names, create_sdk_mcp_server( + name=instance.__class__.__name__, version="1.0.0", tools=sdk_mcp_tools + ) diff --git a/vero/src/vero/agents/events.py b/vero/src/vero/agents/events.py new file mode 100644 index 00000000..de22eb8d --- /dev/null +++ b/vero/src/vero/agents/events.py @@ -0,0 +1,46 @@ +"""Normalized agent event types. + +These are the canonical event shapes that agent backends produce and +logging/callbacks consume. No SDK-specific types should leak beyond +the agent's serialize_event boundary. +""" + +from __future__ import annotations + +from typing import TypedDict + + +class MessageEvent(TypedDict): + kind: str # "message" + text: str + + +class ThinkingEvent(TypedDict): + kind: str # "thinking" + text: str + + +class ToolCallEvent(TypedDict): + kind: str # "tool_call" + name: str + args: str + + +class ToolResultEvent(TypedDict): + kind: str # "tool_result" + name: str + output: str + is_error: bool + + +class SystemEvent(TypedDict): + kind: str # "system" + text: str + + +class ResultEvent(TypedDict): + kind: str # "result" + text: str + + +AgentEvent = MessageEvent | ThinkingEvent | ToolCallEvent | ToolResultEvent | SystemEvent | ResultEvent diff --git a/vero/src/vero/agents/producer.py b/vero/src/vero/agents/producer.py new file mode 100644 index 00000000..9da2e8c1 --- /dev/null +++ b/vero/src/vero/agents/producer.py @@ -0,0 +1,166 @@ +"""Adapt a coding agent into the optimization candidate-producer protocol.""" + +from __future__ import annotations + +import hashlib +import logging +from typing import Any, Callable + +from vero.agents.protocol import AgentContext, AgentRequirements, CodingAgent +from vero.optimization import ( + CandidateChange, + CandidateEvaluationGateway, + CandidateProductionContext, + CandidateProposal, +) +from vero.runtime import ArtifactStore +from vero.utils.general import recursively_serialize +from vero.workspace import Workspace + +logger = logging.getLogger(__name__) + + +class AgentCandidateProducer: + def __init__( + self, + agent: CodingAgent, + *, + prompt: str | None = None, + max_turns: int = 200, + artifacts: ArtifactStore | None = None, + on_event: Callable[[Any], Any] | None = None, + ): + if max_turns < 1: + raise ValueError("max_turns must be positive") + self.agent = agent + self.prompt = prompt + self.max_turns = max_turns + self.artifacts = artifacts + self.on_event = on_event + + def validate_workspace(self, workspace: Workspace) -> None: + requirements = getattr(self.agent, "requirements", AgentRequirements()) + if ( + requirements.host_visible_workspace + and workspace.sandbox.host_path(workspace.project_path) is None + ): + raise ValueError( + f"{type(self.agent).__name__} requires a host-visible workspace; " + f"sandbox {type(workspace.sandbox).__name__} does not expose one" + ) + + def bind_artifacts( + self, + artifacts: ArtifactStore, + *, + producer_id: str = "default", + restore: bool = True, + ) -> None: + """Attach durable storage and restore the latest supported agent state.""" + + self.artifacts = artifacts + if not restore: + return + state_path = self._producer_state_path(producer_id) + if not artifacts.path(state_path).exists(): + return + deserialize = getattr(self.agent, "deserialize_state", None) + if callable(deserialize): + deserialize(artifacts.read_json(state_path)) + + @staticmethod + def _producer_state_path(producer_id: str) -> str: + digest = hashlib.sha256(producer_id.encode()).hexdigest()[:16] + return f"agents/producers/{digest}/state.json" + + def _persist_artifacts( + self, + proposal: CandidateProposal, + *, + state: Any, + trace: Any, + ) -> None: + if self.artifacts is None: + return + digest = hashlib.sha256(proposal.id.encode()).hexdigest()[:16] + if state is not None: + serialized_state = recursively_serialize(state) + self.artifacts.write_json(f"agents/{digest}/state.json", serialized_state) + self.artifacts.write_json( + self._producer_state_path(proposal.producer_id), + serialized_state, + ) + if trace is not None: + self.artifacts.write_json( + f"agents/{digest}/trace.json", + recursively_serialize(trace), + ) + + def _persist_failed_run( + self, + proposal: CandidateProposal, + error: BaseException, + ) -> None: + if self.artifacts is None: + return + try: + serialize_state = getattr(self.agent, "serialize_state", None) + serialize_trace = getattr(self.agent, "serialize_trace", None) + state = serialize_state() if callable(serialize_state) else None + trace = serialize_trace() if callable(serialize_trace) else None + self._persist_artifacts(proposal, state=state, trace=trace) + digest = hashlib.sha256(proposal.id.encode()).hexdigest()[:16] + self.artifacts.write_json( + f"agents/{digest}/failure.json", + { + "type": type(error).__name__, + "message": str(error), + }, + ) + except Exception: + logger.exception("Failed to persist coding-agent failure artifacts") + + async def produce( + self, + *, + proposal: CandidateProposal, + context: CandidateProductionContext, + workspace: Workspace, + evaluation: CandidateEvaluationGateway, + ) -> CandidateChange | None: + self.validate_workspace(workspace) + parent = ( + context.candidates.get(proposal.parent_id) + if proposal.parent_id is not None + else None + ) + if parent is None: + parent = context.baseline + try: + result = await self.agent.run( + context=AgentContext( + session_id=context.session_id, + workspace=workspace, + proposal=proposal, + parent=parent, + evaluation=evaluation, + ), + prompt=proposal.instruction or self.prompt, + max_turns=self.max_turns, + on_event=self.on_event, + ) + except BaseException as error: + self._persist_failed_run(proposal, error) + raise + if result is None: + return None + + self._persist_artifacts( + proposal, + state=result.state, + trace=result.trace, + ) + return CandidateChange( + description=result.description, + metadata={"agent": type(self.agent).__name__, **result.metadata}, + ) diff --git a/vero/src/vero/agents/protocol.py b/vero/src/vero/agents/protocol.py new file mode 100644 index 00000000..7c9e4177 --- /dev/null +++ b/vero/src/vero/agents/protocol.py @@ -0,0 +1,97 @@ +"""Provider-neutral coding-agent contract.""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from typing import Any, Callable, Protocol, runtime_checkable + +from pydantic import Field, JsonValue, field_validator + +from vero.candidate import Candidate +from vero.models import StrictModel +from vero.optimization import ( + CandidateEvaluationGateway, + CandidateProposal, +) +from vero.runtime.context import AGENT_CONTEXT_DIRECTORY +from vero.workspace import Workspace + + +@dataclass(frozen=True) +class AgentRequirements: + """Workspace capabilities required by a coding-agent adapter.""" + + host_visible_workspace: bool = False + + +@dataclass(frozen=True) +class AgentContext: + """Capabilities visible to a coding agent working on one proposal.""" + + session_id: str + workspace: Workspace + proposal: CandidateProposal + parent: Candidate + evaluation: CandidateEvaluationGateway + + @property + def project_path(self) -> Path: + path = self.workspace.sandbox.host_path(self.workspace.project_path) + if path is None: + raise RuntimeError("coding agent requires a host-visible workspace path") + return path + + @property + def sandbox_project_path(self) -> str: + return self.workspace.project_path + + @property + def context_path(self) -> str: + return str(PurePosixPath(self.workspace.project_path) / AGENT_CONTEXT_DIRECTORY) + + @property + def relative_context_path(self) -> str: + return AGENT_CONTEXT_DIRECTORY + + @property + def instructions(self) -> str: + context = ( + f"Read-only evaluation context has been placed in " + f"`{AGENT_CONTEXT_DIRECTORY}/`. Inspect its README, tasks, prior " + "candidates, and evaluation results when useful. Do not modify or " + "commit that directory." + ) + if self.proposal.instruction: + return f"{self.proposal.instruction}\n\n{context}" + return context + + @property + def base_version(self) -> str: + return self.parent.version + + +class AgentRunResult(StrictModel): + description: str = "Apply coding-agent changes" + state: JsonValue | None = None + trace: JsonValue | None = None + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("description") + @classmethod + def validate_description(cls, value: str) -> str: + if not value.strip(): + raise ValueError("agent result description must not be empty") + return value + + +@runtime_checkable +class CodingAgent(Protocol): + async def run( + self, + *, + context: AgentContext, + prompt: str | None, + max_turns: int, + on_event: Callable[[Any], Any] | None = None, + ) -> AgentRunResult | None: ... diff --git a/vero/src/vero/agents/vero.py b/vero/src/vero/agents/vero.py new file mode 100644 index 00000000..a8d7e434 --- /dev/null +++ b/vero/src/vero/agents/vero.py @@ -0,0 +1,413 @@ +"""VeroAgent - native coding agent on the OpenAI Agents SDK. + +The agent harness (model orchestration, tool dispatch, event streaming) runs on +the host. Shell and file effects execute INSIDE a sandbox via plain *function +tools* (see ``vero.tools.sandbox_tools``) that drive an SDK ``SandboxSession``. +The sandbox is bound to the candidate checkout's host directory +(``Manifest(root=...)``), so mid-run evaluation and candidate capture operate on +that directory unchanged; the sandbox *client* is the containment seam +(unix-local by default; docker / modal / e2b for real isolation). + +Using function tools rather than the SDK's hosted ``Shell``/``Filesystem`` +capabilities keeps the native path model-agnostic: it works with any provider +over LiteLLM (ChatCompletions) or the Responses API. Hosted capabilities would +restrict it to the OpenAI Responses API. +""" + +from __future__ import annotations + +import asyncio +import contextlib +import inspect +import logging +from dataclasses import dataclass, field +from typing import Any, Callable + +from agents import ( + Agent, + FunctionTool, + ModelRetrySettings, + ModelSettings, + OpenAIChatCompletionsModel, + RunConfig, + Runner, + RunResultStreaming, + TResponseInputItem, + retry_policies, + set_tracing_disabled, +) +from agents.extensions.models.litellm_model import LitellmModel +from agents.sandbox import Manifest +from agents.sandbox.sandboxes.unix_local import UnixLocalSandboxClient +from pydantic import BaseModel + +from vero.agents.events import AgentEvent +from vero.agents.protocol import AgentContext, AgentRunResult +from vero.tools.base import ToolSet +from vero.tools.evaluation import EvaluationTools +from vero.tools.planning import TodoList, think +from vero.tools.sandbox_tools import build_sandbox_tools +from vero.tools.sub_agent import SubAgentTool +from vero.tools.utils.openai_agents import ( + callable_to_oai_tool, + tool_set_instance_to_oai_tools, +) +from vero.utils.general import recursively_serialize +from vero.utils.openai_agents import stream_events, strict_mode_from_model + +logger = logging.getLogger(__name__) + +set_tracing_disabled(True) + + +def default_tool_sets() -> list[ToolSet | object | Callable]: + """Default vero-specific tools for the VeroAgent. + + Shell and file access are provided by function tools bound to the sandbox + session at run time (see :meth:`VeroAgent._create_agent`), so only + vero-specific tools live here. + """ + + return [ + EvaluationTools(), + TodoList(), + think, + ] + + +def _default_oai_agent(model: str | None = None) -> Agent: + """Build the template Agent. + + The native path drives the sandbox with plain function tools, so any + provider works via LiteLLM. Model is configurable via VERO_OPTIMIZER_MODEL + or VeroAgent.for_model(). + """ + import os + + model = model or os.getenv("VERO_OPTIMIZER_MODEL") or "openai/gpt-5.4" + + litellm_kwargs: dict[str, str] = {} + api_key = os.getenv("LITELLM_API_KEY") + if api_key: + litellm_kwargs["api_key"] = api_key + base_url = os.getenv("LITELLM_BASE_URL") + if base_url: + litellm_kwargs["base_url"] = base_url.rstrip("/") + + configured_model = LitellmModel(model=model, **litellm_kwargs) + + return Agent( + name="VeroAgent", + model=configured_model, + model_settings=ModelSettings( + include_usage=True, + retry=ModelRetrySettings( + max_retries=3, + backoff={ + "initial_delay": 1.0, + "max_delay": 60.0, + "multiplier": 2.0, + "jitter": True, + }, + policy=retry_policies.any( + retry_policies.provider_suggested(), + retry_policies.retry_after(), + retry_policies.network_error(), + retry_policies.http_status([408, 429, 500, 502, 503, 504]), + ), + ), + ), + ) + + +@dataclass +class VeroAgent: + """OpenAI Agents SDK coding agent for a scoped optimization proposal.""" + + oai_agent: Agent = field(default_factory=_default_oai_agent) + tool_sets: list[ToolSet | object | Callable] = field( + default_factory=default_tool_sets + ) + max_retries: int = 3 + event_timeout: int | None = 60 * 12 + sandbox_client_factory: Callable[[], Any] = UnixLocalSandboxClient + state: list[TResponseInputItem] | None = field(default=None, repr=False) + + _context: AgentContext | None = field(default=None, repr=False) + _tools: dict[type | Callable, list[FunctionTool]] = field( + default_factory=dict, repr=False + ) + _run_result: RunResultStreaming | None = field(default=None, repr=False) + + @classmethod + def for_model(cls, model: str) -> VeroAgent: + """Construct an agent using an explicit LiteLLM model identifier.""" + + if not model.strip(): + raise ValueError("model must not be empty") + return cls(oai_agent=_default_oai_agent(model)) + + @property + def trace(self) -> list[TResponseInputItem] | None: + return self.state + + @trace.setter + def trace(self, value: Any) -> None: + self.state = value + + async def run( + self, + *, + context: AgentContext, + prompt: str | None, + max_turns: int, + on_event: Callable | None = None, + ) -> AgentRunResult: + """Run the agent in the candidate workspace.""" + + self._context = context + self._tools = self._create_tools(context) + state = self.state if self.state is not None else [] + input = prompt or "Improve the program, using evaluation feedback when useful." + inputs = state + [{"role": "user", "content": input}] + + client = self.sandbox_client_factory() + session = await client.create( + manifest=Manifest(root=str(context.project_path)) + ) + await session.start() + try: + agent = self._create_agent(session) + run_config = RunConfig( + workflow_name=f"vero::{context.session_id}", + trace_id=context.session_id, + ) + self._run_result = Runner.run_streamed( + agent, + input=inputs, + max_turns=max_turns, + run_config=run_config, + ) + async for event in stream_events( + self._run_result, event_timeout=self.event_timeout + ): + if on_event is not None: + result = on_event(event) + if inspect.isawaitable(result): + await result + await asyncio.sleep(0) + self.state = self._run_result.to_input_list() + finally: + with contextlib.suppress(Exception): + await session.stop() + with contextlib.suppress(Exception): + await client.delete(session) + + metadata = {"usage": self.usage()} + model = self.model_str() + if model is not None: + metadata["model"] = model + return AgentRunResult( + description="Apply Vero coding-agent changes", + state=recursively_serialize(self.serialize_state()), + trace=recursively_serialize(self.serialize_trace()), + metadata=metadata, + ) + + def serialize_event(self, event: Any) -> AgentEvent | None: + """Convert an OpenAI Agents SDK StreamEvent to a normalized AgentEvent. + + Returns None for noise events (raw_response_event, etc.) that + shouldn't be dispatched to callbacks. + """ + if not hasattr(event, "type") or event.type != "run_item_stream_event": + return None + + item = event.item.raw_item + raw = item.model_dump() if isinstance(item, BaseModel) else (item if isinstance(item, dict) else {}) + item_type = raw.get("type", "") + + if item_type == "message": + parts = [] + for block in raw.get("content", []): + if isinstance(block, dict): + text = block.get("text", "") + if text: + parts.append(text) + return {"kind": "message", "text": "\n".join(parts)} if parts else None + + if item_type in ("reasoning", "thinking"): + parts = [] + for block in (raw.get("summary") or raw.get("content") or []): + if isinstance(block, dict): + text = block.get("text") or block.get("summary") or "" + if text: + parts.append(text) + elif isinstance(block, str) and block: + parts.append(block) + return {"kind": "thinking", "text": "\n".join(parts)} if parts else None + + if item_type == "function_call": + return { + "kind": "tool_call", + "name": raw.get("name", "unknown"), + "args": raw.get("arguments", ""), + } + + if item_type == "function_call_output": + output = raw.get("output", "") + return { + "kind": "tool_result", + "name": raw.get("name", ""), + "output": output, + "is_error": isinstance(output, str) and "error" in output.lower()[:80], + } + + return None + + def serialize_trace(self) -> list[TResponseInputItem] | None: + """Serialize the execution trace. For VeroAgent, this is the conversation history.""" + return self.trace + + def serialize_state(self) -> list[TResponseInputItem] | None: + """Serialize state for resumption. Same as trace for VeroAgent.""" + return self.state + + def deserialize_state(self, state: list[TResponseInputItem]) -> None: + """Restore state from conversation history.""" + self.state = state + + def _get_sub_agent_tool(self) -> SubAgentTool | None: + """Get the SubAgentTool instance if present.""" + for tool_set in self.tool_sets: + if isinstance(tool_set, SubAgentTool): + return tool_set + return None + + def model_str(self) -> str | None: + if isinstance(self.oai_agent.model, (LitellmModel, OpenAIChatCompletionsModel)): + return self.oai_agent.model.model + elif isinstance(self.oai_agent.model, str): + return self.oai_agent.model + return None + + def usage(self) -> dict: + """Return usage statistics.""" + + if self._run_result is None: + return {} + + orchestrator_usage = self._run_result.context_wrapper.usage + sub_agent_usages = {} + sub_agent_tool = self._get_sub_agent_tool() + if sub_agent_tool: + sub_agent_usages = { + name: recursively_serialize(sa.session.context_wrapper.usage) # type: ignore + for name, sa in sub_agent_tool.sessions.items() + if sa.session is not None + } + return { + "orchestrator": recursively_serialize(orchestrator_usage), # type: ignore + "sub_agents": sub_agent_usages, + } + + def dict(self) -> dict: + """Return serializable Vero agent configuration.""" + return { + "model": self.model_str(), + "tool_sets": [ + type(ts).__name__ if hasattr(ts, "__class__") else ts.__name__ + for ts in self.tool_sets + ], + "model_settings": recursively_serialize(self.oai_agent.model_settings), # type: ignore + } + + def artifacts(self) -> dict: + try: + sub_agent_tool = self._get_sub_agent_tool() + if sub_agent_tool and sub_agent_tool.sessions: + sub_agent_dicts = { + key: sa.to_dict() for key, sa in sub_agent_tool.sessions.items() + } + return {"subagents": sub_agent_dicts} + except Exception as e: + logger.warning(f"Failed to get sub agent dicts: {e}") + + return {} + + @property + def strict_mode(self) -> bool: + if self.oai_agent.model is None: + return False + return strict_mode_from_model(self.oai_agent.model) + + def _create_agent(self, session: Any) -> Agent: + """Build the Agent: sandbox function tools + vero tools + the template. + + The shell/read_file/write_file tools execute inside ``session`` (bound + to the candidate checkout). Vero-specific tools (evaluation, planning) + run host-side in the harness. + """ + tools = ( + build_sandbox_tools(session) + + self._get_tools() + + list(self.oai_agent.tools) + ) + + instructions = self._context.instructions if self._context else None + + return Agent( + name=self.oai_agent.name, + model=self.oai_agent.model, + model_settings=self.oai_agent.model_settings, + output_type=self.oai_agent.output_type, + tools=tools, + instructions=instructions or self.oai_agent.instructions, + ) + + def _get_tools(self) -> list[FunctionTool]: + tools = [] + for tool_list in self._tools.values(): + tools.extend(tool_list) + tools.sort(key=lambda tool: tool.name) + return tools + + def _create_tools( + self, context: AgentContext + ) -> dict[type | Callable, list[FunctionTool]]: + instances: dict[type | Callable, list[FunctionTool]] = {} + sub_agent_tool = None + + for ts in self.tool_sets: + # Skip SubAgentTool — bind it last + if isinstance(ts, SubAgentTool): + sub_agent_tool = ts + continue + + # Bind if it's a ToolSet + if hasattr(ts, "bind"): + ts.bind(context) # type: ignore + + # Convert to list of FunctionTools + if inspect.isfunction(ts): + # Plain callable like think + instances[ts] = [callable_to_oai_tool(ts, strict_mode=self.strict_mode)] + else: + instances[type(ts)] = self._to_openai_function_tools(ts) + + # Bind SubAgentTool last (needs other tool instances) + if sub_agent_tool is not None: + sub_agent_tool.tool_instances = instances + model = self.oai_agent.model + model_name = model if isinstance(model, str) else model.model + sub_agent_tool.models = {model_name: model} + instances[SubAgentTool] = self._to_openai_function_tools(sub_agent_tool) + + return instances + + def _to_openai_function_tools(self, instance: object) -> list: + strict_mode = self.strict_mode + if inspect.isfunction(instance): + return [callable_to_oai_tool(instance, strict_mode=strict_mode)] + else: + return tool_set_instance_to_oai_tools(instance, strict_mode=strict_mode) diff --git a/vero/src/vero/candidate.py b/vero/src/vero/candidate.py new file mode 100644 index 00000000..9d7c0459 --- /dev/null +++ b/vero/src/vero/candidate.py @@ -0,0 +1,77 @@ +"""Canonical identity for a versioned program candidate.""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +from pydantic import ( + Field, + JsonValue, + field_validator, + model_validator, +) + +from vero.models import StrictModel + + +class Candidate(StrictModel): + """A materialized version of the program being optimized. + + ``version`` is interpreted by the session's candidate repository. A + session uses one repository/workspace family, so the identifier remains + stable when the candidate is materialized in different sandboxes. + """ + + id: str + version: str + parent_id: str | None = None + created_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) + description: str | None = None + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("id", "version") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("candidate identity must not be empty") + return value + + @field_validator("parent_id", "description") + @classmethod + def validate_optional_text(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("optional candidate text must not be empty") + return value + + @field_validator("created_at") + @classmethod + def normalize_created_at(cls, value: datetime) -> datetime: + if value.tzinfo is None or value.utcoffset() is None: + raise ValueError("candidate timestamps must be timezone-aware") + return value.astimezone(UTC) + + @model_validator(mode="after") + def validate_parent(self) -> Candidate: + if self.parent_id == self.id: + raise ValueError("a candidate cannot be its own parent") + return self + + @classmethod + def from_version( + cls, + version: str, + *, + candidate_id: str | None = None, + parent_id: str | None = None, + description: str | None = None, + metadata: dict[str, Any] | None = None, + ) -> Candidate: + """Construct the usual one-candidate-per-workspace-version identity.""" + return cls( + id=candidate_id or version, + version=version, + parent_id=parent_id, + description=description, + metadata=metadata or {}, + ) diff --git a/vero/src/vero/candidate_repository/__init__.py b/vero/src/vero/candidate_repository/__init__.py new file mode 100644 index 00000000..763fe2ae --- /dev/null +++ b/vero/src/vero/candidate_repository/__init__.py @@ -0,0 +1,10 @@ +"""Durable repositories for program candidates.""" + +from vero.candidate_repository.base import CandidateRepository, CandidateRepositoryError +from vero.candidate_repository.git import GitCandidateRepository + +__all__ = [ + "CandidateRepository", + "CandidateRepositoryError", + "GitCandidateRepository", +] diff --git a/vero/src/vero/candidate_repository/base.py b/vero/src/vero/candidate_repository/base.py new file mode 100644 index 00000000..c2bb02e2 --- /dev/null +++ b/vero/src/vero/candidate_repository/base.py @@ -0,0 +1,85 @@ +"""Durable candidate storage and compatible workspace materialization.""" + +from __future__ import annotations + +from abc import ABC, abstractmethod +from contextlib import asynccontextmanager +from typing import AsyncIterator, Generic, Sequence, TypeVar + +from vero.candidate import Candidate +from vero.sandbox import Sandbox +from vero.workspace import Workspace + +WorkspaceT = TypeVar("WorkspaceT", bound=Workspace) + + +class CandidateRepositoryError(RuntimeError): + """Raised when durable candidate state is invalid or cannot be transferred.""" + + +class CandidateRepository(ABC, Generic[WorkspaceT]): + """Own durable candidates for one workspace family within a session.""" + + @property + @abstractmethod + def family(self) -> str: + """Stable workspace/repository family identifier.""" + ... + + @property + @abstractmethod + def format_version(self) -> int: + """Persisted repository format version.""" + ... + + @abstractmethod + def supports(self, workspace: Workspace) -> bool: + """Whether this repository can capture the supplied workspace.""" + ... + + @abstractmethod + async def capture( + self, + candidate: Candidate, + workspace: WorkspaceT, + ) -> Candidate: + """Durably record a clean workspace at the candidate's version.""" + ... + + @abstractmethod + async def materialize_agent_history( + self, + candidates: Sequence[Candidate], + *, + workspace: WorkspaceT, + destination: str, + ) -> None: + """Expose a repository-native view of the visible candidates. + + ``destination`` is a sandbox path reserved for generated agent context. + Implementations may install disposable native references in the supplied + workspace, but must never mutate durable candidate state. + """ + ... + + @abstractmethod + def get(self, candidate_id: str) -> Candidate | None: + """Return one durable candidate by identity.""" + ... + + @abstractmethod + def list(self) -> tuple[Candidate, ...]: + """Return all durable candidates in deterministic order.""" + ... + + @asynccontextmanager + @abstractmethod + async def checkout( + self, + candidate: Candidate, + *, + sandbox: Sandbox, + name: str | None = None, + ) -> AsyncIterator[WorkspaceT]: + """Materialize a temporary isolated workspace for one candidate.""" + yield # pragma: no cover diff --git a/vero/src/vero/candidate_repository/git.py b/vero/src/vero/candidate_repository/git.py new file mode 100644 index 00000000..d0fbb9ba --- /dev/null +++ b/vero/src/vero/candidate_repository/git.py @@ -0,0 +1,862 @@ +"""Git-backed durable candidate repository.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import os +import re +import tempfile +import uuid +from collections.abc import Awaitable, Callable +from contextlib import asynccontextmanager +from pathlib import Path, PurePosixPath +from typing import AsyncIterator, Literal, Sequence + +from pydantic import field_validator + +from vero.candidate import Candidate +from vero.candidate_repository.base import CandidateRepository, CandidateRepositoryError +from vero.evaluation.store.persistence import _atomic_write_json +from vero.models import StrictModel +from vero.sandbox import LocalSandbox, Sandbox +from vero.workspace import GitWorkspace, Workspace + +_OBJECT_ID = re.compile(r"(?:[0-9a-f]{40}|[0-9a-f]{64})\Z") +_AGENT_CONTEXT_DIRECTORY = ".evals" + + +class _GitRepositoryConfig(StrictModel): + schema_version: Literal[1] = 1 + family: Literal["git"] = "git" + project_subpath: str = "." + + @field_validator("project_subpath") + @classmethod + def validate_project_subpath(cls, value: str) -> str: + path = PurePosixPath(value) + if not value or path.is_absolute() or ".." in path.parts or "\\" in value: + raise ValueError("project_subpath must stay within the Git repository") + return value + + +class _CandidateRecord(StrictModel): + schema_version: Literal[1] = 1 + candidate: Candidate + + +def _candidate_digest(candidate_id: str) -> str: + return hashlib.sha256(candidate_id.encode()).hexdigest() + + +def _safe_prefix(name: str | None) -> str: + if name is None: + return "vero-candidate-" + value = "".join( + character for character in name if character.isalnum() or character in "-_" + ) + return f"{value[:48] or 'vero-candidate'}-" + + +class GitCandidateRepository(CandidateRepository[GitWorkspace]): + """A session-owned bare Git repository plus durable candidate records.""" + + def __init__( + self, + *, + root: Path, + host: LocalSandbox, + config: _GitRepositoryConfig, + candidates: dict[str, Candidate], + ) -> None: + self.root = root + self.repository_path = root / "repository.git" + self.records_path = root / "records" + self.config_path = root / "repository.json" + self._host = host + self._config = config + self._candidates = candidates + self._lock = asyncio.Lock() + + @property + def family(self) -> str: + return "git" + + @property + def format_version(self) -> int: + return self._config.schema_version + + @property + def project_subpath(self) -> str: + return self._config.project_subpath + + @classmethod + async def create( + cls, + root: Path | str, + *, + workspace: GitWorkspace, + ) -> GitCandidateRepository: + """Create or open the Git repository paired with ``workspace``.""" + + root = Path(root).expanduser().resolve() + try: + relative = PurePosixPath(workspace.project_path).relative_to(workspace.root) + except ValueError as error: + raise ValueError( + "workspace project path must be within its Git root" + ) from error + project_subpath = str(relative) + config = _GitRepositoryConfig(project_subpath=project_subpath) + root.mkdir(parents=True, exist_ok=True) + host = await LocalSandbox.create(root=root.parent) + config_path = root / "repository.json" + repository_path = root / "repository.git" + records_path = root / "records" + + if config_path.exists(): + stored = _GitRepositoryConfig.model_validate_json( + config_path.read_text(encoding="utf-8") + ) + if stored != config: + raise ValueError( + "Git candidate repository does not match workspace project path" + ) + config = stored + else: + await asyncio.to_thread( + _atomic_write_json, + config_path, + config.model_dump(mode="json"), + ) + + if not repository_path.exists(): + result = await host.run(["git", "init", "--bare", str(repository_path)]) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr or "failed to initialize candidate repository" + ) + records_path.mkdir(parents=True, exist_ok=True) + instance = cls( + root=root, + host=host, + config=config, + candidates={}, + ) + await instance._load_records() + return instance + + @classmethod + async def open(cls, root: Path | str) -> GitCandidateRepository: + """Open an existing Git candidate repository without a source workspace.""" + + root = Path(root).expanduser().resolve() + config_path = root / "repository.json" + repository_path = root / "repository.git" + if not config_path.is_file() or not repository_path.is_dir(): + raise FileNotFoundError(f"candidate repository does not exist: {root}") + config = _GitRepositoryConfig.model_validate_json( + config_path.read_text(encoding="utf-8") + ) + host = await LocalSandbox.create(root=root.parent) + instance = cls(root=root, host=host, config=config, candidates={}) + await instance._load_records() + return instance + + def supports(self, workspace: Workspace) -> bool: + if not isinstance(workspace, GitWorkspace): + return False + try: + relative = PurePosixPath(workspace.project_path).relative_to(workspace.root) + except ValueError: + return False + return str(relative) == self.project_subpath + + def _record_path(self, candidate_id: str) -> Path: + return self.records_path / f"{_candidate_digest(candidate_id)}.json" + + def _candidate_ref(self, candidate_id: str) -> str: + return f"refs/vero/candidates/{_candidate_digest(candidate_id)}" + + def _context_ref(self, candidate_id: str) -> str: + return f"refs/vero/context/{_candidate_digest(candidate_id)}" + + @property + def _reserved_context_path(self) -> str: + if self.project_subpath == ".": + return _AGENT_CONTEXT_DIRECTORY + return str(PurePosixPath(self.project_subpath) / _AGENT_CONTEXT_DIRECTORY) + + async def _host_git(self, *arguments: str, timeout: int = 120) -> str: + result = await self._host.run( + ["git", "--git-dir", str(self.repository_path), *arguments], + timeout=timeout, + env={ + "PATH": os.defpath, + "LANG": "C.UTF-8", + "GIT_CONFIG_GLOBAL": "/dev/null", + }, + ) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr.strip() or result.stdout.strip() or "git command failed" + ) + return result.stdout.strip() + + async def _load_records(self) -> None: + loaded: dict[str, Candidate] = {} + for path in sorted(self.records_path.glob("*.json")): + try: + record = _CandidateRecord.model_validate_json( + path.read_text(encoding="utf-8") + ) + except (OSError, ValueError) as error: + raise CandidateRepositoryError( + f"invalid candidate record {path.name}: {error}" + ) from error + candidate = record.candidate + if path != self._record_path(candidate.id): + raise CandidateRepositoryError( + f"candidate record path does not match identity {candidate.id!r}" + ) + if candidate.id in loaded: + raise CandidateRepositoryError( + f"duplicate candidate record: {candidate.id!r}" + ) + ref = self._candidate_ref(candidate.id) + try: + resolved = await self._host_git( + "rev-parse", "--verify", f"{ref}^{{commit}}", timeout=30 + ) + except CandidateRepositoryError as error: + raise CandidateRepositoryError( + f"candidate {candidate.id!r} references a missing Git object" + ) from error + if resolved != candidate.version: + raise CandidateRepositoryError( + f"candidate {candidate.id!r} ref does not match its version" + ) + loaded[candidate.id] = candidate + self._candidates = loaded + + async def _source_git( + self, + workspace: GitWorkspace, + *arguments: str, + timeout: int = 120, + ) -> str: + result = await workspace.sandbox.run( + [ + "git", + "-c", + f"safe.directory={workspace.root}", + "-c", + "core.hooksPath=/dev/null", + *arguments, + ], + cwd=workspace.root, + timeout=timeout, + env={ + "PATH": os.defpath, + "LANG": "C.UTF-8", + "GIT_CONFIG_GLOBAL": "/dev/null", + }, + ) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr.strip() + or result.stdout.strip() + or "source git command failed" + ) + return result.stdout.strip() + + async def _repository_git( + self, + sandbox: Sandbox, + repository_path: str, + *arguments: str, + timeout: int = 120, + ) -> str: + result = await sandbox.run( + [ + "git", + "-c", + f"safe.directory={repository_path}", + "-c", + "core.hooksPath=/dev/null", + "-C", + repository_path, + *arguments, + ], + cwd=repository_path, + timeout=timeout, + env={ + "PATH": os.defpath, + "LANG": "C.UTF-8", + "GIT_CONFIG_GLOBAL": "/dev/null", + }, + ) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr.strip() + or result.stdout.strip() + or "source git command failed" + ) + return result.stdout.strip() + + async def _fetch_from_workspace( + self, + candidate: Candidate, + workspace: GitWorkspace, + temporary_ref: str, + ) -> None: + source_path = workspace.sandbox.host_path(workspace.root) + if source_path is not None: + await self._host_git( + "-c", + "protocol.file.allow=always", + "fetch", + "--force", + "--no-tags", + "--no-recurse-submodules", + str(source_path), + f"+{candidate.version}:{temporary_ref}", + ) + return + + export_ref = f"refs/vero/export/{uuid.uuid4().hex}" + with tempfile.TemporaryDirectory(prefix="vero-candidate-bundle-") as directory: + local_bundle = Path(directory) / "candidate.bundle" + async with workspace.sandbox.temporary_directory( + prefix="vero-candidate-export-" + ) as remote_directory: + remote_bundle = str( + PurePosixPath(remote_directory) / "candidate.bundle" + ) + try: + await self._source_git( + workspace, + "update-ref", + export_ref, + candidate.version, + timeout=30, + ) + await self._source_git( + workspace, + "bundle", + "create", + remote_bundle, + export_ref, + ) + await workspace.sandbox.download(remote_bundle, str(local_bundle)) + await self._host_git( + "fetch", + "--force", + "--no-tags", + str(local_bundle), + f"+{export_ref}:{temporary_ref}", + ) + finally: + try: + await asyncio.shield( + self._source_git( + workspace, + "update-ref", + "-d", + export_ref, + timeout=30, + ) + ) + except Exception: + pass + + async def _fetch_from_repository( + self, + candidate: Candidate, + *, + sandbox: Sandbox, + repository_path: str, + temporary_ref: str, + ) -> None: + source_path = sandbox.host_path(repository_path) + if source_path is not None: + await self._host_git( + "-c", + "protocol.file.allow=always", + "fetch", + "--force", + "--no-tags", + "--no-recurse-submodules", + str(source_path), + f"+{candidate.version}:{temporary_ref}", + ) + return + + export_ref = f"refs/vero/export/{uuid.uuid4().hex}" + with tempfile.TemporaryDirectory(prefix="vero-candidate-bundle-") as directory: + local_bundle = Path(directory) / "candidate.bundle" + async with sandbox.temporary_directory( + prefix="vero-candidate-export-" + ) as remote_directory: + remote_bundle = str( + PurePosixPath(remote_directory) / "candidate.bundle" + ) + try: + await self._repository_git( + sandbox, + repository_path, + "update-ref", + export_ref, + candidate.version, + timeout=30, + ) + await self._repository_git( + sandbox, + repository_path, + "bundle", + "create", + remote_bundle, + export_ref, + ) + await sandbox.download(remote_bundle, str(local_bundle)) + await self._host_git( + "fetch", + "--force", + "--no-tags", + str(local_bundle), + f"+{export_ref}:{temporary_ref}", + ) + finally: + try: + await asyncio.shield( + self._repository_git( + sandbox, + repository_path, + "update-ref", + "-d", + export_ref, + timeout=30, + ) + ) + except Exception: + pass + + async def _persist_import( + self, + candidate: Candidate, + fetch: Callable[[str], Awaitable[None]], + ) -> Candidate: + if _OBJECT_ID.fullmatch(candidate.version) is None: + raise ValueError("Git candidate version must be a full object ID") + async with self._lock: + existing = self._candidates.get(candidate.id) + if existing is not None: + if existing != candidate: + raise ValueError( + f"candidate ID {candidate.id!r} is already stored " + "with different data" + ) + return existing + + temporary_ref = f"refs/vero/incoming/{uuid.uuid4().hex}" + stable_ref = self._candidate_ref(candidate.id) + try: + await fetch(temporary_ref) + imported = await self._host_git( + "rev-parse", "--verify", f"{temporary_ref}^{{commit}}", timeout=30 + ) + if imported != candidate.version: + raise CandidateRepositoryError( + "imported Git object does not match candidate version" + ) + tracked_context = await self._host_git( + "ls-tree", + "-r", + "--name-only", + temporary_ref, + "--", + self._reserved_context_path, + timeout=30, + ) + if tracked_context: + raise CandidateRepositoryError( + f"candidate tracks reserved agent context path " + f"{self._reserved_context_path!r}" + ) + await self._host_git( + "update-ref", stable_ref, candidate.version, timeout=30 + ) + record = _CandidateRecord(candidate=candidate) + await asyncio.to_thread( + _atomic_write_json, + self._record_path(candidate.id), + record.model_dump(mode="json"), + ) + self._candidates[candidate.id] = candidate + return candidate + finally: + try: + await asyncio.shield( + self._host_git("update-ref", "-d", temporary_ref, timeout=30) + ) + except Exception: + pass + + async def capture( + self, + candidate: Candidate, + workspace: GitWorkspace, + ) -> Candidate: + if not self.supports(workspace): + raise TypeError( + "Git candidate repository requires a compatible GitWorkspace" + ) + if await workspace.is_dirty(): + raise ValueError("candidate workspace must be clean before capture") + actual = await workspace.current_version() + if actual != candidate.version: + raise ValueError( + f"candidate workspace is at {actual!r}, expected {candidate.version!r}" + ) + + async def fetch(temporary_ref: str) -> None: + await self._fetch_from_workspace(candidate, workspace, temporary_ref) + + return await self._persist_import(candidate, fetch) + + async def _exclude_agent_context(self, workspace: GitWorkspace) -> None: + git_directory = await self._source_git( + workspace, + "rev-parse", + "--absolute-git-dir", + timeout=30, + ) + exclude_path = str(PurePosixPath(git_directory) / "info" / "exclude") + pattern = f"/{self._reserved_context_path}/" + existing = ( + await workspace.sandbox.read_file(exclude_path) + if await workspace.sandbox.exists(exclude_path) + else "" + ) + lines = existing.splitlines() + if pattern not in lines: + value = existing + if value and not value.endswith("\n"): + value += "\n" + value += pattern + "\n" + await workspace.sandbox.write_file(exclude_path, value) + + async def _bundle_candidates( + self, + candidates: Sequence[Candidate], + destination: Path, + ) -> None: + refs = [self._candidate_ref(candidate.id) for candidate in candidates] + await self._host_git("bundle", "create", str(destination), *refs) + + async def materialize_agent_history( + self, + candidates: Sequence[Candidate], + *, + workspace: GitWorkspace, + destination: str, + ) -> None: + if not self.supports(workspace): + raise TypeError( + "Git candidate repository requires a compatible GitWorkspace" + ) + visible = sorted(candidates, key=lambda item: (item.created_at, item.id)) + for candidate in visible: + stored = self.get(candidate.id) + if stored != candidate: + raise ValueError( + f"candidate {candidate.id!r} is not present in durable storage" + ) + + await self._exclude_agent_context(workspace) + existing_refs = await self._source_git( + workspace, + "for-each-ref", + "--format=%(refname)", + "refs/vero/context", + timeout=30, + ) + for reference in existing_refs.splitlines(): + if reference: + await self._source_git( + workspace, + "update-ref", + "-d", + reference, + timeout=30, + ) + + if visible: + refspecs = [ + f"+{self._candidate_ref(candidate.id)}:{self._context_ref(candidate.id)}" + for candidate in visible + ] + if workspace.sandbox.host_path(workspace.root) is not None: + await self._source_git( + workspace, + "-c", + "protocol.file.allow=always", + "fetch", + "--force", + "--no-tags", + "--no-recurse-submodules", + str(self.repository_path), + *refspecs, + ) + else: + with tempfile.TemporaryDirectory( + prefix="vero-context-bundle-" + ) as directory: + local_bundle = Path(directory) / "candidates.bundle" + await self._bundle_candidates(visible, local_bundle) + async with workspace.sandbox.temporary_directory( + prefix="vero-context-import-" + ) as remote_directory: + remote_bundle = str( + PurePosixPath(remote_directory) / "candidates.bundle" + ) + await workspace.sandbox.upload( + str(local_bundle), + remote_bundle, + ) + await self._source_git( + workspace, + "fetch", + "--force", + "--no-tags", + "--no-recurse-submodules", + remote_bundle, + *refspecs, + ) + + if await workspace.sandbox.exists(destination): + await workspace.sandbox.remove(destination, recursive=True) + await workspace.sandbox.mkdir(destination) + by_id = {candidate.id: candidate for candidate in visible} + index = [] + for candidate in visible: + digest = _candidate_digest(candidate.id) + candidate_dir = str(PurePosixPath(destination) / digest) + await workspace.sandbox.mkdir(candidate_dir) + native_ref = self._context_ref(candidate.id) + await workspace.sandbox.write_file( + str(PurePosixPath(candidate_dir) / "candidate.json"), + candidate.model_dump_json(indent=2) + "\n", + ) + patch_path = None + parent = by_id.get(candidate.parent_id) if candidate.parent_id else None + if parent is not None: + arguments = [ + "diff", + "--binary", + parent.version, + candidate.version, + ] + if self.project_subpath != ".": + arguments.extend(["--", self.project_subpath]) + patch = await self._source_git(workspace, *arguments) + patch_path = f"{digest}/parent.patch" + await workspace.sandbox.write_file( + str(PurePosixPath(destination) / patch_path), + patch + ("\n" if patch and not patch.endswith("\n") else ""), + ) + index.append( + { + "candidate_id": candidate.id, + "version": candidate.version, + "parent_id": candidate.parent_id, + "native_ref": native_ref, + "metadata_path": f"{digest}/candidate.json", + "parent_patch_path": patch_path, + } + ) + await workspace.sandbox.write_file( + str(PurePosixPath(destination) / "index.json"), + json.dumps( + {"schema_version": 1, "candidates": index}, + ensure_ascii=False, + indent=2, + ) + + "\n", + ) + + async def import_candidate( + self, + candidate: Candidate, + *, + sandbox: Sandbox, + repository_path: str, + ) -> Candidate: + """Import a validated commit without checking out the source repository.""" + + async def fetch(temporary_ref: str) -> None: + await self._fetch_from_repository( + candidate, + sandbox=sandbox, + repository_path=repository_path, + temporary_ref=temporary_ref, + ) + + return await self._persist_import(candidate, fetch) + + def get(self, candidate_id: str) -> Candidate | None: + return self._candidates.get(candidate_id) + + def list(self) -> tuple[Candidate, ...]: + return tuple( + sorted( + self._candidates.values(), + key=lambda candidate: (candidate.created_at, candidate.id), + ) + ) + + async def _bundle_candidate(self, candidate: Candidate, destination: Path) -> None: + await self._host_git( + "bundle", + "create", + str(destination), + self._candidate_ref(candidate.id), + ) + + @asynccontextmanager + async def checkout( + self, + candidate: Candidate, + *, + sandbox: Sandbox, + name: str | None = None, + ) -> AsyncIterator[GitWorkspace]: + stored = self.get(candidate.id) + if stored is None: + raise KeyError(f"unknown candidate: {candidate.id!r}") + if stored != candidate: + raise ValueError( + f"candidate {candidate.id!r} does not match its durable record" + ) + + async with sandbox.temporary_directory(prefix=_safe_prefix(name)) as directory: + checkout_root = str(PurePosixPath(directory) / "repository") + result = await sandbox.run(["git", "init", checkout_root], timeout=30) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr or "failed to initialize candidate checkout" + ) + + host_directory = sandbox.host_path(directory) + if host_directory is not None: + source = str(self.repository_path) + ref = self._candidate_ref(candidate.id) + else: + with tempfile.TemporaryDirectory( + prefix="vero-candidate-checkout-" + ) as host_temporary: + local_bundle = Path(host_temporary) / "candidate.bundle" + await self._bundle_candidate(candidate, local_bundle) + remote_bundle = str(PurePosixPath(directory) / "candidate.bundle") + await sandbox.upload(str(local_bundle), remote_bundle) + source = remote_bundle + ref = self._candidate_ref(candidate.id) + result = await sandbox.run( + [ + "git", + "-c", + f"safe.directory={checkout_root}", + "-C", + checkout_root, + "fetch", + "--force", + "--no-tags", + source, + f"+{ref}:refs/vero/checkout", + ], + timeout=120, + ) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr or "failed to fetch remote candidate checkout" + ) + + if host_directory is not None: + result = await sandbox.run( + [ + "git", + "-c", + "protocol.file.allow=always", + "-c", + f"safe.directory={checkout_root}", + "-C", + checkout_root, + "fetch", + "--force", + "--no-tags", + source, + f"+{ref}:refs/vero/checkout", + ], + timeout=120, + ) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr or "failed to fetch candidate checkout" + ) + + result = await sandbox.run( + [ + "git", + "-c", + "core.hooksPath=/dev/null", + "-c", + f"safe.directory={checkout_root}", + "-C", + checkout_root, + "checkout", + "--detach", + candidate.version, + ], + timeout=60, + ) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr or "failed to check out candidate version" + ) + project_path = ( + checkout_root + if self.project_subpath == "." + else str(PurePosixPath(checkout_root) / self.project_subpath) + ) + workspace = GitWorkspace( + sandbox=sandbox, + root=checkout_root, + project_path=project_path, + ) + if await workspace.current_version() != candidate.version: + raise CandidateRepositoryError( + "candidate checkout resolved incorrectly" + ) + if await workspace.is_dirty(): + raise CandidateRepositoryError( + "candidate checkout is unexpectedly dirty" + ) + try: + yield workspace + finally: + context_path = str( + PurePosixPath(workspace.project_path) / _AGENT_CONTEXT_DIRECTORY + ) + if await workspace.sandbox.exists(context_path): + result = await asyncio.shield( + workspace.sandbox.run( + ["chmod", "-R", "u+w", context_path], + timeout=30, + ) + ) + if result.returncode != 0: + raise CandidateRepositoryError( + result.stderr + or f"failed to unseal agent context {context_path}" + ) diff --git a/vero/src/vero/cli.py b/vero/src/vero/cli.py new file mode 100644 index 00000000..792fd90f --- /dev/null +++ b/vero/src/vero/cli.py @@ -0,0 +1,1026 @@ +"""Command-line interface for generic program optimization.""" + +from __future__ import annotations + +import asyncio +import json +import os +import shlex +import shutil +import subprocess +from datetime import UTC, datetime +from pathlib import Path +from uuid import uuid4 + +import click + +from vero.candidate_repository import GitCandidateRepository +from vero.config import load_config +from vero.evaluation import ( + AllCases, + BudgetLedger, + CaseIds, + CaseRange, + CommandBackend, + CommandBackendConfig, + ConstraintOperator, + DisclosureLevel, + EvaluationDatabase, + EvaluationLimits, + EvaluationPlan, + EvaluationSet, + MetricAggregation, + MetricConstraint, + MetricSelector, + ObjectiveSpec, + RetryPolicy, + project_evaluation, +) +from vero.optimization import ( + CommandCandidateProducer, + CommandCandidateProducerConfig, + SequentialStrategy, +) +from vero.runtime import ( + SessionManifest, + SessionStatus, + WandbEventSink, + create_local_optimization_session, +) + + +def _default_home() -> Path: + return Path(os.environ.get("VERO_HOME", "~/.vero")).expanduser().resolve() + + +_CONFIG_TEMPLATE = '''[target] +root = "./target" +ref = "HEAD" + +[backend] +id = "command" +kind = "command" +harness_root = "./harness" +command = ["python3", "evaluate.py", "{workspace}", "{request}", "{report}"] + +[[evaluations]] +name = "train" +partition = "train" +agent_can_evaluate = true +agent_visible = true +agent_selection = "arbitrary" +disclosure = "full" +expose_case_resources = true + +[[evaluations]] +name = "validation" +partition = "validation" +agent_can_evaluate = true +agent_visible = true +agent_selection = "arbitrary" +disclosure = "aggregate" + +[evaluations.agent_budget] +total_runs = 50 + +[[evaluations]] +name = "test" +partition = "test" +agent_can_evaluate = false +agent_visible = false +agent_selection = "fixed" +disclosure = "none" + +[protocol] +selection_evaluation = "validation" +final_evaluation = "test" +max_proposals = 5 + +[objective] +metric = "score" +direction = "maximize" + +[optimizer] +kind = "claude" +instruction = "Improve the program without changing its intended behavior" +''' + + +def _parse_parameters(values: tuple[str, ...]) -> dict[str, object]: + parameters: dict[str, object] = {} + for value in values: + name, separator, encoded = value.partition("=") + if not separator or not name.strip(): + raise click.BadParameter( + "parameters must use NAME=JSON syntax", + param_hint="--parameter", + ) + if name in parameters: + raise click.BadParameter( + f"duplicate parameter {name!r}", + param_hint="--parameter", + ) + try: + parameters[name] = json.loads(encoded) + except json.JSONDecodeError as error: + raise click.BadParameter( + f"parameter {name!r} is not valid JSON: {error.msg}", + param_hint="--parameter", + ) from error + return parameters + + +def _command(value: str, option: str) -> list[str]: + try: + command = shlex.split(value) + except ValueError as error: + raise click.BadParameter(str(error), param_hint=option) from error + if not command: + raise click.BadParameter("command must not be empty", param_hint=option) + return command + + +def _parse_environment( + values: tuple[str, ...], + *, + option: str, +) -> dict[str, str]: + environment: dict[str, str] = {} + for value in values: + name, separator, content = value.partition("=") + if not separator or not name or "=" in name: + raise click.BadParameter( + "values must use NAME=VALUE syntax", param_hint=option + ) + if name in environment: + raise click.BadParameter( + f"duplicate environment variable {name!r}", param_hint=option + ) + environment[name] = content + return environment + + +def _parse_constraints( + values: tuple[tuple[str, str, str], ...], +) -> list[MetricConstraint]: + constraints: list[MetricConstraint] = [] + for metric_value, operator_value, target_value in values: + metric, separator, aggregation_value = metric_value.partition(":") + if not metric: + raise click.BadParameter("constraint metric must not be empty") + try: + aggregation = ( + MetricAggregation(aggregation_value) + if separator + else MetricAggregation.REPORT + ) + operator = ConstraintOperator(operator_value) + target = float(target_value) + constraints.append( + MetricConstraint( + selector=MetricSelector( + metric=metric, + aggregation=aggregation, + ), + operator=operator, + value=target, + ) + ) + except (ValueError, TypeError) as error: + raise click.BadParameter( + "constraints use METRIC[:AGGREGATION] OP VALUE; " + "OP is one of ==, !=, <, <=, >, >=", + param_hint="--constraint", + ) from error + return constraints + + +def _print_result(session, result) -> None: + click.echo(f"Session: {session.session_dir}") + click.echo( + f"Baseline: {result.baseline.request.candidate.id} " + f"({result.baseline.objective.value if result.baseline.objective else 'n/a'})" + ) + if result.best is None: + click.echo("Best: no feasible candidate") + else: + click.echo( + f"Best: {result.best.request.candidate.id} " + f"({result.best.objective.value if result.best.objective else 'n/a'})" + ) + + +async def _run_configured(config_path: Path, *, optimize: bool): + from vero.config import build_configured_runtime, load_config + + runtime = await build_configured_runtime( + load_config(config_path), + optimize=optimize, + ) + result = await runtime.session.run( + skip_baseline_evaluation=runtime.session.manifest_path.exists(), + max_proposals=None if optimize else 0, + ) + return runtime.session, result + + +@click.group() +def main() -> None: + """VeRO: a harness for agents to optimize programs.""" + + +@main.command(name="init") +@click.argument( + "directory", + type=click.Path(path_type=Path, file_okay=False), + default=Path("."), +) +def initialize_config(directory: Path) -> None: + """Create a commented-safe train/validation/test vero.toml starter.""" + + directory.mkdir(parents=True, exist_ok=True) + destination = directory / "vero.toml" + if destination.exists(): + raise click.ClickException(f"configuration already exists: {destination}") + destination.write_text(_CONFIG_TEMPLATE, encoding="utf-8") + click.echo(f"Created {destination}") + + +@main.command(name="check") +@click.option( + "--config", + "config_path", + type=click.Path(path_type=Path, exists=True, dir_okay=False), + default=Path("vero.toml"), + show_default=True, +) +def check_config(config_path: Path) -> None: + """Validate configuration, paths, Git state, and evaluation references.""" + + try: + config = load_config(config_path) + target = Path(config.target.root) + harness = Path(config.backend.harness_root) + if not target.is_dir(): + raise ValueError(f"target root does not exist: {target}") + if not harness.is_dir(): + raise ValueError(f"evaluation harness root does not exist: {harness}") + for name, path in config.backend.staged_inputs.items(): + if not Path(path).exists(): + raise ValueError(f"staged input {name!r} does not exist: {path}") + subprocess.run( + ["git", "rev-parse", "--verify", config.target.ref], + cwd=target, + check=True, + capture_output=True, + text=True, + ) + dirty = subprocess.run( + ["git", "status", "--porcelain"], + cwd=target, + check=True, + capture_output=True, + text=True, + ).stdout + if dirty.strip(): + raise ValueError("target repository has uncommitted changes") + except Exception as error: + raise click.ClickException(str(error) or type(error).__name__) from error + click.echo( + f"Configuration is valid: {len(config.evaluations)} evaluations, " + f"selection={config.protocol.selection_evaluation!r}, " + f"final={config.protocol.final_evaluation!r}" + ) + + +@main.command(name="evaluate") +@click.option( + "--config", + "config_path", + type=click.Path(path_type=Path, exists=True, dir_okay=False), + default=Path("vero.toml"), + show_default=True, +) +def evaluate_config(config_path: Path) -> None: + """Evaluate the configured baseline without producing candidates.""" + + try: + session, result = asyncio.run(_run_configured(config_path, optimize=False)) + except Exception as error: + raise click.ClickException(str(error) or type(error).__name__) from error + _print_result(session, result) + + +@main.command(name="run") +@click.option( + "--config", + "config_path", + type=click.Path(path_type=Path, exists=True, dir_okay=False), + default=Path("vero.toml"), + show_default=True, +) +def run_config(config_path: Path) -> None: + """Run the optimization declared in vero.toml.""" + + try: + session, result = asyncio.run(_run_configured(config_path, optimize=True)) + except Exception as error: + raise click.ClickException(str(error) or type(error).__name__) from error + _print_result(session, result) + + +@main.command(name="report") +@click.argument( + "session_dir", + type=click.Path(path_type=Path, exists=True, file_okay=False), +) +@click.option( + "--output", + type=click.Path(path_type=Path, dir_okay=False), + help="Portable HTML path; defaults to SESSION_DIR/experiment.html.", +) +def report_session(session_dir: Path, output: Path | None) -> None: + """Build a self-contained visual report for an optimization session.""" + + from vero.report import generate_experiment_report + + try: + destination = asyncio.run(generate_experiment_report(session_dir, output)) + except Exception as error: + raise click.ClickException(str(error) or type(error).__name__) from error + click.echo(destination) + + +@main.command() +@click.argument( + "project_path", + type=click.Path(path_type=Path, exists=True, file_okay=False), +) +@click.option( + "--harness-root", + type=click.Path(path_type=Path, exists=True, file_okay=False), + required=True, + help="Trusted directory containing the evaluation harness.", +) +@click.option( + "--evaluate", + "evaluation_command", + required=True, + help="Evaluation argv with placeholders such as {workspace} and {report}.", +) +@click.option( + "--producer-root", + type=click.Path(path_type=Path, exists=True, file_okay=False), + help="Trusted directory containing an external candidate producer.", +) +@click.option( + "--produce", + "producer_command", + help="Producer argv with placeholders such as {workspace} and {instruction}.", +) +@click.option( + "--agent", + type=click.Choice(["claude", "vero"]), + help="Use a built-in coding-agent producer instead of --produce.", +) +@click.option("--instruction", help="Instruction given to the candidate producer.") +@click.option("--metric", required=True, help="Metric to optimize.") +@click.option( + "--aggregation", + type=click.Choice([value.value for value in MetricAggregation]), + default=MetricAggregation.REPORT.value, + show_default=True, +) +@click.option( + "--case-failure-value", + type=float, + help="Value assigned to failed or missing cases during case aggregation.", +) +@click.option( + "--direction", + type=click.Choice(["maximize", "minimize"]), + required=True, +) +@click.option("--failure-value", type=float) +@click.option( + "--constraint", + type=(str, str, str), + multiple=True, + metavar="METRIC[:AGGREGATION] OP VALUE", + help="Feasibility constraint; repeat for multiple constraints.", +) +@click.option("--evaluation-set", default="default", show_default=True) +@click.option("--partition") +@click.option("--case-id", multiple=True, help="Evaluate only this case; repeatable.") +@click.option("--case-start", default=0, type=click.IntRange(min=0), show_default=True) +@click.option("--case-stop", type=click.IntRange(min=1)) +@click.option( + "--parameter", + multiple=True, + help="Evaluation parameter as NAME=JSON; repeat for multiple values.", +) +@click.option( + "--evaluation-env", + multiple=True, + help="Environment variable to pass through to the evaluation harness.", +) +@click.option( + "--evaluation-variable", + multiple=True, + help="Fixed harness environment variable as NAME=VALUE; repeatable.", +) +@click.option( + "--producer-env", + multiple=True, + help="Environment variable to pass through to an external producer.", +) +@click.option( + "--producer-variable", + multiple=True, + help="Fixed producer environment variable as NAME=VALUE; repeatable.", +) +@click.option("--evaluation-working-directory", default=".", show_default=True) +@click.option("--producer-working-directory", default=".", show_default=True) +@click.option( + "--target-ref", + default="HEAD", + show_default=True, + help="Git ref to use as the immutable baseline.", +) +@click.option( + "--session-dir", + type=click.Path(path_type=Path, file_okay=False), + help="Durable output directory; defaults to $VERO_HOME/sessions/.", +) +@click.option("--session-id", help="Stable session identity.") +@click.option( + "--max-proposals", default=1, type=click.IntRange(min=0), show_default=True +) +@click.option( + "--max-rounds", default=100, type=click.IntRange(min=1), show_default=True +) +@click.option( + "--max-concurrency", default=1, type=click.IntRange(min=1), show_default=True +) +@click.option("--max-turns", default=200, type=click.IntRange(min=1), show_default=True) +@click.option( + "--evaluation-timeout", + "--timeout", + "evaluation_timeout", + default=600.0, + type=click.FloatRange(min=0, min_open=True), + show_default=True, + help="Overall timeout for one evaluation. --timeout is a deprecated alias.", +) +@click.option( + "--producer-timeout", + default=600.0, + type=click.FloatRange(min=0, min_open=True), + show_default=True, + help="Timeout for one external command-producer attempt.", +) +@click.option( + "--case-timeout", + default=180.0, + type=click.FloatRange(min=0, min_open=True), + show_default=True, +) +@click.option( + "--evaluation-concurrency", + default=100, + type=click.IntRange(min=1), + show_default=True, +) +@click.option( + "--error-rate-threshold", + default=0.1, + type=click.FloatRange(min=0, max=1, min_open=True), + show_default=True, + help="Fail an evaluation when this fraction of selected cases errors.", +) +@click.option( + "--retry-max-attempts", + default=3, + type=click.IntRange(min=1), + show_default=True, + help="Maximum attempts for a transient per-case failure.", +) +@click.option( + "--retry-initial-delay", + default=4.0, + type=click.FloatRange(min=0), + show_default=True, +) +@click.option( + "--retry-maximum-delay", + default=120.0, + type=click.FloatRange(min=0), + show_default=True, +) +@click.option( + "--retry-multiplier", + default=2.0, + type=click.FloatRange(min=1), + show_default=True, +) +@click.option( + "--retry-on-timeout/--no-retry-on-timeout", + default=True, + show_default=True, +) +@click.option("--seed", type=int) +@click.option("--wandb-project", help="Log the session to this W&B project.") +@click.option("--wandb-entity") +@click.option("--wandb-name") +@click.option( + "--wandb-mode", + type=click.Choice(["online", "offline", "disabled"]), +) +def optimize( + project_path: Path, + harness_root: Path, + evaluation_command: str, + producer_root: Path | None, + producer_command: str | None, + agent: str | None, + instruction: str | None, + metric: str, + aggregation: str, + case_failure_value: float | None, + direction: str, + failure_value: float | None, + constraint: tuple[tuple[str, str, str], ...], + evaluation_set: str, + partition: str | None, + case_id: tuple[str, ...], + case_start: int, + case_stop: int | None, + parameter: tuple[str, ...], + evaluation_env: tuple[str, ...], + evaluation_variable: tuple[str, ...], + producer_env: tuple[str, ...], + producer_variable: tuple[str, ...], + evaluation_working_directory: str, + producer_working_directory: str, + target_ref: str, + session_dir: Path | None, + session_id: str | None, + max_proposals: int, + max_rounds: int, + max_concurrency: int, + max_turns: int, + evaluation_timeout: float, + producer_timeout: float, + case_timeout: float, + evaluation_concurrency: int, + error_rate_threshold: float, + retry_max_attempts: int, + retry_initial_delay: float, + retry_maximum_delay: float, + retry_multiplier: float, + retry_on_timeout: bool, + seed: int | None, + wandb_project: str | None, + wandb_entity: str | None, + wandb_name: str | None, + wandb_mode: str | None, +) -> None: + """Optimize the versioned program at PROJECT_PATH.""" + + producer_count = int(producer_command is not None) + int(agent is not None) + if producer_count > 1 or (max_proposals > 0 and producer_count != 1): + raise click.UsageError( + "provide exactly one of --produce or --agent when producing candidates" + ) + if producer_command is not None and producer_root is None: + raise click.UsageError("--producer-root is required with --produce") + if producer_command is None and producer_root is not None: + raise click.UsageError("--producer-root is only valid with --produce") + if agent is None and max_turns != 200: + raise click.UsageError("--max-turns is only valid with --agent") + if producer_command is None and ( + producer_env + or producer_variable + or producer_working_directory != "." + or producer_timeout != 600.0 + ): + raise click.UsageError( + "--producer-env, --producer-variable, --producer-working-directory, " + "and --producer-timeout are only valid with --produce" + ) + if wandb_project is None and any( + value is not None for value in (wandb_entity, wandb_name, wandb_mode) + ): + raise click.UsageError( + "--wandb-entity, --wandb-name, and --wandb-mode require --wandb-project" + ) + if case_id and case_stop is not None: + raise click.UsageError("--case-id cannot be combined with --case-stop") + if case_stop is None and case_start != 0: + raise click.UsageError("--case-start requires --case-stop") + if case_stop is not None and case_stop <= case_start: + raise click.UsageError("--case-stop must be greater than --case-start") + + if case_id: + selection = CaseIds(ids=list(case_id)) + elif case_stop is not None: + selection = CaseRange(start=case_start, stop=case_stop) + else: + selection = AllCases() + + if session_id is None and session_dir is not None: + manifest_path = session_dir / "manifest.json" + if manifest_path.exists(): + session_id = SessionManifest.model_validate_json( + manifest_path.read_text(encoding="utf-8") + ).id + resolved_session_id = session_id or ( + session_dir.name if session_dir is not None else str(uuid4()) + ) + resolved_session_dir = ( + session_dir.resolve() + if session_dir is not None + else _default_home() / "sessions" / resolved_session_id + ) + + async def run(): + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(harness_root.resolve()), + command=_command(evaluation_command, "--evaluate"), + working_directory=evaluation_working_directory, + environment=_parse_environment( + evaluation_variable, option="--evaluation-variable" + ), + passthrough_environment=list(evaluation_env), + ) + ) + if producer_command is not None: + assert producer_root is not None + producer = CommandCandidateProducer( + CommandCandidateProducerConfig( + root=str(producer_root.resolve()), + command=_command(producer_command, "--produce"), + working_directory=producer_working_directory, + environment=_parse_environment( + producer_variable, option="--producer-variable" + ), + passthrough_environment=list(producer_env), + timeout_seconds=producer_timeout, + ) + ) + elif agent is not None: + from vero.agents import AgentCandidateProducer + + if agent == "claude": + from vero.agents import ClaudeCodeAgent + + coding_agent = ClaudeCodeAgent() + else: + from vero.agents import VeroAgent + + coding_agent = VeroAgent() + producer = AgentCandidateProducer( + coding_agent, + prompt=instruction, + max_turns=max_turns, + ) + else: + producer = None + + session = await create_local_optimization_session( + project_path=project_path, + session_dir=resolved_session_dir, + session_id=resolved_session_id, + backend_id="command", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector( + metric=metric, + aggregation=MetricAggregation(aggregation), + case_failure_value=case_failure_value, + ), + direction=direction, + failure_value=failure_value, + constraints=_parse_constraints(constraint), + ), + evaluation_plan=EvaluationPlan.single( + EvaluationSet( + name=evaluation_set, + partition=partition, + selection=selection, + ) + ), + strategy=SequentialStrategy(instruction=instruction), + producers={"default": producer} if producer is not None else {}, + parameters=_parse_parameters(parameter), + limits=EvaluationLimits( + timeout_seconds=evaluation_timeout, + case_timeout_seconds=case_timeout, + max_concurrency=evaluation_concurrency, + error_rate_threshold=error_rate_threshold, + retry=RetryPolicy( + max_attempts=retry_max_attempts, + initial_delay_seconds=retry_initial_delay, + maximum_delay_seconds=retry_maximum_delay, + multiplier=retry_multiplier, + retry_on_timeout=retry_on_timeout, + ), + ), + seed=seed, + max_proposals=max_proposals, + max_rounds=max_rounds, + max_concurrency=max_concurrency, + base_ref=target_ref, + metadata={"project_path": str(project_path.resolve())}, + ) + if wandb_project is not None: + assert session.events is not None + session.events.sinks.append( + WandbEventSink( + project=wandb_project, + entity=wandb_entity, + name=wandb_name, + mode=wandb_mode, + session_id=session.id, + session_dir=session.session_dir, + config={ + "vero/target": str(project_path.resolve()), + "vero/evaluation_set": evaluation_set, + "vero/objective_metric": metric, + "vero/objective_direction": direction, + }, + ) + ) + result = await session.run( + skip_baseline_evaluation=session.manifest_path.exists() + ) + return session, result + + try: + session, result = asyncio.run(run()) + except click.ClickException: + raise + except Exception as error: + raise click.ClickException(str(error) or type(error).__name__) from error + + _print_result(session, result) + + +@main.group() +def session() -> None: + """Inspect durable optimization sessions.""" + + +@session.command(name="list") +@click.option( + "--root", + type=click.Path(path_type=Path, file_okay=False), + help="Sessions directory; defaults to $VERO_HOME/sessions.", +) +def session_list(root: Path | None) -> None: + """List session manifests.""" + + root = root.resolve() if root is not None else _default_home() / "sessions" + if not root.exists(): + click.echo("No sessions found.") + return + manifests = sorted(root.rglob("manifest.json")) + if not manifests: + click.echo("No sessions found.") + return + for path in manifests: + try: + manifest = SessionManifest.model_validate_json( + path.read_text(encoding="utf-8") + ) + click.echo( + f"{manifest.id}\t{manifest.status.value}\t" + f"{manifest.best_candidate_id or '-'}\t{path.parent.relative_to(root)}" + ) + except Exception as error: + click.echo(f"{path.parent.relative_to(root)}\tinvalid\t{error}") + + +@session.command(name="inspect") +@click.argument( + "session_dir", + type=click.Path(path_type=Path, exists=True, file_okay=False), +) +def session_inspect(session_dir: Path) -> None: + """Print a canonical session manifest and evaluation summaries as JSON.""" + + manifest_path = session_dir / "manifest.json" + if not manifest_path.exists(): + raise click.ClickException(f"session manifest not found: {manifest_path}") + try: + manifest = SessionManifest.model_validate_json( + manifest_path.read_text(encoding="utf-8") + ) + except Exception as error: + raise click.ClickException(f"invalid session manifest: {error}") from error + database_path = session_dir / "database.json" + try: + database = ( + EvaluationDatabase.load_from_file(database_path) + if database_path.exists() + else EvaluationDatabase.from_evaluations_dir( + session_dir / "evaluations", + database_id=manifest.id, + ) + ) + except Exception as error: + raise click.ClickException(f"invalid evaluation database: {error}") from error + evaluations = sorted( + database.evaluations.values(), + key=lambda record: (record.completed_at, record.id), + ) + try: + if manifest.candidate_repository_family != "git": + raise ValueError( + "unsupported candidate repository family: " + f"{manifest.candidate_repository_family}" + ) + candidate_repository = asyncio.run( + GitCandidateRepository.open(session_dir / "candidates") + ) + candidates = candidate_repository.list() + except Exception as error: + raise click.ClickException(f"invalid candidate repository: {error}") from error + click.echo( + json.dumps( + { + "manifest": manifest.model_dump(mode="json"), + "candidates": [ + candidate.model_dump(mode="json") for candidate in candidates + ], + "evaluations": [ + project_evaluation(record, DisclosureLevel.AGGREGATE).model_dump( + mode="json" + ) + for record in evaluations + ], + }, + ensure_ascii=False, + indent=2, + ) + ) + + +@session.command(name="export") +@click.argument( + "session_dir", + type=click.Path(path_type=Path, exists=True, file_okay=False), +) +@click.option( + "--output", + type=click.Path(path_type=Path, dir_okay=False), + help="Archive path without, or with, a .tar.gz suffix.", +) +def session_export(session_dir: Path, output: Path | None) -> None: + """Export complete durable session state as a portable tar.gz archive.""" + + session_dir = session_dir.resolve() + if not (session_dir / "manifest.json").is_file(): + raise click.ClickException("session manifest not found") + destination = (output or session_dir.with_name(f"{session_dir.name}-export")) + destination = destination.expanduser().resolve() + archive_base = str(destination) + if archive_base.endswith(".tar.gz"): + archive_base = archive_base[: -len(".tar.gz")] + try: + archive = shutil.make_archive( + archive_base, + "gztar", + root_dir=session_dir.parent, + base_dir=session_dir.name, + ) + except Exception as error: + raise click.ClickException(str(error) or type(error).__name__) from error + click.echo(archive) + + +@session.command(name="fork") +@click.argument( + "source", + type=click.Path(path_type=Path, exists=True, file_okay=False), +) +@click.argument( + "destination", + type=click.Path(path_type=Path, file_okay=False), +) +@click.option("--session-id", help="New session ID; defaults to destination name.") +@click.option( + "--max-proposals", + type=click.IntRange(min=0), + help="New protocol proposal limit; edit vero.toml to the same value.", +) +@click.option( + "--reset-budgets", + is_flag=True, + help="Restore configured agent/system budgets instead of carrying balances.", +) +def session_fork( + source: Path, + destination: Path, + session_id: str | None, + max_proposals: int | None, + reset_budgets: bool, +) -> None: + """Fork durable candidates and evaluations into a new resumable session.""" + + source = source.resolve() + destination = destination.expanduser().resolve() + if destination.exists(): + raise click.ClickException(f"destination already exists: {destination}") + if destination.is_relative_to(source): + raise click.ClickException("fork destination must not be inside the source") + try: + manifest = SessionManifest.model_validate_json( + (source / "manifest.json").read_text(encoding="utf-8") + ) + new_id = session_id or destination.name + if not new_id.strip(): + raise ValueError("session ID must not be empty") + shutil.copytree(source, destination) + run = manifest.run.model_copy( + update=( + {"max_proposals": max_proposals} + if max_proposals is not None + else {} + ) + ) + forked_at = datetime.now(UTC) + forked = manifest.model_copy( + update={ + "id": new_id, + "status": SessionStatus.CREATED, + "run": run, + "created_at": forked_at, + "updated_at": forked_at, + "failure": None, + "metadata": { + **manifest.metadata, + "forked_from_session_id": manifest.id, + }, + } + ) + (destination / "manifest.json").write_text( + forked.model_dump_json(indent=2), + encoding="utf-8", + ) + database_path = destination / "database.json" + if database_path.exists(): + database = json.loads(database_path.read_text(encoding="utf-8")) + database["id"] = new_id + database_path.write_text( + json.dumps(database, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + for transient in ("events.jsonl", "agent-context.json"): + (destination / transient).unlink(missing_ok=True) + wandb_state = destination / "artifacts" / "wandb" + if wandb_state.exists(): + shutil.rmtree(wandb_state) + if reset_budgets: + budget_path = destination / "budgets.json" + if forked.evaluation_plan.budgets: + BudgetLedger( + forked.evaluation_plan.budgets, + path=budget_path, + ).save() + else: + budget_path.unlink(missing_ok=True) + except Exception as error: + if destination.exists(): + shutil.rmtree(destination) + raise click.ClickException(str(error) or type(error).__name__) from error + click.echo(f"Forked {manifest.id} to {new_id} at {destination}") + + +@session.command(name="clear") +@click.argument( + "session_dir", + type=click.Path(path_type=Path, exists=True, file_okay=False), +) +@click.option("--yes", is_flag=True, help="Confirm permanent deletion.") +def session_clear(session_dir: Path, yes: bool) -> None: + """Permanently delete one session's control-plane state.""" + + if not yes: + raise click.UsageError("session clear requires --yes") + session_dir = session_dir.resolve() + if not (session_dir / "manifest.json").is_file(): + raise click.ClickException("refusing to clear a directory without manifest.json") + shutil.rmtree(session_dir) + click.echo(f"Deleted {session_dir}") + + +# harbor subcommand registered here, after `main` is defined +from vero.harbor.cli import harbor as harbor_command # noqa: E402, I001 + +main.add_command(harbor_command) + + +if __name__ == "__main__": + main() diff --git a/vero/src/vero/config.py b/vero/src/vero/config.py new file mode 100644 index 00000000..a87b2cba --- /dev/null +++ b/vero/src/vero/config.py @@ -0,0 +1,545 @@ +"""Declarative configuration for generic program optimization.""" + +from __future__ import annotations + +import hashlib +import json +import os +import tomllib +from dataclasses import dataclass +from pathlib import Path +from typing import Annotated, Literal +from uuid import uuid4 + +from pydantic import Field, JsonValue, model_validator + +from vero.evaluation import ( + AgentSelectionMode, + AllCases, + CaseIds, + CaseRange, + CommandBackend, + CommandBackendConfig, + ConstraintOperator, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationBudget, + EvaluationDefinition, + EvaluationLimits, + EvaluationPlan, + EvaluationPrincipal, + EvaluationSet, + MetricAggregation, + MetricConstraint, + MetricSelector, + ObjectiveSpec, + RetryPolicy, +) +from vero.models import StrictModel +from vero.optimization import ( + CommandCandidateProducer, + CommandCandidateProducerConfig, + SequentialStrategy, +) +from vero.runtime import ( + OptimizationComponentSpec, + OptimizationRunSpec, + OptimizationSession, + SessionManifest, + WandbEventSink, + create_local_optimization_session, +) + + +class TargetConfig(StrictModel): + root: str + ref: str = "HEAD" + + +class BackendConfig(StrictModel): + id: str = "command" + kind: Literal["command"] = "command" + harness_root: str + command: list[str] + working_directory: str = "." + environment: dict[str, str] = Field(default_factory=dict) + passthrough_environment: list[str] = Field(default_factory=list) + staged_inputs: dict[str, str] = Field(default_factory=dict) + agent_context_inputs: dict[str, list[str]] = Field(default_factory=dict) + + +class BudgetConfig(StrictModel): + total_runs: int | None = Field(default=None, ge=0) + total_cases: int | None = Field(default=None, ge=0) + + def to_model( + self, + *, + backend_id: str, + evaluation_set: EvaluationSet, + principal: EvaluationPrincipal, + ) -> EvaluationBudget: + return EvaluationBudget( + backend_id=backend_id, + evaluation_set_key=evaluation_set.budget_key(backend_id), + principal=principal, + total_runs=self.total_runs, + total_cases=self.total_cases, + ) + + +class EvaluationConfig(StrictModel): + name: str + partition: str | None = None + case_ids: list[str] | None = None + case_start: int = Field(default=0, ge=0) + case_stop: int | None = Field(default=None, ge=1) + agent_can_evaluate: bool = True + agent_visible: bool = True + agent_selection: AgentSelectionMode = AgentSelectionMode.ARBITRARY + disclosure: DisclosureLevel = DisclosureLevel.FULL + expose_case_resources: bool = False + # Omitted resolves to 5 under aggregate disclosure, 1 otherwise. + min_aggregate_cases: int | None = Field(default=None, ge=1) + agent_budget: BudgetConfig | None = None + system_budget: BudgetConfig | None = None + + @model_validator(mode="after") + def validate_selection(self) -> EvaluationConfig: + if self.case_ids is not None and ( + self.case_start != 0 or self.case_stop is not None + ): + raise ValueError("case_ids and case range cannot both be configured") + if self.case_ids is None and self.case_stop is None and self.case_start != 0: + raise ValueError("case_start requires case_stop") + if self.case_stop is not None and self.case_stop <= self.case_start: + raise ValueError("case_stop must be greater than case_start") + return self + + def to_evaluation_set(self) -> EvaluationSet: + if self.case_ids is not None: + selection = CaseIds(ids=self.case_ids) + elif self.case_stop is not None: + selection = CaseRange(start=self.case_start, stop=self.case_stop) + else: + selection = AllCases() + return EvaluationSet( + name=self.name, + partition=self.partition, + selection=selection, + ) + + def to_definition(self, backend_id: str) -> EvaluationDefinition: + evaluation_set = self.to_evaluation_set() + return EvaluationDefinition( + evaluation_set=evaluation_set, + access=EvaluationAccessPolicy( + agent_can_evaluate=self.agent_can_evaluate, + agent_visible=self.agent_visible, + agent_selection=self.agent_selection, + disclosure=self.disclosure, + expose_case_resources=self.expose_case_resources, + min_aggregate_cases=self.min_aggregate_cases, + ), + agent_budget=( + self.agent_budget.to_model( + backend_id=backend_id, + evaluation_set=evaluation_set, + principal=EvaluationPrincipal.AGENT, + ) + if self.agent_budget is not None + else None + ), + system_budget=( + self.system_budget.to_model( + backend_id=backend_id, + evaluation_set=evaluation_set, + principal=EvaluationPrincipal.SYSTEM, + ) + if self.system_budget is not None + else None + ), + ) + + +class ProtocolConfig(StrictModel): + selection_evaluation: str + final_evaluation: str | None = None + evaluate_final_baseline: bool = True + parameters: dict[str, JsonValue] = Field(default_factory=dict) + seed: int | None = None + timeout_seconds: float = Field(default=600.0, gt=0) + case_timeout_seconds: float = Field(default=180.0, gt=0) + evaluation_concurrency: int = Field(default=100, ge=1) + error_rate_threshold: float | None = Field(default=0.1, gt=0, le=1) + retry: RetryPolicy = Field(default_factory=RetryPolicy) + max_proposals: int = Field(default=1, ge=0) + max_rounds: int = Field(default=100, ge=1) + max_concurrency: int = Field(default=1, ge=1) + + def to_limits(self) -> EvaluationLimits: + return EvaluationLimits( + timeout_seconds=self.timeout_seconds, + case_timeout_seconds=self.case_timeout_seconds, + max_concurrency=self.evaluation_concurrency, + error_rate_threshold=self.error_rate_threshold, + retry=self.retry, + ) + + +class ObjectiveConstraintConfig(StrictModel): + metric: str + aggregation: MetricAggregation = MetricAggregation.REPORT + case_failure_value: float | None = None + operator: ConstraintOperator + value: float + + def to_model(self) -> MetricConstraint: + return MetricConstraint( + selector=MetricSelector( + metric=self.metric, + aggregation=self.aggregation, + case_failure_value=self.case_failure_value, + ), + operator=self.operator, + value=self.value, + ) + + +class ObjectiveConfig(StrictModel): + metric: str + aggregation: MetricAggregation = MetricAggregation.REPORT + case_failure_value: float | None = None + direction: Literal["maximize", "minimize"] + failure_value: float | None = None + constraints: list[ObjectiveConstraintConfig] = Field(default_factory=list) + + def to_model(self) -> ObjectiveSpec: + return ObjectiveSpec( + selector=MetricSelector( + metric=self.metric, + aggregation=self.aggregation, + case_failure_value=self.case_failure_value, + ), + direction=self.direction, + failure_value=self.failure_value, + constraints=[constraint.to_model() for constraint in self.constraints], + ) + + +class BaseOptimizerConfig(StrictModel): + instruction: str | None = None + + +class CommandOptimizerConfig(BaseOptimizerConfig): + kind: Literal["command"] = "command" + root: str = "." + command: list[str] + working_directory: str = "." + environment: dict[str, str] = Field(default_factory=dict) + passthrough_environment: list[str] = Field(default_factory=list) + timeout_seconds: float = Field(default=600.0, gt=0) + description: str = "Optimize candidate" + + @model_validator(mode="after") + def validate_command(self) -> CommandOptimizerConfig: + if not self.command: + raise ValueError("command optimizer requires a non-empty command") + return self + + +class AgentOptimizerConfig(BaseOptimizerConfig): + kind: Literal["vero", "claude"] + model: str | None = None + max_turns: int = Field(default=200, ge=1) + + @model_validator(mode="after") + def validate_model(self) -> AgentOptimizerConfig: + if self.model is not None and not self.model.strip(): + raise ValueError("agent optimizer model must not be empty") + return self + + +OptimizerConfig = Annotated[ + CommandOptimizerConfig | AgentOptimizerConfig, + Field(discriminator="kind"), +] + + +class SessionConfig(StrictModel): + id: str | None = None + directory: str | None = None + + +class WandbConfig(StrictModel): + project: str + run_id: str | None = None + entity: str | None = None + name: str | None = None + group: str | None = None + tags: list[str] = Field(default_factory=list) + mode: Literal["online", "offline", "disabled"] | None = None + notes: str | None = None + config: dict[str, JsonValue] = Field(default_factory=dict) + + +class VeroConfig(StrictModel): + target: TargetConfig + backend: BackendConfig + evaluations: list[EvaluationConfig] + protocol: ProtocolConfig + objective: ObjectiveConfig + optimizer: OptimizerConfig | None = None + session: SessionConfig = Field(default_factory=SessionConfig) + wandb: WandbConfig | None = None + + @model_validator(mode="after") + def validate_plan(self) -> VeroConfig: + self.to_evaluation_plan() + names = {evaluation.name for evaluation in self.evaluations} + unknown_context = sorted(set(self.backend.agent_context_inputs) - names) + if unknown_context: + raise ValueError( + "backend.agent_context_inputs references unknown evaluations: " + f"{unknown_context}" + ) + return self + + def to_evaluation_plan(self) -> EvaluationPlan: + return EvaluationPlan( + evaluations=[ + evaluation.to_definition(self.backend.id) + for evaluation in self.evaluations + ], + selection_evaluation=self.protocol.selection_evaluation, + final_evaluation=self.protocol.final_evaluation, + evaluate_final_baseline=self.protocol.evaluate_final_baseline, + ) + + @staticmethod + def _component_spec( + type_name: str, + payload: dict[str, object], + ) -> OptimizationComponentSpec: + encoded = json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + default=str, + ).encode() + return OptimizationComponentSpec( + type=type_name, + config_digest=hashlib.sha256(encoded).hexdigest(), + ) + + def to_run_spec(self) -> OptimizationRunSpec: + producers = {} + if self.optimizer is not None: + producer_type = ( + "vero.optimization.command.CommandCandidateProducer" + if isinstance(self.optimizer, CommandOptimizerConfig) + else "vero.agents.producer.AgentCandidateProducer" + ) + producers["default"] = self._component_spec( + producer_type, + self.optimizer.model_dump(mode="json"), + ) + return OptimizationRunSpec( + max_proposals=( + self.protocol.max_proposals if self.optimizer is not None else 0 + ), + max_rounds=(self.protocol.max_rounds if self.optimizer is not None else 1), + max_concurrency=( + self.protocol.max_concurrency if self.optimizer is not None else 1 + ), + strategy=self._component_spec( + "vero.optimization.strategy.SequentialStrategy", + { + "producer_id": "default", + "instruction": ( + self.optimizer.instruction + if self.optimizer is not None + else None + ), + }, + ), + producers=producers, + ) + + +def load_config(path: Path | str = Path("vero.toml")) -> VeroConfig: + """Load a trusted config and resolve its filesystem paths beside the file.""" + + config_path = Path(path).expanduser().resolve() + with config_path.open("rb") as config_file: + config = VeroConfig.model_validate(tomllib.load(config_file)) + base = config_path.parent + updates: dict[str, object] = { + "target": config.target.model_copy( + update={"root": str((base / config.target.root).resolve())} + ), + "backend": config.backend.model_copy( + update={ + "harness_root": str((base / config.backend.harness_root).resolve()), + "staged_inputs": { + name: str((base / source).resolve()) + for name, source in config.backend.staged_inputs.items() + }, + } + ), + } + if isinstance(config.optimizer, CommandOptimizerConfig): + updates["optimizer"] = config.optimizer.model_copy( + update={"root": str((base / config.optimizer.root).resolve())} + ) + if config.session.directory is not None: + updates["session"] = config.session.model_copy( + update={"directory": str((base / config.session.directory).resolve())} + ) + return config.model_copy(update=updates) + + +@dataclass(frozen=True) +class ConfiguredRuntime: + config: VeroConfig + session: OptimizationSession + producer: object | None + + +def _session_identity(config: VeroConfig) -> tuple[str, Path]: + configured_dir = config.session.directory + if configured_dir is not None: + session_dir = Path(configured_dir).expanduser().resolve() + manifest_path = session_dir / "manifest.json" + if config.session.id is None and manifest_path.exists(): + session_id = SessionManifest.model_validate_json( + manifest_path.read_text(encoding="utf-8") + ).id + else: + session_id = config.session.id or session_dir.name + return session_id, session_dir + session_id = config.session.id or str(uuid4()) + home = Path(os.environ.get("VERO_HOME", "~/.vero")).expanduser().resolve() + return session_id, home / "sessions" / session_id + + +def _producer(config: CommandOptimizerConfig | AgentOptimizerConfig): + if isinstance(config, CommandOptimizerConfig): + return CommandCandidateProducer( + CommandCandidateProducerConfig( + root=config.root, + command=config.command, + working_directory=config.working_directory, + environment=config.environment, + passthrough_environment=config.passthrough_environment, + timeout_seconds=config.timeout_seconds, + description=config.description, + ) + ) + if config.kind == "claude": + from vero.agents import ClaudeCodeAgent + + agent = ClaudeCodeAgent() + if config.model is not None: + agent.options.model = config.model + else: + from vero.agents import VeroAgent + + agent = ( + VeroAgent.for_model(config.model) + if config.model is not None + else VeroAgent() + ) + from vero.agents import AgentCandidateProducer + + return AgentCandidateProducer( + agent, + prompt=config.instruction, + max_turns=config.max_turns, + ) + + +async def build_configured_runtime( + config: VeroConfig, + *, + optimize: bool, +) -> ConfiguredRuntime: + """Compose a local session from a trusted declarative configuration.""" + + target_root = Path(config.target.root) + harness_root = Path(config.backend.harness_root) + if not target_root.is_dir(): + raise ValueError(f"target root does not exist: {target_root}") + if not harness_root.is_dir(): + raise ValueError(f"evaluation harness root does not exist: {harness_root}") + if optimize and config.optimizer is None: + raise ValueError("vero run requires an [optimizer] configuration") + + optimizer_config = config.optimizer if optimize else None + producer = _producer(optimizer_config) if optimizer_config is not None else None + producers = {"default": producer} if producer is not None else {} + session_id, session_dir = _session_identity(config) + backend = CommandBackend( + CommandBackendConfig( + harness_root=config.backend.harness_root, + command=config.backend.command, + working_directory=config.backend.working_directory, + environment=config.backend.environment, + passthrough_environment=config.backend.passthrough_environment, + staged_inputs=config.backend.staged_inputs, + agent_context_inputs=config.backend.agent_context_inputs, + ) + ) + session = await create_local_optimization_session( + project_path=target_root, + session_dir=session_dir, + session_id=session_id, + backend_id=config.backend.id, + backend=backend, + objective=config.objective.to_model(), + evaluation_plan=config.to_evaluation_plan(), + strategy=SequentialStrategy( + instruction=(optimizer_config.instruction if optimizer_config else None) + ), + producers=producers, + parameters=config.protocol.parameters, + limits=config.protocol.to_limits(), + seed=config.protocol.seed, + max_proposals=(config.protocol.max_proposals if optimizer_config else 0), + max_rounds=(config.protocol.max_rounds if optimizer_config else 1), + max_concurrency=(config.protocol.max_concurrency if optimizer_config else 1), + base_ref=config.target.ref, + metadata={ + "config": "vero.toml", + "project_path": str(target_root), + }, + run_spec=config.to_run_spec(), + ) + if config.wandb is not None: + assert session.events is not None + session.events.sinks.append( + WandbEventSink( + project=config.wandb.project, + session_id=session.id, + session_dir=session.session_dir, + run_id=config.wandb.run_id, + entity=config.wandb.entity, + name=config.wandb.name, + group=config.wandb.group, + tags=config.wandb.tags, + mode=config.wandb.mode, + notes=config.wandb.notes, + config={ + **config.wandb.config, + "vero/target": str(target_root), + "vero/evaluation_plan": config.to_evaluation_plan().model_dump( + mode="json" + ), + "vero/objective": config.objective.to_model().model_dump( + mode="json" + ), + }, + ) + ) + return ConfiguredRuntime(config=config, session=session, producer=producer) diff --git a/vero/src/vero/evals_cli.py b/vero/src/vero/evals_cli.py new file mode 100644 index 00000000..006e7ca1 --- /dev/null +++ b/vero/src/vero/evals_cli.py @@ -0,0 +1,743 @@ +"""`evals` — the agent-facing CLI over the `.evals/` evaluation context. + +One well-named entry point for the whole evaluation loop: run an evaluation +(`evals run`, delegating to the metered sidecar endpoint), then navigate the +disclosure-projected results on disk (`evals list/show/cases/trace/diff`), +browse exposed task resources (`evals tasks/task`), and check what may be run +(`evals plan`). + +The viewers are deliberately dumb and unprivileged: `.evals/` is written by the +trusted control plane already projected to the authorized disclosure, so this +module only reads JSON from disk — stdlib + click, no other vero imports. The +runner subcommands import the sidecar client lazily so the viewers work in +contexts without the harbor extra installed. +""" + +from __future__ import annotations + +import json +import os +import time +from datetime import datetime, timezone +from pathlib import Path + +import click + +CONTEXT_DIRECTORY = ".evals" +_CELL_WIDTH = 48 + + +# -------------------------------------------------------------------------- +# Context discovery and shared helpers +# -------------------------------------------------------------------------- + + +def _find_context(explicit: str | None) -> Path: + """Locate the `.evals` context directory. + + Order: --context flag, $VERO_CONTEXT_PATH, then walk up from the working + directory looking for a `.evals/manifest.json`. + """ + if explicit: + path = Path(explicit).expanduser() + if path.name != CONTEXT_DIRECTORY and (path / CONTEXT_DIRECTORY).is_dir(): + path = path / CONTEXT_DIRECTORY + if not path.is_dir(): + raise click.ClickException(f"context directory not found: {path}") + return path + env = os.environ.get("VERO_CONTEXT_PATH") + if env: + path = Path(env) + if path.is_dir(): + return path + raise click.ClickException(f"$VERO_CONTEXT_PATH is not a directory: {path}") + current = Path.cwd() + for candidate in (current, *current.parents): + path = candidate / CONTEXT_DIRECTORY + if (path / "manifest.json").is_file(): + return path + raise click.ClickException( + f"no {CONTEXT_DIRECTORY}/ directory found from {current} upward; " + "pass --context or set $VERO_CONTEXT_PATH" + ) + + +def _load_json(path: Path) -> object: + try: + return json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError: + raise click.ClickException(f"missing context file: {path}") + except json.JSONDecodeError as error: + raise click.ClickException(f"invalid JSON in {path}: {error}") + + +def _result_index(context: Path) -> list[dict]: + document = _load_json(context / "results" / "index.json") + return list(document.get("evaluations", [])) + + +def _resolve_result(context: Path, identifier: str) -> dict: + """Match an index entry by evaluation id (or unique prefix) or path digest.""" + entries = _result_index(context) + matches = [ + entry + for entry in entries + if entry.get("evaluation_id") == identifier + or str(entry.get("evaluation_id", "")).startswith(identifier) + or str(entry.get("path", "")).startswith(identifier) + ] + if len(matches) == 1: + return matches[0] + available = ", ".join(str(entry.get("evaluation_id")) for entry in entries) or "none" + kind = "ambiguous" if matches else "unknown" + raise click.ClickException( + f"{kind} evaluation {identifier!r}; available ids: {available}" + ) + + +def _result_document(context: Path, entry: dict) -> dict: + return _load_json(context / "results" / str(entry["path"])) + + +def _dig(value: object, *keys: str, default: object = None) -> object: + for key in keys: + if not isinstance(value, dict) or key not in value: + return default + value = value[key] + return value + + +def _result_row(context: Path, entry: dict) -> dict: + """One `evals list` row, tolerant of every disclosure shape.""" + document = _result_document(context, entry) + result = document.get("result", {}) + score = _dig(result, "objective", "value") + if score is None: + score = _dig(result, "metrics", "score") + if score is None: + score = _dig(result, "report", "metrics", "score") + case_files = result.get("case_files") + cases = ( + len(case_files) + if isinstance(case_files, list) + else _dig(result, "total_cases") + ) + return { + "id": entry.get("evaluation_id"), + "evaluation": entry.get("evaluation"), + "partition": entry.get("partition"), + "candidate": entry.get("candidate_id"), + "disclosure": entry.get("disclosure"), + "status": _dig(result, "status") or _dig(result, "report", "status"), + "score": score, + "cases": cases, + "errored": _dig(result, "errored_cases"), + "completed_at": _dig(result, "completed_at"), + } + + +def _case_rows(context: Path, entry: dict) -> list[dict]: + document = _result_document(context, entry) + result = document.get("result", {}) + case_files = result.get("case_files") + if not isinstance(case_files, list): + raise click.ClickException( + f"evaluation {entry.get('evaluation_id')!r} has disclosure " + f"{document.get('disclosure')!r}: per-case results are not available" + ) + root = context / "results" / str(entry["path"]) + rows = [] + for case_file in case_files: + case_document = _load_json(root.parent / str(case_file["path"])) + case = case_document.get("result", {}) + errors = case.get("errors") or [] + rows.append( + { + "case_id": case.get("case_id"), + "task": _dig(case, "input", "task_name"), + "status": case.get("status"), + "score": _dig(case, "metrics", "score"), + "error": ( + errors[0].get("code") + if errors + else _dig(case, "output", "error_category") + ), + "trace": bool( + case_document.get("execution_trace_path") + or case_document.get("evaluation_trace_path") + ), + "_path": str(case_file["path"]), + } + ) + return rows + + +def _resolve_case(context: Path, entry: dict, case_identifier: str) -> dict: + rows = _case_rows(context, entry) + matches = [ + row + for row in rows + if str(row["case_id"]) == case_identifier + or str(row["case_id"]).startswith(case_identifier) + ] + if len(matches) == 1: + return matches[0] + available = ", ".join(str(row["case_id"]) for row in rows) or "none" + kind = "ambiguous" if matches else "unknown" + raise click.ClickException( + f"{kind} case {case_identifier!r}; available case ids: {available}" + ) + + +def _format_cell(value: object) -> str: + if value is None: + return "-" + if isinstance(value, float): + return f"{value:.4g}" + text = str(value) + return text if len(text) <= _CELL_WIDTH else text[: _CELL_WIDTH - 1] + "…" + + +def _print_table(rows: list[dict], columns: list[str]) -> None: + if not rows: + click.echo("(no rows)") + return + cells = [[_format_cell(row.get(column)) for column in columns] for row in rows] + widths = [ + max(len(column), max((len(line[index]) for line in cells), default=0)) + for index, column in enumerate(columns) + ] + click.echo(" ".join(column.ljust(widths[i]) for i, column in enumerate(columns))) + for line in cells: + click.echo(" ".join(cell.ljust(widths[i]) for i, cell in enumerate(line))) + + +def _paginate(rows: list[dict], sort: str | None, desc: bool, limit: int, offset: int): + if sort: + rows = sorted( + rows, + key=lambda row: (row.get(sort) is None, row.get(sort)), + reverse=desc, + ) + total = len(rows) + window = rows[offset : offset + limit if limit else None] + return window, total + + +def _emit(rows: list[dict], columns: list[str], total: int, offset: int, as_json: bool): + public = [ + {key: value for key, value in row.items() if not key.startswith("_")} + for row in rows + ] + if as_json: + click.echo(json.dumps(public, indent=2, default=str)) + return + _print_table(public, columns) + if total > len(rows): + click.echo( + f"({total} total; showing {len(rows)} from offset {offset}; " + "use --limit/--offset)" + ) + + +_CONTEXT_OPTION = click.option( + "--context", + "context_path", + help=f"Path to the {CONTEXT_DIRECTORY}/ directory (default: discovered).", +) +_JSON_OPTION = click.option("--json", "as_json", is_flag=True, help="Emit JSON rows.") + + +def _pagination_options(command): + for option in ( + click.option("--sort", help="Column to sort by."), + click.option("--desc", is_flag=True, help="Sort descending."), + click.option("--limit", default=20, show_default=True, type=click.IntRange(0)), + click.option("--offset", default=0, show_default=True, type=click.IntRange(0)), + ): + command = option(command) + return command + + +# -------------------------------------------------------------------------- +# The command group (runner subcommands attach lazily) +# -------------------------------------------------------------------------- + + +_LAZY_RUNNERS = {"run", "result", "submit"} + + +class _EvalsGroup(click.Group): + """Attach sidecar runner commands only when asked for, so the on-disk + viewers work without the harbor extra installed.""" + + def list_commands(self, ctx): + return sorted({*super().list_commands(ctx), *_LAZY_RUNNERS}) + + def get_command(self, ctx, name): + command = super().get_command(ctx, name) + if command is not None or name not in _LAZY_RUNNERS: + return command + try: + from vero.harbor import cli as harbor_cli + except ImportError as error: + raise click.ClickException( + f"`evals {name}` needs the evaluation sidecar client " + f"(vero[harbor]): {error}" + ) + return { + "run": harbor_cli.evaluate_command, + "result": harbor_cli.evaluation_result_command, + "submit": harbor_cli.submit_command, + }[name] + + +@click.group(cls=_EvalsGroup) +def evals() -> None: + """Run evaluations and navigate their results. + + Everything you are authorized to see lives in the read-only `.evals/` + directory: `results/` (past evaluation results), `tasks/` (exposed task + resources), `candidates/` (prior program versions), and `plan.json` + (what you may evaluate, and remaining budget). + + Typical loop: `evals plan` -> edit + commit -> `evals run` (blocks and + returns the result) -> `evals diff BASELINE CANDIDATE` -> + `evals cases ID --sort score` -> `evals trace ID CASE`. Add `--detach` only + to run several evaluations at once, then poll `evals status JOB`. + """ + + +def _parse_ts(value: object) -> datetime | None: + if not value: + return None + try: + parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00")) + except ValueError: + return None + return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc) + + +def _requested_cases(job: dict) -> int | None: + """How many cases the run requested, when it named a subset. + + A whole-partition run carries no selection, so the total is not known + client-side; return None rather than guess. + """ + selection = _dig(job, "evaluation_set", "selection") + if not isinstance(selection, dict): + return None + ids = selection.get("ids") + if isinstance(ids, list): + return len(ids) + start, stop = selection.get("start"), selection.get("stop") + if isinstance(start, int) and isinstance(stop, int): + return max(0, stop - start) + return None + + +def _enrich_job(job: object) -> object: + """Add client-derived elapsed time and requested-case count to a job status. + + A polling agent otherwise sees only `status: running` with no sense of how + long it has run or how large the evaluation is. + """ + if not isinstance(job, dict): + return job + created = _parse_ts(job.get("created_at")) + if created is not None: + ended = _parse_ts(job.get("completed_at")) or datetime.now(timezone.utc) + job["elapsed_seconds"] = round((ended - created).total_seconds(), 1) + requested = _requested_cases(job) + if requested is not None: + job["requested_cases"] = requested + return job + + +def _sidecar_request(): + try: + from vero.harbor.cli import _request + except ImportError as error: # pragma: no cover - exercised via CLI + raise click.ClickException( + f"this command needs the evaluation sidecar client (vero[harbor]): {error}" + ) + return _request + + +@evals.command("status") +@click.argument("job_id", required=False) +def status_command(job_id): + """Show evaluation access and budgets, or one detached job's status. + + For a specific job this also reports `elapsed_seconds` (how long it has run) + and, for a subset run, `requested_cases` — so a poll shows progress, not just + `running`. + """ + request = _sidecar_request() + if job_id: + click.echo(json.dumps(_enrich_job(request("GET", f"/eval/jobs/{job_id}")), indent=2)) + else: + click.echo(json.dumps(request("GET", "/status"), indent=2)) + + +@evals.command("wait") +@click.argument("job_id") +@click.option( + "--poll-interval", + default=15.0, + show_default=True, + type=click.FloatRange(min=1), + help="Seconds between status polls.", +) +@click.option( + "--timeout", + type=click.FloatRange(min=0), + help="Optional max seconds to wait. On expiry, print the current " + "(still-running) status and exit 0 so you can simply call `evals wait` " + "again. Default: wait until the job finishes.", +) +def wait_command(job_id, poll_interval, timeout): + """Block until a detached job finishes, then print its result. + + The blocking companion to `evals run --detach`: one call that waits, so you + never hand-roll a poll loop. Idempotent — safe to call again if it returns + while the job is still running (only happens when --timeout is set). + """ + request = _sidecar_request() + terminal = {"complete", "failed", "cancelled"} + deadline = None if timeout is None else time.monotonic() + timeout + while True: + job = request("GET", f"/eval/jobs/{job_id}") + status = job.get("status") if isinstance(job, dict) else None + if status in terminal: + break + if deadline is not None and time.monotonic() >= deadline: + click.echo(json.dumps(_enrich_job(job), indent=2)) + return + time.sleep(poll_interval) + if status == "complete": + click.echo(json.dumps(request("GET", f"/eval/jobs/{job_id}/result"), indent=2)) + else: + click.echo(json.dumps(_enrich_job(job), indent=2)) + + +@evals.command("list") +@_CONTEXT_OPTION +@_pagination_options +@_JSON_OPTION +def list_command(context_path, sort, desc, limit, offset, as_json): + """List past evaluation results, one row per evaluation.""" + context = _find_context(context_path) + rows = [_result_row(context, entry) for entry in _result_index(context)] + window, total = _paginate(rows, sort, desc, limit, offset) + _emit( + window, + [ + "id", + "evaluation", + "partition", + "candidate", + "disclosure", + "status", + "score", + "cases", + "completed_at", + ], + total, + offset, + as_json, + ) + + +@evals.command("show") +@click.argument("evaluation_id") +@_CONTEXT_OPTION +@click.option("--raw", is_flag=True, help="Dump the full stored document.") +def show_command(evaluation_id, context_path, raw): + """Show one evaluation result in detail.""" + context = _find_context(context_path) + entry = _resolve_result(context, evaluation_id) + document = _result_document(context, entry) + if raw: + click.echo(json.dumps(document, indent=2, default=str)) + return + compact = json.loads(json.dumps(document, default=str)) + result = compact.get("result", {}) + case_files = result.get("case_files") + if isinstance(case_files, list): + result["case_files"] = ( + f"({len(case_files)} cases; use `evals cases {entry['evaluation_id']}`)" + ) + artifacts = _dig(result, "report", "artifacts") + if isinstance(artifacts, list) and artifacts: + result["report"]["artifacts"] = ( + f"({len(artifacts)} artifacts under " + f"results/{entry['path'].rsplit('/', 1)[0]}/artifacts/)" + ) + click.echo(json.dumps(compact, indent=2)) + + +@evals.command("cases") +@click.argument("evaluation_id") +@_CONTEXT_OPTION +@_pagination_options +@_JSON_OPTION +def cases_command(evaluation_id, context_path, sort, desc, limit, offset, as_json): + """List one evaluation's per-case results (full disclosure only).""" + context = _find_context(context_path) + entry = _resolve_result(context, evaluation_id) + rows = _case_rows(context, entry) + window, total = _paginate(rows, sort, desc, limit, offset) + _emit( + window, + ["case_id", "task", "status", "score", "error", "trace"], + total, + offset, + as_json, + ) + + +@evals.command("trace") +@click.argument("evaluation_id") +@click.argument("case_id") +@_CONTEXT_OPTION +@click.option("--span", type=click.IntRange(0), help="Show one span by index.") +@click.option("--chars", default=10_000, show_default=True, type=click.IntRange(1)) +@click.option("--char-offset", default=0, show_default=True, type=click.IntRange(0)) +def trace_command(evaluation_id, case_id, context_path, span, chars, char_offset): + """Summarize a case's trace, or window one span of it. + + Without --span: span count, shapes, and sizes, plus the case's artifact + files (read those directly with your file tools). With --span N: that + span's JSON, windowed by --char-offset/--chars. + """ + context = _find_context(context_path) + entry = _resolve_result(context, evaluation_id) + case_row = _resolve_case(context, entry, case_id) + case_root = (context / "results" / str(entry["path"])).parent / Path( + str(case_row["_path"]) + ).parent + case_document = _load_json(case_root / "result.json") + + trace = [] + trace_name = case_document.get("execution_trace_path") + if trace_name: + loaded = _load_json(case_root / str(trace_name)) + trace = loaded if isinstance(loaded, list) else [loaded] + + if span is not None: + if not trace: + raise click.ClickException("this case has no execution trace file") + if span >= len(trace): + raise click.ClickException( + f"span {span} out of range; trace has {len(trace)} spans" + ) + text = json.dumps(trace[span], indent=2, default=str) + window = text[char_offset : char_offset + chars] + click.echo( + f"span {span}/{len(trace) - 1}, chars {char_offset}-" + f"{char_offset + len(window)} of {len(text)}" + ) + click.echo(window) + return + + summary: dict[str, object] = {"case_id": case_row["case_id"]} + if trace: + shapes: dict[str, dict[str, int]] = {} + total_chars = 0 + for item in trace: + if isinstance(item, dict): + key = "dict(" + ",".join(sorted(item)) + ")" + elif isinstance(item, list): + key = f"list(len={len(item)})" + else: + key = type(item).__name__ + size = len(json.dumps(item, default=str)) + total_chars += size + shape = shapes.setdefault(key, {"count": 0, "chars": 0}) + shape["count"] += 1 + shape["chars"] += size + summary["execution_trace"] = { + "spans": len(trace), + "total_chars": total_chars, + "shapes": shapes, + "read_with": f"evals trace {entry['evaluation_id']} " + f"{case_row['case_id']} --span N", + } + else: + summary["execution_trace"] = None + + artifacts = [] + for artifact in _dig(case_document, "result", "artifacts", default=[]) or []: + relative = str(artifact.get("path", "")) + on_disk = ( + context / "results" / str(entry["path"]) + ).parent / "artifacts" / relative + artifacts.append( + { + "path": f"results/{str(entry['path']).rsplit('/', 1)[0]}" + f"/artifacts/{relative}", + "bytes": on_disk.stat().st_size if on_disk.is_file() else None, + "description": artifact.get("description"), + } + ) + summary["artifacts"] = artifacts + click.echo(json.dumps(summary, indent=2)) + + +@evals.command("diff") +@click.argument("baseline_id") +@click.argument("candidate_id") +@_CONTEXT_OPTION +@_JSON_OPTION +def diff_command(baseline_id, candidate_id, context_path, as_json): + """Per-case score deltas between two evaluations (matched by case id).""" + context = _find_context(context_path) + baseline_entry = _resolve_result(context, baseline_id) + candidate_entry = _resolve_result(context, candidate_id) + baseline = {row["case_id"]: row for row in _case_rows(context, baseline_entry)} + candidate = {row["case_id"]: row for row in _case_rows(context, candidate_entry)} + rows = [] + for case_id in sorted({*baseline, *candidate}, key=str): + before = _dig(baseline.get(case_id, {}), "score") + after = _dig(candidate.get(case_id, {}), "score") + if case_id not in baseline or case_id not in candidate: + verdict = "unmatched" + elif before is None or after is None: + verdict = "unscored" + elif after > before: + verdict = "improved" + elif after < before: + verdict = "regressed" + else: + verdict = "unchanged" + rows.append( + { + "case_id": case_id, + "baseline": before, + "candidate": after, + "delta": (after - before) + if isinstance(before, (int, float)) and isinstance(after, (int, float)) + else None, + "verdict": verdict, + } + ) + counts: dict[str, int] = {} + for row in rows: + counts[row["verdict"]] = counts.get(row["verdict"], 0) + 1 + if as_json: + click.echo(json.dumps({"cases": rows, "summary": counts}, indent=2)) + return + _print_table(rows, ["case_id", "baseline", "candidate", "delta", "verdict"]) + click.echo(json.dumps(counts)) + + +@evals.command("plan") +@_CONTEXT_OPTION +@_JSON_OPTION +def plan_command(context_path, as_json): + """Show the evaluations you may run, their rules, and remaining budget.""" + context = _find_context(context_path) + document = _load_json(context / "plan.json") + rows = [] + for evaluation in document.get("evaluations", []): + budget = evaluation.get("budget") or {} + rows.append( + { + "evaluation": evaluation.get("name"), + "partition": evaluation.get("partition"), + "backend": evaluation.get("backend"), + "cases": evaluation.get("cases"), + "can_evaluate": evaluation.get("agent_can_evaluate"), + "selection": evaluation.get("agent_selection"), + "disclosure": evaluation.get("disclosure"), + "runs_left": budget.get("remaining_runs"), + "cases_left": budget.get("remaining_cases"), + } + ) + if as_json: + click.echo(json.dumps(rows, indent=2)) + return + _print_table( + rows, + [ + "evaluation", + "partition", + "backend", + "cases", + "can_evaluate", + "selection", + "disclosure", + "runs_left", + "cases_left", + ], + ) + + +def _task_sets(context: Path) -> list[dict]: + document = _load_json(context / "tasks" / "index.json") + return list(document.get("case_resources", [])) + + +@evals.command("tasks") +@click.argument("evaluation_set", required=False) +@_CONTEXT_OPTION +@_JSON_OPTION +def tasks_command(evaluation_set, context_path, as_json): + """List exposed task sets, or one set's task ids and resource paths.""" + context = _find_context(context_path) + sets = _task_sets(context) + if evaluation_set is None: + rows = [] + for item in sets: + resources = _load_json( + context / "tasks" / str(item["path"]) / "resources" / "index.json" + ) + rows.append( + { + "evaluation": _dig(item, "evaluation_set", "name"), + "partition": _dig(item, "evaluation_set", "partition"), + "tasks": len(resources.get("cases", [])), + "path": f"tasks/{item['path']}/resources/", + } + ) + if as_json: + click.echo(json.dumps(rows, indent=2)) + else: + _print_table(rows, ["evaluation", "partition", "tasks", "path"]) + return + matches = [ + item + for item in sets + if evaluation_set + in ( + _dig(item, "evaluation_set", "name"), + _dig(item, "evaluation_set", "partition"), + str(item.get("path")), + ) + ] + if len(matches) != 1: + available = ", ".join( + f"{_dig(item, 'evaluation_set', 'name')}/" + f"{_dig(item, 'evaluation_set', 'partition')}" + for item in sets + ) + raise click.ClickException( + f"unknown or ambiguous task set {evaluation_set!r}; available: " + f"{available or 'none'}" + ) + root = f"tasks/{matches[0]['path']}/resources" + resources = _load_json(context / "tasks" / str(matches[0]["path"]) / "resources" / "index.json") + rows = [ + {"case_id": case.get("case_id"), "path": f"{root}/{case.get('path')}"} + for case in resources.get("cases", []) + ] + if as_json: + click.echo(json.dumps(rows, indent=2)) + else: + _print_table(rows, ["case_id", "path"]) + click.echo("(read task files directly with your file tools)") + + +def main() -> None: + evals() diff --git a/vero/src/vero/evaluation/__init__.py b/vero/src/vero/evaluation/__init__.py new file mode 100644 index 00000000..fa612904 --- /dev/null +++ b/vero/src/vero/evaluation/__init__.py @@ -0,0 +1,174 @@ +"""Backend-neutral evaluation: what a score is, and how one is produced. + +Layered, and the layering is the reading order: ``models``/``exceptions`` define +the vocabulary; ``scoring`` turns raw results into a trustworthy number; +``store`` persists records and the budget ledger; ``backends`` execute one case; +``evaluator`` runs one evaluation; ``engine`` authorizes and orchestrates. + +Nothing here knows about containers or trust boundaries — that is +``vero.sidecar``. Everything below is re-exported here, so the subpackages are an +internal grouping rather than a public surface. +""" + +from vero.evaluation.backends.base import ( + BackendRegistry, + CaseResourceExporter, + CaseStore, + EvaluationBackend, + EvaluationContext, +) +from vero.evaluation.backends.command import CommandBackend, CommandBackendConfig +from vero.evaluation.backends.python_task import ( + PythonTaskBackend, + PythonTaskBackendConfig, + PythonTaskEvaluationConfig, +) +from vero.evaluation.engine import ( + AuthorizationResolver, + EvaluationEngine, + allow_all_evaluations, + authorize_evaluation_plan, +) +from vero.evaluation.evaluator import Evaluator +from vero.evaluation.exceptions import ( + EvaluationBudgetExceeded, + EvaluationCancelledError, + EvaluationDeniedError, + EvaluationError, + EvaluationExecutionError, + EvaluationInfrastructureError, + EvaluationRequestError, + EvaluationTerminatedError, + UnknownBackendError, +) +from vero.evaluation.models import ( + AgentSelectionMode, + AllCases, + BackendProvenance, + CaseError, + CaseIds, + CaseRange, + CaseResult, + CaseSelection, + CaseStatus, + CommandEvaluationInput, + ConstraintOperator, + ConstraintViolation, + DiagnosticSeverity, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationAcknowledgement, + EvaluationArtifact, + EvaluationAuthorization, + EvaluationBudget, + EvaluationCost, + EvaluationDefinition, + EvaluationDiagnostic, + EvaluationLimits, + EvaluationPlan, + EvaluationPrincipal, + EvaluationReceipt, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + EvaluationSummary, + MetricAggregation, + MetricConstraint, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, + RetryPolicy, +) +from vero.evaluation.scoring.objective import ( + compare_evaluation_records, + evaluate_objective, + project_evaluation, + resolve_metric, + select_best_evaluation, +) +from vero.evaluation.store.budget import BudgetLedger +from vero.evaluation.store.persistence import ( + CaseCheckpointStore, + EvaluationDatabase, + EvaluationManifest, + EvaluationStore, + RunningEvaluationManifest, +) + +__all__ = [ + "AgentSelectionMode", + "AllCases", + "BackendProvenance", + "BackendRegistry", + "BudgetLedger", + "CaseError", + "CaseIds", + "CaseRange", + "CaseResourceExporter", + "CaseResult", + "CaseSelection", + "CaseStatus", + "CaseStore", + "CaseCheckpointStore", + "CommandEvaluationInput", + "CommandBackend", + "CommandBackendConfig", + "ConstraintOperator", + "ConstraintViolation", + "DiagnosticSeverity", + "DisclosureLevel", + "EvaluationAcknowledgement", + "EvaluationAccessPolicy", + "EvaluationArtifact", + "EvaluationAuthorization", + "EvaluationBackend", + "EvaluationBudget", + "EvaluationDefinition", + "EvaluationBudgetExceeded", + "EvaluationCancelledError", + "EvaluationContext", + "EvaluationCost", + "EvaluationDeniedError", + "EvaluationDiagnostic", + "EvaluationDatabase", + "EvaluationEngine", + "EvaluationError", + "EvaluationExecutionError", + "EvaluationInfrastructureError", + "EvaluationTerminatedError", + "EvaluationLimits", + "EvaluationManifest", + "EvaluationPlan", + "EvaluationPrincipal", + "EvaluationRecord", + "EvaluationReceipt", + "EvaluationRequestError", + "EvaluationReport", + "EvaluationRequest", + "EvaluationSet", + "EvaluationStatus", + "EvaluationSummary", + "EvaluationStore", + "Evaluator", + "MetricAggregation", + "MetricConstraint", + "MetricSelector", + "ObjectiveResult", + "ObjectiveSpec", + "PythonTaskBackend", + "PythonTaskBackendConfig", + "PythonTaskEvaluationConfig", + "RetryPolicy", + "RunningEvaluationManifest", + "UnknownBackendError", + "AuthorizationResolver", + "compare_evaluation_records", + "allow_all_evaluations", + "authorize_evaluation_plan", + "evaluate_objective", + "project_evaluation", + "resolve_metric", + "select_best_evaluation", +] diff --git a/vero/src/vero/evaluation/backends/__init__.py b/vero/src/vero/evaluation/backends/__init__.py new file mode 100644 index 00000000..bac9b085 --- /dev/null +++ b/vero/src/vero/evaluation/backends/__init__.py @@ -0,0 +1,8 @@ +"""Pluggable evaluation execution. + +``base`` defines the ``EvaluationBackend`` protocol and the registry that +resolves one; ``command`` and ``python_task`` are two implementations. A third, +``HarborBackend``, lives in ``vero.harbor.backend`` — it is a peer of these, kept +in that package because it depends on Harbor machinery that ``vero.evaluation`` +must not. +""" diff --git a/vero/src/vero/evaluation/backends/base.py b/vero/src/vero/evaluation/backends/base.py new file mode 100644 index 00000000..22555419 --- /dev/null +++ b/vero/src/vero/evaluation/backends/base.py @@ -0,0 +1,110 @@ +"""Evaluation backend protocol and trusted backend registry. + +Implementations: ``command`` and ``python_task`` here, plus ``HarborBackend`` in +``vero.harbor.backend`` — a peer of those two, kept in that package because it +depends on Harbor machinery this one must not. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol, runtime_checkable + +from vero.evaluation.models import ( + BackendProvenance, + CaseResult, + EvaluationCost, + EvaluationReport, + EvaluationRequest, + EvaluationSet, +) +from vero.sandbox import Sandbox +from vero.workspace import Workspace + + +@runtime_checkable +class CaseStore(Protocol): + """Checkpoint interface available to streaming evaluation backends.""" + + async def save(self, result: CaseResult) -> None: ... + + async def load(self, case_id: str) -> CaseResult | None: ... + + async def load_all(self) -> list[CaseResult]: ... + + +@dataclass(frozen=True) +class EvaluationContext: + workspace: Workspace + session_id: str + evaluation_id: str + result_dir: Path + artifact_dir: Path + case_store: CaseStore + # True for trusted finalization (admin) evaluations — lets a backend route + # target-agent inference to a reserved scope so the optimizer's search + # evaluations cannot starve the mandatory re-score's inference budget. + finalization: bool = False + + +@runtime_checkable +class EvaluationBackend(Protocol): + @property + def provenance(self) -> BackendProvenance: ... + + async def resolve_cost(self, evaluation_set: EvaluationSet) -> EvaluationCost: ... + + async def evaluate( + self, + *, + context: EvaluationContext, + request: EvaluationRequest, + ) -> EvaluationReport: ... + + +@runtime_checkable +class CaseResourceExporter(Protocol): + """Optional trusted export of an agent-visible evaluation-set view.""" + + async def export_case_resources( + self, + *, + evaluation_set: EvaluationSet, + destination: str, + sandbox: Sandbox, + ) -> None: ... + + +class BackendRegistry: + """Registry of trusted, preconfigured evaluation backends.""" + + def __init__(self, backends: dict[str, EvaluationBackend] | None = None): + self._backends: dict[str, EvaluationBackend] = {} + for backend_id, backend in (backends or {}).items(): + self.register(backend_id, backend) + + def register(self, backend_id: str, backend: EvaluationBackend) -> None: + if not backend_id.strip(): + raise ValueError("backend ID must not be empty") + if backend_id in self._backends: + raise ValueError(f"backend ID {backend_id!r} is already registered") + if not isinstance(backend, EvaluationBackend): + raise TypeError("backend does not implement EvaluationBackend") + self._backends[backend_id] = backend + + def resolve(self, backend_id: str) -> EvaluationBackend: + from vero.evaluation.exceptions import UnknownBackendError + + try: + return self._backends[backend_id] + except KeyError as error: + raise UnknownBackendError( + f"unknown evaluation backend: {backend_id!r}" + ) from error + + def __contains__(self, backend_id: str) -> bool: + return backend_id in self._backends + + def __iter__(self): + return iter(self._backends) diff --git a/vero/src/vero/evaluation/backends/command.py b/vero/src/vero/evaluation/backends/command.py new file mode 100644 index 00000000..5980f998 --- /dev/null +++ b/vero/src/vero/evaluation/backends/command.py @@ -0,0 +1,389 @@ +"""Language- and framework-neutral command evaluation backend.""" + +from __future__ import annotations + +import json +import os +import posixpath +import re +from pathlib import Path, PurePosixPath +from typing import Literal + +from pydantic import Field, field_validator, model_validator + +from vero.evaluation.backends.base import EvaluationContext +from vero.evaluation.models import ( + AllCases, + BackendProvenance, + CaseIds, + CaseRange, + CommandEvaluationInput, + DiagnosticSeverity, + EvaluationArtifact, + EvaluationCost, + EvaluationDiagnostic, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, +) +from vero.evaluation.scoring.security import sanitize_evaluation_report, sanitize_text +from vero.models import StrictModel +from vero.sandbox import Sandbox +from vero.staging import SandboxStagingArea + +_PLACEHOLDERS = {"workspace", "harness", "request", "report", "artifacts"} +_PLACEHOLDER_PATTERN = re.compile(r"\{([^{}]+)\}") +_INPUT_NAME_PATTERN = re.compile(r"^[a-zA-Z][a-zA-Z0-9_-]*$") + + +class CommandBackendConfig(StrictModel): + # Discriminates this from the other backend configs a deployment may name. + type: Literal["command"] = "command" + harness_root: str + command: list[str] + working_directory: str = "." + environment: dict[str, str] = Field(default_factory=dict) + passthrough_environment: list[str] = Field(default_factory=list) + staged_inputs: dict[str, str] = Field(default_factory=dict) + agent_context_inputs: dict[str, list[str]] = Field(default_factory=dict) + + @field_validator("harness_root") + @classmethod + def validate_harness_root(cls, value: str) -> str: + if not value.strip(): + raise ValueError("harness_root must not be empty") + if not Path(value).is_absolute(): + raise ValueError("harness_root must be absolute after config resolution") + return value + + @field_validator("command") + @classmethod + def validate_command(cls, value: list[str]) -> list[str]: + if not value or any(not argument for argument in value): + raise ValueError("command and its arguments must not be empty") + unknown = { + placeholder + for argument in value + for placeholder in _PLACEHOLDER_PATTERN.findall(argument) + if placeholder not in _PLACEHOLDERS and not placeholder.startswith("input:") + } + if unknown: + raise ValueError( + f"unknown command placeholders: {', '.join(sorted(unknown))}" + ) + return value + + @field_validator("working_directory") + @classmethod + def validate_working_directory(cls, value: str) -> str: + path = Path(value) + if not value.strip() or path.is_absolute() or ".." in path.parts: + raise ValueError("working_directory must stay within harness_root") + return value + + @field_validator("environment") + @classmethod + def validate_environment(cls, value: dict[str, str]) -> dict[str, str]: + for name in value: + if not name or "=" in name: + raise ValueError(f"invalid environment variable name: {name!r}") + return value + + @field_validator("passthrough_environment") + @classmethod + def validate_passthrough(cls, value: list[str]) -> list[str]: + if len(value) != len(set(value)): + raise ValueError("passthrough_environment names must be unique") + for name in value: + if not name or "=" in name: + raise ValueError(f"invalid environment variable name: {name!r}") + return value + + @model_validator(mode="after") + def validate_environment_sources(self) -> CommandBackendConfig: + overlap = set(self.environment) & set(self.passthrough_environment) + if overlap: + raise ValueError( + "environment and passthrough_environment overlap for: " + + ", ".join(sorted(overlap)) + ) + invalid = sorted( + name + for name in self.staged_inputs + if not _INPUT_NAME_PATTERN.fullmatch(name) + ) + if invalid: + raise ValueError(f"invalid staged input names: {', '.join(invalid)}") + referenced = { + placeholder.removeprefix("input:") + for argument in self.command + for placeholder in _PLACEHOLDER_PATTERN.findall(argument) + if placeholder.startswith("input:") + } + unknown = sorted(referenced - set(self.staged_inputs)) + if unknown: + raise ValueError(f"unknown staged command inputs: {', '.join(unknown)}") + for evaluation, names in self.agent_context_inputs.items(): + if not evaluation.strip(): + raise ValueError("agent_context_inputs evaluation names must not be empty") + if len(names) != len(set(names)): + raise ValueError( + f"agent_context_inputs for {evaluation!r} must be unique" + ) + unknown_context = sorted( + { + name + for names in self.agent_context_inputs.values() + for name in names + } + - set(self.staged_inputs) + ) + if unknown_context: + raise ValueError( + "agent_context_inputs reference unknown staged inputs: " + + ", ".join(unknown_context) + ) + return self + + +class CommandBackend: + """Invoke a trusted external harness through a versioned JSON contract.""" + + name = "command" + version = "1" + + def __init__(self, config: CommandBackendConfig): + self.config = config + + @property + def provenance(self) -> BackendProvenance: + return BackendProvenance.from_config( + name=self.name, + version=self.version, + config=self.config, + ) + + async def resolve_cost(self, evaluation_set: EvaluationSet) -> EvaluationCost: + selection = evaluation_set.selection + if isinstance(selection, CaseIds): + return EvaluationCost(cases=len(selection.ids)) + if isinstance(selection, CaseRange): + return EvaluationCost(cases=selection.stop - selection.start) + if isinstance(selection, AllCases): + return EvaluationCost(cases=None) + raise AssertionError(f"unsupported case selection: {selection}") + + async def export_case_resources( + self, + *, + evaluation_set: EvaluationSet, + destination: str, + sandbox: Sandbox, + ) -> None: + """Copy only explicitly allowlisted staged inputs into agent context.""" + + resources = [] + for name in self.config.agent_context_inputs.get(evaluation_set.name, []): + source = Path(self.config.staged_inputs[name]).resolve() + if not source.exists(): + raise ValueError( + f"agent context input {name!r} does not exist: {source}" + ) + target = str(PurePosixPath(destination) / name) + await sandbox.upload(str(source), target) + resources.append({"name": name, "path": name}) + await sandbox.write_file( + str(PurePosixPath(destination) / "index.json"), + json.dumps( + { + "schema_version": 1, + "evaluation_set": evaluation_set.model_dump(mode="json"), + "resources": resources, + }, + ensure_ascii=False, + indent=2, + ) + + "\n", + ) + + def _working_directory(self, harness_root: str) -> str: + return posixpath.normpath( + posixpath.join(harness_root, self.config.working_directory) + ) + + def _environment(self) -> dict[str, str]: + environment = {"PATH": os.defpath, "LANG": "C.UTF-8"} + for name in ("TMPDIR", "TMP", "TEMP", "SYSTEMROOT"): + if name in os.environ: + environment[name] = os.environ[name] + environment.update(self.config.environment) + for name in self.config.passthrough_environment: + if name in os.environ: + environment[name] = os.environ[name] + return environment + + def _secrets(self) -> list[str]: + values = list(self.config.environment.values()) + values.extend( + os.environ[name] + for name in self.config.passthrough_environment + if name in os.environ + ) + return values + + def sanitize_error(self, message: str) -> str: + return sanitize_text(message, self._secrets()) + + def validate_request(self, request: EvaluationRequest) -> None: + payload = request.model_dump_json() + if any(secret in payload for secret in self._secrets() if len(secret) >= 4): + raise ValueError( + "evaluation parameters must not contain configured secret values; " + "pass secrets through the backend environment" + ) + + def _expand_command(self, values: dict[str, str]) -> list[str]: + command: list[str] = [] + for argument in self.config.command: + expanded = argument + for placeholder, value in values.items(): + expanded = expanded.replace(f"{{{placeholder}}}", value) + command.append(expanded) + return command + + @staticmethod + def _failure_report( + *, + code: str, + message: str, + artifacts: list[EvaluationArtifact], + ) -> EvaluationReport: + return EvaluationReport( + status=EvaluationStatus.FAILED, + diagnostics=[ + EvaluationDiagnostic( + code=code, + message=message, + severity=DiagnosticSeverity.ERROR, + phase="command", + ) + ], + artifacts=artifacts, + ) + + async def evaluate( + self, + *, + context: EvaluationContext, + request: EvaluationRequest, + ) -> EvaluationReport: + harness_source = Path(self.config.harness_root).resolve() + target_root = context.workspace.sandbox.host_path( + context.workspace.project_path + ) + if target_root is not None: + target_root = target_root.resolve() + if harness_source == target_root or harness_source.is_relative_to( + target_root + ): + raise ValueError( + "command harness must live outside the editable target" + ) + + capture_dir = context.artifact_dir / "command" + capture_dir.mkdir(parents=True, exist_ok=True) + async with SandboxStagingArea( + context.workspace.sandbox, + prefix=f"vero-eval-{context.evaluation_id[:8]}-", + ) as staging: + harness_root = ( + str(harness_source) + if context.workspace.sandbox.capabilities.host_paths + else await staging.upload(harness_source, "harness") + ) + staged_inputs = { + f"input:{name}": await staging.upload(source, f"inputs/{name}") + for name, source in self.config.staged_inputs.items() + } + request_path = await staging.write_text( + "request.json", + CommandEvaluationInput(request=request).model_dump_json(indent=2), + ) + report_path = staging.path("report.json") + artifacts_path = await staging.mkdir("artifacts") + + command = self._expand_command( + { + "workspace": context.workspace.project_path, + "harness": harness_root, + "request": request_path, + "report": report_path, + "artifacts": artifacts_path, + **staged_inputs, + } + ) + result = await context.workspace.sandbox.run( + command, + cwd=self._working_directory(harness_root), + timeout=request.limits.timeout_seconds, + env=self._environment(), + ) + + if await staging.exists("artifacts"): + await staging.download("artifacts", context.artifact_dir) + + report_payload = ( + await staging.read_text("report.json") + if await staging.exists("report.json") + else None + ) + + stdout = self.sanitize_error(result.stdout) + stderr = self.sanitize_error(result.stderr) + (capture_dir / "stdout.log").write_text(stdout, encoding="utf-8") + (capture_dir / "stderr.log").write_text(stderr, encoding="utf-8") + capture_artifacts = [ + EvaluationArtifact( + path="command/stdout.log", + media_type="text/plain", + description="Command harness standard output", + ), + EvaluationArtifact( + path="command/stderr.log", + media_type="text/plain", + description="Command harness standard error", + ), + ] + + if result.returncode != 0: + code = "command_timeout" if result.returncode == -1 else "command_failed" + message = ( + stderr.strip() + or f"evaluation command exited with status {result.returncode}" + ) + return self._failure_report( + code=code, + message=message, + artifacts=capture_artifacts, + ) + if report_payload is None: + return self._failure_report( + code="missing_report", + message="evaluation command did not write a report", + artifacts=capture_artifacts, + ) + try: + report = EvaluationReport.model_validate_json(report_payload) + except Exception as error: + return self._failure_report( + code="invalid_report", + message=self.sanitize_error( + f"evaluation command wrote an invalid report: {error}" + ), + artifacts=capture_artifacts, + ) + report = sanitize_evaluation_report(report, self._secrets()) + return report.model_copy( + update={"artifacts": [*report.artifacts, *capture_artifacts]} + ) diff --git a/vero/src/vero/evaluation/backends/python_task.py b/vero/src/vero/evaluation/backends/python_task.py new file mode 100644 index 00000000..509a02ac --- /dev/null +++ b/vero/src/vero/evaluation/backends/python_task.py @@ -0,0 +1,328 @@ +"""Optional uv-based adapter for Python tasks defined with scale-vero-tasks.""" + +from __future__ import annotations + +import hashlib +import json +import shutil +from pathlib import Path, PurePosixPath + +from pydantic import Field, field_validator, model_validator + +from vero.evaluation.backends.base import EvaluationContext +from vero.evaluation.backends.command import CommandBackend, CommandBackendConfig +from vero.evaluation.models import ( + AllCases, + BackendProvenance, + CaseIds, + CaseRange, + EvaluationCost, + EvaluationReport, + EvaluationRequest, + EvaluationSet, +) +from vero.models import StrictModel +from vero.sandbox import Sandbox + + +def _default_uv() -> str: + executable = shutil.which("uv") + if executable is None: + raise ValueError("uv is required to configure a Python task backend") + return str(Path(executable).resolve()) + + +class PythonTaskEvaluationConfig(StrictModel): + """One named/partitioned dataset owned by a Python task backend.""" + + name: str + cases_path: str + partition: str | None = None + + @field_validator("name") + @classmethod + def validate_name(cls, value: str) -> str: + if not value.strip(): + raise ValueError("Python task evaluation name must not be empty") + return value + + @field_validator("cases_path") + @classmethod + def validate_cases_path(cls, value: str) -> str: + if not value.strip() or not Path(value).is_absolute(): + raise ValueError("Python task cases_path must be absolute") + return value + + @field_validator("partition") + @classmethod + def validate_partition(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("Python task partition must not be empty") + return value + + +class PythonTaskBackendConfig(StrictModel): + """Configuration for an external task harness and editable target package.""" + + harness_root: str + module: str + task: str + evaluations: list[PythonTaskEvaluationConfig] + target_project_directory: str = "." + uv_executable: str = Field(default_factory=_default_uv) + python_executable: str = "python" + environment: dict[str, str] = Field(default_factory=dict) + passthrough_environment: list[str] = Field(default_factory=list) + + @field_validator("harness_root") + @classmethod + def validate_absolute_path(cls, value: str) -> str: + if not value.strip() or not Path(value).is_absolute(): + raise ValueError("Python task backend paths must be absolute") + return value + + @field_validator("module", "task", "python_executable") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("Python task backend identity must not be empty") + return value + + @field_validator("uv_executable") + @classmethod + def validate_uv_executable(cls, value: str) -> str: + if not value.strip(): + raise ValueError("uv_executable must not be empty") + return value + + @field_validator("target_project_directory") + @classmethod + def validate_target_project_directory(cls, value: str) -> str: + path = Path(value) + if not value.strip() or path.is_absolute() or ".." in path.parts: + raise ValueError( + "Python task target_project_directory must stay within the " + "candidate workspace" + ) + return path.as_posix() + + @model_validator(mode="after") + def validate_filesystem(self) -> PythonTaskBackendConfig: + if not Path(self.harness_root).is_dir(): + raise ValueError("Python task harness_root must be an existing directory") + if not self.evaluations: + raise ValueError("Python task backend requires at least one evaluation") + keys = [(item.name, item.partition) for item in self.evaluations] + if len(keys) != len(set(keys)): + raise ValueError("Python task evaluation name/partition pairs must be unique") + for evaluation in self.evaluations: + if not Path(evaluation.cases_path).is_file(): + raise ValueError( + "Python task cases_path must be an existing file: " + f"{evaluation.cases_path}" + ) + return self + + +class PythonTaskBackend: + """Run an external Python task harness against an editable candidate.""" + + name = "python-task" + version = "2" + + def __init__(self, config: PythonTaskBackendConfig): + self.config = config + target = "{workspace}" + if config.target_project_directory != ".": + target += f"/{config.target_project_directory}" + self._target = target + self._commands: dict[tuple[str, str | None], CommandBackend] = {} + + def _source(self, evaluation_set: EvaluationSet) -> PythonTaskEvaluationConfig: + for source in self.config.evaluations: + if (source.name, source.partition) == ( + evaluation_set.name, + evaluation_set.partition, + ): + return source + raise ValueError( + "Python task backend does not own evaluation " + f"{evaluation_set.name!r} partition {evaluation_set.partition!r}" + ) + + def _command(self, evaluation_set: EvaluationSet) -> CommandBackend: + source = self._source(evaluation_set) + key = (source.name, source.partition) + command = self._commands.get(key) + if command is not None: + return command + command = CommandBackend( + CommandBackendConfig( + harness_root=self.config.harness_root, + command=[ + self.config.uv_executable, + "run", + "--project", + "{harness}", + "--with-editable", + self._target, + self.config.python_executable, + "-m", + "vero_tasks.runner", + "--module", + self.config.module, + "--task", + self.config.task, + "--cases", + "{input:cases}", + "--request", + "{request}", + "--report", + "{report}", + ], + environment=self.config.environment, + passthrough_environment=self.config.passthrough_environment, + staged_inputs={"cases": source.cases_path}, + ) + ) + self._commands[key] = command + return command + + @property + def provenance(self) -> BackendProvenance: + return BackendProvenance.from_config( + name=self.name, + version=self.version, + config=self.config, + ) + + def _cases(self, evaluation_set: EvaluationSet) -> list[object]: + path = Path(self._source(evaluation_set).cases_path) + if path.suffix == ".jsonl": + return [ + json.loads(line) + for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + value = json.loads(path.read_text(encoding="utf-8")) + if isinstance(value, dict): + value = value.get("cases") + if not isinstance(value, list): + raise ValueError("Python task case file must contain a case list") + return value + + def _case_ids(self, evaluation_set: EvaluationSet) -> list[str]: + case_ids = [ + str(case["id"]) + if isinstance(case, dict) and case.get("id") is not None + else str(index) + for index, case in enumerate(self._cases(evaluation_set)) + ] + if len(case_ids) != len(set(case_ids)): + raise ValueError("Python task case IDs must be unique") + return case_ids + + def _selected_cases( + self, evaluation_set: EvaluationSet + ) -> list[tuple[str, object]]: + self._validate_evaluation_set(evaluation_set) + cases = self._cases(evaluation_set) + case_ids = self._case_ids(evaluation_set) + selection = evaluation_set.selection + if isinstance(selection, AllCases): + indexes = list(range(len(cases))) + elif isinstance(selection, CaseRange): + indexes = list(range(selection.start, selection.stop)) + elif isinstance(selection, CaseIds): + by_id = {case_id: index for index, case_id in enumerate(case_ids)} + indexes = [by_id[case_id] for case_id in selection.ids] + else: # pragma: no cover - closed discriminated union + raise AssertionError(f"unsupported case selection: {selection}") + return [(case_ids[index], cases[index]) for index in indexes] + + async def export_case_resources( + self, + *, + evaluation_set: EvaluationSet, + destination: str, + sandbox: Sandbox, + ) -> None: + index = [] + for case_id, case in self._selected_cases(evaluation_set): + digest = hashlib.sha256(case_id.encode()).hexdigest() + filename = f"{digest}.json" + await sandbox.write_file( + str(PurePosixPath(destination) / filename), + json.dumps(case, ensure_ascii=False, indent=2, default=str) + "\n", + ) + index.append({"case_id": case_id, "path": filename}) + await sandbox.write_file( + str(PurePosixPath(destination) / "index.json"), + json.dumps( + { + "schema_version": 1, + "evaluation_set": evaluation_set.model_dump(mode="json"), + "cases": index, + }, + ensure_ascii=False, + indent=2, + ) + + "\n", + ) + + def _validate_evaluation_set(self, evaluation_set: EvaluationSet) -> None: + self._source(evaluation_set) + + case_ids = self._case_ids(evaluation_set) + selection = evaluation_set.selection + if isinstance(selection, CaseRange) and selection.stop > len(case_ids): + raise ValueError( + f"case range stops at {selection.stop}, but the evaluation set " + f"contains {len(case_ids)} cases" + ) + if isinstance(selection, CaseIds): + unknown = sorted(set(selection.ids) - set(case_ids)) + if unknown: + raise ValueError(f"unknown Python task case IDs: {unknown}") + + async def resolve_cost(self, evaluation_set: EvaluationSet) -> EvaluationCost: + self._validate_evaluation_set(evaluation_set) + selection = evaluation_set.selection + if isinstance(selection, CaseIds): + return EvaluationCost(cases=len(selection.ids)) + if isinstance(selection, CaseRange): + return EvaluationCost(cases=selection.stop - selection.start) + if isinstance(selection, AllCases): + return EvaluationCost(cases=len(self._case_ids(evaluation_set))) + raise AssertionError(f"unsupported case selection: {selection}") + + def validate_request(self, request: EvaluationRequest) -> None: + self._command(request.evaluation_set).validate_request(request) + self._validate_evaluation_set(request.evaluation_set) + + def sanitize_error(self, message: str) -> str: + source = self.config.evaluations[0] + return self._command( + EvaluationSet(name=source.name, partition=source.partition) + ).sanitize_error(message) + + async def evaluate( + self, + *, + context: EvaluationContext, + request: EvaluationRequest, + ) -> EvaluationReport: + target_root = context.workspace.sandbox.host_path( + context.workspace.project_path + ) + if target_root is not None: + target_root = target_root.resolve() + cases_path = Path(self._source(request.evaluation_set).cases_path).resolve() + if cases_path == target_root or cases_path.is_relative_to(target_root): + raise ValueError( + "Python task cases must live outside the editable target" + ) + return await self._command(request.evaluation_set).evaluate( + context=context, + request=request, + ) diff --git a/vero/src/vero/evaluation/engine.py b/vero/src/vero/evaluation/engine.py new file mode 100644 index 00000000..8e6a0288 --- /dev/null +++ b/vero/src/vero/evaluation/engine.py @@ -0,0 +1,508 @@ +"""Authorization, budget, backend, persistence, and disclosure boundary.""" + +from __future__ import annotations + +import asyncio +import inspect +import logging +from contextlib import asynccontextmanager +from contextvars import ContextVar +from pathlib import Path +from typing import AsyncIterator, Awaitable, Callable + +from vero.evaluation.backends.base import BackendRegistry +from vero.evaluation.evaluator import Evaluator +from vero.evaluation.exceptions import ( + EvaluationCancelledError, + EvaluationDeniedError, + EvaluationExecutionError, + EvaluationInfrastructureError, + EvaluationRequestError, + EvaluationTerminatedError, +) +from vero.evaluation.models import ( + AgentSelectionMode, + AllCases, + DisclosureLevel, + EvaluationAcknowledgement, + EvaluationAuthorization, + EvaluationPlan, + EvaluationPrincipal, + EvaluationRecord, + EvaluationRequest, + EvaluationSummary, + ObjectiveSpec, +) +from vero.evaluation.scoring.error_taxonomy import TERMINATING_DIAGNOSTIC_CODES +from vero.evaluation.scoring.objective import project_evaluation +from vero.evaluation.store.budget import BudgetLedger +from vero.evaluation.store.persistence import EvaluationDatabase, EvaluationStore + +logger = logging.getLogger(__name__) + +AuthorizationResolver = Callable[ + [EvaluationPrincipal, str, EvaluationRequest], + EvaluationAuthorization | Awaitable[EvaluationAuthorization], +] + + +def allow_all_evaluations( + _principal: EvaluationPrincipal, + _backend_id: str, + _request: EvaluationRequest, +) -> EvaluationAuthorization: + """Explicit resolver for trusted runtimes without an evaluation boundary.""" + + return EvaluationAuthorization( + may_evaluate=True, + may_view=True, + expose_case_resources=True, + ) + + +def authorize_evaluation_plan(plan: EvaluationPlan) -> AuthorizationResolver: + """Build the canonical principal-aware resolver for an evaluation plan.""" + + def resolve( + principal: EvaluationPrincipal, + _backend_id: str, + request: EvaluationRequest, + ) -> EvaluationAuthorization: + definition = plan.for_evaluation_set(request.evaluation_set) + if definition is None: + return EvaluationAuthorization( + may_evaluate=False, + may_view=False, + reason="evaluation set is not present in the session plan", + ) + access = definition.access + if principal == EvaluationPrincipal.ADMIN: + return EvaluationAuthorization( + may_evaluate=True, + may_view=False, + meter_budget=False, + disclosure=access.disclosure, + ) + if principal == EvaluationPrincipal.SYSTEM: + return EvaluationAuthorization( + may_evaluate=True, + may_view=access.agent_visible, + meter_budget=definition.system_budget is not None, + disclosure=access.disclosure, + expose_case_resources=False, + ) + if ( + access.agent_selection == AgentSelectionMode.FIXED + and request.evaluation_set.selection != definition.evaluation_set.selection + ): + return EvaluationAuthorization( + may_evaluate=False, + may_view=access.agent_visible, + reason="evaluation set requires its fixed case selection", + ) + return EvaluationAuthorization( + may_evaluate=access.agent_can_evaluate, + may_view=access.agent_visible, + meter_budget=definition.agent_budget is not None, + disclosure=access.disclosure, + expose_case_resources=access.expose_case_resources, + min_aggregate_cases=access.min_aggregate_cases or 1, + reason=( + None + if access.agent_can_evaluate + else "evaluation set is not agent-evaluable" + ), + ) + + return resolve + + +class EvaluationEngine: + """The only runtime path from an evaluation request to a stored record.""" + + def __init__( + self, + *, + evaluator: Evaluator, + backends: BackendRegistry, + database: EvaluationDatabase, + database_path: Path | None = None, + budget_ledger: BudgetLedger | None = None, + authorization_resolver: AuthorizationResolver | None = None, + ): + self.evaluator = evaluator + self.backends = backends + self.database = database + self.database_path = database_path + self.budget_ledger = budget_ledger + self.authorization_resolver = authorization_resolver + self.listeners: list[Callable[[EvaluationRecord], object]] = [] + self._record_lock = asyncio.Lock() + self._agent_evaluations_open = True + self._active_agent_evaluations: dict[object, asyncio.Task[object]] = {} + self._agent_evaluations_idle = asyncio.Event() + self._agent_evaluations_idle.set() + self._agent_evaluation_scope_depth: ContextVar[int] = ContextVar( + f"vero_agent_evaluation_scope_{id(self)}", + default=0, + ) + + def _begin_evaluation( + self, + principal: EvaluationPrincipal, + ) -> object | None: + """Atomically admit and track an agent evaluation on this event loop.""" + + if principal != EvaluationPrincipal.AGENT: + return None + if not self._agent_evaluations_open: + raise EvaluationDeniedError("evaluation finalization has started") + task = asyncio.current_task() + if task is None: # pragma: no cover - async entry points always have a task + raise RuntimeError("evaluation requires an active asyncio task") + token = object() + self._active_agent_evaluations[token] = task + self._agent_evaluations_idle.clear() + return token + + def _finish_evaluation(self, token: object | None) -> None: + if token is None: + return + self._active_agent_evaluations.pop(token, None) + if not self._active_agent_evaluations: + self._agent_evaluations_idle.set() + + @asynccontextmanager + async def agent_evaluation_scope(self) -> AsyncIterator[None]: + """Track a complete agent request, including candidate import and disclosure.""" + + depth = self._agent_evaluation_scope_depth.get() + if depth: + nested = self._agent_evaluation_scope_depth.set(depth + 1) + try: + yield + finally: + self._agent_evaluation_scope_depth.reset(nested) + return + + token = self._begin_evaluation(EvaluationPrincipal.AGENT) + outer = self._agent_evaluation_scope_depth.set(1) + try: + yield + finally: + self._agent_evaluation_scope_depth.reset(outer) + self._finish_evaluation(token) + + async def quiesce_agent_evaluations( + self, + *, + timeout_seconds: float, + cancellation_grace_seconds: float = 30.0, + ) -> int: + """Close agent admission and drain requests accepted before finalization. + + The admission close and active-task snapshot contain no suspension point, + so a new agent evaluation cannot slip between them. If the bounded wait + expires, the remaining requests are cancelled; the normal evaluator + cancellation path persists terminal records and refunds their budgets. + Admin and system evaluations remain available to the verifier. + """ + + if timeout_seconds <= 0: + raise ValueError("evaluation drain timeout must be positive") + if cancellation_grace_seconds <= 0: + raise ValueError("evaluation cancellation grace must be positive") + + self._agent_evaluations_open = False + admitted = len(self._active_agent_evaluations) + if not admitted: + return 0 + + try: + async with asyncio.timeout(timeout_seconds): + await self._agent_evaluations_idle.wait() + return admitted + except TimeoutError: + pending = set(self._active_agent_evaluations.values()) + logger.warning( + "Cancelling %d agent evaluation(s) after a %.1fs finalization drain", + len(pending), + timeout_seconds, + ) + for task in pending: + task.cancel() + try: + async with asyncio.timeout(cancellation_grace_seconds): + await asyncio.gather(*pending, return_exceptions=True) + await self._agent_evaluations_idle.wait() + except TimeoutError: + logger.error( + "%d agent evaluation(s) did not stop within the %.1fs " + "cancellation grace", + len(self._active_agent_evaluations), + cancellation_grace_seconds, + ) + return admitted + + async def authorize( + self, + backend_id: str, + request: EvaluationRequest, + principal: EvaluationPrincipal = EvaluationPrincipal.AGENT, + supplied: EvaluationAuthorization | None = None, + ) -> EvaluationAuthorization: + """Resolve the trusted access decision without executing an evaluation.""" + + if supplied is not None: + return supplied + if self.authorization_resolver is None: + return EvaluationAuthorization( + may_evaluate=False, + reason="evaluation authorization was not configured", + ) + resolved = self.authorization_resolver(principal, backend_id, request) + if inspect.isawaitable(resolved): + resolved = await resolved + return resolved + + async def _evaluate_record( + self, + *, + backend_id: str, + request: EvaluationRequest, + objective_spec: ObjectiveSpec | None, + authorization: EvaluationAuthorization | None, + principal: EvaluationPrincipal, + ) -> tuple[EvaluationRecord, EvaluationAuthorization]: + token = ( + None + if principal == EvaluationPrincipal.AGENT + and self._agent_evaluation_scope_depth.get() + else self._begin_evaluation(principal) + ) + try: + return await self._execute_record( + backend_id=backend_id, + request=request, + objective_spec=objective_spec, + authorization=authorization, + principal=principal, + ) + finally: + self._finish_evaluation(token) + + async def _execute_record( + self, + *, + backend_id: str, + request: EvaluationRequest, + objective_spec: ObjectiveSpec | None, + authorization: EvaluationAuthorization | None, + principal: EvaluationPrincipal, + ) -> tuple[EvaluationRecord, EvaluationAuthorization]: + backend = self.backends.resolve(backend_id) + decision = await self.authorize( + backend_id, + request, + principal, + authorization, + ) + if not decision.may_evaluate: + raise EvaluationDeniedError( + decision.reason or "evaluation is not authorized" + ) + + validate_request = getattr(backend, "validate_request", None) + if callable(validate_request): + try: + validate_request(request) + except ValueError as error: + raise EvaluationRequestError(str(error)) from error + try: + cost = await backend.resolve_cost(request.evaluation_set) + except ValueError as error: + raise EvaluationRequestError(str(error)) from error + # k-anonymity floor: an aggregate-disclosed subset must cover enough + # cases that a hidden per-case result can't be read off one at a time. + # The complete set is exempt — its aggregate is the intended disclosure. + if ( + decision.disclosure == DisclosureLevel.AGGREGATE + and decision.min_aggregate_cases > 1 + and not isinstance(request.evaluation_set.selection, AllCases) + ): + if cost.cases is None: + raise EvaluationDeniedError( + "aggregate subset evaluation requires a backend with " + "exact case costs" + ) + if cost.cases < decision.min_aggregate_cases: + raise EvaluationDeniedError( + f"aggregate subset evaluations must cover at least " + f"{decision.min_aggregate_cases} cases; requested {cost.cases}" + ) + charged = decision.meter_budget and self.budget_ledger is not None + if charged: + await self.budget_ledger.reserve( + backend_id, + request.evaluation_set, + cost, + principal, + ) + + try: + record = await self.evaluator.evaluate( + backend_id=backend_id, + backend=backend, + request=request, + objective_spec=objective_spec, + principal=principal, + ) + except EvaluationCancelledError as error: + cancelled = EvaluationStore( + self.evaluator.evaluations_dir / error.evaluation_id + ).load() + await asyncio.shield(self._record(cancelled)) + if charged: + await asyncio.shield( + self.budget_ledger.refund( + backend_id, + request.evaluation_set, + cost, + principal, + ) + ) + raise + except EvaluationExecutionError as error: + failure = EvaluationStore( + self.evaluator.evaluations_dir / error.evaluation_id + ).load() + # Shield the record + refund like the cancellation handler above: a + # cancellation arriving during this cleanup must not interrupt the + # refund and leak the reservation's budget. + await asyncio.shield(self._record(failure)) + if charged: + await asyncio.shield( + self.budget_ledger.refund( + backend_id, + request.evaluation_set, + cost, + principal, + ) + ) + raise + except asyncio.CancelledError: + # Last line of defence for the reservation. The evaluator's own + # handlers convert cancellation and failure into the two typed errors + # above, but each of them has to await a persist before it can raise, + # and a cancellation delivered during that await -- which + # quiesce_agent_evaluations does deliver, at finalization -- unwinds + # as a raw CancelledError instead. Neither handler above would see + # it, and the reservation would stay charged forever. + # + # Only the refund is recoverable here: a raw cancellation carries no + # evaluation id, so there is no record to load. The evaluator already + # persisted one on its shielded paths. + if charged: + await asyncio.shield( + self.budget_ledger.refund( + backend_id, + request.evaluation_set, + cost, + principal, + ) + ) + raise + # Shielded for the same reason as the two handlers above, and one more: + # the budget was charged for an evaluation that has now actually run, so + # a cancellation landing inside _record would leave the ledger counting + # a run the database never indexed. _record is not atomic either -- it + # mutates the in-memory database before persisting it -- so an + # unshielded cancellation can tear those two apart as well. + await asyncio.shield(self._record(record)) + terminating = next( + ( + diagnostic + for diagnostic in record.report.diagnostics + if diagnostic.code in TERMINATING_DIAGNOSTIC_CODES + ), + None, + ) + if terminating is not None: + # A terminating condition (inference-budget exhaustion or auth + # failure) will not heal: do not refund and do not retry. + raise EvaluationTerminatedError(record.id, terminating.message) + infrastructure = next( + ( + diagnostic + for diagnostic in record.report.diagnostics + if diagnostic.code == "infrastructure_failure" + ), + None, + ) + if infrastructure is not None: + if charged: + # Shield: a cancellation racing this infrastructure-failure + # refund must not leak the reservation's budget. + await asyncio.shield( + self.budget_ledger.refund( + backend_id, + request.evaluation_set, + cost, + principal, + ) + ) + raise EvaluationInfrastructureError(record.id, infrastructure.message) + return record, decision + + async def _record(self, record: EvaluationRecord) -> None: + """Index, persist, and publish a completed evaluation exactly once.""" + async with self._record_lock: + self.database.add_evaluation(record) + if self.database_path is not None: + await asyncio.to_thread( + self.database.save_to_file, + self.database_path, + ) + for listener in self.listeners: + try: + result = listener(record) + if inspect.isawaitable(result): + await result + except Exception: + logger.exception("Evaluation listener failed for %s", record.id) + + async def evaluate_record( + self, + *, + backend_id: str, + request: EvaluationRequest, + objective_spec: ObjectiveSpec | None = None, + authorization: EvaluationAuthorization | None = None, + principal: EvaluationPrincipal = EvaluationPrincipal.AGENT, + ) -> EvaluationRecord: + record, _ = await self._evaluate_record( + backend_id=backend_id, + request=request, + objective_spec=objective_spec, + authorization=authorization, + principal=principal, + ) + return record + + async def evaluate( + self, + *, + backend_id: str, + request: EvaluationRequest, + objective_spec: ObjectiveSpec | None = None, + authorization: EvaluationAuthorization | None = None, + principal: EvaluationPrincipal = EvaluationPrincipal.AGENT, + ) -> EvaluationRecord | EvaluationSummary | EvaluationAcknowledgement: + record, decision = await self._evaluate_record( + backend_id=backend_id, + request=request, + objective_spec=objective_spec, + authorization=authorization, + principal=principal, + ) + return project_evaluation(record, decision.disclosure) diff --git a/vero/src/vero/evaluation/evaluator.py b/vero/src/vero/evaluation/evaluator.py new file mode 100644 index 00000000..96646194 --- /dev/null +++ b/vero/src/vero/evaluation/evaluator.py @@ -0,0 +1,295 @@ +"""Program-neutral evaluator lifecycle.""" + +from __future__ import annotations + +import asyncio +from contextlib import asynccontextmanager +from datetime import UTC, datetime +from pathlib import Path +from typing import AsyncIterator +from uuid import uuid4 + +from vero.candidate import Candidate +from vero.candidate_repository import CandidateRepository +from vero.evaluation.backends.base import EvaluationBackend, EvaluationContext +from vero.evaluation.exceptions import ( + EvaluationCancelledError, + EvaluationExecutionError, +) +from vero.evaluation.models import ( + CaseStatus, + DiagnosticSeverity, + EvaluationDiagnostic, + EvaluationPrincipal, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationStatus, + ObjectiveSpec, +) +from vero.evaluation.scoring.objective import evaluate_objective +from vero.evaluation.store.persistence import EvaluationStore +from vero.sandbox import Sandbox +from vero.workspace import Workspace + + +class Evaluator: + """Run one backend against a clean candidate snapshot and persist it.""" + + def __init__( + self, + *, + candidate_repository: CandidateRepository, + sandbox: Sandbox, + session_dir: Path, + session_id: str | None = None, + ): + self.candidate_repository = candidate_repository + self.sandbox = sandbox + self.session_dir = session_dir + self.session_id = session_id or session_dir.name + + @property + def evaluations_dir(self) -> Path: + return self.session_dir / "evaluations" + + @asynccontextmanager + async def _candidate_workspace( + self, + candidate: Candidate, + ) -> AsyncIterator[Workspace]: + async with self.candidate_repository.checkout( + candidate, + sandbox=self.sandbox, + name=f"vero-evaluation-{candidate.id}", + ) as workspace: + yield workspace + + async def _persist_failure( + self, + *, + store: EvaluationStore, + evaluation_id: str, + backend_id: str, + backend: EvaluationBackend, + request: EvaluationRequest, + objective_spec: ObjectiveSpec | None, + principal: EvaluationPrincipal, + created_at: datetime, + code: str, + message: str, + status: EvaluationStatus = EvaluationStatus.FAILED, + ) -> EvaluationRecord: + report = EvaluationReport( + status=status, + diagnostics=[ + EvaluationDiagnostic( + code=code, + message=message, + severity=DiagnosticSeverity.ERROR, + phase="evaluation", + ) + ], + ) + objective = ( + evaluate_objective(report, objective_spec) + if objective_spec is not None + else None + ) + record = EvaluationRecord( + id=evaluation_id, + request=request, + report=report, + backend_id=backend_id, + backend=backend.provenance, + principal=principal, + objective_spec=objective_spec, + objective=objective, + created_at=created_at, + completed_at=datetime.now(UTC), + ) + await store.save(record) + return record + + async def evaluate( + self, + *, + backend_id: str, + backend: EvaluationBackend, + request: EvaluationRequest, + objective_spec: ObjectiveSpec | None = None, + principal: EvaluationPrincipal = EvaluationPrincipal.SYSTEM, + ) -> EvaluationRecord: + evaluation_id = str(uuid4()) + created_at = datetime.now(UTC) + result_dir = self.evaluations_dir / evaluation_id + store = EvaluationStore(result_dir) + result_dir.mkdir(parents=True, exist_ok=False) + store.artifact_dir.mkdir(parents=True, exist_ok=True) + store.write_running( + evaluation_id=evaluation_id, + request=request, + backend_id=backend_id, + backend=backend.provenance, + objective_spec=objective_spec, + created_at=created_at, + ) + + try: + async with self._candidate_workspace( + request.candidate, + ) as candidate_workspace: + actual_version = await candidate_workspace.current_version() + if actual_version != request.candidate.version: + raise ValueError( + f"candidate workspace is at {actual_version!r}, expected " + f"{request.candidate.version!r}" + ) + if await candidate_workspace.is_dirty(): + raise ValueError( + "candidate workspace must be clean before evaluation" + ) + context = EvaluationContext( + workspace=candidate_workspace, + session_id=self.session_id, + evaluation_id=evaluation_id, + result_dir=result_dir, + artifact_dir=store.artifact_dir, + case_store=store.cases, + finalization=principal == EvaluationPrincipal.ADMIN, + ) + async with asyncio.timeout(request.limits.timeout_seconds): + raw_report = await backend.evaluate( + context=context, + request=request, + ) + report = EvaluationReport.model_validate(raw_report) + report = self._apply_error_rate_threshold( + report, + request.limits.error_rate_threshold, + ) + + objective = ( + evaluate_objective(report, objective_spec) + if objective_spec is not None + else None + ) + record = EvaluationRecord( + id=evaluation_id, + request=request, + report=report, + backend_id=backend_id, + backend=backend.provenance, + principal=principal, + objective_spec=objective_spec, + objective=objective, + created_at=created_at, + completed_at=datetime.now(UTC), + ) + await store.save(record) + return record + except asyncio.CancelledError as error: + message = "evaluation was cancelled" + await asyncio.shield( + self._persist_failure( + store=store, + evaluation_id=evaluation_id, + backend_id=backend_id, + backend=backend, + request=request, + objective_spec=objective_spec, + principal=principal, + created_at=created_at, + code="evaluation_cancelled", + message=message, + status=EvaluationStatus.CANCELLED, + ) + ) + raise EvaluationCancelledError(evaluation_id, message) from error + # The two handlers below shield their persist for the same reason the + # cancellation handler above does. asyncio.timeout absorbs its own + # internal cancellation and re-raises as TimeoutError, but an *external* + # cancel -- quiesce_agent_evaluations draining agent evaluations at + # finalization -- can still be pending, and the first await inside an + # unshielded _persist_failure would deliver it, losing the failure record + # and unwinding as a raw CancelledError instead of the typed error. + # (EvaluationEngine refunds on that raw path too, belt and braces.) + except TimeoutError as error: + message = f"evaluation exceeded {request.limits.timeout_seconds} seconds" + await asyncio.shield( + self._persist_failure( + store=store, + evaluation_id=evaluation_id, + backend_id=backend_id, + backend=backend, + request=request, + objective_spec=objective_spec, + principal=principal, + created_at=created_at, + code="evaluation_timeout", + message=message, + ) + ) + raise EvaluationExecutionError(evaluation_id, message) from error + except Exception as error: + message = str(error) or type(error).__name__ + await asyncio.shield( + self._persist_failure( + store=store, + evaluation_id=evaluation_id, + backend_id=backend_id, + backend=backend, + request=request, + objective_spec=objective_spec, + principal=principal, + created_at=created_at, + code="backend_error", + message=message, + ) + ) + raise EvaluationExecutionError(evaluation_id, message) from error + + @staticmethod + def _apply_error_rate_threshold( + report: EvaluationReport, + threshold: float | None, + ) -> EvaluationReport: + """Mark a report INVALID when too many cases were lost to infrastructure. + + Only infrastructure cases (``CaseStatus.ERROR``) count: a legitimate + agent failure is now an informative ``SUCCESS`` at the failure value, so + it does not push a candidate over the threshold. Crossing the threshold + means the aggregate is unreliable, not that the candidate is bad, so the + report becomes ``INVALID`` rather than ``FAILED``.""" + + if threshold is None or report.status != EvaluationStatus.SUCCESS: + return report + considered = [ + case + for case in report.cases + if case.status in (CaseStatus.SUCCESS, CaseStatus.ERROR) + ] + if considered: + error_rate = sum( + case.status == CaseStatus.ERROR for case in considered + ) / len(considered) + else: + error_rate = report.metrics.get("error_rate") + if error_rate is None or error_rate < threshold: + return report + diagnostic = EvaluationDiagnostic( + code="infrastructure_invalidity_threshold_exceeded", + message=( + f"infrastructure-error rate {error_rate:.6g} reached the configured " + f"threshold {threshold:.6g}; the aggregate score is unreliable" + ), + severity=DiagnosticSeverity.ERROR, + phase="evaluation", + ) + return report.model_copy( + update={ + "status": EvaluationStatus.INVALID, + "metrics": {**report.metrics, "error_rate": error_rate}, + "diagnostics": [*report.diagnostics, diagnostic], + } + ) diff --git a/vero/src/vero/evaluation/exceptions.py b/vero/src/vero/evaluation/exceptions.py new file mode 100644 index 00000000..429696d1 --- /dev/null +++ b/vero/src/vero/evaluation/exceptions.py @@ -0,0 +1,52 @@ +"""Exceptions raised by canonical evaluation components.""" + +import asyncio + + +class EvaluationError(Exception): + """Base exception for evaluation failures.""" + + +class UnknownBackendError(EvaluationError, KeyError): + """Raised when a caller selects an unregistered backend.""" + + +class EvaluationDeniedError(EvaluationError, PermissionError): + """Raised when trusted authorization denies an evaluation.""" + + +class EvaluationBudgetExceeded(EvaluationError): + """Raised when an evaluation budget cannot reserve a cost.""" + + +class EvaluationRequestError(EvaluationError, ValueError): + """Raised when a backend rejects caller-controlled request fields.""" + + +class EvaluationExecutionError(EvaluationError): + """Raised after an evaluation failure has been recorded.""" + + def __init__(self, evaluation_id: str, message: str): + self.evaluation_id = evaluation_id + super().__init__(f"Evaluation {evaluation_id} failed: {message}") + + +class EvaluationInfrastructureError(EvaluationExecutionError): + """Raised after transient/external infrastructure exhausted its retries.""" + + +class EvaluationTerminatedError(EvaluationExecutionError): + """Raised for a non-retryable, terminating condition (inference-budget + exhaustion or authentication failure). + + Unlike :class:`EvaluationInfrastructureError`, this is never refunded and + never retried: the condition will not heal on its own, so the run stops + loudly with the cause preserved rather than burning retries on it.""" + + +class EvaluationCancelledError(asyncio.CancelledError): + """Cancellation propagated after its terminal evaluation record is stored.""" + + def __init__(self, evaluation_id: str, message: str = "evaluation was cancelled"): + self.evaluation_id = evaluation_id + super().__init__(message) diff --git a/vero/src/vero/evaluation/models.py b/vero/src/vero/evaluation/models.py new file mode 100644 index 00000000..9cdbe63c --- /dev/null +++ b/vero/src/vero/evaluation/models.py @@ -0,0 +1,823 @@ +"""Canonical, backend-neutral evaluation contracts.""" + +from __future__ import annotations + +import hashlib +import json +import math +import re +from datetime import UTC, datetime +from enum import Enum +from pathlib import PurePosixPath +from typing import Annotated, Any, Literal, Mapping + +from pydantic import ( + BaseModel, + Field, + JsonValue, + field_validator, + model_validator, +) + +from vero.candidate import Candidate +from vero.models import StrictModel + + +def _non_empty(value: str, field_name: str) -> str: + if not value.strip(): + raise ValueError(f"{field_name} must not be empty") + return value + + +def _optional_non_empty(value: str | None, field_name: str) -> str | None: + if value is not None: + _non_empty(value, field_name) + return value + + +def _finite_metrics(metrics: dict[str, float]) -> dict[str, float]: + for name, value in metrics.items(): + _non_empty(name, "metric name") + if not math.isfinite(value): + raise ValueError(f"metric {name!r} must be finite") + return metrics + + +def _aware_utc(value: datetime, field_name: str) -> datetime: + if value.tzinfo is None or value.utcoffset() is None: + raise ValueError(f"{field_name} must be timezone-aware") + return value.astimezone(UTC) + + +def _canonical_json(value: Any) -> str: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +class AllCases(StrictModel): + kind: Literal["all"] = "all" + + +class CaseIds(StrictModel): + kind: Literal["ids"] = "ids" + ids: list[str] + + @field_validator("ids") + @classmethod + def validate_ids(cls, ids: list[str]) -> list[str]: + if not ids: + raise ValueError("ids must not be empty") + for case_id in ids: + _non_empty(case_id, "case ID") + if len(set(ids)) != len(ids): + raise ValueError("case IDs must be unique") + return ids + + +class CaseRange(StrictModel): + kind: Literal["range"] = "range" + stop: int + start: int = 0 + + @model_validator(mode="after") + def validate_range(self) -> CaseRange: + if self.start < 0: + raise ValueError("start must be non-negative") + if self.stop <= self.start: + raise ValueError("stop must be greater than start") + return self + + +CaseSelection = Annotated[AllCases | CaseIds | CaseRange, Field(discriminator="kind")] + + +class EvaluationSet(StrictModel): + """A backend-owned collection of cases and a selection within it.""" + + name: str = "default" + partition: str | None = None + selection: CaseSelection = Field(default_factory=AllCases) + + @field_validator("name") + @classmethod + def validate_name(cls, value: str) -> str: + return _non_empty(value, "evaluation set name") + + @field_validator("partition") + @classmethod + def validate_partition(cls, value: str | None) -> str | None: + return _optional_non_empty(value, "partition") + + def budget_key(self, backend_id: str) -> str: + _non_empty(backend_id, "backend ID") + return f"{backend_id}:{self.name}:{self.partition or ''}" + + +class EvaluationPrincipal(str, Enum): + """Trusted caller class used for authorization and independent metering.""" + + AGENT = "agent" + SYSTEM = "system" + ADMIN = "admin" + + +class AgentSelectionMode(str, Enum): + """How an agent may vary an evaluation definition's base case selection.""" + + FIXED = "fixed" + ARBITRARY = "arbitrary" + + +class RetryPolicy(StrictModel): + max_attempts: int = Field(default=3, ge=1) + initial_delay_seconds: float = Field(default=4.0, ge=0.0) + maximum_delay_seconds: float = Field(default=120.0, ge=0.0) + multiplier: float = Field(default=2.0, ge=1.0) + retry_on_timeout: bool = True + retry_exception_names: list[str] = Field( + default_factory=lambda: [ + "openai.RateLimitError", + "anthropic.RateLimitError", + ] + ) + retry_status_codes: list[int] = Field(default_factory=lambda: [429, 503, 529]) + retry_message_patterns: list[str] = Field( + default_factory=lambda: ["rate limit", "too many requests"] + ) + + @model_validator(mode="after") + def validate_delays(self) -> RetryPolicy: + if self.maximum_delay_seconds < self.initial_delay_seconds: + raise ValueError("maximum retry delay cannot be less than initial delay") + for name in self.retry_exception_names: + _non_empty(name, "retry exception name") + if len(set(self.retry_exception_names)) != len(self.retry_exception_names): + raise ValueError("retry exception names must be unique") + for status_code in self.retry_status_codes: + if status_code < 100 or status_code > 599: + raise ValueError("retry status codes must be between 100 and 599") + if len(set(self.retry_status_codes)) != len(self.retry_status_codes): + raise ValueError("retry status codes must be unique") + for pattern in self.retry_message_patterns: + _non_empty(pattern, "retry message pattern") + try: + re.compile(pattern) + except re.error as error: + raise ValueError( + f"invalid retry message pattern {pattern!r}: {error}" + ) from error + return self + + @classmethod + def disabled(cls) -> RetryPolicy: + """Return an explicit no-retry policy for backends with their own retries.""" + return cls(max_attempts=1) + + +class EvaluationLimits(StrictModel): + timeout_seconds: float = Field(default=600.0, gt=0.0) + case_timeout_seconds: float = Field(default=180.0, gt=0.0) + max_concurrency: int = Field(default=100, ge=1) + error_rate_threshold: float | None = Field(default=0.1, gt=0.0, le=1.0) + retry: RetryPolicy = Field(default_factory=RetryPolicy) + + +class EvaluationRequest(StrictModel): + candidate: Candidate + evaluation_set: EvaluationSet = Field(default_factory=EvaluationSet) + parameters: dict[str, JsonValue] = Field(default_factory=dict) + limits: EvaluationLimits = Field(default_factory=EvaluationLimits) + seed: int | None = None + + @field_validator("parameters") + @classmethod + def validate_parameter_names( + cls, parameters: dict[str, JsonValue] + ) -> dict[str, JsonValue]: + for name in parameters: + _non_empty(name, "parameter name") + return parameters + + def fingerprint(self) -> str: + """Return the stable identity used to group repeat measurements.""" + payload = { + "candidate": { + "id": self.candidate.id, + "version": self.candidate.version, + }, + "evaluation_set": self.evaluation_set.model_dump(mode="json"), + "parameters": self.parameters, + "limits": self.limits.model_dump(mode="json"), + "seed": self.seed, + } + return hashlib.sha256(_canonical_json(payload).encode()).hexdigest() + + +class CommandEvaluationInput(StrictModel): + """Versioned JSON input passed to an external evaluation harness.""" + + schema_version: Literal[1] = 1 + request: EvaluationRequest + + +class EvaluationArtifact(StrictModel): + path: str + media_type: str | None = None + description: str | None = None + + @field_validator("path") + @classmethod + def validate_path(cls, value: str) -> str: + _non_empty(value, "artifact path") + if "\\" in value or value.startswith("/") or PurePosixPath(value).is_absolute(): + raise ValueError("artifact paths must be relative POSIX paths") + if any(part in {"", ".", ".."} for part in value.split("/")): + raise ValueError( + "artifact paths must not contain empty, '.' or '..' segments" + ) + return value + + @field_validator("media_type", "description") + @classmethod + def validate_optional_text(cls, value: str | None) -> str | None: + return _optional_non_empty(value, "artifact metadata") + + +class CaseStatus(str, Enum): + SUCCESS = "success" + ERROR = "error" + SKIPPED = "skipped" + + +class CaseError(StrictModel): + message: str + code: str | None = None + phase: str | None = None + attempt: int | None = Field(default=None, ge=1) + retryable: bool | None = None + terminal: bool = False + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("message") + @classmethod + def validate_message(cls, value: str) -> str: + return _non_empty(value, "error message") + + @field_validator("code", "phase") + @classmethod + def validate_optional_text(cls, value: str | None) -> str | None: + return _optional_non_empty(value, "error code or phase") + + +class CaseResult(StrictModel): + case_id: str + status: CaseStatus + metrics: dict[str, float] = Field(default_factory=dict) + input: JsonValue | None = None + output: JsonValue | None = None + feedback: str | None = None + errors: list[CaseError] = Field(default_factory=list) + execution_trace: list[JsonValue] | None = None + evaluation_trace: list[JsonValue] | None = None + metadata: dict[str, JsonValue] = Field(default_factory=dict) + artifacts: list[EvaluationArtifact] = Field(default_factory=list) + + @field_validator("case_id") + @classmethod + def validate_case_id(cls, value: str) -> str: + return _non_empty(value, "case ID") + + @field_validator("metrics") + @classmethod + def validate_metrics(cls, value: dict[str, float]) -> dict[str, float]: + return _finite_metrics(value) + + @field_validator("feedback") + @classmethod + def validate_feedback(cls, value: str | None) -> str | None: + return _optional_non_empty(value, "feedback") + + @model_validator(mode="after") + def validate_status_errors(self) -> CaseResult: + has_terminal_error = any(error.terminal for error in self.errors) + if self.status == CaseStatus.ERROR and not has_terminal_error: + raise ValueError("errored cases require at least one terminal error") + if self.status != CaseStatus.ERROR and has_terminal_error: + raise ValueError("only errored cases may contain terminal errors") + return self + + +class EvaluationStatus(str, Enum): + SUCCESS = "success" + FAILED = "failed" + CANCELLED = "cancelled" + #: The evaluation ran, but too many cases were lost to infrastructure for + #: the aggregate score to be trusted. Distinct from FAILED so a benchmark + #: can tell "unreliable due to infrastructure" apart from a feasible low + #: score, and never fold "nothing usable" into a legitimate zero. + INVALID = "invalid" + + +class DiagnosticSeverity(str, Enum): + INFO = "info" + WARNING = "warning" + ERROR = "error" + + +class EvaluationDiagnostic(StrictModel): + code: str + message: str + severity: DiagnosticSeverity + phase: str | None = None + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("code", "message") + @classmethod + def validate_required_text(cls, value: str) -> str: + return _non_empty(value, "diagnostic code or message") + + @field_validator("phase") + @classmethod + def validate_phase(cls, value: str | None) -> str | None: + return _optional_non_empty(value, "diagnostic phase") + + +class EvaluationReport(StrictModel): + schema_version: Literal[1] = 1 + status: EvaluationStatus + metrics: dict[str, float] = Field(default_factory=dict) + cases: list[CaseResult] = Field(default_factory=list) + diagnostics: list[EvaluationDiagnostic] = Field(default_factory=list) + artifacts: list[EvaluationArtifact] = Field(default_factory=list) + + @field_validator("metrics") + @classmethod + def validate_metrics(cls, value: dict[str, float]) -> dict[str, float]: + return _finite_metrics(value) + + @model_validator(mode="after") + def validate_report(self) -> EvaluationReport: + case_ids = [case.case_id for case in self.cases] + if len(case_ids) != len(set(case_ids)): + raise ValueError("case IDs must be unique within an evaluation report") + return self + + +class BackendProvenance(StrictModel): + name: str + version: str + config_digest: str + + @field_validator("name", "version") + @classmethod + def validate_identity(cls, value: str) -> str: + return _non_empty(value, "backend name or version") + + @field_validator("config_digest") + @classmethod + def validate_digest(cls, value: str) -> str: + if re.fullmatch(r"[0-9a-f]{64}", value) is None: + raise ValueError("config_digest must be a lowercase SHA-256 digest") + return value + + @classmethod + def from_config( + cls, + *, + name: str, + version: str, + config: BaseModel | Mapping[str, JsonValue], + ) -> BackendProvenance: + config_value = ( + config.model_dump(mode="json") + if isinstance(config, BaseModel) + else dict(config) + ) + payload = {"name": name, "version": version, "config": config_value} + digest = hashlib.sha256(_canonical_json(payload).encode()).hexdigest() + return cls(name=name, version=version, config_digest=digest) + + +class MetricAggregation(str, Enum): + REPORT = "report" + MEAN = "mean" + MEDIAN = "median" + MIN = "min" + MAX = "max" + + +class MetricSelector(StrictModel): + metric: str + aggregation: MetricAggregation = MetricAggregation.REPORT + case_failure_value: float | None = None + + @field_validator("metric") + @classmethod + def validate_metric(cls, value: str) -> str: + return _non_empty(value, "metric name") + + @field_validator("case_failure_value") + @classmethod + def validate_case_failure_value(cls, value: float | None) -> float | None: + if value is not None and not math.isfinite(value): + raise ValueError("case_failure_value must be finite") + return value + + @model_validator(mode="after") + def validate_case_aggregation(self) -> MetricSelector: + if ( + self.aggregation == MetricAggregation.REPORT + and self.case_failure_value is not None + ): + raise ValueError("case_failure_value requires a case metric aggregation") + return self + + +class ConstraintOperator(str, Enum): + EQ = "==" + NE = "!=" + LT = "<" + LTE = "<=" + GT = ">" + GTE = ">=" + + +class MetricConstraint(StrictModel): + selector: MetricSelector + operator: ConstraintOperator + value: float + + @field_validator("value") + @classmethod + def validate_value(cls, value: float) -> float: + if not math.isfinite(value): + raise ValueError("constraint value must be finite") + return value + + +class ObjectiveSpec(StrictModel): + selector: MetricSelector + direction: Literal["maximize", "minimize"] + failure_value: float | None = None + constraints: list[MetricConstraint] = Field(default_factory=list) + + @field_validator("failure_value") + @classmethod + def validate_failure_value(cls, value: float | None) -> float | None: + if value is not None and not math.isfinite(value): + raise ValueError("failure_value must be finite") + return value + + +class ConstraintViolation(StrictModel): + constraint: MetricConstraint + observed: float | None + reason: str + + @field_validator("observed") + @classmethod + def validate_observed(cls, value: float | None) -> float | None: + if value is not None and not math.isfinite(value): + raise ValueError("observed constraint value must be finite") + return value + + @field_validator("reason") + @classmethod + def validate_reason(cls, value: str) -> str: + return _non_empty(value, "constraint violation reason") + + +class ObjectiveResult(StrictModel): + value: float | None + feasible: bool + violations: list[ConstraintViolation] = Field(default_factory=list) + + @field_validator("value") + @classmethod + def validate_value(cls, value: float | None) -> float | None: + if value is not None and not math.isfinite(value): + raise ValueError("objective value must be finite") + return value + + @model_validator(mode="after") + def validate_result(self) -> ObjectiveResult: + if self.feasible and self.value is None: + raise ValueError("feasible objective results require a value") + if self.feasible and self.violations: + raise ValueError("feasible objective results cannot contain violations") + return self + + +class EvaluationRecord(StrictModel): + schema_version: Literal[2] = 2 + id: str + request: EvaluationRequest + report: EvaluationReport + backend_id: str + backend: BackendProvenance + principal: EvaluationPrincipal = EvaluationPrincipal.SYSTEM + objective_spec: ObjectiveSpec | None = None + objective: ObjectiveResult | None = None + created_at: datetime + completed_at: datetime + + @field_validator("id", "backend_id") + @classmethod + def validate_identity(cls, value: str) -> str: + return _non_empty(value, "evaluation identity") + + @field_validator("created_at", "completed_at") + @classmethod + def normalize_timestamps(cls, value: datetime) -> datetime: + return _aware_utc(value, "evaluation timestamp") + + @model_validator(mode="after") + def validate_record(self) -> EvaluationRecord: + if (self.objective_spec is None) != (self.objective is None): + raise ValueError( + "objective_spec and objective must both be present or absent" + ) + if self.completed_at < self.created_at: + raise ValueError("completed_at must not be before created_at") + return self + + +class DisclosureLevel(str, Enum): + FULL = "full" + AGGREGATE = "aggregate" + NONE = "none" + + +class EvaluationSummary(StrictModel): + evaluation_id: str + candidate_id: str + candidate_version: str + backend_id: str + evaluation_set: EvaluationSet + status: EvaluationStatus + metrics: dict[str, float] + objective: ObjectiveResult | None + total_cases: int = Field(ge=0) + successful_cases: int = Field(ge=0) + errored_cases: int = Field(ge=0) + skipped_cases: int = Field(ge=0) + + @field_validator("metrics") + @classmethod + def validate_metrics(cls, value: dict[str, float]) -> dict[str, float]: + return _finite_metrics(value) + + @model_validator(mode="after") + def validate_counts(self) -> EvaluationSummary: + total = self.successful_cases + self.errored_cases + self.skipped_cases + if total != self.total_cases: + raise ValueError("case status counts must sum to total_cases") + return self + + +class EvaluationAcknowledgement(StrictModel): + evaluation_id: str + status: EvaluationStatus + + +class EvaluationReceipt(StrictModel): + """Bounded agent-facing pointer to filesystem evaluation feedback.""" + + evaluation_id: str + status: EvaluationStatus + disclosure: DisclosureLevel + result: EvaluationSummary | EvaluationAcknowledgement + result_path: str + + @field_validator("evaluation_id") + @classmethod + def validate_evaluation_id(cls, value: str) -> str: + return _non_empty(value, "evaluation ID") + + @field_validator("result_path") + @classmethod + def validate_result_path(cls, value: str) -> str: + path = PurePosixPath(value) + if ( + not value + or "\\" in value + or path.is_absolute() + or any(part in {"", ".", ".."} for part in value.split("/")) + ): + raise ValueError("receipt result_path must be a safe relative POSIX path") + return value + + @model_validator(mode="after") + def validate_projection(self) -> EvaluationReceipt: + if self.disclosure == DisclosureLevel.NONE: + if not isinstance(self.result, EvaluationAcknowledgement): + raise ValueError("none disclosure requires an acknowledgement") + elif not isinstance(self.result, EvaluationSummary): + raise ValueError("aggregate and full disclosure require a summary") + if ( + self.result.evaluation_id != self.evaluation_id + or self.result.status != self.status + ): + raise ValueError("receipt identity and status must match its result") + return self + + +class EvaluationAuthorization(StrictModel): + may_evaluate: bool + may_view: bool | None = None + meter_budget: bool = True + disclosure: DisclosureLevel = DisclosureLevel.FULL + expose_case_resources: bool = False + # Enforced by the engine for agent-chosen aggregate subsets; 1 = no floor + # (trusted principals). + min_aggregate_cases: int = Field(default=1, ge=1) + reason: str | None = None + + @property + def viewable(self) -> bool: + return self.may_evaluate if self.may_view is None else self.may_view + + @field_validator("reason") + @classmethod + def validate_reason(cls, value: str | None) -> str | None: + return _optional_non_empty(value, "authorization reason") + + +class EvaluationCost(StrictModel): + runs: int = Field(default=1, ge=1) + cases: int | None = Field(default=None, ge=0) + + +class EvaluationBudget(StrictModel): + backend_id: str + evaluation_set_key: str + principal: EvaluationPrincipal = EvaluationPrincipal.AGENT + total_runs: int | None = Field(default=None, ge=0) + remaining_runs: int | None = Field(default=None, ge=0) + total_cases: int | None = Field(default=None, ge=0) + remaining_cases: int | None = Field(default=None, ge=0) + + @field_validator("backend_id", "evaluation_set_key") + @classmethod + def validate_keys(cls, value: str) -> str: + return _non_empty(value, "budget key") + + @model_validator(mode="after") + def validate_remaining(self) -> EvaluationBudget: + if ( + self.total_runs is not None + and self.remaining_runs is not None + and self.remaining_runs > self.total_runs + ): + raise ValueError("remaining_runs cannot exceed total_runs") + if ( + self.total_cases is not None + and self.remaining_cases is not None + and self.remaining_cases > self.total_cases + ): + raise ValueError("remaining_cases cannot exceed total_cases") + return self + + +class EvaluationAccessPolicy(StrictModel): + """Agent visibility and invocation rights for one named evaluation set.""" + + agent_can_evaluate: bool = True + agent_visible: bool = True + agent_selection: AgentSelectionMode = AgentSelectionMode.ARBITRARY + disclosure: DisclosureLevel = DisclosureLevel.FULL + expose_case_resources: bool = False + # k-anonymity floor: agent-chosen aggregate subsets must cover at least + # this many cases, so a hidden per-case result can't be read off one case + # at a time. Omitted resolves to 5 under AGGREGATE disclosure (safe rather + # than unfloored) and 1 otherwise; set explicitly to override. + min_aggregate_cases: int | None = Field(default=None, ge=1) + + @model_validator(mode="after") + def validate_visibility(self) -> EvaluationAccessPolicy: + if self.agent_can_evaluate and not self.agent_visible: + raise ValueError("agent-evaluable evaluations must be agent-visible") + if self.expose_case_resources and not self.agent_visible: + raise ValueError("agent-invisible evaluations cannot expose case resources") + if self.min_aggregate_cases is None: + floor = 5 if self.disclosure == DisclosureLevel.AGGREGATE else 1 + object.__setattr__(self, "min_aggregate_cases", floor) + return self + + +class EvaluationDefinition(StrictModel): + """One named evaluation together with access and principal-scoped budgets.""" + + evaluation_set: EvaluationSet + access: EvaluationAccessPolicy = Field(default_factory=EvaluationAccessPolicy) + agent_budget: EvaluationBudget | None = None + system_budget: EvaluationBudget | None = None + + @model_validator(mode="after") + def validate_budgets(self) -> EvaluationDefinition: + expected_suffix = ( + f":{self.evaluation_set.name}:{self.evaluation_set.partition or ''}" + ) + for principal, budget in ( + (EvaluationPrincipal.AGENT, self.agent_budget), + (EvaluationPrincipal.SYSTEM, self.system_budget), + ): + if budget is None: + continue + if budget.principal != principal: + raise ValueError( + f"{principal.value} budget must use principal {principal.value!r}" + ) + if not budget.evaluation_set_key.endswith(expected_suffix): + raise ValueError( + "evaluation budget key does not match its evaluation set" + ) + return self + + +class EvaluationPlan(StrictModel): + """All evaluations available to one optimization protocol.""" + + evaluations: list[EvaluationDefinition] + selection_evaluation: str + final_evaluation: str | None = None + evaluate_final_baseline: bool = True + + @model_validator(mode="after") + def validate_plan(self) -> EvaluationPlan: + names = [item.evaluation_set.name for item in self.evaluations] + if not names: + raise ValueError("evaluation plan must contain at least one evaluation") + if len(names) != len(set(names)): + raise ValueError("evaluation plan names must be unique") + if self.selection_evaluation not in names: + raise ValueError("selection evaluation is not present in the plan") + if self.final_evaluation is not None: + if self.final_evaluation not in names: + raise ValueError("final evaluation is not present in the plan") + final = self.get(self.final_evaluation) + if final.access.agent_can_evaluate or final.access.agent_visible: + raise ValueError( + "final evaluation must be agent-invisible and not agent-evaluable" + ) + return self + + def get(self, name: str) -> EvaluationDefinition: + for definition in self.evaluations: + if definition.evaluation_set.name == name: + return definition + raise KeyError(name) + + def for_evaluation_set( + self, + evaluation_set: EvaluationSet, + ) -> EvaluationDefinition | None: + for definition in self.evaluations: + owned = definition.evaluation_set + if ( + owned.name == evaluation_set.name + and owned.partition == evaluation_set.partition + ): + return definition + return None + + @property + def selection(self) -> EvaluationDefinition: + return self.get(self.selection_evaluation) + + @property + def final(self) -> EvaluationDefinition | None: + if self.final_evaluation is None: + return None + return self.get(self.final_evaluation) + + @property + def budgets(self) -> list[EvaluationBudget]: + values: list[EvaluationBudget] = [] + for definition in self.evaluations: + if definition.agent_budget is not None: + values.append(definition.agent_budget) + if definition.system_budget is not None: + values.append(definition.system_budget) + return values + + @classmethod + def single( + cls, + evaluation_set: EvaluationSet | None = None, + *, + access: EvaluationAccessPolicy | None = None, + agent_budget: EvaluationBudget | None = None, + system_budget: EvaluationBudget | None = None, + ) -> EvaluationPlan: + resolved = evaluation_set or EvaluationSet() + return cls( + evaluations=[ + EvaluationDefinition( + evaluation_set=resolved, + access=access or EvaluationAccessPolicy(), + agent_budget=agent_budget, + system_budget=system_budget, + ) + ], + selection_evaluation=resolved.name, + ) diff --git a/vero/src/vero/evaluation/scoring/__init__.py b/vero/src/vero/evaluation/scoring/__init__.py new file mode 100644 index 00000000..346c5454 --- /dev/null +++ b/vero/src/vero/evaluation/scoring/__init__.py @@ -0,0 +1,6 @@ +"""Turning raw results into a trustworthy number. + +``objective`` resolves a metric and compares candidates, ``error_taxonomy`` +classifies failures so infrastructure noise is not scored as a bad candidate, and +``security`` redacts secrets from anything shown to the agent. +""" diff --git a/vero/src/vero/evaluation/scoring/error_taxonomy.py b/vero/src/vero/evaluation/scoring/error_taxonomy.py new file mode 100644 index 00000000..beacf7e8 --- /dev/null +++ b/vero/src/vero/evaluation/scoring/error_taxonomy.py @@ -0,0 +1,196 @@ +"""Single source of truth for evaluation error categories and their policy. + +Historically the pipeline used one overloaded channel — an HTTP 429 became an +``infrastructure_failure`` diagnostic became a ``failure_value`` reward — to +represent four genuinely different conditions: inference-budget exhaustion, a +transient rate limit, infrastructure that dropped cases, and a legitimate task +failure. Because they were indistinguishable, the system retried permanent +conditions, penalized candidates for infrastructure they did not cause, and +recorded "nothing shipped" as a real score of zero. + +Every layer that classifies or reacts to an error consults this module instead +of maintaining its own vocabulary: + +- the Harbor backend's sub-run classification (``harbor/backend.py``), +- the evaluation engine's retry/refund contract (``evaluation/engine.py``), +- the optimizer agent's client retry policy (``agents/vero.py``), and +- the error-rate / invalidity logic (``evaluation/evaluator.py``). + +Budget exhaustion cannot be recovered from the exception type once the in- +container inference client collapses the gateway's distinct 429 body code into +a generic rate-limit error. It is therefore detected out of band from the +gateway's own usage ledger and assigned directly, not inferred from the +exception here. This module only classifies the signals that *do* survive as +exception type names or messages. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from enum import Enum + + +class ErrorCategory(str, Enum): + """The canonical categories every layer agrees on.""" + + #: The inference gateway's per-scope token/request budget ran out. A + #: permanent condition for this run: stop loudly, never retry or refund. + INFERENCE_BUDGET_EXHAUSTED = "inference_budget_exhausted" + #: The optimizer agent spent its evaluation budget. Expected and benign: + #: surfaced to the agent as a tool result; never terminates a run. + EVALUATION_BUDGET_EXHAUSTED = "evaluation_budget_exhausted" + #: Authentication/authorization failed (e.g. a wrong or cycled key). Does + #: not heal on retry: terminate loudly. + AUTH_FAILURE = "auth_failure" + #: Transient external infrastructure (rate limit, timeout, connection, 5xx, + #: overloaded). Retryable; if it persists it renders the aggregate invalid. + TRANSIENT_INFRA = "transient_infra" + #: The candidate's harness produced no answer, gave up, or crashed on its + #: own. An informative sample: scored at the failure value, not infra. + TASK_FAILURE = "task_failure" + + +@dataclass(frozen=True) +class CategoryPolicy: + """How the pipeline must treat a category, in one place.""" + + #: May a retry plausibly succeed? Drives client, Harbor, and infra retries. + retryable: bool + #: Should the whole run stop immediately rather than continue? + terminating: bool + #: Does a case in this category count toward the infrastructure-loss + #: fraction that can render the aggregate score invalid? + counts_toward_invalidity: bool + #: Is a case in this category a real, scoreable sample (contributes to the + #: aggregate at the failure value) rather than excluded infrastructure noise? + is_informative_sample: bool + #: Stable diagnostic code emitted for this category. + diagnostic_code: str + + +_POLICIES: dict[ErrorCategory, CategoryPolicy] = { + ErrorCategory.INFERENCE_BUDGET_EXHAUSTED: CategoryPolicy( + retryable=False, + terminating=True, + counts_toward_invalidity=False, + is_informative_sample=False, + diagnostic_code="inference_budget_exhausted", + ), + ErrorCategory.EVALUATION_BUDGET_EXHAUSTED: CategoryPolicy( + retryable=False, + terminating=False, + counts_toward_invalidity=False, + is_informative_sample=False, + diagnostic_code="evaluation_budget_exhausted", + ), + ErrorCategory.AUTH_FAILURE: CategoryPolicy( + retryable=False, + terminating=True, + counts_toward_invalidity=False, + is_informative_sample=False, + diagnostic_code="auth_failure", + ), + ErrorCategory.TRANSIENT_INFRA: CategoryPolicy( + retryable=True, + terminating=False, + counts_toward_invalidity=True, + is_informative_sample=False, + diagnostic_code="transient_infrastructure", + ), + ErrorCategory.TASK_FAILURE: CategoryPolicy( + retryable=False, + terminating=False, + counts_toward_invalidity=False, + is_informative_sample=True, + diagnostic_code="task_failure", + ), +} + + +def policy(category: ErrorCategory) -> CategoryPolicy: + """Return the policy for a category.""" + return _POLICIES[category] + + +#: Diagnostic codes whose presence means the run must stop, unretried and +#: unrefunded. Derived from the policy table so there is one source of truth. +TERMINATING_DIAGNOSTIC_CODES: frozenset[str] = frozenset( + category_policy.diagnostic_code + for category_policy in _POLICIES.values() + if category_policy.terminating +) + + +#: The literal exception-type key the Harbor backend records when the candidate +#: produced no answer without raising (see ``harbor/backend.py``). +NO_REWARD_SIGNAL = "no_rewards_recorded" + + +# Ordered most-specific first. Each regex is matched, case-insensitively, +# against a combined " " string. Budget-like +# credit/quota exhaustion from an upstream provider is treated as an inference +# budget condition; authentication/permission never retries; the remainder are +# transient infrastructure. Anything unmatched is deliberately NOT infra — a +# candidate whose own harness crashes is a task failure, an informative low +# score, not an infrastructure error. +_SIGNAL_PATTERNS: list[tuple[re.Pattern[str], ErrorCategory]] = [ + ( + re.compile( + r"budget.?exhausted|quota|insufficient.?credits|billing", + re.IGNORECASE, + ), + ErrorCategory.INFERENCE_BUDGET_EXHAUSTED, + ), + ( + re.compile( + r"authentication|unauthorized|invalid.?api.?key|permission|forbidden", + re.IGNORECASE, + ), + ErrorCategory.AUTH_FAILURE, + ), + ( + re.compile( + r"rate.?limit|too.?many.?requests|time(?:d.?)?out|connection|" + r"service.?unavailable|internal.?server|overloaded|" + r"bad.?gateway|gateway.?time(?:d.?)?out", + re.IGNORECASE, + ), + ErrorCategory.TRANSIENT_INFRA, + ), +] + + +def classify_signal(signal: str) -> ErrorCategory | None: + """Classify one exception-type name or message string. + + Returns ``None`` for the benign no-answer signal and for anything + unrecognized; callers treat both as :attr:`ErrorCategory.TASK_FAILURE` at + the case level. Recognized infrastructure signals return their category. + """ + if not signal or signal == NO_REWARD_SIGNAL: + return None + for pattern, category in _SIGNAL_PATTERNS: + if pattern.search(signal): + return category + return None + + +def classify_case(signals: list[str]) -> ErrorCategory: + """Classify a case from the exception signals its attempts recorded. + + Precedence follows severity: a terminating condition (budget, then auth) + anywhere wins, then transient infrastructure, otherwise the case is a task + failure. An empty signal set — the candidate simply produced no answer — + is a task failure. + """ + categories = {classify_signal(signal) for signal in signals} + categories.discard(None) + for category in ( + ErrorCategory.INFERENCE_BUDGET_EXHAUSTED, + ErrorCategory.AUTH_FAILURE, + ErrorCategory.TRANSIENT_INFRA, + ): + if category in categories: + return category + return ErrorCategory.TASK_FAILURE diff --git a/vero/src/vero/evaluation/scoring/objective.py b/vero/src/vero/evaluation/scoring/objective.py new file mode 100644 index 00000000..29bc2aa0 --- /dev/null +++ b/vero/src/vero/evaluation/scoring/objective.py @@ -0,0 +1,190 @@ +"""Metric resolution, objective evaluation, ranking, and disclosure.""" + +from __future__ import annotations + +import operator +import statistics +from functools import cmp_to_key +from typing import Iterable + +from vero.evaluation.models import ( + CaseStatus, + ConstraintOperator, + ConstraintViolation, + DisclosureLevel, + EvaluationAcknowledgement, + EvaluationRecord, + EvaluationReport, + EvaluationStatus, + EvaluationSummary, + MetricAggregation, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, +) + +_OPERATORS = { + ConstraintOperator.EQ: operator.eq, + ConstraintOperator.NE: operator.ne, + ConstraintOperator.LT: operator.lt, + ConstraintOperator.LTE: operator.le, + ConstraintOperator.GT: operator.gt, + ConstraintOperator.GTE: operator.ge, +} + + +def resolve_metric(report: EvaluationReport, selector: MetricSelector) -> float | None: + """Resolve a report metric or aggregate it across evaluation cases.""" + if selector.aggregation == MetricAggregation.REPORT: + return report.metrics.get(selector.metric) + + values: list[float] = [] + for case in report.cases: + if case.status == CaseStatus.SUCCESS and selector.metric in case.metrics: + values.append(case.metrics[selector.metric]) + elif selector.case_failure_value is not None: + values.append(selector.case_failure_value) + if not values: + return None + if selector.aggregation == MetricAggregation.MEAN: + return float(statistics.fmean(values)) + if selector.aggregation == MetricAggregation.MEDIAN: + return float(statistics.median(values)) + if selector.aggregation == MetricAggregation.MIN: + return min(values) + if selector.aggregation == MetricAggregation.MAX: + return max(values) + raise AssertionError(f"unsupported aggregation: {selector.aggregation}") + + +def evaluate_objective( + report: EvaluationReport, + specification: ObjectiveSpec, +) -> ObjectiveResult: + """Evaluate the optimization objective and all feasibility constraints.""" + value = resolve_metric(report, specification.selector) + violations: list[ConstraintViolation] = [] + + for constraint in specification.constraints: + observed = resolve_metric(report, constraint.selector) + if observed is None: + violations.append( + ConstraintViolation( + constraint=constraint, + observed=None, + reason=f"metric {constraint.selector.metric!r} is unavailable", + ) + ) + continue + if not _OPERATORS[constraint.operator](observed, constraint.value): + violations.append( + ConstraintViolation( + constraint=constraint, + observed=observed, + reason=( + f"observed {observed} does not satisfy " + f"{constraint.operator.value} {constraint.value}" + ), + ) + ) + + if report.status != EvaluationStatus.SUCCESS or value is None: + return ObjectiveResult(value=specification.failure_value, feasible=False) + if violations: + return ObjectiveResult(value=value, feasible=False, violations=violations) + return ObjectiveResult(value=value, feasible=True) + + +def _compare_results( + left: ObjectiveResult, + right: ObjectiveResult, + direction: str, +) -> int: + if left.feasible != right.feasible: + return 1 if left.feasible else -1 + if left.value is None and right.value is None: + return 0 + if left.value is None: + return -1 + if right.value is None: + return 1 + if left.value == right.value: + return 0 + if direction == "maximize": + return 1 if left.value > right.value else -1 + return 1 if left.value < right.value else -1 + + +def compare_evaluation_records(left: EvaluationRecord, right: EvaluationRecord) -> int: + """Compare compatible records with deterministic candidate tie-breaks.""" + if left.objective_spec is None or left.objective is None: + raise ValueError("left record does not contain an objective") + if right.objective_spec is None or right.objective is None: + raise ValueError("right record does not contain an objective") + if left.objective_spec != right.objective_spec: + raise ValueError("records use different objective specifications") + + comparison = _compare_results( + left.objective, + right.objective, + left.objective_spec.direction, + ) + if comparison: + return comparison + + left_created = left.request.candidate.created_at + right_created = right.request.candidate.created_at + if left_created != right_created: + return 1 if left_created > right_created else -1 + + left_id = left.request.candidate.id + right_id = right.request.candidate.id + if left_id == right_id: + return 0 + return 1 if left_id < right_id else -1 + + +def select_best_evaluation( + records: Iterable[EvaluationRecord], +) -> EvaluationRecord | None: + """Return the best feasible record, or ``None`` if none are feasible.""" + feasible = [ + record + for record in records + if record.objective is not None and record.objective.feasible + ] + if not feasible: + return None + return max(feasible, key=cmp_to_key(compare_evaluation_records)) + + +def project_evaluation( + record: EvaluationRecord, + disclosure: DisclosureLevel, +) -> EvaluationRecord | EvaluationSummary | EvaluationAcknowledgement: + """Project a private record into an approved disclosure shape.""" + if disclosure == DisclosureLevel.FULL: + return record + if disclosure == DisclosureLevel.NONE: + return EvaluationAcknowledgement( + evaluation_id=record.id, + status=record.report.status, + ) + + counts = {status: 0 for status in CaseStatus} + for case in record.report.cases: + counts[case.status] += 1 + return EvaluationSummary( + evaluation_id=record.id, + candidate_id=record.request.candidate.id, + candidate_version=record.request.candidate.version, + backend_id=record.backend_id, + evaluation_set=record.request.evaluation_set, + status=record.report.status, + metrics=dict(record.report.metrics), + objective=record.objective, + total_cases=len(record.report.cases), + successful_cases=counts[CaseStatus.SUCCESS], + errored_cases=counts[CaseStatus.ERROR], + skipped_cases=counts[CaseStatus.SKIPPED], + ) diff --git a/vero/src/vero/evaluation/scoring/security.py b/vero/src/vero/evaluation/scoring/security.py new file mode 100644 index 00000000..64967a46 --- /dev/null +++ b/vero/src/vero/evaluation/scoring/security.py @@ -0,0 +1,108 @@ +"""Secret-safe normalization for persisted evaluation content.""" + +from __future__ import annotations + +from typing import Any, Iterable + +from vero.evaluation.models import EvaluationReport + + +def _usable_secrets(secrets: Iterable[str | None]) -> tuple[str, ...]: + return tuple( + sorted( + {secret for secret in secrets if secret and len(secret) >= 4}, + key=len, + reverse=True, + ) + ) + + +def sanitize_text(text: str, secrets: Iterable[str | None]) -> str: + sanitized = text + for secret in _usable_secrets(secrets): + sanitized = sanitized.replace(secret, "[REDACTED]") + return sanitized + + +def _sanitize_value(value: Any, secrets: tuple[str, ...]) -> Any: + if isinstance(value, str): + return sanitize_text(value, secrets) + if isinstance(value, list): + return [_sanitize_value(item, secrets) for item in value] + if isinstance(value, dict): + return {key: _sanitize_value(item, secrets) for key, item in value.items()} + return value + + +def sanitize_evaluation_report( + report: EvaluationReport, + secrets: Iterable[str | None], +) -> EvaluationReport: + """Redact free-form report fields while retaining structural identifiers.""" + secret_values = _usable_secrets(secrets) + if not secret_values: + return report + + diagnostics = [ + diagnostic.model_copy( + update={ + "message": sanitize_text(diagnostic.message, secret_values), + "metadata": _sanitize_value(diagnostic.metadata, secret_values), + } + ) + for diagnostic in report.diagnostics + ] + cases = [] + for case in report.cases: + errors = [ + error.model_copy( + update={ + "message": sanitize_text(error.message, secret_values), + "metadata": _sanitize_value(error.metadata, secret_values), + } + ) + for error in case.errors + ] + artifacts = [ + artifact.model_copy( + update={ + "description": sanitize_text(artifact.description, secret_values) + if artifact.description is not None + else None + } + ) + for artifact in case.artifacts + ] + cases.append( + case.model_copy( + update={ + "input": _sanitize_value(case.input, secret_values), + "output": _sanitize_value(case.output, secret_values), + "feedback": sanitize_text(case.feedback, secret_values) + if case.feedback is not None + else None, + "errors": errors, + "execution_trace": _sanitize_value(case.execution_trace, secret_values), + "evaluation_trace": _sanitize_value(case.evaluation_trace, secret_values), + "metadata": _sanitize_value(case.metadata, secret_values), + "artifacts": artifacts, + } + ) + ) + artifacts = [ + artifact.model_copy( + update={ + "description": sanitize_text(artifact.description, secret_values) + if artifact.description is not None + else None + } + ) + for artifact in report.artifacts + ] + return report.model_copy( + update={ + "diagnostics": diagnostics, + "cases": cases, + "artifacts": artifacts, + } + ) diff --git a/vero/src/vero/evaluation/store/__init__.py b/vero/src/vero/evaluation/store/__init__.py new file mode 100644 index 00000000..4ce97217 --- /dev/null +++ b/vero/src/vero/evaluation/store/__init__.py @@ -0,0 +1,6 @@ +"""Durable evaluation state. + +``persistence`` holds evaluation records, manifests, and the partitioned case +store; ``budget`` holds the ledger that caps runs and cases per partition. Both +write through to disk so a restarted process resumes with its limits intact. +""" diff --git a/vero/src/vero/evaluation/store/budget.py b/vero/src/vero/evaluation/store/budget.py new file mode 100644 index 00000000..c6bd4595 --- /dev/null +++ b/vero/src/vero/evaluation/store/budget.py @@ -0,0 +1,241 @@ +"""Durable, backend-qualified evaluation budget reservations.""" + +from __future__ import annotations + +import asyncio +import json +from pathlib import Path +from typing import Any + +from vero.evaluation.exceptions import EvaluationBudgetExceeded +from vero.evaluation.models import ( + EvaluationBudget, + EvaluationCost, + EvaluationPrincipal, + EvaluationSet, +) +from vero.evaluation.store.persistence import _atomic_write_json + + +async def _drain_write( + task: asyncio.Task[None], +) -> tuple[BaseException | None, asyncio.CancelledError | None]: + """Run a durable write to completion and report both possible outcomes. + + Returns ``(write_error, cancellation)`` and never raises. Draining to + completion matters because a write left running in the background could + overwrite a later one once the ledger lock is released; returning both + outcomes rather than raising one matters because a cancellation must never + be swallowed by a write failure. Every call site below got that second part + wrong at least once while each was hand-rolling this loop, which is why + there is now one copy. + """ + cancellation: asyncio.CancelledError | None = None + while not task.done(): + try: + await asyncio.shield(task) + except asyncio.CancelledError as error: + cancellation = error + except Exception: + # The write failed, so the task is done: stop draining and let the + # caller decide what the failure means. + break + write_error = None if task.cancelled() else task.exception() + return write_error, cancellation + + +class BudgetLedger: + """Atomically meter completed evaluations against configured budgets.""" + + schema_version = 1 + + def __init__( + self, + budgets: list[EvaluationBudget] | None = None, + *, + path: Path | None = None, + ): + self.path = path + self._lock = asyncio.Lock() + self._budgets: dict[tuple[str, str, EvaluationPrincipal], EvaluationBudget] = {} + for budget in budgets or []: + self.add(budget) + + @property + def budgets(self) -> list[EvaluationBudget]: + return list(self._budgets.values()) + + def add(self, budget: EvaluationBudget) -> None: + key = (budget.backend_id, budget.evaluation_set_key, budget.principal) + if key in self._budgets: + raise ValueError(f"duplicate evaluation budget for {key!r}") + updates: dict[str, int] = {} + if budget.total_runs is not None and budget.remaining_runs is None: + updates["remaining_runs"] = budget.total_runs + if budget.total_cases is not None and budget.remaining_cases is None: + updates["remaining_cases"] = budget.total_cases + self._budgets[key] = budget.model_copy(update=updates) + + def get( + self, + backend_id: str, + evaluation_set: EvaluationSet, + principal: EvaluationPrincipal = EvaluationPrincipal.AGENT, + ) -> EvaluationBudget | None: + return self._budgets.get( + (backend_id, evaluation_set.budget_key(backend_id), principal) + ) + + def _write_task( + self, + state: dict[tuple[str, str, EvaluationPrincipal], EvaluationBudget], + ) -> asyncio.Task[None]: + """Start a durable write of one ledger state. Drain it with _drain_write.""" + return asyncio.create_task( + asyncio.to_thread(_atomic_write_json, self.path, self._serialize(state)) + ) + + async def reserve( + self, + backend_id: str, + evaluation_set: EvaluationSet, + cost: EvaluationCost, + principal: EvaluationPrincipal = EvaluationPrincipal.AGENT, + ) -> EvaluationBudget | None: + key = (backend_id, evaluation_set.budget_key(backend_id), principal) + async with self._lock: + budget = self._budgets.get(key) + if budget is None: + return None + if cost.cases is None and budget.remaining_cases is not None: + raise EvaluationBudgetExceeded( + "evaluation case cost is unknown but the budget has a case limit" + ) + if budget.remaining_runs is not None and cost.runs > budget.remaining_runs: + raise EvaluationBudgetExceeded("evaluation run budget exhausted") + if ( + budget.remaining_cases is not None + and cost.cases is not None + and cost.cases > budget.remaining_cases + ): + raise EvaluationBudgetExceeded("evaluation case budget exhausted") + + updates: dict[str, int] = {} + if budget.remaining_runs is not None: + updates["remaining_runs"] = budget.remaining_runs - cost.runs + if budget.remaining_cases is not None and cost.cases is not None: + updates["remaining_cases"] = budget.remaining_cases - cost.cases + updated = budget.model_copy(update=updates) + snapshot = dict(self._budgets) + snapshot[key] = updated + if self.path is not None: + write_error, cancellation = await _drain_write( + self._write_task(snapshot) + ) + if cancellation is not None: + # The run requesting this reservation is being cancelled and + # will never use it. Roll the durable charge back so the + # reservation is atomic — committed in full or not at all — + # and a cancelled reservation never leaks budget. (In-memory + # state was never updated, so we rewrite it as the truth.) + # Nothing to roll back if the charge never landed. + if write_error is None: + write_error, _ = await _drain_write( + self._write_task(self._budgets) + ) + # Chain whichever write failed so the durable-state + # inconsistency stays diagnosable, but always propagate the + # cancellation itself. + raise cancellation from write_error + if write_error is not None: + raise write_error + self._budgets = snapshot + return updated + + async def refund( + self, + backend_id: str, + evaluation_set: EvaluationSet, + cost: EvaluationCost, + principal: EvaluationPrincipal = EvaluationPrincipal.AGENT, + ) -> EvaluationBudget | None: + """Undo a reservation when execution produced no usable result.""" + + key = (backend_id, evaluation_set.budget_key(backend_id), principal) + async with self._lock: + budget = self._budgets.get(key) + if budget is None: + return None + updates: dict[str, int] = {} + if budget.remaining_runs is not None: + restored_runs = budget.remaining_runs + cost.runs + updates["remaining_runs"] = ( + min(restored_runs, budget.total_runs) + if budget.total_runs is not None + else restored_runs + ) + if budget.remaining_cases is not None and cost.cases is not None: + restored_cases = budget.remaining_cases + cost.cases + updates["remaining_cases"] = ( + min(restored_cases, budget.total_cases) + if budget.total_cases is not None + else restored_cases + ) + updated = budget.model_copy(update=updates) + snapshot = dict(self._budgets) + snapshot[key] = updated + if self.path is not None: + write_error, cancellation = await _drain_write( + self._write_task(snapshot) + ) + if write_error is not None: + # A failed durable refund must not swallow the cancellation + # of the run being torn down; chain the write error. + if cancellation is not None: + raise cancellation from write_error + raise write_error + # Unlike reserve, a refund that landed is never rolled back: it + # only ever returns budget, so committing it is safe even for a + # run that is going away. + self._budgets = snapshot + if cancellation is not None: + raise cancellation + return updated + self._budgets = snapshot + return updated + + def _serialize( + self, + budgets: dict[tuple[str, str, EvaluationPrincipal], EvaluationBudget] + | None = None, + ) -> dict[str, Any]: + values = budgets if budgets is not None else self._budgets + return { + "schema_version": self.schema_version, + "budgets": [ + budget.model_dump(mode="json") for _, budget in sorted(values.items()) + ], + } + + def save(self) -> None: + if self.path is None: + raise ValueError("budget ledger has no persistence path") + _atomic_write_json(self.path, self._serialize()) + + @classmethod + def load(cls, path: Path) -> BudgetLedger: + if not path.exists(): + return cls(path=path) + try: + payload = json.loads(path.read_text(encoding="utf-8")) + if payload.get("schema_version") != cls.schema_version: + raise ValueError("unsupported budget ledger schema") + budgets = [ + EvaluationBudget.model_validate(value) + for value in payload.get("budgets", []) + ] + except Exception as error: + raise ValueError( + f"invalid durable budget ledger {path}: {error}" + ) from error + return cls(budgets, path=path) diff --git a/vero/src/vero/evaluation/store/persistence.py b/vero/src/vero/evaluation/store/persistence.py new file mode 100644 index 00000000..e8c3ca49 --- /dev/null +++ b/vero/src/vero/evaluation/store/persistence.py @@ -0,0 +1,495 @@ +"""Atomic persistence for canonical evaluation records.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +import os +import tempfile +import threading +from dataclasses import dataclass, field +from datetime import datetime +from pathlib import Path +from typing import Any, Callable, Literal + +from pydantic import Field, field_validator, model_validator + +from vero.candidate import Candidate +from vero.evaluation.models import ( + BackendProvenance, + CaseResult, + EvaluationPrincipal, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + ObjectiveResult, + ObjectiveSpec, +) +from vero.evaluation.scoring.objective import select_best_evaluation +from vero.models import StrictModel + +logger = logging.getLogger(__name__) + + +def _atomic_write_json(path: Path, value: Any) -> None: + """Write JSON through a same-directory temporary file and atomic replace.""" + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + ) + temporary_path = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(value, handle, ensure_ascii=False, indent=2) + handle.write("\n") + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary_path, path) + except BaseException: + try: + os.close(descriptor) + except OSError: + pass + temporary_path.unlink(missing_ok=True) + raise + + +def _case_digest(case_id: str) -> str: + return hashlib.sha256(case_id.encode()).hexdigest() + + +class CaseFileReference(StrictModel): + case_id: str + path: str + + @field_validator("case_id") + @classmethod + def validate_case_id(cls, value: str) -> str: + if not value.strip(): + raise ValueError("case ID must not be empty") + return value + + @model_validator(mode="after") + def validate_path(self) -> CaseFileReference: + expected = f"cases/{_case_digest(self.case_id)}.json" + if self.path != expected: + raise ValueError(f"case path must be {expected!r}") + return self + + +class EvaluationManifest(StrictModel): + schema_version: Literal[1] = 1 + lifecycle: Literal["complete"] = "complete" + id: str + request: EvaluationRequest + report: EvaluationReport + case_files: list[CaseFileReference] = Field(default_factory=list) + backend_id: str + backend: BackendProvenance + # Persist the principal so a record reconstructed from this source-of-truth + # directory keeps its true provenance instead of silently defaulting to + # SYSTEM. Older manifests without the field read back as SYSTEM, matching + # the historical EvaluationRecord default. + principal: EvaluationPrincipal = EvaluationPrincipal.SYSTEM + objective_spec: ObjectiveSpec | None = None + objective: ObjectiveResult | None = None + created_at: datetime + completed_at: datetime + + @model_validator(mode="after") + def validate_manifest(self) -> EvaluationManifest: + if self.report.cases: + raise ValueError("manifest report must store cases in case_files") + ids = [reference.case_id for reference in self.case_files] + if len(ids) != len(set(ids)): + raise ValueError("case file references must be unique") + if (self.objective_spec is None) != (self.objective is None): + raise ValueError( + "objective_spec and objective must both be present or absent" + ) + return self + + @classmethod + def from_record(cls, record: EvaluationRecord) -> EvaluationManifest: + return cls( + id=record.id, + request=record.request, + report=record.report.model_copy(update={"cases": []}), + case_files=[ + CaseFileReference( + case_id=case.case_id, + path=f"cases/{_case_digest(case.case_id)}.json", + ) + for case in record.report.cases + ], + backend_id=record.backend_id, + backend=record.backend, + principal=record.principal, + objective_spec=record.objective_spec, + objective=record.objective, + created_at=record.created_at, + completed_at=record.completed_at, + ) + + +class RunningEvaluationManifest(StrictModel): + schema_version: Literal[1] = 1 + lifecycle: Literal["running"] = "running" + id: str + request: EvaluationRequest + backend_id: str + backend: BackendProvenance + objective_spec: ObjectiveSpec | None = None + created_at: datetime + + +class CaseCheckpointStore: + """Per-evaluation case checkpoints safe for concurrent async writers.""" + + def __init__(self, cases_dir: Path): + self.cases_dir = cases_dir + self._lock = asyncio.Lock() + + def path_for(self, case_id: str) -> Path: + return self.cases_dir / f"{_case_digest(case_id)}.json" + + async def save(self, result: CaseResult) -> None: + if not isinstance(result, CaseResult): + raise TypeError("result must be a CaseResult") + async with self._lock: + await asyncio.to_thread( + _atomic_write_json, + self.path_for(result.case_id), + result.model_dump(mode="json"), + ) + + async def load(self, case_id: str) -> CaseResult | None: + path = self.path_for(case_id) + async with self._lock: + if not path.exists(): + return None + try: + result = CaseResult.model_validate_json( + path.read_text(encoding="utf-8") + ) + except Exception as error: + raise ValueError(f"invalid case checkpoint {path}: {error}") from error + if result.case_id != case_id: + raise ValueError( + f"case checkpoint {path} contains ID {result.case_id!r}, expected {case_id!r}" + ) + return result + + async def load_all(self) -> list[CaseResult]: + async with self._lock: + paths = ( + sorted(self.cases_dir.glob("*.json")) if self.cases_dir.exists() else [] + ) + results: list[CaseResult] = [] + for path in paths: + try: + results.append( + CaseResult.model_validate_json(path.read_text(encoding="utf-8")) + ) + except Exception as error: + raise ValueError( + f"invalid case checkpoint {path}: {error}" + ) from error + return results + + +class EvaluationStore: + """Persist and reconstruct one evaluation directory.""" + + manifest_basename = "evaluation.json" + + def __init__(self, result_dir: Path): + self.result_dir = result_dir + self.cases = CaseCheckpointStore(result_dir / "cases") + self.artifact_dir = result_dir / "artifacts" + + @property + def manifest_path(self) -> Path: + return self.result_dir / self.manifest_basename + + def _validate_artifacts(self, record: EvaluationRecord) -> None: + artifact_root = self.artifact_dir.resolve() + artifacts = list(record.report.artifacts) + for case in record.report.cases: + artifacts.extend(case.artifacts) + for artifact in artifacts: + resolved = (self.artifact_dir / artifact.path).resolve() + if not resolved.is_relative_to(artifact_root): + raise ValueError( + f"artifact path {artifact.path!r} escapes evaluation artifact directory" + ) + + def write_running( + self, + *, + evaluation_id: str, + request: EvaluationRequest, + backend_id: str, + backend: BackendProvenance, + objective_spec: ObjectiveSpec | None, + created_at: datetime, + ) -> None: + self.result_dir.mkdir(parents=True, exist_ok=True) + manifest = RunningEvaluationManifest( + id=evaluation_id, + request=request, + backend_id=backend_id, + backend=backend, + objective_spec=objective_spec, + created_at=created_at, + ) + _atomic_write_json(self.manifest_path, manifest.model_dump(mode="json")) + + async def save(self, record: EvaluationRecord) -> None: + self.result_dir.mkdir(parents=True, exist_ok=True) + self.artifact_dir.mkdir(parents=True, exist_ok=True) + self._validate_artifacts(record) + for case in record.report.cases: + await self.cases.save(case) + _atomic_write_json( + self.manifest_path, + EvaluationManifest.from_record(record).model_dump(mode="json"), + ) + + def load(self) -> EvaluationRecord: + try: + manifest = EvaluationManifest.model_validate_json( + self.manifest_path.read_text(encoding="utf-8") + ) + except Exception as error: + raise ValueError( + f"invalid evaluation manifest {self.manifest_path}: {error}" + ) from error + + cases: list[CaseResult] = [] + for reference in manifest.case_files: + case_path = self.result_dir / reference.path + try: + case = CaseResult.model_validate_json( + case_path.read_text(encoding="utf-8") + ) + except Exception as error: + raise ValueError( + f"invalid evaluation case file {case_path}: {error}" + ) from error + if case.case_id != reference.case_id: + raise ValueError( + f"evaluation case file {case_path} contains ID {case.case_id!r}, " + f"expected {reference.case_id!r}" + ) + cases.append(case) + + record = EvaluationRecord( + id=manifest.id, + request=manifest.request, + report=manifest.report.model_copy(update={"cases": cases}), + backend_id=manifest.backend_id, + backend=manifest.backend, + principal=manifest.principal, + objective_spec=manifest.objective_spec, + objective=manifest.objective, + created_at=manifest.created_at, + completed_at=manifest.completed_at, + ) + self._validate_artifacts(record) + return record + + +@dataclass +class EvaluationDatabase: + """Thread-safe in-memory index of candidates and evaluation records.""" + + id: str + candidates: dict[str, Candidate] = field(default_factory=dict) + evaluations: dict[str, EvaluationRecord] = field(default_factory=dict) + listeners: list[Callable[[EvaluationRecord], None]] = field( + default_factory=list, + repr=False, + ) + _lock: threading.RLock = field( + default_factory=threading.RLock, init=False, repr=False + ) + + def add_evaluation(self, record: EvaluationRecord) -> None: + with self._lock: + existing_record = self.evaluations.get(record.id) + if existing_record is not None: + if existing_record != record: + raise ValueError( + f"evaluation ID {record.id!r} already has a different record" + ) + return + candidate = record.request.candidate + existing_candidate = self.candidates.get(candidate.id) + if existing_candidate is not None and existing_candidate != candidate: + raise ValueError( + f"candidate ID {candidate.id!r} already has a different identity" + ) + self.candidates[candidate.id] = candidate + self.evaluations[record.id] = record + for listener in self.listeners: + listener(record) + + def get_evaluation(self, evaluation_id: str) -> EvaluationRecord | None: + with self._lock: + return self.evaluations.get(evaluation_id) + + def get_evaluations( + self, + evaluation_ids: list[str] | None = None, + *, + limit: int | None = None, + offset: int = 0, + reverse: bool = False, + filter_fn: Callable[[EvaluationRecord], bool] | None = None, + ) -> list[EvaluationRecord]: + with self._lock: + ids = list(self.evaluations) if evaluation_ids is None else evaluation_ids + records = [self.evaluations[evaluation_id] for evaluation_id in ids] + if filter_fn is not None: + records = [record for record in records if filter_fn(record)] + if reverse: + records.reverse() + records = records[offset:] + return records if limit is None else records[:limit] + + def get_best( + self, + objective_spec: ObjectiveSpec, + *, + backend_ids: set[str] | None = None, + evaluation_sets: list[EvaluationSet] | None = None, + exclude_candidate_id: str | None = None, + ) -> EvaluationRecord | None: + with self._lock: + records = [ + record + for record in self.evaluations.values() + if record.objective_spec == objective_spec + and (backend_ids is None or record.backend_id in backend_ids) + and ( + evaluation_sets is None + or record.request.evaluation_set in evaluation_sets + ) + and record.request.candidate.id != exclude_candidate_id + ] + return select_best_evaluation(records) + + def serialize(self) -> dict[str, Any]: + with self._lock: + return { + "schema_version": 1, + "id": self.id, + "candidates": { + candidate_id: candidate.model_dump(mode="json") + for candidate_id, candidate in self.candidates.items() + }, + "evaluations": { + evaluation_id: record.model_dump(mode="json") + for evaluation_id, record in self.evaluations.items() + }, + } + + @classmethod + def deserialize(cls, data: dict[str, Any]) -> EvaluationDatabase: + if data.get("schema_version") != 1: + raise ValueError("unsupported evaluation database schema") + database = cls(id=data["id"]) + for candidate_id, value in data.get("candidates", {}).items(): + candidate = Candidate.model_validate(value) + if candidate.id != candidate_id: + raise ValueError( + f"candidate map key {candidate_id!r} does not match candidate ID {candidate.id!r}" + ) + database.candidates[candidate_id] = candidate + for evaluation_id, value in data.get("evaluations", {}).items(): + record = EvaluationRecord.model_validate(value) + if record.id != evaluation_id: + raise ValueError( + f"evaluation map key {evaluation_id!r} does not match record ID {record.id!r}" + ) + database.add_evaluation(record) + return database + + def to_json(self) -> str: + return json.dumps(self.serialize(), ensure_ascii=False, indent=2) + + @classmethod + def from_json(cls, value: str) -> EvaluationDatabase: + return cls.deserialize(json.loads(value)) + + def save_to_file(self, path: Path) -> None: + with self._lock: + _atomic_write_json(path, self.serialize()) + + @classmethod + def load_from_file(cls, path: Path) -> EvaluationDatabase: + return cls.from_json(path.read_text(encoding="utf-8")) + + @classmethod + def load_reconciled( + cls, + *, + database_path: Path, + evaluations_dir: Path, + database_id: str, + ) -> EvaluationDatabase: + """Load the index and repair it from canonical completed evaluations.""" + + existed = database_path.exists() + database = cls.load_from_file(database_path) if existed else cls(id=database_id) + if database.id != database_id: + raise ValueError( + f"evaluation database belongs to {database.id!r}, not {database_id!r}" + ) + + completed = cls.from_evaluations_dir( + evaluations_dir, + database_id=database_id, + ) + changed = not existed + for evaluation_id, record in completed.evaluations.items(): + if evaluation_id not in database.evaluations: + changed = True + database.add_evaluation(record) + if changed: + database.save_to_file(database_path) + return database + + @classmethod + def from_evaluations_dir( + cls, + evaluations_dir: Path, + *, + database_id: str | None = None, + ) -> EvaluationDatabase: + database = cls(id=database_id or evaluations_dir.parent.name) + if not evaluations_dir.exists(): + return database + for result_dir in sorted(evaluations_dir.iterdir()): + if not result_dir.is_dir(): + continue + if not (result_dir / EvaluationStore.manifest_basename).exists(): + continue + try: + manifest = json.loads( + (result_dir / EvaluationStore.manifest_basename).read_text( + encoding="utf-8" + ) + ) + if manifest.get("lifecycle", "complete") != "complete": + continue + database.add_evaluation(EvaluationStore(result_dir).load()) + except Exception as error: + logger.warning("Skipping corrupt evaluation %s: %s", result_dir, error) + return database diff --git a/vero/src/vero/exceptions.py b/vero/src/vero/exceptions.py new file mode 100644 index 00000000..b8d4033f --- /dev/null +++ b/vero/src/vero/exceptions.py @@ -0,0 +1,5 @@ +import asyncio + + +class StreamEventTimeout(asyncio.TimeoutError): + pass diff --git a/vero/src/vero/gateway/__init__.py b/vero/src/vero/gateway/__init__.py new file mode 100644 index 00000000..d72a27e2 --- /dev/null +++ b/vero/src/vero/gateway/__init__.py @@ -0,0 +1,12 @@ +"""The inference gateway: the only path from a task container to a model. + +Runs as its own container beside ``main`` and ``eval-sidecar``. It holds the one +real upstream credential and hands each consumer a separate scoped token, so the +optimizer cannot spend from the evaluation or finalization pools. Every proxied +request is checked three ways: the presented token against that scope's digest, +the requested model against the scope's allow-list, and the running budget +against a ledger written through to disk. + +Provider-agnostic by design — it accepts either ``Authorization: Bearer`` or +``x-api-key``, and forwards any endpoint to the upstream proxy. +""" diff --git a/vero/src/vero/gateway/inference.py b/vero/src/vero/gateway/inference.py new file mode 100644 index 00000000..edffd5f7 --- /dev/null +++ b/vero/src/vero/gateway/inference.py @@ -0,0 +1,969 @@ +"""Credential-isolating, budgeted inference gateway for Harbor tasks.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +import os +import re +import secrets +import time +from collections import OrderedDict +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from datetime import UTC, datetime +from pathlib import Path +from typing import Annotated, Any, Literal + +from fastapi import FastAPI, Header, HTTPException, Request +from fastapi.responses import JSONResponse, Response, StreamingResponse +from pydantic import Field, field_validator, model_validator + +from vero.evaluation.store.persistence import _atomic_write_json +from vero.layout import LAYOUT +from vero.models import StrictModel + +logger = logging.getLogger(__name__) + + +def token_digest(token: str) -> str: + """Return the stable digest stored in trusted gateway configuration.""" + if not token.strip(): + raise ValueError("inference token must not be empty") + return hashlib.sha256(token.encode("utf-8")).hexdigest() + + +def generate_inference_token() -> str: + return secrets.token_urlsafe(32) + + +class InferenceScopeConfig(StrictModel): + """One independently authenticated and metered inference consumer.""" + + token_sha256: str + allowed_models: list[str] + max_requests: int | None = Field(default=None, ge=1) + max_tokens: int | None = Field(default=None, ge=1) + max_concurrency: int = Field(default=8, ge=1) + + @field_validator("token_sha256") + @classmethod + def validate_token_digest(cls, value: str) -> str: + if len(value) != 64 or any( + character not in "0123456789abcdef" for character in value + ): + raise ValueError("token_sha256 must be a lowercase SHA-256 digest") + return value + + @field_validator("allowed_models") + @classmethod + def validate_models(cls, value: list[str]) -> list[str]: + if not value or any(not model.strip() for model in value): + raise ValueError("allowed_models must contain non-empty model names") + if len(value) != len(set(value)): + raise ValueError("allowed_models must be unique") + return value + + +class InferenceRequestLogConfig(StrictModel): + """Durable JSONL capture of every request the gateway proxies or denies. + + Operator telemetry: the log lives on the gateway state volume, which is + never agent-visible. Bodies are stored head+tail truncated so the terminal + usage frame of a stream survives; ``body_bytes: 0`` keeps metadata only. + """ + + directory: str = LAYOUT.inference_request_log_dir + body_bytes: int = Field(default=16384, ge=0) + rotate_bytes: int = Field(default=64 * 1024 * 1024, ge=1_048_576) + # Experimental: stamp each record with a provider-agnostic conversation + # thread (see _RequestAttributor) for post-hoc per-trial accounting. + attribution: bool = False + + @field_validator("directory") + @classmethod + def validate_directory(cls, value: str) -> str: + if not Path(value).is_absolute(): + raise ValueError("request log directory must be absolute") + return value + + +class InferenceGatewayConfig(StrictModel): + """Trusted configuration for an OpenAI-compatible inference gateway.""" + + upstream_api_key_env: str = "OPENAI_API_KEY" + upstream_base_url_env: str | None = "OPENAI_BASE_URL" + default_upstream_base_url: str = "https://api.openai.com/v1" + state_path: str = LAYOUT.inference_state + request_log: InferenceRequestLogConfig | None = None + scopes: dict[str, InferenceScopeConfig] + + @field_validator("upstream_api_key_env", "upstream_base_url_env") + @classmethod + def validate_environment_name(cls, value: str | None) -> str | None: + if value is not None and ( + not value + or not (value[0].isalpha() or value[0] == "_") + or any(not (character.isalnum() or character == "_") for character in value) + ): + raise ValueError("gateway environment names must be valid identifiers") + return value + + @field_validator("default_upstream_base_url") + @classmethod + def validate_upstream_url(cls, value: str) -> str: + if not value.startswith(("http://", "https://")): + raise ValueError("default_upstream_base_url must be HTTP(S)") + return value.rstrip("/") + + @field_validator("state_path") + @classmethod + def validate_state_path(cls, value: str) -> str: + if not Path(value).is_absolute(): + raise ValueError("gateway state_path must be absolute") + return value + + @model_validator(mode="after") + def validate_scopes(self) -> InferenceGatewayConfig: + if not self.scopes: + raise ValueError("gateway requires at least one scope") + for name in self.scopes: + if not name or any( + not (character.isalnum() or character in "_-") for character in name + ): + raise ValueError(f"invalid inference scope {name!r}") + return self + + +class InferenceAttributionUsage(StrictModel): + requests: int = Field(default=0, ge=0) + upstream_errors: int = Field(default=0, ge=0) + input_tokens: int = Field(default=0, ge=0) + # Provider-reported cache reads; a subset of input_tokens, not additive. + cached_input_tokens: int = Field(default=0, ge=0) + output_tokens: int = Field(default=0, ge=0) + total_tokens: int = Field(default=0, ge=0) + + +class InferenceScopeUsage(InferenceAttributionUsage): + active_requests: int = Field(default=0, ge=0) + attributions: dict[str, InferenceAttributionUsage] = Field(default_factory=dict) + + +class InferenceUsageLedger(StrictModel): + schema_version: Literal[1] = 1 + scopes: dict[str, InferenceScopeUsage] + + +class InferenceBudgetExceeded(RuntimeError): + pass + + +class InferenceUsageStore: + """Cancellation-safe request reservations and durable provider usage.""" + + def __init__(self, config: InferenceGatewayConfig): + self.config = config + self.path = Path(config.state_path) + self._lock = asyncio.Lock() + self._limits = { + name: asyncio.Semaphore(scope.max_concurrency) + for name, scope in config.scopes.items() + } + self.ledger = self._load() + + def _load(self) -> InferenceUsageLedger: + if self.path.exists(): + loaded = InferenceUsageLedger.model_validate_json( + self.path.read_text(encoding="utf-8") + ) + unknown = set(loaded.scopes) - set(self.config.scopes) + if unknown: + raise ValueError( + "inference usage contains unknown scopes: " + + ", ".join(sorted(unknown)) + ) + scopes = dict(loaded.scopes) + for name in self.config.scopes: + scopes.setdefault(name, InferenceScopeUsage()) + # Active requests cannot survive a gateway restart. + scopes = { + name: value.model_copy(update={"active_requests": 0}) + for name, value in scopes.items() + } + return loaded.model_copy(update={"scopes": scopes}) + return InferenceUsageLedger( + scopes={name: InferenceScopeUsage() for name in self.config.scopes} + ) + + def _save(self) -> None: + _atomic_write_json(self.path, self.ledger.model_dump(mode="json")) + + async def reserve(self, scope_name: str, attribution: str) -> None: + limiter = self._limits[scope_name] + await limiter.acquire() + try: + async with self._lock: + limits = self.config.scopes[scope_name] + usage = self.ledger.scopes[scope_name] + if ( + limits.max_requests is not None + and usage.requests >= limits.max_requests + ): + raise InferenceBudgetExceeded("inference request budget exhausted") + if ( + limits.max_tokens is not None + and usage.total_tokens >= limits.max_tokens + ): + raise InferenceBudgetExceeded("inference token budget exhausted") + attribution_usage = usage.attributions.get( + attribution, InferenceAttributionUsage() + ) + usage = usage.model_copy( + update={ + "requests": usage.requests + 1, + "active_requests": usage.active_requests + 1, + "attributions": { + **usage.attributions, + attribution: attribution_usage.model_copy( + update={"requests": attribution_usage.requests + 1} + ), + }, + } + ) + self.ledger = self.ledger.model_copy( + update={"scopes": {**self.ledger.scopes, scope_name: usage}} + ) + self._save() + except BaseException: + limiter.release() + raise + + async def complete( + self, + scope_name: str, + attribution: str, + *, + input_tokens: int = 0, + output_tokens: int = 0, + total_tokens: int = 0, + cached_input_tokens: int = 0, + upstream_error: bool = False, + ) -> None: + try: + async with self._lock: + usage = self.ledger.scopes[scope_name] + attribution_usage = usage.attributions[attribution] + update = { + "upstream_errors": usage.upstream_errors + int(upstream_error), + "input_tokens": usage.input_tokens + input_tokens, + "cached_input_tokens": usage.cached_input_tokens + + cached_input_tokens, + "output_tokens": usage.output_tokens + output_tokens, + "total_tokens": usage.total_tokens + total_tokens, + "active_requests": max(0, usage.active_requests - 1), + "attributions": { + **usage.attributions, + attribution: attribution_usage.model_copy( + update={ + "upstream_errors": attribution_usage.upstream_errors + + int(upstream_error), + "input_tokens": attribution_usage.input_tokens + + input_tokens, + "cached_input_tokens": ( + attribution_usage.cached_input_tokens + + cached_input_tokens + ), + "output_tokens": attribution_usage.output_tokens + + output_tokens, + "total_tokens": attribution_usage.total_tokens + + total_tokens, + } + ), + }, + } + self.ledger = self.ledger.model_copy( + update={ + "scopes": { + **self.ledger.scopes, + scope_name: usage.model_copy(update=update), + } + } + ) + self._save() + finally: + self._limits[scope_name].release() + + def public_status(self) -> dict[str, Any]: + result: dict[str, Any] = {} + for name, scope in self.config.scopes.items(): + usage = self.ledger.scopes[name] + result[name] = { + **usage.model_dump(mode="json"), + "max_requests": scope.max_requests, + "max_tokens": scope.max_tokens, + "max_concurrency": scope.max_concurrency, + "allowed_models": list(scope.allowed_models), + "remaining_requests": ( + None + if scope.max_requests is None + else max(0, scope.max_requests - usage.requests) + ), + "remaining_tokens": ( + None + if scope.max_tokens is None + else max(0, scope.max_tokens - usage.total_tokens) + ), + } + return result + + +def _bearer_token(authorization: str | None) -> str | None: + prefix = "Bearer " + if authorization is None or not authorization.startswith(prefix): + return None + return authorization[len(prefix) :] + + +def _usage(value: Any) -> tuple[int, int, int, int]: + if not isinstance(value, dict): + return (0, 0, 0, 0) + usage = value.get("usage") + if not isinstance(usage, dict): + response = value.get("response") + usage = response.get("usage") if isinstance(response, dict) else None + if not isinstance(usage, dict): + return (0, 0, 0, 0) + input_tokens = int(usage.get("input_tokens", usage.get("prompt_tokens", 0)) or 0) + output_tokens = int( + usage.get("output_tokens", usage.get("completion_tokens", 0)) or 0 + ) + total_tokens = int(usage.get("total_tokens", input_tokens + output_tokens) or 0) + # Responses API nests cache reads under input_tokens_details; chat + # completions under prompt_tokens_details. + details = usage.get("input_tokens_details") or usage.get("prompt_tokens_details") + cached_tokens = ( + int(details.get("cached_tokens", 0) or 0) if isinstance(details, dict) else 0 + ) + return (input_tokens, output_tokens, total_tokens, cached_tokens) + + +class _StreamingUsage: + def __init__(self): + self.buffer = b"" + self.tokens = (0, 0, 0, 0) + + def feed(self, chunk: bytes) -> None: + self.buffer += chunk + while b"\n" in self.buffer: + line, self.buffer = self.buffer.split(b"\n", 1) + if not line.startswith(b"data:"): + continue + payload = line.removeprefix(b"data:").strip() + if not payload or payload == b"[DONE]": + continue + try: + value = json.loads(payload) + except (json.JSONDecodeError, UnicodeDecodeError): + continue + tokens = _usage(value) + if tokens != (0, 0, 0, 0): + self.tokens = tokens + + +class _BoundedCapture: + """Keep the head and tail of a byte stream within a fixed byte budget.""" + + def __init__(self, cap: int): + self.cap = cap + self.head_limit = cap // 2 + self.tail_limit = cap - self.head_limit + self.head = bytearray() + self.tail = bytearray() + self.total = 0 + + def feed(self, chunk: bytes) -> None: + self.total += len(chunk) + if self.cap <= 0: + return + if len(self.head) < self.head_limit: + take = min(self.head_limit - len(self.head), len(chunk)) + self.head += chunk[:take] + chunk = chunk[take:] + if chunk: + self.tail += chunk + if len(self.tail) > self.tail_limit: + del self.tail[: len(self.tail) - self.tail_limit] + + def snapshot(self) -> dict[str, Any]: + if self.cap <= 0: + return {"bytes": self.total, "truncated": self.total > 0} + kept = len(self.head) + len(self.tail) + result: dict[str, Any] = {"bytes": self.total, "truncated": self.total > kept} + text = bytes(self.head).decode("utf-8", errors="replace") + if self.tail and not result["truncated"]: + text += bytes(self.tail).decode("utf-8", errors="replace") + elif self.tail: + result["tail"] = bytes(self.tail).decode("utf-8", errors="replace") + result["text"] = text + return result + + +class InferenceRequestLog: + """Size-rotated JSONL log of every request the gateway proxies or denies. + + Best-effort by construction: a logging failure must never affect the + proxied request. + """ + + def __init__(self, config: InferenceRequestLogConfig): + self.config = config + self.directory = Path(config.directory) + self.directory.mkdir(parents=True, exist_ok=True) + self._lock = asyncio.Lock() + existing = sorted(self.directory.glob("requests-*.jsonl")) + if existing: + self._current = existing[-1] + self._index = len(existing) + self._size = self._current.stat().st_size + else: + self._index = 1 + self._current = self.directory / "requests-00001.jsonl" + self._size = 0 + + def body(self, data: bytes) -> dict[str, Any]: + capture = self.capture() + capture.feed(data) + return capture.snapshot() + + def capture(self) -> _BoundedCapture: + return _BoundedCapture(self.config.body_bytes) + + async def record(self, **fields: Any) -> None: + try: + line = json.dumps( + { + "schema_version": 1, + "ts": datetime.now(UTC).isoformat(), + **fields, + }, + ensure_ascii=False, + default=str, + ) + encoded = (line + "\n").encode("utf-8") + async with self._lock: + if self._size and self._size + len(encoded) > self.config.rotate_bytes: + self._index += 1 + self._current = self.directory / f"requests-{self._index:05d}.jsonl" + self._size = 0 + await asyncio.to_thread(self._append, self._current, encoded) + self._size += len(encoded) + except Exception: + logger.warning("inference request log write failed", exc_info=True) + + @staticmethod + def _append(path: Path, data: bytes) -> None: + with open(path, "ab") as handle: + handle.write(data) + + +class _RequestAttributor: + """Experimental provider-agnostic conversation threading for the request + log (``request_log.attribution``). + + Stamps each record with a ``thread_id`` so post-hoc analysis can group a + trial's requests without content matching: stateful chains (the responses + API's ``previous_response_id``) inherit their parent's thread, and + stateless full-history surfaces (chat completions, Anthropic Messages) + group by a digest of the conversation's first user message. + + Best-effort by construction — every public method swallows all exceptions + and returns empty results, so it can never affect proxying. Memory is + bounded by two capped FIFO maps; CPU is bounded because the request body + is the dict the proxy already parsed and response scanning is one regex + over an at-most-8KB head. + """ + + _CAP = 100_000 + _SNIPPET_CHARS = 200 + _DIGEST_CHARS = 2048 + _RESPONSE_ID_PATTERN = re.compile(rb'"id"\s*:\s*"(resp_[A-Za-z0-9_-]+)"') + + def __init__(self): + self._response_threads: OrderedDict[str, str] = OrderedDict() + self._root_threads: OrderedDict[str, str] = OrderedDict() + self.errors = 0 + + @staticmethod + def _content_text(content: Any) -> str: + if isinstance(content, str): + return content + if isinstance(content, list): + parts = [] + for block in content: + if isinstance(block, str): + parts.append(block) + elif isinstance(block, dict): + text = block.get("text") or block.get("content") + if isinstance(text, str): + parts.append(text) + return "\n".join(parts) + return "" + + @classmethod + def _first_user_text(cls, value: dict[str, Any]) -> str | None: + """The conversation root: the first user turn, skipping system and + developer content (which is harness-constant, not trial-specific).""" + items = value.get("messages") + if not isinstance(items, list): + items = value.get("input") + if isinstance(items, str): + return items or None + if not isinstance(items, list): + return None + for item in items: + if isinstance(item, dict) and item.get("role") == "user": + text = cls._content_text(item.get("content")) + if text: + return text + return None + + def _remember(self, store: OrderedDict[str, str], key: str, value: str) -> None: + store[key] = value + while len(store) > self._CAP: + store.popitem(last=False) + + def stamp_request(self, value: Any) -> dict[str, str]: + try: + if not isinstance(value, dict): + return {} + previous = value.get("previous_response_id") + if isinstance(previous, str) and previous in self._response_threads: + return {"thread_id": self._response_threads[previous]} + text = self._first_user_text(value) + if not text: + if isinstance(previous, str) and previous: + # Chained onto a response we never saw (e.g. a gateway + # restart): still give the tail of the chain one thread. + thread = secrets.token_hex(8) + self._remember(self._response_threads, previous, thread) + return {"thread_id": thread} + return {} + normalized = "".join( + character for character in text.lower() if character.isalnum() + )[: self._DIGEST_CHARS] + digest = hashlib.sha256(normalized.encode("utf-8")).hexdigest() + thread = self._root_threads.get(digest) + if thread is None: + thread = secrets.token_hex(8) + self._remember(self._root_threads, digest, thread) + return { + "thread_id": thread, + "thread_root_digest": digest, + "root_snippet": text[: self._SNIPPET_CHARS], + } + except Exception: + self.errors += 1 + return {} + + def register_response( + self, fields: dict[str, str] | None, head: bytes + ) -> None: + try: + thread = (fields or {}).get("thread_id") + if not thread or not head: + return + match = self._RESPONSE_ID_PATTERN.search(head[:8192]) + if match is not None: + self._remember( + self._response_threads, match.group(1).decode("ascii"), thread + ) + except Exception: + self.errors += 1 + + +_REQUEST_HEADERS = { + "accept", + "content-type", + "openai-beta", + "user-agent", + "x-stainless-arch", + "x-stainless-async", + "x-stainless-lang", + "x-stainless-os", + "x-stainless-package-version", + "x-stainless-retry-count", + "x-stainless-runtime", + "x-stainless-runtime-version", + "x-stainless-timeout", +} +_RESPONSE_HEADERS = { + "content-type", + "openai-processing-ms", + "request-id", + "x-request-id", +} +def _provider_error(status_code: int, message: str, code: str) -> JSONResponse: + return JSONResponse( + status_code=status_code, + content={ + "error": { + "message": message, + "type": "vero_inference_gateway_error", + "code": code, + } + }, + ) + + +def create_inference_gateway_app( + *, + config: InferenceGatewayConfig, + upstream_api_key: str, + upstream_base_url: str | None = None, + transport: Any = None, +) -> FastAPI: + """Create a scoped reverse proxy without exposing the upstream credential.""" + if not upstream_api_key.strip(): + raise ValueError("upstream API key must not be empty") + try: + import httpx + except ImportError as error: + raise RuntimeError("install scale-vero[harbor] to serve inference") from error + + base_url = (upstream_base_url or config.default_upstream_base_url).rstrip("/") + store = InferenceUsageStore(config) + request_log = ( + InferenceRequestLog(config.request_log) + if config.request_log is not None + else None + ) + attributor = ( + _RequestAttributor() + if request_log is not None and config.request_log.attribution + else None + ) + + @asynccontextmanager + async def lifespan(app: FastAPI): + async with httpx.AsyncClient( + timeout=None, + transport=transport, + follow_redirects=False, + ) as client: + app.state.upstream = client + yield + + app = FastAPI(title="VeRO inference gateway", version="1", lifespan=lifespan) + app.state.usage_store = store + app.state.request_log = request_log + app.state.request_attributor = attributor + + @app.get("/health") + async def health(): + return {"ok": True} + + @app.get("/usage/{scope_name}") + async def usage_status( + scope_name: str, + authorization: Annotated[str | None, Header()] = None, + ): + scope = config.scopes.get(scope_name) + token = _bearer_token(authorization) + if ( + scope is None + or token is None + or not secrets.compare_digest(token_digest(token), scope.token_sha256) + ): + raise HTTPException(status_code=403, detail="invalid inference scope token") + return store.public_status()[scope_name] + + @app.api_route( + LAYOUT.scope_route, + methods=["POST"], + ) + async def proxy( + scope_name: str, + attribution: str, + endpoint: str, + request: Request, + authorization: Annotated[str | None, Header()] = None, + x_api_key: Annotated[str | None, Header()] = None, + ): + scope = config.scopes.get(scope_name) + # Provider-agnostic auth: OpenAI/codex clients send the scope token as + # `Authorization: Bearer`, Anthropic/Claude clients send it as `x-api-key`. + token = _bearer_token(authorization) or ( + x_api_key.strip() if x_api_key and x_api_key.strip() else None + ) + if ( + scope is None + or token is None + or not secrets.compare_digest(token_digest(token), scope.token_sha256) + ): + return _provider_error( + 403, "invalid inference scope token", "invalid_token" + ) + # Provider-agnostic passthrough: forward any inference endpoint + # (responses, chat/completions, messages, embeddings, ...) to the upstream + # litellm proxy, which decides what it supports. Guard only against path + # traversal escaping the scoped upstream base — not a per-provider list. + endpoint = endpoint.strip("/") + if not endpoint or ".." in endpoint.split("/"): + return _provider_error( + 400, "invalid inference endpoint", "invalid_endpoint" + ) + try: + body = await request.body() + value = json.loads(body) if body else {} + except (json.JSONDecodeError, UnicodeDecodeError): + return _provider_error(400, "request body must be JSON", "invalid_request") + + started = time.monotonic() + thread_fields = ( + attributor.stamp_request(value) if attributor is not None else {} + ) + + async def log_request( + *, + status: int, + error: str | None = None, + stream: bool = False, + tokens: tuple[int, int, int, int] = (0, 0, 0, 0), + response: dict[str, Any] | None = None, + upstream_error: bool = False, + ) -> None: + if request_log is None: + return + await request_log.record( + scope=scope_name, + attribution=attribution, + endpoint=endpoint, + model=value.get("model") if isinstance(value, dict) else None, + stream=stream, + status=status, + error=error, + latency_ms=round((time.monotonic() - started) * 1000, 1), + input_tokens=tokens[0], + output_tokens=tokens[1], + total_tokens=tokens[2], + cached_input_tokens=tokens[3], + upstream_error=upstream_error, + request=request_log.body(body), + response=response, + **thread_fields, + ) + # Per-scope allow-list: producer (optimizer) and evaluation (target) have + # separate tokens + allowed_models, so the target is normally confined to + # its fixed eval model. NOTE (known, deferred): this is not a *hard* + # fixed-target guarantee. Both scopes share one gateway host, split only by + # URL path, and the optimizer holds the producer token and authors the + # candidate. An adversarial optimizer can smuggle its producer token into + # the candidate and, from the eval sandbox (which already reaches this host + # for its legit calls), hit /scopes/producer to run the optimizer model. + # Closing this needs structural egress isolation (per-role/per-eval + # endpoints), not path-scoping — deferred as it requires an adversarial + # producer. + model = value.get("model") if isinstance(value, dict) else None + if not isinstance(model, str) or model not in scope.allowed_models: + await log_request(status=403, error="model_denied") + return _provider_error( + 403, "model is not allowed for this scope", "model_denied" + ) + if not attribution or any( + not (character.isalnum() or character in "_.-") for character in attribution + ): + return _provider_error( + 400, "invalid inference attribution", "invalid_attribution" + ) + try: + await store.reserve(scope_name, attribution) + except InferenceBudgetExceeded as error: + # 402, not 429. Budget exhaustion is terminal -- no amount of waiting + # restores the quota -- but 429 means "slow down and try again", and + # every layer above honours that: EvaluationLimits.retry_status_codes + # defaults to [429, 503, 529], and target-agent SDKs retry it too. + # officeqa run #2 reissued 3672 doomed requests after exhausting the + # finalization scope, burning verifier wall-clock to no purpose. + # 402 is in no retry list, so the caller fails fast with the reason. + await log_request(status=402, error="budget_exhausted") + return _provider_error(402, str(error), "budget_exhausted") + + headers = { + name: value + for name, value in request.headers.items() + if name.lower() in _REQUEST_HEADERS + } + headers["authorization"] = f"Bearer {upstream_api_key}" + url = f"{base_url}/{endpoint}" + try: + upstream = await app.state.upstream.send( + app.state.upstream.build_request( + "POST", + url, + params=request.query_params, + content=body, + headers=headers, + ), + stream=True, + ) + except httpx.HTTPError: + await asyncio.shield( + store.complete(scope_name, attribution, upstream_error=True) + ) + await log_request(status=502, error="upstream_error", upstream_error=True) + return _provider_error( + 502, "upstream inference request failed", "upstream_error" + ) + except BaseException: + await asyncio.shield( + store.complete(scope_name, attribution, upstream_error=True) + ) + raise + response_headers = { + name: value + for name, value in upstream.headers.items() + if name.lower() in _RESPONSE_HEADERS + } + is_stream = "text/event-stream" in upstream.headers.get("content-type", "") + if is_stream: + observed = _StreamingUsage() + captured = request_log.capture() if request_log is not None else None + + async def chunks() -> AsyncIterator[bytes]: + failed = upstream.status_code >= 400 + + async def finish() -> None: + try: + await upstream.aclose() + finally: + await store.complete( + scope_name, + attribution, + input_tokens=observed.tokens[0], + output_tokens=observed.tokens[1], + total_tokens=observed.tokens[2], + cached_input_tokens=observed.tokens[3], + upstream_error=failed, + ) + if attributor is not None and captured is not None: + # The response id sits in the first SSE event, so + # the captured head is enough to link the chain. + attributor.register_response( + thread_fields, bytes(captured.head) + ) + await log_request( + status=upstream.status_code, + stream=True, + tokens=observed.tokens, + response=( + captured.snapshot() if captured is not None else None + ), + upstream_error=failed, + ) + + try: + async for chunk in upstream.aiter_bytes(): + observed.feed(chunk) + if captured is not None: + captured.feed(chunk) + yield chunk + except asyncio.CancelledError: + # OpenAI clients commonly stop consuming an SSE stream as soon + # as they observe its terminal response event. That downstream + # cancellation is not an upstream provider failure. + raise + except httpx.HTTPError: + failed = True + raise + except BaseException: + failed = True + raise + finally: + await asyncio.shield(finish()) + + return StreamingResponse( + chunks(), + status_code=upstream.status_code, + headers=response_headers, + media_type="text/event-stream", + ) + + try: + content = await upstream.aread() + try: + tokens = _usage(json.loads(content)) + except (json.JSONDecodeError, UnicodeDecodeError): + tokens = (0, 0, 0, 0) + except BaseException: + await asyncio.shield( + store.complete(scope_name, attribution, upstream_error=True) + ) + raise + finally: + await upstream.aclose() + await asyncio.shield( + store.complete( + scope_name, + attribution, + input_tokens=tokens[0], + output_tokens=tokens[1], + total_tokens=tokens[2], + cached_input_tokens=tokens[3], + upstream_error=upstream.status_code >= 400, + ) + ) + if attributor is not None: + attributor.register_response(thread_fields, content[:8192]) + await log_request( + status=upstream.status_code, + tokens=tokens, + response=request_log.body(content) if request_log is not None else None, + upstream_error=upstream.status_code >= 400, + ) + return Response( + content=content, + status_code=upstream.status_code, + headers=response_headers, + media_type=upstream.headers.get("content-type"), + ) + + return app + + +def load_inference_gateway_config(path: Path | str) -> InferenceGatewayConfig: + return InferenceGatewayConfig.model_validate_json( + Path(path).read_text(encoding="utf-8") + ) + + +def serve_inference_gateway( + *, + config_path: Path | str, + host: str = "0.0.0.0", + port: int = 8001, +) -> None: + import uvicorn + + config = load_inference_gateway_config(config_path) + upstream_api_key = os.environ.get(config.upstream_api_key_env) + if not upstream_api_key: + raise RuntimeError( + f"gateway upstream credential {config.upstream_api_key_env} is missing" + ) + upstream_base_url = ( + os.environ.get(config.upstream_base_url_env) + if config.upstream_base_url_env is not None + else None + ) + uvicorn.run( + create_inference_gateway_app( + config=config, + upstream_api_key=upstream_api_key, + upstream_base_url=upstream_base_url, + ), + host=host, + port=port, + ) diff --git a/vero/src/vero/harbor/__init__.py b/vero/src/vero/harbor/__init__.py new file mode 100644 index 00000000..83a54a62 --- /dev/null +++ b/vero/src/vero/harbor/__init__.py @@ -0,0 +1,45 @@ +"""Harbor as VeRO's task substrate: compile a task, and evaluate on it. + +``build`` compiles a benchmark's build YAML into a runnable Harbor task +directory. ``backend`` evaluates one candidate by nesting a ``harbor run`` per +case — a peer of the backends in ``vero.evaluation.backends``. ``deployment`` is +the standard sidecar factory, and the one module that binds the trusted runtime +in ``vero.sidecar`` to a Harbor-backed evaluation; a task with a command backend +uses the same runtime and no Harbor backend at all. + +The trusted runtime itself lives in ``vero.sidecar``, and the inference gateway +in ``vero.gateway``. Neither is a Harbor concept. +""" + +from vero.harbor.backend import HarborBackend, HarborBackendConfig, HarborCase +from vero.harbor.build import ( + AgentAccessSpec, + CommandBackendSpec, + HarborBuildConfig, + InferenceBudgetSpec, + InferenceGatewaySpec, + VerificationTargetSpec, + WandbSpec, + WorkspaceOverlaySpec, + compile_harbor_task, + load_harbor_build_config, +) +from vero.harbor.deployment import HarborDeploymentConfig, build_harbor_components + +__all__ = [ + "HarborBackend", + "HarborBackendConfig", + "HarborCase", + "HarborDeploymentConfig", + "AgentAccessSpec", + "CommandBackendSpec", + "HarborBuildConfig", + "InferenceBudgetSpec", + "InferenceGatewaySpec", + "VerificationTargetSpec", + "WandbSpec", + "WorkspaceOverlaySpec", + "build_harbor_components", + "compile_harbor_task", + "load_harbor_build_config", +] diff --git a/vero/src/vero/harbor/backend.py b/vero/src/vero/harbor/backend.py new file mode 100644 index 00000000..aed6ed22 --- /dev/null +++ b/vero/src/vero/harbor/backend.py @@ -0,0 +1,1699 @@ +"""Evaluate a candidate program by running it over Harbor tasks.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +import math +import mimetypes +import os +import re +import shutil +import statistics +import tempfile +from collections import defaultdict +from datetime import datetime +from pathlib import Path +from typing import Any, Literal + +from pydantic import Field, JsonValue, field_validator, model_validator + +from vero.evaluation.backends.base import EvaluationContext +from vero.evaluation.models import ( + AllCases, + BackendProvenance, + CaseError, + CaseIds, + CaseRange, + CaseResult, + CaseStatus, + DiagnosticSeverity, + EvaluationArtifact, + EvaluationCost, + EvaluationDiagnostic, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, +) +from vero.evaluation.scoring.error_taxonomy import ( + NO_REWARD_SIGNAL, + ErrorCategory, + classify_case, + policy, +) +from vero.evaluation.scoring.security import sanitize_evaluation_report, sanitize_text +from vero.layout import LAYOUT +from vero.models import StrictModel +from vero.sandbox import CommandResult, Sandbox +from vero.sidecar.isolation import harness_grant_commands, harness_reachability_probe +from vero.staging import SandboxStagingArea + +logger = logging.getLogger(__name__) + + +def _default_uv() -> str: + executable = shutil.which("uv") + if executable is None: + raise ValueError("uv is required to configure a Harbor backend") + return str(Path(executable).resolve()) + + +class HarborCase(StrictModel): + """One canonical case mapped to one Harbor task name.""" + + id: str + task_name: str + result_task_name: str | None = None + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("id", "task_name") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("Harbor case identity must not be empty") + return value + + @field_validator("result_task_name") + @classmethod + def validate_result_identity(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("Harbor result task identity must not be empty") + return value + + @property + def expected_result_task_name(self) -> str: + return self.result_task_name or self.task_name + + +# Dedicated env vars carrying the candidate agent's gateway credentials when +# task_services_use_upstream reroutes OPENAI_* to the real upstream. A custom +# agent reads these to keep its own inference on the metered/allow-listed gateway. +# This name is a contract shared with agents (they cannot import this module). +AGENT_INFERENCE_API_KEY_ENV = "VERO_AGENT_INFERENCE_API_KEY" +AGENT_INFERENCE_BASE_URL_ENV = "VERO_AGENT_INFERENCE_BASE_URL" + + +class HarborBackendConfig(StrictModel): + """Trusted configuration for nested ``harbor run`` evaluation.""" + + # Discriminates this from the other backend configs a deployment may name. + type: Literal["harbor"] = "harbor" + task_source: str + agent_import_path: str + cases_path: str + harbor_requirement: str + evaluation_set_name: str = "harbor" + partition: str | None = None + model: str | None = None + environment_name: str = "modal" + python_version: str = "3.12" + case_timeout_seconds: float = Field(default=180.0, gt=0) + task_agent_timeout_seconds: float = Field(default=600.0, gt=0) + n_attempts: int = Field(default=1, ge=1) + max_retries: int = Field(default=2, ge=0) + retry_wait_multiplier: float = Field(default=2.0, ge=1.0) + retry_min_wait_seconds: float = Field(default=4.0, ge=0.0) + retry_max_wait_seconds: float = Field(default=60.0, ge=0.0) + # Whole-sub-run infrastructure retry. Default 1 (no retry): retry > 1 only + # applies to trusted finalization evaluations (see HarborBackend.evaluate), + # never to competitive agent search, where re-running for best coverage is + # a best-of-N amplifier a candidate can trigger. + infrastructure_max_attempts: int = Field(default=1, ge=1) + infrastructure_retry_delay_seconds: float = Field(default=5.0, ge=0) + reward_key: str | None = None + # Average attempts by default rather than taking the best: best-of-n is an + # optimistic estimator that biases selection above the single-shot final + # score. Selection and final both run through this same aggregation. + aggregate_attempts: Literal["best", "mean"] = "mean" + failure_score: float = 0.0 + feedback_transcripts: bool = False + feedback_max_bytes: int = Field(default=3000, ge=0) + expose_attempt_detail: bool = False + uv_executable: str = Field(default_factory=_default_uv) + default_index: str = "https://pypi.org/simple" + environment: dict[str, str] = Field(default_factory=dict) + passthrough_environment: list[str] = Field(default_factory=list) + inference_gateway_url: str | None = None + inference_gateway_token: str | None = None + # Optional reserved scope for trusted finalization (admin re-score / targets), + # so the optimizer's search evaluations cannot exhaust the budget the mandatory + # re-score needs. When unset, finalization falls back to the evaluation scope. + inference_gateway_finalization_token: str | None = None + # When True, task-owned evaluation services (e.g. LLM user-simulators or graders + # that run *inside* the task containers and cannot reach the compose-internal + # gateway) receive the real upstream credentials via OPENAI_*, while the + # candidate agent still routes through the metered/allow-listed gateway via + # VERO_AGENT_INFERENCE_*. upstream_*_env name the host env vars holding the + # upstream credentials (read from the sidecar process environment). + task_services_use_upstream: bool = False + upstream_api_key_env: str | None = None + upstream_base_url_env: str | None = None + # Unprivileged OS user to execute the untrusted candidate harness as, so it + # cannot read the trusted state (held-out records, budgets, other + # candidates, admin token) that the sidecar process owns. None (the default, + # used for local dev and tests) runs the harness in-process with no drop; + # the compiled sidecar sets this to isolate. + harness_user: str | None = None + case_resources_cache_path: str | None = None + # The gateway's durable usage ledger; when set, each evaluation's report + # carries its own inference token totals as metrics (attribution keyed by + # evaluation id). + inference_usage_path: str | None = None + extra_args: list[str] = Field(default_factory=list) + + @field_validator("cases_path", "case_resources_cache_path") + @classmethod + def validate_absolute_path(cls, value: str | None) -> str | None: + if value is None: + return None + if not value.strip() or not Path(value).is_absolute(): + raise ValueError("Harbor backend file paths must be absolute") + return value + + @field_validator( + "task_source", + "agent_import_path", + "harbor_requirement", + "evaluation_set_name", + "environment_name", + "python_version", + "default_index", + ) + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("Harbor backend identity must not be empty") + return value + + @field_validator("uv_executable") + @classmethod + def validate_uv_executable(cls, value: str) -> str: + if not value.strip(): + raise ValueError("uv_executable must not be empty") + return value + + @field_validator("harbor_requirement") + @classmethod + def validate_pinned_harbor_requirement(cls, value: str) -> str: + exact = re.search( + r"(?:^|\s)harbor(?:\[[A-Za-z0-9_.-]+(?:,[A-Za-z0-9_.-]+)*\])?" + r"\s*==\s*[^*\s,;]+", + value, + ) + pinned_git = re.search(r"@[0-9a-f]{7,64}(?:#.*)?$", value) + if exact is None and pinned_git is None: + raise ValueError( + "harbor_requirement must pin an exact version or Git commit" + ) + return value + + @field_validator( + "partition", + "model", + "reward_key", + "inference_gateway_url", + "inference_gateway_token", + ) + @classmethod + def validate_optional_identity(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("optional Harbor identity must not be empty") + return value + + @field_validator("failure_score") + @classmethod + def validate_failure_score(cls, value: float) -> float: + if not math.isfinite(value): + raise ValueError("failure_score must be finite") + return value + + @field_validator("environment") + @classmethod + def validate_environment(cls, value: dict[str, str]) -> dict[str, str]: + for name in value: + if not name or "=" in name: + raise ValueError(f"invalid environment variable name: {name!r}") + return value + + @field_validator("passthrough_environment") + @classmethod + def validate_passthrough_environment(cls, value: list[str]) -> list[str]: + if len(value) != len(set(value)): + raise ValueError("passthrough environment names must be unique") + for name in value: + if not name or "=" in name: + raise ValueError(f"invalid environment variable name: {name!r}") + return value + + @field_validator("extra_args") + @classmethod + def validate_extra_args(cls, value: list[str]) -> list[str]: + controlled = { + "-a", + "-d", + "-e", + "-i", + "-m", + "-n", + "-p", + "--agent-import-path", + "--jobs-dir", + "--max-retries", + "--n-attempts", + "--agent-timeout-multiplier", + } + conflicts = [ + argument for argument in value if argument.split("=", 1)[0] in controlled + ] + if conflicts: + raise ValueError( + "extra_args cannot override backend-controlled Harbor flags: " + + ", ".join(conflicts) + ) + return value + + return value + + @model_validator(mode="after") + def validate_filesystem_and_environment(self) -> HarborBackendConfig: + if not Path(self.cases_path).is_file(): + raise ValueError("Harbor cases_path must be an existing file") + overlap = set(self.environment) & set(self.passthrough_environment) + if overlap: + raise ValueError( + "environment and passthrough_environment overlap for: " + + ", ".join(sorted(overlap)) + ) + if (self.inference_gateway_url is None) != ( + self.inference_gateway_token is None + ): + raise ValueError( + "inference_gateway_url and inference_gateway_token must be set together" + ) + if ( + self.inference_gateway_url is not None + and not self.inference_gateway_url.startswith(("http://", "https://")) + ): + raise ValueError("inference_gateway_url must be HTTP(S)") + gateway_names = {"OPENAI_API_KEY", "OPENAI_BASE_URL"} + if self.task_services_use_upstream: + gateway_names |= {AGENT_INFERENCE_API_KEY_ENV, AGENT_INFERENCE_BASE_URL_ENV} + gateway_overlap = gateway_names & ( + set(self.environment) | set(self.passthrough_environment) + ) + if self.inference_gateway_url is not None and gateway_overlap: + raise ValueError( + "gateway-managed environment variables must not also be configured: " + + ", ".join(sorted(gateway_overlap)) + ) + if self.task_services_use_upstream: + if self.inference_gateway_url is None: + raise ValueError( + "task_services_use_upstream requires an inference gateway" + ) + if not self.upstream_api_key_env: + raise ValueError( + "task_services_use_upstream requires upstream_api_key_env" + ) + if self.harness_user is not None and self.task_services_use_upstream: + # uid isolation does not hide env vars, so the raw upstream key would + # still reach the isolated harness through OPENAI_*. Refuse the + # combination until task-service credentials are delivered to the + # task containers off the harness environment. + raise ValueError( + "harness_user cannot be combined with task_services_use_upstream: " + "the raw upstream credential would still reach the isolated " + "harness through its environment" + ) + return self + + +class HarborBackend: + """Run Harbor as an external evaluator and collate its trial records.""" + + name = "harbor" + version = "2" + + def __init__(self, config: HarborBackendConfig): + self.config = config + self._cases = self._load_cases() + case_ids = [case.id for case in self._cases] + task_names = [case.task_name for case in self._cases] + if len(case_ids) != len(set(case_ids)): + raise ValueError("Harbor case IDs must be unique") + if len(task_names) != len(set(task_names)): + raise ValueError( + "Harbor task names must be unique within an evaluation set" + ) + + @property + def provenance(self) -> BackendProvenance: + return BackendProvenance.from_config( + name=self.name, + version=self.version, + config=self.config, + ) + + def _load_cases(self) -> list[HarborCase]: + path = Path(self.config.cases_path) + if path.suffix == ".jsonl": + raw = [ + json.loads(line) + for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + else: + raw = json.loads(path.read_text(encoding="utf-8")) + if isinstance(raw, dict): + raw = raw.get("cases") + if not isinstance(raw, list) or not raw: + raise ValueError("Harbor case file must contain a non-empty case list") + return [HarborCase.model_validate(value) for value in raw] + + def _validate_evaluation_set(self, evaluation_set: EvaluationSet) -> None: + if evaluation_set.name != self.config.evaluation_set_name: + raise ValueError( + f"Harbor backend owns evaluation set " + f"{self.config.evaluation_set_name!r}, not {evaluation_set.name!r}" + ) + if evaluation_set.partition != self.config.partition: + raise ValueError( + f"Harbor backend owns partition {self.config.partition!r}, " + f"not {evaluation_set.partition!r}" + ) + selection = evaluation_set.selection + if isinstance(selection, CaseRange) and selection.stop > len(self._cases): + raise ValueError( + f"case range stops at {selection.stop}, but the Harbor evaluation " + f"set contains {len(self._cases)} cases" + ) + if isinstance(selection, CaseIds): + known = {case.id for case in self._cases} + unknown = sorted(set(selection.ids) - known) + if unknown: + raise ValueError(f"unknown Harbor case IDs: {unknown}") + + def _selected_cases(self, evaluation_set: EvaluationSet) -> list[HarborCase]: + self._validate_evaluation_set(evaluation_set) + selection = evaluation_set.selection + if isinstance(selection, AllCases): + return list(self._cases) + if isinstance(selection, CaseRange): + return self._cases[selection.start : selection.stop] + if isinstance(selection, CaseIds): + by_id = {case.id: case for case in self._cases} + return [by_id[case_id] for case_id in selection.ids] + raise AssertionError(f"unsupported Harbor case selection: {selection}") + + async def resolve_cost(self, evaluation_set: EvaluationSet) -> EvaluationCost: + return EvaluationCost(cases=len(self._selected_cases(evaluation_set))) + + async def export_case_resources( + self, + *, + evaluation_set: EvaluationSet, + destination: str, + sandbox: Sandbox, + ) -> None: + """Expose complete task resources for an explicitly authorized partition.""" + + cases = self._selected_cases(evaluation_set) + configured_cache = self.config.case_resources_cache_path + if configured_cache is None: + with tempfile.TemporaryDirectory(prefix="vero-harbor-cases-") as temporary: + root = Path(temporary) + await self._materialize_case_resources(root, cases, evaluation_set) + await sandbox.upload(str(root), destination) + return + + cache = Path(configured_cache) + if not (cache / "index.json").is_file(): + cache.parent.mkdir(parents=True, exist_ok=True) + temporary = Path( + tempfile.mkdtemp( + dir=cache.parent, + prefix=f".{cache.name}.", + ) + ) + try: + await self._materialize_case_resources( + temporary, + cases, + evaluation_set, + ) + if cache.exists(): + shutil.rmtree(cache) + os.replace(temporary, cache) + finally: + shutil.rmtree(temporary, ignore_errors=True) + self._make_agent_readable(cache) + await sandbox.upload(str(cache), destination) + + async def _materialize_case_resources( + self, + root: Path, + cases: list[HarborCase], + evaluation_set: EvaluationSet, + ) -> None: + root.mkdir(parents=True, exist_ok=True) + source = Path(self.config.task_source).expanduser() + if source.exists(): + case_paths = self._materialize_local_case_resources(root, source, cases) + dataset_files_path = None + else: + case_paths, dataset_files_path = await self._materialize_package_resources( + root, + cases, + ) + (root / "index.json").write_text( + json.dumps( + { + "schema_version": 1, + "evaluation_set": evaluation_set.model_dump(mode="json"), + "task_source": self.config.task_source, + "task_source_exposed": True, + "dataset_files_path": dataset_files_path, + "cases": [ + { + "case_id": case.id, + "task_name": case.task_name, + "path": case_paths[case.id], + } + for case in cases + ], + }, + ensure_ascii=False, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + self._make_agent_readable(root) + + @staticmethod + def _make_agent_readable(root: Path) -> None: + """Allow the non-root producer to traverse its read-only context mount.""" + + for path in (root, *root.rglob("*")): + if path.is_symlink(): + continue + mode = path.stat().st_mode & 0o777 + if path.is_dir(): + path.chmod((mode | 0o055) & ~0o022) + elif path.is_file(): + path.chmod((mode | 0o044) & ~0o022) + + def _materialize_local_case_resources( + self, + root: Path, + source: Path, + cases: list[HarborCase], + ) -> dict[str, str]: + tasks = root / "tasks" + tasks.mkdir() + paths: dict[str, str] = {} + resolved_source = source.resolve() + for case in cases: + task = (resolved_source / case.task_name).resolve() + if task.parent != resolved_source or not task.is_dir(): + raise ValueError( + f"local Harbor case {case.task_name!r} is not a direct task directory" + ) + destination = tasks / hashlib.sha256(case.id.encode()).hexdigest() + shutil.copytree(task, destination) + paths[case.id] = destination.relative_to(root).as_posix() + dataset_files = root / "dataset-files" + dataset_files.mkdir() + for path in resolved_source.iterdir(): + if path.is_file() and not path.is_symlink(): + shutil.copy2(path, dataset_files / path.name) + if not any(dataset_files.iterdir()): + dataset_files.rmdir() + return paths + + async def _materialize_package_resources( + self, + root: Path, + cases: list[HarborCase], + ) -> tuple[dict[str, str], str | None]: + try: + from harbor.registry.client.package import PackageDatasetClient + from harbor.tasks.client import TaskClient + except ImportError as error: + raise RuntimeError( + "the pinned Harbor package must be installed to expose remote case resources" + ) from error + + client = PackageDatasetClient() + metadata = await client.get_dataset_metadata(self.config.task_source) + by_name = {task_id.get_name(): task_id for task_id in metadata.task_ids} + missing = sorted( + case.task_name for case in cases if case.task_name not in by_name + ) + if missing: + raise ValueError( + "authorized cases are absent from the pinned Harbor dataset: " + + ", ".join(missing) + ) + task_ids = [by_name[case.task_name] for case in cases] + tasks_root = root / "tasks" + result = await TaskClient().download_tasks( + task_ids=task_ids, + output_dir=tasks_root, + export=True, + ) + paths = { + case.id: download.path.relative_to(root).as_posix() + for case, download in zip(cases, result.results, strict=True) + } + dataset_root = root / "dataset-files" + files = await client.download_dataset_files( + metadata, + output_dir=dataset_root, + ) + return paths, "dataset-files" if files else None + + def _secrets(self) -> list[str]: + values = list(self.config.environment.values()) + values.extend( + os.environ[name] + for name in self.config.passthrough_environment + if name in os.environ + ) + if self.config.inference_gateway_token is not None: + values.append(self.config.inference_gateway_token) + if self.config.inference_gateway_finalization_token is not None: + values.append(self.config.inference_gateway_finalization_token) + return values + + def sanitize_error(self, message: str) -> str: + return sanitize_text(message, self._secrets()) + + def validate_request(self, request: EvaluationRequest) -> None: + self._validate_evaluation_set(request.evaluation_set) + if request.limits.retry.max_attempts > 1: + raise ValueError( + "Harbor does not support generic per-case retries; configure " + "HarborBackendConfig.max_retries instead" + ) + if request.limits.case_timeout_seconds != self.config.case_timeout_seconds: + raise ValueError( + "Harbor case timeout is fixed by the backend at " + f"{self.config.case_timeout_seconds:g} seconds" + ) + if request.seed is not None: + raise ValueError("Harbor does not support the generic evaluation seed") + payload = request.model_dump_json() + if any(secret in payload for secret in self._secrets() if len(secret) >= 4): + raise ValueError( + "evaluation parameters must not contain configured secret values; " + "pass secrets through the backend environment" + ) + + def _environment( + self, evaluation_id: str, *, finalization: bool = False + ) -> dict[str, str]: + environment = {"PATH": os.defpath, "LANG": "C.UTF-8"} + for name in ("TMPDIR", "TMP", "TEMP", "SYSTEMROOT"): + if name in os.environ: + environment[name] = os.environ[name] + environment.update(self.config.environment) + for name in self.config.passthrough_environment: + if name in os.environ: + environment[name] = os.environ[name] + if self.config.harness_user is not None: + # The isolated harness runs as an unprivileged user with no access + # to root's home; point uv and git at a home it owns. + home = f"/home/{self.config.harness_user}" + environment["HOME"] = home + environment["UV_CACHE_DIR"] = f"{home}/.cache/uv" + if self.config.inference_gateway_url is not None: + # Route trusted finalization evals to the reserved scope when configured; + # everything else (and the fallback) uses the shared evaluation scope. + use_finalization = ( + finalization + and self.config.inference_gateway_finalization_token is not None + ) + if finalization and not use_finalization: + # The trusted final re-score asked for the reserved pool but none + # was provisioned, so it silently shares the search budget and can + # be starved. Surface it loudly rather than degrading quietly. + logger.warning( + "finalization evaluation requested the reserved inference " + "scope but no finalization token is configured; falling back " + "to the shared 'evaluation' scope, which search evaluations " + "can exhaust" + ) + scope = "finalization" if use_finalization else "evaluation" + gateway_token = ( + self.config.inference_gateway_finalization_token + if use_finalization + else self.config.inference_gateway_token + ) or "" + gateway_url = ( + f"{self.config.inference_gateway_url.rstrip('/')}" + f"{LAYOUT.scope_path(scope, evaluation_id)}" + ) + if self.config.task_services_use_upstream: + # Task-owned eval services (user-sims, LLM graders) run inside the + # task containers and can't reach the compose-internal gateway; hand + # them the real upstream via OPENAI_*. The candidate agent keeps its + # metered/allow-listed gateway on dedicated VERO_AGENT_INFERENCE_*. + environment["OPENAI_API_KEY"] = os.environ.get( + self.config.upstream_api_key_env or "", "" + ) + if self.config.upstream_base_url_env: + environment["OPENAI_BASE_URL"] = os.environ.get( + self.config.upstream_base_url_env, "" + ) + environment[AGENT_INFERENCE_API_KEY_ENV] = gateway_token + environment[AGENT_INFERENCE_BASE_URL_ENV] = gateway_url + else: + environment["OPENAI_API_KEY"] = gateway_token + environment["OPENAI_BASE_URL"] = gateway_url + # Task packages that template `${OPENAI_API_BASE}` (litellm-style + # name, e.g. swe-atlas-qna's rubric judge) get the same endpoint. + if "OPENAI_BASE_URL" in environment: + environment["OPENAI_API_BASE"] = environment["OPENAI_BASE_URL"] + return environment + + def _source_args(self, task_source: str, *, local: bool) -> list[str]: + return ["-p", task_source] if local else ["-d", task_source] + + def _retry_config_json(self) -> str: + """Partial harbor JobConfig carrying only the retry policy. + + Harbor's ``run`` CLI exposes ``--max-retries``/``--retry-include``/ + ``--retry-exclude`` but no backoff flags, so the backoff is delivered via a + ``--config`` snippet. The CLI loads ``--config`` as the base JobConfig and + then applies our flags on top (``--max-retries`` wins for the count), so a + retry-only partial is sufficient and safe. + """ + return json.dumps( + { + "retry": { + "max_retries": self.config.max_retries, + "wait_multiplier": self.config.retry_wait_multiplier, + "min_wait_sec": self.config.retry_min_wait_seconds, + "max_wait_sec": self.config.retry_max_wait_seconds, + } + } + ) + + def _command( + self, + *, + workspace: str, + request: EvaluationRequest, + cases: list[HarborCase], + jobs_dir: str, + task_source: str, + local_task_source: bool, + retry_config_path: str, + ) -> list[str]: + command = [ + self.config.uv_executable, + "run", + "--python", + self.config.python_version, + "--no-config", + "--no-env-file", + "--default-index", + self.config.default_index, + "--index-strategy", + "first-index", + "--project", + workspace, + "--with", + self.config.harbor_requirement, + "harbor", + "run", + # Non-interactive: a task that declares env vars (e.g. tau3's user-sim + # TAU2_* and grader TAU2_NL_ASSERTIONS_MODEL) makes `harbor run` prompt + # "Proceed? (Y/n)"; with no stdin the sub-run aborts (0 trials). GAIA + # declares none, so it never hit this. + "--yes", + *self._source_args(task_source, local=local_task_source), + "--agent-import-path", + self.config.agent_import_path, + "-e", + self.config.environment_name, + "-n", + str(request.limits.max_concurrency), + "--n-attempts", + str(self.config.n_attempts), + "--config", + retry_config_path, + "--max-retries", + str(self.config.max_retries), + "--agent-timeout-multiplier", + str( + self.config.case_timeout_seconds + / self.config.task_agent_timeout_seconds + ), + ] + model = request.parameters.get("harbor_model_override", self.config.model) + if model is not None: + command.extend(["-m", str(model)]) + for case in cases: + command.extend(["-i", case.task_name]) + command.extend(["--jobs-dir", jobs_dir, *self.config.extra_args]) + return command + + def _trial_groups(self, jobs_dir: Path) -> dict[str, list[dict[str, Any]]]: + groups: dict[str, list[dict[str, Any]]] = defaultdict(list) + if not jobs_dir.exists(): + return groups + jobs_root = jobs_dir.resolve() + for result_path in jobs_dir.rglob("result.json"): + if result_path.is_symlink(): + continue + try: + resolved = result_path.resolve() + resolved.relative_to(jobs_root) + value = json.loads(resolved.read_text(encoding="utf-8")) + modified = resolved.stat().st_mtime + except (json.JSONDecodeError, OSError, ValueError): + continue + task_name = value.get("task_name") if isinstance(value, dict) else None + if not task_name: + continue + value["_trial_dir"] = str(resolved.parent) + value["_mtime"] = modified + groups[str(task_name)].append(value) + for attempts in groups.values(): + attempts.sort( + key=lambda value: ( + value.get("finished_at") is None, + value.get("finished_at") or "", + value.get("trial_name") or "", + value.get("_mtime", 0.0), + ) + ) + return groups + + def _extract_reward(self, rewards: dict[str, Any]) -> float | None: + value: Any + if self.config.reward_key is not None: + value = rewards.get(self.config.reward_key) + else: + value = None + for key in ("pass", "reward"): + if key in rewards: + value = rewards[key] + break + if value is None and len(rewards) == 1: + value = next(iter(rewards.values())) + if value is None: + return None + try: + number = float(value) + except (TypeError, ValueError): + return None + return number if math.isfinite(number) else None + + def _attempt_reward(self, attempt: dict[str, Any]) -> float | None: + rewards = (attempt.get("verifier_result") or {}).get("rewards") or {} + return self._extract_reward(rewards) if rewards else None + + def _attempt_is_infra(self, attempt: dict[str, Any], *, trusted: bool) -> bool: + """Whether a dead attempt died to infrastructure (vs the candidate). + + For a competitive (agent) evaluation a candidate-controlled trial's own + transient-infra exception is not trusted as infrastructure — the + candidate could emit a timeout/connection error to have its dead attempt + excluded from the mean. Only trusted evaluations honor that signal. + """ + info = attempt.get("exception_info") or {} + if not info: + return False + signals = [str(info.get("exception_type") or NO_REWARD_SIGNAL)] + for detail_key in ("message", "detail", "exception_message"): + detail = info.get(detail_key) + if isinstance(detail, str) and detail: + signals.append(detail) + category = classify_case(signals) + if not trusted and category == ErrorCategory.TRANSIENT_INFRA: + return False + return not policy(category).is_informative_sample + + def _best_attempt( + self, + attempts: list[dict[str, Any]], + ) -> tuple[dict[str, Any] | None, float | None]: + scored = [ + (attempt, self._attempt_reward(attempt)) + for attempt in attempts + if self._attempt_reward(attempt) is not None + ] + if not scored: + return None, None + return max( + scored, + key=lambda item: ( + not bool(item[0].get("exception_info")), + item[1], + item[0].get("finished_at") or "", + item[0].get("_mtime", 0.0), + ), + ) + + def _transcript_feedback( + self, + attempts: list[dict[str, Any]], + ) -> str | None: + if not self.config.feedback_transcripts or self.config.feedback_max_bytes == 0: + return None + for attempt in attempts: + trial_dir_value = attempt.get("_trial_dir") + if not trial_dir_value: + continue + trial_dir = Path(trial_dir_value).resolve() + for relative in ("agent/terminus_2.pane", "agent/trajectory.json"): + path = trial_dir / relative + if path.is_symlink(): + continue + try: + resolved = path.resolve() + resolved.relative_to(trial_dir) + data = resolved.read_bytes() + except (OSError, ValueError): + continue + if data: + return data[-self.config.feedback_max_bytes :].decode( + "utf-8", errors="replace" + ) + return None + + def _trial_artifacts( + self, + attempts: list[dict[str, Any]], + artifact_root: Path, + ) -> list[EvaluationArtifact]: + """Reference complete Harbor trial records and redact configured credentials.""" + + resolved_root = artifact_root.resolve() + artifacts: list[EvaluationArtifact] = [] + seen: set[str] = set() + for attempt in attempts: + trial_dir_value = attempt.get("_trial_dir") + if not trial_dir_value: + continue + trial_root = Path(trial_dir_value) + if not trial_root.is_dir() or trial_root.is_symlink(): + continue + for path in sorted(trial_root.rglob("*")): + if not path.is_file() or path.is_symlink(): + continue + try: + resolved = path.resolve(strict=True) + resolved.relative_to(resolved_root) + except (OSError, ValueError): + continue + relative = resolved.relative_to(resolved_root).as_posix() + if relative in seen: + continue + seen.add(relative) + try: + payload = resolved.read_bytes() + if b"\x00" not in payload: + text = payload.decode("utf-8") + sanitized = sanitize_text(text, self._secrets()) + if sanitized != text: + resolved.write_text(sanitized, encoding="utf-8") + except (OSError, UnicodeDecodeError): + pass + media_type = mimetypes.guess_type(resolved.name)[0] + artifacts.append( + EvaluationArtifact( + path=relative, + media_type=media_type, + description=( + "Harbor trial record: " + + path.relative_to(trial_root).as_posix() + ), + ) + ) + return artifacts + + def _inference_usage_metrics(self, evaluation_id: str) -> dict[str, float]: + """This evaluation's gateway token totals, from the durable ledger. + + Attribution is keyed by evaluation id (search and finalization scopes + alike). Best-effort: telemetry must never fail an evaluation. + """ + if self.config.inference_usage_path is None: + return {} + try: + ledger = json.loads( + Path(self.config.inference_usage_path).read_text(encoding="utf-8") + ) + scopes = ledger.get("scopes") + if not isinstance(scopes, dict): + return {} + totals = { + "requests": 0, + "input_tokens": 0, + "cached_input_tokens": 0, + "output_tokens": 0, + "total_tokens": 0, + } + found = False + for scope in scopes.values(): + attribution = (scope.get("attributions") or {}).get(evaluation_id) + if not isinstance(attribution, dict): + continue + found = True + for key in totals: + value = attribution.get(key) + if isinstance(value, (int, float)): + totals[key] += int(value) + if not found: + return {} + return {f"inference_{key}": float(value) for key, value in totals.items()} + except (OSError, ValueError): + logger.warning("inference usage metrics unavailable", exc_info=True) + return {} + + @staticmethod + def _attempt_wall_seconds(attempt: dict[str, Any]) -> float | None: + started = attempt.get("started_at") + finished = attempt.get("finished_at") + if not isinstance(started, str) or not isinstance(finished, str): + return None + try: + delta = datetime.fromisoformat( + finished.replace("Z", "+00:00") + ) - datetime.fromisoformat(started.replace("Z", "+00:00")) + except ValueError: + return None + seconds = delta.total_seconds() + return seconds if seconds >= 0 else None + + def _case_wall_seconds(self, attempts: list[dict[str, Any]]) -> float | None: + durations = [ + seconds + for attempt in attempts + if (seconds := self._attempt_wall_seconds(attempt)) is not None + ] + return max(durations) if durations else None + + @staticmethod + def _phase_seconds(phase: Any) -> float | None: + if not isinstance(phase, dict): + return None + started, finished = phase.get("started_at"), phase.get("finished_at") + if not isinstance(started, str) or not isinstance(finished, str): + return None + try: + delta = datetime.fromisoformat( + finished.replace("Z", "+00:00") + ) - datetime.fromisoformat(started.replace("Z", "+00:00")) + except ValueError: + return None + seconds = delta.total_seconds() + return seconds if seconds >= 0 else None + + # Harbor's own phase names, in the order a trial runs them. + _TRIAL_PHASES = ("environment_setup", "agent_setup", "agent_execution", "verifier") + + def _execution_trace(self, attempts: list[dict[str, Any]]) -> list[JsonValue]: + """One span per trial phase, so `evals trace` has something to show. + + Harbor already records a start/finish window per phase and the agent's own + rollout details; without lifting them into ``CaseResult.execution_trace`` + the whole trace surface (`evals cases` reporting a trace, `evals trace + ID CASE`, `--span N`) is inert on this backend and an agent has to go + spelunking in ``artifacts/harbor/jobs/**`` with raw file tools instead. + + Every span carries the same keys so the summary's shape grouping stays + legible; anything phase-specific and potentially large (rollout details, + tracebacks) goes under ``detail``, which the CLI windows by span. + """ + + spans: list[JsonValue] = [] + for index, attempt in enumerate(attempts): + trial_name = attempt.get("trial_name") + + def span(phase: str, seconds: float | None, detail: Any) -> None: + source = attempt.get(phase) if phase in self._TRIAL_PHASES else None + spans.append( + { + "attempt": index, + "trial_name": trial_name, + "phase": phase, + "started_at": (source or {}).get("started_at") + if isinstance(source, dict) + else attempt.get("started_at"), + "finished_at": (source or {}).get("finished_at") + if isinstance(source, dict) + else attempt.get("finished_at"), + "seconds": seconds, + "detail": detail, + } + ) + + for phase in self._TRIAL_PHASES: + if not isinstance(attempt.get(phase), dict): + continue + detail: dict[str, Any] = {} + if phase == "agent_execution": + detail["agent_info"] = attempt.get("agent_info") + # rollout_details is the harness's own turn-by-turn record + # when it reports one; large by nature, hence span windowing. + detail["agent_result"] = attempt.get("agent_result") + elif phase == "verifier": + detail["rewards"] = ( + attempt.get("verifier_result") or {} + ).get("rewards") + span(phase, self._phase_seconds(attempt.get(phase)), detail or None) + + failure = attempt.get("exception_info") + if isinstance(failure, dict) and failure: + span("exception", None, failure) + return spans + + @staticmethod + def _agent_reported_tokens(attempts: list[dict[str, Any]]) -> dict[str, float]: + """Sum the agent-self-reported token counts across a case's attempts. + + Harbor agents record their own usage in ``agent_result`` (stock + adapters and the baseline agents alike). Self-declared, so telemetry + grade — the gateway's per-evaluation ``inference_*`` metrics remain + the trusted envelope. + """ + names = { + "n_input_tokens": "agent_reported_input_tokens", + "n_cache_tokens": "agent_reported_cached_input_tokens", + "n_output_tokens": "agent_reported_output_tokens", + } + totals: dict[str, float] = {} + for attempt in attempts: + result = attempt.get("agent_result") + if not isinstance(result, dict): + continue + for source, metric in names.items(): + value = result.get(source) + if isinstance(value, (int, float)) and math.isfinite(float(value)): + totals[metric] = totals.get(metric, 0.0) + float(value) + return totals + + @staticmethod + def _case_distribution( + case_results: list[EvaluationCaseResult], metric: str + ) -> dict[str, float]: + """mean/median/max of a per-case cost metric. + + Latency and token distributions are unbounded and heavy-tailed — a few + cases carry much of the total — so the median is reported beside the + mean, and the max names the tail (a case past its wall budget is + stopped and scores the failure value). Accuracy, being bounded, stays + mean plus stddev. + """ + values = [ + value + for case in case_results + if isinstance((value := case.metrics.get(metric)), (int, float)) + ] + if not values: + return {} + return { + f"mean_case_{metric}": sum(values) / len(values), + f"median_case_{metric}": float(statistics.median(values)), + f"max_case_{metric}": float(max(values)), + } + + def _case_result( + self, + case: HarborCase, + attempts: list[dict[str, Any]], + *, + artifact_root: Path, + trusted: bool = False, + ) -> tuple[CaseResult, float]: + trial_artifacts = self._trial_artifacts(attempts, artifact_root) + trace = self._execution_trace(attempts) + wall_seconds = self._case_wall_seconds(attempts) + reported_tokens = self._agent_reported_tokens(attempts) + attempt_detail = [ + { + "reward": self._attempt_reward(attempt), + "exception": (attempt.get("exception_info") or {}).get( + "exception_type" + ), + } + for attempt in attempts + ] + output: dict[str, JsonValue] = { + "task_name": case.task_name, + "result_task_name": case.expected_result_task_name, + } + if self.config.expose_attempt_detail: + output["attempts"] = attempt_detail + + if self.config.aggregate_attempts == "mean" and attempts: + rewards = [self._attempt_reward(attempt) for attempt in attempts] + if any(reward is not None for reward in rewards): + measured = [ + self.config.failure_score if reward is None else reward + for reward in rewards + ] + score = sum(measured) / len(measured) + output["attempt_scores"] = measured + output["aggregate"] = "mean" + # Split the zero-filled dead attempts so an outage-diluted mean + # is distinguishable from a genuinely clean low mean: n_dead_infra + # are dead-to-infrastructure; n_clean are the rest (scored or a + # real candidate failure). + n_dead_infra = sum( + 1 + for attempt, reward in zip(attempts, rewards) + if reward is None + and self._attempt_is_infra(attempt, trusted=trusted) + ) + return ( + CaseResult( + case_id=case.id, + status=CaseStatus.SUCCESS, + metrics={ + "score": score, + "n_attempts": float(len(attempts)), + "n_scored": float( + sum(reward is not None for reward in rewards) + ), + "n_dead_infra": float(n_dead_infra), + "n_clean": float(len(attempts) - n_dead_infra), + **( + {"wall_seconds": wall_seconds} + if wall_seconds is not None + else {} + ), + **reported_tokens, + }, + input={"task_name": case.task_name, **case.metadata}, + output=output, + feedback=( + self._transcript_feedback(attempts) + if score == self.config.failure_score + else None + ), + artifacts=trial_artifacts, + execution_trace=trace, + ), + score, + ) + + best, score = self._best_attempt(attempts) + if best is not None and score is not None: + rewards = (best.get("verifier_result") or {}).get("rewards") or {} + output.update( + { + "trial_name": best.get("trial_name"), + "rewards": rewards, + "aggregate": "best", + } + ) + numeric_rewards = { + key: float(value) + for key, value in rewards.items() + if isinstance(value, (int, float)) and math.isfinite(float(value)) + } + numeric_rewards["score"] = score + if wall_seconds is not None: + numeric_rewards["wall_seconds"] = wall_seconds + numeric_rewards.update(reported_tokens) + return ( + CaseResult( + case_id=case.id, + status=CaseStatus.SUCCESS, + metrics=numeric_rewards, + input={"task_name": case.task_name, **case.metadata}, + output=output, + feedback=( + self._transcript_feedback(attempts) + if score == self.config.failure_score + else None + ), + artifacts=trial_artifacts, + execution_trace=trace, + ), + score, + ) + + # No scored attempt. Decide, from the single source of truth, whether + # this is the candidate's own failure (an informative low score) or an + # infrastructure loss (excluded from the aggregate score). + exception_counts: dict[str, int] = {} + signals: list[str] = [] + for attempt in attempts: + info = attempt.get("exception_info") or {} + name = info.get("exception_type") + key = str(name or NO_REWARD_SIGNAL) + exception_counts[key] = exception_counts.get(key, 0) + 1 + signals.append(key) + # The gateway embeds a distinct "budget_exhausted" code in the error + # body. Classify on the message rather than the status or exception + # type: the in-container client collapses the type to whatever its + # SDK maps the status to, but the message survives intact. + for detail_key in ("message", "detail", "exception_message"): + detail = info.get(detail_key) + if isinstance(detail, str) and detail: + signals.append(detail) + + if not attempts: + # Harbor produced no trial for this task at all: infrastructure + # dropped the case (see the coverage gate), not the candidate. This + # coverage gap is harness-produced and stays excluded even for + # competitive evaluations. + category = ErrorCategory.TRANSIENT_INFRA + else: + category = classify_case(signals) + if not trusted and category == ErrorCategory.TRANSIENT_INFRA: + # A trial ran and died with a transient-looking exception. For + # competitive (agent) selection we cannot trust a candidate- + # controlled process's own exception text to mean + # "infrastructure": excluding it would drop the case from its + # own denominator and let a candidate inflate its mean by + # emitting a timeout/connection error on hard cases. Score it at + # the failure value instead. Genuine infrastructure is caught + # out of band (coverage gaps above; gateway-ledger budget/auth, + # which remain terminating) and via trusted-only retry. + category = ErrorCategory.TASK_FAILURE + category_policy = policy(category) + output["dead_exception_types"] = exception_counts + output["error_category"] = category.value + + if not attempts: + message = ( + f"Harbor produced no trial for task {case.task_name!r} " + "(infrastructure dropped the case)" + ) + else: + message = f"No verifier reward for Harbor task {case.task_name!r}" + if exception_counts: + causes = ", ".join( + f"{name} x{count}" + for name, count in sorted(exception_counts.items()) + ) + message += f"; attempts: {causes}" + + if category_policy.is_informative_sample: + # The candidate produced no answer or crashed on its own: a real, + # scoreable outcome at the failure value, not infrastructure noise. + output["aggregate"] = "task_failure" + return ( + CaseResult( + case_id=case.id, + status=CaseStatus.SUCCESS, + metrics={"score": self.config.failure_score}, + input={"task_name": case.task_name, **case.metadata}, + output=output, + feedback=self._transcript_feedback(attempts), + metadata={"error_category": category.value}, + artifacts=trial_artifacts, + execution_trace=trace, + ), + self.config.failure_score, + ) + + # An infrastructure case: excluded from the aggregate, counted toward + # invalidity, and — for budget/auth — a terminating condition surfaced + # to the aggregation via its category. + return ( + CaseResult( + case_id=case.id, + status=CaseStatus.ERROR, + metrics={"score": self.config.failure_score}, + input={"task_name": case.task_name, **case.metadata}, + output=output, + feedback=self._transcript_feedback(attempts), + metadata={"error_category": category.value}, + artifacts=trial_artifacts, + execution_trace=trace, + errors=[ + CaseError( + message=message, + code=category_policy.diagnostic_code, + phase="harbor", + terminal=True, + ) + ], + ), + self.config.failure_score, + ) + + async def evaluate( + self, + *, + context: EvaluationContext, + request: EvaluationRequest, + ) -> EvaluationReport: + self.validate_request(request) + target_root = context.workspace.sandbox.host_path( + context.workspace.project_path + ) + if target_root is not None: + target_root = target_root.resolve() + cases_path = Path(self.config.cases_path).resolve() + if cases_path == target_root or cases_path.is_relative_to(target_root): + raise ValueError("Harbor cases must live outside the editable target") + source = Path(self.config.task_source).expanduser() + local_task_source = source.exists() + if local_task_source and target_root is not None: + resolved_source = source.resolve() + if resolved_source == target_root or resolved_source.is_relative_to( + target_root + ): + raise ValueError( + "local Harbor tasks must live outside the editable target" + ) + + cases = self._selected_cases(request.evaluation_set) + capture_dir = context.artifact_dir / "harbor" + capture_dir.mkdir(parents=True, exist_ok=True) + requested_tasks = {case.expected_result_task_name for case in cases} + attempts: list[tuple[CommandResult, str, str]] = [] + groups: dict[str, list[dict[str, Any]]] = {} + jobs_dir = capture_dir / "jobs" + # Whole-sub-run infrastructure retry is a trusted-only affordance: for a + # competitive (agent) evaluation, re-running the sub-run and keeping the + # best coverage is a best-of-N amplifier over a candidate that can + # itself trigger the retry. Trusted finalization re-scores, run by the + # operator on a controlled environment, keep the configured retries. + max_infra_attempts = ( + self.config.infrastructure_max_attempts if context.finalization else 1 + ) + for attempt in range(1, max_infra_attempts + 1): + attempt_jobs_dir = ( + jobs_dir if attempt == 1 else capture_dir / f"retry-{attempt}" / "jobs" + ) + attempt_jobs_dir.mkdir(parents=True, exist_ok=True) + async with SandboxStagingArea( + context.workspace.sandbox, + prefix=f"vero-harbor-{context.evaluation_id[:8]}-{attempt}-", + ) as staging: + remote_jobs_dir = await staging.mkdir("jobs") + task_source = ( + ( + str(source.resolve()) + if context.workspace.sandbox.capabilities.host_paths + else await staging.upload(source.resolve(), "tasks") + ) + if local_task_source + else self.config.task_source + ) + retry_config_path = await staging.write_text( + "retry-config.json", self._retry_config_json() + ) + command = self._command( + workspace=context.workspace.project_path, + request=request, + cases=cases, + jobs_dir=remote_jobs_dir, + task_source=task_source, + local_task_source=local_task_source, + retry_config_path=retry_config_path, + ) + # Isolate the untrusted harness: hand the unprivileged user + # ownership of exactly the dirs it must read and write (its + # workspace and the staging tree), then run harbor as that user. + # Everything trusted (session dir, budgets, other candidates, + # admin token) is owned by root and unreadable to it. + if self.config.harness_user is not None: + # Hand the harness user its work dirs and make the checkout + # reachable (the parent that `mktemp -d` left 0700 root); see + # vero.sidecar.isolation for the why. + for provision_command in harness_grant_commands( + self.config.harness_user, + chown_paths=[context.workspace.project_path, staging.root], + checkout_root=context.workspace.root, + ): + provision = await context.workspace.sandbox.run( + provision_command, timeout=120 + ) + if provision.returncode != 0: + raise RuntimeError( + "failed to provision harness workspace " + f"({' '.join(provision_command)}): " + f"{self.sanitize_error(provision.stderr)}" + ) + # Fail fast, at the provisioning site, if the dropped user + # still can't reach its workspace, instead of a cryptic + # "No module named " several retries downstream. + probe = await context.workspace.sandbox.run( + harness_reachability_probe(context.workspace.project_path), + run_as=self.config.harness_user, + ) + if probe.returncode != 0: + raise RuntimeError( + f"harness {self.config.harness_user!r} cannot reach its " + f"workspace {context.workspace.project_path!r} after " + "provisioning; check that every ancestor directory is " + "traversable by the dropped user" + ) + result = await context.workspace.sandbox.run( + command, + cwd=context.workspace.project_path, + timeout=request.limits.timeout_seconds, + env=self._environment( + context.evaluation_id, finalization=context.finalization + ), + run_as=self.config.harness_user, + ) + await staging.download("jobs", attempt_jobs_dir) + stdout = self.sanitize_error(result.stdout) + stderr = self.sanitize_error(result.stderr) + attempts.append((result, stdout, stderr)) + groups = self._trial_groups(attempt_jobs_dir) + # Require FULL coverage before accepting the sub-run. Partial + # coverage used to be accepted silently, scoring the dropped tasks + # as zeros; now we retry, and any tasks still missing after retries + # are surfaced as infrastructure cases (excluded from the mean, + # counted toward invalidity) rather than folded in as zeros. + if requested_tasks <= set(groups): + jobs_dir = attempt_jobs_dir + break + if attempt < max_infra_attempts: + await asyncio.sleep( + self.config.infrastructure_retry_delay_seconds * attempt + ) + + result, stdout, stderr = attempts[-1] + (capture_dir / "stdout.log").write_text(stdout, encoding="utf-8") + (capture_dir / "stderr.log").write_text(stderr, encoding="utf-8") + artifacts = [ + EvaluationArtifact( + path="harbor/stdout.log", + media_type="text/plain", + description="Harbor standard output", + ), + EvaluationArtifact( + path="harbor/stderr.log", + media_type="text/plain", + description="Harbor standard error", + ), + ] + + matching_tasks = requested_tasks & set(groups) + if not matching_tasks: + # Distinguish "sub-run produced nothing" from "produced trials whose + # task_name doesn't match what we requested" — the two collapse to the + # same failure otherwise. Surface the found group keys + exit status. + detail = ( + f"no matching trials for {len(cases)} requested tasks after " + f"{len(attempts)} attempts; requested e.g. {sorted(requested_tasks)[:2]}, " + f"harbor produced {len(groups)} trial group(s) e.g. {sorted(groups)[:3]}, " + f"returncode={result.returncode}" + ) + message = (stderr.strip() + " || " + detail) if stderr.strip() else detail + report = EvaluationReport( + status=EvaluationStatus.FAILED, + diagnostics=[ + EvaluationDiagnostic( + code="infrastructure_failure", + message=message, + severity=DiagnosticSeverity.ERROR, + phase="harbor", + ) + ], + artifacts=artifacts, + ) + return sanitize_evaluation_report(report, self._secrets()) + + case_results: list[CaseResult] = [] + scores: list[float] = [] + for case in cases: + case_result, score = self._case_result( + case, + groups.get(case.expected_result_task_name, []), + artifact_root=context.artifact_dir, + trusted=context.finalization, + ) + case_results.append(case_result) + scores.append(score) + await context.case_store.save(case_result) + diagnostics = [] + if result.returncode != 0: + diagnostics.append( + EvaluationDiagnostic( + code=( + "harbor_partial_timeout" + if result.returncode == -1 + else "harbor_nonzero_exit" + ), + message=stderr.strip() + or f"Harbor exited with status {result.returncode}; partial trials collated", + severity=DiagnosticSeverity.WARNING, + phase="harbor", + ) + ) + + # Coverage: any requested task that produced no trial is an + # infrastructure loss, recorded loudly rather than silently zeroed. + missing_tasks = sorted(requested_tasks - set(groups)) + if missing_tasks: + diagnostics.append( + EvaluationDiagnostic( + code="harbor_incomplete_coverage", + message=( + f"{len(missing_tasks)} of {len(requested_tasks)} requested " + f"tasks produced no trial after {len(attempts)} attempt(s): " + f"{missing_tasks[:5]}" + ), + severity=DiagnosticSeverity.ERROR, + phase="harbor", + ) + ) + + # Infrastructure cases are excluded from the aggregate score; only + # informative samples (successes and legitimate task failures) count. + informative_scores = [ + score + for case, score in zip(case_results, scores) + if case.status != CaseStatus.ERROR + ] + infra_cases = [ + case for case in case_results if case.status == CaseStatus.ERROR + ] + + def _category(case: CaseResult) -> ErrorCategory | None: + raw = case.metadata.get("error_category") + try: + return ErrorCategory(raw) if isinstance(raw, str) else None + except ValueError: + return None + + # A terminating condition (inference-budget exhaustion or auth failure) + # anywhere fails the whole evaluation loudly: distinct code, no score. + terminating = next( + ( + _category(case) + for case in infra_cases + if (category := _category(case)) is not None + and policy(category).terminating + ), + None, + ) + if terminating is not None: + report = EvaluationReport( + status=EvaluationStatus.INVALID, + metrics={ + "error_rate": len(infra_cases) / len(case_results), + }, + cases=case_results, + diagnostics=[ + *diagnostics, + EvaluationDiagnostic( + code=policy(terminating).diagnostic_code, + message=( + f"terminating condition {terminating.value!r} reached; " + "the run cannot continue and must not be retried" + ), + severity=DiagnosticSeverity.ERROR, + phase="harbor", + ), + ], + artifacts=artifacts, + ) + return sanitize_evaluation_report(report, self._secrets()) + + # No informative sample survived (every case was infrastructure): the + # aggregate is undefined, so the whole evaluation is invalid. Retryable + # transient loss keeps the historical infrastructure_failure code so the + # engine still refunds and lets the optimizer retry. + if not informative_scores: + diagnostics.append( + EvaluationDiagnostic( + code="infrastructure_failure", + message=( + "no informative sample survived; every case was lost to " + "infrastructure after Harbor retries were exhausted" + ), + severity=DiagnosticSeverity.ERROR, + phase="harbor", + ) + ) + report = EvaluationReport( + status=EvaluationStatus.INVALID, + metrics={"error_rate": len(infra_cases) / len(case_results)}, + cases=case_results, + diagnostics=diagnostics, + artifacts=artifacts, + ) + return sanitize_evaluation_report(report, self._secrets()) + + # Per-case cost distributions. wall_seconds and the agent-reported token + # counts are recorded per case, so each gets the same mean/median/max + # treatment; the trusted gateway metering below is per evaluation, so + # only its mean is derivable here (per-case attribution of the gateway + # log is post-hoc — see harness-engineering-bench/scripts). + case_distributions: dict[str, float] = {} + for metric in ( + "wall_seconds", + "agent_reported_input_tokens", + "agent_reported_cached_input_tokens", + "agent_reported_output_tokens", + ): + case_distributions.update(self._case_distribution(case_results, metric)) + inference_usage = self._inference_usage_metrics(context.evaluation_id) + mean_case_inference = { + f"mean_case_{key}": value / len(case_results) + for key, value in inference_usage.items() + } + reported_totals: dict[str, float] = {} + for case in case_results: + for key, value in case.metrics.items(): + if key.startswith("agent_reported_") and isinstance( + value, (int, float) + ): + total_key = key.replace("agent_reported_", "agent_reported_total_") + reported_totals[total_key] = ( + reported_totals.get(total_key, 0.0) + float(value) + ) + report = EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={ + "score": sum(informative_scores) / len(informative_scores), + "error_rate": len(infra_cases) / len(case_results), + # Spread across informative cases, so a real difference between + # candidates is distinguishable from evaluation noise. + "score_stddev": ( + statistics.pstdev(informative_scores) + if len(informative_scores) > 1 + else 0.0 + ), + # Cost/latency telemetry: reported alongside accuracy, never + # part of the score. + **case_distributions, + **reported_totals, + **inference_usage, + **mean_case_inference, + }, + cases=case_results, + diagnostics=diagnostics, + artifacts=artifacts, + ) + return sanitize_evaluation_report(report, self._secrets()) diff --git a/vero/src/vero/harbor/build/__init__.py b/vero/src/vero/harbor/build/__init__.py new file mode 100644 index 00000000..ceb851c9 --- /dev/null +++ b/vero/src/vero/harbor/build/__init__.py @@ -0,0 +1,33 @@ +"""Compile a program-optimization setup into a runnable Harbor task. + +Four modules, in the order a build passes through them: specs.py declares the +leaf models a build.yaml composes, config.py assembles them into +HarborBuildConfig and enforces the rules that span groups, loader.py reads the +YAML into one, and compiler.py lowers it into a task directory. +""" + +from vero.harbor.build.compiler import compile_harbor_task +from vero.harbor.build.config import HarborBuildConfig +from vero.harbor.build.loader import load_harbor_build_config +from vero.harbor.build.specs import ( + AgentAccessSpec, + CommandBackendSpec, + InferenceBudgetSpec, + InferenceGatewaySpec, + VerificationTargetSpec, + WandbSpec, + WorkspaceOverlaySpec, +) + +__all__ = [ + "AgentAccessSpec", + "CommandBackendSpec", + "HarborBuildConfig", + "InferenceBudgetSpec", + "InferenceGatewaySpec", + "VerificationTargetSpec", + "WandbSpec", + "WorkspaceOverlaySpec", + "compile_harbor_task", + "load_harbor_build_config", +] diff --git a/vero/src/vero/harbor/build/compiler.py b/vero/src/vero/harbor/build/compiler.py new file mode 100644 index 00000000..a856798f --- /dev/null +++ b/vero/src/vero/harbor/build/compiler.py @@ -0,0 +1,809 @@ +"""Compile a trusted VeRO configuration into a runnable Harbor task.""" + +from __future__ import annotations + +import importlib.resources +import io +import json +import logging +import os +import re +import shutil +import subprocess +import tarfile +import tomllib +from importlib.metadata import version as distribution_version +from pathlib import Path, PurePosixPath + +from vero.evaluation import ( + EvaluationBudget, + EvaluationLimits, + EvaluationSet, + RetryPolicy, +) +from vero.gateway.inference import generate_inference_token, token_digest +from vero.harbor.build.config import HarborBuildConfig +from vero.harbor.build.specs import WorkspaceOverlaySpec +from vero.layout import LAYOUT + +logger = logging.getLogger(__name__) + +_TEMPLATES = Path(__file__).parent / "templates" +_VERO_COPY = ("pyproject.toml", "README.md", "uv.lock", "src") + +SESSION_ID = "trial" +UPSTREAM_API_KEY_ENV = LAYOUT.gateway_upstream_api_key_env +UPSTREAM_BASE_URL_ENV = LAYOUT.gateway_upstream_base_url_env +PRODUCER_BASE_URL = LAYOUT.scope_url("producer", LAYOUT.optimizer_attribution) + +# Credentials the compose template routes through the gateway by setting them +# explicitly, instead of blanking them like every other declared secret. The two +# halves are one invariant: a name here must be set below, and a name set below +# must be here, or the rendered compose emits the key twice. +GATEWAY_ROUTED_CREDENTIALS = frozenset(LAYOUT.routed_credential_envs) + +# Container paths and service identities come from the layout, never from a +# literal here: the templates read the same object, so the two cannot drift. +VERO_DIR = LAYOUT.vero +TRUSTED_REPO = LAYOUT.trusted_repo +AGENT_REPO = LAYOUT.target_repo +CASES_DIR = LAYOUT.cases +TASK_SOURCE_DIR = LAYOUT.task_source +HARNESS_DIR = LAYOUT.harness +SERVE_CONFIG = LAYOUT.serve_config +AGENT_VOLUME = LAYOUT.agent_volume +ADMIN_VOLUME = LAYOUT.admin_volume +SESSION_DIR = LAYOUT.session_dir +TOKEN_PATH = LAYOUT.token_path +INFERENCE_STATE = LAYOUT.inference_state +INFERENCE_REQUEST_LOG_DIR = LAYOUT.inference_request_log_dir +INFERENCE_GATEWAY_URL = LAYOUT.gateway_url + +# How long finalization waits for already-running agent evaluations before +# cancelling them. A grace period, not a ceiling: expiry is graceful (the +# evaluator's cancellation path persists terminal records and refunds budgets), +# so waiting longer buys nothing and only delays the held-out score. Matches +# harbor/deployment.py's own default. +DEFAULT_EVALUATION_DRAIN_SECONDS = 600.0 + + +def _backend_id(partition: str) -> str: + return f"harbor-{partition}" + + +def _is_vero_source(vero_root: Path) -> bool: + return (vero_root / "pyproject.toml").is_file() and ( + vero_root / "src/vero" + ).is_dir() + + +def _copy_vero_source(vero_root: Path, destination: Path) -> None: + destination.mkdir(parents=True, exist_ok=True) + for name in _VERO_COPY: + source = vero_root / name + if not source.exists(): + continue + if source.is_dir(): + shutil.copytree(source, destination / name, dirs_exist_ok=True) + else: + shutil.copy2(source, destination / name) + + +def _rewrite_vero_path(pyproject: Path) -> None: + if not pyproject.exists(): + return + original = pyproject.read_text(encoding="utf-8") + rewritten = re.sub( + r'(scale-vero\s*=\s*\{[^}]*?path\s*=\s*")[^"]*(")', + rf"\g<1>{VERO_DIR}\g<2>", + original, + ) + if rewritten != original: + pyproject.write_text(rewritten, encoding="utf-8") + + +def _safe_extract_tar(payload: bytes, destination: Path) -> None: + with tarfile.open(fileobj=io.BytesIO(payload), mode="r:") as archive: + for member in archive.getmembers(): + path = PurePosixPath(member.name) + if path.is_absolute() or ".." in path.parts: + raise ValueError(f"unsafe path in Git archive: {member.name!r}") + if member.issym() or member.islnk(): + link = PurePosixPath(member.linkname) + if link.is_absolute() or ".." in link.parts: + raise ValueError(f"unsafe link in Git archive: {member.linkname!r}") + # filter="data" strips device files / setuid bits and neutralizes unsafe + # links, matching extract_harbor_session_archive's defensive posture. + archive.extractall(destination, filter="data") + + +def _prepare_baseline_repo( + source: Path, + destination: Path, + *, + rewrite_vero_path: bool, +) -> str: + destination.mkdir(parents=True, exist_ok=True) + repository = subprocess.run( + ["git", "-C", str(source), "rev-parse", "--show-toplevel"], + capture_output=True, + text=True, + ) + if repository.returncode == 0: + root = Path(repository.stdout.strip()) + relative = source.relative_to(root) + treeish = "HEAD" if str(relative) == "." else f"HEAD:{relative.as_posix()}" + archived = subprocess.run( + ["git", "-C", str(root), "archive", "--format=tar", treeish], + capture_output=True, + ) + if archived.returncode != 0: + raise RuntimeError( + "git archive failed: " + + archived.stderr.decode("utf-8", errors="replace").strip() + ) + _safe_extract_tar(archived.stdout, destination) + else: + shutil.copytree( + source, + destination, + dirs_exist_ok=True, + ignore=shutil.ignore_patterns(".git"), + ) + if rewrite_vero_path: + _rewrite_vero_path(destination / "pyproject.toml") + if (destination / ".evals").exists(): + raise ValueError("agent baseline contains reserved path '.evals'") + + def git(*arguments: str) -> str: + result = subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "-C", + str(destination), + *arguments, + ], + check=True, + capture_output=True, + text=True, + ) + return result.stdout.strip() + + git("init", "-q") + git("add", "-A") + git("commit", "-q", "-m", "baseline") + return git("rev-parse", "HEAD") + + +def _local_result_task_name(task_source: Path, selector: str) -> str: + root = task_source.resolve() + task_dir = (root / selector).resolve() + if task_dir.parent != root: + raise ValueError( + f"local Harbor task selector {selector!r} must name a direct child directory" + ) + config_path = task_dir / "task.toml" + if not config_path.is_file(): + raise ValueError(f"local Harbor task selector {selector!r} has no task.toml") + try: + value = tomllib.loads(config_path.read_text(encoding="utf-8")) + task_name = value["task"]["name"] + except (KeyError, TypeError, tomllib.TOMLDecodeError) as error: + raise ValueError( + f"local Harbor task selector {selector!r} has no canonical task.name" + ) from error + if not isinstance(task_name, str) or not task_name.strip(): + raise ValueError( + f"local Harbor task selector {selector!r} has no canonical task.name" + ) + return task_name + + +def _write_cases(config: HarborBuildConfig, destination: Path) -> None: + destination.mkdir(parents=True, exist_ok=True) + # Only a harbor backend names a task_source, and only then does a case id + # correspond to a local Harbor task directory with a canonical name. A + # command backend's ids name whatever its harness understands. + task_source = Path(config.task_source) if config.task_source else None + local = task_source is not None and task_source.exists() + for partition, tasks in config.partitions.items(): + path = destination / f"{partition}.jsonl" + lines = [ + json.dumps( + { + "id": task, + "task_name": task, + **( + { + "result_task_name": _local_result_task_name( + task_source, + task, + ) + } + if local + else {} + ), + }, + ensure_ascii=False, + ) + for task in tasks + ] + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def _deployment_config( + config: HarborBuildConfig, + *, + baseline_version: str, + local_task_source: bool, + evaluation_inference_token: str | None, + finalization_inference_token: str | None, +) -> dict: + task_source = TASK_SOURCE_DIR if local_task_source else config.task_source + backends = {} + if config.command_backend is not None: + # A command backend scores by running a program, so it needs none of the + # nested-`harbor run` plumbing: no task source, no agent import path, no + # model. The harness is baked into the sidecar at HARNESS_DIR, and case + # enumeration is the harness's job — it reads the partition's case file + # from VERO_CASES_DIR, since the backend itself derives a case count from + # the requested selection rather than from a cases file. + specification = config.command_backend + for partition in config.partitions: + backends[_backend_id(partition)] = { + "type": "command", + "harness_root": HARNESS_DIR, + "command": list(specification.command), + "working_directory": specification.working_directory, + "environment": { + "VERO_CASES_DIR": CASES_DIR, + **specification.environment, + }, + "passthrough_environment": list( + dict.fromkeys( + [*specification.passthrough_environment, *config.secrets] + ) + ), + "staged_inputs": dict(specification.staged_inputs), + "agent_context_inputs": { + name: list(paths) + for name, paths in specification.agent_context_inputs.items() + }, + } + else: + # A target may override attempts-per-case on its own (held-out) partition + # without touching search/selection. Backends are per-partition, so the + # override is injected into that partition's backend at compile time; a + # config validator forbids overriding an agent-evaluable partition. + target_n_attempts = { + target.partition: target.n_attempts + for target in config.targets + if target.n_attempts is not None + } + target_aggregate = { + target.partition: target.aggregate_attempts + for target in config.targets + if target.aggregate_attempts is not None + } + for partition in config.partitions: + backends[_backend_id(partition)] = { + "type": "harbor", + "task_source": task_source, + "agent_import_path": config.agent_import_path, + "cases_path": f"{CASES_DIR}/{partition}.jsonl", + "harbor_requirement": config.harbor_requirement, + "evaluation_set_name": config.evaluation_set_name, + "partition": partition, + "model": config.model, + "environment_name": config.environment_name, + "python_version": config.harbor_python_version, + "case_timeout_seconds": config.case_timeout_seconds, + "task_agent_timeout_seconds": config.task_agent_timeout_seconds, + "default_index": config.default_index, + "n_attempts": target_n_attempts.get(partition, config.n_attempts), + "max_retries": config.max_retries, + "retry_wait_multiplier": config.retry_wait_multiplier, + "retry_min_wait_seconds": config.retry_min_wait_seconds, + "retry_max_wait_seconds": config.retry_max_wait_seconds, + "infrastructure_max_attempts": config.infrastructure_max_attempts, + "infrastructure_retry_delay_seconds": ( + config.infrastructure_retry_delay_seconds + ), + "reward_key": config.reward_key, + "aggregate_attempts": target_aggregate.get( + partition, config.aggregate_attempts + ), + "feedback_transcripts": config.feedback_transcripts, + "feedback_max_bytes": config.feedback_max_bytes, + "expose_attempt_detail": config.expose_attempt_detail, + "passthrough_environment": config.secrets, + "environment": config.task_environment, + "inference_gateway_url": ( + INFERENCE_GATEWAY_URL + if config.inference_gateway is not None + else None + ), + "inference_gateway_token": evaluation_inference_token, + "inference_gateway_finalization_token": finalization_inference_token, + "harness_user": config.harness_user, + "task_services_use_upstream": config.task_services_use_upstream, + "upstream_api_key_env": ( + UPSTREAM_API_KEY_ENV + if config.inference_gateway is not None + else None + ), + "upstream_base_url_env": ( + UPSTREAM_BASE_URL_ENV + if config.inference_gateway is not None + and config.inference_gateway.upstream_base_url_env is not None + else None + ), + "case_resources_cache_path": ( + f"{LAYOUT.case_resources_dir}/{partition}" + ), + "inference_usage_path": ( + INFERENCE_STATE if config.inference_gateway is not None else None + ), + "extra_args": config.extra_harbor_args, + } + + limits = EvaluationLimits( + timeout_seconds=config.timeout_seconds, + case_timeout_seconds=config.case_timeout_seconds, + max_concurrency=config.max_concurrency, + error_rate_threshold=config.error_rate_threshold, + retry=RetryPolicy.disabled(), + ) + policies = [] + budgets = [] + for access in config.agent_access: + backend_id = _backend_id(access.partition) + evaluation_set = EvaluationSet( + name=config.evaluation_set_name, + partition=access.partition, + ) + policies.append( + { + "backend_id": backend_id, + "evaluation_set_name": config.evaluation_set_name, + "partition": access.partition, + "objective": config.objective.model_dump(mode="json"), + "access": access.to_access_policy().model_dump(mode="json"), + "parameters": {}, + "allowed_parameters": [], + "limits": limits.model_dump(mode="json"), + } + ) + if access.total_runs is not None or access.total_cases is not None: + budgets.append( + EvaluationBudget( + backend_id=backend_id, + evaluation_set_key=evaluation_set.budget_key(backend_id), + total_runs=access.total_runs, + total_cases=access.total_cases, + ).model_dump(mode="json") + ) + + selection_backend = _backend_id(config.selection_partition) + selection_set = EvaluationSet( + name=config.evaluation_set_name, + partition=config.selection_partition, + ) + targets = [] + for target in config.targets: + parameters = dict(target.parameters) + if target.model is not None: + parameters["harbor_model_override"] = target.model + targets.append( + { + "reward_key": target.reward_key, + "backend_id": _backend_id(target.partition), + "evaluation_set": EvaluationSet( + name=config.evaluation_set_name, + partition=target.partition, + ).model_dump(mode="json"), + "objective": config.objective.model_dump(mode="json"), + "parameters": parameters, + "limits": limits.model_dump(mode="json"), + "failure_value": target.failure_value, + "reward_scale": ( + 1.0 if config.objective.direction == "maximize" else -1.0 + ), + "baseline_reward": target.baseline_reward, + "max_attempts": target.max_attempts, + } + ) + return { + "task_name": config.name, + "task_description": config.description, + "repo_path": TRUSTED_REPO, + "agent_repo_path": AGENT_REPO, + "session_dir": SESSION_DIR, + "session_id": SESSION_ID, + "backends": backends, + "access_policies": policies, + "budgets": budgets, + "selection": { + "mode": config.reward_mode, + # Always populated so auto_best can serve as the fallback when a + # submit-mode run has no submission. + "backend_id": selection_backend, + "evaluation_set": selection_set.model_dump(mode="json"), + "objective": config.objective.model_dump(mode="json"), + "baseline_version": baseline_version, + "parameters": {}, + "limits": limits.model_dump(mode="json"), + "rescore_top_k": config.rescore_top_k, + "rescore_attempts": config.rescore_attempts, + "baseline_floor": config.baseline_floor, + "baseline_selection_score": config.baseline_selection_score, + "selection_coverage_threshold": config.selection_coverage_threshold, + }, + "targets": targets, + "agent_volume": AGENT_VOLUME, + "admin_volume": ADMIN_VOLUME, + "submit_enabled": config.reward_mode == "submit", + "disclose_budget": config.disclose_budget, + "score_baseline": config.score_baseline, + "wandb": ( + config.wandb.model_dump(mode="json") if config.wandb is not None else None + ), + # Must NOT inherit timeout_seconds: that is deliberately sized to be + # unreachable (every trial hitting its per-case cap), so inheriting it + # turns one hung sub-run into a finalization stall of the same order. + # officeqa run #4 sat 6h in exactly that state -- its agent evaluation + # had finished writing results two hours earlier, but the subprocess + # never exited and the drain was waiting out an inherited 21600s. + "evaluation_drain_timeout_seconds": ( + config.evaluation_drain_timeout_seconds + or DEFAULT_EVALUATION_DRAIN_SECONDS + ), + "inference_usage_path": ( + INFERENCE_STATE if config.inference_gateway is not None else None + ), + "inference_request_log_dir": ( + INFERENCE_REQUEST_LOG_DIR + if config.inference_gateway is not None + and config.inference_gateway.log_requests + else None + ), + "inference_limits": ( + { + "producer": config.inference_gateway.producer.model_dump(mode="json"), + "evaluation": config.inference_gateway.evaluation.model_dump( + mode="json" + ), + } + if config.inference_gateway is not None + else {} + ), + } + + +def _render(template: str, destination: Path, **context) -> None: + """Render one template. Every template gets the layout, so no template needs + to spell out a container path, service name, or port of its own.""" + try: + from jinja2 import Environment, FileSystemLoader, StrictUndefined + except ImportError as error: + raise RuntimeError( + "install scale-vero[harbor] to compile Harbor tasks" + ) from error + environment = Environment( + loader=FileSystemLoader(str(_TEMPLATES)), + undefined=StrictUndefined, + keep_trailing_newline=True, + trim_blocks=True, + lstrip_blocks=True, + autoescape=False, + ) + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_text( + environment.get_template(template).render(layout=LAYOUT, **context), + encoding="utf-8", + ) + + +def compile_harbor_task( + config: HarborBuildConfig, + output_dir: Path | str, + *, + vero_root: Path | None = None, +) -> Path: + """Emit a self-contained Harbor task directory from validated config.""" + output = Path(output_dir).expanduser().resolve() + source_root = (vero_root or Path(__file__).parents[4]).resolve() + use_local_vero = _is_vero_source(source_root) + if vero_root is not None and not use_local_vero: + raise ValueError(f"vero_root {source_root} is not a scale-vero source checkout") + protected = [Path(config.agent_repo).resolve()] + if use_local_vero: + protected.append(source_root) + if config.task_source is not None: + task_source_path = Path(config.task_source) + if task_source_path.exists(): + protected.append(task_source_path.resolve()) + for path in protected: + if output == path or output.is_relative_to(path) or path.is_relative_to(output): + raise ValueError( + f"output directory {output} overlaps protected source {path}" + ) + # Imported here, not at module scope: deployment pulls in the whole runtime + # stack, and harbor/__init__ imports this package before it, so a top-level + # import would only work by accident of partial-initialization ordering. + from vero.harbor.deployment import FACTORY_PATH + + gateway_environment: list[str] = [] + credential_sources: list[str] = [] + if config.inference_gateway is not None: + gateway_environment.append(UPSTREAM_API_KEY_ENV) + credential_sources.append(config.inference_gateway.upstream_api_key_env) + if config.inference_gateway.upstream_base_url_env is not None: + gateway_environment.append(UPSTREAM_BASE_URL_ENV) + credential_sources.append(config.inference_gateway.upstream_base_url_env) + task_environment = list(dict.fromkeys([*config.secrets, *gateway_environment])) + if os.environ.get("VERO_SKIP_SECRET_CHECK") is None: + required_sources = list(dict.fromkeys([*config.secrets, *credential_sources])) + missing = [name for name in required_sources if not os.environ.get(name)] + if missing: + raise ValueError( + "declared task credentials are missing: " + ", ".join(missing) + ) + if output.exists(): + shutil.rmtree(output) + environment_dir = output / "environment" + sidecar_dir = environment_dir / "sidecar" + gateway_dir = environment_dir / "gateway" + environment_dir.mkdir(parents=True) + if use_local_vero: + _copy_vero_source(source_root, environment_dir / "vero") + + baseline = _prepare_baseline_repo( + Path(config.agent_repo), + environment_dir / "agent-baseline", + rewrite_vero_path=use_local_vero, + ) + shutil.copytree( + environment_dir / "agent-baseline", + environment_dir / "agent-seed", + ) + # General filesystem overlay: bake arbitrary host files/dirs into the agent + # workspace at build time. A source dir's *contents* land under dest; a source + # file lands as dest/. dest="." is the workspace root. + overlays = list(config.workspace_overlays) + if config.include_evals_skill: + # vero's own packaged skill; resolved from the installed package so the + # baked copy always matches the vero (and `evals` CLI) in the image. + skill_source = importlib.resources.files("vero") / "skills" / "evals" + overlays.append( + WorkspaceOverlaySpec(source=str(skill_source), dest="skills/evals") + ) + overlay_present = bool(overlays) + overlay_excludes: list[str] = [] + if overlay_present: + overlay_root = environment_dir / "overlay" + for spec in overlays: + source = Path(spec.source) + target = overlay_root if spec.dest == "." else overlay_root / spec.dest + if source.is_dir(): + target.mkdir(parents=True, exist_ok=True) + shutil.copytree(source, target, dirs_exist_ok=True) + else: + target.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, target / source.name) + if spec.dest != ".": + top = spec.dest.split("/", 1)[0] + if top not in overlay_excludes: + overlay_excludes.append(top) + _write_cases(config, sidecar_dir / "cases") + if config.command_backend is not None: + # Bake the scoring program into the trusted sidecar, alongside the cases + # it enumerates. Never reachable from the agent workspace. + shutil.copytree( + Path(config.command_backend.harness_source), + sidecar_dir / "harness", + ) + local_task_source = ( + config.task_source is not None and Path(config.task_source).exists() + ) + if local_task_source: + shutil.copytree( + Path(config.task_source), + sidecar_dir / "task-source", + ) + producer_inference_token = ( + generate_inference_token() if config.inference_gateway is not None else None + ) + evaluation_inference_token = ( + generate_inference_token() if config.inference_gateway is not None else None + ) + finalization_inference_token = ( + generate_inference_token() if config.inference_gateway is not None else None + ) + deployment = _deployment_config( + config, + baseline_version=baseline, + local_task_source=local_task_source, + evaluation_inference_token=evaluation_inference_token, + finalization_inference_token=finalization_inference_token, + ) + (sidecar_dir / "serve.json").write_text( + json.dumps(deployment, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + if config.inference_gateway is not None: + assert producer_inference_token is not None + assert evaluation_inference_token is not None + assert finalization_inference_token is not None + # Finalization reserves its own budget; default to the evaluation policy. + finalization_spec = ( + config.inference_gateway.finalization or config.inference_gateway.evaluation + ) + gateway_dir.mkdir(parents=True, exist_ok=True) + gateway_config = { + "upstream_api_key_env": UPSTREAM_API_KEY_ENV, + "upstream_base_url_env": ( + UPSTREAM_BASE_URL_ENV + if config.inference_gateway.upstream_base_url_env is not None + else None + ), + "default_upstream_base_url": ( + config.inference_gateway.default_upstream_base_url + ), + "state_path": INFERENCE_STATE, + "request_log": ( + { + "directory": INFERENCE_REQUEST_LOG_DIR, + "body_bytes": config.inference_gateway.request_log_body_bytes, + "attribution": config.inference_gateway.request_log_attribution, + } + if config.inference_gateway.log_requests + else None + ), + "scopes": { + "producer": { + "token_sha256": token_digest(producer_inference_token), + **config.inference_gateway.producer.model_dump(mode="json"), + }, + "evaluation": { + "token_sha256": token_digest(evaluation_inference_token), + **config.inference_gateway.evaluation.model_dump(mode="json"), + }, + "finalization": { + "token_sha256": token_digest(finalization_inference_token), + **finalization_spec.model_dump(mode="json"), + }, + }, + } + (gateway_dir / "config.json").write_text( + json.dumps(gateway_config, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + (gateway_dir / "launch.json").write_text( + json.dumps( + { + "upstream_api_key_source": ( + config.inference_gateway.upstream_api_key_env + ), + "upstream_api_key_target": UPSTREAM_API_KEY_ENV, + "upstream_base_url_source": ( + config.inference_gateway.upstream_base_url_env + ), + "upstream_base_url_target": UPSTREAM_BASE_URL_ENV, + "producer_api_key": producer_inference_token, + "producer_base_url": (PRODUCER_BASE_URL), + }, + ensure_ascii=False, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + + selection_access = next( + access + for access in config.agent_access + if access.partition == config.selection_partition + ) + context = { + "name_toml": json.dumps(config.name, ensure_ascii=False), + "description_toml": json.dumps(config.description, ensure_ascii=False), + "description": config.description, + "base_image_main": config.base_image_main, + "base_image_sidecar": config.base_image_sidecar, + "use_local_vero": use_local_vero, + # The trusted sidecar needs the wandb extra when W&B reporting is on. + "sidecar_extras": "harbor,wandb" if config.wandb is not None else "harbor", + "vero_requirement": ( + None + if use_local_vero + else ( + "scale-vero[" + + ("harbor,wandb" if config.wandb is not None else "harbor") + + f"]=={distribution_version('scale-vero')}" + ) + ), + "harbor_requirement": config.harbor_requirement, + "secrets": task_environment, + "sidecar_secrets": config.secrets, + "inference_gateway": config.inference_gateway, + "gateway_environment": gateway_environment, + # Names the main container blanks. Default-deny: every declared secret is + # blanked for the optimizer unless it appears in GATEWAY_ROUTED_CREDENTIALS, + # which the compose template sets explicitly just below the blanking loop + # (to the producer token and the gateway URL). The exclusion exists to + # avoid emitting the same key twice, not to permit anything: adding a new + # credential to `secrets` gets it blanked automatically. + "scrubbed_main_environment": [ + name for name in task_environment if name not in GATEWAY_ROUTED_CREDENTIALS + ], + "producer_inference_token": producer_inference_token, + "evaluation_inference_token": evaluation_inference_token, + "inference_gateway_url": INFERENCE_GATEWAY_URL, + "read_only_paths": config.read_only_paths, + "local_task_source": local_task_source, + "sidecar_factory": FACTORY_PATH, + "producer_base_url": PRODUCER_BASE_URL, + "command_harness": config.command_backend is not None, + # The Harbor backend hard-rejects request.seed, so only advertise the + # flag when the build evaluates through a command backend. + "seed_supported": config.command_backend is not None, + "selection_backend": _backend_id(config.selection_partition), + "evaluation_set_name": config.evaluation_set_name, + "selection_partition": config.selection_partition, + "submit_enabled": config.reward_mode == "submit", + "task_services_use_upstream": config.task_services_use_upstream, + "multifidelity": config.instruct_multifidelity, + "minimum_subset_cases": ( + selection_access.min_aggregate_cases + if selection_access.disclosure == "aggregate" + else 1 + ), + "exposed_partitions": [ + access.partition + for access in config.agent_access + if access.expose_case_resources + ], + "exhaust_budget": config.instruct_exhaust_budget, + "disclose_budget": config.disclose_budget, + "build_timeout": config.build_timeout_seconds, + "verifier_timeout": ( + config.verifier_timeout_seconds or max(1, int(config.timeout_seconds)) + ), + "overlay_present": overlay_present, + "overlay_excludes": overlay_excludes, + } + _render("task.toml.j2", output / "task.toml", **context) + _render("instruction.md.j2", output / "instruction.md", **context) + _render("Dockerfile.main.j2", environment_dir / "Dockerfile", **context) + _render( + "Dockerfile.sidecar.j2", + sidecar_dir / "Dockerfile", + **context, + ) + if config.inference_gateway is not None: + _render( + "Dockerfile.gateway.j2", + gateway_dir / "Dockerfile", + **context, + ) + _render( + "docker-compose.yaml.j2", + environment_dir / "docker-compose.yaml", + **context, + ) + _render("seed.sh.j2", environment_dir / "main/seed.sh", **context) + _render("test.sh.j2", output / "tests/test.sh", **context) + _render("solve.sh.j2", output / "solution/solve.sh", **context) + for script in ( + environment_dir / "main/seed.sh", + output / "tests/test.sh", + output / "solution/solve.sh", + ): + script.chmod(0o755) + logger.info("Compiled Harbor task at %s from baseline %s", output, baseline) + return output diff --git a/vero/src/vero/harbor/build/config.py b/vero/src/vero/harbor/build/config.py new file mode 100644 index 00000000..e318bbca --- /dev/null +++ b/vero/src/vero/harbor/build/config.py @@ -0,0 +1,667 @@ +"""Configuration schema for compiling VeRO optimization tasks for Harbor. + +HarborBuildConfig is one flat YAML document, but its fields fall into six groups +that have little to do with each other. Each group is a private base class below, +carrying its own fields, its own docstring, and the validators that concern only +itself. HarborBuildConfig inherits all six, so the YAML stays flat — a build +config written before this split parses identically — while the grammar is +readable one group at a time. + +The grouping also removes a duplicate list. The fields that only mean something +for a nested ``harbor run`` used to be enumerated twice: once as declarations and +once in a hand-maintained frozenset. Now they are exactly the members of +_HarborEvaluationFields, and _HARBOR_ONLY_FIELDS is derived from it. + +Rules that span groups stay on HarborBuildConfig itself, as named validators. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path +from typing import Literal + +from pydantic import Field, field_validator, model_validator + +from vero.evaluation import MetricSelector, ObjectiveSpec +from vero.harbor.build.specs import ( + AgentAccessSpec, + CommandBackendSpec, + InferenceGatewaySpec, + VerificationTargetSpec, + WandbSpec, + WorkspaceOverlaySpec, +) +from vero.models import StrictModel + + +def _require_non_empty(value: str, label: str) -> str: + if not value.strip(): + raise ValueError(f"{label} must not be empty") + return value + + +class _TaskIdentityFields(StrictModel): + """What is being optimized, and the images and case sets it is built from. + + Attributes: + name: Task name written into task.toml. + description: Human-readable task description. + agent_repo: Absolute path to the editable target. Copied twice, into the + immutable agent-baseline and the editable agent-seed. + harbor_requirement: Pinned harbor requirement for the task image. Needed + for either evaluation backend, since the outer optimizer is a Harbor + agent in both cases. + partitions: Partition name to the Harbor task names it holds. Emitted as + one cases/.jsonl per partition. + task_manifest: Optional path to an existing JSON task manifest. When + given, every task named in partitions must appear in it. + base_image_main: Base image for the main container. + base_image_sidecar: Base image for the sidecar container. + """ + + name: str + description: str = "" + agent_repo: str + harbor_requirement: str + partitions: dict[str, list[str]] + task_manifest: str | None = None + base_image_main: str = "ghcr.io/astral-sh/uv:python3.12-bookworm" + base_image_sidecar: str = "ghcr.io/astral-sh/uv:python3.12-bookworm" + + @field_validator("name", "agent_repo", "base_image_main", "base_image_sidecar") + @classmethod + def validate_identity(cls, value: str) -> str: + return _require_non_empty(value, "Harbor build identity") + + @field_validator("agent_repo") + @classmethod + def validate_agent_repo(cls, value: str) -> str: + if not Path(value).is_absolute() or not Path(value).is_dir(): + raise ValueError("agent_repo must be an existing absolute directory") + return value + + @field_validator("harbor_requirement") + @classmethod + def validate_pinned_harbor(cls, value: str) -> str: + exact = re.search( + r"(?:^|\s)harbor(?:\[[A-Za-z0-9_.-]+(?:,[A-Za-z0-9_.-]+)*\])?" + r"\s*==\s*[^*\s,;]+", + value, + ) + pinned_git = re.search(r"@[0-9a-f]{7,64}(?:#.*)?$", value) + if exact is None and pinned_git is None: + raise ValueError( + "harbor_requirement must pin an exact version or Git commit" + ) + return value + + @field_validator("task_manifest") + @classmethod + def validate_task_manifest_path(cls, value: str | None) -> str | None: + if value is None: + return None + if not value.strip() or not Path(value).is_file(): + raise ValueError("task_manifest must be an existing JSON file") + return value + + @field_validator("partitions") + @classmethod + def validate_partitions( + cls, + value: dict[str, list[str]], + ) -> dict[str, list[str]]: + if not value: + raise ValueError("at least one Harbor partition is required") + for partition, tasks in value.items(): + if re.fullmatch(r"[A-Za-z0-9_.-]+", partition) is None: + raise ValueError( + f"partition {partition!r} must use letters, digits, '.', '_', or '-'" + ) + if not tasks or any(not task.strip() for task in tasks): + raise ValueError(f"partition {partition!r} must contain task names") + if len(tasks) != len(set(tasks)): + raise ValueError(f"partition {partition!r} contains duplicate tasks") + return value + + +class _HarborEvaluationFields(StrictModel): + """Knobs for the nested ``harbor run`` that scores a candidate. + + These, and only these, are the fields that mean nothing to a command + evaluation backend. Membership of this class *is* the definition: + _HARBOR_ONLY_FIELDS below is derived from it, and HarborBuildConfig rejects a + command build that sets any of them rather than ignoring it in silence. + + Attributes: + task_source: Local path to the task definitions, or a registry reference + pinned as name@version. + agent_import_path: Import path of the target agent class Harbor loads. + model: Default model the target agent runs on. + environment_name: Harbor environment, e.g. "modal". + harbor_python_version: Python version for the task image. + default_index: Package index used for installs. + n_attempts: Attempts per case in the nested evaluation run. + max_retries: Per-case retries for the nested `harbor run`, forwarded as + --max-retries. Harbor already retries non-excluded exceptions with + backoff; this tunes how hard, so a transient upstream rate-limit + storm during a mandatory evaluation does not fail otherwise-good + candidates. + retry_wait_multiplier: Backoff multiplier. This and the two delays below + are forwarded in a --config JobConfig snippet, since harbor's CLI + exposes no backoff flags. + retry_min_wait_seconds: Minimum backoff delay. + retry_max_wait_seconds: Maximum backoff delay. + infrastructure_max_attempts: Attempts for infrastructure-classed + failures. + infrastructure_retry_delay_seconds: Delay between those attempts. + reward_key: Metric key the backend treats as the reward. A command + harness reports its own reward, so this is Harbor-only. + aggregate_attempts: Combine repeat attempts by "best" or "mean". + feedback_transcripts: Include target transcripts in the agent's feedback. + feedback_max_bytes: Byte cap per feedback transcript. + expose_attempt_detail: Report per-attempt detail, not just the aggregate. + extra_harbor_args: Extra flags for the evaluation sub-run. Rejected if + they override a flag the compiler controls. + task_agent_timeout_seconds: Wall clock declared for the target agent. + Grouped here rather than with the other timeouts because it bounds + the target, which only a Harbor backend runs. + task_environment: Extra environment for the evaluation sub-run, available + to the task's ${VAR} substitutions. Must not name secrets. + task_services_use_upstream: Give task-owned model services (user + simulators, graders) running inside the task containers the real + upstream via OPENAI_*, while the candidate keeps the metered, + allow-listed gateway on VERO_AGENT_INFERENCE_*. Needed for + benchmarks like tau3 whose environment makes its own model calls. + """ + + # Optional because a command evaluation_backend has neither; HarborBuildConfig + # requires both for a harbor backend. + task_source: str | None = None + agent_import_path: str | None = None + + model: str | None = None + environment_name: str = "modal" + harbor_python_version: str = "3.12" + default_index: str = "https://pypi.org/simple" + n_attempts: int = Field(default=1, ge=1) + max_retries: int = Field(default=2, ge=0) + retry_wait_multiplier: float = Field(default=2.0, ge=1.0) + retry_min_wait_seconds: float = Field(default=4.0, ge=0.0) + retry_max_wait_seconds: float = Field(default=60.0, ge=0.0) + infrastructure_max_attempts: int = Field(default=3, ge=1) + infrastructure_retry_delay_seconds: float = Field(default=5.0, ge=0) + reward_key: str | None = None + aggregate_attempts: Literal["best", "mean"] = "mean" + feedback_transcripts: bool = False + feedback_max_bytes: int = Field(default=3000, ge=0) + expose_attempt_detail: bool = False + extra_harbor_args: list[str] = Field(default_factory=list) + task_agent_timeout_seconds: float = Field(default=600.0, gt=0) + task_environment: dict[str, str] = Field(default_factory=dict) + task_services_use_upstream: bool = False + + @field_validator("environment_name", "harbor_python_version", "default_index") + @classmethod + def validate_identity(cls, value: str) -> str: + return _require_non_empty(value, "Harbor build identity") + + @field_validator("model", "reward_key", "agent_import_path", "task_source") + @classmethod + def validate_optional_identity(cls, value: str | None) -> str | None: + if value is not None: + _require_non_empty(value, "optional Harbor identity") + return value + + @field_validator("extra_harbor_args") + @classmethod + def validate_extra_harbor_args(cls, value: list[str]) -> list[str]: + controlled = { + "-a", + "-d", + "-e", + "-i", + "-m", + "-n", + "-p", + "--agent-import-path", + "--jobs-dir", + "--max-retries", + "--n-attempts", + } + conflicts = [ + argument for argument in value if argument.split("=", 1)[0] in controlled + ] + if conflicts: + raise ValueError( + "extra_harbor_args override controlled flags: " + ", ".join(conflicts) + ) + return value + + +# The fields a command build must not set, derived from the class above so the +# two cannot drift. Setting one is a mistake worth reporting: it would be +# silently ignored. +_HARBOR_ONLY_FIELDS = frozenset(_HarborEvaluationFields.model_fields) + + +class _SearchAndSelectionFields(StrictModel): + """What the optimizer may measure, what counts as better, and what ships. + + Attributes: + agent_access: One AgentAccessSpec per partition the optimizer may reach. + A partition with no entry is held out by the absence of a policy. + selection_partition: Partition search optimizes against, normally + validation. + targets: Trusted final scoring passes; see VerificationTargetSpec. + evaluation_set_name: Name given to the evaluation set. + objective: Metric and direction that define fitness. + reward_mode: "submit" scores the candidate the agent submits; + "auto_best" scores the best measured candidate. + baseline_floor: Never ship a candidate scoring below the baseline. + baseline_selection_score: Pin the baseline's selection score rather than + measuring it. + score_baseline: Whether to score the baseline at all. + rescore_top_k: How many leading candidates the trusted side re-scores. + rescore_attempts: Re-score repeats per candidate. + selection_coverage_threshold: Fraction of the selection set an agent + evaluation must cover to be eligible for auto_best ranking. Below + this it is too partial to trust for selection. + """ + + agent_access: list[AgentAccessSpec] + selection_partition: str + targets: list[VerificationTargetSpec] + evaluation_set_name: str = "harbor" + objective: ObjectiveSpec = Field( + default_factory=lambda: ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + ) + reward_mode: Literal["submit", "auto_best"] = "auto_best" + baseline_floor: bool = False + baseline_selection_score: float | None = None + score_baseline: bool = True + rescore_top_k: int = Field(default=3, ge=1) + rescore_attempts: int = Field(default=1, ge=1) + selection_coverage_threshold: float = Field(default=0.9, ge=0.0, le=1.0) + + @field_validator("evaluation_set_name") + @classmethod + def validate_identity(cls, value: str) -> str: + return _require_non_empty(value, "Harbor build identity") + + +class _EvaluationLimitFields(StrictModel): + """Wall clocks and concurrency for the work the sidecar runs. + + Attributes: + timeout_seconds: Wall clock for one evaluation. + case_timeout_seconds: Wall clock for one case. + max_concurrency: Cases evaluated concurrently. + error_rate_threshold: Case error fraction above which an evaluation is + abandoned. + build_timeout_seconds: Wall clock for building the task's images. + verifier_timeout_seconds: Wall clock for the trusted verifier. Falls back + to timeout_seconds. + evaluation_drain_timeout_seconds: Grace period for finalization to wait + on already-running agent evaluations before cancelling them. + Defaults to 600s. Deliberately independent of timeout_seconds, which + is sized to be unreachable: inheriting it would let one hung sub-run + stall the held-out score for hours. Expiry is safe — cancellation + persists terminal records and refunds budgets. + """ + + timeout_seconds: float = Field(default=1800.0, gt=0) + case_timeout_seconds: float = Field(default=180.0, gt=0) + max_concurrency: int = Field(default=8, ge=1) + error_rate_threshold: float | None = Field(default=0.1, gt=0, le=1) + build_timeout_seconds: int = Field(default=1800, ge=1) + verifier_timeout_seconds: int | None = Field(default=None, ge=1) + evaluation_drain_timeout_seconds: float | None = Field(default=None, gt=0) + + +class _TaskEnvironmentFields(StrictModel): + """Credentials, model access, and reporting for the containers. + + Attributes: + secrets: Environment variable names routed into the task. Their presence + on the build host is checked at compile time unless + VERO_SKIP_SECRET_CHECK is set. + inference_gateway: Gateway credential source and per-scope policies; see + InferenceGatewaySpec. Omit for no metered model access. + wandb: Trusted-side Weights & Biases reporting from the evaluation + sidecar. Requires WANDB_API_KEY in secrets, routed to the sidecar and + never to the agent. + agent_env: Environment injected into the optimizer agent's own shell + (setup, install, and run) as harbor --ae KEY=VALUE. Distinct from + extra_harbor_args, which only reaches the evaluation sub-run, and + task_environment, which is that sub-run's environment. Use it for + things like UV_TOOL_BIN_DIR, so `uv tool install` targets a writable + directory on a non-root sandbox. + harness_user: Unprivileged OS user that runs the untrusted candidate + harness in the sidecar, isolating it from trusted state. None + disables that isolation. + """ + + secrets: list[str] = Field(default_factory=list) + inference_gateway: InferenceGatewaySpec | None = None + wandb: WandbSpec | None = None + agent_env: dict[str, str] = Field(default_factory=dict) + harness_user: str | None = "harness" + + @field_validator("secrets") + @classmethod + def validate_secrets(cls, value: list[str]) -> list[str]: + if len(value) != len(set(value)): + raise ValueError("secret environment names must be unique") + for name in value: + if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name) is None: + raise ValueError(f"invalid secret environment name: {name!r}") + return value + + +class _AgentWorkspaceFields(StrictModel): + """What the optimizer finds in its workspace and is told about the task. + + Attributes: + read_only_paths: Candidate-relative paths the optimizer may read but not + edit. Must be safe relative paths. + workspace_overlays: Host files baked into the optimizer's workspace. + include_evals_skill: Bake vero's packaged evals skill into the workspace, + so any coding agent learns the CLI and the .evals layout. + instruct_multifidelity: Tell the optimizer it allocates its own case + budget and may evaluate subsets. Requires disclose_budget. + instruct_exhaust_budget: Tell the optimizer that unspent budget is + wasted. Requires disclose_budget. + disclose_budget: Master switch for budget disclosure, not enforcement. + When False the instruction renders no budget language and the + sidecar's status omits remaining budgets, so the agent cannot reason + about its budget at all. The two instruct_ flags above apply only + when this is True. + """ + + read_only_paths: list[str] = Field(default_factory=list) + workspace_overlays: list[WorkspaceOverlaySpec] = Field(default_factory=list) + include_evals_skill: bool = True + instruct_multifidelity: bool = True + instruct_exhaust_budget: bool = True + disclose_budget: bool = True + + @field_validator("read_only_paths") + @classmethod + def validate_read_only_paths(cls, value: list[str]) -> list[str]: + for item in value: + path = Path(item) + if ( + not item.strip() + or path.is_absolute() + or ".." in path.parts + or re.fullmatch(r"[A-Za-z0-9_./-]+", item) is None + ): + raise ValueError( + "read_only_paths must be safe relative candidate paths" + ) + return value + + +def _read_manifest(manifest_path: str) -> dict: + try: + manifest = json.loads(Path(manifest_path).read_text(encoding="utf-8")) + except json.JSONDecodeError as error: + raise ValueError("task_manifest must contain valid JSON") from error + if not isinstance(manifest, dict): + raise ValueError("task_manifest must be a JSON object") + return manifest + + +def _manifest_source_matches( + manifest: dict, + manifest_path: str, + task_source: str, +) -> bool: + """Whether a manifest's recorded task_source is the build's task_source.""" + recorded = manifest.get("task_source") + if recorded == task_source: + return True + # A vendored local source is recorded relative to the manifest, while the + # loader resolves the build's copy to an absolute path once the directory + # exists; compare resolved locations. + return ( + isinstance(recorded, str) + and (Path(manifest_path).parent / recorded).resolve() + == Path(task_source).resolve() + ) + + +def _manifest_task_names(manifest: dict) -> list[str]: + """The task names a manifest declares, rejecting a malformed tasks list.""" + tasks = manifest.get("tasks") + if not isinstance(tasks, list): + raise ValueError("task_manifest tasks must be a JSON array") + names: list[str] = [] + for item in tasks: + name = item.get("name") if isinstance(item, dict) else item + if not isinstance(name, str) or not name.strip(): + raise ValueError("task_manifest tasks must be names or objects with a name") + names.append(name) + if len(names) != len(set(names)): + raise ValueError("task_manifest contains duplicate task names") + return names + + +class HarborBuildConfig( + _TaskIdentityFields, + _HarborEvaluationFields, + _SearchAndSelectionFields, + _EvaluationLimitFields, + _TaskEnvironmentFields, + _AgentWorkspaceFields, +): + """Everything needed to emit an isolated Harbor optimization task. + + This is the whole grammar of a benchmark's build.yaml: load_harbor_build_config + parses and validates one, and compile_harbor_task lowers it into a task + directory. Unknown keys are rejected, so a typo fails the build instead of + quietly taking a default. + + Most fields come from the six groups this inherits, each documented on its + own class: _TaskIdentityFields, _HarborEvaluationFields, + _SearchAndSelectionFields, _EvaluationLimitFields, _TaskEnvironmentFields, + and _AgentWorkspaceFields. Declared here are only the choice of inner + evaluation backend and the rules that span groups — each of the latter is a + named validator below, so a rejected build points at the rule it broke. + + Attributes: + evaluation_backend: How a candidate is scored. "harbor" drives a target + agent with a nested `harbor run`; "command" runs a program instead, + for a target that is not an agent. The outer optimizer is a Harbor + agent either way. + command_backend: The scoring program, required when evaluation_backend is + "command" and rejected otherwise. + """ + + evaluation_backend: Literal["harbor", "command"] = "harbor" + command_backend: CommandBackendSpec | None = None + + @model_validator(mode="after") + def validate_harness_isolation(self) -> HarborBuildConfig: + if self.harness_user is not None and self.task_services_use_upstream: + raise ValueError( + "harness_user (harness isolation) is incompatible with " + "task_services_use_upstream: the raw upstream credential would " + "reach the isolated harness through its environment. Set " + "harness_user: null to build task-owned upstream services." + ) + return self + + @model_validator(mode="after") + def validate_task_source_is_pinned(self) -> HarborBuildConfig: + if self.task_source is not None and ( + not Path(self.task_source).exists() and "@" not in self.task_source + ): + raise ValueError("registry task_source must include an explicit version") + return self + + @model_validator(mode="after") + def validate_backend_coherence(self) -> HarborBuildConfig: + """Each backend gets its own fields and only its own.""" + if self.evaluation_backend == "command": + if self.command_backend is None: + raise ValueError( + "command evaluation_backend requires a command_backend" + ) + # Reported rather than ignored: these would silently do nothing. + ignored = sorted(_HARBOR_ONLY_FIELDS & self.model_fields_set) + if ignored: + raise ValueError( + "these fields only apply to a harbor evaluation_backend: " + + ", ".join(ignored) + ) + else: + if self.command_backend is not None: + raise ValueError("command_backend requires evaluation_backend: command") + missing = sorted( + name + for name in ("agent_import_path", "task_source") + if getattr(self, name) is None + ) + if missing: + raise ValueError( + "harbor evaluation_backend requires: " + ", ".join(missing) + ) + return self + + @model_validator(mode="after") + def validate_partition_references(self) -> HarborBuildConfig: + """Every partition named elsewhere must exist, and be reachable.""" + known = set(self.partitions) + if self.selection_partition not in known: + raise ValueError("selection_partition is not present in partitions") + access_names = [access.partition for access in self.agent_access] + if len(access_names) != len(set(access_names)): + raise ValueError("agent_access partitions must be unique") + unknown_access = sorted(set(access_names) - known) + if unknown_access: + raise ValueError( + f"agent_access references unknown partitions: {unknown_access}" + ) + if self.selection_partition not in access_names: + raise ValueError("selection_partition must be agent-evaluable") + if not self.targets: + raise ValueError("at least one verification target is required") + unknown_targets = sorted({target.partition for target in self.targets} - known) + if unknown_targets: + raise ValueError(f"targets reference unknown partitions: {unknown_targets}") + reward_keys = [target.reward_key for target in self.targets] + if len(reward_keys) != len(set(reward_keys)): + raise ValueError("target reward keys must be unique") + return self + + @model_validator(mode="after") + def validate_target_attempt_overrides(self) -> HarborBuildConfig: + """A per-target n_attempts/aggregate_attempts override is only honorable + on a held-out target: backends are per-partition, so overriding an + agent-evaluable partition (search or selection) would silently change + those runs too, and a command backend has no attempts to override.""" + overriding = [ + target + for target in self.targets + if target.n_attempts is not None or target.aggregate_attempts is not None + ] + if not overriding: + return self + if self.evaluation_backend == "command": + raise ValueError( + "per-target n_attempts/aggregate_attempts only apply to a harbor " + "evaluation_backend" + ) + shared = {access.partition for access in self.agent_access} | { + self.selection_partition + } + bad = sorted({t.partition for t in overriding} & shared) + if bad: + raise ValueError( + "per-target n_attempts/aggregate_attempts cannot override an " + "agent-evaluable partition (its backend is shared with search/" + f"selection): {', '.join(bad)}" + ) + return self + + @model_validator(mode="after") + def validate_task_manifest_agreement(self) -> HarborBuildConfig: + """The manifest and the partitions must describe the same task set.""" + if self.task_manifest is None or self.task_source is None: + return self + manifest = _read_manifest(self.task_manifest) + if not _manifest_source_matches(manifest, self.task_manifest, self.task_source): + raise ValueError( + "task_manifest task_source does not match build task_source" + ) + declared = set(_manifest_task_names(manifest)) + selected = { + task + for partition_tasks in self.partitions.values() + for task in partition_tasks + } + unknown = sorted(selected - declared) + if unknown: + raise ValueError( + "partitions reference tasks absent from task_manifest: " + + ", ".join(unknown) + ) + return self + + @model_validator(mode="after") + def validate_gateway_credentials_are_not_secrets(self) -> HarborBuildConfig: + """The upstream credential is the gateway's alone. + + Declaring it as a task secret would deliver it to the containers the + gateway exists to keep it away from. + """ + if self.inference_gateway is None: + return self + gateway_environment = {self.inference_gateway.upstream_api_key_env} + if self.inference_gateway.upstream_base_url_env is not None: + gateway_environment.add(self.inference_gateway.upstream_base_url_env) + overlap = sorted(set(self.secrets) & gateway_environment) + if overlap: + raise ValueError( + "inference gateway credentials must not also be sidecar secrets: " + + ", ".join(overlap) + ) + return self + + @model_validator(mode="after") + def validate_requested_models_are_allowed(self) -> HarborBuildConfig: + """Every model that will be requested must be allowed by its scope. + + Otherwise the mismatch surfaces as a gateway 403 at run time — and for a + verification target, only after search has already spent its budget. A + command backend requests no models, so there is nothing to reconcile. + """ + if self.inference_gateway is None or self.evaluation_backend != "harbor": + return self + evaluation_models = self.inference_gateway.evaluation.allowed_models + if self.model is not None and self.model not in evaluation_models: + raise ValueError( + f"model {self.model!r} is not in the inference gateway's " + f"evaluation allowed_models ({', '.join(evaluation_models)})" + ) + # Finalization runs the target agent too, so it must allow the model each + # target scores with: its own override, else the task model. + finalization_models = ( + self.inference_gateway.finalization or self.inference_gateway.evaluation + ).allowed_models + for target in self.targets: + scoring_model = target.model or self.model + if scoring_model is not None and scoring_model not in finalization_models: + raise ValueError( + f"target {target.partition!r} scores with model " + f"{scoring_model!r}, which is not in the inference gateway's " + f"finalization allowed_models " + f"({', '.join(finalization_models)})" + ) + return self diff --git a/vero/src/vero/harbor/build/loader.py b/vero/src/vero/harbor/build/loader.py new file mode 100644 index 00000000..20cf7e08 --- /dev/null +++ b/vero/src/vero/harbor/build/loader.py @@ -0,0 +1,155 @@ +"""Reading a build.yaml: parameter substitution, then path resolution. + +Two things happen before HarborBuildConfig ever sees the document. ``${NAME}`` +placeholders are resolved from --param and the environment, which is how one +checked-in benchmark config serves several run-time configurations. Then the +handful of fields that name host paths are resolved relative to the YAML file, +so a config can be written with paths relative to itself. +""" + +from __future__ import annotations + +import json +import os +import re +from pathlib import Path + +from vero.harbor.build.config import HarborBuildConfig + +_BUILD_PARAM = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::(-|\?)([^}]*))?\}") + + +def _substitute_build_param(text: str, context: dict[str, str]) -> str: + """Resolve ``${NAME}`` / ``${NAME:-default}`` / ``${NAME:?message}`` in one scalar.""" + + def replace(match: re.Match[str]) -> str: + name, operator, argument = match.group(1), match.group(2), match.group(3) + resolved = context.get(name) + if resolved: + return resolved + if operator == "-": + return argument or "" + if operator == "?": + raise ValueError( + f"required build parameter {name!r} is unset: " + f"{argument or 'no message provided'}" + ) + raise ValueError( + f"build parameter {name!r} is unset; pass --param {name}=VALUE " + "or set the environment variable" + ) + + return _BUILD_PARAM.sub(replace, text) + + +def _resolve_build_params(value: object, context: dict[str, str]) -> object: + """Recursively resolve ``${...}`` placeholders in string scalars of a YAML value.""" + if isinstance(value, str): + return _substitute_build_param(value, context) + if isinstance(value, dict): + return { + key: _resolve_build_params(item, context) for key, item in value.items() + } + if isinstance(value, list): + return [_resolve_build_params(item, context) for item in value] + return value + + +def _read_partition_files( + partition_files: object, + base: Path, +) -> dict[str, list[str]]: + """Expand the ``partition_files`` shorthand into inline partitions. + + A partition can hold hundreds of task names, so benchmarks keep them in JSON + files beside the config rather than inline in the YAML. + """ + if not isinstance(partition_files, dict) or not partition_files: + raise ValueError("partition_files must be a non-empty YAML object") + partitions: dict[str, list[str]] = {} + for partition, filename in partition_files.items(): + if not isinstance(partition, str) or not isinstance(filename, str): + raise ValueError("partition_files must map names to JSON files") + partition_path = Path(filename).expanduser() + if not partition_path.is_absolute(): + partition_path = base / partition_path + try: + tasks = json.loads(partition_path.read_text(encoding="utf-8")) + except json.JSONDecodeError as error: + raise ValueError( + f"partition file {partition_path} must contain valid JSON" + ) from error + if not isinstance(tasks, list) or any( + not isinstance(task, str) for task in tasks + ): + raise ValueError( + f"partition file {partition_path} must be a JSON array of task names" + ) + partitions[partition] = tasks + return partitions + + +def _resolve_local_paths(value: dict, base: Path) -> None: + """Rewrite the path-valued fields in place, relative to the config's directory.""" + agent_repo = value.get("agent_repo") + if isinstance(agent_repo, str) and not Path(agent_repo).is_absolute(): + value["agent_repo"] = str((base / agent_repo).resolve()) + task_source = value.get("task_source") + if isinstance(task_source, str): + local_source = base / task_source + # Left alone when no such directory exists: it is a registry reference, + # not a path. + if not Path(task_source).is_absolute() and local_source.exists(): + value["task_source"] = str(local_source.resolve()) + task_manifest = value.get("task_manifest") + if isinstance(task_manifest, str) and not Path(task_manifest).is_absolute(): + value["task_manifest"] = str((base / task_manifest).resolve()) + command_backend = value.get("command_backend") + if isinstance(command_backend, dict): + harness_source = command_backend.get("harness_source") + if isinstance(harness_source, str) and not Path(harness_source).is_absolute(): + command_backend["harness_source"] = str((base / harness_source).resolve()) + overlays = value.get("workspace_overlays") + if isinstance(overlays, list): + for entry in overlays: + source = entry.get("source") if isinstance(entry, dict) else None + if isinstance(source, str) and not Path(source).is_absolute(): + entry["source"] = str((base / source).resolve()) + + +def load_harbor_build_config( + path: Path | str, + *, + params: dict[str, str] | None = None, +) -> HarborBuildConfig: + """Load YAML and resolve local paths relative to the configuration file. + + ``${NAME}`` placeholders in the YAML are substituted at load time from + ``params`` (explicit, e.g. ``--param NAME=VALUE``) layered over the process + environment, so run-time knobs (optimizer model, inner sandbox provider, + concurrency, ...) can be varied without rebuilding the task. Use + ``${NAME:-default}`` for a fallback and ``${NAME:?message}`` to require a + value. Fields left un-templated stay fixed (the reproducible measurement + substrate). + """ + try: + import yaml + except ImportError as error: + raise RuntimeError( + "install scale-vero[harbor] to load Harbor builds" + ) from error + + config_path = Path(path).expanduser().resolve() + value = yaml.safe_load(config_path.read_text(encoding="utf-8")) + if not isinstance(value, dict): + raise ValueError("Harbor build config must be a YAML object") + context = {**os.environ, **(params or {})} + value = _resolve_build_params(value, context) + base = config_path.parent + partition_files = value.pop("partition_files", None) + if partition_files is not None: + if "partitions" in value: + raise ValueError("use either partitions or partition_files, not both") + value["partitions"] = _read_partition_files(partition_files, base) + _resolve_local_paths(value, base) + return HarborBuildConfig.model_validate(value) diff --git a/vero/src/vero/harbor/build/specs.py b/vero/src/vero/harbor/build/specs.py new file mode 100644 index 00000000..1d32e420 --- /dev/null +++ b/vero/src/vero/harbor/build/specs.py @@ -0,0 +1,323 @@ +"""Leaf models a build.yaml composes. + +Each spec here is self-contained: it validates its own fields and knows nothing +about the build that holds it. HarborBuildConfig in config.py assembles them and +adds the rules that span more than one. +""" + +from __future__ import annotations + +import math +import re +from pathlib import Path +from typing import Literal + +from pydantic import Field, JsonValue, field_validator + +from vero.evaluation import DisclosureLevel, EvaluationAccessPolicy +from vero.models import StrictModel + + +class AgentAccessSpec(StrictModel): + """The optimizer agent's access to one named evaluation partition. + + A build declares one spec per partition; to_access_policy translates it into + the runtime EvaluationAccessPolicy the sidecar enforces. This governs what + the optimizer may see and spend while searching, as opposed to how the + trusted side finally scores a candidate (see VerificationTargetSpec). + + Attributes: + partition: Partition this access applies to, e.g. "validation". + disclosure: How much of a result the agent sees — aggregate score vs. + per-case detail. + expose_case_resources: Whether each case's input files are materialized + into the agent's workspace. + min_aggregate_cases: Smallest number of cases an aggregate score may + cover, so the agent cannot request a subset small enough to reveal an + individual case's result. Only consulted when disclosure is AGGREGATE. + total_runs: Optional cap on evaluation runs against this partition. + total_cases: Optional cap on cases scored against this partition. + """ + + partition: str + disclosure: DisclosureLevel = DisclosureLevel.AGGREGATE + expose_case_resources: bool = False + min_aggregate_cases: int = Field(default=5, ge=1) + total_runs: int | None = Field(default=None, ge=0) + total_cases: int | None = Field(default=None, ge=0) + + @field_validator("partition") + @classmethod + def validate_partition(cls, value: str) -> str: + if not value.strip(): + raise ValueError("agent access partition must not be empty") + return value + + def to_access_policy(self) -> EvaluationAccessPolicy: + """The single typed translation from build spec to runtime policy.""" + return EvaluationAccessPolicy( + disclosure=self.disclosure, + expose_case_resources=self.expose_case_resources, + min_aggregate_cases=self.min_aggregate_cases, + ) + + +class VerificationTargetSpec(StrictModel): + """One trusted final scoring pass the verifier runs on a chosen candidate. + + Distinct from the evaluations the optimizer runs during search: after search + picks a best candidate on the selection partition, the trusted verifier + scores it against these targets (normally the held-out test partition) to + produce the final reward. A build declares one spec per pass. + + Attributes: + partition: Partition to score on, e.g. "test". + reward_key: Metric key in the report that is the reward. + model: Optional model override for this pass; passed through as + harbor_model_override so final scoring can differ from search. + parameters: Extra backend parameters for the pass. + failure_value: Score assigned when the pass fails. + baseline_reward: Pin the seed's reward on this target to skip scoring + the immutable baseline every run. + max_attempts: Number of scoring attempts before accepting failure. + n_attempts: Optional per-target override of the global attempts-per-case; + None inherits the global. Set >1 to score each held-out case several + times and combine them (with aggregate_attempts), e.g. 3 to average a + noisy final eval over 3 scorings. Only meaningful on a harbor target + whose partition is not agent-evaluable (see the config validator). + aggregate_attempts: Optional per-target override of how repeat attempts + combine ("best"/"mean"); None inherits the global ("mean"). + """ + + partition: str + reward_key: str = "reward" + model: str | None = None + parameters: dict[str, JsonValue] = Field(default_factory=dict) + failure_value: float = 0.0 + baseline_reward: float | None = None + max_attempts: int = Field(default=1, ge=1) + n_attempts: int | None = Field(default=None, ge=1) + aggregate_attempts: Literal["best", "mean"] | None = None + + @field_validator("partition", "reward_key") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("verification target identity must not be empty") + return value + + @field_validator("failure_value") + @classmethod + def validate_failure_value(cls, value: float) -> float: + if not math.isfinite(value): + raise ValueError("target failure_value must be finite") + return value + + +class InferenceBudgetSpec(StrictModel): + """Routing policy and optional limits for one inference-gateway scope. + + The compiler lowers each spec into an InferenceScopeConfig, adding the + SHA-256 digest of a freshly minted per-scope token. The gateway then checks + the presented token, the requested model, and the running budget on every + proxied request; limits are held server-side in a ledger written through to + disk, so they survive a gateway restart. + + Attributes: + allowed_models: Models this scope may request. A request naming anything + else is refused with 403 model_denied. + max_requests: Cap on proxied requests; unlimited when omitted. + max_tokens: Cap on cumulative tokens; unlimited when omitted. Checked + before a request starts, so a single request can overshoot it. + max_concurrency: Requests this scope may have in flight at once. + """ + + allowed_models: list[str] + max_requests: int | None = Field(default=None, ge=1) + max_tokens: int | None = Field(default=None, ge=1) + max_concurrency: int = Field(default=8, ge=1) + + @field_validator("allowed_models") + @classmethod + def validate_models(cls, value: list[str]) -> list[str]: + if not value or any(not model.strip() for model in value): + raise ValueError("allowed_models must contain non-empty model names") + if len(value) != len(set(value)): + raise ValueError("allowed_models must be unique") + return value + + +class InferenceGatewaySpec(StrictModel): + """Credential source and independent producer/evaluator policies. + + All model access in a compiled task goes through the gateway, which holds + the one upstream credential and hands each consumer a separate token. The + optimizer never sees the evaluation or finalization tokens, so it cannot + spend from those pools. + + The per-scope allow-lists keep the target on its fixed evaluation model, but + only against a non-adversarial optimizer: the scopes share a gateway host and + are split by URL path, so an optimizer that smuggles its own token into the + candidate can reach the producer scope from the eval sandbox. See the proxy + handler in vero.gateway.inference for the full note. + + Attributes: + upstream_api_key_env: Environment variable on the build host holding the + real upstream API key. + upstream_base_url_env: Environment variable holding the upstream base + URL; None to always use default_upstream_base_url. + default_upstream_base_url: Upstream used when no base-URL variable is set. + producer: Policy for the optimizer's own model calls. + evaluation: Policy for the target agent under evaluation. + finalization: Reserved policy for trusted finalization (admin re-score + and verification targets). Defaults to a copy of evaluation, giving + the mandatory re-score a pool that search evaluations cannot starve. + log_requests: Capture every proxied or denied request as JSONL on the + gateway state volume. Never agent-visible; the trusted sidecar + mirrors it to Weights & Biases and the session archive. + request_log_body_bytes: Head+tail budget per logged body, in bytes. + request_log_attribution: Experimental. Stamp log records with + provider-agnostic conversation threads for post-hoc per-trial + accounting. + """ + + upstream_api_key_env: str = "OPENAI_API_KEY" + upstream_base_url_env: str | None = "OPENAI_BASE_URL" + default_upstream_base_url: str = "https://api.openai.com/v1" + producer: InferenceBudgetSpec + evaluation: InferenceBudgetSpec + finalization: InferenceBudgetSpec | None = None + log_requests: bool = True + request_log_body_bytes: int = Field(default=16384, ge=0) + request_log_attribution: bool = False + + @field_validator("upstream_api_key_env", "upstream_base_url_env") + @classmethod + def validate_environment_name(cls, value: str | None) -> str | None: + if value is not None and re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", value) is None: + raise ValueError("gateway environment names must be valid identifiers") + return value + + @field_validator("default_upstream_base_url") + @classmethod + def validate_upstream_url(cls, value: str) -> str: + if not value.startswith(("http://", "https://")): + raise ValueError("default_upstream_base_url must be HTTP(S)") + return value.rstrip("/") + + +class WorkspaceOverlaySpec(StrictModel): + """A host file or directory to copy into the compiled task's agent workspace. + + General-purpose filesystem injection: bake anything (agent definitions, + skills, config, data) into the optimizer's workspace at build time. + + Attributes: + source: Host path, resolved relative to the build YAML. Must exist. A + directory contributes its contents; a file lands as dest/. + dest: Where the contents land, relative to the workspace root. Must be a + safe relative path; "." is the root itself. + """ + + source: str + dest: str = "." + + @field_validator("source") + @classmethod + def validate_source(cls, value: str) -> str: + if not value.strip() or not Path(value).exists(): + raise ValueError("overlay source must be an existing file or directory") + return value + + @field_validator("dest") + @classmethod + def validate_dest(cls, value: str) -> str: + candidate = value.strip() + path = Path(candidate) + if candidate == ".": + return candidate + if ( + not candidate + or path.is_absolute() + or ".." in path.parts + or re.fullmatch(r"[A-Za-z0-9_./-]+", candidate) is None + ): + raise ValueError( + "overlay dest must be a safe relative path within the workspace" + ) + return candidate + + +class WandbSpec(StrictModel): + """Weights & Biases reporting settings, emitted into the sidecar's serve.json. + + Reporting runs on the trusted side only. The field names match the sidecar's + SidecarWandbConfig, which consumes them verbatim. + + Attributes: + project: Weights & Biases project to report into. + entity: Owning user or team; the account default when omitted. + name: Display name for the run. + group: Group to file the run under. + tags: Tags applied to the run. + mode: Client mode, e.g. "online" or "offline". + notes: Free-text note attached to the run. + run_id: Resume into this existing run instead of creating one. + log_traces: Upload each evaluation's trace artifacts. Off by default + because they are large. + """ + + project: str + entity: str | None = None + name: str | None = None + group: str | None = None + tags: list[str] = Field(default_factory=list) + mode: str | None = None + notes: str | None = None + run_id: str | None = None + log_traces: bool = False + + +class CommandBackendSpec(StrictModel): + """Inner evaluation that scores a candidate by running a program. + + Use this when the outer loop is still a Harbor coding agent but the target is + not an agent — a solver, an index build, a data pipeline — so there is nothing + to drive with a nested `harbor run`. The compiler copies harness_source into + the sidecar and points the backend's harness_root at it. + + Attributes: + harness_source: Host directory holding the scoring program, resolved + relative to the build YAML. Must exist. + command: Argument vector to run. Supports the placeholders {harness}, + {workspace}, {request}, {report}, and {artifacts}. + working_directory: Directory to run the command from. + environment: Environment variables set for the command. + passthrough_environment: Names forwarded from the sidecar's environment. + CommandBackend runs the harness with PATH=os.defpath, so a harness + that needs the build image's interpreter must list PATH here. + staged_inputs: Extra files staged into the command's workspace. + agent_context_inputs: Per-case files published into the agent context. + """ + + harness_source: str + command: list[str] + working_directory: str = "." + environment: dict[str, str] = Field(default_factory=dict) + passthrough_environment: list[str] = Field(default_factory=list) + staged_inputs: dict[str, str] = Field(default_factory=dict) + agent_context_inputs: dict[str, list[str]] = Field(default_factory=dict) + + @field_validator("harness_source") + @classmethod + def validate_harness_source(cls, value: str) -> str: + if not value.strip() or not Path(value).is_dir(): + raise ValueError("harness_source must be an existing directory") + return value + + @field_validator("command") + @classmethod + def validate_command(cls, value: list[str]) -> list[str]: + if not value or any(not part.strip() for part in value): + raise ValueError("command must contain non-empty arguments") + return value diff --git a/vero/src/vero/harbor/build/templates/Dockerfile.gateway.j2 b/vero/src/vero/harbor/build/templates/Dockerfile.gateway.j2 new file mode 100644 index 00000000..9ee42e89 --- /dev/null +++ b/vero/src/vero/harbor/build/templates/Dockerfile.gateway.j2 @@ -0,0 +1,17 @@ +# Trusted inference gateway: upstream credential and durable usage ledger. +FROM {{ base_image_sidecar }} + +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +{% if use_local_vero %} +COPY vero {{ layout.vero }} +RUN uv pip install --system "{{ layout.vero }}[harbor]" +{% else %} +RUN uv pip install --system "{{ vero_requirement }}" +{% endif %} + +COPY gateway/config.json {{ layout.inference_config }} + +WORKDIR /opt diff --git a/vero/src/vero/harbor/build/templates/Dockerfile.main.j2 b/vero/src/vero/harbor/build/templates/Dockerfile.main.j2 new file mode 100644 index 00000000..c3e97f17 --- /dev/null +++ b/vero/src/vero/harbor/build/templates/Dockerfile.main.j2 @@ -0,0 +1,22 @@ +# Optimizer workbench: target repository plus the agent-facing VeRO CLI. +FROM {{ base_image_main }} + +RUN apt-get update \ + && apt-get install -y --no-install-recommends git ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +{% if use_local_vero %} +COPY vero {{ layout.vero }} +RUN uv pip install --system "{{ layout.vero }}[harbor]" +{% else %} +RUN uv pip install --system "{{ vero_requirement }}" +{% endif %} + +COPY agent-seed {{ layout.seed_repo }} +{% if overlay_present %} +COPY overlay {{ layout.overlay }} +{% endif %} +COPY main/seed.sh {{ layout.seed_script }} +RUN chmod +x {{ layout.seed_script }} && useradd -m -u 1001 agent + +WORKDIR {{ layout.target_repo }} diff --git a/vero/src/vero/harbor/build/templates/Dockerfile.sidecar.j2 b/vero/src/vero/harbor/build/templates/Dockerfile.sidecar.j2 new file mode 100644 index 00000000..0b30e2f5 --- /dev/null +++ b/vero/src/vero/harbor/build/templates/Dockerfile.sidecar.j2 @@ -0,0 +1,53 @@ +# Trusted evaluation sidecar: baselines, cases, task source, ledger, and secrets. +FROM {{ base_image_sidecar }} + +RUN apt-get update \ + && apt-get install -y --no-install-recommends git ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +{% if use_local_vero %} +COPY vero {{ layout.vero }} +RUN uv pip install --system "{{ layout.vero }}[{{ sidecar_extras }}]" +{% else %} +RUN uv pip install --system "{{ vero_requirement }}" +{% endif %} +RUN uv pip install --system "{{ harbor_requirement }}" + +COPY agent-baseline {{ layout.trusted_repo }} +COPY sidecar/cases {{ layout.cases }} +COPY sidecar/serve.json {{ layout.serve_config }} +{% if local_task_source %} +COPY sidecar/task-source {{ layout.task_source }} +{% endif %} +{% if command_harness %} +COPY sidecar/harness {{ layout.harness }} +{% endif %} + +RUN cd {{ layout.trusted_repo }} && uv sync 2>/dev/null || true +RUN git config --system --add safe.directory {{ layout.trusted_repo }} \ + && git config --system --add safe.directory {{ layout.target_repo }} \ + && git config --system --add safe.directory {{ layout.target_git }} + +# Unprivileged user for executing the untrusted candidate harness. The sidecar +# still starts as root (the trusted control plane); only the `harbor run` +# subprocess drops to this user, so it cannot read the held-out records, the +# budget ledger, other candidates, or the admin token. `-m` gives it a home for +# the uv cache and git; useradd creates a same-named primary group. +RUN useradd -m -u 1002 harness + +# Pre-warm the harness user's uv cache from the build (root) cache. `uv run` as +# the unprivileged harness must resolve the candidate package (and harbor) from +# cache: a fresh empty cache would need network at eval time and fail to import +# it ("No module named ''"). Copy so it is owned by and writable to it. +RUN mkdir -p /home/harness/.cache \ + && cp -a "$(uv cache dir)" /home/harness/.cache/uv \ + && chown -R harness:harness /home/harness/.cache + +# Lock the trusted, root-only files the harness must never read: the case lists +# reveal held-out (validation/test) task membership, and the deployment config +# carries the reserved finalization inference token. (Local task packages under +# {{ layout.task_source }}, which harbor must read to grade host-side, remain readable +# and are addressed by the deeper in-container isolation work.) +RUN chmod 700 {{ layout.cases }} && chmod 600 {{ layout.serve_config }} + +WORKDIR /opt diff --git a/vero/src/vero/harbor/build/templates/docker-compose.yaml.j2 b/vero/src/vero/harbor/build/templates/docker-compose.yaml.j2 new file mode 100644 index 00000000..94d940dc --- /dev/null +++ b/vero/src/vero/harbor/build/templates/docker-compose.yaml.j2 @@ -0,0 +1,108 @@ +# Harbor merges this after its generated main-service configuration. +services: + main: + command: ["{{ layout.seed_script }}"] + environment: + {{ layout.eval_url_env }}: "{{ layout.sidecar_url }}" +{% for secret in scrubbed_main_environment %} + {{ secret }}: "" +{% endfor %} +{% if inference_gateway %} + {{ layout.producer_api_key_env }}: "{{ producer_inference_token }}" + {{ layout.producer_base_url_env }}: "{{ producer_base_url }}" +{% elif layout.producer_api_key_env in secrets %} + {{ layout.producer_api_key_env }}: "" +{% if layout.producer_base_url_env in secrets %} + {{ layout.producer_base_url_env }}: "" +{% endif %} +{% endif %} + volumes: + - agent_repo:{{ layout.target_repo }} + - agent_context:{{ layout.target_evals }}:ro + - token_state:{{ layout.token_dir }}:ro + depends_on: + {{ layout.sidecar_host }}: + condition: service_healthy +{% if inference_gateway %} + {{ layout.gateway_host }}: + condition: service_healthy +{% endif %} + + {{ layout.sidecar_host }}: + build: + context: . + dockerfile: sidecar/Dockerfile + command: + - "vero" + - "harbor" + - "serve" + - "--factory" + - "{{ sidecar_factory }}" + - "--config" + - "{{ layout.serve_config }}" + - "--admin-token" + - "{{ layout.token_path }}" + environment: +{% if sidecar_secrets %} +{% for secret in sidecar_secrets %} + {{ secret }}: "${{ '{' }}{{ secret }}:?{{ secret }} must be set for the eval sidecar{{ '}' }}" +{% endfor %} +{% else %} + LANG: "C.UTF-8" +{% endif %} +{% if task_services_use_upstream %} +{# Trusted sidecar receives the real upstream so it can hand task-owned eval + services (LLM user-sims/graders) a reachable provider. The optimizer/main keeps + these scrubbed to "" so it never sees the raw upstream credential. #} +{% for secret in gateway_environment %} + {{ secret }}: "${{ '{' }}{{ secret }}:?{{ secret }} must be set for the eval sidecar{{ '}' }}" +{% endfor %} +{% endif %} + volumes: + - agent_repo:{{ layout.target_repo }}:ro + - agent_context:{{ layout.agent_volume }} + - admin_state:{{ layout.admin_volume }} + - token_state:{{ layout.token_dir }} +{% if inference_gateway %} + - inference_state:{{ layout.inference_dir }}:ro +{% endif %} + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:{{ layout.sidecar_port }}/health')"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s + +{% if inference_gateway %} + {{ layout.gateway_host }}: + build: + context: . + dockerfile: gateway/Dockerfile + command: + - "vero" + - "harbor" + - "{{ layout.gateway_host }}" + - "--config" + - "{{ layout.inference_config }}" + environment: +{% for secret in gateway_environment %} + {{ secret }}: "${{ '{' }}{{ secret }}:?{{ secret }} must be set for the inference gateway{{ '}' }}" +{% endfor %} + volumes: + - inference_state:{{ layout.inference_dir }} + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:{{ layout.gateway_port }}/health')"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s +{% endif %} + +volumes: + agent_repo: + agent_context: + admin_state: + token_state: +{% if inference_gateway %} + inference_state: +{% endif %} diff --git a/vero/src/vero/harbor/build/templates/instruction.md.j2 b/vero/src/vero/harbor/build/templates/instruction.md.j2 new file mode 100644 index 00000000..f6a9f678 --- /dev/null +++ b/vero/src/vero/harbor/build/templates/instruction.md.j2 @@ -0,0 +1,130 @@ +# Optimize the program + +Improve the program in `{{ layout.target_repo }}` so it scores as highly as possible on the +hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, +{% if disclose_budget %}budget, {% endif %}and final candidate selection. + +{% if description %} +## Objective + +{{ description }} + +{% endif %} +## Workflow + +1. Inspect and edit `{{ layout.target_repo }}`. +2. Commit each candidate with Git. +3. Evaluate the current commit on the selection set. By default `evals run` + **blocks** until scoring finishes and prints the result — one call, no + polling, no job to track: + + ```bash + evals run \ + --backend {{ selection_backend }} \ + --evaluation-set {{ evaluation_set_name }} \ + --partition {{ selection_partition }} + ``` + + An evaluation can take many minutes — that is expected; let the call block + and read the result it returns. Iterate cheaply on a subset first + (`--start 0 --stop N`, or repeated `--case-id ID`). + {% if seed_supported %} + Pass `--seed N` to reproduce a noisy comparison exactly. + {% else %} + To replicate a noisy comparison, re-run the identical selection; this backend + fixes its own sampling and rejects `--seed`. + {% endif %} + Run `evals run --help` for every option. + + **Every result is persisted, so you never need to preserve the printed + output.** The full record — overall metrics, every per-case score, and the + trial artifacts — is written under `.evals/results/` before the command + returns, and stays there for the rest of the run. Piping a result through + `head`/`tail` therefore costs you nothing: recover any of it later with + `evals list` (one row per past evaluation), `evals show ID`, `evals cases ID`, + and `evals diff OLD NEW`. Do not re-run an evaluation to see a number you + truncated — look it up. + + Add `--detach` **only** to run several evaluations at once: it returns a + `job_id` immediately instead of blocking. Then `evals wait JOB_ID` blocks + until that job finishes and prints its result (or poll `evals status JOB_ID`, + which also shows elapsed time). + + **Run every `evals` call in the foreground.** You are a single-shot headless + run: nothing exists to deliver a notification or wake you later. If you put a + long call in a background task, schedule a wake-up, or say you will "report + back when it finishes", the run simply ends there and whatever you have not + submitted is lost. A foreground call that blocks for half an hour is working + correctly — let it block. To wait on two jobs, `evals wait` the first, then + the second. Do not claim an improvement before its comparison baseline has + been scored on the same cases. + +4. Use `evals status` to inspect evaluation jobs and allowed evaluation + sets{% if disclose_budget %}, and to see remaining budgets{% endif %}. + Navigate finished results with `evals list`, `evals diff`, `evals cases`, + and `evals trace` (see `evals --help` and `skills/evals/SKILL.md`). +{% if submit_enabled %} +5. Before finishing, commit and nominate your **best** candidate with + `evals submit` — this is the commit that ships. Measure the baseline and + submit only a candidate you have confirmed beats it on the same cases. If you + never submit, VeRO falls back to auto-best over validation and then your last + commit, so an unvetted or half-finished edit can ship by default — submit + deliberately. +{% else %} +5. VeRO automatically pools your measurements, re-scores a shortlist with the + trusted evaluator, and selects the best candidate. The unmodified baseline is + used as a floor, so a regression is not shipped. +{% endif %} + +{% if multifidelity and disclose_budget %} +## Allocate the evaluation budget + +You decide how to allocate the finite case budget. Use `--start 0 --stop N` or +repeated `--case-id ID` flags for subsets, or omit them to evaluate the complete +set. A request may consume the entire remaining case budget. VeRO does not +reserve cases for later runs, so keep enough only if your strategy requires a +final confirmation. +{% if minimum_subset_cases > 1 %} +Aggregate-only subsets must include at least {{ minimum_subset_cases }} cases; +this prevents singleton aggregates from exposing individual hidden results. +{% else %} +You may use arbitrary subsets, including a single case; validation feedback is +optimization data, while the final evaluation remains hidden. +{% endif %} + +{% endif %} +{% if exposed_partitions %} +## Inspect cases and traces + +Complete task resources for the following authorized partitions are mounted +read-only under `.evals/tasks/`: {{ exposed_partitions | join(", ") }}. Start at +`.evals/tasks/index.json`; each listed partition has its own task index, +attachments, instructions, and task files. + +After a full-disclosure evaluation, `.evals/results/` contains one file per +case plus the complete Harbor trial records: exact failure results, exception +tracebacks, trial logs, and target-agent artifacts. Use these files to inspect +successful as well as failed trajectories without putting long traces into the +CLI response. +{% endif %} +## Rules + +- Only the sidecar evaluates candidates. Only explicitly authorized case + resources are mounted; validation and final targets remain outside this + container. +{% if disclose_budget %} +- Evaluation budgets are finite and are charged by backend and evaluation set. +{% endif %} +- Scores may be noisy; compare candidates on the same selection whenever possible. +- Reusing the same cases removes case-sampling differences, but target-model + randomness remains. Replicate important comparisons before assigning causality. +- Each case has a wall-clock limit: a case that runs past it is stopped and + scores the failure value. A change that is more thorough but slower can + therefore score *worse* by pushing cases over the limit — treat per-case + latency as a real constraint, not only accuracy. +{% if exhaust_budget and disclose_budget %} +- Unspent evaluation budget is wasted. Use remaining calls to re-measure the + strongest candidate or try another plausible change. +{% endif %} +- Paths marked read-only are guardrails. The authoritative evaluator lives in the + sidecar, outside the candidate repository. diff --git a/vero/src/vero/harbor/build/templates/seed.sh.j2 b/vero/src/vero/harbor/build/templates/seed.sh.j2 new file mode 100644 index 00000000..121b3c63 --- /dev/null +++ b/vero/src/vero/harbor/build/templates/seed.sh.j2 @@ -0,0 +1,41 @@ +#!/bin/sh +set -eu + +if [ ! -d {{ layout.target_git }} ]; then + cp -a {{ layout.seed_repo }}/. {{ layout.target_repo }}/ +fi +{% if overlay_present %} +# Inject baked-in workspace overlay (agent definitions, skills, config, ...). +cp -a {{ layout.overlay }}/. {{ layout.target_repo }}/ +{% endif %} + +find {{ layout.target_repo }} -path {{ layout.target_evals }} -prune -o -exec chown agent:agent {} + +git config --system --add safe.directory {{ layout.target_repo }} +printf '%s\n' '/.evals/' >> {{ layout.target_git_exclude }} +{% for path in overlay_excludes %} +printf '%s\n' '/{{ path }}/' >> {{ layout.target_git_exclude }} +{% endfor %} +{% if inference_gateway %} +# Codex otherwise treats the built-in OpenAI provider as WebSocket-capable. +# The VeRO gateway intentionally supports the metered HTTP Responses surface. +mkdir -p /tmp/codex-home +cat > /tmp/codex-home/config.toml <<'TOML' +model_provider = "vero_gateway" + +[model_providers.vero_gateway] +name = "VeRO inference gateway" +base_url = "{{ producer_base_url }}" +env_key = "{{ layout.producer_api_key_env }}" +wire_api = "responses" +supports_websockets = false +TOML +chown -R agent:agent /tmp/codex-home +{% endif %} +{% for path in read_only_paths %} +if [ -e "{{ layout.target_repo }}/{{ path }}" ]; then + chown -R root:root "{{ layout.target_repo }}/{{ path }}" + chmod -R a-w "{{ layout.target_repo }}/{{ path }}" +fi +{% endfor %} + +exec sleep infinity diff --git a/vero/src/vero/harbor/build/templates/solve.sh.j2 b/vero/src/vero/harbor/build/templates/solve.sh.j2 new file mode 100644 index 00000000..b2de3993 --- /dev/null +++ b/vero/src/vero/harbor/build/templates/solve.sh.j2 @@ -0,0 +1,17 @@ +#!/bin/sh +set -eu + +cd {{ layout.target_repo }} +git config user.name optimizer +git config user.email optimizer@example.test +echo "# optimizer candidate" >> README.md 2>/dev/null || echo "candidate" > NOTES.md +git add -A +git commit -m "optimizer candidate" +evals run \ + --backend {{ selection_backend }} \ + --evaluation-set {{ evaluation_set_name }} \ + --partition {{ selection_partition }} +{% if submit_enabled %} +evals submit +{% endif %} +evals status diff --git a/vero/src/vero/harbor/build/templates/task.toml.j2 b/vero/src/vero/harbor/build/templates/task.toml.j2 new file mode 100644 index 00000000..c9d12f3e --- /dev/null +++ b/vero/src/vero/harbor/build/templates/task.toml.j2 @@ -0,0 +1,22 @@ +schema_version = "1.3" + +[task] +name = {{ name_toml }} +description = {{ description_toml }} + +[agent] +user = "agent" + +[verifier] +environment_mode = "shared" +timeout_sec = {{ verifier_timeout }} + +[environment] +build_timeout_sec = {{ build_timeout }} +{% if secrets %} + +[environment.env] +{% for secret in secrets %} +{{ secret }} = "${{ '{' }}{{ secret }}{{ '}' }}" +{% endfor %} +{% endif %} diff --git a/vero/src/vero/harbor/build/templates/test.sh.j2 b/vero/src/vero/harbor/build/templates/test.sh.j2 new file mode 100644 index 00000000..90fc05d4 --- /dev/null +++ b/vero/src/vero/harbor/build/templates/test.sh.j2 @@ -0,0 +1,10 @@ +#!/bin/sh +set -eu + +mkdir -p /logs/verifier +vero harbor finalize \ + --token-file {{ layout.token_path }} \ + --output /logs/verifier/reward.json +vero harbor export-session \ + --token-file {{ layout.token_path }} +cat /logs/verifier/reward.json diff --git a/vero/src/vero/harbor/cli.py b/vero/src/vero/harbor/cli.py new file mode 100644 index 00000000..abbece2c --- /dev/null +++ b/vero/src/vero/harbor/cli.py @@ -0,0 +1,987 @@ +"""CLI clients and server entry point for Harbor sidecar deployments.""" + +from __future__ import annotations + +import asyncio +import json +import os +import re +import shlex +import shutil +import subprocess +import sys +import tempfile +import urllib.error +import urllib.request +from pathlib import Path + +import click + +from vero.evaluation import ( + CaseIds, + CaseRange, + EvaluationLimits, + EvaluationSet, + RetryPolicy, +) +from vero.layout import LAYOUT +from vero.sidecar.auth import read_admin_token +from vero.sidecar.session import ( + create_harbor_session_archive, + extract_harbor_session_archive, + file_sha256, +) +from vero.sidecar.sidecar import SidecarEvaluationRequest + + +def _base_url() -> str: + value = os.environ.get(LAYOUT.eval_url_env) + if not value: + raise click.ClickException(f"{LAYOUT.eval_url_env} is not set") + return value.rstrip("/") + + +def _request( + method: str, + path: str, + *, + payload: dict | None = None, + headers: dict[str, str] | None = None, +): + data = None if payload is None else json.dumps(payload).encode("utf-8") + request = urllib.request.Request( + f"{_base_url()}{path}", + method=method, + data=data, + headers={"Content-Type": "application/json", **(headers or {})}, + ) + try: + with urllib.request.urlopen(request) as response: + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + message = error.read().decode("utf-8", errors="replace") + raise click.ClickException( + f"{method} {path} returned {error.code}: {message}" + ) from error + except urllib.error.URLError as error: + raise click.ClickException( + f"could not reach evaluation sidecar: {error}" + ) from error + + +def _atomic_write_bytes(path: Path, payload: bytes) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + dir=path.parent, + prefix=f".{path.name}.", + suffix=".tmp", + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "wb") as file: + file.write(payload) + file.flush() + os.fsync(file.fileno()) + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) + + +def _download( + path: str, + destination: Path, + *, + headers: dict[str, str] | None = None, +) -> None: + request = urllib.request.Request( + f"{_base_url()}{path}", + method="GET", + headers=headers or {}, + ) + destination.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + dir=destination.parent, + prefix=f".{destination.name}.", + suffix=".tmp", + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "wb") as file: + try: + with urllib.request.urlopen(request) as response: + shutil.copyfileobj(response, file, length=1024 * 1024) + except urllib.error.HTTPError as error: + message = error.read().decode("utf-8", errors="replace") + raise click.ClickException( + f"GET {path} returned {error.code}: {message}" + ) from error + except urllib.error.URLError as error: + raise click.ClickException( + f"could not reach evaluation sidecar: {error}" + ) from error + file.flush() + os.fsync(file.fileno()) + os.replace(temporary, destination) + finally: + temporary.unlink(missing_ok=True) + + +def _redact_trace_text(value: str) -> str: + value = re.sub( + r"(?m)^([A-Z][A-Z0-9_]*(?:API_KEY|TOKEN|SECRET|PASSWORD)=).*$", + r"\1[REDACTED]", + value, + ) + value = re.sub( + r"(?i)(bearer\s+)[A-Za-z0-9._~-]{16,}", + r"\1[REDACTED]", + value, + ) + return re.sub(r"\bsk-[A-Za-z0-9_-]{12,}\b", "[REDACTED]", value) + + +def _redact_trace_value(value: object) -> object: + if isinstance(value, str): + return _redact_trace_text(value) + if isinstance(value, list): + return [_redact_trace_value(item) for item in value] + if isinstance(value, dict): + return {key: _redact_trace_value(item) for key, item in value.items()} + return value + + +def _load_agent_trace(path: Path) -> object: + text = _redact_trace_text(path.read_text(encoding="utf-8", errors="replace")) + try: + return _redact_trace_value(json.loads(text)) + except json.JSONDecodeError: + values = [] + for line in text.splitlines(): + try: + values.append(_redact_trace_value(json.loads(line))) + except json.JSONDecodeError: + continue + if not values: + return [{"role": "assistant", "content": text}] + + entries: list[dict] = [] + for value in values: + if not isinstance(value, dict): + entries.append({"type": "event", "value": value}) + continue + item = value.get("item") + if value.get("type") != "item.completed" or not isinstance(item, dict): + continue + item_type = item.get("type") + if item_type == "agent_message": + entries.append({"role": "assistant", "content": item.get("text", "")}) + elif item_type == "command_execution": + entries.extend( + [ + { + "type": "function_call", + "name": "exec_command", + "arguments": item.get("command", ""), + }, + { + "type": "function_call_output", + "output": item.get("aggregated_output", ""), + }, + ] + ) + elif item_type == "error": + entries.append( + {"type": "error", "message": item.get("message", "unknown error")} + ) + return entries or values + + +def _load_env_file(path: Path) -> dict[str, str]: + """Parse a dotenv-style ``NAME=VALUE`` secrets file. + + Blank lines and ``#`` comments are ignored; a leading ``export`` is + tolerated and surrounding single/double quotes are stripped. Values are + kept verbatim otherwise (no shell/variable expansion) so a secret cannot + be silently mangled. + """ + values: dict[str, str] = {} + for lineno, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + line = raw.strip() + if not line or line.startswith("#"): + continue + if line.startswith("export ") or line.startswith("export\t"): + line = line[len("export ") :].lstrip() + name, separator, value = line.partition("=") + name = name.strip() + if not separator or re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name) is None: + raise click.ClickException( + f"{path}:{lineno}: expected NAME=VALUE with a valid NAME, got {raw!r}" + ) + value = value.strip() + if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'": + value = value[1:-1] + values[name] = value + return values + + +# opencode's own default is 100 agentic iterations, after which it forces a +# text-only response and stops. That is well inside a real optimization run: +# gaia's optimizer used all 100 and never reached `evals submit`. Set high +# enough that the harness never truncates the search -- the case budget and +# the gateway token cap are the intended limits. +OPENCODE_STEP_LIMIT = 1000 + +# Harnesses that drive the model through litellm rather than a provider SDK. +# litellm reads the base URL as _API_BASE; the SDKs read +# _BASE_URL. vero sets the SDK names, so a litellm-based harness sees no +# override and calls the provider's public endpoint instead of the gateway. +_LITELLM_AGENTS = frozenset({"mini-swe-agent", "swe-agent"}) + + +def _litellm_base_url_args(agent: str, task: Path) -> list[str]: + """Give litellm-based harnesses the gateway URL under the name they read. + + mini-swe-agent installs `litellm[proxy]` and resolves `anthropic/` + through it. Without ANTHROPIC_API_BASE it reaches api.anthropic.com holding + only a scoped gateway token, and fails with + ``AuthenticationError: invalid x-api-key`` -- fails closed, so nothing leaks, + but the harness cannot run at all. Set both provider aliases from the + compiled producer scope so whichever provider the model names is covered. + """ + + if agent not in _LITELLM_AGENTS: + return [] + path = task / "environment/gateway/launch.json" + if not path.exists(): + return [] + try: + base_url = json.loads(path.read_text(encoding="utf-8"))["producer_base_url"] + except (OSError, json.JSONDecodeError, KeyError): + return [] + # litellm appends its own route to whatever base it is given, and the two + # providers want different bases. Its openai path adds `/chat/completions`, + # so that one takes the `/v1` producer URL as-is. Its anthropic path adds + # `/v1/messages` unless the base already ends in exactly that (main.py + # `anthropic_chat_completions`), so the same URL there yields a doubled + # `/v1/v1/messages`, which the gateway forwards upstream as a route nobody + # serves -- a 403 that reads like an auth failure. Hand anthropic the fully + # qualified messages path, which litellm leaves alone either way. + root = base_url.rstrip("/") + for suffix in ("/v1/messages", "/v1"): + if root.endswith(suffix): + root = root[: -len(suffix)] + break + return [ + "--ae", + f"OPENAI_API_BASE={base_url}", + "--ae", + f"ANTHROPIC_API_BASE={root}/v1/messages", + ] + + +def _opencode_gateway_args(agent: str, model: str | None, task: Path) -> list[str]: + """Route opencode's non-openai providers through the gateway. + + Harbor's opencode adapter writes a provider ``baseURL`` into + ``opencode.json`` only when the provider half of ``provider/model`` is + ``openai`` (``agents/installed/opencode.py``). For any other provider it + writes none, so opencode calls that provider's public endpoint. That fails + closed rather than leaking -- the optimizer only ever holds a scoped gateway + token, and Anthropic answers ``401 invalid x-api-key`` -- but it does leave + ``openai/`` as the only usable form, which forces the Responses API. Driving + a Claude model that way breaks: litellm's Anthropic-to-Responses translation + emits mixed id namespaces in one stream (a ``resp_`` id, Anthropic ``msg_``/ + ``toolu_`` item ids, and a stray ``chatcmpl-`` id), and opencode dies + resolving a text part under an id it never registered. + + Supplying the baseURL ourselves keeps the traffic on the provider's own API + -- Messages for Anthropic, which is the path claude-code already proves -- + and metered. The adapter deep-merges job kwargs last, so this wins. + """ + + if agent != "opencode" or not model: + return [] + + # opencode caps agentic iterations at 100 by default and then "forces a + # text-only response" (its own config schema's words for `steps`). A gaia + # optimizer hit that cap after ~2h, and the forced final message reads like a + # considered wrap-up, so the truncation is invisible unless you notice the + # step count is exactly 100 -- it never reached `evals submit`. claude-code + # takes harbor's --max-turns instead, so leaving this at the default makes + # the two harnesses incomparable. `build` is opencode's primary agent. + payload: dict[str, object] = { + "agent": {"build": {"steps": OPENCODE_STEP_LIMIT}} + } + + provider, _, _ = model.partition("/") + if "/" in model and provider != "openai": + # The adapter injects a baseURL only for the openai provider; for any + # other it writes none and opencode calls that provider's public + # endpoint. Supply it ourselves so the traffic stays metered. + path = task / "environment/gateway/launch.json" + if path.exists(): + try: + base_url = json.loads(path.read_text(encoding="utf-8"))[ + "producer_base_url" + ] + except (OSError, json.JSONDecodeError, KeyError): + base_url = None + if base_url: + # producer_base_url ends in /v1 and provider SDKs append their + # own route (/messages), matching ANTHROPIC_BASE_URL for + # claude-code. + payload["provider"] = {provider: {"options": {"baseURL": base_url}}} + return ["--ak", f"opencode_config={json.dumps(payload, separators=(',', ':'))}"] + + +def _outer_app_name_args( + environment: str, config_name: str, extra: tuple[str, ...] +) -> list[str]: + """Name the outer trial's Modal app so its sandbox can be found. + + Inner evaluation sandboxes are already grouped by an explicit + ``--ek app_name=...`` in each build's ``extra_harbor_args``, but the outer + trial had none and so landed in Modal's default ``__harbor__`` app. That + costs twice: the workspace holds thousands of containers, so an unnamed outer + sandbox is effectively unfindable in the UI, and recovering the session from + a run that must be killed begins with identifying its container. + + Derived from the build name (``vero/optimize-gaia-baseline`` -> + ``vero-optimize-gaia-baseline``) so outer trials group per benchmark. A + caller passing its own ``--ek app_name=`` wins. + """ + + if environment != "modal": + return [] + if any("app_name=" in argument for argument in extra): + return [] + slug = re.sub(r"[^a-zA-Z0-9_.-]+", "-", config_name).strip("-") + return ["--ek", f"app_name={slug or 'vero'}"] + + +def _compiled_run_environment( + task: Path, overrides: dict[str, str] | None = None +) -> dict[str, str]: + """Separate provider credentials from the environment seen by Harbor agents. + + ``overrides`` (e.g. a loaded ``--env-file``) take precedence over the + ambient environment so a run is reproducible from an explicit secrets file + regardless of what happens to be exported in the shell. + """ + environment = os.environ.copy() + if overrides: + environment.update(overrides) + path = task / "environment/gateway/launch.json" + if not path.exists(): + return environment + try: + launch = json.loads(path.read_text(encoding="utf-8")) + api_source = launch["upstream_api_key_source"] + api_target = launch["upstream_api_key_target"] + producer_api_key = launch["producer_api_key"] + producer_base_url = launch["producer_base_url"] + base_source = launch.get("upstream_base_url_source") + base_target = launch["upstream_base_url_target"] + except (KeyError, json.JSONDecodeError, OSError, TypeError) as error: + raise click.ClickException( + f"invalid compiled gateway launch config: {error}" + ) from error + for name, value in ( + ("upstream_api_key_source", api_source), + ("upstream_api_key_target", api_target), + ("producer_api_key", producer_api_key), + ("producer_base_url", producer_base_url), + ("upstream_base_url_target", base_target), + ): + if not isinstance(value, str) or not value: + raise click.ClickException(f"invalid compiled gateway field {name}") + upstream_api_key = environment.get(api_source) + if not upstream_api_key: + raise click.ClickException( + f"upstream inference credential {api_source} is missing" + ) + environment[api_target] = upstream_api_key + if base_source is not None: + if not isinstance(base_source, str) or not base_source: + raise click.ClickException( + "invalid compiled gateway field upstream_base_url_source" + ) + upstream_base_url = environment.get(base_source) + if not upstream_base_url: + raise click.ClickException( + f"upstream inference base URL {base_source} is missing" + ) + environment[base_target] = upstream_base_url + environment["OPENAI_API_KEY"] = producer_api_key + environment["OPENAI_BASE_URL"] = producer_base_url + # Claude Code (harbor -a claude) reads the Anthropic surface from the host + # env; point it at the same producer scope. The Anthropic SDK re-appends + # "/v1/messages", so hand it the scope root without the trailing "/v1". + # Set unconditionally: codex ignores ANTHROPIC_*, claude ignores OPENAI_*, so + # one compiled task serves either optimizer via `--agent`. + environment["ANTHROPIC_API_KEY"] = producer_api_key + environment["ANTHROPIC_BASE_URL"] = producer_base_url[: -len("/v1")] if ( + producer_base_url.endswith("/v1") + ) else producer_base_url + return environment + + +def _parameters(values: tuple[str, ...]) -> dict: + parameters = {} + for value in values: + name, separator, encoded = value.partition("=") + if not separator or not name.strip(): + raise click.BadParameter("use NAME=JSON", param_hint="--parameter") + if name in parameters: + raise click.BadParameter( + f"duplicate parameter {name!r}", + param_hint="--parameter", + ) + try: + parameters[name] = json.loads(encoded) + except json.JSONDecodeError as error: + raise click.BadParameter( + f"parameter {name!r} is not valid JSON", + param_hint="--parameter", + ) from error + return parameters + + +@click.group() +def harbor() -> None: + """Run VeRO across a Harbor sidecar boundary.""" + + +def _parse_build_params(values: tuple[str, ...]) -> dict[str, str]: + """Parse repeatable ``--param NAME=VALUE`` into a substitution context.""" + params: dict[str, str] = {} + for item in values: + name, separator, value = item.partition("=") + if not separator or re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name) is None: + raise click.ClickException( + f"--param must be NAME=VALUE with a valid NAME: {item!r}" + ) + params[name] = value + return params + + +_PARAM_OPTION = click.option( + "--param", + "params", + multiple=True, + metavar="NAME=VALUE", + help="Substitute ${NAME} in the build YAML; repeatable. Overrides the environment.", +) + + +@harbor.command("build") +@click.option( + "--config", + "config_path", + required=True, + type=click.Path(path_type=Path, exists=True, dir_okay=False), +) +@click.option( + "--output", + required=True, + type=click.Path(path_type=Path, file_okay=False), +) +@_PARAM_OPTION +def build_command(config_path, output, params): + """Compile a build YAML into a runnable Harbor task directory.""" + from vero.harbor.build import compile_harbor_task, load_harbor_build_config + + compiled = compile_harbor_task( + load_harbor_build_config(config_path, params=_parse_build_params(params)), + output, + ) + click.echo(f"Compiled Harbor task: {compiled}") + + +@harbor.command( + "run", + context_settings={"ignore_unknown_options": True}, +) +@click.option( + "--config", + "config_path", + required=True, + type=click.Path(path_type=Path, exists=True, dir_okay=False), +) +@click.option("--agent", required=True, help="Harbor optimizer agent.") +@click.option("--model", help="Model used by the optimizer agent.") +@click.option("--environment", default="modal", show_default=True) +@click.option( + "--env-file", + "env_file", + type=click.Path(path_type=Path, exists=True, dir_okay=False), + help=( + "Load NAME=VALUE run secrets (e.g. the upstream inference key and " + "MODAL_TOKEN_ID/SECRET) from a dotenv file. File values take precedence " + "over the ambient environment and never appear on the command line." + ), +) +@_PARAM_OPTION +@click.argument("extra", nargs=-1, type=click.UNPROCESSED) +def run_command(config_path, agent, model, environment, params, env_file, extra): + """Compile to a temporary directory and invoke `harbor run`.""" + from vero.harbor.build import compile_harbor_task, load_harbor_build_config + + uvx = shutil.which("uvx") + if uvx is None: + raise click.ClickException("uvx is required to run a compiled Harbor task") + overrides = _load_env_file(env_file) if env_file else None + if overrides: + click.echo(f"Loaded {len(overrides)} secret(s) from {env_file}") + # Apply before compiling: the build's declared-credential check and the + # ${NAME} param resolution both read os.environ, so the env-file must + # satisfy them too — not just the launched subprocess. + os.environ.update(overrides) + resolved = _parse_build_params(params) + # The optimizer model is both codex's -m and the producer scope's allow-list; + # expose it as the reserved ${optimizer_model} so a build that references it + # keeps the two in lockstep (no 403 model mismatch). + if model is not None: + resolved.setdefault("optimizer_model", model) + config = load_harbor_build_config(config_path, params=resolved) + with tempfile.TemporaryDirectory(prefix="vero-harbor-") as temporary: + task = compile_harbor_task( + config, + Path(temporary) / "task", + ) + command = [ + uvx, + "--python", + sys.executable, + "--from", + config.harbor_requirement, + "harbor", + "run", + "-p", + str(task), + "-a", + agent, + "-e", + environment, + ] + if model is not None: + command.extend(["-m", model]) + # Forward the build's declared agent env to the optimizer agent's shell. + # Harbor's `--ae KEY=VALUE` populates the agent's extra_env, which harbor + # injects into the agent's setup/install exec (scoped_exec_env). Sorted + # for a deterministic command line. + for key in sorted(config.agent_env): + command.extend(["--ae", f"{key}={config.agent_env[key]}"]) + command.extend(_opencode_gateway_args(agent, model, task)) + command.extend(_litellm_base_url_args(agent, task)) + command.extend(_outer_app_name_args(environment, config.name, extra)) + command.extend(extra) + click.echo(shlex.join(command)) + completed = subprocess.run( + command, + env=_compiled_run_environment(task, overrides), + ) + if completed.returncode: + raise SystemExit(completed.returncode) + + +@harbor.command("serve") +@click.option( + "--factory", "factory_path", required=True, help="Trusted module:factory." +) +@click.option( + "--config", + "config_path", + required=True, + type=click.Path(path_type=Path, exists=True, dir_okay=False), +) +@click.option( + "--admin-token", + "admin_token_path", + required=True, + type=click.Path(path_type=Path, dir_okay=False), +) +@click.option("--host", default="0.0.0.0", show_default=True) +@click.option("--port", default=8000, type=click.IntRange(1, 65535), show_default=True) +def serve_command(factory_path, config_path, admin_token_path, host, port): + """Serve components built by a trusted deployment factory.""" + from vero.sidecar.serve import serve + + serve( + factory_path=factory_path, + config_path=config_path, + admin_token_path=admin_token_path, + host=host, + port=port, + ) + + +@harbor.command("inference-gateway") +@click.option( + "--config", + "config_path", + required=True, + type=click.Path(path_type=Path, exists=True, dir_okay=False), +) +@click.option("--host", default="0.0.0.0", show_default=True) +@click.option("--port", default=8001, type=click.IntRange(1, 65535), show_default=True) +def inference_gateway_command(config_path, host, port): + """Serve the credential-isolating, budgeted inference proxy.""" + from vero.gateway.inference import serve_inference_gateway + + serve_inference_gateway(config_path=config_path, host=host, port=port) + + +@harbor.command("eval") +@click.option( + "--backend", "backend_id", required=True, + help=( + "Evaluation backend to score against. Must match --partition: each " + "partition is served by exactly one backend and asking a different one " + "is denied. `evals plan` lists the pair for every partition you may run." + ), +) +@click.option( + "--evaluation-set", "evaluation_set_name", required=True, + help="Name of the evaluation set to score on.", +) +@click.option( + "--partition", + help="Partition within the evaluation set (e.g. development or validation).", +) +@click.option("--version", help="Candidate version to score; defaults to the agent repository's HEAD commit.") +@click.option( + "--case-id", "case_ids", multiple=True, + help="Score only these case ids (repeatable) — a cheap subset for fast iteration. Cannot combine with --start/--stop.", +) +@click.option( + "--start", type=click.IntRange(min=0), + help="Subset start index (with --stop): score cases [start, stop) for cheap iteration on a slice.", +) +@click.option("--stop", type=click.IntRange(min=1), help="Subset stop index (exclusive); requires --start.") +@click.option("--parameter", multiple=True, help="Evaluation parameter as NAME=JSON (repeatable).") +@click.option( + "--timeout", type=click.FloatRange(min=0, min_open=True), + help="Override the whole-evaluation wall timeout, in seconds.", +) +@click.option( + "--case-timeout", type=click.FloatRange(min=0, min_open=True), + help="Override the per-case wall budget for THIS run, in seconds. A case that exceeds it is stopped and scores the failure value; the final held-out evaluation always uses the configured budget, so keep search runs comparable.", +) +@click.option("--max-concurrency", type=click.IntRange(min=1), help="Override how many cases run in parallel.") +@click.option( + "--error-rate-threshold", + type=click.FloatRange(min=0, max=1, min_open=True), + help="Abort the evaluation if the fraction of errored cases exceeds this.", +) +@click.option("--retry-max-attempts", type=click.IntRange(min=1), help="Max attempts per case on transient failure.") +@click.option("--retry-initial-delay", type=click.FloatRange(min=0), help="Initial retry backoff delay, in seconds.") +@click.option("--retry-maximum-delay", type=click.FloatRange(min=0), help="Maximum retry backoff delay, in seconds.") +@click.option("--retry-multiplier", type=click.FloatRange(min=1), help="Retry backoff growth multiplier.") +@click.option( + "--retry-on-timeout/--no-retry-on-timeout", + default=None, + help="Whether a per-case timeout counts as a retryable failure.", +) +@click.option( + "--seed", type=int, + help=( + "Seed for case sampling / evaluation. Backend-dependent: a backend that " + "fixes its own sampling rejects this with 'invalid evaluation request'. " + "To replicate elsewhere, re-run the identical case selection." + ), +) +@click.option( + "--detach", + is_flag=True, + help="Run several evaluations at once: start a durable job and return a job_id immediately instead of blocking. Poll `evals status JOB` until complete, then read `evals result JOB`. Omit to block and get the result in one call.", +) +def evaluate_command( + backend_id, + evaluation_set_name, + partition, + version, + case_ids, + start, + stop, + parameter, + timeout, + case_timeout, + max_concurrency, + error_rate_threshold, + retry_max_attempts, + retry_initial_delay, + retry_maximum_delay, + retry_multiplier, + retry_on_timeout, + seed, + detach, +): + """Evaluate a candidate through the metered agent endpoint.""" + if case_ids and (start is not None or stop is not None): + raise click.UsageError("--case-id cannot be combined with --start/--stop") + if start is not None and stop is None: + raise click.UsageError("--start requires --stop") + selection = None + if case_ids: + selection = CaseIds(ids=list(case_ids)) + elif stop is not None: + selection = CaseRange(start=start or 0, stop=stop) + evaluation_set = EvaluationSet( + name=evaluation_set_name, + partition=partition, + **({"selection": selection} if selection is not None else {}), + ) + retry_values = { + name: value + for name, value in { + "max_attempts": retry_max_attempts, + "initial_delay_seconds": retry_initial_delay, + "maximum_delay_seconds": retry_maximum_delay, + "multiplier": retry_multiplier, + "retry_on_timeout": retry_on_timeout, + }.items() + if value is not None + } + limit_values = { + name: value + for name, value in { + "timeout_seconds": timeout, + "case_timeout_seconds": case_timeout, + "max_concurrency": max_concurrency, + "error_rate_threshold": error_rate_threshold, + }.items() + if value is not None + } + if retry_values: + limit_values["retry"] = RetryPolicy(**retry_values) + body = SidecarEvaluationRequest( + backend_id=backend_id, + evaluation_set=evaluation_set, + version=version, + parameters=_parameters(parameter), + limits=EvaluationLimits(**limit_values) if limit_values else None, + seed=seed, + ) + click.echo( + json.dumps( + _request( + "POST", + "/eval/jobs" if detach else "/eval", + payload=body.model_dump(mode="json"), + ), + indent=2, + ) + ) + + +@harbor.command("eval-status") +@click.argument("job_id") +def evaluation_status_command(job_id): + """Inspect a detached evaluation job.""" + click.echo(json.dumps(_request("GET", f"/eval/jobs/{job_id}"), indent=2)) + + +@harbor.command("eval-result") +@click.argument("job_id") +def evaluation_result_command(job_id): + """Retrieve a detached evaluation result when it is available.""" + click.echo(json.dumps(_request("GET", f"/eval/jobs/{job_id}/result"), indent=2)) + + +@harbor.command("submit") +@click.option( + "--version", + help="Candidate version to nominate; defaults to the agent repository's HEAD commit.", +) +def submit_command(version): + """Nominate your best candidate as the one to ship. + + This is the deliberate selection that finalization scores on the held-out + set. Submit only a candidate you have confirmed beats the baseline on the + same cases. If you never submit, VeRO falls back to auto-best over + validation and then your last commit, so an unvetted edit can ship by + default -- submit on purpose. + """ + click.echo( + json.dumps(_request("POST", "/submit", payload={"version": version}), indent=2) + ) + + +@harbor.command("status") +def status_command(): + """Show agent-visible evaluation access and remaining budgets.""" + click.echo(json.dumps(_request("GET", "/status"), indent=2)) + + +@harbor.command("score-baseline") +@click.option( + "--token-file", + required=True, + type=click.Path(path_type=Path, exists=True, dir_okay=False), +) +@click.option("--replicates", default=1, show_default=True, type=click.IntRange(min=1)) +def score_baseline_command(token_file, replicates): + """Admin-score the fixed seed N times to produce a pinnable baseline number.""" + token = read_admin_token(token_file) + result = _request( + "POST", + "/score/baseline", + payload={"replicates": replicates}, + headers={"Authorization": f"Bearer {token}"}, + ) + click.echo(json.dumps(result, indent=2)) + + +@harbor.command("finalize") +@click.option( + "--token-file", + required=True, + type=click.Path(path_type=Path, exists=True, dir_okay=False), +) +@click.option( + "--output", + default="/logs/verifier/reward.json", + show_default=True, + type=click.Path(path_type=Path, dir_okay=False), +) +def finalize_command(token_file, output): + """Finalize as the trusted verifier and write Harbor rewards.""" + token = read_admin_token(token_file) + result = _request( + "POST", + "/finalize", + headers={"Authorization": f"Bearer {token}"}, + ) + destination = Path(output) + destination.parent.mkdir(parents=True, exist_ok=True) + # reward.json keeps only the reward map Harbor consumes. + destination.write_text( + json.dumps(result["rewards"], indent=2) + "\n", + encoding="utf-8", + ) + # Persist the full verification result (the shipped flag, verifier errors, + # baseline rewards) alongside it, so "did anything ship, and if not why" is + # answerable without re-running — reward.json alone drops those signals. + (destination.parent / "finalization.json").write_text( + json.dumps(result, indent=2) + "\n", + encoding="utf-8", + ) + click.echo(json.dumps(result, indent=2)) + + +@harbor.command("export-session") +@click.option( + "--token-file", + required=True, + type=click.Path(path_type=Path, exists=True, dir_okay=False), +) +@click.option( + "--output", + default="/logs/verifier/session.tar.gz", + show_default=True, + type=click.Path(path_type=Path, dir_okay=False), +) +@click.option( + "--report-output", + default="/logs/verifier/experiment.html", + show_default=True, + type=click.Path(path_type=Path, dir_okay=False), +) +@click.option( + "--status-output", + default="/logs/verifier/status.json", + show_default=True, + type=click.Path(path_type=Path, dir_okay=False), +) +@click.option( + "--finalization-output", + default="/logs/verifier/finalization.json", + show_default=True, + type=click.Path(path_type=Path, dir_okay=False), +) +@click.option( + "--agent-trace", + default="/logs/agent/trajectory.json", + show_default=True, + type=click.Path(path_type=Path, dir_okay=False), +) +def export_session_command( + token_file, + output, + report_output, + status_output, + finalization_output, + agent_trace, +): + """Persist the complete sidecar session and portable experiment report.""" + from vero.report import generate_experiment_report + + token = read_admin_token(token_file) + headers = {"Authorization": f"Bearer {token}"} + finalization = _request("POST", "/finalize", headers=headers) + status = _request("GET", "/status") + output = Path(output).expanduser().resolve() + report_output = Path(report_output).expanduser().resolve() + status_output = Path(status_output).expanduser().resolve() + finalization_output = Path(finalization_output).expanduser().resolve() + with tempfile.TemporaryDirectory(prefix="vero-harbor-session-") as directory: + temporary = Path(directory) + downloaded = temporary / "sidecar-session.tar.gz" + _download("/session/export", downloaded, headers=headers) + session = extract_harbor_session_archive(downloaded, temporary / "extracted") + encoded_finalization = ( + json.dumps(finalization, ensure_ascii=False, indent=2) + "\n" + ).encode("utf-8") + encoded_status = ( + json.dumps(status, ensure_ascii=False, indent=2) + "\n" + ).encode("utf-8") + _atomic_write_bytes(session / "harbor-finalization.json", encoded_finalization) + _atomic_write_bytes(session / "harbor-status.json", encoded_status) + + requested_trace = Path(agent_trace).expanduser() + trace_path = next( + ( + path + for path in ( + requested_trace, + Path("/logs/agent/trajectory.json"), + Path("/logs/agent/codex.txt"), + ) + if path.is_file() and not path.is_symlink() + ), + None, + ) + if trace_path is not None: + trace = _load_agent_trace(trace_path) + trace_destination = ( + session / "artifacts" / "agents" / "harbor-producer" / "trace.json" + ) + _atomic_write_bytes( + trace_destination, + (json.dumps(trace, ensure_ascii=False, indent=2) + "\n").encode( + "utf-8" + ), + ) + + generated_report = temporary / "experiment.html" + asyncio.run(generate_experiment_report(session, generated_report)) + create_harbor_session_archive(session, output) + digest = file_sha256(output) + _atomic_write_bytes(report_output, generated_report.read_bytes()) + _atomic_write_bytes(status_output, encoded_status) + _atomic_write_bytes(finalization_output, encoded_finalization) + _atomic_write_bytes( + output.with_name(f"{output.name}.sha256"), + f"{digest} {output.name}\n".encode("ascii"), + ) + click.echo( + json.dumps( + { + "session": str(output), + "sha256": digest, + "report": str(report_output), + }, + indent=2, + ) + ) diff --git a/vero/src/vero/harbor/deployment.py b/vero/src/vero/harbor/deployment.py new file mode 100644 index 00000000..e77aaf81 --- /dev/null +++ b/vero/src/vero/harbor/deployment.py @@ -0,0 +1,461 @@ +"""Standard component factory for compiled Harbor optimization tasks.""" + +from __future__ import annotations + +import logging +import shutil +from pathlib import Path, PurePosixPath +from typing import Annotated, Literal + +from pydantic import Field, JsonValue, field_validator, model_validator + +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import ( + BackendRegistry, + BudgetLedger, + EvaluationBackend, + EvaluationBudget, + EvaluationDatabase, + EvaluationLimits, + EvaluationSet, + Evaluator, + ObjectiveSpec, +) +from vero.evaluation.backends.command import CommandBackend, CommandBackendConfig +from vero.evaluation.engine import EvaluationEngine +from vero.harbor.backend import HarborBackend, HarborBackendConfig +from vero.models import StrictModel +from vero.sandbox import LocalSandbox +from vero.sidecar.serve import SidecarComponents +from vero.sidecar.session import initialize_harbor_session_manifest +from vero.sidecar.sidecar import EvaluationSidecar, SidecarEvaluationPolicy +from vero.sidecar.transport import GitCandidateTransport +from vero.sidecar.verifier import ( + CanonicalVerifier, + VerificationSelection, + VerificationTarget, +) +from vero.workspace import GitWorkspace + +logger = logging.getLogger(__name__) + + +class DeploymentSelection(StrictModel): + mode: Literal["submit", "auto_best"] = "auto_best" + backend_id: str | None = None + evaluation_set: EvaluationSet | None = None + objective: ObjectiveSpec | None = None + baseline_version: str | None = "HEAD" + parameters: dict[str, JsonValue] = Field(default_factory=dict) + limits: EvaluationLimits = Field(default_factory=EvaluationLimits) + rescore_top_k: int = Field(default=3, ge=1) + rescore_attempts: int = Field(default=1, ge=1) + baseline_floor: bool = False + baseline_selection_score: float | None = None + selection_coverage_threshold: float = Field(default=0.9, ge=0.0, le=1.0) + + @field_validator("backend_id", "baseline_version") + @classmethod + def validate_optional_identity(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("optional deployment identity must not be empty") + return value + + @model_validator(mode="after") + def validate_selection(self) -> DeploymentSelection: + if self.mode == "auto_best" and ( + self.backend_id is None + or self.evaluation_set is None + or self.objective is None + ): + raise ValueError( + "auto_best deployment requires backend_id, evaluation_set, and objective" + ) + return self + + +class SidecarWandbConfig(StrictModel): + """Trusted-side Weights & Biases config for the eval-sidecar. The W&B + credential (WANDB_API_KEY) is supplied to the sidecar container's + environment, never to the untrusted optimizer agent.""" + + project: str + entity: str | None = None + name: str | None = None + group: str | None = None + tags: list[str] = Field(default_factory=list) + mode: str | None = None + notes: str | None = None + run_id: str | None = None + # Also upload each evaluation's trace artifacts (Harbor trial records, + # stdout/stderr, agent trajectory) to W&B as a per-evaluation artifact. + # Off by default: it uploads files and can be large. + log_traces: bool = False + # How often the sidecar mirrors the gateway's usage ledger and request log + # into the run (when the gateway state paths are configured below). + telemetry_interval_seconds: float = Field(default=30.0, gt=0) + + +DeploymentBackendConfig = Annotated[ + HarborBackendConfig | CommandBackendConfig, + Field(discriminator="type"), +] +"""One partition's evaluation backend. + +Harbor for a nested ``harbor run`` against a target agent, command for a target +that is scored by running a program (a solver, an index build) rather than by +driving an agent. Both live behind the EvaluationBackend protocol, so +everything above this line — disclosure, budgets, verification — is unchanged. +""" + + +def _build_backend(config: DeploymentBackendConfig) -> EvaluationBackend: + if isinstance(config, CommandBackendConfig): + return CommandBackend(config) + return HarborBackend(config) + + +class HarborDeploymentConfig(StrictModel): + task_name: str = "harbor-session" + task_description: str = "" + repo_path: str + agent_repo_path: str + session_dir: str + session_id: str = "trial" + backends: dict[str, DeploymentBackendConfig] + access_policies: list[SidecarEvaluationPolicy] + budgets: list[EvaluationBudget] = Field(default_factory=list) + selection: DeploymentSelection + targets: list[VerificationTarget] + agent_volume: str | None = None + admin_volume: str + inference_usage_path: str | None = None + inference_request_log_dir: str | None = None + inference_limits: dict[str, dict[str, JsonValue]] = Field(default_factory=dict) + submit_enabled: bool = False + disclose_budget: bool = True + score_baseline: bool = True + evaluation_drain_timeout_seconds: float = Field(default=600.0, gt=0) + wandb: SidecarWandbConfig | None = None + + @model_validator(mode="before") + @classmethod + def default_backend_type(cls, value: object) -> object: + """Treat a backend with no ``type`` as Harbor. + + A discriminated union needs the tag present in the input, but serve.json + files compiled before the union existed omit it. Defaulting here keeps + those (and any in-flight run resuming from one) loadable. + """ + if not isinstance(value, dict): + return value + backends = value.get("backends") + if not isinstance(backends, dict): + return value + patched = { + backend_id: ( + {"type": "harbor", **config} + if isinstance(config, dict) and "type" not in config + else config + ) + for backend_id, config in backends.items() + } + return {**value, "backends": patched} + + @field_validator( + "repo_path", + "agent_repo_path", + "session_dir", + "admin_volume", + ) + @classmethod + def validate_absolute_path(cls, value: str) -> str: + path = PurePosixPath(value) + if not value.startswith("/") or ".." in path.parts: + raise ValueError("deployment paths must be absolute") + return value + + @field_validator("inference_usage_path", "inference_request_log_dir") + @classmethod + def validate_optional_file_path(cls, value: str | None) -> str | None: + if value is not None: + path = PurePosixPath(value) + if not value.startswith("/") or ".." in path.parts: + raise ValueError("deployment paths must be absolute") + return value + + @field_validator("agent_volume") + @classmethod + def validate_optional_path(cls, value: str | None) -> str | None: + if value is not None and ( + not value.startswith("/") or ".." in PurePosixPath(value).parts + ): + raise ValueError("deployment paths must be absolute") + return value + + @field_validator("session_id", "task_name") + @classmethod + def validate_session_id(cls, value: str) -> str: + if not value.strip(): + raise ValueError("session_id must not be empty") + return value + + @model_validator(mode="after") + def validate_references(self) -> HarborDeploymentConfig: + if not self.backends: + raise ValueError("deployment requires at least one backend") + trusted = PurePosixPath(self.repo_path) + agent = PurePosixPath(self.agent_repo_path) + if trusted == agent: + raise ValueError("trusted and agent repositories must be distinct") + for name, value in ( + ("session_dir", self.session_dir), + ("admin_volume", self.admin_volume), + *( + (("inference_usage_path", self.inference_usage_path),) + if self.inference_usage_path is not None + else () + ), + *( + (("inference_request_log_dir", self.inference_request_log_dir),) + if self.inference_request_log_dir is not None + else () + ), + ): + path = PurePosixPath(value) + if any( + path == repository or path.is_relative_to(repository) + for repository in (agent, trusted) + ): + raise ValueError(f"{name} must live outside candidate repositories") + if self.submit_enabled != (self.selection.mode == "submit"): + raise ValueError("submit_enabled must match selection mode") + known = set(self.backends) + referenced = {policy.backend_id for policy in self.access_policies} | { + target.backend_id for target in self.targets + } + if self.selection.backend_id is not None: + referenced.add(self.selection.backend_id) + unknown = sorted(referenced - known) + if unknown: + raise ValueError(f"deployment references unknown backends: {unknown}") + return self + + +def _database(session_dir: Path, session_id: str) -> EvaluationDatabase: + database_path = session_dir / "database.json" + return EvaluationDatabase.load_reconciled( + database_path=database_path, + evaluations_dir=session_dir / "evaluations", + database_id=session_id, + ) + + +def _ledger( + session_dir: Path, + budgets: list[EvaluationBudget], +) -> BudgetLedger | None: + path = session_dir / "budgets.json" + if path.exists(): + return BudgetLedger.load(path) + if not budgets: + return None + ledger = BudgetLedger(budgets, path=path) + ledger.save() + return ledger + + +async def build_harbor_components(config: dict) -> SidecarComponents: + """Build the standard compiled-task topology from trusted JSON config.""" + parsed = HarborDeploymentConfig.model_validate(config) + session_dir = Path(parsed.session_dir) + session_dir.mkdir(parents=True, exist_ok=True) + # The session dir holds the trusted state — held-out evaluation records and + # scores, the budget ledger, and every candidate's code. Lock it to the + # owning (trusted) user so the unprivileged harness that executes candidate + # code cannot read it. Harmless where the harness is not isolated (the owner + # retains access); the admin token dir is already hardened in auth.py. + session_dir.chmod(0o700) + sandbox = await LocalSandbox.create(root=Path(parsed.repo_path).parent) + workspace = await GitWorkspace.from_path(sandbox, parsed.repo_path) + candidate_repository = await GitCandidateRepository.create( + session_dir / "candidates", + workspace=workspace, + ) + database = _database(session_dir, parsed.session_id) + ledger = _ledger(session_dir, parsed.budgets) + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=session_dir, + session_id=parsed.session_id, + ), + backends=BackendRegistry( + { + backend_id: _build_backend(backend_config) + for backend_id, backend_config in parsed.backends.items() + } + ), + database=database, + database_path=session_dir / "database.json", + budget_ledger=ledger, + ) + wandb_sink = None + if parsed.wandb is not None: + # Observability must never take down the eval path: on any failure to + # construct the sink, log and continue without W&B. + from vero.runtime.wandb import SidecarWandbSink + + try: + wandb_sink = SidecarWandbSink( + project=parsed.wandb.project, + session_id=parsed.session_id, + session_dir=session_dir, + entity=parsed.wandb.entity, + name=parsed.wandb.name, + group=parsed.wandb.group, + tags=list(parsed.wandb.tags), + mode=parsed.wandb.mode, + notes=parsed.wandb.notes, + run_id=parsed.wandb.run_id, + budget_ledger=ledger, + log_traces=parsed.wandb.log_traces, + ) + engine.listeners.append(wandb_sink) + except Exception: + logger.warning( + "W&B reporting disabled: the sidecar sink could not be " + "initialized; the evaluation sidecar continues without it", + exc_info=True, + ) + wandb_sink = None + + usage_path = ( + Path(parsed.inference_usage_path) + if parsed.inference_usage_path is not None + else None + ) + request_log_dir = ( + Path(parsed.inference_request_log_dir) + if parsed.inference_request_log_dir is not None + else None + ) + telemetry = None + if wandb_sink is not None and ( + usage_path is not None or request_log_dir is not None + ): + from vero.runtime.wandb import InferenceTelemetryPoller + + telemetry = InferenceTelemetryPoller( + sink=wandb_sink, + usage_path=usage_path, + request_log_dir=request_log_dir, + interval_seconds=parsed.wandb.telemetry_interval_seconds, + ) + + def _export_inference_state() -> None: + # Preserve the gateway's usage ledger and request log with the session, + # so /session/export (and the archived run record) carries every + # request-response the gateway saw. + destination = session_dir / "artifacts" / "inference" + try: + if usage_path is not None and usage_path.is_file(): + destination.mkdir(parents=True, exist_ok=True) + shutil.copy2(usage_path, destination / usage_path.name) + if request_log_dir is not None and request_log_dir.is_dir(): + shutil.copytree( + request_log_dir, + destination / "requests", + dirs_exist_ok=True, + ) + except OSError: + logger.warning("inference state export failed", exc_info=True) + + def _finalize_session_telemetry(result: object) -> None: + # Session's end: archive gateway state, flush telemetry (including the + # still-active request log file), and close the W&B run with a summary. + _export_inference_state() + if telemetry is not None: + telemetry.poll_once(final=True) + if wandb_sink is None: + return + rewards = getattr(result, "rewards", {}) or {} + summary = { + "shipped": getattr(result, "shipped", None), + **{f"reward/{key}": value for key, value in rewards.items()}, + } + wandb_sink.finish(summary, failed=not getattr(result, "shipped", True)) + transport = GitCandidateTransport( + workspace=workspace, + candidate_repository=candidate_repository, + agent_repo_path=parsed.agent_repo_path, + ) + baseline = ( + await transport.trusted_candidate(parsed.selection.baseline_version) + if parsed.selection.baseline_version is not None + else None + ) + selection = VerificationSelection( + mode=parsed.selection.mode, + backend_id=parsed.selection.backend_id, + evaluation_set=parsed.selection.evaluation_set, + objective=parsed.selection.objective, + baseline_candidate=baseline, + parameters=parsed.selection.parameters, + limits=parsed.selection.limits, + rescore_top_k=parsed.selection.rescore_top_k, + rescore_attempts=parsed.selection.rescore_attempts, + baseline_floor=parsed.selection.baseline_floor, + baseline_selection_score=parsed.selection.baseline_selection_score, + selection_coverage_threshold=parsed.selection.selection_coverage_threshold, + ) + initialize_harbor_session_manifest( + session_dir, + session_id=parsed.session_id, + task_name=parsed.task_name, + task_description=parsed.task_description, + backends={ + backend_id: engine.backends.resolve(backend_id).provenance + for backend_id in parsed.backends + }, + selection=selection, + targets=parsed.targets, + ) + sidecar = EvaluationSidecar( + engine=engine, + candidate_transport=transport, + access_policies=parsed.access_policies, + agent_volume=( + Path(parsed.agent_volume) if parsed.agent_volume is not None else None + ), + admin_volume=Path(parsed.admin_volume), + submit_enabled=parsed.submit_enabled, + disclose_budget=parsed.disclose_budget, + inference_usage_path=( + Path(parsed.inference_usage_path) + if parsed.inference_usage_path is not None + else None + ), + inference_limits=parsed.inference_limits, + ) + await sidecar.initialize_context() + verifier = CanonicalVerifier( + engine=engine, + selection=selection, + targets=parsed.targets, + admin_volume=Path(parsed.admin_volume), + score_baseline=parsed.score_baseline, + evaluation_drain_timeout_seconds=parsed.evaluation_drain_timeout_seconds, + on_finalized=_finalize_session_telemetry, + ) + return SidecarComponents(sidecar=sidecar, verifier=verifier, telemetry=telemetry) + + +FACTORY_PATH = f"{build_harbor_components.__module__}:{build_harbor_components.__qualname__}" +"""Dotted path the compiled task uses to load this factory. + +Derived rather than written out, so renaming this module or function cannot leave +a stale literal in the compose template. A wrong value here would not fail at +import or in the type checker: it fails when the sidecar container starts. +""" diff --git a/vero/src/vero/harbor/generation.py b/vero/src/vero/harbor/generation.py new file mode 100644 index 00000000..e530979b --- /dev/null +++ b/vero/src/vero/harbor/generation.py @@ -0,0 +1,58 @@ +"""Harbor-backed candidate generation. + +A :class:`~vero.optimization.protocols.GenerationBackend` that produces +candidates by delegating to a ``harbor run`` — symmetric to how +:class:`~vero.harbor.backend.HarborBackend` nests ``harbor run`` for *evaluation*. +This is the contained / untrusted / multi-adapter production path (codex, claude, +terminus, …); the lightweight in-process native producer is the Optimizer's +default backend. + +Status: interface skeleton. The full wiring (drive a ``harbor run`` for the +proposal's agent, then import the resulting commit into the session candidate +repository by object identity) is a follow-on; it reuses the existing +:class:`~vero.sidecar.transport.GitCandidateTransport` and the ``harbor.backend`` +nesting rather than re-hosting Harbor's sidecar/gateway. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Sequence + +from vero.candidate import Candidate +from vero.evaluation import EvaluationRecord +from vero.optimization.models import ( + CandidateProposal, + GenerationOutcome, + OptimizationContext, +) + + +@dataclass +class HarborGenerationBackend: + """Generate candidates by running a Harbor agent over the parent. + + Reuses ``GitCandidateTransport`` to import the untrusted candidate commit + into the session repository by object identity, and returns the Harbor + sidecar's disclosed-partition scores as the generation-time feedback in + ``GenerationOutcome.trial_evaluations``. Selection/target scoring remains the + Optimizer's responsibility. + """ + + # Populated when the harbor-run-as-production wiring lands (task source, + # agent, transport, candidate repository, session dir, …). + + async def generate( + self, + *, + proposal: CandidateProposal, + parent: Candidate, + context: OptimizationContext, + evaluation_records: Sequence[EvaluationRecord], + ) -> GenerationOutcome: + raise NotImplementedError( + "HarborGenerationBackend is an interface skeleton; the native " + "in-process backend (Optimizer default) is the implemented path. " + "Wire this to a `harbor run` + GitCandidateTransport import to enable " + "contained/multi-adapter production." + ) diff --git a/vero/src/vero/layout.py b/vero/src/vero/layout.py new file mode 100644 index 00000000..ffc724eb --- /dev/null +++ b/vero/src/vero/layout.py @@ -0,0 +1,164 @@ +"""The single source of truth for a compiled Harbor task's layout. + +Every container path, service name, and port in a compiled task is defined here +once and referenced everywhere else — the compiler, the Jinja templates, the +runtime configs, and the tests. Before this module the same strings were copied +between Python constants and template literals, so changing one silently failed +to propagate to the other. + +The values are effectively frozen: they are a contract with things outside this +package, including every benchmark's ``read_only_paths`` and the compiled task +directories that are checked in. Rename an attribute freely; changing what it +points at is a breaking change. + +Two conventions worth keeping straight, because both say "agent": + +- The *target* is the program under optimization. It may not be an agent at all + (a solver, an index build), which is why its paths are named ``target_*``. +- The *agent* is the optimizer, which really is an agent. ``agent_volume`` is + its context directory. +""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class TaskLayout: + """Container paths and service identities inside a compiled Harbor task. + + Attributes: + target_repo: The editable target the optimizer may change. + trusted_repo: Immutable copy of the target, the non-regression floor. + seed_repo: The starting point the target repo is seeded from. + vero: Local vero source, when built from a source checkout. + cases: Per-partition case files, root-only. + task_source: Local Harbor task definitions, for a harbor backend. + harness: The scoring program, for a command backend. + overlay: Host files baked into the optimizer's workspace. + serve_config: The trusted deployment config, root-only. + seed_script: Script that seeds the target repo on first boot. + inference_config: The gateway's scope config. + agent_volume: The optimizer's context directory, written by the sidecar. + admin_volume: Trusted state: records, ledger, candidates. + token_dir: Directory holding the admin token. + inference_dir: Gateway state: usage ledger and request log. + sidecar_host: Compose service name of the trusted evaluation sidecar. + sidecar_port: Port the sidecar listens on. + gateway_host: Compose service name of the inference gateway. + gateway_port: Port the gateway listens on. + producer_api_key_env: Container-side variable holding the optimizer's + scoped gateway token. + producer_base_url_env: Container-side variable holding the optimizer's + gateway scope URL. + """ + + target_repo: str = "/work/agent" + trusted_repo: str = "/opt/agent-baseline" + seed_repo: str = "/opt/agent-seed" + vero: str = "/opt/vero" + cases: str = "/opt/cases" + task_source: str = "/opt/task-source" + harness: str = "/opt/harness" + overlay: str = "/opt/overlay" + serve_config: str = "/opt/serve.json" + seed_script: str = "/opt/seed.sh" + inference_config: str = "/opt/inference.json" + agent_volume: str = "/state/agent-context" + admin_volume: str = "/state/admin" + token_dir: str = "/state/token" + inference_dir: str = "/state/inference" + sidecar_host: str = "eval-sidecar" + sidecar_port: int = 8000 + gateway_host: str = "inference-gateway" + gateway_port: int = 8001 + eval_url_env: str = "VERO_EVAL_URL" + gateway_upstream_api_key_env: str = "VERO_INFERENCE_UPSTREAM_API_KEY" + gateway_upstream_base_url_env: str = "VERO_INFERENCE_UPSTREAM_BASE_URL" + optimizer_attribution: str = "optimizer" + # Container-side names an OpenAI-surface client reads. The compose file sets + # both for the optimizer -- key to the producer token, base URL to its + # gateway scope -- and the compiler excludes them from the blanking loop so + # it does not emit the same YAML key twice. The two must stay in step: a name + # excluded from blanking but never set would keep whatever the host passed + # in, so routed_credential_envs below is the single list both sides use. + producer_api_key_env: str = "OPENAI_API_KEY" + producer_base_url_env: str = "OPENAI_BASE_URL" + # The gateway's proxy route, with the FastAPI parameter names it binds. Both + # the route the gateway serves and every URL built for it derive from this one + # string, so a caller cannot construct a path the gateway will not match. + scope_route_base: str = "/scopes/{scope_name}/{attribution}/v1" + + # Derived paths. Defined here rather than spelled out at each use site, so a + # base path and its children cannot drift apart. + + @property + def session_dir(self) -> str: + return f"{self.admin_volume}/session" + + @property + def case_resources_dir(self) -> str: + return f"{self.admin_volume}/case-resources" + + @property + def token_path(self) -> str: + return f"{self.token_dir}/admin.token" + + @property + def inference_state(self) -> str: + return f"{self.inference_dir}/usage.json" + + @property + def inference_request_log_dir(self) -> str: + return f"{self.inference_dir}/requests" + + @property + def target_git(self) -> str: + return f"{self.target_repo}/.git" + + @property + def target_git_exclude(self) -> str: + return f"{self.target_git}/info/exclude" + + @property + def target_evals(self) -> str: + return f"{self.target_repo}/.evals" + + @property + def routed_credential_envs(self) -> tuple[str, ...]: + """Names the compose file sets explicitly, so must not also blank.""" + return (self.producer_api_key_env, self.producer_base_url_env) + + @property + def sidecar_url(self) -> str: + return f"http://{self.sidecar_host}:{self.sidecar_port}" + + @property + def gateway_url(self) -> str: + return f"http://{self.gateway_host}:{self.gateway_port}" + + @property + def scope_route(self) -> str: + """The route the gateway registers, including the proxied endpoint.""" + return f"{self.scope_route_base}/{{endpoint:path}}" + + def scope_path(self, scope: str, attribution: str) -> str: + """Path for one scope, for callers that hold their own base URL. + + The trusted backend reads its gateway URL from the deployment config + rather than assuming this layout's, so it needs the path alone. + """ + return self.scope_route_base.format(scope_name=scope, attribution=attribution) + + def scope_url(self, scope: str, attribution: str) -> str: + """Base URL an OpenAI-compatible client should use for one scope. + + The attribution segment is a free-form label for per-caller accounting, + not a security boundary: the token decides what the scope may do. + """ + return f"{self.gateway_url}{self.scope_path(scope, attribution)}" + + +LAYOUT = TaskLayout() +"""The one instance. There is no reason to construct another.""" diff --git a/vero/src/vero/models.py b/vero/src/vero/models.py new file mode 100644 index 00000000..49fbeca3 --- /dev/null +++ b/vero/src/vero/models.py @@ -0,0 +1,16 @@ +"""Shared base model for VeRO's declarative contracts.""" + +from __future__ import annotations + +from pydantic import BaseModel, ConfigDict + + +class StrictModel(BaseModel): + """Base model that rejects unknown fields. + + Authored configuration and on-disk records both inherit this, so a typo'd + key fails loudly at load time instead of being dropped in silence and + leaving a default in its place. + """ + + model_config = ConfigDict(extra="forbid") diff --git a/vero/src/vero/optimization/__init__.py b/vero/src/vero/optimization/__init__.py new file mode 100644 index 00000000..534f783b --- /dev/null +++ b/vero/src/vero/optimization/__init__.py @@ -0,0 +1,49 @@ +"""Strategy-driven optimization of versioned programs.""" + +from vero.optimization.command import ( + CommandCandidateProducer, + CommandCandidateProducerConfig, +) +from vero.optimization.models import ( + CandidateChange, + CandidateProductionContext, + CandidateProposal, + GenerationOutcome, + OptimizationContext, + OptimizationResult, +) +from vero.optimization.optimizer import Optimizer +from vero.optimization.protocols import ( + CandidateEvaluationGateway, + CandidateProducer, + GenerationBackend, + OptimizationStrategy, + SelectionPolicy, +) +from vero.optimization.strategy import ( + DarwinGodelStrategy, + EvolutionaryStrategy, + ObjectiveSelectionPolicy, + SequentialStrategy, +) + +__all__ = [ + "CandidateChange", + "CandidateProductionContext", + "CandidateEvaluationGateway", + "CandidateProducer", + "CandidateProposal", + "CommandCandidateProducer", + "CommandCandidateProducerConfig", + "DarwinGodelStrategy", + "EvolutionaryStrategy", + "GenerationBackend", + "GenerationOutcome", + "ObjectiveSelectionPolicy", + "OptimizationContext", + "OptimizationResult", + "OptimizationStrategy", + "Optimizer", + "SelectionPolicy", + "SequentialStrategy", +] diff --git a/vero/src/vero/optimization/command.py b/vero/src/vero/optimization/command.py new file mode 100644 index 00000000..d2f13052 --- /dev/null +++ b/vero/src/vero/optimization/command.py @@ -0,0 +1,179 @@ +"""Trusted command candidate producer.""" + +from __future__ import annotations + +import os +import posixpath +import re +from pathlib import Path + +from pydantic import Field, field_validator, model_validator + +from vero.models import StrictModel +from vero.optimization.models import ( + CandidateChange, + CandidateProductionContext, + CandidateProposal, +) +from vero.optimization.protocols import CandidateEvaluationGateway +from vero.staging import SandboxStagingArea +from vero.workspace import Workspace + +_PLACEHOLDERS = { + "workspace", + "context", + "producer", + "round", + "instruction", + "best_candidate_id", + "best_version", + "best_value", +} +_PLACEHOLDER_PATTERN = re.compile(r"\{([^{}]+)\}") + + +class CommandCandidateProducerConfig(StrictModel): + root: str + command: list[str] + working_directory: str = "." + environment: dict[str, str] = Field(default_factory=dict) + passthrough_environment: list[str] = Field(default_factory=list) + timeout_seconds: float = Field(default=600.0, gt=0.0) + description: str = "Optimize candidate" + + @field_validator("root") + @classmethod + def validate_root(cls, value: str) -> str: + if not value.strip() or not Path(value).is_absolute(): + raise ValueError("producer root must be absolute after config resolution") + return value + + @field_validator("command") + @classmethod + def validate_command(cls, value: list[str]) -> list[str]: + if not value or any(not argument for argument in value): + raise ValueError("producer command and its arguments must not be empty") + unknown = { + placeholder + for argument in value + for placeholder in _PLACEHOLDER_PATTERN.findall(argument) + if placeholder not in _PLACEHOLDERS + } + if unknown: + raise ValueError( + f"unknown producer placeholders: {', '.join(sorted(unknown))}" + ) + return value + + @field_validator("working_directory") + @classmethod + def validate_working_directory(cls, value: str) -> str: + path = Path(value) + if not value.strip() or path.is_absolute() or ".." in path.parts: + raise ValueError("producer working_directory must stay within its root") + return value + + @field_validator("passthrough_environment") + @classmethod + def validate_passthrough(cls, value: list[str]) -> list[str]: + if len(value) != len(set(value)): + raise ValueError("producer passthrough environment names must be unique") + return value + + @field_validator("description") + @classmethod + def validate_description(cls, value: str) -> str: + if not value.strip(): + raise ValueError("producer description must not be empty") + return value + + @model_validator(mode="after") + def validate_environment(self) -> CommandCandidateProducerConfig: + overlap = set(self.environment) & set(self.passthrough_environment) + if overlap: + raise ValueError( + "producer environment sources overlap for: " + + ", ".join(sorted(overlap)) + ) + return self + + +class CommandCandidateProducer: + """Run a trusted command that edits the supplied candidate workspace.""" + + def __init__(self, config: CommandCandidateProducerConfig): + self.config = config + + def _environment(self) -> dict[str, str]: + environment = {"PATH": os.defpath, "LANG": "C.UTF-8"} + for name in ("TMPDIR", "TMP", "TEMP", "SYSTEMROOT"): + if name in os.environ: + environment[name] = os.environ[name] + environment.update(self.config.environment) + for name in self.config.passthrough_environment: + if name in os.environ: + environment[name] = os.environ[name] + return environment + + async def produce( + self, + *, + proposal: CandidateProposal, + context: CandidateProductionContext, + workspace: Workspace, + evaluation: CandidateEvaluationGateway, + ) -> CandidateChange | None: + root = Path(self.config.root).resolve() + target = workspace.sandbox.host_path(workspace.project_path) + if target is not None: + target = target.resolve() + if root == target or root.is_relative_to(target): + raise ValueError( + "candidate producer must live outside the editable target" + ) + + best = context.best + async with SandboxStagingArea( + workspace.sandbox, + prefix=f"vero-producer-{proposal.id[:8]}-", + ) as staging: + producer_root = ( + str(root) + if workspace.sandbox.capabilities.host_paths + else await staging.upload(root, "producer") + ) + working_directory = posixpath.normpath( + posixpath.join(producer_root, self.config.working_directory) + ) + values = { + "workspace": workspace.project_path, + "context": posixpath.join(workspace.project_path, ".evals"), + "producer": producer_root, + "round": str(context.round), + "instruction": proposal.instruction or "", + "best_candidate_id": best.id if best else "", + "best_version": best.version if best else "", + # Evaluation values live in the authorized filesystem context. + "best_value": "", + } + command: list[str] = [] + for argument in self.config.command: + expanded = argument + for placeholder, value in values.items(): + expanded = expanded.replace(f"{{{placeholder}}}", value) + command.append(expanded) + + environment = self._environment() + environment["VERO_CONTEXT_PATH"] = values["context"] + result = await workspace.sandbox.run( + command, + cwd=working_directory, + timeout=self.config.timeout_seconds, + env=environment, + ) + if result.returncode != 0: + raise RuntimeError( + result.stderr + or f"candidate producer exited with status {result.returncode}" + ) + return CandidateChange(description=self.config.description) diff --git a/vero/src/vero/optimization/models.py b/vero/src/vero/optimization/models.py new file mode 100644 index 00000000..3c631dc7 --- /dev/null +++ b/vero/src/vero/optimization/models.py @@ -0,0 +1,130 @@ +"""Optimization proposals, context, and results.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Mapping +from uuid import uuid4 + +from pydantic import Field, JsonValue, field_validator, model_validator + +from vero.candidate import Candidate +from vero.evaluation import ( + EvaluationAcknowledgement, + EvaluationRecord, + EvaluationSummary, +) +from vero.models import StrictModel +from vero.workspace import Workspace + + +class CandidateProposal(StrictModel): + """A strategy's request for one producer to explore a parent candidate.""" + + id: str = Field(default_factory=lambda: str(uuid4())) + producer_id: str = "default" + parent_id: str | None = None + instruction: str | None = None + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("id", "producer_id") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("proposal identity must not be empty") + return value + + @field_validator("parent_id", "instruction") + @classmethod + def validate_optional_text(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("optional proposal text must not be empty") + return value + + @model_validator(mode="after") + def validate_parent(self) -> CandidateProposal: + if self.parent_id == self.id: + raise ValueError("a proposal cannot name itself as its parent") + return self + + +class CandidateChange(StrictModel): + """Producer metadata returned after it edits a supplied workspace.""" + + description: str = "Optimize candidate" + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("description") + @classmethod + def validate_description(cls, value: str) -> str: + if not value.strip(): + raise ValueError("candidate description must not be empty") + return value + + +@dataclass(frozen=True) +class OptimizationContext: + """Authorization-projected history supplied to an optimization strategy.""" + + session_id: str + round: int + workspace: Workspace + baseline: Candidate + evaluations: tuple[ + EvaluationRecord | EvaluationSummary | EvaluationAcknowledgement, + ..., + ] + candidates: Mapping[str, Candidate] + best: Candidate | None + + +@dataclass(frozen=True) +class CandidateProductionContext: + """Non-sensitive control context supplied to a candidate producer. + + Authorized evaluation details live in the read-only ``.evals`` tree and are + returned through the evaluation gateway; they are intentionally not + duplicated here as full records. + """ + + session_id: str + round: int + baseline: Candidate + candidates: Mapping[str, Candidate] + best: Candidate | None + + +@dataclass(frozen=True) +class GenerationOutcome: + """Candidates and generation-time feedback from executing one proposal. + + Produced by a :class:`~vero.optimization.protocols.GenerationBackend` (the + native in-process producer by default, or a Harbor run). ``candidate`` is the + produced candidate (``None`` if the producer made no change); + ``trial_candidates`` are the intermediate checkpoints it captured. Crucially, + ``trial_evaluations`` are the **generation-time feedback** evaluations the + producer *observed while iterating* (mid-run self-eval, or a Harbor sidecar's + disclosed-partition scores) — distinct from the orchestrator's later + selection and target scoring, which the Optimizer performs separately on the + returned candidate. + """ + + candidate: Candidate | None + trial_candidates: tuple[Candidate, ...] + trial_evaluations: tuple[EvaluationRecord, ...] + + @property + def candidates(self) -> tuple[Candidate, ...]: + if self.candidate is None: + return self.trial_candidates + return (*self.trial_candidates, self.candidate) + + +@dataclass(frozen=True) +class OptimizationResult: + baseline: EvaluationRecord + evaluations: tuple[EvaluationRecord, ...] + candidates: tuple[Candidate, ...] + best: EvaluationRecord | None + final_baseline: EvaluationRecord | None = None + final: EvaluationRecord | None = None diff --git a/vero/src/vero/optimization/optimizer.py b/vero/src/vero/optimization/optimizer.py new file mode 100644 index 00000000..324d3996 --- /dev/null +++ b/vero/src/vero/optimization/optimizer.py @@ -0,0 +1,739 @@ +"""Batch-oriented optimizer over versioned program candidates.""" + +from __future__ import annotations + +import asyncio +import hashlib +from dataclasses import dataclass, field +from datetime import UTC, datetime +from typing import TYPE_CHECKING + +from pydantic import JsonValue + +from vero.candidate import Candidate +from vero.candidate_repository import CandidateRepository +from vero.evaluation import ( + CaseSelection, + DisclosureLevel, + EvaluationBudget, + EvaluationCancelledError, + EvaluationEngine, + EvaluationExecutionError, + EvaluationLimits, + EvaluationPlan, + EvaluationPrincipal, + EvaluationReceipt, + EvaluationRecord, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + EvaluationSummary, + ObjectiveSpec, + project_evaluation, +) +from vero.optimization.models import ( + CandidateProductionContext, + CandidateProposal, + GenerationOutcome, + OptimizationContext, + OptimizationResult, +) +from vero.optimization.protocols import ( + CandidateEvaluationGateway, + CandidateProducer, + GenerationBackend, + OptimizationStrategy, + SelectionPolicy, +) +from vero.optimization.strategy import ObjectiveSelectionPolicy +from vero.workspace import Workspace + +if TYPE_CHECKING: + from vero.runtime.context import AgentDisclosureLedger, WorkspaceContextManager + + +class _ScopedEvaluationGateway(CandidateEvaluationGateway): + def __init__( + self, + *, + optimizer: Optimizer, + proposal: CandidateProposal, + parent: Candidate, + workspace: Workspace, + workspace_context: WorkspaceContextManager, + round_number: int, + ): + self.optimizer = optimizer + self.proposal = proposal + self.parent = parent + self.workspace = workspace + self.workspace_context = workspace_context + self.round_number = round_number + self._count = 0 + self._last_candidate_id = parent.id + self._trial_candidates: list[Candidate] = [] + self._trial_evaluations: list[EvaluationRecord] = [] + + @property + def last_candidate_id(self) -> str: + return self._last_candidate_id + + @property + def last_candidate_version(self) -> str | None: + if not self._trial_candidates: + return None + return self._trial_candidates[-1].version + + @property + def trial_candidates(self) -> tuple[Candidate, ...]: + return tuple(self._trial_candidates) + + @property + def trial_evaluations(self) -> tuple[EvaluationRecord, ...]: + return tuple(self._trial_evaluations) + + async def evaluate( + self, + *, + evaluation: str, + selection: CaseSelection | None = None, + candidate_id: str | None = None, + description: str = "Evaluate agent checkpoint", + ) -> EvaluationReceipt: + if not description.strip(): + raise ValueError("checkpoint description must not be empty") + try: + definition = self.optimizer.evaluation_plan.get(evaluation) + except KeyError as error: + raise ValueError(f"unknown evaluation: {evaluation!r}") from error + requested_set = definition.evaluation_set.model_copy( + update={"selection": selection or definition.evaluation_set.selection} + ) + captured = candidate_id is None + if candidate_id is not None: + candidate = self.optimizer.candidate_repository.get(candidate_id) + if candidate is None: + raise ValueError(f"unknown candidate: {candidate_id!r}") + else: + version = ( + await self.workspace.save(description) + if await self.workspace.is_dirty() + else await self.workspace.current_version() + ) + self._count += 1 + candidate = Candidate( + id=f"{self.proposal.id}:trial:{self._count}", + version=version, + parent_id=self._last_candidate_id, + created_at=datetime.now(UTC), + description=description, + metadata={ + **self.proposal.metadata, + "producer_id": self.proposal.producer_id, + "proposal_id": self.proposal.id, + "round": self.round_number, + "trial": self._count, + }, + ) + await self.optimizer._capture_candidate(candidate, self.workspace) + request = self.optimizer._request(candidate, requested_set) + try: + result = await self.optimizer.engine.evaluate( + backend_id=self.optimizer.backend_id, + request=request, + objective_spec=self.optimizer.objective, + principal=EvaluationPrincipal.AGENT, + ) + except (EvaluationExecutionError, EvaluationCancelledError) as error: + record = self.optimizer.engine.database.get_evaluation(error.evaluation_id) + if record is not None: + decision = await self.optimizer.engine.authorize( + self.optimizer.backend_id, + request, + EvaluationPrincipal.AGENT, + ) + if decision.viewable: + await asyncio.shield( + self.workspace_context.add_evaluation( + record, + decision.disclosure, + ) + ) + raise + evaluation_id = ( + result.id if isinstance(result, EvaluationRecord) else result.evaluation_id + ) + record = self.optimizer.engine.database.get_evaluation(evaluation_id) + if record is None: + raise RuntimeError( + "evaluation engine did not index completed evaluation " + f"{evaluation_id!r}" + ) + if captured: + self._last_candidate_id = candidate.id + self._trial_candidates.append(candidate) + self._trial_evaluations.append(record) + disclosure = ( + DisclosureLevel.FULL + if isinstance(result, EvaluationRecord) + else ( + DisclosureLevel.AGGREGATE + if isinstance(result, EvaluationSummary) + else DisclosureLevel.NONE + ) + ) + return await self.workspace_context.add_evaluation(record, disclosure) + + def budgets(self) -> dict[str, EvaluationBudget | None]: + ledger = self.optimizer.engine.budget_ledger + return { + definition.evaluation_set.name: ( + ledger.get( + self.optimizer.backend_id, + definition.evaluation_set, + EvaluationPrincipal.AGENT, + ) + if ledger is not None + else None + ) + for definition in self.optimizer.evaluation_plan.evaluations + if definition.access.agent_can_evaluate + } + + +@dataclass +class Optimizer: + """Schedule proposal, production, evaluation, and selection rounds.""" + + workspace: Workspace + candidate_repository: CandidateRepository + engine: EvaluationEngine + backend_id: str + evaluation_plan: EvaluationPlan + objective: ObjectiveSpec + strategy: OptimizationStrategy + producers: dict[str, CandidateProducer] + selection: SelectionPolicy = field(default_factory=ObjectiveSelectionPolicy) + generation_backend: GenerationBackend | None = None + parameters: dict[str, JsonValue] = field(default_factory=dict) + limits: EvaluationLimits = field(default_factory=EvaluationLimits) + seed: int | None = None + max_proposals: int = 1 + max_rounds: int = 100 + max_concurrency: int = 1 + session_id: str | None = None + _context_ledger: AgentDisclosureLedger | None = field( + default=None, + init=False, + repr=False, + ) + _producer_locks: dict[str, asyncio.Lock] = field( + default_factory=dict, + init=False, + repr=False, + ) + + def _best(self, records: list[EvaluationRecord]) -> EvaluationRecord | None: + selection_set = self.evaluation_plan.selection.evaluation_set + compatible = [ + record + for record in records + if record.request.evaluation_set == selection_set + ] + return self.selection.select(compatible, self.objective) + + def _request( + self, + candidate: Candidate, + evaluation_set: EvaluationSet | None = None, + ) -> EvaluationRequest: + return EvaluationRequest( + candidate=candidate, + evaluation_set=evaluation_set or self.evaluation_plan.selection.evaluation_set, + parameters=self.parameters, + limits=self.limits, + seed=self.seed, + ) + + async def _capture_candidate( + self, + candidate: Candidate, + workspace: Workspace, + ) -> Candidate: + return await self.candidate_repository.capture(candidate, workspace) + + async def evaluate_candidate( + self, + candidate: Candidate, + evaluation_set: EvaluationSet | None = None, + *, + principal: EvaluationPrincipal = EvaluationPrincipal.SYSTEM, + ) -> EvaluationRecord: + return await self.engine.evaluate_record( + backend_id=self.backend_id, + request=self._request(candidate, evaluation_set), + objective_spec=self.objective, + principal=principal, + ) + + @staticmethod + def _workspace_name(proposal: CandidateProposal) -> str: + digest = hashlib.sha256(proposal.id.encode()).hexdigest()[:12] + return f"vero-candidate-{digest}" + + async def _produce_candidate( + self, + *, + proposal: CandidateProposal, + context: OptimizationContext, + parent: Candidate, + evaluation_records: tuple[EvaluationRecord, ...], + ) -> GenerationOutcome: + try: + producer = self.producers[proposal.producer_id] + except KeyError as error: + raise ValueError( + f"unknown candidate producer: {proposal.producer_id!r}" + ) from error + + async with self.candidate_repository.checkout( + parent, + sandbox=self.workspace.sandbox, + name=self._workspace_name(proposal), + ) as candidate_workspace: + if proposal.parent_id is None: + proposal = proposal.model_copy(update={"parent_id": parent.id}) + before = await candidate_workspace.current_version() + if before != parent.version: + raise ValueError( + f"candidate workspace is at {before!r}, " + f"expected parent {parent.version!r}" + ) + if self._context_ledger is None: + from vero.runtime.context import AgentDisclosureLedger + + self._context_ledger = AgentDisclosureLedger( + self.engine.evaluator.session_dir / "agent-context.json" + ) + from vero.runtime.context import WorkspaceContextManager + + workspace_context = WorkspaceContextManager( + session_id=context.session_id, + session_dir=self.engine.evaluator.session_dir, + round_number=context.round, + proposal_id=proposal.id, + parent=parent, + workspace=candidate_workspace, + candidate_repository=self.candidate_repository, + engine=self.engine, + backend_id=self.backend_id, + evaluation_plan=self.evaluation_plan, + candidates=tuple(context.candidates.values()), + evaluations=evaluation_records, + ledger=self._context_ledger, + ) + await workspace_context.initialize() + evaluation = _ScopedEvaluationGateway( + optimizer=self, + proposal=proposal, + parent=parent, + workspace=candidate_workspace, + workspace_context=workspace_context, + round_number=context.round, + ) + # A producer often owns mutable conversation state. Different producer + # IDs may run concurrently, but one producer is deliberately + # non-reentrant so parallel proposals cannot corrupt that state. + producer_lock = self._producer_locks.setdefault( + proposal.producer_id, + asyncio.Lock(), + ) + production_context = CandidateProductionContext( + session_id=context.session_id, + round=context.round, + baseline=context.baseline, + candidates=context.candidates, + best=( + context.best if context.best is not None else None + ), + ) + async with producer_lock: + change = await producer.produce( + proposal=proposal, + context=production_context, + workspace=candidate_workspace, + evaluation=evaluation, + ) + if change is None: + return GenerationOutcome( + candidate=None, + trial_candidates=evaluation.trial_candidates, + trial_evaluations=evaluation.trial_evaluations, + ) + version = ( + await candidate_workspace.save(change.description) + if await candidate_workspace.is_dirty() + else await candidate_workspace.current_version() + ) + if version == parent.version and not evaluation.trial_candidates: + return GenerationOutcome( + candidate=None, + trial_candidates=(), + trial_evaluations=(), + ) + if version == evaluation.last_candidate_version: + return GenerationOutcome( + candidate=None, + trial_candidates=evaluation.trial_candidates, + trial_evaluations=evaluation.trial_evaluations, + ) + metadata = dict(proposal.metadata) + metadata.update(change.metadata) + metadata["producer_id"] = proposal.producer_id + metadata["proposal_id"] = proposal.id + metadata["round"] = context.round + candidate = Candidate( + id=proposal.id, + version=version, + parent_id=evaluation.last_candidate_id, + created_at=datetime.now(UTC), + description=change.description, + metadata=metadata, + ) + await self._capture_candidate(candidate, candidate_workspace) + return GenerationOutcome( + candidate=candidate, + trial_candidates=evaluation.trial_candidates, + trial_evaluations=evaluation.trial_evaluations, + ) + + async def run( + self, + *, + baseline: Candidate | None = None, + skip_baseline_evaluation: bool = False, + max_proposals: int | None = None, + ) -> OptimizationResult: + proposal_limit = self.max_proposals if max_proposals is None else max_proposals + # Check the session-level limit first: otherwise a negative + # self.max_proposals always trips the proposal_limit guard below and its + # specific message is unreachable. + if self.max_proposals < 0: + raise ValueError("max_proposals must be non-negative") + if proposal_limit < 0 or proposal_limit > self.max_proposals: + raise ValueError( + "run max_proposals must be between zero and the session protocol limit" + ) + if self.max_rounds < 1: + raise ValueError("max_rounds must be positive") + if self.max_concurrency < 1: + raise ValueError("max_concurrency must be positive") + if not self.producers and proposal_limit: + raise ValueError("at least one candidate producer is required") + + if baseline is None: + version = await self.workspace.current_version() + baseline = Candidate.from_version(version) + stored_baseline = self.candidate_repository.get(baseline.id) + if stored_baseline is None: + baseline = await self._capture_candidate(baseline, self.workspace) + elif stored_baseline != baseline: + raise ValueError("baseline does not match its durable candidate record") + + backend_provenance = self.engine.backends.resolve(self.backend_id).provenance + selection_set = self.evaluation_plan.selection.evaluation_set + existing_baselines = [ + record + for record in self.engine.database.evaluations.values() + if record.request.candidate.id == baseline.id + and record.request.candidate.version == baseline.version + and record.backend_id == self.backend_id + and record.request.evaluation_set == selection_set + and record.request.parameters == self.parameters + and record.request.limits == self.limits + and record.request.seed == self.seed + and record.backend == backend_provenance + and record.objective_spec == self.objective + ] + if skip_baseline_evaluation: + if not existing_baselines: + raise ValueError( + "skip_baseline_evaluation requires an existing compatible baseline" + ) + baseline_record = existing_baselines[-1] + else: + baseline_record = await self.evaluate_candidate( + baseline, + selection_set, + principal=EvaluationPrincipal.SYSTEM, + ) + + final_baseline: EvaluationRecord | None = None + final_definition = self.evaluation_plan.final + if final_definition is not None and self.evaluation_plan.evaluate_final_baseline: + final_set = final_definition.evaluation_set + existing_final_baselines = [ + record + for record in self.engine.database.evaluations.values() + if record.request.candidate.id == baseline.id + and record.request.candidate.version == baseline.version + and record.backend_id == self.backend_id + and record.request.evaluation_set == final_set + and record.request.parameters == self.parameters + and record.request.limits == self.limits + and record.request.seed == self.seed + and record.backend == backend_provenance + and record.objective_spec == self.objective + ] + if skip_baseline_evaluation and existing_final_baselines: + final_baseline = existing_final_baselines[-1] + else: + final_baseline = await self.evaluate_candidate( + baseline, + final_set, + principal=EvaluationPrincipal.SYSTEM, + ) + + compatible = [ + record + for record in self.engine.database.evaluations.values() + if record.backend_id == self.backend_id + and self.evaluation_plan.for_evaluation_set( + record.request.evaluation_set + ) + is not None + and record.request.parameters == self.parameters + and record.request.limits == self.limits + and record.request.seed == self.seed + and record.backend == backend_provenance + and record.objective_spec == self.objective + ] + candidate_records = { + candidate.id: candidate for candidate in self.candidate_repository.list() + } + reachable = {baseline.id} + changed = True + while changed: + changed = False + for candidate in candidate_records.values(): + if candidate.id in reachable: + continue + if candidate.parent_id in reachable: + reachable.add(candidate.id) + changed = True + evaluations = [ + record for record in compatible if record.request.candidate.id in reachable + ] + if baseline_record not in evaluations: + evaluations.insert(0, baseline_record) + evaluations.sort(key=lambda record: (record.completed_at, record.id)) + candidates: dict[str, Candidate] = { + candidate_id: candidate + for candidate_id, candidate in candidate_records.items() + if candidate_id in reachable + } + candidates[baseline.id] = baseline + proposal_ids = { + str(candidate.metadata.get("proposal_id", candidate.id)) + for candidate in candidates.values() + if candidate.id != baseline.id and "producer_id" in candidate.metadata + } + generated = len(proposal_ids) + completed_rounds = [ + int(candidate.metadata["round"]) + for candidate in candidates.values() + if isinstance(candidate.metadata.get("round"), int) + ] + start_round = max(completed_rounds, default=generated - 1) + 1 + semaphore = asyncio.Semaphore(self.max_concurrency) + + evaluated_candidate_ids = { + record.request.candidate.id for record in evaluations + if record.request.evaluation_set == selection_set + } + pending = [ + candidate + for candidate in candidates.values() + if candidate.id != baseline.id + and candidate.id not in evaluated_candidate_ids + and "producer_id" in candidate.metadata + and "trial" not in candidate.metadata + ] + pending.sort( + key=lambda candidate: ( + int(candidate.metadata.get("round", 0)), + int(candidate.metadata.get("trial", 0)), + candidate.created_at, + candidate.id, + ) + ) + + async def evaluate(candidate: Candidate) -> EvaluationRecord: + async with semaphore: + return await self.evaluate_candidate( + candidate, + selection_set, + principal=EvaluationPrincipal.SYSTEM, + ) + + if pending: + async with asyncio.TaskGroup() as group: + pending_tasks = [ + group.create_task(evaluate(candidate)) for candidate in pending + ] + evaluations.extend(task.result() for task in pending_tasks) + + for round_number in range(start_round, self.max_rounds): + if generated >= proposal_limit: + break + best = self._best(evaluations) + visible_evaluations = tuple( + record + for record in evaluations + if ( + definition := self.evaluation_plan.for_evaluation_set( + record.request.evaluation_set + ) + ) + is not None + and definition.access.agent_visible + ) + evaluation_views = tuple( + project_evaluation( + record, + self.evaluation_plan.for_evaluation_set( + record.request.evaluation_set + ).access.disclosure, + ) + for record in visible_evaluations + ) + context = OptimizationContext( + session_id=self.session_id or self.engine.evaluator.session_dir.name, + round=round_number, + workspace=self.workspace, + baseline=baseline, + evaluations=evaluation_views, + candidates=dict(candidates), + best=(best.request.candidate if best is not None else None), + ) + proposals = list(await self.strategy.propose(context)) + if not proposals: + break + remaining = proposal_limit - generated + proposals = proposals[:remaining] + proposal_ids = [proposal.id for proposal in proposals] + if len(proposal_ids) != len(set(proposal_ids)): + raise ValueError("strategy returned duplicate proposal IDs") + if any(proposal_id in candidates for proposal_id in proposal_ids): + raise ValueError("strategy reused an existing candidate ID") + + async def produce(proposal: CandidateProposal) -> GenerationOutcome: + parent_id = proposal.parent_id or ( + best.request.candidate.id if best is not None else baseline.id + ) + try: + parent = candidates[parent_id] + except KeyError as error: + raise ValueError( + f"proposal {proposal.id!r} names unknown parent {parent_id!r}" + ) from error + async with semaphore: + if self.generation_backend is not None: + return await self.generation_backend.generate( + proposal=proposal, + context=context, + parent=parent, + evaluation_records=visible_evaluations, + ) + return await self._produce_candidate( + proposal=proposal, + context=context, + parent=parent, + evaluation_records=visible_evaluations, + ) + + async with asyncio.TaskGroup() as group: + production_tasks = [ + group.create_task(produce(proposal)) for proposal in proposals + ] + outcomes = [task.result() for task in production_tasks] + meaningful_outcomes = [ + outcome for outcome in outcomes if outcome.candidates + ] + if not meaningful_outcomes: + break + generated += len(meaningful_outcomes) + for outcome in meaningful_outcomes: + evaluations.extend(outcome.trial_evaluations) + for candidate in outcome.candidates: + if candidate.id in candidates: + raise ValueError( + f"candidate producer reused candidate ID {candidate.id!r}" + ) + candidates[candidate.id] = candidate + + produced = [ + outcome.candidate + for outcome in meaningful_outcomes + if outcome.candidate is not None + ] + + async with asyncio.TaskGroup() as group: + evaluation_tasks = [ + group.create_task(evaluate(candidate)) for candidate in produced + ] + evaluations.extend(task.result() for task in evaluation_tasks) + + best = self._best(evaluations) + final_record: EvaluationRecord | None = None + if final_definition is not None and best is not None: + if ( + final_baseline is not None + and best.request.candidate.id == baseline.id + and best.request.candidate.version == baseline.version + ): + final_record = final_baseline + else: + final_set = final_definition.evaluation_set + existing_final = [ + record + for record in self.engine.database.evaluations.values() + if record.request.candidate.id == best.request.candidate.id + and record.request.candidate.version + == best.request.candidate.version + and record.backend_id == self.backend_id + and record.request.evaluation_set == final_set + and record.request.parameters == self.parameters + and record.request.limits == self.limits + and record.request.seed == self.seed + and record.backend == backend_provenance + and record.objective_spec == self.objective + # Only reuse a genuine, completed final result produced by the + # same trusted path: never a failed/cancelled/invalid record, + # and never a non-SYSTEM (e.g. agent-visible) evaluation, so a + # resume can't be marked complete on a broken final. + and record.report.status == EvaluationStatus.SUCCESS + and record.principal == EvaluationPrincipal.SYSTEM + ] + if existing_final: + # On resume (or a re-run of a completed session) the winner + # may already have a compatible final evaluation. Reuse it + # rather than repeating a hidden-test run and spending the + # system/inference budget an exact allocation may not have. + final_record = existing_final[-1] + else: + final_record = await self.evaluate_candidate( + best.request.candidate, + final_set, + principal=EvaluationPrincipal.SYSTEM, + ) + evaluations.append(final_record) + + return OptimizationResult( + baseline=baseline_record, + evaluations=tuple(evaluations), + candidates=tuple(candidates.values()), + best=best, + final_baseline=final_baseline, + final=final_record, + ) diff --git a/vero/src/vero/optimization/protocols.py b/vero/src/vero/optimization/protocols.py new file mode 100644 index 00000000..48312963 --- /dev/null +++ b/vero/src/vero/optimization/protocols.py @@ -0,0 +1,90 @@ +"""Extension protocols for optimization strategies and candidate producers.""" + +from __future__ import annotations + +from typing import Mapping, Protocol, Sequence, runtime_checkable + +from vero.candidate import Candidate +from vero.evaluation import ( + CaseSelection, + EvaluationBudget, + EvaluationReceipt, + EvaluationRecord, + ObjectiveSpec, +) +from vero.optimization.models import ( + CandidateChange, + CandidateProductionContext, + CandidateProposal, + GenerationOutcome, + OptimizationContext, +) +from vero.workspace import Workspace + + +@runtime_checkable +class OptimizationStrategy(Protocol): + async def propose( + self, + context: OptimizationContext, + ) -> Sequence[CandidateProposal]: ... + + +@runtime_checkable +class CandidateEvaluationGateway(Protocol): + """Evaluation capability scoped to one producer workspace.""" + + async def evaluate( + self, + *, + evaluation: str, + selection: CaseSelection | None = None, + candidate_id: str | None = None, + description: str = "Evaluate agent checkpoint", + ) -> EvaluationReceipt: ... + + def budgets(self) -> Mapping[str, EvaluationBudget | None]: ... + + +@runtime_checkable +class CandidateProducer(Protocol): + async def produce( + self, + *, + proposal: CandidateProposal, + context: CandidateProductionContext, + workspace: Workspace, + evaluation: CandidateEvaluationGateway, + ) -> CandidateChange | None: ... + + +@runtime_checkable +class GenerationBackend(Protocol): + """Turn a parent + proposal into candidate(s) plus generation-time feedback. + + This is the swappable unit the Optimizer drives. The default is the + Optimizer's built-in native production (an agent or operator editing a + checkout in a sandbox, with mid-run self-eval). A Harbor implementation + delegates to a ``harbor run`` instead. Either way, the orchestrator performs + selection and target scoring *separately* on the returned candidate; the + ``GenerationOutcome.trial_evaluations`` are only the generation-time feedback + the producer observed while iterating. + """ + + async def generate( + self, + *, + proposal: CandidateProposal, + parent: Candidate, + context: OptimizationContext, + evaluation_records: Sequence[EvaluationRecord], + ) -> GenerationOutcome: ... + + +@runtime_checkable +class SelectionPolicy(Protocol): + def select( + self, + records: Sequence[EvaluationRecord], + objective: ObjectiveSpec, + ) -> EvaluationRecord | None: ... diff --git a/vero/src/vero/optimization/strategy.py b/vero/src/vero/optimization/strategy.py new file mode 100644 index 00000000..4cca44da --- /dev/null +++ b/vero/src/vero/optimization/strategy.py @@ -0,0 +1,329 @@ +"""Built-in optimization and selection strategies.""" + +from __future__ import annotations + +import random +from collections.abc import Sequence + +from vero.evaluation import ( + EvaluationRecord, + EvaluationSummary, + ObjectiveResult, + ObjectiveSpec, + select_best_evaluation, +) +from vero.optimization.models import CandidateProposal, OptimizationContext + + +class SequentialStrategy: + """Request one candidate from the same producer on every round.""" + + def __init__( + self, + *, + producer_id: str = "default", + instruction: str | None = None, + ): + self.producer_id = producer_id + self.instruction = instruction + + @property + def config(self) -> dict[str, object]: + """Identity-relevant settings, folded into the resume digest.""" + return {"producer_id": self.producer_id, "instruction": self.instruction} + + async def propose(self, context: OptimizationContext) -> Sequence[CandidateProposal]: + parent_id = ( + context.best.id + if context.best is not None + else context.baseline.id + ) + return [ + CandidateProposal( + producer_id=self.producer_id, + parent_id=parent_id, + instruction=self.instruction, + ) + ] + + +def _record_objective( + view: object, +) -> tuple[str, str | None, ObjectiveResult] | None: + """Extract (candidate_id, evaluation_set_name, objective) from a projected view. + + Handles full ``EvaluationRecord`` and aggregate ``EvaluationSummary`` views; + returns ``None`` for target/none views (``EvaluationAcknowledgement``) that + carry no objective — so the strategy never sees held-out scores. + """ + if isinstance(view, EvaluationRecord): + if view.objective is None: + return None + return view.request.candidate.id, view.request.evaluation_set.name, view.objective + if isinstance(view, EvaluationSummary): + if view.objective is None: + return None + return view.candidate_id, view.evaluation_set.name, view.objective + return None + + +class EvolutionaryStrategy: + """A population-based strategy: select parents, emit N mutated offspring. + + Each round it ingests the disclosed evaluations into a fitness archive + (direction-aware via ``objective``), keeps the fittest ``population_size`` + candidates, and emits ``num_offspring`` proposals whose parents are chosen by + tournament selection. Before any feasible candidate exists it seeds offspring + from the current best (or the baseline). This is the native runner's + distinctive capability over Harbor's single-agent-as-optimizer. + + Only mutation (parent + instruction) is implemented; crossover (combining two + parents) is a follow-on — it needs the producer/instruction to reference a + second candidate's contents, exposed through the read-only ``.evals`` tree. + + ``evaluation_set`` should name the selection set when multiple sets are + visible, so fitness is ranked on one consistent metric; left ``None`` it + ranks on all disclosed evaluations (correct for single-set setups). + """ + + def __init__( + self, + *, + objective: ObjectiveSpec, + producer_id: str = "default", + population_size: int = 8, + num_offspring: int = 4, + tournament_size: int = 3, + instruction: str | None = None, + evaluation_set: str | None = None, + seed: int | None = None, + ): + if population_size < 1: + raise ValueError("population_size must be >= 1") + if num_offspring < 1: + raise ValueError("num_offspring must be >= 1") + if tournament_size < 1: + raise ValueError("tournament_size must be >= 1") + self.objective = objective + self.producer_id = producer_id + self.population_size = population_size + self.num_offspring = num_offspring + self.tournament_size = tournament_size + self.instruction = instruction + self.evaluation_set = evaluation_set + self.seed = seed + self._rng = random.Random(seed) + self._fitness: dict[str, float] = {} + self._generation = 0 + + @property + def config(self) -> dict[str, object]: + """Every setting that changes search semantics, so the resume digest + rejects a resume under a materially different configuration.""" + return { + "producer_id": self.producer_id, + "population_size": self.population_size, + "num_offspring": self.num_offspring, + "tournament_size": self.tournament_size, + "instruction": self.instruction, + "evaluation_set": self.evaluation_set, + "seed": self.seed, + } + + async def propose(self, context: OptimizationContext) -> Sequence[CandidateProposal]: + self._ingest(context) + parents = self._select_parents(context) + proposals = [ + CandidateProposal( + producer_id=self.producer_id, + parent_id=parent_id, + instruction=self.instruction, + metadata={ + "strategy": "evolutionary", + "generation": self._generation, + "parent_id": parent_id, + }, + ) + for parent_id in parents + ] + self._generation += 1 + return proposals + + def _to_fitness(self, objective: ObjectiveResult) -> float: + """Map an objective result to an internal fitness where higher is better.""" + if not objective.feasible or objective.value is None: + return float("-inf") + return ( + objective.value + if self.objective.direction == "maximize" + else -objective.value + ) + + def _ingest(self, context: OptimizationContext) -> None: + for view in context.evaluations: + parsed = _record_objective(view) + if parsed is None: + continue + candidate_id, set_name, objective = parsed + if self.evaluation_set is not None and set_name != self.evaluation_set: + continue + fitness = self._to_fitness(objective) + previous = self._fitness.get(candidate_id) + if previous is None or fitness > previous: + self._fitness[candidate_id] = fitness + + def _population(self, context: OptimizationContext) -> list[str]: + scored = [ + (candidate_id, fitness) + for candidate_id, fitness in self._fitness.items() + if candidate_id in context.candidates and fitness > float("-inf") + ] + scored.sort(key=lambda item: item[1], reverse=True) + return [candidate_id for candidate_id, _ in scored[: self.population_size]] + + def _select_parents(self, context: OptimizationContext) -> list[str]: + population = self._population(context) + if not population: + seed_parent = ( + context.best.id if context.best is not None else context.baseline.id + ) + return [seed_parent] * self.num_offspring + parents: list[str] = [] + for _ in range(self.num_offspring): + k = min(self.tournament_size, len(population)) + contenders = self._rng.sample(population, k) + parents.append(max(contenders, key=lambda cid: self._fitness[cid])) + return parents + + +class DarwinGodelStrategy: + """Open-ended archive evolution in the style of the Darwin Gödel Machine. + + Where ``EvolutionaryStrategy`` runs a tournament over a fittest-``K`` + population, DGM keeps the **entire archive** as candidate parents and samples + parent ``p`` with probability proportional to ``score(p) / (1 + children(p))`` + — every archived agent keeps a non-zero probability, so lower-performing + nodes remain reachable as *stepping stones* (many innovations traverse them). + + The intended pairing is a self-application producer: each offspring is the + parent's own harness modifying itself (analyze its ``.evals`` trace corpus, + implement one improvement), which is the self-referential loop the original + Gödel Machine wanted but validated empirically rather than by proof. + + ``base_weight`` is the floor performance given to not-yet-scored or infeasible + agents so brand-new lineages are still explorable. ``evaluation_set`` names + the selection set to rank on when several are visible (held-out target scores + never appear in ``context.evaluations``, so they cannot leak into selection). + """ + + def __init__( + self, + *, + objective: ObjectiveSpec, + producer_id: str = "self", + num_offspring: int = 2, + instruction: str | None = None, + evaluation_set: str | None = None, + base_weight: float = 0.2, + seed: int | None = None, + ): + if num_offspring < 1: + raise ValueError("num_offspring must be >= 1") + if base_weight <= 0: + raise ValueError("base_weight must be > 0 so every agent stays reachable") + self.objective = objective + self.producer_id = producer_id + self.num_offspring = num_offspring + self.instruction = instruction + self.evaluation_set = evaluation_set + self.base_weight = base_weight + self.seed = seed + self._rng = random.Random(seed) + self._score: dict[str, float] = {} + self._children: dict[str, int] = {} + self._generation = 0 + + @property + def config(self) -> dict[str, object]: + """Every setting that changes search semantics, so the resume digest + rejects a resume under a materially different configuration.""" + return { + "producer_id": self.producer_id, + "num_offspring": self.num_offspring, + "instruction": self.instruction, + "evaluation_set": self.evaluation_set, + "base_weight": self.base_weight, + "seed": self.seed, + } + + async def propose(self, context: OptimizationContext) -> Sequence[CandidateProposal]: + self._ingest(context) + parents = self._select_parents(context) + proposals = [ + CandidateProposal( + producer_id=self.producer_id, + parent_id=parent_id, + instruction=self.instruction, + metadata={ + "strategy": "darwin-godel", + "generation": self._generation, + "parent_id": parent_id, + }, + ) + for parent_id in parents + ] + for parent_id in parents: + self._children[parent_id] = self._children.get(parent_id, 0) + 1 + self._generation += 1 + return proposals + + def _ingest(self, context: OptimizationContext) -> None: + """Record each candidate's best feasible objective as its performance.""" + for view in context.evaluations: + parsed = _record_objective(view) + if parsed is None: + continue + candidate_id, set_name, objective = parsed + if self.evaluation_set is not None and set_name != self.evaluation_set: + continue + if not objective.feasible or objective.value is None: + continue + quality = ( + objective.value + if self.objective.direction == "maximize" + else -objective.value + ) + previous = self._score.get(candidate_id) + if previous is None or quality > previous: + self._score[candidate_id] = quality + + def _weight(self, candidate_id: str) -> float: + """DGM selection weight: performance, damped by how much it's been explored.""" + performance = self._score.get(candidate_id) + quality = self.base_weight if performance is None else max(self.base_weight, performance) + return quality / (1 + self._children.get(candidate_id, 0)) + + def _select_parents(self, context: OptimizationContext) -> list[str]: + archive = list(context.candidates) + if not archive: + seed_parent = ( + context.best.id if context.best is not None else context.baseline.id + ) + return [seed_parent] * self.num_offspring + weights = [self._weight(candidate_id) for candidate_id in archive] + if sum(weights) <= 0: + weights = [1.0] * len(archive) + # sample WITH replacement: a strong lineage may spawn several children a round + return self._rng.choices(archive, weights=weights, k=self.num_offspring) + + +class ObjectiveSelectionPolicy: + """Select the best feasible value of the configured objective.""" + + def select( + self, + records: Sequence[EvaluationRecord], + objective: ObjectiveSpec, + ) -> EvaluationRecord | None: + compatible = [record for record in records if record.objective_spec == objective] + return select_best_evaluation(compatible) diff --git a/vero/src/vero/report.py b/vero/src/vero/report.py new file mode 100644 index 00000000..59c4fc34 --- /dev/null +++ b/vero/src/vero/report.py @@ -0,0 +1,529 @@ +"""Self-contained, read-only reports for durable optimization sessions.""" + +from __future__ import annotations + +import base64 +import hashlib +import importlib.resources +import json +import mimetypes +import subprocess +from pathlib import Path +from typing import Any + +from vero.candidate import Candidate +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import EvaluationDatabase +from vero.runtime.events import RuntimeEvent +from vero.runtime.session import SessionManifest +from vero.sidecar.session import HarborSessionManifest +from vero.sidecar.verifier import VerificationResult + +_MAX_EMBEDDED_ARTIFACT_BYTES = 5_000_000 +_MAX_EMBEDDED_ARTIFACTS_BYTES = 50_000_000 +_MAX_DIFF_CHARACTERS = 500_000 + + +def _read_events(path: Path) -> list[dict[str, Any]]: + if not path.is_file(): + return [] + events: list[dict[str, Any]] = [] + for line_number, line in enumerate( + path.read_text(encoding="utf-8").splitlines(), 1 + ): + if not line.strip(): + continue + try: + event = RuntimeEvent.model_validate_json(line) + except Exception as error: + raise ValueError( + f"invalid runtime event on line {line_number}: {error}" + ) from error + events.append(event.model_dump(mode="json")) + return events + + +def _git_diff( + repository_path: Path, + candidate: Candidate, + parent: Candidate | None, + *, + project_subpath: str, +) -> dict[str, Any]: + if parent is None: + arguments = [ + "show", + "--format=", + "--no-ext-diff", + "--no-color", + candidate.version, + ] + label = "Initial program" + else: + arguments = [ + "diff", + "--no-ext-diff", + "--no-color", + parent.version, + candidate.version, + ] + label = f"Changes from {parent.id}" + if project_subpath != ".": + arguments.extend(["--", project_subpath]) + result = subprocess.run( + ["git", "--git-dir", str(repository_path), *arguments], + check=False, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + env={"PATH": "/usr/bin:/bin:/usr/sbin:/sbin", "LANG": "C.UTF-8"}, + ) + if result.returncode != 0: + return { + "label": label, + "text": "", + "error": result.stderr.strip() or "Git could not render this diff.", + "truncated": False, + } + text = result.stdout + truncated = len(text) > _MAX_DIFF_CHARACTERS + if truncated: + text = text[:_MAX_DIFF_CHARACTERS] + return {"label": label, "text": text, "error": None, "truncated": truncated} + + +def _embed_artifact( + path: Path, + *, + media_type: str | None, + description: str | None, + relative_path: str, + remaining_bytes: int, +) -> tuple[dict[str, Any], int]: + resolved_media_type = media_type or mimetypes.guess_type(path.name)[0] + artifact: dict[str, Any] = { + "path": relative_path, + "media_type": resolved_media_type, + "description": description, + "exists": path.is_file(), + "size": path.stat().st_size if path.is_file() else None, + "kind": "missing", + "content": None, + "omitted_reason": None, + } + if not path.is_file(): + artifact["omitted_reason"] = "Artifact file is missing." + return artifact, 0 + size = path.stat().st_size + if size > _MAX_EMBEDDED_ARTIFACT_BYTES: + artifact["kind"] = "omitted" + artifact["omitted_reason"] = ( + f"Artifact is larger than {_MAX_EMBEDDED_ARTIFACT_BYTES:,} bytes." + ) + return artifact, 0 + if size > remaining_bytes: + artifact["kind"] = "omitted" + artifact["omitted_reason"] = "The report's embedded-artifact limit was reached." + return artifact, 0 + + payload = path.read_bytes() + if resolved_media_type and ( + resolved_media_type.startswith("image/") + or resolved_media_type == "application/pdf" + ): + artifact["kind"] = ( + "image" if resolved_media_type.startswith("image/") else "binary" + ) + if artifact["kind"] == "image": + encoded = base64.b64encode(payload).decode("ascii") + artifact["content"] = f"data:{resolved_media_type};base64,{encoded}" + else: + artifact["omitted_reason"] = "Binary preview is not supported." + elif ( + resolved_media_type is None + or resolved_media_type.startswith("text/") + or resolved_media_type in {"application/json", "application/xml"} + ): + artifact["kind"] = "text" + artifact["content"] = payload.decode("utf-8", errors="replace") + else: + artifact["kind"] = "binary" + artifact["omitted_reason"] = "Binary preview is not supported." + return artifact, size + + +def _trace_entries(value: Any) -> list[dict[str, Any]]: + values = value if isinstance(value, list) else [value] + entries: list[dict[str, Any]] = [] + for item in values: + if not isinstance(item, dict): + entries.append({"kind": "event", "title": "Event", "body": item}) + continue + item_type = item.get("type") + role = item.get("role") + if item_type == "function_call": + entries.append( + { + "kind": "tool-call", + "title": str(item.get("name") or "Tool call"), + "body": item.get("arguments"), + } + ) + elif item_type == "function_call_output": + entries.append( + { + "kind": "tool-result", + "title": "Tool result", + "body": item.get("output"), + } + ) + elif role in {"user", "assistant", "system", "developer"}: + entries.append( + { + "kind": str(role), + "title": str(role).capitalize(), + "body": item.get("content"), + } + ) + else: + entries.append( + { + "kind": str(item_type or "event"), + "title": str(item_type or "Event").replace("_", " ").title(), + "body": item, + } + ) + return entries + + +def _read_traces(session_dir: Path) -> list[dict[str, Any]]: + root = session_dir / "artifacts" / "agents" + if not root.is_dir(): + return [] + traces: list[dict[str, Any]] = [] + for directory in sorted(path for path in root.iterdir() if path.is_dir()): + trace_path = directory / "trace.json" + if directory.name == "producers" or not trace_path.is_file(): + continue + try: + raw = json.loads(trace_path.read_text(encoding="utf-8")) + except Exception as error: + raw = {"error": f"Could not parse trace: {error}"} + failure_path = directory / "failure.json" + failure = None + if failure_path.is_file(): + try: + failure = json.loads(failure_path.read_text(encoding="utf-8")) + except Exception as error: + failure = {"message": f"Could not parse failure: {error}"} + traces.append( + { + "id": directory.name, + "entries": _trace_entries(raw), + "failure": failure, + "path": str(trace_path.relative_to(session_dir)), + } + ) + return traces + + +def _load_database(session_dir: Path, database_id: str) -> EvaluationDatabase: + database_path = session_dir / "database.json" + database = ( + EvaluationDatabase.load_from_file(database_path) + if database_path.is_file() + else EvaluationDatabase(id=database_id) + ) + if database.id != database_id: + raise ValueError( + f"evaluation database belongs to {database.id!r}, not {database_id!r}" + ) + completed = EvaluationDatabase.from_evaluations_dir( + session_dir / "evaluations", database_id=database_id + ) + for record in completed.evaluations.values(): + if record.id not in database.evaluations: + database.add_evaluation(record) + return database + + +async def _build_report_data( + session_dir: Path, + *, + manifest: dict[str, Any], + database: EvaluationDatabase, + default_trace_id: str | None = None, +) -> dict[str, Any]: + evaluations = sorted( + database.evaluations.values(), + key=lambda record: (record.completed_at, record.id), + ) + + if manifest["candidate_repository_family"] != "git": + candidates = tuple( + sorted( + database.candidates.values(), + key=lambda value: (value.created_at, value.id), + ) + ) + repository = None + else: + repository = await GitCandidateRepository.open(session_dir / "candidates") + candidates = repository.list() + + candidate_by_id = {candidate.id: candidate for candidate in candidates} + traces = _read_traces(session_dir) + trace_ids = {trace["id"] for trace in traces} + candidate_data: list[dict[str, Any]] = [] + for candidate in candidates: + proposal_id = candidate.metadata.get("proposal_id") + trace_id = ( + hashlib.sha256(str(proposal_id).encode()).hexdigest()[:16] + if proposal_id is not None + else default_trace_id + ) + item = candidate.model_dump(mode="json") + item["trace_id"] = trace_id if trace_id in trace_ids else None + if repository is not None: + item["diff"] = _git_diff( + repository.repository_path, + candidate, + candidate_by_id.get(candidate.parent_id) + if candidate.parent_id + else None, + project_subpath=repository.project_subpath, + ) + else: + item["diff"] = { + "label": "Program changes", + "text": "", + "error": "Diffs are unavailable for this candidate repository family.", + "truncated": False, + } + candidate_data.append(item) + + embedded_bytes = 0 + evaluation_data: list[dict[str, Any]] = [] + for step, record in enumerate(evaluations): + references = list(record.report.artifacts) + for case in record.report.cases: + references.extend(case.artifacts) + seen_paths: set[str] = set() + artifacts: list[dict[str, Any]] = [] + for reference in references: + if reference.path in seen_paths: + continue + seen_paths.add(reference.path) + artifact, consumed = _embed_artifact( + session_dir / "evaluations" / record.id / "artifacts" / reference.path, + media_type=reference.media_type, + description=reference.description, + relative_path=reference.path, + remaining_bytes=_MAX_EMBEDDED_ARTIFACTS_BYTES - embedded_bytes, + ) + embedded_bytes += consumed + artifacts.append(artifact) + item = record.model_dump(mode="json") + item["step"] = step + item["artifacts"] = artifacts + evaluation_data.append(item) + + events = _read_events(session_dir / "events.jsonl") + return { + "schema_version": 1, + "generated_from": str(session_dir), + "manifest": manifest, + "candidates": candidate_data, + "evaluations": evaluation_data, + "events": events, + "traces": traces, + } + + +def _harbor_presentation_manifest( + manifest: HarborSessionManifest, + database: EvaluationDatabase, + finalization: VerificationResult | None, +) -> dict[str, Any]: + selection = manifest.selection + selected = finalization.candidate if finalization is not None else None + selection_records = [ + record + for record in database.evaluations.values() + if selected is not None + and selection.backend_id is not None + and selection.evaluation_set is not None + and record.request.candidate.id == selected.id + and record.backend_id == selection.backend_id + and record.request.evaluation_set == selection.evaluation_set + and record.objective is not None + and record.objective.feasible + and record.objective.value is not None + ] + if selection.objective is not None: + selection_records.sort(key=lambda record: record.id) + selection_records.sort( + key=lambda record: record.objective.value, + reverse=selection.objective.direction == "maximize", + ) + best_evaluation_id = selection_records[0].id if selection_records else None + final_evaluation_id = None + if finalization is not None and finalization.evaluation_ids: + final_evaluation_id = next(iter(finalization.evaluation_ids.values())) + completed_at = max( + (record.completed_at for record in database.evaluations.values()), + default=manifest.created_at, + ) + errors = finalization.errors if finalization is not None else {} + objective = selection.objective or manifest.targets[0].objective + backend_id = selection.backend_id or manifest.targets[0].backend_id + selection_set = selection.evaluation_set + return { + "schema_version": 1, + "id": f"{manifest.task_name} · {manifest.id}", + "status": "failed" if finalization is None or errors else "completed", + "backend_id": backend_id, + "candidate_repository_family": manifest.candidate_repository_family, + "candidate_repository_format_version": ( + manifest.candidate_repository_format_version + ), + "evaluation_plan": { + "selection_evaluation": ( + selection_set.name if selection_set is not None else "selection" + ), + "final_evaluation": ( + manifest.targets[0].evaluation_set.name if manifest.targets else None + ), + }, + "selection_evaluation_set": ( + selection_set.model_dump(mode="json") if selection_set is not None else None + ), + "objective": objective.model_dump(mode="json"), + "baseline": ( + selection.baseline_candidate.model_dump(mode="json") + if selection.baseline_candidate is not None + else None + ), + "best_candidate_id": selected.id if selected is not None else None, + "best_evaluation_id": best_evaluation_id, + "final_baseline_evaluation_id": None, + "final_evaluation_id": final_evaluation_id, + "created_at": manifest.created_at.isoformat(), + "updated_at": completed_at.isoformat(), + "failure": ( + {"type": "verification", "message": json.dumps(errors, sort_keys=True)} + if errors + else None + ), + "metadata": { + "task_description": manifest.task_description, + "verification": ( + finalization.model_dump(mode="json") + if finalization is not None + else None + ), + }, + } + + +async def build_experiment_report_data(session_dir: Path | str) -> dict[str, Any]: + """Load a local or Harbor session into the portable report data model.""" + + session_dir = Path(session_dir).expanduser().resolve() + manifest_path = session_dir / "manifest.json" + if manifest_path.is_file(): + parsed = SessionManifest.model_validate_json( + manifest_path.read_text(encoding="utf-8") + ) + manifest = parsed.model_dump(mode="json") + manifest["selection_evaluation_set"] = ( + parsed.evaluation_plan.selection.evaluation_set.model_dump(mode="json") + ) + return await _build_report_data( + session_dir, + manifest=manifest, + database=_load_database(session_dir, parsed.id), + ) + + harbor_path = session_dir / "harbor-session.json" + if not harbor_path.is_file(): + raise FileNotFoundError(f"session manifest not found: {manifest_path}") + harbor = HarborSessionManifest.model_validate_json( + harbor_path.read_text(encoding="utf-8") + ) + database = _load_database(session_dir, harbor.id) + finalization_path = session_dir / "harbor-finalization.json" + finalization = ( + VerificationResult.model_validate_json( + finalization_path.read_text(encoding="utf-8") + ) + if finalization_path.is_file() + else None + ) + traces = _read_traces(session_dir) + data = await _build_report_data( + session_dir, + manifest=_harbor_presentation_manifest(harbor, database, finalization), + database=database, + default_trace_id=traces[0]["id"] if traces else None, + ) + if not data["events"]: + data["events"] = [ + { + "created_at": evaluation["completed_at"], + "kind": "evaluation.completed", + "payload": { + "evaluation_id": evaluation["id"], + "candidate_id": evaluation["request"]["candidate"]["id"], + "backend_id": evaluation["backend_id"], + "evaluation_set": evaluation["request"]["evaluation_set"], + "status": evaluation["report"]["status"], + "objective": evaluation["objective"], + }, + } + for evaluation in data["evaluations"] + ] + if finalization is not None: + data["events"].append( + { + "created_at": data["manifest"]["updated_at"], + "kind": "verification.completed", + "payload": finalization.model_dump(mode="json"), + } + ) + return data + + +def _safe_json(value: Any) -> str: + return ( + json.dumps(value, ensure_ascii=False, separators=(",", ":")) + .replace("<", "\\u003c") + .replace(">", "\\u003e") + .replace("&", "\\u0026") + ) + + +async def generate_experiment_report( + session_dir: Path | str, + output: Path | str | None = None, +) -> Path: + """Generate one portable HTML report without modifying the session.""" + + resolved_session = Path(session_dir).expanduser().resolve() + destination = ( + Path(output).expanduser().resolve() + if output is not None + else resolved_session / "experiment.html" + ) + data = await build_experiment_report_data(resolved_session) + template = ( + importlib.resources.files("vero") + .joinpath("templates/report.html") + .read_text(encoding="utf-8") + ) + html = template.replace("__VERO_REPORT_DATA__", _safe_json(data)) + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_text(html, encoding="utf-8") + return destination + diff --git a/vero/src/vero/runtime/__init__.py b/vero/src/vero/runtime/__init__.py new file mode 100644 index 00000000..18e519f6 --- /dev/null +++ b/vero/src/vero/runtime/__init__.py @@ -0,0 +1,59 @@ +"""Durable runtime state for optimization sessions.""" + +from vero.runtime.artifacts import ArtifactStore +from vero.runtime.context import ( + AGENT_CONTEXT_DIRECTORY, + AgentContextDirectory, + AgentDisclosureLedger, + WorkspaceContextManager, + evaluation_result_path, + make_evaluation_receipt, + narrower_disclosure, +) +from vero.runtime.events import ( + EventBus, + EventSink, + JsonlEventSink, + RuntimeEvent, + agent_event_emitter, +) +from vero.runtime.factory import ( + create_local_optimization_session, + create_optimization_session, +) +from vero.runtime.session import ( + OptimizationComponentSpec, + OptimizationRunSpec, + OptimizationSession, + SessionFailure, + SessionManifest, + SessionStatus, +) +from vero.runtime.wandb import WandbEventSink +from vero.staging import SandboxStagingArea + +__all__ = [ + "ArtifactStore", + "AGENT_CONTEXT_DIRECTORY", + "AgentContextDirectory", + "AgentDisclosureLedger", + "EventBus", + "EventSink", + "JsonlEventSink", + "OptimizationSession", + "OptimizationComponentSpec", + "OptimizationRunSpec", + "RuntimeEvent", + "SandboxStagingArea", + "SessionFailure", + "SessionManifest", + "SessionStatus", + "WandbEventSink", + "WorkspaceContextManager", + "agent_event_emitter", + "create_optimization_session", + "evaluation_result_path", + "make_evaluation_receipt", + "narrower_disclosure", + "create_local_optimization_session", +] diff --git a/vero/src/vero/runtime/artifacts.py b/vero/src/vero/runtime/artifacts.py new file mode 100644 index 00000000..64389a17 --- /dev/null +++ b/vero/src/vero/runtime/artifacts.py @@ -0,0 +1,42 @@ +"""Safe storage for session-level runtime artifacts.""" + +from __future__ import annotations + +import json +from pathlib import Path, PurePosixPath +from typing import Any + +from vero.evaluation.store.persistence import _atomic_write_json + + +class ArtifactStore: + def __init__(self, root: Path): + self.root = root + + def path(self, relative_path: str) -> Path: + value = PurePosixPath(relative_path) + if ( + not relative_path + or "\\" in relative_path + or value.is_absolute() + or any(part in {"", ".", ".."} for part in relative_path.split("/")) + ): + raise ValueError("artifact path must be a safe relative POSIX path") + resolved = (self.root / Path(*value.parts)).resolve() + if not resolved.is_relative_to(self.root.resolve()): + raise ValueError("artifact path escapes the session artifact directory") + return resolved + + def write_text(self, relative_path: str, value: str) -> Path: + path = self.path(relative_path) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(value, encoding="utf-8") + return path + + def write_json(self, relative_path: str, value: Any) -> Path: + path = self.path(relative_path) + _atomic_write_json(path, value) + return path + + def read_json(self, relative_path: str) -> Any: + return json.loads(self.path(relative_path).read_text(encoding="utf-8")) diff --git a/vero/src/vero/runtime/context.py b/vero/src/vero/runtime/context.py new file mode 100644 index 00000000..a9515383 --- /dev/null +++ b/vero/src/vero/runtime/context.py @@ -0,0 +1,604 @@ +"""Authorized filesystem context exposed to optimization agents.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import posixpath +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from typing import Literal, Sequence + +from pydantic import Field + +from vero.candidate import Candidate +from vero.candidate_repository import CandidateRepository +from vero.evaluation import ( + CaseResourceExporter, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationAcknowledgement, + EvaluationBudget, + EvaluationEngine, + EvaluationPlan, + EvaluationPrincipal, + EvaluationReceipt, + EvaluationRecord, + EvaluationSet, + EvaluationSummary, + project_evaluation, +) +from vero.evaluation.store.persistence import _atomic_write_json +from vero.models import StrictModel +from vero.sandbox import Sandbox +from vero.workspace import Workspace + +AGENT_CONTEXT_DIRECTORY = ".evals" +# Top-level names inside the context tree. The `evals` CLI, the skill, and the +# harbor instruction templates all address these paths by name. +RESULTS_SUBDIRECTORY = "results" +TASKS_SUBDIRECTORY = "tasks" +CANDIDATES_SUBDIRECTORY = "candidates" +PLAN_FILENAME = "plan.json" +_DISCLOSURE_RANK = { + DisclosureLevel.NONE: 0, + DisclosureLevel.AGGREGATE: 1, + DisclosureLevel.FULL: 2, +} + + +def context_digest(value: str) -> str: + """Return a path-safe identity for an arbitrary public identifier.""" + + return hashlib.sha256(value.encode()).hexdigest() + + +def evaluation_result_path(evaluation_id: str) -> str: + return ( + f"{AGENT_CONTEXT_DIRECTORY}/{RESULTS_SUBDIRECTORY}/" + f"{context_digest(evaluation_id)}/evaluation.json" + ) + + +@dataclass(frozen=True) +class ContextPlanEntry: + """One evaluation set as the agent may see it, resolved by the caller. + + Callers own policy filtering nuances that differ per topology (which sets + exist, budget lookup, budget disclosure); the directory owns rendering. + """ + + backend_id: str + backend: object + evaluation_set: EvaluationSet + access: EvaluationAccessPolicy + budget: EvaluationBudget | None = None + + +def narrower_disclosure( + left: DisclosureLevel, + right: DisclosureLevel, +) -> DisclosureLevel: + return left if _DISCLOSURE_RANK[left] <= _DISCLOSURE_RANK[right] else right + + +def make_evaluation_receipt( + record: EvaluationRecord, + disclosure: DisclosureLevel, +) -> EvaluationReceipt: + result: EvaluationSummary | EvaluationAcknowledgement + if disclosure == DisclosureLevel.NONE: + projected = project_evaluation(record, DisclosureLevel.NONE) + assert isinstance(projected, EvaluationAcknowledgement) + result = projected + else: + projected = project_evaluation(record, DisclosureLevel.AGGREGATE) + assert isinstance(projected, EvaluationSummary) + result = projected + return EvaluationReceipt( + evaluation_id=record.id, + status=record.report.status, + disclosure=disclosure, + result=result, + result_path=evaluation_result_path(record.id), + ) + + +class AgentDisclosureEntry(StrictModel): + evaluation_id: str + maximum_disclosure: DisclosureLevel + + +class AgentDisclosureLedgerModel(StrictModel): + schema_version: Literal[1] = 1 + evaluations: dict[str, AgentDisclosureEntry] = Field(default_factory=dict) + + +class AgentDisclosureLedger: + """Durably records which evaluations may enter agent-facing context.""" + + def __init__(self, path: Path): + self.path = path + self._lock = asyncio.Lock() + if path.exists(): + self.model = AgentDisclosureLedgerModel.model_validate_json( + path.read_text(encoding="utf-8") + ) + else: + self.model = AgentDisclosureLedgerModel() + + def get(self, evaluation_id: str) -> DisclosureLevel | None: + entry = self.model.evaluations.get(evaluation_id) + return entry.maximum_disclosure if entry is not None else None + + async def remember( + self, + evaluation_id: str, + disclosure: DisclosureLevel, + ) -> DisclosureLevel: + """Record first exposure and never broaden it on a later call.""" + + async with self._lock: + existing = self.model.evaluations.get(evaluation_id) + effective = ( + disclosure + if existing is None + else narrower_disclosure(existing.maximum_disclosure, disclosure) + ) + if existing is None or effective != existing.maximum_disclosure: + self.model.evaluations[evaluation_id] = AgentDisclosureEntry( + evaluation_id=evaluation_id, + maximum_disclosure=effective, + ) + await asyncio.to_thread( + _atomic_write_json, + self.path, + self.model.model_dump(mode="json"), + ) + return effective + + +class AgentContextDirectory: + """Render authorized records into one sandbox filesystem tree.""" + + def __init__( + self, + *, + sandbox: Sandbox, + root: str, + session_dir: Path, + ): + self.sandbox = sandbox + self.root = root.rstrip("/") + self.session_dir = session_dir + + def path(self, *parts: str) -> str: + path = PurePosixPath(self.root) + for part in parts: + path /= part + return path.as_posix() + + async def write_json(self, path: str, value: object) -> None: + await self.sandbox.write_file( + path, + json.dumps(value, ensure_ascii=False, indent=2, default=str) + "\n", + ) + + async def reset(self) -> None: + if await self.sandbox.exists(self.root): + await self.unseal() + for name in await self.sandbox.list_dir(self.root): + await self.sandbox.remove(self.path(name), recursive=True) + else: + await self.sandbox.mkdir(self.root) + + async def unseal(self) -> None: + if not await self.sandbox.exists(self.root): + return + result = await self.sandbox.run(["chmod", "-R", "u+w", self.root]) + if result.returncode != 0: + raise RuntimeError(result.stderr or f"failed to unseal {self.root}") + + async def seal(self) -> None: + result = await self.sandbox.run(["chmod", "-R", "a-w", self.root]) + if result.returncode != 0: + raise RuntimeError(result.stderr or f"failed to seal {self.root}") + + async def write_header( + self, + *, + session_id: str, + round_number: int | None, + proposal_id: str | None, + parent_candidate_id: str | None, + ) -> None: + readme = """# Evaluation context + +This directory is generated and read-only. It contains everything you are +authorized to inspect about how your program is evaluated: + +- `results/` — past evaluation results (scores, per-case results, traces, + artifacts). Start at `results/index.json`. +- `tasks/` — the task resources you may read, when exposed. Start at + `tasks/index.json`. +- `candidates/` — prior program versions, with Git refs you can pass to + `git show` or `git diff`. Start at `candidates/index.json`. +- `plan.json` — the named evaluations you may invoke, their base case + selection, disclosure level, and remaining budget. + +Full-disclosure results put each case and long trace in its own file; their +artifact paths resolve below that result's `artifacts/` directory. + +Use ordinary filesystem and Git commands to analyze this context. Do not copy +it into the program: candidate versions that track `.evals` are rejected. +""" + await self.sandbox.write_file(self.path("README.md"), readme) + await self.write_json( + self.path("manifest.json"), + { + "schema_version": 1, + "session_id": session_id, + "round": round_number, + "proposal_id": proposal_id, + "parent_candidate_id": parent_candidate_id, + "snapshot_semantics": "generation", + }, + ) + + async def write_evaluations( + self, + projections: Sequence[ + tuple[ + EvaluationRecord, + DisclosureLevel, + EvaluationRecord | EvaluationSummary | EvaluationAcknowledgement, + ] + ], + ) -> None: + root = self.path(RESULTS_SUBDIRECTORY) + if await self.sandbox.exists(root): + await self.sandbox.remove(root, recursive=True) + await self.sandbox.mkdir(root) + index: list[dict[str, object]] = [] + for record, disclosure, projection in sorted( + projections, + key=lambda item: (item[0].completed_at, item[0].id), + ): + digest = context_digest(record.id) + relative_path = f"{digest}/evaluation.json" + evaluation_root = self.path(RESULTS_SUBDIRECTORY, digest) + await self.sandbox.mkdir(evaluation_root) + missing_artifacts: list[str] = [] + if isinstance(projection, EvaluationRecord): + payload = projection.model_dump(mode="json") + cases = payload["report"].pop("cases") + case_files: list[dict[str, object]] = [] + for case_model, case_payload in zip( + projection.report.cases, + cases, + strict=True, + ): + case_digest = context_digest(case_model.case_id) + case_root = self.path( + RESULTS_SUBDIRECTORY, digest, "cases", case_digest + ) + await self.sandbox.mkdir(case_root) + execution_trace = case_payload.pop("execution_trace", None) + evaluation_trace = case_payload.pop("evaluation_trace", None) + case_document: dict[str, object] = { + "schema_version": 1, + "result": case_payload, + } + if execution_trace is not None: + await self.write_json( + posixpath.join(case_root, "execution-trace.json"), + execution_trace, + ) + case_document["execution_trace_path"] = "execution-trace.json" + if evaluation_trace is not None: + await self.write_json( + posixpath.join(case_root, "evaluation-trace.json"), + evaluation_trace, + ) + case_document["evaluation_trace_path"] = "evaluation-trace.json" + await self.write_json( + posixpath.join(case_root, "result.json"), + case_document, + ) + case_files.append( + { + "case_id": case_model.case_id, + "path": f"cases/{case_digest}/result.json", + } + ) + payload["case_files"] = case_files + source_root = self.session_dir / "evaluations" / record.id / "artifacts" + resolved_source_root = source_root.resolve() + artifact_paths = { + artifact.path for artifact in projection.report.artifacts + } + for case in projection.report.cases: + artifact_paths.update(artifact.path for artifact in case.artifacts) + for artifact_path in sorted(artifact_paths): + source = source_root / Path(*PurePosixPath(artifact_path).parts) + try: + resolved_source = source.resolve(strict=True) + except (OSError, RuntimeError): + missing_artifacts.append(artifact_path) + continue + contains_symlink = source.is_symlink() or ( + source.is_dir() + and any(path.is_symlink() for path in source.rglob("*")) + ) + if ( + not resolved_source.is_relative_to(resolved_source_root) + or contains_symlink + ): + missing_artifacts.append(artifact_path) + continue + await self.sandbox.upload( + str(source), + self.path(RESULTS_SUBDIRECTORY, digest, "artifacts", artifact_path), + ) + document = { + "schema_version": 1, + "disclosure": disclosure.value, + "result": payload, + "artifacts_path": "artifacts", + "missing_artifacts": missing_artifacts, + } + else: + document = { + "schema_version": 1, + "disclosure": disclosure.value, + "result": projection.model_dump(mode="json"), + } + await self.write_json( + self.path(RESULTS_SUBDIRECTORY, relative_path), + document, + ) + index.append( + { + "evaluation_id": record.id, + "candidate_id": record.request.candidate.id, + "candidate_version": record.request.candidate.version, + "evaluation": record.request.evaluation_set.name, + "partition": record.request.evaluation_set.partition, + "disclosure": disclosure.value, + "path": relative_path, + } + ) + await self.write_json( + self.path(RESULTS_SUBDIRECTORY, "index.json"), + {"schema_version": 1, "evaluations": index}, + ) + + async def write_case_resources( + self, + entries: Sequence[ContextPlanEntry], + ) -> None: + root = self.path(TASKS_SUBDIRECTORY) + if await self.sandbox.exists(root): + await self.sandbox.remove(root, recursive=True) + await self.sandbox.mkdir(root) + index: list[dict[str, object]] = [] + for entry in entries: + # expose_case_resources implies agent_visible (model validation). + if not ( + entry.access.expose_case_resources + and isinstance(entry.backend, CaseResourceExporter) + ): + continue + digest = context_digest( + entry.evaluation_set.budget_key(entry.backend_id) + ) + resource_root = self.path(TASKS_SUBDIRECTORY, digest) + await self.sandbox.mkdir(resource_root) + await self.write_json( + posixpath.join(resource_root, "manifest.json"), + { + "schema_version": 1, + "backend_id": entry.backend_id, + "evaluation_set": entry.evaluation_set.model_dump(mode="json"), + "resources_path": "resources", + }, + ) + resources = posixpath.join(resource_root, "resources") + await self.sandbox.mkdir(resources) + await entry.backend.export_case_resources( + evaluation_set=entry.evaluation_set, + destination=resources, + sandbox=self.sandbox, + ) + index.append( + { + "backend_id": entry.backend_id, + "evaluation_set": entry.evaluation_set.model_dump(mode="json"), + "path": digest, + } + ) + await self.write_json( + posixpath.join(root, "index.json"), + {"schema_version": 1, "case_resources": index}, + ) + + async def write_evaluation_plan( + self, + entries: Sequence[ContextPlanEntry], + ) -> None: + evaluations = [] + for entry in entries: + access = entry.access + if not access.agent_visible and not access.agent_can_evaluate: + continue + # Partition size, so the agent can size a subset instead of guessing + # a range and learning the bound from a rejected request. Advisory + # only: a backend that cannot cost its own base selection just + # reports no count rather than failing the whole plan. + resolve_cost = getattr(entry.backend, "resolve_cost", None) + case_count = None + if callable(resolve_cost): + try: + case_count = (await resolve_cost(entry.evaluation_set)).cases + except Exception: + case_count = None + evaluations.append( + { + "name": entry.evaluation_set.name, + "partition": entry.evaluation_set.partition, + # Which backend serves this partition. `evals run` needs the + # pair to match — asking one partition's backend for another + # is denied — and `--backend`'s help points here for it. + "backend": entry.backend_id, + "cases": case_count, + "base_selection": entry.evaluation_set.selection.model_dump( + mode="json" + ), + "agent_can_evaluate": access.agent_can_evaluate, + "agent_selection": access.agent_selection.value, + "disclosure": access.disclosure.value, + "expose_case_resources": access.expose_case_resources, + "budget": ( + entry.budget.model_dump(mode="json") + if entry.budget is not None + else None + ), + } + ) + await self.write_json( + self.path(PLAN_FILENAME), + {"schema_version": 1, "evaluations": evaluations}, + ) + + +class WorkspaceContextManager: + """Build and refresh one proposal's generation-scoped context snapshot.""" + + def __init__( + self, + *, + session_id: str, + session_dir: Path, + round_number: int, + proposal_id: str, + parent: Candidate, + workspace: Workspace, + candidate_repository: CandidateRepository, + engine: EvaluationEngine, + backend_id: str, + evaluation_plan: EvaluationPlan, + candidates: Sequence[Candidate], + evaluations: Sequence[EvaluationRecord], + ledger: AgentDisclosureLedger, + ): + self.session_id = session_id + self.session_dir = session_dir + self.round_number = round_number + self.proposal_id = proposal_id + self.parent = parent + self.workspace = workspace + self.candidate_repository = candidate_repository + self.engine = engine + self.backend_id = backend_id + self.evaluation_plan = evaluation_plan + self.candidates = {candidate.id: candidate for candidate in candidates} + self.evaluations = {record.id: record for record in evaluations} + self.ledger = ledger + self.root = posixpath.join( + self.workspace.project_path, + AGENT_CONTEXT_DIRECTORY, + ) + self.directory = AgentContextDirectory( + sandbox=workspace.sandbox, + root=self.root, + session_dir=session_dir, + ) + self._lock = asyncio.Lock() + + async def _projections( + self, + ) -> list[ + tuple[ + EvaluationRecord, + DisclosureLevel, + EvaluationRecord | EvaluationSummary | EvaluationAcknowledgement, + ] + ]: + projections = [] + for record in self.evaluations.values(): + decision = await self.engine.authorize( + record.backend_id, + record.request, + EvaluationPrincipal.AGENT, + ) + if not decision.viewable: + continue + maximum = await self.ledger.remember(record.id, decision.disclosure) + disclosure = narrower_disclosure(maximum, decision.disclosure) + projections.append( + (record, disclosure, project_evaluation(record, disclosure)) + ) + return projections + + async def _write_candidate_history(self) -> None: + await self.candidate_repository.materialize_agent_history( + tuple(self.candidates.values()), + workspace=self.workspace, + destination=self.directory.path(CANDIDATES_SUBDIRECTORY), + ) + + def _context_entries(self) -> list[ContextPlanEntry]: + ledger = self.engine.budget_ledger + backend = self.engine.backends.resolve(self.backend_id) + return [ + ContextPlanEntry( + backend_id=self.backend_id, + backend=backend, + evaluation_set=definition.evaluation_set, + access=definition.access, + budget=( + ledger.get( + self.backend_id, + definition.evaluation_set, + EvaluationPrincipal.AGENT, + ) + if ledger is not None + else None + ), + ) + for definition in self.evaluation_plan.evaluations + ] + + async def initialize(self) -> None: + async with self._lock: + await self.directory.reset() + await self.directory.write_header( + session_id=self.session_id, + round_number=self.round_number, + proposal_id=self.proposal_id, + parent_candidate_id=self.parent.id, + ) + entries = self._context_entries() + await self._write_candidate_history() + await self.directory.write_evaluations(await self._projections()) + await self.directory.write_evaluation_plan(entries) + await self.directory.write_case_resources(entries) + await self.directory.seal() + + async def add_evaluation( + self, + record: EvaluationRecord, + disclosure: DisclosureLevel, + ) -> EvaluationReceipt: + async with self._lock: + self.candidates[record.request.candidate.id] = record.request.candidate + self.evaluations[record.id] = record + maximum = await self.ledger.remember(record.id, disclosure) + effective = narrower_disclosure(maximum, disclosure) + await self.directory.unseal() + await self._write_candidate_history() + await self.directory.write_evaluations(await self._projections()) + await self.directory.write_evaluation_plan(self._context_entries()) + await self.directory.seal() + return make_evaluation_receipt(record, effective) diff --git a/vero/src/vero/runtime/events.py b/vero/src/vero/runtime/events.py new file mode 100644 index 00000000..248fa533 --- /dev/null +++ b/vero/src/vero/runtime/events.py @@ -0,0 +1,120 @@ +"""Session-scoped runtime events and sinks.""" + +from __future__ import annotations + +import asyncio +import inspect +import json +import logging +from collections.abc import Awaitable, Callable +from datetime import UTC, datetime +from pathlib import Path +from typing import Protocol +from uuid import uuid4 + +from pydantic import Field, JsonValue, field_validator + +from vero.models import StrictModel + +logger = logging.getLogger(__name__) + + +class RuntimeEvent(StrictModel): + id: str = Field(default_factory=lambda: str(uuid4())) + session_id: str + kind: str + created_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) + payload: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("id", "session_id", "kind") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("event identity must not be empty") + return value + + @field_validator("created_at") + @classmethod + def normalize_created_at(cls, value: datetime) -> datetime: + if value.tzinfo is None or value.utcoffset() is None: + raise ValueError("event timestamps must be timezone-aware") + return value.astimezone(UTC) + + +class EventSink(Protocol): + def __call__(self, event: RuntimeEvent) -> object: ... + + +class EventBus: + """Publish runtime events without coupling execution to observability sinks.""" + + def __init__(self, sinks: list[EventSink] | None = None): + self.sinks: list[EventSink] = list(sinks or []) + + async def emit( + self, + *, + session_id: str, + kind: str, + payload: dict[str, JsonValue] | None = None, + ) -> RuntimeEvent: + event = RuntimeEvent( + session_id=session_id, + kind=kind, + payload=payload or {}, + ) + for sink in self.sinks: + try: + result = sink(event) + if inspect.isawaitable(result): + await result + except Exception: + logger.exception("Runtime event sink failed for %s", event.id) + return event + + +def agent_event_emitter( + bus: EventBus, + session_id: str, + agent: object, + *, + kind: str = "agent", +) -> Callable[[object], Awaitable[None]] | None: + """Build an ``on_event`` callback that publishes agent activity to ``bus``. + + The coding agent streams SDK-specific stream events; the agent normalizes + them via ``serialize_event`` into ``AgentEvent`` dicts (message / thinking / + tool_call / tool_result). This adapts that normalized stream onto the runtime + ``EventBus`` so tool calls, reasoning, and messages land in ``events.jsonl`` + and any registered sink (e.g. W&B) live — the native runner's introspection + advantage over opaque environments. Returns ``None`` if the agent cannot + normalize its events. + """ + serialize = getattr(agent, "serialize_event", None) + if not callable(serialize): + return None + + async def _emit(raw_event: object) -> None: + normalized = serialize(raw_event) + if normalized: + await bus.emit( + session_id=session_id, kind=kind, payload=dict(normalized) + ) + + return _emit + + +class JsonlEventSink: + """Append canonical runtime events to a session JSONL file.""" + + def __init__(self, path: Path): + self.path = path + self._lock = asyncio.Lock() + + async def __call__(self, event: RuntimeEvent) -> None: + line = json.dumps(event.model_dump(mode="json"), ensure_ascii=False) + async with self._lock: + self.path.parent.mkdir(parents=True, exist_ok=True) + with self.path.open("a", encoding="utf-8") as handle: + handle.write(line) + handle.write("\n") diff --git a/vero/src/vero/runtime/factory.py b/vero/src/vero/runtime/factory.py new file mode 100644 index 00000000..74084d21 --- /dev/null +++ b/vero/src/vero/runtime/factory.py @@ -0,0 +1,282 @@ +"""Composition helpers for local optimization sessions.""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path + +from pydantic import JsonValue + +from vero.candidate import Candidate +from vero.candidate_repository import CandidateRepository, GitCandidateRepository +from vero.evaluation import ( + AuthorizationResolver, + BackendRegistry, + BudgetLedger, + EvaluationBackend, + EvaluationBudget, + EvaluationDatabase, + EvaluationEngine, + EvaluationLimits, + EvaluationPlan, + Evaluator, + ObjectiveSpec, + authorize_evaluation_plan, +) +from vero.optimization import ( + CandidateProducer, + ObjectiveSelectionPolicy, + OptimizationStrategy, + Optimizer, + SelectionPolicy, + SequentialStrategy, +) +from vero.runtime.session import ( + OptimizationRunSpec, + OptimizationSession, + SessionManifest, +) +from vero.sandbox import LocalSandbox +from vero.workspace import GitWorkspace, Workspace + + +def _load_database(session_dir: Path, session_id: str) -> EvaluationDatabase: + database_path = session_dir / "database.json" + return EvaluationDatabase.load_reconciled( + database_path=database_path, + evaluations_dir=session_dir / "evaluations", + database_id=session_id, + ) + + +def _load_budget_ledger( + session_dir: Path, + budgets: list[EvaluationBudget], +) -> BudgetLedger | None: + budget_path = session_dir / "budgets.json" + if budget_path.exists(): + return BudgetLedger.load(budget_path) + if not budgets: + return None + ledger = BudgetLedger(budgets, path=budget_path) + ledger.save() + return ledger + + +async def create_optimization_session( + *, + workspace: Workspace, + candidate_repository: CandidateRepository, + session_dir: Path | str, + backend_id: str, + backend: EvaluationBackend, + objective: ObjectiveSpec, + producers: Mapping[str, CandidateProducer], + evaluation_plan: EvaluationPlan, + session_id: str | None = None, + strategy: OptimizationStrategy | None = None, + selection: SelectionPolicy | None = None, + parameters: dict[str, JsonValue] | None = None, + limits: EvaluationLimits | None = None, + authorization_resolver: AuthorizationResolver | None = None, + metadata: dict[str, JsonValue] | None = None, + run_spec: OptimizationRunSpec | None = None, + seed: int | None = None, + max_proposals: int = 1, + max_rounds: int = 100, + max_concurrency: int = 1, + base_ref: str | None = None, +) -> OptimizationSession: + """Build a durable session around an already-provisioned workspace. + + ``session_dir`` and ``candidate_repository`` are durable control-plane state + on the host. The original workspace supplies the baseline, context, and + default execution sandbox; all candidate checkouts come from the compatible + repository. The evaluation plan is the default fail-closed authorization + boundary; advanced deployments may supply an equivalent trusted resolver. + """ + + session_dir = Path(session_dir).expanduser().resolve() + manifest_path = session_dir / "manifest.json" + if session_id is None and manifest_path.exists(): + session_id = SessionManifest.model_validate_json( + manifest_path.read_text(encoding="utf-8") + ).id + session_id = session_id or session_dir.name + if not session_id.strip(): + raise ValueError("session ID must not be empty") + if not candidate_repository.supports(workspace): + raise ValueError( + "candidate repository and workspace belong to different families" + ) + mismatched_budgets = [ + budget.backend_id + for budget in evaluation_plan.budgets + if budget.backend_id != backend_id + ] + if mismatched_budgets: + raise ValueError( + "evaluation plan budgets must use the session backend " + f"{backend_id!r}" + ) + if not producers and max_proposals: + raise ValueError("at least one candidate producer is required") + for producer in producers.values(): + validate_workspace = getattr(producer, "validate_workspace", None) + if callable(validate_workspace): + validate_workspace(workspace) + + persisted_manifest = ( + SessionManifest.model_validate_json(manifest_path.read_text(encoding="utf-8")) + if manifest_path.exists() + else None + ) + if persisted_manifest is not None: + baseline = persisted_manifest.baseline + if baseline is None: + raise ValueError("persisted session is missing its baseline candidate") + stored = candidate_repository.get(baseline.id) + if stored != baseline: + raise ValueError( + "persisted baseline is missing from the candidate repository" + ) + else: + if await workspace.is_dirty(): + raise ValueError("target workspace must be clean before optimization") + baseline_version = ( + await workspace.resolve_ref(base_ref) + if base_ref is not None + else await workspace.current_version() + ) + baseline = candidate_repository.get(baseline_version) + if baseline is None: + baseline = Candidate.from_version(baseline_version) + if await workspace.current_version() == baseline_version: + await candidate_repository.capture(baseline, workspace) + else: + async with workspace.temp_copy( + from_version=baseline_version + ) as baseline_workspace: + await candidate_repository.capture(baseline, baseline_workspace) + elif ( + baseline.version != baseline_version + or baseline.parent_id is not None + or baseline.description is not None + or baseline.metadata + ): + raise ValueError( + "candidate repository contains a conflicting baseline record" + ) + + session_dir.mkdir(parents=True, exist_ok=True) + database_path = session_dir / "database.json" + database = _load_database(session_dir, session_id) + budget_ledger = _load_budget_ledger(session_dir, evaluation_plan.budgets) + evaluator = Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=session_dir, + session_id=session_id, + ) + engine = EvaluationEngine( + evaluator=evaluator, + backends=BackendRegistry({backend_id: backend}), + database=database, + database_path=database_path, + budget_ledger=budget_ledger, + authorization_resolver=( + authorization_resolver or authorize_evaluation_plan(evaluation_plan) + ), + ) + optimizer = Optimizer( + workspace=workspace, + candidate_repository=candidate_repository, + engine=engine, + backend_id=backend_id, + evaluation_plan=evaluation_plan, + objective=objective, + strategy=strategy or SequentialStrategy(), + producers=dict(producers), + selection=selection or ObjectiveSelectionPolicy(), + parameters=parameters or {}, + limits=limits or EvaluationLimits(), + seed=seed, + max_proposals=max_proposals, + max_rounds=max_rounds, + max_concurrency=max_concurrency, + session_id=session_id, + ) + session = OptimizationSession( + id=session_id, + session_dir=session_dir, + optimizer=optimizer, + baseline=baseline, + metadata=metadata or {}, + run_spec=run_spec, + ) + for producer_id, producer in producers.items(): + bind_artifacts = getattr(producer, "bind_artifacts", None) + if callable(bind_artifacts): + bind_artifacts(session.artifacts, producer_id=producer_id) + return session + + +async def create_local_optimization_session( + *, + project_path: Path | str, + session_dir: Path | str, + backend_id: str, + backend: EvaluationBackend, + objective: ObjectiveSpec, + producers: Mapping[str, CandidateProducer], + evaluation_plan: EvaluationPlan, + session_id: str | None = None, + strategy: OptimizationStrategy | None = None, + selection: SelectionPolicy | None = None, + parameters: dict[str, JsonValue] | None = None, + limits: EvaluationLimits | None = None, + authorization_resolver: AuthorizationResolver | None = None, + metadata: dict[str, JsonValue] | None = None, + run_spec: OptimizationRunSpec | None = None, + seed: int | None = None, + max_proposals: int = 1, + max_rounds: int = 100, + max_concurrency: int = 1, + base_ref: str | None = None, +) -> OptimizationSession: + """Provision a local Git workspace and build an optimization session.""" + + project_path = Path(project_path).expanduser().resolve() + session_path = Path(session_dir).expanduser().resolve() + sandbox = await LocalSandbox.create(root=project_path.parent) + workspace = await GitWorkspace.from_path(sandbox, str(project_path)) + repository_root = Path(workspace.root).resolve() + if session_path == repository_root or session_path.is_relative_to(repository_root): + raise ValueError("session directory must live outside the target repository") + candidate_repository = await GitCandidateRepository.create( + session_path / "candidates", + workspace=workspace, + ) + return await create_optimization_session( + workspace=workspace, + candidate_repository=candidate_repository, + session_dir=session_path, + backend_id=backend_id, + backend=backend, + objective=objective, + producers=producers, + evaluation_plan=evaluation_plan, + session_id=session_id, + strategy=strategy, + selection=selection, + parameters=parameters, + limits=limits, + authorization_resolver=authorization_resolver, + metadata=metadata, + run_spec=run_spec, + seed=seed, + max_proposals=max_proposals, + max_rounds=max_rounds, + max_concurrency=max_concurrency, + base_ref=base_ref, + ) diff --git a/vero/src/vero/runtime/session.py b/vero/src/vero/runtime/session.py new file mode 100644 index 00000000..778bbfdb --- /dev/null +++ b/vero/src/vero/runtime/session.py @@ -0,0 +1,499 @@ +"""Generic optimization session lifecycle and durable manifest.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +from dataclasses import dataclass, field +from datetime import UTC, datetime +from enum import Enum +from pathlib import Path +from typing import Literal + +from pydantic import BaseModel, Field, JsonValue, field_validator + +from vero.candidate import Candidate +from vero.candidate_repository import CandidateRepository +from vero.evaluation import ( + BackendProvenance, + CaseStatus, + EvaluationLimits, + EvaluationPlan, + EvaluationRecord, + ObjectiveSpec, +) +from vero.evaluation.store.persistence import _atomic_write_json +from vero.models import StrictModel +from vero.optimization import OptimizationResult, Optimizer +from vero.runtime.artifacts import ArtifactStore +from vero.runtime.events import EventBus, JsonlEventSink, agent_event_emitter +from vero.workspace import Workspace + + +class SessionStatus(str, Enum): + CREATED = "created" + RUNNING = "running" + COMPLETED = "completed" + FAILED = "failed" + + +class SessionFailure(StrictModel): + type: str + message: str + + @field_validator("type", "message") + @classmethod + def validate_text(cls, value: str) -> str: + if not value.strip(): + raise ValueError("session failure fields must not be empty") + return value + + +class OptimizationComponentSpec(StrictModel): + """Stable type and configuration identity for a protocol component.""" + + type: str + config_digest: str + + @field_validator("type", "config_digest") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("component identity must not be empty") + return value + + @field_validator("config_digest") + @classmethod + def validate_digest(cls, value: str) -> str: + if len(value) != 64 or any( + character not in "0123456789abcdef" for character in value + ): + raise ValueError("component config_digest must be a SHA-256 hex digest") + return value + + +class OptimizationRunSpec(StrictModel): + """Execution choices that must remain stable when a session resumes.""" + + max_proposals: int = Field(ge=0) + max_rounds: int = Field(ge=1) + max_concurrency: int = Field(ge=1) + strategy: OptimizationComponentSpec + producers: dict[str, OptimizationComponentSpec] + # Optional so manifests written before this field can still load; a resume + # under a changed selection policy is then caught as a run-protocol mismatch. + selection: OptimizationComponentSpec | None = None + + +class SessionManifest(StrictModel): + schema_version: Literal[3] = 3 + id: str + status: SessionStatus + backend_id: str + backend: BackendProvenance + candidate_repository_family: str + candidate_repository_format_version: int + evaluation_plan: EvaluationPlan + objective: ObjectiveSpec + run: OptimizationRunSpec + parameters: dict[str, JsonValue] = Field(default_factory=dict) + limits: EvaluationLimits = Field(default_factory=EvaluationLimits) + seed: int | None = None + baseline: Candidate | None = None + best_candidate_id: str | None = None + best_evaluation_id: str | None = None + final_baseline_evaluation_id: str | None = None + final_evaluation_id: str | None = None + created_at: datetime + updated_at: datetime + failure: SessionFailure | None = None + metadata: dict[str, JsonValue] = Field(default_factory=dict) + + @field_validator("id", "backend_id", "candidate_repository_family") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("session identity must not be empty") + return value + + @field_validator("created_at", "updated_at") + @classmethod + def normalize_timestamps(cls, value: datetime) -> datetime: + if value.tzinfo is None or value.utcoffset() is None: + raise ValueError("session timestamps must be timezone-aware") + return value.astimezone(UTC) + + +@dataclass +class OptimizationSession: + """Own the durable state and lifecycle of one optimization run.""" + + id: str + session_dir: Path + optimizer: Optimizer + baseline: Candidate | None = None + metadata: dict[str, JsonValue] = field(default_factory=dict) + events: EventBus | None = None + run_spec: OptimizationRunSpec | None = None + + def __post_init__(self) -> None: + if not self.id.strip(): + raise ValueError("session ID must not be empty") + expected = self.session_dir.resolve() + evaluator_session = self.optimizer.engine.evaluator.session_dir.resolve() + if evaluator_session != expected: + raise ValueError( + "optimizer evaluator session directory must match OptimizationSession" + ) + optimizer_session_id = self.optimizer.session_id + if optimizer_session_id is not None and optimizer_session_id != self.id: + raise ValueError("optimizer session ID does not match OptimizationSession") + self.optimizer.session_id = self.id + evaluator_session_id = getattr( + self.optimizer.engine.evaluator, "session_id", None + ) + if evaluator_session_id is not None and evaluator_session_id != self.id: + raise ValueError("evaluator session ID does not match OptimizationSession") + self.optimizer.engine.evaluator.session_id = self.id + if ( + self.optimizer.engine.evaluator.candidate_repository + is not self.optimizer.candidate_repository + ): + raise ValueError( + "optimizer and evaluator must share one candidate repository" + ) + self.session_dir.mkdir(parents=True, exist_ok=True) + if self.events is None: + self.events = EventBus([JsonlEventSink(self.events_path)]) + self.artifacts = ArtifactStore(self.session_dir / "artifacts") + self._event_step = len(self.optimizer.engine.database.evaluations) + self.optimizer.engine.listeners.append(self._on_evaluation_completed) + self._wire_agent_events() + + def _wire_agent_events(self) -> None: + """Publish agent activity (tool calls, reasoning, messages) to the bus. + + Agent producers that expose a settable ``on_event`` and a normalizing + agent get their events routed onto the session event bus. A + caller-provided ``on_event`` is left untouched. + """ + assert self.events is not None + for producer in self.optimizer.producers.values(): + if getattr(producer, "on_event", None) is not None: + continue + if not hasattr(producer, "on_event"): + continue + agent = getattr(producer, "agent", None) + if agent is None: + continue + emitter = agent_event_emitter(self.events, self.id, agent) + if emitter is not None: + producer.on_event = emitter + + @property + def manifest_path(self) -> Path: + return self.session_dir / "manifest.json" + + @property + def events_path(self) -> Path: + return self.session_dir / "events.jsonl" + + @property + def database(self): + return self.optimizer.engine.database + + @property + def budget_ledger(self): + return self.optimizer.engine.budget_ledger + + @property + def candidate_repository(self) -> CandidateRepository: + return self.optimizer.candidate_repository + + @property + def workspace(self) -> Workspace: + """The original workspace supplied when the session was created.""" + + return self.optimizer.workspace + + def _initial_manifest(self, baseline: Candidate) -> SessionManifest: + now = datetime.now(UTC) + return SessionManifest( + id=self.id, + status=SessionStatus.CREATED, + backend_id=self.optimizer.backend_id, + backend=self.optimizer.engine.backends.resolve( + self.optimizer.backend_id + ).provenance, + candidate_repository_family=self.candidate_repository.family, + candidate_repository_format_version=( + self.candidate_repository.format_version + ), + evaluation_plan=self.optimizer.evaluation_plan, + objective=self.optimizer.objective, + run=self._run_spec(), + parameters=self.optimizer.parameters, + limits=self.optimizer.limits, + seed=self.optimizer.seed, + baseline=baseline, + created_at=now, + updated_at=now, + metadata=self.metadata, + ) + + @staticmethod + def _component_spec(value: object) -> OptimizationComponentSpec: + kind = type(value) + type_name = f"{kind.__module__}.{kind.__qualname__}" + payload: dict[str, object] = {} + config = getattr(value, "config", None) + if isinstance(config, BaseModel): + payload["config"] = config.model_dump(mode="json") + elif isinstance(config, dict): + payload["config"] = config + for name in ("producer_id", "instruction", "prompt", "max_turns"): + if hasattr(value, name): + payload[name] = getattr(value, name) + agent = getattr(value, "agent", None) + serialize_agent = getattr(agent, "dict", None) + if callable(serialize_agent): + payload["agent"] = serialize_agent() + encoded = json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + default=str, + ).encode() + return OptimizationComponentSpec( + type=type_name, + config_digest=hashlib.sha256(encoded).hexdigest(), + ) + + def _run_spec(self) -> OptimizationRunSpec: + if self.run_spec is not None: + return self.run_spec + return OptimizationRunSpec( + max_proposals=self.optimizer.max_proposals, + max_rounds=self.optimizer.max_rounds, + max_concurrency=self.optimizer.max_concurrency, + strategy=self._component_spec(self.optimizer.strategy), + producers={ + producer_id: self._component_spec(producer) + for producer_id, producer in sorted(self.optimizer.producers.items()) + }, + selection=self._component_spec(self.optimizer.selection), + ) + + async def _on_evaluation_completed(self, record: EvaluationRecord) -> None: + """Publish evaluations as they finish, rather than replaying them at exit.""" + + assert self.events is not None + step = self._event_step + self._event_step += 1 + await self.events.emit( + session_id=self.id, + kind="evaluation_completed", + payload=self._evaluation_event_payload(record, step=step), + ) + + async def _save_manifest(self, manifest: SessionManifest) -> None: + await asyncio.to_thread( + _atomic_write_json, + self.manifest_path, + manifest.model_dump(mode="json"), + ) + + @staticmethod + def _evaluation_event_payload( + record: EvaluationRecord, + *, + step: int, + ) -> dict[str, JsonValue]: + counts = {status: 0 for status in CaseStatus} + for case in record.report.cases: + counts[case.status] += 1 + payload: dict[str, JsonValue] = { + "step": step, + "evaluation_id": record.id, + "candidate_id": record.request.candidate.id, + "candidate_version": record.request.candidate.version, + "evaluation": record.request.evaluation_set.name, + "partition": record.request.evaluation_set.partition, + "principal": record.principal.value, + "status": record.report.status.value, + "cases/total": len(record.report.cases), + "cases/success": counts[CaseStatus.SUCCESS], + "cases/error": counts[CaseStatus.ERROR], + "cases/skipped": counts[CaseStatus.SKIPPED], + } + payload.update( + {f"metrics/{name}": value for name, value in record.report.metrics.items()} + ) + if record.objective is not None: + payload["objective/value"] = record.objective.value + payload["objective/feasible"] = record.objective.feasible + return payload + + def load_manifest(self) -> SessionManifest: + return SessionManifest.model_validate_json( + self.manifest_path.read_text(encoding="utf-8") + ) + + async def run( + self, + *, + baseline: Candidate | None = None, + skip_baseline_evaluation: bool = False, + max_proposals: int | None = None, + ) -> OptimizationResult: + manifest = self.load_manifest() if self.manifest_path.exists() else None + if baseline is None: + if manifest is not None and manifest.baseline is not None: + baseline = manifest.baseline + elif self.baseline is not None: + baseline = self.baseline + else: + baseline = Candidate.from_version( + await self.optimizer.workspace.current_version() + ) + if manifest is None: + manifest = self._initial_manifest(baseline) + if manifest.id != self.id: + raise ValueError("session manifest ID does not match runtime session") + if manifest.backend_id != self.optimizer.backend_id: + raise ValueError("session backend does not match the persisted manifest") + if manifest.candidate_repository_family != self.candidate_repository.family: + raise ValueError( + "session candidate repository does not match the persisted manifest" + ) + if ( + manifest.candidate_repository_format_version + != self.candidate_repository.format_version + ): + raise ValueError( + "session candidate repository format does not match the " + "persisted manifest" + ) + backend = self.optimizer.engine.backends.resolve(self.optimizer.backend_id) + if manifest.backend != backend.provenance: + raise ValueError( + "session backend configuration does not match the persisted manifest" + ) + if manifest.evaluation_plan != self.optimizer.evaluation_plan: + raise ValueError( + "session evaluation plan does not match the persisted manifest" + ) + if manifest.objective != self.optimizer.objective: + raise ValueError("session objective does not match the persisted manifest") + if manifest.run != self._run_spec(): + raise ValueError("session run protocol does not match the persisted manifest") + if manifest.parameters != self.optimizer.parameters: + raise ValueError( + "session evaluation parameters do not match the persisted manifest" + ) + if manifest.limits != self.optimizer.limits: + raise ValueError( + "session evaluation limits do not match the persisted manifest" + ) + if manifest.seed != self.optimizer.seed: + raise ValueError( + "session evaluation seed does not match the persisted manifest" + ) + if manifest.baseline is None or ( + manifest.baseline.id, + manifest.baseline.version, + ) != (baseline.id, baseline.version): + raise ValueError("session baseline does not match the persisted manifest") + + manifest = manifest.model_copy( + update={ + "status": SessionStatus.RUNNING, + "updated_at": datetime.now(UTC), + "failure": None, + } + ) + await self._save_manifest(manifest) + assert self.events is not None + await self.events.emit( + session_id=self.id, + kind="session_started", + payload={"baseline_candidate_id": baseline.id}, + ) + + try: + result = await self.optimizer.run( + baseline=baseline, + skip_baseline_evaluation=skip_baseline_evaluation, + max_proposals=max_proposals, + ) + except BaseException as error: + failure = SessionFailure( + type=f"{type(error).__module__}.{type(error).__name__}", + message=str(error) or type(error).__name__, + ) + await self._save_manifest( + manifest.model_copy( + update={ + "status": SessionStatus.FAILED, + "updated_at": datetime.now(UTC), + "failure": failure, + } + ) + ) + await self.events.emit( + session_id=self.id, + kind="session_failed", + payload={"error_type": failure.type, "message": failure.message}, + ) + raise + + best = result.best + completed = manifest.model_copy( + update={ + "status": SessionStatus.COMPLETED, + "updated_at": datetime.now(UTC), + "best_candidate_id": ( + best.request.candidate.id if best is not None else None + ), + "best_evaluation_id": best.id if best is not None else None, + "final_baseline_evaluation_id": ( + result.final_baseline.id + if result.final_baseline is not None + else None + ), + "final_evaluation_id": ( + result.final.id if result.final is not None else None + ), + } + ) + await self._save_manifest(completed) + await self.events.emit( + session_id=self.id, + kind="session_completed", + payload={ + "best_candidate_id": completed.best_candidate_id, + "best_evaluation_id": completed.best_evaluation_id, + "evaluation_count": len(result.evaluations), + "status": "completed", + "baseline_candidate_id": result.baseline.request.candidate.id, + "baseline_objective": ( + result.baseline.objective.value + if result.baseline.objective is not None + else None + ), + "best_objective": ( + best.objective.value + if best is not None and best.objective is not None + else None + ), + "final_objective": ( + result.final.objective.value + if result.final is not None + and result.final.objective is not None + else None + ), + }, + ) + return result diff --git a/vero/src/vero/runtime/wandb.py b/vero/src/vero/runtime/wandb.py new file mode 100644 index 00000000..e467e3f3 --- /dev/null +++ b/vero/src/vero/runtime/wandb.py @@ -0,0 +1,455 @@ +"""Optional Weights & Biases reporting for canonical runtime events.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +import tempfile +from pathlib import Path +from typing import Any +from uuid import uuid4 + +from vero.evaluation.models import CaseStatus, EvaluationRecord +from vero.evaluation.store.budget import BudgetLedger +from vero.runtime.artifacts import ArtifactStore +from vero.runtime.events import RuntimeEvent + +logger = logging.getLogger(__name__) + + +def _open_wandb_run( + *, + project: str, + session_id: str, + wandb_dir: Path, + client: Any | None, + entity: str | None, + name: str | None, + group: str | None, + tags: list[str] | None, + mode: str | None, + notes: str | None, + config: dict[str, Any] | None, + run_id: str | None, +) -> Any: + """Open one resumable W&B run keyed stably to the session. W&B is imported + lazily so nothing here depends on it unless a sink is constructed.""" + if client is None: + try: + import wandb as client + except ImportError as error: + raise RuntimeError( + "W&B reporting requires `pip install scale-vero[wandb]`" + ) from error + wandb_dir.mkdir(parents=True, exist_ok=True) + stable_id = run_id or ("vero-" + hashlib.sha256(session_id.encode()).hexdigest()[:16]) + init_kwargs: dict[str, Any] = { + "project": project, + "id": stable_id, + "resume": "allow", + "dir": str(wandb_dir), + "config": {**(config or {}), "vero/session_id": session_id}, + } + for key, value in { + "entity": entity, + "name": name, + "group": group, + "tags": tags or None, + "mode": mode, + "notes": notes, + }.items(): + if value is not None: + init_kwargs[key] = value + return client.init(**init_kwargs) + + +class WandbEventSink: + """Log one optimization session as one W&B run. + + W&B is imported only when this sink is constructed, so the core runtime has + no mandatory tracking dependency. + """ + + def __init__( + self, + *, + project: str, + session_id: str, + session_dir: Path, + entity: str | None = None, + name: str | None = None, + group: str | None = None, + tags: list[str] | None = None, + mode: str | None = None, + notes: str | None = None, + config: dict[str, Any] | None = None, + run_id: str | None = None, + client: Any | None = None, + ): + if client is None: + try: + import wandb as client + except ImportError as error: + raise RuntimeError( + "W&B reporting requires `pip install scale-vero[wandb]`" + ) from error + + wandb_dir = session_dir / "artifacts" / "wandb" + wandb_dir.mkdir(parents=True, exist_ok=True) + self.artifacts = ArtifactStore(session_dir / "artifacts") + self.state_path = "wandb/state.json" + if self.artifacts.path(self.state_path).exists(): + state = self.artifacts.read_json(self.state_path) + self.logged_evaluations = set(state.get("evaluation_ids", [])) + self.next_step = int(state.get("next_step", len(self.logged_evaluations))) + else: + self.logged_evaluations: set[str] = set() + self.next_step = 0 + stable_id = run_id or ( + "vero-" + hashlib.sha256(session_id.encode()).hexdigest()[:16] + ) + init_kwargs: dict[str, Any] = { + "project": project, + "id": stable_id, + "resume": "allow", + "dir": str(wandb_dir), + "config": {**(config or {}), "vero/session_id": session_id}, + } + for key, value in { + "entity": entity, + "name": name, + "group": group, + "tags": tags or None, + "mode": mode, + "notes": notes, + }.items(): + if value is not None: + init_kwargs[key] = value + self.run = client.init(**init_kwargs) + + def _save_state(self) -> None: + self.artifacts.write_json( + self.state_path, + { + "evaluation_ids": sorted(self.logged_evaluations), + "next_step": self.next_step, + }, + ) + + def __call__(self, event: RuntimeEvent) -> None: + if event.kind == "evaluation_completed": + payload = dict(event.payload) + payload.pop("step") + evaluation_id = str(payload["evaluation_id"]) + if evaluation_id in self.logged_evaluations: + return + self.run.log(payload, step=self.next_step) + self.logged_evaluations.add(evaluation_id) + self.next_step += 1 + self._save_state() + return + if event.kind == "session_completed": + self.run.summary.update(event.payload) + self.run.finish() + return + if event.kind == "session_failed": + self.run.summary.update( + { + "status": "failed", + "error_type": event.payload.get("error_type"), + "error_message": event.payload.get("message"), + } + ) + self.run.finish(exit_code=1) + + +class SidecarWandbSink: + """Log the trusted eval-sidecar's evaluation stream to one W&B run. + + Subscribed to ``EvaluationEngine.listeners``, so it sees every completed + evaluation the sidecar produces — the agent's search evals and the trusted + SYSTEM/ADMIN re-scores — with the real scores, statuses, diagnostics, and + remaining budget the sidecar holds. This is the harbor-path live-watch + surface: it does not depend on the untrusted optimizer agent cooperating, + and the W&B credentials live only in the trusted container. + """ + + def __init__( + self, + *, + project: str, + session_id: str, + session_dir: Path, + entity: str | None = None, + name: str | None = None, + group: str | None = None, + tags: list[str] | None = None, + mode: str | None = None, + notes: str | None = None, + config: dict[str, Any] | None = None, + run_id: str | None = None, + budget_ledger: BudgetLedger | None = None, + log_traces: bool = False, + client: Any | None = None, + ): + if client is None: + try: + import wandb as client + except ImportError as error: + raise RuntimeError( + "W&B reporting requires `pip install scale-vero[wandb]`" + ) from error + self._wandb = client + self.session_dir = session_dir + self.log_traces = log_traces + self.artifacts = ArtifactStore(session_dir / "artifacts") + self.state_path = "wandb/state.json" + stored_run_id = None + if self.artifacts.path(self.state_path).exists(): + state = self.artifacts.read_json(self.state_path) + self.logged_evaluations = set(state.get("evaluation_ids", [])) + self.next_step = int(state.get("next_step", len(self.logged_evaluations))) + stored_run_id = state.get("run_id") + self._shipped_request_logs = dict(state.get("request_log_files", {})) + else: + self.logged_evaluations: set[str] = set() + self.next_step = 0 + self._shipped_request_logs: dict[str, int] = {} + self._last_inference_usage: dict[str, Any] = {} + # One W&B run per invocation. A fresh session volume mints a new id; a + # sidecar restart within the same run reuses the one persisted in state, + # so it resumes rather than colliding with other invocations. + self.run_id = run_id or stored_run_id or f"vero-{uuid4().hex[:16]}" + self._save_state() + self.budget_ledger = budget_ledger + # Keep W&B's symlink-laden run dir out of session_dir, which is archived + # verbatim for export (a symlink there would break the archive). + wandb_run_dir = Path(tempfile.mkdtemp(prefix="vero-wandb-")) + self.run = _open_wandb_run( + project=project, + session_id=session_id, + wandb_dir=wandb_run_dir, + client=client, + entity=entity, + name=name, + group=group, + tags=tags, + mode=mode, + notes=notes, + config=config, + run_id=self.run_id, + ) + + def _save_state(self) -> None: + self.artifacts.write_json( + self.state_path, + { + "evaluation_ids": sorted(self.logged_evaluations), + "next_step": self.next_step, + "run_id": self.run_id, + "request_log_files": self._shipped_request_logs, + }, + ) + + def _payload(self, record: EvaluationRecord) -> dict[str, Any]: + report = record.report + objective = record.objective + evaluation_set = record.request.evaluation_set + partition = evaluation_set.partition or "none" + principal = record.principal.value + # Scope metrics by partition/principal so unlike evals don't share a series. + scope = f"{partition}/{principal}" + counts = {status: 0 for status in CaseStatus} + for case in report.cases: + counts[case.status] += 1 + payload: dict[str, Any] = { + # Identity / context — flat, shared across every evaluation. + "evaluation_id": record.id, + "candidate_id": record.request.candidate.id, + "candidate_version": record.request.candidate.version, + "evaluation_set": evaluation_set.name, + "partition": partition, + "principal": principal, + "status": report.status.value, + "latency_seconds": ( + record.completed_at - record.created_at + ).total_seconds(), + # Scoped quality metrics. `num_cases` is the evaluation's sample + # size (number of trials): evaluations cover different case counts, + # so a score is only interpretable next to how many cases produced + # it. + f"{scope}/num_cases": len(report.cases), + f"{scope}/cases/success": counts[CaseStatus.SUCCESS], + f"{scope}/cases/error": counts[CaseStatus.ERROR], + f"{scope}/cases/skipped": counts[CaseStatus.SKIPPED], + f"{scope}/feasible": ( + bool(objective.feasible) if objective is not None else False + ), + } + if objective is not None and objective.value is not None: + payload[f"{scope}/score"] = objective.value + for key, value in report.metrics.items(): + payload[f"{scope}/metric/{key}"] = value + if report.diagnostics: + payload["diagnostics"] = ",".join( + diagnostic.code for diagnostic in report.diagnostics + ) + if self.budget_ledger is not None: + for budget in self.budget_ledger.budgets: + scope = f"{budget.backend_id}/{budget.principal.value}" + if budget.remaining_runs is not None: + payload[f"budget/{scope}/remaining_runs"] = budget.remaining_runs + if budget.remaining_cases is not None: + payload[f"budget/{scope}/remaining_cases"] = budget.remaining_cases + return payload + + def _log_trace(self, record: EvaluationRecord) -> None: + """Upload an evaluation's trace artifacts as one W&B artifact. + + Full job data is on the per-case artifacts, not just the report-level + harbor logs, so walk both. Best-effort.""" + entries = list(record.report.artifacts) + for case in record.report.cases: + entries.extend(case.artifacts) + if not entries: + return + eval_dir = self.session_dir / "evaluations" / record.id / "artifacts" + artifact = self._wandb.Artifact( + name=f"trace-{record.id}", type="evaluation_trace" + ) + added = 0 + seen: set[str] = set() + for entry in entries: + if entry.path in seen: + continue + seen.add(entry.path) + path = eval_dir / entry.path + if path.is_file(): + artifact.add_file(str(path), name=entry.path) + added += 1 + if added: + self.run.log_artifact(artifact) + + def log_inference_usage(self, scopes: dict[str, Any]) -> None: + """Log the gateway's per-scope usage counters as W&B series.""" + payload: dict[str, Any] = {} + for name, usage in sorted(scopes.items()): + if not isinstance(usage, dict): + continue + for key in ( + "requests", + "upstream_errors", + "input_tokens", + "cached_input_tokens", + "output_tokens", + "total_tokens", + "active_requests", + ): + value = usage.get(key) + if isinstance(value, (int, float)): + payload[f"inference/{name}/{key}"] = value + if not payload or payload == self._last_inference_usage: + return + self.run.log(payload, step=self.next_step) + self.next_step += 1 + self._last_inference_usage = payload + self._save_state() + + def ship_request_logs(self, directory: Path, *, final: bool = False) -> None: + """Upload the gateway's rotated request-log files as one W&B artifact. + + The highest-numbered file is still being appended to, so it is only + included on the final call. Unchanged files dedupe by digest on the + W&B side, so re-adding them per version is cheap. + """ + files = sorted(directory.glob("requests-*.jsonl")) + if not final: + files = files[:-1] + if not files: + return + snapshot = {path.name: path.stat().st_size for path in files} + if snapshot == self._shipped_request_logs: + return + artifact = self._wandb.Artifact( + name="inference-requests", type="inference_request_log" + ) + for path in files: + artifact.add_file(str(path), name=path.name) + self.run.log_artifact(artifact) + self._shipped_request_logs = snapshot + self._save_state() + + def __call__(self, record: EvaluationRecord) -> None: + if record.id in self.logged_evaluations: + return + self.run.log(self._payload(record), step=self.next_step) + if self.log_traces: + try: + self._log_trace(record) + except Exception: # tracing must never drop the metric log + pass + self.logged_evaluations.add(record.id) + self.next_step += 1 + self._save_state() + + def finish( + self, + summary: dict[str, Any] | None = None, + *, + failed: bool = False, + ) -> None: + """Update the run summary (e.g. shipped/rewards at finalize) and close.""" + if summary: + self.run.summary.update(summary) + self.run.finish(exit_code=1 if failed else 0) + + +class InferenceTelemetryPoller: + """Mirror the gateway's durable state into the sidecar's W&B run. + + The gateway state volume is mounted read-only in the trusted sidecar, so + this gives live inference telemetry (including the untrusted optimizer's + producer-scope burn) without W&B credentials leaving the sidecar. + Best-effort throughout: telemetry must never affect the eval path. + """ + + def __init__( + self, + *, + sink: SidecarWandbSink, + usage_path: Path | None = None, + request_log_dir: Path | None = None, + interval_seconds: float = 30.0, + ): + if usage_path is None and request_log_dir is None: + raise ValueError("telemetry poller requires at least one source") + if interval_seconds <= 0: + raise ValueError("telemetry interval must be positive") + self.sink = sink + self.usage_path = usage_path + self.request_log_dir = request_log_dir + self.interval_seconds = interval_seconds + + def poll_once(self, *, final: bool = False) -> None: + if self.usage_path is not None: + try: + if self.usage_path.exists(): + value = json.loads(self.usage_path.read_text(encoding="utf-8")) + scopes = value.get("scopes") if isinstance(value, dict) else None + if isinstance(scopes, dict): + self.sink.log_inference_usage(scopes) + except Exception: + logger.warning("inference usage telemetry failed", exc_info=True) + if self.request_log_dir is not None: + try: + if self.request_log_dir.is_dir(): + self.sink.ship_request_logs(self.request_log_dir, final=final) + except Exception: + logger.warning("inference request log shipping failed", exc_info=True) + + async def run(self) -> None: + while True: + await asyncio.sleep(self.interval_seconds) + await asyncio.to_thread(self.poll_once) diff --git a/vero/src/vero/sandbox.py b/vero/src/vero/sandbox.py new file mode 100644 index 00000000..5324734b --- /dev/null +++ b/vero/src/vero/sandbox.py @@ -0,0 +1,752 @@ +"""Abstract sandbox: filesystem + shell execution. + +A Sandbox provides a unified interface for file I/O and shell commands. +Tools call sandbox methods instead of using pathlib/subprocess directly, +making it possible to swap in different backends (local, container, remote VM). + +The default implementation (LocalSandbox) wraps pathlib.Path and +asyncio.create_subprocess_exec. Access control is handled by Workspace, +not Sandbox — the sandbox is a dumb I/O layer. +""" + +from __future__ import annotations + +import asyncio +import os +import posixpath +import shutil +import signal +import tempfile +import uuid +from abc import ABC, abstractmethod +from contextlib import asynccontextmanager +from dataclasses import dataclass +from pathlib import Path +from typing import AsyncIterator, Awaitable, Callable, NamedTuple + + +async def _terminate_host_process_tree( + process: asyncio.subprocess.Process, + *, + grace_seconds: float = 5, +) -> None: + """Terminate a host subprocess and every descendant in its process group.""" + + if process.returncode is not None and os.name != "posix": + return + + try: + if os.name == "posix": + os.killpg(process.pid, signal.SIGTERM) + else: + process.terminate() + except ProcessLookupError: + return + + if process.returncode is None: + try: + await asyncio.wait_for(process.wait(), timeout=grace_seconds) + except asyncio.TimeoutError: + pass + + # The group leader may exit before descendants that ignore SIGTERM. Always + # sweep the original process group with SIGKILL before returning. + try: + if os.name == "posix": + os.killpg(process.pid, signal.SIGKILL) + elif process.returncode is None: + process.kill() + except ProcessLookupError: + pass + if process.returncode is None: + await process.wait() + + +async def _cleanup_host_process( + process: asyncio.subprocess.Process, + before_terminate: Callable[[], Awaitable[None]] | None = None, +) -> None: + """Run backend cleanup, then unconditionally reap the host process group.""" + + try: + if before_terminate is not None: + await before_terminate() + finally: + await _terminate_host_process_tree(process) + + +# ============================================================================= +# Data types +# ============================================================================= + + +class FileStat(NamedTuple): + """Minimal file stat info.""" + + st_size: int + + +class CommandResult(NamedTuple): + """Result of a shell command execution.""" + + stdout: str + stderr: str + returncode: int + + +@dataclass(frozen=True) +class SandboxCapabilities: + """Execution features exposed by a sandbox implementation.""" + + posix: bool = True + host_paths: bool = False + + +# ============================================================================= +# Abstract base +# ============================================================================= + + +class Sandbox(ABC): + """Abstract sandbox providing filesystem access and shell execution. + + A Sandbox represents a "computer" — an execution environment where vero + runs. For a local sandbox the root is the user's home directory; for a + Docker sandbox it would be ``/``. Vero home (``~/.vero/``) and the git + workspace both live *within* the sandbox. + + Access control is **not** a sandbox concern — it lives on Workspace. + The sandbox is pure I/O: read, write, run, exist-checks. + """ + + # ── Construction ─────────────────────────────────────────────────────── + + @classmethod + @abstractmethod + async def create(cls, **kwargs) -> Sandbox: + """Create a new, bare sandbox instance. + + Each subclass knows how to spin up its own environment. + """ + ... + + # ── Properties ────────────────────────────────────────────────────── + + @property + @abstractmethod + def root(self) -> str: + """Root directory of the sandbox.""" + ... + + @property + def capabilities(self) -> SandboxCapabilities: + return SandboxCapabilities() + + def host_path(self, path: str) -> Path | None: + """Return a host-visible equivalent, or ``None`` for isolated paths.""" + + return None + + # ── Path resolution ──────────────────────────────────────────────── + + @abstractmethod + def resolve_path(self, path: str) -> str: + """Resolve a path relative to sandbox root. For sandbox-internal use.""" + ... + + # ── Filesystem (async) ────────────────────────────────────────────── + + @abstractmethod + async def read_file(self, path: str, encoding: str = "utf-8") -> str: ... + + @abstractmethod + async def read_file_bytes(self, path: str, limit: int | None = None) -> bytes: ... + + @abstractmethod + async def write_file( + self, path: str, content: str, encoding: str = "utf-8" + ) -> None: ... + + @abstractmethod + async def exists(self, path: str) -> bool: ... + + @abstractmethod + async def is_file(self, path: str) -> bool: ... + + @abstractmethod + async def is_dir(self, path: str) -> bool: ... + + @abstractmethod + async def mkdir(self, path: str, parents: bool = True) -> None: ... + + @abstractmethod + async def stat(self, path: str) -> FileStat: ... + + @abstractmethod + async def list_dir(self, path: str) -> list[str]: ... + + # ── Shell (async) ─────────────────────────────────────────────────── + + @abstractmethod + async def run( + self, + command: str | list[str], + cwd: str | None = None, + timeout: int | None = 30, + env: dict[str, str] | None = None, + run_as: str | None = None, + ) -> CommandResult: + """Run a command. ``run_as`` drops the process to that unprivileged + user (and its same-named primary group), shedding supplementary + groups; it requires the caller to be privileged and is how untrusted + code is isolated from the trusted process that launches it.""" + ... + + async def canonicalize(self, path: str) -> str: + """Resolve a sandbox path, including symlinks, inside the sandbox.""" + + result = await self.run(["realpath", path]) + if result.returncode != 0: + raise FileNotFoundError(result.stderr or path) + return result.stdout.strip() + + async def remove(self, path: str, *, recursive: bool = False) -> None: + """Remove a sandbox path.""" + + command = ["rm"] + if recursive: + command.append("-rf") + else: + command.append("-f") + command.extend(["--", path]) + result = await self.run(command) + if result.returncode != 0: + raise RuntimeError(result.stderr or f"failed to remove {path}") + + @asynccontextmanager + async def temporary_directory(self, prefix: str = "vero-") -> AsyncIterator[str]: + """Create and clean up a temporary directory inside the sandbox.""" + + safe_prefix = "".join( + character + for character in prefix + if character.isalnum() or character in "-_" + ) + template = f"/tmp/{safe_prefix or 'vero-'}XXXXXX" + result = await self.run(["mktemp", "-d", template]) + if result.returncode != 0: + raise RuntimeError( + result.stderr or "failed to create sandbox temporary directory" + ) + path = await self.canonicalize(result.stdout.strip()) + try: + yield path + finally: + await asyncio.shield(self.remove(path, recursive=True)) + + # ── Host ↔ Sandbox file transfer (async) ─────────────────────────── + + @abstractmethod + async def upload(self, local_path: str, remote_path: str) -> None: + """Copy a file or directory from the host into the sandbox. + + For LocalSandbox this is a local copy (no-op if paths match). + For remote sandboxes this would be docker cp, scp, etc. + """ + ... + + @abstractmethod + async def download(self, remote_path: str, local_path: str) -> None: + """Copy a file or directory from the sandbox to the host. + + For LocalSandbox this is a local copy (no-op if paths match). + For remote sandboxes this would be docker cp, scp, etc. + """ + ... + + async def close(self) -> None: + """Release resources owned by this sandbox. Default: no-op.""" + + return None + + +# ============================================================================= +# Local implementation +# ============================================================================= + + +class LocalSandbox(Sandbox): + """Sandbox backed by the local filesystem and subprocess execution. + + File I/O uses pathlib, shell execution uses asyncio.create_subprocess_exec. + No access control — that is handled by Workspace. + + Uses pathlib.Path for path resolution and manipulation. + """ + + def __init__(self, root: Path | str) -> None: + self._root = Path(root).resolve() + + @classmethod + async def create(cls, root: Path | str | None = None, **kwargs) -> LocalSandbox: + """Create a LocalSandbox rooted at the user's home directory.""" + if root is None: + root = Path.home() + root = Path(root).resolve() + return cls(root=root) + + # ── Properties ────────────────────────────────────────────────────── + + @property + def root(self) -> str: + return str(self._root) + + @property + def capabilities(self) -> SandboxCapabilities: + return SandboxCapabilities(host_paths=True) + + def host_path(self, path: str) -> Path: + return Path(self.resolve_path(path)) + + # ── Path resolution ──────────────────────────────────────────────── + + def resolve_path(self, path: str) -> str: + p = Path(path) + if p.is_absolute(): + return str(p.resolve()) + return str((self._root / p).resolve()) + + async def canonicalize(self, path: str) -> str: + resolved = Path(self.resolve_path(path)) + if not resolved.exists(): + raise FileNotFoundError(path) + return str(resolved.resolve()) + + # ── Filesystem ────────────────────────────────────────────────────── + + async def read_file(self, path: str, encoding: str = "utf-8") -> str: + resolved = Path(self.resolve_path(path)) + return resolved.read_text(encoding=encoding) + + async def read_file_bytes(self, path: str, limit: int | None = None) -> bytes: + resolved = Path(self.resolve_path(path)) + with open(resolved, "rb") as f: + return f.read(limit) if limit is not None else f.read() + + async def write_file( + self, path: str, content: str, encoding: str = "utf-8" + ) -> None: + resolved = Path(self.resolve_path(path)) + resolved.parent.mkdir(parents=True, exist_ok=True) + resolved.write_text(content, encoding=encoding) + + async def exists(self, path: str) -> bool: + return Path(self.resolve_path(path)).exists() + + async def is_file(self, path: str) -> bool: + return Path(self.resolve_path(path)).is_file() + + async def is_dir(self, path: str) -> bool: + return Path(self.resolve_path(path)).is_dir() + + async def mkdir(self, path: str, parents: bool = True) -> None: + Path(self.resolve_path(path)).mkdir(parents=parents, exist_ok=True) + + async def stat(self, path: str) -> FileStat: + st = Path(self.resolve_path(path)).stat() + return FileStat(st_size=st.st_size) + + async def list_dir(self, path: str) -> list[str]: + resolved = Path(self.resolve_path(path)) + return sorted(entry.name for entry in resolved.iterdir()) + + # ── Shell ─────────────────────────────────────────────────────────── + + async def run( + self, + command: str | list[str], + cwd: str | None = None, + timeout: int | None = 30, + env: dict[str, str] | None = None, + run_as: str | None = None, + ) -> CommandResult: + """Run a command via subprocess. + + Returns CommandResult with stdout, stderr, returncode. + Does NOT raise on non-zero exit — caller decides how to handle. + """ + if isinstance(command, str): + command = ["bash", "-c", command] + + if cwd is None: + cwd = str(self._root) + + privilege_kwargs: dict[str, object] = {} + if run_as is not None: + # Drop to the unprivileged user and its same-named primary group, + # shedding supplementary groups. Requires the launching process to + # be privileged (root); otherwise create_subprocess_exec raises + # PermissionError — the intended fail-closed behavior for isolation. + privilege_kwargs = { + "user": run_as, + "group": run_as, + "extra_groups": [], + } + + proc = await asyncio.create_subprocess_exec( + *command, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + cwd=str(cwd), + env=env, + start_new_session=os.name == "posix", + **privilege_kwargs, + ) + + try: + stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout) + return CommandResult( + stdout=stdout.decode().strip(), + stderr=stderr.decode().strip(), + returncode=proc.returncode or 0, + ) + except asyncio.TimeoutError: + await asyncio.shield(_cleanup_host_process(proc)) + cmd_str = " ".join(command) + return CommandResult( + stdout="", + stderr=f"Command '{cmd_str}' timed out after {timeout} seconds", + returncode=-1, + ) + except (KeyboardInterrupt, asyncio.CancelledError): + await asyncio.shield(_cleanup_host_process(proc)) + raise + + # ── Host ↔ Sandbox file transfer ─────────────────────────────────── + + async def upload(self, local_path: str, remote_path: str) -> None: + """Copy from host to sandbox. No-op if paths resolve to the same location.""" + import shutil + + src = Path(local_path).resolve() + dst = Path(remote_path).resolve() + if src == dst: + return + if src.is_dir(): + shutil.copytree(src, dst, dirs_exist_ok=True) + elif src.is_file(): + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + async def download(self, remote_path: str, local_path: str) -> None: + """Copy from sandbox to host. No-op if paths resolve to the same location.""" + import shutil + + src = Path(remote_path).resolve() + dst = Path(local_path).resolve() + if src == dst: + return + if src.is_dir(): + shutil.copytree(src, dst, dirs_exist_ok=True) + elif src.is_file(): + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + +class DockerSandbox(Sandbox): + """POSIX sandbox backed by a Docker container with no shared filesystem.""" + + def __init__( + self, + container_id: str, + *, + root: str = "/workspace", + docker_executable: str = "docker", + owns_container: bool = False, + ) -> None: + self.container_id = container_id + self._root = posixpath.normpath(root) + self.docker_executable = docker_executable + self.owns_container = owns_container + self._closed = False + + @classmethod + async def create( + cls, + *, + image: str, + root: str = "/workspace", + name: str | None = None, + docker_executable: str | None = None, + **kwargs, + ) -> DockerSandbox: + executable = docker_executable or shutil.which("docker") + if executable is None: + raise ValueError("docker is required to create a DockerSandbox") + command = [ + executable, + "run", + "--detach", + "--rm", + "--workdir", + root, + ] + if name is not None: + command.extend(["--name", name]) + command.extend([image, "sh", "-c", "while :; do sleep 3600; done"]) + result = await cls._host_command(command, timeout=60) + if result.returncode != 0: + raise RuntimeError(result.stderr or "failed to create Docker sandbox") + sandbox = cls( + result.stdout.strip(), + root=root, + docker_executable=executable, + owns_container=True, + ) + mkdir_result = await sandbox.run(["mkdir", "-p", root]) + if mkdir_result.returncode != 0: + await sandbox.close() + raise RuntimeError(mkdir_result.stderr or f"failed to create {root}") + return sandbox + + @classmethod + async def from_container( + cls, + container_id: str, + *, + root: str = "/workspace", + docker_executable: str | None = None, + ) -> DockerSandbox: + executable = docker_executable or shutil.which("docker") + if executable is None: + raise ValueError("docker is required to attach a DockerSandbox") + sandbox = cls( + container_id, + root=root, + docker_executable=executable, + owns_container=False, + ) + result = await sandbox.run(["mkdir", "-p", root]) + if result.returncode != 0: + raise RuntimeError( + result.stderr or f"cannot access Docker container {container_id}" + ) + return sandbox + + @staticmethod + async def _host_command( + command: list[str], + *, + timeout: int | float | None = 30, + before_terminate: Callable[[], Awaitable[None]] | None = None, + ) -> CommandResult: + process = await asyncio.create_subprocess_exec( + *command, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + start_new_session=os.name == "posix", + ) + try: + stdout, stderr = await asyncio.wait_for( + process.communicate(), timeout=timeout + ) + except asyncio.TimeoutError: + await asyncio.shield(_cleanup_host_process(process, before_terminate)) + return CommandResult("", f"Command timed out after {timeout} seconds", -1) + except (KeyboardInterrupt, asyncio.CancelledError): + await asyncio.shield(_cleanup_host_process(process, before_terminate)) + raise + return CommandResult( + stdout.decode(errors="replace").strip(), + stderr.decode(errors="replace").strip(), + process.returncode or 0, + ) + + async def _docker( + self, *arguments: str, timeout: int | float | None = 30 + ) -> CommandResult: + return await self._host_command( + [self.docker_executable, *arguments], + timeout=timeout, + ) + + @property + def root(self) -> str: + return self._root + + @property + def capabilities(self) -> SandboxCapabilities: + return SandboxCapabilities(posix=True, host_paths=False) + + def resolve_path(self, path: str) -> str: + if posixpath.isabs(path): + return posixpath.normpath(path) + return posixpath.normpath(posixpath.join(self._root, path)) + + async def read_file(self, path: str, encoding: str = "utf-8") -> str: + return (await self.read_file_bytes(path)).decode(encoding) + + async def read_file_bytes(self, path: str, limit: int | None = None) -> bytes: + command = [self.docker_executable, "exec", self.container_id] + if limit is None: + command.extend(["cat", self.resolve_path(path)]) + else: + command.extend(["head", "-c", str(limit), self.resolve_path(path)]) + process = await asyncio.create_subprocess_exec( + *command, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await process.communicate() + if process.returncode != 0: + raise FileNotFoundError(stderr.decode(errors="replace") or path) + return stdout + + async def write_file( + self, path: str, content: str, encoding: str = "utf-8" + ) -> None: + with tempfile.TemporaryDirectory(prefix="vero-docker-") as directory: + local_path = Path(directory) / "content" + local_path.write_text(content, encoding=encoding) + await self.upload(str(local_path), self.resolve_path(path)) + + async def exists(self, path: str) -> bool: + return (await self.run(["test", "-e", self.resolve_path(path)])).returncode == 0 + + async def is_file(self, path: str) -> bool: + return (await self.run(["test", "-f", self.resolve_path(path)])).returncode == 0 + + async def is_dir(self, path: str) -> bool: + return (await self.run(["test", "-d", self.resolve_path(path)])).returncode == 0 + + async def mkdir(self, path: str, parents: bool = True) -> None: + command = ["mkdir"] + if parents: + command.append("-p") + command.append(self.resolve_path(path)) + result = await self.run(command) + if result.returncode != 0: + raise RuntimeError(result.stderr or f"failed to create {path}") + + async def stat(self, path: str) -> FileStat: + result = await self.run(["stat", "-c", "%s", self.resolve_path(path)]) + if result.returncode != 0: + raise FileNotFoundError(result.stderr or path) + return FileStat(st_size=int(result.stdout)) + + async def list_dir(self, path: str) -> list[str]: + result = await self.run(["ls", "-1A", self.resolve_path(path)]) + if result.returncode != 0: + raise FileNotFoundError(result.stderr or path) + return sorted(line for line in result.stdout.splitlines() if line) + + async def run( + self, + command: str | list[str], + cwd: str | None = None, + timeout: int | None = 30, + env: dict[str, str] | None = None, + run_as: str | None = None, + ) -> CommandResult: + pid_file = f"/tmp/vero-exec-{uuid.uuid4().hex}.pid" + arguments = ["exec"] + if run_as is not None: + arguments.extend(["-u", run_as]) + if cwd is not None: + arguments.extend(["--workdir", self.resolve_path(cwd)]) + for name, value in (env or {}).items(): + arguments.extend(["--env", f"{name}={value}"]) + arguments.append(self.container_id) + payload = ["sh", "-c", command] if isinstance(command, str) else command + wrapper = ( + 'pid_file="$1"; shift; ' + 'printf \'%s\\n\' "$$" > "$pid_file"; ' + "trap 'rm -f \"$pid_file\"' EXIT; " + '"$@"' + ) + arguments.extend( + ["setsid", "--wait", "sh", "-c", wrapper, "sh", pid_file, *payload] + ) + + async def terminate_container_group() -> None: + script = """ +pid_file=$1 +attempt=0 +while [ ! -s "$pid_file" ] && [ "$attempt" -lt 20 ]; do + sleep 0.05 + attempt=$((attempt + 1)) +done +if [ -s "$pid_file" ]; then + pgid=$(cat "$pid_file") + kill -TERM -- "-$pgid" 2>/dev/null || true + attempt=0 + while kill -0 -- "-$pgid" 2>/dev/null && [ "$attempt" -lt 10 ]; do + sleep 0.5 + attempt=$((attempt + 1)) + done + kill -KILL -- "-$pgid" 2>/dev/null || true +fi +rm -f "$pid_file" +""" + await self._docker( + "exec", + self.container_id, + "sh", + "-c", + script, + "sh", + pid_file, + timeout=10, + ) + + return await self._host_command( + [self.docker_executable, *arguments], + timeout=timeout, + before_terminate=terminate_container_group, + ) + + async def canonicalize(self, path: str) -> str: + result = await self.run(["readlink", "-f", self.resolve_path(path)]) + if result.returncode != 0: + raise FileNotFoundError(result.stderr or path) + return result.stdout.strip() + + async def upload(self, local_path: str, remote_path: str) -> None: + source = Path(local_path).resolve() + if not source.exists(): + raise FileNotFoundError(local_path) + destination = self.resolve_path(remote_path) + if source.is_dir(): + await self.mkdir(destination) + docker_source = f"{source}/." + else: + await self.mkdir(posixpath.dirname(destination)) + docker_source = str(source) + result = await self._docker( + "cp", docker_source, f"{self.container_id}:{destination}", timeout=120 + ) + if result.returncode != 0: + raise RuntimeError(result.stderr or f"failed to upload {local_path}") + + async def download(self, remote_path: str, local_path: str) -> None: + source = self.resolve_path(remote_path) + destination = Path(local_path).resolve() + if not await self.exists(source): + raise FileNotFoundError(remote_path) + if await self.is_dir(source): + destination.mkdir(parents=True, exist_ok=True) + docker_source = f"{self.container_id}:{source}/." + else: + destination.parent.mkdir(parents=True, exist_ok=True) + docker_source = f"{self.container_id}:{source}" + result = await self._docker("cp", docker_source, str(destination), timeout=120) + if result.returncode != 0: + raise RuntimeError(result.stderr or f"failed to download {remote_path}") + + async def close(self) -> None: + if self._closed or not self.owns_container: + return + self._closed = True + result = await self._docker("rm", "--force", self.container_id, timeout=30) + if result.returncode != 0 and "No such container" not in result.stderr: + raise RuntimeError(result.stderr or "failed to remove Docker sandbox") diff --git a/vero/src/vero/sidecar/__init__.py b/vero/src/vero/sidecar/__init__.py new file mode 100644 index 00000000..3213efad --- /dev/null +++ b/vero/src/vero/sidecar/__init__.py @@ -0,0 +1,72 @@ +"""The trusted side of the evaluation boundary. + +Owns held-out partitions, disclosure, budgets, and final scoring — everything the +optimizer must not be able to reach. Deployed as the ``eval-sidecar`` container +beside the optimizer's ``main`` container, which is where the name comes from. + +Two invariants worth knowing: + +- Nothing here is Harbor-specific. This package depends on ``vero.evaluation`` + only; ``vero.harbor.deployment`` is the single module that binds this stack to + a Harbor-backed evaluation, and a compiled task can just as well use a command + backend (see ``examples/harbor-circle-packing``). +- Co-location is load-bearing. HTTP carries the API, but candidate code, the + ``.evals`` result context, and the admin token all move through volumes shared + with ``main`` — so this is not, today, deployable away from the task it serves. + +``sidecar`` is the agent-facing API and is transport-neutral; ``app`` is an +optional FastAPI transport over it; ``serve`` is the composition root that loads +a factory, builds the components, and runs them. +""" + +from vero.sidecar.session import HarborSessionManifest +from vero.sidecar.sidecar import ( + EvaluationAccessError, + EvaluationAccessStatus, + EvaluationJobNotFoundError, + EvaluationJobStatus, + EvaluationSidecar, + SidecarEvaluationJob, + SidecarEvaluationPolicy, + SidecarEvaluationRequest, + SidecarEvaluationResult, + SidecarStatus, + Submission, + SubmissionDisabledError, +) +from vero.sidecar.transport import ( + CandidateTransferError, + CandidateTransport, + GitCandidateTransport, +) +from vero.sidecar.verifier import ( + CanonicalVerifier, + NoCandidateError, + VerificationResult, + VerificationSelection, + VerificationTarget, +) + +__all__ = [ + "CandidateTransferError", + "CandidateTransport", + "CanonicalVerifier", + "EvaluationAccessError", + "EvaluationAccessStatus", + "EvaluationJobNotFoundError", + "EvaluationJobStatus", + "EvaluationSidecar", + "GitCandidateTransport", + "HarborSessionManifest", + "NoCandidateError", + "SidecarEvaluationJob", + "SidecarEvaluationPolicy", + "SidecarEvaluationRequest", + "SidecarEvaluationResult", + "SidecarStatus", + "Submission", + "SubmissionDisabledError", + "VerificationResult", + "VerificationSelection", + "VerificationTarget", +] diff --git a/vero/src/vero/sidecar/app.py b/vero/src/vero/sidecar/app.py new file mode 100644 index 00000000..98851094 --- /dev/null +++ b/vero/src/vero/sidecar/app.py @@ -0,0 +1,230 @@ +"""Optional FastAPI transport for the canonical Harbor sidecar.""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +import shutil +import tempfile +from contextlib import asynccontextmanager +from pathlib import Path +from typing import TYPE_CHECKING, Annotated + +from fastapi import FastAPI, Header, HTTPException, Query +from fastapi.responses import FileResponse, JSONResponse +from starlette.background import BackgroundTask + +from vero.evaluation import ( + EvaluationBudgetExceeded, + EvaluationDeniedError, + EvaluationRequestError, +) +from vero.evaluation.exceptions import EvaluationExecutionError +from vero.models import StrictModel +from vero.sidecar.auth import check_admin_token +from vero.sidecar.session import create_harbor_session_archive +from vero.sidecar.sidecar import ( + EvaluationAccessError, + EvaluationJobNotFoundError, + EvaluationJobStatus, + EvaluationSidecar, + SidecarEvaluationRequest, + SidecarEvaluationResult, + SubmissionDisabledError, +) +from vero.sidecar.transport import CandidateTransferError +from vero.sidecar.verifier import CanonicalVerifier + +if TYPE_CHECKING: + from vero.runtime.wandb import InferenceTelemetryPoller + +logger = logging.getLogger(__name__) + + +class SubmitRequest(StrictModel): + version: str | None = None + + +class ScoreBaselineRequest(StrictModel): + replicates: int = 1 + + +def _error(status_code: int, message: str, *, detail: bool = False): + async def handler(_request, error): + text = message or str(error) + # `detail` opts a category into appending the raised message. Use it only + # where the message states a backend capability the agent must know to + # fix its own request — never for denials, whose whole point is opacity. + if detail and message and str(error): + text = f"{message}: {error}" + return JSONResponse(status_code=status_code, content={"error": text}) + + return handler + + +def create_app( + *, + sidecar: EvaluationSidecar, + verifier: CanonicalVerifier, + admin_token: str, + telemetry: "InferenceTelemetryPoller | None" = None, +) -> FastAPI: + """Expose agent endpoints and token-gated admin endpoints on one app.""" + if not admin_token.strip(): + raise ValueError("admin_token must not be empty") + + @asynccontextmanager + async def lifespan(_app: FastAPI): + task = ( + asyncio.create_task(telemetry.run()) if telemetry is not None else None + ) + try: + yield + finally: + if task is not None: + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + app = FastAPI(title="VeRO evaluation sidecar", version="1", lifespan=lifespan) + app.add_exception_handler( + EvaluationBudgetExceeded, + _error(429, "evaluation budget exhausted"), + ) + app.add_exception_handler(EvaluationDeniedError, _error(403, "evaluation denied")) + app.add_exception_handler(EvaluationAccessError, _error(403, "evaluation denied")) + app.add_exception_handler( + EvaluationRequestError, + # The agent cannot repair a request it is only told is "invalid": these + # messages are backend-authored capability statements (an unsupported + # flag, a fixed limit), so pass them through. + _error(400, "invalid evaluation request", detail=True), + ) + app.add_exception_handler( + CandidateTransferError, + _error(400, "candidate version could not be imported"), + ) + app.add_exception_handler( + SubmissionDisabledError, + _error(409, "candidate submission is disabled"), + ) + app.add_exception_handler( + EvaluationJobNotFoundError, + _error(404, "evaluation job not found"), + ) + + @app.exception_handler(EvaluationExecutionError) + async def evaluation_failure(_request, error: EvaluationExecutionError): + return JSONResponse( + status_code=502, + content={ + "error": "evaluation failed", + "evaluation_id": error.evaluation_id, + }, + ) + + def require_admin(authorization: str | None) -> None: + if not check_admin_token(authorization, admin_token): + raise HTTPException(status_code=403, detail="admin token required") + + @app.get("/health") + async def health(): + return {"ok": True} + + @app.post("/eval") + async def evaluate(body: SidecarEvaluationRequest): + return await sidecar.evaluate(body) + + @app.post("/eval/jobs", status_code=202) + async def start_evaluation_job(body: SidecarEvaluationRequest): + return await sidecar.start_evaluation_job(body) + + @app.get("/eval/jobs/{job_id}") + async def evaluation_job(job_id: str): + return sidecar.evaluation_job(job_id) + + @app.get("/eval/jobs/{job_id}/result") + async def evaluation_job_result(job_id: str): + job = sidecar.evaluation_job(job_id) + if job.receipt is None: + status_code = ( + 202 + if job.status + in {EvaluationJobStatus.QUEUED, EvaluationJobStatus.RUNNING} + else 409 + ) + return JSONResponse( + status_code=status_code, + content=job.model_dump(mode="json"), + ) + return SidecarEvaluationResult( + disclosure=job.receipt.disclosure, + receipt=job.receipt, + ) + + @app.post("/submit") + async def submit(body: SubmitRequest): + return await sidecar.submit(body.version) + + @app.get("/status") + async def status(): + return sidecar.status() + + @app.post("/finalize") + async def finalize(authorization: Annotated[str | None, Header()] = None): + require_admin(authorization) + return await verifier.finalize() + + @app.post("/score/baseline") + async def score_baseline( + body: ScoreBaselineRequest, + authorization: Annotated[str | None, Header()] = None, + ): + require_admin(authorization) + return await verifier.measure_baseline(replicates=body.replicates) + + @app.get("/evaluations") + async def evaluations( + authorization: Annotated[str | None, Header()] = None, + limit: Annotated[int, Query(ge=1, le=1000)] = 100, + offset: Annotated[int, Query(ge=0)] = 0, + ): + require_admin(authorization) + records = sidecar.engine.database.get_evaluations( + limit=limit, + offset=offset, + reverse=True, + ) + return {"evaluations": records} + + @app.get("/session/export") + async def export_session( + authorization: Annotated[str | None, Header()] = None, + ): + require_admin(authorization) + directory = Path(tempfile.mkdtemp(prefix="vero-harbor-export-")) + archive = directory / "session.tar.gz" + try: + await asyncio.to_thread( + create_harbor_session_archive, + sidecar.engine.evaluator.session_dir, + archive, + ) + except BaseException as error: + shutil.rmtree(directory, ignore_errors=True) + # Surface the real cause (admin-only endpoint) rather than a bare 500. + logger.exception("session export failed") + if isinstance(error, Exception): + raise HTTPException( + status_code=500, detail=f"session export failed: {error}" + ) from error + raise + return FileResponse( + archive, + media_type="application/gzip", + filename="vero-session.tar.gz", + background=BackgroundTask(shutil.rmtree, directory, ignore_errors=True), + ) + + return app diff --git a/vero/src/vero/sidecar/auth.py b/vero/src/vero/sidecar/auth.py new file mode 100644 index 00000000..46f029d1 --- /dev/null +++ b/vero/src/vero/sidecar/auth.py @@ -0,0 +1,79 @@ +"""Admin bearer-token helpers for the Harbor evaluation sidecar.""" + +from __future__ import annotations + +import os +import secrets +import tempfile +from pathlib import Path + + +def generate_admin_token() -> str: + return secrets.token_urlsafe(32) + + +def write_admin_token( + path: Path | str, + token: str, + *, + mode: int = 0o400, +) -> Path: + """Atomically write a read-only token file for the trusted verifier. + + The token gates the full-disclosure admin endpoints (``/finalize``, + ``/evaluations``, ``/session/export``). Under the shared-verifier topology + the token volume is also mounted into the *untrusted* agent container, so + the file (``0o400``) and its parent directory (``0o700``) are locked to the + writing user — root in the sidecar. The agent runs unprivileged + (``task.toml`` ``[agent].user``) and can therefore neither read the file nor + traverse into its directory; only the root-run verifier phase can. This is + what stops an untrusted agent from bypassing the aggregate-disclosure floor + through the admin endpoints. + """ + if not token.strip() or "\x00" in token: + raise ValueError("admin token must not be empty") + destination = Path(path) + destination.parent.mkdir(parents=True, exist_ok=True) + # Block traversal by any non-owner (e.g. the unprivileged agent user), so + # the token is unreachable even if its own mode were ever relaxed. + try: + destination.parent.chmod(0o700) + except OSError: + pass + descriptor, name = tempfile.mkstemp( + dir=destination.parent, + prefix=f".{destination.name}.", + suffix=".tmp", + ) + temporary = Path(name) + try: + os.fchmod(descriptor, mode) + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + handle.write(token) + handle.write("\n") + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, destination) + destination.chmod(mode) + except BaseException: + try: + os.close(descriptor) + except OSError: + pass + temporary.unlink(missing_ok=True) + raise + return destination + + +def read_admin_token(path: Path | str) -> str: + token = Path(path).read_text(encoding="utf-8").strip() + if not token: + raise ValueError("admin token file is empty") + return token + + +def check_admin_token(authorization: str | None, expected_token: str) -> bool: + prefix = "Bearer " + if authorization is None or not authorization.startswith(prefix): + return False + return secrets.compare_digest(authorization[len(prefix) :], expected_token) diff --git a/vero/src/vero/sidecar/isolation.py b/vero/src/vero/sidecar/isolation.py new file mode 100644 index 00000000..aa841dbe --- /dev/null +++ b/vero/src/vero/sidecar/isolation.py @@ -0,0 +1,43 @@ +"""Filesystem provisioning for the isolated harness user. + +Single source of truth for the commands that hand the dropped-privilege harness +user access to its workspace — the eval launch (``backend.py``) and the isolation +integration test both use these so they can't drift. Kept stdlib-only and free of +``vero`` package imports so the test can load it standalone inside a minimal +container (see ``tests/test_v05_harbor_isolation_container.py``). +""" + +from __future__ import annotations + +from collections.abc import Sequence +from pathlib import PurePosixPath + + +def harness_grant_commands( + user: str, *, chown_paths: Sequence[str], checkout_root: str +) -> list[list[str]]: + """Commands to hand ``user`` its work dirs and make the checkout reachable. + + Chowns each work dir to the user, then grants traversal (``o+x``) on the + checkout's parent: ``mktemp -d`` leaves that parent ``0700`` root, and the + dropped user runs as its own uid+gid (so it is "other" for the root-owned + parent) and otherwise can't traverse in to resolve the editable candidate + package's absolute path — the import fails with "No module named " + while harbor itself still loads from the user-owned cache. The parent holds + only candidate code, so widening traversal exposes no trusted data. + """ + owner = f"{user}:{user}" + commands = [["chown", "-R", owner, str(path)] for path in chown_paths] + checkout_parent = str(PurePosixPath(checkout_root).parent) + commands.append(["chmod", "o+x", checkout_parent]) + return commands + + +def harness_reachability_probe(project_path: str) -> list[str]: + """Command (run as the dropped user) that fails if the workspace is unreachable. + + Readability of ``project_path`` requires traversing every ancestor, so a + non-zero exit means a provisioning gap left the workspace out of reach — caught + at the launch site rather than as a cryptic "No module named " downstream. + """ + return ["test", "-r", str(project_path)] diff --git a/vero/src/vero/sidecar/serve.py b/vero/src/vero/sidecar/serve.py new file mode 100644 index 00000000..d31bc59e --- /dev/null +++ b/vero/src/vero/sidecar/serve.py @@ -0,0 +1,104 @@ +"""Build and serve Harbor sidecar components from a trusted factory.""" + +from __future__ import annotations + +import asyncio +import importlib +import inspect +import json +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any, Awaitable, Callable + +from vero.sidecar.auth import generate_admin_token, write_admin_token +from vero.sidecar.sidecar import EvaluationSidecar +from vero.sidecar.verifier import CanonicalVerifier + +if TYPE_CHECKING: + from vero.runtime.wandb import InferenceTelemetryPoller + + +@dataclass(frozen=True) +class SidecarComponents: + sidecar: EvaluationSidecar + verifier: CanonicalVerifier + telemetry: "InferenceTelemetryPoller | None" = None + + +SidecarFactory = Callable[ + [dict[str, Any]], + SidecarComponents | Awaitable[SidecarComponents], +] + + +def load_factory(import_path: str) -> SidecarFactory: + module_name, separator, attribute_name = import_path.partition(":") + if not separator or not module_name.strip() or not attribute_name.strip(): + raise ValueError("sidecar factory must use module:attribute syntax") + module = importlib.import_module(module_name) + factory = getattr(module, attribute_name) + if not callable(factory): + raise TypeError("sidecar factory is not callable") + return factory + + +async def build_components( + *, + factory_path: str, + config_path: Path | str, +) -> SidecarComponents: + path = Path(config_path) + try: + config = json.loads(path.read_text(encoding="utf-8")) + except Exception as error: + raise ValueError(f"invalid sidecar config {path}: {error}") from error + if not isinstance(config, dict): + raise ValueError("sidecar config must be a JSON object") + built = load_factory(factory_path)(config) + if inspect.isawaitable(built): + built = await built + if not isinstance(built, SidecarComponents): + raise TypeError("sidecar factory must return SidecarComponents") + return built + + +async def build_app( + *, + factory_path: str, + config_path: Path | str, + admin_token_path: Path | str, +): + from vero.sidecar.app import create_app + + components = await build_components( + factory_path=factory_path, + config_path=config_path, + ) + token = generate_admin_token() + write_admin_token(admin_token_path, token) + return create_app( + sidecar=components.sidecar, + verifier=components.verifier, + admin_token=token, + telemetry=components.telemetry, + ) + + +def serve( + *, + factory_path: str, + config_path: Path | str, + admin_token_path: Path | str, + host: str = "0.0.0.0", + port: int = 8000, +) -> None: + import uvicorn + + app = asyncio.run( + build_app( + factory_path=factory_path, + config_path=config_path, + admin_token_path=admin_token_path, + ) + ) + uvicorn.run(app, host=host, port=port) diff --git a/vero/src/vero/sidecar/session.py b/vero/src/vero/sidecar/session.py new file mode 100644 index 00000000..e2874006 --- /dev/null +++ b/vero/src/vero/sidecar/session.py @@ -0,0 +1,235 @@ +"""Portable, reportable session snapshots for ephemeral Harbor environments.""" + +from __future__ import annotations + +import hashlib +import io +import json +import logging +import os +import tarfile +import tempfile +from collections import deque +from datetime import UTC, datetime +from pathlib import Path, PurePosixPath +from typing import Literal + +from pydantic import field_validator + +from vero.evaluation import BackendProvenance +from vero.evaluation.store.persistence import _atomic_write_json +from vero.models import StrictModel +from vero.sidecar.verifier import VerificationSelection, VerificationTarget + +logger = logging.getLogger(__name__) + + +class HarborSessionManifest(StrictModel): + """Trusted metadata needed to interpret an exported Harbor session.""" + + schema_version: Literal[1] = 1 + id: str + task_name: str + task_description: str = "" + created_at: datetime + candidate_repository_family: Literal["git"] = "git" + candidate_repository_format_version: Literal[1] = 1 + backends: dict[str, BackendProvenance] + selection: VerificationSelection + targets: list[VerificationTarget] + + @field_validator("id", "task_name") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("Harbor session identity must not be empty") + return value + + @field_validator("created_at") + @classmethod + def normalize_timestamp(cls, value: datetime) -> datetime: + if value.tzinfo is None or value.utcoffset() is None: + raise ValueError("Harbor session timestamp must be timezone-aware") + return value.astimezone(UTC) + + +def initialize_harbor_session_manifest( + session_dir: Path, + *, + session_id: str, + task_name: str, + task_description: str, + backends: dict[str, BackendProvenance], + selection: VerificationSelection, + targets: list[VerificationTarget], +) -> HarborSessionManifest: + """Create the immutable session identity or validate it on restart.""" + + path = session_dir / "harbor-session.json" + if path.is_file(): + stored = HarborSessionManifest.model_validate_json( + path.read_text(encoding="utf-8") + ) + expected = HarborSessionManifest( + id=session_id, + task_name=task_name, + task_description=task_description, + created_at=stored.created_at, + backends=backends, + selection=selection, + targets=targets, + ) + if stored != expected: + raise ValueError("Harbor session manifest is incompatible with deployment") + return stored + + manifest = HarborSessionManifest( + id=session_id, + task_name=task_name, + task_description=task_description, + created_at=datetime.now(UTC), + backends=backends, + selection=selection, + targets=targets, + ) + _atomic_write_json(path, manifest.model_dump(mode="json")) + return manifest + + +def _walk_without_following_links(root: Path): + """Yield every entry beneath ``root`` without ever descending through a + symbolic link (so the walk cannot escape the tree or loop).""" + queue: deque[Path] = deque([root]) + while queue: + current = queue.popleft() + try: + children = sorted(current.iterdir()) + except OSError: + continue + for child in children: + yield child + if child.is_dir() and not child.is_symlink(): + queue.append(child) + + +def create_harbor_session_archive( + session_dir: Path | str, + destination: Path | str, +) -> Path: + """Atomically archive a session beneath a stable ``session/`` root. + + Resilient: no single entry can abort the export. Symlinks are recorded as + inert ``.symlink`` metadata (not link members, which the extractor + refuses), unreadable/special files are skipped, and every omission is listed + in ``vero-export-skipped.json`` inside the archive. + """ + + source = Path(session_dir).expanduser().resolve() + if not (source / "harbor-session.json").is_file(): + raise FileNotFoundError("Harbor session manifest not found") + + output = Path(destination).expanduser().resolve() + output.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + dir=output.parent, + prefix=f".{output.name}.", + suffix=".tmp", + ) + os.close(descriptor) + temporary = Path(temporary_name) + skipped: list[dict[str, str]] = [] + + def _text_member(archive: tarfile.TarFile, arcname: str, payload: bytes) -> None: + info = tarfile.TarInfo(arcname) + info.size = len(payload) + archive.addfile(info, io.BytesIO(payload)) + + try: + with tarfile.open(temporary, "w:gz") as archive: + archive.add(source, arcname="session", recursive=False) + for path in _walk_without_following_links(source): + arcname = "session/" + path.relative_to(source).as_posix() + if path.is_symlink(): + try: + target = os.readlink(path) + except OSError as error: + target = f"" + _text_member( + archive, + arcname + ".symlink", + (f"symlink -> {target}\n").encode("utf-8"), + ) + skipped.append( + {"path": arcname, "reason": "symlink", "target": target} + ) + elif path.is_dir(): + archive.add(path, arcname=arcname, recursive=False) + elif path.is_file(): + try: + archive.add(path, arcname=arcname, recursive=False) + except OSError as error: + skipped.append( + { + "path": arcname, + "reason": f"unreadable: {error.__class__.__name__}", + } + ) + else: + skipped.append({"path": arcname, "reason": "special-file"}) + if skipped: + _text_member( + archive, + "session/vero-export-skipped.json", + (json.dumps({"skipped": skipped}, indent=2) + "\n").encode("utf-8"), + ) + os.replace(temporary, output) + finally: + temporary.unlink(missing_ok=True) + if skipped: + logger.warning( + "session export: %d entr%s not archived verbatim " + "(symlinks recorded as metadata; special/unreadable files skipped); " + "see session/vero-export-skipped.json", + len(skipped), + "y" if len(skipped) == 1 else "ies", + ) + return output + + +def extract_harbor_session_archive( + archive_path: Path | str, + destination: Path | str, +) -> Path: + """Extract a trusted sidecar export without permitting link or path traversal.""" + + archive_path = Path(archive_path).expanduser().resolve() + destination = Path(destination).expanduser().resolve() + destination.mkdir(parents=True, exist_ok=True) + with tarfile.open(archive_path, "r:gz") as archive: + members = archive.getmembers() + for member in members: + path = PurePosixPath(member.name) + if ( + path.is_absolute() + or not path.parts + or path.parts[0] != "session" + or ".." in path.parts + or member.issym() + or member.islnk() + or member.isdev() + ): + raise ValueError(f"unsafe Harbor session archive member: {member.name}") + archive.extractall(destination, members=members, filter="data") + session = destination / "session" + HarborSessionManifest.model_validate_json( + (session / "harbor-session.json").read_text(encoding="utf-8") + ) + return session + + +def file_sha256(path: Path | str) -> str: + digest = hashlib.sha256() + with Path(path).open("rb") as file: + for chunk in iter(lambda: file.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() diff --git a/vero/src/vero/sidecar/sidecar.py b/vero/src/vero/sidecar/sidecar.py new file mode 100644 index 00000000..1c8b8083 --- /dev/null +++ b/vero/src/vero/sidecar/sidecar.py @@ -0,0 +1,850 @@ +"""Transport-neutral agent frontend over the canonical evaluation engine.""" + +from __future__ import annotations + +import asyncio +import json +import posixpath +from datetime import UTC, datetime +from enum import Enum +from pathlib import Path +from uuid import uuid4 + +from pydantic import Field, JsonValue, field_validator, model_validator + +from vero.candidate import Candidate +from vero.evaluation import ( + AllCases, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationAcknowledgement, + EvaluationAuthorization, + EvaluationBudget, + EvaluationBudgetExceeded, + EvaluationCancelledError, + EvaluationDeniedError, + EvaluationExecutionError, + EvaluationInfrastructureError, + EvaluationLimits, + EvaluationReceipt, + EvaluationRecord, + EvaluationRequest, + EvaluationRequestError, + EvaluationSet, + EvaluationSummary, + EvaluationTerminatedError, + ObjectiveSpec, + project_evaluation, +) +from vero.evaluation.engine import EvaluationEngine +from vero.evaluation.store.persistence import _atomic_write_json +from vero.models import StrictModel +from vero.runtime.context import ( + CANDIDATES_SUBDIRECTORY, + AgentContextDirectory, + AgentDisclosureLedger, + ContextPlanEntry, + context_digest, + make_evaluation_receipt, + narrower_disclosure, +) +from vero.sandbox import LocalSandbox +from vero.sidecar.transport import CandidateTransferError, CandidateTransport + + +class EvaluationAccessError(RuntimeError): + """Raised when an agent requests an evaluation it may not perform.""" + + +class SubmissionDisabledError(RuntimeError): + """Raised when submission is disabled for this optimization task.""" + + +class EvaluationJobNotFoundError(LookupError): + """Raised when an agent requests an unknown evaluation job.""" + + +class SidecarEvaluationPolicy(StrictModel): + """Agent access to one backend-owned evaluation-set partition.""" + + backend_id: str + evaluation_set_name: str + partition: str | None = None + objective: ObjectiveSpec | None = None + # The runtime access truth, compiled by AgentAccessSpec.to_access_policy() + # (disclosure, case-resource exposure, and the k-anonymity floor for + # aggregate subsets all live here, not as parallel flat fields). + access: EvaluationAccessPolicy + parameters: dict[str, JsonValue] = Field(default_factory=dict) + allowed_parameters: list[str] = Field(default_factory=list) + limits: EvaluationLimits | None = None + + @field_validator("backend_id", "evaluation_set_name") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("evaluation access identity must not be empty") + return value + + @field_validator("partition") + @classmethod + def validate_partition(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("evaluation access partition must not be empty") + return value + + @property + def key(self) -> tuple[str, str, str | None]: + return (self.backend_id, self.evaluation_set_name, self.partition) + + @model_validator(mode="after") + def validate_parameters(self) -> SidecarEvaluationPolicy: + if any(not name.strip() for name in self.parameters): + raise ValueError("fixed evaluation parameter names must not be empty") + if len(self.allowed_parameters) != len(set(self.allowed_parameters)): + raise ValueError("allowed evaluation parameters must be unique") + if any(not name.strip() for name in self.allowed_parameters): + raise ValueError("allowed evaluation parameter names must not be empty") + overlap = set(self.parameters) & set(self.allowed_parameters) + if overlap: + raise ValueError( + "fixed and agent-controlled parameters overlap: " + + ", ".join(sorted(overlap)) + ) + return self + + +class SidecarEvaluationRequest(StrictModel): + """Agent request; candidate identity is established by the transport.""" + + backend_id: str + evaluation_set: EvaluationSet + version: str | None = None + parameters: dict[str, JsonValue] = Field(default_factory=dict) + limits: EvaluationLimits | None = None + seed: int | None = None + + @field_validator("backend_id") + @classmethod + def validate_backend_id(cls, value: str) -> str: + if not value.strip(): + raise ValueError("backend_id must not be empty") + return value + + @field_validator("version") + @classmethod + def validate_version(cls, value: str | None) -> str | None: + if value is not None and not value.strip(): + raise ValueError("candidate version must not be empty") + return value + + +EvaluationProjection = EvaluationRecord | EvaluationSummary | EvaluationAcknowledgement + + +class SidecarEvaluationResult(StrictModel): + disclosure: DisclosureLevel + receipt: EvaluationReceipt + + @model_validator(mode="after") + def validate_projection(self) -> SidecarEvaluationResult: + if self.receipt.disclosure != self.disclosure: + raise ValueError("sidecar disclosure must match its receipt") + return self + + +class EvaluationJobStatus(str, Enum): + QUEUED = "queued" + RUNNING = "running" + COMPLETE = "complete" + FAILED = "failed" + CANCELLED = "cancelled" + + +class SidecarEvaluationJob(StrictModel): + """Durable agent-facing lifecycle for one sidecar evaluation request.""" + + job_id: str + status: EvaluationJobStatus + backend_id: str + evaluation_set: EvaluationSet + version: str | None = None + evaluation_id: str | None = None + receipt: EvaluationReceipt | None = None + error: str | None = None + created_at: datetime + completed_at: datetime | None = None + + @model_validator(mode="after") + def validate_lifecycle(self) -> SidecarEvaluationJob: + terminal = self.status in { + EvaluationJobStatus.COMPLETE, + EvaluationJobStatus.FAILED, + EvaluationJobStatus.CANCELLED, + } + if terminal != (self.completed_at is not None): + raise ValueError("only terminal evaluation jobs have completed_at") + if self.status == EvaluationJobStatus.COMPLETE and self.receipt is None: + raise ValueError("complete evaluation jobs require a receipt") + if self.receipt is not None: + if self.status != EvaluationJobStatus.COMPLETE: + raise ValueError("only complete evaluation jobs may have a receipt") + if self.evaluation_id != self.receipt.evaluation_id: + raise ValueError("evaluation job and receipt identities must match") + if self.error is not None and self.status not in { + EvaluationJobStatus.FAILED, + EvaluationJobStatus.CANCELLED, + }: + raise ValueError("only failed or cancelled jobs may have an error") + return self + + +class EvaluationAccessStatus(StrictModel): + backend_id: str + evaluation_set_name: str + partition: str | None + disclosure: DisclosureLevel + expose_case_resources: bool + min_aggregate_cases: int + allowed_parameters: list[str] + limits: EvaluationLimits | None = None + budget: EvaluationBudget | None = None + + +class SidecarStatus(StrictModel): + submit_enabled: bool + evaluation_access: list[EvaluationAccessStatus] + inference_usage: dict[str, JsonValue] | None = None + evaluation_jobs: list[SidecarEvaluationJob] = Field(default_factory=list) + + +class Submission(StrictModel): + candidate: Candidate + + +class EvaluationSidecar: + """Meter, evaluate, and disclose candidates imported from an agent repo. + + The sidecar supports any number of registered backends. Unknown evaluation + sets fail closed, and every accepted request supplies an explicit canonical + authorization to the engine. + """ + + def __init__( + self, + *, + engine: EvaluationEngine, + candidate_transport: CandidateTransport, + access_policies: list[SidecarEvaluationPolicy], + agent_volume: Path | None = None, + admin_volume: Path | None = None, + inference_usage_path: Path | None = None, + inference_limits: dict[str, dict[str, JsonValue]] | None = None, + submit_enabled: bool = False, + disclose_budget: bool = True, + ): + self.engine = engine + self.candidate_transport = candidate_transport + self.agent_volume = Path(agent_volume) if agent_volume is not None else None + self.admin_volume = Path(admin_volume) if admin_volume is not None else None + self.inference_usage_path = ( + Path(inference_usage_path) if inference_usage_path is not None else None + ) + self.inference_limits = inference_limits or {} + self.submit_enabled = submit_enabled + self.disclose_budget = disclose_budget + self._context_lock = asyncio.Lock() + self._context_initialized = False + self._evaluation_jobs_dir = ( + self.engine.evaluator.session_dir / "evaluation-jobs" + ) + self._evaluation_jobs_dir.mkdir(parents=True, exist_ok=True) + self._evaluation_jobs: dict[str, SidecarEvaluationJob] = {} + self._evaluation_job_tasks: dict[str, asyncio.Task[None]] = {} + self._load_evaluation_jobs() + self._disclosures = AgentDisclosureLedger( + self.engine.evaluator.session_dir / "agent-context.json" + ) + self._context_directory = ( + AgentContextDirectory( + sandbox=LocalSandbox(self.agent_volume.parent), + root=str(self.agent_volume), + session_dir=self.engine.evaluator.session_dir, + ) + if self.agent_volume is not None + else None + ) + self._policies: dict[tuple[str, str, str | None], SidecarEvaluationPolicy] = {} + for policy in access_policies: + if policy.key in self._policies: + raise ValueError( + f"duplicate evaluation access policy for {policy.key!r}" + ) + if policy.backend_id not in engine.backends: + raise ValueError( + f"access policy references unknown backend {policy.backend_id!r}" + ) + self._policies[policy.key] = policy + + def _load_evaluation_jobs(self) -> None: + """Restore terminal jobs and mark interrupted in-flight jobs explicitly.""" + + for path in sorted(self._evaluation_jobs_dir.glob("*.json")): + try: + job = SidecarEvaluationJob.model_validate_json( + path.read_text(encoding="utf-8") + ) + except (OSError, ValueError): + continue + if job.status in { + EvaluationJobStatus.QUEUED, + EvaluationJobStatus.RUNNING, + }: + job = job.model_copy( + update={ + "status": EvaluationJobStatus.FAILED, + "error": "evaluation job was interrupted by a sidecar restart", + "completed_at": datetime.now(UTC), + } + ) + _atomic_write_json(path, job.model_dump(mode="json")) + self._evaluation_jobs[job.job_id] = job + + async def _save_evaluation_job(self, job: SidecarEvaluationJob) -> None: + self._evaluation_jobs[job.job_id] = job + await asyncio.to_thread( + _atomic_write_json, + self._evaluation_jobs_dir / f"{job.job_id}.json", + job.model_dump(mode="json"), + ) + + async def _update_evaluation_job( + self, + job_id: str, + **updates: object, + ) -> SidecarEvaluationJob: + job = self._evaluation_jobs[job_id].model_copy(update=updates) + job = SidecarEvaluationJob.model_validate(job) + await self._save_evaluation_job(job) + return job + + def evaluation_job(self, job_id: str) -> SidecarEvaluationJob: + try: + return self._evaluation_jobs[job_id] + except KeyError as error: + raise EvaluationJobNotFoundError( + f"unknown evaluation job {job_id!r}" + ) from error + + async def start_evaluation_job( + self, + request: SidecarEvaluationRequest, + ) -> SidecarEvaluationJob: + """Admit an evaluation and return without waiting for its result.""" + + job = SidecarEvaluationJob( + job_id=str(uuid4()), + status=EvaluationJobStatus.QUEUED, + backend_id=request.backend_id, + evaluation_set=request.evaluation_set, + version=request.version, + created_at=datetime.now(UTC), + ) + await self._save_evaluation_job(job) + admitted = asyncio.Event() + task = asyncio.create_task( + self._run_detached_job(job.job_id, request, admitted), + name=f"vero-evaluation-job-{job.job_id}", + ) + self._evaluation_job_tasks[job.job_id] = task + task.add_done_callback( + lambda _task, job_id=job.job_id: self._evaluation_job_tasks.pop( + job_id, None + ) + ) + await admitted.wait() + return self.evaluation_job(job.job_id) + + async def _execute_tracked_job( + self, + job_id: str, + request: SidecarEvaluationRequest, + admitted: asyncio.Event, + ) -> SidecarEvaluationResult: + """Run one evaluation, recording it through its job lifecycle. + + Shared by the synchronous and detached entry points so every + evaluation is a tracked job visible in status. Re-raises on failure + after recording the terminal state, so the synchronous caller surfaces + the real error (mapped to HTTP by the app handlers) and the detached + wrapper can record-and-swallow. + """ + try: + async with self.engine.agent_evaluation_scope(): + policy, canonical_request = await self._prepare_evaluation(request) + await self._update_evaluation_job( + job_id, + status=EvaluationJobStatus.RUNNING, + version=canonical_request.candidate.version, + ) + admitted.set() + result = await self._evaluate_prepared( + backend_id=request.backend_id, + policy=policy, + request=canonical_request, + ) + await self._update_evaluation_job( + job_id, + status=EvaluationJobStatus.COMPLETE, + evaluation_id=result.receipt.evaluation_id, + receipt=result.receipt, + completed_at=datetime.now(UTC), + ) + return result + except asyncio.CancelledError: + admitted.set() + await asyncio.shield( + self._update_evaluation_job( + job_id, + status=EvaluationJobStatus.CANCELLED, + receipt=None, + error="evaluation job was cancelled", + completed_at=datetime.now(UTC), + ) + ) + raise + except Exception as error: # the terminal record remains in agent context + admitted.set() + evaluation_id = getattr(error, "evaluation_id", None) + await self._update_evaluation_job( + job_id, + status=EvaluationJobStatus.FAILED, + evaluation_id=evaluation_id, + receipt=None, + error=self._evaluation_job_error(error), + completed_at=datetime.now(UTC), + ) + raise + + async def _run_detached_job( + self, + job_id: str, + request: SidecarEvaluationRequest, + admitted: asyncio.Event, + ) -> None: + """Detached wrapper: the terminal state is already recorded on the job + (which the agent polls), so a non-cancellation failure is swallowed + here to avoid an unretrieved background-task exception.""" + try: + await self._execute_tracked_job(job_id, request, admitted) + except asyncio.CancelledError: + raise + except Exception: + pass + + @staticmethod + def _evaluation_job_error(error: Exception) -> str: + # Terminating conditions subclass EvaluationExecutionError, so surface + # their reason first: an out-of-budget or auth-failed run should say so. + if isinstance(error, EvaluationTerminatedError): + return f"evaluation terminated: {error}" + if isinstance(error, EvaluationBudgetExceeded): + return "evaluation budget exhausted" + if isinstance(error, EvaluationInfrastructureError): + return "infrastructure failure" + if isinstance(error, (EvaluationDeniedError, EvaluationAccessError)): + return "evaluation denied" + if isinstance(error, EvaluationRequestError): + # Same reasoning as the blocking path in sidecar/app.py: the message + # names the backend capability that rejected the request, which the + # agent needs in order to reissue it correctly. + return f"invalid evaluation request: {error}" if str(error) else ( + "invalid evaluation request" + ) + if isinstance(error, CandidateTransferError): + return "candidate version could not be imported" + # Unmapped failure: include the exception type (a class name, so no + # secret leak) so the agent and operators can tell failures apart + # instead of guessing from an opaque "evaluation failed". + return f"evaluation failed: {type(error).__name__}" + + def _policy( + self, + backend_id: str, + evaluation_set: EvaluationSet, + ) -> SidecarEvaluationPolicy: + key = (backend_id, evaluation_set.name, evaluation_set.partition) + policy = self._policies.get(key) + if policy is None or not policy.access.agent_can_evaluate: + raise EvaluationAccessError( + "the requested backend and evaluation set are not agent-evaluable" + ) + return policy + + async def _enforce_aggregate_floor( + self, + policy: SidecarEvaluationPolicy, + evaluation_set: EvaluationSet, + ) -> None: + if policy.access.disclosure != DisclosureLevel.AGGREGATE: + return + if isinstance(evaluation_set.selection, AllCases): + return + backend = self.engine.backends.resolve(policy.backend_id) + cost = await backend.resolve_cost(evaluation_set) + if cost.cases is None: + raise EvaluationAccessError( + "aggregate subset evaluation requires a backend with exact case costs" + ) + if cost.cases < policy.access.min_aggregate_cases: + raise EvaluationAccessError( + f"aggregate subset evaluations must cover at least " + f"{policy.access.min_aggregate_cases} cases; requested {cost.cases}" + ) + + _BUDGET_METRIC_MARKERS = ("inference_", "agent_reported_") + + def _redact_budget_metrics(self, record: EvaluationRecord) -> EvaluationRecord: + """Budget-blind mode: cost metrics — gateway-metered ``inference_*`` + and self-declared ``agent_reported_*``, report- and case-level — are + budget signal and must not reach the agent (enforcement unchanged). + + Matched anywhere in the key, not just as a prefix: the report also + carries distribution wrappers of these metrics + (``mean_case_inference_input_tokens``, + ``median_case_agent_reported_output_tokens``), which are the same + signal under a different name. Latency keys carry no marker and stay + visible. + """ + if self.disclose_budget: + return record + + def keep(metrics: dict[str, float]) -> dict[str, float]: + return { + key: value + for key, value in metrics.items() + if not any(marker in key for marker in self._BUDGET_METRIC_MARKERS) + } + + metrics = keep(record.report.metrics) + cases = [ + case.model_copy(update={"metrics": kept}) + if len(kept := keep(case.metrics)) != len(case.metrics) + else case + for case in record.report.cases + ] + if len(metrics) == len(record.report.metrics) and all( + kept is original for kept, original in zip(cases, record.report.cases) + ): + return record + return record.model_copy( + update={ + "report": record.report.model_copy( + update={"metrics": metrics, "cases": cases} + ) + } + ) + + def _visible_projections( + self, + ) -> list[tuple[EvaluationRecord, DisclosureLevel, EvaluationProjection]]: + projections = [] + for evaluation_id, entry in self._disclosures.model.evaluations.items(): + record = self.engine.database.get_evaluation(evaluation_id) + if record is None: + continue + record = self._redact_budget_metrics(record) + policy = self._policies.get( + ( + record.backend_id, + record.request.evaluation_set.name, + record.request.evaluation_set.partition, + ) + ) + if policy is None or not policy.access.agent_can_evaluate: + continue + disclosure = narrower_disclosure( + entry.maximum_disclosure, + policy.access.disclosure, + ) + projections.append( + (record, disclosure, project_evaluation(record, disclosure)) + ) + return projections + + async def _write_candidate_index( + self, + projections: list[ + tuple[EvaluationRecord, DisclosureLevel, EvaluationProjection] + ], + ) -> None: + assert self._context_directory is not None + root = self._context_directory.path(CANDIDATES_SUBDIRECTORY) + sandbox = self._context_directory.sandbox + if await sandbox.exists(root): + await sandbox.remove(root, recursive=True) + await sandbox.mkdir(root) + candidates = { + record.request.candidate.id: record.request.candidate + for record, _, _ in projections + } + index = [] + for candidate in sorted( + candidates.values(), + key=lambda item: (item.created_at, item.id), + ): + digest = context_digest(candidate.id) + directory = self._context_directory.path("candidates", digest) + await sandbox.mkdir(directory) + await sandbox.write_file( + posixpath.join(directory, "candidate.json"), + candidate.model_dump_json(indent=2) + "\n", + ) + index.append( + { + "candidate_id": candidate.id, + "version": candidate.version, + "parent_id": candidate.parent_id, + "native_ref": candidate.version, + "metadata_path": f"{digest}/candidate.json", + "parent_patch_path": None, + } + ) + await self._context_directory.write_json( + self._context_directory.path(CANDIDATES_SUBDIRECTORY, "index.json"), + {"schema_version": 1, "candidates": index}, + ) + + def _context_entries(self) -> list[ContextPlanEntry]: + ledger = self.engine.budget_ledger + entries = [] + for policy in self._policies.values(): + evaluation_set = EvaluationSet( + name=policy.evaluation_set_name, + partition=policy.partition, + ) + entries.append( + ContextPlanEntry( + backend_id=policy.backend_id, + backend=self.engine.backends.resolve(policy.backend_id), + evaluation_set=evaluation_set, + access=policy.access, + # Budget is enforced regardless; disclosure is what we gate. + budget=( + ledger.get(policy.backend_id, evaluation_set) + if ledger is not None and self.disclose_budget + else None + ), + ) + ) + return entries + + async def initialize_context(self) -> None: + if self._context_directory is None: + return + async with self._context_lock: + await self._context_directory.reset() + await self._context_directory.write_header( + session_id=self.engine.evaluator.session_id, + round_number=None, + proposal_id=None, + parent_candidate_id=None, + ) + projections = self._visible_projections() + entries = self._context_entries() + await self._write_candidate_index(projections) + await self._context_directory.write_evaluations(projections) + await self._context_directory.write_evaluation_plan(entries) + await self._context_directory.write_case_resources(entries) + self._context_initialized = True + + async def _refresh_context(self) -> None: + if self._context_directory is None: + return + if not self._context_initialized: + await self.initialize_context() + return + async with self._context_lock: + projections = self._visible_projections() + await self._write_candidate_index(projections) + await self._context_directory.write_evaluations(projections) + await self._context_directory.write_evaluation_plan( + self._context_entries() + ) + + async def evaluate( + self, + request: SidecarEvaluationRequest, + ) -> SidecarEvaluationResult: + """Run an evaluation synchronously, recording it as a tracked job so it + is visible in status exactly like a detached one, then return its + result (or re-raise its failure).""" + job = SidecarEvaluationJob( + job_id=str(uuid4()), + status=EvaluationJobStatus.QUEUED, + backend_id=request.backend_id, + evaluation_set=request.evaluation_set, + version=request.version, + created_at=datetime.now(UTC), + ) + await self._save_evaluation_job(job) + return await self._execute_tracked_job(job.job_id, request, asyncio.Event()) + + async def _prepare_evaluation( + self, + request: SidecarEvaluationRequest, + ) -> tuple[SidecarEvaluationPolicy, EvaluationRequest]: + policy = self._policy(request.backend_id, request.evaluation_set) + unknown_parameters = sorted( + set(request.parameters) - set(policy.allowed_parameters) + ) + if unknown_parameters: + raise EvaluationAccessError( + "evaluation parameters are not agent-controllable: " + + ", ".join(unknown_parameters) + ) + if policy.limits is not None and request.limits is not None: + raise EvaluationAccessError( + "evaluation limits are fixed by the access policy and cannot be " + "overridden by the agent" + ) + await self._enforce_aggregate_floor(policy, request.evaluation_set) + candidate = await self.candidate_transport.import_candidate(request.version) + parameters = {**policy.parameters, **request.parameters} + return policy, EvaluationRequest( + candidate=candidate, + evaluation_set=request.evaluation_set, + parameters=parameters, + limits=policy.limits or request.limits or EvaluationLimits(), + seed=request.seed, + ) + + async def _evaluate_prepared( + self, + *, + backend_id: str, + policy: SidecarEvaluationPolicy, + request: EvaluationRequest, + ) -> SidecarEvaluationResult: + try: + result = await self.engine.evaluate( + backend_id=backend_id, + request=request, + objective_spec=policy.objective, + authorization=EvaluationAuthorization( + may_evaluate=True, + meter_budget=True, + disclosure=policy.access.disclosure, + expose_case_resources=policy.access.expose_case_resources, + min_aggregate_cases=policy.access.min_aggregate_cases or 1, + ), + ) + except (EvaluationExecutionError, EvaluationCancelledError) as error: + record = self.engine.database.get_evaluation(error.evaluation_id) + if record is not None: + await self._disclosures.remember(record.id, policy.access.disclosure) + await asyncio.shield(self._refresh_context()) + raise + evaluation_id = ( + result.id if isinstance(result, EvaluationRecord) else result.evaluation_id + ) + record = self.engine.database.get_evaluation(evaluation_id) + if record is None: + raise RuntimeError( + f"evaluation engine did not index completed evaluation {evaluation_id!r}" + ) + maximum = await self._disclosures.remember(record.id, policy.access.disclosure) + disclosure = narrower_disclosure(maximum, policy.access.disclosure) + await self._refresh_context() + return SidecarEvaluationResult( + disclosure=disclosure, + receipt=make_evaluation_receipt( + self._redact_budget_metrics(record), disclosure + ), + ) + + async def submit(self, version: str | None = None) -> Submission: + if not self.submit_enabled: + raise SubmissionDisabledError("candidate submission is disabled") + candidate = await self.candidate_transport.import_candidate(version) + submission = Submission(candidate=candidate) + if self.admin_volume is not None: + await asyncio.to_thread( + _atomic_write_json, + self.admin_volume / "submission.json", + submission.model_dump(mode="json"), + ) + return submission + + def status(self) -> SidecarStatus: + access: list[EvaluationAccessStatus] = [] + for policy in self._policies.values(): + if not policy.access.agent_can_evaluate: + continue + evaluation_set = EvaluationSet( + name=policy.evaluation_set_name, + partition=policy.partition, + ) + budget = ( + self.engine.budget_ledger.get(policy.backend_id, evaluation_set) + if self.engine.budget_ledger is not None + else None + ) + access.append( + EvaluationAccessStatus( + backend_id=policy.backend_id, + evaluation_set_name=policy.evaluation_set_name, + partition=policy.partition, + disclosure=policy.access.disclosure, + expose_case_resources=policy.access.expose_case_resources, + min_aggregate_cases=policy.access.min_aggregate_cases, + allowed_parameters=list(policy.allowed_parameters), + limits=policy.limits, + # Budget is enforced regardless; disclosure is what we gate. + budget=budget if self.disclose_budget else None, + ) + ) + inference_usage: dict[str, JsonValue] | None = None + observed_scopes: dict[str, object] = {} + if self.inference_usage_path is not None and self.inference_usage_path.exists(): + try: + value = self.inference_usage_path.read_text(encoding="utf-8") + parsed = json.loads(value) + scopes = parsed.get("scopes") if isinstance(parsed, dict) else None + if isinstance(scopes, dict): + observed_scopes = scopes + except (OSError, ValueError): + observed_scopes = {} + if self.inference_limits and self.disclose_budget: + inference_usage = {} + for name, limits in self.inference_limits.items(): + observed = observed_scopes.get(name) + usage = observed if isinstance(observed, dict) else {} + requests = usage.get("requests", 0) + total_tokens = usage.get("total_tokens", 0) + max_requests = limits.get("max_requests") + max_tokens = limits.get("max_tokens") + inference_usage[name] = { + **limits, + **usage, + "remaining_requests": ( + None + if max_requests is None + else max(0, int(max_requests) - int(requests)) + ), + "remaining_tokens": ( + None + if max_tokens is None + else max(0, int(max_tokens) - int(total_tokens)) + ), + } + return SidecarStatus( + submit_enabled=self.submit_enabled, + evaluation_access=access, + inference_usage=inference_usage, + evaluation_jobs=sorted( + self._evaluation_jobs.values(), + key=lambda job: (job.created_at, job.job_id), + reverse=True, + )[:100], + ) diff --git a/vero/src/vero/sidecar/transport.py b/vero/src/vero/sidecar/transport.py new file mode 100644 index 00000000..519bbac8 --- /dev/null +++ b/vero/src/vero/sidecar/transport.py @@ -0,0 +1,190 @@ +"""Candidate transfer across the Harbor sidecar trust boundary.""" + +from __future__ import annotations + +import os +import re +from datetime import datetime +from pathlib import PurePosixPath +from typing import Protocol, runtime_checkable + +from vero.candidate import Candidate +from vero.candidate_repository import GitCandidateRepository +from vero.workspace import Workspace + + +class CandidateTransferError(RuntimeError): + """Raised when an untrusted candidate cannot be imported safely.""" + + +@runtime_checkable +class CandidateTransport(Protocol): + """Import an external program version into durable candidate storage.""" + + async def import_candidate(self, version: str | None = None) -> Candidate: ... + + +class GitCandidateTransport: + """Copy a commit from an agent repository into a trusted candidate repository. + + The source ref is resolved before fetching, the object is fetched through a + unique temporary ref, and the imported commit receives a durable candidate + record. Later verifier evaluations do not depend on the agent repository + still retaining the object. + """ + + _OBJECT_ID = re.compile(r"(?:[0-9a-f]{40}|[0-9a-f]{64})\Z") + + def __init__( + self, + *, + workspace: Workspace, + candidate_repository: GitCandidateRepository, + agent_repo_path: str, + ): + if not agent_repo_path.startswith("/"): + raise ValueError("agent_repo_path must be an absolute sandbox path") + if PurePosixPath(agent_repo_path) == PurePosixPath(workspace.root): + raise ValueError("agent and trusted repositories must be distinct") + self.workspace = workspace + self.candidate_repository = candidate_repository + self.agent_repo_path = agent_repo_path.rstrip("/") or "/" + self._candidates: dict[str, Candidate] = {} + + async def _run( + self, + command: list[str], + *, + cwd: str, + timeout: int, + ) -> str: + if command[0] != "git": + raise ValueError("candidate transport only permits Git commands") + command = [ + "git", + "-c", + f"safe.directory={cwd}", + *command[1:], + ] + result = await self.workspace.sandbox.run( + command, + cwd=cwd, + timeout=timeout, + env={ + "PATH": os.defpath, + "LANG": "C.UTF-8", + "GIT_CONFIG_GLOBAL": "/dev/null", + }, + ) + if result.returncode != 0: + message = result.stderr.strip() or result.stdout.strip() or "git failed" + raise CandidateTransferError(message) + return result.stdout.strip() + + async def _resolve_ref(self, version: str, *, repository: str) -> str: + value = await self._run( + [ + "git", + "-c", + "core.hooksPath=/dev/null", + "rev-parse", + "--verify", + "--end-of-options", + f"{version}^{{commit}}", + ], + cwd=repository, + timeout=30, + ) + if self._OBJECT_ID.fullmatch(value) is None: + raise CandidateTransferError( + "source ref did not resolve to a Git object ID" + ) + return value + + async def trusted_candidate(self, version: str | None = None) -> Candidate: + """Resolve an existing commit already owned by the trusted workspace.""" + source_version = version or "HEAD" + if not source_version.strip() or "\x00" in source_version: + raise CandidateTransferError("candidate version must not be empty") + object_id = await self._resolve_ref( + source_version, + repository=self.workspace.root, + ) + cached = self._candidates.get(object_id) + if cached is None: + cached = await self._candidate_metadata( + object_id, + repository=self.workspace.root, + ) + cached = await self.candidate_repository.import_candidate( + cached, + sandbox=self.workspace.sandbox, + repository_path=self.workspace.root, + ) + self._candidates[object_id] = cached + return cached + + async def _candidate_metadata( + self, + object_id: str, + *, + repository: str, + ) -> Candidate: + value = await self._run( + [ + "git", + "-c", + "core.hooksPath=/dev/null", + "show", + "-s", + "--format=%cI%x00%P%x00%T%x00%s", + "--end-of-options", + object_id, + ], + cwd=repository, + timeout=30, + ) + parts = value.split("\x00", 3) + if len(parts) != 4: + raise CandidateTransferError("could not read imported commit metadata") + timestamp, parents, tree, subject = parts + if self._OBJECT_ID.fullmatch(tree) is None: + raise CandidateTransferError("imported commit has an invalid tree identity") + try: + created_at = datetime.fromisoformat(timestamp) + except ValueError as error: + raise CandidateTransferError( + "imported commit has an invalid timestamp" + ) from error + parent_id = parents.split()[0] if parents.strip() else None + return Candidate( + id=object_id, + version=object_id, + parent_id=parent_id, + created_at=created_at, + description=subject.strip() or None, + metadata={"transport": "git", "content_digest": tree}, + ) + + async def import_candidate(self, version: str | None = None) -> Candidate: + source_version = version or "HEAD" + if not source_version.strip() or "\x00" in source_version: + raise CandidateTransferError("candidate version must not be empty") + object_id = await self._resolve_ref( + source_version, + repository=self.agent_repo_path, + ) + cached = self._candidates.get(object_id) + if cached is not None: + return cached + candidate = await self._candidate_metadata( + object_id, + repository=self.agent_repo_path, + ) + candidate = await self.candidate_repository.import_candidate( + candidate, + sandbox=self.workspace.sandbox, + repository_path=self.agent_repo_path, + ) + self._candidates[object_id] = candidate + return candidate diff --git a/vero/src/vero/sidecar/verifier.py b/vero/src/vero/sidecar/verifier.py new file mode 100644 index 00000000..64a287d9 --- /dev/null +++ b/vero/src/vero/sidecar/verifier.py @@ -0,0 +1,582 @@ +"""Admin-side candidate selection and final evaluation for Harbor tasks.""" + +from __future__ import annotations + +import asyncio +import logging +import math +import statistics +from collections import defaultdict +from pathlib import Path +from typing import Callable, Literal + +from pydantic import Field, JsonValue, field_validator, model_validator + +from vero.candidate import Candidate +from vero.evaluation import ( + EvaluationAuthorization, + EvaluationLimits, + EvaluationPrincipal, + EvaluationRecord, + EvaluationRequest, + EvaluationSet, + ObjectiveSpec, +) +from vero.evaluation.engine import EvaluationEngine +from vero.evaluation.store.persistence import _atomic_write_json +from vero.models import StrictModel +from vero.sidecar.sidecar import Submission + +logger = logging.getLogger(__name__) + + +class NoCandidateError(RuntimeError): + """Raised when finalization has no submitted or evaluated candidate.""" + + +class VerificationTarget(StrictModel): + """One trusted final evaluation projected to one Harbor reward key.""" + + reward_key: str + backend_id: str + evaluation_set: EvaluationSet + objective: ObjectiveSpec + parameters: dict[str, JsonValue] = Field(default_factory=dict) + limits: EvaluationLimits = Field(default_factory=EvaluationLimits) + failure_value: float = 0.0 + reward_scale: float = 1.0 + reward_offset: float = 0.0 + # Pin the seed's reward on this target (post scale/offset) to skip scoring + # the fixed seed each run — reproducibility + one fewer eval per run. + baseline_reward: float | None = None + # Keep one attempt by default for adversarial candidates. Retrying an + # unmeasurable candidate-controlled run creates a one-sided re-roll. + max_attempts: int = Field(default=1, ge=1) + + @field_validator("reward_key", "backend_id") + @classmethod + def validate_identity(cls, value: str) -> str: + if not value.strip(): + raise ValueError("verification target identity must not be empty") + return value + + @field_validator("failure_value", "reward_scale", "reward_offset") + @classmethod + def validate_failure_value(cls, value: float) -> float: + if not math.isfinite(value): + raise ValueError("verification reward values must be finite") + return value + + @field_validator("baseline_reward") + @classmethod + def validate_baseline_reward(cls, value: float | None) -> float | None: + if value is not None and not math.isfinite(value): + raise ValueError("baseline_reward must be finite") + return value + + +class VerificationSelection(StrictModel): + """How finalization chooses a candidate before scoring its targets.""" + + mode: Literal["submit", "auto_best"] = "auto_best" + backend_id: str | None = None + evaluation_set: EvaluationSet | None = None + objective: ObjectiveSpec | None = None + baseline_candidate: Candidate | None = None + parameters: dict[str, JsonValue] = Field(default_factory=dict) + limits: EvaluationLimits = Field(default_factory=EvaluationLimits) + rescore_top_k: int = Field(default=3, ge=1) + rescore_attempts: int = Field(default=1, ge=1) + # Off by default: the floor gates a ship on a validation comparison while + # the reward is on the (possibly differently-distributed) target. + baseline_floor: bool = False + # Pin the seed's selection-partition score to skip re-scoring it each run. + baseline_selection_score: float | None = None + selection_coverage_threshold: float = Field(default=0.9, ge=0.0, le=1.0) + + @model_validator(mode="after") + def validate_auto_best(self) -> VerificationSelection: + fields = (self.backend_id, self.evaluation_set, self.objective) + if self.mode == "auto_best" and any(value is None for value in fields): + raise ValueError( + "auto_best selection requires backend_id, evaluation_set, and objective" + ) + if self.backend_id is not None and not self.backend_id.strip(): + raise ValueError("selection backend_id must not be empty") + if ( + self.baseline_floor + and self.mode == "auto_best" + and self.baseline_candidate is None + ): + raise ValueError("baseline_floor requires baseline_candidate") + return self + + +class VerificationResult(StrictModel): + """Durable, idempotent output consumed by Harbor's verifier.""" + + candidate: Candidate | None = None + #: Whether a candidate was actually shipped. False means selection produced + #: nothing (the rewards are failure values recording "no candidate"), which + #: a benchmark must treat as a distinct outcome from a candidate that + #: shipped and legitimately scored the failure value. + shipped: bool = True + rewards: dict[str, float] + evaluation_ids: dict[str, str] = Field(default_factory=dict) + baseline_rewards: dict[str, float] = Field(default_factory=dict) + #: Per reward key: the scoring evaluation's full report metrics (accuracy + #: plus cost/latency telemetry such as inference_*_tokens and + #: mean_case_wall_seconds). Informational — the reward itself remains the + #: objective value alone. + reward_metrics: dict[str, dict[str, float]] = Field(default_factory=dict) + baseline_reward_metrics: dict[str, dict[str, float]] = Field(default_factory=dict) + errors: dict[str, str] = Field(default_factory=dict) + + @field_validator("rewards", "baseline_rewards") + @classmethod + def validate_rewards(cls, value: dict[str, float]) -> dict[str, float]: + for key, reward in value.items(): + if not key.strip() or not math.isfinite(reward): + raise ValueError("verification rewards require names and finite values") + return value + + +class CanonicalVerifier: + """Select, re-measure, and score candidates through one evaluation engine.""" + + def __init__( + self, + *, + engine: EvaluationEngine, + selection: VerificationSelection, + targets: list[VerificationTarget], + admin_volume: Path, + score_baseline: bool = True, + evaluation_drain_timeout_seconds: float = 600.0, + on_finalized: Callable[[VerificationResult], None] | None = None, + ): + if not targets: + raise ValueError("at least one verification target is required") + reward_keys = [target.reward_key for target in targets] + if len(reward_keys) != len(set(reward_keys)): + raise ValueError("verification reward keys must be unique") + for target in targets: + if target.backend_id not in engine.backends: + raise ValueError( + f"verification target references unknown backend {target.backend_id!r}" + ) + if ( + selection.backend_id is not None + and selection.backend_id not in engine.backends + ): + raise ValueError( + f"selection references unknown backend {selection.backend_id!r}" + ) + self.engine = engine + self.selection = selection + self.targets = targets + self.admin_volume = Path(admin_volume) + self.score_baseline = score_baseline + if evaluation_drain_timeout_seconds <= 0: + raise ValueError("evaluation drain timeout must be positive") + self.evaluation_drain_timeout_seconds = evaluation_drain_timeout_seconds + self._on_finalized = on_finalized + self._lock = asyncio.Lock() + self._result: VerificationResult | None = None + + @property + def result_path(self) -> Path: + return self.admin_volume / "finalize.json" + + @property + def submission_path(self) -> Path: + return self.admin_volume / "submission.json" + + def _try_load_submission(self) -> Candidate | None: + if not self.submission_path.exists(): + return None + try: + submission = Submission.model_validate_json( + self.submission_path.read_text(encoding="utf-8") + ) + except Exception as error: + raise ValueError(f"invalid durable submission: {error}") from error + return submission.candidate + + def _selection_records(self) -> list[EvaluationRecord]: + assert self.selection.backend_id is not None + assert self.selection.evaluation_set is not None + assert self.selection.objective is not None + baseline_version = ( + self.selection.baseline_candidate.version + if self.selection.baseline_candidate is not None + else None + ) + # Match on backend, partition, and objective (not exact evaluation_set + # equality) so a range/subset that samples the set still counts. + selection_set = self.selection.evaluation_set + matching: list[EvaluationRecord] = [] + for record in self.engine.database.evaluations.values(): + objective = record.objective + evaluation_set = record.request.evaluation_set + if ( + record.backend_id != self.selection.backend_id + or evaluation_set.name != selection_set.name + or evaluation_set.partition != selection_set.partition + or record.objective_spec != self.selection.objective + or objective is None + or not objective.feasible + or objective.value is None + or record.request.candidate.version == baseline_version + ): + continue + matching.append(record) + if not matching: + return [] + # Coverage threshold relative to the best-covered eval seen — the closest + # proxy for the full set without re-enumerating it. An under-measured + # eval is too noisy to rank on; the admin re-score on the full selection + # set (in _auto_best) is the actual ship gate. + coverage = { + id(record): len({case.case_id for case in record.report.cases}) + for record in matching + } + best = max(coverage.values()) + min_cases = math.ceil(best * self.selection.selection_coverage_threshold) + return [record for record in matching if coverage[id(record)] >= min_cases] + + def _shortlist(self) -> list[Candidate]: + records = self._selection_records() + if not records: + return [] + + by_content: dict[str, list[EvaluationRecord]] = defaultdict(list) + for record in records: + candidate = record.request.candidate + content = candidate.metadata.get("content_digest") + key = str(content) if content is not None else candidate.version + by_content[key].append(record) + + pooled: list[tuple[float, Candidate]] = [] + for group in by_content.values(): + values = [record.objective.value for record in group] + representative = min( + (record.request.candidate for record in group), + key=lambda candidate: candidate.id, + ) + pooled.append((statistics.fmean(values), representative)) + pooled.sort(key=lambda item: item[1].id) + pooled.sort( + key=lambda item: item[0], + reverse=self.selection.objective.direction == "maximize", + ) + return [candidate for _, candidate in pooled[: self.selection.rescore_top_k]] + + async def _evaluate( + self, + *, + candidate: Candidate, + backend_id: str, + evaluation_set: EvaluationSet, + objective: ObjectiveSpec, + parameters: dict[str, JsonValue], + limits: EvaluationLimits, + max_attempts: int, + ) -> tuple[EvaluationRecord | None, str | None]: + last_error: str | None = None + for attempt in range(1, max_attempts + 1): + try: + record = await self.engine.evaluate_record( + backend_id=backend_id, + request=EvaluationRequest( + candidate=candidate, + evaluation_set=evaluation_set, + parameters=parameters, + limits=limits, + ), + objective_spec=objective, + authorization=EvaluationAuthorization( + may_evaluate=True, + meter_budget=False, + disclosure="full", + ), + principal=EvaluationPrincipal.ADMIN, + ) + if ( + record.objective is not None + and record.objective.feasible + and record.objective.value is not None + ): + return record, None + last_error = "evaluation did not produce a feasible objective" + except Exception as error: + last_error = str(error) or type(error).__name__ + if attempt == max_attempts: + break + return None, last_error + + async def _rescore_candidate( + self, + candidate: Candidate, + ) -> tuple[EvaluationRecord | None, str | None]: + assert self.selection.backend_id is not None + assert self.selection.evaluation_set is not None + assert self.selection.objective is not None + return await self._evaluate( + candidate=candidate, + backend_id=self.selection.backend_id, + evaluation_set=self.selection.evaluation_set, + objective=self.selection.objective, + parameters=self.selection.parameters, + limits=self.selection.limits, + max_attempts=self.selection.rescore_attempts, + ) + + def _strictly_beats( + self, + candidate: EvaluationRecord, + baseline: EvaluationRecord, + ) -> bool: + assert candidate.objective is not None and candidate.objective.value is not None + assert baseline.objective is not None and baseline.objective.value is not None + if not candidate.objective.feasible: + return False + if not baseline.objective.feasible: + return True + if self.selection.objective.direction == "maximize": + return candidate.objective.value > baseline.objective.value + return candidate.objective.value < baseline.objective.value + + def _beats_value(self, candidate: EvaluationRecord, baseline_value: float) -> bool: + assert candidate.objective is not None and candidate.objective.value is not None + if not candidate.objective.feasible: + return False + if self.selection.objective.direction == "maximize": + return candidate.objective.value > baseline_value + return candidate.objective.value < baseline_value + + def _pick_last(self) -> Candidate | None: + """Last-resort fallback: the most recently measured candidate.""" + records = list(self.engine.database.evaluations.values()) + if not records: + return None + latest = max(records, key=lambda record: record.created_at) + return latest.request.candidate + + async def _auto_best(self) -> Candidate | None: + rescored: list[EvaluationRecord] = [] + for candidate in self._shortlist(): + record, _ = await self._rescore_candidate(candidate) + if record is not None: + rescored.append(record) + if not rescored: + return None + assert self.selection.objective is not None + rescored.sort(key=lambda record: record.request.candidate.id) + rescored.sort( + key=lambda record: record.objective.value, + reverse=self.selection.objective.direction == "maximize", + ) + best = rescored[0] + + baseline_candidate = self.selection.baseline_candidate + if self.selection.baseline_floor and baseline_candidate is not None: + pinned = self.selection.baseline_selection_score + if pinned is not None: + if not self._beats_value(best, pinned): + return baseline_candidate + else: + baseline, _ = await self._rescore_candidate(baseline_candidate) + if baseline is None: + # Fail safe: can't verify the seed → don't ship it unverified. + raise NoCandidateError( + "baseline floor could not measure the seed (infrastructure)" + ) + if not self._strictly_beats(best, baseline): + return baseline_candidate + return best.request.candidate + + async def _select_candidate(self) -> Candidate: + # Chain: the agent's explicit submission wins; else auto_best over + # coverage-qualified evals; else the current/last candidate. Only when + # there is no candidate at all do we fail (and ship nothing). + submission = self._try_load_submission() + if submission is not None: + return submission + candidate = await self._auto_best() + if candidate is not None: + return candidate + last = self._pick_last() + if last is not None: + return last + raise NoCandidateError("no submitted, selectable, or prior candidate to ship") + + async def _score_target( + self, + candidate: Candidate, + target: VerificationTarget, + ) -> tuple[float, str | None, dict[str, float], str | None]: + record, error = await self._evaluate( + candidate=candidate, + backend_id=target.backend_id, + evaluation_set=target.evaluation_set, + objective=target.objective, + parameters=target.parameters, + limits=target.limits, + max_attempts=target.max_attempts, + ) + if record is None: + return target.failure_value, None, {}, error + assert record.objective is not None and record.objective.value is not None + reward = ( + target.reward_scale * float(record.objective.value) + target.reward_offset + ) + return reward, record.id, dict(record.report.metrics), None + + async def measure_baseline(self, *, replicates: int = 1) -> dict[str, JsonValue]: + """Admin-score the fixed seed to produce pinnable baseline numbers. + + Runs `replicates` trusted evaluations of the baseline on the selection + partition and each target and returns per-key mean/stddev, so a stable + value can be pinned (baseline_selection_score / target baseline_reward) + instead of re-scoring the seed every run.""" + if replicates < 1: + raise ValueError("replicates must be >= 1") + baseline = self.selection.baseline_candidate + if baseline is None: + raise NoCandidateError("no baseline candidate to score") + + def _aggregate(values: list[float | None]) -> dict[str, JsonValue]: + clean = [value for value in values if value is not None] + return { + "values": values, + "n": len(clean), + "mean": statistics.fmean(clean) if clean else None, + "stddev": statistics.pstdev(clean) if len(clean) > 1 else 0.0, + } + + result: dict[str, JsonValue] = { + "candidate_version": baseline.version, + "replicates": replicates, + } + if ( + self.selection.backend_id is not None + and self.selection.evaluation_set is not None + and self.selection.objective is not None + ): + selection_values: list[float | None] = [] + for _ in range(replicates): + record, _ = await self._rescore_candidate(baseline) + selection_values.append( + record.objective.value + if record is not None and record.objective is not None + else None + ) + result["selection"] = _aggregate(selection_values) + targets: dict[str, JsonValue] = {} + for target in self.targets: + target_values: list[float | None] = [] + for _ in range(replicates): + reward, _, _, error = await self._score_target(baseline, target) + target_values.append(None if error is not None else reward) + targets[target.reward_key] = _aggregate(target_values) + result["targets"] = targets + return result + + async def _finalize(self) -> VerificationResult: + try: + candidate = await self._select_candidate() + except NoCandidateError as error: + return VerificationResult( + shipped=False, + rewards={ + target.reward_key: target.failure_value for target in self.targets + }, + errors={"selection": str(error)}, + ) + + rewards: dict[str, float] = {} + evaluation_ids: dict[str, str] = {} + reward_metrics: dict[str, dict[str, float]] = {} + errors: dict[str, str] = {} + for target in self.targets: + reward, evaluation_id, metrics, error = await self._score_target( + candidate, target + ) + rewards[target.reward_key] = reward + if evaluation_id is not None: + evaluation_ids[target.reward_key] = evaluation_id + if metrics: + reward_metrics[target.reward_key] = metrics + if error is not None: + errors[target.reward_key] = error + + baseline_rewards: dict[str, float] = {} + baseline_reward_metrics: dict[str, dict[str, float]] = {} + baseline = self.selection.baseline_candidate + if self.score_baseline and baseline is not None: + if baseline.version == candidate.version: + baseline_rewards = dict(rewards) + baseline_reward_metrics = dict(reward_metrics) + else: + for target in self.targets: + if target.baseline_reward is not None: + baseline_rewards[target.reward_key] = target.baseline_reward + continue + reward, _, metrics, error = await self._score_target( + baseline, target + ) + baseline_rewards[target.reward_key] = reward + if metrics: + baseline_reward_metrics[target.reward_key] = metrics + if error is not None: + errors[f"baseline:{target.reward_key}"] = error + + return VerificationResult( + candidate=candidate, + rewards=rewards, + evaluation_ids=evaluation_ids, + baseline_rewards=baseline_rewards, + reward_metrics=reward_metrics, + baseline_reward_metrics=baseline_reward_metrics, + errors=errors, + ) + + async def finalize(self) -> VerificationResult: + """Return the first durable finalization result on every invocation.""" + async with self._lock: + if self._result is not None: + return self._result + if self.result_path.exists(): + try: + self._result = VerificationResult.model_validate_json( + self.result_path.read_text(encoding="utf-8") + ) + except Exception as error: + raise ValueError( + f"invalid durable finalization result: {error}" + ) from error + return self._result + drained = await self.engine.quiesce_agent_evaluations( + timeout_seconds=self.evaluation_drain_timeout_seconds, + ) + if drained: + logger.info( + "Finalization drained %d accepted agent evaluation(s)", + drained, + ) + result = await self._finalize() + await asyncio.to_thread( + _atomic_write_json, + self.result_path, + result.model_dump(mode="json"), + ) + self._result = result + if self._on_finalized is not None: + # Best-effort session-end hook (e.g. finish the W&B run); must + # never affect the durable finalization result. + try: + self._on_finalized(result) + except Exception: + logger.warning("finalization hook failed", exc_info=True) + return result diff --git a/vero/src/vero/skills/evals/SKILL.md b/vero/src/vero/skills/evals/SKILL.md new file mode 100644 index 00000000..3381c081 --- /dev/null +++ b/vero/src/vero/skills/evals/SKILL.md @@ -0,0 +1,120 @@ +--- +name: evals +description: >- + Run evaluations and navigate their results with the `evals` CLI over the + read-only `.evals/` directory. Use whenever you need to score a candidate + program, compare two candidates case by case, inspect why a case failed, or + check what you are allowed to evaluate and how much budget remains. +--- + +# evals: run evaluations and navigate their results + +Your program is scored by a trusted evaluator you cannot see into. Everything +you *are* allowed to see lands in the read-only `.evals/` directory, and the +`evals` CLI is the structured way to work with it. `evals --help` lists every +subcommand. + +## The loop + +```bash +evals plan # backend+partition pairs, sizes, budget +evals run --backend B --evaluation-set S --partition P # BLOCKS, returns the result +evals list --sort score --desc # every past result, one row each +evals diff BASELINE_ID CANDIDATE_ID # which cases improved / regressed +evals cases ID --sort score # per-case scores for one result +evals trace ID CASE_ID # trace summary + artifact files for a case +evals trace ID CASE_ID --span 3 # one span of the trace, char-windowed +evals submit --version COMMIT # nominate your best candidate to ship +``` + +Evaluate the baseline first: a candidate only counts as an improvement against +a measured baseline on the same case selection. + +**Results are persisted, so printed output is disposable.** Before `evals run` +returns, the complete record — overall metrics, every per-case score, and the +trial artifacts — is written under `.evals/results/` and stays there for the rest +of the run. So piping through `head`/`tail` loses nothing recoverable: use +`evals list` to find any past evaluation and `evals show` / `evals cases` / +`evals diff` to read it back. Never re-run an evaluation just to recover a number +you truncated — it is on disk. + +## Blocking vs detached + +By default `evals run` **blocks** until scoring finishes and returns the result +— one call, nothing to track. Evaluations can take many minutes; that is +expected, so let it block. This is what you want almost always. + +Use `--detach` **only** to run several evaluations concurrently: it returns a +`job_id` immediately instead of blocking. Then `evals wait JOB_ID` blocks until +that job finishes and prints its result; or poll `evals status JOB_ID`, which +now also reports `elapsed_seconds` (and `requested_cases` for a subset) so you +can see it is progressing. To wait on two jobs, wait the first, then the second. + +Run every `evals` call in the **foreground**. If you are a headless single-shot +run, nothing can wake you: putting a long call in a background task, scheduling +a wake-up, or promising to "report back when it finishes" ends the run right +there. A call that blocks for half an hour is working correctly. + +## Run options worth knowing (`evals run --help` for all) + +- **`--start N --stop M`** or repeated **`--case-id ID`** — score a *subset*. + Iterate on a handful of cases (cheap, fast) before spending budget on the full + set. A range past the end of the partition is rejected, so read the `cases` + column of `evals plan` for the partition's size rather than guessing. +- **`--seed N`** — backend-dependent, and several backends reject it outright + (you get an `invalid evaluation request` naming the reason). Where it is + refused, replicate a noisy comparison by re-running the *identical selection* + instead — same `--start/--stop` or same `--case-id` set. +- **`--max-concurrency`**, **`--case-timeout`**, **`--timeout`** — override + parallelism and wall budgets for a run. Note the final held-out evaluation + always uses the configured per-case budget, so don't loosen `--case-timeout` + in search or your slow candidate will look fine, then get stopped on test. + +## The `.evals/` tree + +``` +.evals/ +├── README.md, manifest.json +├── plan.json # evaluations you may invoke: selection rules, disclosure, budget +├── results/ # past evaluation results -> evals list / show / cases / trace / diff +│ ├── index.json +│ └── /evaluation.json, cases//result.json, artifacts/ +├── tasks/ # exposed task resources -> evals tasks +└── candidates/ # prior program versions (Git refs for `git show` / `git diff`) +``` + +Everything is plain JSON and files — `evals` is navigation and aggregation; +read individual artifact or task files directly with your file tools once a +command hands you the path. + +## Disclosure + +Each result is projected to an authorized disclosure level before it reaches +you: `full` (per-case results, traces, artifacts), `aggregate` (overall metrics +only — `evals cases` will refuse), or `none` (bare acknowledgement). Budgets +are enforced by the evaluator; `evals plan` shows what remains. + +## Common mistakes + +- Comparing evaluations run on different case selections — deltas are then + case-sampling noise, not signal. Use `evals diff`, which matches by case id. +- Claiming an improvement before the baseline's own evaluation finished. +- Dumping whole traces into context. Go metadata-first: `evals trace` summary, + then `--span N`, then windowed chars; artifacts are files — `grep` them. +- Re-running an evaluation because you truncated its output. Every result is on + disk under `.evals/results/`; read it back with `evals show` / `evals cases`. +- Confusing ids: evaluation ids come from `evals list`; case ids from + `evals cases`; job ids only from `evals run --detach`. +- Mismatching `--backend` and `--partition`. Each partition is served by exactly + one backend, and asking a different one is refused as `evaluation denied` — a + denial is deliberately opaque, so it will not tell you that is the reason. Take + both from the same row of `evals plan`. +- Scores may be noisy: replicate an important comparison before acting on it, by + re-running the identical case selection. +- Backgrounding the wait. Putting `evals run`/`evals wait` in a background task + (or scheduling a wake-up) looks like waiting but ends a headless run: the + notification you are counting on never arrives. Block in the foreground. +- Finishing without `evals submit`: your best candidate must be nominated + deliberately, or a fallback (auto-best, then your last commit) ships instead. +- Optimizing accuracy while ignoring per-case latency: a slower candidate can be + stopped at the wall budget and score the failure value. diff --git a/vero/src/vero/staging.py b/vero/src/vero/staging.py new file mode 100644 index 00000000..1ac50500 --- /dev/null +++ b/vero/src/vero/staging.py @@ -0,0 +1,72 @@ +"""Managed exchange directory between the host and a sandbox.""" + +from __future__ import annotations + +from pathlib import Path, PurePosixPath +from typing import Self + +from vero.sandbox import Sandbox + + +class SandboxStagingArea: + """Temporary sandbox directory with explicit host transfer operations.""" + + def __init__(self, sandbox: Sandbox, *, prefix: str = "vero-") -> None: + self.sandbox = sandbox + self.prefix = prefix + self.root: str | None = None + self._temporary_directory = None + + async def __aenter__(self) -> Self: + self._temporary_directory = self.sandbox.temporary_directory(self.prefix) + self.root = await self._temporary_directory.__aenter__() + return self + + async def __aexit__(self, exc_type, exc, traceback) -> None: + assert self._temporary_directory is not None + await self._temporary_directory.__aexit__(exc_type, exc, traceback) + self.root = None + + @staticmethod + def _relative_path(relative_path: str) -> PurePosixPath: + value = PurePosixPath(relative_path) + if ( + not relative_path + or "\\" in relative_path + or value.is_absolute() + or any(part in {"", ".", ".."} for part in relative_path.split("/")) + ): + raise ValueError("staging path must be a safe relative POSIX path") + return value + + def path(self, relative_path: str) -> str: + if self.root is None: + raise RuntimeError("sandbox staging area is not active") + value = self._relative_path(relative_path) + return (PurePosixPath(self.root) / value).as_posix() + + async def mkdir(self, relative_path: str) -> str: + path = self.path(relative_path) + await self.sandbox.mkdir(path) + return path + + async def write_text(self, relative_path: str, value: str) -> str: + path = self.path(relative_path) + await self.sandbox.write_file(path, value) + return path + + async def read_text(self, relative_path: str) -> str: + return await self.sandbox.read_file(self.path(relative_path)) + + async def exists(self, relative_path: str) -> bool: + return await self.sandbox.exists(self.path(relative_path)) + + async def upload(self, local_path: Path | str, relative_path: str) -> str: + remote_path = self.path(relative_path) + await self.sandbox.upload(str(local_path), remote_path) + return remote_path + + async def download(self, relative_path: str, local_path: Path | str) -> Path: + destination = Path(local_path) + await self.sandbox.download(self.path(relative_path), str(destination)) + return destination diff --git a/vero/src/vero/templates/report.html b/vero/src/vero/templates/report.html new file mode 100644 index 00000000..1d845d28 --- /dev/null +++ b/vero/src/vero/templates/report.html @@ -0,0 +1,151 @@ + + + + + + + VeRO experiment report + + + +
VeRO experiment report

Optimization session

+
+
This portable report can contain source diffs, evaluation artifacts, prompts, and agent tool output. Treat it as sensitive experiment data.
+
+
+

Score trajectories by split

+

Candidate lineage

Click a node to inspect itbaselinebest
+
+

Candidates

+
+

Producer traces

+

Event timeline

+
+
+
+ + + + diff --git a/vero/src/vero/tools/__init__.py b/vero/src/vero/tools/__init__.py new file mode 100644 index 00000000..888735cc --- /dev/null +++ b/vero/src/vero/tools/__init__.py @@ -0,0 +1,35 @@ +from .base import ToolSet +from .evaluation import EvaluationTools +from .planning import TodoList, think +from .registry import ToolDefinition, ToolRegistry, ToolSetInstance +from .sub_agent import SubAgentTool + +# Register all tool classes +ToolRegistry.register(EvaluationTools) +ToolRegistry.register(SubAgentTool) +ToolRegistry.register(TodoList) +ToolRegistry.register_callable(think) + + +def __getattr__(name: str): + if name in {"WebFetch", "WebSearch"}: + from .web import WebFetch, WebSearch + + return {"WebFetch": WebFetch, "WebSearch": WebSearch}[name] + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + + +__all__ = [ + # Tool classes (these ARE the keys now) + "EvaluationTools", + "SubAgentTool", + "TodoList", + "think", + "WebFetch", + "WebSearch", + # Registry + "ToolRegistry", + "ToolSetInstance", + "ToolDefinition", + "ToolSet", +] diff --git a/vero/src/vero/tools/base.py b/vero/src/vero/tools/base.py new file mode 100644 index 00000000..14d517a1 --- /dev/null +++ b/vero/src/vero/tools/base.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol, runtime_checkable + +if TYPE_CHECKING: + from vero.agents.protocol import AgentContext + + +@runtime_checkable +class ToolSet(Protocol): + """Protocol for tool sets that can be bound to an agent context. + + ToolSets group related tool methods (decorated with @is_tool). + They are pre-created instances that self-wire to policy resources + via bind(). + """ + + exclude_tools: list[str] + + def bind(self, context: AgentContext) -> None: ... diff --git a/vero/src/vero/tools/evaluation.py b/vero/src/vero/tools/evaluation.py new file mode 100644 index 00000000..e6334374 --- /dev/null +++ b/vero/src/vero/tools/evaluation.py @@ -0,0 +1,95 @@ +"""Agent-facing tools for the scoped evaluation capability.""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from typing import TYPE_CHECKING + +from vero.evaluation import CaseSelection, EvaluationBudgetExceeded +from vero.optimization import CandidateEvaluationGateway +from vero.tools.utils import is_tool + +if TYPE_CHECKING: + from vero.agents.protocol import AgentContext + + +@dataclass +class EvaluationTools: + """Evaluate current or prior programs through the session's named evaluations.""" + + exclude_tools: list[str] = field(default_factory=list) + evaluation: CandidateEvaluationGateway | None = field(default=None, repr=False) + + def bind(self, context: AgentContext) -> None: + self.evaluation = context.evaluation + + def _gateway(self) -> CandidateEvaluationGateway: + if self.evaluation is None: + raise RuntimeError("evaluation tools are not bound to an agent context") + return self.evaluation + + @is_tool + async def evaluate( + self, + evaluation: str, + selection: CaseSelection | None = None, + candidate_id: str | None = None, + description: str = "Evaluate agent checkpoint", + ) -> str: + """Evaluate a program, returning authorized feedback and a filesystem path. + + Args: + evaluation: Name from ``.evals/plan.json``. + selection: Optional case IDs or range. Omit to use the base selection. + candidate_id: Existing candidate to re-evaluate. Omit to save and + evaluate the current workspace. + description: A short description of the program changes being evaluated. + + Returns: + A bounded JSON receipt with an authorized summary and the path to the + complete permitted feedback under ``.evals/results``. + """ + + try: + result = await self._gateway().evaluate( + evaluation=evaluation, + selection=selection, + candidate_id=candidate_id, + description=description, + ) + except EvaluationBudgetExceeded as error: + # Running out of evaluation budget is expected and benign: hand the + # agent a clean result rather than raising, so it stops evaluating + # and proceeds with what it already knows. The final held-out + # evaluation runs on a separate trusted path and is unaffected. + return json.dumps( + { + "status": "evaluation_budget_exhausted", + "message": str(error), + "detail": ( + "The evaluation budget for this evaluation is spent. " + "Further evaluations of this kind are unavailable; " + "proceed with the feedback you already have." + ), + }, + indent=2, + ) + return result.model_dump_json(indent=2) + + @is_tool + def get_evaluation_budgets(self) -> str: + """Return remaining agent budgets for every available evaluation.""" + + budgets = self._gateway().budgets() + return json.dumps( + { + name: ( + budget.model_dump(mode="json") + if budget is not None + else None + ) + for name, budget in sorted(budgets.items()) + }, + indent=2, + ) diff --git a/vero/src/vero/tools/planning.py b/vero/src/vero/tools/planning.py new file mode 100644 index 00000000..4c31746b --- /dev/null +++ b/vero/src/vero/tools/planning.py @@ -0,0 +1,96 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from enum import StrEnum + +from vero.tools.utils import is_tool + + +def think(thought: str) -> str: + """Think and reason about a thought.""" + return "" + + +class TodoStatus(StrEnum): + """The status of a todo item.""" + + NOT_STARTED = "NOT_STARTED" + IN_PROGRESS = "IN_PROGRESS" + COMPLETED = "COMPLETED" + SKIPPED = "SKIPPED" + + +@dataclass +class TodoList: + """Maintain and keep track of a list of todos.""" + + exclude_tools: list[str] = field(default_factory=list) + todos: list[str] = field(default_factory=list) + todo_statuses: dict[str, TodoStatus] = field(default_factory=dict) + + @is_tool + def add_todos(self, tasks: str | list[str]) -> str: + """Add a todo item or severalitems to the list of todos. + + Args: + tasks: The task or tasks to add to the list. + Returns: + The id of the todo item or items. + """ + if isinstance(tasks, str): + tasks = [tasks] + + task_ids = [] + for task in tasks: + task_ids.append(len(self.todos)) + self.todos.append(task) + self.todo_statuses[task] = TodoStatus.NOT_STARTED + return f"{len(tasks)} items have been added to the todo list with ids {', '.join([str(task_id) for task_id in task_ids])}." + + @is_tool + def update_todo_status( + self, status: TodoStatus, task: str | None = None, task_id: int | None = None + ) -> str: + """Update the status of a todo item. Either provide the task or the task_id. + + Args: + status: The status to update the todo item to. + task: The task to update the status of. + task_id: The id of the todo item to update the status of. + Returns: + A message indicating the status of the todo item has been updated. + """ + + if task is None and task_id is None: + raise ValueError("Either task or task_id must be provided") + + if task_id is not None: + task = self.todos[task_id] + + self.todo_statuses[task] = status + return f"Todo item has been updated to status {status}" + + @is_tool + def list_todos(self, status: list[TodoStatus] | None = None) -> str: + """List the todos with the given status. Defaults to not started and in progress todos. + + Args: + status: The statuses to list the todos for. + + Returns: + A JSON string mapping task ids to task details and their statuses. + """ + + if status is None: + status = [TodoStatus.NOT_STARTED, TodoStatus.IN_PROGRESS] + + if not isinstance(status, list): + status = [status] + + todos = {} + for task_id, task in enumerate(self.todos): + if self.todo_statuses[task] in status: + todos[task_id] = {"task": task, "status": self.todo_statuses[task]} + + return f"""Here are the todos with statuses: {status}\n```json{json.dumps(todos, indent=2)}```""" diff --git a/vero/src/vero/tools/registry.py b/vero/src/vero/tools/registry.py new file mode 100644 index 00000000..aac8645e --- /dev/null +++ b/vero/src/vero/tools/registry.py @@ -0,0 +1,89 @@ +"""Tool Registry for managing available tools.""" + +from dataclasses import dataclass, field +from typing import Callable, Generic, TypeVar + +ToolT = TypeVar("ToolT") + + +@dataclass(frozen=True, slots=True) +class ToolSetInstance(Generic[ToolT]): + """An instance of a tool set with its function tools.""" + + instance: Callable | object + tools: list[ToolT] = field(repr=False) + + +@dataclass +class ToolDefinition: + """Definition of a tool in the registry.""" + + tool_class: type | Callable + description: str + is_callable: bool = False + + @property + def name(self) -> str: + """The original class/function name.""" + return self.tool_class.__name__ + + +class ToolRegistry: + """Registry for tools available to agents. + + Uses the class/callable itself as the key. + + Example: + >>> ToolRegistry.register(BashTool) + >>> ToolRegistry.get(BashTool) + ToolDefinition(tool_class=BashTool, ...) + """ + + _tools: dict[type | Callable, ToolDefinition] = {} + + @classmethod + def register(cls, tool_class: type, description: str | None = None) -> type: + """Register a tool class. Returns the class itself as the key.""" + desc = description or (tool_class.__doc__ or "").strip() + cls._tools[tool_class] = ToolDefinition( + tool_class=tool_class, description=desc, is_callable=False + ) + return tool_class + + @classmethod + def register_callable(cls, func: Callable, description: str | None = None) -> Callable: + """Register a callable (function) as a tool. Returns the callable itself.""" + desc = description or (func.__doc__ or "").split("\n")[0].strip() + cls._tools[func] = ToolDefinition( + tool_class=func, description=desc, is_callable=True + ) + return func + + @classmethod + def get(cls, key: type | Callable) -> ToolDefinition: + """Get a tool definition by class/callable.""" + if key not in cls._tools: + raise KeyError( + f"Tool '{key}' not found in registry. Available: {[t.__name__ for t in cls._tools.keys()]}" + ) + return cls._tools[key] + + @classmethod + def describe(cls, key: type | Callable) -> str: + """Describe a tool by class/callable.""" + return cls.get(key).description + + @classmethod + def exists(cls, key: type | Callable) -> bool: + """Check if a tool exists in the registry.""" + return key in cls._tools + + @classmethod + def all_keys(cls) -> set[type | Callable]: + """Get all registered tool keys.""" + return set(cls._tools.keys()) + + @classmethod + def items(cls) -> list[tuple[type | Callable, ToolDefinition]]: + """Get all registered tools as (key, definition) pairs.""" + return list(cls._tools.items()) diff --git a/vero/src/vero/tools/sandbox_tools.py b/vero/src/vero/tools/sandbox_tools.py new file mode 100644 index 00000000..278ec2cc --- /dev/null +++ b/vero/src/vero/tools/sandbox_tools.py @@ -0,0 +1,79 @@ +"""Function tools that drive an SDK ``SandboxSession``. + +These give the coding agent a real shell plus file read/write that execute +INSIDE the sandbox — so they inherit the sandbox's isolation, workspace binding +(``Manifest(root=...)``), and containment (the sandbox client is the seam: +unix-local / docker / modal / e2b). Crucially they are *plain function tools*, +not the SDK's hosted ``Shell``/``Filesystem`` capabilities, so they work with +ANY model/provider over ChatCompletions *or* Responses. The hosted capabilities +would restrict the native path to the OpenAI Responses API. + +All operations go through ``session.exec`` (a real shell), which keeps the tool +surface tiny and provider-neutral. +""" + +from __future__ import annotations + +import base64 +import shlex +from typing import TYPE_CHECKING + +from agents import function_tool + +if TYPE_CHECKING: + from agents.sandbox.session.sandbox_session import SandboxSession + + +def _format(result: object) -> str: + """Render an ExecResult as text (stdout, optional stderr, non-zero exit).""" + out = getattr(result, "stdout", None) + if out is None: + out = getattr(result, "output", "") + if isinstance(out, (bytes, bytearray)): + out = bytes(out).decode(errors="replace") + err = getattr(result, "stderr", "") or "" + if isinstance(err, (bytes, bytearray)): + err = bytes(err).decode(errors="replace") + code = getattr(result, "exit_code", getattr(result, "returncode", 0)) + text = str(out) + if str(err).strip(): + text += f"\n[stderr]\n{err}" + if code not in (0, None): + text = f"[exit={code}]\n{text}" + return text or f"[exit={code}]" + + +def build_sandbox_tools(session: "SandboxSession") -> list: + """Build shell / read_file / write_file function tools bound to ``session``.""" + + @function_tool + async def shell(command: str) -> str: + """Run a shell command in the program workspace and return its output. + + The working directory is the program's workspace root. Use this to run + the program, tests, git, package managers, and any Unix tooling. + """ + return _format(await session.exec(command)) + + @function_tool + async def read_file(path: str) -> str: + """Read a UTF-8 text file. ``path`` is relative to the workspace root.""" + return _format(await session.exec(f"cat -- {shlex.quote(path)}")) + + @function_tool + async def write_file(path: str, content: str) -> str: + """Create or overwrite a file with ``content``. + + ``path`` is relative to the workspace root; parent directories are + created as needed. + """ + encoded = base64.b64encode(content.encode()).decode() + quoted = shlex.quote(path) + command = ( + f'mkdir -p "$(dirname -- {quoted})" && ' + f"printf %s {shlex.quote(encoded)} | base64 -d > {quoted} && " + f"printf 'wrote %s bytes to %s\\n' {len(content)} {quoted}" + ) + return _format(await session.exec(command)) + + return [shell, read_file, write_file] diff --git a/vero/src/vero/tools/sub_agent.py b/vero/src/vero/tools/sub_agent.py new file mode 100644 index 00000000..3791f1c5 --- /dev/null +++ b/vero/src/vero/tools/sub_agent.py @@ -0,0 +1,182 @@ +"""Sub-agent tool for spawning child agents with access to vero tools.""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Callable, NamedTuple + +from vero.exceptions import StreamEventTimeout +from vero.tools.utils import is_tool + +if TYPE_CHECKING: + from agents import Agent, RunResultStreaming, Tool + +logger = logging.getLogger(__name__) + +MAX_SANITIZATION_RETRIES = 3 + + +def default_sub_agent_tools() -> set[type | Callable]: + """Default tools available to sub-agents.""" + from vero.tools.evaluation import EvaluationTools + from vero.tools.planning import TodoList, think + from vero.tools.web import WebFetch + + return { + EvaluationTools, + TodoList, + WebFetch, + think, + } + + +class SubAgentSession(NamedTuple): + """A sub-agent with its agent and session.""" + + agent: Agent + session: RunResultStreaming | None = None + + def to_dict(self) -> dict: + from agents import RunResultStreaming + + def serialize_agent(agent: Agent) -> dict: + agent_dict: dict[str, Any] = {"name": agent.name} + if agent.instructions is not None and isinstance(agent.instructions, str): + agent_dict["instructions"] = agent.instructions + agent_dict["tools"] = [tool.name for tool in agent.tools] + return agent_dict + + session_json = None + if isinstance(self.session, RunResultStreaming): + session_json = self.session.to_input_list() + + return {"agent": serialize_agent(self.agent), "session": session_json} + + +@dataclass +class SubAgentTool: + """Spawn child agents with access to a subset of vero tools. + + The orchestrator invokes call_sub_agent to delegate tasks. + Sub-agents can use tools from `allowed_tools` (filtered from `tool_instances`). + """ + + exclude_tools: list[str] = field(default_factory=list) + allowed_tools: set[type | Callable] = field(default_factory=default_sub_agent_tools) + sessions: dict[str, SubAgentSession] = field(default_factory=dict) + + # Runtime fields — set by the agent after binding other ToolSets + tool_instances: dict[type | Callable, list[Tool]] = field(default_factory=dict) + models: dict[str, Any] = field(default_factory=dict) + + def bind(self, session) -> None: + pass # tool_instances and models are set by the agent after binding other ToolSets + + @property + def available_tool_sets(self) -> dict[type | Callable, list[Tool]]: + return {k: v for k, v in self.tool_instances.items() if k in self.allowed_tools} + + @is_tool + async def call_sub_agent( + self, + prompt: str, + instructions: str | None = None, + max_turns: int = 50, + agent_name: str | None = None, + tool_sets: list[str] | None = None, + model_alias: str | None = None, + ) -> str: + """Invoke a sub-agent to perform a task with the given tools. + + Use sub-agents to delegate tasks like summarizing code, analyzing experiments, + searching the web, or getting feedback on solutions. + + Args: + prompt: The prompt for the sub-agent. + instructions: Sub-agent system instructions. + max_turns: Maximum turns for the sub-agent. + agent_name: Name of the sub-agent (reuse to continue a conversation). + tool_sets: Tool set names to provide (e.g. ["FileRead", "Grep"]). Defaults to all available. + model_alias: Model alias to use. Defaults to first available model. + """ + import asyncio + + import agents as agents_lib + from agents import Agent, Runner + + from vero.utils.openai_agents import stream_events + + if model_alias is None: + model_alias = next(iter(self.models.keys())) + + available = self.available_tool_sets + + # Resolve tool_sets by name + tools: list[Tool] = [] + if tool_sets: + name_to_key = { + k.__name__ if hasattr(k, "__name__") else str(k): k for k in available + } + for name in tool_sets: + if name not in name_to_key: + raise ValueError( + f"Unknown tool set: {name}. Available: {list(name_to_key.keys())}" + ) + tools.extend(available[name_to_key[name]]) + else: + for tool_list in available.values(): + tools.extend(tool_list) + + try: + model = self.models[model_alias] + except KeyError: + raise KeyError( + f"Invalid model alias: {model_alias}. Available: {list(self.models.keys())}" + ) + + if ( + agent_name is not None + and agent_name in self.sessions + and self.sessions[agent_name].session is not None + ): + previous = self.sessions[agent_name].session + if previous: + input = previous.to_input_list() + [{"content": prompt, "role": "user"}] + else: + input = [{"content": prompt, "role": "user"}] + + elif agent_name is not None: + input = prompt + else: + agent_name = f"Sub-Agent {len(self.sessions) + 1}" + input = prompt + + if agent_name in self.sessions: + subagent = self.sessions[agent_name].agent + subagent.instructions = instructions or subagent.instructions + else: + subagent = Agent( + name=agent_name, + model=model, + tools=tools, + instructions=instructions, + ) + + result = Runner.run_streamed(subagent, input=input, max_turns=max_turns) + error: Exception | None = None + try: + async for _event in stream_events(result): + await asyncio.sleep(0) + except Exception as exc: # surfaced via the isinstance checks below + error = exc + self.sessions[agent_name] = SubAgentSession(agent=subagent, session=result) + + if isinstance(error, StreamEventTimeout): + return f"Sub-agent '{agent_name}' timed out. Use the same agent name to continue." + elif isinstance(error, agents_lib.MaxTurnsExceeded): + return f"Sub-agent '{agent_name}' reached max turns ({max_turns}). Use the same agent name to continue." + elif error is not None: + raise error + + return f"Sub-agent '{agent_name}' response:\n\n{result.final_output}" diff --git a/vero/src/vero/tools/utils/__init__.py b/vero/src/vero/tools/utils/__init__.py new file mode 100644 index 00000000..4f783b4f --- /dev/null +++ b/vero/src/vero/tools/utils/__init__.py @@ -0,0 +1,32 @@ +from typing import Callable + +_IS_TOOL = "_is_tool" + + +def is_tool(method: Callable) -> Callable: + """A decorator to mark a method of a class as a tool.""" + setattr(method, _IS_TOOL, True) + return method + + +def get_tools_from_class( + cls_or_instance: type | object, +) -> list[Callable]: + """Get all methods marked with @is_tool from a class or instance. + + If the instance has an `exclude_tools` attribute (from the ToolSet protocol), + those method names are excluded from the result. + + Args: + cls_or_instance: A class type or an instance to get tools from. + + Returns: + List of methods marked with @is_tool decorator. + """ + exclude = getattr(cls_or_instance, "exclude_tools", []) or [] + return [ + getattr(cls_or_instance, attr) + for attr in dir(cls_or_instance) + if getattr(getattr(cls_or_instance, attr), _IS_TOOL, False) + and attr not in exclude + ] diff --git a/vero/src/vero/tools/utils/openai_agents.py b/vero/src/vero/tools/utils/openai_agents.py new file mode 100644 index 00000000..ee63f539 --- /dev/null +++ b/vero/src/vero/tools/utils/openai_agents.py @@ -0,0 +1,80 @@ +from inspect import isclass, isfunction, ismethod +from typing import Any, Callable + +from agents import FunctionTool, function_tool + +from vero.tools.utils import get_tools_from_class +from vero.utils.general import camel_to_snake + + +def callable_to_oai_tool( + tool: Callable | object, method_name: str | None = None, strict_mode: bool = True, **kwargs: Any +) -> FunctionTool: + """Converts a function or class method to an OpenAI Agents FunctionTool. + + Args: + tool: The function or callable object to convert. + method_name: If the tool is a class, the name of the method to convert. If None, we assume the tool is a callable object and use __call__. + strict_mode: Strict mode toggle for function_tool adapter. + **kwargs: Additional keyword arguments to the OpenAI function_tool adapter. + + Returns: + An OpenAI Agents FunctionTool. + """ + + if isfunction(tool) or ismethod(tool): + return function_tool(tool, strict_mode=strict_mode, **kwargs) + + if isclass(tool): + raise ValueError( + "Cannot convert a class to an OpenAI Agents FunctionTool. Pass an instance instead." + ) + + tool_cls = tool.__class__ + + if method_name is None: + method_name = "__call__" + + if "name_override" not in kwargs: + kwargs["name_override"] = camel_to_snake(tool_cls.__name__) + else: + if "name_override" not in kwargs: + kwargs["name_override"] = camel_to_snake(tool_cls.__name__) + "_" + method_name + + # use the method of the instance not the class + method = getattr(tool, method_name) + return function_tool(method, strict_mode=strict_mode, **kwargs) + + +def tool_set_instance_to_oai_tools( + instance: object, + strict_mode: bool = True, + prefix_class_name: bool = True, + exclude_methods: list[str] | None = None, +) -> list[FunctionTool]: + """Creates a list of OpenAI Agents FunctionTools from an instance.""" + + exclude_methods = exclude_methods or [] + + if exclude_methods: + tools = [ + getattr(instance, tool.__name__) + for tool in get_tools_from_class(instance.__class__) + if tool.__name__ not in exclude_methods + ] + else: + tools = [ + getattr(instance, tool.__name__) for tool in get_tools_from_class(instance.__class__) + ] + + # Defaault to using the __call__ method of the instance if no tools are found. + if not tools: + return [callable_to_oai_tool(instance, strict_mode=strict_mode)] + + oai_tools = [] + for tool in tools: + kwargs = {"strict_mode": strict_mode} + if prefix_class_name: + kwargs["name_override"] = f"{instance.__class__.__name__}_{tool.__name__}" + oai_tools.append(callable_to_oai_tool(tool, **kwargs)) + return oai_tools diff --git a/vero/src/vero/tools/web.py b/vero/src/vero/tools/web.py new file mode 100644 index 00000000..bb858096 --- /dev/null +++ b/vero/src/vero/tools/web.py @@ -0,0 +1,304 @@ +from __future__ import annotations + +import io +import json +import os +from dataclasses import dataclass, field + +from async_lru import alru_cache + +from vero.tools.registry import ToolRegistry + +SCRAPING_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" + + +def _fetch_and_parse_webpage_trafilatura(url: str) -> str | None: + """Fetch and parse a webpage using trafilatura.""" + import trafilatura + from trafilatura.settings import use_config + + trafilatura_config = use_config() + trafilatura_config["DEFAULT"]["USER_AGENT"] = SCRAPING_AGENT + + downloaded = trafilatura.fetch_url(url, config=trafilatura_config, no_ssl=True) + if downloaded is None: + return None + + return trafilatura.extract(downloaded) + + +async def _fetch_and_parse_webpage_beautifulsoup(url: str) -> str | None: + import httpx + from bs4 import BeautifulSoup + + try: + async with httpx.AsyncClient(verify=False, follow_redirects=True, timeout=30.0) as client: + response = await client.get( + url, + headers={ + "User-Agent": SCRAPING_AGENT, + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", + }, + ) + response.raise_for_status() + downloaded = response.text + soup = BeautifulSoup(downloaded, "lxml") + for script in soup(["script", "style"]): + script.extract() + content = soup.get_text(separator="\n", strip=True) + return content + except (httpx.HTTPError, Exception): + return None + + +async def _fetch_and_parse_pdf(url: str) -> str | None: + """Fetch and parse a PDF document.""" + import httpx + from pypdf import PdfReader + + try: + async with httpx.AsyncClient(verify=False, follow_redirects=True, timeout=30.0) as client: + response = await client.get( + url, + headers={ + "User-Agent": SCRAPING_AGENT, + "Accept": "application/pdf", + }, + ) + response.raise_for_status() + + # Check if content is actually a PDF + content_type = response.headers.get("content-type", "").lower() + if "pdf" not in content_type and not url.lower().endswith(".pdf"): + return None + + # Parse PDF + pdf_file = io.BytesIO(response.content) + reader = PdfReader(pdf_file) + + # Extract text from all pages + text_content = [] + for page in reader.pages: + text = page.extract_text() + if text: + text_content.append(text) + + return "\n\n".join(text_content) if text_content else None + + except Exception: + return None + + +@alru_cache(maxsize=128) +async def _fetch_and_parse_webpage(url: str) -> str | None: + """Fetch and parse a webpage or PDF using multiple methods with fallbacks.""" + # Check if URL likely points to a PDF + if url.lower().endswith(".pdf"): + content = await _fetch_and_parse_pdf(url) + if content is not None: + return content + + # Try HTML parsing first + content = _fetch_and_parse_webpage_trafilatura(url) + if content is not None: + return content + + content = await _fetch_and_parse_webpage_beautifulsoup(url) + if content is not None: + return content + + # If HTML parsing failed, try PDF as fallback (might be PDF with non-.pdf extension) + content = await _fetch_and_parse_pdf(url) + if content is not None: + return content + + return None + + +@dataclass +class WebFetch: + """Fetches and extracts text content from a webpage given a URL.""" + + exclude_tools: list[str] = field(default_factory=list) + max_chars: int = 50_000 + + async def __call__(self, url: str, offset: int = 0, max_chars: int = 50_000) -> str: + """ + Fetches the content of a webpage and returns the extracted text. + + Args: + url: The URL of the webpage to fetch + + Returns: + Extracted text content from the webpage + """ + + if offset < 0: + raise ValueError("offset must be greater than or equal to 0.") + + if max_chars > self.max_chars: + raise ValueError( + f"max_chars must be less than or equal to {self.max_chars}. Use pagination to fetch more content." + ) + + content = await _fetch_and_parse_webpage(url) + + if content is None: + raise RuntimeError(f"Failed to fetch content from {url}") + + if offset > len(content): + raise ValueError(f"Offset was greater than the length of the content {len(content)}") + + # Apply offset and max_chars + content = content[offset : offset + max_chars] + + return content + + +def _contains_chinese(text: str) -> bool: + """Check if text contains Chinese characters.""" + return any("\u4E00" <= char <= "\u9FFF" for char in text) + + +@dataclass +class WebSearch: + """Searches the web for information using the Serper API (google.serper.dev).""" + + exclude_tools: list[str] = field(default_factory=list) + max_chars_per_result: int = 10_000 + max_results: int = 10 + max_retries: int = 3 + fetch_full_content: bool = True + base_url: str = "https://google.serper.dev/search" + + def __post_init__(self): + api_key = os.getenv("SERPER_KEY_ID") or os.getenv("SERPER_API_KEY") + if not api_key: + raise ValueError( + "`SERPER_KEY_ID` or `SERPER_API_KEY` must be set as an environment variable to use `WebSearch` tool." + ) + + async def __call__(self, query: str) -> str: # noqa: C901 + """ + Searches the web for information and returns a list of results with URL, title, and content. + + If fetch_full_content is True (default), fetches and parses full webpage content. + Otherwise, returns only snippets from the search results. + + Args: + query: The query string to search for + Returns: + JSON string containing search results (URL, Title, Content/Snippet) + """ + api_key = os.getenv("SERPER_KEY_ID") or os.getenv("SERPER_API_KEY") + if not api_key: + raise RuntimeError( + "`SERPER_KEY_ID` or `SERPER_API_KEY` must be set as an environment variable to use `WebSearch` tool." + ) + + # Configure payload based on language + if _contains_chinese(query): + payload = { + "q": query, + "location": "China", + "gl": "cn", + "hl": "zh-cn", + "num": min(self.max_results, 10), + } + else: + payload = { + "q": query, + "location": "United States", + "gl": "us", + "hl": "en", + "num": min(self.max_results, 10), + } + + headers = { + "X-API-KEY": api_key, + "Content-Type": "application/json", + } + + import httpx + + # Retry logic + last_error = None + for attempt in range(self.max_retries): + try: + async with httpx.AsyncClient(timeout=30.0) as client: + response = await client.post( + self.base_url, + json=payload, + headers=headers, + ) + response.raise_for_status() + + results = response.json() + + if "organic" not in results: + return json.dumps( + { + "error": f"No results found for query: '{query}'", + "suggestion": "Try a less specific query", + "results": [], + } + ) + + # Parse and optionally fetch full content + parsed_results = [] + for item in results["organic"][: self.max_results]: + result_item = { + "title": item.get("title", ""), + "url": item.get("link", ""), + } + + if self.fetch_full_content: + # Fetch full page content + content = await _fetch_and_parse_webpage(result_item["url"]) + if content is not None: + if len(content) > self.max_chars_per_result: + content = content[: self.max_chars_per_result] + "... (truncated)" + result_item["content"] = content + else: + # Fall back to snippet if full content fetch fails + result_item["content"] = item.get("snippet", "") + else: + # Just use the snippet + result_item["content"] = item.get("snippet", "") + + # Add optional metadata + if "date" in item: + result_item["date"] = item["date"] + if "source" in item: + result_item["source"] = item["source"] + + parsed_results.append(result_item) + + return json.dumps(parsed_results, indent=4) + + except httpx.TimeoutException as e: + last_error = e + if attempt < self.max_retries - 1: + continue + except httpx.HTTPStatusError as e: + last_error = e + if attempt < self.max_retries - 1: + continue + except Exception as e: + last_error = e + if attempt < self.max_retries - 1: + continue + + # All retries failed + return json.dumps( + { + "error": f"Search failed after {self.max_retries} attempts: {str(last_error)}", + "results": [], + } + ) + + +# These tools have optional scraping dependencies, so register them only when +# the web module is imported. +ToolRegistry.register(WebFetch) +ToolRegistry.register(WebSearch) diff --git a/vero/src/vero/utils/__init__.py b/vero/src/vero/utils/__init__.py new file mode 100644 index 00000000..fcebecf0 --- /dev/null +++ b/vero/src/vero/utils/__init__.py @@ -0,0 +1,27 @@ +from .asyncio import ( + SubprocessCancelledError, + SubprocessResult, + SubprocessTimeoutError, + anext_with_timeout, + run_bash_command, + run_subprocess_with_tee, +) +from .general import ( + camel_to_snake, + paginate, + recursively_serialize, + strip_ansi, +) + +__all__ = [ + "anext_with_timeout", + "camel_to_snake", + "paginate", + "recursively_serialize", + "run_bash_command", + "run_subprocess_with_tee", + "strip_ansi", + "SubprocessCancelledError", + "SubprocessResult", + "SubprocessTimeoutError", +] diff --git a/vero/src/vero/utils/asyncio.py b/vero/src/vero/utils/asyncio.py new file mode 100644 index 00000000..5991faf2 --- /dev/null +++ b/vero/src/vero/utils/asyncio.py @@ -0,0 +1,254 @@ +import asyncio +import sys +from dataclasses import dataclass +from subprocess import CalledProcessError +from typing import Any, AsyncIterator, TypeVar + +T = TypeVar("T") + + +@dataclass +class SubprocessResult: + """Result of a subprocess execution, always includes captured output.""" + + args: list[str] + returncode: int | None # None if process didn't complete + stdout: str + stderr: str + pid: int | None = None # Process ID, useful for debugging orphaned processes + timed_out: bool = False + cancelled: bool = False + + +class SubprocessTimeoutError(Exception): + """Timeout with captured output preserved.""" + + def __init__(self, result: SubprocessResult): + self.result = result + super().__init__(f"Subprocess timed out: {result.args}") + + +class SubprocessCancelledError(Exception): + """Cancellation with captured output preserved.""" + + def __init__(self, result: SubprocessResult): + self.result = result + super().__init__(f"Subprocess cancelled: {result.args}") + + +async def anext_with_timeout(it: AsyncIterator[T], timeout: int = 5) -> T: + """Get the next item from an async iterator with a timeout.""" + async with asyncio.timeout(timeout): + return await anext(it) + + +async def run_bash_command(cmd: list[str], timeout: int | None = None) -> str: + """Run a Bash command and return its output. + + Deprecated: tools should use sandbox.run() instead. + """ + proc = await asyncio.create_subprocess_exec( + *cmd, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + + async def terminate_process(): + """Terminate process, shielded from cancellation.""" + proc.terminate() + try: + await asyncio.wait_for(proc.wait(), timeout=5) + except asyncio.TimeoutError: + proc.kill() + await proc.wait() + + try: + stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout) + stdout_str = stdout.decode().strip() + stderr_str = stderr.decode().strip() + if proc.returncode == 0: + return stdout_str + else: + raise CalledProcessError( + returncode=proc.returncode, + cmd=cmd, + output=stdout_str, + stderr=stderr_str, + ) + except asyncio.TimeoutError: + await asyncio.shield(terminate_process()) + cmd_str = " ".join(cmd) + raise TimeoutError(f"Command '{cmd_str}' timed out after {timeout} seconds") + except (KeyboardInterrupt, asyncio.CancelledError): + await asyncio.shield(terminate_process()) + raise + + +async def run_subprocess_with_tee( + cmd: list[str], + timeout: float | None = None, + cwd: str | None = None, + check: bool = False, + flush: bool = False, + chunk_size: int = 8192, + tee_stdout: bool = True, + tee_stderr: bool = True, + **kwargs: Any, +) -> SubprocessResult: + """Run a subprocess, streaming output to console while capturing it. + + Args: + cmd: The command to run. + timeout: Timeout in seconds (None = no timeout). + cwd: Working directory for the subprocess. + check: Raise CalledProcessError on non-zero exit. + flush: Flush console output after each chunk. + chunk_size: Bytes to read per iteration (default 8192). + tee_stdout: Whether to print stdout to console. + tee_stderr: Whether to print stderr to console. + **kwargs: Additional arguments to asyncio.create_subprocess_exec. + + Returns: + SubprocessResult with captured stdout/stderr and status. + + Raises: + SubprocessTimeoutError: If timeout exceeded (includes partial output). + SubprocessCancelledError: If cancelled (includes partial output). + CalledProcessError: If check=True and non-zero exit code. + """ + process: asyncio.subprocess.Process | None = None + stdout_chunks: list[str] = [] + stderr_chunks: list[str] = [] + + def build_result( + returncode: int | None = None, + timed_out: bool = False, + cancelled: bool = False, + ) -> SubprocessResult: + return SubprocessResult( + args=cmd, + returncode=returncode, + stdout="".join(stdout_chunks), + stderr="".join(stderr_chunks), + pid=process.pid if process else None, + timed_out=timed_out, + cancelled=cancelled, + ) + + async def read_stream( + stream: asyncio.StreamReader | None, + chunks: list[str], + output_stream, + tee: bool, + ): + if not stream: + return + + while True: + chunk = await stream.read(chunk_size) + if not chunk: + break + decoded = chunk.decode(errors="replace") + chunks.append(decoded) + if tee: + output_stream.write(decoded) + if flush: + output_stream.flush() + + async def terminate_process_safe(): + """Terminate process, handling edge cases.""" + if process is None or process.returncode is not None: + return + try: + process.terminate() + try: + await asyncio.wait_for(process.wait(), timeout=5) + except asyncio.TimeoutError: + process.kill() + await process.wait() + except ProcessLookupError: + pass # Already dead + + try: + process = await asyncio.create_subprocess_exec( + *cmd, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + cwd=cwd, + **kwargs, + ) + + async def run_and_wait(): + # Run stream readers as tasks so we can cancel them + stdout_task = asyncio.create_task( + read_stream(process.stdout, stdout_chunks, sys.stdout, tee_stdout) + ) + stderr_task = asyncio.create_task( + read_stream(process.stderr, stderr_chunks, sys.stderr, tee_stderr) + ) + + try: + # Wait for process to exit (don't wait for streams - child processes may hold them open) + await process.wait() + finally: + # Always cleanup stream tasks, even on cancellation/timeout + _, pending = await asyncio.wait( + [stdout_task, stderr_task], + timeout=1.0, + ) + for task in pending: + task.cancel() + try: + await asyncio.wait_for(task, timeout=0.5) + except (asyncio.CancelledError, asyncio.TimeoutError): + pass + + if timeout is not None: + await asyncio.wait_for(run_and_wait(), timeout=timeout) + else: + await run_and_wait() + + except asyncio.TimeoutError: + # Terminate but preserve partial output + try: + await asyncio.shield(terminate_process_safe()) + except asyncio.CancelledError: + # Shield was pierced, do sync cleanup + if process and process.returncode is None: + process.kill() + raise SubprocessTimeoutError( + build_result( + returncode=process.returncode if process else None, + timed_out=True, + ) + ) + + except asyncio.CancelledError: + try: + await asyncio.shield(terminate_process_safe()) + except asyncio.CancelledError: + if process and process.returncode is None: + process.kill() + raise SubprocessCancelledError( + build_result( + returncode=process.returncode if process else None, + cancelled=True, + ) + ) + + result = build_result(returncode=process.returncode) + + if check and process.returncode != 0: + if process.returncode is None: + returncode = -1 + else: + returncode = process.returncode + + raise CalledProcessError( + returncode=returncode, + cmd=cmd, + output=result.stdout, + stderr=result.stderr, + ) + + return result diff --git a/vero/src/vero/utils/general.py b/vero/src/vero/utils/general.py new file mode 100644 index 00000000..e323f70a --- /dev/null +++ b/vero/src/vero/utils/general.py @@ -0,0 +1,104 @@ +from __future__ import annotations + +import re +from dataclasses import asdict, is_dataclass +from pathlib import Path +from typing import ( + Protocol, + TypeVar, + overload, + runtime_checkable, +) + +from pydantic import BaseModel, JsonValue + +JsonT = TypeVar("JsonT", bound=JsonValue) + + +@runtime_checkable +class IsDataclass(Protocol): + __dataclass_fields__: dict # all dataclasses have this attribute + + +@overload +def recursively_serialize(data: BaseModel) -> dict[str, JsonValue]: ... + + +@overload +def recursively_serialize(data: JsonT) -> JsonT: ... + + +@overload +def recursively_serialize(data: Path) -> str: ... + + +@overload +def recursively_serialize(data: IsDataclass) -> dict[str, JsonValue]: ... + + +def recursively_serialize( + data: BaseModel | JsonValue | Path | IsDataclass, +) -> JsonValue: + """Recursively serialize a BaseModel or JsonValue or dataclass to a JSON value.""" + if is_dataclass(data): + return recursively_serialize(asdict(data)) + + if isinstance(data, dict): + return {k: recursively_serialize(v) for k, v in data.items()} + + if isinstance(data, (list, tuple, set)): + return [recursively_serialize(item) for item in data] + + if isinstance(data, BaseModel): + return recursively_serialize(data.model_dump()) + + if isinstance(data, Path): + return str(data) + + return data # type: ignore + + +def camel_to_snake(s: str) -> str: + """Convert a camel case string to a snake case string.""" + snake = [] + for char in s: + if char.isupper() and snake: + snake.append("_") + snake.append(char.lower()) + return "".join(snake) + + +def strip_ansi(text: str) -> str: + """Strip ANSI escape codes from a string.""" + _ANSI_ESCAPE_PATTERN = re.compile(r"\x1b(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])") + return _ANSI_ESCAPE_PATTERN.sub("", text) + + +def paginate( + items: list[str], max_chars: int, offset: int = 0, limit: int | None = None +) -> list[str]: + """Paginate a list of items into a list of strings. max_chars is not respected for single item lists.""" + + if offset < 0: + raise ValueError("offset must be greater than or equal to 0.") + + if offset > len(items): + raise ValueError( + f"offset must be less than the length of the items {len(items)}." + ) + + items = items[offset:] + if limit is None: + limit = len(items) + + paginated_items = [] + paginated_items_length = 0 + + for item in items[:limit]: + paginated_items.append(item) + paginated_items_length += len(item) + + if paginated_items_length > max_chars: + break + + return paginated_items diff --git a/vero/src/vero/utils/openai_agents.py b/vero/src/vero/utils/openai_agents.py new file mode 100644 index 00000000..75b7ae3f --- /dev/null +++ b/vero/src/vero/utils/openai_agents.py @@ -0,0 +1,54 @@ +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING, AsyncGenerator + +from vero.exceptions import StreamEventTimeout +from vero.utils import anext_with_timeout + +if TYPE_CHECKING: + from agents import OpenAIChatCompletionsModel, RunResultStreaming, StreamEvent + from agents.extensions.models.litellm_model import LitellmModel + + +async def stream_events( + result: RunResultStreaming, + event_timeout: int | None = None, +) -> AsyncGenerator[StreamEvent, None]: + """Stream events from an OpenAI agent run. + + Args: + result: The RunResultStreaming object to stream events from. + event_timeout: Timeout in seconds for each event. None means no timeout. + """ + + event_iter = aiter(result.stream_events()) + + while True: + try: + if event_timeout is not None: + event = await anext_with_timeout(event_iter, event_timeout) + else: + event = await anext(event_iter) + yield event + except StopAsyncIteration: + break + except asyncio.TimeoutError: + result.cancel(mode="immediate") + await asyncio.sleep(0.1) + raise StreamEventTimeout( + f"Timeout of {event_timeout} seconds reached for an event on iterator {event_iter}!" + ) + + +def strict_mode_from_model( + model: str | OpenAIChatCompletionsModel | LitellmModel, +) -> bool: + """Gets the strict mode of the tool schema from the model name.""" + + if not isinstance(model, str): + model = model.model + + return ( + "/" not in model or model.startswith("openai") or model.startswith("anthropic") + ) diff --git a/vero/src/vero/workspace/__init__.py b/vero/src/vero/workspace/__init__.py new file mode 100644 index 00000000..d8a8d027 --- /dev/null +++ b/vero/src/vero/workspace/__init__.py @@ -0,0 +1,6 @@ +from __future__ import annotations + +from vero.workspace.base import Workspace +from vero.workspace.git import GitWorkspace # noqa: E402 + +__all__ = ["GitWorkspace", "Workspace"] diff --git a/vero/src/vero/workspace/base.py b/vero/src/vero/workspace/base.py new file mode 100644 index 00000000..6c5befdb --- /dev/null +++ b/vero/src/vero/workspace/base.py @@ -0,0 +1,141 @@ +from __future__ import annotations + +import posixpath +from abc import ABC, abstractmethod +from contextlib import asynccontextmanager +from pathlib import PurePosixPath +from typing import TYPE_CHECKING, AsyncGenerator + +if TYPE_CHECKING: + from vero.sandbox import Sandbox + + +class Workspace(ABC): + """Abstract version-control workspace. + + Combines an isolated working directory with snapshot/restore versioning. + Workspace depends on a Sandbox for file and shell operations. All + version-control methods are async to support non-local sandbox backends. + """ + + # ── Properties ───────────────────────────────────────────────── + + @property + @abstractmethod + def sandbox(self) -> Sandbox: + """The execution environment this workspace operates in.""" + ... + + @property + @abstractmethod + def root(self) -> str: + """Workspace root directory (e.g. git repo root).""" + ... + + @property + @abstractmethod + def project_path(self) -> str: + """Project directory within this workspace.""" + ... + + @property + @abstractmethod + def name(self) -> str: + """Human-readable name for this workspace.""" + ... + + # ── History ───────────────────────────────────────────────────── + + @abstractmethod + async def current_version(self) -> str: + """Return the current version ID (commit hash, change ID, etc.).""" + ... + + @abstractmethod + async def save(self, message: str = "Save") -> str: + """Create a new version from current state. Returns the version ID.""" + ... + + @abstractmethod + async def restore(self, version_id: str, message: str | None = None) -> str: + """Revert to a previous version, preserving history. Returns new version ID.""" + ... + + # ── History inspection ────────────────────────────────────────── + + @abstractmethod + async def diff( + self, from_version: str | None = None, to_version: str | None = None + ) -> str: + """Diff between two versions.""" + ... + + @abstractmethod + async def log(self, max_count: int = 10, since_version: str | None = None) -> str: + """Version history.""" + ... + + @abstractmethod + async def is_ancestor(self, version_a: str, version_b: str) -> bool: + """Is version_a an ancestor of version_b?""" + ... + + # ── Copies ────────────────────────────────────────────────────── + + @abstractmethod + async def copy( + self, name: str | None = None, from_version: str | None = None + ) -> Workspace: + """Create a persistent isolated copy of this workspace.""" + ... + + @asynccontextmanager + async def temp_copy( + self, from_version: str | None = None + ) -> AsyncGenerator[Workspace, None]: + """Temporary isolated copy, cleaned up on exit.""" + yield self # pragma: no cover + + # ── Execution at a version ────────────────────────────────────── + + @asynccontextmanager + async def at(self, version_id: str) -> AsyncGenerator[None, None]: + """Temporarily switch to a version, restore on exit.""" + yield # pragma: no cover + + # ── State ─────────────────────────────────────────────────────── + + @abstractmethod + async def is_dirty(self) -> bool: + """Are there unsaved changes not yet captured in a version?""" + ... + + async def destroy(self) -> None: + """Clean up this workspace. Default: no-op.""" + pass + + # ── Path resolution ──────────────────────────────────────────── + + def resolve_path(self, path: str) -> str: + """Resolve a path relative to ``project_path``. + + Absolute paths pass through. Relative paths (including ``"."``) + are resolved against ``project_path``, not ``sandbox.root``. + """ + value = PurePosixPath(str(path)) + if value.is_absolute(): + return posixpath.normpath(value.as_posix()) + root = posixpath.normpath(PurePosixPath(str(self.project_path)).as_posix()) + return posixpath.normpath(posixpath.join(root, value.as_posix())) + + def get_relative_path(self, path: str) -> str | None: + """Get path relative to ``project_path``, or None if outside.""" + resolved = PurePosixPath(self.resolve_path(path)) + root = PurePosixPath( + posixpath.normpath(PurePosixPath(str(self.project_path)).as_posix()) + ) + try: + relative = resolved.relative_to(root) + except ValueError: + return None + return relative.as_posix() diff --git a/vero/src/vero/workspace/git.py b/vero/src/vero/workspace/git.py new file mode 100644 index 00000000..7fd20597 --- /dev/null +++ b/vero/src/vero/workspace/git.py @@ -0,0 +1,408 @@ +from __future__ import annotations + +import asyncio +import logging +import uuid +from contextlib import asynccontextmanager +from pathlib import PurePosixPath +from typing import AsyncGenerator + +from vero.sandbox import Sandbox +from vero.workspace.base import Workspace + +logger = logging.getLogger(__name__) + + +def _basename(path: str) -> str: + """Extract the last component of a path.""" + return PurePosixPath(path).name + + +def _parent(path: str) -> str: + """Extract the parent directory of a path.""" + return str(PurePosixPath(path).parent) + + +def _join(parent: str, child: str) -> str: + """Join two path components.""" + return str(PurePosixPath(parent) / child) + + +class GitWorkspace(Workspace): + """Workspace backed by git, using sandbox.run() for all git operations. + + Works with any Sandbox implementation — local, Docker, remote. + All path manipulation uses string operations (no pathlib/os.path) + so it remains correct for non-local sandboxes. + """ + + def __init__( + self, + sandbox: Sandbox, + root: str, + project_path: str | None = None, + name: str | None = None, + worktree_owner_root: str | None = None, + ) -> None: + self._sandbox = sandbox + self._root = root + self._project_path = project_path or root + self._name = name or _basename(root) + self._worktree_owner_root = worktree_owner_root + self._lock = asyncio.Lock() + + @property + def sandbox(self) -> Sandbox: + return self._sandbox + + @property + def root(self) -> str: + return self._root + + @property + def project_path(self) -> str: + return self._project_path + + @property + def name(self) -> str: + return self._name + + # ── Git helper ────────────────────────────────────────────────── + + async def _git(self, *args: str) -> str: + """Run a git command via sandbox.run(), returning stdout. Raises on non-zero exit.""" + result = await self._sandbox.run( + ["git", "-c", f"safe.directory={self._root}", *args], + cwd=self._root, + ) + if result.returncode != 0: + raise RuntimeError(f"git {' '.join(args)} failed: {result.stderr}") + return result.stdout + + # ── Construction helpers ──────────────────────────────────────── + + @classmethod + async def from_path( + cls, + sandbox: Sandbox, + project_path: str, + ) -> GitWorkspace: + """Create a GitWorkspace from a project path. + + Finds the git repo root and determines the project-relative path. + All resolution happens via sandbox commands, not local path operations. + """ + project_path = await sandbox.canonicalize(str(project_path)) + + result = await sandbox.run( + [ + "git", + "-c", + "safe.directory=*", + "rev-parse", + "--show-toplevel", + ], + cwd=project_path, + ) + if result.returncode != 0: + detail = f": {result.stderr}" if result.stderr else "" + raise RuntimeError(f"Not a git repository: {project_path}{detail}") + repo_root = await sandbox.canonicalize(result.stdout.strip()) + + # Find the main repo name (handles worktrees whose common dir differs) + repo_name = _basename(repo_root) + try: + result = await sandbox.run( + [ + "git", + "-c", + f"safe.directory={repo_root}", + "rev-parse", + "--git-common-dir", + ], + cwd=project_path, + ) + if result.returncode == 0: + git_common_dir = result.stdout.strip() + # Only use common-dir if it's an absolute path (worktree case). + # Regular repos return ".git" (relative), which is not useful. + if git_common_dir.startswith("/"): + main_root = _parent(git_common_dir) + repo_name = _basename(main_root) + except RuntimeError: + pass + + return cls( + sandbox=sandbox, + root=repo_root, + project_path=project_path, + name=repo_name, + ) + + @classmethod + async def create(cls, project_path: str) -> GitWorkspace: + """Create a GitWorkspace with a default local sandbox. + + Convenience factory for non-Policy callers (evaluator, trace analysis, + etc.) that just need a workspace for git operations without agent + access control. + """ + from vero.sandbox import LocalSandbox + + sandbox = await LocalSandbox.create() + return await cls.from_path(sandbox, str(project_path)) + + # ── History ───────────────────────────────────────────────────── + + async def current_version(self) -> str: + return await self._git("rev-parse", "HEAD") + + async def save(self, message: str = "Save") -> str: + async with self._lock: + if not await self.is_dirty(): + return await self.current_version() + + # Stage changes scoped to the project path + if self._project_path == self._root: + await self._git("add", "--all") + else: + await self._git("add", self._project_path) + + # Commit (skip hooks for automated commits) + await self._git( + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + message, + "--no-verify", + ) + + return await self.current_version() + + async def restore(self, version_id: str, message: str | None = None) -> str: + async with self._lock: + message = message or f"Restore to {version_id[:8]}" + + # Checkout files from the target version + await self._git("checkout", version_id, "--", ".") + + # Stage everything + await self._git("add", "--all") + + # Only commit if there are actual changes + try: + await self._git("diff", "--cached", "--quiet", "--exit-code") + # No changes — already at that state + except RuntimeError: + # There are staged changes — commit them + await self._git( + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + message, + "--no-verify", + ) + + return await self.current_version() + + # ── History inspection ────────────────────────────────────────── + + async def diff( + self, from_version: str | None = None, to_version: str | None = None + ) -> str: + args = ["diff"] + if from_version: + args.append(from_version) + if to_version: + args.append(to_version) + return await self._git(*args) + + async def log(self, max_count: int = 10, since_version: str | None = None) -> str: + args = ["log", f"-n{max_count}", "--pretty=format:%h - %s (%cr) <%an>"] + if since_version: + args.append(f"{since_version}..HEAD") + return await self._git(*args) + + async def is_ancestor(self, version_a: str, version_b: str) -> bool: + try: + await self._git("merge-base", "--is-ancestor", version_a, version_b) + return True + except RuntimeError: + return False + + def _copied_project_path(self, copied_root: str) -> str: + relative = PurePosixPath(self._project_path).relative_to(self._root) + if str(relative) == ".": + return copied_root + return _join(copied_root, str(relative)) + + # ── Copies ────────────────────────────────────────────────────── + + async def _remove_worktree(self, target_path: str) -> None: + result = await self._sandbox.run( + ["git", "worktree", "remove", "--force", target_path], + cwd=self._root, + ) + if result.returncode != 0 and await self._sandbox.exists(target_path): + await self._sandbox.remove(target_path, recursive=True) + await self._sandbox.run( + ["git", "worktree", "prune"], + cwd=self._root, + ) + + async def _add_worktree(self, target_path: str, from_version: str | None) -> None: + if await self._sandbox.exists(target_path): + raise FileExistsError(target_path) + arguments = ["worktree", "add", "--detach", target_path] + if from_version is not None: + arguments.append(from_version) + try: + await self._git(*arguments) + except BaseException: + await asyncio.shield(self._remove_worktree(target_path)) + raise + + async def copy( + self, name: str | None = None, from_version: str | None = None + ) -> GitWorkspace: + """Create a new git worktree as an isolated copy.""" + async with self._lock: + if name is None: + name = f"worktree-{uuid.uuid4().hex[:8]}" + + target_path = _join(_parent(self._root), name) + await self._add_worktree(target_path, from_version) + + return GitWorkspace( + sandbox=self._sandbox, + root=target_path, + project_path=self._copied_project_path(target_path), + name=self._name, + worktree_owner_root=self._root, + ) + + @asynccontextmanager + async def temp_copy( + self, from_version: str | None = None + ) -> AsyncGenerator[GitWorkspace, None]: + """Create a temporary worktree, cleaned up on exit.""" + copy_name = f"tmp-{uuid.uuid4().hex[:8]}" + + # Ask the sandbox for a temp directory + result = await self._sandbox.run(["mktemp", "-d"]) + if result.returncode != 0: + raise RuntimeError(f"Failed to create temp dir: {result.stderr}") + temporary_root = result.stdout.strip() + target_path = _join(temporary_root, copy_name) + + try: + async with self._lock: + await self._add_worktree(target_path, from_version) + except BaseException: + await asyncio.shield(self._sandbox.remove(temporary_root, recursive=True)) + raise + + target_path = await self._sandbox.canonicalize(target_path) + + copied = GitWorkspace( + sandbox=self._sandbox, + root=target_path, + project_path=self._copied_project_path(target_path), + name=self._name, + worktree_owner_root=self._root, + ) + + try: + yield copied + finally: + await asyncio.shield(copied.destroy()) + await asyncio.shield(self._sandbox.remove(temporary_root, recursive=True)) + + # ── Execution at a version ────────────────────────────────────── + + @asynccontextmanager + async def at(self, version_id: str) -> AsyncGenerator[None, None]: + """Temporarily checkout a version, restore previous state on exit.""" + async with self._lock: + # Remember current state + try: + previous_branch = await self._git("symbolic-ref", "--short", "HEAD") + except RuntimeError: + previous_branch = None + previous_commit = await self.current_version() + + try: + await self._git("checkout", version_id) + yield + finally: + if previous_branch: + await self._git("checkout", previous_branch) + else: + await self._git("checkout", previous_commit) + + # ── Optional ──────────────────────────────────────────────────── + + async def is_dirty(self) -> bool: + status = await self._git("status", "--porcelain", self._project_path) + return bool(status) + + async def destroy(self) -> None: + """Remove this worktree.""" + if self._worktree_owner_root is None: + return + async with self._lock: + owner = GitWorkspace( + sandbox=self._sandbox, + root=self._worktree_owner_root, + ) + await owner._remove_worktree(self._root) + self._worktree_owner_root = None + + # ── Git-specific helpers (used by Policy and git tools) ───────── + + async def resolve_ref(self, ref: str) -> str: + """Resolve a git ref (branch, tag, short hash) to a full commit hash.""" + return await self._git("rev-parse", ref) + + async def current_branch(self) -> str | None: + """Return current branch name, or None if detached HEAD.""" + try: + return await self._git("symbolic-ref", "--short", "HEAD") + except RuntimeError: + return None + + async def branch_exists(self, branch_name: str) -> bool: + try: + await self._git("rev-parse", "--verify", f"refs/heads/{branch_name}") + return True + except RuntimeError: + return False + + async def get_head_commit(self, branch_name: str) -> str: + return await self._git("rev-parse", f"refs/heads/{branch_name}") + + async def checkout_branch( + self, branch_name: str, create: bool = False, from_ref: str | None = None + ) -> None: + async with self._lock: + args = ["checkout"] + if create: + args.append("-b") + args.append(branch_name) + if from_ref: + args.append(from_ref) + await self._git(*args) + + async def maybe_fetch(self) -> None: + """Fetch from remote if one exists.""" + try: + await self._git("remote", "get-url", "origin") + await self._git("fetch", "--all") + except RuntimeError: + pass diff --git a/vero/tests/__init__.py b/vero/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/vero/tests/test_lockfile.py b/vero/tests/test_lockfile.py new file mode 100644 index 00000000..eac0589c --- /dev/null +++ b/vero/tests/test_lockfile.py @@ -0,0 +1,33 @@ +"""Tests to ensure project configuration files are up-to-date.""" + +import subprocess +import warnings +from pathlib import Path + +# Get the project root (parent of tests directory) +PROJECT_ROOT = Path(__file__).parent.parent + + +class TestLockfile: + """Tests for uv.lock file synchronization.""" + + def test_uv_lock_is_up_to_date(self): + """Ensure uv.lock is synchronized with pyproject.toml. + + If this test warns, run `uv lock` to update the lock file. + """ + result = subprocess.run( + ["uv", "lock", "--check"], + cwd=PROJECT_ROOT, + capture_output=True, + text=True, + ) + + if result.returncode != 0: + warnings.warn( + f"uv.lock is out of sync with pyproject.toml. " + f"Run `uv lock` to update it.\n" + f"stdout: {result.stdout}\n" + f"stderr: {result.stderr}", + UserWarning, + ) diff --git a/vero/tests/test_remote_sandbox.py b/vero/tests/test_remote_sandbox.py new file mode 100644 index 00000000..ef9cab63 --- /dev/null +++ b/vero/tests/test_remote_sandbox.py @@ -0,0 +1,182 @@ +"""Tests verifying sandbox boundary correctness with a simulated remote sandbox. + +MockRemoteSandbox uses two separate directories — "host" and "remote" — to +simulate a non-local sandbox where host and sandbox don't share a filesystem. +upload() copies host→remote, download() copies remote→host. All file ops +happen in the remote dir. This catches any code that assumes shared paths. +""" + +from __future__ import annotations + +import shutil +from pathlib import Path + +import pytest + +from vero.sandbox import CommandResult, FileStat, Sandbox +from vero.staging import SandboxStagingArea + + +class MockRemoteSandbox(Sandbox): + """Sandbox that simulates a remote environment. + + File ops happen in `remote_root`. The host writes to `host_root`. + upload/download actually copy between them. This catches any code + that assumes host paths are accessible inside the sandbox. + """ + + def __init__(self, host_root: Path, remote_root: Path): + self._host_root = host_root + self._remote_root = remote_root + + @classmethod + async def create(cls, **kwargs) -> MockRemoteSandbox: + raise NotImplementedError("MockRemoteSandbox requires explicit host/remote roots") + + @property + def root(self) -> str: + return str(self._remote_root) + + def resolve_path(self, path: str) -> str: + p = Path(path) + if p.is_absolute(): + return str(p.resolve()) + return str((self._remote_root / p).resolve()) + + async def read_file(self, path: str, encoding: str = "utf-8") -> str: + resolved = Path(self.resolve_path(path)) + return resolved.read_text(encoding=encoding) + + async def read_file_bytes(self, path: str, limit: int | None = None) -> bytes: + resolved = Path(self.resolve_path(path)) + with open(resolved, "rb") as f: + return f.read(limit) if limit is not None else f.read() + + async def write_file(self, path: str, content: str, encoding: str = "utf-8") -> None: + resolved = Path(self.resolve_path(path)) + resolved.parent.mkdir(parents=True, exist_ok=True) + resolved.write_text(content, encoding=encoding) + + async def exists(self, path: str) -> bool: + return Path(self.resolve_path(path)).exists() + + async def is_file(self, path: str) -> bool: + return Path(self.resolve_path(path)).is_file() + + async def is_dir(self, path: str) -> bool: + return Path(self.resolve_path(path)).is_dir() + + async def mkdir(self, path: str, parents: bool = True) -> None: + Path(self.resolve_path(path)).mkdir(parents=parents, exist_ok=True) + + async def stat(self, path: str) -> FileStat: + st = Path(self.resolve_path(path)).stat() + return FileStat(st_size=st.st_size) + + async def list_dir(self, path: str) -> list[str]: + return sorted(e.name for e in Path(self.resolve_path(path)).iterdir()) + + async def run(self, command, cwd=None, timeout=30, env=None) -> CommandResult: + import asyncio + + if isinstance(command, str): + command = ["bash", "-c", command] + if cwd is None: + cwd = str(self._remote_root) + proc = await asyncio.create_subprocess_exec( + *command, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + cwd=str(cwd), + env=env, + ) + stdout, stderr = await proc.communicate() + return CommandResult( + stdout=stdout.decode().strip(), + stderr=stderr.decode().strip(), + returncode=proc.returncode or 0, + ) + + async def upload(self, local_path: str, remote_path: str) -> None: + """Copy from host to remote. This MUST actually copy (not no-op).""" + src = Path(local_path) + dst = Path(remote_path) + if not src.exists(): + return + if src.is_dir(): + shutil.copytree(src, dst, dirs_exist_ok=True) + else: + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + async def download(self, remote_path: str, local_path: str) -> None: + """Copy from remote to host. This MUST actually copy (not no-op).""" + src = Path(remote_path) + dst = Path(local_path) + if not src.exists(): + return + if src.is_dir(): + shutil.copytree(src, dst, dirs_exist_ok=True) + else: + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + +class TestMockRemoteSandboxBasics: + """Verify the mock itself works correctly.""" + + @pytest.fixture + def sandbox(self, tmp_path): + host = tmp_path / "host" + remote = tmp_path / "remote" + host.mkdir() + remote.mkdir() + return MockRemoteSandbox( + host_root=host, + remote_root=remote, + ) + + @pytest.mark.asyncio + async def test_host_and_remote_are_separate(self, sandbox, tmp_path): + """Writing to sandbox doesn't create files on the host.""" + await sandbox.write_file("test.txt", "hello") + + # File exists in remote + assert (tmp_path / "remote" / "test.txt").exists() + # File does NOT exist in host + assert not (tmp_path / "host" / "test.txt").exists() + + @pytest.mark.asyncio + async def test_upload_copies_to_remote(self, sandbox, tmp_path): + host_file = tmp_path / "host" / "data.txt" + host_file.write_text("from host") + + remote_path = str(tmp_path / "remote" / "data.txt") + await sandbox.upload(str(host_file), remote_path) + + assert (tmp_path / "remote" / "data.txt").read_text() == "from host" + + @pytest.mark.asyncio + async def test_download_copies_from_remote(self, sandbox, tmp_path): + await sandbox.write_file("result.txt", "from remote") + + local_path = str(tmp_path / "host" / "result.txt") + await sandbox.download(str(tmp_path / "remote" / "result.txt"), local_path) + + assert (tmp_path / "host" / "result.txt").read_text() == "from remote" + + @pytest.mark.asyncio + async def test_staging_area_exchanges_data_and_cleans_up(self, sandbox, tmp_path): + source = tmp_path / "host" / "input.txt" + source.write_text("input", encoding="utf-8") + destination = tmp_path / "host" / "output.txt" + + async with SandboxStagingArea(sandbox) as staging: + staging_root = Path(staging.root) + await staging.upload(source, "inputs/input.txt") + assert await staging.read_text("inputs/input.txt") == "input" + await staging.write_text("outputs/output.txt", "output") + await staging.download("outputs/output.txt", destination) + + assert destination.read_text(encoding="utf-8") == "output" + assert not staging_root.exists() diff --git a/vero/tests/test_sandbox.py b/vero/tests/test_sandbox.py new file mode 100644 index 00000000..4e92a803 --- /dev/null +++ b/vero/tests/test_sandbox.py @@ -0,0 +1,363 @@ +"""Tests for the Sandbox abstraction (LocalSandbox).""" + +from __future__ import annotations + +import asyncio +import sys +from pathlib import Path + +import pytest + +import vero.sandbox as sandbox_module +from vero.sandbox import CommandResult, FileStat, LocalSandbox, Sandbox + + +@pytest.fixture +def sandbox(tmp_path): + """Create a LocalSandbox for I/O testing.""" + (tmp_path / "hello.txt").write_text("hello world\n") + (tmp_path / "subdir").mkdir() + (tmp_path / "subdir" / "nested.py").write_text("x = 1\n") + (tmp_path / "private").mkdir() + (tmp_path / "private" / "secret.txt").write_text("top secret\n") + + return LocalSandbox(root=tmp_path) + + +@pytest.fixture +def fast_process_group_termination(monkeypatch): + terminate = sandbox_module._terminate_host_process_tree + + async def terminate_quickly(process): + await terminate(process, grace_seconds=0.1) + + monkeypatch.setattr( + sandbox_module, + "_terminate_host_process_tree", + terminate_quickly, + ) + + +class TestSandboxABC: + def test_cannot_instantiate(self): + with pytest.raises(TypeError): + Sandbox() + + +class TestProperties: + def test_root(self, sandbox, tmp_path): + assert sandbox.root == str(tmp_path) + + def test_resolve_path(self, sandbox, tmp_path): + result = sandbox.resolve_path("hello.txt") + assert result == str(tmp_path / "hello.txt") + + def test_local_paths_are_host_visible(self, sandbox, tmp_path): + assert sandbox.capabilities.host_paths + assert sandbox.host_path("hello.txt") == tmp_path / "hello.txt" + + @pytest.mark.asyncio + async def test_temporary_directory_is_cleaned_up(self, sandbox): + async with sandbox.temporary_directory("vero-test-") as directory: + assert await sandbox.is_dir(directory) + assert Path(directory).name.startswith("vero-test-") + assert directory == str(Path(directory).resolve()) + assert not await sandbox.exists(directory) + + +class TestFileOperations: + @pytest.mark.asyncio + async def test_read_file(self, sandbox): + content = await sandbox.read_file("hello.txt") + assert content == "hello world\n" + + @pytest.mark.asyncio + async def test_read_file_bytes(self, sandbox): + data = await sandbox.read_file_bytes("hello.txt") + assert data == b"hello world\n" + + @pytest.mark.asyncio + async def test_read_file_bytes_with_limit(self, sandbox): + data = await sandbox.read_file_bytes("hello.txt", limit=5) + assert data == b"hello" + + @pytest.mark.asyncio + async def test_write_file(self, sandbox, tmp_path): + await sandbox.write_file("new.txt", "new content\n") + assert (tmp_path / "new.txt").read_text() == "new content\n" + + @pytest.mark.asyncio + async def test_write_file_creates_parents(self, sandbox, tmp_path): + await sandbox.write_file("a/b/c.txt", "deep\n") + assert (tmp_path / "a" / "b" / "c.txt").read_text() == "deep\n" + + @pytest.mark.asyncio + async def test_exists(self, sandbox): + assert await sandbox.exists("hello.txt") is True + assert await sandbox.exists("nonexistent.txt") is False + + @pytest.mark.asyncio + async def test_is_file(self, sandbox): + assert await sandbox.is_file("hello.txt") is True + assert await sandbox.is_file("subdir") is False + + @pytest.mark.asyncio + async def test_is_dir(self, sandbox): + assert await sandbox.is_dir("subdir") is True + assert await sandbox.is_dir("hello.txt") is False + + @pytest.mark.asyncio + async def test_mkdir(self, sandbox, tmp_path): + await sandbox.mkdir("new_dir/nested") + assert (tmp_path / "new_dir" / "nested").is_dir() + + @pytest.mark.asyncio + async def test_stat(self, sandbox): + result = await sandbox.stat("hello.txt") + assert isinstance(result, FileStat) + assert result.st_size == len("hello world\n") + + @pytest.mark.asyncio + async def test_list_dir(self, sandbox): + entries = await sandbox.list_dir(sandbox.root) + assert "hello.txt" in entries + assert "subdir" in entries + assert "private" in entries + assert entries == sorted(entries) + + @pytest.mark.asyncio + async def test_list_dir_subdir(self, sandbox): + entries = await sandbox.list_dir("subdir") + assert entries == ["nested.py"] + + +class TestShellExecution: + @pytest.mark.asyncio + async def test_run_simple_command(self, sandbox): + result = await sandbox.run(["echo", "hello"]) + assert isinstance(result, CommandResult) + assert result.stdout == "hello" + assert result.returncode == 0 + + @pytest.mark.asyncio + async def test_run_returns_stderr(self, sandbox): + result = await sandbox.run(["ls", "/nonexistent_path_xyz"]) + assert result.returncode != 0 + assert result.stderr != "" + + @pytest.mark.asyncio + async def test_run_does_not_raise_on_nonzero(self, sandbox): + result = await sandbox.run(["false"]) + assert result.returncode != 0 + + @pytest.mark.asyncio + async def test_run_default_cwd_is_root(self, sandbox): + result = await sandbox.run(["pwd"]) + assert result.stdout == sandbox.root + + @pytest.mark.asyncio + async def test_run_custom_cwd(self, sandbox): + result = await sandbox.run(["pwd"], cwd=sandbox.resolve_path("subdir")) + assert result.stdout.endswith("subdir") + + @pytest.mark.asyncio + async def test_run_string_command(self, sandbox): + result = await sandbox.run("echo hello && echo world") + assert "hello" in result.stdout + assert "world" in result.stdout + + @pytest.mark.asyncio + async def test_run_timeout(self, sandbox): + result = await sandbox.run(["sleep", "10"], timeout=1) + assert result.returncode == -1 + assert "timed out" in result.stderr + + @pytest.mark.asyncio + async def test_timeout_terminates_descendant_processes( + self, + sandbox, + tmp_path, + fast_process_group_termination, + ): + marker = tmp_path / "timeout-descendant-survived" + descendant = ( + "import signal, time; from pathlib import Path; " + "signal.signal(signal.SIGTERM, signal.SIG_IGN); " + f"time.sleep(0.8); Path({str(marker)!r}).write_text('leaked')" + ) + parent = ( + "import subprocess, sys, time; " + f"subprocess.Popen([sys.executable, '-c', {descendant!r}]); " + "time.sleep(60)" + ) + + result = await sandbox.run([sys.executable, "-c", parent], timeout=0.1) + assert result.returncode == -1 + await asyncio.sleep(1) + assert not marker.exists() + + @pytest.mark.asyncio + async def test_cancellation_terminates_descendant_processes( + self, + sandbox, + tmp_path, + fast_process_group_termination, + ): + marker = tmp_path / "cancelled-descendant-survived" + descendant = ( + "import signal, time; from pathlib import Path; " + "signal.signal(signal.SIGTERM, signal.SIG_IGN); " + f"time.sleep(0.8); Path({str(marker)!r}).write_text('leaked')" + ) + parent = ( + "import subprocess, sys, time; " + f"subprocess.Popen([sys.executable, '-c', {descendant!r}]); " + "time.sleep(60)" + ) + task = asyncio.create_task( + sandbox.run([sys.executable, "-c", parent], timeout=None) + ) + await asyncio.sleep(0.1) + + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + await asyncio.sleep(1) + assert not marker.exists() + + @pytest.mark.asyncio + async def test_timeout_cleans_descendants_after_group_leader_exits( + self, + sandbox, + tmp_path, + fast_process_group_termination, + ): + marker = tmp_path / "detached-descendant-survived" + descendant = ( + "import signal, time; from pathlib import Path; " + "signal.signal(signal.SIGTERM, signal.SIG_IGN); " + f"time.sleep(0.8); Path({str(marker)!r}).write_text('leaked')" + ) + parent = ( + "import subprocess, sys; " + f"subprocess.Popen([sys.executable, '-c', {descendant!r}])" + ) + + result = await sandbox.run([sys.executable, "-c", parent], timeout=0.1) + assert result.returncode == -1 + await asyncio.sleep(1) + assert not marker.exists() + + @pytest.mark.asyncio + async def test_run_env(self, sandbox): + import os + + env = {**os.environ, "MY_TEST_VAR": "test_value_123"} + result = await sandbox.run(["printenv", "MY_TEST_VAR"], env=env) + assert result.stdout == "test_value_123" + + +class TestFileTransfer: + @pytest.mark.asyncio + async def test_upload_file(self, sandbox, tmp_path): + # Create a file outside the sandbox + external = tmp_path / "external" + external.mkdir() + (external / "data.txt").write_text("uploaded\n") + + dest = sandbox.resolve_path("uploaded.txt") + await sandbox.upload(str(external / "data.txt"), dest) + + content = await sandbox.read_file("uploaded.txt") + assert content == "uploaded\n" + + @pytest.mark.asyncio + async def test_upload_directory(self, sandbox, tmp_path): + external = tmp_path / "external_dir" + external.mkdir() + (external / "a.txt").write_text("aaa\n") + (external / "b.txt").write_text("bbb\n") + + dest = sandbox.resolve_path("imported") + await sandbox.upload(str(external), dest) + + assert await sandbox.is_dir("imported") + assert await sandbox.read_file(str(Path(dest) / "a.txt")) == "aaa\n" + assert await sandbox.read_file(str(Path(dest) / "b.txt")) == "bbb\n" + + @pytest.mark.asyncio + async def test_download_file(self, sandbox, tmp_path): + await sandbox.write_file("to_download.txt", "download me\n") + + local_dest = tmp_path / "downloaded.txt" + await sandbox.download(sandbox.resolve_path("to_download.txt"), str(local_dest)) + + assert local_dest.read_text() == "download me\n" + + @pytest.mark.asyncio + async def test_download_directory(self, sandbox, tmp_path): + await sandbox.mkdir("export_dir") + await sandbox.write_file( + str(Path(sandbox.resolve_path("export_dir")) / "x.txt"), "xxx\n" + ) + + local_dest = tmp_path / "exported" + await sandbox.download(sandbox.resolve_path("export_dir"), str(local_dest)) + + assert local_dest.is_dir() + assert (local_dest / "x.txt").read_text() == "xxx\n" + + @pytest.mark.asyncio + async def test_upload_noop_same_path(self, sandbox): + """upload is a no-op when source and dest resolve to the same path.""" + path = sandbox.resolve_path("hello.txt") + await sandbox.upload(path, path) # should not crash or duplicate + content = await sandbox.read_file("hello.txt") + assert content == "hello world\n" + + @pytest.mark.asyncio + async def test_download_noop_same_path(self, sandbox): + """download is a no-op when source and dest resolve to the same path.""" + path = sandbox.resolve_path("hello.txt") + await sandbox.download(path, path) + content = await sandbox.read_file("hello.txt") + assert content == "hello world\n" + + +class _ImmediateProcess: + returncode = 0 + + async def communicate(self): + return (b"", b"") + + +@pytest.mark.asyncio +async def test_run_as_drops_to_unprivileged_user(sandbox, monkeypatch): + """run_as forwards the user/group and sheds supplementary groups.""" + captured: dict = {} + + async def fake_exec(*args, **kwargs): + captured["kwargs"] = kwargs + return _ImmediateProcess() + + monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_exec) + await sandbox.run(["true"], run_as="harness") + + assert captured["kwargs"]["user"] == "harness" + assert captured["kwargs"]["group"] == "harness" + assert captured["kwargs"]["extra_groups"] == [] + + +@pytest.mark.asyncio +async def test_run_without_run_as_does_not_drop_privileges(sandbox, monkeypatch): + captured: dict = {} + + async def fake_exec(*args, **kwargs): + captured["kwargs"] = kwargs + return _ImmediateProcess() + + monkeypatch.setattr(asyncio, "create_subprocess_exec", fake_exec) + await sandbox.run(["true"]) + + assert "user" not in captured["kwargs"] + assert "group" not in captured["kwargs"] diff --git a/vero/tests/test_subprocess.py b/vero/tests/test_subprocess.py new file mode 100644 index 00000000..3b71b7b2 --- /dev/null +++ b/vero/tests/test_subprocess.py @@ -0,0 +1,108 @@ +"""Tests for subprocess termination on cancellation.""" + +import asyncio +import sys + +import pytest + +from vero.utils.asyncio import ( + SubprocessCancelledError, + SubprocessTimeoutError, + run_bash_command, + run_subprocess_with_tee, +) + + +class TestSubprocessTermination: + """Tests that subprocesses are properly terminated on cancellation.""" + + @pytest.mark.asyncio + async def test_run_subprocess_with_tee_terminates_on_cancellation(self): + """Test that run_subprocess_with_tee terminates subprocess on CancelledError.""" + # Start a long-running subprocess + task = asyncio.create_task(run_subprocess_with_tee(["sleep", "60"], timeout=None)) + + # Give subprocess time to start + await asyncio.sleep(0.1) + + # Cancel the task + task.cancel() + + # Wait for cancellation to complete + with pytest.raises(SubprocessCancelledError): + await task + + # Give OS time to clean up + await asyncio.sleep(0.1) + + # Verify no zombie sleep processes from our test + # (This is a best-effort check - the process should be gone) + + @pytest.mark.asyncio + async def test_run_subprocess_with_tee_terminates_on_timeout(self): + """Test that run_subprocess_with_tee terminates subprocess on timeout.""" + with pytest.raises(SubprocessTimeoutError): + await run_subprocess_with_tee(["sleep", "60"], timeout=1) + + @pytest.mark.asyncio + async def test_run_bash_command_terminates_on_cancellation(self): + """Test that run_bash_command terminates subprocess on CancelledError.""" + task = asyncio.create_task(run_bash_command(["sleep", "60"])) + + await asyncio.sleep(0.1) + task.cancel() + + with pytest.raises(asyncio.CancelledError): + await task + + @pytest.mark.asyncio + async def test_run_bash_command_terminates_on_timeout(self): + """Test that run_bash_command terminates subprocess on timeout.""" + with pytest.raises(TimeoutError): + await run_bash_command(["sleep", "60"], timeout=1) + + @pytest.mark.asyncio + async def test_subprocess_actually_terminates(self): + """Test that the subprocess is actually killed, not just abandoned.""" + # Use a marker file to detect if subprocess is still running + import tempfile + from pathlib import Path + + with tempfile.TemporaryDirectory() as tmpdir: + marker = Path(tmpdir) / "running" + + # Script that creates marker file while running, removes on clean exit + script = f""" +import time +from pathlib import Path +marker = Path("{marker}") +marker.write_text("running") +try: + time.sleep(60) +finally: + marker.unlink(missing_ok=True) +""" + task = asyncio.create_task( + run_subprocess_with_tee([sys.executable, "-c", script], timeout=None) + ) + + # Wait for subprocess to start and create marker + for _ in range(20): + await asyncio.sleep(0.1) + if marker.exists(): + break + else: + pytest.fail("Subprocess didn't start in time") + + # Cancel and wait + task.cancel() + with pytest.raises(SubprocessCancelledError): + await task + + # Give time for cleanup + await asyncio.sleep(0.5) + + # Marker should be gone (subprocess ran finally block) or + # subprocess was killed (marker still exists but process is dead) + # Either way, let's verify no python process is still sleeping + # by checking that we can complete quickly diff --git a/vero/tests/test_tools.py b/vero/tests/test_tools.py new file mode 100644 index 00000000..106f9c0c --- /dev/null +++ b/vero/tests/test_tools.py @@ -0,0 +1,64 @@ +"""Tests for the tools.""" + +import json +import re + +import pytest + +from vero.tools.planning import TodoList, TodoStatus + + +class TestTodoList: + """Tests for TodoList tool.""" + + def test_add_todos(self): + """Test adding todo items.""" + todo_list = TodoList() + todo_list.add_todos("Write tests") + assert len(todo_list.todos) == 1 + assert todo_list.todo_statuses["Write tests"] == TodoStatus.NOT_STARTED + + def test_update_todo_status(self): + """Test updating todo status by task_id.""" + todo_list = TodoList() + todo_list.add_todos("Task 1") + todo_list.add_todos("Task 2") + todo_list.update_todo_status(status=TodoStatus.COMPLETED, task_id=0) + + assert len(todo_list.todo_statuses) == 2 + assert todo_list.todo_statuses["Task 1"] == TodoStatus.COMPLETED + assert todo_list.todo_statuses["Task 2"] == TodoStatus.NOT_STARTED + + result = todo_list.list_todos(status=[TodoStatus.COMPLETED]) + assert "Task 1" in result + assert "Task 2" not in result + + result = todo_list.list_todos(status=[TodoStatus.NOT_STARTED]) + assert "Task 1" not in result + assert "Task 2" in result + + def test_update_status_error_handling(self): + """Test that updating status without task or task_id raises ValueError.""" + todo_list = TodoList() + todo_list.add_todos("Task 1") + + with pytest.raises(ValueError): + todo_list.update_todo_status(status=TodoStatus.COMPLETED) + + def test_list_todos(self): + """Test listing todos with status filters.""" + todo_list = TodoList() + todo_list.add_todos("Task 1") + todo_list.add_todos("Task 2") + todo_list.add_todos("Task 3") + + # Update statuses + todo_list.update_todo_status(status=TodoStatus.COMPLETED, task_id=0) + todo_list.update_todo_status(status=TodoStatus.IN_PROGRESS, task_id=1) + + # List completed tasks + result = todo_list.list_todos(status=TodoStatus.COMPLETED) + json_match = re.search(r"```json(.+?)```", result, re.DOTALL) + todos_dict = json.loads(json_match.group(1)) + assert len(todos_dict) == 1 + assert todos_dict["0"]["task"] == "Task 1" diff --git a/vero/tests/test_uv_with_editable.py b/vero/tests/test_uv_with_editable.py new file mode 100644 index 00000000..07171aec --- /dev/null +++ b/vero/tests/test_uv_with_editable.py @@ -0,0 +1,117 @@ +"""Test that uv run --with-editable correctly overlays a package version. + +Creates two copies of a 'greeter' package with different behavior, +and a 'consumer' package that depends on greeter. Verifies that: +1. uv run --project consumer → uses the installed greeter (v1) +2. uv run --project consumer --with-editable greeter_v2 → uses greeter v2 +""" + +from __future__ import annotations + +import subprocess +import textwrap +from pathlib import Path + +import pytest + + +def _write_package(root: Path, name: str, version: str, code: str) -> Path: + """Create a minimal uv package.""" + pkg_dir = root / name + pkg_dir.mkdir(parents=True) + src_dir = pkg_dir / "src" / name.replace("-", "_") + src_dir.mkdir(parents=True) + + (pkg_dir / "pyproject.toml").write_text(textwrap.dedent(f"""\ + [project] + name = "{name}" + version = "{version}" + requires-python = ">=3.11" + + [build-system] + requires = ["hatchling"] + build-backend = "hatchling.build" + + [tool.hatch.build.targets.wheel] + packages = ["src/{name.replace('-', '_')}"] + """)) + + (src_dir / "__init__.py").write_text(code) + return pkg_dir + + +@pytest.fixture +def workspace(tmp_path): + """Create a workspace with greeter_v1, greeter_v2, and consumer.""" + + # greeter v1: returns "hello from v1" + greeter_v1 = _write_package( + tmp_path, "greeter", "1.0.0", + 'def greet(): return "hello from v1"\n', + ) + + # greeter v2: returns "hello from v2" (same package name, different code) + greeter_v2_dir = tmp_path / "greeter_v2_src" + greeter_v2_dir.mkdir() + greeter_v2 = _write_package( + greeter_v2_dir, "greeter", "2.0.0", + 'def greet(): return "hello from v2"\n', + ) + + # consumer: depends on greeter (path dep to v1) + consumer = tmp_path / "consumer" + consumer.mkdir() + (consumer / "pyproject.toml").write_text(textwrap.dedent(f"""\ + [project] + name = "consumer" + version = "0.1.0" + requires-python = ">=3.11" + dependencies = ["greeter"] + + [tool.uv.sources] + greeter = {{ path = "{greeter_v1}" }} + """)) + + # Sync consumer to install greeter v1 + subprocess.run( + ["uv", "sync", "--project", str(consumer)], + capture_output=True, + check=True, + ) + + return greeter_v1, greeter_v2, consumer + + +@pytest.mark.asyncio +async def test_with_editable_overlays_package(workspace): + """Verify --with-editable replaces the installed package.""" + greeter_v1, greeter_v2, consumer = workspace + + script = "from greeter import greet; print(greet())" + + # Run normally → should get v1 + result_v1 = subprocess.run( + ["uv", "run", "--project", str(consumer), "python", "-c", script], + capture_output=True, + text=True, + ) + assert result_v1.returncode == 0, f"v1 failed: {result_v1.stderr}" + assert "hello from v1" in result_v1.stdout + + # Run with --with-editable pointing to v2 → should get v2 + result_v2 = subprocess.run( + ["uv", "run", "--project", str(consumer), "--with-editable", str(greeter_v2), "python", "-c", script], + capture_output=True, + text=True, + ) + assert result_v2.returncode == 0, f"v2 failed: {result_v2.stderr}" + assert "hello from v2" in result_v2.stdout + + # Run normally again → still v1 (no mutation) + result_v1_again = subprocess.run( + ["uv", "run", "--project", str(consumer), "python", "-c", script], + capture_output=True, + text=True, + ) + assert result_v1_again.returncode == 0 + assert "hello from v1" in result_v1_again.stdout diff --git a/vero/tests/test_v05_agent_context.py b/vero/tests/test_v05_agent_context.py new file mode 100644 index 00000000..733a52f9 --- /dev/null +++ b/vero/tests/test_v05_agent_context.py @@ -0,0 +1,300 @@ +from __future__ import annotations + +import json +import os +import stat +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + +from vero.candidate import Candidate +from vero.evaluation import ( + BackendProvenance, + CaseResult, + CaseStatus, + DisclosureLevel, + EvaluationArtifact, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + EvaluationSummary, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, + project_evaluation, +) +from vero.runtime.context import ( + AgentContextDirectory, + AgentDisclosureLedger, + context_digest, + make_evaluation_receipt, +) +from vero.sandbox import LocalSandbox + + +def record( + evaluation_id: str, + *, + artifact: bool = False, + trace_marker: str = "trace-marker", +) -> EvaluationRecord: + created_at = datetime(2026, 1, 1, tzinfo=UTC) + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + artifacts = ( + [ + EvaluationArtifact(path="logs/output.txt", media_type="text/plain"), + EvaluationArtifact(path="logs/leak.txt", media_type="text/plain"), + ] + if artifact + else [] + ) + return EvaluationRecord( + id=evaluation_id, + request=EvaluationRequest( + candidate=Candidate( + id=f"candidate:{evaluation_id}", + version="a" * 40, + created_at=created_at, + ), + evaluation_set=EvaluationSet(name="validation"), + ), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 0.75}, + cases=[ + CaseResult( + case_id="case/one", + status=CaseStatus.SUCCESS, + metrics={"score": 0.75}, + input={"prompt": "private case"}, + output={"answer": "candidate answer"}, + execution_trace=[{"message": trace_marker}], + evaluation_trace=[{"grader": "accepted"}], + artifacts=artifacts, + ) + ], + ), + backend_id="backend", + backend=BackendProvenance( + name="test", + version="1", + config_digest="0" * 64, + ), + objective_spec=objective, + objective=ObjectiveResult(value=0.75, feasible=True), + created_at=created_at, + completed_at=created_at + timedelta(seconds=1), + ) + + +@pytest.mark.asyncio +async def test_agent_context_splits_full_traces_and_honors_disclosure(tmp_path: Path): + session_dir = tmp_path / "session" + full = record("evaluation:full", artifact=True, trace_marker="x" * 10_000) + aggregate = record("evaluation:aggregate") + hidden = record("evaluation:hidden") + source_artifact = ( + session_dir / "evaluations" / full.id / "artifacts" / "logs" / "output.txt" + ) + source_artifact.parent.mkdir(parents=True) + source_artifact.write_text("artifact details\n", encoding="utf-8") + secret = tmp_path / "secret.txt" + secret.write_text("must not be copied\n", encoding="utf-8") + os.symlink(secret, source_artifact.parent / "leak.txt") + + project = tmp_path / "project" + project.mkdir() + directory = AgentContextDirectory( + sandbox=await LocalSandbox.create(root=tmp_path), + root=str(project / ".evals"), + session_dir=session_dir, + ) + await directory.reset() + await directory.write_header( + session_id="session", + round_number=2, + proposal_id="proposal", + parent_candidate_id="parent", + ) + await directory.write_evaluations( + [ + ( + full, + DisclosureLevel.FULL, + project_evaluation(full, DisclosureLevel.FULL), + ), + ( + aggregate, + DisclosureLevel.AGGREGATE, + project_evaluation(aggregate, DisclosureLevel.AGGREGATE), + ), + ( + hidden, + DisclosureLevel.NONE, + project_evaluation(hidden, DisclosureLevel.NONE), + ), + ] + ) + await directory.seal() + + try: + evaluations = project / ".evals" / "results" + full_root = evaluations / context_digest(full.id) + full_document = json.loads( + (full_root / "evaluation.json").read_text(encoding="utf-8") + ) + assert full_document["disclosure"] == "full" + assert "cases" not in full_document["result"]["report"] + case_path = full_document["result"]["case_files"][0]["path"] + case_document = json.loads((full_root / case_path).read_text(encoding="utf-8")) + assert case_document["execution_trace_path"] == "execution-trace.json" + trace = (full_root / Path(case_path).parent / "execution-trace.json").read_text( + encoding="utf-8" + ) + assert "x" * 10_000 in trace + assert (full_root / "artifacts" / "logs" / "output.txt").read_text() == ( + "artifact details\n" + ) + assert full_document["missing_artifacts"] == ["logs/leak.txt"] + assert not (full_root / "artifacts" / "logs" / "leak.txt").exists() + + aggregate_root = evaluations / context_digest(aggregate.id) + aggregate_document = json.loads( + (aggregate_root / "evaluation.json").read_text(encoding="utf-8") + ) + assert aggregate_document["disclosure"] == "aggregate" + assert aggregate_document["result"]["metrics"] == {"score": 0.75} + assert not (aggregate_root / "cases").exists() + + hidden_document = json.loads( + (evaluations / context_digest(hidden.id) / "evaluation.json").read_text( + encoding="utf-8" + ) + ) + assert hidden_document == { + "schema_version": 1, + "disclosure": "none", + "result": { + "evaluation_id": hidden.id, + "status": "success", + }, + } + mode = stat.S_IMODE((project / ".evals" / "manifest.json").stat().st_mode) + assert mode & 0o222 == 0 + + receipt = make_evaluation_receipt(full, DisclosureLevel.FULL) + assert isinstance(receipt.result, EvaluationSummary) + assert receipt.result_path == ( + f".evals/results/{context_digest(full.id)}/evaluation.json" + ) + assert "x" * 100 not in receipt.model_dump_json() + finally: + await directory.unseal() + + +@pytest.mark.asyncio +async def test_agent_context_reset_preserves_mounted_root( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +): + root = tmp_path / "agent-context" + root.mkdir() + (root / "stale.txt").write_text("stale\n", encoding="utf-8") + sandbox = await LocalSandbox.create(root=tmp_path) + remove = sandbox.remove + + async def reject_root_removal(path: str, *, recursive: bool = False) -> None: + if Path(path).resolve() == root.resolve(): + raise AssertionError("context mount root must not be removed") + await remove(path, recursive=recursive) + + monkeypatch.setattr(sandbox, "remove", reject_root_removal) + directory = AgentContextDirectory( + sandbox=sandbox, + root=str(root), + session_dir=tmp_path / "session", + ) + await directory.seal() + + await directory.reset() + + assert root.is_dir() + assert list(root.iterdir()) == [] + + +@pytest.mark.asyncio +async def test_disclosure_ledger_survives_restart_and_never_broadens(tmp_path: Path): + path = tmp_path / "agent-context.json" + ledger = AgentDisclosureLedger(path) + + assert await ledger.remember("evaluation", DisclosureLevel.AGGREGATE) == ( + DisclosureLevel.AGGREGATE + ) + reopened = AgentDisclosureLedger(path) + assert await reopened.remember("evaluation", DisclosureLevel.FULL) == ( + DisclosureLevel.AGGREGATE + ) + assert await reopened.remember("evaluation", DisclosureLevel.NONE) == ( + DisclosureLevel.NONE + ) + assert AgentDisclosureLedger(path).get("evaluation") == DisclosureLevel.NONE + + +@pytest.mark.asyncio +async def test_evaluation_plan_reports_partition_size(tmp_path: Path): + """The plan carries each partition's case count. + + Without it an agent sizing a subset can only guess a `--stop`, and learns the + bound from a rejected request — observed live, where an optimizer asked for + cases 20-70 of a 49-case partition and lost two evaluations to it. + """ + from vero.evaluation import EvaluationAccessPolicy, EvaluationCost, EvaluationSet + from vero.runtime.context import ContextPlanEntry + + class SizedBackend: + async def resolve_cost(self, evaluation_set) -> EvaluationCost: + return EvaluationCost(cases=49) + + class UncostableBackend: + async def resolve_cost(self, evaluation_set): + raise RuntimeError("cannot cost this set") + + directory = AgentContextDirectory( + sandbox=await LocalSandbox.create(root=tmp_path), + root=str(tmp_path / "ctx"), + session_dir=tmp_path / "session", + ) + await directory.reset() + await directory.write_evaluation_plan( + [ + ContextPlanEntry( + backend_id="harbor-development", + backend=SizedBackend(), + evaluation_set=EvaluationSet(name="bench", partition="development"), + access=EvaluationAccessPolicy(agent_can_evaluate=True), + ), + ContextPlanEntry( + backend_id="harbor-validation", + backend=UncostableBackend(), + evaluation_set=EvaluationSet(name="bench", partition="validation"), + access=EvaluationAccessPolicy(agent_can_evaluate=True), + ), + ] + ) + + plan = json.loads((tmp_path / "ctx" / "plan.json").read_text(encoding="utf-8")) + # The backend serving each partition, so `evals run --backend/--partition` + # can be taken from one row. Mismatching them is refused as an opaque + # "evaluation denied", which cost swe-atlas-qna's optimizer a round trip. + assert {row["partition"]: row["backend"] for row in plan["evaluations"]} == { + "development": "harbor-development", + "validation": "harbor-validation", + } + sizes = {row["partition"]: row["cases"] for row in plan["evaluations"]} + # A backend that cannot cost itself degrades to no count, not a broken plan. + assert sizes == {"development": 49, "validation": None} diff --git a/vero/tests/test_v05_agent_events.py b/vero/tests/test_v05_agent_events.py new file mode 100644 index 00000000..cc245d1c --- /dev/null +++ b/vero/tests/test_v05_agent_events.py @@ -0,0 +1,52 @@ +from __future__ import annotations + +import pytest + +from vero.runtime.events import EventBus, RuntimeEvent, agent_event_emitter + + +class _FakeAgent: + """Normalizes raw stream events the way VeroAgent.serialize_event does.""" + + def serialize_event(self, raw): + if raw == "noise": + return None # noise events are dropped, not published + return {"kind": "tool_call", "name": "shell", "args": raw} + + +@pytest.mark.asyncio +async def test_agent_event_emitter_publishes_normalized_events_to_bus(): + captured: list[RuntimeEvent] = [] + bus = EventBus([captured.append]) + emit = agent_event_emitter(bus, "sess-1", _FakeAgent()) + assert emit is not None + + await emit("ls -la") + await emit("noise") # serialize_event -> None, so nothing is published + + assert len(captured) == 1 + event = captured[0] + assert isinstance(event, RuntimeEvent) + assert event.session_id == "sess-1" + assert event.kind == "agent" + assert event.payload == {"kind": "tool_call", "name": "shell", "args": "ls -la"} + + +@pytest.mark.asyncio +async def test_agent_event_emitter_supports_async_sinks(): + captured: list[RuntimeEvent] = [] + + async def async_sink(event: RuntimeEvent) -> None: + captured.append(event) + + bus = EventBus([async_sink]) + emit = agent_event_emitter(bus, "s", _FakeAgent()) + assert emit is not None + await emit("pwd") + + assert [e.payload["args"] for e in captured] == ["pwd"] + + +def test_agent_event_emitter_is_none_without_serialize_event(): + bus = EventBus() + assert agent_event_emitter(bus, "s", object()) is None diff --git a/vero/tests/test_v05_agent_producer.py b/vero/tests/test_v05_agent_producer.py new file mode 100644 index 00000000..18a03189 --- /dev/null +++ b/vero/tests/test_v05_agent_producer.py @@ -0,0 +1,350 @@ +from __future__ import annotations + +import hashlib +import json +import subprocess +import sys +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from vero.agents import AgentCandidateProducer, AgentRequirements, AgentRunResult +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import ( + BackendRegistry, + CommandBackend, + CommandBackendConfig, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationDatabase, + EvaluationDefinition, + EvaluationEngine, + EvaluationPlan, + EvaluationReceipt, + EvaluationSet, + Evaluator, + MetricSelector, + ObjectiveSpec, + authorize_evaluation_plan, +) +from vero.optimization import CandidateProposal, Optimizer, SequentialStrategy +from vero.runtime import ArtifactStore +from vero.sandbox import LocalSandbox +from vero.workspace import GitWorkspace + + +def initialize_repository(path: Path) -> str: + subprocess.run( + ["git", "init", "-b", "main"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=path, + check=True, + capture_output=True, + ) + return subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=path, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +class CheckpointingCodingAgent: + def __init__(self): + self.feedback: EvaluationReceipt | None = None + self.initial_candidate_ids: set[str] = set() + self.initial_evaluation_count = 0 + self.case_resource = None + + async def run(self, *, context, prompt, max_turns, on_event=None): + assert prompt == "Make the program faster" + assert max_turns == 5 + context_root = Path(context.workspace.project_path) / ".evals" + assert not await context.workspace.is_dirty() + manifest = json.loads((context_root / "manifest.json").read_text()) + assert manifest["parent_candidate_id"] == context.parent.id + candidate_index = json.loads( + (context_root / "candidates" / "index.json").read_text() + ) + self.initial_candidate_ids = { + candidate["candidate_id"] for candidate in candidate_index["candidates"] + } + evaluation_index = json.loads( + (context_root / "results" / "index.json").read_text() + ) + self.initial_evaluation_count = len(evaluation_index["evaluations"]) + cases_index = json.loads((context_root / "tasks" / "index.json").read_text()) + resource_path = cases_index["case_resources"][0]["path"] + resource_index = json.loads( + ( + context_root / "tasks" / resource_path / "resources" / "index.json" + ).read_text() + ) + self.case_resource = json.loads( + ( + context_root + / "tasks" + / resource_path + / "resources" + / resource_index["resources"][0]["path"] + ).read_text() + ) + program = Path(context.workspace.project_path) / "program.txt" + program.write_text("fast\n", encoding="utf-8") + feedback = await context.evaluation.evaluate( + evaluation="performance", + description="Try the fast implementation" + ) + assert isinstance(feedback, EvaluationReceipt) + self.feedback = feedback + assert (Path(context.workspace.project_path) / feedback.result_path).is_file() + refreshed = json.loads( + (context_root / "results" / "index.json").read_text() + ) + assert len(refreshed["evaluations"]) == self.initial_evaluation_count + 1 + + # A later edit regresses. The evaluated checkpoint must remain selectable. + program.write_text("slow\n", encoding="utf-8") + return AgentRunResult( + description="Finish agent attempt", + state={"turn": 2}, + trace=[{"objective": feedback.result.objective.value}], + metadata={"provider": "test"}, + ) + + +def test_host_native_agent_rejects_isolated_workspace(): + class HostNativeAgent: + requirements = AgentRequirements(host_visible_workspace=True) + + class IsolatedSandbox: + def host_path(self, path): + return None + + producer = AgentCandidateProducer(HostNativeAgent()) + + with pytest.raises(ValueError, match="requires a host-visible workspace"): + producer.validate_workspace( + SimpleNamespace( + project_path="/workspace/target", + sandbox=IsolatedSandbox(), + ) + ) + + +@pytest.mark.asyncio +async def test_failed_agent_run_persists_state_and_trace(tmp_path: Path): + class FailingAgent: + async def run(self, **kwargs): + raise RuntimeError("turn limit reached") + + def serialize_state(self): + return {"turn": 5} + + def serialize_trace(self): + return [{"tool": "file_read"}] + + artifacts = ArtifactStore(tmp_path / "artifacts") + producer = AgentCandidateProducer(FailingAgent(), artifacts=artifacts) + proposal = CandidateProposal(id="proposal", producer_id="default") + baseline = object() + context = SimpleNamespace( + session_id="session", + candidates={}, + baseline=baseline, + ) + + with pytest.raises(RuntimeError, match="turn limit reached"): + await producer.produce( + proposal=proposal, + context=context, + workspace=SimpleNamespace(), + evaluation=SimpleNamespace(), + ) + + digest = hashlib.sha256(proposal.id.encode()).hexdigest()[:16] + assert artifacts.read_json(f"agents/{digest}/state.json") == {"turn": 5} + assert artifacts.read_json(f"agents/{digest}/trace.json") == [{"tool": "file_read"}] + assert artifacts.read_json(f"agents/{digest}/failure.json") == { + "type": "RuntimeError", + "message": "turn limit reached", + } + assert artifacts.read_json(producer._producer_state_path("default")) == {"turn": 5} + + +@pytest.mark.asyncio +async def test_agent_checkpoint_is_a_selectable_candidate(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("slow\n", encoding="utf-8") + baseline_version = initialize_repository(target) + + harness = tmp_path / "harness" + harness.mkdir() + harness_script = harness / "evaluate.py" + harness_script.write_text( + """ +import json +import sys +from pathlib import Path + +workspace, report_path = map(Path, sys.argv[1:]) +program = (workspace / "program.txt").read_text().strip() +latency = 1.0 if program == "fast" else 10.0 +report_path.write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"latency_ms": latency}, +})) +""", + encoding="utf-8", + ) + cases = harness / "cases.json" + cases.write_text( + json.dumps([{"id": "case-1", "size": 128}]) + "\n", + encoding="utf-8", + ) + + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(target)) + session_dir = tmp_path / "sessions" / "agent" + candidate_repository = await GitCandidateRepository.create( + session_dir / "candidates", workspace=workspace + ) + database = EvaluationDatabase(id="agent") + + plan = EvaluationPlan( + evaluations=[ + EvaluationDefinition( + evaluation_set=EvaluationSet(name="performance"), + access=EvaluationAccessPolicy( + disclosure=DisclosureLevel.AGGREGATE, + expose_case_resources=True, + ), + ), + EvaluationDefinition( + evaluation_set=EvaluationSet(name="test", partition="test"), + access=EvaluationAccessPolicy( + agent_can_evaluate=False, + agent_visible=False, + disclosure=DisclosureLevel.NONE, + ), + ), + ], + selection_evaluation="performance", + final_evaluation="test", + ) + + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=session_dir, + ), + backends=BackendRegistry( + { + "command": CommandBackend( + CommandBackendConfig( + harness_root=str(harness), + command=[ + sys.executable, + str(harness_script), + "{workspace}", + "{report}", + ], + staged_inputs={"cases": str(cases)}, + agent_context_inputs={"performance": ["cases"]}, + ) + ) + } + ), + database=database, + database_path=session_dir / "database.json", + authorization_resolver=authorize_evaluation_plan(plan), + ) + agent = CheckpointingCodingAgent() + artifacts = ArtifactStore(session_dir / "artifacts") + optimizer = Optimizer( + workspace=workspace, + candidate_repository=candidate_repository, + engine=engine, + backend_id="command", + evaluation_plan=plan, + objective=ObjectiveSpec( + selector=MetricSelector(metric="latency_ms"), + direction="minimize", + ), + strategy=SequentialStrategy(instruction="Make the program faster"), + producers={ + "default": AgentCandidateProducer( + agent, + max_turns=5, + artifacts=artifacts, + ) + }, + max_proposals=1, + ) + + result = await optimizer.run() + + assert agent.feedback is not None + assert agent.feedback.result.objective.value == 1.0 + assert agent.feedback.result_path.startswith(".evals/results/") + assert agent.initial_candidate_ids == {baseline_version} + assert agent.initial_evaluation_count == 1 + assert agent.case_resource == [{"id": "case-1", "size": 128}] + assert len(result.evaluations) == 5 + assert len(result.candidates) == 3 + assert result.best.objective.value == 1.0 + assert result.best.request.candidate.id.endswith(":trial:1") + assert result.final is not None + assert result.final.request.evaluation_set.name == "test" + assert result.best.request.candidate.parent_id == baseline_version + final = next( + candidate + for candidate in result.candidates + if candidate.id not in {baseline_version, result.best.request.candidate.id} + ) + assert final.parent_id == result.best.request.candidate.id + assert (target / "program.txt").read_text(encoding="utf-8") == "slow\n" + assert len(database.evaluations) == 5 + + agent_artifacts = list((session_dir / "artifacts" / "agents").iterdir()) + proposal_artifacts = [path for path in agent_artifacts if path.name != "producers"] + assert len(proposal_artifacts) == 1 + assert (proposal_artifacts[0] / "state.json").exists() + assert (proposal_artifacts[0] / "trace.json").exists() + + class ResumedAgent: + def __init__(self): + self.state = None + + def deserialize_state(self, state): + self.state = state + + resumed_agent = ResumedAgent() + resumed_producer = AgentCandidateProducer(resumed_agent) + resumed_producer.bind_artifacts(artifacts) + assert resumed_agent.state == {"turn": 2} diff --git a/vero/tests/test_v05_c_example.py b/vero/tests/test_v05_c_example.py new file mode 100644 index 00000000..ad2cf8fe --- /dev/null +++ b/vero/tests/test_v05_c_example.py @@ -0,0 +1,90 @@ +from __future__ import annotations + +import json +import shutil +import subprocess +from pathlib import Path + +import pytest +from click.testing import CliRunner + +from vero.cli import main + + +def initialize_repository(path: Path) -> str: + subprocess.run( + ["git", "init", "-b", "main"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=path, + check=True, + capture_output=True, + ) + return subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=path, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +@pytest.mark.skipif(shutil.which("cc") is None, reason="a C compiler is required") +def test_declarative_c_example_optimizes_a_non_python_target(tmp_path: Path): + source = Path(__file__).parents[1] / "examples" / "c-matmul" + example = tmp_path / "c-matmul" + shutil.copytree(source, example, ignore=shutil.ignore_patterns("__pycache__")) + baseline_source = (example / "target" / "matmul.c").read_text(encoding="utf-8") + baseline_version = initialize_repository(example / "target") + config = example / "vero.toml" + runner = CliRunner() + + evaluated = runner.invoke(main, ["evaluate", "--config", str(config)]) + assert evaluated.exit_code == 0, evaluated.output + assert f"Baseline: {baseline_version}" in evaluated.output + + optimized = runner.invoke(main, ["run", "--config", str(config)]) + assert optimized.exit_code == 0, optimized.output + assert "Best: no feasible candidate" not in optimized.output + assert f"Best: {baseline_version}" not in optimized.output + assert (example / "target" / "matmul.c").read_text( + encoding="utf-8" + ) == baseline_source + + database = json.loads( + (example / ".vero" / "session" / "database.json").read_text(encoding="utf-8") + ) + records = list(database["evaluations"].values()) + assert len(records) == 2 + assert all(record["objective"]["feasible"] for record in records) + assert min(record["objective"]["value"] for record in records) < max( + record["objective"]["value"] for record in records + ) + + worktrees = subprocess.run( + ["git", "worktree", "list", "--porcelain"], + cwd=example / "target", + check=True, + capture_output=True, + text=True, + ).stdout + assert worktrees.count("worktree ") == 1 diff --git a/vero/tests/test_v05_candidate_repository.py b/vero/tests/test_v05_candidate_repository.py new file mode 100644 index 00000000..ca244ad0 --- /dev/null +++ b/vero/tests/test_v05_candidate_repository.py @@ -0,0 +1,298 @@ +from __future__ import annotations + +import json +import os +import shutil +import stat +import subprocess +from pathlib import Path + +import pytest + +from vero.candidate import Candidate +from vero.candidate_repository import CandidateRepositoryError, GitCandidateRepository +from vero.sandbox import LocalSandbox, SandboxCapabilities +from vero.workspace import GitWorkspace + + +class OpaqueLocalSandbox(LocalSandbox): + """Local execution with paths hidden to exercise remote bundle transfer.""" + + @property + def capabilities(self) -> SandboxCapabilities: + return SandboxCapabilities(host_paths=False) + + def host_path(self, path: str) -> None: + return None + + +def git(path: Path, *arguments: str) -> str: + result = subprocess.run( + ["git", *arguments], + cwd=path, + check=True, + capture_output=True, + text=True, + ) + return result.stdout.strip() + + +def initialize(path: Path) -> str: + git(path, "init", "-b", "main") + git(path, "add", "--all") + git( + path, + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ) + return git(path, "rev-parse", "HEAD") + + +@pytest.mark.asyncio +async def test_git_repository_captures_remote_candidates_and_recreates_them( + tmp_path: Path, +): + source = tmp_path / "source" + source.mkdir() + program = source / "program.sh" + program.write_text("#!/bin/sh\necho baseline\n", encoding="utf-8") + program.chmod(0o755) + os.symlink("program.sh", source / "program-link") + baseline_version = initialize(source) + + source_sandbox = OpaqueLocalSandbox(tmp_path) + workspace = await GitWorkspace.from_path(source_sandbox, str(source)) + repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", + workspace=workspace, + ) + baseline = Candidate.from_version(baseline_version) + await repository.capture(baseline, workspace) + + program.write_text("#!/bin/sh\necho candidate\n", encoding="utf-8") + candidate_version = await workspace.save("candidate") + candidate = Candidate.from_version( + candidate_version, + candidate_id="candidate-1", + parent_id=baseline.id, + description="Improve the program", + metadata={"producer_id": "test"}, + ) + await repository.capture(candidate, workspace) + assert await repository.capture(candidate, workspace) == candidate + + shutil.rmtree(source) + reopened = await GitCandidateRepository.open(repository.root) + assert reopened.list() == (baseline, candidate) + + destination_sandbox = OpaqueLocalSandbox(tmp_path) + checkout_root: Path | None = None + async with reopened.checkout( + candidate, + sandbox=destination_sandbox, + name="inspect", + ) as checkout: + checkout_root = Path(checkout.root) + assert await checkout.current_version() == candidate.version + assert (Path(checkout.project_path) / "program.sh").read_text() == ( + "#!/bin/sh\necho candidate\n" + ) + mode = (Path(checkout.project_path) / "program.sh").stat().st_mode + assert mode & stat.S_IXUSR + assert (Path(checkout.project_path) / "program-link").is_symlink() + assert not await checkout.is_dirty() + assert checkout_root is not None + assert not checkout_root.exists() + + +@pytest.mark.asyncio +async def test_git_repository_rejects_conflicting_candidate_identity(tmp_path: Path): + source = tmp_path / "source" + source.mkdir() + (source / "program.txt").write_text("baseline\n", encoding="utf-8") + baseline_version = initialize(source) + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(source)) + repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", + workspace=workspace, + ) + candidate = Candidate.from_version(baseline_version, candidate_id="candidate") + await repository.capture(candidate, workspace) + + conflicting = candidate.model_copy(update={"description": "different"}) + with pytest.raises(ValueError, match="already stored with different data"): + await repository.capture(conflicting, workspace) + + +@pytest.mark.asyncio +async def test_git_repository_preserves_nested_project_path(tmp_path: Path): + source = tmp_path / "source" + project = source / "packages" / "target" + project.mkdir(parents=True) + (project / "program.txt").write_text("baseline\n", encoding="utf-8") + version = initialize(source) + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(project)) + repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", + workspace=workspace, + ) + candidate = Candidate.from_version(version) + await repository.capture(candidate, workspace) + + async with repository.checkout(candidate, sandbox=sandbox) as checkout: + assert Path(checkout.project_path).relative_to(checkout.root) == Path( + "packages/target" + ) + assert (Path(checkout.project_path) / "program.txt").read_text() == ( + "baseline\n" + ) + + +@pytest.mark.asyncio +async def test_git_repository_fails_loudly_when_a_record_loses_its_ref( + tmp_path: Path, +): + source = tmp_path / "source" + source.mkdir() + (source / "program.txt").write_text("baseline\n", encoding="utf-8") + version = initialize(source) + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(source)) + repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", + workspace=workspace, + ) + await repository.capture(Candidate.from_version(version), workspace) + retained_ref = git( + repository.repository_path, + "for-each-ref", + "--format=%(refname)", + "refs/vero/candidates", + ) + git(repository.repository_path, "update-ref", "-d", retained_ref) + + with pytest.raises(CandidateRepositoryError, match="missing Git object"): + await GitCandidateRepository.open(repository.root) + + +@pytest.mark.asyncio +async def test_git_repository_rejects_candidates_that_track_agent_context( + tmp_path: Path, +): + source = tmp_path / "source" + source.mkdir() + (source / "program.txt").write_text("baseline\n", encoding="utf-8") + baseline_version = initialize(source) + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(source)) + repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", + workspace=workspace, + ) + await repository.capture(Candidate.from_version(baseline_version), workspace) + + context = source / ".evals" + context.mkdir() + (context / "private.json").write_text('{"secret": true}\n', encoding="utf-8") + git(source, "add", "-f", ".evals/private.json") + git( + source, + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "try to capture context", + ) + candidate = Candidate.from_version( + git(source, "rev-parse", "HEAD"), + candidate_id="candidate-with-context", + ) + + with pytest.raises(CandidateRepositoryError, match="reserved agent context"): + await repository.capture(candidate, workspace) + assert repository.get(candidate.id) is None + + +@pytest.mark.asyncio +async def test_git_repository_materializes_visible_history_in_opaque_workspace( + tmp_path: Path, +): + source = tmp_path / "source" + source.mkdir() + program = source / "program.txt" + program.write_text("baseline\n", encoding="utf-8") + baseline_version = initialize(source) + sandbox = OpaqueLocalSandbox(tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(source)) + repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", + workspace=workspace, + ) + baseline = Candidate.from_version(baseline_version) + await repository.capture(baseline, workspace) + + program.write_text("first\n", encoding="utf-8") + first_version = await workspace.save("first candidate") + first = Candidate.from_version( + first_version, + candidate_id="first", + parent_id=baseline.id, + ) + await repository.capture(first, workspace) + + git(source, "checkout", "--detach", baseline.version) + program.write_text("sibling\n", encoding="utf-8") + sibling_version = await workspace.save("sibling candidate") + sibling = Candidate.from_version( + sibling_version, + candidate_id="sibling", + parent_id=baseline.id, + ) + await repository.capture(sibling, workspace) + + async with repository.checkout( + baseline, + sandbox=OpaqueLocalSandbox(tmp_path), + name="history", + ) as checkout: + destination = f"{checkout.project_path}/.evals/candidates" + await repository.materialize_agent_history( + (baseline, first, sibling), + workspace=checkout, + destination=destination, + ) + index = json.loads( + (Path(destination) / "index.json").read_text(encoding="utf-8") + )["candidates"] + assert {item["candidate_id"] for item in index} == { + baseline.id, + first.id, + sibling.id, + } + assert all( + item["native_ref"].startswith("refs/vero/context/") for item in index + ) + for item in index: + assert ( + git( + Path(checkout.root), + "rev-parse", + "--verify", + f"{item['native_ref']}^{{commit}}", + ) + == item["version"] + ) + if item["candidate_id"] != baseline.id: + patch = Path(destination) / item["parent_patch_path"] + assert patch.is_file() + assert "program.txt" in patch.read_text(encoding="utf-8") + assert not await checkout.is_dirty() diff --git a/vero/tests/test_v05_circle_packing_example.py b/vero/tests/test_v05_circle_packing_example.py new file mode 100644 index 00000000..53f2be17 --- /dev/null +++ b/vero/tests/test_v05_circle_packing_example.py @@ -0,0 +1,180 @@ +from __future__ import annotations + +import json +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest +from click.testing import CliRunner + +from vero.cli import main + + +def test_circle_packing_harness_applies_request_seed(tmp_path: Path): + example = Path(__file__).parents[1] / "examples" / "circle-packing" + workspace = tmp_path / "candidate" + workspace.mkdir() + (workspace / "packing.py").write_text( + """\ +import random + +def run_packing(): + centers = [] + for row in range(6): + for column in range(5): + if len(centers) == 26: + break + centers.append([0.1 + 0.2 * column, 0.1 + 0.16 * row]) + radii = [random.uniform(0.01, 0.015) for _ in centers] + return centers, radii, sum(radii) +""", + encoding="utf-8", + ) + request = tmp_path / "request.json" + request.write_text( + json.dumps({"schema_version": 1, "request": {"seed": 12345}}), + encoding="utf-8", + ) + + scores = [] + layouts = [] + for attempt in range(2): + report = tmp_path / f"report-{attempt}.json" + artifacts = tmp_path / f"artifacts-{attempt}" + completed = subprocess.run( + [ + sys.executable, + str(example / "harness" / "evaluate.py"), + "--workspace", + str(workspace), + "--request", + str(request), + "--report", + str(report), + "--artifacts", + str(artifacts), + ], + check=True, + capture_output=True, + text=True, + ) + assert completed.stderr == "" + payload = json.loads(report.read_text(encoding="utf-8")) + assert payload["metrics"]["valid"] == 1.0 + scores.append(payload["metrics"]["sum_radii"]) + layouts.append( + (artifacts / "circle-packing" / "layout.json").read_text( + encoding="utf-8" + ) + ) + + assert scores[0] == scores[1] + assert layouts[0] == layouts[1] + + +def initialize_repository(path: Path) -> str: + subprocess.run( + ["git", "init", "-b", "main"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=path, + check=True, + capture_output=True, + ) + return subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=path, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +@pytest.mark.skipif(shutil.which("uv") is None, reason="uv is required") +def test_circle_packing_example_evaluates_and_preserves_artifacts( + tmp_path: Path, monkeypatch +): + source = Path(__file__).parents[1] / "examples" / "circle-packing" + example = tmp_path / "circle-packing" + shutil.copytree( + source, + example, + ignore=shutil.ignore_patterns(".git", ".vero", ".evals", ".venv", "__pycache__"), + ) + baseline_version = initialize_repository(example / "target") + monkeypatch.setenv("UV_CACHE_DIR", str(tmp_path / "uv-cache")) + + result = CliRunner().invoke( + main, + ["evaluate", "--config", str(example / "vero.toml")], + ) + + assert result.exit_code == 0, result.output + assert f"Baseline: {baseline_version}" in result.output + database = json.loads( + (example / ".vero" / "session" / "database.json").read_text( + encoding="utf-8" + ) + ) + records = list(database["evaluations"].values()) + assert len(records) == 2 + assert {record["request"]["evaluation_set"]["name"] for record in records} == { + "development", + "final", + } + assert all(record["objective"]["feasible"] for record in records) + assert all(record["request"]["seed"] == 1337 for record in records) + assert all( + record["report"]["metrics"]["sum_radii"] + == pytest.approx(0.9597642169962064) + for record in records + ) + + for record in records: + artifact_root = ( + example + / ".vero" + / "session" + / "evaluations" + / record["id"] + / "artifacts" + / "circle-packing" + ) + assert (artifact_root / "layout.json").is_file() + assert (artifact_root / "layout.svg").read_text( + encoding="utf-8" + ).startswith(" AgentContext: + baseline = Candidate(id="baseline", version="baseline-version") + proposal = CandidateProposal( + id="proposal", + parent_id=baseline.id, + instruction="Optimize matrix multiplication", + ) + return AgentContext( + session_id="session-1", + workspace=StubWorkspace(tmp_path), + proposal=proposal, + parent=baseline, + evaluation=gateway, + ) + + +@pytest.mark.asyncio +async def test_claude_agent_implements_canonical_coding_agent_contract( + tmp_path: Path, +): + gateway = StubEvaluationGateway() + evaluation_tools = EvaluationTools() + agent = ClaudeCodeAgent(tool_sets=[evaluation_tools]) + result_message = ResultMessage( + subtype="success", + duration_ms=10, + duration_api_ms=8, + is_error=False, + num_turns=1, + session_id="claude-session", + usage={"input_tokens": 10, "output_tokens": 3}, + result="done", + ) + client = FakeClaudeClient(result_message) + agent._create_client = lambda max_turns=None: client + events = [] + + async def capture(event): + events.append(event) + + context = agent_context(tmp_path, gateway) + result = await agent.run( + context=context, + prompt="Try a tiled kernel", + max_turns=7, + on_event=capture, + ) + + assert isinstance(agent, CodingAgent) + assert client.queries == ["Try a tiled kernel"] + assert events == [result_message] + assert evaluation_tools.evaluation is gateway + assert agent._context is context + assert agent.state == {"session_id": "claude-session"} + assert result.state == {"session_id": "claude-session"} + assert result.metadata["usage"]["input_tokens"] == 10 + assert context.project_path == tmp_path + assert context.instructions.startswith("Optimize matrix multiplication\n\n") + assert "evaluation context has been placed in `.evals/`" in context.instructions + assert context.base_version == "baseline-version" diff --git a/vero/tests/test_v05_cli.py b/vero/tests/test_v05_cli.py new file mode 100644 index 00000000..a5f61a71 --- /dev/null +++ b/vero/tests/test_v05_cli.py @@ -0,0 +1,474 @@ +from __future__ import annotations + +import json +import shlex +import subprocess +import sys +from pathlib import Path + +from click.testing import CliRunner + +import vero +from vero.candidate import Candidate +from vero.cli import main + + +def test_root_package_exposes_only_canonical_program_api(): + assert vero.Candidate is Candidate + assert not hasattr(vero, "Experiment") + + +def test_canonical_root_and_tools_do_not_import_legacy_core(): + result = subprocess.run( + [ + sys.executable, + "-c", + ( + "import sys, vero, vero.tools; " + "assert not any(name == 'vero.core' or name.startswith('vero.core.') " + "for name in sys.modules)" + ), + ], + check=False, + capture_output=True, + text=True, + ) + + assert result.returncode == 0, result.stderr + + +def initialize_repository(path: Path) -> None: + subprocess.run( + ["git", "init", "-b", "main"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=path, + check=True, + capture_output=True, + ) + + +def test_cli_optimizes_non_python_program_and_inspects_session(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("slow\n", encoding="utf-8") + initialize_repository(target) + + harness = tmp_path / "harness" + harness.mkdir() + harness_script = harness / "evaluate.py" + harness_script.write_text( + """ +import json +import sys +from pathlib import Path +workspace, report = map(Path, sys.argv[1:]) +latency = 1.0 if (workspace / "program.txt").read_text().strip() == "fast" else 9.0 +report.write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"latency_ms": latency, "correct": 1.0}, +})) +""", + encoding="utf-8", + ) + producer = tmp_path / "producer" + producer.mkdir() + producer_script = producer / "improve.py" + producer_script.write_text( + """ +import sys +from pathlib import Path +Path(sys.argv[1], "program.txt").write_text("fast\\n") +""", + encoding="utf-8", + ) + session_dir = tmp_path / "sessions" / "nested" / "cli-run" + runner = CliRunner() + + result = runner.invoke( + main, + [ + "optimize", + str(target), + "--harness-root", + str(harness), + "--evaluate", + shlex.join( + [ + sys.executable, + str(harness_script), + "{workspace}", + "{report}", + ] + ), + "--producer-root", + str(producer), + "--produce", + shlex.join([sys.executable, str(producer_script), "{workspace}"]), + "--metric", + "latency_ms", + "--direction", + "minimize", + "--constraint", + "correct", + "==", + "1", + "--evaluation-set", + "performance", + "--parameter", + "threshold=0.5", + "--session-dir", + str(session_dir), + "--session-id", + "cli-session", + ], + ) + + assert result.exit_code == 0, result.output + assert "Baseline:" in result.output + assert "(9.0)" in result.output + assert "Best:" in result.output + assert "(1.0)" in result.output + assert (target / "program.txt").read_text(encoding="utf-8") == "slow\n" + + inspect_result = runner.invoke(main, ["session", "inspect", str(session_dir)]) + assert inspect_result.exit_code == 0, inspect_result.output + inspection = json.loads(inspect_result.output) + assert inspection["manifest"]["id"] == "cli-session" + assert inspection["manifest"]["status"] == "completed" + assert len(inspection["candidates"]) == 2 + assert len(inspection["evaluations"]) == 2 + + list_result = runner.invoke( + main, + ["session", "list", "--root", str(tmp_path / "sessions")], + ) + assert list_result.exit_code == 0, list_result.output + assert "cli-session\tcompleted" in list_result.output + assert "nested/cli-run" in list_result.output + + archive = tmp_path / "cli-export.tar.gz" + export_result = runner.invoke( + main, + ["session", "export", str(session_dir), "--output", str(archive)], + ) + assert export_result.exit_code == 0, export_result.output + assert archive.is_file() + + fork_dir = tmp_path / "sessions" / "forked" + fork_result = runner.invoke( + main, + [ + "session", + "fork", + str(session_dir), + str(fork_dir), + "--max-proposals", + "2", + "--reset-budgets", + ], + ) + assert fork_result.exit_code == 0, fork_result.output + forked = json.loads((fork_dir / "manifest.json").read_text()) + assert forked["id"] == "forked" + assert forked["run"]["max_proposals"] == 2 + assert json.loads((fork_dir / "database.json").read_text())["id"] == "forked" + + clear_result = runner.invoke( + main, + ["session", "clear", str(fork_dir), "--yes"], + ) + assert clear_result.exit_code == 0, clear_result.output + assert not fork_dir.exists() + + evaluate_only_dir = tmp_path / "sessions" / "evaluate-only" + evaluate_only = runner.invoke( + main, + [ + "optimize", + str(target), + "--harness-root", + str(harness), + "--evaluate", + shlex.join( + [ + sys.executable, + str(harness_script), + "{workspace}", + "{report}", + ] + ), + "--metric", + "latency_ms", + "--direction", + "minimize", + "--max-proposals", + "0", + "--session-dir", + str(evaluate_only_dir), + ], + ) + assert evaluate_only.exit_code == 0, evaluate_only.output + assert "(9.0)" in evaluate_only.output + + +def test_cli_init_and_check_config(tmp_path: Path): + runner = CliRunner() + result = runner.invoke(main, ["init", str(tmp_path)]) + assert result.exit_code == 0, result.output + + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + initialize_repository(target) + (tmp_path / "harness").mkdir() + + checked = runner.invoke( + main, + ["check", "--config", str(tmp_path / "vero.toml")], + ) + assert checked.exit_code == 0, checked.output + assert "selection='validation'" in checked.output + + +def test_cli_requires_exactly_one_producer(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + harness = tmp_path / "harness" + harness.mkdir() + runner = CliRunner() + + result = runner.invoke( + main, + [ + "optimize", + str(target), + "--harness-root", + str(harness), + "--evaluate", + "evaluate {workspace} {report}", + "--metric", + "score", + "--direction", + "maximize", + ], + ) + + assert result.exit_code == 2 + assert "exactly one of --produce or --agent" in result.output + + +def test_cli_rejects_options_that_do_not_apply(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + harness = tmp_path / "harness" + harness.mkdir() + base = [ + "optimize", + str(target), + "--harness-root", + str(harness), + "--evaluate", + "evaluate {workspace} {report}", + "--metric", + "score", + "--direction", + "maximize", + "--max-proposals", + "0", + ] + runner = CliRunner() + + max_turns = runner.invoke(main, [*base, "--max-turns", "10"]) + producer_timeout = runner.invoke(main, [*base, "--producer-timeout", "10"]) + wandb = runner.invoke(main, [*base, "--wandb-mode", "offline"]) + + assert max_turns.exit_code == 2 + assert "--max-turns is only valid with --agent" in max_turns.output + assert producer_timeout.exit_code == 2 + assert "are only valid with --produce" in producer_timeout.output + assert wandb.exit_code == 2 + assert "require --wandb-project" in wandb.output + + +def test_opencode_non_openai_provider_gets_a_gateway_base_url(tmp_path): + """opencode reaches non-openai providers only if we supply the baseURL. + + Harbor's adapter injects one for `openai/...` alone, so `anthropic/...` + otherwise calls api.anthropic.com and dies on 401 (the optimizer holds only a + scoped token). Forcing `openai/` instead puts a Claude model on the Responses + API, whose litellm translation opencode cannot parse. This keeps Anthropic on + Messages, through the gateway. + """ + from vero.harbor.cli import _opencode_gateway_args + + task = tmp_path / "task" + (task / "environment" / "gateway").mkdir(parents=True) + (task / "environment" / "gateway" / "launch.json").write_text( + json.dumps({"producer_base_url": "http://inference-gateway:8001/scopes/p/o/v1"}), + encoding="utf-8", + ) + + args = _opencode_gateway_args("opencode", "anthropic/claude-sonnet-5", task) + assert args[0] == "--ak" + key, _, value = args[1].partition("=") + assert key == "opencode_config" + payload = json.loads(value) + assert payload["provider"] == { + "anthropic": { + "options": {"baseURL": "http://inference-gateway:8001/scopes/p/o/v1"} + } + } + # The same config carries the step limit; see the step-limit test below. + assert payload["agent"]["build"]["steps"] > 100 + + # openai is the one provider the adapter already handles for baseURL; don't + # fight it -- but opencode still needs its step limit raised. + openai = _opencode_gateway_args("opencode", "openai/gpt-5.4", task) + assert "provider" not in json.loads(openai[1].removeprefix("opencode_config=")) + # Other agents are untouched -- claude-code controls turns via --max-turns. + assert _opencode_gateway_args("claude-code", "anthropic/claude-sonnet-5", task) == [] + # For opencode the step limit is unconditional: it does not depend on the + # model spelling or on a gateway being present, because a truncated search is + # a problem either way. Only the baseURL injection is conditional. + bare = _opencode_gateway_args("opencode", "claude-sonnet-5", task) + assert json.loads(bare[1].removeprefix("opencode_config="))["agent"]["build"][ + "steps" + ] > 100 + no_gateway = _opencode_gateway_args("opencode", "anthropic/x", tmp_path / "none") + payload = json.loads(no_gateway[1].removeprefix("opencode_config=")) + assert payload["agent"]["build"]["steps"] > 100 + assert "provider" not in payload + + +def test_outer_modal_trial_gets_a_named_app(): + """The outer trial must not land in Modal's anonymous default app. + + Inner eval sandboxes are already grouped by app_name via each build's + extra_harbor_args, but the outer trial had none, so it went to __harbor__ + alongside every other workspace container -- ~1800 of them -- which makes it + unfindable in the UI and turns "copy the session out before killing this run" + into a search problem. + """ + from vero.harbor.cli import _outer_app_name_args + + assert _outer_app_name_args("modal", "vero/optimize-gaia-baseline", ()) == [ + "--ek", + "app_name=vero-optimize-gaia-baseline", + ] + # Docker outer trials are found via `docker ps`; no app applies. + assert _outer_app_name_args("docker", "vero/optimize-gaia-baseline", ()) == [] + # An explicit choice wins. + assert _outer_app_name_args( + "modal", "vero/optimize-gaia-baseline", ("--ek", "app_name=mine") + ) == [] + # A name that is entirely punctuation still yields a usable app. + assert _outer_app_name_args("modal", "///", ()) == ["--ek", "app_name=vero"] + + +def test_opencode_gets_a_step_limit_that_does_not_truncate_the_search(tmp_path): + """opencode caps agentic iterations at 100 and then forces a text-only reply. + + That is inside a real run: gaia's optimizer used exactly 100 and stopped + without ever calling `evals submit`, and because the cap produces a fluent + final message the truncation is invisible unless you count the steps. + claude-code takes harbor's --max-turns instead, so leaving opencode at its + default makes the two harnesses incomparable on the one axis the grid varies. + """ + from vero.harbor.cli import OPENCODE_STEP_LIMIT, _opencode_gateway_args + + args = _opencode_gateway_args("opencode", "anthropic/claude-sonnet-5", tmp_path) + assert args[0] == "--ak" + payload = json.loads(args[1].removeprefix("opencode_config=")) + assert payload["agent"]["build"]["steps"] == OPENCODE_STEP_LIMIT + assert OPENCODE_STEP_LIMIT > 100 + + # Still set for the openai provider, which needs no baseURL injection. + openai = json.loads( + _opencode_gateway_args("opencode", "openai/gpt-5.4", tmp_path)[1] + .removeprefix("opencode_config=") + ) + assert openai["agent"]["build"]["steps"] == OPENCODE_STEP_LIMIT + assert "provider" not in openai + + # claude-code is untouched; it has its own turn control. + assert _opencode_gateway_args("claude-code", "claude-sonnet-5", tmp_path) == [] + + +def test_litellm_harnesses_get_the_gateway_url_under_the_name_they_read(tmp_path): + """litellm reads _API_BASE; the provider SDKs read _BASE_URL. + + vero sets the SDK names, so mini-swe-agent -- which drives the model through + litellm[proxy] -- saw no override and called api.anthropic.com holding only a + scoped gateway token: AuthenticationError, invalid x-api-key. It fails closed, + so nothing leaked, but the harness could not run at all. + """ + from vero.harbor.cli import _litellm_base_url_args + + task = tmp_path / "task" + (task / "environment" / "gateway").mkdir(parents=True) + (task / "environment" / "gateway" / "launch.json").write_text( + json.dumps({"producer_base_url": "http://inference-gateway:8001/scopes/p/o/v1"}), + encoding="utf-8", + ) + + args = _litellm_base_url_args("mini-swe-agent", task) + assert args[::2] == ["--ae", "--ae"] + values = args[1::2] + # openai: litellm appends /chat/completions, so the /v1 base passes through. + assert "OPENAI_API_BASE=http://inference-gateway:8001/scopes/p/o/v1" in values + # anthropic: litellm appends /v1/messages unless the base already ends in it. + # Passing the /v1 base produced /v1/v1/messages and a 403 from upstream. + assert ( + "ANTHROPIC_API_BASE=http://inference-gateway:8001/scopes/p/o/v1/messages" + in values + ) + assert not any("/v1/v1" in value for value in values) + + # Same wire path whichever form the compiled gateway hands us. + for producer, expected in ( + ("http://gw:8001/scopes/p/o", "http://gw:8001/scopes/p/o/v1/messages"), + ("http://gw:8001/scopes/p/o/v1/", "http://gw:8001/scopes/p/o/v1/messages"), + ( + "http://gw:8001/scopes/p/o/v1/messages", + "http://gw:8001/scopes/p/o/v1/messages", + ), + ): + (task / "environment" / "gateway" / "launch.json").write_text( + json.dumps({"producer_base_url": producer}), encoding="utf-8" + ) + assert ( + f"ANTHROPIC_API_BASE={expected}" + in _litellm_base_url_args("mini-swe-agent", task)[1::2] + ) + + (task / "environment" / "gateway" / "launch.json").write_text( + json.dumps({"producer_base_url": "http://inference-gateway:8001/scopes/p/o/v1"}), + encoding="utf-8", + ) + + # Harnesses that use provider SDKs already get _BASE_URL and need nothing. + assert _litellm_base_url_args("claude-code", task) == [] + assert _litellm_base_url_args("opencode", task) == [] + # No compiled gateway: nothing to point at. + assert _litellm_base_url_args("mini-swe-agent", tmp_path / "none") == [] diff --git a/vero/tests/test_v05_command_backend.py b/vero/tests/test_v05_command_backend.py new file mode 100644 index 00000000..7430a507 --- /dev/null +++ b/vero/tests/test_v05_command_backend.py @@ -0,0 +1,331 @@ +import json +import os +import sys +from datetime import UTC, datetime +from pathlib import Path +from types import SimpleNamespace + +import pytest +from pydantic import ValidationError + +from vero.candidate import Candidate +from vero.evaluation import ( + AllCases, + CaseCheckpointStore, + CaseIds, + CaseRange, + CommandBackend, + CommandBackendConfig, + EvaluationContext, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, +) +from vero.sandbox import LocalSandbox + + +def write_harness(root: Path, body: str) -> Path: + root.mkdir(parents=True, exist_ok=True) + script = root / "harness.py" + script.write_text(body) + return script + + +async def context(tmp_path: Path, workspace_path: Path) -> EvaluationContext: + workspace_path.mkdir(parents=True, exist_ok=True) + result_dir = tmp_path / "result" + artifact_dir = result_dir / "artifacts" + artifact_dir.mkdir(parents=True) + workspace = SimpleNamespace( + project_path=str(workspace_path), + sandbox=await LocalSandbox.create(root=tmp_path), + ) + return EvaluationContext( + workspace=workspace, + session_id="session", + evaluation_id="evaluation", + result_dir=result_dir, + artifact_dir=artifact_dir, + case_store=CaseCheckpointStore(result_dir / "cases"), + ) + + +def request() -> EvaluationRequest: + return EvaluationRequest( + candidate=Candidate( + id="candidate", + version="version-1", + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + ) + + +@pytest.mark.asyncio +async def test_command_backend_uses_argv_without_shell_interpolation(tmp_path: Path): + harness_root = tmp_path / "harness" + script = write_harness( + harness_root, + """ +import json +import sys +from pathlib import Path + +workspace, request_path, report_path = map(Path, sys.argv[1:]) +payload = json.loads(request_path.read_text()) +Path(report_path).write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": { + "workspace_exists": float(workspace.exists()), + "request_schema": float(payload["schema_version"]), + }, +})) +print(workspace) +""", + ) + workspace_path = tmp_path / "target;touch SHOULD_NOT_EXIST" + runtime_context = await context(tmp_path, workspace_path) + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(harness_root), + command=[ + sys.executable, + str(script), + "{workspace}", + "{request}", + "{report}", + ], + ) + ) + + report = await backend.evaluate(context=runtime_context, request=request()) + + assert report.status == EvaluationStatus.SUCCESS + assert report.metrics == {"workspace_exists": 1.0, "request_schema": 1.0} + assert not (tmp_path / "SHOULD_NOT_EXIST").exists() + assert [artifact.path for artifact in report.artifacts] == [ + "command/stdout.log", + "command/stderr.log", + ] + + +@pytest.mark.asyncio +async def test_command_backend_passes_only_declared_environment(tmp_path: Path): + harness_root = tmp_path / "harness" + script = write_harness( + harness_root, + """ +import json +import os +import sys +from pathlib import Path + +Path(sys.argv[1]).write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": { + "configured": float(os.environ["CONFIGURED"]), + "passed": float(os.environ["PASSED"]), + "hidden_absent": float("HIDDEN" not in os.environ), + }, +})) +""", + ) + os.environ["PASSED"] = "2" + os.environ["HIDDEN"] = "3" + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(harness_root), + command=[sys.executable, str(script), "{report}"], + environment={"CONFIGURED": "1"}, + passthrough_environment=["PASSED"], + ) + ) + + report = await backend.evaluate( + context=await context(tmp_path, tmp_path / "target"), + request=request(), + ) + + assert report.metrics == { + "configured": 1.0, + "passed": 2.0, + "hidden_absent": 1.0, + } + + +@pytest.mark.asyncio +async def test_command_backend_redacts_secrets(tmp_path: Path): + harness_root = tmp_path / "harness" + secret = "highly-sensitive-token" + script = write_harness( + harness_root, + """ +import json +import os +import sys +from pathlib import Path + +secret = os.environ["SECRET_TOKEN"] +print(f"stdout leaked {secret}") +print(f"stderr leaked {secret}", file=sys.stderr) +Path(sys.argv[1]).write_text(json.dumps({ + "schema_version": 1, + "status": "failed", + "diagnostics": [{ + "code": "harness_failed", + "message": f"diagnostic leaked {secret}", + "severity": "error" + }] +})) +""", + ) + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(harness_root), + command=[sys.executable, str(script), "{report}"], + environment={"SECRET_TOKEN": secret}, + ) + ) + runtime_context = await context(tmp_path, tmp_path / "target") + + report = await backend.evaluate(context=runtime_context, request=request()) + + persisted_text = ( + report.model_dump_json() + + (runtime_context.artifact_dir / "command" / "stdout.log").read_text() + + (runtime_context.artifact_dir / "command" / "stderr.log").read_text() + ) + assert secret not in persisted_text + assert "[REDACTED]" in persisted_text + with pytest.raises(ValueError, match="must not contain configured secret"): + backend.validate_request( + request().model_copy(update={"parameters": {"token": secret}}) + ) + + +@pytest.mark.parametrize( + ("body", "arguments", "expected_code"), + [ + ("raise SystemExit(7)", [], "command_failed"), + ("print('no report')", [], "missing_report"), + ( + "from pathlib import Path; import sys; Path(sys.argv[1]).write_text('{')", + ["{report}"], + "invalid_report", + ), + ], +) +@pytest.mark.asyncio +async def test_command_failures_return_failed_reports( + tmp_path: Path, + body: str, + arguments: list[str], + expected_code: str, +): + harness_root = tmp_path / "harness" + script = write_harness(harness_root, body) + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(harness_root), + command=[sys.executable, str(script), *arguments], + ) + ) + + report = await backend.evaluate( + context=await context(tmp_path, tmp_path / "target"), + request=request(), + ) + + assert report.status == EvaluationStatus.FAILED + assert report.diagnostics[0].code == expected_code + + +@pytest.mark.parametrize( + ("selection", "expected"), + [ + (AllCases(), None), + (CaseIds(ids=["a", "b"]), 2), + (CaseRange(stop=5), 5), + (CaseRange(start=5, stop=9), 4), + ], +) +@pytest.mark.asyncio +async def test_command_backend_resolves_cost(tmp_path: Path, selection, expected): + backend = CommandBackend( + CommandBackendConfig(harness_root=str(tmp_path), command=["run"]) + ) + + cost = await backend.resolve_cost(EvaluationSet(selection=selection)) + + assert cost.cases == expected + + +@pytest.mark.asyncio +async def test_command_backend_exports_only_allowlisted_agent_inputs(tmp_path: Path): + visible = tmp_path / "visible.json" + visible.write_text('{"visible": true}\n', encoding="utf-8") + hidden = tmp_path / "hidden.json" + hidden.write_text('{"hidden": true}\n', encoding="utf-8") + destination = tmp_path / "context" + destination.mkdir() + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(tmp_path), + command=["run"], + staged_inputs={"visible": str(visible), "hidden": str(hidden)}, + agent_context_inputs={"validation": ["visible"]}, + ) + ) + + await backend.export_case_resources( + evaluation_set=EvaluationSet(name="validation"), + destination=str(destination), + sandbox=await LocalSandbox.create(root=tmp_path), + ) + + index = json.loads((destination / "index.json").read_text()) + assert index["resources"] == [{"name": "visible", "path": "visible"}] + assert json.loads((destination / "visible").read_text()) == {"visible": True} + assert not (destination / "hidden").exists() + + +def test_command_config_rejects_unsafe_shapes(tmp_path: Path): + with pytest.raises(ValidationError, match="must be absolute"): + CommandBackendConfig(harness_root="relative", command=["run"]) + with pytest.raises(ValidationError, match="must not be empty"): + CommandBackendConfig(harness_root=str(tmp_path), command=[]) + with pytest.raises(ValidationError, match="unknown command placeholders"): + CommandBackendConfig( + harness_root=str(tmp_path), + command=["run", "{secret}"], + ) + with pytest.raises(ValidationError, match="overlap"): + CommandBackendConfig( + harness_root=str(tmp_path), + command=["run"], + environment={"TOKEN": "value"}, + passthrough_environment=["TOKEN"], + ) + with pytest.raises(ValidationError, match="unknown staged inputs"): + CommandBackendConfig( + harness_root=str(tmp_path), + command=["run"], + agent_context_inputs={"validation": ["missing"]}, + ) + + +@pytest.mark.asyncio +async def test_harness_cannot_live_inside_target(tmp_path: Path): + target = tmp_path / "target" + harness = target / "harness" + harness.mkdir(parents=True) + backend = CommandBackend( + CommandBackendConfig(harness_root=str(harness), command=["run"]) + ) + + with pytest.raises(ValueError, match="outside the editable target"): + await backend.evaluate( + context=await context(tmp_path, target), + request=request(), + ) diff --git a/vero/tests/test_v05_config.py b/vero/tests/test_v05_config.py new file mode 100644 index 00000000..0e826500 --- /dev/null +++ b/vero/tests/test_v05_config.py @@ -0,0 +1,148 @@ +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from vero.config import AgentOptimizerConfig, VeroConfig, _producer, load_config + + +def _config(optimizer: dict) -> dict: + return { + "target": {"root": "."}, + "backend": { + "harness_root": ".", + "command": ["evaluate", "{workspace}", "{report}"], + }, + "evaluations": [{"name": "default"}], + "protocol": {"selection_evaluation": "default"}, + "objective": {"metric": "score", "direction": "maximize"}, + "optimizer": optimizer, + } + + +def test_agent_optimizer_accepts_only_agent_fields(): + config = VeroConfig.model_validate( + _config( + { + "kind": "vero", + "instruction": "Improve the program", + "model": "openai/gpt-5", + "max_turns": 10, + } + ) + ) + + assert isinstance(config.optimizer, AgentOptimizerConfig) + assert config.optimizer.model == "openai/gpt-5" + assert config.optimizer.max_turns == 10 + + +def test_agent_optimizer_rejects_empty_model(): + with pytest.raises(ValidationError, match="model must not be empty"): + VeroConfig.model_validate(_config({"kind": "vero", "model": " "})) + + +@pytest.mark.parametrize( + ("kind", "model"), + [("vero", "openai/gpt-5"), ("claude", "claude-opus-4-1")], +) +def test_agent_optimizer_applies_configured_model(kind, model): + producer = _producer(AgentOptimizerConfig(kind=kind, model=model)) + + configured = ( + producer.agent.model_str() + if kind == "vero" + else producer.agent.options.model + ) + assert configured == model + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("root", "producer"), + ("working_directory", "src"), + ("environment", {"NAME": "value"}), + ("passthrough_environment", ["TOKEN"]), + ("timeout_seconds", 10), + ("description", "ignored"), + ("command", ["ignored"]), + ], +) +def test_agent_optimizer_rejects_command_only_fields(field, value): + optimizer = {"kind": "claude", field: value} + + with pytest.raises(ValidationError, match=field): + VeroConfig.model_validate(_config(optimizer)) + + +def test_config_rejects_case_ids_combined_with_range_start(): + value = _config({"kind": "vero"}) + value["evaluations"][0].update({"case_ids": ["a"], "case_start": 1}) + + with pytest.raises(ValidationError, match="case_ids and case range"): + VeroConfig.model_validate(value) + + +def test_config_wires_retry_policy_into_evaluation_limits(): + value = _config({"kind": "vero"}) + value["protocol"]["retry"] = { + "max_attempts": 5, + "initial_delay_seconds": 0.5, + } + + config = VeroConfig.model_validate(value) + + assert config.protocol.to_limits().retry.max_attempts == 5 + assert config.protocol.to_limits().retry.initial_delay_seconds == 0.5 + + +def test_config_wires_case_failure_and_error_rate_policy(): + value = _config({"kind": "vero"}) + value["protocol"]["error_rate_threshold"] = 0.25 + value["objective"].update( + {"aggregation": "mean", "case_failure_value": 0.0} + ) + + config = VeroConfig.model_validate(value) + + assert config.protocol.to_limits().error_rate_threshold == 0.25 + assert config.objective.to_model().selector.case_failure_value == 0.0 + + +def test_config_resolves_agent_context_inputs_with_staged_inputs(tmp_path): + config_path = tmp_path / "vero.toml" + config_path.write_text( + """ +[target] +root = "target" + +[backend] +harness_root = "harness" +command = ["run", "{input:cases}"] + +[backend.staged_inputs] +cases = "data/cases.json" + +[backend.agent_context_inputs] +train = ["cases"] + +[[evaluations]] +name = "train" + +[protocol] +selection_evaluation = "train" + +[objective] +metric = "score" +direction = "maximize" +""", + encoding="utf-8", + ) + + config = load_config(config_path) + + assert config.backend.agent_context_inputs == {"train": ["cases"]} + assert config.backend.staged_inputs == { + "cases": str((tmp_path / "data/cases.json").resolve()) + } diff --git a/vero/tests/test_v05_darwin_godel_strategy.py b/vero/tests/test_v05_darwin_godel_strategy.py new file mode 100644 index 00000000..068153cd --- /dev/null +++ b/vero/tests/test_v05_darwin_godel_strategy.py @@ -0,0 +1,120 @@ +from __future__ import annotations + +import pytest + +from vero.candidate import Candidate +from vero.evaluation import ( + EvaluationSet, + EvaluationStatus, + EvaluationSummary, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, +) +from vero.optimization import DarwinGodelStrategy +from vero.optimization.models import OptimizationContext + +OBJ_MAX = ObjectiveSpec(selector=MetricSelector(metric="score"), direction="maximize") + + +def _summary(candidate_id: str, value: float | None, *, feasible: bool = True) -> EvaluationSummary: + return EvaluationSummary( + evaluation_id=f"eval-{candidate_id}", + candidate_id=candidate_id, + candidate_version=f"{candidate_id}-v", + backend_id="cmd", + evaluation_set=EvaluationSet(name="perf"), + status=EvaluationStatus.SUCCESS, + metrics={"score": value} if value is not None else {}, + objective=ObjectiveResult(value=value, feasible=feasible), + total_cases=1, + successful_cases=1, + errored_cases=0, + skipped_cases=0, + ) + + +def _context(*, candidates, evaluations=(), best=None, baseline=None) -> OptimizationContext: + baseline = baseline or next(iter(candidates.values())) + return OptimizationContext( + session_id="s", + round=0, + workspace=object(), # unused by the strategy + baseline=baseline, + evaluations=tuple(evaluations), + candidates=candidates, + best=best, + ) + + +@pytest.mark.asyncio +async def test_dgm_seeds_from_baseline_when_archive_has_only_the_baseline(): + strat = DarwinGodelStrategy(objective=OBJ_MAX, num_offspring=3, seed=0) + base = Candidate(id="base", version="base-v") + ctx = _context(candidates={"base": base}, baseline=base) + + proposals = await strat.propose(ctx) + + assert len(proposals) == 3 + assert all(p.parent_id == "base" for p in proposals) + assert all(p.metadata["strategy"] == "darwin-godel" for p in proposals) + assert proposals[0].producer_id == "self" # self-application producer by default + + +@pytest.mark.asyncio +async def test_dgm_keeps_the_whole_archive_reachable_including_low_and_unscored(): + # Unlike EvolutionaryStrategy's fittest-K population, every archived agent — + # even a weak or not-yet-scored one — can be a parent (stepping stones). + strat = DarwinGodelStrategy(objective=OBJ_MAX, num_offspring=300, seed=1) + cands = {c: Candidate(id=c, version=f"{c}-v") for c in ["base", "strong", "weak", "unscored"]} + evals = (_summary("base", 0.10), _summary("strong", 0.90), _summary("weak", 0.05)) + ctx = _context(candidates=cands, evaluations=evals, best=cands["strong"], baseline=cands["base"]) + + parents = [p.parent_id for p in await strat.propose(ctx)] + seen = set(parents) + + assert "weak" in seen and "unscored" in seen # low + unscored stay reachable + assert parents.count("strong") > parents.count("weak") # performance still biases selection + + +@pytest.mark.asyncio +async def test_dgm_inverse_children_downweights_explored_lineages(): + # Equal performance, but the more-explored lineage gets a lower selection weight. + strat = DarwinGodelStrategy(objective=OBJ_MAX, seed=0) + strat._score = {"a": 0.8, "b": 0.8} + strat._children = {"a": 3, "b": 0} + + assert strat._weight("a") == pytest.approx(0.8 / 4) + assert strat._weight("b") == pytest.approx(0.8 / 1) + assert strat._weight("a") < strat._weight("b") + + +@pytest.mark.asyncio +async def test_dgm_tracks_children_across_rounds(): + strat = DarwinGodelStrategy(objective=OBJ_MAX, num_offspring=4, seed=3) + cands = {c: Candidate(id=c, version=f"{c}-v") for c in ["a", "b"]} + ctx = _context(candidates=cands, evaluations=(_summary("a", 0.8), _summary("b", 0.8)), baseline=cands["a"]) + + await strat.propose(ctx) + await strat.propose(ctx) + + # Every selection increments the parent's child count; 2 rounds × 4 offspring = 8. + assert sum(strat._children.values()) == 8 + assert strat._generation == 2 + + +@pytest.mark.asyncio +async def test_dgm_minimize_direction_prefers_lower_values(): + strat = DarwinGodelStrategy( + objective=ObjectiveSpec(selector=MetricSelector(metric="score"), direction="minimize"), + num_offspring=200, + base_weight=0.01, # sharpen so the performance signal dominates + seed=0, + ) + cands = {c: Candidate(id=c, version=f"{c}-v") for c in ["hi", "lo"]} + ctx = _context(candidates=cands, evaluations=(_summary("hi", 9.0), _summary("lo", 1.0)), baseline=cands["lo"]) + + parents = [p.parent_id for p in await strat.propose(ctx)] + + # Under minimize, the lower value ("lo") is the higher performer → picked more. + assert parents.count("lo") > parents.count("hi") diff --git a/vero/tests/test_v05_docker_sandbox.py b/vero/tests/test_v05_docker_sandbox.py new file mode 100644 index 00000000..aeda19d0 --- /dev/null +++ b/vero/tests/test_v05_docker_sandbox.py @@ -0,0 +1,210 @@ +from __future__ import annotations + +import asyncio +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import ( + CommandBackend, + CommandBackendConfig, + EvaluationPlan, + MetricSelector, + ObjectiveSpec, +) +from vero.optimization import CommandCandidateProducer, CommandCandidateProducerConfig +from vero.runtime import create_optimization_session +from vero.sandbox import CommandResult, DockerSandbox +from vero.workspace import GitWorkspace + + +def _docker_available() -> bool: + executable = shutil.which("docker") + if executable is None: + return False + return ( + subprocess.run( + [executable, "info"], + capture_output=True, + timeout=10, + check=False, + ).returncode + == 0 + ) + + +@pytest.mark.asyncio +async def test_docker_sandbox_owns_an_unmounted_container(monkeypatch): + commands: list[list[str]] = [] + + async def fake_host_command(command, *, timeout=30, before_terminate=None): + commands.append(command) + if command[1] == "run": + return CommandResult("container-id", "", 0) + return CommandResult("", "", 0) + + monkeypatch.setattr( + DockerSandbox, + "_host_command", + staticmethod(fake_host_command), + ) + + sandbox = await DockerSandbox.create( + image="example/image:locked", + docker_executable="docker", + ) + await sandbox.close() + + run_command = commands[0] + assert run_command[:3] == ["docker", "run", "--detach"] + assert "--volume" not in run_command + assert "-v" not in run_command + assert sandbox.host_path("/workspace/project") is None + exec_command = next(command for command in commands if command[1] == "exec") + assert "setsid" in exec_command + assert commands[-1] == ["docker", "rm", "--force", "container-id"] + + +@pytest.mark.asyncio +async def test_docker_timeout_terminates_the_in_container_process_group(monkeypatch): + host_commands: list[list[str]] = [] + cleanup_commands: list[tuple[str, ...]] = [] + + async def fake_host_command( + command, + *, + timeout=30, + before_terminate=None, + ): + host_commands.append(command) + assert before_terminate is not None + await before_terminate() + return CommandResult("", "timed out", -1) + + async def fake_docker(*arguments, timeout=30): + cleanup_commands.append(arguments) + return CommandResult("", "", 0) + + monkeypatch.setattr( + DockerSandbox, + "_host_command", + staticmethod(fake_host_command), + ) + sandbox = DockerSandbox("container-id", docker_executable="docker") + monkeypatch.setattr(sandbox, "_docker", fake_docker) + + result = await sandbox.run(["sh", "-c", "sleep 60"], timeout=0.1) + + assert result.returncode == -1 + assert "setsid" in host_commands[0] + assert cleanup_commands + assert cleanup_commands[0][:2] == ("exec", "container-id") + assert "kill -TERM" in cleanup_commands[0][4] + + +@pytest.mark.skipif(not _docker_available(), reason="Docker daemon is unavailable") +@pytest.mark.asyncio +async def test_generic_optimization_without_a_shared_filesystem(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.c").write_text( + '#include \nint main(void) { printf("1.0\\n"); return 0; }\n', + encoding="utf-8", + ) + subprocess.run(["git", "init", "-b", "main"], cwd=target, check=True) + subprocess.run(["git", "add", "--all"], cwd=target, check=True) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=target, + check=True, + ) + + harness = tmp_path / "harness" + harness.mkdir() + (harness / "evaluate.sh").write_text( + """#!/bin/sh +set -eu +workspace=$1 +report=$2 +artifacts=$3 +cc "$workspace/program.c" -o "$artifacts/program" +score=$("$artifacts/program") +printf '{"schema_version":1,"status":"success","metrics":{"score":%s}}' "$score" > "$report" +""", + encoding="utf-8", + ) + producer_root = tmp_path / "producer" + producer_root.mkdir() + (producer_root / "improve.sh").write_text( + "sed -i 's/1.0/2.0/' \"$1/program.c\"\n", + encoding="utf-8", + ) + + sandbox = await DockerSandbox.create( + image=os.environ.get("VERO_DOCKER_TEST_IMAGE", "gcc:14-bookworm") + ) + try: + remote_target = "/workspace/target" + assert sandbox.host_path(remote_target) is None + await sandbox.upload(target, remote_target) + workspace = await GitWorkspace.from_path(sandbox, remote_target) + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(harness), + command=[ + "sh", + "{harness}/evaluate.sh", + "{workspace}", + "{report}", + "{artifacts}", + ], + ) + ) + producer = CommandCandidateProducer( + CommandCandidateProducerConfig( + root=str(producer_root), + command=["sh", "{producer}/improve.sh", "{workspace}"], + ) + ) + candidate_repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", + workspace=workspace, + ) + session = await create_optimization_session( + workspace=workspace, + candidate_repository=candidate_repository, + session_dir=tmp_path / "session", + backend_id="command", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + evaluation_plan=EvaluationPlan.single(), + producers={"default": producer}, + max_proposals=1, + ) + + result = await asyncio.wait_for(session.run(), timeout=180) + + assert result.baseline.objective.value == 1.0 + assert result.best.objective.value == 2.0 + assert (session.session_dir / "database.json").is_file() + assert await sandbox.read_file(f"{remote_target}/program.c") == ( + '#include \nint main(void) { printf("1.0\\n"); return 0; }\n' + ) + finally: + await sandbox.close() diff --git a/vero/tests/test_v05_error_taxonomy.py b/vero/tests/test_v05_error_taxonomy.py new file mode 100644 index 00000000..6beb23d7 --- /dev/null +++ b/vero/tests/test_v05_error_taxonomy.py @@ -0,0 +1,60 @@ +"""Tests for the single source of truth error taxonomy.""" + +from vero.evaluation.scoring.error_taxonomy import ( + ErrorCategory, + classify_case, + classify_signal, + policy, +) + + +def test_classify_signal_recognizes_infrastructure_categories(): + assert classify_signal("openai.RateLimitError") is ErrorCategory.TRANSIENT_INFRA + assert classify_signal("APITimeoutError") is ErrorCategory.TRANSIENT_INFRA + assert classify_signal("ConnectionError") is ErrorCategory.TRANSIENT_INFRA + assert classify_signal("AuthenticationError") is ErrorCategory.AUTH_FAILURE + assert classify_signal("PermissionDeniedError") is ErrorCategory.AUTH_FAILURE + assert ( + classify_signal("insufficient_quota") + is ErrorCategory.INFERENCE_BUDGET_EXHAUSTED + ) + + +def test_classify_signal_leaves_task_failures_unclassified(): + # The benign "produced no answer" marker and a candidate's own bug are not + # infrastructure — the case-level classifier turns both into task failures. + assert classify_signal("no_rewards_recorded") is None + assert classify_signal("ValueError") is None + assert classify_signal("") is None + + +def test_classify_case_precedence_and_defaults(): + assert classify_case([]) is ErrorCategory.TASK_FAILURE + assert classify_case(["no_rewards_recorded"]) is ErrorCategory.TASK_FAILURE + assert classify_case(["KeyError"]) is ErrorCategory.TASK_FAILURE + assert classify_case(["openai.RateLimitError"]) is ErrorCategory.TRANSIENT_INFRA + # Auth (terminating) outranks a transient rate limit seen on another attempt. + assert ( + classify_case(["openai.RateLimitError", "AuthenticationError"]) + is ErrorCategory.AUTH_FAILURE + ) + + +def test_policy_encodes_the_intended_treatment(): + budget = policy(ErrorCategory.INFERENCE_BUDGET_EXHAUSTED) + assert budget.terminating and not budget.retryable + assert not budget.counts_toward_invalidity + + eval_budget = policy(ErrorCategory.EVALUATION_BUDGET_EXHAUSTED) + assert not eval_budget.terminating and not eval_budget.retryable + + auth = policy(ErrorCategory.AUTH_FAILURE) + assert auth.terminating and not auth.retryable + + transient = policy(ErrorCategory.TRANSIENT_INFRA) + assert transient.retryable and transient.counts_toward_invalidity + assert not transient.is_informative_sample + + task = policy(ErrorCategory.TASK_FAILURE) + assert task.is_informative_sample + assert not task.counts_toward_invalidity and not task.terminating diff --git a/vero/tests/test_v05_evals_cli.py b/vero/tests/test_v05_evals_cli.py new file mode 100644 index 00000000..090844e2 --- /dev/null +++ b/vero/tests/test_v05_evals_cli.py @@ -0,0 +1,364 @@ +from __future__ import annotations + +import json +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest +import pytest_asyncio +from click.testing import CliRunner + +from vero.candidate import Candidate +from vero.evals_cli import _enrich_job, evals +from vero.evaluation import ( + BackendProvenance, + CaseResult, + CaseStatus, + DisclosureLevel, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, + project_evaluation, +) +from vero.runtime.context import AgentContextDirectory +from vero.sandbox import LocalSandbox + + +def _record(evaluation_id: str, scores: dict[str, float], value: float): + created_at = datetime(2026, 1, 1, tzinfo=UTC) + return EvaluationRecord( + id=evaluation_id, + request=EvaluationRequest( + candidate=Candidate( + id=f"candidate:{evaluation_id}", + version="a" * 40, + created_at=created_at, + ), + evaluation_set=EvaluationSet(name="validation", partition="validation"), + ), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": value}, + cases=[ + CaseResult( + case_id=case_id, + status=CaseStatus.SUCCESS, + metrics={"score": score}, + input={"task_name": f"task-{case_id}"}, + output={"answer": "answer"}, + execution_trace=[ + {"turn": 1, "message": f"marker-{case_id}"}, + {"turn": 2, "tool": "run", "result": {"stdout": "ok"}}, + ], + ) + for case_id, score in scores.items() + ], + ), + backend_id="backend", + backend=BackendProvenance(name="test", version="1", config_digest="0" * 64), + objective_spec=ObjectiveSpec( + selector=MetricSelector(metric="score"), direction="maximize" + ), + objective=ObjectiveResult(value=value, feasible=True), + created_at=created_at, + completed_at=created_at + timedelta(seconds=1), + ) + + +@pytest_asyncio.fixture +async def context_dir(tmp_path: Path) -> Path: + baseline = _record("evaluation:baseline", {"one": 0.25, "two": 1.0}, 0.625) + candidate = _record("evaluation:candidate", {"one": 0.75, "two": 0.5}, 0.7) + aggregate = _record("evaluation:aggregate", {"one": 0.5}, 0.5) + project = tmp_path / "project" + project.mkdir() + directory = AgentContextDirectory( + sandbox=await LocalSandbox.create(root=tmp_path), + root=str(project / ".evals"), + session_dir=tmp_path / "session", + ) + await directory.reset() + await directory.write_header( + session_id="session", + round_number=1, + proposal_id="proposal", + parent_candidate_id="parent", + ) + await directory.write_evaluations( + [ + (item, disclosure, project_evaluation(item, disclosure)) + for item, disclosure in ( + (baseline, DisclosureLevel.FULL), + (candidate, DisclosureLevel.FULL), + (aggregate, DisclosureLevel.AGGREGATE), + ) + ] + ) + root = project / ".evals" + (root / "plan.json").write_text( + json.dumps( + { + "schema_version": 1, + "evaluations": [ + { + "name": "validation", + "partition": "validation", + "base_selection": {"kind": "all"}, + "agent_can_evaluate": True, + "agent_selection": "arbitrary", + "disclosure": "full", + "expose_case_resources": True, + "backend": "harbor-validation", + "cases": 12, + "budget": {"remaining_runs": 3, "remaining_cases": 40}, + } + ], + } + ) + ) + resources = root / "tasks" / "digest" / "resources" + resources.mkdir(parents=True) + (resources / "one.json").write_text('{"prompt": "task one"}') + (resources / "index.json").write_text( + json.dumps({"schema_version": 1, "cases": [{"case_id": "one", "path": "one.json"}]}) + ) + (root / "tasks" / "index.json").write_text( + json.dumps( + { + "schema_version": 1, + "case_resources": [ + { + "backend_id": "backend", + "evaluation_set": {"name": "validation", "partition": "validation"}, + "path": "digest", + } + ], + } + ) + ) + return root + + +def _invoke(context_dir: Path, *arguments: str): + result = CliRunner().invoke( + evals, [*arguments, "--context", str(context_dir)], catch_exceptions=False + ) + return result + + +@pytest.mark.asyncio +async def test_list_shows_every_result_and_sorts(context_dir: Path): + result = _invoke(context_dir, "list", "--json") + assert result.exit_code == 0 + rows = json.loads(result.output) + assert {row["id"] for row in rows} == { + "evaluation:baseline", + "evaluation:candidate", + "evaluation:aggregate", + } + by_id = {row["id"]: row for row in rows} + assert by_id["evaluation:baseline"]["score"] == 0.625 + assert by_id["evaluation:baseline"]["cases"] == 2 + assert by_id["evaluation:aggregate"]["disclosure"] == "aggregate" + assert by_id["evaluation:aggregate"]["cases"] == 1 # summary count only + + ordered = json.loads( + _invoke(context_dir, "list", "--json", "--sort", "score", "--desc").output + ) + assert ordered[0]["id"] == "evaluation:candidate" + + +@pytest.mark.asyncio +async def test_show_summarizes_case_files(context_dir: Path): + result = _invoke(context_dir, "show", "evaluation:baseline") + assert result.exit_code == 0 + assert "2 cases" in result.output + assert "evals cases" in result.output + + +@pytest.mark.asyncio +async def test_cases_lists_per_case_results_and_respects_disclosure(context_dir: Path): + result = _invoke(context_dir, "cases", "evaluation:baseline", "--json") + assert result.exit_code == 0 + rows = json.loads(result.output) + assert [(row["case_id"], row["score"]) for row in rows] == [ + ("one", 0.25), + ("two", 1.0), + ] + assert all(row["trace"] for row in rows) + + denied = CliRunner().invoke( + evals, ["cases", "evaluation:aggregate", "--context", str(context_dir)] + ) + assert denied.exit_code != 0 + assert "disclosure" in denied.output + + +@pytest.mark.asyncio +async def test_trace_summary_and_span_window(context_dir: Path): + summary = _invoke(context_dir, "trace", "evaluation:baseline", "one") + assert summary.exit_code == 0 + payload = json.loads(summary.output) + assert payload["execution_trace"]["spans"] == 2 + assert any("turn" in shape for shape in payload["execution_trace"]["shapes"]) + + span = _invoke(context_dir, "trace", "evaluation:baseline", "one", "--span", "0") + assert span.exit_code == 0 + assert "marker-one" in span.output + + +@pytest.mark.asyncio +async def test_diff_reports_per_case_verdicts(context_dir: Path): + result = _invoke( + context_dir, "diff", "evaluation:baseline", "evaluation:candidate", "--json" + ) + assert result.exit_code == 0 + payload = json.loads(result.output) + verdicts = {row["case_id"]: row["verdict"] for row in payload["cases"]} + assert verdicts == {"one": "improved", "two": "regressed"} + assert payload["summary"] == {"improved": 1, "regressed": 1} + + +@pytest.mark.asyncio +async def test_plan_shows_budget(context_dir: Path): + result = _invoke(context_dir, "plan", "--json") + assert result.exit_code == 0 + rows = json.loads(result.output) + assert rows == [ + { + "evaluation": "validation", + "partition": "validation", + "backend": "harbor-validation", + "cases": 12, + "can_evaluate": True, + "selection": "arbitrary", + "disclosure": "full", + "runs_left": 3, + "cases_left": 40, + } + ] + + +@pytest.mark.asyncio +async def test_tasks_lists_sets_then_task_paths(context_dir: Path): + sets = json.loads(_invoke(context_dir, "tasks", "--json").output) + assert sets == [ + { + "evaluation": "validation", + "partition": "validation", + "tasks": 1, + "path": "tasks/digest/resources/", + } + ] + tasks = json.loads(_invoke(context_dir, "tasks", "validation", "--json").output) + assert tasks == [{"case_id": "one", "path": "tasks/digest/resources/one.json"}] + + +@pytest.mark.asyncio +async def test_context_is_discovered_from_workspace(context_dir: Path, monkeypatch): + monkeypatch.chdir(context_dir.parent) + monkeypatch.delenv("VERO_CONTEXT_PATH", raising=False) + result = CliRunner().invoke(evals, ["list", "--json"], catch_exceptions=False) + assert result.exit_code == 0 + assert len(json.loads(result.output)) == 3 + + +def test_enrich_job_adds_elapsed_and_requested_cases(): + # terminal job: elapsed from created->completed, exactly. + done = _enrich_job( + { + "job_id": "j", + "status": "complete", + "created_at": "2026-01-01T00:00:00Z", + "completed_at": "2026-01-01T00:05:00Z", + } + ) + assert done["elapsed_seconds"] == 300.0 + # subset range run: requested_cases = stop - start. + ranged = _enrich_job( + { + "job_id": "j", + "status": "running", + "created_at": "2026-01-01T00:00:00Z", + "evaluation_set": {"selection": {"start": 0, "stop": 8}}, + } + ) + assert ranged["requested_cases"] == 8 + assert ranged["elapsed_seconds"] >= 0 + # explicit case ids -> len; whole-partition run -> no requested_cases. + assert _enrich_job({"evaluation_set": {"selection": {"ids": ["a", "b"]}}})[ + "requested_cases" + ] == 2 + assert "requested_cases" not in _enrich_job( + {"status": "running", "evaluation_set": {"name": "v"}} + ) + + +def test_status_job_output_is_enriched(monkeypatch): + import vero.harbor.cli as harbor_cli + + monkeypatch.setattr( + harbor_cli, + "_request", + lambda method, path, **kw: { + "job_id": "j", + "status": "running", + "created_at": "2026-01-01T00:00:00Z", + "evaluation_set": {"selection": {"start": 0, "stop": 10}}, + }, + ) + result = CliRunner().invoke(evals, ["status", "j"], catch_exceptions=False) + assert result.exit_code == 0 + payload = json.loads(result.output) + assert payload["requested_cases"] == 10 + assert "elapsed_seconds" in payload + + +def test_wait_blocks_until_complete_then_prints_result(monkeypatch): + import vero.harbor.cli as harbor_cli + + statuses = iter(["running", "running", "complete"]) + calls: list[str] = [] + + def fake_request(method, path, **kw): + calls.append(path) + if path.endswith("/result"): + return {"result": {"objective": {"value": 0.5}}} + return {"job_id": "j", "status": next(statuses), "created_at": "2026-01-01T00:00:00Z"} + + monkeypatch.setattr(harbor_cli, "_request", fake_request) + monkeypatch.setattr("time.sleep", lambda _seconds: None) + result = CliRunner().invoke( + evals, ["wait", "j", "--poll-interval", "1"], catch_exceptions=False + ) + assert result.exit_code == 0 + assert json.loads(result.output)["result"]["objective"]["value"] == 0.5 + assert calls[-1] == "/eval/jobs/j/result" # result fetched only after terminal + + +def test_wait_timeout_returns_still_running_and_enriched(monkeypatch): + import vero.harbor.cli as harbor_cli + + monkeypatch.setattr( + harbor_cli, + "_request", + lambda method, path, **kw: { + "job_id": "j", + "status": "running", + "created_at": "2026-01-01T00:00:00Z", + "evaluation_set": {"selection": {"ids": ["a", "b", "c"]}}, + }, + ) + monkeypatch.setattr("time.sleep", lambda _seconds: None) + result = CliRunner().invoke( + evals, ["wait", "j", "--timeout", "0"], catch_exceptions=False + ) + assert result.exit_code == 0 # clean exit, not Exit 143 + payload = json.loads(result.output) + assert payload["status"] == "running" + assert payload["requested_cases"] == 3 diff --git a/vero/tests/test_v05_evaluation_models.py b/vero/tests/test_v05_evaluation_models.py new file mode 100644 index 00000000..9f12b773 --- /dev/null +++ b/vero/tests/test_v05_evaluation_models.py @@ -0,0 +1,338 @@ +from datetime import UTC, datetime, timedelta + +import pytest +from pydantic import ValidationError + +from vero.candidate import Candidate +from vero.evaluation import ( + AllCases, + BackendProvenance, + CaseError, + CaseIds, + CaseRange, + CaseResult, + CaseStatus, + DiagnosticSeverity, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationArtifact, + EvaluationBudget, + EvaluationDefinition, + EvaluationDiagnostic, + EvaluationLimits, + EvaluationPlan, + EvaluationPrincipal, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, + RetryPolicy, +) + + +def candidate(candidate_id: str = "candidate-1") -> Candidate: + return Candidate( + id=candidate_id, + version=f"snapshot:{candidate_id}", + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + + +def test_candidate_identity_is_workspace_neutral(): + value = Candidate.from_version( + "remote-revision:42", + candidate_id="idea-7", + parent_id="baseline", + metadata={"producer": "coding-agent"}, + ) + + assert value.id == "idea-7" + assert value.version == "remote-revision:42" + assert value.parent_id == "baseline" + assert not hasattr(value, "commit") + assert not hasattr(value, "repo_name") + + +def test_candidate_rejects_naive_timestamps_and_self_parent(): + with pytest.raises(ValidationError, match="timezone-aware"): + Candidate(id="a", version="1", created_at=datetime(2026, 1, 1)) + with pytest.raises(ValidationError, match="own parent"): + Candidate(id="a", version="1", parent_id="a") + + +@pytest.mark.parametrize( + "selection", + [ + AllCases(), + CaseIds(ids=["case-2", "case-7"]), + CaseRange(stop=10), + CaseRange(start=10, stop=20), + ], +) +def test_evaluation_set_round_trips_selection(selection): + evaluation_set = EvaluationSet( + name="performance", + partition="validation", + selection=selection, + ) + + restored = EvaluationSet.model_validate_json(evaluation_set.model_dump_json()) + + assert restored == evaluation_set + assert type(restored.selection) is type(selection) + assert restored.budget_key("command") == "command:performance:validation" + + +@pytest.mark.parametrize( + ("model", "kwargs"), + [ + (EvaluationSet, {"name": " "}), + (EvaluationSet, {"partition": ""}), + (CaseIds, {"ids": []}), + (CaseIds, {"ids": ["same", "same"]}), + (CaseRange, {"start": -1, "stop": 2}), + (CaseRange, {"start": 2, "stop": 2}), + ], +) +def test_selection_rejects_invalid_values(model, kwargs): + with pytest.raises(ValidationError): + model(**kwargs) + + +def test_request_fingerprint_ignores_candidate_display_metadata(): + first = EvaluationRequest( + candidate=Candidate( + id="same", + version="version-1", + created_at=datetime(2026, 1, 1, tzinfo=UTC), + description="first description", + ), + parameters={"outer": {"z": 1, "a": 2}, "alpha": True}, + ) + second = EvaluationRequest( + candidate=Candidate( + id="same", + version="version-1", + created_at=datetime(2026, 1, 2, tzinfo=UTC), + description="updated description", + ), + parameters={"alpha": True, "outer": {"a": 2, "z": 1}}, + ) + + assert first.fingerprint() == second.fingerprint() + + +def test_retry_policy_restores_transient_provider_defaults(): + policy = RetryPolicy() + + assert policy.max_attempts == 3 + assert policy.retry_on_timeout is True + assert policy.retry_status_codes == [429, 503, 529] + assert RetryPolicy.disabled().max_attempts == 1 + + +@pytest.mark.parametrize( + "values", + [ + {"initial_delay_seconds": 2, "maximum_delay_seconds": 1}, + {"retry_exception_names": ["same", "same"]}, + {"retry_status_codes": [99]}, + {"retry_message_patterns": ["["]}, + ], +) +def test_retry_policy_rejects_invalid_configuration(values): + with pytest.raises(ValidationError): + RetryPolicy(**values) + + +def test_case_failure_value_requires_case_aggregation(): + with pytest.raises(ValidationError, match="requires a case metric aggregation"): + MetricSelector(metric="score", case_failure_value=0.0) + + +@pytest.mark.parametrize("threshold", [0.0, 1.1]) +def test_error_rate_threshold_must_be_a_positive_fraction(threshold): + with pytest.raises(ValidationError): + EvaluationLimits(error_rate_threshold=threshold) + + +@pytest.mark.parametrize( + "path", + [ + "", + "/absolute.log", + "../escape.log", + "logs/../escape.log", + "logs//run.log", + "logs\\run.log", + ], +) +def test_artifacts_reject_unsafe_paths(path): + with pytest.raises(ValidationError): + EvaluationArtifact(path=path) + + +def test_case_result_preserves_multiple_attempt_errors(): + result = CaseResult( + case_id="1", + status=CaseStatus.ERROR, + errors=[ + CaseError(message="timed out", attempt=1, retryable=True), + CaseError( + message="failed again", + attempt=2, + retryable=False, + terminal=True, + ), + ], + ) + + assert [error.attempt for error in result.errors] == [1, 2] + + +@pytest.mark.parametrize( + "value", + [ + {"case_id": "1", "status": "error", "errors": []}, + { + "case_id": "1", + "status": "success", + "errors": [{"message": "fatal", "terminal": True}], + }, + { + "case_id": "1", + "status": "skipped", + "errors": [{"message": "fatal", "terminal": True}], + }, + ], +) +def test_case_status_and_terminal_errors_agree(value): + with pytest.raises(ValidationError): + CaseResult.model_validate(value) + + +def test_report_preserves_structured_diagnostics_and_rejects_duplicate_cases(): + case = CaseResult(case_id="same", status=CaseStatus.SUCCESS) + with pytest.raises(ValidationError, match="case IDs must be unique"): + EvaluationReport( + status=EvaluationStatus.SUCCESS, + cases=[case, case], + ) + + report = EvaluationReport( + status=EvaluationStatus.FAILED, + diagnostics=[ + EvaluationDiagnostic( + code="compile_failed", + message="compiler returned 1", + severity=DiagnosticSeverity.ERROR, + phase="compile", + ) + ], + ) + assert report.diagnostics[0].code == "compile_failed" + assert not hasattr(report, "error") + + +def test_backend_provenance_digest_is_stable_across_key_order(): + first = BackendProvenance.from_config( + name="command", version="1", config={"command": ["run"], "timeout": 10} + ) + second = BackendProvenance.from_config( + name="command", version="1", config={"timeout": 10, "command": ["run"]} + ) + + assert first == second + + +def test_record_is_schema_one_and_requires_aware_ordered_timestamps(): + specification = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + created_at = datetime(2026, 1, 1, tzinfo=UTC) + record = EvaluationRecord( + id="evaluation-1", + request=EvaluationRequest(candidate=candidate()), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 1.0}, + ), + backend_id="command", + backend=BackendProvenance( + name="command", + version="1", + config_digest="0" * 64, + ), + objective_spec=specification, + objective=ObjectiveResult(value=1.0, feasible=True), + created_at=created_at, + completed_at=created_at + timedelta(seconds=1), + ) + + assert record.schema_version == 2 + assert record.request.candidate.version == "snapshot:candidate-1" + + with pytest.raises(ValidationError, match="must not be before"): + record.model_copy( + update={"completed_at": created_at - timedelta(seconds=1)} + ).__class__.model_validate( + record.model_copy( + update={"completed_at": created_at - timedelta(seconds=1)} + ).model_dump() + ) + + +def test_evaluation_plan_models_selection_visibility_and_principal_budgets(): + validation = EvaluationSet(name="validation", partition="validation") + test = EvaluationSet(name="test", partition="test") + plan = EvaluationPlan( + evaluations=[ + EvaluationDefinition( + evaluation_set=validation, + access=EvaluationAccessPolicy( + disclosure=DisclosureLevel.AGGREGATE, + ), + agent_budget=EvaluationBudget( + backend_id="command", + evaluation_set_key=validation.budget_key("command"), + principal=EvaluationPrincipal.AGENT, + total_runs=10, + ), + system_budget=EvaluationBudget( + backend_id="command", + evaluation_set_key=validation.budget_key("command"), + principal=EvaluationPrincipal.SYSTEM, + total_runs=100, + ), + ), + EvaluationDefinition( + evaluation_set=test, + access=EvaluationAccessPolicy( + agent_can_evaluate=False, + agent_visible=False, + disclosure=DisclosureLevel.NONE, + ), + ), + ], + selection_evaluation="validation", + final_evaluation="test", + ) + + assert plan.selection.evaluation_set == validation + assert plan.final.evaluation_set == test + assert {budget.principal for budget in plan.budgets} == { + EvaluationPrincipal.AGENT, + EvaluationPrincipal.SYSTEM, + } + + with pytest.raises(ValidationError, match="final evaluation must be"): + EvaluationPlan( + evaluations=[EvaluationDefinition(evaluation_set=test)], + selection_evaluation="test", + final_evaluation="test", + ) diff --git a/vero/tests/test_v05_evaluation_objective.py b/vero/tests/test_v05_evaluation_objective.py new file mode 100644 index 00000000..bfa312df --- /dev/null +++ b/vero/tests/test_v05_evaluation_objective.py @@ -0,0 +1,223 @@ +from datetime import UTC, datetime, timedelta + +import pytest + +from vero.candidate import Candidate +from vero.evaluation import ( + BackendProvenance, + BackendRegistry, + CaseError, + CaseResult, + CaseStatus, + ConstraintOperator, + DisclosureLevel, + EvaluationAcknowledgement, + EvaluationCost, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + EvaluationSummary, + MetricAggregation, + MetricConstraint, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, + UnknownBackendError, + compare_evaluation_records, + evaluate_objective, + project_evaluation, + resolve_metric, + select_best_evaluation, +) + + +def report(status=EvaluationStatus.SUCCESS): + return EvaluationReport( + status=status, + metrics={"latency": 10.0, "correct": 1.0}, + cases=[ + CaseResult( + case_id="1", + status=CaseStatus.SUCCESS, + metrics={"score": 1.0}, + ), + CaseResult( + case_id="2", + status=CaseStatus.SUCCESS, + metrics={"score": 3.0}, + ), + CaseResult( + case_id="3", + status=CaseStatus.ERROR, + metrics={"score": 100.0}, + errors=[CaseError(message="failed", terminal=True)], + ), + CaseResult( + case_id="4", + status=CaseStatus.SKIPPED, + metrics={"score": 200.0}, + ), + ], + ) + + +@pytest.mark.parametrize( + ("aggregation", "expected"), + [ + (MetricAggregation.MEAN, 2.0), + (MetricAggregation.MEDIAN, 2.0), + (MetricAggregation.MIN, 1.0), + (MetricAggregation.MAX, 3.0), + ], +) +def test_case_metric_aggregations_use_only_successful_cases(aggregation, expected): + assert resolve_metric( + report(), MetricSelector(metric="score", aggregation=aggregation) + ) == expected + + +def test_case_metric_aggregation_penalizes_failed_and_missing_cases(): + selector = MetricSelector( + metric="score", + aggregation=MetricAggregation.MEAN, + case_failure_value=0.0, + ) + + assert resolve_metric(report(), selector) == 1.0 + + +@pytest.mark.parametrize( + ("operator", "threshold", "feasible"), + [ + (ConstraintOperator.EQ, 1.0, True), + (ConstraintOperator.NE, 0.0, True), + (ConstraintOperator.LT, 2.0, True), + (ConstraintOperator.LTE, 1.0, True), + (ConstraintOperator.GT, 0.0, True), + (ConstraintOperator.GTE, 1.0, True), + (ConstraintOperator.EQ, 0.0, False), + ], +) +def test_objective_supports_every_constraint_operator(operator, threshold, feasible): + specification = ObjectiveSpec( + selector=MetricSelector(metric="latency"), + direction="minimize", + constraints=[ + MetricConstraint( + selector=MetricSelector(metric="correct"), + operator=operator, + value=threshold, + ) + ], + ) + + result = evaluate_objective(report(), specification) + + assert result.value == 10.0 + assert result.feasible is feasible + assert len(result.violations) == (0 if feasible else 1) + + +def make_record( + candidate_id: str, + value: float, + *, + feasible: bool = True, + direction: str = "minimize", + candidate_created_at: datetime | None = None, +) -> EvaluationRecord: + specification = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction=direction, + ) + created_at = datetime(2026, 1, 1, tzinfo=UTC) + return EvaluationRecord( + id=f"evaluation:{candidate_id}", + request=EvaluationRequest( + candidate=Candidate( + id=candidate_id, + version=f"version:{candidate_id}", + created_at=candidate_created_at or created_at, + ) + ), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": value}, + ), + backend_id="default", + backend=BackendProvenance( + name="fake", + version="1", + config_digest="0" * 64, + ), + objective_spec=specification, + objective=ObjectiveResult(value=value, feasible=feasible), + created_at=created_at, + completed_at=created_at + timedelta(seconds=1), + ) + + +def test_selection_respects_direction_feasibility_and_tie_breaks(): + assert ( + select_best_evaluation( + [ + make_record("a", 2.0, direction="maximize"), + make_record("b", 1.0, direction="maximize"), + ] + ).request.candidate.id + == "a" + ) + feasible = make_record("feasible", 100.0) + infeasible = make_record("infeasible", 1.0, feasible=False) + assert compare_evaluation_records(feasible, infeasible) > 0 + assert select_best_evaluation([infeasible]) is None + + created_at = datetime(2026, 1, 1, tzinfo=UTC) + assert ( + select_best_evaluation( + [ + make_record("b", 1.0, candidate_created_at=created_at), + make_record("a", 1.0, candidate_created_at=created_at), + ] + ).request.candidate.id + == "a" + ) + + +def test_disclosure_projections_exclude_case_details(): + record = make_record("a", 1.0) + aggregate = project_evaluation(record, DisclosureLevel.AGGREGATE) + hidden = project_evaluation(record, DisclosureLevel.NONE) + + assert isinstance(aggregate, EvaluationSummary) + assert aggregate.candidate_id == "a" + assert "cases" not in aggregate.model_dump() + assert isinstance(hidden, EvaluationAcknowledgement) + assert set(hidden.model_dump()) == {"evaluation_id", "status"} + + +class FakeBackend: + provenance = BackendProvenance( + name="fake", + version="1", + config_digest="0" * 64, + ) + + async def resolve_cost(self, evaluation_set: EvaluationSet) -> EvaluationCost: + return EvaluationCost() + + async def evaluate(self, *, context, request): + return EvaluationReport(status=EvaluationStatus.SUCCESS) + + +def test_backend_registry_is_explicit_and_rejects_unknown_ids(): + backend = FakeBackend() + registry = BackendRegistry({"trusted": backend}) + + assert registry.resolve("trusted") is backend + with pytest.raises(UnknownBackendError): + registry.resolve("missing") + with pytest.raises(ValueError, match="already registered"): + registry.register("trusted", backend) diff --git a/vero/tests/test_v05_evaluation_runtime.py b/vero/tests/test_v05_evaluation_runtime.py new file mode 100644 index 00000000..36793da4 --- /dev/null +++ b/vero/tests/test_v05_evaluation_runtime.py @@ -0,0 +1,1249 @@ +from __future__ import annotations + +import asyncio +import json +import threading +from contextlib import asynccontextmanager +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + +import vero.evaluation.store.budget as budget_module +import vero.evaluation.store.persistence as persistence +from vero.candidate import Candidate +from vero.evaluation import ( + AgentSelectionMode, + AllCases, + BackendProvenance, + BackendRegistry, + BudgetLedger, + CaseError, + CaseRange, + CaseResult, + CaseStatus, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationAuthorization, + EvaluationBudget, + EvaluationBudgetExceeded, + EvaluationCost, + EvaluationDatabase, + EvaluationDefinition, + EvaluationDeniedError, + EvaluationEngine, + EvaluationExecutionError, + EvaluationLimits, + EvaluationPlan, + EvaluationPrincipal, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + EvaluationStore, + Evaluator, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, + allow_all_evaluations, + authorize_evaluation_plan, +) +from vero.workspace import Workspace + + +class StubWorkspace(Workspace): + def __init__(self, root: Path, version: str = "main", *, dirty: bool = False): + root.mkdir(parents=True, exist_ok=True) + self._root = root + self._version = version + self._dirty = dirty + self.at_calls: list[str] = [] + self.copy_calls: list[str] = [] + + @property + def sandbox(self): + return None + + @property + def root(self) -> str: + return str(self._root) + + @property + def project_path(self) -> str: + return str(self._root) + + @property + def name(self) -> str: + return "workspace" + + async def current_version(self) -> str: + return self._version + + async def save(self, message: str = "Save") -> str: + return self._version + + async def restore(self, version_id: str, message: str | None = None) -> str: + self._version = version_id + return version_id + + async def diff(self, from_version=None, to_version=None) -> str: + return "" + + async def log(self, max_count=10, since_version=None) -> str: + return "" + + async def is_ancestor(self, version_a: str, version_b: str) -> bool: + return True + + async def copy(self, name=None, from_version=None): + return StubWorkspace(self._root, from_version or self._version) + + @asynccontextmanager + async def temp_copy(self, from_version=None): + version = from_version or self._version + self.copy_calls.append(version) + yield StubWorkspace(self._root, version) + + @asynccontextmanager + async def at(self, version_id: str): + previous = self._version + self.at_calls.append(version_id) + self._version = version_id + try: + yield + finally: + self._version = previous + + async def is_dirty(self) -> bool: + return self._dirty + + +class StubCandidateRepository: + family = "stub" + + def __init__(self, workspace: StubWorkspace): + self.workspace = workspace + self.checkout_calls: list[str] = [] + + @asynccontextmanager + async def checkout(self, candidate, *, sandbox, name=None): + self.checkout_calls.append(candidate.version) + yield StubWorkspace(self.workspace._root, candidate.version) + + +class StubBackend: + def __init__( + self, + *, + report: EvaluationReport | None = None, + cost: EvaluationCost | None = None, + error: Exception | None = None, + ): + self.report = report or EvaluationReport(status=EvaluationStatus.SUCCESS) + self.cost = cost or EvaluationCost(cases=0) + self.error = error + self.resolve_calls = 0 + self.evaluate_calls = 0 + self.running_manifests: list[dict] = [] + + @property + def provenance(self) -> BackendProvenance: + return BackendProvenance( + name="stub", + version="1", + config_digest="0" * 64, + ) + + async def resolve_cost(self, evaluation_set: EvaluationSet) -> EvaluationCost: + self.resolve_calls += 1 + return self.cost + + async def evaluate(self, *, context, request): + self.evaluate_calls += 1 + self.running_manifests.append( + json.loads((context.result_dir / "evaluation.json").read_text()) + ) + await context.case_store.save( + CaseResult(case_id="checkpoint", status=CaseStatus.SUCCESS) + ) + if self.error is not None: + raise self.error + return self.report + + +def request(version: str = "candidate") -> EvaluationRequest: + return EvaluationRequest( + candidate=Candidate( + id=f"id:{version}", + version=version, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ), + evaluation_set=EvaluationSet(name="performance"), + ) + + +def evaluator(tmp_path: Path, workspace: StubWorkspace) -> Evaluator: + runtime = Evaluator( + candidate_repository=StubCandidateRepository(workspace), + sandbox=workspace.sandbox, + session_dir=tmp_path / "sessions" / "session", + ) + return runtime + + +def record(candidate_id: str = "candidate") -> EvaluationRecord: + created_at = datetime(2026, 1, 1, tzinfo=UTC) + specification = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + return EvaluationRecord( + id=f"evaluation:{candidate_id}", + request=EvaluationRequest( + candidate=Candidate( + id=candidate_id, + version=f"version:{candidate_id}", + created_at=created_at, + ) + ), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 1.0}, + cases=[ + CaseResult( + case_id="case/with/path-characters", + status=CaseStatus.SUCCESS, + metrics={"score": 1.0}, + ) + ], + ), + backend_id="default", + backend=BackendProvenance( + name="stub", + version="1", + config_digest="0" * 64, + ), + objective_spec=specification, + objective=ObjectiveResult(value=1.0, feasible=True), + created_at=created_at, + completed_at=created_at + timedelta(seconds=1), + ) + + +@pytest.mark.asyncio +async def test_store_splits_cases_from_manifest_and_round_trips(tmp_path: Path): + value = record() + store = EvaluationStore(tmp_path / value.id) + + await store.save(value) + + manifest = json.loads(store.manifest_path.read_text()) + assert manifest["schema_version"] == 1 + assert manifest["lifecycle"] == "complete" + assert manifest["report"]["cases"] == [] + assert manifest["case_files"][0]["case_id"] == "case/with/path-characters" + assert "/" not in Path(manifest["case_files"][0]["path"]).name + assert store.load() == value + + +@pytest.mark.asyncio +async def test_store_preserves_principal_across_reconstruction(tmp_path: Path): + value = record().model_copy(update={"principal": EvaluationPrincipal.ADMIN}) + store = EvaluationStore(tmp_path / value.id) + + await store.save(value) + + # Reconstructing from the source-of-truth directory must keep the real + # provenance, not silently default it to SYSTEM. + assert store.load().principal == EvaluationPrincipal.ADMIN + assert store.load() == value + + +def test_running_manifest_is_not_a_completed_record(tmp_path: Path): + value = record() + store = EvaluationStore(tmp_path / value.id) + store.write_running( + evaluation_id=value.id, + request=value.request, + backend_id=value.backend_id, + backend=value.backend, + objective_spec=value.objective_spec, + created_at=value.created_at, + ) + + with pytest.raises(ValueError, match="invalid evaluation manifest"): + store.load() + + +def test_atomic_json_write_closes_descriptor_when_fdopen_fails( + tmp_path: Path, + monkeypatch, +): + closed = [] + real_close = persistence.os.close + + def fail_fdopen(*_args, **_kwargs): + raise RuntimeError("fdopen failed") + + def tracked_close(descriptor): + closed.append(descriptor) + real_close(descriptor) + + monkeypatch.setattr(persistence.os, "fdopen", fail_fdopen) + monkeypatch.setattr(persistence.os, "close", tracked_close) + + with pytest.raises(RuntimeError, match="fdopen failed"): + persistence._atomic_write_json(tmp_path / "value.json", {"value": 1}) + + assert len(closed) == 1 + assert not list(tmp_path.glob("*.tmp")) + + +def test_database_round_trips_schema_one_and_distinguishes_empty_filter(tmp_path: Path): + database = EvaluationDatabase(id="session") + value = record() + database.add_evaluation(value) + path = tmp_path / "database.json" + database.save_to_file(path) + + restored = EvaluationDatabase.load_from_file(path) + + assert restored.get_evaluations() == [value] + assert restored.get_evaluations([]) == [] + assert restored.get_best(value.objective_spec) == value + assert json.loads(path.read_text())["schema_version"] == 1 + + +@pytest.mark.asyncio +async def test_database_repairs_crash_window_from_completed_evaluations( + tmp_path: Path, +): + database_path = tmp_path / "database.json" + EvaluationDatabase(id="session").save_to_file(database_path) + value = record() + await EvaluationStore(tmp_path / "evaluations" / value.id).save(value) + + restored = EvaluationDatabase.load_reconciled( + database_path=database_path, + evaluations_dir=tmp_path / "evaluations", + database_id="session", + ) + + assert restored.get_evaluation(value.id) == value + assert ( + EvaluationDatabase.load_from_file(database_path).get_evaluation(value.id) + == value + ) + + +def test_database_reconciliation_ignores_running_evaluations(tmp_path: Path): + value = record() + store = EvaluationStore(tmp_path / "evaluations" / value.id) + store.write_running( + evaluation_id=value.id, + request=value.request, + backend_id=value.backend_id, + backend=value.backend, + objective_spec=value.objective_spec, + created_at=value.created_at, + ) + + restored = EvaluationDatabase.load_reconciled( + database_path=tmp_path / "database.json", + evaluations_dir=tmp_path / "evaluations", + database_id="session", + ) + + assert restored.evaluations == {} + + +def test_database_rejects_conflicting_candidate_identity(): + database = EvaluationDatabase(id="session") + database.add_evaluation(record("same")) + conflicting = record("other").model_copy( + update={ + "request": EvaluationRequest( + candidate=Candidate( + id="same", + version="different-version", + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + ) + } + ) + + with pytest.raises(ValueError, match="different identity"): + database.add_evaluation(conflicting) + + +@pytest.mark.asyncio +async def test_evaluator_runs_at_candidate_version_and_persists(tmp_path: Path): + workspace = StubWorkspace(tmp_path / "repo") + backend = StubBackend( + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"latency": 2.0}, + ) + ) + specification = ObjectiveSpec( + selector=MetricSelector(metric="latency"), + direction="minimize", + ) + + runtime = evaluator(tmp_path, workspace) + value = await runtime.evaluate( + backend_id="command", + backend=backend, + request=request(), + objective_spec=specification, + ) + + assert runtime.candidate_repository.checkout_calls == ["candidate"] + assert backend.running_manifests[0]["lifecycle"] == "running" + assert value.objective.value == 2.0 + result_dir = runtime.evaluations_dir / value.id + assert EvaluationStore(result_dir).load() == value + + +@pytest.mark.asyncio +async def test_evaluator_marks_reports_invalid_at_the_error_rate_threshold( + tmp_path: Path, +): + workspace = StubWorkspace(tmp_path / "repo") + cases = [ + CaseResult( + case_id=f"success-{index}", + status=CaseStatus.SUCCESS, + metrics={"score": 1.0}, + ) + for index in range(9) + ] + cases.append( + CaseResult( + case_id="error", + status=CaseStatus.ERROR, + errors=[CaseError(message="failed", terminal=True)], + ) + ) + backend = StubBackend( + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 1.0}, + cases=cases, + ) + ) + + value = await evaluator(tmp_path, workspace).evaluate( + backend_id="default", + backend=backend, + request=request(), + objective_spec=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + failure_value=0.0, + ), + ) + + assert value.report.status == EvaluationStatus.INVALID + assert value.report.metrics["error_rate"] == pytest.approx(0.1) + assert ( + value.report.diagnostics[-1].code + == "infrastructure_invalidity_threshold_exceeded" + ) + assert value.objective == ObjectiveResult(value=0.0, feasible=False) + + +@pytest.mark.asyncio +async def test_error_rate_threshold_can_be_disabled(tmp_path: Path): + workspace = StubWorkspace(tmp_path / "repo") + backend = StubBackend( + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 1.0, "error_rate": 1.0}, + ) + ) + evaluation_request = request().model_copy( + update={"limits": EvaluationLimits(error_rate_threshold=None)} + ) + + value = await evaluator(tmp_path, workspace).evaluate( + backend_id="default", + backend=backend, + request=evaluation_request, + ) + + assert value.report.status == EvaluationStatus.SUCCESS + + +@pytest.mark.asyncio +async def test_evaluator_uses_isolated_copy(tmp_path: Path): + workspace = StubWorkspace(tmp_path / "repo") + backend = StubBackend() + + runtime = evaluator(tmp_path, workspace) + await runtime.evaluate( + backend_id="default", + backend=backend, + request=request(), + ) + + assert runtime.candidate_repository.checkout_calls == ["candidate"] + assert workspace.copy_calls == [] + assert workspace.at_calls == [] + + +@pytest.mark.asyncio +async def test_backend_exception_is_recorded_then_raised(tmp_path: Path): + workspace = StubWorkspace(tmp_path / "repo") + backend = StubBackend(error=RuntimeError("boom")) + runtime = evaluator(tmp_path, workspace) + + with pytest.raises(EvaluationExecutionError) as captured: + await runtime.evaluate( + backend_id="default", + backend=backend, + request=request(), + ) + + failure = EvaluationStore( + runtime.evaluations_dir / captured.value.evaluation_id + ).load() + assert failure.report.status == EvaluationStatus.FAILED + assert failure.report.diagnostics[0].code == "backend_error" + + +@pytest.mark.asyncio +async def test_engine_indexes_a_durable_backend_exception(tmp_path: Path): + workspace = StubWorkspace(tmp_path / "repo") + backend = StubBackend(error=RuntimeError("boom")) + database = EvaluationDatabase(id="session") + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": backend}), + database=database, + database_path=tmp_path / "database.json", + authorization_resolver=allow_all_evaluations, + ) + + with pytest.raises(EvaluationExecutionError) as captured: + await engine.evaluate_record( + backend_id="default", + request=request(), + ) + + failure = database.get_evaluation(captured.value.evaluation_id) + assert failure is not None + assert failure.report.status == EvaluationStatus.FAILED + assert ( + EvaluationDatabase.load_from_file(tmp_path / "database.json").get_evaluation( + failure.id + ) + == failure + ) + + +@pytest.mark.asyncio +async def test_budget_ledger_reserves_and_restores(tmp_path: Path): + evaluation_set = EvaluationSet(name="performance") + path = tmp_path / "budget.json" + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="command", + evaluation_set_key=evaluation_set.budget_key("command"), + total_runs=2, + total_cases=10, + ) + ], + path=path, + ) + + remaining = await ledger.reserve( + "command", evaluation_set, EvaluationCost(runs=1, cases=7) + ) + + assert remaining.remaining_runs == 1 + assert remaining.remaining_cases == 3 + assert BudgetLedger.load(path).get("command", evaluation_set) == remaining + + with pytest.raises(EvaluationBudgetExceeded): + await ledger.reserve("command", evaluation_set, EvaluationCost(runs=1, cases=4)) + + +@pytest.mark.asyncio +async def test_budget_reservation_rolls_back_when_cancelled( + tmp_path: Path, + monkeypatch, +): + evaluation_set = EvaluationSet(name="performance") + path = tmp_path / "budget.json" + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="command", + evaluation_set_key=evaluation_set.budget_key("command"), + total_runs=2, + ) + ], + path=path, + ) + ledger.save() + started = threading.Event() + release = threading.Event() + real_write = budget_module._atomic_write_json + + def delayed_write(write_path, value): + started.set() + assert release.wait(timeout=5) + real_write(write_path, value) + + monkeypatch.setattr(budget_module, "_atomic_write_json", delayed_write) + reservation = asyncio.create_task( + ledger.reserve("command", evaluation_set, EvaluationCost()) + ) + assert await asyncio.to_thread(started.wait, 5) + reservation.cancel() + release.set() + + with pytest.raises(asyncio.CancelledError): + await reservation + + # The reservation is atomic: a run cancelled while the charge was being + # written keeps its full budget rather than leaking it, and disk and memory + # agree on the rolled-back state. + in_memory = ledger.get("command", evaluation_set) + on_disk = BudgetLedger.load(path).get("command", evaluation_set) + assert in_memory is not None + assert in_memory.remaining_runs == 2 + assert on_disk == in_memory + + +@pytest.mark.asyncio +async def test_reserve_rollback_write_failure_preserves_cancellation( + tmp_path: Path, + monkeypatch, +): + # If the rollback write itself fails, the run's cancellation must still + # propagate (not be replaced by the write's OSError), with the write error + # chained for diagnosis. + evaluation_set = EvaluationSet(name="performance") + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="command", + evaluation_set_key=evaluation_set.budget_key("command"), + total_runs=2, + ) + ], + path=tmp_path / "budget.json", + ) + ledger.save() + started = threading.Event() + release = threading.Event() + real_write = budget_module._atomic_write_json + calls = {"n": 0} + + def failing_rollback_write(write_path, value): + calls["n"] += 1 + if calls["n"] == 1: + # the charge write: block so the reservation can be cancelled here + started.set() + assert release.wait(timeout=5) + real_write(write_path, value) + else: + # the rollback write fails durably + raise OSError("disk gone") + + monkeypatch.setattr(budget_module, "_atomic_write_json", failing_rollback_write) + reservation = asyncio.create_task( + ledger.reserve("command", evaluation_set, EvaluationCost()) + ) + assert await asyncio.to_thread(started.wait, 5) + reservation.cancel() + release.set() + + with pytest.raises(asyncio.CancelledError) as excinfo: + await reservation + assert isinstance(excinfo.value.__cause__, OSError) + + +@pytest.mark.asyncio +async def test_reserve_charge_write_failure_preserves_cancellation( + tmp_path: Path, + monkeypatch, +): + # The third of the three durable writes in this module, and the one that + # was still hand-rolling the drain loop: if the charge write itself fails + # while the run is being cancelled, the caller must still see the + # cancellation. Returning OSError instead breaks structured cancellation + # and hides that the run went away. + evaluation_set = EvaluationSet(name="performance") + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="command", + evaluation_set_key=evaluation_set.budget_key("command"), + total_runs=2, + ) + ], + path=tmp_path / "budget.json", + ) + started = threading.Event() + release = threading.Event() + + def blocking_failing_write(write_path, value): + started.set() + assert release.wait(timeout=5) + raise OSError("disk gone") + + monkeypatch.setattr(budget_module, "_atomic_write_json", blocking_failing_write) + reservation = asyncio.create_task( + ledger.reserve("command", evaluation_set, EvaluationCost(runs=1)) + ) + assert await asyncio.to_thread(started.wait, 5) + reservation.cancel() + release.set() + + with pytest.raises(asyncio.CancelledError) as excinfo: + await reservation + assert isinstance(excinfo.value.__cause__, OSError) + # The charge never landed, so the budget is untouched on both sides. + assert ledger.get("command", evaluation_set).remaining_runs == 2 + + +@pytest.mark.asyncio +async def test_refund_write_failure_preserves_cancellation(tmp_path: Path, monkeypatch): + # A cancellation racing the durable refund write must win over the write's + # own failure rather than being swallowed. + evaluation_set = EvaluationSet(name="performance") + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="command", + evaluation_set_key=evaluation_set.budget_key("command"), + total_runs=2, + ) + ], + path=tmp_path / "budget.json", + ) + await ledger.reserve("command", evaluation_set, EvaluationCost(runs=1)) + started = threading.Event() + release = threading.Event() + + def blocking_failing_write(write_path, value): + started.set() + assert release.wait(timeout=5) + raise OSError("disk gone") + + monkeypatch.setattr(budget_module, "_atomic_write_json", blocking_failing_write) + refund = asyncio.create_task( + ledger.refund("command", evaluation_set, EvaluationCost(runs=1)) + ) + assert await asyncio.to_thread(started.wait, 5) + refund.cancel() + release.set() + + with pytest.raises(asyncio.CancelledError) as excinfo: + await refund + assert isinstance(excinfo.value.__cause__, OSError) + + +@pytest.mark.asyncio +async def test_execution_error_refunds_reservation(tmp_path: Path): + # A backend exception (EvaluationExecutionError) must refund the reservation + # via the shielded cleanup path, restoring the budget in memory and on disk. + workspace = StubWorkspace(tmp_path / "repo") + backend = StubBackend(error=RuntimeError("boom")) + evaluation_set = request().evaluation_set + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="default", + evaluation_set_key=evaluation_set.budget_key("default"), + total_runs=2, + ) + ], + path=tmp_path / "budgets.json", + ) + ledger.save() + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": backend}), + database=EvaluationDatabase(id="session"), + database_path=tmp_path / "database.json", + budget_ledger=ledger, + authorization_resolver=allow_all_evaluations, + ) + + with pytest.raises(EvaluationExecutionError): + await engine.evaluate_record(backend_id="default", request=request()) + + remaining = ledger.get("default", evaluation_set) + assert remaining is not None and remaining.remaining_runs == 2 + assert ( + BudgetLedger.load(tmp_path / "budgets.json") + .get("default", evaluation_set) + .remaining_runs + == 2 + ) + + +@pytest.mark.asyncio +async def test_cancelled_evaluation_is_terminal_indexed_and_refunded(tmp_path: Path): + class BlockingBackend(StubBackend): + async def evaluate(self, *, context, request): + self.evaluate_calls += 1 + await asyncio.Event().wait() + + workspace = StubWorkspace(tmp_path / "repo") + backend = BlockingBackend() + evaluation_set = request().evaluation_set + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="default", + evaluation_set_key=evaluation_set.budget_key("default"), + total_runs=1, + ) + ], + path=tmp_path / "budgets.json", + ) + ledger.save() + database = EvaluationDatabase(id="session") + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": backend}), + database=database, + database_path=tmp_path / "database.json", + budget_ledger=ledger, + authorization_resolver=allow_all_evaluations, + ) + evaluation = asyncio.create_task( + engine.evaluate_record( + backend_id="default", + request=request(), + ) + ) + while backend.evaluate_calls == 0: + await asyncio.sleep(0) + evaluation.cancel() + + with pytest.raises(asyncio.CancelledError): + await evaluation + + assert len(database.evaluations) == 1 + cancelled = next(iter(database.evaluations.values())) + assert cancelled.report.status == EvaluationStatus.CANCELLED + assert cancelled.report.diagnostics[0].code == "evaluation_cancelled" + assert ( + EvaluationStore( + evaluator(tmp_path, workspace).evaluations_dir / cancelled.id + ).load() + == cancelled + ) + remaining = ledger.get("default", evaluation_set) + assert remaining is not None + assert remaining.remaining_runs == 1 + + +@pytest.mark.asyncio +async def test_finalization_drains_admitted_agent_evaluations_and_closes_admission( + tmp_path: Path, +): + class BlockingBackend(StubBackend): + def __init__(self): + super().__init__( + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 0.75}, + ) + ) + self.started = asyncio.Event() + self.release = asyncio.Event() + + async def evaluate(self, *, context, request): + self.evaluate_calls += 1 + self.started.set() + await self.release.wait() + return self.report + + workspace = StubWorkspace(tmp_path / "repo") + backend = BlockingBackend() + database = EvaluationDatabase(id="session") + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": backend}), + database=database, + authorization_resolver=allow_all_evaluations, + ) + evaluation = asyncio.create_task( + engine.evaluate_record(backend_id="default", request=request()) + ) + await backend.started.wait() + + drain = asyncio.create_task(engine.quiesce_agent_evaluations(timeout_seconds=1.0)) + await asyncio.sleep(0) + assert not drain.done() + with pytest.raises(EvaluationDeniedError, match="finalization has started"): + await engine.evaluate_record(backend_id="default", request=request("late")) + + backend.release.set() + + assert await drain == 1 + assert (await evaluation).report.metrics == {"score": 0.75} + assert len(database.evaluations) == 1 + admin = await engine.evaluate_record( + backend_id="default", + request=request("admin"), + principal=EvaluationPrincipal.ADMIN, + ) + assert admin.request.candidate.version == "admin" + + +@pytest.mark.asyncio +async def test_finalization_cancels_an_agent_evaluation_after_drain_timeout( + tmp_path: Path, +): + class BlockingBackend(StubBackend): + def __init__(self): + super().__init__() + self.started = asyncio.Event() + + async def evaluate(self, *, context, request): + self.evaluate_calls += 1 + self.started.set() + await asyncio.Event().wait() + + workspace = StubWorkspace(tmp_path / "repo") + backend = BlockingBackend() + database = EvaluationDatabase(id="session") + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": backend}), + database=database, + authorization_resolver=allow_all_evaluations, + ) + evaluation = asyncio.create_task( + engine.evaluate_record(backend_id="default", request=request()) + ) + await backend.started.wait() + + assert ( + await engine.quiesce_agent_evaluations( + timeout_seconds=0.01, + cancellation_grace_seconds=1.0, + ) + == 1 + ) + with pytest.raises(asyncio.CancelledError): + await evaluation + + assert len(database.evaluations) == 1 + cancelled = next(iter(database.evaluations.values())) + assert cancelled.report.status == EvaluationStatus.CANCELLED + + +@pytest.mark.asyncio +async def test_cancellation_during_the_success_record_still_completes_it( + tmp_path: Path, +): + """A charged run must finish being recorded even if the caller goes away. + + The success path was the one _record call not shielded, so a cancellation + arriving while it ran abandoned it part-way: the budget stayed charged for + an evaluation the engine never finished publishing. + + The cancellable window is the listener loop, not the database write -- + asyncio.to_thread cannot be interrupted, so that part completes either way. + Listeners are where the trusted side mirrors an evaluation to W&B and the + session archive, so silently skipping them loses the run from both. + """ + workspace = StubWorkspace(tmp_path / "repo") + database = EvaluationDatabase(id="session") + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": StubBackend()}), + database=database, + authorization_resolver=allow_all_evaluations, + ) + entered = asyncio.Event() + release = asyncio.Event() + finished = asyncio.Event() + + async def slow_listener(record: EvaluationRecord) -> None: + entered.set() + await release.wait() + finished.set() + + engine.listeners.append(slow_listener) + evaluation = asyncio.create_task( + engine.evaluate_record(backend_id="default", request=request()) + ) + await asyncio.wait_for(entered.wait(), timeout=5) + + evaluation.cancel() + with pytest.raises(asyncio.CancelledError): + await evaluation + # shield lets the caller unwind at once while _record finishes in the + # background, so wait for the listener rather than assuming it ran. + release.set() + await asyncio.wait_for(finished.wait(), timeout=5) + assert len(database.evaluations) == 1 + + +@pytest.mark.asyncio +async def test_plan_authorization_separates_agent_and_system_selection_rights( + tmp_path: Path, +): + canonical = EvaluationSet( + name="validation", + selection=CaseRange(stop=10), + ) + plan = EvaluationPlan( + evaluations=[ + EvaluationDefinition( + evaluation_set=canonical, + access=EvaluationAccessPolicy( + agent_selection=AgentSelectionMode.FIXED, + disclosure=DisclosureLevel.AGGREGATE, + ), + ) + ], + selection_evaluation="validation", + ) + workspace = StubWorkspace(tmp_path / "repo") + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": StubBackend()}), + database=EvaluationDatabase(id="session"), + authorization_resolver=authorize_evaluation_plan(plan), + ) + subset_request = request().model_copy( + update={ + "evaluation_set": canonical.model_copy( + update={"selection": CaseRange(stop=2)} + ) + } + ) + + agent = await engine.authorize( + "default", + subset_request, + EvaluationPrincipal.AGENT, + ) + system = await engine.authorize( + "default", + subset_request, + EvaluationPrincipal.SYSTEM, + ) + + assert agent.may_evaluate is False + assert agent.viewable is True + assert system.may_evaluate is True + + +@pytest.mark.asyncio +async def test_engine_denial_stops_before_cost_and_evaluation(tmp_path: Path): + workspace = StubWorkspace(tmp_path / "repo") + backend = StubBackend() + database = EvaluationDatabase(id="session") + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"default": backend}), + database=database, + ) + + with pytest.raises(EvaluationDeniedError, match="private set"): + await engine.evaluate( + backend_id="default", + request=request(), + authorization=EvaluationAuthorization( + may_evaluate=False, + reason="private set", + ), + ) + + assert backend.resolve_calls == 0 + assert backend.evaluate_calls == 0 + assert database.evaluations == {} + + +@pytest.mark.asyncio +async def test_engine_denies_by_default_without_authorization(tmp_path: Path): + backend = StubBackend() + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, StubWorkspace(tmp_path / "repo")), + backends=BackendRegistry({"default": backend}), + database=EvaluationDatabase(id="session"), + ) + + with pytest.raises( + EvaluationDeniedError, + match="authorization was not configured", + ): + await engine.evaluate_record(backend_id="default", request=request()) + + assert backend.resolve_calls == 0 + assert backend.evaluate_calls == 0 + + +@pytest.mark.asyncio +async def test_engine_selects_backend_persists_and_projects(tmp_path: Path): + workspace = StubWorkspace(tmp_path / "repo") + first = StubBackend( + report=EvaluationReport(status=EvaluationStatus.SUCCESS, metrics={"value": 1}) + ) + second = StubBackend( + report=EvaluationReport(status=EvaluationStatus.SUCCESS, metrics={"value": 2}) + ) + database = EvaluationDatabase(id="session") + database_path = tmp_path / "database.json" + engine = EvaluationEngine( + evaluator=evaluator(tmp_path, workspace), + backends=BackendRegistry({"first": first, "second": second}), + database=database, + database_path=database_path, + ) + + summary = await engine.evaluate( + backend_id="second", + request=request(), + authorization=EvaluationAuthorization( + may_evaluate=True, + meter_budget=False, + disclosure=DisclosureLevel.AGGREGATE, + ), + ) + + assert first.resolve_calls == 0 + assert second.resolve_calls == 1 + assert summary.metrics == {"value": 2.0} + assert len(database.evaluations) == 1 + assert database_path.exists() + + +@pytest.mark.asyncio +async def test_engine_enforces_aggregate_k_anonymity_floor(tmp_path: Path): + """The floor is a core-engine property, not a sidecar courtesy.""" + canonical = EvaluationSet(name="validation", selection=CaseRange(stop=10)) + plan = EvaluationPlan( + evaluations=[ + EvaluationDefinition( + evaluation_set=canonical, + # min_aggregate_cases omitted: resolves to the safe floor (5). + access=EvaluationAccessPolicy(disclosure=DisclosureLevel.AGGREGATE), + ) + ], + selection_evaluation="validation", + ) + + def engine_with(cost: EvaluationCost) -> EvaluationEngine: + return EvaluationEngine( + evaluator=evaluator(tmp_path, StubWorkspace(tmp_path / "repo")), + backends=BackendRegistry({"default": StubBackend(cost=cost)}), + database=EvaluationDatabase(id="session"), + authorization_resolver=authorize_evaluation_plan(plan), + ) + + def request_with(selection) -> EvaluationRequest: + return request().model_copy( + update={ + "evaluation_set": canonical.model_copy(update={"selection": selection}) + } + ) + + # 1) An agent-chosen 2-case aggregate subset is refused by the engine. + with pytest.raises(EvaluationDeniedError, match="at least 5 cases"): + await engine_with(EvaluationCost(cases=2)).evaluate( + backend_id="default", + request=request_with(CaseRange(stop=2)), + principal=EvaluationPrincipal.AGENT, + ) + + # 2) A subset without exact case costs cannot be floored: refused. + with pytest.raises(EvaluationDeniedError, match="exact case costs"): + await engine_with(EvaluationCost(cases=None)).evaluate( + backend_id="default", + request=request_with(CaseRange(stop=2)), + principal=EvaluationPrincipal.AGENT, + ) + + # 3) A subset at the floor passes. + await engine_with(EvaluationCost(cases=5)).evaluate( + backend_id="default", + request=request_with(CaseRange(stop=5)), + principal=EvaluationPrincipal.AGENT, + ) + + # 4) The complete selection is exempt: its aggregate is the intended + # disclosure, however small the set. + await engine_with(EvaluationCost(cases=2)).evaluate( + backend_id="default", + request=request_with(AllCases()), + principal=EvaluationPrincipal.AGENT, + ) + + # 5) Trusted principals are never floored. + await engine_with(EvaluationCost(cases=2)).evaluate( + backend_id="default", + request=request_with(CaseRange(stop=2)), + principal=EvaluationPrincipal.ADMIN, + ) + + +@pytest.mark.asyncio +async def test_raw_cancellation_during_cleanup_still_refunds(tmp_path: Path): + """A CancelledError that escapes the evaluator must not leak the reservation. + + The evaluator turns cancellation and failure into two typed errors, but each + has to await a persist before it can raise them. A cancellation delivered + during that await -- which quiesce_agent_evaluations does deliver, when it + drains agent evaluations at finalization -- unwinds as a raw CancelledError + instead, matching neither typed handler. Without a bare handler in the engine + the reservation stays charged forever. + """ + + class CancellingEvaluator: + """Stands in for an evaluator interrupted mid-cleanup.""" + + def __init__(self, evaluations_dir): + self.evaluations_dir = evaluations_dir + + async def evaluate(self, **_kwargs): + raise asyncio.CancelledError() + + workspace = StubWorkspace(tmp_path / "repo") + evaluation_set = request().evaluation_set + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="default", + evaluation_set_key=evaluation_set.budget_key("default"), + total_runs=2, + ) + ], + path=tmp_path / "budgets.json", + ) + ledger.save() + engine = EvaluationEngine( + evaluator=CancellingEvaluator(evaluator(tmp_path, workspace).evaluations_dir), + backends=BackendRegistry({"default": StubBackend()}), + database=EvaluationDatabase(id="session"), + database_path=tmp_path / "database.json", + budget_ledger=ledger, + authorization_resolver=allow_all_evaluations, + ) + + # The cancellation still propagates -- it is not swallowed. + with pytest.raises(asyncio.CancelledError): + await engine.evaluate_record(backend_id="default", request=request()) + + remaining = ledger.get("default", evaluation_set) + assert remaining is not None and remaining.remaining_runs == 2 + assert ( + BudgetLedger.load(tmp_path / "budgets.json") + .get("default", evaluation_set) + .remaining_runs + == 2 + ) diff --git a/vero/tests/test_v05_evaluation_tools.py b/vero/tests/test_v05_evaluation_tools.py new file mode 100644 index 00000000..b0ab0721 --- /dev/null +++ b/vero/tests/test_v05_evaluation_tools.py @@ -0,0 +1,105 @@ +from __future__ import annotations + +import json + +import pytest + +from vero.evaluation import ( + EvaluationAcknowledgement, + EvaluationBudget, + EvaluationBudgetExceeded, + EvaluationSet, + EvaluationStatus, +) +from vero.tools.evaluation import EvaluationTools +from vero.tools.utils import get_tools_from_class + + +class StubEvaluationGateway: + def __init__(self): + self.descriptions: list[str] = [] + self.remaining_runs = 2 + + async def evaluate( + self, + *, + evaluation, + selection=None, + candidate_id=None, + description="Evaluate agent checkpoint", + ): + assert evaluation == "validation" + assert selection is None + assert candidate_id is None + self.descriptions.append(description) + self.remaining_runs -= 1 + return EvaluationAcknowledgement( + evaluation_id="evaluation-1", + status=EvaluationStatus.SUCCESS, + ) + + def budgets(self): + return { + "validation": EvaluationBudget( + backend_id="command", + evaluation_set_key=EvaluationSet(name="validation").budget_key( + "command" + ), + total_runs=2, + remaining_runs=self.remaining_runs, + ) + } + + +@pytest.mark.asyncio +async def test_evaluation_tools_expose_only_scoped_feedback_and_budget(): + gateway = StubEvaluationGateway() + tools = EvaluationTools(evaluation=gateway) + + result = json.loads( + await tools.evaluate( + evaluation="validation", + description="Try vectorized implementation", + ) + ) + budget = json.loads(tools.get_evaluation_budgets()) + + assert result == { + "evaluation_id": "evaluation-1", + "status": "success", + } + assert gateway.descriptions == ["Try vectorized implementation"] + assert budget["validation"]["remaining_runs"] == 1 + assert {tool.__name__ for tool in get_tools_from_class(tools)} == { + "evaluate", + "get_evaluation_budgets", + } + + +@pytest.mark.asyncio +async def test_evaluate_returns_clean_result_when_evaluation_budget_is_spent(): + class ExhaustedGateway(StubEvaluationGateway): + async def evaluate(self, **_kwargs): + raise EvaluationBudgetExceeded("evaluation run budget exhausted") + + result = json.loads( + await EvaluationTools(evaluation=ExhaustedGateway()).evaluate( + evaluation="validation", + ) + ) + + # A benign, non-terminating signal the agent can absorb — not a raised error. + assert result["status"] == "evaluation_budget_exhausted" + assert "budget" in result["message"] + + +def test_evaluation_tools_report_unmetered_and_require_binding(): + class UnmeteredGateway(StubEvaluationGateway): + def budgets(self): + return {"validation": None} + + assert json.loads( + EvaluationTools(evaluation=UnmeteredGateway()).get_evaluation_budgets() + ) == {"validation": None} + with pytest.raises(RuntimeError, match="not bound"): + EvaluationTools().get_evaluation_budgets() diff --git a/vero/tests/test_v05_evolutionary_strategy.py b/vero/tests/test_v05_evolutionary_strategy.py new file mode 100644 index 00000000..2a329602 --- /dev/null +++ b/vero/tests/test_v05_evolutionary_strategy.py @@ -0,0 +1,119 @@ +from __future__ import annotations + +import pytest + +from vero.candidate import Candidate +from vero.evaluation import ( + EvaluationSet, + EvaluationStatus, + EvaluationSummary, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, +) +from vero.optimization import EvolutionaryStrategy +from vero.optimization.models import OptimizationContext + +OBJ_MAX = ObjectiveSpec(selector=MetricSelector(metric="score"), direction="maximize") + + +def _summary(candidate_id: str, value: float | None, *, feasible: bool = True) -> EvaluationSummary: + return EvaluationSummary( + evaluation_id=f"eval-{candidate_id}", + candidate_id=candidate_id, + candidate_version=f"{candidate_id}-v", + backend_id="cmd", + evaluation_set=EvaluationSet(name="perf"), + status=EvaluationStatus.SUCCESS, + metrics={"score": value} if value is not None else {}, + objective=ObjectiveResult(value=value, feasible=feasible), + total_cases=1, + successful_cases=1, + errored_cases=0, + skipped_cases=0, + ) + + +def _context(*, candidates, evaluations=(), best=None, baseline=None) -> OptimizationContext: + baseline = baseline or next(iter(candidates.values())) + return OptimizationContext( + session_id="s", + round=0, + workspace=object(), # unused by the strategy + baseline=baseline, + evaluations=tuple(evaluations), + candidates=candidates, + best=best, + ) + + +@pytest.mark.asyncio +async def test_seeds_offspring_from_baseline_before_any_feasible_candidate(): + strat = EvolutionaryStrategy(objective=OBJ_MAX, num_offspring=3, seed=0) + baseline = Candidate(id="base", version="base-v") + ctx = _context(candidates={"base": baseline}, baseline=baseline) + + proposals = await strat.propose(ctx) + + assert len(proposals) == 3 + assert all(p.parent_id == "base" for p in proposals) + assert len({p.id for p in proposals}) == 3 # unique proposal ids + assert all(p.metadata["strategy"] == "evolutionary" for p in proposals) + assert all(p.metadata["generation"] == 0 for p in proposals) + + +@pytest.mark.asyncio +async def test_selects_parents_from_the_fittest_population(): + strat = EvolutionaryStrategy( + objective=OBJ_MAX, + num_offspring=6, + population_size=2, + tournament_size=1, # each pick is a uniform draw from the top-2 + seed=0, + ) + candidates = {cid: Candidate(id=cid, version=f"{cid}-v") for cid in ["base", "a", "b", "c"]} + evaluations = ( + _summary("base", 0.10), + _summary("a", 0.90), + _summary("b", 0.80), + _summary("c", 0.20), + _summary("d", None, feasible=False), # infeasible: excluded from the population + ) + ctx = _context( + candidates=candidates, + evaluations=evaluations, + best=candidates["a"], + baseline=candidates["base"], + ) + + proposals = await strat.propose(ctx) + + assert len(proposals) == 6 + # Population is the top-2 by value: {a: 0.9, b: 0.8}; the weaker/infeasible ones + # and the unknown "d" never become parents. + assert {p.parent_id for p in proposals} <= {"a", "b"} + assert "c" not in {p.parent_id for p in proposals} + + +@pytest.mark.asyncio +async def test_minimize_direction_prefers_lower_values(): + strat = EvolutionaryStrategy( + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), direction="minimize" + ), + num_offspring=5, + population_size=1, + tournament_size=1, + seed=0, + ) + candidates = {cid: Candidate(id=cid, version=f"{cid}-v") for cid in ["hi", "lo"]} + ctx = _context( + candidates=candidates, + evaluations=(_summary("hi", 9.0), _summary("lo", 1.0)), + best=candidates["lo"], + ) + + proposals = await strat.propose(ctx) + + # Under minimize, the single-slot population is the lowest value ("lo"). + assert {p.parent_id for p in proposals} == {"lo"} diff --git a/vero/tests/test_v05_harbor_backend.py b/vero/tests/test_v05_harbor_backend.py new file mode 100644 index 00000000..1883eac1 --- /dev/null +++ b/vero/tests/test_v05_harbor_backend.py @@ -0,0 +1,1223 @@ +from __future__ import annotations + +import json +import sys +from datetime import UTC, datetime +from pathlib import Path, PurePosixPath +from types import ModuleType, SimpleNamespace + +import pytest + +from vero.candidate import Candidate +from vero.evaluation import ( + CaseCheckpointStore, + CaseIds, + CaseRange, + CaseStatus, + EvaluationContext, + EvaluationLimits, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + RetryPolicy, +) +from vero.harbor import HarborBackend, HarborBackendConfig +from vero.layout import LAYOUT +from vero.sandbox import CommandResult, LocalSandbox + + +def _cases(path: Path) -> Path: + path.write_text( + json.dumps( + [ + {"id": "case-a", "task_name": "example/alpha"}, + {"id": "case-b", "task_name": "example/beta"}, + {"id": "case-c", "task_name": "example/gamma"}, + ] + ), + encoding="utf-8", + ) + return path + + +def _config(tmp_path: Path, **updates) -> HarborBackendConfig: + values = { + "task_source": "example/tasks@1.0", + "agent_import_path": "candidate.agent:Agent", + "cases_path": str(_cases(tmp_path / "cases.json")), + "harbor_requirement": "harbor==0.1.17", + "evaluation_set_name": "harbor-bench", + "partition": "test", + "uv_executable": sys.executable, + "infrastructure_max_attempts": 1, + "infrastructure_retry_delay_seconds": 0, + } + values.update(updates) + return HarborBackendConfig(**values) + + +def _request(selection=None) -> EvaluationRequest: + return EvaluationRequest( + candidate=Candidate( + id="candidate", + version="version", + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ), + evaluation_set=EvaluationSet( + name="harbor-bench", + partition="test", + **({"selection": selection} if selection is not None else {}), + ), + limits=EvaluationLimits( + timeout_seconds=90, + max_concurrency=7, + retry=RetryPolicy.disabled(), + ), + ) + + +def test_backend_accepts_pinned_environment_extra(tmp_path): + config = _config( + tmp_path, + harbor_requirement="harbor[modal]==0.20.0", + ) + + assert config.harbor_requirement == "harbor[modal]==0.20.0" + + +def test_environment_default_routes_agent_through_gateway_on_openai(tmp_path): + config = _config( + tmp_path, + inference_gateway_url="http://inference-gateway:8001", + inference_gateway_token="gw-token", + ) + env = HarborBackend(config)._environment("eval-1") + + assert env["OPENAI_API_KEY"] == "gw-token" + assert env["OPENAI_BASE_URL"] == ( + "http://inference-gateway:8001/scopes/evaluation/eval-1/v1" + ) + # litellm-style alias used by task verifier env templates + assert env["OPENAI_API_BASE"] == env["OPENAI_BASE_URL"] + assert "VERO_AGENT_INFERENCE_API_KEY" not in env + + +def test_environment_routes_task_services_to_upstream(tmp_path, monkeypatch): + monkeypatch.setenv("VERO_INFERENCE_UPSTREAM_API_KEY", "upstream-key") + monkeypatch.setenv("VERO_INFERENCE_UPSTREAM_BASE_URL", "https://upstream/v1") + config = _config( + tmp_path, + inference_gateway_url="http://inference-gateway:8001", + inference_gateway_token="gw-token", + task_services_use_upstream=True, + upstream_api_key_env="VERO_INFERENCE_UPSTREAM_API_KEY", + upstream_base_url_env="VERO_INFERENCE_UPSTREAM_BASE_URL", + ) + env = HarborBackend(config)._environment("eval-1") + + # task-owned eval services reach the real upstream via OPENAI_* + assert env["OPENAI_API_KEY"] == "upstream-key" + assert env["OPENAI_BASE_URL"] == "https://upstream/v1" + assert env["OPENAI_API_BASE"] == "https://upstream/v1" + # the candidate agent keeps the metered gateway on dedicated vars + assert env["VERO_AGENT_INFERENCE_API_KEY"] == "gw-token" + assert env["VERO_AGENT_INFERENCE_BASE_URL"] == ( + "http://inference-gateway:8001/scopes/evaluation/eval-1/v1" + ) + + +def test_task_services_use_upstream_requires_gateway_and_upstream_env(tmp_path): + with pytest.raises(ValueError, match="requires an inference gateway"): + _config(tmp_path, task_services_use_upstream=True) + with pytest.raises(ValueError, match="requires upstream_api_key_env"): + _config( + tmp_path, + task_services_use_upstream=True, + inference_gateway_url="http://inference-gateway:8001", + inference_gateway_token="gw-token", + ) + + +@pytest.mark.parametrize( + ("request_factory", "message"), + [ + ( + lambda: _request().model_copy( + update={ + "limits": _request().limits.model_copy( + update={"retry": RetryPolicy(max_attempts=2)} + ) + } + ), + "generic per-case retries", + ), + ( + lambda: _request().model_copy( + update={ + "limits": _request().limits.model_copy( + update={"case_timeout_seconds": 30} + ) + } + ), + "case timeout is fixed by the backend", + ), + (lambda: _request().model_copy(update={"seed": 7}), "evaluation seed"), + ], +) +def test_harbor_backend_rejects_unsupported_generic_controls( + tmp_path, request_factory, message +): + backend = HarborBackend(_config(tmp_path)) + + with pytest.raises(ValueError, match=message): + backend.validate_request(request_factory()) + + +class FakeSandbox(LocalSandbox): + def __init__( + self, + root: Path, + trials: dict[str, list[dict]], + result: CommandResult | None = None, + ): + super().__init__(root) + self.trials = trials + self.result = result or CommandResult("harbor output", "", 0) + self.command = None + self.cwd = None + self.timeout = None + self.env = None + self.run_as = None + self.chown_commands: list[list[str]] = [] + self.chmod_commands: list[list[str]] = [] + self.probe_commands: list[tuple[list[str], str | None]] = [] + + async def run(self, command, cwd=None, timeout=30, env=None, run_as=None): + if isinstance(command, list) and command[:1] == ["chown"]: + # Record the harness-isolation provisioning without touching real uids. + self.chown_commands.append(command) + return CommandResult("", "", 0) + if isinstance(command, list) and command[:1] == ["chmod"]: + self.chmod_commands.append(command) + return CommandResult("", "", 0) + if isinstance(command, list) and command[:1] == ["test"]: + # The workspace-reachability probe, run as the dropped user. + self.probe_commands.append((command, run_as)) + return CommandResult("", "", 0) + if not isinstance(command, list) or "--jobs-dir" not in command: + return await super().run( + command, cwd=cwd, timeout=timeout, env=env, run_as=run_as + ) + self.command = command + self.cwd = cwd + self.timeout = timeout + self.env = env + self.run_as = run_as + jobs_dir = Path(command[command.index("--jobs-dir") + 1]) + for task_name, attempts in self.trials.items(): + for index, attempt in enumerate(attempts): + trial_dir = ( + jobs_dir / f"job-{index}" / f"trial-{task_name.split('/')[-1]}" + ) + trial_dir.mkdir(parents=True, exist_ok=True) + payload = { + "task_name": task_name, + "trial_name": f"trial-{index}", + "finished_at": f"2026-01-01T00:00:0{index}Z", + **attempt, + } + agent_files = payload.pop("_agent_files", {}) + (trial_dir / "result.json").write_text(json.dumps(payload)) + for relative_path, content in agent_files.items(): + path = trial_dir / "agent" / relative_path + path.parent.mkdir(parents=True, exist_ok=True) + if isinstance(content, bytes): + path.write_bytes(content) + else: + path.write_text(content, encoding="utf-8") + return self.result + + +async def _context( + tmp_path: Path, sandbox: FakeSandbox, *, finalization: bool = False +) -> EvaluationContext: + target = tmp_path / "target" + target.mkdir(parents=True, exist_ok=True) + result_dir = tmp_path / "result" + artifact_dir = result_dir / "artifacts" + artifact_dir.mkdir(parents=True) + return EvaluationContext( + workspace=SimpleNamespace( + project_path=str(target), root=str(target), sandbox=sandbox + ), + session_id="session", + evaluation_id="evaluation", + result_dir=result_dir, + artifact_dir=artifact_dir, + case_store=CaseCheckpointStore(result_dir / "cases"), + finalization=finalization, + ) + + +@pytest.mark.asyncio +async def test_harbor_backend_resolves_canonical_case_selections(tmp_path): + backend = HarborBackend(_config(tmp_path)) + evaluation_set = EvaluationSet(name="harbor-bench", partition="test") + + assert (await backend.resolve_cost(evaluation_set)).cases == 3 + assert ( + await backend.resolve_cost( + evaluation_set.model_copy(update={"selection": CaseRange(start=1, stop=3)}) + ) + ).cases == 2 + assert ( + await backend.resolve_cost( + evaluation_set.model_copy( + update={"selection": CaseIds(ids=["case-c", "case-a"])} + ) + ) + ).cases == 2 + + with pytest.raises(ValueError, match="unknown Harbor case IDs"): + await backend.resolve_cost( + evaluation_set.model_copy(update={"selection": CaseIds(ids=["missing"])}) + ) + + +@pytest.mark.asyncio +async def test_harbor_backend_exports_complete_authorized_local_tasks(tmp_path): + task_source = tmp_path / "tasks" + task_source.mkdir() + (task_source / "metric.py").write_text("def score(): return 1\n") + cases_path = tmp_path / "local-cases.json" + cases_path.write_text( + json.dumps( + [ + {"id": "case-a", "task_name": "alpha"}, + {"id": "case-b", "task_name": "beta"}, + ] + ) + ) + for task_name in ("alpha", "beta"): + task = task_source / task_name + task.mkdir() + (task / "task.toml").write_text(f'[task]\nname="example/{task_name}"\n') + (task / "instruction.md").write_text(f"Question for {task_name}\n") + (task / "attachment.txt").write_text(f"attachment for {task_name}\n") + backend = HarborBackend( + _config( + tmp_path, + task_source=str(task_source), + cases_path=str(cases_path), + ) + ) + destination = tmp_path / "context" + destination.mkdir() + + await backend.export_case_resources( + evaluation_set=EvaluationSet( + name="harbor-bench", + partition="test", + selection=CaseIds(ids=["case-b"]), + ), + destination=str(destination), + sandbox=await LocalSandbox.create(root=tmp_path), + ) + + index = json.loads((destination / "index.json").read_text()) + assert index["task_source_exposed"] is True + assert [item["case_id"] for item in index["cases"]] == ["case-b"] + exported = destination / index["cases"][0]["path"] + assert (exported / "task.toml").is_file() + assert (exported / "instruction.md").read_text() == "Question for beta\n" + assert (exported / "attachment.txt").read_text() == "attachment for beta\n" + assert (destination / "dataset-files/metric.py").is_file() + assert not (destination / "tasks" / "alpha").exists() + assert destination.stat().st_mode & 0o005 == 0o005 + assert exported.stat().st_mode & 0o005 == 0o005 + assert (exported / "attachment.txt").stat().st_mode & 0o004 == 0o004 + + +@pytest.mark.asyncio +async def test_harbor_backend_maps_remote_download_results_by_request_order( + tmp_path, monkeypatch +): + class FakeTaskId: + def __init__(self, name: str): + self.name = name + + def get_name(self) -> str: + return self.name + + async def get_dataset_metadata(_client, _source): + return SimpleNamespace( + task_ids=[ + FakeTaskId("example/alpha"), + FakeTaskId("example/beta"), + FakeTaskId("example/gamma"), + ] + ) + + async def download_tasks(_client, *, task_ids, output_dir, export): + assert export is True + results = [] + for task_id in task_ids: + path = output_dir / task_id.get_name().split("/")[-1] + path.mkdir(parents=True) + (path / "instruction.md").write_text(task_id.get_name()) + results.append(SimpleNamespace(path=path)) + return SimpleNamespace(results=results) + + async def download_dataset_files(_client, _metadata, *, output_dir): + output_dir.mkdir() + path = output_dir / "metric.py" + path.write_text("def score(): return 1\n") + return [path] + + class PackageDatasetClient: + pass + + class TaskClient: + pass + + PackageDatasetClient.get_dataset_metadata = get_dataset_metadata + PackageDatasetClient.download_dataset_files = download_dataset_files + TaskClient.download_tasks = download_tasks + + modules = { + name: ModuleType(name) + for name in ( + "harbor", + "harbor.registry", + "harbor.registry.client", + "harbor.registry.client.package", + "harbor.tasks", + "harbor.tasks.client", + ) + } + modules[ + "harbor.registry.client.package" + ].PackageDatasetClient = PackageDatasetClient + modules["harbor.tasks.client"].TaskClient = TaskClient + for name, module in modules.items(): + monkeypatch.setitem(sys.modules, name, module) + backend = HarborBackend(_config(tmp_path)) + destination = tmp_path / "context" + destination.mkdir() + + await backend.export_case_resources( + evaluation_set=EvaluationSet( + name="harbor-bench", + partition="test", + selection=CaseIds(ids=["case-c", "case-a"]), + ), + destination=str(destination), + sandbox=await LocalSandbox.create(root=tmp_path), + ) + + index = json.loads((destination / "index.json").read_text()) + assert [item["case_id"] for item in index["cases"]] == ["case-c", "case-a"] + assert [ + (destination / item["path"] / "instruction.md").read_text() + for item in index["cases"] + ] == ["example/gamma", "example/alpha"] + assert (destination / "dataset-files/metric.py").is_file() + + +@pytest.mark.asyncio +async def test_harbor_backend_exposes_complete_successful_trial_records(tmp_path): + secret = "evaluation-scope-secret" + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [ + { + "verifier_result": {"rewards": {"reward": 1.0}}, + "_agent_files": { + "trajectory.json": json.dumps( + {"steps": [{"message": f"used {secret}"}]} + ), + "gaia-trace.jsonl": ( + '{"turn":1,"action":"search"}\n{"turn":2,"answer":"done"}\n' + ), + }, + } + ] + }, + ) + backend = HarborBackend(_config(tmp_path, environment={"EVALUATION_TOKEN": secret})) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a"])), + ) + + artifacts = report.cases[0].artifacts + assert {Path(artifact.path).name for artifact in artifacts} == { + "trajectory.json", + "gaia-trace.jsonl", + "result.json", + } + for artifact in artifacts: + content = (tmp_path / "result/artifacts" / artifact.path).read_text() + assert secret not in content + trajectory = next( + artifact + for artifact in artifacts + if Path(artifact.path).name == "trajectory.json" + ) + assert "[REDACTED]" in (tmp_path / "result/artifacts" / trajectory.path).read_text() + + +@pytest.mark.asyncio +async def test_harbor_backend_reports_latency_and_inference_telemetry(tmp_path): + usage_path = tmp_path / "usage.json" + usage_path.write_text( + json.dumps( + { + "schema_version": 1, + "scopes": { + "evaluation": { + "requests": 99, + "attributions": { + "evaluation": { + "requests": 12, + "input_tokens": 1000, + "cached_input_tokens": 400, + "output_tokens": 250, + "total_tokens": 1250, + }, + "other-eval": {"requests": 5, "total_tokens": 9999}, + }, + }, + "finalization": { + "attributions": { + "evaluation": { + "requests": 2, + "input_tokens": 100, + "cached_input_tokens": 0, + "output_tokens": 50, + "total_tokens": 150, + } + } + }, + }, + } + ) + ) + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [ + { + "verifier_result": {"rewards": {"reward": 1.0}}, + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:01:30Z", + "agent_result": { + "n_input_tokens": 1000, + "n_cache_tokens": 600, + "n_output_tokens": 50, + }, + } + ], + "example/beta": [ + { + "verifier_result": {"rewards": {"reward": 0.0}}, + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:00:30Z", + "agent_result": { + "n_input_tokens": 200, + "n_cache_tokens": 0, + "n_output_tokens": 25, + }, + } + ], + }, + ) + backend = HarborBackend( + _config(tmp_path, inference_usage_path=str(usage_path)) + ) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a", "case-b"])), + ) + + by_id = {case.case_id: case for case in report.cases} + assert by_id["case-a"].metrics["wall_seconds"] == 90.0 + assert by_id["case-b"].metrics["wall_seconds"] == 30.0 + assert report.metrics["mean_case_wall_seconds"] == 60.0 + assert report.metrics["max_case_wall_seconds"] == 90.0 + assert report.metrics["median_case_wall_seconds"] == 60.0 + # this evaluation's attribution only, summed across gateway scopes + assert report.metrics["inference_requests"] == 14.0 + assert report.metrics["inference_input_tokens"] == 1100.0 + assert report.metrics["inference_cached_input_tokens"] == 400.0 + assert report.metrics["inference_output_tokens"] == 300.0 + assert report.metrics["inference_total_tokens"] == 1400.0 + # the trusted gateway total is per evaluation, so only its per-case mean is + # derivable here (median/max need post-hoc per-case attribution) + assert report.metrics["mean_case_inference_input_tokens"] == 550.0 + assert report.metrics["mean_case_inference_total_tokens"] == 700.0 + assert report.metrics["mean_case_inference_requests"] == 7.0 + # agent-reported tokens are per case, so they get the same mean/median/max + # treatment as latency rather than only a report-level sum + assert report.metrics["mean_case_agent_reported_input_tokens"] == 600.0 + assert report.metrics["max_case_agent_reported_input_tokens"] == 1000.0 + assert report.metrics["mean_case_agent_reported_cached_input_tokens"] == 300.0 + assert report.metrics["mean_case_agent_reported_output_tokens"] == 37.5 + assert report.metrics["max_case_agent_reported_output_tokens"] == 50.0 + # agent-self-reported usage: per case and summed at report level, under a + # prefix distinct from the trusted gateway-metered inference_* metrics + assert by_id["case-a"].metrics["agent_reported_input_tokens"] == 1000.0 + assert by_id["case-a"].metrics["agent_reported_cached_input_tokens"] == 600.0 + assert by_id["case-b"].metrics["agent_reported_output_tokens"] == 25.0 + assert report.metrics["agent_reported_total_input_tokens"] == 1200.0 + assert report.metrics["agent_reported_total_cached_input_tokens"] == 600.0 + assert report.metrics["agent_reported_total_output_tokens"] == 75.0 + # accuracy remains the primary metric, untouched by telemetry + assert report.metrics["score"] == 0.5 + + +@pytest.mark.asyncio +async def test_harbor_backend_exposes_exact_failed_trial_result(tmp_path): + detail = "Invalid schema for function 'transcribe_audio': Missing 'language'." + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [ + { + "verifier_result": None, + "exception_info": { + "exception_type": "BadRequestError", + "exception_message": detail, + }, + } + ] + }, + ) + backend = HarborBackend(_config(tmp_path)) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a"])), + ) + + artifacts = report.cases[0].artifacts + result_artifact = next( + artifact for artifact in artifacts if Path(artifact.path).name == "result.json" + ) + result = json.loads( + (tmp_path / "result/artifacts" / result_artifact.path).read_text() + ) + assert result["exception_info"]["exception_message"] == detail + assert result_artifact.description.endswith("result.json") + + +@pytest.mark.asyncio +async def test_harbor_backend_scores_agent_crash_as_informative_task_failure(tmp_path): + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [{"verifier_result": {"rewards": {"reward": 1.0}}}], + "example/beta": [ + { + "verifier_result": None, + "exception_info": {"exception_type": "AgentCrash"}, + } + ], + }, + ) + backend = HarborBackend(_config(tmp_path)) + runtime_context = await _context(tmp_path, sandbox) + + report = await backend.evaluate( + context=runtime_context, + request=_request(CaseRange(stop=2)), + ) + + # A candidate whose own harness crashes is an informative task failure: + # scored at the failure value, a real SUCCESS sample that counts toward the + # mean, and NOT an infrastructure error. + assert report.status == EvaluationStatus.SUCCESS + assert report.metrics == {"score": 0.5, "error_rate": 0.0, "score_stddev": 0.5} + assert [case.status for case in report.cases] == [ + CaseStatus.SUCCESS, + CaseStatus.SUCCESS, + ] + assert report.cases[1].metrics["score"] == 0.0 + assert report.cases[1].output["error_category"] == "task_failure" + assert sandbox.command[:15] == [ + sys.executable, + "run", + "--python", + "3.12", + "--no-config", + "--no-env-file", + "--default-index", + "https://pypi.org/simple", + "--index-strategy", + "first-index", + "--project", + str(tmp_path / "target"), + "--with", + "harbor==0.1.17", + "harbor", + ] + assert sandbox.command.count("-i") == 2 + assert sandbox.command[sandbox.command.index("-n") + 1] == "7" + assert ( + sandbox.command[sandbox.command.index("--agent-timeout-multiplier") + 1] + == "0.3" + ) + assert sandbox.timeout == 90 + assert [artifact.path for artifact in report.artifacts] == [ + "harbor/stdout.log", + "harbor/stderr.log", + ] + checkpoints = await runtime_context.case_store.load_all() + assert [case.case_id for case in checkpoints] == ["case-a", "case-b"] + + +@pytest.mark.asyncio +async def test_harbor_backend_excludes_transient_infrastructure_only_when_trusted( + tmp_path, +): + # A within-trial transient-infra exception is trusted as infrastructure only + # for finalization (admin) evaluations. For a competitive agent evaluation + # it is scored at the failure value, so a candidate cannot inflate its mean + # by emitting a timeout/connection error on a hard case. + def _sandbox(): + return FakeSandbox( + tmp_path, + { + "example/alpha": [{"verifier_result": {"rewards": {"reward": 1.0}}}], + "example/beta": [ + { + "verifier_result": None, + "exception_info": {"exception_type": "openai.RateLimitError"}, + } + ], + }, + ) + + backend = HarborBackend(_config(tmp_path)) + + # Agent (untrusted) evaluation: the rate-limited case is a scored failure. + agent_report = await backend.evaluate( + context=await _context(tmp_path / "agent", _sandbox()), + request=_request(CaseRange(stop=2)), + ) + assert agent_report.metrics["score"] == 0.5 + assert agent_report.metrics["error_rate"] == 0.0 + assert [case.status for case in agent_report.cases] == [ + CaseStatus.SUCCESS, + CaseStatus.SUCCESS, + ] + assert agent_report.cases[1].output["error_category"] == "task_failure" + + # Trusted finalization: infrastructure is excluded from the mean and + # reported as the error rate, as before. + trusted_report = await backend.evaluate( + context=await _context(tmp_path / "trusted", _sandbox(), finalization=True), + request=_request(CaseRange(stop=2)), + ) + assert trusted_report.metrics["score"] == 1.0 + assert trusted_report.metrics["error_rate"] == 0.5 + assert [case.status for case in trusted_report.cases] == [ + CaseStatus.SUCCESS, + CaseStatus.ERROR, + ] + assert trusted_report.cases[1].output["error_category"] == "transient_infra" + assert trusted_report.cases[1].errors[0].code == "transient_infrastructure" + + +@pytest.mark.asyncio +async def test_harbor_backend_marks_inference_budget_exhaustion_invalid(tmp_path): + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [{"verifier_result": {"rewards": {"reward": 1.0}}}], + "example/beta": [ + { + "verifier_result": None, + "exception_info": { + "exception_type": "openai.RateLimitError", + # The gateway's distinct code survives in the body message + # even though the client collapses the type to a 429. + "message": "Error code: 429 - inference token budget " + "exhausted (code: budget_exhausted)", + }, + } + ], + }, + ) + backend = HarborBackend(_config(tmp_path)) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseRange(stop=2)), + ) + + # Budget exhaustion is terminating: the whole evaluation is INVALID with a + # distinct, non-retryable code — never a rate-limit, never a score of zero. + assert report.status == EvaluationStatus.INVALID + assert "score" not in report.metrics + assert any( + diagnostic.code == "inference_budget_exhausted" + for diagnostic in report.diagnostics + ) + + +@pytest.mark.asyncio +async def test_harbor_backend_treats_missing_coverage_as_infrastructure(tmp_path): + # Only alpha produces a trial; beta is dropped entirely by the sub-run. + sandbox = FakeSandbox( + tmp_path, + {"example/alpha": [{"verifier_result": {"rewards": {"reward": 1.0}}}]}, + ) + backend = HarborBackend(_config(tmp_path)) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseRange(stop=2)), + ) + + # The dropped case is infrastructure (excluded from the mean, counted as + # error rate) rather than a silent zero, and coverage loss is logged. + assert report.metrics["score"] == 1.0 + assert report.metrics["error_rate"] == 0.5 + assert [case.status for case in report.cases] == [ + CaseStatus.SUCCESS, + CaseStatus.ERROR, + ] + assert report.cases[1].output["error_category"] == "transient_infra" + assert any( + diagnostic.code == "harbor_incomplete_coverage" + for diagnostic in report.diagnostics + ) + + +@pytest.mark.asyncio +async def test_infrastructure_retry_is_trusted_only(tmp_path): + # With incomplete coverage (beta never produced), a competitive evaluation + # must not re-run the sub-run (no best-of-N amplifier a candidate can + # trigger); a trusted finalization re-scores with the configured retries. + def _sandbox(): + return FakeSandbox( + tmp_path, + {"example/alpha": [{"verifier_result": {"rewards": {"reward": 1.0}}}]}, + ) + + backend = HarborBackend( + _config( + tmp_path, + infrastructure_max_attempts=3, + infrastructure_retry_delay_seconds=0, + ) + ) + + agent_ctx = await _context(tmp_path / "agent", _sandbox()) + await backend.evaluate(context=agent_ctx, request=_request(CaseRange(stop=2))) + assert not (agent_ctx.artifact_dir / "harbor" / "retry-2").exists() + + trusted_ctx = await _context( + tmp_path / "trusted", _sandbox(), finalization=True + ) + await backend.evaluate(context=trusted_ctx, request=_request(CaseRange(stop=2))) + assert (trusted_ctx.artifact_dir / "harbor" / "retry-2").exists() + assert (trusted_ctx.artifact_dir / "harbor" / "retry-3").exists() + + +@pytest.mark.asyncio +async def test_harbor_backend_runs_harness_as_unprivileged_user(tmp_path): + sandbox = FakeSandbox( + tmp_path, + {"example/alpha": [{"verifier_result": {"rewards": {"reward": 1.0}}}]}, + ) + backend = HarborBackend(_config(tmp_path, harness_user="harness")) + + await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseRange(stop=1)), + ) + + # `harbor run` drops to the unprivileged user, and its work dirs (workspace + + # staging) are handed to that user first so it never needs the trusted state. + assert sandbox.run_as == "harness" + assert sandbox.chown_commands, "expected the harness work dirs to be chowned" + for command in sandbox.chown_commands: + assert command[:3] == ["chown", "-R", "harness:harness"] + # The checkout's mktemp parent (0700 root) is made traversable so the + # dropped-uid harness can resolve the editable candidate package's absolute + # path — otherwise `import ` fails with "No module named ...". + checkout_parent = str(PurePosixPath(str(tmp_path / "target")).parent) + assert ["chmod", "o+x", checkout_parent] in sandbox.chmod_commands + # And it probes, as the dropped user, that the workspace is actually reachable + # before running harbor — so a permission gap fails fast and clearly here. + assert (["test", "-r", str(tmp_path / "target")], "harness") in sandbox.probe_commands + + +def test_harbor_config_rejects_harness_isolation_with_upstream_task_services(tmp_path): + # uid isolation cannot hide env vars, so the raw upstream key would still + # reach the isolated harness; the combination must be refused. + with pytest.raises(ValueError, match="harness_user cannot be combined"): + _config( + tmp_path, + inference_gateway_url="http://inference-gateway:8001", + inference_gateway_token="gw-token", + task_services_use_upstream=True, + upstream_api_key_env="VERO_INFERENCE_UPSTREAM_API_KEY", + harness_user="harness", + ) + + +@pytest.mark.asyncio +async def test_harbor_backend_routes_candidate_inference_through_scoped_gateway( + tmp_path, monkeypatch +): + monkeypatch.setenv("OPENAI_API_KEY", "raw-provider-key") + sandbox = FakeSandbox( + tmp_path, + {"example/alpha": [{"verifier_result": {"rewards": {"reward": 1.0}}}]}, + ) + backend = HarborBackend( + _config( + tmp_path, + inference_gateway_url="http://inference-gateway:8001", + inference_gateway_token="evaluation-scope-token", + ) + ) + + await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a"])), + ) + + assert sandbox.env["OPENAI_API_KEY"] == "evaluation-scope-token" + assert sandbox.env["OPENAI_BASE_URL"] == ( + "http://inference-gateway:8001/scopes/evaluation/evaluation/v1" + ) + assert "raw-provider-key" not in sandbox.env.values() + + +@pytest.mark.asyncio +async def test_harbor_backend_matches_canonical_result_task_name(tmp_path): + cases_path = tmp_path / "canonical-cases.json" + cases_path.write_text( + json.dumps( + [ + { + "id": "local-task", + "task_name": "local-task", + "result_task_name": "org/local-task", + } + ] + ), + encoding="utf-8", + ) + sandbox = FakeSandbox( + tmp_path, + {"org/local-task": [{"verifier_result": {"rewards": {"reward": 1.0}}}]}, + ) + backend = HarborBackend(_config(tmp_path, cases_path=str(cases_path))) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(), + ) + + assert report.status == EvaluationStatus.SUCCESS + assert report.metrics["score"] == 1.0 + assert report.cases[0].output["task_name"] == "local-task" + assert report.cases[0].output["result_task_name"] == "org/local-task" + + +@pytest.mark.asyncio +async def test_harbor_backend_mean_counts_dead_attempts_as_failures(tmp_path): + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [ + {"verifier_result": {"rewards": {"pass": 1.0}}}, + { + "verifier_result": None, + "exception_info": {"exception_type": "TimeoutError"}, + }, + ] + }, + ) + backend = HarborBackend(_config(tmp_path, aggregate_attempts="mean")) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a"])), + ) + + assert report.metrics == {"score": 0.5, "error_rate": 0.0, "score_stddev": 0.0} + assert report.cases[0].metrics == { + "score": 0.5, + "n_attempts": 2.0, + "n_scored": 1.0, + # This is a competitive (untrusted) evaluation, so the dead attempt's own + # TimeoutError is not trusted as infrastructure — it counts as a clean + # zero-filled failure, not infra dilution. + "n_dead_infra": 0.0, + "n_clean": 2.0, + } + + +@pytest.mark.asyncio +async def test_harbor_backend_fails_when_no_requested_trials_match(tmp_path): + secret = "sensitive-token" + sandbox = FakeSandbox( + tmp_path, + {}, + CommandResult("", f"runner failed with {secret}", 1), + ) + backend = HarborBackend(_config(tmp_path, environment={"TOKEN": secret})) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a"])), + ) + + assert report.status == EvaluationStatus.FAILED + assert report.diagnostics[0].code == "infrastructure_failure" + assert secret not in report.diagnostics[0].message + assert secret not in (tmp_path / "result/artifacts/harbor/stderr.log").read_text() + + +def test_harbor_backend_rejects_controlled_extra_flags(tmp_path): + with pytest.raises(ValueError, match="backend-controlled"): + _config(tmp_path, extra_args=["--jobs-dir=/tmp/forged"]) + + +def test_environment_routes_finalization_to_reserved_scope(tmp_path): + config = _config( + tmp_path, + inference_gateway_url="http://inference-gateway:8001", + inference_gateway_token="gw-eval", + inference_gateway_finalization_token="gw-fin", + ) + backend = HarborBackend(config) + + search = backend._environment("eval-1", finalization=False) + assert search["OPENAI_API_KEY"] == "gw-eval" + assert "/scopes/evaluation/eval-1/" in search["OPENAI_BASE_URL"] + + final = backend._environment("eval-1", finalization=True) + assert final["OPENAI_API_KEY"] == "gw-fin" + assert "/scopes/finalization/eval-1/" in final["OPENAI_BASE_URL"] + + +def test_environment_finalization_falls_back_when_no_reserved_token(tmp_path): + config = _config( + tmp_path, + inference_gateway_url="http://inference-gateway:8001", + inference_gateway_token="gw-eval", + ) + final = HarborBackend(config)._environment("eval-1", finalization=True) + assert final["OPENAI_API_KEY"] == "gw-eval" + assert "/scopes/evaluation/eval-1/" in final["OPENAI_BASE_URL"] + + +def test_retry_config_json_carries_editable_backoff(tmp_path): + backend = HarborBackend( + _config( + tmp_path, + max_retries=5, + retry_wait_multiplier=3.0, + retry_min_wait_seconds=2.0, + retry_max_wait_seconds=45.0, + ) + ) + + payload = json.loads(backend._retry_config_json()) + + assert payload == { + "retry": { + "max_retries": 5, + "wait_multiplier": 3.0, + "min_wait_sec": 2.0, + "max_wait_sec": 45.0, + } + } + + +def test_command_forwards_retry_config_and_max_retries(tmp_path): + backend = HarborBackend(_config(tmp_path, max_retries=5)) + + command = backend._command( + workspace=LAYOUT.target_repo, + request=_request(), + cases=[], + jobs_dir="/staging/jobs", + task_source="example/tasks@1.0", + local_task_source=False, + retry_config_path="/staging/retry-config.json", + ) + + # backoff arrives via the JobConfig snippet ... + assert "--config" in command + assert command[command.index("--config") + 1] == "/staging/retry-config.json" + # ... while the count stays a flag (harbor applies it over the --config base). + assert command[command.index("--max-retries") + 1] == "5" + + +@pytest.mark.asyncio +async def test_harbor_backend_reports_median_for_skewed_cost_distributions(tmp_path): + # Heavy tail: one case dominates both latency and tokens. The mean alone + # hides that shape, which is why cost metrics carry a median and a max. + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [ + { + "verifier_result": {"rewards": {"reward": 1.0}}, + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:00:10Z", + "agent_result": { + "n_input_tokens": 100, + "n_cache_tokens": 10, + "n_output_tokens": 5, + }, + } + ], + "example/beta": [ + { + "verifier_result": {"rewards": {"reward": 1.0}}, + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:00:20Z", + "agent_result": { + "n_input_tokens": 200, + "n_cache_tokens": 20, + "n_output_tokens": 10, + }, + } + ], + "example/gamma": [ + { + "verifier_result": {"rewards": {"reward": 1.0}}, + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:05:00Z", + "agent_result": { + "n_input_tokens": 3000, + "n_cache_tokens": 300, + "n_output_tokens": 150, + }, + } + ], + }, + ) + backend = HarborBackend(_config(tmp_path)) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a", "case-b", "case-c"])), + ) + + # latency: mean is dragged up by the tail case, median is not + assert report.metrics["mean_case_wall_seconds"] == 110.0 + assert report.metrics["median_case_wall_seconds"] == 20.0 + assert report.metrics["max_case_wall_seconds"] == 300.0 + # tokens get the same treatment, and show the same divergence + assert report.metrics["mean_case_agent_reported_input_tokens"] == 1100.0 + assert report.metrics["median_case_agent_reported_input_tokens"] == 200.0 + assert report.metrics["max_case_agent_reported_input_tokens"] == 3000.0 + assert report.metrics["mean_case_agent_reported_output_tokens"] == 55.0 + assert report.metrics["median_case_agent_reported_output_tokens"] == 10.0 + assert report.metrics["max_case_agent_reported_output_tokens"] == 150.0 + assert report.metrics["median_case_agent_reported_cached_input_tokens"] == 20.0 + + +@pytest.mark.asyncio +async def test_case_result_carries_a_phase_execution_trace(tmp_path: Path): + """Harbor trial phases are lifted into CaseResult.execution_trace. + + Without this the whole trace surface is inert on the only backend the + benchmarks use: `evals cases` reports trace=False for every case and + `evals trace ID CASE` has nothing to read, so an agent that wants to know + why a case failed has to walk artifacts/harbor/jobs/** with raw file tools. + Observed live in a conformance run, where the optimizer correctly skipped + `evals trace` because the interface told it there was nothing there. + """ + sandbox = FakeSandbox( + tmp_path, + { + "example/alpha": [ + { + "trial_name": "alpha__abc", + "verifier_result": {"rewards": {"reward": 1.0}}, + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:01:30Z", + "agent_info": {"name": "seed", "version": "0.1.0"}, + "agent_result": {"n_input_tokens": 10, "rollout_details": ["turn"]}, + "environment_setup": { + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:00:10Z", + }, + "agent_setup": { + "started_at": "2026-01-01T00:00:10Z", + "finished_at": "2026-01-01T00:00:20Z", + }, + "agent_execution": { + "started_at": "2026-01-01T00:00:20Z", + "finished_at": "2026-01-01T00:01:20Z", + }, + "verifier": { + "started_at": "2026-01-01T00:01:20Z", + "finished_at": "2026-01-01T00:01:30Z", + }, + } + ], + # A case that died: the exception has to reach the trace too, since + # that is the case an agent most needs to inspect. + "example/beta": [ + { + "trial_name": "beta__xyz", + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:00:30Z", + "environment_setup": { + "started_at": "2026-01-01T00:00:00Z", + "finished_at": "2026-01-01T00:00:05Z", + }, + "exception_info": { + "exception_type": "TimeoutError", + "message": "agent exceeded its clock", + }, + } + ], + }, + ) + backend = HarborBackend(_config(tmp_path)) + + report = await backend.evaluate( + context=await _context(tmp_path, sandbox), + request=_request(CaseIds(ids=["case-a", "case-b"])), + ) + by_id = {case.case_id: case for case in report.cases} + + scored = by_id["case-a"].execution_trace + assert [span["phase"] for span in scored] == [ + "environment_setup", + "agent_setup", + "agent_execution", + "verifier", + ] + assert [span["seconds"] for span in scored] == [10.0, 10.0, 60.0, 10.0] + assert {span["attempt"] for span in scored} == {0} + assert {span["trial_name"] for span in scored} == {"alpha__abc"} + # The harness's own turn-by-turn record rides on the agent_execution span, + # which is why the CLI windows spans rather than printing them whole. + agent_span = next(s for s in scored if s["phase"] == "agent_execution") + assert agent_span["detail"]["agent_result"]["rollout_details"] == ["turn"] + assert agent_span["detail"]["agent_info"]["name"] == "seed" + verifier_span = next(s for s in scored if s["phase"] == "verifier") + assert verifier_span["detail"]["rewards"] == {"reward": 1.0} + # Uniform key set across spans keeps `evals trace`'s shape summary legible. + assert len({tuple(sorted(span)) for span in scored}) == 1 + + failed = by_id["case-b"].execution_trace + assert [span["phase"] for span in failed] == ["environment_setup", "exception"] + assert failed[-1]["detail"]["exception_type"] == "TimeoutError" diff --git a/vero/tests/test_v05_harbor_build.py b/vero/tests/test_v05_harbor_build.py new file mode 100644 index 00000000..6f54bc68 --- /dev/null +++ b/vero/tests/test_v05_harbor_build.py @@ -0,0 +1,1335 @@ +from __future__ import annotations + +import json +import shlex +import subprocess +import tomllib +from pathlib import Path + +import pytest +import yaml +from pydantic import ValidationError + +from vero.harbor import ( + AgentAccessSpec, + CommandBackendSpec, + HarborBuildConfig, + HarborDeploymentConfig, + InferenceBudgetSpec, + InferenceGatewaySpec, + VerificationTargetSpec, + WandbSpec, + WorkspaceOverlaySpec, + compile_harbor_task, + load_harbor_build_config, +) +from vero.harbor.build.compiler import GATEWAY_ROUTED_CREDENTIALS +from vero.harbor.build.config import ( + _HARBOR_ONLY_FIELDS, + _AgentWorkspaceFields, + _EvaluationLimitFields, + _HarborEvaluationFields, + _SearchAndSelectionFields, + _TaskEnvironmentFields, + _TaskIdentityFields, +) +from vero.harbor.deployment import FACTORY_PATH +from vero.layout import LAYOUT +from vero.sidecar.serve import load_factory + +_OMIT = object() + + +def test_task_layout_values_are_pinned(): + """The one place the layout's literal values are written down twice. + + Every other test references LAYOUT, so without this they would all be + tautological: changing a path would silently keep them green. These values + are a contract with the benchmarks' read_only_paths and with the compiled + task directories that are checked in, so a deliberate change should have to + edit this list too. + """ + assert LAYOUT.target_repo == "/work/agent" + assert LAYOUT.trusted_repo == "/opt/agent-baseline" + assert LAYOUT.seed_repo == "/opt/agent-seed" + assert LAYOUT.vero == "/opt/vero" + assert LAYOUT.cases == "/opt/cases" + assert LAYOUT.task_source == "/opt/task-source" + assert LAYOUT.harness == "/opt/harness" + assert LAYOUT.overlay == "/opt/overlay" + assert LAYOUT.serve_config == "/opt/serve.json" + assert LAYOUT.seed_script == "/opt/seed.sh" + assert LAYOUT.inference_config == "/opt/inference.json" + assert LAYOUT.agent_volume == "/state/agent-context" + assert LAYOUT.admin_volume == "/state/admin" + assert LAYOUT.token_dir == "/state/token" + assert LAYOUT.inference_dir == "/state/inference" + assert LAYOUT.sidecar_host == "eval-sidecar" + assert LAYOUT.sidecar_port == 8000 + assert LAYOUT.gateway_host == "inference-gateway" + assert LAYOUT.gateway_port == 8001 + # Derived paths, so a base and its children cannot drift apart. + assert LAYOUT.session_dir == "/state/admin/session" + assert LAYOUT.case_resources_dir == "/state/admin/case-resources" + assert LAYOUT.token_path == "/state/token/admin.token" + assert LAYOUT.inference_state == "/state/inference/usage.json" + assert LAYOUT.inference_request_log_dir == "/state/inference/requests" + assert LAYOUT.target_git == "/work/agent/.git" + assert LAYOUT.target_git_exclude == "/work/agent/.git/info/exclude" + assert LAYOUT.target_evals == "/work/agent/.evals" + assert LAYOUT.sidecar_url == "http://eval-sidecar:8000" + assert LAYOUT.gateway_url == "http://inference-gateway:8001" + + +def _git(path: Path, *arguments: str) -> str: + result = subprocess.run( + ["git", *arguments], + cwd=path, + check=True, + text=True, + capture_output=True, + ) + return result.stdout.strip() + + +def test_build_param_substitution_semantics(): + from vero.harbor.build.loader import _substitute_build_param as sub + + context = {"A": "x"} + assert sub("${A}", context) == "x" + assert sub("${MISSING:-fallback}", context) == "fallback" + assert sub("pre-${A}-post", context) == "pre-x-post" + with pytest.raises(ValueError, match="required build parameter 'MISSING'"): + sub("${MISSING:?please set it}", context) + with pytest.raises(ValueError, match="build parameter 'MISSING' is unset"): + sub("${MISSING}", context) + + +def _target_repo(path: Path) -> Path: + path.mkdir(parents=True) + _git(path, "init", "-q") + _git(path, "config", "user.name", "VeRO Test") + _git(path, "config", "user.email", "vero@example.test") + (path / "README.md").write_text("# Target\n", encoding="utf-8") + (path / "pyproject.toml").write_text( + '[project]\nname="target"\nversion="0.1.0"\n', + encoding="utf-8", + ) + _git(path, "add", ".") + _git(path, "commit", "-q", "-m", "target baseline") + return path + + +def _config(tmp_path: Path, **updates) -> HarborBuildConfig: + target = _target_repo(tmp_path / "target") + task_source = tmp_path / "protected-tasks" + task_source.mkdir() + task_names = ["task-a", "task-b", "task-c", "task-d", "task-e", "task-hidden"] + for task_name in task_names: + task = task_source / task_name + task.mkdir() + (task / "task.toml").write_text( + f'[task]\nname="org/{task_name}"\n', + encoding="utf-8", + ) + values = { + "name": 'org/optimize-"program"', + "description": "Improve the program", + "agent_repo": str(target), + "task_source": str(task_source), + "agent_import_path": "target.agent:Agent", + "harbor_requirement": "harbor==0.1.17", + "partitions": { + "validation": ["task-a", "task-b", "task-c", "task-d", "task-e"], + "test": ["task-hidden"], + }, + "agent_access": [ + AgentAccessSpec( + partition="validation", + expose_case_resources=True, + total_runs=5, + total_cases=25, + ) + ], + "selection_partition": "validation", + "targets": [VerificationTargetSpec(partition="test")], + } + values.update(updates) + # Pass _OMIT to leave a key out entirely, which is different from passing + # None: the build config reports Harbor-only keys that were set at all. + return HarborBuildConfig(**{k: v for k, v in values.items() if v is not _OMIT}) + + +def test_compiler_bakes_workspace_overlay_into_agent_environment(tmp_path): + bundle = tmp_path / "bundle" + (bundle / ".claude" / "agents").mkdir(parents=True) + (bundle / ".claude" / "agents" / "insights.md").write_text( + "# a\n", encoding="utf-8" + ) + (bundle / "skills" / "insights").mkdir(parents=True) + (bundle / "skills" / "insights" / "SKILL.md").write_text("# s\n", encoding="utf-8") + + config = _config( + tmp_path, + workspace_overlays=[ + WorkspaceOverlaySpec(source=str(bundle / ".claude"), dest=".claude"), + WorkspaceOverlaySpec(source=str(bundle / "skills"), dest="skills"), + ], + ) + output = compile_harbor_task(config, tmp_path / "compiled") + env = output / "environment" + + # staged into the build context under environment/overlay/ + assert (env / "overlay" / ".claude" / "agents" / "insights.md").is_file() + assert (env / "overlay" / "skills" / "insights" / "SKILL.md").is_file() + # Dockerfile copies it, seed applies it, injected tooling is git-excluded + assert f"COPY overlay {LAYOUT.overlay}" in (env / "Dockerfile").read_text() + seed = (env / "main" / "seed.sh").read_text() + assert f"cp -a {LAYOUT.overlay}/. {LAYOUT.target_repo}/" in seed + assert "/.claude/" in seed and "/skills/" in seed + + +def test_compiler_plumbs_wandb_into_serve_config(tmp_path): + output = compile_harbor_task( + _config(tmp_path, wandb=WandbSpec(project="vero-bench", group="gaia")), + tmp_path / "compiled", + ) + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert serve["wandb"]["project"] == "vero-bench" + assert serve["wandb"]["group"] == "gaia" + + +def test_compiler_omits_wandb_when_unset(tmp_path): + output = compile_harbor_task(_config(tmp_path), tmp_path / "compiled") + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert serve["wandb"] is None + + +def test_compiler_omits_overlay_when_unset(tmp_path): + output = compile_harbor_task( + _config(tmp_path, include_evals_skill=False), tmp_path / "compiled" + ) + env = output / "environment" + assert not (env / "overlay").exists() + assert "COPY overlay" not in (env / "Dockerfile").read_text() + assert LAYOUT.overlay not in (env / "main" / "seed.sh").read_text() + + +def test_compiler_bakes_evals_skill_by_default(tmp_path): + output = compile_harbor_task(_config(tmp_path), tmp_path / "compiled") + env = output / "environment" + skill = env / "overlay" / "skills" / "evals" / "SKILL.md" + assert skill.is_file() + assert "evals run" in skill.read_text(encoding="utf-8") + assert f"'/skills/' >> {LAYOUT.target_git_exclude}" in ( + env / "main" / "seed.sh" + ).read_text(encoding="utf-8") + + +def test_compiler_budget_disclosure_toggle(tmp_path): + disclosed = compile_harbor_task(_config(tmp_path / "on"), tmp_path / "on/out") + instruction_on = (disclosed / "instruction.md").read_text(encoding="utf-8") + serve_on = json.loads( + (disclosed / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert "budget" in instruction_on.lower() + assert serve_on["disclose_budget"] is True + + blind = compile_harbor_task( + _config(tmp_path / "off", disclose_budget=False), tmp_path / "off/out" + ) + instruction_off = (blind / "instruction.md").read_text(encoding="utf-8") + serve_off = json.loads( + (blind / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert "budget" not in instruction_off.lower() + assert serve_off["disclose_budget"] is False + + +def test_compiler_plumbs_task_services_use_upstream(tmp_path, monkeypatch): + monkeypatch.setenv("TEST_UPSTREAM_KEY", "real-provider-secret") + monkeypatch.setenv("TEST_UPSTREAM_URL", "https://provider.example/v1") + monkeypatch.setenv("TEST_MODAL_TOKEN", "modal-secret") + config = _config( + tmp_path, + secrets=["TEST_MODAL_TOKEN"], + task_services_use_upstream=True, + # Task-owned upstream services are incompatible with harness isolation + # (the key would reach the harness env), so this build opts out. + harness_user=None, + task_environment={"TAU2_USER_MODEL": "openai/gpt-target"}, + inference_gateway=InferenceGatewaySpec( + upstream_api_key_env="TEST_UPSTREAM_KEY", + upstream_base_url_env="TEST_UPSTREAM_URL", + producer=InferenceBudgetSpec(allowed_models=["gpt-producer"]), + evaluation=InferenceBudgetSpec(allowed_models=["gpt-target"]), + ), + ) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + backend = next(iter(serve["backends"].values())) + assert backend["task_services_use_upstream"] is True + assert backend["upstream_api_key_env"] == "VERO_INFERENCE_UPSTREAM_API_KEY" + assert backend["upstream_base_url_env"] == "VERO_INFERENCE_UPSTREAM_BASE_URL" + assert backend["environment"] == {"TAU2_USER_MODEL": "openai/gpt-target"} + # validates against the deployment schema (backend config accepts the flags) + for partition, b in serve["backends"].items(): + b["cases_path"] = str( + output + / f"environment/sidecar/cases/{partition.removeprefix('harbor-')}.jsonl" + ) + HarborDeploymentConfig.model_validate(serve) + # the trusted eval-sidecar receives the real upstream (main keeps it scrubbed) + compose = yaml.safe_load((output / "environment/docker-compose.yaml").read_text()) + sidecar_env = compose["services"]["eval-sidecar"]["environment"] + assert "VERO_INFERENCE_UPSTREAM_API_KEY" in sidecar_env + assert "VERO_INFERENCE_UPSTREAM_BASE_URL" in sidecar_env + assert ( + compose["services"]["main"]["environment"]["VERO_INFERENCE_UPSTREAM_API_KEY"] + == "" + ) + + +def test_compiler_reserves_finalization_scope_defaulting_to_evaluation(tmp_path): + config = _config( + tmp_path, + inference_gateway=InferenceGatewaySpec( + producer=InferenceBudgetSpec(allowed_models=["gpt-producer"]), + evaluation=InferenceBudgetSpec( + allowed_models=["gpt-target"], max_requests=15000, max_tokens=100000000 + ), + ), + ) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + gateway = json.loads( + (output / "environment/gateway/config.json").read_text(encoding="utf-8") + ) + scopes = gateway["scopes"] + # a reserved finalization scope exists, defaulting to the evaluation policy + assert "finalization" in scopes + assert scopes["finalization"]["allowed_models"] == ["gpt-target"] + assert scopes["finalization"]["max_tokens"] == 100000000 + # its token is distinct from the evaluation token (optimizer can't drain it) + assert ( + scopes["finalization"]["token_sha256"] != scopes["evaluation"]["token_sha256"] + ) + # and the sidecar backend carries a finalization token distinct from the eval one + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + backend = next(iter(serve["backends"].values())) + assert backend["inference_gateway_finalization_token"] is not None + assert ( + backend["inference_gateway_finalization_token"] + != backend["inference_gateway_token"] + ) + # Per-request logging is on by default: the gateway captures every + # request/response on its state volume and the sidecar mirrors it. The + # experimental thread-attribution stamping stays off unless opted into. + assert gateway["request_log"] == { + "directory": LAYOUT.inference_request_log_dir, + "body_bytes": 16384, + "attribution": False, + } + assert serve["inference_request_log_dir"] == LAYOUT.inference_request_log_dir + + +def test_compiler_emits_request_log_attribution_when_enabled(tmp_path): + config = _config( + tmp_path, + inference_gateway=InferenceGatewaySpec( + producer=InferenceBudgetSpec(allowed_models=["gpt-producer"]), + evaluation=InferenceBudgetSpec(allowed_models=["gpt-target"]), + request_log_attribution=True, + ), + ) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + gateway = json.loads( + (output / "environment/gateway/config.json").read_text(encoding="utf-8") + ) + assert gateway["request_log"]["attribution"] is True + + +def test_compiler_omits_request_log_when_disabled(tmp_path): + config = _config( + tmp_path, + inference_gateway=InferenceGatewaySpec( + producer=InferenceBudgetSpec(allowed_models=["gpt-producer"]), + evaluation=InferenceBudgetSpec(allowed_models=["gpt-target"]), + log_requests=False, + ), + ) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + gateway = json.loads( + (output / "environment/gateway/config.json").read_text(encoding="utf-8") + ) + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert gateway["request_log"] is None + assert serve["inference_request_log_dir"] is None + + +def test_workspace_overlay_rejects_unsafe_dest_and_missing_source(tmp_path): + present = tmp_path / "present" + present.mkdir() + with pytest.raises(ValidationError): + WorkspaceOverlaySpec(source=str(present), dest="../escape") + with pytest.raises(ValidationError): + WorkspaceOverlaySpec(source=str(tmp_path / "missing"), dest="ok") + + +def test_build_config_requires_pins_and_valid_partition_references(tmp_path): + assert ( + _config( + tmp_path / "modal", + harbor_requirement="harbor[modal]==0.20.0", + ).harbor_requirement + == "harbor[modal]==0.20.0" + ) + with pytest.raises(ValidationError, match="pin an exact version"): + _config(tmp_path / "unpinned", harbor_requirement="harbor>=0.1") + with pytest.raises(ValidationError, match="selection_partition"): + _config(tmp_path / "unknown", selection_partition="missing") + with pytest.raises(ValidationError, match="controlled flags"): + _config(tmp_path / "flags", extra_harbor_args=["--jobs-dir=/forged"]) + with pytest.raises(ValidationError, match="explicit version"): + _config(tmp_path / "source", task_source="org/unversioned") + + +def test_build_config_requires_requested_models_to_be_allowed_by_their_scope(tmp_path): + """A model the gateway would refuse must fail the build, not the run. + + The verification case is the costly one: finalization runs the target agent + too, so a target scoring with a model its scope disallows only 403s after + search has already spent its whole budget. + """ + + def gateway(**updates) -> InferenceGatewaySpec: + values = { + "producer": InferenceBudgetSpec(allowed_models=["gpt-producer"]), + "evaluation": InferenceBudgetSpec(allowed_models=["gpt-target"]), + } + values.update(updates) + return InferenceGatewaySpec(**values) + + # The matching case builds, and finalization inherits the evaluation policy. + assert ( + _config(tmp_path / "ok", model="gpt-target", inference_gateway=gateway()).model + == "gpt-target" + ) + + with pytest.raises(ValidationError, match="evaluation allowed_models"): + _config(tmp_path / "search", model="gpt-typo", inference_gateway=gateway()) + + # An explicit finalization scope that forgets the task model starves the + # target agent during verification. + with pytest.raises(ValidationError, match="finalization allowed_models"): + _config( + tmp_path / "narrowed", + model="gpt-target", + inference_gateway=gateway( + finalization=InferenceBudgetSpec(allowed_models=["gpt-judge"]) + ), + ) + + # A per-target override is checked against finalization, not evaluation. + with pytest.raises(ValidationError, match="finalization allowed_models"): + _config( + tmp_path / "override", + model="gpt-target", + targets=[VerificationTargetSpec(partition="test", model="gpt-other")], + inference_gateway=gateway(), + ) + + # No gateway means no metering, so there is nothing to check against. + assert _config(tmp_path / "ungated", model="gpt-anything").model == "gpt-anything" + + +def _command_backend(tmp_path: Path) -> CommandBackendSpec: + harness = tmp_path / "harness" + harness.mkdir(parents=True) + (harness / "score.py").write_text("print('score')\n", encoding="utf-8") + return CommandBackendSpec( + harness_source=str(harness), + command=["python", "{harness}/score.py", "--report", "{report}"], + ) + + +def test_command_backend_compiles_a_task_with_no_target_agent(tmp_path): + """The outer loop stays a Harbor agent while the target is scored by a program. + + This is the Family B shape: a solver or index build has no agent to drive, so + there is nothing for a nested `harbor run` to do. + """ + config = _config( + tmp_path, + evaluation_backend="command", + command_backend=_command_backend(tmp_path / "cmd"), + agent_import_path=_OMIT, + task_source=_OMIT, + ) + assert config.agent_import_path is None + + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + backend = serve["backends"]["harbor-validation"] + assert backend["type"] == "command" + assert backend["harness_root"] == LAYOUT.harness + # Case enumeration is the harness's job, so it is told where the case files are. + assert backend["environment"]["VERO_CASES_DIR"] == LAYOUT.cases + # None of the nested-harbor plumbing leaks into a command backend. + assert "agent_import_path" not in backend + assert "task_source" not in backend + + # The harness is baked into the trusted sidecar, never the agent workspace. + assert (output / "environment/sidecar/harness/score.py").is_file() + dockerfile = (output / "environment/sidecar/Dockerfile").read_text(encoding="utf-8") + assert f"COPY sidecar/harness {LAYOUT.harness}" in dockerfile + + # And the trusted side still parses what the compiler emitted. + assert HarborDeploymentConfig.model_validate(serve).backends["harbor-validation"] + + +def test_deployment_treats_a_backend_without_a_type_as_harbor(tmp_path): + """serve.json written before the backend union existed must still load. + + A discriminated union needs the tag in the input, so without a default the + union would break any run resuming from an older compiled task. + """ + from vero.harbor.backend import HarborBackendConfig + + cases = tmp_path / "cases.jsonl" + cases.write_text('{"id": "a", "task_name": "a"}\n', encoding="utf-8") + backend = HarborBackendConfig( + task_source=str(tmp_path), + agent_import_path="target.agent:Agent", + cases_path=str(cases), + harbor_requirement="harbor==0.20.0", + ) + legacy = { + "repo_path": LAYOUT.trusted_repo, + "agent_repo_path": LAYOUT.target_repo, + "session_dir": LAYOUT.session_dir, + "admin_volume": LAYOUT.admin_volume, + "access_policies": [], + "targets": [], + "selection": {"mode": "submit"}, + "submit_enabled": True, + # The tag the older compiler never wrote. + "backends": { + "harbor-validation": backend.model_dump(mode="json", exclude={"type"}) + }, + } + parsed = HarborDeploymentConfig.model_validate(legacy) + assert parsed.backends["harbor-validation"].type == "harbor" + + +def test_build_config_keeps_the_two_backend_kinds_apart(tmp_path): + with pytest.raises(ValidationError, match="requires a command_backend"): + _config( + tmp_path / "missing", + evaluation_backend="command", + agent_import_path=_OMIT, + task_source=_OMIT, + ) + with pytest.raises(ValidationError, match="requires evaluation_backend: command"): + _config(tmp_path / "stray", command_backend=_command_backend(tmp_path / "s")) + with pytest.raises(ValidationError, match="requires: agent_import_path"): + _config(tmp_path / "noagent", agent_import_path=None) + with pytest.raises(ValidationError, match="only apply to a harbor"): + _config( + tmp_path / "tasksource", + evaluation_backend="command", + command_backend=_command_backend(tmp_path / "t"), + agent_import_path=_OMIT, + ) + # Harbor-only knobs are reported rather than silently ignored. + with pytest.raises(ValidationError, match="only apply to a harbor"): + _config( + tmp_path / "ignored", + evaluation_backend="command", + command_backend=_command_backend(tmp_path / "i"), + agent_import_path=_OMIT, + task_source=_OMIT, + max_retries=5, + feedback_transcripts=True, + ) + + +def test_every_harbor_only_field_is_rejected_by_a_command_build(tmp_path): + """The rejection list is derived, so this sweep needs no maintenance. + + The previous hand-written frozenset had already drifted: reward_key is + Harbor-only in fact -- a command harness reports its own reward -- but was + missing from the list, so setting it on a command build passed validation and + did nothing. Looping over the group means a field added to + _HarborEvaluationFields is covered the day it lands. + """ + assert _HARBOR_ONLY_FIELDS == frozenset(_HarborEvaluationFields.model_fields) + + for index, name in enumerate(sorted(_HARBOR_ONLY_FIELDS)): + # Its own default is enough: the check reads model_fields_set, so setting + # a field at all is the mistake, whatever the value. + default = HarborBuildConfig.model_fields[name].get_default( + call_default_factory=True + ) + # Listed last so it overrides the _OMIT when it is one of these two. + overrides = { + "evaluation_backend": "command", + "command_backend": _command_backend(tmp_path / f"h{index}"), + "agent_import_path": _OMIT, + "task_source": _OMIT, + name: default, + } + with pytest.raises(ValidationError, match="only apply to a harbor") as caught: + _config(tmp_path / f"harbor-only-{index}", **overrides) + assert name in str(caught.value) + + +def test_field_groups_are_inherited_not_nested(): + """Grouping must not change the wire format. + + The groups exist so the schema can be read one concern at a time, but a + build.yaml is flat and the checked-in benchmark configs depend on that. + Turning a group into a sub-object would break every one of them silently, so + assert the keys stay top-level -- and that every field belongs to a group, + which is what keeps each one documented somewhere. + """ + groups = ( + _TaskIdentityFields, + _HarborEvaluationFields, + _SearchAndSelectionFields, + _EvaluationLimitFields, + _TaskEnvironmentFields, + _AgentWorkspaceFields, + ) + top_level = set(HarborBuildConfig.model_fields) + for group in groups: + assert set(group.model_fields) <= top_level + + grouped = {name for group in groups for name in group.model_fields} + # Only the backend choice itself is declared on HarborBuildConfig. + assert grouped | {"evaluation_backend", "command_backend"} == top_level + + +def test_compiled_task_factory_path_resolves(tmp_path): + """The factory named in the compiled compose file must actually import. + + A stale dotted path fails when the sidecar container starts, not at import, + so it is invisible to the type checker and to every other test. Asserting on + the rendered artifact rather than on FACTORY_PATH alone means this still + holds if someone puts a literal back in the template. + """ + output = compile_harbor_task(_config(tmp_path), tmp_path / "compiled") + compose = yaml.safe_load( + (output / "environment/docker-compose.yaml").read_text(encoding="utf-8") + ) + command = compose["services"][LAYOUT.sidecar_host]["command"] + factory = command[command.index("--factory") + 1] + + assert factory == FACTORY_PATH + assert callable(load_factory(factory)) + + +def test_compiled_producer_base_url_matches_the_gateway_route(tmp_path): + """Every artifact naming the producer scope must agree with the served route. + + The URL used to be rebuilt by string concatenation in four places while the + route was declared in a fifth. A mismatch 404s or 403s inside the container + at run time, so asserting on the rendered artifacts is the only check that + would catch a template going back to concatenation. + """ + expected = LAYOUT.scope_url("producer", LAYOUT.optimizer_attribution) + config = _config( + tmp_path, + inference_gateway=InferenceGatewaySpec( + producer=InferenceBudgetSpec(allowed_models=["gpt-producer"]), + evaluation=InferenceBudgetSpec(allowed_models=["gpt-target"]), + ), + model="gpt-target", + ) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + + compose = yaml.safe_load( + (output / "environment/docker-compose.yaml").read_text(encoding="utf-8") + ) + assert compose["services"]["main"]["environment"]["OPENAI_BASE_URL"] == expected + launch = json.loads( + (output / "environment/gateway/launch.json").read_text(encoding="utf-8") + ) + assert launch["producer_base_url"] == expected + seed = (output / "environment/main/seed.sh").read_text(encoding="utf-8") + assert f'base_url = "{expected}"' in seed + + # And the path the gateway actually serves is the same string, not a copy. + assert LAYOUT.scope_route.startswith(LAYOUT.scope_route_base) + assert expected.endswith( + LAYOUT.scope_path("producer", LAYOUT.optimizer_attribution) + ) + + +def test_routed_credentials_are_set_not_merely_left_unblanked(tmp_path): + """Every name excluded from blanking must be positively set instead. + + The compiler blanks each declared secret in the main container, skipping + GATEWAY_ROUTED_CREDENTIALS because the compose template assigns those + itself. That exclusion is only safe while the two agree. A name dropped + from the blanking loop but never assigned would be neither blanked nor + set, so whatever the host exported would survive into the optimizer's + container -- which is the one thing the scrub exists to prevent. + """ + config = _config( + tmp_path, + inference_gateway=InferenceGatewaySpec( + producer=InferenceBudgetSpec(allowed_models=["gpt-producer"]), + evaluation=InferenceBudgetSpec(allowed_models=["gpt-target"]), + ), + ) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + compose = yaml.safe_load( + (output / "environment/docker-compose.yaml").read_text(encoding="utf-8") + ) + environment = compose["services"]["main"]["environment"] + + assert set(GATEWAY_ROUTED_CREDENTIALS) == set(LAYOUT.routed_credential_envs) + for name in GATEWAY_ROUTED_CREDENTIALS: + assert name in environment, f"{name} is skipped by the scrub but never set" + assert environment[name], f"{name} is set to an empty value" + + # And they carry the scoped values, not the upstream ones. + assert environment[LAYOUT.producer_api_key_env] != "" + assert environment[LAYOUT.producer_base_url_env] == LAYOUT.scope_url( + "producer", LAYOUT.optimizer_attribution + ) + # The raw upstream is blanked in the same container. + assert environment[LAYOUT.gateway_upstream_api_key_env] == "" + + +def test_load_build_config_resolves_relative_local_paths(tmp_path): + target = _target_repo(tmp_path / "target") + tasks = tmp_path / "tasks" + tasks.mkdir() + config_path = tmp_path / "build.yaml" + config_path.write_text( + "\n".join( + [ + "name: org/task", + "agent_repo: target", + "task_source: tasks", + "agent_import_path: target.agent:Agent", + "harbor_requirement: harbor==0.1.17", + "partitions:", + " validation: [org/a]", + " test: [org/b]", + "agent_access:", + " - partition: validation", + "selection_partition: validation", + "targets:", + " - partition: test", + ] + ) + + "\n", + encoding="utf-8", + ) + + loaded = load_harbor_build_config(config_path) + + assert loaded.agent_repo == str(target) + assert loaded.task_source == str(tasks) + + +def test_load_build_config_resolves_a_relative_harness_source(tmp_path): + """A command build's harness is relocatable like every other path field.""" + target = _target_repo(tmp_path / "target") + harness = tmp_path / "harness" + harness.mkdir() + (harness / "score.py").write_text("print('score')\n", encoding="utf-8") + config_path = tmp_path / "build.yaml" + config_path.write_text( + "\n".join( + [ + "name: org/solve", + f"agent_repo: {target.name}", + "harbor_requirement: harbor==0.1.17", + "partitions:", + " validation: [s0]", + " test: [s1]", + "agent_access:", + " - partition: validation", + " min_aggregate_cases: 1", + "selection_partition: validation", + "targets:", + " - partition: test", + "evaluation_backend: command", + "command_backend:", + f" harness_source: {harness.name}", + ' command: [python, "{harness}/score.py"]', + ] + ) + + "\n", + encoding="utf-8", + ) + + loaded = load_harbor_build_config(config_path) + + assert loaded.command_backend is not None + assert loaded.command_backend.harness_source == str(harness) + + +def test_load_build_config_supports_partition_files_and_validates_manifest(tmp_path): + target = _target_repo(tmp_path / "target") + partitions = tmp_path / "partitions" + partitions.mkdir() + (partitions / "validation.json").write_text( + '["task-a", "task-b"]\n', encoding="utf-8" + ) + (partitions / "test.json").write_text('["task-c"]\n', encoding="utf-8") + source = "org/benchmark@sha256:abc" + manifest = partitions / "manifest.json" + manifest.write_text( + json.dumps( + { + "task_source": source, + "tasks": [ + {"name": "task-a", "ref": "sha256:a"}, + {"name": "task-b", "ref": "sha256:b"}, + {"name": "task-c", "ref": "sha256:c"}, + ], + } + ), + encoding="utf-8", + ) + config_path = tmp_path / "build.yaml" + config_path.write_text( + "\n".join( + [ + "name: org/task", + "agent_repo: target", + f"task_source: {source}", + "task_manifest: partitions/manifest.json", + "agent_import_path: target.agent:Agent", + "harbor_requirement: harbor==0.20.0", + "partition_files:", + " validation: partitions/validation.json", + " test: partitions/test.json", + "agent_access:", + " - partition: validation", + "selection_partition: validation", + "targets:", + " - partition: test", + ] + ) + + "\n", + encoding="utf-8", + ) + + loaded = load_harbor_build_config(config_path) + + assert loaded.agent_repo == str(target) + assert loaded.partitions == { + "validation": ["task-a", "task-b"], + "test": ["task-c"], + } + assert loaded.task_manifest == str(manifest) + + (partitions / "test.json").write_text('["task-missing"]\n', encoding="utf-8") + with pytest.raises(ValidationError, match="absent from task_manifest"): + load_harbor_build_config(config_path) + + +def test_load_build_config_matches_vendored_local_task_source(tmp_path): + # A local (vendored) task_source is resolved to an absolute path by the + # loader once the directory exists, while the committed manifest records it + # relative to itself; the two must still be recognized as the same source. + _target_repo(tmp_path / "target") + tasks = tmp_path / "tasks" + tasks.mkdir() + partitions = tmp_path / "partitions" + partitions.mkdir() + (partitions / "validation.json").write_text('["task-a"]\n', encoding="utf-8") + (partitions / "test.json").write_text('["task-a"]\n', encoding="utf-8") + (partitions / "manifest.json").write_text( + json.dumps( + { + "task_source": "../tasks", + "tasks": [{"name": "task-a", "ref": "sha256:a"}], + } + ), + encoding="utf-8", + ) + config_path = tmp_path / "build.yaml" + config_path.write_text( + "\n".join( + [ + "name: org/task", + "agent_repo: target", + "task_source: tasks", + "task_manifest: partitions/manifest.json", + "agent_import_path: target.agent:Agent", + "harbor_requirement: harbor==0.20.0", + "partition_files:", + " validation: partitions/validation.json", + " test: partitions/test.json", + "agent_access:", + " - partition: validation", + "selection_partition: validation", + "targets:", + " - partition: test", + ] + ) + + "\n", + encoding="utf-8", + ) + + loaded = load_harbor_build_config(config_path) + assert loaded.task_source == str(tasks.resolve()) + + # A genuinely different source still fails. + (partitions / "manifest.json").write_text( + json.dumps( + { + "task_source": "../elsewhere", + "tasks": [{"name": "task-a", "ref": "sha256:a"}], + } + ), + encoding="utf-8", + ) + with pytest.raises(ValidationError, match="does not match build task_source"): + load_harbor_build_config(config_path) + + +def test_compiler_emits_isolated_canonical_harbor_task(tmp_path): + config = _config(tmp_path) + output = compile_harbor_task( + config, + tmp_path / "compiled", + vero_root=Path(__file__).parents[1], + ) + + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert set(serve["backends"]) == {"harbor-validation", "harbor-test"} + access = serve["access_policies"][0]["access"] + assert access["disclosure"] == "aggregate" + assert access["expose_case_resources"] is True + assert access["min_aggregate_cases"] == 5 # safe floor survives compilation + assert serve["budgets"][0]["total_runs"] == 5 + assert serve["selection"]["backend_id"] == "harbor-validation" + assert serve["targets"][0]["backend_id"] == "harbor-test" + assert serve["targets"][0]["reward_scale"] == 1.0 + # A grace period, not a ceiling — and explicitly NOT timeout_seconds, which is + # sized to be unreachable. Inheriting it stalled officeqa run #4's + # finalization for hours behind a sub-run that had already finished its work. + assert serve["evaluation_drain_timeout_seconds"] == 600.0 + assert serve["evaluation_drain_timeout_seconds"] != config.timeout_seconds + assert serve["backends"]["harbor-test"]["task_source"] == LAYOUT.task_source + assert serve["backends"]["harbor-test"]["python_version"] == "3.12" + assert serve["backends"]["harbor-test"]["case_timeout_seconds"] == 180.0 + assert serve["backends"]["harbor-test"]["task_agent_timeout_seconds"] == 600.0 + assert ( + serve["backends"]["harbor-validation"]["case_resources_cache_path"] + == f"{LAYOUT.case_resources_dir}/validation" + ) + assert serve["access_policies"][0]["limits"]["retry"]["max_attempts"] == 1 + assert serve["access_policies"][0]["limits"]["case_timeout_seconds"] == 180.0 + assert "use_evaluation_copies" not in serve + instruction = (output / "instruction.md").read_text() + assert "--detach" in instruction + assert "evals status JOB_ID" in instruction + assert "randomness remains" in instruction + for partition, backend in serve["backends"].items(): + partition_name = partition.removeprefix("harbor-") + backend["cases_path"] = str( + output / f"environment/sidecar/cases/{partition_name}.jsonl" + ) + HarborDeploymentConfig.model_validate(serve) + test_case = json.loads( + (output / "environment/sidecar/cases/test.jsonl").read_text(encoding="utf-8") + ) + assert test_case == { + "id": "task-hidden", + "task_name": "task-hidden", + "result_task_name": "org/task-hidden", + } + assert (output / "environment/sidecar/task-source/task-hidden/task.toml").is_file() + assert not (output / "environment/agent-seed/protected-tasks").exists() + instruction = (output / "instruction.md").read_text(encoding="utf-8") + assert "## Objective\n\nImprove the program" in instruction + assert "--backend harbor-validation" in instruction + # k-anonymity floor (default 5) is disclosed to the agent + assert "must include at least 5 cases" in instruction + assert "Complete task resources" in instruction + assert ".evals/results/" in instruction + task_toml = (output / "task.toml").read_text(encoding="utf-8") + assert 'name = "org/optimize-\\"program\\""' in task_toml + assert tomllib.loads(task_toml)["task"]["name"] == 'org/optimize-"program"' + compose = (output / "environment/docker-compose.yaml").read_text() + assert "vero.harbor.deployment:build_harbor_components" in compose + assert f"admin_state:{LAYOUT.admin_volume}" in compose + assert f"agent_context:{LAYOUT.target_evals}:ro" in compose + assert f"agent_context:{LAYOUT.agent_volume}" in compose + assert set(yaml.safe_load(compose)["services"]) == {"main", "eval-sidecar"} + sidecar_dockerfile = (output / "environment/sidecar/Dockerfile").read_text() + assert 'uv pip install --system "harbor==0.1.17"' in sidecar_dockerfile + # the unprivileged harness user exists and gets a pre-warmed uv cache, so + # `uv run` as harness can resolve the candidate package offline + assert "useradd -m -u 1002 harness" in sidecar_dockerfile + assert "chown -R harness:harness /home/harness/.cache" in sidecar_dockerfile + seed = (output / "environment/main/seed.sh").read_text() + assert f"-path {LAYOUT.target_evals} -prune" in seed + assert f"'/.evals/' >> {LAYOUT.target_git_exclude}" in seed + test_script = output / "tests/test.sh" + assert test_script.stat().st_mode & 0o111 + assert "vero harbor export-session" in test_script.read_text() + + +def test_compiler_checks_secrets_before_writing_and_rejects_source_overlap( + tmp_path, + monkeypatch, +): + config = _config(tmp_path, secrets=["MISSING_TEST_SECRET"]) + output = tmp_path / "compiled" + monkeypatch.delenv("MISSING_TEST_SECRET", raising=False) + + with pytest.raises(ValueError, match="MISSING_TEST_SECRET"): + compile_harbor_task( + config, + output, + vero_root=Path(__file__).parents[1], + ) + assert not output.exists() + + monkeypatch.setenv("MISSING_TEST_SECRET", "configured") + compile_harbor_task( + config, + output, + vero_root=Path(__file__).parents[1], + ) + task = tomllib.loads((output / "task.toml").read_text(encoding="utf-8")) + assert task["environment"]["env"] == { + "MISSING_TEST_SECRET": "${MISSING_TEST_SECRET}" + } + + safe = config.model_copy(update={"secrets": []}) + with pytest.raises(ValueError, match="overlaps protected source"): + compile_harbor_task( + safe, + safe.agent_repo, + vero_root=Path(__file__).parents[1], + ) + + (Path(safe.agent_repo) / ".evals").mkdir() + (Path(safe.agent_repo) / ".evals" / "context.json").write_text("{}\n") + _git(Path(safe.agent_repo), "add", "-f", ".evals/context.json") + _git(Path(safe.agent_repo), "commit", "-q", "-m", "reserved context") + with pytest.raises(ValueError, match="reserved path"): + compile_harbor_task( + safe, + tmp_path / "reserved-context", + vero_root=Path(__file__).parents[1], + ) + + +def test_compiler_isolates_upstream_inference_credentials(tmp_path, monkeypatch): + monkeypatch.setenv("TEST_UPSTREAM_KEY", "real-provider-secret") + monkeypatch.setenv("TEST_UPSTREAM_URL", "https://provider.example/v1") + monkeypatch.setenv("TEST_MODAL_TOKEN", "modal-secret") + config = _config( + tmp_path, + secrets=["TEST_MODAL_TOKEN"], + inference_gateway=InferenceGatewaySpec( + upstream_api_key_env="TEST_UPSTREAM_KEY", + upstream_base_url_env="TEST_UPSTREAM_URL", + producer=InferenceBudgetSpec( + allowed_models=["gpt-producer"], + max_requests=10, + max_tokens=1000, + ), + evaluation=InferenceBudgetSpec( + allowed_models=["gpt-target"], + max_requests=20, + max_tokens=2000, + ), + ), + ) + + output = compile_harbor_task( + config, + tmp_path / "compiled", + vero_root=Path(__file__).parents[1], + ) + + task = tomllib.loads((output / "task.toml").read_text()) + assert task["environment"]["env"] == { + "TEST_MODAL_TOKEN": "${TEST_MODAL_TOKEN}", + "VERO_INFERENCE_UPSTREAM_API_KEY": "${VERO_INFERENCE_UPSTREAM_API_KEY}", + "VERO_INFERENCE_UPSTREAM_BASE_URL": "${VERO_INFERENCE_UPSTREAM_BASE_URL}", + } + compose = yaml.safe_load((output / "environment/docker-compose.yaml").read_text()) + assert set(compose["services"]) == { + "main", + "eval-sidecar", + "inference-gateway", + } + main_environment = compose["services"]["main"]["environment"] + assert main_environment["TEST_MODAL_TOKEN"] == "" + assert main_environment["VERO_INFERENCE_UPSTREAM_API_KEY"] == "" + assert main_environment["OPENAI_API_KEY"] + assert main_environment["OPENAI_API_KEY"] != "real-provider-secret" + assert main_environment["OPENAI_BASE_URL"].endswith("/scopes/producer/optimizer/v1") + assert compose["services"]["eval-sidecar"]["environment"] == { + "TEST_MODAL_TOKEN": "${TEST_MODAL_TOKEN:?TEST_MODAL_TOKEN must be set for the eval sidecar}" + } + assert compose["services"]["inference-gateway"]["environment"] == { + "VERO_INFERENCE_UPSTREAM_API_KEY": "${VERO_INFERENCE_UPSTREAM_API_KEY:?VERO_INFERENCE_UPSTREAM_API_KEY must be set for the inference gateway}", + "VERO_INFERENCE_UPSTREAM_BASE_URL": "${VERO_INFERENCE_UPSTREAM_BASE_URL:?VERO_INFERENCE_UPSTREAM_BASE_URL must be set for the inference gateway}", + } + gateway = json.loads((output / "environment/gateway/config.json").read_text()) + assert "real-provider-secret" not in json.dumps(gateway) + assert gateway["scopes"]["producer"]["token_sha256"] + launch = json.loads((output / "environment/gateway/launch.json").read_text()) + assert launch["upstream_api_key_source"] == "TEST_UPSTREAM_KEY" + assert launch["upstream_api_key_target"] == "VERO_INFERENCE_UPSTREAM_API_KEY" + assert launch["producer_api_key"] == main_environment["OPENAI_API_KEY"] + seed = (output / "environment/main/seed.sh").read_text() + assert 'model_provider = "vero_gateway"' in seed + assert "supports_websockets = false" in seed + serve = json.loads((output / "environment/sidecar/serve.json").read_text()) + backend = serve["backends"]["harbor-validation"] + assert backend["passthrough_environment"] == ["TEST_MODAL_TOKEN"] + assert backend["inference_gateway_token"] + assert backend["inference_gateway_token"] != "real-provider-secret" + assert "real-provider-secret" not in json.dumps(serve) + assert (output / "environment/gateway/Dockerfile").is_file() + + +def test_compiler_uses_published_version_outside_a_source_checkout( + tmp_path, + monkeypatch, +): + config = _config(tmp_path) + from vero.harbor.build import compiler + + monkeypatch.setattr( + compiler, + "__file__", + "/installed/site-packages/vero/harbor/build/compiler.py", + ) + monkeypatch.setattr(compiler, "distribution_version", lambda _name: "0.5.0") + + output = compiler.compile_harbor_task(config, tmp_path / "compiled") + + assert not (output / "environment/vero").exists() + dockerfile = (output / "environment/Dockerfile").read_text(encoding="utf-8") + assert "scale-vero[harbor]==0.5.0" in dockerfile + + +def test_agent_access_defaults_to_safe_k_anonymity_floor(): + # Omitting min_aggregate_cases must yield a real floor (5), not a no-op (1). + assert AgentAccessSpec(partition="validation").min_aggregate_cases == 5 + + +def test_run_command_forwards_agent_env_as_harbor_ae(tmp_path, monkeypatch): + # `config.agent_env` must be rendered as repeated `--ae KEY=VALUE` on the + # optimizer's `harbor run` so it reaches the agent's setup/install exec + # (harbor injects --ae into the agent's extra_env / scoped_exec_env). This is + # the only supported channel for e.g. UV_TOOL_BIN_DIR on a non-root sandbox; + # extra_harbor_args only flows into the eval sub-run, not this command. + from click.testing import CliRunner + + from vero.harbor import build as harbor_build + from vero.harbor import cli as harbor_cli + + config = _config( + tmp_path, + agent_env={"UV_TOOL_BIN_DIR": "/home/agent/.local/bin", "FOO": "bar"}, + ) + config_path = tmp_path / "build.yaml" + config_path.write_text("name: placeholder\n", encoding="utf-8") + + captured: dict[str, list[str]] = {} + + def fake_run(command, *args, **kwargs): + captured["command"] = list(command) + return subprocess.CompletedProcess(command, 0) + + monkeypatch.setattr(harbor_cli.shutil, "which", lambda _name: "/usr/bin/uvx") + monkeypatch.setattr(harbor_cli.subprocess, "run", fake_run) + monkeypatch.setattr( + harbor_build, "load_harbor_build_config", lambda *a, **k: config + ) + monkeypatch.setattr(harbor_build, "compile_harbor_task", lambda cfg, out: out) + + result = CliRunner().invoke( + harbor_cli.harbor, + [ + "run", + "--config", + str(config_path), + "--agent", + "codex", + "--model", + "gpt-5.4", + "--environment", + "modal", + ], + ) + + assert result.exit_code == 0, result.output + command = captured["command"] + assert "--ae" in command + assert "--ae UV_TOOL_BIN_DIR=/home/agent/.local/bin" in shlex.join(command) + assert "--ae FOO=bar" in shlex.join(command) + # Deterministic (sorted-by-key) ordering: FOO before UV_TOOL_BIN_DIR. + joined = shlex.join(command) + assert joined.index("--ae FOO=bar") < joined.index("--ae UV_TOOL_BIN_DIR=") + + +def _serve_backends(tmp_path: Path, **updates) -> dict: + config = _config(tmp_path, **updates) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + serve = json.loads( + (output / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + return serve["backends"] + + +def test_verification_target_spec_defaults_n_attempts_none(): + target = VerificationTargetSpec(partition="test") + assert target.n_attempts is None + assert target.aggregate_attempts is None + + +def test_compiler_applies_per_target_n_attempts_override(tmp_path): + # Only the held-out test backend gets 3x/mean; validation (search/selection) + # keeps the global n_attempts=1. + backends = _serve_backends( + tmp_path, + n_attempts=1, + targets=[ + VerificationTargetSpec( + partition="test", n_attempts=3, aggregate_attempts="mean" + ) + ], + ) + assert backends["harbor-test"]["n_attempts"] == 3 + assert backends["harbor-test"]["aggregate_attempts"] == "mean" + assert backends["harbor-validation"]["n_attempts"] == 1 + + +def test_compiler_defaults_target_n_attempts_to_global(tmp_path): + backends = _serve_backends(tmp_path, n_attempts=2) + # No per-target override -> every backend inherits the global (backward-compat). + assert backends["harbor-test"]["n_attempts"] == 2 + assert backends["harbor-validation"]["n_attempts"] == 2 + + +def test_build_rejects_n_attempts_override_on_agent_partition(tmp_path): + # validation is agent-evaluable + the selection partition; its backend is + # shared with search, so an override there must be rejected. + with pytest.raises(ValidationError, match="agent-evaluable partition"): + _config( + tmp_path, + targets=[VerificationTargetSpec(partition="validation", n_attempts=3)], + ) + + +def test_instruction_omits_retired_insights_paragraph(tmp_path): + # The insights-generator subagent / skills/insights overlay was retired; the + # rendered optimizer instruction must not advertise it (it did whenever the + # always-baked evals skill made overlay_present true). + config = _config(tmp_path) + output = compile_harbor_task( + config, tmp_path / "compiled", vero_root=Path(__file__).parents[1] + ) + instruction = (output / "instruction.md").read_text(encoding="utf-8") + assert "insights-generator" not in instruction + assert "skills/insights/" not in instruction + + +def test_instruction_advertises_seed_only_where_the_backend_accepts_it(tmp_path): + # HarborBackend.validate_request rejects request.seed outright, so telling a + # Harbor-scored optimizer to "pass --seed N to reproduce a comparison" sends + # it into a guaranteed 400. Observed live: the optimizer followed that advice + # and burned a round trip on `invalid evaluation request`. + harbor = compile_harbor_task( + _config(tmp_path), tmp_path / "harbor", vero_root=Path(__file__).parents[1] + ) + instruction = (harbor / "instruction.md").read_text(encoding="utf-8") + assert "--seed" in instruction # still named, so the rejection is not a surprise + assert "rejects `--seed`" in instruction + assert "re-run the identical selection" in instruction + + # A command backend does its own sampling and accepts the seed. + command = compile_harbor_task( + _config( + tmp_path / "cmd-config", + evaluation_backend="command", + command_backend=_command_backend(tmp_path / "cmd"), + agent_import_path=_OMIT, + task_source=_OMIT, + ), + tmp_path / "command", + vero_root=Path(__file__).parents[1], + ) + instruction = (command / "instruction.md").read_text(encoding="utf-8") + assert "reproduce a noisy comparison exactly" in instruction + assert "rejects `--seed`" not in instruction + + +def test_drain_timeout_is_independent_of_the_unreachable_eval_ceiling(tmp_path): + # timeout_seconds is deliberately set above ceil(trials/concurrency) x + # case_timeout so it can never fire. The drain is the opposite kind of clock: + # it decides how long finalization waits on already-running agent evaluations + # before cancelling them, and cancellation is graceful. Tying the two makes a + # hung sub-run cost hours of held-out scoring, which is what happened. + default = compile_harbor_task( + _config(tmp_path / "cfg-default", timeout_seconds=90000.0), + tmp_path / "default", + vero_root=Path(__file__).parents[1], + ) + serve = json.loads( + (default / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert serve["evaluation_drain_timeout_seconds"] == 600.0 + + # An explicit value still wins, for a benchmark that genuinely wants to wait. + explicit = compile_harbor_task( + _config( + tmp_path / "cfg-explicit", + timeout_seconds=90000.0, + evaluation_drain_timeout_seconds=1800.0, + ), + tmp_path / "explicit", + vero_root=Path(__file__).parents[1], + ) + serve = json.loads( + (explicit / "environment/sidecar/serve.json").read_text(encoding="utf-8") + ) + assert serve["evaluation_drain_timeout_seconds"] == 1800.0 diff --git a/vero/tests/test_v05_harbor_deployment.py b/vero/tests/test_v05_harbor_deployment.py new file mode 100644 index 00000000..da61367c --- /dev/null +++ b/vero/tests/test_v05_harbor_deployment.py @@ -0,0 +1,272 @@ +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from vero.evaluation import ( + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationBudget, + EvaluationSet, + MetricSelector, + ObjectiveSpec, +) +from vero.harbor import ( + HarborBackendConfig, + build_harbor_components, +) +from vero.report import generate_experiment_report +from vero.sidecar import ( + SidecarEvaluationPolicy, + VerificationTarget, +) + + +def _git(path: Path, *arguments: str) -> str: + result = subprocess.run( + ["git", *arguments], + cwd=path, + check=True, + text=True, + capture_output=True, + ) + return result.stdout.strip() + + +def _repo(path: Path, content: str) -> str: + path.mkdir(parents=True) + _git(path, "init", "-q") + _git(path, "config", "user.name", "VeRO Test") + _git(path, "config", "user.email", "vero@example.test") + (path / "program.py").write_text(content, encoding="utf-8") + _git(path, "add", "program.py") + _git(path, "commit", "-q", "-m", "baseline") + return _git(path, "rev-parse", "HEAD") + + +@pytest.mark.asyncio +async def test_standard_deployment_factory_builds_one_canonical_runtime(tmp_path): + trusted = tmp_path / "trusted" + agent = tmp_path / "agent" + baseline = _repo(trusted, "VALUE = 1\n") + _repo(agent, "VALUE = 1\n") + cases = tmp_path / "cases.jsonl" + cases.write_text( + json.dumps({"id": "task", "task_name": "org/task"}) + "\n", + encoding="utf-8", + ) + evaluation_set = EvaluationSet(name="benchmark", partition="validation") + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + backend_config = HarborBackendConfig( + task_source="org/benchmark@1.0", + agent_import_path="program:Agent", + cases_path=str(cases), + harbor_requirement="harbor==0.1.17", + evaluation_set_name="benchmark", + partition="validation", + uv_executable=sys.executable, + ) + budget = EvaluationBudget( + backend_id="validation", + evaluation_set_key=evaluation_set.budget_key("validation"), + total_runs=4, + total_cases=10, + ) + config = { + "repo_path": str(trusted), + "agent_repo_path": str(agent), + "session_dir": str(tmp_path / "state/session"), + "session_id": "trial", + "backends": {"validation": backend_config.model_dump(mode="json")}, + "access_policies": [ + SidecarEvaluationPolicy( + backend_id="validation", + evaluation_set_name="benchmark", + partition="validation", + objective=objective, + access=EvaluationAccessPolicy( + disclosure=DisclosureLevel.AGGREGATE + ), + ).model_dump(mode="json") + ], + "budgets": [budget.model_dump(mode="json")], + "selection": { + "mode": "auto_best", + "backend_id": "validation", + "evaluation_set": evaluation_set.model_dump(mode="json"), + "objective": objective.model_dump(mode="json"), + "baseline_version": "HEAD", + }, + "targets": [ + VerificationTarget( + reward_key="reward", + backend_id="validation", + evaluation_set=evaluation_set, + objective=objective, + ).model_dump(mode="json") + ], + "agent_volume": str(tmp_path / "state/agent"), + "admin_volume": str(tmp_path / "state/admin"), + } + + components = await build_harbor_components(config) + + assert components.sidecar.engine is components.verifier.engine + assert components.verifier.selection.baseline_candidate.version == baseline + assert components.sidecar.status().evaluation_access[0].budget.remaining_runs == 4 + assert (tmp_path / "state/session/budgets.json").is_file() + # The trusted session dir is locked to its owner so the unprivileged harness + # cannot read the held-out records, budgets, or candidates it contains. + assert ((tmp_path / "state/session").stat().st_mode & 0o777) == 0o700 + manifest = json.loads((tmp_path / "state/session/harbor-session.json").read_text()) + assert manifest["id"] == "trial" + assert manifest["selection"]["evaluation_set"] == { + "name": "benchmark", + "partition": "validation", + "selection": {"kind": "all"}, + } + assert (tmp_path / "state/agent/manifest.json").is_file() + assert json.loads( + (tmp_path / "state/agent/results/index.json").read_text() + ) == {"schema_version": 1, "evaluations": []} + report = await generate_experiment_report( + tmp_path / "state/session", + tmp_path / "experiment.html", + ) + assert baseline in report.read_text() + + +@pytest.mark.asyncio +async def test_standard_deployment_fails_closed_on_corrupt_budget_state(tmp_path): + trusted = tmp_path / "trusted" + agent = tmp_path / "agent" + _repo(trusted, "VALUE = 1\n") + _repo(agent, "VALUE = 1\n") + cases = tmp_path / "cases.json" + cases.write_text('[{"id":"task","task_name":"org/task"}]') + session = tmp_path / "state/session" + session.mkdir(parents=True) + (session / "budgets.json").write_text("not json", encoding="utf-8") + evaluation_set = EvaluationSet(name="benchmark") + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + config = { + "repo_path": str(trusted), + "agent_repo_path": str(agent), + "session_dir": str(session), + "backends": { + "backend": { + "task_source": "org/benchmark@1.0", + "agent_import_path": "program:Agent", + "cases_path": str(cases), + "harbor_requirement": "harbor==0.1.17", + "uv_executable": sys.executable, + } + }, + "access_policies": [], + "budgets": [], + "selection": { + "mode": "auto_best", + "backend_id": "backend", + "evaluation_set": evaluation_set.model_dump(mode="json"), + "objective": objective.model_dump(mode="json"), + "baseline_version": "HEAD", + "baseline_floor": False, + }, + "targets": [ + { + "reward_key": "reward", + "backend_id": "backend", + "evaluation_set": evaluation_set.model_dump(mode="json"), + "objective": objective.model_dump(mode="json"), + } + ], + "admin_volume": str(tmp_path / "state/admin"), + } + + with pytest.raises(ValueError, match="invalid durable budget ledger"): + await build_harbor_components(config) + + +@pytest.mark.asyncio +async def test_deployment_finalize_hook_archives_gateway_state(tmp_path): + # Even without W&B configured, finalize must copy the gateway's usage + # ledger and request log into the session so /session/export carries them. + trusted = tmp_path / "trusted" + agent = tmp_path / "agent" + _repo(trusted, "VALUE = 1\n") + _repo(agent, "VALUE = 1\n") + cases = tmp_path / "cases.json" + cases.write_text('[{"id":"task","task_name":"org/task"}]') + usage_path = tmp_path / "state/inference/usage.json" + usage_path.parent.mkdir(parents=True) + usage_path.write_text('{"schema_version": 1, "scopes": {}}') + requests_dir = tmp_path / "state/inference/requests" + requests_dir.mkdir() + (requests_dir / "requests-00001.jsonl").write_text('{"scope":"producer"}\n') + evaluation_set = EvaluationSet(name="benchmark") + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + session_dir = tmp_path / "state/session" + config = { + "repo_path": str(trusted), + "agent_repo_path": str(agent), + "session_dir": str(session_dir), + "backends": { + "backend": { + "task_source": "org/benchmark@1.0", + "agent_import_path": "program:Agent", + "cases_path": str(cases), + "harbor_requirement": "harbor==0.1.17", + "uv_executable": sys.executable, + } + }, + "access_policies": [], + "budgets": [], + "selection": { + "mode": "auto_best", + "backend_id": "backend", + "evaluation_set": evaluation_set.model_dump(mode="json"), + "objective": objective.model_dump(mode="json"), + "baseline_version": "HEAD", + }, + "targets": [ + { + "reward_key": "reward", + "backend_id": "backend", + "evaluation_set": evaluation_set.model_dump(mode="json"), + "objective": objective.model_dump(mode="json"), + } + ], + "admin_volume": str(tmp_path / "state/admin"), + "inference_usage_path": str(usage_path), + "inference_request_log_dir": str(requests_dir), + } + + components = await build_harbor_components(config) + # No W&B -> no poller, but the finalize hook is still installed. + assert components.telemetry is None + assert components.verifier._on_finalized is not None + + class Result: + shipped = True + rewards = {} + + components.verifier._on_finalized(Result()) + + exported = session_dir / "artifacts/inference" + assert (exported / "usage.json").read_text().startswith('{"schema_version"') + assert ( + exported / "requests/requests-00001.jsonl" + ).read_text() == '{"scope":"producer"}\n' diff --git a/vero/tests/test_v05_harbor_http.py b/vero/tests/test_v05_harbor_http.py new file mode 100644 index 00000000..0b4ae9d6 --- /dev/null +++ b/vero/tests/test_v05_harbor_http.py @@ -0,0 +1,668 @@ +from __future__ import annotations + +import json +import os +import stat +from datetime import UTC, datetime +from types import SimpleNamespace + +import click +import pytest +from click.testing import CliRunner +from fastapi.testclient import TestClient + +from vero.candidate import Candidate +from vero.cli import main +from vero.evaluation import ( + DisclosureLevel, + EvaluationAcknowledgement, + EvaluationDatabase, + EvaluationReceipt, + EvaluationRequestError, + EvaluationStatus, +) +from vero.harbor.cli import ( + _compiled_run_environment, + _load_agent_trace, + _load_env_file, + harbor, +) +from vero.sidecar.app import create_app +from vero.sidecar.auth import ( + check_admin_token, + read_admin_token, + write_admin_token, +) +from vero.sidecar.sidecar import ( + EvaluationAccessError, + EvaluationJobNotFoundError, + EvaluationJobStatus, + SidecarEvaluationJob, + SidecarEvaluationResult, + SidecarStatus, + Submission, +) +from vero.sidecar.verifier import VerificationResult + + +class FakeSidecar: + def __init__(self): + self.requests = [] + self.raise_access_error = False + self.raise_request_error = False + self.job = None + self.engine = SimpleNamespace(database=EvaluationDatabase(id="session")) + + async def evaluate(self, request): + if self.raise_access_error: + raise EvaluationAccessError("private details") + if self.raise_request_error: + raise EvaluationRequestError("unknown case") + self.requests.append(request) + return SidecarEvaluationResult( + disclosure=DisclosureLevel.NONE, + receipt=EvaluationReceipt( + evaluation_id="evaluation", + status=EvaluationStatus.SUCCESS, + disclosure=DisclosureLevel.NONE, + result=EvaluationAcknowledgement( + evaluation_id="evaluation", + status=EvaluationStatus.SUCCESS, + ), + result_path=".evals/results/evaluation/evaluation.json", + ), + ) + + async def submit(self, version=None): + return Submission( + candidate=Candidate( + id="candidate", + version=version or "HEAD", + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + ) + + async def start_evaluation_job(self, request): + result = await self.evaluate(request) + self.job = SidecarEvaluationJob( + job_id="job-1", + status=EvaluationJobStatus.COMPLETE, + backend_id=request.backend_id, + evaluation_set=request.evaluation_set, + version=request.version, + evaluation_id=result.receipt.evaluation_id, + receipt=result.receipt, + created_at=datetime(2026, 1, 1, tzinfo=UTC), + completed_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + return self.job + + def evaluation_job(self, job_id): + if self.job is None or self.job.job_id != job_id: + raise EvaluationJobNotFoundError(job_id) + return self.job + + def status(self): + return SidecarStatus(submit_enabled=True, evaluation_access=[]) + + +class FakeVerifier: + def __init__(self): + self.calls = 0 + + async def finalize(self): + self.calls += 1 + return VerificationResult(rewards={"reward": 0.75}) + + +def test_admin_token_is_atomic_restrictive_and_constant_time_checked(tmp_path): + path = write_admin_token(tmp_path / "admin/token", "secret-token") + + assert read_admin_token(path) == "secret-token" + # Read-only file inside a root-only directory: an unprivileged agent that + # shares the token volume can neither read the file nor traverse to it. + assert stat.S_IMODE(path.stat().st_mode) == 0o400 + assert stat.S_IMODE(path.parent.stat().st_mode) == 0o700 + assert check_admin_token("Bearer secret-token", "secret-token") + assert not check_admin_token("Bearer wrong", "secret-token") + assert not check_admin_token(None, "secret-token") + + +def test_compiled_run_environment_keeps_upstream_credentials_from_agent( + tmp_path, monkeypatch +): + monkeypatch.setenv("OPENAI_API_KEY", "upstream-secret") + monkeypatch.setenv("OPENAI_BASE_URL", "https://provider.example/v1") + launch = tmp_path / "environment/gateway/launch.json" + launch.parent.mkdir(parents=True) + launch.write_text( + json.dumps( + { + "upstream_api_key_source": "OPENAI_API_KEY", + "upstream_api_key_target": "VERO_INFERENCE_UPSTREAM_API_KEY", + "upstream_base_url_source": "OPENAI_BASE_URL", + "upstream_base_url_target": "VERO_INFERENCE_UPSTREAM_BASE_URL", + "producer_api_key": "producer-scope-token", + "producer_base_url": "http://inference/scopes/producer/optimizer/v1", + } + ) + ) + + environment = _compiled_run_environment(tmp_path) + + assert environment["OPENAI_API_KEY"] == "producer-scope-token" + assert environment["OPENAI_BASE_URL"].startswith("http://inference/") + assert environment["VERO_INFERENCE_UPSTREAM_API_KEY"] == "upstream-secret" + assert environment["VERO_INFERENCE_UPSTREAM_BASE_URL"] == ( + "https://provider.example/v1" + ) + # Claude Code (harbor -a claude) is pointed at the same producer scope; the + # Anthropic SDK re-appends "/v1/messages" so the base drops the trailing /v1. + assert environment["ANTHROPIC_API_KEY"] == "producer-scope-token" + assert environment["ANTHROPIC_BASE_URL"] == ( + "http://inference/scopes/producer/optimizer" + ) + + +def test_load_env_file_parses_comments_export_and_quotes(tmp_path): + path = tmp_path / "secrets.env" + path.write_text( + "# a comment\n" + "\n" + "OPENAI_API_KEY=upstream-secret\n" + "export MODAL_TOKEN_ID='mt-id'\n" + 'MODAL_TOKEN_SECRET="mt-secret"\n' + ) + + values = _load_env_file(path) + + assert values == { + "OPENAI_API_KEY": "upstream-secret", + "MODAL_TOKEN_ID": "mt-id", + "MODAL_TOKEN_SECRET": "mt-secret", + } + + +def test_load_env_file_rejects_malformed_line(tmp_path): + path = tmp_path / "secrets.env" + path.write_text("NOT_AN_ASSIGNMENT\n") + + with pytest.raises(click.ClickException): + _load_env_file(path) + + +def test_env_file_overrides_take_precedence_over_ambient_environment( + tmp_path, monkeypatch +): + monkeypatch.setenv("OPENAI_API_KEY", "ambient-wrong-key") + monkeypatch.delenv("MODAL_TOKEN_ID", raising=False) + launch = tmp_path / "environment/gateway/launch.json" + launch.parent.mkdir(parents=True) + launch.write_text( + json.dumps( + { + "upstream_api_key_source": "OPENAI_API_KEY", + "upstream_api_key_target": "VERO_INFERENCE_UPSTREAM_API_KEY", + "producer_api_key": "producer-scope-token", + "producer_base_url": "http://inference/scopes/producer/optimizer/v1", + "upstream_base_url_target": "VERO_INFERENCE_UPSTREAM_BASE_URL", + } + ) + ) + + environment = _compiled_run_environment( + tmp_path, + {"OPENAI_API_KEY": "file-upstream-secret", "MODAL_TOKEN_ID": "mt-id"}, + ) + + # The upstream credential relocated to the gateway target comes from the + # env-file, not the ambient shell. + assert environment["VERO_INFERENCE_UPSTREAM_API_KEY"] == "file-upstream-secret" + assert environment["OPENAI_API_KEY"] == "producer-scope-token" + assert environment["MODAL_TOKEN_ID"] == "mt-id" + + +def test_codex_jsonl_is_converted_to_a_redacted_producer_trace(tmp_path): + path = tmp_path / "codex.txt" + path.write_text( + "\n".join( + [ + json.dumps( + { + "type": "item.completed", + "item": {"type": "agent_message", "text": "Inspecting."}, + } + ), + json.dumps( + { + "type": "item.completed", + "item": { + "type": "command_execution", + "command": "env", + "aggregated_output": "OPENAI_API_KEY=sk-secretvalue123\n", + }, + } + ), + ] + ) + + "\n" + ) + + trace = _load_agent_trace(path) + + assert trace[0] == {"role": "assistant", "content": "Inspecting."} + assert trace[-1]["output"] == "OPENAI_API_KEY=[REDACTED]\n" + + +def test_harbor_run_uses_current_python_and_pinned_harbor_extra(tmp_path, monkeypatch): + import vero.harbor.build as harbor_build + import vero.harbor.cli as harbor_cli + + config_path = tmp_path / "build.yaml" + config_path.write_text("task_name: unused\n") + config = SimpleNamespace( + harbor_requirement="harbor[modal]==0.20.0", + agent_env={}, + # Real configs always carry a name; the outer trial derives its + # Modal app name from it so the sandbox is findable. + name="vero/stub-benchmark", + ) + observed = {} + + def compile_task(_config, output): + output.mkdir(parents=True) + return output + + def run(command, *, env): + observed["command"] = command + observed["environment"] = env + return SimpleNamespace(returncode=0) + + monkeypatch.setattr( + harbor_build, "load_harbor_build_config", lambda _path, **_kwargs: config + ) + monkeypatch.setattr(harbor_build, "compile_harbor_task", compile_task) + monkeypatch.setattr(harbor_cli.shutil, "which", lambda _name: "/usr/bin/uvx") + monkeypatch.setattr(harbor_cli.subprocess, "run", run) + + result = CliRunner().invoke( + harbor, + [ + "run", + "--config", + str(config_path), + "--agent", + "codex", + "--environment", + "modal", + ], + ) + + assert result.exit_code == 0, result.output + assert observed["command"][:6] == [ + "/usr/bin/uvx", + "--python", + harbor_cli.sys.executable, + "--from", + "harbor[modal]==0.20.0", + "harbor", + ] + + +def test_harbor_run_env_file_secrets_reach_subprocess_not_command_line( + tmp_path, monkeypatch +): + import vero.harbor.build as harbor_build + import vero.harbor.cli as harbor_cli + + config_path = tmp_path / "build.yaml" + config_path.write_text("task_name: unused\n") + env_file = tmp_path / "secrets.env" + env_file.write_text("MODAL_TOKEN_ID=mt-id\nMODAL_TOKEN_SECRET=mt-secret\n") + config = SimpleNamespace( + harbor_requirement="harbor[modal]==0.20.0", + agent_env={}, + # Real configs always carry a name; the outer trial derives its + # Modal app name from it so the sandbox is findable. + name="vero/stub-benchmark", + ) + observed = {} + + def compile_task(_config, output): + # The build's declared-credential check reads os.environ at compile time, + # so the env-file must already be applied here (not just at subprocess launch). + observed["modal_at_compile"] = os.environ.get("MODAL_TOKEN_ID") + output.mkdir(parents=True) + return output + + def run(command, *, env): + observed["command"] = command + observed["environment"] = env + return SimpleNamespace(returncode=0) + + monkeypatch.setattr( + harbor_build, "load_harbor_build_config", lambda _path, **_kwargs: config + ) + monkeypatch.setattr(harbor_build, "compile_harbor_task", compile_task) + monkeypatch.setattr(harbor_cli.shutil, "which", lambda _name: "/usr/bin/uvx") + monkeypatch.setattr(harbor_cli.subprocess, "run", run) + + result = CliRunner().invoke( + harbor, + [ + "run", + "--config", + str(config_path), + "--agent", + "codex", + "--environment", + "docker", + "--env-file", + str(env_file), + ], + ) + + assert result.exit_code == 0, result.output + assert observed["modal_at_compile"] == "mt-id" + assert observed["environment"]["MODAL_TOKEN_ID"] == "mt-id" + assert observed["environment"]["MODAL_TOKEN_SECRET"] == "mt-secret" + # Secrets are passed via the environment only, never on argv. + assert "mt-secret" not in " ".join(observed["command"]) + assert "Loaded 2 secret(s)" in result.output + assert "mt-secret" not in result.output + + +def test_http_app_separates_agent_and_admin_surfaces(tmp_path, monkeypatch): + sidecar = FakeSidecar() + sidecar.engine.evaluator = SimpleNamespace(session_dir=tmp_path / "session") + verifier = FakeVerifier() + + def create_archive(_session_dir, destination): + destination.write_bytes(b"portable-session") + return destination + + monkeypatch.setattr("vero.sidecar.app.create_harbor_session_archive", create_archive) + client = TestClient( + create_app( + sidecar=sidecar, + verifier=verifier, + admin_token="admin-secret", + ) + ) + + assert client.get("/health").json() == {"ok": True} + response = client.post( + "/eval", + json={ + "backend_id": "backend", + "evaluation_set": {"name": "public"}, + }, + ) + assert response.status_code == 200 + assert response.json()["disclosure"] == "none" + assert sidecar.requests[0].evaluation_set.name == "public" + assert client.get("/status").json()["submit_enabled"] is True + assert client.post("/finalize").status_code == 403 + assert client.get("/session/export").status_code == 403 + exported = client.get( + "/session/export", + headers={"Authorization": "Bearer admin-secret"}, + ) + assert exported.content == b"portable-session" + finalized = client.post( + "/finalize", + headers={"Authorization": "Bearer admin-secret"}, + ) + assert finalized.json()["rewards"] == {"reward": 0.75} + assert verifier.calls == 1 + assert client.get("/evaluations").status_code == 403 + assert client.get( + "/evaluations", + headers={"Authorization": "Bearer admin-secret"}, + ).json() == {"evaluations": []} + + +def test_http_app_exposes_agent_evaluation_jobs(): + sidecar = FakeSidecar() + client = TestClient( + create_app( + sidecar=sidecar, + verifier=FakeVerifier(), + admin_token="admin-secret", + ) + ) + + started = client.post( + "/eval/jobs", + json={ + "backend_id": "backend", + "evaluation_set": {"name": "public"}, + }, + ) + + assert started.status_code == 202 + assert started.json()["job_id"] == "job-1" + assert client.get("/eval/jobs/job-1").json()["status"] == "complete" + result = client.get("/eval/jobs/job-1/result") + assert result.status_code == 200 + assert result.json()["receipt"]["evaluation_id"] == "evaluation" + assert client.get("/eval/jobs/missing").status_code == 404 + + +def test_http_app_redacts_access_denial_details(): + sidecar = FakeSidecar() + sidecar.raise_access_error = True + client = TestClient( + create_app( + sidecar=sidecar, + verifier=FakeVerifier(), + admin_token="admin-secret", + ) + ) + + response = client.post( + "/eval", + json={ + "backend_id": "backend", + "evaluation_set": {"name": "hidden"}, + }, + ) + + assert response.status_code == 403 + assert response.json() == {"error": "evaluation denied"} + + +def test_http_app_maps_backend_request_rejection_to_400(): + sidecar = FakeSidecar() + sidecar.raise_request_error = True + client = TestClient( + create_app( + sidecar=sidecar, + verifier=FakeVerifier(), + admin_token="admin-secret", + ) + ) + + response = client.post( + "/eval", + json={ + "backend_id": "backend", + "evaluation_set": {"name": "hidden"}, + }, + ) + + # The reason is passed through: an agent told only "invalid" cannot repair + # its request, and these messages state a backend capability, not policy. + # Contrast test_http_app_redacts_access_denial_details, which stays opaque. + assert response.status_code == 400 + assert response.json() == {"error": "invalid evaluation request: unknown case"} + + +def test_harbor_cli_builds_canonical_selection(monkeypatch): + captured = {} + + def fake_request(method, path, *, payload=None, headers=None): + captured.update(method=method, path=path, payload=payload, headers=headers) + return {"ok": True} + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + result = CliRunner().invoke( + main, + [ + "harbor", + "eval", + "--backend", + "backend", + "--evaluation-set", + "benchmark", + "--partition", + "validation", + "--case-id", + "a", + "--case-id", + "b", + "--parameter", + "temperature=0.2", + ], + ) + + assert result.exit_code == 0, result.output + assert captured["method"] == "POST" + assert captured["path"] == "/eval" + assert captured["payload"]["evaluation_set"]["selection"] == { + "kind": "ids", + "ids": ["a", "b"], + } + assert captured["payload"]["parameters"] == {"temperature": 0.2} + assert captured["payload"]["limits"] is None + + +def test_harbor_cli_supports_detached_evaluation_jobs(monkeypatch): + requests = [] + + def fake_request(method, path, *, payload=None, headers=None): + requests.append((method, path, payload)) + return {"job_id": "job-1", "status": "running"} + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + runner = CliRunner() + started = runner.invoke( + main, + [ + "harbor", + "eval", + "--backend", + "backend", + "--evaluation-set", + "benchmark", + "--detach", + ], + ) + status = runner.invoke(main, ["harbor", "eval-status", "job-1"]) + result = runner.invoke(main, ["harbor", "eval-result", "job-1"]) + + assert started.exit_code == 0, started.output + assert status.exit_code == 0, status.output + assert result.exit_code == 0, result.output + assert requests[0][:2] == ("POST", "/eval/jobs") + assert requests[1][:2] == ("GET", "/eval/jobs/job-1") + assert requests[2][:2] == ("GET", "/eval/jobs/job-1/result") + + +def test_harbor_finalize_cli_writes_only_rewards(tmp_path, monkeypatch): + token_file = write_admin_token(tmp_path / "token", "admin-secret") + output = tmp_path / "logs/reward.json" + + def fake_request(method, path, *, payload=None, headers=None): + assert headers == {"Authorization": "Bearer admin-secret"} + return { + "rewards": {"accuracy": 0.9}, + "baseline_rewards": {"accuracy": 0.7}, + } + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + result = CliRunner().invoke( + main, + [ + "harbor", + "finalize", + "--token-file", + str(token_file), + "--output", + str(output), + ], + ) + + assert result.exit_code == 0, result.output + assert json.loads(output.read_text()) == {"accuracy": 0.9} + + +def test_harbor_export_session_persists_archive_report_and_checksum( + tmp_path, monkeypatch +): + import vero.harbor.cli as harbor_cli + import vero.report as report_module + + token_file = write_admin_token(tmp_path / "token", "admin-secret") + output = tmp_path / "logs/session.tar.gz" + report = tmp_path / "logs/experiment.html" + status_output = tmp_path / "logs/status.json" + finalization_output = tmp_path / "logs/finalization.json" + trace = tmp_path / "trajectory.json" + trace.write_text("[]\n") + + def fake_request(method, path, *, payload=None, headers=None): + if path == "/finalize": + assert method == "POST" + assert headers == {"Authorization": "Bearer admin-secret"} + return {"candidate": None, "rewards": {"reward": 0.0}, "errors": {}} + assert (method, path) == ("GET", "/status") + return {"submit_enabled": False, "evaluation_access": []} + + def fake_download(path, destination, *, headers=None): + assert path == "/session/export" + assert headers == {"Authorization": "Bearer admin-secret"} + destination.write_bytes(b"sidecar archive") + + def fake_extract(_archive, destination): + session = destination / "session" + session.mkdir(parents=True) + (session / "harbor-session.json").write_text("{}\n") + return session + + async def fake_report(_session, destination): + destination.write_text("experiment\n") + return destination + + monkeypatch.setattr(harbor_cli, "_request", fake_request) + monkeypatch.setattr(harbor_cli, "_download", fake_download) + monkeypatch.setattr(harbor_cli, "extract_harbor_session_archive", fake_extract) + monkeypatch.setattr(report_module, "generate_experiment_report", fake_report) + + result = CliRunner().invoke( + main, + [ + "harbor", + "export-session", + "--token-file", + str(token_file), + "--output", + str(output), + "--report-output", + str(report), + "--status-output", + str(status_output), + "--finalization-output", + str(finalization_output), + "--agent-trace", + str(trace), + ], + ) + + assert result.exit_code == 0, result.output + assert output.is_file() + assert report.read_text() == "experiment\n" + assert json.loads(status_output.read_text())["submit_enabled"] is False + assert json.loads(finalization_output.read_text())["rewards"] == {"reward": 0.0} + checksum = output.with_name(f"{output.name}.sha256").read_text() + assert output.name in checksum diff --git a/vero/tests/test_v05_harbor_inference.py b/vero/tests/test_v05_harbor_inference.py new file mode 100644 index 00000000..f53d684c --- /dev/null +++ b/vero/tests/test_v05_harbor_inference.py @@ -0,0 +1,708 @@ +from __future__ import annotations + +import asyncio +import json + +import httpx +from fastapi.testclient import TestClient + +from vero.gateway.inference import ( + InferenceGatewayConfig, + InferenceScopeConfig, + create_inference_gateway_app, + token_digest, +) + + +def _config(tmp_path, *, max_requests=2, max_tokens=100): + return InferenceGatewayConfig( + state_path=str(tmp_path / "usage.json"), + scopes={ + "producer": InferenceScopeConfig( + token_sha256=token_digest("scoped-token"), + allowed_models=["gpt-test"], + max_requests=max_requests, + max_tokens=max_tokens, + max_concurrency=1, + ) + }, + ) + + +def test_gateway_replaces_credentials_enforces_scope_and_persists_usage(tmp_path): + observed = [] + + def upstream(request: httpx.Request): + observed.append(request) + return httpx.Response( + 200, + json={ + "id": "response", + "usage": { + "input_tokens": 11, + "output_tokens": 7, + "total_tokens": 18, + }, + }, + ) + + app = create_inference_gateway_app( + config=_config(tmp_path, max_requests=1), + upstream_api_key="upstream-secret", + upstream_base_url="https://provider.example/v1", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + denied = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer wrong"}, + json={"model": "gpt-test", "input": "hello"}, + ) + wrong_model = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "other", "input": "hello"}, + ) + accepted = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hello"}, + ) + exhausted = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "again"}, + ) + usage = client.get( + "/usage/producer", + headers={"Authorization": "Bearer scoped-token"}, + ) + + assert denied.status_code == 403 + assert wrong_model.status_code == 403 + assert accepted.status_code == 200 + # 402, not 429: budget exhaustion is terminal, and 429 would be retried + # by EvaluationLimits.retry_status_codes and by target-agent SDKs. + assert exhausted.status_code == 402 + assert len(observed) == 1 + assert observed[0].headers["authorization"] == "Bearer upstream-secret" + assert b"scoped-token" not in observed[0].content + assert usage.json()["requests"] == 1 + assert usage.json()["total_tokens"] == 18 + assert usage.json()["remaining_requests"] == 0 + persisted = json.loads((tmp_path / "usage.json").read_text()) + assert persisted["scopes"]["producer"]["active_requests"] == 0 + assert persisted["scopes"]["producer"]["attributions"]["optimizer"] == { + "requests": 1, + "upstream_errors": 0, + "input_tokens": 11, + "cached_input_tokens": 0, + "output_tokens": 7, + "total_tokens": 18, + } + + +def test_gateway_records_cached_input_tokens(tmp_path): + def upstream(request: httpx.Request): + if b'"stream": true' in request.content or b'"stream":true' in request.content: + payload = ( + 'event: response.completed\ndata: {"type":"response.completed",' + '"response":{"usage":{"input_tokens":40,"output_tokens":4,' + '"total_tokens":44,"input_tokens_details":{"cached_tokens":32}}}}\n\n' + ) + return httpx.Response( + 200, content=payload, headers={"content-type": "text/event-stream"} + ) + return httpx.Response( + 200, + json={ + "usage": { + "prompt_tokens": 20, + "completion_tokens": 5, + "total_tokens": 25, + "prompt_tokens_details": {"cached_tokens": 16}, + } + }, + ) + + app = create_inference_gateway_app( + config=_config(tmp_path, max_requests=None, max_tokens=None), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + client.post( + "/scopes/producer/optimizer/v1/chat/completions", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "messages": []}, + ) + client.post( + "/scopes/producer/eval-1/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hi", "stream": True}, + ) + + persisted = json.loads((tmp_path / "usage.json").read_text()) + scope = persisted["scopes"]["producer"] + # both the chat (prompt_tokens_details) and responses (input_tokens_details) + # shapes are recognized, per-attribution and in the scope total + assert scope["cached_input_tokens"] == 48 + assert scope["attributions"]["optimizer"]["cached_input_tokens"] == 16 + assert scope["attributions"]["eval-1"]["cached_input_tokens"] == 32 + + +def test_gateway_records_usage_without_enforcing_omitted_limits(tmp_path): + def upstream(_request: httpx.Request): + return httpx.Response( + 200, + json={ + "usage": { + "input_tokens": 11, + "output_tokens": 7, + "total_tokens": 18, + } + }, + ) + + app = create_inference_gateway_app( + config=_config(tmp_path, max_requests=None, max_tokens=None), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + responses = [ + client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": f"request {index}"}, + ) + for index in range(3) + ] + usage = client.get( + "/usage/producer", + headers={"Authorization": "Bearer scoped-token"}, + ).json() + + assert [response.status_code for response in responses] == [200, 200, 200] + assert usage["requests"] == 3 + assert usage["total_tokens"] == 54 + assert usage["max_requests"] is None + assert usage["max_tokens"] is None + assert usage["remaining_requests"] is None + assert usage["remaining_tokens"] is None + + +def test_gateway_meters_streaming_responses(tmp_path): + payload = ( + 'event: response.created\ndata: {"type":"response.created"}\n\n' + 'event: response.completed\ndata: {"type":"response.completed",' + '"response":{"usage":{"input_tokens":5,"output_tokens":3,' + '"total_tokens":8}}}\n\n' + ) + + def upstream(_request: httpx.Request): + return httpx.Response( + 200, + content=payload, + headers={"content-type": "text/event-stream"}, + ) + + app = create_inference_gateway_app( + config=_config(tmp_path), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + response = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hello", "stream": True}, + ) + + assert response.status_code == 200 + assert "response.completed" in response.text + persisted = json.loads((tmp_path / "usage.json").read_text()) + usage = persisted["scopes"]["producer"] + assert usage["active_requests"] == 0 + assert usage["total_tokens"] == 8 + + +def test_gateway_reloads_usage_and_enforces_budget(tmp_path): + config = _config(tmp_path, max_requests=1) + (tmp_path / "usage.json").write_text( + json.dumps( + { + "schema_version": 1, + "scopes": { + "producer": { + "requests": 1, + "upstream_errors": 0, + "input_tokens": 1, + "output_tokens": 1, + "total_tokens": 2, + "active_requests": 4, + "attributions": {}, + } + }, + } + ) + ) + app = create_inference_gateway_app( + config=config, + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(lambda _request: httpx.Response(200, json={})), + ) + with TestClient(app) as client: + exhausted = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hello"}, + ) + + assert exhausted.status_code == 402 + assert app.state.usage_store.ledger.scopes["producer"].active_requests == 0 + + +def test_gateway_forwards_arbitrary_endpoints_to_upstream(tmp_path): + # Provider-agnostic passthrough: an endpoint the gateway has never heard of + # (here the Anthropic Messages surface) is forwarded to the upstream proxy, + # with the same scope-token + model + budget enforcement as any other call. + observed = [] + + def upstream(request: httpx.Request): + observed.append(request) + return httpx.Response( + 200, json={"usage": {"input_tokens": 3, "output_tokens": 2}} + ) + + app = create_inference_gateway_app( + config=_config(tmp_path), + upstream_api_key="upstream-secret", + upstream_base_url="https://provider.example/v1", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + forwarded = client.post( + "/scopes/producer/optimizer/v1/messages", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "messages": []}, + ) + wrong_model = client.post( + "/scopes/producer/optimizer/v1/messages", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "other", "messages": []}, + ) + + assert forwarded.status_code == 200 + assert wrong_model.status_code == 403 # model allow-list still applies + assert str(observed[0].url) == "https://provider.example/v1/messages" + assert observed[0].headers["authorization"] == "Bearer upstream-secret" + + +def test_gateway_accepts_scope_token_via_x_api_key_header(tmp_path): + # Anthropic/Claude clients send the scope token as `x-api-key`, not + # `Authorization: Bearer`; the gateway must accept either. + observed = [] + + def upstream(request: httpx.Request): + observed.append(request) + return httpx.Response(200, json={"usage": {"input_tokens": 1}}) + + app = create_inference_gateway_app( + config=_config(tmp_path), + upstream_api_key="upstream-secret", + upstream_base_url="https://provider.example/v1", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + via_x_api_key = client.post( + "/scopes/producer/optimizer/v1/messages", + headers={"x-api-key": "scoped-token"}, + json={"model": "gpt-test", "messages": []}, + ) + no_token = client.post( + "/scopes/producer/optimizer/v1/messages", + headers={"x-api-key": "wrong"}, + json={"model": "gpt-test", "messages": []}, + ) + + assert via_x_api_key.status_code == 200 + assert no_token.status_code == 403 + # the upstream never sees the scope token; it gets the upstream credential + assert observed[0].headers["authorization"] == "Bearer upstream-secret" + + +def test_gateway_rejects_path_traversal_endpoints(tmp_path): + app = create_inference_gateway_app( + config=_config(tmp_path), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(lambda _request: httpx.Response(200, json={})), + ) + with TestClient(app) as client: + # Percent-encoded so the client doesn't normalize `..` away before it + # reaches the route; the server-side guard must still reject it. + traversal = client.post( + "/scopes/producer/optimizer/v1/%2e%2e/admin", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test"}, + ) + + assert traversal.status_code == 400 + assert traversal.json()["error"]["code"] == "invalid_endpoint" + + +def test_gateway_releases_concurrency_reservation_on_upstream_failure(tmp_path): + def unavailable(request: httpx.Request): + raise httpx.ConnectError("offline", request=request) + + app = create_inference_gateway_app( + config=_config(tmp_path), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(unavailable), + ) + with TestClient(app) as client: + response = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hello"}, + ) + + assert response.status_code == 502 + usage = app.state.usage_store.ledger.scopes["producer"] + assert usage.active_requests == 0 + assert usage.upstream_errors == 1 + + +def _logged_config(tmp_path, **kwargs): + from vero.gateway.inference import InferenceRequestLogConfig + + config = _config(tmp_path, **kwargs) + return config.model_copy( + update={ + "request_log": InferenceRequestLogConfig( + directory=str(tmp_path / "requests"), + body_bytes=64, + ) + } + ) + + +def _log_records(tmp_path): + records = [] + for path in sorted((tmp_path / "requests").glob("requests-*.jsonl")): + for line in path.read_text().splitlines(): + records.append(json.loads(line)) + return records + + +def test_gateway_request_log_captures_responses_streams_and_denials(tmp_path): + stream_payload = ( + 'event: response.completed\ndata: {"type":"response.completed",' + '"response":{"usage":{"input_tokens":5,"output_tokens":3,' + '"total_tokens":8}}}\n\n' + ) + + def upstream(request: httpx.Request): + if b'"stream": true' in request.content or b'"stream":true' in request.content: + return httpx.Response( + 200, + content=stream_payload, + headers={"content-type": "text/event-stream"}, + ) + return httpx.Response( + 200, + json={"id": "r", "usage": {"input_tokens": 11, "output_tokens": 7}}, + ) + + app = create_inference_gateway_app( + config=_logged_config(tmp_path, max_requests=2), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hi"}, + ) + client.post( + "/scopes/producer/eval-1/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hi", "stream": True}, + ) + client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "denied-model", "input": "hi"}, + ) + client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "over budget"}, + ) + + records = _log_records(tmp_path) + assert [record["status"] for record in records] == [200, 200, 403, 402] + plain, stream, denied, exhausted = records + assert plain["scope"] == "producer" + assert plain["attribution"] == "optimizer" + assert plain["model"] == "gpt-test" + assert plain["input_tokens"] == 11 and plain["output_tokens"] == 7 + assert "gpt-test" in plain["request"]["text"] + assert '"id":"r"' in plain["response"]["text"] + assert plain["latency_ms"] >= 0 + assert stream["stream"] is True + assert stream["attribution"] == "eval-1" + assert stream["total_tokens"] == 8 + # the head+tail capture keeps the stream's terminal usage frame + assert "total_tokens" in ( + stream["response"]["text"] + stream["response"].get("tail", "") + ) + assert denied["error"] == "model_denied" and denied["response"] is None + assert exhausted["error"] == "budget_exhausted" + + +def test_gateway_request_log_truncates_and_rotates(tmp_path): + from vero.gateway.inference import InferenceRequestLogConfig + + config = _config(tmp_path, max_requests=None, max_tokens=None).model_copy( + update={ + "request_log": InferenceRequestLogConfig( + directory=str(tmp_path / "requests"), + body_bytes=32, + rotate_bytes=1_048_576, + ) + } + ) + app = create_inference_gateway_app( + config=config, + upstream_api_key="upstream-secret", + transport=httpx.MockTransport( + lambda _request: httpx.Response(200, json={"data": "y" * 200}) + ), + ) + with TestClient(app) as client: + client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "x" * 500}, + ) + + (record,) = _log_records(tmp_path) + assert record["request"]["truncated"] is True + assert record["request"]["bytes"] > 500 + assert len(record["request"]["text"]) <= 16 + assert "tail" in record["request"] + assert record["response"]["truncated"] is True + + +def test_gateway_request_log_rotation_boundary(tmp_path): + from vero.gateway.inference import InferenceRequestLog, InferenceRequestLogConfig + + log = InferenceRequestLog( + InferenceRequestLogConfig( + directory=str(tmp_path / "requests"), + body_bytes=16384, + rotate_bytes=1_048_576, + ) + ) + # Force a tiny rotation threshold without violating the config floor. + log.config = log.config.model_copy(update={"rotate_bytes": 400}) + + async def fill(): + for index in range(6): + await log.record(scope="producer", index=index, payload="z" * 100) + + asyncio.run(fill()) + files = sorted((tmp_path / "requests").glob("requests-*.jsonl")) + assert len(files) > 1 + total = sum( + len(path.read_text().splitlines()) for path in files + ) + assert total == 6 + + +def _attributed_config(tmp_path, **kwargs): + from vero.gateway.inference import InferenceRequestLogConfig + + return _config(tmp_path, **kwargs).model_copy( + update={ + "request_log": InferenceRequestLogConfig( + directory=str(tmp_path / "requests"), + body_bytes=1024, + attribution=True, + ) + } + ) + + +def test_gateway_attribution_threads_stateful_and_stateless_requests(tmp_path): + calls = {"n": 0} + + def upstream(request: httpx.Request): + calls["n"] += 1 + if b'"stream": true' in request.content or b'"stream":true' in request.content: + payload = ( + f'event: response.created\ndata: {{"type":"response.created",' + f'"response":{{"id":"resp_stream{calls["n"]}"}}}}\n\n' + 'event: response.completed\ndata: {"type":"response.completed",' + '"response":{"usage":{"input_tokens":1,"output_tokens":1,' + '"total_tokens":2}}}\n\n' + ) + return httpx.Response( + 200, content=payload, headers={"content-type": "text/event-stream"} + ) + return httpx.Response( + 200, + json={"id": f"resp_{calls['n']}", "usage": {"input_tokens": 1}}, + ) + + app = create_inference_gateway_app( + config=_attributed_config(tmp_path, max_requests=None, max_tokens=None), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport(upstream), + ) + with TestClient(app) as client: + headers = {"Authorization": "Bearer scoped-token"} + # responses API: root turn, then a stateful follow-up with no prompt + client.post( + "/scopes/producer/optimizer/v1/responses", + headers=headers, + json={"model": "gpt-test", "input": "Solve task 42: find the answer"}, + ) + client.post( + "/scopes/producer/optimizer/v1/responses", + headers=headers, + json={"model": "gpt-test", "previous_response_id": "resp_1", "input": []}, + ) + # a streamed root and a follow-up chained onto its SSE-delivered id + client.post( + "/scopes/producer/optimizer/v1/responses", + headers=headers, + json={"model": "gpt-test", "input": "Another task entirely", "stream": True}, + ) + client.post( + "/scopes/producer/optimizer/v1/responses", + headers=headers, + json={ + "model": "gpt-test", + "previous_response_id": "resp_stream3", + "input": [], + }, + ) + # anthropic-messages shape: giant system prompt, threading keys off the + # first user message; the second call resends history (stateless) + system = "x" * 30000 + client.post( + "/scopes/producer/optimizer/v1/messages", + headers=headers, + json={ + "model": "gpt-test", + "system": system, + "messages": [ + {"role": "user", "content": "Solve task 42: find the answer"}, + ], + }, + ) + client.post( + "/scopes/producer/optimizer/v1/messages", + headers=headers, + json={ + "model": "gpt-test", + "system": system, + "messages": [ + {"role": "user", "content": "Solve task 42: find the answer"}, + {"role": "assistant", "content": [{"type": "text", "text": "hm"}]}, + {"role": "user", "content": "continue"}, + ], + }, + ) + + records = _log_records(tmp_path) + assert len(records) == 6 + threads = [record.get("thread_id") for record in records] + assert all(threads) + # stateful follow-ups inherit their root's thread + assert threads[1] == threads[0] + assert threads[3] == threads[2] + assert threads[2] != threads[0] + # stateless resends group by first-user-message digest — and the messages + # conversation shares its root text with the first responses thread + assert threads[4] == threads[5] == threads[0] + assert records[0]["root_snippet"].startswith("Solve task 42") + assert records[0]["thread_root_digest"] == records[4]["thread_root_digest"] + # chained records carry the thread but no root fields + assert "root_snippet" not in records[1] + + +def test_gateway_attribution_never_breaks_proxying(tmp_path): + app = create_inference_gateway_app( + config=_attributed_config(tmp_path, max_requests=None, max_tokens=None), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport( + lambda _request: httpx.Response(200, json={"id": 7, "usage": {}}) + ), + ) + hostile_bodies = [ + {"model": "gpt-test", "input": {"nested": {"weird": True}}}, + {"model": "gpt-test", "messages": [{"role": "user", "content": [1, None]}]}, + {"model": "gpt-test", "messages": "not-a-list", "previous_response_id": 9}, + {"model": "gpt-test", "input": [{"role": "user", "content": {"a": "b"}}]}, + ] + with TestClient(app) as client: + for body in hostile_bodies: + response = client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json=body, + ) + assert response.status_code == 200 + assert len(_log_records(tmp_path)) == len(hostile_bodies) + + +def test_gateway_attribution_disabled_by_default_and_memory_bounded(tmp_path): + from vero.gateway.inference import _RequestAttributor + + # default config: no attributor, no thread fields in records + app = create_inference_gateway_app( + config=_logged_config(tmp_path, max_requests=None, max_tokens=None), + upstream_api_key="upstream-secret", + transport=httpx.MockTransport( + lambda _request: httpx.Response(200, json={"usage": {}}) + ), + ) + assert app.state.request_attributor is None + with TestClient(app) as client: + client.post( + "/scopes/producer/optimizer/v1/responses", + headers={"Authorization": "Bearer scoped-token"}, + json={"model": "gpt-test", "input": "hello"}, + ) + (record,) = _log_records(tmp_path) + assert "thread_id" not in record + + # FIFO caps hold under churn + attributor = _RequestAttributor() + attributor._CAP = 10 + for index in range(50): + fields = attributor.stamp_request({"input": f"root {index}"}) + attributor.register_response(fields, f'{{"id":"resp_{index}"}}'.encode()) + assert len(attributor._root_threads) <= 10 + assert len(attributor._response_threads) <= 10 + assert attributor.errors == 0 + + +def test_budget_exhaustion_status_is_not_retryable(): + """Budget exhaustion must not wear a status anything will retry. + + It is terminal -- waiting never restores the quota -- so returning 429 made + every layer above retry it: EvaluationLimits.retry_status_codes defaults to + [429, 503, 529], and target-agent SDKs retry 429 on their own. officeqa run + #2 reissued 3672 doomed requests after exhausting its finalization scope. + """ + from vero.evaluation.models import EvaluationLimits + + budget_exhausted_status = 402 + assert budget_exhausted_status not in EvaluationLimits().retry.retry_status_codes + # The transient ones stay retryable. + assert 429 in EvaluationLimits().retry.retry_status_codes diff --git a/vero/tests/test_v05_harbor_isolation_container.py b/vero/tests/test_v05_harbor_isolation_container.py new file mode 100644 index 00000000..7053ddc5 --- /dev/null +++ b/vero/tests/test_v05_harbor_isolation_container.py @@ -0,0 +1,158 @@ +"""Container integration test for harness-isolation filesystem invariants. + +Runs inside a throwaway Linux container (the CI/dev host may be macOS, where the +uid-drop is unavailable) and asserts, against the *real* privilege drop +(``vero.sandbox.LocalSandbox.run(run_as=...)``) and the *real* provisioning +commands (``vero.sidecar.isolation``), that: + +* a candidate checkout under a ``mktemp -d`` (0700 root) parent is UNreachable to + the dropped user when only the leaf is chowned (the regression sentinel — this + is the bug that shipped), +* it becomes reachable after ``harness_grant_commands`` (and the reachability + probe agrees), and +* trusted root-only state stays unreadable to the dropped user. + +The container loads ``sandbox.py``/``isolation.py`` standalone (they are +stdlib-only) so no vero install or heavy dependency is needed. +""" + +from __future__ import annotations + +import shutil +import subprocess +from pathlib import Path + +import pytest + +_IMAGE = "ghcr.io/astral-sh/uv:python3.12-bookworm" +_VERO_SRC = Path(__file__).resolve().parents[1] / "src" + + +def _docker_usable() -> bool: + if shutil.which("docker") is None: + return False + try: + return ( + subprocess.run( + ["docker", "info"], + capture_output=True, + timeout=30, + check=False, + ).returncode + == 0 + ) + except (OSError, subprocess.SubprocessError): + return False + + +pytestmark = pytest.mark.skipif( + not _docker_usable(), reason="requires a usable docker daemon" +) + + +# Executed as root inside the container. +_CONTAINER_SCRIPT = r''' +import asyncio, importlib.util, os, subprocess, sys + + +def _load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module # dataclass/typing introspection reads sys.modules + spec.loader.exec_module(module) + return module + + +# Real modules, loaded standalone (stdlib-only) — no vero install needed. +sbx = _load("_vero_sandbox", "/vero-src/vero/sandbox.py") +iso = _load("_vero_isolation", "/vero-src/vero/sidecar/isolation.py") + +subprocess.run(["useradd", "-m", "-u", "1002", "harness"], check=True) +sandbox = sbx.LocalSandbox("/") # run_as drops privileges from this root process +MARKER = os.path.join("src", "tinyagent", "__init__.py") + + +def make_checkout(): + # Mirror candidate_repository.git.checkout: /repository. mktemp -d + # leaves the parent 0700 root — the traversal the harness needs. + parent = subprocess.run( + ["mktemp", "-d", "/tmp/vero-candidate-checkoutXXXXXX"], + capture_output=True, text=True, check=True, + ).stdout.strip() + repo = os.path.join(parent, "repository") + os.makedirs(os.path.join(repo, "src", "tinyagent")) + open(os.path.join(repo, MARKER), "w").close() + return parent, repo + + +async def as_harness(command): + return await sandbox.run(command, run_as="harness") + + +async def main(): + failures = [] + + # Trusted seal: a root-only 0700 dir + 0600 file (stands in for the session + # dir / serve.json / admin token). + os.makedirs("/trusted", exist_ok=True) + with open("/trusted/secret", "w") as handle: + handle.write("held-out") + os.chmod("/trusted/secret", 0o600) + os.chmod("/trusted", 0o700) + + # 1) Regression sentinel: chown only the leaf (the shipped bug). The dropped + # user must NOT be able to reach a file under the still-0700 parent. + _parent, repo = make_checkout() + subprocess.run(["chown", "-R", "harness:harness", repo], check=True) + if (await as_harness(["test", "-r", os.path.join(repo, MARKER)])).returncode == 0: + failures.append( + "regression sentinel: workspace reachable WITHOUT the traversal grant" + ) + + # 2) Positive: the real provisioning makes it reachable, and the probe agrees. + _parent, repo = make_checkout() + for command in iso.harness_grant_commands( + "harness", chown_paths=[repo], checkout_root=repo + ): + subprocess.run(command, check=True) + if (await as_harness(iso.harness_reachability_probe(repo))).returncode != 0: + failures.append("positive: reachability probe failed after provisioning") + if (await as_harness(["test", "-r", os.path.join(repo, MARKER)])).returncode != 0: + failures.append("positive: provisioned workspace unreadable to the harness") + + # 3) Negative: trusted state stays sealed from the dropped user. + if (await as_harness(["cat", "/trusted/secret"])).returncode == 0: + failures.append("negative: harness read the trusted /trusted/secret") + + if failures: + print("ISOLATION INVARIANTS VIOLATED:") + for item in failures: + print(" -", item) + sys.exit(1) + print("all isolation invariants hold") + + +asyncio.run(main()) +''' + + +def test_harness_isolation_invariants_on_container(tmp_path): + script = tmp_path / "probe.py" + script.write_text(_CONTAINER_SCRIPT) + result = subprocess.run( + [ + "docker", "run", "--rm", + "-v", f"{_VERO_SRC}:/vero-src:ro", + "-v", f"{script}:/probe.py:ro", + _IMAGE, + "python3", "/probe.py", + ], + capture_output=True, + text=True, + timeout=600, + check=False, + ) + assert result.returncode == 0, ( + f"isolation invariants failed:\n{result.stdout}\n{result.stderr}" + ) + assert "all isolation invariants hold" in result.stdout diff --git a/vero/tests/test_v05_harbor_session.py b/vero/tests/test_v05_harbor_session.py new file mode 100644 index 00000000..e793d4dc --- /dev/null +++ b/vero/tests/test_v05_harbor_session.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +import io +import json +import tarfile +from datetime import UTC, datetime + +import pytest + +from vero.evaluation import ( + BackendProvenance, + EvaluationSet, + MetricSelector, + ObjectiveSpec, +) +from vero.sidecar.session import ( + HarborSessionManifest, + create_harbor_session_archive, + extract_harbor_session_archive, + file_sha256, +) +from vero.sidecar.verifier import VerificationSelection, VerificationTarget + + +def _manifest() -> HarborSessionManifest: + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + evaluation_set = EvaluationSet(name="benchmark", partition="validation") + return HarborSessionManifest( + id="trial", + task_name="org/optimize", + created_at=datetime(2026, 7, 16, tzinfo=UTC), + backends={ + "validation": BackendProvenance( + name="harbor", + version="2", + config_digest="a" * 64, + ) + }, + selection=VerificationSelection(mode="submit"), + targets=[ + VerificationTarget( + reward_key="reward", + backend_id="validation", + evaluation_set=evaluation_set, + objective=objective, + ) + ], + ) + + +def test_harbor_session_archive_round_trip_and_checksum(tmp_path): + session = tmp_path / "source" + session.mkdir() + (session / "harbor-session.json").write_text( + _manifest().model_dump_json(indent=2) + "\n" + ) + (session / "database.json").write_text('{"id":"trial"}\n') + archive = create_harbor_session_archive(session, tmp_path / "session.tar.gz") + + extracted = extract_harbor_session_archive(archive, tmp_path / "extracted") + + assert ( + HarborSessionManifest.model_validate_json( + (extracted / "harbor-session.json").read_text() + ).id + == "trial" + ) + assert len(file_sha256(archive)) == 64 + + +def test_harbor_session_archive_records_symlinks_as_metadata_without_failing(tmp_path): + # Regression: a single symlink anywhere in the tree used to raise and 500 the + # whole export, discarding an entire run's data. It must instead succeed, + # preserve each link's target as inert metadata, and record the omission. + session = tmp_path / "source" + session.mkdir() + (session / "harbor-session.json").write_text( + _manifest().model_dump_json(indent=2) + "\n" + ) + (session / "database.json").write_text('{"id":"trial"}\n') + (session / "link_in").symlink_to("database.json") # relative, in-tree + (session / "link_abs").symlink_to("/etc/hostname") # absolute, out-of-tree + + archive = create_harbor_session_archive(session, tmp_path / "session.tar.gz") + + # No link members survive (keeps the archive extractable + traversal-safe). + with tarfile.open(archive) as tar: + members = tar.getmembers() + assert not any(m.issym() or m.islnk() for m in members) + names = {m.name for m in members} + assert "session/link_in.symlink" in names + assert "session/link_abs.symlink" in names + assert "session/vero-export-skipped.json" in names + + extracted = extract_harbor_session_archive(archive, tmp_path / "extracted") + assert (extracted / "database.json").read_text() == '{"id":"trial"}\n' + assert "database.json" in (extracted / "link_in.symlink").read_text() + assert "/etc/hostname" in (extracted / "link_abs.symlink").read_text() + skipped = json.loads((extracted / "vero-export-skipped.json").read_text()) + reasons = {entry["path"]: entry["reason"] for entry in skipped["skipped"]} + assert reasons["session/link_in"] == "symlink" + assert reasons["session/link_abs"] == "symlink" + + +def test_harbor_session_archive_rejects_traversal(tmp_path): + archive = tmp_path / "unsafe.tar.gz" + with tarfile.open(archive, "w:gz") as output: + member = tarfile.TarInfo("session/../../escape") + payload = b"bad" + member.size = len(payload) + output.addfile(member, io.BytesIO(payload)) + + with pytest.raises(ValueError, match="unsafe Harbor session archive member"): + extract_harbor_session_archive(archive, tmp_path / "extracted") diff --git a/vero/tests/test_v05_harbor_sidecar.py b/vero/tests/test_v05_harbor_sidecar.py new file mode 100644 index 00000000..27792dcd --- /dev/null +++ b/vero/tests/test_v05_harbor_sidecar.py @@ -0,0 +1,897 @@ +from __future__ import annotations + +import asyncio +import json +import shutil +import subprocess +from contextlib import asynccontextmanager +from datetime import UTC, datetime +from pathlib import Path + +import pytest + +from vero.candidate import Candidate +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import ( + BackendProvenance, + BackendRegistry, + BudgetLedger, + CaseIds, + CaseResult, + CaseStatus, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationAuthorization, + EvaluationBudget, + EvaluationCost, + EvaluationDatabase, + EvaluationDeniedError, + EvaluationEngine, + EvaluationLimits, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + Evaluator, + MetricSelector, + ObjectiveSpec, +) +from vero.runtime.context import context_digest +from vero.sandbox import LocalSandbox +from vero.sidecar import ( + EvaluationAccessError, + EvaluationJobStatus, + EvaluationSidecar, + GitCandidateTransport, + SidecarEvaluationPolicy, + SidecarEvaluationRequest, + SubmissionDisabledError, +) +from vero.workspace import GitWorkspace, Workspace + + +class StubWorkspace(Workspace): + def __init__(self, root: Path, version: str): + root.mkdir(parents=True, exist_ok=True) + self._root = root + self._version = version + + @property + def sandbox(self): + return None + + @property + def root(self) -> str: + return str(self._root) + + @property + def project_path(self) -> str: + return str(self._root) + + @property + def name(self) -> str: + return "stub" + + async def current_version(self) -> str: + return self._version + + async def save(self, message="Save") -> str: + return self._version + + async def restore(self, version_id, message=None) -> str: + self._version = version_id + return version_id + + async def diff(self, from_version=None, to_version=None) -> str: + return "" + + async def log(self, max_count=10, since_version=None) -> str: + return "" + + async def is_ancestor(self, version_a, version_b) -> bool: + return True + + async def copy(self, name=None, from_version=None): + return StubWorkspace(self._root, from_version or self._version) + + @asynccontextmanager + async def temp_copy(self, from_version=None): + yield StubWorkspace(self._root, from_version or self._version) + + @asynccontextmanager + async def at(self, version_id): + previous = self._version + self._version = version_id + try: + yield + finally: + self._version = previous + + async def is_dirty(self) -> bool: + return False + + +class StubCandidateRepository: + family = "stub" + + def __init__(self, root: Path): + self.root = root + + @asynccontextmanager + async def checkout(self, candidate, *, sandbox, name=None): + yield StubWorkspace(self.root, candidate.version) + + +class StubBackend: + @property + def provenance(self): + return BackendProvenance( + name="stub", + version="1", + config_digest="0" * 64, + ) + + async def resolve_cost(self, evaluation_set): + selection = evaluation_set.selection + if isinstance(selection, CaseIds): + return EvaluationCost(cases=len(selection.ids)) + return EvaluationCost(cases=8) + + async def evaluate(self, *, context, request): + selection = request.evaluation_set.selection + ids = ( + selection.ids + if isinstance(selection, CaseIds) + else [f"case-{i}" for i in range(8)] + ) + cases = [ + CaseResult( + case_id=case_id, + status=CaseStatus.SUCCESS, + metrics={"score": 0.75}, + ) + for case_id in ids + ] + for case in cases: + await context.case_store.save(case) + return EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 0.75}, + cases=cases, + ) + + +class StubTransport: + def __init__(self, candidate: Candidate): + self.candidate = candidate + self.calls: list[str | None] = [] + + async def import_candidate(self, version=None): + self.calls.append(version) + return self.candidate + + +def _sidecar(tmp_path: Path, *, submit_enabled=False, fixed_limits=False, disclose_budget=True): + candidate = Candidate( + id="candidate", + version="candidate-version", + created_at=datetime(2026, 1, 1, tzinfo=UTC), + ) + workspace = StubWorkspace(tmp_path / "repo", candidate.version) + backend = StubBackend() + evaluation_set = EvaluationSet(name="benchmark", partition="validation") + ledger = BudgetLedger( + [ + EvaluationBudget( + backend_id="primary", + evaluation_set_key=evaluation_set.budget_key("primary"), + total_runs=3, + total_cases=20, + ) + ] + ) + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=StubCandidateRepository(tmp_path / "stub-workspace"), + sandbox=workspace.sandbox, + session_dir=tmp_path / "session", + ), + backends=BackendRegistry({"primary": backend, "secondary": StubBackend()}), + database=EvaluationDatabase(id="session"), + budget_ledger=ledger, + ) + transport = StubTransport(candidate) + sidecar = EvaluationSidecar( + engine=engine, + candidate_transport=transport, + access_policies=[ + SidecarEvaluationPolicy( + backend_id="primary", + evaluation_set_name="benchmark", + partition="validation", + access=EvaluationAccessPolicy( + disclosure=DisclosureLevel.AGGREGATE, + min_aggregate_cases=5, + ), + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + limits=EvaluationLimits() if fixed_limits else None, + ), + SidecarEvaluationPolicy( + backend_id="secondary", + evaluation_set_name="public", + access=EvaluationAccessPolicy(disclosure=DisclosureLevel.FULL), + ), + ], + agent_volume=tmp_path / "agent-volume", + admin_volume=tmp_path / "admin-volume", + submit_enabled=submit_enabled, + disclose_budget=disclose_budget, + ) + return sidecar, transport, ledger + + +@pytest.mark.asyncio +async def test_sidecar_uses_canonical_disclosure_budget_and_multiple_backends(tmp_path): + sidecar, transport, ledger = _sidecar(tmp_path) + evaluation_set = EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=[f"case-{i}" for i in range(5)]), + ) + + response = await sidecar.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=evaluation_set, + version="HEAD", + ) + ) + + assert response.disclosure == DisclosureLevel.AGGREGATE + assert response.receipt.result.metrics == {"score": 0.75} + assert response.receipt.result.total_cases == 5 + assert transport.calls == ["HEAD"] + assert response.receipt.result_path == ( + f".evals/results/{context_digest(response.receipt.evaluation_id)}/" + "evaluation.json" + ) + assert ( + tmp_path + / "agent-volume" + / "results" + / context_digest(response.receipt.evaluation_id) + / "evaluation.json" + ).is_file() + budget = ledger.get("primary", evaluation_set) + assert budget.remaining_runs == 2 + assert budget.remaining_cases == 15 + status = sidecar.status() + assert [item.backend_id for item in status.evaluation_access] == [ + "primary", + "secondary", + ] + assert status.evaluation_access[0].expose_case_resources is False + # The synchronous evaluation is now a tracked job, so it is visible in + # status exactly like a detached one. + assert len(status.evaluation_jobs) == 1 + assert status.evaluation_jobs[0].status == EvaluationJobStatus.COMPLETE + assert status.evaluation_jobs[0].evaluation_id == response.receipt.evaluation_id + + +@pytest.mark.asyncio +async def test_sidecar_detached_job_returns_before_evaluation_and_persists_result( + tmp_path, +): + sidecar, _, _ = _sidecar(tmp_path) + + class BlockingBackend(StubBackend): + def __init__(self): + self.started = asyncio.Event() + self.release = asyncio.Event() + + async def evaluate(self, *, context, request): + self.started.set() + await self.release.wait() + return await super().evaluate(context=context, request=request) + + backend = BlockingBackend() + sidecar.engine.backends._backends["primary"] = backend + job = await sidecar.start_evaluation_job( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=[f"case-{i}" for i in range(5)]), + ), + version="HEAD", + ) + ) + + assert job.status == EvaluationJobStatus.RUNNING + assert job.version == "candidate-version" + assert job.receipt is None + await backend.started.wait() + assert sidecar.status().evaluation_jobs[0].job_id == job.job_id + drain = asyncio.create_task( + sidecar.engine.quiesce_agent_evaluations(timeout_seconds=1.0) + ) + await asyncio.sleep(0) + assert not drain.done() + + backend.release.set() + task = sidecar._evaluation_job_tasks[job.job_id] + await task + assert await drain == 1 + + completed = sidecar.evaluation_job(job.job_id) + assert completed.status == EvaluationJobStatus.COMPLETE + assert completed.receipt is not None + assert completed.receipt.result.metrics == {"score": 0.75} + persisted = tmp_path / "session/evaluation-jobs" / f"{job.job_id}.json" + assert json.loads(persisted.read_text())["status"] == "complete" + + +@pytest.mark.asyncio +async def test_sidecar_detached_job_reports_safe_admission_failure(tmp_path): + sidecar, _, _ = _sidecar(tmp_path) + + job = await sidecar.start_evaluation_job( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet(name="private", partition="validation"), + ) + ) + + assert job.status == EvaluationJobStatus.FAILED + assert job.error == "evaluation denied" + assert job.receipt is None + + +@pytest.mark.asyncio +async def test_sidecar_tracks_candidate_import_as_part_of_an_agent_evaluation( + tmp_path, +): + sidecar, original_transport, _ = _sidecar(tmp_path) + + class BlockingTransport(StubTransport): + def __init__(self, candidate): + super().__init__(candidate) + self.started = asyncio.Event() + self.release = asyncio.Event() + + async def import_candidate(self, version=None): + self.calls.append(version) + self.started.set() + await self.release.wait() + return self.candidate + + transport = BlockingTransport(original_transport.candidate) + sidecar.candidate_transport = transport + evaluation = asyncio.create_task( + sidecar.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + ), + version="HEAD", + ) + ) + ) + await transport.started.wait() + + drain = asyncio.create_task( + sidecar.engine.quiesce_agent_evaluations(timeout_seconds=1.0) + ) + await asyncio.sleep(0) + assert not drain.done() + transport.release.set() + + assert (await evaluation).receipt.status == EvaluationStatus.SUCCESS + assert await drain == 1 + with pytest.raises(EvaluationDeniedError, match="finalization has started"): + await sidecar.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + ), + ) + ) + + +def test_sidecar_status_reports_inference_usage_and_remaining_budget(tmp_path): + sidecar, _, _ = _sidecar(tmp_path) + usage_path = tmp_path / "inference/usage.json" + usage_path.parent.mkdir() + usage_path.write_text( + json.dumps( + { + "schema_version": 1, + "scopes": { + "producer": { + "requests": 3, + "total_tokens": 120, + "active_requests": 0, + } + }, + } + ) + ) + sidecar.inference_usage_path = usage_path + sidecar.inference_limits = { + "producer": { + "allowed_models": ["gpt-test"], + "max_requests": 10, + "max_tokens": 1000, + "max_concurrency": 2, + } + } + + usage = sidecar.status().inference_usage + + assert usage is not None + assert usage["producer"]["requests"] == 3 + assert usage["producer"]["remaining_requests"] == 7 + assert usage["producer"]["remaining_tokens"] == 880 + + +def test_sidecar_hides_budget_when_disclosure_disabled(tmp_path): + # Budget-blind mode: enforcement is unchanged but /status must not reveal + # per-set case budgets or inference remaining, closing the live leak. + disclosed, _, _ = _sidecar(tmp_path) + assert any(a.budget is not None for a in disclosed.status().evaluation_access) + + blind, _, _ = _sidecar(tmp_path, disclose_budget=False) + usage_path = tmp_path / "inference/usage.json" + usage_path.parent.mkdir() + usage_path.write_text(json.dumps({"schema_version": 1, "scopes": {}})) + blind.inference_usage_path = usage_path + blind.inference_limits = {"producer": {"max_requests": 10, "max_tokens": 1000}} + + status = blind.status() + assert all(a.budget is None for a in status.evaluation_access) + assert status.inference_usage is None + + +@pytest.mark.asyncio +async def test_sidecar_context_survives_restart_without_disclosing_admin_runs( + tmp_path: Path, +): + sidecar, transport, _ = _sidecar(tmp_path) + admin = await sidecar.engine.evaluate_record( + backend_id="secondary", + request=EvaluationRequest( + candidate=transport.candidate, + evaluation_set=EvaluationSet(name="public"), + ), + authorization=EvaluationAuthorization( + may_evaluate=True, + meter_budget=False, + disclosure=DisclosureLevel.FULL, + ), + ) + response = await sidecar.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=[f"case-{index}" for index in range(5)]), + ), + ) + ) + + restarted = EvaluationSidecar( + engine=sidecar.engine, + candidate_transport=transport, + access_policies=list(sidecar._policies.values()), + agent_volume=tmp_path / "agent-volume", + admin_volume=tmp_path / "admin-volume", + ) + await restarted.initialize_context() + + index = json.loads((tmp_path / "agent-volume/results/index.json").read_text())[ + "evaluations" + ] + assert [entry["evaluation_id"] for entry in index] == [ + response.receipt.evaluation_id + ] + assert admin.id not in json.dumps(index) + + +class CostReportingBackend(StubBackend): + async def evaluate(self, *, context, request): + report = await super().evaluate(context=context, request=request) + cases = [ + case.model_copy( + update={ + "metrics": { + **case.metrics, + "agent_reported_input_tokens": 111.0, + "wall_seconds": 7.0, + } + } + ) + for case in report.cases + ] + return report.model_copy( + update={ + "cases": cases, + "metrics": { + **report.metrics, + "inference_total_tokens": 12345.0, + "inference_cached_input_tokens": 678.0, + "agent_reported_total_input_tokens": 555.0, + # distribution wrappers of the same budget signal + "mean_case_inference_input_tokens": 99.0, + "median_case_agent_reported_output_tokens": 33.0, + "mean_case_wall_seconds": 42.0, + "median_case_wall_seconds": 40.0, + } + } + ) + + +@pytest.mark.asyncio +async def test_budget_blind_sidecar_redacts_inference_metrics(tmp_path): + # Cost telemetry is budget signal: in budget-blind mode the agent's receipt + # and context must not carry inference_* metrics; latency stays visible and + # the admin record keeps everything. + blind, _, _ = _sidecar(tmp_path, disclose_budget=False) + blind.engine.backends = BackendRegistry( + {"primary": CostReportingBackend(), "secondary": CostReportingBackend()} + ) + response = await blind.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=[f"case-{i}" for i in range(5)]), + ), + ) + ) + + receipt_metrics = response.receipt.result.metrics + assert "inference_total_tokens" not in receipt_metrics + assert "inference_cached_input_tokens" not in receipt_metrics + # per-case distribution wrappers are the same signal renamed, so they are + # redacted too; latency distributions stay visible + assert "mean_case_inference_input_tokens" not in receipt_metrics + assert "median_case_agent_reported_output_tokens" not in receipt_metrics + assert receipt_metrics["mean_case_wall_seconds"] == 42.0 + assert receipt_metrics["median_case_wall_seconds"] == 40.0 + + document = json.loads( + ( + tmp_path + / "agent-volume/results" + / context_digest(response.receipt.evaluation_id) + / "evaluation.json" + ).read_text() + ) + context_metrics = document["result"]["metrics"] + assert "inference_total_tokens" not in context_metrics + assert "agent_reported_total_input_tokens" not in context_metrics + assert context_metrics["mean_case_wall_seconds"] == 42.0 + + # the trusted record is untouched, including per-case cost metrics + record = blind.engine.database.get_evaluation(response.receipt.evaluation_id) + assert record.report.metrics["inference_total_tokens"] == 12345.0 + assert record.report.metrics["agent_reported_total_input_tokens"] == 555.0 + assert record.report.cases[0].metrics["agent_reported_input_tokens"] == 111.0 + + # full disclosure in budget-blind mode: per-case cost metrics are redacted + # from the agent-visible case files too, while wall_seconds stays + full = await blind.evaluate( + SidecarEvaluationRequest( + backend_id="secondary", + evaluation_set=EvaluationSet(name="public"), + ) + ) + full_doc = json.loads( + ( + tmp_path + / "agent-volume/results" + / context_digest(full.receipt.evaluation_id) + / "evaluation.json" + ).read_text() + ) + case_file = full_doc["result"]["case_files"][0]["path"] + case_doc = json.loads( + ( + tmp_path + / "agent-volume/results" + / context_digest(full.receipt.evaluation_id) + / case_file + ).read_text() + ) + assert "agent_reported_input_tokens" not in case_doc["result"]["metrics"] + assert case_doc["result"]["metrics"]["wall_seconds"] == 7.0 + + # with budget disclosure on, the same metrics flow to the agent + disclosed, _, _ = _sidecar(tmp_path / "disclosed", disclose_budget=True) + disclosed.engine.backends = BackendRegistry( + {"primary": CostReportingBackend(), "secondary": CostReportingBackend()} + ) + open_response = await disclosed.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=[f"case-{i}" for i in range(5)]), + ), + ) + ) + assert open_response.receipt.result.metrics["inference_total_tokens"] == 12345.0 + + +@pytest.mark.asyncio +async def test_sidecar_context_writes_plan_and_tasks(tmp_path): + sidecar, _, _ = _sidecar(tmp_path) + await sidecar.initialize_context() + + volume = tmp_path / "agent-volume" + # Case resources live under tasks/ (the documented name), not cases/. + tasks_index = json.loads((volume / "tasks/index.json").read_text()) + assert tasks_index["case_resources"] == [] + assert not (volume / "cases").exists() + + plan = json.loads((volume / "plan.json").read_text()) + by_name = {entry["name"]: entry for entry in plan["evaluations"]} + assert set(by_name) == {"benchmark", "public"} + assert by_name["benchmark"]["disclosure"] == "aggregate" + assert by_name["benchmark"]["budget"]["remaining_runs"] == 3 + assert by_name["benchmark"]["budget"]["remaining_cases"] == 20 + + # Each evaluation refreshes the plan with the depleted budget. + await sidecar.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=[f"case-{i}" for i in range(5)]), + ), + ) + ) + plan = json.loads((volume / "plan.json").read_text()) + by_name = {entry["name"]: entry for entry in plan["evaluations"]} + assert by_name["benchmark"]["budget"]["remaining_runs"] == 2 + assert by_name["benchmark"]["budget"]["remaining_cases"] == 15 + + # Budget-blind mode withholds the budget column, nothing else. + blind, _, _ = _sidecar(tmp_path, disclose_budget=False) + await blind.initialize_context() + plan = json.loads((volume / "plan.json").read_text()) + assert all(entry["budget"] is None for entry in plan["evaluations"]) + + +@pytest.mark.asyncio +async def test_sidecar_fails_closed_before_transfer_or_budget(tmp_path): + sidecar, transport, ledger = _sidecar(tmp_path) + too_small = SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=["single"]), + ), + ) + + with pytest.raises(EvaluationAccessError, match="at least 5"): + await sidecar.evaluate(too_small) + # The synchronous failure is recorded as a tracked job (visible in status) + # even though the error is still re-raised to the caller. + assert sidecar.status().evaluation_jobs[0].status == EvaluationJobStatus.FAILED + assert sidecar.status().evaluation_jobs[0].error == "evaluation denied" + with pytest.raises(EvaluationAccessError, match="not agent-evaluable"): + await sidecar.evaluate( + too_small.model_copy( + update={"evaluation_set": EvaluationSet(name="hidden")} + ) + ) + with pytest.raises(EvaluationAccessError, match="not agent-controllable"): + await sidecar.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=EvaluationSet( + name="benchmark", + partition="validation", + ), + parameters={"harbor_model_override": "untrusted-model"}, + ) + ) + + assert transport.calls == [] + budget = ledger.get("primary", too_small.evaluation_set) + assert budget.remaining_runs == 3 + assert budget.remaining_cases == 20 + + +@pytest.mark.asyncio +async def test_sidecar_rejects_agent_limits_when_policy_fixes_them(tmp_path): + sidecar, transport, ledger = _sidecar(tmp_path, fixed_limits=True) + evaluation_set = EvaluationSet( + name="benchmark", + partition="validation", + selection=CaseIds(ids=[f"case-{i}" for i in range(5)]), + ) + + with pytest.raises(EvaluationAccessError, match="limits are fixed"): + await sidecar.evaluate( + SidecarEvaluationRequest( + backend_id="primary", + evaluation_set=evaluation_set, + limits=EvaluationLimits(timeout_seconds=10), + ) + ) + + assert transport.calls == [] + budget = ledger.get("primary", evaluation_set) + assert budget.remaining_runs == 3 + assert budget.remaining_cases == 20 + + +@pytest.mark.asyncio +async def test_sidecar_submission_is_explicit_and_durable(tmp_path): + sidecar, _, _ = _sidecar(tmp_path) + with pytest.raises(SubmissionDisabledError): + await sidecar.submit() + + enabled, _, _ = _sidecar(tmp_path / "enabled", submit_enabled=True) + submission = await enabled.submit("candidate-ref") + + assert submission.candidate.version == "candidate-version" + assert (tmp_path / "enabled/admin-volume/submission.json").is_file() + + +def _git(path: Path, *arguments: str) -> str: + result = subprocess.run( + ["git", *arguments], + cwd=path, + check=True, + text=True, + capture_output=True, + ) + return result.stdout.strip() + + +def _init_repo(path: Path, content: str) -> str: + path.mkdir(parents=True) + _git(path, "init", "-q") + _git(path, "config", "user.name", "VeRO Test") + _git(path, "config", "user.email", "vero@example.test") + (path / "program.txt").write_text(content, encoding="utf-8") + _git(path, "add", "program.txt") + _git(path, "commit", "-q", "-m", f"program {content}") + return _git(path, "rev-parse", "HEAD") + + +@pytest.mark.asyncio +async def test_git_candidate_transport_fetches_to_stable_ref(tmp_path, monkeypatch): + agent_repo = tmp_path / "agent repo" + trusted_repo = tmp_path / "trusted" + agent_commit = _init_repo(agent_repo, "candidate") + _init_repo(trusted_repo, "baseline") + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(trusted_repo)) + system_config = tmp_path / "trusted-system-gitconfig" + _git( + tmp_path, + "config", + "--file", + str(system_config), + "--add", + "safe.directory", + str(trusted_repo), + ) + _git( + tmp_path, + "config", + "--file", + str(system_config), + "--add", + "safe.directory", + str(agent_repo), + ) + _git( + tmp_path, + "config", + "--file", + str(system_config), + "--add", + "safe.directory", + str(agent_repo / ".git"), + ) + original_run = sandbox.run + + async def run_as_different_owner(command, cwd=None, timeout=30, env=None): + return await original_run( + command, + cwd=cwd, + timeout=timeout, + env={ + **(env or {}), + "GIT_TEST_ASSUME_DIFFERENT_OWNER": "1", + "GIT_CONFIG_SYSTEM": str(system_config), + }, + ) + + monkeypatch.setattr(sandbox, "run", run_as_different_owner) + candidate_repository = await GitCandidateRepository.create( + tmp_path / "candidate-store", + workspace=workspace, + ) + transport = GitCandidateTransport( + workspace=workspace, + candidate_repository=candidate_repository, + agent_repo_path=str(agent_repo), + ) + + candidate = await transport.import_candidate() + repeated = await transport.import_candidate(agent_commit) + + assert candidate == repeated + assert candidate.version == agent_commit + assert candidate.description == "program candidate" + shutil.rmtree(agent_repo) + checkout_sandbox = await LocalSandbox.create(root=tmp_path) + async with candidate_repository.checkout( + candidate, + sandbox=checkout_sandbox, + ) as checkout: + assert await checkout.current_version() == agent_commit + retained_refs = _git( + candidate_repository.repository_path, + "for-each-ref", + "--format=%(refname)", + "refs/vero/candidates", + ).splitlines() + assert len(retained_refs) == 1 + assert ( + _git(candidate_repository.repository_path, "rev-parse", retained_refs[0]) + == agent_commit + ) + assert ( + _git( + candidate_repository.repository_path, + "for-each-ref", + "--format=%(refname)", + "refs/vero/incoming", + ) + == "" + ) + + +def test_access_policy_defaults_to_safe_k_anonymity_floor(): + # The floor now lives on the core policy: omitted resolves to 5 under + # aggregate disclosure and 1 otherwise. + aggregate = EvaluationAccessPolicy(disclosure=DisclosureLevel.AGGREGATE) + assert aggregate.min_aggregate_cases == 5 + assert EvaluationAccessPolicy().min_aggregate_cases == 1 + + +def test_detached_job_error_names_the_reason_a_request_was_rejected(): + # The detached path mirrors the blocking one (sidecar/app.py): a request + # rejection must say *what* the backend refused, or a polling agent sees + # only "invalid evaluation request" and cannot reissue a valid call. + # Denials stay opaque on purpose — they are policy, not capability. + from vero.evaluation import EvaluationRequestError + + describe = EvaluationSidecar._evaluation_job_error + assert describe(EvaluationRequestError("backend does not support the seed")) == ( + "invalid evaluation request: backend does not support the seed" + ) + assert describe(EvaluationRequestError("")) == "invalid evaluation request" + assert describe(EvaluationDeniedError("private policy detail")) == ( + "evaluation denied" + ) diff --git a/vero/tests/test_v05_harbor_verifier.py b/vero/tests/test_v05_harbor_verifier.py new file mode 100644 index 00000000..bb2cbe11 --- /dev/null +++ b/vero/tests/test_v05_harbor_verifier.py @@ -0,0 +1,595 @@ +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + +from vero.candidate import Candidate +from vero.evaluation import ( + BackendProvenance, + BackendRegistry, + CaseResult, + CaseStatus, + EvaluationDatabase, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, +) +from vero.sidecar import ( + CanonicalVerifier, + Submission, + VerificationSelection, + VerificationTarget, +) + +OBJECTIVE = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", +) + + +class StubBackend: + @property + def provenance(self): + return BackendProvenance( + name="stub", + version="1", + config_digest="0" * 64, + ) + + async def resolve_cost(self, evaluation_set): + raise AssertionError("fake engine handles evaluation directly") + + async def evaluate(self, *, context, request): + raise AssertionError("fake engine handles evaluation directly") + + +class FakeEngine: + def __init__(self, scores): + self.backends = BackendRegistry({"backend": StubBackend()}) + self.database = EvaluationDatabase(id="session") + self.scores = scores + self.calls = [] + self.drain_calls = [] + self.on_drain = None + self._sequence = 0 + + async def quiesce_agent_evaluations(self, *, timeout_seconds): + self.drain_calls.append(timeout_seconds) + if self.on_drain is not None: + self.on_drain() + return 0 + + async def evaluate_record( + self, + *, + backend_id, + request, + objective_spec, + authorization, + principal, + ): + self.calls.append((request.candidate.version, request.evaluation_set.name)) + key = (request.candidate.version, request.evaluation_set.name) + value = self.scores[key] + if isinstance(value, list): + value = value.pop(0) + if isinstance(value, Exception): + raise value + self._sequence += 1 + record = _record( + f"admin-{self._sequence}", + request.candidate, + request.evaluation_set, + float(value), + objective_spec, + backend_id=backend_id, + ) + self.database.add_evaluation(record) + assert authorization.meter_budget is False + assert principal.value == "admin" + return record + + +def _candidate(name: str, *, content: str | None = None, seconds: int = 0): + return Candidate( + id=name, + version=name, + created_at=datetime(2026, 1, 1, tzinfo=UTC) + timedelta(seconds=seconds), + metadata={"content_digest": content or name}, + ) + + +def _record( + record_id: str, + candidate: Candidate, + evaluation_set: EvaluationSet, + score: float, + objective: ObjectiveSpec = OBJECTIVE, + *, + backend_id: str = "backend", +): + now = datetime(2026, 2, 1, tzinfo=UTC) + from vero.evaluation import EvaluationRequest + + return EvaluationRecord( + id=record_id, + request=EvaluationRequest( + candidate=candidate, + evaluation_set=evaluation_set, + ), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": score}, + ), + backend_id=backend_id, + backend=StubBackend().provenance, + objective_spec=objective, + objective=ObjectiveResult(value=score, feasible=True), + created_at=now, + completed_at=now, + ) + + +def _verifier( + tmp_path: Path, + engine: FakeEngine, + *, + baseline: Candidate, + top_k: int = 1, + score_baseline: bool = True, + baseline_floor: bool = False, +): + return CanonicalVerifier( + engine=engine, + selection=VerificationSelection( + mode="auto_best", + backend_id="backend", + evaluation_set=EvaluationSet(name="selection"), + objective=OBJECTIVE, + baseline_candidate=baseline, + rescore_top_k=top_k, + rescore_attempts=1, + baseline_floor=baseline_floor, + ), + targets=[ + VerificationTarget( + reward_key="reward", + backend_id="backend", + evaluation_set=EvaluationSet(name="test"), + objective=OBJECTIVE, + max_attempts=1, + ) + ], + admin_volume=tmp_path, + score_baseline=score_baseline, + ) + + +@pytest.mark.asyncio +async def test_verifier_pools_repeats_then_admin_rescores_and_scores_baseline(tmp_path): + baseline = _candidate("baseline") + farmed = _candidate("farmed", content="same-code", seconds=1) + duplicate = _candidate("duplicate", content="same-code", seconds=2) + steady = _candidate("steady", seconds=3) + engine = FakeEngine( + { + ("steady", "selection"): 0.65, + ("steady", "test"): 0.8, + ("baseline", "selection"): 0.6, + ("baseline", "test"): 0.5, + } + ) + for index, (candidate, score) in enumerate( + [(farmed, 0.95), (duplicate, 0.05), (steady, 0.7)] + ): + engine.database.add_evaluation( + _record( + f"record-{index}", candidate, EvaluationSet(name="selection"), score + ) + ) + + result = await _verifier( + tmp_path, engine, baseline=baseline, baseline_floor=True + ).finalize() + + assert result.candidate == steady + assert result.rewards == {"reward": 0.8} + assert result.baseline_rewards == {"reward": 0.5} + # the scoring evaluations' full report metrics ride along in the durable + # result (accuracy plus cost/latency telemetry when the backend emits it) + assert result.reward_metrics == {"reward": {"score": 0.8}} + assert result.baseline_reward_metrics == {"reward": {"score": 0.5}} + assert engine.calls == [ + ("steady", "selection"), + ("baseline", "selection"), + ("steady", "test"), + ("baseline", "test"), + ] + assert engine.drain_calls == [600.0] + + +@pytest.mark.asyncio +async def test_verifier_waits_for_an_inflight_selection_evaluation(tmp_path): + baseline = _candidate("baseline") + candidate = _candidate("candidate", seconds=1) + engine = FakeEngine( + { + ("candidate", "selection"): 0.8, + ("baseline", "selection"): 0.5, + ("candidate", "test"): 0.9, + } + ) + + def complete_inflight(): + engine.database.add_evaluation( + _record( + "agent-selection", + candidate, + EvaluationSet(name="selection"), + 0.75, + ) + ) + + engine.on_drain = complete_inflight + + result = await _verifier( + tmp_path, + engine, + baseline=baseline, + score_baseline=False, + ).finalize() + + assert result.candidate == candidate + assert result.rewards == {"reward": 0.9} + assert engine.drain_calls == [600.0] + + +@pytest.mark.asyncio +async def test_verifier_baseline_floor_prevents_shipping_a_regression(tmp_path): + baseline = _candidate("baseline") + candidate = _candidate("candidate", seconds=1) + engine = FakeEngine( + { + ("candidate", "selection"): 0.55, + ("baseline", "selection"): 0.6, + ("baseline", "test"): 0.5, + } + ) + engine.database.add_evaluation( + _record("selection", candidate, EvaluationSet(name="selection"), 0.9) + ) + + result = await _verifier( + tmp_path, engine, baseline=baseline, baseline_floor=True + ).finalize() + + assert result.candidate == baseline + assert result.rewards == {"reward": 0.5} + assert result.baseline_rewards == result.rewards + assert engine.calls.count(("baseline", "test")) == 1 + + +@pytest.mark.asyncio +async def test_submit_finalization_is_durable_and_idempotent(tmp_path): + candidate = _candidate("submitted") + engine = FakeEngine({("submitted", "test"): 0.9}) + tmp_path.mkdir(exist_ok=True) + (tmp_path / "submission.json").write_text( + Submission(candidate=candidate).model_dump_json(), + encoding="utf-8", + ) + selection = VerificationSelection(mode="submit", baseline_floor=False) + target = VerificationTarget( + reward_key="reward", + backend_id="backend", + evaluation_set=EvaluationSet(name="test"), + objective=OBJECTIVE, + max_attempts=1, + ) + + first = await CanonicalVerifier( + engine=engine, + selection=selection, + targets=[target], + admin_volume=tmp_path, + score_baseline=False, + ).finalize() + engine.scores[("submitted", "test")] = 0.1 + replayed = await CanonicalVerifier( + engine=engine, + selection=selection, + targets=[target], + admin_volume=tmp_path, + score_baseline=False, + ).finalize() + + assert first == replayed + assert replayed.shipped is True + assert replayed.rewards == {"reward": 0.9} + assert engine.calls == [("submitted", "test")] + + +@pytest.mark.asyncio +async def test_verifier_floors_rewards_when_no_candidate_exists(tmp_path): + baseline = _candidate("baseline") + engine = FakeEngine({}) + result = await _verifier( + tmp_path, + engine, + baseline=baseline, + score_baseline=False, + ).finalize() + + assert result.candidate is None + # "Nothing shipped" is an explicit, distinct outcome — not a real zero. + assert result.shipped is False + assert result.rewards == {"reward": 0.0} + assert "selection" in result.errors + + +@pytest.mark.asyncio +async def test_verifier_transforms_minimization_objective_into_higher_reward(tmp_path): + candidate = _candidate("submitted") + minimize = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="minimize", + ) + engine = FakeEngine({("submitted", "latency"): 2.5}) + (tmp_path / "submission.json").write_text( + Submission(candidate=candidate).model_dump_json(), + encoding="utf-8", + ) + verifier = CanonicalVerifier( + engine=engine, + selection=VerificationSelection(mode="submit", baseline_floor=False), + targets=[ + VerificationTarget( + reward_key="latency_reward", + backend_id="backend", + evaluation_set=EvaluationSet(name="latency"), + objective=minimize, + reward_scale=-1.0, + max_attempts=1, + ) + ], + admin_volume=tmp_path, + score_baseline=False, + ) + + result = await verifier.finalize() + + assert result.rewards == {"latency_reward": -2.5} + + +def _record_with_cases(record_id, candidate, evaluation_set, score, case_ids): + now = datetime(2026, 2, 1, tzinfo=UTC) + return EvaluationRecord( + id=record_id, + request=EvaluationRequest(candidate=candidate, evaluation_set=evaluation_set), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": score}, + cases=[ + CaseResult( + case_id=cid, status=CaseStatus.SUCCESS, metrics={"score": score} + ) + for cid in case_ids + ], + ), + backend_id="backend", + backend=StubBackend().provenance, + objective_spec=OBJECTIVE, + objective=ObjectiveResult(value=score, feasible=True), + created_at=now, + completed_at=now, + ) + + +@pytest.mark.asyncio +async def test_verifier_prefers_agent_submission_over_auto_best(tmp_path): + baseline = _candidate("baseline") + picked = _candidate("picked", seconds=5) + engine = FakeEngine({("picked", "test"): 0.9, ("baseline", "test"): 0.5}) + engine.database.add_evaluation( + _record("r", _candidate("other", seconds=1), EvaluationSet(name="selection"), 0.7) + ) + (tmp_path / "submission.json").write_text( + Submission(candidate=picked).model_dump_json() + ) + + result = await _verifier(tmp_path, engine, baseline=baseline).finalize() + + assert result.candidate == picked + assert result.shipped is True + assert result.rewards == {"reward": 0.9} + # The submitted candidate wins outright; auto_best never re-scored 'other'. + assert ("other", "selection") not in engine.calls + + +@pytest.mark.asyncio +async def test_verifier_falls_back_to_last_candidate_when_no_selection_eval(tmp_path): + baseline = _candidate("baseline") + last = _candidate("last", seconds=9) + engine = FakeEngine({("last", "test"): 0.42, ("baseline", "test"): 0.5}) + # Only a non-selection-partition eval exists -> no qualifying selection records. + engine.database.add_evaluation( + _record("r", last, EvaluationSet(name="development"), 0.7) + ) + + result = await _verifier(tmp_path, engine, baseline=baseline).finalize() + + # pick-last ships the most recent candidate instead of shipping nothing. + assert result.candidate == last + assert result.shipped is True + assert result.rewards == {"reward": 0.42} + + +@pytest.mark.asyncio +async def test_verifier_coverage_threshold_excludes_undermeasured_candidates(tmp_path): + baseline = _candidate("baseline") + well = _candidate("well", seconds=1) + thin = _candidate("thin", seconds=2) + engine = FakeEngine( + { + ("well", "selection"): 0.6, + ("well", "test"): 0.7, + ("baseline", "selection"): 0.5, + ("baseline", "test"): 0.4, + } + ) + engine.database.add_evaluation( + _record_with_cases( + "rw", well, EvaluationSet(name="selection"), 0.6, [f"c{i}" for i in range(10)] + ) + ) + # Higher score but only 2/10 coverage -> below the 0.9 threshold -> excluded. + engine.database.add_evaluation( + _record_with_cases("rt", thin, EvaluationSet(name="selection"), 0.99, ["c0", "c1"]) + ) + + result = await _verifier(tmp_path, engine, baseline=baseline, top_k=5).finalize() + + assert result.candidate == well + assert ("thin", "selection") not in engine.calls + assert ("well", "selection") in engine.calls + + +@pytest.mark.asyncio +async def test_verifier_uses_pinned_baseline_reward_without_scoring(tmp_path): + # A pinned target baseline_reward is used verbatim; the seed is never scored. + baseline = _candidate("baseline") + cand = _candidate("cand", seconds=1) + engine = FakeEngine({("cand", "selection"): 0.8, ("cand", "test"): 0.7}) + engine.database.add_evaluation( + _record("r", cand, EvaluationSet(name="selection"), 0.8) + ) + verifier = CanonicalVerifier( + engine=engine, + selection=VerificationSelection( + mode="auto_best", + backend_id="backend", + evaluation_set=EvaluationSet(name="selection"), + objective=OBJECTIVE, + baseline_candidate=baseline, + rescore_top_k=1, + rescore_attempts=1, + ), + targets=[ + VerificationTarget( + reward_key="reward", + backend_id="backend", + evaluation_set=EvaluationSet(name="test"), + objective=OBJECTIVE, + max_attempts=1, + baseline_reward=0.55, + ) + ], + admin_volume=tmp_path, + score_baseline=True, + ) + + result = await verifier.finalize() + + assert result.candidate == cand + assert result.rewards == {"reward": 0.7} + assert result.baseline_rewards == {"reward": 0.55} + assert ("baseline", "test") not in engine.calls # seed never scored + + +@pytest.mark.asyncio +async def test_verifier_uses_pinned_baseline_selection_score(tmp_path): + # With a pinned selection score, the floor compares without re-scoring the seed. + baseline = _candidate("baseline") + cand = _candidate("cand", seconds=1) + engine = FakeEngine({("cand", "selection"): 0.4, ("baseline", "test"): 0.3}) + engine.database.add_evaluation( + _record("r", cand, EvaluationSet(name="selection"), 0.4) + ) + verifier = CanonicalVerifier( + engine=engine, + selection=VerificationSelection( + mode="auto_best", + backend_id="backend", + evaluation_set=EvaluationSet(name="selection"), + objective=OBJECTIVE, + baseline_candidate=baseline, + rescore_top_k=1, + rescore_attempts=1, + baseline_floor=True, + baseline_selection_score=0.6, + ), + targets=[ + VerificationTarget( + reward_key="reward", + backend_id="backend", + evaluation_set=EvaluationSet(name="test"), + objective=OBJECTIVE, + max_attempts=1, + ) + ], + admin_volume=tmp_path, + score_baseline=False, + ) + + result = await verifier.finalize() + + # best (0.4) does not beat the pinned seed (0.6) → floor keeps the seed. + assert result.candidate == baseline + assert ("baseline", "selection") not in engine.calls # seed never re-scored + + +@pytest.mark.asyncio +async def test_verifier_floor_fails_safe_when_seed_unmeasurable(tmp_path): + # Floor on, unpinned, seed re-score fails → ship nothing (inconclusive), + # not the best candidate unverified. + baseline = _candidate("baseline") + cand = _candidate("cand", seconds=1) + engine = FakeEngine( + {("cand", "selection"): 0.9, ("baseline", "selection"): RuntimeError("outage")} + ) + engine.database.add_evaluation( + _record("r", cand, EvaluationSet(name="selection"), 0.9) + ) + + result = await _verifier( + tmp_path, engine, baseline=baseline, baseline_floor=True + ).finalize() + + assert result.shipped is False + assert "baseline floor" in result.errors.get("selection", "") + + +@pytest.mark.asyncio +async def test_verifier_score_baseline_produces_replicated_means(tmp_path): + baseline = _candidate("baseline") + engine = FakeEngine( + { + ("baseline", "selection"): [0.5, 0.6], + ("baseline", "test"): [0.4, 0.5], + } + ) + + out = await _verifier(tmp_path, engine, baseline=baseline).measure_baseline( + replicates=2 + ) + + assert out["candidate_version"] == "baseline" + assert out["replicates"] == 2 + assert out["selection"]["n"] == 2 + assert out["selection"]["mean"] == 0.55 + assert out["targets"]["reward"]["n"] == 2 + assert out["targets"]["reward"]["mean"] == 0.45 + # 2 selection re-scores + 2 target scores, all admin. + assert engine.calls == [ + ("baseline", "selection"), + ("baseline", "selection"), + ("baseline", "test"), + ("baseline", "test"), + ] diff --git a/vero/tests/test_v05_optimizer.py b/vero/tests/test_v05_optimizer.py new file mode 100644 index 00000000..f5af089e --- /dev/null +++ b/vero/tests/test_v05_optimizer.py @@ -0,0 +1,535 @@ +from __future__ import annotations + +import asyncio +import subprocess +import sys +from pathlib import Path + +import pytest + +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import ( + BackendRegistry, + CommandBackend, + CommandBackendConfig, + EvaluationDatabase, + EvaluationEngine, + EvaluationPlan, + EvaluationSet, + Evaluator, + MetricSelector, + ObjectiveSpec, + allow_all_evaluations, +) +from vero.optimization import ( + CandidateChange, + CandidateProposal, + CommandCandidateProducer, + CommandCandidateProducerConfig, + Optimizer, + SequentialStrategy, +) +from vero.sandbox import LocalSandbox +from vero.workspace import GitWorkspace + + +def initialize_repository(path: Path) -> str: + subprocess.run( + ["git", "init", "-b", "main"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=path, + check=True, + capture_output=True, + ) + return subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=path, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +@pytest.mark.asyncio +async def test_optimizer_improves_a_non_python_program_via_external_commands( + tmp_path: Path, +): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("slow\n") + baseline_version = initialize_repository(target) + + harness = tmp_path / "harness" + harness.mkdir() + harness_script = harness / "evaluate.py" + harness_script.write_text( + """ +import json +import sys +from pathlib import Path + +workspace, report_path = map(Path, sys.argv[1:]) +program = (workspace / "program.txt").read_text().strip() +latency = 1.0 if program == "fast" else 10.0 +report_path.write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"latency_ms": latency, "correct": 1.0}, +})) +""" + ) + + producer_root = tmp_path / "producer" + producer_root.mkdir() + producer_script = producer_root / "optimize.py" + producer_script.write_text( + """ +import json +import os +import sys +from pathlib import Path + +workspace = Path(sys.argv[1]) +context = Path(sys.argv[2]) +assert Path(os.environ["VERO_CONTEXT_PATH"]) == context +evaluations = json.loads((context / "results/index.json").read_text()) +assert len(evaluations["evaluations"]) == 1 +(workspace / "program.txt").write_text("fast\\n") +""" + ) + + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(target)) + session_dir = tmp_path / "sessions" / "optimization" + candidate_repository = await GitCandidateRepository.create( + session_dir / "candidates", workspace=workspace + ) + database = EvaluationDatabase(id="optimization") + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=session_dir, + ), + backends=BackendRegistry( + { + "command": CommandBackend( + CommandBackendConfig( + harness_root=str(harness), + command=[ + sys.executable, + str(harness_script), + "{workspace}", + "{report}", + ], + ) + ) + } + ), + database=database, + database_path=session_dir / "database.json", + authorization_resolver=allow_all_evaluations, + ) + optimizer = Optimizer( + workspace=workspace, + candidate_repository=candidate_repository, + engine=engine, + backend_id="command", + evaluation_plan=EvaluationPlan.single(EvaluationSet(name="performance")), + objective=ObjectiveSpec( + selector=MetricSelector(metric="latency_ms"), + direction="minimize", + ), + strategy=SequentialStrategy(), + producers={ + "default": CommandCandidateProducer( + CommandCandidateProducerConfig( + root=str(producer_root), + command=[ + sys.executable, + str(producer_script), + "{workspace}", + "{context}", + ], + description="Use fast implementation", + ) + ) + }, + max_proposals=1, + ) + + result = await optimizer.run() + + assert result.baseline.request.candidate.version == baseline_version + assert result.baseline.objective.value == 10.0 + assert len(result.evaluations) == 2 + assert len(result.candidates) == 2 + assert result.best.objective.value == 1.0 + assert result.best.request.candidate.parent_id == baseline_version + assert result.best.request.candidate.version != baseline_version + assert (target / "program.txt").read_text() == "slow\n" + assert len(database.evaluations) == 2 + assert (session_dir / "database.json").exists() + assert len(list((session_dir / "evaluations").iterdir())) == 2 + worktrees = subprocess.run( + ["git", "worktree", "list", "--porcelain"], + cwd=target, + check=True, + capture_output=True, + text=True, + ).stdout + assert worktrees.count("worktree ") == 1 + candidate_branches = subprocess.run( + [ + "git", + "for-each-ref", + "--format=%(refname)", + "refs/heads/vero-candidate-", + ], + cwd=target, + check=True, + capture_output=True, + text=True, + ).stdout + assert candidate_branches == "" + assert len(candidate_repository.list()) == 2 + assert (candidate_repository.repository_path / "HEAD").exists() + + +class _SpyGenerationBackend: + """A GenerationBackend that records calls and delegates to native production.""" + + def __init__(self, optimizer: Optimizer): + self._optimizer = optimizer + self.calls = 0 + + async def generate(self, **kwargs): + self.calls += 1 + return await self._optimizer._produce_candidate(**kwargs) + + +@pytest.mark.asyncio +async def test_optimizer_routes_production_through_generation_backend(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("slow\n") + baseline_version = initialize_repository(target) + + harness = tmp_path / "harness" + harness.mkdir() + harness_script = harness / "evaluate.py" + harness_script.write_text( + """ +import json +import sys +from pathlib import Path + +workspace, report_path = map(Path, sys.argv[1:]) +program = (workspace / "program.txt").read_text().strip() +latency = 1.0 if program == "fast" else 10.0 +report_path.write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"latency_ms": latency, "correct": 1.0}, +})) +""" + ) + + producer_root = tmp_path / "producer" + producer_root.mkdir() + producer_script = producer_root / "optimize.py" + producer_script.write_text( + """ +import sys +from pathlib import Path + +workspace = Path(sys.argv[1]) +(workspace / "program.txt").write_text("fast\\n") +""" + ) + + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(target)) + session_dir = tmp_path / "sessions" / "optimization" + candidate_repository = await GitCandidateRepository.create( + session_dir / "candidates", workspace=workspace + ) + database = EvaluationDatabase(id="optimization") + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=session_dir, + ), + backends=BackendRegistry( + { + "command": CommandBackend( + CommandBackendConfig( + harness_root=str(harness), + command=[ + sys.executable, + str(harness_script), + "{workspace}", + "{report}", + ], + ) + ) + } + ), + database=database, + database_path=session_dir / "database.json", + authorization_resolver=allow_all_evaluations, + ) + optimizer = Optimizer( + workspace=workspace, + candidate_repository=candidate_repository, + engine=engine, + backend_id="command", + evaluation_plan=EvaluationPlan.single(EvaluationSet(name="performance")), + objective=ObjectiveSpec( + selector=MetricSelector(metric="latency_ms"), + direction="minimize", + ), + strategy=SequentialStrategy(), + producers={ + "default": CommandCandidateProducer( + CommandCandidateProducerConfig( + root=str(producer_root), + command=[sys.executable, str(producer_script), "{workspace}"], + description="Use fast implementation", + ) + ) + }, + max_proposals=1, + ) + spy = _SpyGenerationBackend(optimizer) + optimizer.generation_backend = spy + + result = await optimizer.run() + + # The Optimizer routed production through the injected backend... + assert spy.calls == 1 + # ...and the run still improved the program through it. + assert result.best.objective.value == 1.0 + assert result.best.request.candidate.version != baseline_version + + +class ReusedIdStrategy: + async def propose(self, context): + return [ + CandidateProposal( + id=context.baseline.id, + producer_id="default", + ) + ] + + +class FailingBatchProducer: + def __init__(self, sibling_started): + self.sibling_started = sibling_started + + async def produce(self, **_kwargs): + await self.sibling_started.wait() + raise RuntimeError("producer failed") + + +class CancelledBatchProducer: + def __init__(self): + self.started = asyncio.Event() + self.cancelled = False + + async def produce(self, **_kwargs): + self.started.set() + try: + await asyncio.Event().wait() + except asyncio.CancelledError: + self.cancelled = True + raise + return CandidateChange() + + +class FailingBatchStrategy: + async def propose(self, _context): + return [ + CandidateProposal(id="failing-proposal", producer_id="failing"), + CandidateProposal(id="cancelled-proposal", producer_id="cancelled"), + ] + + +@pytest.mark.asyncio +async def test_optimizer_rejects_strategy_candidate_id_reuse(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("slow\n") + initialize_repository(target) + harness = tmp_path / "harness" + harness.mkdir() + harness_script = harness / "evaluate.py" + harness_script.write_text( + """ +import json +import sys +from pathlib import Path +Path(sys.argv[1]).write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"score": 1.0}, +})) +""" + ) + producer_root = tmp_path / "producer" + producer_root.mkdir() + producer_script = producer_root / "noop.py" + producer_script.write_text("pass\n") + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(target)) + candidate_repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", workspace=workspace + ) + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=tmp_path / "session", + ), + backends=BackendRegistry( + { + "command": CommandBackend( + CommandBackendConfig( + harness_root=str(harness), + command=[sys.executable, str(harness_script), "{report}"], + ) + ) + } + ), + database=EvaluationDatabase(id="session"), + authorization_resolver=allow_all_evaluations, + ) + optimizer = Optimizer( + workspace=workspace, + candidate_repository=candidate_repository, + engine=engine, + backend_id="command", + evaluation_plan=EvaluationPlan.single(EvaluationSet()), + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + strategy=ReusedIdStrategy(), + producers={ + "default": CommandCandidateProducer( + CommandCandidateProducerConfig( + root=str(producer_root), + command=[sys.executable, str(producer_script)], + ) + ) + }, + ) + + with pytest.raises(ValueError, match="reused an existing candidate ID"): + await optimizer.run() + + +@pytest.mark.asyncio +async def test_optimizer_cancels_sibling_producers_and_cleans_worktrees( + tmp_path: Path, +): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("slow\n") + initialize_repository(target) + harness = tmp_path / "harness" + harness.mkdir() + harness_script = harness / "evaluate.py" + harness_script.write_text( + """ +import json +import sys +from pathlib import Path +Path(sys.argv[1]).write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"score": 1.0}, +})) +""" + ) + sandbox = await LocalSandbox.create(root=tmp_path) + workspace = await GitWorkspace.from_path(sandbox, str(target)) + candidate_repository = await GitCandidateRepository.create( + tmp_path / "session" / "candidates", workspace=workspace + ) + engine = EvaluationEngine( + evaluator=Evaluator( + candidate_repository=candidate_repository, + sandbox=workspace.sandbox, + session_dir=tmp_path / "session", + ), + backends=BackendRegistry( + { + "command": CommandBackend( + CommandBackendConfig( + harness_root=str(harness), + command=[sys.executable, str(harness_script), "{report}"], + ) + ) + } + ), + database=EvaluationDatabase(id="session"), + authorization_resolver=allow_all_evaluations, + ) + cancelled = CancelledBatchProducer() + optimizer = Optimizer( + workspace=workspace, + candidate_repository=candidate_repository, + engine=engine, + backend_id="command", + evaluation_plan=EvaluationPlan.single(EvaluationSet()), + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + strategy=FailingBatchStrategy(), + producers={ + "failing": FailingBatchProducer(cancelled.started), + "cancelled": cancelled, + }, + max_proposals=2, + max_concurrency=2, + ) + + with pytest.raises(ExceptionGroup) as captured: + await optimizer.run() + + assert any( + isinstance(error, RuntimeError) and str(error) == "producer failed" + for error in captured.value.exceptions + ) + assert cancelled.cancelled + worktrees = subprocess.run( + ["git", "worktree", "list", "--porcelain"], + cwd=target, + check=True, + capture_output=True, + text=True, + ).stdout + assert worktrees.count("worktree ") == 1 diff --git a/vero/tests/test_v05_python_example.py b/vero/tests/test_v05_python_example.py new file mode 100644 index 00000000..15925e18 --- /dev/null +++ b/vero/tests/test_v05_python_example.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + + +def test_advertised_python_matmul_evaluation_runs_end_to_end(tmp_path: Path): + script = ( + Path(__file__).parents[2] + / "vero-tasks" + / "examples" + / "matmul-kernel" + / "run.py" + ) + environment = dict(os.environ) + environment["UV_CACHE_DIR"] = str( + Path(os.environ.get("UV_CACHE_DIR", tmp_path / "uv-cache")).resolve() + ) + + result = subprocess.run( + [ + sys.executable, + str(script), + "--eval-only", + "--work-dir", + str(tmp_path / "example"), + ], + cwd=script.parents[2], + env=environment, + capture_output=True, + text=True, + timeout=120, + check=False, + ) + + assert result.returncode == 0, result.stdout + result.stderr + assert "Baseline score:" in result.stdout + assert (tmp_path / "example" / "session" / "manifest.json").exists() diff --git a/vero/tests/test_v05_python_task_backend.py b/vero/tests/test_v05_python_task_backend.py new file mode 100644 index 00000000..0979dbf4 --- /dev/null +++ b/vero/tests/test_v05_python_task_backend.py @@ -0,0 +1,323 @@ +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from vero.evaluation import ( + AllCases, + CaseIds, + CaseRange, + EvaluationPlan, + EvaluationSet, + MetricSelector, + ObjectiveSpec, + PythonTaskBackend, + PythonTaskBackendConfig, + PythonTaskEvaluationConfig, +) +from vero.optimization import ( + CommandCandidateProducer, + CommandCandidateProducerConfig, +) +from vero.runtime import create_local_optimization_session +from vero.sandbox import LocalSandbox + + +def initialize_repository(path: Path) -> None: + subprocess.run( + ["git", "init", "-b", "main"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=path, + check=True, + capture_output=True, + ) + + +@pytest.mark.asyncio +async def test_python_task_backend_builds_uv_command_and_resolves_cost(tmp_path: Path): + cases = tmp_path / "cases.json" + cases.write_text( + json.dumps([{"id": "a"}, {"id": "b"}, {"id": "c"}]), + encoding="utf-8", + ) + backend = PythonTaskBackend( + PythonTaskBackendConfig( + harness_root=str(tmp_path), + module="target.tasks", + task="quality", + evaluations=[ + PythonTaskEvaluationConfig(name="default", cases_path=str(cases)) + ], + target_project_directory="packages/target", + uv_executable=sys.executable, + ) + ) + + command = backend._command(EvaluationSet()).config.command + assert command[:6] == [ + sys.executable, + "run", + "--project", + "{harness}", + "--with-editable", + "{workspace}/packages/target", + ] + assert command[-4:] == ["--request", "{request}", "--report", "{report}"] + assert "{input:cases}" in command + assert (await backend.resolve_cost(EvaluationSet(selection=AllCases()))).cases == 3 + assert ( + await backend.resolve_cost(EvaluationSet(selection=CaseRange(start=1, stop=3))) + ).cases == 2 + assert ( + await backend.resolve_cost(EvaluationSet(selection=CaseIds(ids=["a", "c"]))) + ).cases == 2 + + with pytest.raises(ValueError, match="contains 3 cases"): + await backend.resolve_cost(EvaluationSet(selection=CaseRange(stop=4))) + + with pytest.raises(ValueError, match="unknown Python task case IDs"): + await backend.resolve_cost(EvaluationSet(selection=CaseIds(ids=["missing"]))) + + +def test_python_task_backend_requires_external_case_file(tmp_path: Path): + missing = tmp_path / "missing.json" + + with pytest.raises(ValueError, match="existing file"): + PythonTaskBackendConfig( + harness_root=str(tmp_path), + module="target.tasks", + task="quality", + evaluations=[ + PythonTaskEvaluationConfig(name="default", cases_path=str(missing)) + ], + uv_executable=sys.executable, + ) + + +@pytest.mark.asyncio +async def test_python_task_backend_exports_only_selected_cases(tmp_path: Path): + cases = tmp_path / "cases.json" + cases.write_text( + json.dumps( + [ + {"id": "a", "prompt": "alpha"}, + {"id": "b", "prompt": "beta"}, + {"id": "c", "prompt": "gamma"}, + ] + ), + encoding="utf-8", + ) + destination = tmp_path / "context" + destination.mkdir() + backend = PythonTaskBackend( + PythonTaskBackendConfig( + harness_root=str(tmp_path), + module="target.tasks", + task="quality", + evaluations=[ + PythonTaskEvaluationConfig(name="default", cases_path=str(cases)) + ], + uv_executable=sys.executable, + ) + ) + + await backend.export_case_resources( + evaluation_set=EvaluationSet(selection=CaseIds(ids=["c", "a"])), + destination=str(destination), + sandbox=await LocalSandbox.create(root=tmp_path), + ) + + index = json.loads((destination / "index.json").read_text()) + assert [item["case_id"] for item in index["cases"]] == ["c", "a"] + exported = [ + json.loads((destination / item["path"]).read_text()) for item in index["cases"] + ] + assert exported == [ + {"id": "c", "prompt": "gamma"}, + {"id": "a", "prompt": "alpha"}, + ] + + +@pytest.mark.asyncio +async def test_python_task_backend_routes_named_partitions_to_distinct_datasets( + tmp_path: Path, +): + train = tmp_path / "train.json" + train.write_text(json.dumps([{"id": "train-a"}]), encoding="utf-8") + validation = tmp_path / "validation.json" + validation.write_text( + json.dumps([{"id": "val-a"}, {"id": "val-b"}]), + encoding="utf-8", + ) + backend = PythonTaskBackend( + PythonTaskBackendConfig( + harness_root=str(tmp_path), + module="target.tasks", + task="quality", + evaluations=[ + PythonTaskEvaluationConfig( + name="train", + partition="train", + cases_path=str(train), + ), + PythonTaskEvaluationConfig( + name="validation", + partition="validation", + cases_path=str(validation), + ), + ], + uv_executable=sys.executable, + ) + ) + + assert ( + await backend.resolve_cost(EvaluationSet(name="train", partition="train")) + ).cases == 1 + assert ( + await backend.resolve_cost( + EvaluationSet(name="validation", partition="validation") + ) + ).cases == 2 + with pytest.raises(ValueError, match="does not own evaluation"): + await backend.resolve_cost(EvaluationSet(name="test", partition="test")) + + +@pytest.mark.asyncio +async def test_python_task_backend_optimizes_target_task_module(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "factor.txt").write_text("1\n", encoding="utf-8") + (target / "target_program.py").write_text( + """ +from pathlib import Path + +def multiply(value): + factor = int(Path(__file__).with_name("factor.txt").read_text()) + return value * factor +""", + encoding="utf-8", + ) + (tmp_path / "evaluation_tasks.py").write_text( + """ +from vero_tasks import TaskOutput, TaskResult, create_task +from target_program import multiply + +task = create_task("multiply") + +@task.inference() +async def infer(case, context): + return TaskOutput(output=multiply(case["value"])) + +@task.evaluation() +async def evaluate(case, output, context): + return TaskResult.from_task_output( + output, + score=float(output.output == case["expected"]), + ) +""", + encoding="utf-8", + ) + initialize_repository(target) + + cases = tmp_path / "cases.json" + cases.write_text( + json.dumps( + [ + {"id": "one", "value": 2, "expected": 4}, + {"id": "two", "value": 3, "expected": 6}, + ] + ), + encoding="utf-8", + ) + tasks_source = Path(__file__).parents[2] / "vero-tasks" / "src" + fake_uv = tmp_path / "uv" + fake_uv.write_text( + f"""#!{sys.executable} +import runpy +import sys + +arguments = sys.argv[1:] +project = arguments[arguments.index("--project") + 1] +target = arguments[arguments.index("--with-editable") + 1] +module_index = arguments.index("-m") +module = arguments[module_index + 1] +sys.path[:0] = [project, target, {str(tasks_source)!r}] +sys.argv = [module, *arguments[module_index + 2:]] +runpy.run_module(module, run_name="__main__") +""", + encoding="utf-8", + ) + fake_uv.chmod(0o755) + + producer_root = tmp_path / "producer" + producer_root.mkdir() + producer_script = producer_root / "improve.py" + producer_script.write_text( + """ +import sys +from pathlib import Path +Path(sys.argv[1], "factor.txt").write_text("2\\n") +""", + encoding="utf-8", + ) + backend = PythonTaskBackend( + PythonTaskBackendConfig( + harness_root=str(tmp_path), + module="evaluation_tasks", + task="multiply", + evaluations=[ + PythonTaskEvaluationConfig(name="default", cases_path=str(cases)) + ], + uv_executable=str(fake_uv), + ) + ) + producer = CommandCandidateProducer( + CommandCandidateProducerConfig( + root=str(producer_root), + command=[sys.executable, str(producer_script), "{workspace}"], + ) + ) + session = await create_local_optimization_session( + project_path=target, + session_dir=tmp_path / "sessions" / "python-task", + backend_id="python-task", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + evaluation_plan=EvaluationPlan.single(), + producers={"default": producer}, + max_proposals=1, + ) + + result = await session.run() + + assert result.baseline.objective.value == 0.0 + assert result.best.objective.value == 1.0 + assert [case.case_id for case in result.best.report.cases] == ["one", "two"] + assert (target / "factor.txt").read_text(encoding="utf-8") == "1\n" diff --git a/vero/tests/test_v05_report.py b/vero/tests/test_v05_report.py new file mode 100644 index 00000000..a5d316e5 --- /dev/null +++ b/vero/tests/test_v05_report.py @@ -0,0 +1,228 @@ +from __future__ import annotations + +import asyncio +import hashlib +import json +import subprocess +from datetime import UTC, datetime, timedelta +from pathlib import Path + +from click.testing import CliRunner + +from vero.candidate import Candidate +from vero.candidate_repository import GitCandidateRepository +from vero.cli import main +from vero.evaluation import ( + BackendProvenance, + EvaluationArtifact, + EvaluationDatabase, + EvaluationPlan, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, +) +from vero.runtime import ( + OptimizationComponentSpec, + OptimizationRunSpec, + RuntimeEvent, + SessionManifest, + SessionStatus, +) +from vero.sandbox import LocalSandbox +from vero.workspace import GitWorkspace + + +def git(path: Path, *arguments: str) -> str: + result = subprocess.run( + ["git", *arguments], + cwd=path, + check=True, + capture_output=True, + text=True, + ) + return result.stdout.strip() + + +async def build_session(root: Path) -> Path: + source = root / "source" + source.mkdir() + program = source / "program.py" + program.write_text("score = 1\n", encoding="utf-8") + git(source, "init", "-b", "main") + git(source, "add", "--all") + git( + source, + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ) + + sandbox = await LocalSandbox.create(root=root) + workspace = await GitWorkspace.from_path(sandbox, str(source)) + session_dir = root / "session" + repository = await GitCandidateRepository.create( + session_dir / "candidates", workspace=workspace + ) + baseline = Candidate.from_version( + git(source, "rev-parse", "HEAD"), candidate_id="baseline" + ) + await repository.capture(baseline, workspace) + + program.write_text("score = 2\n", encoding="utf-8") + version = await workspace.save("improve score") + proposal_id = "proposal-1" + candidate = Candidate.from_version( + version, + candidate_id="candidate-1", + parent_id=baseline.id, + description="Improve ", + metadata={"proposal_id": proposal_id, "producer_id": "test"}, + ) + await repository.capture(candidate, workspace) + + objective = ObjectiveSpec(selector=MetricSelector(metric="score"), direction="maximize") + provenance = BackendProvenance.from_config(name="test", version="1", config={}) + created = datetime(2026, 1, 1, tzinfo=UTC) + database = EvaluationDatabase(id="report-test") + for index, (evaluated, value) in enumerate(((baseline, 1.0), (candidate, 2.0))): + evaluation_id = f"evaluation-{index}" + record = EvaluationRecord( + id=evaluation_id, + request=EvaluationRequest( + candidate=evaluated, + evaluation_set=EvaluationSet(name="development"), + ), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": value}, + artifacts=( + [ + EvaluationArtifact( + path="preview.svg", + media_type="image/svg+xml", + description="Program output", + ) + ] + if evaluated is candidate + else [] + ), + ), + backend_id="test", + backend=provenance, + objective_spec=objective, + objective=ObjectiveResult(value=value, feasible=True), + created_at=created + timedelta(minutes=index), + completed_at=created + timedelta(minutes=index, seconds=1), + ) + database.add_evaluation(record) + if evaluated is candidate: + artifact_dir = session_dir / "evaluations" / evaluation_id / "artifacts" + artifact_dir.mkdir(parents=True) + (artifact_dir / "preview.svg").write_text( + '', + encoding="utf-8", + ) + database.save_to_file(session_dir / "database.json") + + component = OptimizationComponentSpec(type="test", config_digest="0" * 64) + manifest = SessionManifest( + id="report-test", + status=SessionStatus.COMPLETED, + backend_id="test", + backend=provenance, + candidate_repository_family="git", + candidate_repository_format_version=1, + evaluation_plan=EvaluationPlan.single(EvaluationSet(name="development")), + objective=objective, + run=OptimizationRunSpec( + max_proposals=1, + max_rounds=1, + max_concurrency=1, + strategy=component, + producers={"test": component}, + ), + baseline=baseline, + best_candidate_id=candidate.id, + best_evaluation_id="evaluation-1", + created_at=created, + updated_at=created + timedelta(minutes=2), + ) + (session_dir / "manifest.json").write_text( + manifest.model_dump_json(indent=2), encoding="utf-8" + ) + event = RuntimeEvent( + session_id=manifest.id, + kind="evaluation_completed", + created_at=created + timedelta(minutes=1), + payload={"evaluation_id": "evaluation-1", "objective/value": 2.0}, + ) + (session_dir / "events.jsonl").write_text( + event.model_dump_json() + "\n", encoding="utf-8" + ) + trace_id = hashlib.sha256(proposal_id.encode()).hexdigest()[:16] + trace_dir = session_dir / "artifacts" / "agents" / trace_id + trace_dir.mkdir(parents=True) + (trace_dir / "trace.json").write_text( + json.dumps( + [ + {"role": "user", "content": "Improve the score"}, + { + "type": "function_call", + "name": "edit", + "arguments": '{"path":"program.py"}', + }, + ] + ), + encoding="utf-8", + ) + return session_dir + + +def report_payload(html: str) -> dict[str, object]: + opening = '", 1)[0] + return json.loads(encoded) + + +def test_report_command_builds_portable_full_experiment_view(tmp_path: Path): + session_dir = asyncio.run(build_session(tmp_path)) + output = tmp_path / "report.html" + + result = CliRunner().invoke( + main, ["report", str(session_dir), "--output", str(output)] + ) + + assert result.exit_code == 0, result.output + assert str(output) in result.output + html = output.read_text(encoding="utf-8") + payload = report_payload(html) + assert len(payload["candidates"]) == 2 + assert len(payload["evaluations"]) == 2 + assert payload["candidates"][1]["trace_id"] is not None + assert "+score = 2" in payload["candidates"][1]["diff"]["text"] + assert payload["evaluations"][1]["artifacts"][0]["kind"] == "image" + assert payload["evaluations"][1]["artifacts"][0]["content"].startswith( + "data:image/svg+xml;base64," + ) + assert payload["traces"][0]["entries"][1]["kind"] == "tool-call" + assert payload["events"][0]["kind"] == "evaluation_completed" + assert "" not in html + assert "Score trajectories by split" in html + assert "Lines connect exact case selections only" in html + assert "Comparable baseline" in html + + +def test_report_command_fails_clearly_without_a_manifest(tmp_path: Path): + result = CliRunner().invoke(main, ["report", str(tmp_path)]) + + assert result.exit_code == 1 + assert "session manifest not found" in result.output diff --git a/vero/tests/test_v05_runtime_factory.py b/vero/tests/test_v05_runtime_factory.py new file mode 100644 index 00000000..5354fb3d --- /dev/null +++ b/vero/tests/test_v05_runtime_factory.py @@ -0,0 +1,510 @@ +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from vero.candidate import Candidate +from vero.candidate_repository import GitCandidateRepository +from vero.evaluation import ( + CommandBackend, + CommandBackendConfig, + DisclosureLevel, + EvaluationAccessPolicy, + EvaluationDefinition, + EvaluationPlan, + EvaluationSet, + MetricSelector, + ObjectiveSpec, + allow_all_evaluations, +) +from vero.optimization import ( + CommandCandidateProducer, + CommandCandidateProducerConfig, +) +from vero.runtime import ( + SessionStatus, + create_local_optimization_session, + create_optimization_session, +) +from vero.sandbox import LocalSandbox +from vero.workspace import GitWorkspace + + +def initialize_repository(path: Path) -> str: + subprocess.run( + ["git", "init", "-b", "main"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + ["git", "add", "--all"], + cwd=path, + check=True, + capture_output=True, + ) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "baseline", + ], + cwd=path, + check=True, + capture_output=True, + ) + return subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=path, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def command_components(tmp_path: Path): + harness = tmp_path / "harness" + harness.mkdir() + harness_script = harness / "evaluate.py" + harness_script.write_text( + """ +import json +import sys +from pathlib import Path + +workspace, report_path = map(Path, sys.argv[1:]) +value = (workspace / "program.txt").read_text().strip() +score = {"improved": 1.0, "improved-again": 2.0}.get(value, 0.0) +report_path.write_text(json.dumps({ + "schema_version": 1, + "status": "success", + "metrics": {"score": score}, +})) +""", + encoding="utf-8", + ) + producer_root = tmp_path / "producer" + producer_root.mkdir() + producer_script = producer_root / "improve.py" + producer_script.write_text( + """ +import sys +from pathlib import Path +value = "improved" if sys.argv[2] == "0" else "improved-again" +Path(sys.argv[1], "program.txt").write_text(value + "\\n") +""", + encoding="utf-8", + ) + backend = CommandBackend( + CommandBackendConfig( + harness_root=str(harness), + command=[ + sys.executable, + str(harness_script), + "{workspace}", + "{report}", + ], + ) + ) + producer = CommandCandidateProducer( + CommandCandidateProducerConfig( + root=str(producer_root), + command=[ + sys.executable, + str(producer_script), + "{workspace}", + "{round}", + ], + description="Improve the program", + ) + ) + return backend, producer + + +@pytest.mark.asyncio +async def test_generic_factory_accepts_a_provisioned_workspace(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + initialize_repository(target) + backend, _ = command_components(tmp_path) + workspace = await GitWorkspace.from_path(LocalSandbox(tmp_path), str(target)) + session_dir = tmp_path / "sessions" / "generic" + candidate_repository = await GitCandidateRepository.create( + session_dir / "candidates", workspace=workspace + ) + + session = await create_optimization_session( + workspace=workspace, + candidate_repository=candidate_repository, + session_dir=session_dir, + backend_id="command", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + evaluation_plan=EvaluationPlan.single(), + producers={}, + max_proposals=0, + authorization_resolver=allow_all_evaluations, + ) + result = await session.run() + + assert result.baseline.objective.value == 0.0 + + +@pytest.mark.asyncio +async def test_local_factory_builds_and_resumes_generic_session(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + baseline_version = initialize_repository(target) + backend, producer = command_components(tmp_path) + session_dir = tmp_path / "sessions" / "factory" + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + + session = await create_local_optimization_session( + project_path=target, + session_dir=session_dir, + session_id="stable-id", + backend_id="command", + backend=backend, + objective=objective, + evaluation_plan=EvaluationPlan.single(EvaluationSet(name="quality")), + producers={"default": producer}, + max_proposals=1, + ) + result = await session.run() + + assert session.id == "stable-id" + assert result.baseline.request.candidate.version == baseline_version + assert result.best.objective.value == 1.0 + assert (target / "program.txt").read_text(encoding="utf-8") == "baseline\n" + assert session.load_manifest().status == SessionStatus.COMPLETED + assert len(session.database.evaluations) == 2 + + # Simulate a crash after the canonical evaluation directory was committed + # but before database.json was updated. + database_path = session_dir / "database.json" + stale_database = json.loads(database_path.read_text(encoding="utf-8")) + stale_database["evaluations"].pop(result.best.id) + stale_database["candidates"].pop(result.best.request.candidate.id) + database_path.write_text(json.dumps(stale_database), encoding="utf-8") + + resumed = await create_local_optimization_session( + project_path=target, + session_dir=session_dir, + session_id=None, + backend_id="command", + backend=backend, + objective=objective, + evaluation_plan=EvaluationPlan.single(EvaluationSet(name="quality")), + producers={"default": producer}, + max_proposals=1, + ) + resumed_result = await resumed.run(skip_baseline_evaluation=True) + + assert len(resumed.database.evaluations) == 2 + assert resumed.id == "stable-id" + assert resumed_result.baseline.id == result.baseline.id + assert len(resumed_result.evaluations) == 2 + assert resumed_result.best.id == result.best.id + + +@pytest.mark.asyncio +async def test_resume_evaluates_a_durable_candidate_captured_before_crash( + tmp_path: Path, +): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + initialize_repository(target) + backend, _ = command_components(tmp_path) + session_dir = tmp_path / "sessions" / "captured" + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + session = await create_local_optimization_session( + project_path=target, + session_dir=session_dir, + session_id="captured", + backend_id="command", + backend=backend, + objective=objective, + evaluation_plan=EvaluationPlan.single(), + producers={}, + max_proposals=0, + ) + initial = await session.run() + baseline = initial.baseline.request.candidate + + async with session.candidate_repository.checkout( + baseline, + sandbox=session.workspace.sandbox, + ) as candidate_workspace: + candidate_path = Path(candidate_workspace.project_path) / "program.txt" + candidate_path.write_text("improved\n", encoding="utf-8") + version = await candidate_workspace.save("captured before evaluation") + candidate = Candidate.from_version( + version, + candidate_id="captured-candidate", + parent_id=baseline.id, + description="Captured before evaluation", + metadata={ + "producer_id": "default", + "proposal_id": "captured-candidate", + "round": 0, + }, + ) + await session.candidate_repository.capture(candidate, candidate_workspace) + + resumed = await create_local_optimization_session( + project_path=target, + session_dir=session_dir, + session_id=None, + backend_id="command", + backend=backend, + objective=objective, + evaluation_plan=EvaluationPlan.single(), + producers={}, + max_proposals=0, + ) + result = await resumed.run(skip_baseline_evaluation=True) + + assert len(result.evaluations) == 2 + assert result.best.request.candidate.id == "captured-candidate" + assert result.best.objective.value == 1.0 + + +@pytest.mark.asyncio +async def test_factory_reuses_baseline_captured_before_manifest_write(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + initialize_repository(target) + backend, _ = command_components(tmp_path) + session_dir = tmp_path / "sessions" / "baseline-crash" + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + first = await create_local_optimization_session( + project_path=target, + session_dir=session_dir, + backend_id="command", + backend=backend, + objective=objective, + evaluation_plan=EvaluationPlan.single(), + producers={}, + max_proposals=0, + ) + assert not first.manifest_path.exists() + captured = first.baseline + + recreated = await create_local_optimization_session( + project_path=target, + session_dir=session_dir, + backend_id="command", + backend=backend, + objective=objective, + evaluation_plan=EvaluationPlan.single(), + producers={}, + max_proposals=0, + ) + + assert recreated.baseline == captured + assert recreated.candidate_repository.list() == (captured,) + + +@pytest.mark.asyncio +async def test_local_factory_rejects_changed_run_protocol_after_resume(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + initialize_repository(target) + backend, producer = command_components(tmp_path) + session_dir = tmp_path / "sessions" / "resume" + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + kwargs = { + "project_path": target, + "session_dir": session_dir, + "session_id": "resume", + "backend_id": "command", + "backend": backend, + "objective": objective, + "evaluation_plan": EvaluationPlan.single(EvaluationSet(name="quality")), + "producers": {"default": producer}, + } + + first = await create_local_optimization_session(max_proposals=1, **kwargs) + first_result = await first.run() + resumed = await create_local_optimization_session(max_proposals=2, **kwargs) + + assert len(first_result.evaluations) == 2 + with pytest.raises(ValueError, match="run protocol"): + await resumed.run(skip_baseline_evaluation=True) + + +@pytest.mark.asyncio +async def test_local_factory_can_evaluate_an_older_target_ref(tmp_path: Path): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + baseline_version = initialize_repository(target) + (target / "program.txt").write_text("improved\n", encoding="utf-8") + subprocess.run(["git", "add", "--all"], cwd=target, check=True) + subprocess.run( + [ + "git", + "-c", + "user.name=vero", + "-c", + "user.email=vero@localhost", + "commit", + "-m", + "new head", + ], + cwd=target, + check=True, + capture_output=True, + ) + head_version = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=target, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + backend, _ = command_components(tmp_path) + + session = await create_local_optimization_session( + project_path=target, + session_dir=tmp_path / "sessions" / "old-ref", + session_id="old-ref", + backend_id="command", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + evaluation_plan=EvaluationPlan.single(), + producers={}, + max_proposals=0, + base_ref=baseline_version, + ) + result = await session.run() + + assert result.baseline.request.candidate.version == baseline_version + assert result.baseline.objective.value == 0.0 + assert head_version != baseline_version + assert ( + subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=target, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + == head_version + ) + + +@pytest.mark.asyncio +async def test_local_factory_rejects_state_inside_or_outside_version_control( + tmp_path: Path, +): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + initialize_repository(target) + backend, producer = command_components(tmp_path) + kwargs = { + "project_path": target, + "backend_id": "command", + "backend": backend, + "objective": ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + "evaluation_plan": EvaluationPlan.single(), + "producers": {"default": producer}, + } + + with pytest.raises(ValueError, match="outside the target repository"): + await create_local_optimization_session( + session_dir=target / ".vero" / "session", + **kwargs, + ) + + (target / "program.txt").write_text("dirty\n", encoding="utf-8") + with pytest.raises(ValueError, match="must be clean"): + await create_local_optimization_session( + session_dir=tmp_path / "sessions" / "dirty", + **kwargs, + ) + + +@pytest.mark.asyncio +async def test_session_uses_canonical_selection_and_hidden_final_evaluations( + tmp_path: Path, +): + target = tmp_path / "target" + target.mkdir() + (target / "program.txt").write_text("baseline\n", encoding="utf-8") + initialize_repository(target) + backend, _ = command_components(tmp_path) + validation = EvaluationSet(name="validation", partition="validation") + final = EvaluationSet(name="test", partition="test") + plan = EvaluationPlan( + evaluations=[ + EvaluationDefinition(evaluation_set=validation), + EvaluationDefinition( + evaluation_set=final, + access=EvaluationAccessPolicy( + agent_can_evaluate=False, + agent_visible=False, + disclosure=DisclosureLevel.NONE, + ), + ), + ], + selection_evaluation="validation", + final_evaluation="test", + ) + + session = await create_local_optimization_session( + project_path=target, + session_dir=tmp_path / "sessions" / "final", + backend_id="command", + backend=backend, + objective=ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ), + evaluation_plan=plan, + producers={}, + max_proposals=0, + ) + result = await session.run() + + assert result.baseline.request.evaluation_set == validation + assert result.final_baseline is not None + assert result.final_baseline.request.evaluation_set == final + assert result.final == result.final_baseline + assert session.load_manifest().final_evaluation_id == result.final.id diff --git a/vero/tests/test_v05_runtime_session.py b/vero/tests/test_v05_runtime_session.py new file mode 100644 index 00000000..9544cbd6 --- /dev/null +++ b/vero/tests/test_v05_runtime_session.py @@ -0,0 +1,300 @@ +import json +from datetime import UTC, datetime, timedelta +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from vero.candidate import Candidate +from vero.evaluation import ( + BackendProvenance, + EvaluationDatabase, + EvaluationLimits, + EvaluationPlan, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, +) +from vero.optimization import OptimizationResult +from vero.runtime import ArtifactStore, EventBus, OptimizationSession, SessionStatus + + +def evaluation(candidate: Candidate) -> EvaluationRecord: + created_at = datetime(2026, 1, 1, tzinfo=UTC) + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + return EvaluationRecord( + id=f"evaluation:{candidate.id}", + request=EvaluationRequest(candidate=candidate), + report=EvaluationReport( + status=EvaluationStatus.SUCCESS, + metrics={"score": 1.0}, + ), + backend_id="default", + backend=BackendProvenance( + name="fake", + version="1", + config_digest="0" * 64, + ), + objective_spec=objective, + objective=ObjectiveResult(value=1.0, feasible=True), + created_at=created_at, + completed_at=created_at + timedelta(seconds=1), + ) + + +class StubWorkspace: + async def current_version(self): + return "baseline-version" + + +class StubCandidateRepository: + family = "stub" + format_version = 1 + + +class StubOptimizer: + def __init__(self, session_dir: Path, *, failure: Exception | None = None): + self.workspace = StubWorkspace() + self.candidate_repository = StubCandidateRepository() + self.backend_id = "default" + self.evaluation_plan = EvaluationPlan.single( + EvaluationSet(name="performance") + ) + self.objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), + direction="maximize", + ) + self.parameters = {} + self.limits = EvaluationLimits() + self.seed = None + self.session_id = None + self.max_proposals = 0 + self.max_rounds = 1 + self.max_concurrency = 1 + self.strategy = SimpleNamespace() + self.producers = {} + self.selection = SimpleNamespace() + self.engine = SimpleNamespace( + evaluator=SimpleNamespace( + session_dir=session_dir, + candidate_repository=self.candidate_repository, + ), + database=EvaluationDatabase(id=session_dir.name), + budget_ledger=None, + listeners=[], + backends=SimpleNamespace( + resolve=lambda _: SimpleNamespace( + provenance=BackendProvenance( + name="fake", + version="1", + config_digest="0" * 64, + ) + ) + ), + ) + self.failure = failure + self.calls: list[tuple[Candidate, bool]] = [] + + async def run( + self, + *, + baseline, + skip_baseline_evaluation=False, + max_proposals=None, + ): + self.calls.append((baseline, skip_baseline_evaluation)) + if self.failure is not None: + raise self.failure + baseline_record = evaluation(baseline) + for listener in self.engine.listeners: + result = listener(baseline_record) + if result is not None: + await result + return OptimizationResult( + baseline=baseline_record, + evaluations=(baseline_record,), + candidates=(baseline,), + best=baseline_record, + ) + + +@pytest.mark.asyncio +async def test_session_persists_lifecycle_events_and_best_result(tmp_path: Path): + session_dir = tmp_path / "sessions" / "run-1" + optimizer = StubOptimizer(session_dir) + session = OptimizationSession( + id="run-1", + session_dir=session_dir, + optimizer=optimizer, + metadata={"purpose": "test"}, + ) + + result = await session.run() + + manifest = session.load_manifest() + assert manifest.schema_version == 3 + assert manifest.status == SessionStatus.COMPLETED + assert manifest.baseline.version == "baseline-version" + assert manifest.best_candidate_id == "baseline-version" + assert manifest.best_evaluation_id == result.best.id + assert manifest.metadata == {"purpose": "test"} + events = [json.loads(line) for line in session.events_path.read_text().splitlines()] + assert [event["kind"] for event in events] == [ + "session_started", + "evaluation_completed", + "session_completed", + ] + assert session.database is optimizer.engine.database + + +@pytest.mark.asyncio +async def test_session_persists_failure_before_reraising(tmp_path: Path): + session_dir = tmp_path / "sessions" / "failed" + session = OptimizationSession( + id="failed", + session_dir=session_dir, + optimizer=StubOptimizer(session_dir, failure=RuntimeError("producer exploded")), + ) + + with pytest.raises(RuntimeError, match="producer exploded"): + await session.run() + + manifest = session.load_manifest() + assert manifest.status == SessionStatus.FAILED + assert manifest.failure.type.endswith("RuntimeError") + assert manifest.failure.message == "producer exploded" + events = [json.loads(line) for line in session.events_path.read_text().splitlines()] + assert events[-1]["kind"] == "session_failed" + + +@pytest.mark.asyncio +async def test_session_rejects_resume_with_changed_objective(tmp_path: Path): + session_dir = tmp_path / "sessions" / "changed-objective" + optimizer = StubOptimizer(session_dir) + session = OptimizationSession( + id="changed-objective", + session_dir=session_dir, + optimizer=optimizer, + ) + await session.run() + optimizer.objective = ObjectiveSpec( + selector=MetricSelector(metric="different"), + direction="minimize", + ) + + with pytest.raises(ValueError, match="objective does not match"): + await session.run(skip_baseline_evaluation=True) + + +@pytest.mark.asyncio +async def test_session_rejects_resume_with_changed_evaluation_parameters( + tmp_path: Path, +): + session_dir = tmp_path / "sessions" / "changed-parameters" + optimizer = StubOptimizer(session_dir) + session = OptimizationSession( + id="changed-parameters", + session_dir=session_dir, + optimizer=optimizer, + ) + await session.run() + optimizer.parameters = {"temperature": 0.5} + + with pytest.raises(ValueError, match="parameters do not match"): + await session.run(skip_baseline_evaluation=True) + + +def test_component_spec_reflects_strategy_settings(): + from vero.optimization import EvolutionaryStrategy + + objective = ObjectiveSpec( + selector=MetricSelector(metric="score"), direction="maximize" + ) + base = EvolutionaryStrategy(objective=objective, population_size=8, seed=1) + bigger = EvolutionaryStrategy(objective=objective, population_size=16, seed=1) + reseeded = EvolutionaryStrategy(objective=objective, population_size=8, seed=2) + + base_digest = OptimizationSession._component_spec(base).config_digest + # Built-in strategy settings and the seed now change the resume identity, so + # a resume under materially different search semantics is rejected. + assert base_digest != OptimizationSession._component_spec(bigger).config_digest + assert base_digest != OptimizationSession._component_spec(reseeded).config_digest + + +def test_component_spec_reflects_selection_policy(): + from vero.optimization.strategy import ObjectiveSelectionPolicy + + class _AltSelectionPolicy: + def select(self, records, objective): + return None + + # The selection policy is part of the run identity, so swapping it to a + # different policy changes the component spec (by type) and a resume under a + # different policy is rejected. + objective_spec = OptimizationSession._component_spec(ObjectiveSelectionPolicy()) + alt_spec = OptimizationSession._component_spec(_AltSelectionPolicy()) + assert objective_spec != alt_spec + assert objective_spec.type != alt_spec.type + + +def test_session_rejects_mismatched_evaluator_directory(tmp_path: Path): + with pytest.raises(ValueError, match="must match"): + OptimizationSession( + id="run", + session_dir=tmp_path / "expected", + optimizer=StubOptimizer(tmp_path / "different"), + ) + + +def test_session_binds_and_validates_optimizer_session_id(tmp_path: Path): + session_dir = tmp_path / "directory-name-can-differ" + optimizer = StubOptimizer(session_dir) + + OptimizationSession( + id="canonical-session-id", + session_dir=session_dir, + optimizer=optimizer, + ) + + assert optimizer.session_id == "canonical-session-id" + optimizer.session_id = "different" + with pytest.raises(ValueError, match="session ID"): + OptimizationSession( + id="canonical-session-id", + session_dir=session_dir, + optimizer=optimizer, + ) + + +def test_artifact_store_rejects_escaping_paths(tmp_path: Path): + store = ArtifactStore(tmp_path / "artifacts") + + path = store.write_json("agent/state.json", {"turn": 3}) + + assert path == tmp_path / "artifacts" / "agent" / "state.json" + assert store.read_json("agent/state.json") == {"turn": 3} + for unsafe in ("", "../escape", "/absolute", "a//b", "a\\b"): + with pytest.raises(ValueError): + store.write_text(unsafe, "no") + + +@pytest.mark.asyncio +async def test_event_sink_failure_does_not_break_runtime(): + captured = [] + + def broken(_event): + raise RuntimeError("sink failed") + + bus = EventBus([broken, captured.append]) + + event = await bus.emit(session_id="session", kind="test") + + assert captured == [event] diff --git a/vero/tests/test_v05_vero_agent.py b/vero/tests/test_v05_vero_agent.py new file mode 100644 index 00000000..fa32a191 --- /dev/null +++ b/vero/tests/test_v05_vero_agent.py @@ -0,0 +1,118 @@ +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace + +import pytest + +pytest.importorskip("agents") + +from agents import Agent, Runner + +from vero.agents import AgentContext, CodingAgent +from vero.agents.vero import VeroAgent, default_tool_sets +from vero.candidate import Candidate +from vero.optimization import CandidateProposal +from vero.tools.evaluation import EvaluationTools + + +class StubSandbox: + def host_path(self, path): + return Path(path) + + +class StubWorkspace: + def __init__(self, project_path: Path): + self.project_path = str(project_path) + self.sandbox = StubSandbox() + self.accesses = [] + + +class StubEvaluationGateway: + async def evaluate(self, **kwargs): + raise AssertionError("the fake run does not invoke tools") + + def budgets(self): + return {} + + +class FakeRunResult: + def __init__(self): + self.context_wrapper = SimpleNamespace( + usage={"requests": 1, "input_tokens": 8, "output_tokens": 2} + ) + + def stream_events(self): + async def _gen(): + return + yield # pragma: no cover - marks this as an async generator + + return _gen() + + def to_input_list(self): + return [ + {"role": "user", "content": "Try a tiled kernel"}, + {"role": "assistant", "content": "Done"}, + ] + + +def test_default_tools_use_canonical_evaluation_capability(): + names = {type(tool).__name__ for tool in default_tool_sets()} + + assert "EvaluationTools" in names + assert not any("Experiment" in name or "Dataset" in name for name in names) + + +def agent_context(tmp_path: Path, gateway: StubEvaluationGateway) -> AgentContext: + baseline = Candidate(id="baseline", version="baseline-version") + proposal = CandidateProposal( + id="proposal", + parent_id=baseline.id, + instruction="Optimize matrix multiplication", + ) + return AgentContext( + session_id="session-1", + workspace=StubWorkspace(tmp_path), + proposal=proposal, + parent=baseline, + evaluation=gateway, + ) + + +@pytest.mark.asyncio +async def test_vero_agent_implements_canonical_coding_agent_contract( + tmp_path: Path, + monkeypatch, +): + gateway = StubEvaluationGateway() + evaluation_tools = EvaluationTools() + agent = VeroAgent( + oai_agent=Agent(name="test-agent", model="gpt-4.1"), + tool_sets=[evaluation_tools], + ) + captured = {} + + def fake_run_streamed(agent, *, input, max_turns, run_config=None): + captured["agent"] = agent + captured["input"] = input + captured["max_turns"] = max_turns + captured["run_config"] = run_config + return FakeRunResult() + + monkeypatch.setattr(Runner, "run_streamed", staticmethod(fake_run_streamed)) + context = agent_context(tmp_path, gateway) + result = await agent.run( + context=context, + prompt="Try a tiled kernel", + max_turns=7, + ) + + assert isinstance(agent, CodingAgent) + assert captured["input"] == [{"role": "user", "content": "Try a tiled kernel"}] + assert captured["max_turns"] == 7 + assert captured["run_config"].workflow_name == "vero::session-1" + assert evaluation_tools.evaluation is gateway + assert agent._context is context + assert result.state[-1] == {"role": "assistant", "content": "Done"} + assert result.metadata["model"] == "gpt-4.1" + assert result.metadata["usage"]["orchestrator"]["input_tokens"] == 8 diff --git a/vero/tests/test_v05_wandb.py b/vero/tests/test_v05_wandb.py new file mode 100644 index 00000000..963a59d1 --- /dev/null +++ b/vero/tests/test_v05_wandb.py @@ -0,0 +1,404 @@ +from __future__ import annotations + +import json +from datetime import UTC, datetime, timedelta +from pathlib import Path + +from vero.candidate import Candidate +from vero.evaluation import ( + BackendProvenance, + CaseResult, + CaseStatus, + EvaluationPrincipal, + EvaluationRecord, + EvaluationReport, + EvaluationRequest, + EvaluationSet, + EvaluationStatus, + MetricSelector, + ObjectiveResult, + ObjectiveSpec, +) +from vero.runtime import RuntimeEvent, WandbEventSink +from vero.runtime.wandb import SidecarWandbSink + + +class FakeArtifact: + def __init__(self, name, type): + self.name = name + self.type = type + self.files = [] + + def add_file(self, path, name): + self.files.append((path, name)) + + +class FakeRun: + def __init__(self): + self.logged = [] + self.summary = {} + self.finished = [] + self.artifacts = [] + + def log(self, payload, *, step): + self.logged.append((payload, step)) + + def log_artifact(self, artifact): + self.artifacts.append(artifact) + + def finish(self, *, exit_code=0): + self.finished.append(exit_code) + + +class FakeWandb: + def __init__(self): + self.kwargs = None + self.run = FakeRun() + + def init(self, **kwargs): + self.kwargs = kwargs + return self.run + + def Artifact(self, *, name, type): + return FakeArtifact(name=name, type=type) + + +def test_wandb_sink_tracks_canonical_runtime_events(tmp_path: Path): + client = FakeWandb() + sink = WandbEventSink( + project="vero-tests", + session_id="session/with:unsafe-run-id-characters", + session_dir=tmp_path / "session", + mode="offline", + config={"vero/objective_metric": "latency_ms"}, + client=client, + ) + + assert client.kwargs["project"] == "vero-tests" + assert client.kwargs["id"].startswith("vero-") + assert client.kwargs["resume"] == "allow" + assert client.kwargs["mode"] == "offline" + assert client.kwargs["config"]["vero/session_id"].startswith("session/") + + sink( + RuntimeEvent( + session_id="session", + kind="evaluation_completed", + payload={ + "step": 2, + "candidate_id": "candidate", + "evaluation_id": "evaluation-1", + "metrics/latency_ms": 1.25, + "objective/value": 1.25, + "objective/feasible": True, + }, + ) + ) + assert client.run.logged == [ + ( + { + "candidate_id": "candidate", + "evaluation_id": "evaluation-1", + "metrics/latency_ms": 1.25, + "objective/value": 1.25, + "objective/feasible": True, + }, + 0, + ) + ] + assert json.loads( + (tmp_path / "session" / "artifacts" / "wandb" / "state.json").read_text() + ) == {"evaluation_ids": ["evaluation-1"], "next_step": 1} + + # Replayed history on resume is not sent twice or logged with a stale step. + sink( + RuntimeEvent( + session_id="session", + kind="evaluation_completed", + payload={"step": 0, "evaluation_id": "evaluation-1"}, + ) + ) + assert len(client.run.logged) == 1 + + sink( + RuntimeEvent( + session_id="session", + kind="session_completed", + payload={"status": "completed", "best_objective": 1.25}, + ) + ) + assert client.run.summary["best_objective"] == 1.25 + assert client.run.finished == [0] + + +def _evaluation_record( + *, status=EvaluationStatus.SUCCESS, score=0.75 +) -> EvaluationRecord: + created = datetime(2026, 1, 1, tzinfo=UTC) + return EvaluationRecord( + id="eval-1", + request=EvaluationRequest( + candidate=Candidate(id="cand", version="v1", created_at=created), + evaluation_set=EvaluationSet(name="benchmark", partition="validation"), + ), + report=EvaluationReport( + status=status, + metrics={"score": score, "score_stddev": 0.1, "error_rate": 0.0}, + cases=[ + CaseResult( + case_id="c1", status=CaseStatus.SUCCESS, metrics={"score": score} + ) + ], + ), + backend_id="primary", + backend=BackendProvenance(name="harbor", version="1", config_digest="0" * 64), + principal=EvaluationPrincipal.SYSTEM, + objective_spec=ObjectiveSpec( + selector=MetricSelector(metric="score"), direction="maximize" + ), + objective=ObjectiveResult(value=score, feasible=True), + created_at=created, + completed_at=created + timedelta(seconds=3), + ) + + +def test_sidecar_wandb_sink_logs_evaluation_records(tmp_path: Path): + client = FakeWandb() + sink = SidecarWandbSink( + project="vero", + session_id="s", + session_dir=tmp_path / "session", + client=client, + ) + + sink(_evaluation_record()) + + payload, step = client.run.logged[0] + assert step == 0 + assert payload["evaluation_id"] == "eval-1" + assert payload["status"] == "success" + assert payload["principal"] == "system" + assert payload["partition"] == "validation" + # Quality metrics are scoped by partition/principal so dev, validation, and + # the admin held-out re-score do not share an axis. + assert payload["validation/system/score"] == 0.75 + assert payload["validation/system/metric/score_stddev"] == 0.1 + assert payload["validation/system/cases/success"] == 1 + # num_cases records the evaluation's sample size (number of trials). + assert payload["validation/system/num_cases"] == 1 + # The bare, partition-blind keys are gone — nothing conflates partitions. + assert "score" not in payload + assert "cases/success" not in payload + + # Same record is not logged twice. + sink(_evaluation_record()) + assert len(client.run.logged) == 1 + + +def test_sidecar_wandb_run_id_is_unique_per_invocation_but_stable_on_restart( + tmp_path: Path, +): + # Distinct session volumes must get distinct W&B runs (not all resume one + # shared run), while a restart against the same volume resumes its run. + a = FakeWandb() + SidecarWandbSink(project="v", session_id="trial", session_dir=tmp_path / "a", client=a) + b = FakeWandb() + SidecarWandbSink(project="v", session_id="trial", session_dir=tmp_path / "b", client=b) + assert a.kwargs["id"] != b.kwargs["id"] + + # Same session dir (persisted state) -> same run id on restart. + a2 = FakeWandb() + SidecarWandbSink(project="v", session_id="trial", session_dir=tmp_path / "a", client=a2) + assert a2.kwargs["id"] == a.kwargs["id"] + + +def test_sidecar_wandb_run_dir_is_outside_the_archived_session_dir(tmp_path: Path): + # Regression: W&B's working directory accumulates symlinks (`latest-run`, + # debug logs, an absolute cache link). If it lived under session_dir it would + # be swept into create_harbor_session_archive, whose symlink refusal 500s the + # entire /session/export and loses the run's eval data. The run dir must be + # outside session_dir; only symlink-free resume state may live in artifacts. + client = FakeWandb() + session_dir = tmp_path / "session" + SidecarWandbSink( + project="vero", + session_id="s", + session_dir=session_dir, + client=client, + ) + + wandb_dir = Path(client.kwargs["dir"]).resolve() + resolved_session = session_dir.resolve() + assert not wandb_dir.is_relative_to(resolved_session) + + +def test_sidecar_wandb_sink_logs_full_trace_including_per_case_artifacts( + tmp_path: Path, +): + from vero.evaluation import EvaluationArtifact + + client = FakeWandb() + session_dir = tmp_path / "session" + sink = SidecarWandbSink( + project="vero", + session_id="s", + session_dir=session_dir, + log_traces=True, + client=client, + ) + + # Lay out an evaluation's on-disk artifacts: report-level harbor logs plus + # the full per-trial record attached to the case. + eval_dir = session_dir / "evaluations" / "eval-1" / "artifacts" + (eval_dir / "harbor").mkdir(parents=True) + (eval_dir / "harbor" / "stdout.log").write_text("out", encoding="utf-8") + trial = eval_dir / "harbor" / "jobs" / "trial-0" / "agent" + trial.mkdir(parents=True) + (trial.parent / "result.json").write_text("{}", encoding="utf-8") + (trial / "trajectory.json").write_text("[]", encoding="utf-8") + + record = _evaluation_record() + record = record.model_copy( + update={ + "report": record.report.model_copy( + update={ + "artifacts": [ + EvaluationArtifact(path="harbor/stdout.log"), + ], + "cases": [ + record.report.cases[0].model_copy( + update={ + "artifacts": [ + EvaluationArtifact( + path="harbor/jobs/trial-0/result.json" + ), + EvaluationArtifact( + path="harbor/jobs/trial-0/agent/trajectory.json" + ), + ] + } + ) + ], + } + ) + } + ) + + sink(record) + + assert len(client.run.artifacts) == 1 + artifact = client.run.artifacts[0] + assert artifact.type == "evaluation_trace" + uploaded = {name for _, name in artifact.files} + # Both the report-level harbor log AND the full per-case trial records land. + assert uploaded == { + "harbor/stdout.log", + "harbor/jobs/trial-0/result.json", + "harbor/jobs/trial-0/agent/trajectory.json", + } + + +def _gateway_state(tmp_path: Path, requests: int = 3) -> tuple[Path, Path]: + usage_path = tmp_path / "inference" / "usage.json" + requests_dir = tmp_path / "inference" / "requests" + requests_dir.mkdir(parents=True, exist_ok=True) + usage_path.write_text( + json.dumps( + { + "schema_version": 1, + "scopes": { + "producer": { + "requests": requests, + "upstream_errors": 0, + "input_tokens": 10, + "output_tokens": 5, + "total_tokens": 15, + "active_requests": 1, + "attributions": {}, + } + }, + } + ) + ) + return usage_path, requests_dir + + +def test_sidecar_wandb_mirrors_gateway_usage_and_request_logs(tmp_path: Path): + from vero.runtime.wandb import InferenceTelemetryPoller + + client = FakeWandb() + sink = SidecarWandbSink( + project="v", session_id="s", session_dir=tmp_path / "session", client=client + ) + usage_path, requests_dir = _gateway_state(tmp_path) + (requests_dir / "requests-00001.jsonl").write_text('{"a":1}\n') + (requests_dir / "requests-00002.jsonl").write_text('{"a":2}\n') + + poller = InferenceTelemetryPoller( + sink=sink, + usage_path=usage_path, + request_log_dir=requests_dir, + interval_seconds=5, + ) + poller.poll_once() + + payload, step = client.run.logged[0] + assert step == 0 + assert payload["inference/producer/requests"] == 3 + assert payload["inference/producer/total_tokens"] == 15 + assert payload["inference/producer/active_requests"] == 1 + # Only the rotated file ships; the highest-numbered one is still growing. + assert len(client.run.artifacts) == 1 + assert client.run.artifacts[0].type == "inference_request_log" + assert [name for _, name in client.run.artifacts[0].files] == [ + "requests-00001.jsonl" + ] + + # Unchanged gateway state: no duplicate series point, no new artifact. + poller.poll_once() + assert len(client.run.logged) == 1 + assert len(client.run.artifacts) == 1 + + # Usage moved -> a new point; eval records share the same step counter. + _gateway_state(tmp_path, requests=4) + poller.poll_once() + assert client.run.logged[1][1] == 1 + sink(_evaluation_record()) + assert client.run.logged[2][1] == 2 + + # The final flush ships the active file too. + poller.poll_once(final=True) + assert [name for _, name in client.run.artifacts[-1].files] == [ + "requests-00001.jsonl", + "requests-00002.jsonl", + ] + + # A restarted sink remembers what shipped and does not re-upload. + restarted = FakeWandb() + resumed = SidecarWandbSink( + project="v", session_id="s", session_dir=tmp_path / "session", client=restarted + ) + resumed.ship_request_logs(requests_dir, final=True) + assert restarted.run.artifacts == [] + + +def test_sidecar_wandb_telemetry_is_best_effort(tmp_path: Path): + from vero.runtime.wandb import InferenceTelemetryPoller + + client = FakeWandb() + sink = SidecarWandbSink( + project="v", session_id="s", session_dir=tmp_path / "session", client=client + ) + usage_path = tmp_path / "inference" / "usage.json" + usage_path.parent.mkdir(parents=True) + usage_path.write_text("{corrupt") + + poller = InferenceTelemetryPoller( + sink=sink, + usage_path=usage_path, + request_log_dir=tmp_path / "inference" / "missing", + interval_seconds=5, + ) + poller.poll_once() # must not raise + assert client.run.logged == [] + assert client.run.artifacts == [] diff --git a/vero/tests/test_web.py b/vero/tests/test_web.py new file mode 100644 index 00000000..38a40040 --- /dev/null +++ b/vero/tests/test_web.py @@ -0,0 +1,156 @@ +"""Tests for web tools (WebSearch, WebFetch).""" + +import json +import os + +import pytest + +# Check if Serper API key is available +SERPER_API_KEY_AVAILABLE = bool(os.getenv("SERPER_KEY_ID") or os.getenv("SERPER_API_KEY")) + +requires_serper_key = pytest.mark.skipif( + not SERPER_API_KEY_AVAILABLE, + reason="SERPER_KEY_ID or SERPER_API_KEY not set", +) + + +class TestWebSearch: + """Tests for WebSearch tool.""" + + def test_init_without_api_key(self, monkeypatch): + """Test that WebSearch raises ValueError if no API key is set.""" + from vero.tools.web import WebSearch + + # Remove both possible API key env vars + monkeypatch.delenv("SERPER_KEY_ID", raising=False) + monkeypatch.delenv("SERPER_API_KEY", raising=False) + + with pytest.raises(ValueError, match="SERPER_KEY_ID.*SERPER_API_KEY"): + WebSearch() + + @requires_serper_key + def test_init_with_api_key(self): + """Test that WebSearch initializes successfully with API key.""" + from vero.tools.web import WebSearch + + search = WebSearch() + assert search.max_results == 10 + assert search.max_retries == 3 + + @requires_serper_key + @pytest.mark.asyncio + async def test_search_returns_results(self): + """Test that WebSearch returns valid results.""" + from vero.tools.web import WebSearch + + search = WebSearch(fetch_full_content=False) # Faster - just snippets + result = await search("Python programming language") + + # Parse the JSON result + parsed = json.loads(result) + + # Should not have an error + assert "error" not in parsed or parsed.get("results") is not None + + # Should be a list of results + assert isinstance(parsed, list) + assert len(parsed) > 0 + + # Each result should have required fields + for item in parsed: + assert "title" in item + assert "url" in item + assert "content" in item + + @requires_serper_key + @pytest.mark.asyncio + async def test_search_with_full_content(self): + """Test that WebSearch can fetch full page content.""" + from vero.tools.web import WebSearch + + search = WebSearch(fetch_full_content=True, max_results=2) + result = await search("Python official website") + + parsed = json.loads(result) + assert isinstance(parsed, list) + + # At least one result should have substantial content + has_content = any(len(item.get("content", "")) > 100 for item in parsed) + assert has_content, "Expected at least one result with fetched content" + + @requires_serper_key + @pytest.mark.asyncio + async def test_search_chinese_query(self): + """Test that WebSearch handles Chinese queries.""" + from vero.tools.web import WebSearch + + search = WebSearch(fetch_full_content=False, max_results=3) + result = await search("Python 编程语言") + + parsed = json.loads(result) + assert isinstance(parsed, list) + # Should return some results (might be empty depending on search) + + @requires_serper_key + @pytest.mark.asyncio + async def test_search_no_results_query(self): + """Test that WebSearch handles queries with no results gracefully.""" + from vero.tools.web import WebSearch + + search = WebSearch(fetch_full_content=False) + # Very unlikely to have results + result = await search("xyzzy123456789abcdefghijklmnop") + + parsed = json.loads(result) + # Should either be empty list or have error message + assert isinstance(parsed, (list, dict)) + + +class TestWebFetch: + """Tests for WebFetch tool.""" + + @pytest.mark.asyncio + async def test_fetch_webpage(self): + """Test fetching a simple webpage.""" + from vero.tools.web import WebFetch + + fetch = WebFetch() + # Use a reliable, stable URL + content = await fetch("https://example.com") + + assert isinstance(content, str) + assert len(content) > 0 + assert "Example Domain" in content or "example" in content.lower() + + @pytest.mark.asyncio + async def test_fetch_with_offset(self): + """Test fetching with offset.""" + from vero.tools.web import WebFetch + + fetch = WebFetch() + full_content = await fetch("https://example.com") + offset_content = await fetch("https://example.com", offset=10) + + # The offset content should be the full content minus the first 10 chars + assert offset_content == full_content[10:] + assert len(offset_content) == len(full_content) - 10 + + @pytest.mark.asyncio + async def test_fetch_invalid_url(self): + """Test that fetching invalid URL raises error.""" + from vero.tools.web import WebFetch + + fetch = WebFetch() + + with pytest.raises(RuntimeError, match="Failed to fetch"): + await fetch("https://this-url-definitely-does-not-exist-12345.com") + + @pytest.mark.asyncio + async def test_fetch_with_max_chars(self): + """Test that max_chars truncates content.""" + from vero.tools.web import WebFetch + + fetch = WebFetch() + content = await fetch("https://example.com", max_chars=50) + + assert len(content) <= 50 diff --git a/vero/tests/test_workspace.py b/vero/tests/test_workspace.py new file mode 100644 index 00000000..5824f252 --- /dev/null +++ b/vero/tests/test_workspace.py @@ -0,0 +1,391 @@ +"""Tests for the Workspace abstraction (GitWorkspace).""" + +from __future__ import annotations + +import asyncio +import subprocess +from pathlib import Path + +import pytest +import pytest_asyncio + +from vero.sandbox import LocalSandbox +from vero.workspace.git import GitWorkspace + + +def _init_git_repo(path: Path) -> None: + """Initialize a git repo with an initial commit.""" + subprocess.run(["git", "init"], cwd=path, capture_output=True, check=True) + (path / "main.py").write_text("x = 1\n") + subprocess.run(["git", "add", "."], cwd=path, capture_output=True, check=True) + subprocess.run( + [ + "git", + "-c", + "user.name=test", + "-c", + "user.email=test@test", + "commit", + "-m", + "init", + ], + cwd=path, + capture_output=True, + check=True, + ) + + +@pytest_asyncio.fixture +async def workspace(tmp_path): + """Create a GitWorkspace backed by a LocalSandbox.""" + _init_git_repo(tmp_path) + sandbox = LocalSandbox(root=tmp_path) + return await GitWorkspace.from_path(sandbox, tmp_path) + + +class TestConstruction: + @pytest.mark.asyncio + async def test_from_path(self, workspace): + assert workspace.root is not None + assert workspace.project_path is not None + assert workspace.name is not None + + @pytest.mark.asyncio + async def test_sandbox_property(self, workspace): + assert workspace.sandbox is not None + assert isinstance(workspace.sandbox, LocalSandbox) + + @pytest.mark.asyncio + async def test_from_path_subdir(self, tmp_path): + """When project_path is a subdirectory, root is the repo root and project_path is the subdir.""" + _init_git_repo(tmp_path) + subdir = tmp_path / "agents" / "my-agent" + subdir.mkdir(parents=True) + (subdir / "main.py").write_text("agent code\n") + + sandbox = LocalSandbox(root=tmp_path) + ws = await GitWorkspace.from_path(sandbox, subdir) + + assert ws.root == str(tmp_path) + assert ws.project_path == str(subdir) + assert ws.root != ws.project_path + + @pytest.mark.asyncio + async def test_not_a_git_repo(self, tmp_path): + non_git = tmp_path / "not_a_repo" + non_git.mkdir() + sandbox = LocalSandbox(root=non_git) + with pytest.raises(RuntimeError, match="Not a git repository"): + await GitWorkspace.from_path(sandbox, non_git) + + +class TestVersioning: + @pytest.mark.asyncio + async def test_current_version(self, workspace): + version = await workspace.current_version() + assert len(version) == 40 # full SHA + + @pytest.mark.asyncio + async def test_save(self, workspace): + # Make a change + await workspace.sandbox.write_file( + str(Path(workspace.root) / "new.txt"), "new content\n" + ) + assert await workspace.is_dirty() + + new_version = await workspace.save("add new.txt") + assert len(new_version) == 40 + assert not await workspace.is_dirty() + + @pytest.mark.asyncio + async def test_save_no_changes(self, workspace): + v1 = await workspace.current_version() + v2 = await workspace.save("nothing changed") + assert v1 == v2 + + @pytest.mark.asyncio + async def test_restore(self, workspace): + v1 = await workspace.current_version() + + # Make and save a change + main_py = str(Path(workspace.root) / "main.py") + await workspace.sandbox.write_file(main_py, "x = 999\n") + await workspace.save("change main.py") + v2 = await workspace.current_version() + assert v1 != v2 + + # Restore to v1 + await workspace.restore(v1) + + # main.py should be back to original + content = await workspace.sandbox.read_file(main_py) + assert content == "x = 1\n" + + +class TestSubdirProject: + """Tests for when project_path is a subdirectory of the repo root.""" + + @pytest_asyncio.fixture + async def subdir_workspace(self, tmp_path): + _init_git_repo(tmp_path) + subdir = tmp_path / "agents" / "my-agent" + subdir.mkdir(parents=True) + (subdir / "agent.py").write_text("v1\n") + # Commit the subdir + subprocess.run( + ["git", "add", "."], cwd=tmp_path, capture_output=True, check=True + ) + subprocess.run( + [ + "git", + "-c", + "user.name=test", + "-c", + "user.email=test@test", + "commit", + "-m", + "add agent", + ], + cwd=tmp_path, + capture_output=True, + check=True, + ) + sandbox = LocalSandbox(root=tmp_path) + return await GitWorkspace.from_path(sandbox, subdir) + + @pytest.mark.asyncio + async def test_save_scoped_to_project(self, subdir_workspace): + """save() only stages files within project_path, not the whole repo.""" + ws = subdir_workspace + repo_root = ws.root + + # Create a file outside the project subdir + outside = str(Path(repo_root) / "outside.txt") + await ws.sandbox.write_file(outside, "outside\n") + + # Create a file inside the project subdir + inside = str(Path(ws.project_path) / "new.py") + await ws.sandbox.write_file(inside, "new\n") + + v1 = await ws.current_version() + await ws.save("save inside only") + v2 = await ws.current_version() + + # A commit was made (inside file was staged) + assert v1 != v2 + + # The outside file should still be untracked + result = await ws.sandbox.run( + ["git", "status", "--porcelain", "outside.txt"], cwd=repo_root + ) + assert "??" in result.stdout # untracked + + @pytest.mark.asyncio + async def test_is_dirty_scoped_to_project(self, subdir_workspace): + """is_dirty() only reports changes within the project subdirectory.""" + ws = subdir_workspace + assert not await ws.is_dirty() + + # Modify file OUTSIDE project — should NOT be dirty + await ws.sandbox.write_file(str(Path(ws.root) / "outside.txt"), "unrelated\n") + assert not await ws.is_dirty(), ( + "Changes outside project_path should not make workspace dirty" + ) + + # Modify file INSIDE project — should be dirty + await ws.sandbox.write_file(str(Path(ws.project_path) / "agent.py"), "v2\n") + assert await ws.is_dirty() + + +class TestDirtyState: + @pytest.mark.asyncio + async def test_clean_initially(self, workspace): + assert not await workspace.is_dirty() + + @pytest.mark.asyncio + async def test_dirty_after_edit(self, workspace): + await workspace.sandbox.write_file( + str(Path(workspace.root) / "main.py"), "x = 2\n" + ) + assert await workspace.is_dirty() + + @pytest.mark.asyncio + async def test_clean_after_save(self, workspace): + await workspace.sandbox.write_file( + str(Path(workspace.root) / "main.py"), "x = 2\n" + ) + await workspace.save("update") + assert not await workspace.is_dirty() + + +class TestHistory: + @pytest.mark.asyncio + async def test_diff(self, workspace): + v1 = await workspace.current_version() + await workspace.sandbox.write_file( + str(Path(workspace.root) / "main.py"), "x = 2\n" + ) + await workspace.save("update") + v2 = await workspace.current_version() + + diff = await workspace.diff(v1, v2) + assert "-x = 1" in diff + assert "+x = 2" in diff + + @pytest.mark.asyncio + async def test_log(self, workspace): + log = await workspace.log() + assert "init" in log + + @pytest.mark.asyncio + async def test_is_ancestor(self, workspace): + v1 = await workspace.current_version() + await workspace.sandbox.write_file( + str(Path(workspace.root) / "main.py"), "x = 2\n" + ) + await workspace.save("update") + v2 = await workspace.current_version() + + assert await workspace.is_ancestor(v1, v2) + assert not await workspace.is_ancestor(v2, v1) + + +class TestCopies: + @pytest.mark.asyncio + async def test_temp_copy(self, workspace): + v1 = await workspace.current_version() + + async with workspace.temp_copy(from_version=v1) as copy_ws: + assert copy_ws.root != workspace.root + assert copy_ws.root == str(Path(copy_ws.root).resolve()) + # The copy is a separate git worktree + assert Path(copy_ws.root).exists() + # main.py should exist in the copy + assert (Path(copy_ws.root) / "main.py").exists() + assert (Path(copy_ws.root) / "main.py").read_text() == "x = 1\n" + + # After context exit, the temp worktree is cleaned up + assert not Path(copy_ws.root).exists() + + # Original is unchanged + content = await workspace.sandbox.read_file( + str(Path(workspace.root) / "main.py") + ) + assert content == "x = 1\n" + + @pytest.mark.asyncio + async def test_copies_preserve_subdirectory_project(self, tmp_path): + _init_git_repo(tmp_path) + subdir = tmp_path / "packages" / "target" + subdir.mkdir(parents=True) + (subdir / "program.py").write_text("value = 1\n") + subprocess.run( + ["git", "add", "."], cwd=tmp_path, capture_output=True, check=True + ) + subprocess.run( + [ + "git", + "-c", + "user.name=test", + "-c", + "user.email=test@test", + "commit", + "-m", + "add target", + ], + cwd=tmp_path, + capture_output=True, + check=True, + ) + workspace = await GitWorkspace.from_path(LocalSandbox(root=tmp_path), subdir) + + async with workspace.temp_copy() as copy_workspace: + relative = Path(copy_workspace.project_path).relative_to( + copy_workspace.root + ) + assert relative == Path("packages/target") + assert Path(copy_workspace.project_path, "program.py").is_file() + + persistent = await workspace.copy(name="subproject-copy") + try: + relative = Path(persistent.project_path).relative_to(persistent.root) + assert relative == Path("packages/target") + finally: + await persistent.destroy() + + @pytest.mark.asyncio + async def test_persistent_copy_does_not_leak_a_branch( + self, + workspace, + ): + branches_before = await workspace._git( + "for-each-ref", "--format=%(refname)", "refs/heads" + ) + copied = await workspace.copy(name="candidate-copy") + await copied.sandbox.write_file( + str(Path(copied.root) / "candidate.txt"), + "candidate\n", + ) + await copied.save("candidate") + await copied.destroy() + + branches_after = await workspace._git( + "for-each-ref", "--format=%(refname)", "refs/heads" + ) + assert branches_after == branches_before + assert not Path(copied.root).exists() + + @pytest.mark.asyncio + async def test_copy_rolls_back_if_cancelled_after_git_creates_worktree( + self, + workspace, + monkeypatch, + ): + original_run = workspace.sandbox.run + injected = False + + async def cancel_after_add(command, **kwargs): + nonlocal injected + result = await original_run(command, **kwargs) + if ( + "worktree" in command + and command[command.index("worktree") + 1] == "add" + and not injected + ): + injected = True + raise asyncio.CancelledError + return result + + monkeypatch.setattr(workspace.sandbox, "run", cancel_after_add) + + with pytest.raises(asyncio.CancelledError): + await workspace.copy(name="cancelled-copy") + + worktrees = await workspace._git("worktree", "list", "--porcelain") + assert worktrees.count("worktree ") == 1 + assert not Path(workspace.root).parent.joinpath("cancelled-copy").exists() + + +class TestAtVersion: + @pytest.mark.asyncio + async def test_at_version(self, workspace): + v1 = await workspace.current_version() + + # Make a change + await workspace.sandbox.write_file( + str(Path(workspace.root) / "main.py"), "x = 2\n" + ) + await workspace.save("update") + + # Temporarily switch back to v1 + async with workspace.at(v1): + content = await workspace.sandbox.read_file( + str(Path(workspace.root) / "main.py") + ) + assert content == "x = 1\n" + + # After exit, back to latest + content = await workspace.sandbox.read_file( + str(Path(workspace.root) / "main.py") + ) + assert content == "x = 2\n" diff --git a/vero/uv.lock b/vero/uv.lock new file mode 100644 index 00000000..6c439360 --- /dev/null +++ b/vero/uv.lock @@ -0,0 +1,2877 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.12' and sys_platform == 'win32'", + "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.12' and sys_platform == 'emscripten'", + "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] + +[[package]] +name = "aiohappyeyeballs" +version = "2.6.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/26/30/f84a107a9c4331c14b2b586036f40965c128aa4fee4dda5d3d51cb14ad54/aiohappyeyeballs-2.6.1.tar.gz", hash = "sha256:c3f9d0113123803ccadfdf3f0faa505bc78e6a72d1cc4806cbd719826e943558", size = 22760, upload-time = "2025-03-12T01:42:48.764Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0f/15/5bf3b99495fb160b63f95972b81750f18f7f4e02ad051373b669d17d44f2/aiohappyeyeballs-2.6.1-py3-none-any.whl", hash = "sha256:f349ba8f4b75cb25c99c5c2d84e997e485204d2902a9597802b0371f09331fb8", size = 15265, upload-time = "2025-03-12T01:42:47.083Z" }, +] + +[[package]] +name = "aiohttp" +version = "3.13.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohappyeyeballs" }, + { name = "aiosignal" }, + { name = "attrs" }, + { name = "frozenlist" }, + { name = "multidict" }, + { name = "propcache" }, + { name = "yarl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/77/9a/152096d4808df8e4268befa55fba462f440f14beab85e8ad9bf990516918/aiohttp-3.13.5.tar.gz", hash = "sha256:9d98cc980ecc96be6eb4c1994ce35d28d8b1f5e5208a23b421187d1209dbb7d1", size = 7858271, upload-time = "2026-03-31T22:01:03.343Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d6/f5/a20c4ac64aeaef1679e25c9983573618ff765d7aa829fa2b84ae7573169e/aiohttp-3.13.5-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:7ab7229b6f9b5c1ba4910d6c41a9eb11f543eadb3f384df1b4c293f4e73d44d6", size = 757513, upload-time = "2026-03-31T21:57:02.146Z" }, + { url = "https://files.pythonhosted.org/packages/75/0a/39fa6c6b179b53fcb3e4b3d2b6d6cad0180854eda17060c7218540102bef/aiohttp-3.13.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:8f14c50708bb156b3a3ca7230b3d820199d56a48e3af76fa21c2d6087190fe3d", size = 506748, upload-time = "2026-03-31T21:57:04.275Z" }, + { url = "https://files.pythonhosted.org/packages/87/ec/e38ce072e724fd7add6243613f8d1810da084f54175353d25ccf9f9c7e5a/aiohttp-3.13.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:e7d2f8616f0ff60bd332022279011776c3ac0faa0f1b463f7bb12326fbc97a1c", size = 501673, upload-time = "2026-03-31T21:57:06.208Z" }, + { url = "https://files.pythonhosted.org/packages/ba/ba/3bc7525d7e2beaa11b309a70d48b0d3cfc3c2089ec6a7d0820d59c657053/aiohttp-3.13.5-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a2567b72e1ffc3ab25510db43f355b29eeada56c0a622e58dcdb19530eb0a3cb", size = 1763757, upload-time = "2026-03-31T21:57:07.882Z" }, + { url = "https://files.pythonhosted.org/packages/5e/ab/e87744cf18f1bd78263aba24924d4953b41086bd3a31d22452378e9028a0/aiohttp-3.13.5-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:fb0540c854ac9c0c5ad495908fdfd3e332d553ec731698c0e29b1877ba0d2ec6", size = 1720152, upload-time = "2026-03-31T21:57:09.946Z" }, + { url = "https://files.pythonhosted.org/packages/6b/f3/ed17a6f2d742af17b50bae2d152315ed1b164b07a5fd5cc1754d99e4dfa5/aiohttp-3.13.5-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c9883051c6972f58bfc4ebb2116345ee2aa151178e99c3f2b2bbe2af712abd13", size = 1818010, upload-time = "2026-03-31T21:57:12.157Z" }, + { url = "https://files.pythonhosted.org/packages/53/06/ecbc63dc937192e2a5cb46df4d3edb21deb8225535818802f210a6ea5816/aiohttp-3.13.5-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2294172ce08a82fb7c7273485895de1fa1186cc8294cfeb6aef4af42ad261174", size = 1907251, upload-time = "2026-03-31T21:57:14.023Z" }, + { url = "https://files.pythonhosted.org/packages/7e/a5/0521aa32c1ddf3aa1e71dcc466be0b7db2771907a13f18cddaa45967d97b/aiohttp-3.13.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3a807cabd5115fb55af198b98178997a5e0e57dead43eb74a93d9c07d6d4a7dc", size = 1759969, upload-time = "2026-03-31T21:57:16.146Z" }, + { url = "https://files.pythonhosted.org/packages/f6/78/a38f8c9105199dd3b9706745865a8a59d0041b6be0ca0cc4b2ccf1bab374/aiohttp-3.13.5-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:aa6d0d932e0f39c02b80744273cd5c388a2d9bc07760a03164f229c8e02662f6", size = 1616871, upload-time = "2026-03-31T21:57:17.856Z" }, + { url = "https://files.pythonhosted.org/packages/6f/41/27392a61ead8ab38072105c71aa44ff891e71653fe53d576a7067da2b4e8/aiohttp-3.13.5-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:60869c7ac4aaabe7110f26499f3e6e5696eae98144735b12a9c3d9eae2b51a49", size = 1739844, upload-time = "2026-03-31T21:57:19.679Z" }, + { url = "https://files.pythonhosted.org/packages/6e/55/5564e7ae26d94f3214250009a0b1c65a0c6af4bf88924ccb6fdab901de28/aiohttp-3.13.5-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:26d2f8546f1dfa75efa50c3488215a903c0168d253b75fba4210f57ab77a0fb8", size = 1731969, upload-time = "2026-03-31T21:57:22.006Z" }, + { url = "https://files.pythonhosted.org/packages/6d/c5/705a3929149865fc941bcbdd1047b238e4a72bcb215a9b16b9d7a2e8d992/aiohttp-3.13.5-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:f1162a1492032c82f14271e831c8f4b49f2b6078f4f5fc74de2c912fa225d51d", size = 1795193, upload-time = "2026-03-31T21:57:24.256Z" }, + { url = "https://files.pythonhosted.org/packages/a6/19/edabed62f718d02cff7231ca0db4ef1c72504235bc467f7b67adb1679f48/aiohttp-3.13.5-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:8b14eb3262fad0dc2f89c1a43b13727e709504972186ff6a99a3ecaa77102b6c", size = 1606477, upload-time = "2026-03-31T21:57:26.364Z" }, + { url = "https://files.pythonhosted.org/packages/de/fc/76f80ef008675637d88d0b21584596dc27410a990b0918cb1e5776545b5b/aiohttp-3.13.5-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:ca9ac61ac6db4eb6c2a0cd1d0f7e1357647b638ccc92f7e9d8d133e71ed3c6ac", size = 1813198, upload-time = "2026-03-31T21:57:28.316Z" }, + { url = "https://files.pythonhosted.org/packages/e5/67/5b3ac26b80adb20ea541c487f73730dc8fa107d632c998f25bbbab98fcda/aiohttp-3.13.5-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:7996023b2ed59489ae4762256c8516df9820f751cf2c5da8ed2fb20ee50abab3", size = 1752321, upload-time = "2026-03-31T21:57:30.549Z" }, + { url = "https://files.pythonhosted.org/packages/88/06/e4a2e49255ea23fa4feeb5ab092d90240d927c15e47b5b5c48dff5a9ce29/aiohttp-3.13.5-cp311-cp311-win32.whl", hash = "sha256:77dfa48c9f8013271011e51c00f8ada19851f013cde2c48fca1ba5e0caf5bb06", size = 439069, upload-time = "2026-03-31T21:57:32.388Z" }, + { url = "https://files.pythonhosted.org/packages/c0/43/8c7163a596dab4f8be12c190cf467a1e07e4734cf90eebb39f7f5d53fc6a/aiohttp-3.13.5-cp311-cp311-win_amd64.whl", hash = "sha256:d3a4834f221061624b8887090637db9ad4f61752001eae37d56c52fddade2dc8", size = 462859, upload-time = "2026-03-31T21:57:34.455Z" }, + { url = "https://files.pythonhosted.org/packages/be/6f/353954c29e7dcce7cf00280a02c75f30e133c00793c7a2ed3776d7b2f426/aiohttp-3.13.5-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:023ecba036ddd840b0b19bf195bfae970083fd7024ce1ac22e9bba90464620e9", size = 748876, upload-time = "2026-03-31T21:57:36.319Z" }, + { url = "https://files.pythonhosted.org/packages/f5/1b/428a7c64687b3b2e9cd293186695affc0e1e54a445d0361743b231f11066/aiohttp-3.13.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:15c933ad7920b7d9a20de151efcd05a6e38302cbf0e10c9b2acb9a42210a2416", size = 499557, upload-time = "2026-03-31T21:57:38.236Z" }, + { url = "https://files.pythonhosted.org/packages/29/47/7be41556bfbb6917069d6a6634bb7dd5e163ba445b783a90d40f5ac7e3a7/aiohttp-3.13.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ab2899f9fa2f9f741896ebb6fa07c4c883bfa5c7f2ddd8cf2aafa86fa981b2d2", size = 500258, upload-time = "2026-03-31T21:57:39.923Z" }, + { url = "https://files.pythonhosted.org/packages/67/84/c9ecc5828cb0b3695856c07c0a6817a99d51e2473400f705275a2b3d9239/aiohttp-3.13.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a60eaa2d440cd4707696b52e40ed3e2b0f73f65be07fd0ef23b6b539c9c0b0b4", size = 1749199, upload-time = "2026-03-31T21:57:41.938Z" }, + { url = "https://files.pythonhosted.org/packages/f0/d3/3c6d610e66b495657622edb6ae7c7fd31b2e9086b4ec50b47897ad6042a9/aiohttp-3.13.5-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:55b3bdd3292283295774ab585160c4004f4f2f203946997f49aac032c84649e9", size = 1721013, upload-time = "2026-03-31T21:57:43.904Z" }, + { url = "https://files.pythonhosted.org/packages/49/a0/24409c12217456df0bae7babe3b014e460b0b38a8e60753d6cb339f6556d/aiohttp-3.13.5-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c2b2355dc094e5f7d45a7bb262fe7207aa0460b37a0d87027dcf21b5d890e7d5", size = 1781501, upload-time = "2026-03-31T21:57:46.285Z" }, + { url = "https://files.pythonhosted.org/packages/98/9d/b65ec649adc5bccc008b0957a9a9c691070aeac4e41cea18559fef49958b/aiohttp-3.13.5-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b38765950832f7d728297689ad78f5f2cf79ff82487131c4d26fe6ceecdc5f8e", size = 1878981, upload-time = "2026-03-31T21:57:48.734Z" }, + { url = "https://files.pythonhosted.org/packages/57/d8/8d44036d7eb7b6a8ec4c5494ea0c8c8b94fbc0ed3991c1a7adf230df03bf/aiohttp-3.13.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b18f31b80d5a33661e08c89e202edabf1986e9b49c42b4504371daeaa11b47c1", size = 1767934, upload-time = "2026-03-31T21:57:51.171Z" }, + { url = "https://files.pythonhosted.org/packages/31/04/d3f8211f273356f158e3464e9e45484d3fb8c4ce5eb2f6fe9405c3273983/aiohttp-3.13.5-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:33add2463dde55c4f2d9635c6ab33ce154e5ecf322bd26d09af95c5f81cfa286", size = 1566671, upload-time = "2026-03-31T21:57:53.326Z" }, + { url = "https://files.pythonhosted.org/packages/41/db/073e4ebe00b78e2dfcacff734291651729a62953b48933d765dc513bf798/aiohttp-3.13.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:327cc432fdf1356fb4fbc6fe833ad4e9f6aacb71a8acaa5f1855e4b25910e4a9", size = 1705219, upload-time = "2026-03-31T21:57:55.385Z" }, + { url = "https://files.pythonhosted.org/packages/48/45/7dfba71a2f9fd97b15c95c06819de7eb38113d2cdb6319669195a7d64270/aiohttp-3.13.5-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:7c35b0bf0b48a70b4cb4fc5d7bed9b932532728e124874355de1a0af8ec4bc88", size = 1743049, upload-time = "2026-03-31T21:57:57.341Z" }, + { url = "https://files.pythonhosted.org/packages/18/71/901db0061e0f717d226386a7f471bb59b19566f2cae5f0d93874b017271f/aiohttp-3.13.5-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:df23d57718f24badef8656c49743e11a89fd6f5358fa8a7b96e728fda2abf7d3", size = 1749557, upload-time = "2026-03-31T21:57:59.626Z" }, + { url = "https://files.pythonhosted.org/packages/08/d5/41eebd16066e59cd43728fe74bce953d7402f2b4ddfdfef2c0e9f17ca274/aiohttp-3.13.5-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:02e048037a6501a5ec1f6fc9736135aec6eb8a004ce48838cb951c515f32c80b", size = 1558931, upload-time = "2026-03-31T21:58:01.972Z" }, + { url = "https://files.pythonhosted.org/packages/30/e6/4a799798bf05740e66c3a1161079bda7a3dd8e22ca392481d7a7f9af82a6/aiohttp-3.13.5-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:31cebae8b26f8a615d2b546fee45d5ffb76852ae6450e2a03f42c9102260d6fe", size = 1774125, upload-time = "2026-03-31T21:58:04.007Z" }, + { url = "https://files.pythonhosted.org/packages/84/63/7749337c90f92bc2cb18f9560d67aa6258c7060d1397d21529b8004fcf6f/aiohttp-3.13.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:888e78eb5ca55a615d285c3c09a7a91b42e9dd6fc699b166ebd5dee87c9ccf14", size = 1732427, upload-time = "2026-03-31T21:58:06.337Z" }, + { url = "https://files.pythonhosted.org/packages/98/de/cf2f44ff98d307e72fb97d5f5bbae3bfcb442f0ea9790c0bf5c5c2331404/aiohttp-3.13.5-cp312-cp312-win32.whl", hash = "sha256:8bd3ec6376e68a41f9f95f5ed170e2fcf22d4eb27a1f8cb361d0508f6e0557f3", size = 433534, upload-time = "2026-03-31T21:58:08.712Z" }, + { url = "https://files.pythonhosted.org/packages/aa/ca/eadf6f9c8fa5e31d40993e3db153fb5ed0b11008ad5d9de98a95045bed84/aiohttp-3.13.5-cp312-cp312-win_amd64.whl", hash = "sha256:110e448e02c729bcebb18c60b9214a87ba33bac4a9fa5e9a5f139938b56c6cb1", size = 460446, upload-time = "2026-03-31T21:58:10.945Z" }, + { url = "https://files.pythonhosted.org/packages/78/e9/d76bf503005709e390122d34e15256b88f7008e246c4bdbe915cd4f1adce/aiohttp-3.13.5-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:a5029cc80718bbd545123cd8fe5d15025eccaaaace5d0eeec6bd556ad6163d61", size = 742930, upload-time = "2026-03-31T21:58:13.155Z" }, + { url = "https://files.pythonhosted.org/packages/57/00/4b7b70223deaebd9bb85984d01a764b0d7bd6526fcdc73cca83bcbe7243e/aiohttp-3.13.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4bb6bf5811620003614076bdc807ef3b5e38244f9d25ca5fe888eaccea2a9832", size = 496927, upload-time = "2026-03-31T21:58:15.073Z" }, + { url = "https://files.pythonhosted.org/packages/9c/f5/0fb20fb49f8efdcdce6cd8127604ad2c503e754a8f139f5e02b01626523f/aiohttp-3.13.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a84792f8631bf5a94e52d9cc881c0b824ab42717165a5579c760b830d9392ac9", size = 497141, upload-time = "2026-03-31T21:58:17.009Z" }, + { url = "https://files.pythonhosted.org/packages/3b/86/b7c870053e36a94e8951b803cb5b909bfbc9b90ca941527f5fcafbf6b0fa/aiohttp-3.13.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:57653eac22c6a4c13eb22ecf4d673d64a12f266e72785ab1c8b8e5940d0e8090", size = 1732476, upload-time = "2026-03-31T21:58:18.925Z" }, + { url = "https://files.pythonhosted.org/packages/b5/e5/4e161f84f98d80c03a238671b4136e6530453d65262867d989bbe78244d0/aiohttp-3.13.5-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e5e5f7debc7a57af53fdf5c5009f9391d9f4c12867049d509bf7bb164a6e295b", size = 1706507, upload-time = "2026-03-31T21:58:21.094Z" }, + { url = "https://files.pythonhosted.org/packages/d4/56/ea11a9f01518bd5a2a2fcee869d248c4b8a0cfa0bb13401574fa31adf4d4/aiohttp-3.13.5-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c719f65bebcdf6716f10e9eff80d27567f7892d8988c06de12bbbd39307c6e3a", size = 1773465, upload-time = "2026-03-31T21:58:23.159Z" }, + { url = "https://files.pythonhosted.org/packages/eb/40/333ca27fb74b0383f17c90570c748f7582501507307350a79d9f9f3c6eb1/aiohttp-3.13.5-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d97f93fdae594d886c5a866636397e2bcab146fd7a132fd6bb9ce182224452f8", size = 1873523, upload-time = "2026-03-31T21:58:25.59Z" }, + { url = "https://files.pythonhosted.org/packages/f0/d2/e2f77eef1acb7111405433c707dc735e63f67a56e176e72e9e7a2cd3f493/aiohttp-3.13.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3df334e39d4c2f899a914f1dba283c1aadc311790733f705182998c6f7cae665", size = 1754113, upload-time = "2026-03-31T21:58:27.624Z" }, + { url = "https://files.pythonhosted.org/packages/fb/56/3f653d7f53c89669301ec9e42c95233e2a0c0a6dd051269e6e678db4fdb0/aiohttp-3.13.5-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fe6970addfea9e5e081401bcbadf865d2b6da045472f58af08427e108d618540", size = 1562351, upload-time = "2026-03-31T21:58:29.918Z" }, + { url = "https://files.pythonhosted.org/packages/ec/a6/9b3e91eb8ae791cce4ee736da02211c85c6f835f1bdfac0594a8a3b7018c/aiohttp-3.13.5-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7becdf835feff2f4f335d7477f121af787e3504b48b449ff737afb35869ba7bb", size = 1693205, upload-time = "2026-03-31T21:58:32.214Z" }, + { url = "https://files.pythonhosted.org/packages/98/fc/bfb437a99a2fcebd6b6eaec609571954de2ed424f01c352f4b5504371dd3/aiohttp-3.13.5-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:676e5651705ad5d8a70aeb8eb6936c436d8ebbd56e63436cb7dd9bb36d2a9a46", size = 1730618, upload-time = "2026-03-31T21:58:34.728Z" }, + { url = "https://files.pythonhosted.org/packages/e4/b6/c8534862126191a034f68153194c389addc285a0f1347d85096d349bbc15/aiohttp-3.13.5-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:9b16c653d38eb1a611cc898c41e76859ca27f119d25b53c12875fd0474ae31a8", size = 1745185, upload-time = "2026-03-31T21:58:36.909Z" }, + { url = "https://files.pythonhosted.org/packages/0b/93/4ca8ee2ef5236e2707e0fd5fecb10ce214aee1ff4ab307af9c558bda3b37/aiohttp-3.13.5-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:999802d5fa0389f58decd24b537c54aa63c01c3219ce17d1214cbda3c2b22d2d", size = 1557311, upload-time = "2026-03-31T21:58:39.38Z" }, + { url = "https://files.pythonhosted.org/packages/57/ae/76177b15f18c5f5d094f19901d284025db28eccc5ae374d1d254181d33f4/aiohttp-3.13.5-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:ec707059ee75732b1ba130ed5f9580fe10ff75180c812bc267ded039db5128c6", size = 1773147, upload-time = "2026-03-31T21:58:41.476Z" }, + { url = "https://files.pythonhosted.org/packages/01/a4/62f05a0a98d88af59d93b7fcac564e5f18f513cb7471696ac286db970d6a/aiohttp-3.13.5-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2d6d44a5b48132053c2f6cd5c8cb14bc67e99a63594e336b0f2af81e94d5530c", size = 1730356, upload-time = "2026-03-31T21:58:44.049Z" }, + { url = "https://files.pythonhosted.org/packages/e4/85/fc8601f59dfa8c9523808281f2da571f8b4699685f9809a228adcc90838d/aiohttp-3.13.5-cp313-cp313-win32.whl", hash = "sha256:329f292ed14d38a6c4c435e465f48bebb47479fd676a0411936cc371643225cc", size = 432637, upload-time = "2026-03-31T21:58:46.167Z" }, + { url = "https://files.pythonhosted.org/packages/c0/1b/ac685a8882896acf0f6b31d689e3792199cfe7aba37969fa91da63a7fa27/aiohttp-3.13.5-cp313-cp313-win_amd64.whl", hash = "sha256:69f571de7500e0557801c0b51f4780482c0ec5fe2ac851af5a92cfce1af1cb83", size = 458896, upload-time = "2026-03-31T21:58:48.119Z" }, + { url = "https://files.pythonhosted.org/packages/5d/ce/46572759afc859e867a5bc8ec3487315869013f59281ce61764f76d879de/aiohttp-3.13.5-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:eb4639f32fd4a9904ab8fb45bf3383ba71137f3d9d4ba25b3b3f3109977c5b8c", size = 745721, upload-time = "2026-03-31T21:58:50.229Z" }, + { url = "https://files.pythonhosted.org/packages/13/fe/8a2efd7626dbe6049b2ef8ace18ffda8a4dfcbe1bcff3ac30c0c7575c20b/aiohttp-3.13.5-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:7e5dc4311bd5ac493886c63cbf76ab579dbe4641268e7c74e48e774c74b6f2be", size = 497663, upload-time = "2026-03-31T21:58:52.232Z" }, + { url = "https://files.pythonhosted.org/packages/9b/91/cc8cc78a111826c54743d88651e1687008133c37e5ee615fee9b57990fac/aiohttp-3.13.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:756c3c304d394977519824449600adaf2be0ccee76d206ee339c5e76b70ded25", size = 499094, upload-time = "2026-03-31T21:58:54.566Z" }, + { url = "https://files.pythonhosted.org/packages/0a/33/a8362cb15cf16a3af7e86ed11962d5cd7d59b449202dc576cdc731310bde/aiohttp-3.13.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ecc26751323224cf8186efcf7fbcbc30f4e1d8c7970659daf25ad995e4032a56", size = 1726701, upload-time = "2026-03-31T21:58:56.864Z" }, + { url = "https://files.pythonhosted.org/packages/45/0c/c091ac5c3a17114bd76cbf85d674650969ddf93387876cf67f754204bd77/aiohttp-3.13.5-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:10a75acfcf794edf9d8db50e5a7ec5fc818b2a8d3f591ce93bc7b1210df016d2", size = 1683360, upload-time = "2026-03-31T21:58:59.072Z" }, + { url = "https://files.pythonhosted.org/packages/23/73/bcee1c2b79bc275e964d1446c55c54441a461938e70267c86afaae6fba27/aiohttp-3.13.5-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:0f7a18f258d124cd678c5fe072fe4432a4d5232b0657fca7c1847f599233c83a", size = 1773023, upload-time = "2026-03-31T21:59:01.776Z" }, + { url = "https://files.pythonhosted.org/packages/c7/ef/720e639df03004fee2d869f771799d8c23046dec47d5b81e396c7cda583a/aiohttp-3.13.5-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:df6104c009713d3a89621096f3e3e88cc323fd269dbd7c20afe18535094320be", size = 1853795, upload-time = "2026-03-31T21:59:04.568Z" }, + { url = "https://files.pythonhosted.org/packages/bd/c9/989f4034fb46841208de7aeeac2c6d8300745ab4f28c42f629ba77c2d916/aiohttp-3.13.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:241a94f7de7c0c3b616627aaad530fe2cb620084a8b144d3be7b6ecfe95bae3b", size = 1730405, upload-time = "2026-03-31T21:59:07.221Z" }, + { url = "https://files.pythonhosted.org/packages/ce/75/ee1fd286ca7dc599d824b5651dad7b3be7ff8d9a7e7b3fe9820d9180f7db/aiohttp-3.13.5-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c974fb66180e58709b6fc402846f13791240d180b74de81d23913abe48e96d94", size = 1558082, upload-time = "2026-03-31T21:59:09.484Z" }, + { url = "https://files.pythonhosted.org/packages/c3/20/1e9e6650dfc436340116b7aa89ff8cb2bbdf0abc11dfaceaad8f74273a10/aiohttp-3.13.5-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:6e27ea05d184afac78aabbac667450c75e54e35f62238d44463131bd3f96753d", size = 1692346, upload-time = "2026-03-31T21:59:12.068Z" }, + { url = "https://files.pythonhosted.org/packages/d8/40/8ebc6658d48ea630ac7903912fe0dd4e262f0e16825aa4c833c56c9f1f56/aiohttp-3.13.5-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:a79a6d399cef33a11b6f004c67bb07741d91f2be01b8d712d52c75711b1e07c7", size = 1698891, upload-time = "2026-03-31T21:59:14.552Z" }, + { url = "https://files.pythonhosted.org/packages/d8/78/ea0ae5ec8ba7a5c10bdd6e318f1ba5e76fcde17db8275188772afc7917a4/aiohttp-3.13.5-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:c632ce9c0b534fbe25b52c974515ed674937c5b99f549a92127c85f771a78772", size = 1742113, upload-time = "2026-03-31T21:59:17.068Z" }, + { url = "https://files.pythonhosted.org/packages/8a/66/9d308ed71e3f2491be1acb8769d96c6f0c47d92099f3bc9119cada27b357/aiohttp-3.13.5-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:fceedde51fbd67ee2bcc8c0b33d0126cc8b51ef3bbde2f86662bd6d5a6f10ec5", size = 1553088, upload-time = "2026-03-31T21:59:19.541Z" }, + { url = "https://files.pythonhosted.org/packages/da/a6/6cc25ed8dfc6e00c90f5c6d126a98e2cf28957ad06fa1036bd34b6f24a2c/aiohttp-3.13.5-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:f92995dfec9420bb69ae629abf422e516923ba79ba4403bc750d94fb4a6c68c1", size = 1757976, upload-time = "2026-03-31T21:59:22.311Z" }, + { url = "https://files.pythonhosted.org/packages/c1/2b/cce5b0ffe0de99c83e5e36d8f828e4161e415660a9f3e58339d07cce3006/aiohttp-3.13.5-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:20ae0ff08b1f2c8788d6fb85afcb798654ae6ba0b747575f8562de738078457b", size = 1712444, upload-time = "2026-03-31T21:59:24.635Z" }, + { url = "https://files.pythonhosted.org/packages/6c/cf/9e1795b4160c58d29421eafd1a69c6ce351e2f7c8d3c6b7e4ca44aea1a5b/aiohttp-3.13.5-cp314-cp314-win32.whl", hash = "sha256:b20df693de16f42b2472a9c485e1c948ee55524786a0a34345511afdd22246f3", size = 438128, upload-time = "2026-03-31T21:59:27.291Z" }, + { url = "https://files.pythonhosted.org/packages/22/4d/eaedff67fc805aeba4ba746aec891b4b24cebb1a7d078084b6300f79d063/aiohttp-3.13.5-cp314-cp314-win_amd64.whl", hash = "sha256:f85c6f327bf0b8c29da7d93b1cabb6363fb5e4e160a32fa241ed2dce21b73162", size = 464029, upload-time = "2026-03-31T21:59:29.429Z" }, + { url = "https://files.pythonhosted.org/packages/79/11/c27d9332ee20d68dd164dc12a6ecdef2e2e35ecc97ed6cf0d2442844624b/aiohttp-3.13.5-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:1efb06900858bb618ff5cee184ae2de5828896c448403d51fb633f09e109be0a", size = 778758, upload-time = "2026-03-31T21:59:31.547Z" }, + { url = "https://files.pythonhosted.org/packages/04/fb/377aead2e0a3ba5f09b7624f702a964bdf4f08b5b6728a9799830c80041e/aiohttp-3.13.5-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:fee86b7c4bd29bdaf0d53d14739b08a106fdda809ca5fe032a15f52fae5fe254", size = 512883, upload-time = "2026-03-31T21:59:34.098Z" }, + { url = "https://files.pythonhosted.org/packages/bb/a6/aa109a33671f7a5d3bd78b46da9d852797c5e665bfda7d6b373f56bff2ec/aiohttp-3.13.5-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:20058e23909b9e65f9da62b396b77dfa95965cbe840f8def6e572538b1d32e36", size = 516668, upload-time = "2026-03-31T21:59:36.497Z" }, + { url = "https://files.pythonhosted.org/packages/79/b3/ca078f9f2fa9563c36fb8ef89053ea2bb146d6f792c5104574d49d8acb63/aiohttp-3.13.5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8cf20a8d6868cb15a73cab329ffc07291ba8c22b1b88176026106ae39aa6df0f", size = 1883461, upload-time = "2026-03-31T21:59:38.723Z" }, + { url = "https://files.pythonhosted.org/packages/b7/e3/a7ad633ca1ca497b852233a3cce6906a56c3225fb6d9217b5e5e60b7419d/aiohttp-3.13.5-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:330f5da04c987f1d5bdb8ae189137c77139f36bd1cb23779ca1a354a4b027800", size = 1747661, upload-time = "2026-03-31T21:59:41.187Z" }, + { url = "https://files.pythonhosted.org/packages/33/b9/cd6fe579bed34a906d3d783fe60f2fa297ef55b27bb4538438ee49d4dc41/aiohttp-3.13.5-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6f1cbf0c7926d315c3c26c2da41fd2b5d2fe01ac0e157b78caefc51a782196cf", size = 1863800, upload-time = "2026-03-31T21:59:43.84Z" }, + { url = "https://files.pythonhosted.org/packages/c0/3f/2c1e2f5144cefa889c8afd5cf431994c32f3b29da9961698ff4e3811b79a/aiohttp-3.13.5-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:53fc049ed6390d05423ba33103ded7281fe897cf97878f369a527070bd95795b", size = 1958382, upload-time = "2026-03-31T21:59:46.187Z" }, + { url = "https://files.pythonhosted.org/packages/66/1d/f31ec3f1013723b3babe3609e7f119c2c2fb6ef33da90061a705ef3e1bc8/aiohttp-3.13.5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:898703aa2667e3c5ca4c54ca36cd73f58b7a38ef87a5606414799ebce4d3fd3a", size = 1803724, upload-time = "2026-03-31T21:59:48.656Z" }, + { url = "https://files.pythonhosted.org/packages/0e/b4/57712dfc6f1542f067daa81eb61da282fab3e6f1966fca25db06c4fc62d5/aiohttp-3.13.5-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0494a01ca9584eea1e5fbd6d748e61ecff218c51b576ee1999c23db7066417d8", size = 1640027, upload-time = "2026-03-31T21:59:51.284Z" }, + { url = "https://files.pythonhosted.org/packages/25/3c/734c878fb43ec083d8e31bf029daae1beafeae582d1b35da234739e82ee7/aiohttp-3.13.5-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:6cf81fe010b8c17b09495cbd15c1d35afbc8fb405c0c9cf4738e5ae3af1d65be", size = 1806644, upload-time = "2026-03-31T21:59:53.753Z" }, + { url = "https://files.pythonhosted.org/packages/20/a5/f671e5cbec1c21d044ff3078223f949748f3a7f86b14e34a365d74a5d21f/aiohttp-3.13.5-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:c564dd5f09ddc9d8f2c2d0a301cd30a79a2cc1b46dd1a73bef8f0038863d016b", size = 1791630, upload-time = "2026-03-31T21:59:56.239Z" }, + { url = "https://files.pythonhosted.org/packages/0b/63/fb8d0ad63a0b8a99be97deac8c04dacf0785721c158bdf23d679a87aa99e/aiohttp-3.13.5-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2994be9f6e51046c4f864598fd9abeb4fba6e88f0b2152422c9666dcd4aea9c6", size = 1809403, upload-time = "2026-03-31T21:59:59.103Z" }, + { url = "https://files.pythonhosted.org/packages/59/0c/bfed7f30662fcf12206481c2aac57dedee43fe1c49275e85b3a1e1742294/aiohttp-3.13.5-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:157826e2fa245d2ef46c83ea8a5faf77ca19355d278d425c29fda0beb3318037", size = 1634924, upload-time = "2026-03-31T22:00:02.116Z" }, + { url = "https://files.pythonhosted.org/packages/17/d6/fd518d668a09fd5a3319ae5e984d4d80b9a4b3df4e21c52f02251ef5a32e/aiohttp-3.13.5-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:a8aca50daa9493e9e13c0f566201a9006f080e7c50e5e90d0b06f53146a54500", size = 1836119, upload-time = "2026-03-31T22:00:04.756Z" }, + { url = "https://files.pythonhosted.org/packages/78/b7/15fb7a9d52e112a25b621c67b69c167805cb1f2ab8f1708a5c490d1b52fe/aiohttp-3.13.5-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:3b13560160d07e047a93f23aaa30718606493036253d5430887514715b67c9d9", size = 1772072, upload-time = "2026-03-31T22:00:07.494Z" }, + { url = "https://files.pythonhosted.org/packages/7e/df/57ba7f0c4a553fc2bd8b6321df236870ec6fd64a2a473a8a13d4f733214e/aiohttp-3.13.5-cp314-cp314t-win32.whl", hash = "sha256:9a0f4474b6ea6818b41f82172d799e4b3d29e22c2c520ce4357856fced9af2f8", size = 471819, upload-time = "2026-03-31T22:00:10.277Z" }, + { url = "https://files.pythonhosted.org/packages/62/29/2f8418269e46454a26171bfdd6a055d74febf32234e474930f2f60a17145/aiohttp-3.13.5-cp314-cp314t-win_amd64.whl", hash = "sha256:18a2f6c1182c51baa1d28d68fea51513cb2a76612f038853c0ad3c145423d3d9", size = 505441, upload-time = "2026-03-31T22:00:12.791Z" }, +] + +[[package]] +name = "aiosignal" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "frozenlist" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/62/06741b579156360248d1ec624842ad0edf697050bbaf7c3e46394e106ad1/aiosignal-1.4.0.tar.gz", hash = "sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7", size = 25007, upload-time = "2025-07-03T22:54:43.528Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" }, +] + +[[package]] +name = "annotated-doc" +version = "0.0.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/57/ba/046ceea27344560984e26a590f90bc7f4a75b06701f653222458922b558c/annotated_doc-0.0.4.tar.gz", hash = "sha256:fbcda96e87e9c92ad167c2e53839e57503ecfda18804ea28102353485033faa4", size = 7288, upload-time = "2025-11-10T22:07:42.062Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/d3/26bf1008eb3d2daa8ef4cacc7f3bfdc11818d111f7e2d0201bc6e3b49d45/annotated_doc-0.0.4-py3-none-any.whl", hash = "sha256:571ac1dc6991c450b25a9c2d84a3705e2ae7a53467b5d111c24fa8baabbed320", size = 5303, upload-time = "2025-11-10T22:07:40.673Z" }, +] + +[[package]] +name = "annotated-types" +version = "0.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" }, +] + +[[package]] +name = "anyio" +version = "4.13.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/19/14/2c5dd9f512b66549ae92767a9c7b330ae88e1932ca57876909410251fe13/anyio-4.13.0.tar.gz", hash = "sha256:334b70e641fd2221c1505b3890c69882fe4a2df910cba14d97019b90b24439dc", size = 231622, upload-time = "2026-03-24T12:59:09.671Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" }, +] + +[[package]] +name = "async-lru" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e8/1f/989ecfef8e64109a489fff357450cb73fa73a865a92bd8c272170a6922c2/async_lru-2.3.0.tar.gz", hash = "sha256:89bdb258a0140d7313cf8f4031d816a042202faa61d0ab310a0a538baa1c24b6", size = 16332, upload-time = "2026-03-19T01:04:32.413Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/e2/c2e3abf398f80732e58b03be77bde9022550d221dd8781bf586bd4d97cc1/async_lru-2.3.0-py3-none-any.whl", hash = "sha256:eea27b01841909316f2cc739807acea1c623df2be8c5cfad7583286397bb8315", size = 8403, upload-time = "2026-03-19T01:04:30.883Z" }, +] + +[[package]] +name = "attrs" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, +] + +[[package]] +name = "babel" +version = "2.18.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" }, +] + +[[package]] +name = "beautifulsoup4" +version = "4.14.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "soupsieve" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c3/b0/1c6a16426d389813b48d95e26898aff79abbde42ad353958ad95cc8c9b21/beautifulsoup4-4.14.3.tar.gz", hash = "sha256:6292b1c5186d356bba669ef9f7f051757099565ad9ada5dd630bd9de5fa7fb86", size = 627737, upload-time = "2025-11-30T15:08:26.084Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1a/39/47f9197bdd44df24d67ac8893641e16f386c984a0619ef2ee4c51fbbc019/beautifulsoup4-4.14.3-py3-none-any.whl", hash = "sha256:0918bfe44902e6ad8d57732ba310582e98da931428d231a5ecb9e7c703a735bb", size = 107721, upload-time = "2025-11-30T15:08:24.087Z" }, +] + +[[package]] +name = "bracex" +version = "2.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/63/9a/fec38644694abfaaeca2798b58e276a8e61de49e2e37494ace423395febc/bracex-2.6.tar.gz", hash = "sha256:98f1347cd77e22ee8d967a30ad4e310b233f7754dbf31ff3fceb76145ba47dc7", size = 26642, upload-time = "2025-06-22T19:12:31.254Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9d/2a/9186535ce58db529927f6cf5990a849aa9e052eea3e2cfefe20b9e1802da/bracex-2.6-py3-none-any.whl", hash = "sha256:0b0049264e7340b3ec782b5cb99beb325f36c3782a32e36e876452fd49a09952", size = 11508, upload-time = "2025-06-22T19:12:29.781Z" }, +] + +[[package]] +name = "certifi" +version = "2026.2.25" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/af/2d/7bf41579a8986e348fa033a31cdd0e4121114f6bce2457e8876010b092dd/certifi-2026.2.25.tar.gz", hash = "sha256:e887ab5cee78ea814d3472169153c2d12cd43b14bd03329a39a9c6e2e80bfba7", size = 155029, upload-time = "2026-02-25T02:54:17.342Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9a/3c/c17fb3ca2d9c3acff52e30b309f538586f9f5b9c9cf454f3845fc9af4881/certifi-2026.2.25-py3-none-any.whl", hash = "sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa", size = 153684, upload-time = "2026-02-25T02:54:15.766Z" }, +] + +[[package]] +name = "cffi" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pycparser", marker = "implementation_name != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/eb/56/b1ba7935a17738ae8453301356628e8147c79dbb825bcbc73dc7401f9846/cffi-2.0.0.tar.gz", hash = "sha256:44d1b5909021139fe36001ae048dbdde8214afa20200eda0f64c068cac5d5529", size = 523588, upload-time = "2025-09-08T23:24:04.541Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/4a/3dfd5f7850cbf0d06dc84ba9aa00db766b52ca38d8b86e3a38314d52498c/cffi-2.0.0-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:b4c854ef3adc177950a8dfc81a86f5115d2abd545751a304c5bcf2c2c7283cfe", size = 184344, upload-time = "2025-09-08T23:22:26.456Z" }, + { url = "https://files.pythonhosted.org/packages/4f/8b/f0e4c441227ba756aafbe78f117485b25bb26b1c059d01f137fa6d14896b/cffi-2.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2de9a304e27f7596cd03d16f1b7c72219bd944e99cc52b84d0145aefb07cbd3c", size = 180560, upload-time = "2025-09-08T23:22:28.197Z" }, + { url = "https://files.pythonhosted.org/packages/b1/b7/1200d354378ef52ec227395d95c2576330fd22a869f7a70e88e1447eb234/cffi-2.0.0-cp311-cp311-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:baf5215e0ab74c16e2dd324e8ec067ef59e41125d3eade2b863d294fd5035c92", size = 209613, upload-time = "2025-09-08T23:22:29.475Z" }, + { url = "https://files.pythonhosted.org/packages/b8/56/6033f5e86e8cc9bb629f0077ba71679508bdf54a9a5e112a3c0b91870332/cffi-2.0.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:730cacb21e1bdff3ce90babf007d0a0917cc3e6492f336c2f0134101e0944f93", size = 216476, upload-time = "2025-09-08T23:22:31.063Z" }, + { url = "https://files.pythonhosted.org/packages/dc/7f/55fecd70f7ece178db2f26128ec41430d8720f2d12ca97bf8f0a628207d5/cffi-2.0.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:6824f87845e3396029f3820c206e459ccc91760e8fa24422f8b0c3d1731cbec5", size = 203374, upload-time = "2025-09-08T23:22:32.507Z" }, + { url = "https://files.pythonhosted.org/packages/84/ef/a7b77c8bdc0f77adc3b46888f1ad54be8f3b7821697a7b89126e829e676a/cffi-2.0.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:9de40a7b0323d889cf8d23d1ef214f565ab154443c42737dfe52ff82cf857664", size = 202597, upload-time = "2025-09-08T23:22:34.132Z" }, + { url = "https://files.pythonhosted.org/packages/d7/91/500d892b2bf36529a75b77958edfcd5ad8e2ce4064ce2ecfeab2125d72d1/cffi-2.0.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8941aaadaf67246224cee8c3803777eed332a19d909b47e29c9842ef1e79ac26", size = 215574, upload-time = "2025-09-08T23:22:35.443Z" }, + { url = "https://files.pythonhosted.org/packages/44/64/58f6255b62b101093d5df22dcb752596066c7e89dd725e0afaed242a61be/cffi-2.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:a05d0c237b3349096d3981b727493e22147f934b20f6f125a3eba8f994bec4a9", size = 218971, upload-time = "2025-09-08T23:22:36.805Z" }, + { url = "https://files.pythonhosted.org/packages/ab/49/fa72cebe2fd8a55fbe14956f9970fe8eb1ac59e5df042f603ef7c8ba0adc/cffi-2.0.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:94698a9c5f91f9d138526b48fe26a199609544591f859c870d477351dc7b2414", size = 211972, upload-time = "2025-09-08T23:22:38.436Z" }, + { url = "https://files.pythonhosted.org/packages/0b/28/dd0967a76aab36731b6ebfe64dec4e981aff7e0608f60c2d46b46982607d/cffi-2.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5fed36fccc0612a53f1d4d9a816b50a36702c28a2aa880cb8a122b3466638743", size = 217078, upload-time = "2025-09-08T23:22:39.776Z" }, + { url = "https://files.pythonhosted.org/packages/2b/c0/015b25184413d7ab0a410775fdb4a50fca20f5589b5dab1dbbfa3baad8ce/cffi-2.0.0-cp311-cp311-win32.whl", hash = "sha256:c649e3a33450ec82378822b3dad03cc228b8f5963c0c12fc3b1e0ab940f768a5", size = 172076, upload-time = "2025-09-08T23:22:40.95Z" }, + { url = "https://files.pythonhosted.org/packages/ae/8f/dc5531155e7070361eb1b7e4c1a9d896d0cb21c49f807a6c03fd63fc877e/cffi-2.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:66f011380d0e49ed280c789fbd08ff0d40968ee7b665575489afa95c98196ab5", size = 182820, upload-time = "2025-09-08T23:22:42.463Z" }, + { url = "https://files.pythonhosted.org/packages/95/5c/1b493356429f9aecfd56bc171285a4c4ac8697f76e9bbbbb105e537853a1/cffi-2.0.0-cp311-cp311-win_arm64.whl", hash = "sha256:c6638687455baf640e37344fe26d37c404db8b80d037c3d29f58fe8d1c3b194d", size = 177635, upload-time = "2025-09-08T23:22:43.623Z" }, + { url = "https://files.pythonhosted.org/packages/ea/47/4f61023ea636104d4f16ab488e268b93008c3d0bb76893b1b31db1f96802/cffi-2.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d02d6655b0e54f54c4ef0b94eb6be0607b70853c45ce98bd278dc7de718be5d", size = 185271, upload-time = "2025-09-08T23:22:44.795Z" }, + { url = "https://files.pythonhosted.org/packages/df/a2/781b623f57358e360d62cdd7a8c681f074a71d445418a776eef0aadb4ab4/cffi-2.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8eca2a813c1cb7ad4fb74d368c2ffbbb4789d377ee5bb8df98373c2cc0dee76c", size = 181048, upload-time = "2025-09-08T23:22:45.938Z" }, + { url = "https://files.pythonhosted.org/packages/ff/df/a4f0fbd47331ceeba3d37c2e51e9dfc9722498becbeec2bd8bc856c9538a/cffi-2.0.0-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:21d1152871b019407d8ac3985f6775c079416c282e431a4da6afe7aefd2bccbe", size = 212529, upload-time = "2025-09-08T23:22:47.349Z" }, + { url = "https://files.pythonhosted.org/packages/d5/72/12b5f8d3865bf0f87cf1404d8c374e7487dcf097a1c91c436e72e6badd83/cffi-2.0.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b21e08af67b8a103c71a250401c78d5e0893beff75e28c53c98f4de42f774062", size = 220097, upload-time = "2025-09-08T23:22:48.677Z" }, + { url = "https://files.pythonhosted.org/packages/c2/95/7a135d52a50dfa7c882ab0ac17e8dc11cec9d55d2c18dda414c051c5e69e/cffi-2.0.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:1e3a615586f05fc4065a8b22b8152f0c1b00cdbc60596d187c2a74f9e3036e4e", size = 207983, upload-time = "2025-09-08T23:22:50.06Z" }, + { url = "https://files.pythonhosted.org/packages/3a/c8/15cb9ada8895957ea171c62dc78ff3e99159ee7adb13c0123c001a2546c1/cffi-2.0.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:81afed14892743bbe14dacb9e36d9e0e504cd204e0b165062c488942b9718037", size = 206519, upload-time = "2025-09-08T23:22:51.364Z" }, + { url = "https://files.pythonhosted.org/packages/78/2d/7fa73dfa841b5ac06c7b8855cfc18622132e365f5b81d02230333ff26e9e/cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3e17ed538242334bf70832644a32a7aae3d83b57567f9fd60a26257e992b79ba", size = 219572, upload-time = "2025-09-08T23:22:52.902Z" }, + { url = "https://files.pythonhosted.org/packages/07/e0/267e57e387b4ca276b90f0434ff88b2c2241ad72b16d31836adddfd6031b/cffi-2.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3925dd22fa2b7699ed2617149842d2e6adde22b262fcbfada50e3d195e4b3a94", size = 222963, upload-time = "2025-09-08T23:22:54.518Z" }, + { url = "https://files.pythonhosted.org/packages/b6/75/1f2747525e06f53efbd878f4d03bac5b859cbc11c633d0fb81432d98a795/cffi-2.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2c8f814d84194c9ea681642fd164267891702542f028a15fc97d4674b6206187", size = 221361, upload-time = "2025-09-08T23:22:55.867Z" }, + { url = "https://files.pythonhosted.org/packages/7b/2b/2b6435f76bfeb6bbf055596976da087377ede68df465419d192acf00c437/cffi-2.0.0-cp312-cp312-win32.whl", hash = "sha256:da902562c3e9c550df360bfa53c035b2f241fed6d9aef119048073680ace4a18", size = 172932, upload-time = "2025-09-08T23:22:57.188Z" }, + { url = "https://files.pythonhosted.org/packages/f8/ed/13bd4418627013bec4ed6e54283b1959cf6db888048c7cf4b4c3b5b36002/cffi-2.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:da68248800ad6320861f129cd9c1bf96ca849a2771a59e0344e88681905916f5", size = 183557, upload-time = "2025-09-08T23:22:58.351Z" }, + { url = "https://files.pythonhosted.org/packages/95/31/9f7f93ad2f8eff1dbc1c3656d7ca5bfd8fb52c9d786b4dcf19b2d02217fa/cffi-2.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:4671d9dd5ec934cb9a73e7ee9676f9362aba54f7f34910956b84d727b0d73fb6", size = 177762, upload-time = "2025-09-08T23:22:59.668Z" }, + { url = "https://files.pythonhosted.org/packages/4b/8d/a0a47a0c9e413a658623d014e91e74a50cdd2c423f7ccfd44086ef767f90/cffi-2.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:00bdf7acc5f795150faa6957054fbbca2439db2f775ce831222b66f192f03beb", size = 185230, upload-time = "2025-09-08T23:23:00.879Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d2/a6c0296814556c68ee32009d9c2ad4f85f2707cdecfd7727951ec228005d/cffi-2.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:45d5e886156860dc35862657e1494b9bae8dfa63bf56796f2fb56e1679fc0bca", size = 181043, upload-time = "2025-09-08T23:23:02.231Z" }, + { url = "https://files.pythonhosted.org/packages/b0/1e/d22cc63332bd59b06481ceaac49d6c507598642e2230f201649058a7e704/cffi-2.0.0-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:07b271772c100085dd28b74fa0cd81c8fb1a3ba18b21e03d7c27f3436a10606b", size = 212446, upload-time = "2025-09-08T23:23:03.472Z" }, + { url = "https://files.pythonhosted.org/packages/a9/f5/a2c23eb03b61a0b8747f211eb716446c826ad66818ddc7810cc2cc19b3f2/cffi-2.0.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d48a880098c96020b02d5a1f7d9251308510ce8858940e6fa99ece33f610838b", size = 220101, upload-time = "2025-09-08T23:23:04.792Z" }, + { url = "https://files.pythonhosted.org/packages/f2/7f/e6647792fc5850d634695bc0e6ab4111ae88e89981d35ac269956605feba/cffi-2.0.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f93fd8e5c8c0a4aa1f424d6173f14a892044054871c771f8566e4008eaa359d2", size = 207948, upload-time = "2025-09-08T23:23:06.127Z" }, + { url = "https://files.pythonhosted.org/packages/cb/1e/a5a1bd6f1fb30f22573f76533de12a00bf274abcdc55c8edab639078abb6/cffi-2.0.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:dd4f05f54a52fb558f1ba9f528228066954fee3ebe629fc1660d874d040ae5a3", size = 206422, upload-time = "2025-09-08T23:23:07.753Z" }, + { url = "https://files.pythonhosted.org/packages/98/df/0a1755e750013a2081e863e7cd37e0cdd02664372c754e5560099eb7aa44/cffi-2.0.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c8d3b5532fc71b7a77c09192b4a5a200ea992702734a2e9279a37f2478236f26", size = 219499, upload-time = "2025-09-08T23:23:09.648Z" }, + { url = "https://files.pythonhosted.org/packages/50/e1/a969e687fcf9ea58e6e2a928ad5e2dd88cc12f6f0ab477e9971f2309b57c/cffi-2.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d9b29c1f0ae438d5ee9acb31cadee00a58c46cc9c0b2f9038c6b0b3470877a8c", size = 222928, upload-time = "2025-09-08T23:23:10.928Z" }, + { url = "https://files.pythonhosted.org/packages/36/54/0362578dd2c9e557a28ac77698ed67323ed5b9775ca9d3fe73fe191bb5d8/cffi-2.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6d50360be4546678fc1b79ffe7a66265e28667840010348dd69a314145807a1b", size = 221302, upload-time = "2025-09-08T23:23:12.42Z" }, + { url = "https://files.pythonhosted.org/packages/eb/6d/bf9bda840d5f1dfdbf0feca87fbdb64a918a69bca42cfa0ba7b137c48cb8/cffi-2.0.0-cp313-cp313-win32.whl", hash = "sha256:74a03b9698e198d47562765773b4a8309919089150a0bb17d829ad7b44b60d27", size = 172909, upload-time = "2025-09-08T23:23:14.32Z" }, + { url = "https://files.pythonhosted.org/packages/37/18/6519e1ee6f5a1e579e04b9ddb6f1676c17368a7aba48299c3759bbc3c8b3/cffi-2.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:19f705ada2530c1167abacb171925dd886168931e0a7b78f5bffcae5c6b5be75", size = 183402, upload-time = "2025-09-08T23:23:15.535Z" }, + { url = "https://files.pythonhosted.org/packages/cb/0e/02ceeec9a7d6ee63bb596121c2c8e9b3a9e150936f4fbef6ca1943e6137c/cffi-2.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:256f80b80ca3853f90c21b23ee78cd008713787b1b1e93eae9f3d6a7134abd91", size = 177780, upload-time = "2025-09-08T23:23:16.761Z" }, + { url = "https://files.pythonhosted.org/packages/92/c4/3ce07396253a83250ee98564f8d7e9789fab8e58858f35d07a9a2c78de9f/cffi-2.0.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fc33c5141b55ed366cfaad382df24fe7dcbc686de5be719b207bb248e3053dc5", size = 185320, upload-time = "2025-09-08T23:23:18.087Z" }, + { url = "https://files.pythonhosted.org/packages/59/dd/27e9fa567a23931c838c6b02d0764611c62290062a6d4e8ff7863daf9730/cffi-2.0.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c654de545946e0db659b3400168c9ad31b5d29593291482c43e3564effbcee13", size = 181487, upload-time = "2025-09-08T23:23:19.622Z" }, + { url = "https://files.pythonhosted.org/packages/d6/43/0e822876f87ea8a4ef95442c3d766a06a51fc5298823f884ef87aaad168c/cffi-2.0.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:24b6f81f1983e6df8db3adc38562c83f7d4a0c36162885ec7f7b77c7dcbec97b", size = 220049, upload-time = "2025-09-08T23:23:20.853Z" }, + { url = "https://files.pythonhosted.org/packages/b4/89/76799151d9c2d2d1ead63c2429da9ea9d7aac304603de0c6e8764e6e8e70/cffi-2.0.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:12873ca6cb9b0f0d3a0da705d6086fe911591737a59f28b7936bdfed27c0d47c", size = 207793, upload-time = "2025-09-08T23:23:22.08Z" }, + { url = "https://files.pythonhosted.org/packages/bb/dd/3465b14bb9e24ee24cb88c9e3730f6de63111fffe513492bf8c808a3547e/cffi-2.0.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:d9b97165e8aed9272a6bb17c01e3cc5871a594a446ebedc996e2397a1c1ea8ef", size = 206300, upload-time = "2025-09-08T23:23:23.314Z" }, + { url = "https://files.pythonhosted.org/packages/47/d9/d83e293854571c877a92da46fdec39158f8d7e68da75bf73581225d28e90/cffi-2.0.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:afb8db5439b81cf9c9d0c80404b60c3cc9c3add93e114dcae767f1477cb53775", size = 219244, upload-time = "2025-09-08T23:23:24.541Z" }, + { url = "https://files.pythonhosted.org/packages/2b/0f/1f177e3683aead2bb00f7679a16451d302c436b5cbf2505f0ea8146ef59e/cffi-2.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:737fe7d37e1a1bffe70bd5754ea763a62a066dc5913ca57e957824b72a85e205", size = 222828, upload-time = "2025-09-08T23:23:26.143Z" }, + { url = "https://files.pythonhosted.org/packages/c6/0f/cafacebd4b040e3119dcb32fed8bdef8dfe94da653155f9d0b9dc660166e/cffi-2.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:38100abb9d1b1435bc4cc340bb4489635dc2f0da7456590877030c9b3d40b0c1", size = 220926, upload-time = "2025-09-08T23:23:27.873Z" }, + { url = "https://files.pythonhosted.org/packages/3e/aa/df335faa45b395396fcbc03de2dfcab242cd61a9900e914fe682a59170b1/cffi-2.0.0-cp314-cp314-win32.whl", hash = "sha256:087067fa8953339c723661eda6b54bc98c5625757ea62e95eb4898ad5e776e9f", size = 175328, upload-time = "2025-09-08T23:23:44.61Z" }, + { url = "https://files.pythonhosted.org/packages/bb/92/882c2d30831744296ce713f0feb4c1cd30f346ef747b530b5318715cc367/cffi-2.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:203a48d1fb583fc7d78a4c6655692963b860a417c0528492a6bc21f1aaefab25", size = 185650, upload-time = "2025-09-08T23:23:45.848Z" }, + { url = "https://files.pythonhosted.org/packages/9f/2c/98ece204b9d35a7366b5b2c6539c350313ca13932143e79dc133ba757104/cffi-2.0.0-cp314-cp314-win_arm64.whl", hash = "sha256:dbd5c7a25a7cb98f5ca55d258b103a2054f859a46ae11aaf23134f9cc0d356ad", size = 180687, upload-time = "2025-09-08T23:23:47.105Z" }, + { url = "https://files.pythonhosted.org/packages/3e/61/c768e4d548bfa607abcda77423448df8c471f25dbe64fb2ef6d555eae006/cffi-2.0.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:9a67fc9e8eb39039280526379fb3a70023d77caec1852002b4da7e8b270c4dd9", size = 188773, upload-time = "2025-09-08T23:23:29.347Z" }, + { url = "https://files.pythonhosted.org/packages/2c/ea/5f76bce7cf6fcd0ab1a1058b5af899bfbef198bea4d5686da88471ea0336/cffi-2.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7a66c7204d8869299919db4d5069a82f1561581af12b11b3c9f48c584eb8743d", size = 185013, upload-time = "2025-09-08T23:23:30.63Z" }, + { url = "https://files.pythonhosted.org/packages/be/b4/c56878d0d1755cf9caa54ba71e5d049479c52f9e4afc230f06822162ab2f/cffi-2.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7cc09976e8b56f8cebd752f7113ad07752461f48a58cbba644139015ac24954c", size = 221593, upload-time = "2025-09-08T23:23:31.91Z" }, + { url = "https://files.pythonhosted.org/packages/e0/0d/eb704606dfe8033e7128df5e90fee946bbcb64a04fcdaa97321309004000/cffi-2.0.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:92b68146a71df78564e4ef48af17551a5ddd142e5190cdf2c5624d0c3ff5b2e8", size = 209354, upload-time = "2025-09-08T23:23:33.214Z" }, + { url = "https://files.pythonhosted.org/packages/d8/19/3c435d727b368ca475fb8742ab97c9cb13a0de600ce86f62eab7fa3eea60/cffi-2.0.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b1e74d11748e7e98e2f426ab176d4ed720a64412b6a15054378afdb71e0f37dc", size = 208480, upload-time = "2025-09-08T23:23:34.495Z" }, + { url = "https://files.pythonhosted.org/packages/d0/44/681604464ed9541673e486521497406fadcc15b5217c3e326b061696899a/cffi-2.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:28a3a209b96630bca57cce802da70c266eb08c6e97e5afd61a75611ee6c64592", size = 221584, upload-time = "2025-09-08T23:23:36.096Z" }, + { url = "https://files.pythonhosted.org/packages/25/8e/342a504ff018a2825d395d44d63a767dd8ebc927ebda557fecdaca3ac33a/cffi-2.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7553fb2090d71822f02c629afe6042c299edf91ba1bf94951165613553984512", size = 224443, upload-time = "2025-09-08T23:23:37.328Z" }, + { url = "https://files.pythonhosted.org/packages/e1/5e/b666bacbbc60fbf415ba9988324a132c9a7a0448a9a8f125074671c0f2c3/cffi-2.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6c6c373cfc5c83a975506110d17457138c8c63016b563cc9ed6e056a82f13ce4", size = 223437, upload-time = "2025-09-08T23:23:38.945Z" }, + { url = "https://files.pythonhosted.org/packages/a0/1d/ec1a60bd1a10daa292d3cd6bb0b359a81607154fb8165f3ec95fe003b85c/cffi-2.0.0-cp314-cp314t-win32.whl", hash = "sha256:1fc9ea04857caf665289b7a75923f2c6ed559b8298a1b8c49e59f7dd95c8481e", size = 180487, upload-time = "2025-09-08T23:23:40.423Z" }, + { url = "https://files.pythonhosted.org/packages/bf/41/4c1168c74fac325c0c8156f04b6749c8b6a8f405bbf91413ba088359f60d/cffi-2.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d68b6cef7827e8641e8ef16f4494edda8b36104d79773a334beaa1e3521430f6", size = 191726, upload-time = "2025-09-08T23:23:41.742Z" }, + { url = "https://files.pythonhosted.org/packages/ae/3a/dbeec9d1ee0844c679f6bb5d6ad4e9f198b1224f4e7a32825f47f6192b0c/cffi-2.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9", size = 184195, upload-time = "2025-09-08T23:23:43.004Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.4.7" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e7/a1/67fe25fac3c7642725500a3f6cfe5821ad557c3abb11c9d20d12c7008d3e/charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5", size = 144271, upload-time = "2026-04-02T09:28:39.342Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/d7/b5b7020a0565c2e9fa8c09f4b5fa6232feb326b8c20081ccded47ea368fd/charset_normalizer-3.4.7-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:7641bb8895e77f921102f72833904dcd9901df5d6d72a2ab8f31d04b7e51e4e7", size = 309705, upload-time = "2026-04-02T09:26:02.191Z" }, + { url = "https://files.pythonhosted.org/packages/5a/53/58c29116c340e5456724ecd2fff4196d236b98f3da97b404bc5e51ac3493/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:202389074300232baeb53ae2569a60901f7efadd4245cf3a3bf0617d60b439d7", size = 206419, upload-time = "2026-04-02T09:26:03.583Z" }, + { url = "https://files.pythonhosted.org/packages/b2/02/e8146dc6591a37a00e5144c63f29fb7c97a734ea8a111190783c0e60ab63/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:30b8d1d8c52a48c2c5690e152c169b673487a2a58de1ec7393196753063fcd5e", size = 227901, upload-time = "2026-04-02T09:26:04.738Z" }, + { url = "https://files.pythonhosted.org/packages/fb/73/77486c4cd58f1267bf17db420e930c9afa1b3be3fe8c8b8ebbebc9624359/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:532bc9bf33a68613fd7d65e4b1c71a6a38d7d42604ecf239c77392e9b4e8998c", size = 222742, upload-time = "2026-04-02T09:26:06.36Z" }, + { url = "https://files.pythonhosted.org/packages/a1/fa/f74eb381a7d94ded44739e9d94de18dc5edc9c17fb8c11f0a6890696c0a9/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2fe249cb4651fd12605b7288b24751d8bfd46d35f12a20b1ba33dea122e690df", size = 214061, upload-time = "2026-04-02T09:26:08.347Z" }, + { url = "https://files.pythonhosted.org/packages/dc/92/42bd3cefcf7687253fb86694b45f37b733c97f59af3724f356fa92b8c344/charset_normalizer-3.4.7-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:65bcd23054beab4d166035cabbc868a09c1a49d1efe458fe8e4361215df40265", size = 199239, upload-time = "2026-04-02T09:26:09.823Z" }, + { url = "https://files.pythonhosted.org/packages/4c/3d/069e7184e2aa3b3cddc700e3dd267413dc259854adc3380421c805c6a17d/charset_normalizer-3.4.7-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:08e721811161356f97b4059a9ba7bafb23ea5ee2255402c42881c214e173c6b4", size = 210173, upload-time = "2026-04-02T09:26:10.953Z" }, + { url = "https://files.pythonhosted.org/packages/62/51/9d56feb5f2e7074c46f93e0ebdbe61f0848ee246e2f0d89f8e20b89ebb8f/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:e060d01aec0a910bdccb8be71faf34e7799ce36950f8294c8bf612cba65a2c9e", size = 209841, upload-time = "2026-04-02T09:26:12.142Z" }, + { url = "https://files.pythonhosted.org/packages/d2/59/893d8f99cc4c837dda1fe2f1139079703deb9f321aabcb032355de13b6c7/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:38c0109396c4cfc574d502df99742a45c72c08eff0a36158b6f04000043dbf38", size = 200304, upload-time = "2026-04-02T09:26:13.711Z" }, + { url = "https://files.pythonhosted.org/packages/7d/1d/ee6f3be3464247578d1ed5c46de545ccc3d3ff933695395c402c21fa6b77/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:1c2a768fdd44ee4a9339a9b0b130049139b8ce3c01d2ce09f67f5a68048d477c", size = 229455, upload-time = "2026-04-02T09:26:14.941Z" }, + { url = "https://files.pythonhosted.org/packages/54/bb/8fb0a946296ea96a488928bdce8ef99023998c48e4713af533e9bb98ef07/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:1a87ca9d5df6fe460483d9a5bbf2b18f620cbed41b432e2bddb686228282d10b", size = 210036, upload-time = "2026-04-02T09:26:16.478Z" }, + { url = "https://files.pythonhosted.org/packages/9a/bc/015b2387f913749f82afd4fcba07846d05b6d784dd16123cb66860e0237d/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:d635aab80466bc95771bb78d5370e74d36d1fe31467b6b29b8b57b2a3cd7d22c", size = 224739, upload-time = "2026-04-02T09:26:17.751Z" }, + { url = "https://files.pythonhosted.org/packages/17/ab/63133691f56baae417493cba6b7c641571a2130eb7bceba6773367ab9ec5/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ae196f021b5e7c78e918242d217db021ed2a6ace2bc6ae94c0fc596221c7f58d", size = 216277, upload-time = "2026-04-02T09:26:18.981Z" }, + { url = "https://files.pythonhosted.org/packages/06/6d/3be70e827977f20db77c12a97e6a9f973631a45b8d186c084527e53e77a4/charset_normalizer-3.4.7-cp311-cp311-win32.whl", hash = "sha256:adb2597b428735679446b46c8badf467b4ca5f5056aae4d51a19f9570301b1ad", size = 147819, upload-time = "2026-04-02T09:26:20.295Z" }, + { url = "https://files.pythonhosted.org/packages/20/d9/5f67790f06b735d7c7637171bbfd89882ad67201891b7275e51116ed8207/charset_normalizer-3.4.7-cp311-cp311-win_amd64.whl", hash = "sha256:8e385e4267ab76874ae30db04c627faaaf0b509e1ccc11a95b3fc3e83f855c00", size = 159281, upload-time = "2026-04-02T09:26:21.74Z" }, + { url = "https://files.pythonhosted.org/packages/ca/83/6413f36c5a34afead88ce6f66684d943d91f233d76dd083798f9602b75ae/charset_normalizer-3.4.7-cp311-cp311-win_arm64.whl", hash = "sha256:d4a48e5b3c2a489fae013b7589308a40146ee081f6f509e047e0e096084ceca1", size = 147843, upload-time = "2026-04-02T09:26:22.901Z" }, + { url = "https://files.pythonhosted.org/packages/0c/eb/4fc8d0a7110eb5fc9cc161723a34a8a6c200ce3b4fbf681bc86feee22308/charset_normalizer-3.4.7-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:eca9705049ad3c7345d574e3510665cb2cf844c2f2dcfe675332677f081cbd46", size = 311328, upload-time = "2026-04-02T09:26:24.331Z" }, + { url = "https://files.pythonhosted.org/packages/f8/e3/0fadc706008ac9d7b9b5be6dc767c05f9d3e5df51744ce4cc9605de7b9f4/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6178f72c5508bfc5fd446a5905e698c6212932f25bcdd4b47a757a50605a90e2", size = 208061, upload-time = "2026-04-02T09:26:25.568Z" }, + { url = "https://files.pythonhosted.org/packages/42/f0/3dd1045c47f4a4604df85ec18ad093912ae1344ac706993aff91d38773a2/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e1421b502d83040e6d7fb2fb18dff63957f720da3d77b2fbd3187ceb63755d7b", size = 229031, upload-time = "2026-04-02T09:26:26.865Z" }, + { url = "https://files.pythonhosted.org/packages/dc/67/675a46eb016118a2fbde5a277a5d15f4f69d5f3f5f338e5ee2f8948fcf43/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:edac0f1ab77644605be2cbba52e6b7f630731fc42b34cb0f634be1a6eface56a", size = 225239, upload-time = "2026-04-02T09:26:28.044Z" }, + { url = "https://files.pythonhosted.org/packages/4b/f8/d0118a2f5f23b02cd166fa385c60f9b0d4f9194f574e2b31cef350ad7223/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5649fd1c7bade02f320a462fdefd0b4bd3ce036065836d4f42e0de958038e116", size = 216589, upload-time = "2026-04-02T09:26:29.239Z" }, + { url = "https://files.pythonhosted.org/packages/b1/f1/6d2b0b261b6c4ceef0fcb0d17a01cc5bc53586c2d4796fa04b5c540bc13d/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:203104ed3e428044fd943bc4bf45fa73c0730391f9621e37fe39ecf477b128cb", size = 202733, upload-time = "2026-04-02T09:26:30.5Z" }, + { url = "https://files.pythonhosted.org/packages/6f/c0/7b1f943f7e87cc3db9626ba17807d042c38645f0a1d4415c7a14afb5591f/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:298930cec56029e05497a76988377cbd7457ba864beeea92ad7e844fe74cd1f1", size = 212652, upload-time = "2026-04-02T09:26:31.709Z" }, + { url = "https://files.pythonhosted.org/packages/38/dd/5a9ab159fe45c6e72079398f277b7d2b523e7f716acc489726115a910097/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:708838739abf24b2ceb208d0e22403dd018faeef86ddac04319a62ae884c4f15", size = 211229, upload-time = "2026-04-02T09:26:33.282Z" }, + { url = "https://files.pythonhosted.org/packages/d5/ff/531a1cad5ca855d1c1a8b69cb71abfd6d85c0291580146fda7c82857caa1/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:0f7eb884681e3938906ed0434f20c63046eacd0111c4ba96f27b76084cd679f5", size = 203552, upload-time = "2026-04-02T09:26:34.845Z" }, + { url = "https://files.pythonhosted.org/packages/c1/4c/a5fb52d528a8ca41f7598cb619409ece30a169fbdf9cdce592e53b46c3a6/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4dc1e73c36828f982bfe79fadf5919923f8a6f4df2860804db9a98c48824ce8d", size = 230806, upload-time = "2026-04-02T09:26:36.152Z" }, + { url = "https://files.pythonhosted.org/packages/59/7a/071feed8124111a32b316b33ae4de83d36923039ef8cf48120266844285b/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:aed52fea0513bac0ccde438c188c8a471c4e0f457c2dd20cdbf6ea7a450046c7", size = 212316, upload-time = "2026-04-02T09:26:37.672Z" }, + { url = "https://files.pythonhosted.org/packages/fd/35/f7dba3994312d7ba508e041eaac39a36b120f32d4c8662b8814dab876431/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464", size = 227274, upload-time = "2026-04-02T09:26:38.93Z" }, + { url = "https://files.pythonhosted.org/packages/8a/2d/a572df5c9204ab7688ec1edc895a73ebded3b023bb07364710b05dd1c9be/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bb6d88045545b26da47aa879dd4a89a71d1dce0f0e549b1abcb31dfe4a8eac49", size = 218468, upload-time = "2026-04-02T09:26:40.17Z" }, + { url = "https://files.pythonhosted.org/packages/86/eb/890922a8b03a568ca2f336c36585a4713c55d4d67bf0f0c78924be6315ca/charset_normalizer-3.4.7-cp312-cp312-win32.whl", hash = "sha256:2257141f39fe65a3fdf38aeccae4b953e5f3b3324f4ff0daf9f15b8518666a2c", size = 148460, upload-time = "2026-04-02T09:26:41.416Z" }, + { url = "https://files.pythonhosted.org/packages/35/d9/0e7dffa06c5ab081f75b1b786f0aefc88365825dfcd0ac544bdb7b2b6853/charset_normalizer-3.4.7-cp312-cp312-win_amd64.whl", hash = "sha256:5ed6ab538499c8644b8a3e18debabcd7ce684f3fa91cf867521a7a0279cab2d6", size = 159330, upload-time = "2026-04-02T09:26:42.554Z" }, + { url = "https://files.pythonhosted.org/packages/9e/5d/481bcc2a7c88ea6b0878c299547843b2521ccbc40980cb406267088bc701/charset_normalizer-3.4.7-cp312-cp312-win_arm64.whl", hash = "sha256:56be790f86bfb2c98fb742ce566dfb4816e5a83384616ab59c49e0604d49c51d", size = 147828, upload-time = "2026-04-02T09:26:44.075Z" }, + { url = "https://files.pythonhosted.org/packages/c1/3b/66777e39d3ae1ddc77ee606be4ec6d8cbd4c801f65e5a1b6f2b11b8346dd/charset_normalizer-3.4.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f496c9c3cc02230093d8330875c4c3cdfc3b73612a5fd921c65d39cbcef08063", size = 309627, upload-time = "2026-04-02T09:26:45.198Z" }, + { url = "https://files.pythonhosted.org/packages/2e/4e/b7f84e617b4854ade48a1b7915c8ccfadeba444d2a18c291f696e37f0d3b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ea948db76d31190bf08bd371623927ee1339d5f2a0b4b1b4a4439a65298703c", size = 207008, upload-time = "2026-04-02T09:26:46.824Z" }, + { url = "https://files.pythonhosted.org/packages/c4/bb/ec73c0257c9e11b268f018f068f5d00aa0ef8c8b09f7753ebd5f2880e248/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a277ab8928b9f299723bc1a2dabb1265911b1a76341f90a510368ca44ad9ab66", size = 228303, upload-time = "2026-04-02T09:26:48.397Z" }, + { url = "https://files.pythonhosted.org/packages/85/fb/32d1f5033484494619f701e719429c69b766bfc4dbc61aa9e9c8c166528b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3bec022aec2c514d9cf199522a802bd007cd588ab17ab2525f20f9c34d067c18", size = 224282, upload-time = "2026-04-02T09:26:49.684Z" }, + { url = "https://files.pythonhosted.org/packages/fa/07/330e3a0dda4c404d6da83b327270906e9654a24f6c546dc886a0eb0ffb23/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e044c39e41b92c845bc815e5ae4230804e8e7bc29e399b0437d64222d92809dd", size = 215595, upload-time = "2026-04-02T09:26:50.915Z" }, + { url = "https://files.pythonhosted.org/packages/e3/7c/fc890655786e423f02556e0216d4b8c6bcb6bdfa890160dc66bf52dee468/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:f495a1652cf3fbab2eb0639776dad966c2fb874d79d87ca07f9d5f059b8bd215", size = 201986, upload-time = "2026-04-02T09:26:52.197Z" }, + { url = "https://files.pythonhosted.org/packages/d8/97/bfb18b3db2aed3b90cf54dc292ad79fdd5ad65c4eae454099475cbeadd0d/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e712b419df8ba5e42b226c510472b37bd57b38e897d3eca5e8cfd410a29fa859", size = 211711, upload-time = "2026-04-02T09:26:53.49Z" }, + { url = "https://files.pythonhosted.org/packages/6f/a5/a581c13798546a7fd557c82614a5c65a13df2157e9ad6373166d2a3e645d/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7804338df6fcc08105c7745f1502ba68d900f45fd770d5bdd5288ddccb8a42d8", size = 210036, upload-time = "2026-04-02T09:26:54.975Z" }, + { url = "https://files.pythonhosted.org/packages/8c/bf/b3ab5bcb478e4193d517644b0fb2bf5497fbceeaa7a1bc0f4d5b50953861/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:481551899c856c704d58119b5025793fa6730adda3571971af568f66d2424bb5", size = 202998, upload-time = "2026-04-02T09:26:56.303Z" }, + { url = "https://files.pythonhosted.org/packages/e7/4e/23efd79b65d314fa320ec6017b4b5834d5c12a58ba4610aa353af2e2f577/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:f59099f9b66f0d7145115e6f80dd8b1d847176df89b234a5a6b3f00437aa0832", size = 230056, upload-time = "2026-04-02T09:26:57.554Z" }, + { url = "https://files.pythonhosted.org/packages/b9/9f/1e1941bc3f0e01df116e68dc37a55c4d249df5e6fa77f008841aef68264f/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6", size = 211537, upload-time = "2026-04-02T09:26:58.843Z" }, + { url = "https://files.pythonhosted.org/packages/80/0f/088cbb3020d44428964a6c97fe1edfb1b9550396bf6d278330281e8b709c/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:3dedcc22d73ec993f42055eff4fcfed9318d1eeb9a6606c55892a26964964e48", size = 226176, upload-time = "2026-04-02T09:27:00.437Z" }, + { url = "https://files.pythonhosted.org/packages/6a/9f/130394f9bbe06f4f63e22641d32fc9b202b7e251c9aef4db044324dac493/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:64f02c6841d7d83f832cd97ccf8eb8a906d06eb95d5276069175c696b024b60a", size = 217723, upload-time = "2026-04-02T09:27:02.021Z" }, + { url = "https://files.pythonhosted.org/packages/73/55/c469897448a06e49f8fa03f6caae97074fde823f432a98f979cc42b90e69/charset_normalizer-3.4.7-cp313-cp313-win32.whl", hash = "sha256:4042d5c8f957e15221d423ba781e85d553722fc4113f523f2feb7b188cc34c5e", size = 148085, upload-time = "2026-04-02T09:27:03.192Z" }, + { url = "https://files.pythonhosted.org/packages/5d/78/1b74c5bbb3f99b77a1715c91b3e0b5bdb6fe302d95ace4f5b1bec37b0167/charset_normalizer-3.4.7-cp313-cp313-win_amd64.whl", hash = "sha256:3946fa46a0cf3e4c8cb1cc52f56bb536310d34f25f01ca9b6c16afa767dab110", size = 158819, upload-time = "2026-04-02T09:27:04.454Z" }, + { url = "https://files.pythonhosted.org/packages/68/86/46bd42279d323deb8687c4a5a811fd548cb7d1de10cf6535d099877a9a9f/charset_normalizer-3.4.7-cp313-cp313-win_arm64.whl", hash = "sha256:80d04837f55fc81da168b98de4f4b797ef007fc8a79ab71c6ec9bc4dd662b15b", size = 147915, upload-time = "2026-04-02T09:27:05.971Z" }, + { url = "https://files.pythonhosted.org/packages/97/c8/c67cb8c70e19ef1960b97b22ed2a1567711de46c4ddf19799923adc836c2/charset_normalizer-3.4.7-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:c36c333c39be2dbca264d7803333c896ab8fa7d4d6f0ab7edb7dfd7aea6e98c0", size = 309234, upload-time = "2026-04-02T09:27:07.194Z" }, + { url = "https://files.pythonhosted.org/packages/99/85/c091fdee33f20de70d6c8b522743b6f831a2f1cd3ff86de4c6a827c48a76/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1c2aed2e5e41f24ea8ef1590b8e848a79b56f3a5564a65ceec43c9d692dc7d8a", size = 208042, upload-time = "2026-04-02T09:27:08.749Z" }, + { url = "https://files.pythonhosted.org/packages/87/1c/ab2ce611b984d2fd5d86a5a8a19c1ae26acac6bad967da4967562c75114d/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:54523e136b8948060c0fa0bc7b1b50c32c186f2fceee897a495406bb6e311d2b", size = 228706, upload-time = "2026-04-02T09:27:09.951Z" }, + { url = "https://files.pythonhosted.org/packages/a8/29/2b1d2cb00bf085f59d29eb773ce58ec2d325430f8c216804a0a5cd83cbca/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:715479b9a2802ecac752a3b0efa2b0b60285cf962ee38414211abdfccc233b41", size = 224727, upload-time = "2026-04-02T09:27:11.175Z" }, + { url = "https://files.pythonhosted.org/packages/47/5c/032c2d5a07fe4d4855fea851209cca2b6f03ebeb6d4e3afdb3358386a684/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bd6c2a1c7573c64738d716488d2cdd3c00e340e4835707d8fdb8dc1a66ef164e", size = 215882, upload-time = "2026-04-02T09:27:12.446Z" }, + { url = "https://files.pythonhosted.org/packages/2c/c2/356065d5a8b78ed04499cae5f339f091946a6a74f91e03476c33f0ab7100/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:c45e9440fb78f8ddabcf714b68f936737a121355bf59f3907f4e17721b9d1aae", size = 200860, upload-time = "2026-04-02T09:27:13.721Z" }, + { url = "https://files.pythonhosted.org/packages/0c/cd/a32a84217ced5039f53b29f460962abb2d4420def55afabe45b1c3c7483d/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3534e7dcbdcf757da6b85a0bbf5b6868786d5982dd959b065e65481644817a18", size = 211564, upload-time = "2026-04-02T09:27:15.272Z" }, + { url = "https://files.pythonhosted.org/packages/44/86/58e6f13ce26cc3b8f4a36b94a0f22ae2f00a72534520f4ae6857c4b81f89/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e8ac484bf18ce6975760921bb6148041faa8fef0547200386ea0b52b5d27bf7b", size = 211276, upload-time = "2026-04-02T09:27:16.834Z" }, + { url = "https://files.pythonhosted.org/packages/8f/fe/d17c32dc72e17e155e06883efa84514ca375f8a528ba2546bee73fc4df81/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:a5fe03b42827c13cdccd08e6c0247b6a6d4b5e3cdc53fd1749f5896adcdc2356", size = 201238, upload-time = "2026-04-02T09:27:18.229Z" }, + { url = "https://files.pythonhosted.org/packages/6a/29/f33daa50b06525a237451cdb6c69da366c381a3dadcd833fa5676bc468b3/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:2d6eb928e13016cea4f1f21d1e10c1cebd5a421bc57ddf5b1142ae3f86824fab", size = 230189, upload-time = "2026-04-02T09:27:19.445Z" }, + { url = "https://files.pythonhosted.org/packages/b6/6e/52c84015394a6a0bdcd435210a7e944c5f94ea1055f5cc5d56c5fe368e7b/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e74327fb75de8986940def6e8dee4f127cc9752bee7355bb323cc5b2659b6d46", size = 211352, upload-time = "2026-04-02T09:27:20.79Z" }, + { url = "https://files.pythonhosted.org/packages/8c/d7/4353be581b373033fb9198bf1da3cf8f09c1082561e8e922aa7b39bf9fe8/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d6038d37043bced98a66e68d3aa2b6a35505dc01328cd65217cefe82f25def44", size = 227024, upload-time = "2026-04-02T09:27:22.063Z" }, + { url = "https://files.pythonhosted.org/packages/30/45/99d18aa925bd1740098ccd3060e238e21115fffbfdcb8f3ece837d0ace6c/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:7579e913a5339fb8fa133f6bbcfd8e6749696206cf05acdbdca71a1b436d8e72", size = 217869, upload-time = "2026-04-02T09:27:23.486Z" }, + { url = "https://files.pythonhosted.org/packages/5c/05/5ee478aa53f4bb7996482153d4bfe1b89e0f087f0ab6b294fcf92d595873/charset_normalizer-3.4.7-cp314-cp314-win32.whl", hash = "sha256:5b77459df20e08151cd6f8b9ef8ef1f961ef73d85c21a555c7eed5b79410ec10", size = 148541, upload-time = "2026-04-02T09:27:25.146Z" }, + { url = "https://files.pythonhosted.org/packages/48/77/72dcb0921b2ce86420b2d79d454c7022bf5be40202a2a07906b9f2a35c97/charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl", hash = "sha256:92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f", size = 159634, upload-time = "2026-04-02T09:27:26.642Z" }, + { url = "https://files.pythonhosted.org/packages/c6/a3/c2369911cd72f02386e4e340770f6e158c7980267da16af8f668217abaa0/charset_normalizer-3.4.7-cp314-cp314-win_arm64.whl", hash = "sha256:67f6279d125ca0046a7fd386d01b311c6363844deac3e5b069b514ba3e63c246", size = 148384, upload-time = "2026-04-02T09:27:28.271Z" }, + { url = "https://files.pythonhosted.org/packages/94/09/7e8a7f73d24dba1f0035fbbf014d2c36828fc1bf9c88f84093e57d315935/charset_normalizer-3.4.7-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:effc3f449787117233702311a1b7d8f59cba9ced946ba727bdc329ec69028e24", size = 330133, upload-time = "2026-04-02T09:27:29.474Z" }, + { url = "https://files.pythonhosted.org/packages/8d/da/96975ddb11f8e977f706f45cddd8540fd8242f71ecdb5d18a80723dcf62c/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79", size = 216257, upload-time = "2026-04-02T09:27:30.793Z" }, + { url = "https://files.pythonhosted.org/packages/e5/e8/1d63bf8ef2d388e95c64b2098f45f84758f6d102a087552da1485912637b/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:733784b6d6def852c814bce5f318d25da2ee65dd4839a0718641c696e09a2960", size = 234851, upload-time = "2026-04-02T09:27:32.44Z" }, + { url = "https://files.pythonhosted.org/packages/9b/40/e5ff04233e70da2681fa43969ad6f66ca5611d7e669be0246c4c7aaf6dc8/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a89c23ef8d2c6b27fd200a42aa4ac72786e7c60d40efdc76e6011260b6e949c4", size = 233393, upload-time = "2026-04-02T09:27:34.03Z" }, + { url = "https://files.pythonhosted.org/packages/be/c1/06c6c49d5a5450f76899992f1ee40b41d076aee9279b49cf9974d2f313d5/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c114670c45346afedc0d947faf3c7f701051d2518b943679c8ff88befe14f8e", size = 223251, upload-time = "2026-04-02T09:27:35.369Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9f/f2ff16fb050946169e3e1f82134d107e5d4ae72647ec8a1b1446c148480f/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:a180c5e59792af262bf263b21a3c49353f25945d8d9f70628e73de370d55e1e1", size = 206609, upload-time = "2026-04-02T09:27:36.661Z" }, + { url = "https://files.pythonhosted.org/packages/69/d5/a527c0cd8d64d2eab7459784fb4169a0ac76e5a6fc5237337982fd61347e/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3c9a494bc5ec77d43cea229c4f6db1e4d8fe7e1bbffa8b6f0f0032430ff8ab44", size = 220014, upload-time = "2026-04-02T09:27:38.019Z" }, + { url = "https://files.pythonhosted.org/packages/7e/80/8a7b8104a3e203074dc9aa2c613d4b726c0e136bad1cc734594b02867972/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8d828b6667a32a728a1ad1d93957cdf37489c57b97ae6c4de2860fa749b8fc1e", size = 218979, upload-time = "2026-04-02T09:27:39.37Z" }, + { url = "https://files.pythonhosted.org/packages/02/9a/b759b503d507f375b2b5c153e4d2ee0a75aa215b7f2489cf314f4541f2c0/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cf1493cd8607bec4d8a7b9b004e699fcf8f9103a9284cc94962cb73d20f9d4a3", size = 209238, upload-time = "2026-04-02T09:27:40.722Z" }, + { url = "https://files.pythonhosted.org/packages/c2/4e/0f3f5d47b86bdb79256e7290b26ac847a2832d9a4033f7eb2cd4bcf4bb5b/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0c96c3b819b5c3e9e165495db84d41914d6894d55181d2d108cc1a69bfc9cce0", size = 236110, upload-time = "2026-04-02T09:27:42.33Z" }, + { url = "https://files.pythonhosted.org/packages/96/23/bce28734eb3ed2c91dcf93abeb8a5cf393a7b2749725030bb630e554fdd8/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:752a45dc4a6934060b3b0dab47e04edc3326575f82be64bc4fc293914566503e", size = 219824, upload-time = "2026-04-02T09:27:43.924Z" }, + { url = "https://files.pythonhosted.org/packages/2c/6f/6e897c6984cc4d41af319b077f2f600fc8214eb2fe2d6bcb79141b882400/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:8778f0c7a52e56f75d12dae53ae320fae900a8b9b4164b981b9c5ce059cd1fcb", size = 233103, upload-time = "2026-04-02T09:27:45.348Z" }, + { url = "https://files.pythonhosted.org/packages/76/22/ef7bd0fe480a0ae9b656189ec00744b60933f68b4f42a7bb06589f6f576a/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ce3412fbe1e31eb81ea42f4169ed94861c56e643189e1e75f0041f3fe7020abe", size = 225194, upload-time = "2026-04-02T09:27:46.706Z" }, + { url = "https://files.pythonhosted.org/packages/c5/a7/0e0ab3e0b5bc1219bd80a6a0d4d72ca74d9250cb2382b7c699c147e06017/charset_normalizer-3.4.7-cp314-cp314t-win32.whl", hash = "sha256:c03a41a8784091e67a39648f70c5f97b5b6a37f216896d44d2cdcb82615339a0", size = 159827, upload-time = "2026-04-02T09:27:48.053Z" }, + { url = "https://files.pythonhosted.org/packages/7a/1d/29d32e0fb40864b1f878c7f5a0b343ae676c6e2b271a2d55cc3a152391da/charset_normalizer-3.4.7-cp314-cp314t-win_amd64.whl", hash = "sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c", size = 174168, upload-time = "2026-04-02T09:27:49.795Z" }, + { url = "https://files.pythonhosted.org/packages/de/32/d92444ad05c7a6e41fb2036749777c163baf7a0301a040cb672d6b2b1ae9/charset_normalizer-3.4.7-cp314-cp314t-win_arm64.whl", hash = "sha256:c35abb8bfff0185efac5878da64c45dafd2b37fb0383add1be155a763c1f083d", size = 153018, upload-time = "2026-04-02T09:27:51.116Z" }, + { url = "https://files.pythonhosted.org/packages/db/8f/61959034484a4a7c527811f4721e75d02d653a35afb0b6054474d8185d4c/charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d", size = 61958, upload-time = "2026-04-02T09:28:37.794Z" }, +] + +[[package]] +name = "claude-agent-sdk" +version = "0.1.56" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "mcp" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ee/c1/c7afb1a08cecef0644693ccd9975651e45cf23a50272b94b6eca2c1a7dc8/claude_agent_sdk-0.1.56.tar.gz", hash = "sha256:a95bc14e59f9d6c8e7fa2e6581008a3f24f10e1b57302719823f62cfb5beccdc", size = 121659, upload-time = "2026-04-04T00:56:30.512Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b0/73/4e3d13c4d43614de35a113c87ec96b3db605baa23f9f5c4a38536837e18e/claude_agent_sdk-0.1.56-py3-none-macosx_11_0_arm64.whl", hash = "sha256:9f5a7c87617101e6bb0f23408104ac6f40f9b5adec91dcfe5b8de5f65a7df73a", size = 58585662, upload-time = "2026-04-04T00:56:34.935Z" }, + { url = "https://files.pythonhosted.org/packages/1e/6d/78347c2efa1526f1f6e7edecabe636575f622bcaa7921965457f95dd12dc/claude_agent_sdk-0.1.56-py3-none-macosx_11_0_x86_64.whl", hash = "sha256:824f4a10340f46dd26fee8e74aeed4fc64fec95084e327ab1ebb6058b349e1c3", size = 60419564, upload-time = "2026-04-04T00:56:39.64Z" }, + { url = "https://files.pythonhosted.org/packages/87/c1/708262318926c8393d494a5dcaafd9bc7d6ba547c0a5fad4eff5f9aa0ecd/claude_agent_sdk-0.1.56-py3-none-manylinux_2_17_aarch64.whl", hash = "sha256:ff60dedc06b62b52e5937a9a2c4b0ec4ad0dd6764c20be656d01aeb8b11fba1d", size = 71893844, upload-time = "2026-04-04T00:56:44.402Z" }, + { url = "https://files.pythonhosted.org/packages/1b/4f/24918a596b0d61c3a691af2a9ee52b8c54f1769ce2c5fef1d64350056e53/claude_agent_sdk-0.1.56-py3-none-manylinux_2_17_x86_64.whl", hash = "sha256:fe866b2119f69e99d9637acc27b588670c610fed1c4a096287874db5744d029b", size = 72030943, upload-time = "2026-04-04T00:56:49.892Z" }, + { url = "https://files.pythonhosted.org/packages/f3/d8/5ded242e55f0b5f295d4ee2cbe5ae3bca914eb0a2a291f81e38b68d3ef58/claude_agent_sdk-0.1.56-py3-none-win_amd64.whl", hash = "sha256:5934e082e1ccf975d65cd7412f2eaf2c5ffa6b9019a2ca2a9fb228310df7ddc8", size = 74141451, upload-time = "2026-04-04T00:56:57.683Z" }, +] + +[[package]] +name = "click" +version = "8.3.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/57/75/31212c6bf2503fdf920d87fee5d7a86a2e3bcf444984126f13d8e4016804/click-8.3.2.tar.gz", hash = "sha256:14162b8b3b3550a7d479eafa77dfd3c38d9dc8951f6f69c78913a8f9a7540fd5", size = 302856, upload-time = "2026-04-03T19:14:45.118Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e4/20/71885d8b97d4f3dde17b1fdb92dbd4908b00541c5a3379787137285f602e/click-8.3.2-py3-none-any.whl", hash = "sha256:1924d2c27c5653561cd2cae4548d1406039cb79b858b747cfea24924bbc1616d", size = 108379, upload-time = "2026-04-03T19:14:43.505Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "courlan" +version = "1.3.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "babel" }, + { name = "tld" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/6f/54/6d6ceeff4bed42e7a10d6064d35ee43a810e7b3e8beb4abeae8cff4713ae/courlan-1.3.2.tar.gz", hash = "sha256:0b66f4db3a9c39a6e22dd247c72cfaa57d68ea660e94bb2c84ec7db8712af190", size = 206382, upload-time = "2024-10-29T16:40:20.994Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8e/ca/6a667ccbe649856dcd3458bab80b016681b274399d6211187c6ab969fc50/courlan-1.3.2-py3-none-any.whl", hash = "sha256:d0dab52cf5b5b1000ee2839fbc2837e93b2514d3cb5bb61ae158a55b7a04c6be", size = 33848, upload-time = "2024-10-29T16:40:18.325Z" }, +] + +[[package]] +name = "cryptography" +version = "46.0.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cffi", marker = "platform_python_implementation != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a4/ba/04b1bd4218cbc58dc90ce967106d51582371b898690f3ae0402876cc4f34/cryptography-46.0.6.tar.gz", hash = "sha256:27550628a518c5c6c903d84f637fbecf287f6cb9ced3804838a1295dc1fd0759", size = 750542, upload-time = "2026-03-25T23:34:53.396Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/47/23/9285e15e3bc57325b0a72e592921983a701efc1ee8f91c06c5f0235d86d9/cryptography-46.0.6-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:64235194bad039a10bb6d2d930ab3323baaec67e2ce36215fd0952fad0930ca8", size = 7176401, upload-time = "2026-03-25T23:33:22.096Z" }, + { url = "https://files.pythonhosted.org/packages/60/f8/e61f8f13950ab6195b31913b42d39f0f9afc7d93f76710f299b5ec286ae6/cryptography-46.0.6-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:26031f1e5ca62fcb9d1fcb34b2b60b390d1aacaa15dc8b895a9ed00968b97b30", size = 4275275, upload-time = "2026-03-25T23:33:23.844Z" }, + { url = "https://files.pythonhosted.org/packages/19/69/732a736d12c2631e140be2348b4ad3d226302df63ef64d30dfdb8db7ad1c/cryptography-46.0.6-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9a693028b9cbe51b5a1136232ee8f2bc242e4e19d456ded3fa7c86e43c713b4a", size = 4425320, upload-time = "2026-03-25T23:33:25.703Z" }, + { url = "https://files.pythonhosted.org/packages/d4/12/123be7292674abf76b21ac1fc0e1af50661f0e5b8f0ec8285faac18eb99e/cryptography-46.0.6-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:67177e8a9f421aa2d3a170c3e56eca4e0128883cf52a071a7cbf53297f18b175", size = 4278082, upload-time = "2026-03-25T23:33:27.423Z" }, + { url = "https://files.pythonhosted.org/packages/5b/ba/d5e27f8d68c24951b0a484924a84c7cdaed7502bac9f18601cd357f8b1d2/cryptography-46.0.6-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:d9528b535a6c4f8ff37847144b8986a9a143585f0540fbcb1a98115b543aa463", size = 4926514, upload-time = "2026-03-25T23:33:29.206Z" }, + { url = "https://files.pythonhosted.org/packages/34/71/1ea5a7352ae516d5512d17babe7e1b87d9db5150b21f794b1377eac1edc0/cryptography-46.0.6-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:22259338084d6ae497a19bae5d4c66b7ca1387d3264d1c2c0e72d9e9b6a77b97", size = 4457766, upload-time = "2026-03-25T23:33:30.834Z" }, + { url = "https://files.pythonhosted.org/packages/01/59/562be1e653accee4fdad92c7a2e88fced26b3fdfce144047519bbebc299e/cryptography-46.0.6-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:760997a4b950ff00d418398ad73fbc91aa2894b5c1db7ccb45b4f68b42a63b3c", size = 3986535, upload-time = "2026-03-25T23:33:33.02Z" }, + { url = "https://files.pythonhosted.org/packages/d6/8b/b1ebfeb788bf4624d36e45ed2662b8bd43a05ff62157093c1539c1288a18/cryptography-46.0.6-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:3dfa6567f2e9e4c5dceb8ccb5a708158a2a871052fa75c8b78cb0977063f1507", size = 4277618, upload-time = "2026-03-25T23:33:34.567Z" }, + { url = "https://files.pythonhosted.org/packages/dd/52/a005f8eabdb28df57c20f84c44d397a755782d6ff6d455f05baa2785bd91/cryptography-46.0.6-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:cdcd3edcbc5d55757e5f5f3d330dd00007ae463a7e7aa5bf132d1f22a4b62b19", size = 4890802, upload-time = "2026-03-25T23:33:37.034Z" }, + { url = "https://files.pythonhosted.org/packages/ec/4d/8e7d7245c79c617d08724e2efa397737715ca0ec830ecb3c91e547302555/cryptography-46.0.6-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:d4e4aadb7fc1f88687f47ca20bb7227981b03afaae69287029da08096853b738", size = 4457425, upload-time = "2026-03-25T23:33:38.904Z" }, + { url = "https://files.pythonhosted.org/packages/1d/5c/f6c3596a1430cec6f949085f0e1a970638d76f81c3ea56d93d564d04c340/cryptography-46.0.6-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:2b417edbe8877cda9022dde3a008e2deb50be9c407eef034aeeb3a8b11d9db3c", size = 4405530, upload-time = "2026-03-25T23:33:40.842Z" }, + { url = "https://files.pythonhosted.org/packages/7e/c9/9f9cea13ee2dbde070424e0c4f621c091a91ffcc504ffea5e74f0e1daeff/cryptography-46.0.6-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:380343e0653b1c9d7e1f55b52aaa2dbb2fdf2730088d48c43ca1c7c0abb7cc2f", size = 4667896, upload-time = "2026-03-25T23:33:42.781Z" }, + { url = "https://files.pythonhosted.org/packages/ad/b5/1895bc0821226f129bc74d00eccfc6a5969e2028f8617c09790bf89c185e/cryptography-46.0.6-cp311-abi3-win32.whl", hash = "sha256:bcb87663e1f7b075e48c3be3ecb5f0b46c8fc50b50a97cf264e7f60242dca3f2", size = 3026348, upload-time = "2026-03-25T23:33:45.021Z" }, + { url = "https://files.pythonhosted.org/packages/c3/f8/c9bcbf0d3e6ad288b9d9aa0b1dee04b063d19e8c4f871855a03ab3a297ab/cryptography-46.0.6-cp311-abi3-win_amd64.whl", hash = "sha256:6739d56300662c468fddb0e5e291f9b4d084bead381667b9e654c7dd81705124", size = 3483896, upload-time = "2026-03-25T23:33:46.649Z" }, + { url = "https://files.pythonhosted.org/packages/01/41/3a578f7fd5c70611c0aacba52cd13cb364a5dee895a5c1d467208a9380b0/cryptography-46.0.6-cp314-cp314t-macosx_10_9_universal2.whl", hash = "sha256:2ef9e69886cbb137c2aef9772c2e7138dc581fad4fcbcf13cc181eb5a3ab6275", size = 7117147, upload-time = "2026-03-25T23:33:48.249Z" }, + { url = "https://files.pythonhosted.org/packages/fa/87/887f35a6fca9dde90cad08e0de0c89263a8e59b2d2ff904fd9fcd8025b6f/cryptography-46.0.6-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7f417f034f91dcec1cb6c5c35b07cdbb2ef262557f701b4ecd803ee8cefed4f4", size = 4266221, upload-time = "2026-03-25T23:33:49.874Z" }, + { url = "https://files.pythonhosted.org/packages/aa/a8/0a90c4f0b0871e0e3d1ed126aed101328a8a57fd9fd17f00fb67e82a51ca/cryptography-46.0.6-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d24c13369e856b94892a89ddf70b332e0b70ad4a5c43cf3e9cb71d6d7ffa1f7b", size = 4408952, upload-time = "2026-03-25T23:33:52.128Z" }, + { url = "https://files.pythonhosted.org/packages/16/0b/b239701eb946523e4e9f329336e4ff32b1247e109cbab32d1a7b61da8ed7/cryptography-46.0.6-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:aad75154a7ac9039936d50cf431719a2f8d4ed3d3c277ac03f3339ded1a5e707", size = 4270141, upload-time = "2026-03-25T23:33:54.11Z" }, + { url = "https://files.pythonhosted.org/packages/0f/a8/976acdd4f0f30df7b25605f4b9d3d89295351665c2091d18224f7ad5cdbf/cryptography-46.0.6-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:3c21d92ed15e9cfc6eb64c1f5a0326db22ca9c2566ca46d845119b45b4400361", size = 4904178, upload-time = "2026-03-25T23:33:55.725Z" }, + { url = "https://files.pythonhosted.org/packages/b1/1b/bf0e01a88efd0e59679b69f42d4afd5bced8700bb5e80617b2d63a3741af/cryptography-46.0.6-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:4668298aef7cddeaf5c6ecc244c2302a2b8e40f384255505c22875eebb47888b", size = 4441812, upload-time = "2026-03-25T23:33:57.364Z" }, + { url = "https://files.pythonhosted.org/packages/bb/8b/11df86de2ea389c65aa1806f331cae145f2ed18011f30234cc10ca253de8/cryptography-46.0.6-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:8ce35b77aaf02f3b59c90b2c8a05c73bac12cea5b4e8f3fbece1f5fddea5f0ca", size = 3963923, upload-time = "2026-03-25T23:33:59.361Z" }, + { url = "https://files.pythonhosted.org/packages/91/e0/207fb177c3a9ef6a8108f234208c3e9e76a6aa8cf20d51932916bd43bda0/cryptography-46.0.6-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:c89eb37fae9216985d8734c1afd172ba4927f5a05cfd9bf0e4863c6d5465b013", size = 4269695, upload-time = "2026-03-25T23:34:00.909Z" }, + { url = "https://files.pythonhosted.org/packages/21/5e/19f3260ed1e95bced52ace7501fabcd266df67077eeb382b79c81729d2d3/cryptography-46.0.6-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:ed418c37d095aeddf5336898a132fba01091f0ac5844e3e8018506f014b6d2c4", size = 4869785, upload-time = "2026-03-25T23:34:02.796Z" }, + { url = "https://files.pythonhosted.org/packages/10/38/cd7864d79aa1d92ef6f1a584281433419b955ad5a5ba8d1eb6c872165bcb/cryptography-46.0.6-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:69cf0056d6947edc6e6760e5f17afe4bea06b56a9ac8a06de9d2bd6b532d4f3a", size = 4441404, upload-time = "2026-03-25T23:34:04.35Z" }, + { url = "https://files.pythonhosted.org/packages/09/0a/4fe7a8d25fed74419f91835cf5829ade6408fd1963c9eae9c4bce390ecbb/cryptography-46.0.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8e7304c4f4e9490e11efe56af6713983460ee0780f16c63f219984dab3af9d2d", size = 4397549, upload-time = "2026-03-25T23:34:06.342Z" }, + { url = "https://files.pythonhosted.org/packages/5f/a0/7d738944eac6513cd60a8da98b65951f4a3b279b93479a7e8926d9cd730b/cryptography-46.0.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b928a3ca837c77a10e81a814a693f2295200adb3352395fad024559b7be7a736", size = 4651874, upload-time = "2026-03-25T23:34:07.916Z" }, + { url = "https://files.pythonhosted.org/packages/cb/f1/c2326781ca05208845efca38bf714f76939ae446cd492d7613808badedf1/cryptography-46.0.6-cp314-cp314t-win32.whl", hash = "sha256:97c8115b27e19e592a05c45d0dd89c57f81f841cc9880e353e0d3bf25b2139ed", size = 3001511, upload-time = "2026-03-25T23:34:09.892Z" }, + { url = "https://files.pythonhosted.org/packages/c9/57/fe4a23eb549ac9d903bd4698ffda13383808ef0876cc912bcb2838799ece/cryptography-46.0.6-cp314-cp314t-win_amd64.whl", hash = "sha256:c797e2517cb7880f8297e2c0f43bb910e91381339336f75d2c1c2cbf811b70b4", size = 3471692, upload-time = "2026-03-25T23:34:11.613Z" }, + { url = "https://files.pythonhosted.org/packages/c4/cc/f330e982852403da79008552de9906804568ae9230da8432f7496ce02b71/cryptography-46.0.6-cp38-abi3-macosx_10_9_universal2.whl", hash = "sha256:12cae594e9473bca1a7aceb90536060643128bb274fcea0fc459ab90f7d1ae7a", size = 7162776, upload-time = "2026-03-25T23:34:13.308Z" }, + { url = "https://files.pythonhosted.org/packages/49/b3/dc27efd8dcc4bff583b3f01d4a3943cd8b5821777a58b3a6a5f054d61b79/cryptography-46.0.6-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:639301950939d844a9e1c4464d7e07f902fe9a7f6b215bb0d4f28584729935d8", size = 4270529, upload-time = "2026-03-25T23:34:15.019Z" }, + { url = "https://files.pythonhosted.org/packages/e6/05/e8d0e6eb4f0d83365b3cb0e00eb3c484f7348db0266652ccd84632a3d58d/cryptography-46.0.6-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ed3775295fb91f70b4027aeba878d79b3e55c0b3e97eaa4de71f8f23a9f2eb77", size = 4414827, upload-time = "2026-03-25T23:34:16.604Z" }, + { url = "https://files.pythonhosted.org/packages/2f/97/daba0f5d2dc6d855e2dcb70733c812558a7977a55dd4a6722756628c44d1/cryptography-46.0.6-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:8927ccfbe967c7df312ade694f987e7e9e22b2425976ddbf28271d7e58845290", size = 4271265, upload-time = "2026-03-25T23:34:18.586Z" }, + { url = "https://files.pythonhosted.org/packages/89/06/fe1fce39a37ac452e58d04b43b0855261dac320a2ebf8f5260dd55b201a9/cryptography-46.0.6-cp38-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:b12c6b1e1651e42ab5de8b1e00dc3b6354fdfd778e7fa60541ddacc27cd21410", size = 4916800, upload-time = "2026-03-25T23:34:20.561Z" }, + { url = "https://files.pythonhosted.org/packages/ff/8a/b14f3101fe9c3592603339eb5d94046c3ce5f7fc76d6512a2d40efd9724e/cryptography-46.0.6-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:063b67749f338ca9c5a0b7fe438a52c25f9526b851e24e6c9310e7195aad3b4d", size = 4448771, upload-time = "2026-03-25T23:34:22.406Z" }, + { url = "https://files.pythonhosted.org/packages/01/b3/0796998056a66d1973fd52ee89dc1bb3b6581960a91ad4ac705f182d398f/cryptography-46.0.6-cp38-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:02fad249cb0e090b574e30b276a3da6a149e04ee2f049725b1f69e7b8351ec70", size = 3978333, upload-time = "2026-03-25T23:34:24.281Z" }, + { url = "https://files.pythonhosted.org/packages/c5/3d/db200af5a4ffd08918cd55c08399dc6c9c50b0bc72c00a3246e099d3a849/cryptography-46.0.6-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:7e6142674f2a9291463e5e150090b95a8519b2fb6e6aaec8917dd8d094ce750d", size = 4271069, upload-time = "2026-03-25T23:34:25.895Z" }, + { url = "https://files.pythonhosted.org/packages/d7/18/61acfd5b414309d74ee838be321c636fe71815436f53c9f0334bf19064fa/cryptography-46.0.6-cp38-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:456b3215172aeefb9284550b162801d62f5f264a081049a3e94307fe20792cfa", size = 4878358, upload-time = "2026-03-25T23:34:27.67Z" }, + { url = "https://files.pythonhosted.org/packages/8b/65/5bf43286d566f8171917cae23ac6add941654ccf085d739195a4eacf1674/cryptography-46.0.6-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:341359d6c9e68834e204ceaf25936dffeafea3829ab80e9503860dcc4f4dac58", size = 4448061, upload-time = "2026-03-25T23:34:29.375Z" }, + { url = "https://files.pythonhosted.org/packages/e0/25/7e49c0fa7205cf3597e525d156a6bce5b5c9de1fd7e8cb01120e459f205a/cryptography-46.0.6-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:9a9c42a2723999a710445bc0d974e345c32adfd8d2fac6d8a251fa829ad31cfb", size = 4399103, upload-time = "2026-03-25T23:34:32.036Z" }, + { url = "https://files.pythonhosted.org/packages/44/46/466269e833f1c4718d6cd496ffe20c56c9c8d013486ff66b4f69c302a68d/cryptography-46.0.6-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6617f67b1606dfd9fe4dbfa354a9508d4a6d37afe30306fe6c101b7ce3274b72", size = 4659255, upload-time = "2026-03-25T23:34:33.679Z" }, + { url = "https://files.pythonhosted.org/packages/0a/09/ddc5f630cc32287d2c953fc5d32705e63ec73e37308e5120955316f53827/cryptography-46.0.6-cp38-abi3-win32.whl", hash = "sha256:7f6690b6c55e9c5332c0b59b9c8a3fb232ebf059094c17f9019a51e9827df91c", size = 3010660, upload-time = "2026-03-25T23:34:35.418Z" }, + { url = "https://files.pythonhosted.org/packages/1b/82/ca4893968aeb2709aacfb57a30dec6fa2ab25b10fa9f064b8882ce33f599/cryptography-46.0.6-cp38-abi3-win_amd64.whl", hash = "sha256:79e865c642cfc5c0b3eb12af83c35c5aeff4fa5c672dc28c43721c2c9fdd2f0f", size = 3471160, upload-time = "2026-03-25T23:34:37.191Z" }, + { url = "https://files.pythonhosted.org/packages/2e/84/7ccff00ced5bac74b775ce0beb7d1be4e8637536b522b5df9b73ada42da2/cryptography-46.0.6-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:2ea0f37e9a9cf0df2952893ad145fd9627d326a59daec9b0802480fa3bcd2ead", size = 3475444, upload-time = "2026-03-25T23:34:38.944Z" }, + { url = "https://files.pythonhosted.org/packages/bc/1f/4c926f50df7749f000f20eede0c896769509895e2648db5da0ed55db711d/cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:a3e84d5ec9ba01f8fd03802b2147ba77f0c8f2617b2aff254cedd551844209c8", size = 4218227, upload-time = "2026-03-25T23:34:40.871Z" }, + { url = "https://files.pythonhosted.org/packages/c6/65/707be3ffbd5f786028665c3223e86e11c4cda86023adbc56bd72b1b6bab5/cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:12f0fa16cc247b13c43d56d7b35287ff1569b5b1f4c5e87e92cc4fcc00cd10c0", size = 4381399, upload-time = "2026-03-25T23:34:42.609Z" }, + { url = "https://files.pythonhosted.org/packages/f3/6d/73557ed0ef7d73d04d9aba745d2c8e95218213687ee5e76b7d236a5030fc/cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:50575a76e2951fe7dbd1f56d181f8c5ceeeb075e9ff88e7ad997d2f42af06e7b", size = 4217595, upload-time = "2026-03-25T23:34:44.205Z" }, + { url = "https://files.pythonhosted.org/packages/9e/c5/e1594c4eec66a567c3ac4400008108a415808be2ce13dcb9a9045c92f1a0/cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:90e5f0a7b3be5f40c3a0a0eafb32c681d8d2c181fc2a1bdabe9b3f611d9f6b1a", size = 4380912, upload-time = "2026-03-25T23:34:46.328Z" }, + { url = "https://files.pythonhosted.org/packages/1a/89/843b53614b47f97fe1abc13f9a86efa5ec9e275292c457af1d4a60dc80e0/cryptography-46.0.6-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:6728c49e3b2c180ef26f8e9f0a883a2c585638db64cf265b49c9ba10652d430e", size = 3409955, upload-time = "2026-03-25T23:34:48.465Z" }, +] + +[[package]] +name = "dateparser" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "python-dateutil" }, + { name = "pytz" }, + { name = "regex" }, + { name = "tzlocal" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/46/2d/a0ccdb78788064fa0dc901b8524e50615c42be1d78b78d646d0b28d09180/dateparser-1.4.0.tar.gz", hash = "sha256:97a21840d5ecdf7630c584f673338a5afac5dfe84f647baf4d7e8df98f9354a4", size = 321512, upload-time = "2026-03-26T09:56:10.292Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b4/0b/3c3bb7cbe757279e693a0be6049048012f794d01f81099609ecd53b899f0/dateparser-1.4.0-py3-none-any.whl", hash = "sha256:7902b8e85d603494bf70a5a0b1decdddb2270b9c6e6b2bc8a57b93476c0df378", size = 300379, upload-time = "2026-03-26T09:56:08.409Z" }, +] + +[[package]] +name = "distro" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz", hash = "sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed", size = 60722, upload-time = "2023-12-24T09:54:32.31Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" }, +] + +[[package]] +name = "fastapi" +version = "0.139.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d3/af/a5f50ccfa659ec1802cb4ca842c23f06d906a8cc9aef6016a2caeea3d4ed/fastapi-0.139.0.tar.gz", hash = "sha256:99ab7b2d92223c76d6cf10757ab3f89d45b38267fc20b2a136cf02f6beac3145", size = 423016, upload-time = "2026-07-01T16:35:33.436Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/7c/8e3c6ad324ea5cb36604fc3f968554887891c316d9dfde57761611d907ad/fastapi-0.139.0-py3-none-any.whl", hash = "sha256:cf15e1e9e667ddb0ad63811e60bd11390d1aac838ca4a7a23f421807b2308189", size = 130339, upload-time = "2026-07-01T16:35:32.19Z" }, +] + +[[package]] +name = "fastuuid" +version = "0.14.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/7d/d9daedf0f2ebcacd20d599928f8913e9d2aea1d56d2d355a93bfa2b611d7/fastuuid-0.14.0.tar.gz", hash = "sha256:178947fc2f995b38497a74172adee64fdeb8b7ec18f2a5934d037641ba265d26", size = 18232, upload-time = "2025-10-19T22:19:22.402Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/98/f3/12481bda4e5b6d3e698fbf525df4443cc7dce746f246b86b6fcb2fba1844/fastuuid-0.14.0-cp311-cp311-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:73946cb950c8caf65127d4e9a325e2b6be0442a224fd51ba3b6ac44e1912ce34", size = 516386, upload-time = "2025-10-19T22:42:40.176Z" }, + { url = "https://files.pythonhosted.org/packages/59/19/2fc58a1446e4d72b655648eb0879b04e88ed6fa70d474efcf550f640f6ec/fastuuid-0.14.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:12ac85024637586a5b69645e7ed986f7535106ed3013640a393a03e461740cb7", size = 264569, upload-time = "2025-10-19T22:25:50.977Z" }, + { url = "https://files.pythonhosted.org/packages/78/29/3c74756e5b02c40cfcc8b1d8b5bac4edbd532b55917a6bcc9113550e99d1/fastuuid-0.14.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:05a8dde1f395e0c9b4be515b7a521403d1e8349443e7641761af07c7ad1624b1", size = 254366, upload-time = "2025-10-19T22:29:49.166Z" }, + { url = "https://files.pythonhosted.org/packages/52/96/d761da3fccfa84f0f353ce6e3eb8b7f76b3aa21fd25e1b00a19f9c80a063/fastuuid-0.14.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:09378a05020e3e4883dfdab438926f31fea15fd17604908f3d39cbeb22a0b4dc", size = 278978, upload-time = "2025-10-19T22:35:41.306Z" }, + { url = "https://files.pythonhosted.org/packages/fc/c2/f84c90167cc7765cb82b3ff7808057608b21c14a38531845d933a4637307/fastuuid-0.14.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bbb0c4b15d66b435d2538f3827f05e44e2baafcc003dd7d8472dc67807ab8fd8", size = 279692, upload-time = "2025-10-19T22:25:36.997Z" }, + { url = "https://files.pythonhosted.org/packages/af/7b/4bacd03897b88c12348e7bd77943bac32ccf80ff98100598fcff74f75f2e/fastuuid-0.14.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:cd5a7f648d4365b41dbf0e38fe8da4884e57bed4e77c83598e076ac0c93995e7", size = 303384, upload-time = "2025-10-19T22:29:46.578Z" }, + { url = "https://files.pythonhosted.org/packages/c0/a2/584f2c29641df8bd810d00c1f21d408c12e9ad0c0dafdb8b7b29e5ddf787/fastuuid-0.14.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:c0a94245afae4d7af8c43b3159d5e3934c53f47140be0be624b96acd672ceb73", size = 460921, upload-time = "2025-10-19T22:36:42.006Z" }, + { url = "https://files.pythonhosted.org/packages/24/68/c6b77443bb7764c760e211002c8638c0c7cce11cb584927e723215ba1398/fastuuid-0.14.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:2b29e23c97e77c3a9514d70ce343571e469098ac7f5a269320a0f0b3e193ab36", size = 480575, upload-time = "2025-10-19T22:28:18.975Z" }, + { url = "https://files.pythonhosted.org/packages/5a/87/93f553111b33f9bb83145be12868c3c475bf8ea87c107063d01377cc0e8e/fastuuid-0.14.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:1e690d48f923c253f28151b3a6b4e335f2b06bf669c68a02665bc150b7839e94", size = 452317, upload-time = "2025-10-19T22:25:32.75Z" }, + { url = "https://files.pythonhosted.org/packages/9e/8c/a04d486ca55b5abb7eaa65b39df8d891b7b1635b22db2163734dc273579a/fastuuid-0.14.0-cp311-cp311-win32.whl", hash = "sha256:a6f46790d59ab38c6aa0e35c681c0484b50dc0acf9e2679c005d61e019313c24", size = 154804, upload-time = "2025-10-19T22:24:15.615Z" }, + { url = "https://files.pythonhosted.org/packages/9c/b2/2d40bf00820de94b9280366a122cbaa60090c8cf59e89ac3938cf5d75895/fastuuid-0.14.0-cp311-cp311-win_amd64.whl", hash = "sha256:e150eab56c95dc9e3fefc234a0eedb342fac433dacc273cd4d150a5b0871e1fa", size = 156099, upload-time = "2025-10-19T22:24:31.646Z" }, + { url = "https://files.pythonhosted.org/packages/02/a2/e78fcc5df65467f0d207661b7ef86c5b7ac62eea337c0c0fcedbeee6fb13/fastuuid-0.14.0-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:77e94728324b63660ebf8adb27055e92d2e4611645bf12ed9d88d30486471d0a", size = 510164, upload-time = "2025-10-19T22:31:45.635Z" }, + { url = "https://files.pythonhosted.org/packages/2b/b3/c846f933f22f581f558ee63f81f29fa924acd971ce903dab1a9b6701816e/fastuuid-0.14.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:caa1f14d2102cb8d353096bc6ef6c13b2c81f347e6ab9d6fbd48b9dea41c153d", size = 261837, upload-time = "2025-10-19T22:38:38.53Z" }, + { url = "https://files.pythonhosted.org/packages/54/ea/682551030f8c4fa9a769d9825570ad28c0c71e30cf34020b85c1f7ee7382/fastuuid-0.14.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d23ef06f9e67163be38cece704170486715b177f6baae338110983f99a72c070", size = 251370, upload-time = "2025-10-19T22:40:26.07Z" }, + { url = "https://files.pythonhosted.org/packages/14/dd/5927f0a523d8e6a76b70968e6004966ee7df30322f5fc9b6cdfb0276646a/fastuuid-0.14.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0c9ec605ace243b6dbe3bd27ebdd5d33b00d8d1d3f580b39fdd15cd96fd71796", size = 277766, upload-time = "2025-10-19T22:37:23.779Z" }, + { url = "https://files.pythonhosted.org/packages/16/6e/c0fb547eef61293153348f12e0f75a06abb322664b34a1573a7760501336/fastuuid-0.14.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:808527f2407f58a76c916d6aa15d58692a4a019fdf8d4c32ac7ff303b7d7af09", size = 278105, upload-time = "2025-10-19T22:26:56.821Z" }, + { url = "https://files.pythonhosted.org/packages/2d/b1/b9c75e03b768f61cf2e84ee193dc18601aeaf89a4684b20f2f0e9f52b62c/fastuuid-0.14.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2fb3c0d7fef6674bbeacdd6dbd386924a7b60b26de849266d1ff6602937675c8", size = 301564, upload-time = "2025-10-19T22:30:31.604Z" }, + { url = "https://files.pythonhosted.org/packages/fc/fa/f7395fdac07c7a54f18f801744573707321ca0cee082e638e36452355a9d/fastuuid-0.14.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ab3f5d36e4393e628a4df337c2c039069344db5f4b9d2a3c9cea48284f1dd741", size = 459659, upload-time = "2025-10-19T22:31:32.341Z" }, + { url = "https://files.pythonhosted.org/packages/66/49/c9fd06a4a0b1f0f048aacb6599e7d96e5d6bc6fa680ed0d46bf111929d1b/fastuuid-0.14.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:b9a0ca4f03b7e0b01425281ffd44e99d360e15c895f1907ca105854ed85e2057", size = 478430, upload-time = "2025-10-19T22:26:22.962Z" }, + { url = "https://files.pythonhosted.org/packages/be/9c/909e8c95b494e8e140e8be6165d5fc3f61fdc46198c1554df7b3e1764471/fastuuid-0.14.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:3acdf655684cc09e60fb7e4cf524e8f42ea760031945aa8086c7eae2eeeabeb8", size = 450894, upload-time = "2025-10-19T22:27:01.647Z" }, + { url = "https://files.pythonhosted.org/packages/90/eb/d29d17521976e673c55ef7f210d4cdd72091a9ec6755d0fd4710d9b3c871/fastuuid-0.14.0-cp312-cp312-win32.whl", hash = "sha256:9579618be6280700ae36ac42c3efd157049fe4dd40ca49b021280481c78c3176", size = 154374, upload-time = "2025-10-19T22:29:19.879Z" }, + { url = "https://files.pythonhosted.org/packages/cc/fc/f5c799a6ea6d877faec0472d0b27c079b47c86b1cdc577720a5386483b36/fastuuid-0.14.0-cp312-cp312-win_amd64.whl", hash = "sha256:d9e4332dc4ba054434a9594cbfaf7823b57993d7d8e7267831c3e059857cf397", size = 156550, upload-time = "2025-10-19T22:27:49.658Z" }, + { url = "https://files.pythonhosted.org/packages/a5/83/ae12dd39b9a39b55d7f90abb8971f1a5f3c321fd72d5aa83f90dc67fe9ed/fastuuid-0.14.0-cp313-cp313-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:77a09cb7427e7af74c594e409f7731a0cf887221de2f698e1ca0ebf0f3139021", size = 510720, upload-time = "2025-10-19T22:42:34.633Z" }, + { url = "https://files.pythonhosted.org/packages/53/b0/a4b03ff5d00f563cc7546b933c28cb3f2a07344b2aec5834e874f7d44143/fastuuid-0.14.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:9bd57289daf7b153bfa3e8013446aa144ce5e8c825e9e366d455155ede5ea2dc", size = 262024, upload-time = "2025-10-19T22:30:25.482Z" }, + { url = "https://files.pythonhosted.org/packages/9c/6d/64aee0a0f6a58eeabadd582e55d0d7d70258ffdd01d093b30c53d668303b/fastuuid-0.14.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ac60fc860cdf3c3f327374db87ab8e064c86566ca8c49d2e30df15eda1b0c2d5", size = 251679, upload-time = "2025-10-19T22:36:14.096Z" }, + { url = "https://files.pythonhosted.org/packages/60/f5/a7e9cda8369e4f7919d36552db9b2ae21db7915083bc6336f1b0082c8b2e/fastuuid-0.14.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ab32f74bd56565b186f036e33129da77db8be09178cd2f5206a5d4035fb2a23f", size = 277862, upload-time = "2025-10-19T22:36:23.302Z" }, + { url = "https://files.pythonhosted.org/packages/f0/d3/8ce11827c783affffd5bd4d6378b28eb6cc6d2ddf41474006b8d62e7448e/fastuuid-0.14.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:33e678459cf4addaedd9936bbb038e35b3f6b2061330fd8f2f6a1d80414c0f87", size = 278278, upload-time = "2025-10-19T22:29:43.809Z" }, + { url = "https://files.pythonhosted.org/packages/a2/51/680fb6352d0bbade04036da46264a8001f74b7484e2fd1f4da9e3db1c666/fastuuid-0.14.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1e3cc56742f76cd25ecb98e4b82a25f978ccffba02e4bdce8aba857b6d85d87b", size = 301788, upload-time = "2025-10-19T22:36:06.825Z" }, + { url = "https://files.pythonhosted.org/packages/fa/7c/2014b5785bd8ebdab04ec857635ebd84d5ee4950186a577db9eff0fb8ff6/fastuuid-0.14.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:cb9a030f609194b679e1660f7e32733b7a0f332d519c5d5a6a0a580991290022", size = 459819, upload-time = "2025-10-19T22:35:31.623Z" }, + { url = "https://files.pythonhosted.org/packages/01/d2/524d4ceeba9160e7a9bc2ea3e8f4ccf1ad78f3bde34090ca0c51f09a5e91/fastuuid-0.14.0-cp313-cp313-musllinux_1_1_i686.whl", hash = "sha256:09098762aad4f8da3a888eb9ae01c84430c907a297b97166b8abc07b640f2995", size = 478546, upload-time = "2025-10-19T22:26:03.023Z" }, + { url = "https://files.pythonhosted.org/packages/bc/17/354d04951ce114bf4afc78e27a18cfbd6ee319ab1829c2d5fb5e94063ac6/fastuuid-0.14.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:1383fff584fa249b16329a059c68ad45d030d5a4b70fb7c73a08d98fd53bcdab", size = 450921, upload-time = "2025-10-19T22:31:02.151Z" }, + { url = "https://files.pythonhosted.org/packages/fb/be/d7be8670151d16d88f15bb121c5b66cdb5ea6a0c2a362d0dcf30276ade53/fastuuid-0.14.0-cp313-cp313-win32.whl", hash = "sha256:a0809f8cc5731c066c909047f9a314d5f536c871a7a22e815cc4967c110ac9ad", size = 154559, upload-time = "2025-10-19T22:36:36.011Z" }, + { url = "https://files.pythonhosted.org/packages/22/1d/5573ef3624ceb7abf4a46073d3554e37191c868abc3aecd5289a72f9810a/fastuuid-0.14.0-cp313-cp313-win_amd64.whl", hash = "sha256:0df14e92e7ad3276327631c9e7cec09e32572ce82089c55cb1bb8df71cf394ed", size = 156539, upload-time = "2025-10-19T22:33:35.898Z" }, + { url = "https://files.pythonhosted.org/packages/16/c9/8c7660d1fe3862e3f8acabd9be7fc9ad71eb270f1c65cce9a2b7a31329ab/fastuuid-0.14.0-cp314-cp314-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:b852a870a61cfc26c884af205d502881a2e59cc07076b60ab4a951cc0c94d1ad", size = 510600, upload-time = "2025-10-19T22:43:44.17Z" }, + { url = "https://files.pythonhosted.org/packages/4c/f4/a989c82f9a90d0ad995aa957b3e572ebef163c5299823b4027986f133dfb/fastuuid-0.14.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:c7502d6f54cd08024c3ea9b3514e2d6f190feb2f46e6dbcd3747882264bb5f7b", size = 262069, upload-time = "2025-10-19T22:43:38.38Z" }, + { url = "https://files.pythonhosted.org/packages/da/6c/a1a24f73574ac995482b1326cf7ab41301af0fabaa3e37eeb6b3df00e6e2/fastuuid-0.14.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1ca61b592120cf314cfd66e662a5b54a578c5a15b26305e1b8b618a6f22df714", size = 251543, upload-time = "2025-10-19T22:32:22.537Z" }, + { url = "https://files.pythonhosted.org/packages/1a/20/2a9b59185ba7a6c7b37808431477c2d739fcbdabbf63e00243e37bd6bf49/fastuuid-0.14.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:aa75b6657ec129d0abded3bec745e6f7ab642e6dba3a5272a68247e85f5f316f", size = 277798, upload-time = "2025-10-19T22:33:53.821Z" }, + { url = "https://files.pythonhosted.org/packages/ef/33/4105ca574f6ded0af6a797d39add041bcfb468a1255fbbe82fcb6f592da2/fastuuid-0.14.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a8a0dfea3972200f72d4c7df02c8ac70bad1bb4c58d7e0ec1e6f341679073a7f", size = 278283, upload-time = "2025-10-19T22:29:02.812Z" }, + { url = "https://files.pythonhosted.org/packages/fe/8c/fca59f8e21c4deb013f574eae05723737ddb1d2937ce87cb2a5d20992dc3/fastuuid-0.14.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1bf539a7a95f35b419f9ad105d5a8a35036df35fdafae48fb2fd2e5f318f0d75", size = 301627, upload-time = "2025-10-19T22:35:54.985Z" }, + { url = "https://files.pythonhosted.org/packages/cb/e2/f78c271b909c034d429218f2798ca4e89eeda7983f4257d7865976ddbb6c/fastuuid-0.14.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:9a133bf9cc78fdbd1179cb58a59ad0100aa32d8675508150f3658814aeefeaa4", size = 459778, upload-time = "2025-10-19T22:28:00.999Z" }, + { url = "https://files.pythonhosted.org/packages/1e/f0/5ff209d865897667a2ff3e7a572267a9ced8f7313919f6d6043aed8b1caa/fastuuid-0.14.0-cp314-cp314-musllinux_1_1_i686.whl", hash = "sha256:f54d5b36c56a2d5e1a31e73b950b28a0d83eb0c37b91d10408875a5a29494bad", size = 478605, upload-time = "2025-10-19T22:36:21.764Z" }, + { url = "https://files.pythonhosted.org/packages/e0/c8/2ce1c78f983a2c4987ea865d9516dbdfb141a120fd3abb977ae6f02ba7ca/fastuuid-0.14.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:ec27778c6ca3393ef662e2762dba8af13f4ec1aaa32d08d77f71f2a70ae9feb8", size = 450837, upload-time = "2025-10-19T22:34:37.178Z" }, + { url = "https://files.pythonhosted.org/packages/df/60/dad662ec9a33b4a5fe44f60699258da64172c39bd041da2994422cdc40fe/fastuuid-0.14.0-cp314-cp314-win32.whl", hash = "sha256:e23fc6a83f112de4be0cc1990e5b127c27663ae43f866353166f87df58e73d06", size = 154532, upload-time = "2025-10-19T22:35:18.217Z" }, + { url = "https://files.pythonhosted.org/packages/1f/f6/da4db31001e854025ffd26bc9ba0740a9cbba2c3259695f7c5834908b336/fastuuid-0.14.0-cp314-cp314-win_amd64.whl", hash = "sha256:df61342889d0f5e7a32f7284e55ef95103f2110fee433c2ae7c2c0956d76ac8a", size = 156457, upload-time = "2025-10-19T22:33:44.579Z" }, +] + +[[package]] +name = "filelock" +version = "3.25.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/b8/00651a0f559862f3bb7d6f7477b192afe3f583cc5e26403b44e59a55ab34/filelock-3.25.2.tar.gz", hash = "sha256:b64ece2b38f4ca29dd3e810287aa8c48182bbecd1ae6e9ae126c9b35f1382694", size = 40480, upload-time = "2026-03-11T20:45:38.487Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a4/a5/842ae8f0c08b61d6484b52f99a03510a3a72d23141942d216ebe81fefbce/filelock-3.25.2-py3-none-any.whl", hash = "sha256:ca8afb0da15f229774c9ad1b455ed96e85a81373065fb10446672f64444ddf70", size = 26759, upload-time = "2026-03-11T20:45:37.437Z" }, +] + +[[package]] +name = "frozenlist" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2d/f5/c831fac6cc817d26fd54c7eaccd04ef7e0288806943f7cc5bbf69f3ac1f0/frozenlist-1.8.0.tar.gz", hash = "sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad", size = 45875, upload-time = "2025-10-06T05:38:17.865Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bc/03/077f869d540370db12165c0aa51640a873fb661d8b315d1d4d67b284d7ac/frozenlist-1.8.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:09474e9831bc2b2199fad6da3c14c7b0fbdd377cce9d3d77131be28906cb7d84", size = 86912, upload-time = "2025-10-06T05:35:45.98Z" }, + { url = "https://files.pythonhosted.org/packages/df/b5/7610b6bd13e4ae77b96ba85abea1c8cb249683217ef09ac9e0ae93f25a91/frozenlist-1.8.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:17c883ab0ab67200b5f964d2b9ed6b00971917d5d8a92df149dc2c9779208ee9", size = 50046, upload-time = "2025-10-06T05:35:47.009Z" }, + { url = "https://files.pythonhosted.org/packages/6e/ef/0e8f1fe32f8a53dd26bdd1f9347efe0778b0fddf62789ea683f4cc7d787d/frozenlist-1.8.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:fa47e444b8ba08fffd1c18e8cdb9a75db1b6a27f17507522834ad13ed5922b93", size = 50119, upload-time = "2025-10-06T05:35:48.38Z" }, + { url = "https://files.pythonhosted.org/packages/11/b1/71a477adc7c36e5fb628245dfbdea2166feae310757dea848d02bd0689fd/frozenlist-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:2552f44204b744fba866e573be4c1f9048d6a324dfe14475103fd51613eb1d1f", size = 231067, upload-time = "2025-10-06T05:35:49.97Z" }, + { url = "https://files.pythonhosted.org/packages/45/7e/afe40eca3a2dc19b9904c0f5d7edfe82b5304cb831391edec0ac04af94c2/frozenlist-1.8.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:957e7c38f250991e48a9a73e6423db1bb9dd14e722a10f6b8bb8e16a0f55f695", size = 233160, upload-time = "2025-10-06T05:35:51.729Z" }, + { url = "https://files.pythonhosted.org/packages/a6/aa/7416eac95603ce428679d273255ffc7c998d4132cfae200103f164b108aa/frozenlist-1.8.0-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:8585e3bb2cdea02fc88ffa245069c36555557ad3609e83be0ec71f54fd4abb52", size = 228544, upload-time = "2025-10-06T05:35:53.246Z" }, + { url = "https://files.pythonhosted.org/packages/8b/3d/2a2d1f683d55ac7e3875e4263d28410063e738384d3adc294f5ff3d7105e/frozenlist-1.8.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:edee74874ce20a373d62dc28b0b18b93f645633c2943fd90ee9d898550770581", size = 243797, upload-time = "2025-10-06T05:35:54.497Z" }, + { url = "https://files.pythonhosted.org/packages/78/1e/2d5565b589e580c296d3bb54da08d206e797d941a83a6fdea42af23be79c/frozenlist-1.8.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c9a63152fe95756b85f31186bddf42e4c02c6321207fd6601a1c89ebac4fe567", size = 247923, upload-time = "2025-10-06T05:35:55.861Z" }, + { url = "https://files.pythonhosted.org/packages/aa/c3/65872fcf1d326a7f101ad4d86285c403c87be7d832b7470b77f6d2ed5ddc/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:b6db2185db9be0a04fecf2f241c70b63b1a242e2805be291855078f2b404dd6b", size = 230886, upload-time = "2025-10-06T05:35:57.399Z" }, + { url = "https://files.pythonhosted.org/packages/a0/76/ac9ced601d62f6956f03cc794f9e04c81719509f85255abf96e2510f4265/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:f4be2e3d8bc8aabd566f8d5b8ba7ecc09249d74ba3c9ed52e54dc23a293f0b92", size = 245731, upload-time = "2025-10-06T05:35:58.563Z" }, + { url = "https://files.pythonhosted.org/packages/b9/49/ecccb5f2598daf0b4a1415497eba4c33c1e8ce07495eb07d2860c731b8d5/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:c8d1634419f39ea6f5c427ea2f90ca85126b54b50837f31497f3bf38266e853d", size = 241544, upload-time = "2025-10-06T05:35:59.719Z" }, + { url = "https://files.pythonhosted.org/packages/53/4b/ddf24113323c0bbcc54cb38c8b8916f1da7165e07b8e24a717b4a12cbf10/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:1a7fa382a4a223773ed64242dbe1c9c326ec09457e6b8428efb4118c685c3dfd", size = 241806, upload-time = "2025-10-06T05:36:00.959Z" }, + { url = "https://files.pythonhosted.org/packages/a7/fb/9b9a084d73c67175484ba2789a59f8eebebd0827d186a8102005ce41e1ba/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:11847b53d722050808926e785df837353bd4d75f1d494377e59b23594d834967", size = 229382, upload-time = "2025-10-06T05:36:02.22Z" }, + { url = "https://files.pythonhosted.org/packages/95/a3/c8fb25aac55bf5e12dae5c5aa6a98f85d436c1dc658f21c3ac73f9fa95e5/frozenlist-1.8.0-cp311-cp311-win32.whl", hash = "sha256:27c6e8077956cf73eadd514be8fb04d77fc946a7fe9f7fe167648b0b9085cc25", size = 39647, upload-time = "2025-10-06T05:36:03.409Z" }, + { url = "https://files.pythonhosted.org/packages/0a/f5/603d0d6a02cfd4c8f2a095a54672b3cf967ad688a60fb9faf04fc4887f65/frozenlist-1.8.0-cp311-cp311-win_amd64.whl", hash = "sha256:ac913f8403b36a2c8610bbfd25b8013488533e71e62b4b4adce9c86c8cea905b", size = 44064, upload-time = "2025-10-06T05:36:04.368Z" }, + { url = "https://files.pythonhosted.org/packages/5d/16/c2c9ab44e181f043a86f9a8f84d5124b62dbcb3a02c0977ec72b9ac1d3e0/frozenlist-1.8.0-cp311-cp311-win_arm64.whl", hash = "sha256:d4d3214a0f8394edfa3e303136d0575eece0745ff2b47bd2cb2e66dd92d4351a", size = 39937, upload-time = "2025-10-06T05:36:05.669Z" }, + { url = "https://files.pythonhosted.org/packages/69/29/948b9aa87e75820a38650af445d2ef2b6b8a6fab1a23b6bb9e4ef0be2d59/frozenlist-1.8.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1", size = 87782, upload-time = "2025-10-06T05:36:06.649Z" }, + { url = "https://files.pythonhosted.org/packages/64/80/4f6e318ee2a7c0750ed724fa33a4bdf1eacdc5a39a7a24e818a773cd91af/frozenlist-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b", size = 50594, upload-time = "2025-10-06T05:36:07.69Z" }, + { url = "https://files.pythonhosted.org/packages/2b/94/5c8a2b50a496b11dd519f4a24cb5496cf125681dd99e94c604ccdea9419a/frozenlist-1.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4", size = 50448, upload-time = "2025-10-06T05:36:08.78Z" }, + { url = "https://files.pythonhosted.org/packages/6a/bd/d91c5e39f490a49df14320f4e8c80161cfcce09f1e2cde1edd16a551abb3/frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383", size = 242411, upload-time = "2025-10-06T05:36:09.801Z" }, + { url = "https://files.pythonhosted.org/packages/8f/83/f61505a05109ef3293dfb1ff594d13d64a2324ac3482be2cedc2be818256/frozenlist-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4", size = 243014, upload-time = "2025-10-06T05:36:11.394Z" }, + { url = "https://files.pythonhosted.org/packages/d8/cb/cb6c7b0f7d4023ddda30cf56b8b17494eb3a79e3fda666bf735f63118b35/frozenlist-1.8.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8", size = 234909, upload-time = "2025-10-06T05:36:12.598Z" }, + { url = "https://files.pythonhosted.org/packages/31/c5/cd7a1f3b8b34af009fb17d4123c5a778b44ae2804e3ad6b86204255f9ec5/frozenlist-1.8.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b", size = 250049, upload-time = "2025-10-06T05:36:14.065Z" }, + { url = "https://files.pythonhosted.org/packages/c0/01/2f95d3b416c584a1e7f0e1d6d31998c4a795f7544069ee2e0962a4b60740/frozenlist-1.8.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52", size = 256485, upload-time = "2025-10-06T05:36:15.39Z" }, + { url = "https://files.pythonhosted.org/packages/ce/03/024bf7720b3abaebcff6d0793d73c154237b85bdf67b7ed55e5e9596dc9a/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29", size = 237619, upload-time = "2025-10-06T05:36:16.558Z" }, + { url = "https://files.pythonhosted.org/packages/69/fa/f8abdfe7d76b731f5d8bd217827cf6764d4f1d9763407e42717b4bed50a0/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3", size = 250320, upload-time = "2025-10-06T05:36:17.821Z" }, + { url = "https://files.pythonhosted.org/packages/f5/3c/b051329f718b463b22613e269ad72138cc256c540f78a6de89452803a47d/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143", size = 246820, upload-time = "2025-10-06T05:36:19.046Z" }, + { url = "https://files.pythonhosted.org/packages/0f/ae/58282e8f98e444b3f4dd42448ff36fa38bef29e40d40f330b22e7108f565/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608", size = 250518, upload-time = "2025-10-06T05:36:20.763Z" }, + { url = "https://files.pythonhosted.org/packages/8f/96/007e5944694d66123183845a106547a15944fbbb7154788cbf7272789536/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa", size = 239096, upload-time = "2025-10-06T05:36:22.129Z" }, + { url = "https://files.pythonhosted.org/packages/66/bb/852b9d6db2fa40be96f29c0d1205c306288f0684df8fd26ca1951d461a56/frozenlist-1.8.0-cp312-cp312-win32.whl", hash = "sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf", size = 39985, upload-time = "2025-10-06T05:36:23.661Z" }, + { url = "https://files.pythonhosted.org/packages/b8/af/38e51a553dd66eb064cdf193841f16f077585d4d28394c2fa6235cb41765/frozenlist-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746", size = 44591, upload-time = "2025-10-06T05:36:24.958Z" }, + { url = "https://files.pythonhosted.org/packages/a7/06/1dc65480ab147339fecc70797e9c2f69d9cea9cf38934ce08df070fdb9cb/frozenlist-1.8.0-cp312-cp312-win_arm64.whl", hash = "sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd", size = 40102, upload-time = "2025-10-06T05:36:26.333Z" }, + { url = "https://files.pythonhosted.org/packages/2d/40/0832c31a37d60f60ed79e9dfb5a92e1e2af4f40a16a29abcc7992af9edff/frozenlist-1.8.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a", size = 85717, upload-time = "2025-10-06T05:36:27.341Z" }, + { url = "https://files.pythonhosted.org/packages/30/ba/b0b3de23f40bc55a7057bd38434e25c34fa48e17f20ee273bbde5e0650f3/frozenlist-1.8.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7", size = 49651, upload-time = "2025-10-06T05:36:28.855Z" }, + { url = "https://files.pythonhosted.org/packages/0c/ab/6e5080ee374f875296c4243c381bbdef97a9ac39c6e3ce1d5f7d42cb78d6/frozenlist-1.8.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40", size = 49417, upload-time = "2025-10-06T05:36:29.877Z" }, + { url = "https://files.pythonhosted.org/packages/d5/4e/e4691508f9477ce67da2015d8c00acd751e6287739123113a9fca6f1604e/frozenlist-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027", size = 234391, upload-time = "2025-10-06T05:36:31.301Z" }, + { url = "https://files.pythonhosted.org/packages/40/76/c202df58e3acdf12969a7895fd6f3bc016c642e6726aa63bd3025e0fc71c/frozenlist-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822", size = 233048, upload-time = "2025-10-06T05:36:32.531Z" }, + { url = "https://files.pythonhosted.org/packages/f9/c0/8746afb90f17b73ca5979c7a3958116e105ff796e718575175319b5bb4ce/frozenlist-1.8.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121", size = 226549, upload-time = "2025-10-06T05:36:33.706Z" }, + { url = "https://files.pythonhosted.org/packages/7e/eb/4c7eefc718ff72f9b6c4893291abaae5fbc0c82226a32dcd8ef4f7a5dbef/frozenlist-1.8.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5", size = 239833, upload-time = "2025-10-06T05:36:34.947Z" }, + { url = "https://files.pythonhosted.org/packages/c2/4e/e5c02187cf704224f8b21bee886f3d713ca379535f16893233b9d672ea71/frozenlist-1.8.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e", size = 245363, upload-time = "2025-10-06T05:36:36.534Z" }, + { url = "https://files.pythonhosted.org/packages/1f/96/cb85ec608464472e82ad37a17f844889c36100eed57bea094518bf270692/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11", size = 229314, upload-time = "2025-10-06T05:36:38.582Z" }, + { url = "https://files.pythonhosted.org/packages/5d/6f/4ae69c550e4cee66b57887daeebe006fe985917c01d0fff9caab9883f6d0/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1", size = 243365, upload-time = "2025-10-06T05:36:40.152Z" }, + { url = "https://files.pythonhosted.org/packages/7a/58/afd56de246cf11780a40a2c28dc7cbabbf06337cc8ddb1c780a2d97e88d8/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1", size = 237763, upload-time = "2025-10-06T05:36:41.355Z" }, + { url = "https://files.pythonhosted.org/packages/cb/36/cdfaf6ed42e2644740d4a10452d8e97fa1c062e2a8006e4b09f1b5fd7d63/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8", size = 240110, upload-time = "2025-10-06T05:36:42.716Z" }, + { url = "https://files.pythonhosted.org/packages/03/a8/9ea226fbefad669f11b52e864c55f0bd57d3c8d7eb07e9f2e9a0b39502e1/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed", size = 233717, upload-time = "2025-10-06T05:36:44.251Z" }, + { url = "https://files.pythonhosted.org/packages/1e/0b/1b5531611e83ba7d13ccc9988967ea1b51186af64c42b7a7af465dcc9568/frozenlist-1.8.0-cp313-cp313-win32.whl", hash = "sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496", size = 39628, upload-time = "2025-10-06T05:36:45.423Z" }, + { url = "https://files.pythonhosted.org/packages/d8/cf/174c91dbc9cc49bc7b7aab74d8b734e974d1faa8f191c74af9b7e80848e6/frozenlist-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231", size = 43882, upload-time = "2025-10-06T05:36:46.796Z" }, + { url = "https://files.pythonhosted.org/packages/c1/17/502cd212cbfa96eb1388614fe39a3fc9ab87dbbe042b66f97acb57474834/frozenlist-1.8.0-cp313-cp313-win_arm64.whl", hash = "sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62", size = 39676, upload-time = "2025-10-06T05:36:47.8Z" }, + { url = "https://files.pythonhosted.org/packages/d2/5c/3bbfaa920dfab09e76946a5d2833a7cbdf7b9b4a91c714666ac4855b88b4/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94", size = 89235, upload-time = "2025-10-06T05:36:48.78Z" }, + { url = "https://files.pythonhosted.org/packages/d2/d6/f03961ef72166cec1687e84e8925838442b615bd0b8854b54923ce5b7b8a/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c", size = 50742, upload-time = "2025-10-06T05:36:49.837Z" }, + { url = "https://files.pythonhosted.org/packages/1e/bb/a6d12b7ba4c3337667d0e421f7181c82dda448ce4e7ad7ecd249a16fa806/frozenlist-1.8.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52", size = 51725, upload-time = "2025-10-06T05:36:50.851Z" }, + { url = "https://files.pythonhosted.org/packages/bc/71/d1fed0ffe2c2ccd70b43714c6cab0f4188f09f8a67a7914a6b46ee30f274/frozenlist-1.8.0-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51", size = 284533, upload-time = "2025-10-06T05:36:51.898Z" }, + { url = "https://files.pythonhosted.org/packages/c9/1f/fb1685a7b009d89f9bf78a42d94461bc06581f6e718c39344754a5d9bada/frozenlist-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65", size = 292506, upload-time = "2025-10-06T05:36:53.101Z" }, + { url = "https://files.pythonhosted.org/packages/e6/3b/b991fe1612703f7e0d05c0cf734c1b77aaf7c7d321df4572e8d36e7048c8/frozenlist-1.8.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82", size = 274161, upload-time = "2025-10-06T05:36:54.309Z" }, + { url = "https://files.pythonhosted.org/packages/ca/ec/c5c618767bcdf66e88945ec0157d7f6c4a1322f1473392319b7a2501ded7/frozenlist-1.8.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714", size = 294676, upload-time = "2025-10-06T05:36:55.566Z" }, + { url = "https://files.pythonhosted.org/packages/7c/ce/3934758637d8f8a88d11f0585d6495ef54b2044ed6ec84492a91fa3b27aa/frozenlist-1.8.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d", size = 300638, upload-time = "2025-10-06T05:36:56.758Z" }, + { url = "https://files.pythonhosted.org/packages/fc/4f/a7e4d0d467298f42de4b41cbc7ddaf19d3cfeabaf9ff97c20c6c7ee409f9/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506", size = 283067, upload-time = "2025-10-06T05:36:57.965Z" }, + { url = "https://files.pythonhosted.org/packages/dc/48/c7b163063d55a83772b268e6d1affb960771b0e203b632cfe09522d67ea5/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51", size = 292101, upload-time = "2025-10-06T05:36:59.237Z" }, + { url = "https://files.pythonhosted.org/packages/9f/d0/2366d3c4ecdc2fd391e0afa6e11500bfba0ea772764d631bbf82f0136c9d/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e", size = 289901, upload-time = "2025-10-06T05:37:00.811Z" }, + { url = "https://files.pythonhosted.org/packages/b8/94/daff920e82c1b70e3618a2ac39fbc01ae3e2ff6124e80739ce5d71c9b920/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0", size = 289395, upload-time = "2025-10-06T05:37:02.115Z" }, + { url = "https://files.pythonhosted.org/packages/e3/20/bba307ab4235a09fdcd3cc5508dbabd17c4634a1af4b96e0f69bfe551ebd/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41", size = 283659, upload-time = "2025-10-06T05:37:03.711Z" }, + { url = "https://files.pythonhosted.org/packages/fd/00/04ca1c3a7a124b6de4f8a9a17cc2fcad138b4608e7a3fc5877804b8715d7/frozenlist-1.8.0-cp313-cp313t-win32.whl", hash = "sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b", size = 43492, upload-time = "2025-10-06T05:37:04.915Z" }, + { url = "https://files.pythonhosted.org/packages/59/5e/c69f733a86a94ab10f68e496dc6b7e8bc078ebb415281d5698313e3af3a1/frozenlist-1.8.0-cp313-cp313t-win_amd64.whl", hash = "sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888", size = 48034, upload-time = "2025-10-06T05:37:06.343Z" }, + { url = "https://files.pythonhosted.org/packages/16/6c/be9d79775d8abe79b05fa6d23da99ad6e7763a1d080fbae7290b286093fd/frozenlist-1.8.0-cp313-cp313t-win_arm64.whl", hash = "sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042", size = 41749, upload-time = "2025-10-06T05:37:07.431Z" }, + { url = "https://files.pythonhosted.org/packages/f1/c8/85da824b7e7b9b6e7f7705b2ecaf9591ba6f79c1177f324c2735e41d36a2/frozenlist-1.8.0-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:cee686f1f4cadeb2136007ddedd0aaf928ab95216e7691c63e50a8ec066336d0", size = 86127, upload-time = "2025-10-06T05:37:08.438Z" }, + { url = "https://files.pythonhosted.org/packages/8e/e8/a1185e236ec66c20afd72399522f142c3724c785789255202d27ae992818/frozenlist-1.8.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:119fb2a1bd47307e899c2fac7f28e85b9a543864df47aa7ec9d3c1b4545f096f", size = 49698, upload-time = "2025-10-06T05:37:09.48Z" }, + { url = "https://files.pythonhosted.org/packages/a1/93/72b1736d68f03fda5fdf0f2180fb6caaae3894f1b854d006ac61ecc727ee/frozenlist-1.8.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4970ece02dbc8c3a92fcc5228e36a3e933a01a999f7094ff7c23fbd2beeaa67c", size = 49749, upload-time = "2025-10-06T05:37:10.569Z" }, + { url = "https://files.pythonhosted.org/packages/a7/b2/fabede9fafd976b991e9f1b9c8c873ed86f202889b864756f240ce6dd855/frozenlist-1.8.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:cba69cb73723c3f329622e34bdbf5ce1f80c21c290ff04256cff1cd3c2036ed2", size = 231298, upload-time = "2025-10-06T05:37:11.993Z" }, + { url = "https://files.pythonhosted.org/packages/3a/3b/d9b1e0b0eed36e70477ffb8360c49c85c8ca8ef9700a4e6711f39a6e8b45/frozenlist-1.8.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:778a11b15673f6f1df23d9586f83c4846c471a8af693a22e066508b77d201ec8", size = 232015, upload-time = "2025-10-06T05:37:13.194Z" }, + { url = "https://files.pythonhosted.org/packages/dc/94/be719d2766c1138148564a3960fc2c06eb688da592bdc25adcf856101be7/frozenlist-1.8.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0325024fe97f94c41c08872db482cf8ac4800d80e79222c6b0b7b162d5b13686", size = 225038, upload-time = "2025-10-06T05:37:14.577Z" }, + { url = "https://files.pythonhosted.org/packages/e4/09/6712b6c5465f083f52f50cf74167b92d4ea2f50e46a9eea0523d658454ae/frozenlist-1.8.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:97260ff46b207a82a7567b581ab4190bd4dfa09f4db8a8b49d1a958f6aa4940e", size = 240130, upload-time = "2025-10-06T05:37:15.781Z" }, + { url = "https://files.pythonhosted.org/packages/f8/d4/cd065cdcf21550b54f3ce6a22e143ac9e4836ca42a0de1022da8498eac89/frozenlist-1.8.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:54b2077180eb7f83dd52c40b2750d0a9f175e06a42e3213ce047219de902717a", size = 242845, upload-time = "2025-10-06T05:37:17.037Z" }, + { url = "https://files.pythonhosted.org/packages/62/c3/f57a5c8c70cd1ead3d5d5f776f89d33110b1addae0ab010ad774d9a44fb9/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2f05983daecab868a31e1da44462873306d3cbfd76d1f0b5b69c473d21dbb128", size = 229131, upload-time = "2025-10-06T05:37:18.221Z" }, + { url = "https://files.pythonhosted.org/packages/6c/52/232476fe9cb64f0742f3fde2b7d26c1dac18b6d62071c74d4ded55e0ef94/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:33f48f51a446114bc5d251fb2954ab0164d5be02ad3382abcbfe07e2531d650f", size = 240542, upload-time = "2025-10-06T05:37:19.771Z" }, + { url = "https://files.pythonhosted.org/packages/5f/85/07bf3f5d0fb5414aee5f47d33c6f5c77bfe49aac680bfece33d4fdf6a246/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:154e55ec0655291b5dd1b8731c637ecdb50975a2ae70c606d100750a540082f7", size = 237308, upload-time = "2025-10-06T05:37:20.969Z" }, + { url = "https://files.pythonhosted.org/packages/11/99/ae3a33d5befd41ac0ca2cc7fd3aa707c9c324de2e89db0e0f45db9a64c26/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:4314debad13beb564b708b4a496020e5306c7333fa9a3ab90374169a20ffab30", size = 238210, upload-time = "2025-10-06T05:37:22.252Z" }, + { url = "https://files.pythonhosted.org/packages/b2/60/b1d2da22f4970e7a155f0adde9b1435712ece01b3cd45ba63702aea33938/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:073f8bf8becba60aa931eb3bc420b217bb7d5b8f4750e6f8b3be7f3da85d38b7", size = 231972, upload-time = "2025-10-06T05:37:23.5Z" }, + { url = "https://files.pythonhosted.org/packages/3f/ab/945b2f32de889993b9c9133216c068b7fcf257d8595a0ac420ac8677cab0/frozenlist-1.8.0-cp314-cp314-win32.whl", hash = "sha256:bac9c42ba2ac65ddc115d930c78d24ab8d4f465fd3fc473cdedfccadb9429806", size = 40536, upload-time = "2025-10-06T05:37:25.581Z" }, + { url = "https://files.pythonhosted.org/packages/59/ad/9caa9b9c836d9ad6f067157a531ac48b7d36499f5036d4141ce78c230b1b/frozenlist-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:3e0761f4d1a44f1d1a47996511752cf3dcec5bbdd9cc2b4fe595caf97754b7a0", size = 44330, upload-time = "2025-10-06T05:37:26.928Z" }, + { url = "https://files.pythonhosted.org/packages/82/13/e6950121764f2676f43534c555249f57030150260aee9dcf7d64efda11dd/frozenlist-1.8.0-cp314-cp314-win_arm64.whl", hash = "sha256:d1eaff1d00c7751b7c6662e9c5ba6eb2c17a2306ba5e2a37f24ddf3cc953402b", size = 40627, upload-time = "2025-10-06T05:37:28.075Z" }, + { url = "https://files.pythonhosted.org/packages/c0/c7/43200656ecc4e02d3f8bc248df68256cd9572b3f0017f0a0c4e93440ae23/frozenlist-1.8.0-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:d3bb933317c52d7ea5004a1c442eef86f426886fba134ef8cf4226ea6ee1821d", size = 89238, upload-time = "2025-10-06T05:37:29.373Z" }, + { url = "https://files.pythonhosted.org/packages/d1/29/55c5f0689b9c0fb765055629f472c0de484dcaf0acee2f7707266ae3583c/frozenlist-1.8.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:8009897cdef112072f93a0efdce29cd819e717fd2f649ee3016efd3cd885a7ed", size = 50738, upload-time = "2025-10-06T05:37:30.792Z" }, + { url = "https://files.pythonhosted.org/packages/ba/7d/b7282a445956506fa11da8c2db7d276adcbf2b17d8bb8407a47685263f90/frozenlist-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2c5dcbbc55383e5883246d11fd179782a9d07a986c40f49abe89ddf865913930", size = 51739, upload-time = "2025-10-06T05:37:32.127Z" }, + { url = "https://files.pythonhosted.org/packages/62/1c/3d8622e60d0b767a5510d1d3cf21065b9db874696a51ea6d7a43180a259c/frozenlist-1.8.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:39ecbc32f1390387d2aa4f5a995e465e9e2f79ba3adcac92d68e3e0afae6657c", size = 284186, upload-time = "2025-10-06T05:37:33.21Z" }, + { url = "https://files.pythonhosted.org/packages/2d/14/aa36d5f85a89679a85a1d44cd7a6657e0b1c75f61e7cad987b203d2daca8/frozenlist-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92db2bf818d5cc8d9c1f1fc56b897662e24ea5adb36ad1f1d82875bd64e03c24", size = 292196, upload-time = "2025-10-06T05:37:36.107Z" }, + { url = "https://files.pythonhosted.org/packages/05/23/6bde59eb55abd407d34f77d39a5126fb7b4f109a3f611d3929f14b700c66/frozenlist-1.8.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2dc43a022e555de94c3b68a4ef0b11c4f747d12c024a520c7101709a2144fb37", size = 273830, upload-time = "2025-10-06T05:37:37.663Z" }, + { url = "https://files.pythonhosted.org/packages/d2/3f/22cff331bfad7a8afa616289000ba793347fcd7bc275f3b28ecea2a27909/frozenlist-1.8.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:cb89a7f2de3602cfed448095bab3f178399646ab7c61454315089787df07733a", size = 294289, upload-time = "2025-10-06T05:37:39.261Z" }, + { url = "https://files.pythonhosted.org/packages/a4/89/5b057c799de4838b6c69aa82b79705f2027615e01be996d2486a69ca99c4/frozenlist-1.8.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:33139dc858c580ea50e7e60a1b0ea003efa1fd42e6ec7fdbad78fff65fad2fd2", size = 300318, upload-time = "2025-10-06T05:37:43.213Z" }, + { url = "https://files.pythonhosted.org/packages/30/de/2c22ab3eb2a8af6d69dc799e48455813bab3690c760de58e1bf43b36da3e/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:168c0969a329b416119507ba30b9ea13688fafffac1b7822802537569a1cb0ef", size = 282814, upload-time = "2025-10-06T05:37:45.337Z" }, + { url = "https://files.pythonhosted.org/packages/59/f7/970141a6a8dbd7f556d94977858cfb36fa9b66e0892c6dd780d2219d8cd8/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:28bd570e8e189d7f7b001966435f9dac6718324b5be2990ac496cf1ea9ddb7fe", size = 291762, upload-time = "2025-10-06T05:37:46.657Z" }, + { url = "https://files.pythonhosted.org/packages/c1/15/ca1adae83a719f82df9116d66f5bb28bb95557b3951903d39135620ef157/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b2a095d45c5d46e5e79ba1e5b9cb787f541a8dee0433836cea4b96a2c439dcd8", size = 289470, upload-time = "2025-10-06T05:37:47.946Z" }, + { url = "https://files.pythonhosted.org/packages/ac/83/dca6dc53bf657d371fbc88ddeb21b79891e747189c5de990b9dfff2ccba1/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:eab8145831a0d56ec9c4139b6c3e594c7a83c2c8be25d5bcf2d86136a532287a", size = 289042, upload-time = "2025-10-06T05:37:49.499Z" }, + { url = "https://files.pythonhosted.org/packages/96/52/abddd34ca99be142f354398700536c5bd315880ed0a213812bc491cff5e4/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:974b28cf63cc99dfb2188d8d222bc6843656188164848c4f679e63dae4b0708e", size = 283148, upload-time = "2025-10-06T05:37:50.745Z" }, + { url = "https://files.pythonhosted.org/packages/af/d3/76bd4ed4317e7119c2b7f57c3f6934aba26d277acc6309f873341640e21f/frozenlist-1.8.0-cp314-cp314t-win32.whl", hash = "sha256:342c97bf697ac5480c0a7ec73cd700ecfa5a8a40ac923bd035484616efecc2df", size = 44676, upload-time = "2025-10-06T05:37:52.222Z" }, + { url = "https://files.pythonhosted.org/packages/89/76/c615883b7b521ead2944bb3480398cbb07e12b7b4e4d073d3752eb721558/frozenlist-1.8.0-cp314-cp314t-win_amd64.whl", hash = "sha256:06be8f67f39c8b1dc671f5d83aaefd3358ae5cdcf8314552c57e7ed3e6475bdd", size = 49451, upload-time = "2025-10-06T05:37:53.425Z" }, + { url = "https://files.pythonhosted.org/packages/e0/a3/5982da14e113d07b325230f95060e2169f5311b1017ea8af2a29b374c289/frozenlist-1.8.0-cp314-cp314t-win_arm64.whl", hash = "sha256:102e6314ca4da683dca92e3b1355490fed5f313b768500084fbe6371fddfdb79", size = 42507, upload-time = "2025-10-06T05:37:54.513Z" }, + { url = "https://files.pythonhosted.org/packages/9a/9a/e35b4a917281c0b8419d4207f4334c8e8c5dbf4f3f5f9ada73958d937dcc/frozenlist-1.8.0-py3-none-any.whl", hash = "sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d", size = 13409, upload-time = "2025-10-06T05:38:16.721Z" }, +] + +[[package]] +name = "fsspec" +version = "2026.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/51/7c/f60c259dcbf4f0c47cc4ddb8f7720d2dcdc8888c8e5ad84c73ea4531cc5b/fsspec-2026.2.0.tar.gz", hash = "sha256:6544e34b16869f5aacd5b90bdf1a71acb37792ea3ddf6125ee69a22a53fb8bff", size = 313441, upload-time = "2026-02-05T21:50:53.743Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e6/ab/fb21f4c939bb440104cc2b396d3be1d9b7a9fd3c6c2a53d98c45b3d7c954/fsspec-2026.2.0-py3-none-any.whl", hash = "sha256:98de475b5cb3bd66bedd5c4679e87b4fdfe1a3bf4d707b151b3c07e58c9a2437", size = 202505, upload-time = "2026-02-05T21:50:51.819Z" }, +] + +[[package]] +name = "gitdb" +version = "4.0.12" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "smmap" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/72/94/63b0fc47eb32792c7ba1fe1b694daec9a63620db1e313033d18140c2320a/gitdb-4.0.12.tar.gz", hash = "sha256:5ef71f855d191a3326fcfbc0d5da835f26b13fbcba60c32c21091c349ffdb571", size = 394684, upload-time = "2025-01-02T07:20:46.413Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl", hash = "sha256:67073e15955400952c6565cc3e707c554a4eea2e428946f7a4c162fab9bd9bcf", size = 62794, upload-time = "2025-01-02T07:20:43.624Z" }, +] + +[[package]] +name = "gitpython" +version = "3.1.46" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "gitdb" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/b5/59d16470a1f0dfe8c793f9ef56fd3826093fc52b3bd96d6b9d6c26c7e27b/gitpython-3.1.46.tar.gz", hash = "sha256:400124c7d0ef4ea03f7310ac2fbf7151e09ff97f2a3288d64a440c584a29c37f", size = 215371, upload-time = "2026-01-01T15:37:32.073Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/09/e21df6aef1e1ffc0c816f0522ddc3f6dcded766c3261813131c78a704470/gitpython-3.1.46-py3-none-any.whl", hash = "sha256:79812ed143d9d25b6d176a10bb511de0f9c67b1fa641d82097b0ab90398a2058", size = 208620, upload-time = "2026-01-01T15:37:30.574Z" }, +] + +[[package]] +name = "griffelib" +version = "2.0.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9d/82/74f4a3310cdabfbb10da554c3a672847f1ed33c6f61dd472681ce7f1fe67/griffelib-2.0.2.tar.gz", hash = "sha256:3cf20b3bc470e83763ffbf236e0076b1211bac1bc67de13daf494640f2de707e", size = 166461, upload-time = "2026-03-27T11:34:51.091Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/11/8c/c9138d881c79aa0ea9ed83cbd58d5ca75624378b38cee225dcf5c42cc91f/griffelib-2.0.2-py3-none-any.whl", hash = "sha256:925c857658fb1ba40c0772c37acbc2ab650bd794d9c1b9726922e36ea4117ea1", size = 142357, upload-time = "2026-03-27T11:34:46.275Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "hf-xet" +version = "1.4.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/92/ec9ad04d0b5728dca387a45af7bc98fbb0d73b2118759f5f6038b61a57e8/hf_xet-1.4.3.tar.gz", hash = "sha256:8ddedb73c8c08928c793df2f3401ec26f95be7f7e516a7bee2fbb546f6676113", size = 670477, upload-time = "2026-03-31T22:40:07.874Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/72/43/724d307b34e353da0abd476e02f72f735cdd2bc86082dee1b32ea0bfee1d/hf_xet-1.4.3-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:7551659ba4f1e1074e9623996f28c3873682530aee0a846b7f2f066239228144", size = 3800935, upload-time = "2026-03-31T22:39:49.618Z" }, + { url = "https://files.pythonhosted.org/packages/2b/d2/8bee5996b699262edb87dbb54118d287c0e1b2fc78af7cdc41857ba5e3c4/hf_xet-1.4.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:bee693ada985e7045997f05f081d0e12c4c08bd7626dc397f8a7c487e6c04f7f", size = 3558942, upload-time = "2026-03-31T22:39:47.938Z" }, + { url = "https://files.pythonhosted.org/packages/c3/a1/e993d09cbe251196fb60812b09a58901c468127b7259d2bf0f68bf6088eb/hf_xet-1.4.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:21644b404bb0100fe3857892f752c4d09642586fd988e61501c95bbf44b393a3", size = 4207657, upload-time = "2026-03-31T22:39:39.69Z" }, + { url = "https://files.pythonhosted.org/packages/64/44/9eb6d21e5c34c63e5e399803a6932fa983cabdf47c0ecbcfe7ea97684b8c/hf_xet-1.4.3-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:987f09cfe418237812896a6736b81b1af02a3a6dcb4b4944425c4c4fca7a7cf8", size = 3986765, upload-time = "2026-03-31T22:39:37.936Z" }, + { url = "https://files.pythonhosted.org/packages/ea/7b/8ad6f16fdb82f5f7284a34b5ec48645bd575bdcd2f6f0d1644775909c486/hf_xet-1.4.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:60cf7fc43a99da0a853345cf86d23738c03983ee5249613a6305d3e57a5dca74", size = 4188162, upload-time = "2026-03-31T22:39:58.382Z" }, + { url = "https://files.pythonhosted.org/packages/1b/c4/39d6e136cbeea9ca5a23aad4b33024319222adbdc059ebcda5fc7d9d5ff4/hf_xet-1.4.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2815a49a7a59f3e2edf0cf113ae88e8cb2ca2a221bf353fb60c609584f4884d4", size = 4424525, upload-time = "2026-03-31T22:40:00.225Z" }, + { url = "https://files.pythonhosted.org/packages/46/f2/adc32dae6bdbc367853118b9878139ac869419a4ae7ba07185dc31251b76/hf_xet-1.4.3-cp313-cp313t-win_amd64.whl", hash = "sha256:42ee323265f1e6a81b0e11094564fb7f7e0ec75b5105ffd91ae63f403a11931b", size = 3671610, upload-time = "2026-03-31T22:40:10.42Z" }, + { url = "https://files.pythonhosted.org/packages/e2/19/25d897dcc3f81953e0c2cde9ec186c7a0fee413eb0c9a7a9130d87d94d3a/hf_xet-1.4.3-cp313-cp313t-win_arm64.whl", hash = "sha256:27c976ba60079fb8217f485b9c5c7fcd21c90b0367753805f87cb9f3cdc4418a", size = 3528529, upload-time = "2026-03-31T22:40:09.106Z" }, + { url = "https://files.pythonhosted.org/packages/ec/36/3e8f85ca9fe09b8de2b2e10c63b3b3353d7dda88a0b3d426dffbe7b8313b/hf_xet-1.4.3-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:5251d5ece3a81815bae9abab41cf7ddb7bcb8f56411bce0827f4a3071c92fdc6", size = 3801019, upload-time = "2026-03-31T22:39:56.651Z" }, + { url = "https://files.pythonhosted.org/packages/b5/9c/defb6cb1de28bccb7bd8d95f6e60f72a3d3fa4cb3d0329c26fb9a488bfe7/hf_xet-1.4.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1feb0f3abeacee143367c326a128a2e2b60868ec12a36c225afb1d6c5a05e6d2", size = 3558746, upload-time = "2026-03-31T22:39:54.766Z" }, + { url = "https://files.pythonhosted.org/packages/c1/bd/8d001191893178ff8e826e46ad5299446e62b93cd164e17b0ffea08832ec/hf_xet-1.4.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8b301fc150290ca90b4fccd079829b84bb4786747584ae08b94b4577d82fb791", size = 4207692, upload-time = "2026-03-31T22:39:46.246Z" }, + { url = "https://files.pythonhosted.org/packages/ce/48/6790b402803250e9936435613d3a78b9aaeee7973439f0918848dde58309/hf_xet-1.4.3-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:d972fbe95ddc0d3c0fc49b31a8a69f47db35c1e3699bf316421705741aab6653", size = 3986281, upload-time = "2026-03-31T22:39:44.648Z" }, + { url = "https://files.pythonhosted.org/packages/51/56/ea62552fe53db652a9099eda600b032d75554d0e86c12a73824bfedef88b/hf_xet-1.4.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c5b48db1ee344a805a1b9bd2cda9b6b65fe77ed3787bd6e87ad5521141d317cd", size = 4187414, upload-time = "2026-03-31T22:40:04.951Z" }, + { url = "https://files.pythonhosted.org/packages/7d/f5/bc1456d4638061bea997e6d2db60a1a613d7b200e0755965ec312dc1ef79/hf_xet-1.4.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:22bdc1f5fb8b15bf2831440b91d1c9bbceeb7e10c81a12e8d75889996a5c9da8", size = 4424368, upload-time = "2026-03-31T22:40:06.347Z" }, + { url = "https://files.pythonhosted.org/packages/e4/76/ab597bae87e1f06d18d3ecb8ed7f0d3c9a37037fc32ce76233d369273c64/hf_xet-1.4.3-cp314-cp314t-win_amd64.whl", hash = "sha256:0392c79b7cf48418cd61478c1a925246cf10639f4cd9d94368d8ca1e8df9ea07", size = 3672280, upload-time = "2026-03-31T22:40:16.401Z" }, + { url = "https://files.pythonhosted.org/packages/62/05/2e462d34e23a09a74d73785dbed71cc5dbad82a72eee2ad60a72a554155d/hf_xet-1.4.3-cp314-cp314t-win_arm64.whl", hash = "sha256:681c92a07796325778a79d76c67011764ecc9042a8c3579332b61b63ae512075", size = 3528945, upload-time = "2026-03-31T22:40:14.995Z" }, + { url = "https://files.pythonhosted.org/packages/ac/9f/9c23e4a447b8f83120798f9279d0297a4d1360bdbf59ef49ebec78fe2545/hf_xet-1.4.3-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:d0da85329eaf196e03e90b84c2d0aca53bd4573d097a75f99609e80775f98025", size = 3805048, upload-time = "2026-03-31T22:39:53.105Z" }, + { url = "https://files.pythonhosted.org/packages/0b/f8/7aacb8e5f4a7899d39c787b5984e912e6c18b11be136ef13947d7a66d265/hf_xet-1.4.3-cp37-abi3-macosx_11_0_arm64.whl", hash = "sha256:e23717ce4186b265f69afa66e6f0069fe7efbf331546f5c313d00e123dc84583", size = 3562178, upload-time = "2026-03-31T22:39:51.295Z" }, + { url = "https://files.pythonhosted.org/packages/df/9a/a24b26dc8a65f0ecc0fe5be981a19e61e7ca963b85e062c083f3a9100529/hf_xet-1.4.3-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fc360b70c815bf340ed56c7b8c63aacf11762a4b099b2fe2c9bd6d6068668c08", size = 4212320, upload-time = "2026-03-31T22:39:42.922Z" }, + { url = "https://files.pythonhosted.org/packages/53/60/46d493db155d2ee2801b71fb1b0fd67696359047fdd8caee2c914cc50c79/hf_xet-1.4.3-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:39f2d2e9654cd9b4319885733993807aab6de9dfbd34c42f0b78338d6617421f", size = 3991546, upload-time = "2026-03-31T22:39:41.335Z" }, + { url = "https://files.pythonhosted.org/packages/bc/f5/067363e1c96c6b17256910830d1b54099d06287e10f4ec6ec4e7e08371fc/hf_xet-1.4.3-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:49ad8a8cead2b56051aa84d7fce3e1335efe68df3cf6c058f22a65513885baac", size = 4193200, upload-time = "2026-03-31T22:40:01.936Z" }, + { url = "https://files.pythonhosted.org/packages/42/4b/53951592882d9c23080c7644542fda34a3813104e9e11fa1a7d82d419cb8/hf_xet-1.4.3-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:7716d62015477a70ea272d2d68cd7cad140f61c52ee452e133e139abfe2c17ba", size = 4429392, upload-time = "2026-03-31T22:40:03.492Z" }, + { url = "https://files.pythonhosted.org/packages/8a/21/75a6c175b4e79662ad8e62f46a40ce341d8d6b206b06b4320d07d55b188c/hf_xet-1.4.3-cp37-abi3-win_amd64.whl", hash = "sha256:6b591fcad34e272a5b02607485e4f2a1334aebf1bc6d16ce8eb1eb8978ac2021", size = 3677359, upload-time = "2026-03-31T22:40:13.619Z" }, + { url = "https://files.pythonhosted.org/packages/8a/7c/44314ecd0e89f8b2b51c9d9e5e7a60a9c1c82024ac471d415860557d3cd8/hf_xet-1.4.3-cp37-abi3-win_arm64.whl", hash = "sha256:7c2c7e20bcfcc946dc67187c203463f5e932e395845d098cc2a93f5b67ca0b47", size = 3533664, upload-time = "2026-03-31T22:40:12.152Z" }, +] + +[[package]] +name = "htmldate" +version = "1.9.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "charset-normalizer" }, + { name = "dateparser" }, + { name = "lxml" }, + { name = "python-dateutil" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9d/10/ead9dabc999f353c3aa5d0dc0835b1e355215a5ecb489a7f4ef2ddad5e33/htmldate-1.9.4.tar.gz", hash = "sha256:1129063e02dd0354b74264de71e950c0c3fcee191178321418ccad2074cc8ed0", size = 44690, upload-time = "2025-11-04T17:46:44.983Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a1/bd/adfcdaaad5805c0c5156aeefd64c1e868c05e9c1cd6fd21751f168cd88c7/htmldate-1.9.4-py3-none-any.whl", hash = "sha256:1b94bcc4e08232a5b692159903acf95548b6a7492dddca5bb123d89d6325921c", size = 31558, upload-time = "2025-11-04T17:46:43.258Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "httpx-sse" +version = "0.4.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0f/4c/751061ffa58615a32c31b2d82e8482be8dd4a89154f003147acee90f2be9/httpx_sse-0.4.3.tar.gz", hash = "sha256:9b1ed0127459a66014aec3c56bebd93da3c1bc8bb6618c8082039a44889a755d", size = 15943, upload-time = "2025-10-10T21:48:22.271Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d2/fd/6668e5aec43ab844de6fc74927e155a3b37bf40d7c3790e49fc0406b6578/httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc", size = 8960, upload-time = "2025-10-10T21:48:21.158Z" }, +] + +[[package]] +name = "huggingface-hub" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "filelock" }, + { name = "fsspec" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" }, + { name = "httpx" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "tqdm" }, + { name = "typer" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/88/bb/62c7aa86f63a05e2f9b96642fdef9b94526a23979820b09f5455deff4983/huggingface_hub-1.9.0.tar.gz", hash = "sha256:0ea5be7a56135c91797cae6ad726e38eaeb6eb4b77cefff5c9d38ba0ecf874f7", size = 750326, upload-time = "2026-04-03T08:35:55.888Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/73/37/0d15d16150e1829f3e90962c99f28257f6de9e526a680b4c6f5acdb54fd2/huggingface_hub-1.9.0-py3-none-any.whl", hash = "sha256:2999328c058d39fd19ab748dd09bd4da2fbaa4f4c1ddea823eab103051e14a1f", size = 637355, upload-time = "2026-04-03T08:35:53.897Z" }, +] + +[[package]] +name = "idna" +version = "3.11" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/0703ccc57f3a7233505399edb88de3cbd678da106337b9fcde432b65ed60/idna-3.11.tar.gz", hash = "sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902", size = 194582, upload-time = "2025-10-12T14:55:20.501Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" }, +] + +[[package]] +name = "importlib-metadata" +version = "8.7.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "zipp" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f3/49/3b30cad09e7771a4982d9975a8cbf64f00d4a1ececb53297f1d9a7be1b10/importlib_metadata-8.7.1.tar.gz", hash = "sha256:49fef1ae6440c182052f407c8d34a68f72efc36db9ca90dc0113398f2fdde8bb", size = 57107, upload-time = "2025-12-21T10:00:19.278Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fa/5e/f8e9a1d23b9c20a551a8a02ea3637b4642e22c2626e3a13a9a29cdea99eb/importlib_metadata-8.7.1-py3-none-any.whl", hash = "sha256:5a1f80bf1daa489495071efbb095d75a634cf28a8bc299581244063b53176151", size = 27865, upload-time = "2025-12-21T10:00:18.329Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "jinja2" +version = "3.1.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, +] + +[[package]] +name = "jiter" +version = "0.13.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0d/5e/4ec91646aee381d01cdb9974e30882c9cd3b8c5d1079d6b5ff4af522439a/jiter-0.13.0.tar.gz", hash = "sha256:f2839f9c2c7e2dffc1bc5929a510e14ce0a946be9365fd1219e7ef342dae14f4", size = 164847, upload-time = "2026-02-02T12:37:56.441Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/29/499f8c9eaa8a16751b1c0e45e6f5f1761d180da873d417996cc7bddc8eef/jiter-0.13.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:ea026e70a9a28ebbdddcbcf0f1323128a8db66898a06eaad3a4e62d2f554d096", size = 311157, upload-time = "2026-02-02T12:35:37.758Z" }, + { url = "https://files.pythonhosted.org/packages/50/f6/566364c777d2ab450b92100bea11333c64c38d32caf8dc378b48e5b20c46/jiter-0.13.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:66aa3e663840152d18cc8ff1e4faad3dd181373491b9cfdc6004b92198d67911", size = 319729, upload-time = "2026-02-02T12:35:39.246Z" }, + { url = "https://files.pythonhosted.org/packages/73/dd/560f13ec5e4f116d8ad2658781646cca91b617ae3b8758d4a5076b278f70/jiter-0.13.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c3524798e70655ff19aec58c7d05adb1f074fecff62da857ea9be2b908b6d701", size = 354766, upload-time = "2026-02-02T12:35:40.662Z" }, + { url = "https://files.pythonhosted.org/packages/7c/0d/061faffcfe94608cbc28a0d42a77a74222bdf5055ccdbe5fd2292b94f510/jiter-0.13.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ec7e287d7fbd02cb6e22f9a00dd9c9cd504c40a61f2c61e7e1f9690a82726b4c", size = 362587, upload-time = "2026-02-02T12:35:42.025Z" }, + { url = "https://files.pythonhosted.org/packages/92/c9/c66a7864982fd38a9773ec6e932e0398d1262677b8c60faecd02ffb67bf3/jiter-0.13.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:47455245307e4debf2ce6c6e65a717550a0244231240dcf3b8f7d64e4c2f22f4", size = 487537, upload-time = "2026-02-02T12:35:43.459Z" }, + { url = "https://files.pythonhosted.org/packages/6c/86/84eb4352cd3668f16d1a88929b5888a3fe0418ea8c1dfc2ad4e7bf6e069a/jiter-0.13.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ee9da221dca6e0429c2704c1b3655fe7b025204a71d4d9b73390c759d776d165", size = 373717, upload-time = "2026-02-02T12:35:44.928Z" }, + { url = "https://files.pythonhosted.org/packages/6e/09/9fe4c159358176f82d4390407a03f506a8659ed13ca3ac93a843402acecf/jiter-0.13.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:24ab43126d5e05f3d53a36a8e11eb2f23304c6c1117844aaaf9a0aa5e40b5018", size = 362683, upload-time = "2026-02-02T12:35:46.636Z" }, + { url = "https://files.pythonhosted.org/packages/c9/5e/85f3ab9caca0c1d0897937d378b4a515cae9e119730563572361ea0c48ae/jiter-0.13.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:9da38b4fedde4fb528c740c2564628fbab737166a0e73d6d46cb4bb5463ff411", size = 392345, upload-time = "2026-02-02T12:35:48.088Z" }, + { url = "https://files.pythonhosted.org/packages/12/4c/05b8629ad546191939e6f0c2f17e29f542a398f4a52fb987bc70b6d1eb8b/jiter-0.13.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:0b34c519e17658ed88d5047999a93547f8889f3c1824120c26ad6be5f27b6cf5", size = 517775, upload-time = "2026-02-02T12:35:49.482Z" }, + { url = "https://files.pythonhosted.org/packages/4d/88/367ea2eb6bc582c7052e4baf5ddf57ebe5ab924a88e0e09830dfb585c02d/jiter-0.13.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:d2a6394e6af690d462310a86b53c47ad75ac8c21dc79f120714ea449979cb1d3", size = 551325, upload-time = "2026-02-02T12:35:51.104Z" }, + { url = "https://files.pythonhosted.org/packages/f3/12/fa377ffb94a2f28c41afaed093e0d70cfe512035d5ecb0cad0ae4792d35e/jiter-0.13.0-cp311-cp311-win32.whl", hash = "sha256:0f0c065695f616a27c920a56ad0d4fc46415ef8b806bf8fc1cacf25002bd24e1", size = 204709, upload-time = "2026-02-02T12:35:52.467Z" }, + { url = "https://files.pythonhosted.org/packages/cb/16/8e8203ce92f844dfcd3d9d6a5a7322c77077248dbb12da52d23193a839cd/jiter-0.13.0-cp311-cp311-win_amd64.whl", hash = "sha256:0733312953b909688ae3c2d58d043aa040f9f1a6a75693defed7bc2cc4bf2654", size = 204560, upload-time = "2026-02-02T12:35:53.925Z" }, + { url = "https://files.pythonhosted.org/packages/44/26/97cc40663deb17b9e13c3a5cf29251788c271b18ee4d262c8f94798b8336/jiter-0.13.0-cp311-cp311-win_arm64.whl", hash = "sha256:5d9b34ad56761b3bf0fbe8f7e55468704107608512350962d3317ffd7a4382d5", size = 189608, upload-time = "2026-02-02T12:35:55.304Z" }, + { url = "https://files.pythonhosted.org/packages/2e/30/7687e4f87086829955013ca12a9233523349767f69653ebc27036313def9/jiter-0.13.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:0a2bd69fc1d902e89925fc34d1da51b2128019423d7b339a45d9e99c894e0663", size = 307958, upload-time = "2026-02-02T12:35:57.165Z" }, + { url = "https://files.pythonhosted.org/packages/c3/27/e57f9a783246ed95481e6749cc5002a8a767a73177a83c63ea71f0528b90/jiter-0.13.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f917a04240ef31898182f76a332f508f2cc4b57d2b4d7ad2dbfebbfe167eb505", size = 318597, upload-time = "2026-02-02T12:35:58.591Z" }, + { url = "https://files.pythonhosted.org/packages/cf/52/e5719a60ac5d4d7c5995461a94ad5ef962a37c8bf5b088390e6fad59b2ff/jiter-0.13.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c1e2b199f446d3e82246b4fd9236d7cb502dc2222b18698ba0d986d2fecc6152", size = 348821, upload-time = "2026-02-02T12:36:00.093Z" }, + { url = "https://files.pythonhosted.org/packages/61/db/c1efc32b8ba4c740ab3fc2d037d8753f67685f475e26b9d6536a4322bcdd/jiter-0.13.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:04670992b576fa65bd056dbac0c39fe8bd67681c380cb2b48efa885711d9d726", size = 364163, upload-time = "2026-02-02T12:36:01.937Z" }, + { url = "https://files.pythonhosted.org/packages/55/8a/fb75556236047c8806995671a18e4a0ad646ed255276f51a20f32dceaeec/jiter-0.13.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5a1aff1fbdb803a376d4d22a8f63f8e7ccbce0b4890c26cc7af9e501ab339ef0", size = 483709, upload-time = "2026-02-02T12:36:03.41Z" }, + { url = "https://files.pythonhosted.org/packages/7e/16/43512e6ee863875693a8e6f6d532e19d650779d6ba9a81593ae40a9088ff/jiter-0.13.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3b3fb8c2053acaef8580809ac1d1f7481a0a0bdc012fd7f5d8b18fb696a5a089", size = 370480, upload-time = "2026-02-02T12:36:04.791Z" }, + { url = "https://files.pythonhosted.org/packages/f8/4c/09b93e30e984a187bc8aaa3510e1ec8dcbdcd71ca05d2f56aac0492453aa/jiter-0.13.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bdaba7d87e66f26a2c45d8cbadcbfc4bf7884182317907baf39cfe9775bb4d93", size = 360735, upload-time = "2026-02-02T12:36:06.994Z" }, + { url = "https://files.pythonhosted.org/packages/1a/1b/46c5e349019874ec5dfa508c14c37e29864ea108d376ae26d90bee238cd7/jiter-0.13.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7b88d649135aca526da172e48083da915ec086b54e8e73a425ba50999468cc08", size = 391814, upload-time = "2026-02-02T12:36:08.368Z" }, + { url = "https://files.pythonhosted.org/packages/15/9e/26184760e85baee7162ad37b7912797d2077718476bf91517641c92b3639/jiter-0.13.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:e404ea551d35438013c64b4f357b0474c7abf9f781c06d44fcaf7a14c69ff9e2", size = 513990, upload-time = "2026-02-02T12:36:09.993Z" }, + { url = "https://files.pythonhosted.org/packages/e9/34/2c9355247d6debad57a0a15e76ab1566ab799388042743656e566b3b7de1/jiter-0.13.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1f4748aad1b4a93c8bdd70f604d0f748cdc0e8744c5547798acfa52f10e79228", size = 548021, upload-time = "2026-02-02T12:36:11.376Z" }, + { url = "https://files.pythonhosted.org/packages/ac/4a/9f2c23255d04a834398b9c2e0e665382116911dc4d06b795710503cdad25/jiter-0.13.0-cp312-cp312-win32.whl", hash = "sha256:0bf670e3b1445fc4d31612199f1744f67f889ee1bbae703c4b54dc097e5dd394", size = 203024, upload-time = "2026-02-02T12:36:12.682Z" }, + { url = "https://files.pythonhosted.org/packages/09/ee/f0ae675a957ae5a8f160be3e87acea6b11dc7b89f6b7ab057e77b2d2b13a/jiter-0.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:15db60e121e11fe186c0b15236bd5d18381b9ddacdcf4e659feb96fc6c969c92", size = 205424, upload-time = "2026-02-02T12:36:13.93Z" }, + { url = "https://files.pythonhosted.org/packages/1b/02/ae611edf913d3cbf02c97cdb90374af2082c48d7190d74c1111dde08bcdd/jiter-0.13.0-cp312-cp312-win_arm64.whl", hash = "sha256:41f92313d17989102f3cb5dd533a02787cdb99454d494344b0361355da52fcb9", size = 186818, upload-time = "2026-02-02T12:36:15.308Z" }, + { url = "https://files.pythonhosted.org/packages/91/9c/7ee5a6ff4b9991e1a45263bfc46731634c4a2bde27dfda6c8251df2d958c/jiter-0.13.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1f8a55b848cbabf97d861495cd65f1e5c590246fabca8b48e1747c4dfc8f85bf", size = 306897, upload-time = "2026-02-02T12:36:16.748Z" }, + { url = "https://files.pythonhosted.org/packages/7c/02/be5b870d1d2be5dd6a91bdfb90f248fbb7dcbd21338f092c6b89817c3dbf/jiter-0.13.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f556aa591c00f2c45eb1b89f68f52441a016034d18b65da60e2d2875bbbf344a", size = 317507, upload-time = "2026-02-02T12:36:18.351Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/b25d2ec333615f5f284f3a4024f7ce68cfa0604c322c6808b2344c7f5d2b/jiter-0.13.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f7e1d61da332ec412350463891923f960c3073cf1aae93b538f0bb4c8cd46efb", size = 350560, upload-time = "2026-02-02T12:36:19.746Z" }, + { url = "https://files.pythonhosted.org/packages/be/ec/74dcb99fef0aca9fbe56b303bf79f6bd839010cb18ad41000bf6cc71eec0/jiter-0.13.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3097d665a27bc96fd9bbf7f86178037db139f319f785e4757ce7ccbf390db6c2", size = 363232, upload-time = "2026-02-02T12:36:21.243Z" }, + { url = "https://files.pythonhosted.org/packages/1b/37/f17375e0bb2f6a812d4dd92d7616e41917f740f3e71343627da9db2824ce/jiter-0.13.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9d01ecc3a8cbdb6f25a37bd500510550b64ddf9f7d64a107d92f3ccb25035d0f", size = 483727, upload-time = "2026-02-02T12:36:22.688Z" }, + { url = "https://files.pythonhosted.org/packages/77/d2/a71160a5ae1a1e66c1395b37ef77da67513b0adba73b993a27fbe47eb048/jiter-0.13.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ed9bbc30f5d60a3bdf63ae76beb3f9db280d7f195dfcfa61af792d6ce912d159", size = 370799, upload-time = "2026-02-02T12:36:24.106Z" }, + { url = "https://files.pythonhosted.org/packages/01/99/ed5e478ff0eb4e8aa5fd998f9d69603c9fd3f32de3bd16c2b1194f68361c/jiter-0.13.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:98fbafb6e88256f4454de33c1f40203d09fc33ed19162a68b3b257b29ca7f663", size = 359120, upload-time = "2026-02-02T12:36:25.519Z" }, + { url = "https://files.pythonhosted.org/packages/16/be/7ffd08203277a813f732ba897352797fa9493faf8dc7995b31f3d9cb9488/jiter-0.13.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5467696f6b827f1116556cb0db620440380434591e93ecee7fd14d1a491b6daa", size = 390664, upload-time = "2026-02-02T12:36:26.866Z" }, + { url = "https://files.pythonhosted.org/packages/d1/84/e0787856196d6d346264d6dcccb01f741e5f0bd014c1d9a2ebe149caf4f3/jiter-0.13.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:2d08c9475d48b92892583df9da592a0e2ac49bcd41fae1fec4f39ba6cf107820", size = 513543, upload-time = "2026-02-02T12:36:28.217Z" }, + { url = "https://files.pythonhosted.org/packages/65/50/ecbd258181c4313cf79bca6c88fb63207d04d5bf5e4f65174114d072aa55/jiter-0.13.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:aed40e099404721d7fcaf5b89bd3b4568a4666358bcac7b6b15c09fb6252ab68", size = 547262, upload-time = "2026-02-02T12:36:29.678Z" }, + { url = "https://files.pythonhosted.org/packages/27/da/68f38d12e7111d2016cd198161b36e1f042bd115c169255bcb7ec823a3bf/jiter-0.13.0-cp313-cp313-win32.whl", hash = "sha256:36ebfbcffafb146d0e6ffb3e74d51e03d9c35ce7c625c8066cdbfc7b953bdc72", size = 200630, upload-time = "2026-02-02T12:36:31.808Z" }, + { url = "https://files.pythonhosted.org/packages/25/65/3bd1a972c9a08ecd22eb3b08a95d1941ebe6938aea620c246cf426ae09c2/jiter-0.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:8d76029f077379374cf0dbc78dbe45b38dec4a2eb78b08b5194ce836b2517afc", size = 202602, upload-time = "2026-02-02T12:36:33.679Z" }, + { url = "https://files.pythonhosted.org/packages/15/fe/13bd3678a311aa67686bb303654792c48206a112068f8b0b21426eb6851e/jiter-0.13.0-cp313-cp313-win_arm64.whl", hash = "sha256:bb7613e1a427cfcb6ea4544f9ac566b93d5bf67e0d48c787eca673ff9c9dff2b", size = 185939, upload-time = "2026-02-02T12:36:35.065Z" }, + { url = "https://files.pythonhosted.org/packages/49/19/a929ec002ad3228bc97ca01dbb14f7632fffdc84a95ec92ceaf4145688ae/jiter-0.13.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fa476ab5dd49f3bf3a168e05f89358c75a17608dbabb080ef65f96b27c19ab10", size = 316616, upload-time = "2026-02-02T12:36:36.579Z" }, + { url = "https://files.pythonhosted.org/packages/52/56/d19a9a194afa37c1728831e5fb81b7722c3de18a3109e8f282bfc23e587a/jiter-0.13.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ade8cb6ff5632a62b7dbd4757d8c5573f7a2e9ae285d6b5b841707d8363205ef", size = 346850, upload-time = "2026-02-02T12:36:38.058Z" }, + { url = "https://files.pythonhosted.org/packages/36/4a/94e831c6bf287754a8a019cb966ed39ff8be6ab78cadecf08df3bb02d505/jiter-0.13.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9950290340acc1adaded363edd94baebcee7dabdfa8bee4790794cd5cfad2af6", size = 358551, upload-time = "2026-02-02T12:36:39.417Z" }, + { url = "https://files.pythonhosted.org/packages/a2/ec/a4c72c822695fa80e55d2b4142b73f0012035d9fcf90eccc56bc060db37c/jiter-0.13.0-cp313-cp313t-win_amd64.whl", hash = "sha256:2b4972c6df33731aac0742b64fd0d18e0a69bc7d6e03108ce7d40c85fd9e3e6d", size = 201950, upload-time = "2026-02-02T12:36:40.791Z" }, + { url = "https://files.pythonhosted.org/packages/b6/00/393553ec27b824fbc29047e9c7cd4a3951d7fbe4a76743f17e44034fa4e4/jiter-0.13.0-cp313-cp313t-win_arm64.whl", hash = "sha256:701a1e77d1e593c1b435315ff625fd071f0998c5f02792038a5ca98899261b7d", size = 185852, upload-time = "2026-02-02T12:36:42.077Z" }, + { url = "https://files.pythonhosted.org/packages/6e/f5/f1997e987211f6f9bd71b8083047b316208b4aca0b529bb5f8c96c89ef3e/jiter-0.13.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:cc5223ab19fe25e2f0bf2643204ad7318896fe3729bf12fde41b77bfc4fafff0", size = 308804, upload-time = "2026-02-02T12:36:43.496Z" }, + { url = "https://files.pythonhosted.org/packages/cd/8f/5482a7677731fd44881f0204981ce2d7175db271f82cba2085dd2212e095/jiter-0.13.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9776ebe51713acf438fd9b4405fcd86893ae5d03487546dae7f34993217f8a91", size = 318787, upload-time = "2026-02-02T12:36:45.071Z" }, + { url = "https://files.pythonhosted.org/packages/f3/b9/7257ac59778f1cd025b26a23c5520a36a424f7f1b068f2442a5b499b7464/jiter-0.13.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:879e768938e7b49b5e90b7e3fecc0dbec01b8cb89595861fb39a8967c5220d09", size = 353880, upload-time = "2026-02-02T12:36:47.365Z" }, + { url = "https://files.pythonhosted.org/packages/c3/87/719eec4a3f0841dad99e3d3604ee4cba36af4419a76f3cb0b8e2e691ad67/jiter-0.13.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:682161a67adea11e3aae9038c06c8b4a9a71023228767477d683f69903ebc607", size = 366702, upload-time = "2026-02-02T12:36:48.871Z" }, + { url = "https://files.pythonhosted.org/packages/d2/65/415f0a75cf6921e43365a1bc227c565cb949caca8b7532776e430cbaa530/jiter-0.13.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a13b68cd1cd8cc9de8f244ebae18ccb3e4067ad205220ef324c39181e23bbf66", size = 486319, upload-time = "2026-02-02T12:36:53.006Z" }, + { url = "https://files.pythonhosted.org/packages/54/a2/9e12b48e82c6bbc6081fd81abf915e1443add1b13d8fc586e1d90bb02bb8/jiter-0.13.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:87ce0f14c6c08892b610686ae8be350bf368467b6acd5085a5b65441e2bf36d2", size = 372289, upload-time = "2026-02-02T12:36:54.593Z" }, + { url = "https://files.pythonhosted.org/packages/4e/c1/e4693f107a1789a239c759a432e9afc592366f04e901470c2af89cfd28e1/jiter-0.13.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c365005b05505a90d1c47856420980d0237adf82f70c4aff7aebd3c1cc143ad", size = 360165, upload-time = "2026-02-02T12:36:56.112Z" }, + { url = "https://files.pythonhosted.org/packages/17/08/91b9ea976c1c758240614bd88442681a87672eebc3d9a6dde476874e706b/jiter-0.13.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1317fdffd16f5873e46ce27d0e0f7f4f90f0cdf1d86bf6abeaea9f63ca2c401d", size = 389634, upload-time = "2026-02-02T12:36:57.495Z" }, + { url = "https://files.pythonhosted.org/packages/18/23/58325ef99390d6d40427ed6005bf1ad54f2577866594bcf13ce55675f87d/jiter-0.13.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:c05b450d37ba0c9e21c77fef1f205f56bcee2330bddca68d344baebfc55ae0df", size = 514933, upload-time = "2026-02-02T12:36:58.909Z" }, + { url = "https://files.pythonhosted.org/packages/5b/25/69f1120c7c395fd276c3996bb8adefa9c6b84c12bb7111e5c6ccdcd8526d/jiter-0.13.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:775e10de3849d0631a97c603f996f518159272db00fdda0a780f81752255ee9d", size = 548842, upload-time = "2026-02-02T12:37:00.433Z" }, + { url = "https://files.pythonhosted.org/packages/18/05/981c9669d86850c5fbb0d9e62bba144787f9fba84546ba43d624ee27ef29/jiter-0.13.0-cp314-cp314-win32.whl", hash = "sha256:632bf7c1d28421c00dd8bbb8a3bac5663e1f57d5cd5ed962bce3c73bf62608e6", size = 202108, upload-time = "2026-02-02T12:37:01.718Z" }, + { url = "https://files.pythonhosted.org/packages/8d/96/cdcf54dd0b0341db7d25413229888a346c7130bd20820530905fdb65727b/jiter-0.13.0-cp314-cp314-win_amd64.whl", hash = "sha256:f22ef501c3f87ede88f23f9b11e608581c14f04db59b6a801f354397ae13739f", size = 204027, upload-time = "2026-02-02T12:37:03.075Z" }, + { url = "https://files.pythonhosted.org/packages/fb/f9/724bcaaab7a3cd727031fe4f6995cb86c4bd344909177c186699c8dec51a/jiter-0.13.0-cp314-cp314-win_arm64.whl", hash = "sha256:07b75fe09a4ee8e0c606200622e571e44943f47254f95e2436c8bdcaceb36d7d", size = 187199, upload-time = "2026-02-02T12:37:04.414Z" }, + { url = "https://files.pythonhosted.org/packages/62/92/1661d8b9fd6a3d7a2d89831db26fe3c1509a287d83ad7838831c7b7a5c7e/jiter-0.13.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:964538479359059a35fb400e769295d4b315ae61e4105396d355a12f7fef09f0", size = 318423, upload-time = "2026-02-02T12:37:05.806Z" }, + { url = "https://files.pythonhosted.org/packages/4f/3b/f77d342a54d4ebcd128e520fc58ec2f5b30a423b0fd26acdfc0c6fef8e26/jiter-0.13.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e104da1db1c0991b3eaed391ccd650ae8d947eab1480c733e5a3fb28d4313e40", size = 351438, upload-time = "2026-02-02T12:37:07.189Z" }, + { url = "https://files.pythonhosted.org/packages/76/b3/ba9a69f0e4209bd3331470c723c2f5509e6f0482e416b612431a5061ed71/jiter-0.13.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0e3a5f0cde8ff433b8e88e41aa40131455420fb3649a3c7abdda6145f8cb7202", size = 364774, upload-time = "2026-02-02T12:37:08.579Z" }, + { url = "https://files.pythonhosted.org/packages/b3/16/6cdb31fa342932602458dbb631bfbd47f601e03d2e4950740e0b2100b570/jiter-0.13.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:57aab48f40be1db920a582b30b116fe2435d184f77f0e4226f546794cedd9cf0", size = 487238, upload-time = "2026-02-02T12:37:10.066Z" }, + { url = "https://files.pythonhosted.org/packages/ed/b1/956cc7abaca8d95c13aa8d6c9b3f3797241c246cd6e792934cc4c8b250d2/jiter-0.13.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7772115877c53f62beeb8fd853cab692dbc04374ef623b30f997959a4c0e7e95", size = 372892, upload-time = "2026-02-02T12:37:11.656Z" }, + { url = "https://files.pythonhosted.org/packages/26/c4/97ecde8b1e74f67b8598c57c6fccf6df86ea7861ed29da84629cdbba76c4/jiter-0.13.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1211427574b17b633cfceba5040de8081e5abf114f7a7602f73d2e16f9fdaa59", size = 360309, upload-time = "2026-02-02T12:37:13.244Z" }, + { url = "https://files.pythonhosted.org/packages/4b/d7/eabe3cf46715854ccc80be2cd78dd4c36aedeb30751dbf85a1d08c14373c/jiter-0.13.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7beae3a3d3b5212d3a55d2961db3c292e02e302feb43fce6a3f7a31b90ea6dfe", size = 389607, upload-time = "2026-02-02T12:37:14.881Z" }, + { url = "https://files.pythonhosted.org/packages/df/2d/03963fc0804e6109b82decfb9974eb92df3797fe7222428cae12f8ccaa0c/jiter-0.13.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:e5562a0f0e90a6223b704163ea28e831bd3a9faa3512a711f031611e6b06c939", size = 514986, upload-time = "2026-02-02T12:37:16.326Z" }, + { url = "https://files.pythonhosted.org/packages/f6/6c/8c83b45eb3eb1c1e18d841fe30b4b5bc5619d781267ca9bc03e005d8fd0a/jiter-0.13.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:6c26a424569a59140fb51160a56df13f438a2b0967365e987889186d5fc2f6f9", size = 548756, upload-time = "2026-02-02T12:37:17.736Z" }, + { url = "https://files.pythonhosted.org/packages/47/66/eea81dfff765ed66c68fd2ed8c96245109e13c896c2a5015c7839c92367e/jiter-0.13.0-cp314-cp314t-win32.whl", hash = "sha256:24dc96eca9f84da4131cdf87a95e6ce36765c3b156fc9ae33280873b1c32d5f6", size = 201196, upload-time = "2026-02-02T12:37:19.101Z" }, + { url = "https://files.pythonhosted.org/packages/ff/32/4ac9c7a76402f8f00d00842a7f6b83b284d0cf7c1e9d4227bc95aa6d17fa/jiter-0.13.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0a8d76c7524087272c8ae913f5d9d608bd839154b62c4322ef65723d2e5bb0b8", size = 204215, upload-time = "2026-02-02T12:37:20.495Z" }, + { url = "https://files.pythonhosted.org/packages/f9/8e/7def204fea9f9be8b3c21a6f2dd6c020cf56c7d5ff753e0e23ed7f9ea57e/jiter-0.13.0-cp314-cp314t-win_arm64.whl", hash = "sha256:2c26cf47e2cad140fa23b6d58d435a7c0161f5c514284802f25e87fddfe11024", size = 187152, upload-time = "2026-02-02T12:37:22.124Z" }, + { url = "https://files.pythonhosted.org/packages/79/b3/3c29819a27178d0e461a8571fb63c6ae38be6dc36b78b3ec2876bbd6a910/jiter-0.13.0-graalpy311-graalpy242_311_native-macosx_10_12_x86_64.whl", hash = "sha256:b1cbfa133241d0e6bdab48dcdc2604e8ba81512f6bbd68ec3e8e1357dd3c316c", size = 307016, upload-time = "2026-02-02T12:37:42.755Z" }, + { url = "https://files.pythonhosted.org/packages/eb/ae/60993e4b07b1ac5ebe46da7aa99fdbb802eb986c38d26e3883ac0125c4e0/jiter-0.13.0-graalpy311-graalpy242_311_native-macosx_11_0_arm64.whl", hash = "sha256:db367d8be9fad6e8ebbac4a7578b7af562e506211036cba2c06c3b998603c3d2", size = 305024, upload-time = "2026-02-02T12:37:44.774Z" }, + { url = "https://files.pythonhosted.org/packages/77/fa/2227e590e9cf98803db2811f172b2d6460a21539ab73006f251c66f44b14/jiter-0.13.0-graalpy311-graalpy242_311_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:45f6f8efb2f3b0603092401dc2df79fa89ccbc027aaba4174d2d4133ed661434", size = 339337, upload-time = "2026-02-02T12:37:46.668Z" }, + { url = "https://files.pythonhosted.org/packages/2d/92/015173281f7eb96c0ef580c997da8ef50870d4f7f4c9e03c845a1d62ae04/jiter-0.13.0-graalpy311-graalpy242_311_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:597245258e6ad085d064780abfb23a284d418d3e61c57362d9449c6c7317ee2d", size = 346395, upload-time = "2026-02-02T12:37:48.09Z" }, + { url = "https://files.pythonhosted.org/packages/80/60/e50fa45dd7e2eae049f0ce964663849e897300433921198aef94b6ffa23a/jiter-0.13.0-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:3d744a6061afba08dd7ae375dcde870cffb14429b7477e10f67e9e6d68772a0a", size = 305169, upload-time = "2026-02-02T12:37:50.376Z" }, + { url = "https://files.pythonhosted.org/packages/d2/73/a009f41c5eed71c49bec53036c4b33555afcdee70682a18c6f66e396c039/jiter-0.13.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:ff732bd0a0e778f43d5009840f20b935e79087b4dc65bd36f1cd0f9b04b8ff7f", size = 303808, upload-time = "2026-02-02T12:37:52.092Z" }, + { url = "https://files.pythonhosted.org/packages/c4/10/528b439290763bff3d939268085d03382471b442f212dca4ff5f12802d43/jiter-0.13.0-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ab44b178f7981fcaea7e0a5df20e773c663d06ffda0198f1a524e91b2fde7e59", size = 337384, upload-time = "2026-02-02T12:37:53.582Z" }, + { url = "https://files.pythonhosted.org/packages/67/8a/a342b2f0251f3dac4ca17618265d93bf244a2a4d089126e81e4c1056ac50/jiter-0.13.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7bb00b6d26db67a05fe3e12c76edc75f32077fb51deed13822dc648fa373bc19", size = 343768, upload-time = "2026-02-02T12:37:55.055Z" }, +] + +[[package]] +name = "jsonschema" +version = "4.26.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "jsonschema-specifications" }, + { name = "referencing" }, + { name = "rpds-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/fc/e067678238fa451312d4c62bf6e6cf5ec56375422aee02f9cb5f909b3047/jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326", size = 366583, upload-time = "2026-01-07T13:41:07.246Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/90/f63fb5873511e014207a475e2bb4e8b2e570d655b00ac19a9a0ca0a385ee/jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce", size = 90630, upload-time = "2026-01-07T13:41:05.306Z" }, +] + +[[package]] +name = "jsonschema-specifications" +version = "2025.9.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "referencing" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/19/74/a633ee74eb36c44aa6d1095e7cc5569bebf04342ee146178e2d36600708b/jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d", size = 32855, upload-time = "2025-09-08T01:34:59.186Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" }, +] + +[[package]] +name = "justext" +version = "3.0.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml", extra = ["html-clean"] }, +] +sdist = { url = "https://files.pythonhosted.org/packages/49/f3/45890c1b314f0d04e19c1c83d534e611513150939a7cf039664d9ab1e649/justext-3.0.2.tar.gz", hash = "sha256:13496a450c44c4cd5b5a75a5efcd9996066d2a189794ea99a49949685a0beb05", size = 828521, upload-time = "2025-02-25T20:21:49.934Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f2/ac/52f4e86d1924a7fc05af3aeb34488570eccc39b4af90530dd6acecdf16b5/justext-3.0.2-py2.py3-none-any.whl", hash = "sha256:62b1c562b15c3c6265e121cc070874243a443bfd53060e869393f09d6b6cc9a7", size = 837940, upload-time = "2025-02-25T20:21:44.179Z" }, +] + +[[package]] +name = "litellm" +version = "1.92.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohttp" }, + { name = "click" }, + { name = "fastuuid" }, + { name = "httpx" }, + { name = "importlib-metadata" }, + { name = "jinja2" }, + { name = "jsonschema" }, + { name = "openai" }, + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "tiktoken" }, + { name = "tokenizers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/46/ea/5522be7e0dbcaa7c3fddfd2834f387c6e8eab47c0cc65f2a74657c815af7/litellm-1.92.0.tar.gz", hash = "sha256:773adf5503ee1793289689c899394a83df8122993760d9acd782e32aa798db9d", size = 15098143, upload-time = "2026-07-12T01:15:59.828Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ed/40/401dfb16c88f22fcb285eafc074cb995047feb24b98bcf47e9552207837c/litellm-1.92.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:f0ec8327aacc7914cc16310a1a3cc14340f1140b0a761403c5a4a98b01684373", size = 19292358, upload-time = "2026-07-12T01:15:43.628Z" }, + { url = "https://files.pythonhosted.org/packages/51/29/f4d671762611a88ff7818e891094984202c7502e2984eed0e440a11332bd/litellm-1.92.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:b4b65395c5d7b15fa24e08a13871c0617f6d156c46cee0eb920f3a5954e50adf", size = 19290890, upload-time = "2026-07-12T01:15:46.237Z" }, + { url = "https://files.pythonhosted.org/packages/cf/26/8061907b67aa04b7cdf2a41cbc4f8aebfbc722c909a7e4b2aea25def7053/litellm-1.92.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:9c7b2849591a35f7039dc7386a81a8e68fccb5ac2fecaa5122192c1172556b29", size = 19291180, upload-time = "2026-07-12T01:15:48.728Z" }, + { url = "https://files.pythonhosted.org/packages/ba/6c/dc9b0b580ee8af9117abd729d1de25d74fae487a0029ca77049a09f3ad33/litellm-1.92.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:f17499155366afd1eaaeb6fd0c7b55cbd2239dcd72f2fe1f8071a969cc402fae", size = 19289559, upload-time = "2026-07-12T01:15:51.376Z" }, + { url = "https://files.pythonhosted.org/packages/10/34/1671376f228443330471416e24ee51bc8364c0ae6155d4ec6217b70a4753/litellm-1.92.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:437a0e403136996430795ead76502103dcf74769b6909e12d3e2511061ea8ff8", size = 19291560, upload-time = "2026-07-12T01:15:54.66Z" }, + { url = "https://files.pythonhosted.org/packages/3d/4f/141cf1162eeee762fc839c335e3f78fc309a65f2385023479a20b09e680a/litellm-1.92.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:8f03047a73154f33c2733899e4bf9b5ed129e2e345caa305895a318452e4a807", size = 19289770, upload-time = "2026-07-12T01:15:57.136Z" }, +] + +[[package]] +name = "lxml" +version = "6.0.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/88/262177de60548e5a2bfc46ad28232c9e9cbde697bd94132aeb80364675cb/lxml-6.0.2.tar.gz", hash = "sha256:cd79f3367bd74b317dda655dc8fcfa304d9eb6e4fb06b7168c5cf27f96e0cd62", size = 4073426, upload-time = "2025-09-22T04:04:59.287Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/d5/becbe1e2569b474a23f0c672ead8a29ac50b2dc1d5b9de184831bda8d14c/lxml-6.0.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:13e35cbc684aadf05d8711a5d1b5857c92e5e580efa9a0d2be197199c8def607", size = 8634365, upload-time = "2025-09-22T04:00:45.672Z" }, + { url = "https://files.pythonhosted.org/packages/28/66/1ced58f12e804644426b85d0bb8a4478ca77bc1761455da310505f1a3526/lxml-6.0.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3b1675e096e17c6fe9c0e8c81434f5736c0739ff9ac6123c87c2d452f48fc938", size = 4650793, upload-time = "2025-09-22T04:00:47.783Z" }, + { url = "https://files.pythonhosted.org/packages/11/84/549098ffea39dfd167e3f174b4ce983d0eed61f9d8d25b7bf2a57c3247fc/lxml-6.0.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8ac6e5811ae2870953390452e3476694196f98d447573234592d30488147404d", size = 4944362, upload-time = "2025-09-22T04:00:49.845Z" }, + { url = "https://files.pythonhosted.org/packages/ac/bd/f207f16abf9749d2037453d56b643a7471d8fde855a231a12d1e095c4f01/lxml-6.0.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5aa0fc67ae19d7a64c3fe725dc9a1bb11f80e01f78289d05c6f62545affec438", size = 5083152, upload-time = "2025-09-22T04:00:51.709Z" }, + { url = "https://files.pythonhosted.org/packages/15/ae/bd813e87d8941d52ad5b65071b1affb48da01c4ed3c9c99e40abb266fbff/lxml-6.0.2-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de496365750cc472b4e7902a485d3f152ecf57bd3ba03ddd5578ed8ceb4c5964", size = 5023539, upload-time = "2025-09-22T04:00:53.593Z" }, + { url = "https://files.pythonhosted.org/packages/02/cd/9bfef16bd1d874fbe0cb51afb00329540f30a3283beb9f0780adbb7eec03/lxml-6.0.2-cp311-cp311-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:200069a593c5e40b8f6fc0d84d86d970ba43138c3e68619ffa234bc9bb806a4d", size = 5344853, upload-time = "2025-09-22T04:00:55.524Z" }, + { url = "https://files.pythonhosted.org/packages/b8/89/ea8f91594bc5dbb879734d35a6f2b0ad50605d7fb419de2b63d4211765cc/lxml-6.0.2-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7d2de809c2ee3b888b59f995625385f74629707c9355e0ff856445cdcae682b7", size = 5225133, upload-time = "2025-09-22T04:00:57.269Z" }, + { url = "https://files.pythonhosted.org/packages/b9/37/9c735274f5dbec726b2db99b98a43950395ba3d4a1043083dba2ad814170/lxml-6.0.2-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:b2c3da8d93cf5db60e8858c17684c47d01fee6405e554fb55018dd85fc23b178", size = 4677944, upload-time = "2025-09-22T04:00:59.052Z" }, + { url = "https://files.pythonhosted.org/packages/20/28/7dfe1ba3475d8bfca3878365075abe002e05d40dfaaeb7ec01b4c587d533/lxml-6.0.2-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:442de7530296ef5e188373a1ea5789a46ce90c4847e597856570439621d9c553", size = 5284535, upload-time = "2025-09-22T04:01:01.335Z" }, + { url = "https://files.pythonhosted.org/packages/e7/cf/5f14bc0de763498fc29510e3532bf2b4b3a1c1d5d0dff2e900c16ba021ef/lxml-6.0.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2593c77efde7bfea7f6389f1ab249b15ed4aa5bc5cb5131faa3b843c429fbedb", size = 5067343, upload-time = "2025-09-22T04:01:03.13Z" }, + { url = "https://files.pythonhosted.org/packages/1c/b0/bb8275ab5472f32b28cfbbcc6db7c9d092482d3439ca279d8d6fa02f7025/lxml-6.0.2-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:3e3cb08855967a20f553ff32d147e14329b3ae70ced6edc2f282b94afbc74b2a", size = 4725419, upload-time = "2025-09-22T04:01:05.013Z" }, + { url = "https://files.pythonhosted.org/packages/25/4c/7c222753bc72edca3b99dbadba1b064209bc8ed4ad448af990e60dcce462/lxml-6.0.2-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:2ed6c667fcbb8c19c6791bbf40b7268ef8ddf5a96940ba9404b9f9a304832f6c", size = 5275008, upload-time = "2025-09-22T04:01:07.327Z" }, + { url = "https://files.pythonhosted.org/packages/6c/8c/478a0dc6b6ed661451379447cdbec77c05741a75736d97e5b2b729687828/lxml-6.0.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:b8f18914faec94132e5b91e69d76a5c1d7b0c73e2489ea8929c4aaa10b76bbf7", size = 5248906, upload-time = "2025-09-22T04:01:09.452Z" }, + { url = "https://files.pythonhosted.org/packages/2d/d9/5be3a6ab2784cdf9accb0703b65e1b64fcdd9311c9f007630c7db0cfcce1/lxml-6.0.2-cp311-cp311-win32.whl", hash = "sha256:6605c604e6daa9e0d7f0a2137bdc47a2e93b59c60a65466353e37f8272f47c46", size = 3610357, upload-time = "2025-09-22T04:01:11.102Z" }, + { url = "https://files.pythonhosted.org/packages/e2/7d/ca6fb13349b473d5732fb0ee3eec8f6c80fc0688e76b7d79c1008481bf1f/lxml-6.0.2-cp311-cp311-win_amd64.whl", hash = "sha256:e5867f2651016a3afd8dd2c8238baa66f1e2802f44bc17e236f547ace6647078", size = 4036583, upload-time = "2025-09-22T04:01:12.766Z" }, + { url = "https://files.pythonhosted.org/packages/ab/a2/51363b5ecd3eab46563645f3a2c3836a2fc67d01a1b87c5017040f39f567/lxml-6.0.2-cp311-cp311-win_arm64.whl", hash = "sha256:4197fb2534ee05fd3e7afaab5d8bfd6c2e186f65ea7f9cd6a82809c887bd1285", size = 3680591, upload-time = "2025-09-22T04:01:14.874Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c8/8ff2bc6b920c84355146cd1ab7d181bc543b89241cfb1ebee824a7c81457/lxml-6.0.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a59f5448ba2ceccd06995c95ea59a7674a10de0810f2ce90c9006f3cbc044456", size = 8661887, upload-time = "2025-09-22T04:01:17.265Z" }, + { url = "https://files.pythonhosted.org/packages/37/6f/9aae1008083bb501ef63284220ce81638332f9ccbfa53765b2b7502203cf/lxml-6.0.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:e8113639f3296706fbac34a30813929e29247718e88173ad849f57ca59754924", size = 4667818, upload-time = "2025-09-22T04:01:19.688Z" }, + { url = "https://files.pythonhosted.org/packages/f1/ca/31fb37f99f37f1536c133476674c10b577e409c0a624384147653e38baf2/lxml-6.0.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a8bef9b9825fa8bc816a6e641bb67219489229ebc648be422af695f6e7a4fa7f", size = 4950807, upload-time = "2025-09-22T04:01:21.487Z" }, + { url = "https://files.pythonhosted.org/packages/da/87/f6cb9442e4bada8aab5ae7e1046264f62fdbeaa6e3f6211b93f4c0dd97f1/lxml-6.0.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:65ea18d710fd14e0186c2f973dc60bb52039a275f82d3c44a0e42b43440ea534", size = 5109179, upload-time = "2025-09-22T04:01:23.32Z" }, + { url = "https://files.pythonhosted.org/packages/c8/20/a7760713e65888db79bbae4f6146a6ae5c04e4a204a3c48896c408cd6ed2/lxml-6.0.2-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c371aa98126a0d4c739ca93ceffa0fd7a5d732e3ac66a46e74339acd4d334564", size = 5023044, upload-time = "2025-09-22T04:01:25.118Z" }, + { url = "https://files.pythonhosted.org/packages/a2/b0/7e64e0460fcb36471899f75831509098f3fd7cd02a3833ac517433cb4f8f/lxml-6.0.2-cp312-cp312-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:700efd30c0fa1a3581d80a748157397559396090a51d306ea59a70020223d16f", size = 5359685, upload-time = "2025-09-22T04:01:27.398Z" }, + { url = "https://files.pythonhosted.org/packages/b9/e1/e5df362e9ca4e2f48ed6411bd4b3a0ae737cc842e96877f5bf9428055ab4/lxml-6.0.2-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c33e66d44fe60e72397b487ee92e01da0d09ba2d66df8eae42d77b6d06e5eba0", size = 5654127, upload-time = "2025-09-22T04:01:29.629Z" }, + { url = "https://files.pythonhosted.org/packages/c6/d1/232b3309a02d60f11e71857778bfcd4acbdb86c07db8260caf7d008b08f8/lxml-6.0.2-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90a345bbeaf9d0587a3aaffb7006aa39ccb6ff0e96a57286c0cb2fd1520ea192", size = 5253958, upload-time = "2025-09-22T04:01:31.535Z" }, + { url = "https://files.pythonhosted.org/packages/35/35/d955a070994725c4f7d80583a96cab9c107c57a125b20bb5f708fe941011/lxml-6.0.2-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:064fdadaf7a21af3ed1dcaa106b854077fbeada827c18f72aec9346847cd65d0", size = 4711541, upload-time = "2025-09-22T04:01:33.801Z" }, + { url = "https://files.pythonhosted.org/packages/1e/be/667d17363b38a78c4bd63cfd4b4632029fd68d2c2dc81f25ce9eb5224dd5/lxml-6.0.2-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fbc74f42c3525ac4ffa4b89cbdd00057b6196bcefe8bce794abd42d33a018092", size = 5267426, upload-time = "2025-09-22T04:01:35.639Z" }, + { url = "https://files.pythonhosted.org/packages/ea/47/62c70aa4a1c26569bc958c9ca86af2bb4e1f614e8c04fb2989833874f7ae/lxml-6.0.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6ddff43f702905a4e32bc24f3f2e2edfe0f8fde3277d481bffb709a4cced7a1f", size = 5064917, upload-time = "2025-09-22T04:01:37.448Z" }, + { url = "https://files.pythonhosted.org/packages/bd/55/6ceddaca353ebd0f1908ef712c597f8570cc9c58130dbb89903198e441fd/lxml-6.0.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:6da5185951d72e6f5352166e3da7b0dc27aa70bd1090b0eb3f7f7212b53f1bb8", size = 4788795, upload-time = "2025-09-22T04:01:39.165Z" }, + { url = "https://files.pythonhosted.org/packages/cf/e8/fd63e15da5e3fd4c2146f8bbb3c14e94ab850589beab88e547b2dbce22e1/lxml-6.0.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:57a86e1ebb4020a38d295c04fc79603c7899e0df71588043eb218722dabc087f", size = 5676759, upload-time = "2025-09-22T04:01:41.506Z" }, + { url = "https://files.pythonhosted.org/packages/76/47/b3ec58dc5c374697f5ba37412cd2728f427d056315d124dd4b61da381877/lxml-6.0.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2047d8234fe735ab77802ce5f2297e410ff40f5238aec569ad7c8e163d7b19a6", size = 5255666, upload-time = "2025-09-22T04:01:43.363Z" }, + { url = "https://files.pythonhosted.org/packages/19/93/03ba725df4c3d72afd9596eef4a37a837ce8e4806010569bedfcd2cb68fd/lxml-6.0.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6f91fd2b2ea15a6800c8e24418c0775a1694eefc011392da73bc6cef2623b322", size = 5277989, upload-time = "2025-09-22T04:01:45.215Z" }, + { url = "https://files.pythonhosted.org/packages/c6/80/c06de80bfce881d0ad738576f243911fccf992687ae09fd80b734712b39c/lxml-6.0.2-cp312-cp312-win32.whl", hash = "sha256:3ae2ce7d6fedfb3414a2b6c5e20b249c4c607f72cb8d2bb7cc9c6ec7c6f4e849", size = 3611456, upload-time = "2025-09-22T04:01:48.243Z" }, + { url = "https://files.pythonhosted.org/packages/f7/d7/0cdfb6c3e30893463fb3d1e52bc5f5f99684a03c29a0b6b605cfae879cd5/lxml-6.0.2-cp312-cp312-win_amd64.whl", hash = "sha256:72c87e5ee4e58a8354fb9c7c84cbf95a1c8236c127a5d1b7683f04bed8361e1f", size = 4011793, upload-time = "2025-09-22T04:01:50.042Z" }, + { url = "https://files.pythonhosted.org/packages/ea/7b/93c73c67db235931527301ed3785f849c78991e2e34f3fd9a6663ffda4c5/lxml-6.0.2-cp312-cp312-win_arm64.whl", hash = "sha256:61cb10eeb95570153e0c0e554f58df92ecf5109f75eacad4a95baa709e26c3d6", size = 3672836, upload-time = "2025-09-22T04:01:52.145Z" }, + { url = "https://files.pythonhosted.org/packages/53/fd/4e8f0540608977aea078bf6d79f128e0e2c2bba8af1acf775c30baa70460/lxml-6.0.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:9b33d21594afab46f37ae58dfadd06636f154923c4e8a4d754b0127554eb2e77", size = 8648494, upload-time = "2025-09-22T04:01:54.242Z" }, + { url = "https://files.pythonhosted.org/packages/5d/f4/2a94a3d3dfd6c6b433501b8d470a1960a20ecce93245cf2db1706adf6c19/lxml-6.0.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6c8963287d7a4c5c9a432ff487c52e9c5618667179c18a204bdedb27310f022f", size = 4661146, upload-time = "2025-09-22T04:01:56.282Z" }, + { url = "https://files.pythonhosted.org/packages/25/2e/4efa677fa6b322013035d38016f6ae859d06cac67437ca7dc708a6af7028/lxml-6.0.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1941354d92699fb5ffe6ed7b32f9649e43c2feb4b97205f75866f7d21aa91452", size = 4946932, upload-time = "2025-09-22T04:01:58.989Z" }, + { url = "https://files.pythonhosted.org/packages/ce/0f/526e78a6d38d109fdbaa5049c62e1d32fdd70c75fb61c4eadf3045d3d124/lxml-6.0.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bb2f6ca0ae2d983ded09357b84af659c954722bbf04dea98030064996d156048", size = 5100060, upload-time = "2025-09-22T04:02:00.812Z" }, + { url = "https://files.pythonhosted.org/packages/81/76/99de58d81fa702cc0ea7edae4f4640416c2062813a00ff24bd70ac1d9c9b/lxml-6.0.2-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb2a12d704f180a902d7fa778c6d71f36ceb7b0d317f34cdc76a5d05aa1dd1df", size = 5019000, upload-time = "2025-09-22T04:02:02.671Z" }, + { url = "https://files.pythonhosted.org/packages/b5/35/9e57d25482bc9a9882cb0037fdb9cc18f4b79d85df94fa9d2a89562f1d25/lxml-6.0.2-cp313-cp313-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:6ec0e3f745021bfed19c456647f0298d60a24c9ff86d9d051f52b509663feeb1", size = 5348496, upload-time = "2025-09-22T04:02:04.904Z" }, + { url = "https://files.pythonhosted.org/packages/a6/8e/cb99bd0b83ccc3e8f0f528e9aa1f7a9965dfec08c617070c5db8d63a87ce/lxml-6.0.2-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:846ae9a12d54e368933b9759052d6206a9e8b250291109c48e350c1f1f49d916", size = 5643779, upload-time = "2025-09-22T04:02:06.689Z" }, + { url = "https://files.pythonhosted.org/packages/d0/34/9e591954939276bb679b73773836c6684c22e56d05980e31d52a9a8deb18/lxml-6.0.2-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ef9266d2aa545d7374938fb5c484531ef5a2ec7f2d573e62f8ce722c735685fd", size = 5244072, upload-time = "2025-09-22T04:02:08.587Z" }, + { url = "https://files.pythonhosted.org/packages/8d/27/b29ff065f9aaca443ee377aff699714fcbffb371b4fce5ac4ca759e436d5/lxml-6.0.2-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:4077b7c79f31755df33b795dc12119cb557a0106bfdab0d2c2d97bd3cf3dffa6", size = 4718675, upload-time = "2025-09-22T04:02:10.783Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9f/f756f9c2cd27caa1a6ef8c32ae47aadea697f5c2c6d07b0dae133c244fbe/lxml-6.0.2-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a7c5d5e5f1081955358533be077166ee97ed2571d6a66bdba6ec2f609a715d1a", size = 5255171, upload-time = "2025-09-22T04:02:12.631Z" }, + { url = "https://files.pythonhosted.org/packages/61/46/bb85ea42d2cb1bd8395484fd72f38e3389611aa496ac7772da9205bbda0e/lxml-6.0.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:8f8d0cbd0674ee89863a523e6994ac25fd5be9c8486acfc3e5ccea679bad2679", size = 5057175, upload-time = "2025-09-22T04:02:14.718Z" }, + { url = "https://files.pythonhosted.org/packages/95/0c/443fc476dcc8e41577f0af70458c50fe299a97bb6b7505bb1ae09aa7f9ac/lxml-6.0.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:2cbcbf6d6e924c28f04a43f3b6f6e272312a090f269eff68a2982e13e5d57659", size = 4785688, upload-time = "2025-09-22T04:02:16.957Z" }, + { url = "https://files.pythonhosted.org/packages/48/78/6ef0b359d45bb9697bc5a626e1992fa5d27aa3f8004b137b2314793b50a0/lxml-6.0.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:dfb874cfa53340009af6bdd7e54ebc0d21012a60a4e65d927c2e477112e63484", size = 5660655, upload-time = "2025-09-22T04:02:18.815Z" }, + { url = "https://files.pythonhosted.org/packages/ff/ea/e1d33808f386bc1339d08c0dcada6e4712d4ed8e93fcad5f057070b7988a/lxml-6.0.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:fb8dae0b6b8b7f9e96c26fdd8121522ce5de9bb5538010870bd538683d30e9a2", size = 5247695, upload-time = "2025-09-22T04:02:20.593Z" }, + { url = "https://files.pythonhosted.org/packages/4f/47/eba75dfd8183673725255247a603b4ad606f4ae657b60c6c145b381697da/lxml-6.0.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:358d9adae670b63e95bc59747c72f4dc97c9ec58881d4627fe0120da0f90d314", size = 5269841, upload-time = "2025-09-22T04:02:22.489Z" }, + { url = "https://files.pythonhosted.org/packages/76/04/5c5e2b8577bc936e219becb2e98cdb1aca14a4921a12995b9d0c523502ae/lxml-6.0.2-cp313-cp313-win32.whl", hash = "sha256:e8cd2415f372e7e5a789d743d133ae474290a90b9023197fd78f32e2dc6873e2", size = 3610700, upload-time = "2025-09-22T04:02:24.465Z" }, + { url = "https://files.pythonhosted.org/packages/fe/0a/4643ccc6bb8b143e9f9640aa54e38255f9d3b45feb2cbe7ae2ca47e8782e/lxml-6.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:b30d46379644fbfc3ab81f8f82ae4de55179414651f110a1514f0b1f8f6cb2d7", size = 4010347, upload-time = "2025-09-22T04:02:26.286Z" }, + { url = "https://files.pythonhosted.org/packages/31/ef/dcf1d29c3f530577f61e5fe2f1bd72929acf779953668a8a47a479ae6f26/lxml-6.0.2-cp313-cp313-win_arm64.whl", hash = "sha256:13dcecc9946dca97b11b7c40d29fba63b55ab4170d3c0cf8c0c164343b9bfdcf", size = 3671248, upload-time = "2025-09-22T04:02:27.918Z" }, + { url = "https://files.pythonhosted.org/packages/03/15/d4a377b385ab693ce97b472fe0c77c2b16ec79590e688b3ccc71fba19884/lxml-6.0.2-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:b0c732aa23de8f8aec23f4b580d1e52905ef468afb4abeafd3fec77042abb6fe", size = 8659801, upload-time = "2025-09-22T04:02:30.113Z" }, + { url = "https://files.pythonhosted.org/packages/c8/e8/c128e37589463668794d503afaeb003987373c5f94d667124ffd8078bbd9/lxml-6.0.2-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:4468e3b83e10e0317a89a33d28f7aeba1caa4d1a6fd457d115dd4ffe90c5931d", size = 4659403, upload-time = "2025-09-22T04:02:32.119Z" }, + { url = "https://files.pythonhosted.org/packages/00/ce/74903904339decdf7da7847bb5741fc98a5451b42fc419a86c0c13d26fe2/lxml-6.0.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:abd44571493973bad4598a3be7e1d807ed45aa2adaf7ab92ab7c62609569b17d", size = 4966974, upload-time = "2025-09-22T04:02:34.155Z" }, + { url = "https://files.pythonhosted.org/packages/1f/d3/131dec79ce61c5567fecf82515bd9bc36395df42501b50f7f7f3bd065df0/lxml-6.0.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:370cd78d5855cfbffd57c422851f7d3864e6ae72d0da615fca4dad8c45d375a5", size = 5102953, upload-time = "2025-09-22T04:02:36.054Z" }, + { url = "https://files.pythonhosted.org/packages/3a/ea/a43ba9bb750d4ffdd885f2cd333572f5bb900cd2408b67fdda07e85978a0/lxml-6.0.2-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:901e3b4219fa04ef766885fb40fa516a71662a4c61b80c94d25336b4934b71c0", size = 5055054, upload-time = "2025-09-22T04:02:38.154Z" }, + { url = "https://files.pythonhosted.org/packages/60/23/6885b451636ae286c34628f70a7ed1fcc759f8d9ad382d132e1c8d3d9bfd/lxml-6.0.2-cp314-cp314-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:a4bf42d2e4cf52c28cc1812d62426b9503cdb0c87a6de81442626aa7d69707ba", size = 5352421, upload-time = "2025-09-22T04:02:40.413Z" }, + { url = "https://files.pythonhosted.org/packages/48/5b/fc2ddfc94ddbe3eebb8e9af6e3fd65e2feba4967f6a4e9683875c394c2d8/lxml-6.0.2-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b2c7fdaa4d7c3d886a42534adec7cfac73860b89b4e5298752f60aa5984641a0", size = 5673684, upload-time = "2025-09-22T04:02:42.288Z" }, + { url = "https://files.pythonhosted.org/packages/29/9c/47293c58cc91769130fbf85531280e8cc7868f7fbb6d92f4670071b9cb3e/lxml-6.0.2-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:98a5e1660dc7de2200b00d53fa00bcd3c35a3608c305d45a7bbcaf29fa16e83d", size = 5252463, upload-time = "2025-09-22T04:02:44.165Z" }, + { url = "https://files.pythonhosted.org/packages/9b/da/ba6eceb830c762b48e711ded880d7e3e89fc6c7323e587c36540b6b23c6b/lxml-6.0.2-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:dc051506c30b609238d79eda75ee9cab3e520570ec8219844a72a46020901e37", size = 4698437, upload-time = "2025-09-22T04:02:46.524Z" }, + { url = "https://files.pythonhosted.org/packages/a5/24/7be3f82cb7990b89118d944b619e53c656c97dc89c28cfb143fdb7cd6f4d/lxml-6.0.2-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8799481bbdd212470d17513a54d568f44416db01250f49449647b5ab5b5dccb9", size = 5269890, upload-time = "2025-09-22T04:02:48.812Z" }, + { url = "https://files.pythonhosted.org/packages/1b/bd/dcfb9ea1e16c665efd7538fc5d5c34071276ce9220e234217682e7d2c4a5/lxml-6.0.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9261bb77c2dab42f3ecd9103951aeca2c40277701eb7e912c545c1b16e0e4917", size = 5097185, upload-time = "2025-09-22T04:02:50.746Z" }, + { url = "https://files.pythonhosted.org/packages/21/04/a60b0ff9314736316f28316b694bccbbabe100f8483ad83852d77fc7468e/lxml-6.0.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:65ac4a01aba353cfa6d5725b95d7aed6356ddc0a3cd734de00124d285b04b64f", size = 4745895, upload-time = "2025-09-22T04:02:52.968Z" }, + { url = "https://files.pythonhosted.org/packages/d6/bd/7d54bd1846e5a310d9c715921c5faa71cf5c0853372adf78aee70c8d7aa2/lxml-6.0.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:b22a07cbb82fea98f8a2fd814f3d1811ff9ed76d0fc6abc84eb21527596e7cc8", size = 5695246, upload-time = "2025-09-22T04:02:54.798Z" }, + { url = "https://files.pythonhosted.org/packages/fd/32/5643d6ab947bc371da21323acb2a6e603cedbe71cb4c99c8254289ab6f4e/lxml-6.0.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:d759cdd7f3e055d6bc8d9bec3ad905227b2e4c785dc16c372eb5b5e83123f48a", size = 5260797, upload-time = "2025-09-22T04:02:57.058Z" }, + { url = "https://files.pythonhosted.org/packages/33/da/34c1ec4cff1eea7d0b4cd44af8411806ed943141804ac9c5d565302afb78/lxml-6.0.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:945da35a48d193d27c188037a05fec5492937f66fb1958c24fc761fb9d40d43c", size = 5277404, upload-time = "2025-09-22T04:02:58.966Z" }, + { url = "https://files.pythonhosted.org/packages/82/57/4eca3e31e54dc89e2c3507e1cd411074a17565fa5ffc437c4ae0a00d439e/lxml-6.0.2-cp314-cp314-win32.whl", hash = "sha256:be3aaa60da67e6153eb15715cc2e19091af5dc75faef8b8a585aea372507384b", size = 3670072, upload-time = "2025-09-22T04:03:38.05Z" }, + { url = "https://files.pythonhosted.org/packages/e3/e0/c96cf13eccd20c9421ba910304dae0f619724dcf1702864fd59dd386404d/lxml-6.0.2-cp314-cp314-win_amd64.whl", hash = "sha256:fa25afbadead523f7001caf0c2382afd272c315a033a7b06336da2637d92d6ed", size = 4080617, upload-time = "2025-09-22T04:03:39.835Z" }, + { url = "https://files.pythonhosted.org/packages/d5/5d/b3f03e22b3d38d6f188ef044900a9b29b2fe0aebb94625ce9fe244011d34/lxml-6.0.2-cp314-cp314-win_arm64.whl", hash = "sha256:063eccf89df5b24e361b123e257e437f9e9878f425ee9aae3144c77faf6da6d8", size = 3754930, upload-time = "2025-09-22T04:03:41.565Z" }, + { url = "https://files.pythonhosted.org/packages/5e/5c/42c2c4c03554580708fc738d13414801f340c04c3eff90d8d2d227145275/lxml-6.0.2-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:6162a86d86893d63084faaf4ff937b3daea233e3682fb4474db07395794fa80d", size = 8910380, upload-time = "2025-09-22T04:03:01.645Z" }, + { url = "https://files.pythonhosted.org/packages/bf/4f/12df843e3e10d18d468a7557058f8d3733e8b6e12401f30b1ef29360740f/lxml-6.0.2-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:414aaa94e974e23a3e92e7ca5b97d10c0cf37b6481f50911032c69eeb3991bba", size = 4775632, upload-time = "2025-09-22T04:03:03.814Z" }, + { url = "https://files.pythonhosted.org/packages/e4/0c/9dc31e6c2d0d418483cbcb469d1f5a582a1cd00a1f4081953d44051f3c50/lxml-6.0.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:48461bd21625458dd01e14e2c38dd0aea69addc3c4f960c30d9f59d7f93be601", size = 4975171, upload-time = "2025-09-22T04:03:05.651Z" }, + { url = "https://files.pythonhosted.org/packages/e7/2b/9b870c6ca24c841bdd887504808f0417aa9d8d564114689266f19ddf29c8/lxml-6.0.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:25fcc59afc57d527cfc78a58f40ab4c9b8fd096a9a3f964d2781ffb6eb33f4ed", size = 5110109, upload-time = "2025-09-22T04:03:07.452Z" }, + { url = "https://files.pythonhosted.org/packages/bf/0c/4f5f2a4dd319a178912751564471355d9019e220c20d7db3fb8307ed8582/lxml-6.0.2-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5179c60288204e6ddde3f774a93350177e08876eaf3ab78aa3a3649d43eb7d37", size = 5041061, upload-time = "2025-09-22T04:03:09.297Z" }, + { url = "https://files.pythonhosted.org/packages/12/64/554eed290365267671fe001a20d72d14f468ae4e6acef1e179b039436967/lxml-6.0.2-cp314-cp314t-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:967aab75434de148ec80597b75062d8123cadf2943fb4281f385141e18b21338", size = 5306233, upload-time = "2025-09-22T04:03:11.651Z" }, + { url = "https://files.pythonhosted.org/packages/7a/31/1d748aa275e71802ad9722df32a7a35034246b42c0ecdd8235412c3396ef/lxml-6.0.2-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d100fcc8930d697c6561156c6810ab4a508fb264c8b6779e6e61e2ed5e7558f9", size = 5604739, upload-time = "2025-09-22T04:03:13.592Z" }, + { url = "https://files.pythonhosted.org/packages/8f/41/2c11916bcac09ed561adccacceaedd2bf0e0b25b297ea92aab99fd03d0fa/lxml-6.0.2-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2ca59e7e13e5981175b8b3e4ab84d7da57993eeff53c07764dcebda0d0e64ecd", size = 5225119, upload-time = "2025-09-22T04:03:15.408Z" }, + { url = "https://files.pythonhosted.org/packages/99/05/4e5c2873d8f17aa018e6afde417c80cc5d0c33be4854cce3ef5670c49367/lxml-6.0.2-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:957448ac63a42e2e49531b9d6c0fa449a1970dbc32467aaad46f11545be9af1d", size = 4633665, upload-time = "2025-09-22T04:03:17.262Z" }, + { url = "https://files.pythonhosted.org/packages/0f/c9/dcc2da1bebd6275cdc723b515f93edf548b82f36a5458cca3578bc899332/lxml-6.0.2-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b7fc49c37f1786284b12af63152fe1d0990722497e2d5817acfe7a877522f9a9", size = 5234997, upload-time = "2025-09-22T04:03:19.14Z" }, + { url = "https://files.pythonhosted.org/packages/9c/e2/5172e4e7468afca64a37b81dba152fc5d90e30f9c83c7c3213d6a02a5ce4/lxml-6.0.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e19e0643cc936a22e837f79d01a550678da8377d7d801a14487c10c34ee49c7e", size = 5090957, upload-time = "2025-09-22T04:03:21.436Z" }, + { url = "https://files.pythonhosted.org/packages/a5/b3/15461fd3e5cd4ddcb7938b87fc20b14ab113b92312fc97afe65cd7c85de1/lxml-6.0.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:1db01e5cf14345628e0cbe71067204db658e2fb8e51e7f33631f5f4735fefd8d", size = 4764372, upload-time = "2025-09-22T04:03:23.27Z" }, + { url = "https://files.pythonhosted.org/packages/05/33/f310b987c8bf9e61c4dd8e8035c416bd3230098f5e3cfa69fc4232de7059/lxml-6.0.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:875c6b5ab39ad5291588aed6925fac99d0097af0dd62f33c7b43736043d4a2ec", size = 5634653, upload-time = "2025-09-22T04:03:25.767Z" }, + { url = "https://files.pythonhosted.org/packages/70/ff/51c80e75e0bc9382158133bdcf4e339b5886c6ee2418b5199b3f1a61ed6d/lxml-6.0.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:cdcbed9ad19da81c480dfd6dd161886db6096083c9938ead313d94b30aadf272", size = 5233795, upload-time = "2025-09-22T04:03:27.62Z" }, + { url = "https://files.pythonhosted.org/packages/56/4d/4856e897df0d588789dd844dbed9d91782c4ef0b327f96ce53c807e13128/lxml-6.0.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:80dadc234ebc532e09be1975ff538d154a7fa61ea5031c03d25178855544728f", size = 5257023, upload-time = "2025-09-22T04:03:30.056Z" }, + { url = "https://files.pythonhosted.org/packages/0f/85/86766dfebfa87bea0ab78e9ff7a4b4b45225df4b4d3b8cc3c03c5cd68464/lxml-6.0.2-cp314-cp314t-win32.whl", hash = "sha256:da08e7bb297b04e893d91087df19638dc7a6bb858a954b0cc2b9f5053c922312", size = 3911420, upload-time = "2025-09-22T04:03:32.198Z" }, + { url = "https://files.pythonhosted.org/packages/fe/1a/b248b355834c8e32614650b8008c69ffeb0ceb149c793961dd8c0b991bb3/lxml-6.0.2-cp314-cp314t-win_amd64.whl", hash = "sha256:252a22982dca42f6155125ac76d3432e548a7625d56f5a273ee78a5057216eca", size = 4406837, upload-time = "2025-09-22T04:03:34.027Z" }, + { url = "https://files.pythonhosted.org/packages/92/aa/df863bcc39c5e0946263454aba394de8a9084dbaff8ad143846b0d844739/lxml-6.0.2-cp314-cp314t-win_arm64.whl", hash = "sha256:bb4c1847b303835d89d785a18801a883436cdfd5dc3d62947f9c49e24f0f5a2c", size = 3822205, upload-time = "2025-09-22T04:03:36.249Z" }, + { url = "https://files.pythonhosted.org/packages/0b/11/29d08bc103a62c0eba8016e7ed5aeebbf1e4312e83b0b1648dd203b0e87d/lxml-6.0.2-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:1c06035eafa8404b5cf475bb37a9f6088b0aca288d4ccc9d69389750d5543700", size = 3949829, upload-time = "2025-09-22T04:04:45.608Z" }, + { url = "https://files.pythonhosted.org/packages/12/b3/52ab9a3b31e5ab8238da241baa19eec44d2ab426532441ee607165aebb52/lxml-6.0.2-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c7d13103045de1bdd6fe5d61802565f1a3537d70cd3abf596aa0af62761921ee", size = 4226277, upload-time = "2025-09-22T04:04:47.754Z" }, + { url = "https://files.pythonhosted.org/packages/a0/33/1eaf780c1baad88224611df13b1c2a9dfa460b526cacfe769103ff50d845/lxml-6.0.2-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0a3c150a95fbe5ac91de323aa756219ef9cf7fde5a3f00e2281e30f33fa5fa4f", size = 4330433, upload-time = "2025-09-22T04:04:49.907Z" }, + { url = "https://files.pythonhosted.org/packages/7a/c1/27428a2ff348e994ab4f8777d3a0ad510b6b92d37718e5887d2da99952a2/lxml-6.0.2-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:60fa43be34f78bebb27812ed90f1925ec99560b0fa1decdb7d12b84d857d31e9", size = 4272119, upload-time = "2025-09-22T04:04:51.801Z" }, + { url = "https://files.pythonhosted.org/packages/f0/d0/3020fa12bcec4ab62f97aab026d57c2f0cfd480a558758d9ca233bb6a79d/lxml-6.0.2-pp311-pypy311_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:21c73b476d3cfe836be731225ec3421fa2f048d84f6df6a8e70433dff1376d5a", size = 4417314, upload-time = "2025-09-22T04:04:55.024Z" }, + { url = "https://files.pythonhosted.org/packages/6c/77/d7f491cbc05303ac6801651aabeb262d43f319288c1ea96c66b1d2692ff3/lxml-6.0.2-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:27220da5be049e936c3aca06f174e8827ca6445a4353a1995584311487fc4e3e", size = 3518768, upload-time = "2025-09-22T04:04:57.097Z" }, +] + +[package.optional-dependencies] +html-clean = [ + { name = "lxml-html-clean" }, +] + +[[package]] +name = "lxml-html-clean" +version = "0.4.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9a/a4/5c62acfacd69ff4f5db395100f5cfb9b54e7ac8c69a235e4e939fd13f021/lxml_html_clean-0.4.4.tar.gz", hash = "sha256:58f39a9d632711202ed1d6d0b9b47a904e306c85de5761543b90e3e3f736acfb", size = 23899, upload-time = "2026-02-27T09:35:52.911Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d9/76/7ffc1d3005cf7749123bc47cb3ea343cd97b0ac2211bab40f57283577d0e/lxml_html_clean-0.4.4-py3-none-any.whl", hash = "sha256:ce2ef506614ecb85ee1c5fe0a2aa45b06a19514ec7949e9c8f34f06925cfabcb", size = 14565, upload-time = "2026-02-27T09:35:51.86Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5b/f5/4ec618ed16cc4f8fb3b701563655a69816155e79e24a17b651541804721d/markdown_it_py-4.0.0.tar.gz", hash = "sha256:cb0a2b4aa34f932c007117b194e945bd74e0ec24133ceb5bac59009cda1cb9f3", size = 73070, upload-time = "2025-08-11T12:57:52.854Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/94/54/e7d793b573f298e1c9013b8c4dade17d481164aa517d1d7148619c2cedbf/markdown_it_py-4.0.0-py3-none-any.whl", hash = "sha256:87327c59b172c5011896038353a81343b6754500a08cd7a4973bb48c6d578147", size = 87321, upload-time = "2025-08-11T12:57:51.923Z" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/08/db/fefacb2136439fc8dd20e797950e749aa1f4997ed584c62cfb8ef7c2be0e/markupsafe-3.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1cc7ea17a6824959616c525620e387f6dd30fec8cb44f649e31712db02123dad", size = 11631, upload-time = "2025-09-27T18:36:18.185Z" }, + { url = "https://files.pythonhosted.org/packages/e1/2e/5898933336b61975ce9dc04decbc0a7f2fee78c30353c5efba7f2d6ff27a/markupsafe-3.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4bd4cd07944443f5a265608cc6aab442e4f74dff8088b0dfc8238647b8f6ae9a", size = 12058, upload-time = "2025-09-27T18:36:19.444Z" }, + { url = "https://files.pythonhosted.org/packages/1d/09/adf2df3699d87d1d8184038df46a9c80d78c0148492323f4693df54e17bb/markupsafe-3.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b5420a1d9450023228968e7e6a9ce57f65d148ab56d2313fcd589eee96a7a50", size = 24287, upload-time = "2025-09-27T18:36:20.768Z" }, + { url = "https://files.pythonhosted.org/packages/30/ac/0273f6fcb5f42e314c6d8cd99effae6a5354604d461b8d392b5ec9530a54/markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0bf2a864d67e76e5c9a34dc26ec616a66b9888e25e7b9460e1c76d3293bd9dbf", size = 22940, upload-time = "2025-09-27T18:36:22.249Z" }, + { url = "https://files.pythonhosted.org/packages/19/ae/31c1be199ef767124c042c6c3e904da327a2f7f0cd63a0337e1eca2967a8/markupsafe-3.0.3-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:bc51efed119bc9cfdf792cdeaa4d67e8f6fcccab66ed4bfdd6bde3e59bfcbb2f", size = 21887, upload-time = "2025-09-27T18:36:23.535Z" }, + { url = "https://files.pythonhosted.org/packages/b2/76/7edcab99d5349a4532a459e1fe64f0b0467a3365056ae550d3bcf3f79e1e/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a", size = 23692, upload-time = "2025-09-27T18:36:24.823Z" }, + { url = "https://files.pythonhosted.org/packages/a4/28/6e74cdd26d7514849143d69f0bf2399f929c37dc2b31e6829fd2045b2765/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:7be7b61bb172e1ed687f1754f8e7484f1c8019780f6f6b0786e76bb01c2ae115", size = 21471, upload-time = "2025-09-27T18:36:25.95Z" }, + { url = "https://files.pythonhosted.org/packages/62/7e/a145f36a5c2945673e590850a6f8014318d5577ed7e5920a4b3448e0865d/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a", size = 22923, upload-time = "2025-09-27T18:36:27.109Z" }, + { url = "https://files.pythonhosted.org/packages/0f/62/d9c46a7f5c9adbeeeda52f5b8d802e1094e9717705a645efc71b0913a0a8/markupsafe-3.0.3-cp311-cp311-win32.whl", hash = "sha256:0db14f5dafddbb6d9208827849fad01f1a2609380add406671a26386cdf15a19", size = 14572, upload-time = "2025-09-27T18:36:28.045Z" }, + { url = "https://files.pythonhosted.org/packages/83/8a/4414c03d3f891739326e1783338e48fb49781cc915b2e0ee052aa490d586/markupsafe-3.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:de8a88e63464af587c950061a5e6a67d3632e36df62b986892331d4620a35c01", size = 15077, upload-time = "2025-09-27T18:36:29.025Z" }, + { url = "https://files.pythonhosted.org/packages/35/73/893072b42e6862f319b5207adc9ae06070f095b358655f077f69a35601f0/markupsafe-3.0.3-cp311-cp311-win_arm64.whl", hash = "sha256:3b562dd9e9ea93f13d53989d23a7e775fdfd1066c33494ff43f5418bc8c58a5c", size = 13876, upload-time = "2025-09-27T18:36:29.954Z" }, + { url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" }, + { url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" }, + { url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" }, + { url = "https://files.pythonhosted.org/packages/3c/2e/8d0c2ab90a8c1d9a24f0399058ab8519a3279d1bd4289511d74e909f060e/markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d", size = 22947, upload-time = "2025-09-27T18:36:33.86Z" }, + { url = "https://files.pythonhosted.org/packages/2c/54/887f3092a85238093a0b2154bd629c89444f395618842e8b0c41783898ea/markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a", size = 21962, upload-time = "2025-09-27T18:36:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/c9/2f/336b8c7b6f4a4d95e91119dc8521402461b74a485558d8f238a68312f11c/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b", size = 23760, upload-time = "2025-09-27T18:36:36.001Z" }, + { url = "https://files.pythonhosted.org/packages/32/43/67935f2b7e4982ffb50a4d169b724d74b62a3964bc1a9a527f5ac4f1ee2b/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f", size = 21529, upload-time = "2025-09-27T18:36:36.906Z" }, + { url = "https://files.pythonhosted.org/packages/89/e0/4486f11e51bbba8b0c041098859e869e304d1c261e59244baa3d295d47b7/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b", size = 23015, upload-time = "2025-09-27T18:36:37.868Z" }, + { url = "https://files.pythonhosted.org/packages/2f/e1/78ee7a023dac597a5825441ebd17170785a9dab23de95d2c7508ade94e0e/markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d", size = 14540, upload-time = "2025-09-27T18:36:38.761Z" }, + { url = "https://files.pythonhosted.org/packages/aa/5b/bec5aa9bbbb2c946ca2733ef9c4ca91c91b6a24580193e891b5f7dbe8e1e/markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c", size = 15105, upload-time = "2025-09-27T18:36:39.701Z" }, + { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" }, + { url = "https://files.pythonhosted.org/packages/38/2f/907b9c7bbba283e68f20259574b13d005c121a0fa4c175f9bed27c4597ff/markupsafe-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795", size = 11622, upload-time = "2025-09-27T18:36:41.777Z" }, + { url = "https://files.pythonhosted.org/packages/9c/d9/5f7756922cdd676869eca1c4e3c0cd0df60ed30199ffd775e319089cb3ed/markupsafe-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219", size = 12029, upload-time = "2025-09-27T18:36:43.257Z" }, + { url = "https://files.pythonhosted.org/packages/00/07/575a68c754943058c78f30db02ee03a64b3c638586fba6a6dd56830b30a3/markupsafe-3.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6", size = 24374, upload-time = "2025-09-27T18:36:44.508Z" }, + { url = "https://files.pythonhosted.org/packages/a9/21/9b05698b46f218fc0e118e1f8168395c65c8a2c750ae2bab54fc4bd4e0e8/markupsafe-3.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676", size = 22980, upload-time = "2025-09-27T18:36:45.385Z" }, + { url = "https://files.pythonhosted.org/packages/7f/71/544260864f893f18b6827315b988c146b559391e6e7e8f7252839b1b846a/markupsafe-3.0.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9", size = 21990, upload-time = "2025-09-27T18:36:46.916Z" }, + { url = "https://files.pythonhosted.org/packages/c2/28/b50fc2f74d1ad761af2f5dcce7492648b983d00a65b8c0e0cb457c82ebbe/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1", size = 23784, upload-time = "2025-09-27T18:36:47.884Z" }, + { url = "https://files.pythonhosted.org/packages/ed/76/104b2aa106a208da8b17a2fb72e033a5a9d7073c68f7e508b94916ed47a9/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc", size = 21588, upload-time = "2025-09-27T18:36:48.82Z" }, + { url = "https://files.pythonhosted.org/packages/b5/99/16a5eb2d140087ebd97180d95249b00a03aa87e29cc224056274f2e45fd6/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12", size = 23041, upload-time = "2025-09-27T18:36:49.797Z" }, + { url = "https://files.pythonhosted.org/packages/19/bc/e7140ed90c5d61d77cea142eed9f9c303f4c4806f60a1044c13e3f1471d0/markupsafe-3.0.3-cp313-cp313-win32.whl", hash = "sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed", size = 14543, upload-time = "2025-09-27T18:36:51.584Z" }, + { url = "https://files.pythonhosted.org/packages/05/73/c4abe620b841b6b791f2edc248f556900667a5a1cf023a6646967ae98335/markupsafe-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5", size = 15113, upload-time = "2025-09-27T18:36:52.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/3a/fa34a0f7cfef23cf9500d68cb7c32dd64ffd58a12b09225fb03dd37d5b80/markupsafe-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485", size = 13911, upload-time = "2025-09-27T18:36:53.513Z" }, + { url = "https://files.pythonhosted.org/packages/e4/d7/e05cd7efe43a88a17a37b3ae96e79a19e846f3f456fe79c57ca61356ef01/markupsafe-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73", size = 11658, upload-time = "2025-09-27T18:36:54.819Z" }, + { url = "https://files.pythonhosted.org/packages/99/9e/e412117548182ce2148bdeacdda3bb494260c0b0184360fe0d56389b523b/markupsafe-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37", size = 12066, upload-time = "2025-09-27T18:36:55.714Z" }, + { url = "https://files.pythonhosted.org/packages/bc/e6/fa0ffcda717ef64a5108eaa7b4f5ed28d56122c9a6d70ab8b72f9f715c80/markupsafe-3.0.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19", size = 25639, upload-time = "2025-09-27T18:36:56.908Z" }, + { url = "https://files.pythonhosted.org/packages/96/ec/2102e881fe9d25fc16cb4b25d5f5cde50970967ffa5dddafdb771237062d/markupsafe-3.0.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025", size = 23569, upload-time = "2025-09-27T18:36:57.913Z" }, + { url = "https://files.pythonhosted.org/packages/4b/30/6f2fce1f1f205fc9323255b216ca8a235b15860c34b6798f810f05828e32/markupsafe-3.0.3-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6", size = 23284, upload-time = "2025-09-27T18:36:58.833Z" }, + { url = "https://files.pythonhosted.org/packages/58/47/4a0ccea4ab9f5dcb6f79c0236d954acb382202721e704223a8aafa38b5c8/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f", size = 24801, upload-time = "2025-09-27T18:36:59.739Z" }, + { url = "https://files.pythonhosted.org/packages/6a/70/3780e9b72180b6fecb83a4814d84c3bf4b4ae4bf0b19c27196104149734c/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb", size = 22769, upload-time = "2025-09-27T18:37:00.719Z" }, + { url = "https://files.pythonhosted.org/packages/98/c5/c03c7f4125180fc215220c035beac6b9cb684bc7a067c84fc69414d315f5/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009", size = 23642, upload-time = "2025-09-27T18:37:01.673Z" }, + { url = "https://files.pythonhosted.org/packages/80/d6/2d1b89f6ca4bff1036499b1e29a1d02d282259f3681540e16563f27ebc23/markupsafe-3.0.3-cp313-cp313t-win32.whl", hash = "sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354", size = 14612, upload-time = "2025-09-27T18:37:02.639Z" }, + { url = "https://files.pythonhosted.org/packages/2b/98/e48a4bfba0a0ffcf9925fe2d69240bfaa19c6f7507b8cd09c70684a53c1e/markupsafe-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218", size = 15200, upload-time = "2025-09-27T18:37:03.582Z" }, + { url = "https://files.pythonhosted.org/packages/0e/72/e3cc540f351f316e9ed0f092757459afbc595824ca724cbc5a5d4263713f/markupsafe-3.0.3-cp313-cp313t-win_arm64.whl", hash = "sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287", size = 13973, upload-time = "2025-09-27T18:37:04.929Z" }, + { url = "https://files.pythonhosted.org/packages/33/8a/8e42d4838cd89b7dde187011e97fe6c3af66d8c044997d2183fbd6d31352/markupsafe-3.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe", size = 11619, upload-time = "2025-09-27T18:37:06.342Z" }, + { url = "https://files.pythonhosted.org/packages/b5/64/7660f8a4a8e53c924d0fa05dc3a55c9cee10bbd82b11c5afb27d44b096ce/markupsafe-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026", size = 12029, upload-time = "2025-09-27T18:37:07.213Z" }, + { url = "https://files.pythonhosted.org/packages/da/ef/e648bfd021127bef5fa12e1720ffed0c6cbb8310c8d9bea7266337ff06de/markupsafe-3.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737", size = 24408, upload-time = "2025-09-27T18:37:09.572Z" }, + { url = "https://files.pythonhosted.org/packages/41/3c/a36c2450754618e62008bf7435ccb0f88053e07592e6028a34776213d877/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97", size = 23005, upload-time = "2025-09-27T18:37:10.58Z" }, + { url = "https://files.pythonhosted.org/packages/bc/20/b7fdf89a8456b099837cd1dc21974632a02a999ec9bf7ca3e490aacd98e7/markupsafe-3.0.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d", size = 22048, upload-time = "2025-09-27T18:37:11.547Z" }, + { url = "https://files.pythonhosted.org/packages/9a/a7/591f592afdc734f47db08a75793a55d7fbcc6902a723ae4cfbab61010cc5/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda", size = 23821, upload-time = "2025-09-27T18:37:12.48Z" }, + { url = "https://files.pythonhosted.org/packages/7d/33/45b24e4f44195b26521bc6f1a82197118f74df348556594bd2262bda1038/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf", size = 21606, upload-time = "2025-09-27T18:37:13.485Z" }, + { url = "https://files.pythonhosted.org/packages/ff/0e/53dfaca23a69fbfbbf17a4b64072090e70717344c52eaaaa9c5ddff1e5f0/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe", size = 23043, upload-time = "2025-09-27T18:37:14.408Z" }, + { url = "https://files.pythonhosted.org/packages/46/11/f333a06fc16236d5238bfe74daccbca41459dcd8d1fa952e8fbd5dccfb70/markupsafe-3.0.3-cp314-cp314-win32.whl", hash = "sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9", size = 14747, upload-time = "2025-09-27T18:37:15.36Z" }, + { url = "https://files.pythonhosted.org/packages/28/52/182836104b33b444e400b14f797212f720cbc9ed6ba34c800639d154e821/markupsafe-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581", size = 15341, upload-time = "2025-09-27T18:37:16.496Z" }, + { url = "https://files.pythonhosted.org/packages/6f/18/acf23e91bd94fd7b3031558b1f013adfa21a8e407a3fdb32745538730382/markupsafe-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4", size = 14073, upload-time = "2025-09-27T18:37:17.476Z" }, + { url = "https://files.pythonhosted.org/packages/3c/f0/57689aa4076e1b43b15fdfa646b04653969d50cf30c32a102762be2485da/markupsafe-3.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab", size = 11661, upload-time = "2025-09-27T18:37:18.453Z" }, + { url = "https://files.pythonhosted.org/packages/89/c3/2e67a7ca217c6912985ec766c6393b636fb0c2344443ff9d91404dc4c79f/markupsafe-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175", size = 12069, upload-time = "2025-09-27T18:37:19.332Z" }, + { url = "https://files.pythonhosted.org/packages/f0/00/be561dce4e6ca66b15276e184ce4b8aec61fe83662cce2f7d72bd3249d28/markupsafe-3.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634", size = 25670, upload-time = "2025-09-27T18:37:20.245Z" }, + { url = "https://files.pythonhosted.org/packages/50/09/c419f6f5a92e5fadde27efd190eca90f05e1261b10dbd8cbcb39cd8ea1dc/markupsafe-3.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50", size = 23598, upload-time = "2025-09-27T18:37:21.177Z" }, + { url = "https://files.pythonhosted.org/packages/22/44/a0681611106e0b2921b3033fc19bc53323e0b50bc70cffdd19f7d679bb66/markupsafe-3.0.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e", size = 23261, upload-time = "2025-09-27T18:37:22.167Z" }, + { url = "https://files.pythonhosted.org/packages/5f/57/1b0b3f100259dc9fffe780cfb60d4be71375510e435efec3d116b6436d43/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5", size = 24835, upload-time = "2025-09-27T18:37:23.296Z" }, + { url = "https://files.pythonhosted.org/packages/26/6a/4bf6d0c97c4920f1597cc14dd720705eca0bf7c787aebc6bb4d1bead5388/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523", size = 22733, upload-time = "2025-09-27T18:37:24.237Z" }, + { url = "https://files.pythonhosted.org/packages/14/c7/ca723101509b518797fedc2fdf79ba57f886b4aca8a7d31857ba3ee8281f/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc", size = 23672, upload-time = "2025-09-27T18:37:25.271Z" }, + { url = "https://files.pythonhosted.org/packages/fb/df/5bd7a48c256faecd1d36edc13133e51397e41b73bb77e1a69deab746ebac/markupsafe-3.0.3-cp314-cp314t-win32.whl", hash = "sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d", size = 14819, upload-time = "2025-09-27T18:37:26.285Z" }, + { url = "https://files.pythonhosted.org/packages/1a/8a/0402ba61a2f16038b48b39bccca271134be00c5c9f0f623208399333c448/markupsafe-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9", size = 15426, upload-time = "2025-09-27T18:37:27.316Z" }, + { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" }, +] + +[[package]] +name = "mcp" +version = "1.27.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "httpx" }, + { name = "httpx-sse" }, + { name = "jsonschema" }, + { name = "pydantic" }, + { name = "pydantic-settings" }, + { name = "pyjwt", extra = ["crypto"] }, + { name = "python-multipart" }, + { name = "pywin32", marker = "sys_platform == 'win32'" }, + { name = "sse-starlette" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, + { name = "uvicorn", marker = "sys_platform != 'emscripten'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8b/eb/c0cfc62075dc6e1ec1c64d352ae09ac051d9334311ed226f1f425312848a/mcp-1.27.0.tar.gz", hash = "sha256:d3dc35a7eec0d458c1da4976a48f982097ddaab87e278c5511d5a4a56e852b83", size = 607509, upload-time = "2026-04-02T14:48:08.88Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9c/46/f6b4ad632c67ef35209a66127e4bddc95759649dd595f71f13fba11bdf9a/mcp-1.27.0-py3-none-any.whl", hash = "sha256:5ce1fa81614958e267b21fb2aa34e0aea8e2c6ede60d52aba45fd47246b4d741", size = 215967, upload-time = "2026-04-02T14:48:07.24Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "multidict" +version = "6.7.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1a/c2/c2d94cbe6ac1753f3fc980da97b3d930efe1da3af3c9f5125354436c073d/multidict-6.7.1.tar.gz", hash = "sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d", size = 102010, upload-time = "2026-01-26T02:46:45.979Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ce/f1/a90635c4f88fb913fbf4ce660b83b7445b7a02615bda034b2f8eb38fd597/multidict-6.7.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:7ff981b266af91d7b4b3793ca3382e53229088d193a85dfad6f5f4c27fc73e5d", size = 76626, upload-time = "2026-01-26T02:43:26.485Z" }, + { url = "https://files.pythonhosted.org/packages/a6/9b/267e64eaf6fc637a15b35f5de31a566634a2740f97d8d094a69d34f524a4/multidict-6.7.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:844c5bca0b5444adb44a623fb0a1310c2f4cd41f402126bb269cd44c9b3f3e1e", size = 44706, upload-time = "2026-01-26T02:43:27.607Z" }, + { url = "https://files.pythonhosted.org/packages/dd/a4/d45caf2b97b035c57267791ecfaafbd59c68212004b3842830954bb4b02e/multidict-6.7.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f2a0a924d4c2e9afcd7ec64f9de35fcd96915149b2216e1cb2c10a56df483855", size = 44356, upload-time = "2026-01-26T02:43:28.661Z" }, + { url = "https://files.pythonhosted.org/packages/fd/d2/0a36c8473f0cbaeadd5db6c8b72d15bbceeec275807772bfcd059bef487d/multidict-6.7.1-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:8be1802715a8e892c784c0197c2ace276ea52702a0ede98b6310c8f255a5afb3", size = 244355, upload-time = "2026-01-26T02:43:31.165Z" }, + { url = "https://files.pythonhosted.org/packages/5d/16/8c65be997fd7dd311b7d39c7b6e71a0cb449bad093761481eccbbe4b42a2/multidict-6.7.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2e2d2ed645ea29f31c4c7ea1552fcfd7cb7ba656e1eafd4134a6620c9f5fdd9e", size = 246433, upload-time = "2026-01-26T02:43:32.581Z" }, + { url = "https://files.pythonhosted.org/packages/01/fb/4dbd7e848d2799c6a026ec88ad39cf2b8416aa167fcc903baa55ecaa045c/multidict-6.7.1-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:95922cee9a778659e91db6497596435777bd25ed116701a4c034f8e46544955a", size = 225376, upload-time = "2026-01-26T02:43:34.417Z" }, + { url = "https://files.pythonhosted.org/packages/b6/8a/4a3a6341eac3830f6053062f8fbc9a9e54407c80755b3f05bc427295c2d0/multidict-6.7.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6b83cabdc375ffaaa15edd97eb7c0c672ad788e2687004990074d7d6c9b140c8", size = 257365, upload-time = "2026-01-26T02:43:35.741Z" }, + { url = "https://files.pythonhosted.org/packages/f7/a2/dd575a69c1aa206e12d27d0770cdf9b92434b48a9ef0cd0d1afdecaa93c4/multidict-6.7.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:38fb49540705369bab8484db0689d86c0a33a0a9f2c1b197f506b71b4b6c19b0", size = 254747, upload-time = "2026-01-26T02:43:36.976Z" }, + { url = "https://files.pythonhosted.org/packages/5a/56/21b27c560c13822ed93133f08aa6372c53a8e067f11fbed37b4adcdac922/multidict-6.7.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:439cbebd499f92e9aa6793016a8acaa161dfa749ae86d20960189f5398a19144", size = 246293, upload-time = "2026-01-26T02:43:38.258Z" }, + { url = "https://files.pythonhosted.org/packages/5a/a4/23466059dc3854763423d0ad6c0f3683a379d97673b1b89ec33826e46728/multidict-6.7.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6d3bc717b6fe763b8be3f2bee2701d3c8eb1b2a8ae9f60910f1b2860c82b6c49", size = 242962, upload-time = "2026-01-26T02:43:40.034Z" }, + { url = "https://files.pythonhosted.org/packages/1f/67/51dd754a3524d685958001e8fa20a0f5f90a6a856e0a9dcabff69be3dbb7/multidict-6.7.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:619e5a1ac57986dbfec9f0b301d865dddf763696435e2962f6d9cf2fdff2bb71", size = 237360, upload-time = "2026-01-26T02:43:41.752Z" }, + { url = "https://files.pythonhosted.org/packages/64/3f/036dfc8c174934d4b55d86ff4f978e558b0e585cef70cfc1ad01adc6bf18/multidict-6.7.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:0b38ebffd9be37c1170d33bc0f36f4f262e0a09bc1aac1c34c7aa51a7293f0b3", size = 245940, upload-time = "2026-01-26T02:43:43.042Z" }, + { url = "https://files.pythonhosted.org/packages/3d/20/6214d3c105928ebc353a1c644a6ef1408bc5794fcb4f170bb524a3c16311/multidict-6.7.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:10ae39c9cfe6adedcdb764f5e8411d4a92b055e35573a2eaa88d3323289ef93c", size = 253502, upload-time = "2026-01-26T02:43:44.371Z" }, + { url = "https://files.pythonhosted.org/packages/b1/e2/c653bc4ae1be70a0f836b82172d643fcf1dade042ba2676ab08ec08bff0f/multidict-6.7.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:25167cc263257660290fba06b9318d2026e3c910be240a146e1f66dd114af2b0", size = 247065, upload-time = "2026-01-26T02:43:45.745Z" }, + { url = "https://files.pythonhosted.org/packages/c8/11/a854b4154cd3bd8b1fd375e8a8ca9d73be37610c361543d56f764109509b/multidict-6.7.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:128441d052254f42989ef98b7b6a6ecb1e6f708aa962c7984235316db59f50fa", size = 241870, upload-time = "2026-01-26T02:43:47.054Z" }, + { url = "https://files.pythonhosted.org/packages/13/bf/9676c0392309b5fdae322333d22a829715b570edb9baa8016a517b55b558/multidict-6.7.1-cp311-cp311-win32.whl", hash = "sha256:d62b7f64ffde3b99d06b707a280db04fb3855b55f5a06df387236051d0668f4a", size = 41302, upload-time = "2026-01-26T02:43:48.753Z" }, + { url = "https://files.pythonhosted.org/packages/c9/68/f16a3a8ba6f7b6dc92a1f19669c0810bd2c43fc5a02da13b1cbf8e253845/multidict-6.7.1-cp311-cp311-win_amd64.whl", hash = "sha256:bdbf9f3b332abd0cdb306e7c2113818ab1e922dc84b8f8fd06ec89ed2a19ab8b", size = 45981, upload-time = "2026-01-26T02:43:49.921Z" }, + { url = "https://files.pythonhosted.org/packages/ac/ad/9dd5305253fa00cd3c7555dbef69d5bf4133debc53b87ab8d6a44d411665/multidict-6.7.1-cp311-cp311-win_arm64.whl", hash = "sha256:b8c990b037d2fff2f4e33d3f21b9b531c5745b33a49a7d6dbe7a177266af44f6", size = 43159, upload-time = "2026-01-26T02:43:51.635Z" }, + { url = "https://files.pythonhosted.org/packages/8d/9c/f20e0e2cf80e4b2e4b1c365bf5fe104ee633c751a724246262db8f1a0b13/multidict-6.7.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172", size = 76893, upload-time = "2026-01-26T02:43:52.754Z" }, + { url = "https://files.pythonhosted.org/packages/fe/cf/18ef143a81610136d3da8193da9d80bfe1cb548a1e2d1c775f26b23d024a/multidict-6.7.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd", size = 45456, upload-time = "2026-01-26T02:43:53.893Z" }, + { url = "https://files.pythonhosted.org/packages/a9/65/1caac9d4cd32e8433908683446eebc953e82d22b03d10d41a5f0fefe991b/multidict-6.7.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7", size = 43872, upload-time = "2026-01-26T02:43:55.041Z" }, + { url = "https://files.pythonhosted.org/packages/cf/3b/d6bd75dc4f3ff7c73766e04e705b00ed6dbbaccf670d9e05a12b006f5a21/multidict-6.7.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53", size = 251018, upload-time = "2026-01-26T02:43:56.198Z" }, + { url = "https://files.pythonhosted.org/packages/fd/80/c959c5933adedb9ac15152e4067c702a808ea183a8b64cf8f31af8ad3155/multidict-6.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75", size = 258883, upload-time = "2026-01-26T02:43:57.499Z" }, + { url = "https://files.pythonhosted.org/packages/86/85/7ed40adafea3d4f1c8b916e3b5cc3a8e07dfcdcb9cd72800f4ed3ca1b387/multidict-6.7.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b", size = 242413, upload-time = "2026-01-26T02:43:58.755Z" }, + { url = "https://files.pythonhosted.org/packages/d2/57/b8565ff533e48595503c785f8361ff9a4fde4d67de25c207cd0ba3befd03/multidict-6.7.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733", size = 268404, upload-time = "2026-01-26T02:44:00.216Z" }, + { url = "https://files.pythonhosted.org/packages/e0/50/9810c5c29350f7258180dfdcb2e52783a0632862eb334c4896ac717cebcb/multidict-6.7.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a", size = 269456, upload-time = "2026-01-26T02:44:02.202Z" }, + { url = "https://files.pythonhosted.org/packages/f3/8d/5e5be3ced1d12966fefb5c4ea3b2a5b480afcea36406559442c6e31d4a48/multidict-6.7.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961", size = 256322, upload-time = "2026-01-26T02:44:03.56Z" }, + { url = "https://files.pythonhosted.org/packages/31/6e/d8a26d81ac166a5592782d208dd90dfdc0a7a218adaa52b45a672b46c122/multidict-6.7.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582", size = 253955, upload-time = "2026-01-26T02:44:04.845Z" }, + { url = "https://files.pythonhosted.org/packages/59/4c/7c672c8aad41534ba619bcd4ade7a0dc87ed6b8b5c06149b85d3dd03f0cd/multidict-6.7.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e", size = 251254, upload-time = "2026-01-26T02:44:06.133Z" }, + { url = "https://files.pythonhosted.org/packages/7b/bd/84c24de512cbafbdbc39439f74e967f19570ce7924e3007174a29c348916/multidict-6.7.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3", size = 252059, upload-time = "2026-01-26T02:44:07.518Z" }, + { url = "https://files.pythonhosted.org/packages/fa/ba/f5449385510825b73d01c2d4087bf6d2fccc20a2d42ac34df93191d3dd03/multidict-6.7.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6", size = 263588, upload-time = "2026-01-26T02:44:09.382Z" }, + { url = "https://files.pythonhosted.org/packages/d7/11/afc7c677f68f75c84a69fe37184f0f82fce13ce4b92f49f3db280b7e92b3/multidict-6.7.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a", size = 259642, upload-time = "2026-01-26T02:44:10.73Z" }, + { url = "https://files.pythonhosted.org/packages/2b/17/ebb9644da78c4ab36403739e0e6e0e30ebb135b9caf3440825001a0bddcb/multidict-6.7.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba", size = 251377, upload-time = "2026-01-26T02:44:12.042Z" }, + { url = "https://files.pythonhosted.org/packages/ca/a4/840f5b97339e27846c46307f2530a2805d9d537d8b8bd416af031cad7fa0/multidict-6.7.1-cp312-cp312-win32.whl", hash = "sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511", size = 41887, upload-time = "2026-01-26T02:44:14.245Z" }, + { url = "https://files.pythonhosted.org/packages/80/31/0b2517913687895f5904325c2069d6a3b78f66cc641a86a2baf75a05dcbb/multidict-6.7.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19", size = 46053, upload-time = "2026-01-26T02:44:15.371Z" }, + { url = "https://files.pythonhosted.org/packages/0c/5b/aba28e4ee4006ae4c7df8d327d31025d760ffa992ea23812a601d226e682/multidict-6.7.1-cp312-cp312-win_arm64.whl", hash = "sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf", size = 43307, upload-time = "2026-01-26T02:44:16.852Z" }, + { url = "https://files.pythonhosted.org/packages/f2/22/929c141d6c0dba87d3e1d38fbdf1ba8baba86b7776469f2bc2d3227a1e67/multidict-6.7.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23", size = 76174, upload-time = "2026-01-26T02:44:18.509Z" }, + { url = "https://files.pythonhosted.org/packages/c7/75/bc704ae15fee974f8fccd871305e254754167dce5f9e42d88a2def741a1d/multidict-6.7.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2", size = 45116, upload-time = "2026-01-26T02:44:19.745Z" }, + { url = "https://files.pythonhosted.org/packages/79/76/55cd7186f498ed080a18440c9013011eb548f77ae1b297206d030eb1180a/multidict-6.7.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445", size = 43524, upload-time = "2026-01-26T02:44:21.571Z" }, + { url = "https://files.pythonhosted.org/packages/e9/3c/414842ef8d5a1628d68edee29ba0e5bcf235dbfb3ccd3ea303a7fe8c72ff/multidict-6.7.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177", size = 249368, upload-time = "2026-01-26T02:44:22.803Z" }, + { url = "https://files.pythonhosted.org/packages/f6/32/befed7f74c458b4a525e60519fe8d87eef72bb1e99924fa2b0f9d97a221e/multidict-6.7.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23", size = 256952, upload-time = "2026-01-26T02:44:24.306Z" }, + { url = "https://files.pythonhosted.org/packages/03/d6/c878a44ba877f366630c860fdf74bfb203c33778f12b6ac274936853c451/multidict-6.7.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060", size = 240317, upload-time = "2026-01-26T02:44:25.772Z" }, + { url = "https://files.pythonhosted.org/packages/68/49/57421b4d7ad2e9e60e25922b08ceb37e077b90444bde6ead629095327a6f/multidict-6.7.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d", size = 267132, upload-time = "2026-01-26T02:44:27.648Z" }, + { url = "https://files.pythonhosted.org/packages/b7/fe/ec0edd52ddbcea2a2e89e174f0206444a61440b40f39704e64dc807a70bd/multidict-6.7.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed", size = 268140, upload-time = "2026-01-26T02:44:29.588Z" }, + { url = "https://files.pythonhosted.org/packages/b0/73/6e1b01cbeb458807aa0831742232dbdd1fa92bfa33f52a3f176b4ff3dc11/multidict-6.7.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429", size = 254277, upload-time = "2026-01-26T02:44:30.902Z" }, + { url = "https://files.pythonhosted.org/packages/6a/b2/5fb8c124d7561a4974c342bc8c778b471ebbeb3cc17df696f034a7e9afe7/multidict-6.7.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6", size = 252291, upload-time = "2026-01-26T02:44:32.31Z" }, + { url = "https://files.pythonhosted.org/packages/5a/96/51d4e4e06bcce92577fcd488e22600bd38e4fd59c20cb49434d054903bd2/multidict-6.7.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9", size = 250156, upload-time = "2026-01-26T02:44:33.734Z" }, + { url = "https://files.pythonhosted.org/packages/db/6b/420e173eec5fba721a50e2a9f89eda89d9c98fded1124f8d5c675f7a0c0f/multidict-6.7.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c", size = 249742, upload-time = "2026-01-26T02:44:35.222Z" }, + { url = "https://files.pythonhosted.org/packages/44/a3/ec5b5bd98f306bc2aa297b8c6f11a46714a56b1e6ef5ebda50a4f5d7c5fb/multidict-6.7.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84", size = 262221, upload-time = "2026-01-26T02:44:36.604Z" }, + { url = "https://files.pythonhosted.org/packages/cd/f7/e8c0d0da0cd1e28d10e624604e1a36bcc3353aaebdfdc3a43c72bc683a12/multidict-6.7.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d", size = 258664, upload-time = "2026-01-26T02:44:38.008Z" }, + { url = "https://files.pythonhosted.org/packages/52/da/151a44e8016dd33feed44f730bd856a66257c1ee7aed4f44b649fb7edeb3/multidict-6.7.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33", size = 249490, upload-time = "2026-01-26T02:44:39.386Z" }, + { url = "https://files.pythonhosted.org/packages/87/af/a3b86bf9630b732897f6fc3f4c4714b90aa4361983ccbdcd6c0339b21b0c/multidict-6.7.1-cp313-cp313-win32.whl", hash = "sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3", size = 41695, upload-time = "2026-01-26T02:44:41.318Z" }, + { url = "https://files.pythonhosted.org/packages/b2/35/e994121b0e90e46134673422dd564623f93304614f5d11886b1b3e06f503/multidict-6.7.1-cp313-cp313-win_amd64.whl", hash = "sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5", size = 45884, upload-time = "2026-01-26T02:44:42.488Z" }, + { url = "https://files.pythonhosted.org/packages/ca/61/42d3e5dbf661242a69c97ea363f2d7b46c567da8eadef8890022be6e2ab0/multidict-6.7.1-cp313-cp313-win_arm64.whl", hash = "sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df", size = 43122, upload-time = "2026-01-26T02:44:43.664Z" }, + { url = "https://files.pythonhosted.org/packages/6d/b3/e6b21c6c4f314bb956016b0b3ef2162590a529b84cb831c257519e7fde44/multidict-6.7.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1", size = 83175, upload-time = "2026-01-26T02:44:44.894Z" }, + { url = "https://files.pythonhosted.org/packages/fb/76/23ecd2abfe0957b234f6c960f4ade497f55f2c16aeb684d4ecdbf1c95791/multidict-6.7.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963", size = 48460, upload-time = "2026-01-26T02:44:46.106Z" }, + { url = "https://files.pythonhosted.org/packages/c4/57/a0ed92b23f3a042c36bc4227b72b97eca803f5f1801c1ab77c8a212d455e/multidict-6.7.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34", size = 46930, upload-time = "2026-01-26T02:44:47.278Z" }, + { url = "https://files.pythonhosted.org/packages/b5/66/02ec7ace29162e447f6382c495dc95826bf931d3818799bbef11e8f7df1a/multidict-6.7.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65", size = 242582, upload-time = "2026-01-26T02:44:48.604Z" }, + { url = "https://files.pythonhosted.org/packages/58/18/64f5a795e7677670e872673aca234162514696274597b3708b2c0d276cce/multidict-6.7.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292", size = 250031, upload-time = "2026-01-26T02:44:50.544Z" }, + { url = "https://files.pythonhosted.org/packages/c8/ed/e192291dbbe51a8290c5686f482084d31bcd9d09af24f63358c3d42fd284/multidict-6.7.1-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43", size = 228596, upload-time = "2026-01-26T02:44:51.951Z" }, + { url = "https://files.pythonhosted.org/packages/1e/7e/3562a15a60cf747397e7f2180b0a11dc0c38d9175a650e75fa1b4d325e15/multidict-6.7.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca", size = 257492, upload-time = "2026-01-26T02:44:53.902Z" }, + { url = "https://files.pythonhosted.org/packages/24/02/7d0f9eae92b5249bb50ac1595b295f10e263dd0078ebb55115c31e0eaccd/multidict-6.7.1-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd", size = 255899, upload-time = "2026-01-26T02:44:55.316Z" }, + { url = "https://files.pythonhosted.org/packages/00/e3/9b60ed9e23e64c73a5cde95269ef1330678e9c6e34dd4eb6b431b85b5a10/multidict-6.7.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7", size = 247970, upload-time = "2026-01-26T02:44:56.783Z" }, + { url = "https://files.pythonhosted.org/packages/3e/06/538e58a63ed5cfb0bd4517e346b91da32fde409d839720f664e9a4ae4f9d/multidict-6.7.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3", size = 245060, upload-time = "2026-01-26T02:44:58.195Z" }, + { url = "https://files.pythonhosted.org/packages/b2/2f/d743a3045a97c895d401e9bd29aaa09b94f5cbdf1bd561609e5a6c431c70/multidict-6.7.1-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4", size = 235888, upload-time = "2026-01-26T02:44:59.57Z" }, + { url = "https://files.pythonhosted.org/packages/38/83/5a325cac191ab28b63c52f14f1131f3b0a55ba3b9aa65a6d0bf2a9b921a0/multidict-6.7.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8", size = 243554, upload-time = "2026-01-26T02:45:01.054Z" }, + { url = "https://files.pythonhosted.org/packages/20/1f/9d2327086bd15da2725ef6aae624208e2ef828ed99892b17f60c344e57ed/multidict-6.7.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c", size = 252341, upload-time = "2026-01-26T02:45:02.484Z" }, + { url = "https://files.pythonhosted.org/packages/e8/2c/2a1aa0280cf579d0f6eed8ee5211c4f1730bd7e06c636ba2ee6aafda302e/multidict-6.7.1-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52", size = 246391, upload-time = "2026-01-26T02:45:03.862Z" }, + { url = "https://files.pythonhosted.org/packages/e5/03/7ca022ffc36c5a3f6e03b179a5ceb829be9da5783e6fe395f347c0794680/multidict-6.7.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108", size = 243422, upload-time = "2026-01-26T02:45:05.296Z" }, + { url = "https://files.pythonhosted.org/packages/dc/1d/b31650eab6c5778aceed46ba735bd97f7c7d2f54b319fa916c0f96e7805b/multidict-6.7.1-cp313-cp313t-win32.whl", hash = "sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32", size = 47770, upload-time = "2026-01-26T02:45:06.754Z" }, + { url = "https://files.pythonhosted.org/packages/ac/5b/2d2d1d522e51285bd61b1e20df8f47ae1a9d80839db0b24ea783b3832832/multidict-6.7.1-cp313-cp313t-win_amd64.whl", hash = "sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8", size = 53109, upload-time = "2026-01-26T02:45:08.044Z" }, + { url = "https://files.pythonhosted.org/packages/3d/a3/cc409ba012c83ca024a308516703cf339bdc4b696195644a7215a5164a24/multidict-6.7.1-cp313-cp313t-win_arm64.whl", hash = "sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118", size = 45573, upload-time = "2026-01-26T02:45:09.349Z" }, + { url = "https://files.pythonhosted.org/packages/91/cc/db74228a8be41884a567e88a62fd589a913708fcf180d029898c17a9a371/multidict-6.7.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee", size = 75190, upload-time = "2026-01-26T02:45:10.651Z" }, + { url = "https://files.pythonhosted.org/packages/d5/22/492f2246bb5b534abd44804292e81eeaf835388901f0c574bac4eeec73c5/multidict-6.7.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2", size = 44486, upload-time = "2026-01-26T02:45:11.938Z" }, + { url = "https://files.pythonhosted.org/packages/f1/4f/733c48f270565d78b4544f2baddc2fb2a245e5a8640254b12c36ac7ac68e/multidict-6.7.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1", size = 43219, upload-time = "2026-01-26T02:45:14.346Z" }, + { url = "https://files.pythonhosted.org/packages/24/bb/2c0c2287963f4259c85e8bcbba9182ced8d7fca65c780c38e99e61629d11/multidict-6.7.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d", size = 245132, upload-time = "2026-01-26T02:45:15.712Z" }, + { url = "https://files.pythonhosted.org/packages/a7/f9/44d4b3064c65079d2467888794dea218d1601898ac50222ab8a9a8094460/multidict-6.7.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31", size = 252420, upload-time = "2026-01-26T02:45:17.293Z" }, + { url = "https://files.pythonhosted.org/packages/8b/13/78f7275e73fa17b24c9a51b0bd9d73ba64bb32d0ed51b02a746eb876abe7/multidict-6.7.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048", size = 233510, upload-time = "2026-01-26T02:45:19.356Z" }, + { url = "https://files.pythonhosted.org/packages/4b/25/8167187f62ae3cbd52da7893f58cb036b47ea3fb67138787c76800158982/multidict-6.7.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362", size = 264094, upload-time = "2026-01-26T02:45:20.834Z" }, + { url = "https://files.pythonhosted.org/packages/a1/e7/69a3a83b7b030cf283fb06ce074a05a02322359783424d7edf0f15fe5022/multidict-6.7.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37", size = 260786, upload-time = "2026-01-26T02:45:22.818Z" }, + { url = "https://files.pythonhosted.org/packages/fe/3b/8ec5074bcfc450fe84273713b4b0a0dd47c0249358f5d82eb8104ffe2520/multidict-6.7.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709", size = 248483, upload-time = "2026-01-26T02:45:24.368Z" }, + { url = "https://files.pythonhosted.org/packages/48/5a/d5a99e3acbca0e29c5d9cba8f92ceb15dce78bab963b308ae692981e3a5d/multidict-6.7.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0", size = 248403, upload-time = "2026-01-26T02:45:25.982Z" }, + { url = "https://files.pythonhosted.org/packages/35/48/e58cd31f6c7d5102f2a4bf89f96b9cf7e00b6c6f3d04ecc44417c00a5a3c/multidict-6.7.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb", size = 240315, upload-time = "2026-01-26T02:45:27.487Z" }, + { url = "https://files.pythonhosted.org/packages/94/33/1cd210229559cb90b6786c30676bb0c58249ff42f942765f88793b41fdce/multidict-6.7.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd", size = 245528, upload-time = "2026-01-26T02:45:28.991Z" }, + { url = "https://files.pythonhosted.org/packages/64/f2/6e1107d226278c876c783056b7db43d800bb64c6131cec9c8dfb6903698e/multidict-6.7.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601", size = 258784, upload-time = "2026-01-26T02:45:30.503Z" }, + { url = "https://files.pythonhosted.org/packages/4d/c1/11f664f14d525e4a1b5327a82d4de61a1db604ab34c6603bb3c2cc63ad34/multidict-6.7.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1", size = 251980, upload-time = "2026-01-26T02:45:32.603Z" }, + { url = "https://files.pythonhosted.org/packages/e1/9f/75a9ac888121d0c5bbd4ecf4eead45668b1766f6baabfb3b7f66a410e231/multidict-6.7.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b", size = 243602, upload-time = "2026-01-26T02:45:34.043Z" }, + { url = "https://files.pythonhosted.org/packages/9a/e7/50bf7b004cc8525d80dbbbedfdc7aed3e4c323810890be4413e589074032/multidict-6.7.1-cp314-cp314-win32.whl", hash = "sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d", size = 40930, upload-time = "2026-01-26T02:45:36.278Z" }, + { url = "https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl", hash = "sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f", size = 45074, upload-time = "2026-01-26T02:45:37.546Z" }, + { url = "https://files.pythonhosted.org/packages/97/ab/22803b03285fa3a525f48217963da3a65ae40f6a1b6f6cf2768879e208f9/multidict-6.7.1-cp314-cp314-win_arm64.whl", hash = "sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5", size = 42471, upload-time = "2026-01-26T02:45:38.889Z" }, + { url = "https://files.pythonhosted.org/packages/e0/6d/f9293baa6146ba9507e360ea0292b6422b016907c393e2f63fc40ab7b7b5/multidict-6.7.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581", size = 82401, upload-time = "2026-01-26T02:45:40.254Z" }, + { url = "https://files.pythonhosted.org/packages/7a/68/53b5494738d83558d87c3c71a486504d8373421c3e0dbb6d0db48ad42ee0/multidict-6.7.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a", size = 48143, upload-time = "2026-01-26T02:45:41.635Z" }, + { url = "https://files.pythonhosted.org/packages/37/e8/5284c53310dcdc99ce5d66563f6e5773531a9b9fe9ec7a615e9bc306b05f/multidict-6.7.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c", size = 46507, upload-time = "2026-01-26T02:45:42.99Z" }, + { url = "https://files.pythonhosted.org/packages/e4/fc/6800d0e5b3875568b4083ecf5f310dcf91d86d52573160834fb4bfcf5e4f/multidict-6.7.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262", size = 239358, upload-time = "2026-01-26T02:45:44.376Z" }, + { url = "https://files.pythonhosted.org/packages/41/75/4ad0973179361cdf3a113905e6e088173198349131be2b390f9fa4da5fc6/multidict-6.7.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59", size = 246884, upload-time = "2026-01-26T02:45:47.167Z" }, + { url = "https://files.pythonhosted.org/packages/c3/9c/095bb28b5da139bd41fb9a5d5caff412584f377914bd8787c2aa98717130/multidict-6.7.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889", size = 225878, upload-time = "2026-01-26T02:45:48.698Z" }, + { url = "https://files.pythonhosted.org/packages/07/d0/c0a72000243756e8f5a277b6b514fa005f2c73d481b7d9e47cd4568aa2e4/multidict-6.7.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4", size = 253542, upload-time = "2026-01-26T02:45:50.164Z" }, + { url = "https://files.pythonhosted.org/packages/c0/6b/f69da15289e384ecf2a68837ec8b5ad8c33e973aa18b266f50fe55f24b8c/multidict-6.7.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d", size = 252403, upload-time = "2026-01-26T02:45:51.779Z" }, + { url = "https://files.pythonhosted.org/packages/a2/76/b9669547afa5a1a25cd93eaca91c0da1c095b06b6d2d8ec25b713588d3a1/multidict-6.7.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609", size = 244889, upload-time = "2026-01-26T02:45:53.27Z" }, + { url = "https://files.pythonhosted.org/packages/7e/a9/a50d2669e506dad33cfc45b5d574a205587b7b8a5f426f2fbb2e90882588/multidict-6.7.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489", size = 241982, upload-time = "2026-01-26T02:45:54.919Z" }, + { url = "https://files.pythonhosted.org/packages/c5/bb/1609558ad8b456b4827d3c5a5b775c93b87878fd3117ed3db3423dfbce1b/multidict-6.7.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c", size = 232415, upload-time = "2026-01-26T02:45:56.981Z" }, + { url = "https://files.pythonhosted.org/packages/d8/59/6f61039d2aa9261871e03ab9dc058a550d240f25859b05b67fd70f80d4b3/multidict-6.7.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e", size = 240337, upload-time = "2026-01-26T02:45:58.698Z" }, + { url = "https://files.pythonhosted.org/packages/a1/29/fdc6a43c203890dc2ae9249971ecd0c41deaedfe00d25cb6564b2edd99eb/multidict-6.7.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c", size = 248788, upload-time = "2026-01-26T02:46:00.862Z" }, + { url = "https://files.pythonhosted.org/packages/a9/14/a153a06101323e4cf086ecee3faadba52ff71633d471f9685c42e3736163/multidict-6.7.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9", size = 242842, upload-time = "2026-01-26T02:46:02.824Z" }, + { url = "https://files.pythonhosted.org/packages/41/5f/604ae839e64a4a6efc80db94465348d3b328ee955e37acb24badbcd24d83/multidict-6.7.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2", size = 240237, upload-time = "2026-01-26T02:46:05.898Z" }, + { url = "https://files.pythonhosted.org/packages/5f/60/c3a5187bf66f6fb546ff4ab8fb5a077cbdd832d7b1908d4365c7f74a1917/multidict-6.7.1-cp314-cp314t-win32.whl", hash = "sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7", size = 48008, upload-time = "2026-01-26T02:46:07.468Z" }, + { url = "https://files.pythonhosted.org/packages/0c/f7/addf1087b860ac60e6f382240f64fb99f8bfb532bb06f7c542b83c29ca61/multidict-6.7.1-cp314-cp314t-win_amd64.whl", hash = "sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5", size = 53542, upload-time = "2026-01-26T02:46:08.809Z" }, + { url = "https://files.pythonhosted.org/packages/4c/81/4629d0aa32302ef7b2ec65c75a728cc5ff4fa410c50096174c1632e70b3e/multidict-6.7.1-cp314-cp314t-win_arm64.whl", hash = "sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2", size = 44719, upload-time = "2026-01-26T02:46:11.146Z" }, + { url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" }, +] + +[[package]] +name = "openai" +version = "2.46.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "distro" }, + { name = "httpx" }, + { name = "jiter" }, + { name = "pydantic" }, + { name = "sniffio" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/af/ac/f725c4efbda8657d02be684607e5a2e5ce362e4790fdbcbdfb7c15018647/openai-2.46.0.tar.gz", hash = "sha256:0421e0735ac41451cad894af4cddf0435bfbf8cbc538ac0e15b3c062f2ddc06a", size = 1114628, upload-time = "2026-07-17T02:48:06.05Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ea/7b/206238ebcb50b235942b1c66dba4974776f2057402a8d91c399be587d66a/openai-2.46.0-py3-none-any.whl", hash = "sha256:672381db55efb3a1e2610f29304c130cccdd0b319bace4d492b2443cb64c1e7c", size = 1637556, upload-time = "2026-07-17T02:48:03.695Z" }, +] + +[[package]] +name = "openai-agents" +version = "0.18.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "griffelib" }, + { name = "mcp" }, + { name = "openai" }, + { name = "pydantic" }, + { name = "requests" }, + { name = "typing-extensions" }, + { name = "websockets" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/80/3f/b1162cad8720fafc9cf658d6896027385967f5006adfb92ae0dab2b54a70/openai_agents-0.18.3.tar.gz", hash = "sha256:e637f5f5a50692ccbedb0e4f7f2e4f8e2facfcddd41142f35faf90c89b700fc3", size = 5577652, upload-time = "2026-07-17T03:40:25.208Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/7c/08b9929a15131081898e38c5612d44ee293851d5c80f31ca1153efe82143/openai_agents-0.18.3-py3-none-any.whl", hash = "sha256:c6ed971fdeb34d39a9931787bd3960c1e84dc5d7345705794cc5cab8a1158d07", size = 880799, upload-time = "2026-07-17T03:40:23.163Z" }, +] + +[package.optional-dependencies] +litellm = [ + { name = "litellm" }, +] + +[[package]] +name = "orjson" +version = "3.11.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/0c/964746fcafbd16f8ff53219ad9f6b412b34f345c75f384ad434ceaadb538/orjson-3.11.9.tar.gz", hash = "sha256:4fef17e1f8722c11587a6ef18e35902450221da0028e65dbaaa543619e68e48f", size = 5599163, upload-time = "2026-05-06T15:11:08.309Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/51/3fb9e65ae76ee97bd611869a503fa3fc0a6e81dd8b737cf3003f682df7ff/orjson-3.11.9-cp311-cp311-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:f01c4818b3fc9b0da8e096722a84318071eaa118df35f6ed2344da0e73a5444f", size = 228522, upload-time = "2026-05-06T15:09:35.362Z" }, + { url = "https://files.pythonhosted.org/packages/16/fa/9d54b07cb3f3b0bfd57841478e42d7a0ece4a9f49f9907eecf5a45461687/orjson-3.11.9-cp311-cp311-macosx_15_0_arm64.whl", hash = "sha256:3ebca4179031ee716ed076ffadc29428e900512f6fccee8614c9983157fcf19c", size = 128463, upload-time = "2026-05-06T15:09:37.063Z" }, + { url = "https://files.pythonhosted.org/packages/88/b1/6ceafc2eefd0a553e3be77ce6c49d107e772485d9568629376171c50e634/orjson-3.11.9-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:48ee05097750de0ff69ed5b7bbcf0732182fd57a24043dcc2a1da780a5ead3a5", size = 132306, upload-time = "2026-05-06T15:09:38.299Z" }, + { url = "https://files.pythonhosted.org/packages/ea/76/f11311285324a40aab1e3031385c50b635a7cd0734fdaf60c7e89a696f60/orjson-3.11.9-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a6082706765a95a6680d812e1daf1c0cfe8adec7831b3ff3b625693f3b461b1c", size = 127988, upload-time = "2026-05-06T15:09:39.597Z" }, + { url = "https://files.pythonhosted.org/packages/9e/85/0ef63bcf1337f44031ce9b91b1919563f62a37527b3ea4368bb15a22e5d7/orjson-3.11.9-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:277fefe9d76ee17eb14debf399e3533d4d63b5f677a4d3719eb763536af1f4bd", size = 135188, upload-time = "2026-05-06T15:09:40.957Z" }, + { url = "https://files.pythonhosted.org/packages/05/94/b0d27090ea8a2095db3c2bd1b1c96f96f19bbb494d7fef33130e846e613d/orjson-3.11.9-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:03db380e3780fa0015ed776a90f20e8e20bb11dde13b216ce19e5718e3dfba62", size = 145937, upload-time = "2026-05-06T15:09:42.249Z" }, + { url = "https://files.pythonhosted.org/packages/09/eb/75d50c29c05b8054013e221e598820a365c8e64065312e75e202ed880709/orjson-3.11.9-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:33d7d766701847dc6729846362dc27895d2f2d2251264f9d10e7cb9878194877", size = 132758, upload-time = "2026-05-06T15:09:43.945Z" }, + { url = "https://files.pythonhosted.org/packages/49/bd/360686f39348aa88827cb6fbf7dc606fd41c831a35235e1abf1db8e3a9e6/orjson-3.11.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:147302878da387104b66bb4a8b0227d1d487e976ce41a8501916161072ed87b1", size = 133971, upload-time = "2026-05-06T15:09:45.239Z" }, + { url = "https://files.pythonhosted.org/packages/0e/30/3178eb16f3221aeef068b6f1f1ebe05f656ea5c6dffe9f6c917329fe17a3/orjson-3.11.9-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:3513550321f8c8c811a7c3297b8a630e82dc08e4c10216d07703c997776236cd", size = 141685, upload-time = "2026-05-06T15:09:46.858Z" }, + { url = "https://files.pythonhosted.org/packages/5f/f1/ff2f19ed0225f9680fafa42febca3570dd59444ebf190980738d376214c2/orjson-3.11.9-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:c5d001196b89fa9cf0a4ab79766cd835b991a166e4b621ba95089edc50c429ff", size = 415167, upload-time = "2026-05-06T15:09:48.312Z" }, + { url = "https://files.pythonhosted.org/packages/9b/61/863bddf0da6e9e586765414debd54b4e58db05f560902b6d00658cb88636/orjson-3.11.9-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:16969c9d369c98eb084889c6e4d2d39b77c7eb38ceccf8da2a9fff62ae908980", size = 147913, upload-time = "2026-05-06T15:09:49.733Z" }, + { url = "https://files.pythonhosted.org/packages/b6/8a/4081492586d75b073d60c5271a8d0f05a0955cabf1e34c8473f6fcd84235/orjson-3.11.9-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:63e0efbc991250c0b3143488fa57d95affcabbfc63c99c48d625dd37779aafe2", size = 136959, upload-time = "2026-05-06T15:09:51.311Z" }, + { url = "https://files.pythonhosted.org/packages/0d/bd/70b6ab193594d7abb875320c0a7c8335e846f28968c432c31042409c3c8d/orjson-3.11.9-cp311-cp311-win32.whl", hash = "sha256:14ed654580c1ed2bc217352ec82f91b047aef82951aa71c7f64e0dcb03c0e180", size = 131533, upload-time = "2026-05-06T15:09:52.637Z" }, + { url = "https://files.pythonhosted.org/packages/3f/17/1a1a228183d62d1b77e2c30d210f47dd4768b310ebe1607c63e3c0e3a71e/orjson-3.11.9-cp311-cp311-win_amd64.whl", hash = "sha256:57ea77fb70a448ce87d18fca050193202a3da5e54598f6501ca5476fb66cfe02", size = 127106, upload-time = "2026-05-06T15:09:54.204Z" }, + { url = "https://files.pythonhosted.org/packages/b8/95/285de5fa296d09681ee9c546cd4a8aeb773b701cf343dc125994f4d52953/orjson-3.11.9-cp311-cp311-win_arm64.whl", hash = "sha256:19b72ed11572a2ee51a67a903afbe5af504f84ed6f529c0fe44b0ab3fb5cc697", size = 126848, upload-time = "2026-05-06T15:09:55.551Z" }, + { url = "https://files.pythonhosted.org/packages/16/6d/11867a3ffa3a3608d84a4de51ef4dd0896d6b5cc9132fbe1daf593e677bc/orjson-3.11.9-cp312-cp312-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:9ef6fe90aadef185c7b128859f40beb24720b4ecea95379fc9000931179c3a49", size = 228515, upload-time = "2026-05-06T15:09:57.265Z" }, + { url = "https://files.pythonhosted.org/packages/24/75/05912954c8b288f34fcf5cd4b9b071cb4f6e77b9961e175e56ebb258089f/orjson-3.11.9-cp312-cp312-macosx_15_0_arm64.whl", hash = "sha256:e5c9b8f28e726e97d97696c826bc7bea5d71cecd63576dba92924a32c1961291", size = 128409, upload-time = "2026-05-06T15:09:59.063Z" }, + { url = "https://files.pythonhosted.org/packages/ab/86/1c3a47df3bc8191ea9ac51603bbb872a95167a364320c269f2557911f406/orjson-3.11.9-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:26a473dbb4162108b27901492546f83c76fdcea3d0eadff00ae7a07e18dcce09", size = 132106, upload-time = "2026-05-06T15:10:00.798Z" }, + { url = "https://files.pythonhosted.org/packages/d7/cf/b33b5f3e695ae7d63feef9d915c37cc3b8f465493dcd4f8e0b4c697a2366/orjson-3.11.9-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:011382e2a60fda9d46f1cdee31068cfc52ffe952b587d683ec0463002802a0f4", size = 127864, upload-time = "2026-05-06T15:10:02.15Z" }, + { url = "https://files.pythonhosted.org/packages/31/6a/6cf69385a58208024fcb8c014e2141b8ce838aba6492b589f8acfff97fab/orjson-3.11.9-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c2d3dc759490128c5c1711a53eeaa8ee1d437fd0038ffd2b6008abf46db3f882", size = 135213, upload-time = "2026-05-06T15:10:03.515Z" }, + { url = "https://files.pythonhosted.org/packages/e8/f8/0b1bd3e8f2efcdd376af5c8cfd79eaf13f018080c0089c80ebd724e3c7fb/orjson-3.11.9-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d8ea516b3726d190e1b4297e6f4e7a8650347ae053868a18163b4dd3641d1fff", size = 145994, upload-time = "2026-05-06T15:10:05.083Z" }, + { url = "https://files.pythonhosted.org/packages/f3/59/dab79f61044c529d2c81aecdc589b1f833a1c8dec11ba3b1c2498a02ca7e/orjson-3.11.9-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:380cdce7ba24989af81d0a7013d0aaec5d0e2a21734c0e2681b1bc4f141957fe", size = 132744, upload-time = "2026-05-06T15:10:06.853Z" }, + { url = "https://files.pythonhosted.org/packages/0e/a4/82b7a2fe5d8a67a59ed831b24d59a3d46ea7d207b66e1602d376541d94a6/orjson-3.11.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:be4fa4f0af7fa18951f7ab3fc2148e223af211bf03f59e1c6034ec3f97f21d61", size = 134014, upload-time = "2026-05-06T15:10:08.213Z" }, + { url = "https://files.pythonhosted.org/packages/50/c7/375e83a76851b73b2e39f3bcf0e5a19e2b89bad13e5bca97d0b293d27f24/orjson-3.11.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a8f5f8bc7ce7d59f08d9f99fa510c06496164a24cb5f3d34537dbd9ca30132e2", size = 141509, upload-time = "2026-05-06T15:10:09.595Z" }, + { url = "https://files.pythonhosted.org/packages/7f/7c/49d5d82a3d3097f641f094f552131f1e2723b0b8cb0fa2874ab65ecfffa6/orjson-3.11.9-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:4d7fde5501b944f83b3e665e1b31343ff6e154b15560a16b7130ea1e594a4206", size = 415127, upload-time = "2026-05-06T15:10:11.049Z" }, + { url = "https://files.pythonhosted.org/packages/3a/dc/7446c538590d55f455647e5f3c61fc33f7108714e7afcffa6a2a033f8350/orjson-3.11.9-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:cde1a448023ba7d5bb4c01c5afb48894380b5e4956e0627266526587ef4e535f", size = 148025, upload-time = "2026-05-06T15:10:12.842Z" }, + { url = "https://files.pythonhosted.org/packages/df/e5/4d2d8af06f788329b4f78f8cc3679bb395392fcaa1e4d8d3c33e85308fa4/orjson-3.11.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:71e63adb0e1f1ed5d9e168f50a91ceb93ae6420731d222dc7da5c69409aa47aa", size = 136943, upload-time = "2026-05-06T15:10:14.405Z" }, + { url = "https://files.pythonhosted.org/packages/06/69/850264ccf6d80f6b174620d30a87f65c9b1490aba33fe6b62798e618cad3/orjson-3.11.9-cp312-cp312-win32.whl", hash = "sha256:2d057a602cdd19a0ad680417527c45b6961a095081c0f46fe0e03e304aac6470", size = 131606, upload-time = "2026-05-06T15:10:15.791Z" }, + { url = "https://files.pythonhosted.org/packages/b9/d5/973a43fc9c55e20f2051e9830997649f669be0cb3ca52192087c0143f118/orjson-3.11.9-cp312-cp312-win_amd64.whl", hash = "sha256:59e403b1cc5a676da8eaf31f6254801b7341b3e29efa85f92b48d272637e77be", size = 127101, upload-time = "2026-05-06T15:10:17.129Z" }, + { url = "https://files.pythonhosted.org/packages/fe/ae/495470f0e4a18f73fa10b7f6b84b464ec4cc5291c4e0c7c2a6c400bef006/orjson-3.11.9-cp312-cp312-win_arm64.whl", hash = "sha256:9af678d6488357948f1f84c6cd1c1d397c014e1ae2f98ae082a44eb48f602624", size = 126736, upload-time = "2026-05-06T15:10:18.645Z" }, + { url = "https://files.pythonhosted.org/packages/32/33/93fcc25907235c344ae73122f8a4e01d2d393ef062b4af7d2e2487a32c37/orjson-3.11.9-cp313-cp313-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:4bab1b2d6141fe7b32ae71dac905666ece4f94936efbfb13d55bb7739a3a6021", size = 228458, upload-time = "2026-05-06T15:10:20.079Z" }, + { url = "https://files.pythonhosted.org/packages/8f/27/b1e6dadb3c080313c03fdd8067b85e6a0460c7d8d6a1c3984ef77b904e4d/orjson-3.11.9-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:844417969855fc7a41be124aafe83dc424592a7f77cd4501900c67307122b92c", size = 128368, upload-time = "2026-05-06T15:10:21.549Z" }, + { url = "https://files.pythonhosted.org/packages/21/0f/c9ede0bf052f6b4051e64a7d4fa91b725cccf8321a6a786e86eb03519f00/orjson-3.11.9-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffe02797b5e9f3a9d8292ddcd289b474ad13e81ad83cd1891a240811f1d2cb81", size = 132070, upload-time = "2026-05-06T15:10:23.371Z" }, + { url = "https://files.pythonhosted.org/packages/fd/26/d398e28048dc18205bbe812f2c88cb9b40313db2470778e25964796458fe/orjson-3.11.9-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0e4eed3b200023042814d2fc8a5d2e880f13b52e1ed2485e83da4f3962f7dc1a", size = 127892, upload-time = "2026-05-06T15:10:24.714Z" }, + { url = "https://files.pythonhosted.org/packages/66/60/52b0054c4c700d5aa7fc5b7ca96917400d8f061307778578e67a10e25852/orjson-3.11.9-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8aff7da9952a5ad1cef8e68017724d96c7b9a66e99e91d6252e1b133d67a7b10", size = 135217, upload-time = "2026-05-06T15:10:26.084Z" }, + { url = "https://files.pythonhosted.org/packages/d5/97/1e3dc2b2a28b7b2528f403d2fc1d79ec5f39af3bc143ab65d3ec26426385/orjson-3.11.9-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4d4e98d6f3b8afed8bc8cd9718ec0cdf46661826beefb53fe8eafb37f2bf0362", size = 145980, upload-time = "2026-05-06T15:10:28.062Z" }, + { url = "https://files.pythonhosted.org/packages/fc/39/31fbfe7850f2de32dee7e7e5c09f26d403ab01e440ac96001c6b01ad3c99/orjson-3.11.9-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3a81d52442a7c99b3662333235b3adf96a1715864658b35bb797212be7bddb97", size = 132738, upload-time = "2026-05-06T15:10:29.727Z" }, + { url = "https://files.pythonhosted.org/packages/a1/08/dca0082dd2a194acb93e5457e73455388e2e2ca464a2672449a9ddbb679d/orjson-3.11.9-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4e39364e726a8fff737309aff059ff67d8a8c8d5b677be7bb49a8b3e84b7e218", size = 134033, upload-time = "2026-05-06T15:10:31.152Z" }, + { url = "https://files.pythonhosted.org/packages/11/d4/5bdb0626801230139987385554c5d4c42255218ac906525bf4347f22cd95/orjson-3.11.9-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4fd66214623f1b17501df9f0543bef0b833979ab5b6ded1e1d123222866aa8c9", size = 141492, upload-time = "2026-05-06T15:10:32.641Z" }, + { url = "https://files.pythonhosted.org/packages/fa/88/a21fb53b3ede6703aede6dce4710ed4111e5b201cfa6bbff5e544f9d47d7/orjson-3.11.9-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:8ecc30f10465fa1e0ce13fd01d9e22c316e5053a719a8d915d4545a09a5ff677", size = 415087, upload-time = "2026-05-06T15:10:34.438Z" }, + { url = "https://files.pythonhosted.org/packages/3d/57/1b30daf70f0d8180e9a73cefbfbdd99e4bf19eb020466502b01fba7e0e50/orjson-3.11.9-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:97db4c94a7db398a5bd636273324f0b3fd58b350bbbac8bb380ceb825a9b40f4", size = 148031, upload-time = "2026-05-06T15:10:36.358Z" }, + { url = "https://files.pythonhosted.org/packages/04/83/45fbb6d962e260807f99441db9613cee868ceda4baceda59b3720a563f97/orjson-3.11.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9f78cf8fec5bd627f4082b8dfeac7871b43d7f3274904492a43dab39f18a19a0", size = 136915, upload-time = "2026-05-06T15:10:38.013Z" }, + { url = "https://files.pythonhosted.org/packages/5f/cc/2d10025f9056d376e4127ec05a5808b218d46f035fdc08178a5411b34250/orjson-3.11.9-cp313-cp313-win32.whl", hash = "sha256:d4087e5c0209a0a8efe4de3303c234b9c44d1174161dcd851e8eea07c7560b32", size = 131613, upload-time = "2026-05-06T15:10:39.569Z" }, + { url = "https://files.pythonhosted.org/packages/67/bd/2775ff28bfe883b9aa1ff348300542eb2ef1ee18d8ae0e3a49846817a865/orjson-3.11.9-cp313-cp313-win_amd64.whl", hash = "sha256:051b102c93b4f634e89f3866b07b9a9a98915ada541f4ec30f177067b2694979", size = 127086, upload-time = "2026-05-06T15:10:41.262Z" }, + { url = "https://files.pythonhosted.org/packages/91/2b/d26799e580939e32a7da9a39531bc9e58e15ca32ffaa6a8cb3e9bb0d22cd/orjson-3.11.9-cp313-cp313-win_arm64.whl", hash = "sha256:cce9127885941bd28f080cecf1f1d288336b7e0d812c345b08be88b572796254", size = 126696, upload-time = "2026-05-06T15:10:42.651Z" }, + { url = "https://files.pythonhosted.org/packages/8e/eb/5da01e356015aee6ecfa1187ced87aef51364e306f5e695dd52719bf0e78/orjson-3.11.9-cp314-cp314-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:b6ef1979adc4bc243523f1a2ba91418030a8e29b0a99cbe7e0e2d6807d4dce6e", size = 228465, upload-time = "2026-05-06T15:10:44.097Z" }, + { url = "https://files.pythonhosted.org/packages/64/62/3e0e0c14c957133bcd855395c62b55ed4e3b0af23ffea11b032cb1dcbdb1/orjson-3.11.9-cp314-cp314-macosx_15_0_arm64.whl", hash = "sha256:f36b7f32c7c0db4a719f1fc5824db4a9c6f8bd1a354debb91faf26ebf3a4c71e", size = 128364, upload-time = "2026-05-06T15:10:45.839Z" }, + { url = "https://files.pythonhosted.org/packages/5a/5a/07d8aa117211a8ed7630bda80c8c0b14d04e0f8dcf99bcf49656e4a710eb/orjson-3.11.9-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:08f4d8ebb44925c794e535b2bebc507cebf32209df81de22ae285fb0d8d66de0", size = 132063, upload-time = "2026-05-06T15:10:47.267Z" }, + { url = "https://files.pythonhosted.org/packages/d6/ec/4acaf21483e18aa945be74a474c74b434f284b549f275a0a39b9f98956e9/orjson-3.11.9-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6cc7923789694fd58f001cbcac7e47abc13af4d560ebbfcf3b41a8b1a0748124", size = 122356, upload-time = "2026-05-06T15:10:48.765Z" }, + { url = "https://files.pythonhosted.org/packages/13/d8/5f0555e7638801323b7a75850f92e7dfa891bc84fe27a1ba4449170d1200/orjson-3.11.9-cp314-cp314-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ea5c46eb2d3af39e806b986f4b09d5c2706a1f5afde3cbf7544ce6616127173c", size = 129592, upload-time = "2026-05-06T15:10:50.13Z" }, + { url = "https://files.pythonhosted.org/packages/b6/30/ed9860412a3603ceb3c5955bfd72d28b9d0e7ba6ed81add14f83d7114236/orjson-3.11.9-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f5d89a2ed90731df3be64bab0aa44f78bff39fdc9d71c291f4a8023aa46425b7", size = 140491, upload-time = "2026-05-06T15:10:51.582Z" }, + { url = "https://files.pythonhosted.org/packages/d0/17/adc514dea7ac7c505527febf884934b815d34f0c7b8693c1a8b39c5c4a57/orjson-3.11.9-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:25e4aed0312d292c09f61af25bba34e0b2c88546041472b09088c39a4d828af1", size = 127309, upload-time = "2026-05-06T15:10:53.329Z" }, + { url = "https://files.pythonhosted.org/packages/76/3e/c0b690253f0b82d86e99949af13533363acfb5432ecb5d53dd5b3bce9c34/orjson-3.11.9-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aaea64f3f467d22e70eeed68bdccb3bc4f83f650446c4a03c59f2cba28a108db", size = 134030, upload-time = "2026-05-06T15:10:54.988Z" }, + { url = "https://files.pythonhosted.org/packages/c1/7a/bc82a0bb25e9faaf92dc4d9ef002732efc09737706af83e346788641d4a7/orjson-3.11.9-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a028425d1b440c5d92a6be1e1a020739dfe67ea87d96c6dbe828c1b30041728b", size = 141482, upload-time = "2026-05-06T15:10:56.663Z" }, + { url = "https://files.pythonhosted.org/packages/01/55/e69188b939f77d5d32a9833745ace31ea5ccae3ab613a1ec185d3cd2c4fb/orjson-3.11.9-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:5b192c6cf397e4455b11523c5cf2b18ed084c1bbd61b6c0926344d2129481972", size = 415178, upload-time = "2026-05-06T15:10:58.446Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1a/b8a5a7ac527e80b9cb11d51e3f6689b709279183264b9ec5c7bc680bb8b5/orjson-3.11.9-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:ea407d4ccf5891d667d045fecae97a7a1e5e87b3b97f97ae1803c2e741130be0", size = 148089, upload-time = "2026-05-06T15:11:00.441Z" }, + { url = "https://files.pythonhosted.org/packages/97/4e/00503f64204bf859b37213a63927028f30fb6268cd8677fb0a5ad48155e1/orjson-3.11.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5f63aaf97afd9f6dec5b1a68e1b8da12bfccb4cb9a9a65c3e0b6c847849e7586", size = 136921, upload-time = "2026-05-06T15:11:02.176Z" }, + { url = "https://files.pythonhosted.org/packages/0d/ba/a23b82a0a8d0ed7bed4e5f5035aae751cad4ff6a1e8d2ecd14d8860f5929/orjson-3.11.9-cp314-cp314-win32.whl", hash = "sha256:e30ab17845bb9fa54ccf67fa4f9f5282652d54faa6d17452f47d0f369d038673", size = 131638, upload-time = "2026-05-06T15:11:03.696Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c3/0c6798456bade745c75c452342dabacce5798196483e77e643be1f53877d/orjson-3.11.9-cp314-cp314-win_amd64.whl", hash = "sha256:32ef5f4283a3be81913947d19608eacb7c6608026851123790cd9cc8982af34b", size = 127078, upload-time = "2026-05-06T15:11:05.123Z" }, + { url = "https://files.pythonhosted.org/packages/16/21/5a3f1e8913103b703a436a5664238e5b965ec392b555fe68943ea3691e6b/orjson-3.11.9-cp314-cp314-win_arm64.whl", hash = "sha256:eebdbdeef0094e4f5aefa20dcd4eb2368ab5e7a3b4edea27f1e7b2892e009cf9", size = 126687, upload-time = "2026-05-06T15:11:06.602Z" }, +] + +[[package]] +name = "packaging" +version = "26.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/65/ee/299d360cdc32edc7d2cf530f3accf79c4fca01e96ffc950d8a52213bd8e4/packaging-26.0.tar.gz", hash = "sha256:00243ae351a257117b6a241061796684b084ed1c516a08c48a3f7e147a9d80b4", size = 143416, upload-time = "2026-01-21T20:50:39.064Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/b9/c538f279a4e237a006a2c98387d081e9eb060d203d8ed34467cc0f0b9b53/packaging-26.0-py3-none-any.whl", hash = "sha256:b36f1fef9334a5588b4166f8bcd26a14e521f2b55e6b9de3aaa80d3ff7a37529", size = 74366, upload-time = "2026-01-21T20:50:37.788Z" }, +] + +[[package]] +name = "platformdirs" +version = "4.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/47/e4501f49c178ae1d9f4a75073fda4204f52647993f075a9db4d14930e0c5/platformdirs-4.10.0.tar.gz", hash = "sha256:31e761a6a0ca04faf7353ea759bdba55652be214725111e5aac52dfa29d4bef7", size = 31224, upload-time = "2026-05-28T03:32:53.587Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl", hash = "sha256:fb516cdb12eb0d857d0cd85a7c57cea4d060bee4578d6cf5a14dfdf8cbf8784a", size = 22743, upload-time = "2026-05-28T03:32:52.175Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "propcache" +version = "0.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9e/da/e9fc233cf63743258bff22b3dfa7ea5baef7b5bc324af47a0ad89b8ffc6f/propcache-0.4.1.tar.gz", hash = "sha256:f48107a8c637e80362555f37ecf49abe20370e557cc4ab374f04ec4423c97c3d", size = 46442, upload-time = "2025-10-08T19:49:02.291Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8c/d4/4e2c9aaf7ac2242b9358f98dccd8f90f2605402f5afeff6c578682c2c491/propcache-0.4.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:60a8fda9644b7dfd5dece8c61d8a85e271cb958075bfc4e01083c148b61a7caf", size = 80208, upload-time = "2025-10-08T19:46:24.597Z" }, + { url = "https://files.pythonhosted.org/packages/c2/21/d7b68e911f9c8e18e4ae43bdbc1e1e9bbd971f8866eb81608947b6f585ff/propcache-0.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c30b53e7e6bda1d547cabb47c825f3843a0a1a42b0496087bb58d8fedf9f41b5", size = 45777, upload-time = "2025-10-08T19:46:25.733Z" }, + { url = "https://files.pythonhosted.org/packages/d3/1d/11605e99ac8ea9435651ee71ab4cb4bf03f0949586246476a25aadfec54a/propcache-0.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:6918ecbd897443087a3b7cd978d56546a812517dcaaca51b49526720571fa93e", size = 47647, upload-time = "2025-10-08T19:46:27.304Z" }, + { url = "https://files.pythonhosted.org/packages/58/1a/3c62c127a8466c9c843bccb503d40a273e5cc69838805f322e2826509e0d/propcache-0.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3d902a36df4e5989763425a8ab9e98cd8ad5c52c823b34ee7ef307fd50582566", size = 214929, upload-time = "2025-10-08T19:46:28.62Z" }, + { url = "https://files.pythonhosted.org/packages/56/b9/8fa98f850960b367c4b8fe0592e7fc341daa7a9462e925228f10a60cf74f/propcache-0.4.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a9695397f85973bb40427dedddf70d8dc4a44b22f1650dd4af9eedf443d45165", size = 221778, upload-time = "2025-10-08T19:46:30.358Z" }, + { url = "https://files.pythonhosted.org/packages/46/a6/0ab4f660eb59649d14b3d3d65c439421cf2f87fe5dd68591cbe3c1e78a89/propcache-0.4.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2bb07ffd7eaad486576430c89f9b215f9e4be68c4866a96e97db9e97fead85dc", size = 228144, upload-time = "2025-10-08T19:46:32.607Z" }, + { url = "https://files.pythonhosted.org/packages/52/6a/57f43e054fb3d3a56ac9fc532bc684fc6169a26c75c353e65425b3e56eef/propcache-0.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fd6f30fdcf9ae2a70abd34da54f18da086160e4d7d9251f81f3da0ff84fc5a48", size = 210030, upload-time = "2025-10-08T19:46:33.969Z" }, + { url = "https://files.pythonhosted.org/packages/40/e2/27e6feebb5f6b8408fa29f5efbb765cd54c153ac77314d27e457a3e993b7/propcache-0.4.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:fc38cba02d1acba4e2869eef1a57a43dfbd3d49a59bf90dda7444ec2be6a5570", size = 208252, upload-time = "2025-10-08T19:46:35.309Z" }, + { url = "https://files.pythonhosted.org/packages/9e/f8/91c27b22ccda1dbc7967f921c42825564fa5336a01ecd72eb78a9f4f53c2/propcache-0.4.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:67fad6162281e80e882fb3ec355398cf72864a54069d060321f6cd0ade95fe85", size = 202064, upload-time = "2025-10-08T19:46:36.993Z" }, + { url = "https://files.pythonhosted.org/packages/f2/26/7f00bd6bd1adba5aafe5f4a66390f243acab58eab24ff1a08bebb2ef9d40/propcache-0.4.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:f10207adf04d08bec185bae14d9606a1444715bc99180f9331c9c02093e1959e", size = 212429, upload-time = "2025-10-08T19:46:38.398Z" }, + { url = "https://files.pythonhosted.org/packages/84/89/fd108ba7815c1117ddca79c228f3f8a15fc82a73bca8b142eb5de13b2785/propcache-0.4.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:e9b0d8d0845bbc4cfcdcbcdbf5086886bc8157aa963c31c777ceff7846c77757", size = 216727, upload-time = "2025-10-08T19:46:39.732Z" }, + { url = "https://files.pythonhosted.org/packages/79/37/3ec3f7e3173e73f1d600495d8b545b53802cbf35506e5732dd8578db3724/propcache-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:981333cb2f4c1896a12f4ab92a9cc8f09ea664e9b7dbdc4eff74627af3a11c0f", size = 205097, upload-time = "2025-10-08T19:46:41.025Z" }, + { url = "https://files.pythonhosted.org/packages/61/b0/b2631c19793f869d35f47d5a3a56fb19e9160d3c119f15ac7344fc3ccae7/propcache-0.4.1-cp311-cp311-win32.whl", hash = "sha256:f1d2f90aeec838a52f1c1a32fe9a619fefd5e411721a9117fbf82aea638fe8a1", size = 38084, upload-time = "2025-10-08T19:46:42.693Z" }, + { url = "https://files.pythonhosted.org/packages/f4/78/6cce448e2098e9f3bfc91bb877f06aa24b6ccace872e39c53b2f707c4648/propcache-0.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:364426a62660f3f699949ac8c621aad6977be7126c5807ce48c0aeb8e7333ea6", size = 41637, upload-time = "2025-10-08T19:46:43.778Z" }, + { url = "https://files.pythonhosted.org/packages/9c/e9/754f180cccd7f51a39913782c74717c581b9cc8177ad0e949f4d51812383/propcache-0.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:e53f3a38d3510c11953f3e6a33f205c6d1b001129f972805ca9b42fc308bc239", size = 38064, upload-time = "2025-10-08T19:46:44.872Z" }, + { url = "https://files.pythonhosted.org/packages/a2/0f/f17b1b2b221d5ca28b4b876e8bb046ac40466513960646bda8e1853cdfa2/propcache-0.4.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:e153e9cd40cc8945138822807139367f256f89c6810c2634a4f6902b52d3b4e2", size = 80061, upload-time = "2025-10-08T19:46:46.075Z" }, + { url = "https://files.pythonhosted.org/packages/76/47/8ccf75935f51448ba9a16a71b783eb7ef6b9ee60f5d14c7f8a8a79fbeed7/propcache-0.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:cd547953428f7abb73c5ad82cbb32109566204260d98e41e5dfdc682eb7f8403", size = 46037, upload-time = "2025-10-08T19:46:47.23Z" }, + { url = "https://files.pythonhosted.org/packages/0a/b6/5c9a0e42df4d00bfb4a3cbbe5cf9f54260300c88a0e9af1f47ca5ce17ac0/propcache-0.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f048da1b4f243fc44f205dfd320933a951b8d89e0afd4c7cacc762a8b9165207", size = 47324, upload-time = "2025-10-08T19:46:48.384Z" }, + { url = "https://files.pythonhosted.org/packages/9e/d3/6c7ee328b39a81ee877c962469f1e795f9db87f925251efeb0545e0020d0/propcache-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ec17c65562a827bba85e3872ead335f95405ea1674860d96483a02f5c698fa72", size = 225505, upload-time = "2025-10-08T19:46:50.055Z" }, + { url = "https://files.pythonhosted.org/packages/01/5d/1c53f4563490b1d06a684742cc6076ef944bc6457df6051b7d1a877c057b/propcache-0.4.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:405aac25c6394ef275dee4c709be43745d36674b223ba4eb7144bf4d691b7367", size = 230242, upload-time = "2025-10-08T19:46:51.815Z" }, + { url = "https://files.pythonhosted.org/packages/20/e1/ce4620633b0e2422207c3cb774a0ee61cac13abc6217763a7b9e2e3f4a12/propcache-0.4.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0013cb6f8dde4b2a2f66903b8ba740bdfe378c943c4377a200551ceb27f379e4", size = 238474, upload-time = "2025-10-08T19:46:53.208Z" }, + { url = "https://files.pythonhosted.org/packages/46/4b/3aae6835b8e5f44ea6a68348ad90f78134047b503765087be2f9912140ea/propcache-0.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15932ab57837c3368b024473a525e25d316d8353016e7cc0e5ba9eb343fbb1cf", size = 221575, upload-time = "2025-10-08T19:46:54.511Z" }, + { url = "https://files.pythonhosted.org/packages/6e/a5/8a5e8678bcc9d3a1a15b9a29165640d64762d424a16af543f00629c87338/propcache-0.4.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:031dce78b9dc099f4c29785d9cf5577a3faf9ebf74ecbd3c856a7b92768c3df3", size = 216736, upload-time = "2025-10-08T19:46:56.212Z" }, + { url = "https://files.pythonhosted.org/packages/f1/63/b7b215eddeac83ca1c6b934f89d09a625aa9ee4ba158338854c87210cc36/propcache-0.4.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:ab08df6c9a035bee56e31af99be621526bd237bea9f32def431c656b29e41778", size = 213019, upload-time = "2025-10-08T19:46:57.595Z" }, + { url = "https://files.pythonhosted.org/packages/57/74/f580099a58c8af587cac7ba19ee7cb418506342fbbe2d4a4401661cca886/propcache-0.4.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4d7af63f9f93fe593afbf104c21b3b15868efb2c21d07d8732c0c4287e66b6a6", size = 220376, upload-time = "2025-10-08T19:46:59.067Z" }, + { url = "https://files.pythonhosted.org/packages/c4/ee/542f1313aff7eaf19c2bb758c5d0560d2683dac001a1c96d0774af799843/propcache-0.4.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:cfc27c945f422e8b5071b6e93169679e4eb5bf73bbcbf1ba3ae3a83d2f78ebd9", size = 226988, upload-time = "2025-10-08T19:47:00.544Z" }, + { url = "https://files.pythonhosted.org/packages/8f/18/9c6b015dd9c6930f6ce2229e1f02fb35298b847f2087ea2b436a5bfa7287/propcache-0.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:35c3277624a080cc6ec6f847cbbbb5b49affa3598c4535a0a4682a697aaa5c75", size = 215615, upload-time = "2025-10-08T19:47:01.968Z" }, + { url = "https://files.pythonhosted.org/packages/80/9e/e7b85720b98c45a45e1fca6a177024934dc9bc5f4d5dd04207f216fc33ed/propcache-0.4.1-cp312-cp312-win32.whl", hash = "sha256:671538c2262dadb5ba6395e26c1731e1d52534bfe9ae56d0b5573ce539266aa8", size = 38066, upload-time = "2025-10-08T19:47:03.503Z" }, + { url = "https://files.pythonhosted.org/packages/54/09/d19cff2a5aaac632ec8fc03737b223597b1e347416934c1b3a7df079784c/propcache-0.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:cb2d222e72399fcf5890d1d5cc1060857b9b236adff2792ff48ca2dfd46c81db", size = 41655, upload-time = "2025-10-08T19:47:04.973Z" }, + { url = "https://files.pythonhosted.org/packages/68/ab/6b5c191bb5de08036a8c697b265d4ca76148efb10fa162f14af14fb5f076/propcache-0.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:204483131fb222bdaaeeea9f9e6c6ed0cac32731f75dfc1d4a567fc1926477c1", size = 37789, upload-time = "2025-10-08T19:47:06.077Z" }, + { url = "https://files.pythonhosted.org/packages/bf/df/6d9c1b6ac12b003837dde8a10231a7344512186e87b36e855bef32241942/propcache-0.4.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:43eedf29202c08550aac1d14e0ee619b0430aaef78f85864c1a892294fbc28cf", size = 77750, upload-time = "2025-10-08T19:47:07.648Z" }, + { url = "https://files.pythonhosted.org/packages/8b/e8/677a0025e8a2acf07d3418a2e7ba529c9c33caf09d3c1f25513023c1db56/propcache-0.4.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d62cdfcfd89ccb8de04e0eda998535c406bf5e060ffd56be6c586cbcc05b3311", size = 44780, upload-time = "2025-10-08T19:47:08.851Z" }, + { url = "https://files.pythonhosted.org/packages/89/a4/92380f7ca60f99ebae761936bc48a72a639e8a47b29050615eef757cb2a7/propcache-0.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cae65ad55793da34db5f54e4029b89d3b9b9490d8abe1b4c7ab5d4b8ec7ebf74", size = 46308, upload-time = "2025-10-08T19:47:09.982Z" }, + { url = "https://files.pythonhosted.org/packages/2d/48/c5ac64dee5262044348d1d78a5f85dd1a57464a60d30daee946699963eb3/propcache-0.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:333ddb9031d2704a301ee3e506dc46b1fe5f294ec198ed6435ad5b6a085facfe", size = 208182, upload-time = "2025-10-08T19:47:11.319Z" }, + { url = "https://files.pythonhosted.org/packages/c6/0c/cd762dd011a9287389a6a3eb43aa30207bde253610cca06824aeabfe9653/propcache-0.4.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:fd0858c20f078a32cf55f7e81473d96dcf3b93fd2ccdb3d40fdf54b8573df3af", size = 211215, upload-time = "2025-10-08T19:47:13.146Z" }, + { url = "https://files.pythonhosted.org/packages/30/3e/49861e90233ba36890ae0ca4c660e95df565b2cd15d4a68556ab5865974e/propcache-0.4.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:678ae89ebc632c5c204c794f8dab2837c5f159aeb59e6ed0539500400577298c", size = 218112, upload-time = "2025-10-08T19:47:14.913Z" }, + { url = "https://files.pythonhosted.org/packages/f1/8b/544bc867e24e1bd48f3118cecd3b05c694e160a168478fa28770f22fd094/propcache-0.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d472aeb4fbf9865e0c6d622d7f4d54a4e101a89715d8904282bb5f9a2f476c3f", size = 204442, upload-time = "2025-10-08T19:47:16.277Z" }, + { url = "https://files.pythonhosted.org/packages/50/a6/4282772fd016a76d3e5c0df58380a5ea64900afd836cec2c2f662d1b9bb3/propcache-0.4.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4d3df5fa7e36b3225954fba85589da77a0fe6a53e3976de39caf04a0db4c36f1", size = 199398, upload-time = "2025-10-08T19:47:17.962Z" }, + { url = "https://files.pythonhosted.org/packages/3e/ec/d8a7cd406ee1ddb705db2139f8a10a8a427100347bd698e7014351c7af09/propcache-0.4.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:ee17f18d2498f2673e432faaa71698032b0127ebf23ae5974eeaf806c279df24", size = 196920, upload-time = "2025-10-08T19:47:19.355Z" }, + { url = "https://files.pythonhosted.org/packages/f6/6c/f38ab64af3764f431e359f8baf9e0a21013e24329e8b85d2da32e8ed07ca/propcache-0.4.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:580e97762b950f993ae618e167e7be9256b8353c2dcd8b99ec100eb50f5286aa", size = 203748, upload-time = "2025-10-08T19:47:21.338Z" }, + { url = "https://files.pythonhosted.org/packages/d6/e3/fa846bd70f6534d647886621388f0a265254d30e3ce47e5c8e6e27dbf153/propcache-0.4.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:501d20b891688eb8e7aa903021f0b72d5a55db40ffaab27edefd1027caaafa61", size = 205877, upload-time = "2025-10-08T19:47:23.059Z" }, + { url = "https://files.pythonhosted.org/packages/e2/39/8163fc6f3133fea7b5f2827e8eba2029a0277ab2c5beee6c1db7b10fc23d/propcache-0.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9a0bd56e5b100aef69bd8562b74b46254e7c8812918d3baa700c8a8009b0af66", size = 199437, upload-time = "2025-10-08T19:47:24.445Z" }, + { url = "https://files.pythonhosted.org/packages/93/89/caa9089970ca49c7c01662bd0eeedfe85494e863e8043565aeb6472ce8fe/propcache-0.4.1-cp313-cp313-win32.whl", hash = "sha256:bcc9aaa5d80322bc2fb24bb7accb4a30f81e90ab8d6ba187aec0744bc302ad81", size = 37586, upload-time = "2025-10-08T19:47:25.736Z" }, + { url = "https://files.pythonhosted.org/packages/f5/ab/f76ec3c3627c883215b5c8080debb4394ef5a7a29be811f786415fc1e6fd/propcache-0.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:381914df18634f5494334d201e98245c0596067504b9372d8cf93f4bb23e025e", size = 40790, upload-time = "2025-10-08T19:47:26.847Z" }, + { url = "https://files.pythonhosted.org/packages/59/1b/e71ae98235f8e2ba5004d8cb19765a74877abf189bc53fc0c80d799e56c3/propcache-0.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:8873eb4460fd55333ea49b7d189749ecf6e55bf85080f11b1c4530ed3034cba1", size = 37158, upload-time = "2025-10-08T19:47:27.961Z" }, + { url = "https://files.pythonhosted.org/packages/83/ce/a31bbdfc24ee0dcbba458c8175ed26089cf109a55bbe7b7640ed2470cfe9/propcache-0.4.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:92d1935ee1f8d7442da9c0c4fa7ac20d07e94064184811b685f5c4fada64553b", size = 81451, upload-time = "2025-10-08T19:47:29.445Z" }, + { url = "https://files.pythonhosted.org/packages/25/9c/442a45a470a68456e710d96cacd3573ef26a1d0a60067e6a7d5e655621ed/propcache-0.4.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:473c61b39e1460d386479b9b2f337da492042447c9b685f28be4f74d3529e566", size = 46374, upload-time = "2025-10-08T19:47:30.579Z" }, + { url = "https://files.pythonhosted.org/packages/f4/bf/b1d5e21dbc3b2e889ea4327044fb16312a736d97640fb8b6aa3f9c7b3b65/propcache-0.4.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:c0ef0aaafc66fbd87842a3fe3902fd889825646bc21149eafe47be6072725835", size = 48396, upload-time = "2025-10-08T19:47:31.79Z" }, + { url = "https://files.pythonhosted.org/packages/f4/04/5b4c54a103d480e978d3c8a76073502b18db0c4bc17ab91b3cb5092ad949/propcache-0.4.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f95393b4d66bfae908c3ca8d169d5f79cd65636ae15b5e7a4f6e67af675adb0e", size = 275950, upload-time = "2025-10-08T19:47:33.481Z" }, + { url = "https://files.pythonhosted.org/packages/b4/c1/86f846827fb969c4b78b0af79bba1d1ea2156492e1b83dea8b8a6ae27395/propcache-0.4.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c07fda85708bc48578467e85099645167a955ba093be0a2dcba962195676e859", size = 273856, upload-time = "2025-10-08T19:47:34.906Z" }, + { url = "https://files.pythonhosted.org/packages/36/1d/fc272a63c8d3bbad6878c336c7a7dea15e8f2d23a544bda43205dfa83ada/propcache-0.4.1-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:af223b406d6d000830c6f65f1e6431783fc3f713ba3e6cc8c024d5ee96170a4b", size = 280420, upload-time = "2025-10-08T19:47:36.338Z" }, + { url = "https://files.pythonhosted.org/packages/07/0c/01f2219d39f7e53d52e5173bcb09c976609ba30209912a0680adfb8c593a/propcache-0.4.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a78372c932c90ee474559c5ddfffd718238e8673c340dc21fe45c5b8b54559a0", size = 263254, upload-time = "2025-10-08T19:47:37.692Z" }, + { url = "https://files.pythonhosted.org/packages/2d/18/cd28081658ce597898f0c4d174d4d0f3c5b6d4dc27ffafeef835c95eb359/propcache-0.4.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:564d9f0d4d9509e1a870c920a89b2fec951b44bf5ba7d537a9e7c1ccec2c18af", size = 261205, upload-time = "2025-10-08T19:47:39.659Z" }, + { url = "https://files.pythonhosted.org/packages/7a/71/1f9e22eb8b8316701c2a19fa1f388c8a3185082607da8e406a803c9b954e/propcache-0.4.1-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:17612831fda0138059cc5546f4d12a2aacfb9e47068c06af35c400ba58ba7393", size = 247873, upload-time = "2025-10-08T19:47:41.084Z" }, + { url = "https://files.pythonhosted.org/packages/4a/65/3d4b61f36af2b4eddba9def857959f1016a51066b4f1ce348e0cf7881f58/propcache-0.4.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:41a89040cb10bd345b3c1a873b2bf36413d48da1def52f268a055f7398514874", size = 262739, upload-time = "2025-10-08T19:47:42.51Z" }, + { url = "https://files.pythonhosted.org/packages/2a/42/26746ab087faa77c1c68079b228810436ccd9a5ce9ac85e2b7307195fd06/propcache-0.4.1-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:e35b88984e7fa64aacecea39236cee32dd9bd8c55f57ba8a75cf2399553f9bd7", size = 263514, upload-time = "2025-10-08T19:47:43.927Z" }, + { url = "https://files.pythonhosted.org/packages/94/13/630690fe201f5502d2403dd3cfd451ed8858fe3c738ee88d095ad2ff407b/propcache-0.4.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6f8b465489f927b0df505cbe26ffbeed4d6d8a2bbc61ce90eb074ff129ef0ab1", size = 257781, upload-time = "2025-10-08T19:47:45.448Z" }, + { url = "https://files.pythonhosted.org/packages/92/f7/1d4ec5841505f423469efbfc381d64b7b467438cd5a4bbcbb063f3b73d27/propcache-0.4.1-cp313-cp313t-win32.whl", hash = "sha256:2ad890caa1d928c7c2965b48f3a3815c853180831d0e5503d35cf00c472f4717", size = 41396, upload-time = "2025-10-08T19:47:47.202Z" }, + { url = "https://files.pythonhosted.org/packages/48/f0/615c30622316496d2cbbc29f5985f7777d3ada70f23370608c1d3e081c1f/propcache-0.4.1-cp313-cp313t-win_amd64.whl", hash = "sha256:f7ee0e597f495cf415bcbd3da3caa3bd7e816b74d0d52b8145954c5e6fd3ff37", size = 44897, upload-time = "2025-10-08T19:47:48.336Z" }, + { url = "https://files.pythonhosted.org/packages/fd/ca/6002e46eccbe0e33dcd4069ef32f7f1c9e243736e07adca37ae8c4830ec3/propcache-0.4.1-cp313-cp313t-win_arm64.whl", hash = "sha256:929d7cbe1f01bb7baffb33dc14eb5691c95831450a26354cd210a8155170c93a", size = 39789, upload-time = "2025-10-08T19:47:49.876Z" }, + { url = "https://files.pythonhosted.org/packages/8e/5c/bca52d654a896f831b8256683457ceddd490ec18d9ec50e97dfd8fc726a8/propcache-0.4.1-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:3f7124c9d820ba5548d431afb4632301acf965db49e666aa21c305cbe8c6de12", size = 78152, upload-time = "2025-10-08T19:47:51.051Z" }, + { url = "https://files.pythonhosted.org/packages/65/9b/03b04e7d82a5f54fb16113d839f5ea1ede58a61e90edf515f6577c66fa8f/propcache-0.4.1-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:c0d4b719b7da33599dfe3b22d3db1ef789210a0597bc650b7cee9c77c2be8c5c", size = 44869, upload-time = "2025-10-08T19:47:52.594Z" }, + { url = "https://files.pythonhosted.org/packages/b2/fa/89a8ef0468d5833a23fff277b143d0573897cf75bd56670a6d28126c7d68/propcache-0.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9f302f4783709a78240ebc311b793f123328716a60911d667e0c036bc5dcbded", size = 46596, upload-time = "2025-10-08T19:47:54.073Z" }, + { url = "https://files.pythonhosted.org/packages/86/bd/47816020d337f4a746edc42fe8d53669965138f39ee117414c7d7a340cfe/propcache-0.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c80ee5802e3fb9ea37938e7eecc307fb984837091d5fd262bb37238b1ae97641", size = 206981, upload-time = "2025-10-08T19:47:55.715Z" }, + { url = "https://files.pythonhosted.org/packages/df/f6/c5fa1357cc9748510ee55f37173eb31bfde6d94e98ccd9e6f033f2fc06e1/propcache-0.4.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ed5a841e8bb29a55fb8159ed526b26adc5bdd7e8bd7bf793ce647cb08656cdf4", size = 211490, upload-time = "2025-10-08T19:47:57.499Z" }, + { url = "https://files.pythonhosted.org/packages/80/1e/e5889652a7c4a3846683401a48f0f2e5083ce0ec1a8a5221d8058fbd1adf/propcache-0.4.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:55c72fd6ea2da4c318e74ffdf93c4fe4e926051133657459131a95c846d16d44", size = 215371, upload-time = "2025-10-08T19:47:59.317Z" }, + { url = "https://files.pythonhosted.org/packages/b2/f2/889ad4b2408f72fe1a4f6a19491177b30ea7bf1a0fd5f17050ca08cfc882/propcache-0.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8326e144341460402713f91df60ade3c999d601e7eb5ff8f6f7862d54de0610d", size = 201424, upload-time = "2025-10-08T19:48:00.67Z" }, + { url = "https://files.pythonhosted.org/packages/27/73/033d63069b57b0812c8bd19f311faebeceb6ba31b8f32b73432d12a0b826/propcache-0.4.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:060b16ae65bc098da7f6d25bf359f1f31f688384858204fe5d652979e0015e5b", size = 197566, upload-time = "2025-10-08T19:48:02.604Z" }, + { url = "https://files.pythonhosted.org/packages/dc/89/ce24f3dc182630b4e07aa6d15f0ff4b14ed4b9955fae95a0b54c58d66c05/propcache-0.4.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:89eb3fa9524f7bec9de6e83cf3faed9d79bffa560672c118a96a171a6f55831e", size = 193130, upload-time = "2025-10-08T19:48:04.499Z" }, + { url = "https://files.pythonhosted.org/packages/a9/24/ef0d5fd1a811fb5c609278d0209c9f10c35f20581fcc16f818da959fc5b4/propcache-0.4.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:dee69d7015dc235f526fe80a9c90d65eb0039103fe565776250881731f06349f", size = 202625, upload-time = "2025-10-08T19:48:06.213Z" }, + { url = "https://files.pythonhosted.org/packages/f5/02/98ec20ff5546f68d673df2f7a69e8c0d076b5abd05ca882dc7ee3a83653d/propcache-0.4.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:5558992a00dfd54ccbc64a32726a3357ec93825a418a401f5cc67df0ac5d9e49", size = 204209, upload-time = "2025-10-08T19:48:08.432Z" }, + { url = "https://files.pythonhosted.org/packages/a0/87/492694f76759b15f0467a2a93ab68d32859672b646aa8a04ce4864e7932d/propcache-0.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c9b822a577f560fbd9554812526831712c1436d2c046cedee4c3796d3543b144", size = 197797, upload-time = "2025-10-08T19:48:09.968Z" }, + { url = "https://files.pythonhosted.org/packages/ee/36/66367de3575db1d2d3f3d177432bd14ee577a39d3f5d1b3d5df8afe3b6e2/propcache-0.4.1-cp314-cp314-win32.whl", hash = "sha256:ab4c29b49d560fe48b696cdcb127dd36e0bc2472548f3bf56cc5cb3da2b2984f", size = 38140, upload-time = "2025-10-08T19:48:11.232Z" }, + { url = "https://files.pythonhosted.org/packages/0c/2a/a758b47de253636e1b8aef181c0b4f4f204bf0dd964914fb2af90a95b49b/propcache-0.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:5a103c3eb905fcea0ab98be99c3a9a5ab2de60228aa5aceedc614c0281cf6153", size = 41257, upload-time = "2025-10-08T19:48:12.707Z" }, + { url = "https://files.pythonhosted.org/packages/34/5e/63bd5896c3fec12edcbd6f12508d4890d23c265df28c74b175e1ef9f4f3b/propcache-0.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:74c1fb26515153e482e00177a1ad654721bf9207da8a494a0c05e797ad27b992", size = 38097, upload-time = "2025-10-08T19:48:13.923Z" }, + { url = "https://files.pythonhosted.org/packages/99/85/9ff785d787ccf9bbb3f3106f79884a130951436f58392000231b4c737c80/propcache-0.4.1-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:824e908bce90fb2743bd6b59db36eb4f45cd350a39637c9f73b1c1ea66f5b75f", size = 81455, upload-time = "2025-10-08T19:48:15.16Z" }, + { url = "https://files.pythonhosted.org/packages/90/85/2431c10c8e7ddb1445c1f7c4b54d886e8ad20e3c6307e7218f05922cad67/propcache-0.4.1-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:c2b5e7db5328427c57c8e8831abda175421b709672f6cfc3d630c3b7e2146393", size = 46372, upload-time = "2025-10-08T19:48:16.424Z" }, + { url = "https://files.pythonhosted.org/packages/01/20/b0972d902472da9bcb683fa595099911f4d2e86e5683bcc45de60dd05dc3/propcache-0.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6f6ff873ed40292cd4969ef5310179afd5db59fdf055897e282485043fc80ad0", size = 48411, upload-time = "2025-10-08T19:48:17.577Z" }, + { url = "https://files.pythonhosted.org/packages/e2/e3/7dc89f4f21e8f99bad3d5ddb3a3389afcf9da4ac69e3deb2dcdc96e74169/propcache-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:49a2dc67c154db2c1463013594c458881a069fcf98940e61a0569016a583020a", size = 275712, upload-time = "2025-10-08T19:48:18.901Z" }, + { url = "https://files.pythonhosted.org/packages/20/67/89800c8352489b21a8047c773067644e3897f02ecbbd610f4d46b7f08612/propcache-0.4.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:005f08e6a0529984491e37d8dbc3dd86f84bd78a8ceb5fa9a021f4c48d4984be", size = 273557, upload-time = "2025-10-08T19:48:20.762Z" }, + { url = "https://files.pythonhosted.org/packages/e2/a1/b52b055c766a54ce6d9c16d9aca0cad8059acd9637cdf8aa0222f4a026ef/propcache-0.4.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5c3310452e0d31390da9035c348633b43d7e7feb2e37be252be6da45abd1abcc", size = 280015, upload-time = "2025-10-08T19:48:22.592Z" }, + { url = "https://files.pythonhosted.org/packages/48/c8/33cee30bd890672c63743049f3c9e4be087e6780906bfc3ec58528be59c1/propcache-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4c3c70630930447f9ef1caac7728c8ad1c56bc5015338b20fed0d08ea2480b3a", size = 262880, upload-time = "2025-10-08T19:48:23.947Z" }, + { url = "https://files.pythonhosted.org/packages/0c/b1/8f08a143b204b418285c88b83d00edbd61afbc2c6415ffafc8905da7038b/propcache-0.4.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8e57061305815dfc910a3634dcf584f08168a8836e6999983569f51a8544cd89", size = 260938, upload-time = "2025-10-08T19:48:25.656Z" }, + { url = "https://files.pythonhosted.org/packages/cf/12/96e4664c82ca2f31e1c8dff86afb867348979eb78d3cb8546a680287a1e9/propcache-0.4.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:521a463429ef54143092c11a77e04056dd00636f72e8c45b70aaa3140d639726", size = 247641, upload-time = "2025-10-08T19:48:27.207Z" }, + { url = "https://files.pythonhosted.org/packages/18/ed/e7a9cfca28133386ba52278136d42209d3125db08d0a6395f0cba0c0285c/propcache-0.4.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:120c964da3fdc75e3731aa392527136d4ad35868cc556fd09bb6d09172d9a367", size = 262510, upload-time = "2025-10-08T19:48:28.65Z" }, + { url = "https://files.pythonhosted.org/packages/f5/76/16d8bf65e8845dd62b4e2b57444ab81f07f40caa5652b8969b87ddcf2ef6/propcache-0.4.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:d8f353eb14ee3441ee844ade4277d560cdd68288838673273b978e3d6d2c8f36", size = 263161, upload-time = "2025-10-08T19:48:30.133Z" }, + { url = "https://files.pythonhosted.org/packages/e7/70/c99e9edb5d91d5ad8a49fa3c1e8285ba64f1476782fed10ab251ff413ba1/propcache-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ab2943be7c652f09638800905ee1bab2c544e537edb57d527997a24c13dc1455", size = 257393, upload-time = "2025-10-08T19:48:31.567Z" }, + { url = "https://files.pythonhosted.org/packages/08/02/87b25304249a35c0915d236575bc3574a323f60b47939a2262b77632a3ee/propcache-0.4.1-cp314-cp314t-win32.whl", hash = "sha256:05674a162469f31358c30bcaa8883cb7829fa3110bf9c0991fe27d7896c42d85", size = 42546, upload-time = "2025-10-08T19:48:32.872Z" }, + { url = "https://files.pythonhosted.org/packages/cb/ef/3c6ecf8b317aa982f309835e8f96987466123c6e596646d4e6a1dfcd080f/propcache-0.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:990f6b3e2a27d683cb7602ed6c86f15ee6b43b1194736f9baaeb93d0016633b1", size = 46259, upload-time = "2025-10-08T19:48:34.226Z" }, + { url = "https://files.pythonhosted.org/packages/c4/2d/346e946d4951f37eca1e4f55be0f0174c52cd70720f84029b02f296f4a38/propcache-0.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:ecef2343af4cc68e05131e45024ba34f6095821988a9d0a02aa7c73fcc448aa9", size = 40428, upload-time = "2025-10-08T19:48:35.441Z" }, + { url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" }, +] + +[[package]] +name = "protobuf" +version = "7.35.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/da/01/9ef0afd7999eb9badb3a768b4aedd78c86d4c65cfaf1958ab276199e76b4/protobuf-7.35.1.tar.gz", hash = "sha256:ce115a26fe0c39a2c29973d914d327e516a6455464489fe3cd1e51a1b354f81a", size = 458717, upload-time = "2026-06-11T21:55:40.257Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/03/8aeeb7458d22546bf64b5250ca1daeb5ff757d900e8e4a7476c6f0db843e/protobuf-7.35.1-cp310-abi3-macosx_10_9_universal2.whl", hash = "sha256:24f857477359a85c0c235261b8ba905fd51b2562f4a64ca1df5473f29850cbf6", size = 433226, upload-time = "2026-06-11T21:55:31.719Z" }, + { url = "https://files.pythonhosted.org/packages/37/4b/dfb89eb0e652a1ff073c39a59fb5e3a83cfe9b57a2c83fa6d78270101767/protobuf-7.35.1-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:11d6b0ec246892d85215b0a13ca6e0233cf5284b68f0ac02646427f4ff88a799", size = 328847, upload-time = "2026-06-11T21:55:34.035Z" }, + { url = "https://files.pythonhosted.org/packages/0f/58/dc12f2cd484951524af6e3382c785869b9b3fb5e52ee95ae23add53ee8f9/protobuf-7.35.1-cp310-abi3-manylinux2014_s390x.whl", hash = "sha256:b73f9489a4b8b1c9cb1f8ed951c736392592edb24b9d6819f36d2e10b171d5b4", size = 344030, upload-time = "2026-06-11T21:55:34.941Z" }, + { url = "https://files.pythonhosted.org/packages/e4/be/5b3cfe508bfab6761414ff944e3366eb13be4fd71efcd69450f89ba39f43/protobuf-7.35.1-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:74758715c53d7158fb76caf4f0cfdacc5329a4b1bb994f865d6cf302d413a1c4", size = 327130, upload-time = "2026-06-11T21:55:35.921Z" }, + { url = "https://files.pythonhosted.org/packages/d8/bc/6d6c7ba8709c85f8f2c390b2b118d6fb08a783676a572271851bf45a7d22/protobuf-7.35.1-cp310-abi3-win32.whl", hash = "sha256:353652e4efd0bca5b5fc2656abf8307ef351f0cf938c9eba09f0e09c20a25c30", size = 428945, upload-time = "2026-06-11T21:55:37.034Z" }, + { url = "https://files.pythonhosted.org/packages/0a/19/8d0cb6f20a1ef7b18f1c8986ad5783f22f84cce39c6ce9a6e645ea55192e/protobuf-7.35.1-cp310-abi3-win_amd64.whl", hash = "sha256:230a75ddfc2de4806e56696ce9640c1cdfdb6543b7cfce98d42a4c0a0e7bdb87", size = 439996, upload-time = "2026-06-11T21:55:38.123Z" }, + { url = "https://files.pythonhosted.org/packages/19/c7/5f7c636ec43e0c545e28d1f1db71990108306f7bdcb89f069ba97e428e7f/protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9", size = 171659, upload-time = "2026-06-11T21:55:39.155Z" }, +] + +[[package]] +name = "pycparser" +version = "3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" }, +] + +[[package]] +name = "pydantic" +version = "2.12.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/69/44/36f1a6e523abc58ae5f928898e4aca2e0ea509b5aa6f6f392a5d882be928/pydantic-2.12.5.tar.gz", hash = "sha256:4d351024c75c0f085a9febbb665ce8c0c6ec5d30e903bdb6394b7ede26aebb49", size = 821591, upload-time = "2025-11-26T15:11:46.471Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/87/b70ad306ebb6f9b585f114d0ac2137d792b48be34d732d60e597c2f8465a/pydantic-2.12.5-py3-none-any.whl", hash = "sha256:e561593fccf61e8a20fc46dfc2dfe075b8be7d0188df33f221ad1f0139180f9d", size = 463580, upload-time = "2025-11-26T15:11:44.605Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.41.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/71/70/23b021c950c2addd24ec408e9ab05d59b035b39d97cdc1130e1bce647bb6/pydantic_core-2.41.5.tar.gz", hash = "sha256:08daa51ea16ad373ffd5e7606252cc32f07bc72b28284b6bc9c6df804816476e", size = 460952, upload-time = "2025-11-04T13:43:49.098Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e8/72/74a989dd9f2084b3d9530b0915fdda64ac48831c30dbf7c72a41a5232db8/pydantic_core-2.41.5-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:a3a52f6156e73e7ccb0f8cced536adccb7042be67cb45f9562e12b319c119da6", size = 2105873, upload-time = "2025-11-04T13:39:31.373Z" }, + { url = "https://files.pythonhosted.org/packages/12/44/37e403fd9455708b3b942949e1d7febc02167662bf1a7da5b78ee1ea2842/pydantic_core-2.41.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7f3bf998340c6d4b0c9a2f02d6a400e51f123b59565d74dc60d252ce888c260b", size = 1899826, upload-time = "2025-11-04T13:39:32.897Z" }, + { url = "https://files.pythonhosted.org/packages/33/7f/1d5cab3ccf44c1935a359d51a8a2a9e1a654b744b5e7f80d41b88d501eec/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:378bec5c66998815d224c9ca994f1e14c0c21cb95d2f52b6021cc0b2a58f2a5a", size = 1917869, upload-time = "2025-11-04T13:39:34.469Z" }, + { url = "https://files.pythonhosted.org/packages/6e/6a/30d94a9674a7fe4f4744052ed6c5e083424510be1e93da5bc47569d11810/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e7b576130c69225432866fe2f4a469a85a54ade141d96fd396dffcf607b558f8", size = 2063890, upload-time = "2025-11-04T13:39:36.053Z" }, + { url = "https://files.pythonhosted.org/packages/50/be/76e5d46203fcb2750e542f32e6c371ffa9b8ad17364cf94bb0818dbfb50c/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6cb58b9c66f7e4179a2d5e0f849c48eff5c1fca560994d6eb6543abf955a149e", size = 2229740, upload-time = "2025-11-04T13:39:37.753Z" }, + { url = "https://files.pythonhosted.org/packages/d3/ee/fed784df0144793489f87db310a6bbf8118d7b630ed07aa180d6067e653a/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:88942d3a3dff3afc8288c21e565e476fc278902ae4d6d134f1eeda118cc830b1", size = 2350021, upload-time = "2025-11-04T13:39:40.94Z" }, + { url = "https://files.pythonhosted.org/packages/c8/be/8fed28dd0a180dca19e72c233cbf58efa36df055e5b9d90d64fd1740b828/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f31d95a179f8d64d90f6831d71fa93290893a33148d890ba15de25642c5d075b", size = 2066378, upload-time = "2025-11-04T13:39:42.523Z" }, + { url = "https://files.pythonhosted.org/packages/b0/3b/698cf8ae1d536a010e05121b4958b1257f0b5522085e335360e53a6b1c8b/pydantic_core-2.41.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c1df3d34aced70add6f867a8cf413e299177e0c22660cc767218373d0779487b", size = 2175761, upload-time = "2025-11-04T13:39:44.553Z" }, + { url = "https://files.pythonhosted.org/packages/b8/ba/15d537423939553116dea94ce02f9c31be0fa9d0b806d427e0308ec17145/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:4009935984bd36bd2c774e13f9a09563ce8de4abaa7226f5108262fa3e637284", size = 2146303, upload-time = "2025-11-04T13:39:46.238Z" }, + { url = "https://files.pythonhosted.org/packages/58/7f/0de669bf37d206723795f9c90c82966726a2ab06c336deba4735b55af431/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_armv7l.whl", hash = "sha256:34a64bc3441dc1213096a20fe27e8e128bd3ff89921706e83c0b1ac971276594", size = 2340355, upload-time = "2025-11-04T13:39:48.002Z" }, + { url = "https://files.pythonhosted.org/packages/e5/de/e7482c435b83d7e3c3ee5ee4451f6e8973cff0eb6007d2872ce6383f6398/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:c9e19dd6e28fdcaa5a1de679aec4141f691023916427ef9bae8584f9c2fb3b0e", size = 2319875, upload-time = "2025-11-04T13:39:49.705Z" }, + { url = "https://files.pythonhosted.org/packages/fe/e6/8c9e81bb6dd7560e33b9053351c29f30c8194b72f2d6932888581f503482/pydantic_core-2.41.5-cp311-cp311-win32.whl", hash = "sha256:2c010c6ded393148374c0f6f0bf89d206bf3217f201faa0635dcd56bd1520f6b", size = 1987549, upload-time = "2025-11-04T13:39:51.842Z" }, + { url = "https://files.pythonhosted.org/packages/11/66/f14d1d978ea94d1bc21fc98fcf570f9542fe55bfcc40269d4e1a21c19bf7/pydantic_core-2.41.5-cp311-cp311-win_amd64.whl", hash = "sha256:76ee27c6e9c7f16f47db7a94157112a2f3a00e958bc626e2f4ee8bec5c328fbe", size = 2011305, upload-time = "2025-11-04T13:39:53.485Z" }, + { url = "https://files.pythonhosted.org/packages/56/d8/0e271434e8efd03186c5386671328154ee349ff0354d83c74f5caaf096ed/pydantic_core-2.41.5-cp311-cp311-win_arm64.whl", hash = "sha256:4bc36bbc0b7584de96561184ad7f012478987882ebf9f9c389b23f432ea3d90f", size = 1972902, upload-time = "2025-11-04T13:39:56.488Z" }, + { url = "https://files.pythonhosted.org/packages/5f/5d/5f6c63eebb5afee93bcaae4ce9a898f3373ca23df3ccaef086d0233a35a7/pydantic_core-2.41.5-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:f41a7489d32336dbf2199c8c0a215390a751c5b014c2c1c5366e817202e9cdf7", size = 2110990, upload-time = "2025-11-04T13:39:58.079Z" }, + { url = "https://files.pythonhosted.org/packages/aa/32/9c2e8ccb57c01111e0fd091f236c7b371c1bccea0fa85247ac55b1e2b6b6/pydantic_core-2.41.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:070259a8818988b9a84a449a2a7337c7f430a22acc0859c6b110aa7212a6d9c0", size = 1896003, upload-time = "2025-11-04T13:39:59.956Z" }, + { url = "https://files.pythonhosted.org/packages/68/b8/a01b53cb0e59139fbc9e4fda3e9724ede8de279097179be4ff31f1abb65a/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e96cea19e34778f8d59fe40775a7a574d95816eb150850a85a7a4c8f4b94ac69", size = 1919200, upload-time = "2025-11-04T13:40:02.241Z" }, + { url = "https://files.pythonhosted.org/packages/38/de/8c36b5198a29bdaade07b5985e80a233a5ac27137846f3bc2d3b40a47360/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ed2e99c456e3fadd05c991f8f437ef902e00eedf34320ba2b0842bd1c3ca3a75", size = 2052578, upload-time = "2025-11-04T13:40:04.401Z" }, + { url = "https://files.pythonhosted.org/packages/00/b5/0e8e4b5b081eac6cb3dbb7e60a65907549a1ce035a724368c330112adfdd/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:65840751b72fbfd82c3c640cff9284545342a4f1eb1586ad0636955b261b0b05", size = 2208504, upload-time = "2025-11-04T13:40:06.072Z" }, + { url = "https://files.pythonhosted.org/packages/77/56/87a61aad59c7c5b9dc8caad5a41a5545cba3810c3e828708b3d7404f6cef/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e536c98a7626a98feb2d3eaf75944ef6f3dbee447e1f841eae16f2f0a72d8ddc", size = 2335816, upload-time = "2025-11-04T13:40:07.835Z" }, + { url = "https://files.pythonhosted.org/packages/0d/76/941cc9f73529988688a665a5c0ecff1112b3d95ab48f81db5f7606f522d3/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eceb81a8d74f9267ef4081e246ffd6d129da5d87e37a77c9bde550cb04870c1c", size = 2075366, upload-time = "2025-11-04T13:40:09.804Z" }, + { url = "https://files.pythonhosted.org/packages/d3/43/ebef01f69baa07a482844faaa0a591bad1ef129253ffd0cdaa9d8a7f72d3/pydantic_core-2.41.5-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d38548150c39b74aeeb0ce8ee1d8e82696f4a4e16ddc6de7b1d8823f7de4b9b5", size = 2171698, upload-time = "2025-11-04T13:40:12.004Z" }, + { url = "https://files.pythonhosted.org/packages/b1/87/41f3202e4193e3bacfc2c065fab7706ebe81af46a83d3e27605029c1f5a6/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:c23e27686783f60290e36827f9c626e63154b82b116d7fe9adba1fda36da706c", size = 2132603, upload-time = "2025-11-04T13:40:13.868Z" }, + { url = "https://files.pythonhosted.org/packages/49/7d/4c00df99cb12070b6bccdef4a195255e6020a550d572768d92cc54dba91a/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:482c982f814460eabe1d3bb0adfdc583387bd4691ef00b90575ca0d2b6fe2294", size = 2329591, upload-time = "2025-11-04T13:40:15.672Z" }, + { url = "https://files.pythonhosted.org/packages/cc/6a/ebf4b1d65d458f3cda6a7335d141305dfa19bdc61140a884d165a8a1bbc7/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:bfea2a5f0b4d8d43adf9d7b8bf019fb46fdd10a2e5cde477fbcb9d1fa08c68e1", size = 2319068, upload-time = "2025-11-04T13:40:17.532Z" }, + { url = "https://files.pythonhosted.org/packages/49/3b/774f2b5cd4192d5ab75870ce4381fd89cf218af999515baf07e7206753f0/pydantic_core-2.41.5-cp312-cp312-win32.whl", hash = "sha256:b74557b16e390ec12dca509bce9264c3bbd128f8a2c376eaa68003d7f327276d", size = 1985908, upload-time = "2025-11-04T13:40:19.309Z" }, + { url = "https://files.pythonhosted.org/packages/86/45/00173a033c801cacf67c190fef088789394feaf88a98a7035b0e40d53dc9/pydantic_core-2.41.5-cp312-cp312-win_amd64.whl", hash = "sha256:1962293292865bca8e54702b08a4f26da73adc83dd1fcf26fbc875b35d81c815", size = 2020145, upload-time = "2025-11-04T13:40:21.548Z" }, + { url = "https://files.pythonhosted.org/packages/f9/22/91fbc821fa6d261b376a3f73809f907cec5ca6025642c463d3488aad22fb/pydantic_core-2.41.5-cp312-cp312-win_arm64.whl", hash = "sha256:1746d4a3d9a794cacae06a5eaaccb4b8643a131d45fbc9af23e353dc0a5ba5c3", size = 1976179, upload-time = "2025-11-04T13:40:23.393Z" }, + { url = "https://files.pythonhosted.org/packages/87/06/8806241ff1f70d9939f9af039c6c35f2360cf16e93c2ca76f184e76b1564/pydantic_core-2.41.5-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:941103c9be18ac8daf7b7adca8228f8ed6bb7a1849020f643b3a14d15b1924d9", size = 2120403, upload-time = "2025-11-04T13:40:25.248Z" }, + { url = "https://files.pythonhosted.org/packages/94/02/abfa0e0bda67faa65fef1c84971c7e45928e108fe24333c81f3bfe35d5f5/pydantic_core-2.41.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:112e305c3314f40c93998e567879e887a3160bb8689ef3d2c04b6cc62c33ac34", size = 1896206, upload-time = "2025-11-04T13:40:27.099Z" }, + { url = "https://files.pythonhosted.org/packages/15/df/a4c740c0943e93e6500f9eb23f4ca7ec9bf71b19e608ae5b579678c8d02f/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0cbaad15cb0c90aa221d43c00e77bb33c93e8d36e0bf74760cd00e732d10a6a0", size = 1919307, upload-time = "2025-11-04T13:40:29.806Z" }, + { url = "https://files.pythonhosted.org/packages/9a/e3/6324802931ae1d123528988e0e86587c2072ac2e5394b4bc2bc34b61ff6e/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:03ca43e12fab6023fc79d28ca6b39b05f794ad08ec2feccc59a339b02f2b3d33", size = 2063258, upload-time = "2025-11-04T13:40:33.544Z" }, + { url = "https://files.pythonhosted.org/packages/c9/d4/2230d7151d4957dd79c3044ea26346c148c98fbf0ee6ebd41056f2d62ab5/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:dc799088c08fa04e43144b164feb0c13f9a0bc40503f8df3e9fde58a3c0c101e", size = 2214917, upload-time = "2025-11-04T13:40:35.479Z" }, + { url = "https://files.pythonhosted.org/packages/e6/9f/eaac5df17a3672fef0081b6c1bb0b82b33ee89aa5cec0d7b05f52fd4a1fa/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:97aeba56665b4c3235a0e52b2c2f5ae9cd071b8a8310ad27bddb3f7fb30e9aa2", size = 2332186, upload-time = "2025-11-04T13:40:37.436Z" }, + { url = "https://files.pythonhosted.org/packages/cf/4e/35a80cae583a37cf15604b44240e45c05e04e86f9cfd766623149297e971/pydantic_core-2.41.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:406bf18d345822d6c21366031003612b9c77b3e29ffdb0f612367352aab7d586", size = 2073164, upload-time = "2025-11-04T13:40:40.289Z" }, + { url = "https://files.pythonhosted.org/packages/bf/e3/f6e262673c6140dd3305d144d032f7bd5f7497d3871c1428521f19f9efa2/pydantic_core-2.41.5-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:b93590ae81f7010dbe380cdeab6f515902ebcbefe0b9327cc4804d74e93ae69d", size = 2179146, upload-time = "2025-11-04T13:40:42.809Z" }, + { url = "https://files.pythonhosted.org/packages/75/c7/20bd7fc05f0c6ea2056a4565c6f36f8968c0924f19b7d97bbfea55780e73/pydantic_core-2.41.5-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:01a3d0ab748ee531f4ea6c3e48ad9dac84ddba4b0d82291f87248f2f9de8d740", size = 2137788, upload-time = "2025-11-04T13:40:44.752Z" }, + { url = "https://files.pythonhosted.org/packages/3a/8d/34318ef985c45196e004bc46c6eab2eda437e744c124ef0dbe1ff2c9d06b/pydantic_core-2.41.5-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:6561e94ba9dacc9c61bce40e2d6bdc3bfaa0259d3ff36ace3b1e6901936d2e3e", size = 2340133, upload-time = "2025-11-04T13:40:46.66Z" }, + { url = "https://files.pythonhosted.org/packages/9c/59/013626bf8c78a5a5d9350d12e7697d3d4de951a75565496abd40ccd46bee/pydantic_core-2.41.5-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:915c3d10f81bec3a74fbd4faebe8391013ba61e5a1a8d48c4455b923bdda7858", size = 2324852, upload-time = "2025-11-04T13:40:48.575Z" }, + { url = "https://files.pythonhosted.org/packages/1a/d9/c248c103856f807ef70c18a4f986693a46a8ffe1602e5d361485da502d20/pydantic_core-2.41.5-cp313-cp313-win32.whl", hash = "sha256:650ae77860b45cfa6e2cdafc42618ceafab3a2d9a3811fcfbd3bbf8ac3c40d36", size = 1994679, upload-time = "2025-11-04T13:40:50.619Z" }, + { url = "https://files.pythonhosted.org/packages/9e/8b/341991b158ddab181cff136acd2552c9f35bd30380422a639c0671e99a91/pydantic_core-2.41.5-cp313-cp313-win_amd64.whl", hash = "sha256:79ec52ec461e99e13791ec6508c722742ad745571f234ea6255bed38c6480f11", size = 2019766, upload-time = "2025-11-04T13:40:52.631Z" }, + { url = "https://files.pythonhosted.org/packages/73/7d/f2f9db34af103bea3e09735bb40b021788a5e834c81eedb541991badf8f5/pydantic_core-2.41.5-cp313-cp313-win_arm64.whl", hash = "sha256:3f84d5c1b4ab906093bdc1ff10484838aca54ef08de4afa9de0f5f14d69639cd", size = 1981005, upload-time = "2025-11-04T13:40:54.734Z" }, + { url = "https://files.pythonhosted.org/packages/ea/28/46b7c5c9635ae96ea0fbb779e271a38129df2550f763937659ee6c5dbc65/pydantic_core-2.41.5-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:3f37a19d7ebcdd20b96485056ba9e8b304e27d9904d233d7b1015db320e51f0a", size = 2119622, upload-time = "2025-11-04T13:40:56.68Z" }, + { url = "https://files.pythonhosted.org/packages/74/1a/145646e5687e8d9a1e8d09acb278c8535ebe9e972e1f162ed338a622f193/pydantic_core-2.41.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1d1d9764366c73f996edd17abb6d9d7649a7eb690006ab6adbda117717099b14", size = 1891725, upload-time = "2025-11-04T13:40:58.807Z" }, + { url = "https://files.pythonhosted.org/packages/23/04/e89c29e267b8060b40dca97bfc64a19b2a3cf99018167ea1677d96368273/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:25e1c2af0fce638d5f1988b686f3b3ea8cd7de5f244ca147c777769e798a9cd1", size = 1915040, upload-time = "2025-11-04T13:41:00.853Z" }, + { url = "https://files.pythonhosted.org/packages/84/a3/15a82ac7bd97992a82257f777b3583d3e84bdb06ba6858f745daa2ec8a85/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:506d766a8727beef16b7adaeb8ee6217c64fc813646b424d0804d67c16eddb66", size = 2063691, upload-time = "2025-11-04T13:41:03.504Z" }, + { url = "https://files.pythonhosted.org/packages/74/9b/0046701313c6ef08c0c1cf0e028c67c770a4e1275ca73131563c5f2a310a/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4819fa52133c9aa3c387b3328f25c1facc356491e6135b459f1de698ff64d869", size = 2213897, upload-time = "2025-11-04T13:41:05.804Z" }, + { url = "https://files.pythonhosted.org/packages/8a/cd/6bac76ecd1b27e75a95ca3a9a559c643b3afcd2dd62086d4b7a32a18b169/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2b761d210c9ea91feda40d25b4efe82a1707da2ef62901466a42492c028553a2", size = 2333302, upload-time = "2025-11-04T13:41:07.809Z" }, + { url = "https://files.pythonhosted.org/packages/4c/d2/ef2074dc020dd6e109611a8be4449b98cd25e1b9b8a303c2f0fca2f2bcf7/pydantic_core-2.41.5-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:22f0fb8c1c583a3b6f24df2470833b40207e907b90c928cc8d3594b76f874375", size = 2064877, upload-time = "2025-11-04T13:41:09.827Z" }, + { url = "https://files.pythonhosted.org/packages/18/66/e9db17a9a763d72f03de903883c057b2592c09509ccfe468187f2a2eef29/pydantic_core-2.41.5-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2782c870e99878c634505236d81e5443092fba820f0373997ff75f90f68cd553", size = 2180680, upload-time = "2025-11-04T13:41:12.379Z" }, + { url = "https://files.pythonhosted.org/packages/d3/9e/3ce66cebb929f3ced22be85d4c2399b8e85b622db77dad36b73c5387f8f8/pydantic_core-2.41.5-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:0177272f88ab8312479336e1d777f6b124537d47f2123f89cb37e0accea97f90", size = 2138960, upload-time = "2025-11-04T13:41:14.627Z" }, + { url = "https://files.pythonhosted.org/packages/a6/62/205a998f4327d2079326b01abee48e502ea739d174f0a89295c481a2272e/pydantic_core-2.41.5-cp314-cp314-musllinux_1_1_armv7l.whl", hash = "sha256:63510af5e38f8955b8ee5687740d6ebf7c2a0886d15a6d65c32814613681bc07", size = 2339102, upload-time = "2025-11-04T13:41:16.868Z" }, + { url = "https://files.pythonhosted.org/packages/3c/0d/f05e79471e889d74d3d88f5bd20d0ed189ad94c2423d81ff8d0000aab4ff/pydantic_core-2.41.5-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:e56ba91f47764cc14f1daacd723e3e82d1a89d783f0f5afe9c364b8bb491ccdb", size = 2326039, upload-time = "2025-11-04T13:41:18.934Z" }, + { url = "https://files.pythonhosted.org/packages/ec/e1/e08a6208bb100da7e0c4b288eed624a703f4d129bde2da475721a80cab32/pydantic_core-2.41.5-cp314-cp314-win32.whl", hash = "sha256:aec5cf2fd867b4ff45b9959f8b20ea3993fc93e63c7363fe6851424c8a7e7c23", size = 1995126, upload-time = "2025-11-04T13:41:21.418Z" }, + { url = "https://files.pythonhosted.org/packages/48/5d/56ba7b24e9557f99c9237e29f5c09913c81eeb2f3217e40e922353668092/pydantic_core-2.41.5-cp314-cp314-win_amd64.whl", hash = "sha256:8e7c86f27c585ef37c35e56a96363ab8de4e549a95512445b85c96d3e2f7c1bf", size = 2015489, upload-time = "2025-11-04T13:41:24.076Z" }, + { url = "https://files.pythonhosted.org/packages/4e/bb/f7a190991ec9e3e0ba22e4993d8755bbc4a32925c0b5b42775c03e8148f9/pydantic_core-2.41.5-cp314-cp314-win_arm64.whl", hash = "sha256:e672ba74fbc2dc8eea59fb6d4aed6845e6905fc2a8afe93175d94a83ba2a01a0", size = 1977288, upload-time = "2025-11-04T13:41:26.33Z" }, + { url = "https://files.pythonhosted.org/packages/92/ed/77542d0c51538e32e15afe7899d79efce4b81eee631d99850edc2f5e9349/pydantic_core-2.41.5-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8566def80554c3faa0e65ac30ab0932b9e3a5cd7f8323764303d468e5c37595a", size = 2120255, upload-time = "2025-11-04T13:41:28.569Z" }, + { url = "https://files.pythonhosted.org/packages/bb/3d/6913dde84d5be21e284439676168b28d8bbba5600d838b9dca99de0fad71/pydantic_core-2.41.5-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:b80aa5095cd3109962a298ce14110ae16b8c1aece8b72f9dafe81cf597ad80b3", size = 1863760, upload-time = "2025-11-04T13:41:31.055Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f0/e5e6b99d4191da102f2b0eb9687aaa7f5bea5d9964071a84effc3e40f997/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3006c3dd9ba34b0c094c544c6006cc79e87d8612999f1a5d43b769b89181f23c", size = 1878092, upload-time = "2025-11-04T13:41:33.21Z" }, + { url = "https://files.pythonhosted.org/packages/71/48/36fb760642d568925953bcc8116455513d6e34c4beaa37544118c36aba6d/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:72f6c8b11857a856bcfa48c86f5368439f74453563f951e473514579d44aa612", size = 2053385, upload-time = "2025-11-04T13:41:35.508Z" }, + { url = "https://files.pythonhosted.org/packages/20/25/92dc684dd8eb75a234bc1c764b4210cf2646479d54b47bf46061657292a8/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5cb1b2f9742240e4bb26b652a5aeb840aa4b417c7748b6f8387927bc6e45e40d", size = 2218832, upload-time = "2025-11-04T13:41:37.732Z" }, + { url = "https://files.pythonhosted.org/packages/e2/09/f53e0b05023d3e30357d82eb35835d0f6340ca344720a4599cd663dca599/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:bd3d54f38609ff308209bd43acea66061494157703364ae40c951f83ba99a1a9", size = 2327585, upload-time = "2025-11-04T13:41:40Z" }, + { url = "https://files.pythonhosted.org/packages/aa/4e/2ae1aa85d6af35a39b236b1b1641de73f5a6ac4d5a7509f77b814885760c/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2ff4321e56e879ee8d2a879501c8e469414d948f4aba74a2d4593184eb326660", size = 2041078, upload-time = "2025-11-04T13:41:42.323Z" }, + { url = "https://files.pythonhosted.org/packages/cd/13/2e215f17f0ef326fc72afe94776edb77525142c693767fc347ed6288728d/pydantic_core-2.41.5-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d0d2568a8c11bf8225044aa94409e21da0cb09dcdafe9ecd10250b2baad531a9", size = 2173914, upload-time = "2025-11-04T13:41:45.221Z" }, + { url = "https://files.pythonhosted.org/packages/02/7a/f999a6dcbcd0e5660bc348a3991c8915ce6599f4f2c6ac22f01d7a10816c/pydantic_core-2.41.5-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:a39455728aabd58ceabb03c90e12f71fd30fa69615760a075b9fec596456ccc3", size = 2129560, upload-time = "2025-11-04T13:41:47.474Z" }, + { url = "https://files.pythonhosted.org/packages/3a/b1/6c990ac65e3b4c079a4fb9f5b05f5b013afa0f4ed6780a3dd236d2cbdc64/pydantic_core-2.41.5-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:239edca560d05757817c13dc17c50766136d21f7cd0fac50295499ae24f90fdf", size = 2329244, upload-time = "2025-11-04T13:41:49.992Z" }, + { url = "https://files.pythonhosted.org/packages/d9/02/3c562f3a51afd4d88fff8dffb1771b30cfdfd79befd9883ee094f5b6c0d8/pydantic_core-2.41.5-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:2a5e06546e19f24c6a96a129142a75cee553cc018ffee48a460059b1185f4470", size = 2331955, upload-time = "2025-11-04T13:41:54.079Z" }, + { url = "https://files.pythonhosted.org/packages/5c/96/5fb7d8c3c17bc8c62fdb031c47d77a1af698f1d7a406b0f79aaa1338f9ad/pydantic_core-2.41.5-cp314-cp314t-win32.whl", hash = "sha256:b4ececa40ac28afa90871c2cc2b9ffd2ff0bf749380fbdf57d165fd23da353aa", size = 1988906, upload-time = "2025-11-04T13:41:56.606Z" }, + { url = "https://files.pythonhosted.org/packages/22/ed/182129d83032702912c2e2d8bbe33c036f342cc735737064668585dac28f/pydantic_core-2.41.5-cp314-cp314t-win_amd64.whl", hash = "sha256:80aa89cad80b32a912a65332f64a4450ed00966111b6615ca6816153d3585a8c", size = 1981607, upload-time = "2025-11-04T13:41:58.889Z" }, + { url = "https://files.pythonhosted.org/packages/9f/ed/068e41660b832bb0b1aa5b58011dea2a3fe0ba7861ff38c4d4904c1c1a99/pydantic_core-2.41.5-cp314-cp314t-win_arm64.whl", hash = "sha256:35b44f37a3199f771c3eaa53051bc8a70cd7b54f333531c59e29fd4db5d15008", size = 1974769, upload-time = "2025-11-04T13:42:01.186Z" }, + { url = "https://files.pythonhosted.org/packages/11/72/90fda5ee3b97e51c494938a4a44c3a35a9c96c19bba12372fb9c634d6f57/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-macosx_10_12_x86_64.whl", hash = "sha256:b96d5f26b05d03cc60f11a7761a5ded1741da411e7fe0909e27a5e6a0cb7b034", size = 2115441, upload-time = "2025-11-04T13:42:39.557Z" }, + { url = "https://files.pythonhosted.org/packages/1f/53/8942f884fa33f50794f119012dc6a1a02ac43a56407adaac20463df8e98f/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-macosx_11_0_arm64.whl", hash = "sha256:634e8609e89ceecea15e2d61bc9ac3718caaaa71963717bf3c8f38bfde64242c", size = 1930291, upload-time = "2025-11-04T13:42:42.169Z" }, + { url = "https://files.pythonhosted.org/packages/79/c8/ecb9ed9cd942bce09fc888ee960b52654fbdbede4ba6c2d6e0d3b1d8b49c/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:93e8740d7503eb008aa2df04d3b9735f845d43ae845e6dcd2be0b55a2da43cd2", size = 1948632, upload-time = "2025-11-04T13:42:44.564Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1b/687711069de7efa6af934e74f601e2a4307365e8fdc404703afc453eab26/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f15489ba13d61f670dcc96772e733aad1a6f9c429cc27574c6cdaed82d0146ad", size = 2138905, upload-time = "2025-11-04T13:42:47.156Z" }, + { url = "https://files.pythonhosted.org/packages/09/32/59b0c7e63e277fa7911c2fc70ccfb45ce4b98991e7ef37110663437005af/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:7da7087d756b19037bc2c06edc6c170eeef3c3bafcb8f532ff17d64dc427adfd", size = 2110495, upload-time = "2025-11-04T13:42:49.689Z" }, + { url = "https://files.pythonhosted.org/packages/aa/81/05e400037eaf55ad400bcd318c05bb345b57e708887f07ddb2d20e3f0e98/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:aabf5777b5c8ca26f7824cb4a120a740c9588ed58df9b2d196ce92fba42ff8dc", size = 1915388, upload-time = "2025-11-04T13:42:52.215Z" }, + { url = "https://files.pythonhosted.org/packages/6e/0d/e3549b2399f71d56476b77dbf3cf8937cec5cd70536bdc0e374a421d0599/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c007fe8a43d43b3969e8469004e9845944f1a80e6acd47c150856bb87f230c56", size = 1942879, upload-time = "2025-11-04T13:42:56.483Z" }, + { url = "https://files.pythonhosted.org/packages/f7/07/34573da085946b6a313d7c42f82f16e8920bfd730665de2d11c0c37a74b5/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:76d0819de158cd855d1cbb8fcafdf6f5cf1eb8e470abe056d5d161106e38062b", size = 2139017, upload-time = "2025-11-04T13:42:59.471Z" }, + { url = "https://files.pythonhosted.org/packages/5f/9b/1b3f0e9f9305839d7e84912f9e8bfbd191ed1b1ef48083609f0dabde978c/pydantic_core-2.41.5-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:b2379fa7ed44ddecb5bfe4e48577d752db9fc10be00a6b7446e9663ba143de26", size = 2101980, upload-time = "2025-11-04T13:43:25.97Z" }, + { url = "https://files.pythonhosted.org/packages/a4/ed/d71fefcb4263df0da6a85b5d8a7508360f2f2e9b3bf5814be9c8bccdccc1/pydantic_core-2.41.5-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:266fb4cbf5e3cbd0b53669a6d1b039c45e3ce651fd5442eff4d07c2cc8d66808", size = 1923865, upload-time = "2025-11-04T13:43:28.763Z" }, + { url = "https://files.pythonhosted.org/packages/ce/3a/626b38db460d675f873e4444b4bb030453bbe7b4ba55df821d026a0493c4/pydantic_core-2.41.5-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58133647260ea01e4d0500089a8c4f07bd7aa6ce109682b1426394988d8aaacc", size = 2134256, upload-time = "2025-11-04T13:43:31.71Z" }, + { url = "https://files.pythonhosted.org/packages/83/d9/8412d7f06f616bbc053d30cb4e5f76786af3221462ad5eee1f202021eb4e/pydantic_core-2.41.5-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:287dad91cfb551c363dc62899a80e9e14da1f0e2b6ebde82c806612ca2a13ef1", size = 2174762, upload-time = "2025-11-04T13:43:34.744Z" }, + { url = "https://files.pythonhosted.org/packages/55/4c/162d906b8e3ba3a99354e20faa1b49a85206c47de97a639510a0e673f5da/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_aarch64.whl", hash = "sha256:03b77d184b9eb40240ae9fd676ca364ce1085f203e1b1256f8ab9984dca80a84", size = 2143141, upload-time = "2025-11-04T13:43:37.701Z" }, + { url = "https://files.pythonhosted.org/packages/1f/f2/f11dd73284122713f5f89fc940f370d035fa8e1e078d446b3313955157fe/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_armv7l.whl", hash = "sha256:a668ce24de96165bb239160b3d854943128f4334822900534f2fe947930e5770", size = 2330317, upload-time = "2025-11-04T13:43:40.406Z" }, + { url = "https://files.pythonhosted.org/packages/88/9d/b06ca6acfe4abb296110fb1273a4d848a0bfb2ff65f3ee92127b3244e16b/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_x86_64.whl", hash = "sha256:f14f8f046c14563f8eb3f45f499cc658ab8d10072961e07225e507adb700e93f", size = 2316992, upload-time = "2025-11-04T13:43:43.602Z" }, + { url = "https://files.pythonhosted.org/packages/36/c7/cfc8e811f061c841d7990b0201912c3556bfeb99cdcb7ed24adc8d6f8704/pydantic_core-2.41.5-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:56121965f7a4dc965bff783d70b907ddf3d57f6eba29b6d2e5dabfaf07799c51", size = 2145302, upload-time = "2025-11-04T13:43:46.64Z" }, +] + +[[package]] +name = "pydantic-settings" +version = "2.13.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/52/6d/fffca34caecc4a3f97bda81b2098da5e8ab7efc9a66e819074a11955d87e/pydantic_settings-2.13.1.tar.gz", hash = "sha256:b4c11847b15237fb0171e1462bf540e294affb9b86db4d9aa5c01730bdbe4025", size = 223826, upload-time = "2026-02-19T13:45:08.055Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/00/4b/ccc026168948fec4f7555b9164c724cf4125eac006e176541483d2c959be/pydantic_settings-2.13.1-py3-none-any.whl", hash = "sha256:d56fd801823dbeae7f0975e1f8c8e25c258eb75d278ea7abb5d9cebb01b56237", size = 58929, upload-time = "2026-02-19T13:45:06.034Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pyjwt" +version = "2.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c2/27/a3b6e5bf6ff856d2509292e95c8f57f0df7017cf5394921fc4e4ef40308a/pyjwt-2.12.1.tar.gz", hash = "sha256:c74a7a2adf861c04d002db713dd85f84beb242228e671280bf709d765b03672b", size = 102564, upload-time = "2026-03-13T19:27:37.25Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/7a/8dd906bd22e79e47397a61742927f6747fe93242ef86645ee9092e610244/pyjwt-2.12.1-py3-none-any.whl", hash = "sha256:28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c", size = 29726, upload-time = "2026-03-13T19:27:35.677Z" }, +] + +[package.optional-dependencies] +crypto = [ + { name = "cryptography" }, +] + +[[package]] +name = "pypdf" +version = "6.9.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/31/83/691bdb309306232362503083cb15777491045dd54f45393a317dc7d8082f/pypdf-6.9.2.tar.gz", hash = "sha256:7f850faf2b0d4ab936582c05da32c52214c2b089d61a316627b5bfb5b0dab46c", size = 5311837, upload-time = "2026-03-23T14:53:27.983Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a5/7e/c85f41243086a8fe5d1baeba527cb26a1918158a565932b41e0f7c0b32e9/pypdf-6.9.2-py3-none-any.whl", hash = "sha256:662cf29bcb419a36a1365232449624ab40b7c2d0cfc28e54f42eeecd1fd7e844", size = 333744, upload-time = "2026-03-23T14:53:26.573Z" }, +] + +[[package]] +name = "pytest" +version = "9.0.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d1/db/7ef3487e0fb0049ddb5ce41d3a49c235bf9ad299b6a25d5780a89f19230f/pytest-9.0.2.tar.gz", hash = "sha256:75186651a92bd89611d1d9fc20f0b4345fd827c41ccd5c299a868a05d70edf11", size = 1568901, upload-time = "2025-12-06T21:30:51.014Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3b/ab/b3226f0bd7cdcf710fbede2b3548584366da3b19b5021e74f5bde2a8fa3f/pytest-9.0.2-py3-none-any.whl", hash = "sha256:711ffd45bf766d5264d487b917733b453d917afd2b0ad65223959f59089f875b", size = 374801, upload-time = "2025-12-06T21:30:49.154Z" }, +] + +[[package]] +name = "pytest-asyncio" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pytest" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/90/2c/8af215c0f776415f3590cac4f9086ccefd6fd463befeae41cd4d3f193e5a/pytest_asyncio-1.3.0.tar.gz", hash = "sha256:d7f52f36d231b80ee124cd216ffb19369aa168fc10095013c6b014a34d3ee9e5", size = 50087, upload-time = "2025-11-10T16:07:47.256Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/35/f8b19922b6a25bc0880171a2f1a003eaeb93657475193ab516fd87cac9da/pytest_asyncio-1.3.0-py3-none-any.whl", hash = "sha256:611e26147c7f77640e6d0a92a38ed17c3e9848063698d5c93d5aa7aa11cebff5", size = 15075, upload-time = "2025-11-10T16:07:45.537Z" }, +] + +[[package]] +name = "pytest-json-report" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pytest" }, + { name = "pytest-metadata" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4f/d3/765dae9712fcd68d820338908c1337e077d5fdadccd5cacf95b9b0bea278/pytest-json-report-1.5.0.tar.gz", hash = "sha256:2dde3c647851a19b5f3700729e8310a6e66efb2077d674f27ddea3d34dc615de", size = 21241, upload-time = "2022-03-15T21:03:10.2Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/81/35/d07400c715bf8a88aa0c1ee9c9eb6050ca7fe5b39981f0eea773feeb0681/pytest_json_report-1.5.0-py3-none-any.whl", hash = "sha256:9897b68c910b12a2e48dd849f9a284b2c79a732a8a9cb398452ddd23d3c8c325", size = 13222, upload-time = "2022-03-15T21:03:08.65Z" }, +] + +[[package]] +name = "pytest-metadata" +version = "3.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a6/85/8c969f8bec4e559f8f2b958a15229a35495f5b4ce499f6b865eac54b878d/pytest_metadata-3.1.1.tar.gz", hash = "sha256:d2a29b0355fbc03f168aa96d41ff88b1a3b44a3b02acbe491801c98a048017c8", size = 9952, upload-time = "2024-02-12T19:38:44.887Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/43/7e7b2ec865caa92f67b8f0e9231a798d102724ca4c0e1f414316be1c1ef2/pytest_metadata-3.1.1-py3-none-any.whl", hash = "sha256:c8e0844db684ee1c798cfa38908d20d67d0463ecb6137c72e91f418558dd5f4b", size = 11428, upload-time = "2024-02-12T19:38:42.531Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "python-dotenv" +version = "1.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/82/ed/0301aeeac3e5353ef3d94b6ec08bbcabd04a72018415dcb29e588514bba8/python_dotenv-1.2.2.tar.gz", hash = "sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3", size = 50135, upload-time = "2026-03-01T16:00:26.196Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" }, +] + +[[package]] +name = "python-multipart" +version = "0.0.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/01/979e98d542a70714b0cb2b6728ed0b7c46792b695e3eaec3e20711271ca3/python_multipart-0.0.22.tar.gz", hash = "sha256:7340bef99a7e0032613f56dc36027b959fd3b30a787ed62d310e951f7c3a3a58", size = 37612, upload-time = "2026-01-25T10:15:56.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1b/d0/397f9626e711ff749a95d96b7af99b9c566a9bb5129b8e4c10fc4d100304/python_multipart-0.0.22-py3-none-any.whl", hash = "sha256:2b2cd894c83d21bf49d702499531c7bafd057d730c201782048f7945d82de155", size = 24579, upload-time = "2026-01-25T10:15:54.811Z" }, +] + +[[package]] +name = "pytz" +version = "2026.1.post1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/56/db/b8721d71d945e6a8ac63c0fc900b2067181dbb50805958d4d4661cf7d277/pytz-2026.1.post1.tar.gz", hash = "sha256:3378dde6a0c3d26719182142c56e60c7f9af7e968076f31aae569d72a0358ee1", size = 321088, upload-time = "2026-03-03T07:47:50.683Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/99/781fe0c827be2742bcc775efefccb3b048a3a9c6ce9aec0cbf4a101677e5/pytz-2026.1.post1-py2.py3-none-any.whl", hash = "sha256:f2fd16142fda348286a75e1a524be810bb05d444e5a081f37f7affc635035f7a", size = 510489, upload-time = "2026-03-03T07:47:49.167Z" }, +] + +[[package]] +name = "pywin32" +version = "311" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7c/af/449a6a91e5d6db51420875c54f6aff7c97a86a3b13a0b4f1a5c13b988de3/pywin32-311-cp311-cp311-win32.whl", hash = "sha256:184eb5e436dea364dcd3d2316d577d625c0351bf237c4e9a5fabbcfa5a58b151", size = 8697031, upload-time = "2025-07-14T20:13:13.266Z" }, + { url = "https://files.pythonhosted.org/packages/51/8f/9bb81dd5bb77d22243d33c8397f09377056d5c687aa6d4042bea7fbf8364/pywin32-311-cp311-cp311-win_amd64.whl", hash = "sha256:3ce80b34b22b17ccbd937a6e78e7225d80c52f5ab9940fe0506a1a16f3dab503", size = 9508308, upload-time = "2025-07-14T20:13:15.147Z" }, + { url = "https://files.pythonhosted.org/packages/44/7b/9c2ab54f74a138c491aba1b1cd0795ba61f144c711daea84a88b63dc0f6c/pywin32-311-cp311-cp311-win_arm64.whl", hash = "sha256:a733f1388e1a842abb67ffa8e7aad0e70ac519e09b0f6a784e65a136ec7cefd2", size = 8703930, upload-time = "2025-07-14T20:13:16.945Z" }, + { url = "https://files.pythonhosted.org/packages/e7/ab/01ea1943d4eba0f850c3c61e78e8dd59757ff815ff3ccd0a84de5f541f42/pywin32-311-cp312-cp312-win32.whl", hash = "sha256:750ec6e621af2b948540032557b10a2d43b0cee2ae9758c54154d711cc852d31", size = 8706543, upload-time = "2025-07-14T20:13:20.765Z" }, + { url = "https://files.pythonhosted.org/packages/d1/a8/a0e8d07d4d051ec7502cd58b291ec98dcc0c3fff027caad0470b72cfcc2f/pywin32-311-cp312-cp312-win_amd64.whl", hash = "sha256:b8c095edad5c211ff31c05223658e71bf7116daa0ecf3ad85f3201ea3190d067", size = 9495040, upload-time = "2025-07-14T20:13:22.543Z" }, + { url = "https://files.pythonhosted.org/packages/ba/3a/2ae996277b4b50f17d61f0603efd8253cb2d79cc7ae159468007b586396d/pywin32-311-cp312-cp312-win_arm64.whl", hash = "sha256:e286f46a9a39c4a18b319c28f59b61de793654af2f395c102b4f819e584b5852", size = 8710102, upload-time = "2025-07-14T20:13:24.682Z" }, + { url = "https://files.pythonhosted.org/packages/a5/be/3fd5de0979fcb3994bfee0d65ed8ca9506a8a1260651b86174f6a86f52b3/pywin32-311-cp313-cp313-win32.whl", hash = "sha256:f95ba5a847cba10dd8c4d8fefa9f2a6cf283b8b88ed6178fa8a6c1ab16054d0d", size = 8705700, upload-time = "2025-07-14T20:13:26.471Z" }, + { url = "https://files.pythonhosted.org/packages/e3/28/e0a1909523c6890208295a29e05c2adb2126364e289826c0a8bc7297bd5c/pywin32-311-cp313-cp313-win_amd64.whl", hash = "sha256:718a38f7e5b058e76aee1c56ddd06908116d35147e133427e59a3983f703a20d", size = 9494700, upload-time = "2025-07-14T20:13:28.243Z" }, + { url = "https://files.pythonhosted.org/packages/04/bf/90339ac0f55726dce7d794e6d79a18a91265bdf3aa70b6b9ca52f35e022a/pywin32-311-cp313-cp313-win_arm64.whl", hash = "sha256:7b4075d959648406202d92a2310cb990fea19b535c7f4a78d3f5e10b926eeb8a", size = 8709318, upload-time = "2025-07-14T20:13:30.348Z" }, + { url = "https://files.pythonhosted.org/packages/c9/31/097f2e132c4f16d99a22bfb777e0fd88bd8e1c634304e102f313af69ace5/pywin32-311-cp314-cp314-win32.whl", hash = "sha256:b7a2c10b93f8986666d0c803ee19b5990885872a7de910fc460f9b0c2fbf92ee", size = 8840714, upload-time = "2025-07-14T20:13:32.449Z" }, + { url = "https://files.pythonhosted.org/packages/90/4b/07c77d8ba0e01349358082713400435347df8426208171ce297da32c313d/pywin32-311-cp314-cp314-win_amd64.whl", hash = "sha256:3aca44c046bd2ed8c90de9cb8427f581c479e594e99b5c0bb19b29c10fd6cb87", size = 9656800, upload-time = "2025-07-14T20:13:34.312Z" }, + { url = "https://files.pythonhosted.org/packages/c0/d2/21af5c535501a7233e734b8af901574572da66fcc254cb35d0609c9080dd/pywin32-311-cp314-cp314-win_arm64.whl", hash = "sha256:a508e2d9025764a8270f93111a970e1d0fbfc33f4153b388bb649b7eec4f9b42", size = 8932540, upload-time = "2025-07-14T20:13:36.379Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6d/16/a95b6757765b7b031c9374925bb718d55e0a9ba8a1b6a12d25962ea44347/pyyaml-6.0.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e", size = 185826, upload-time = "2025-09-25T21:31:58.655Z" }, + { url = "https://files.pythonhosted.org/packages/16/19/13de8e4377ed53079ee996e1ab0a9c33ec2faf808a4647b7b4c0d46dd239/pyyaml-6.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824", size = 175577, upload-time = "2025-09-25T21:32:00.088Z" }, + { url = "https://files.pythonhosted.org/packages/0c/62/d2eb46264d4b157dae1275b573017abec435397aa59cbcdab6fc978a8af4/pyyaml-6.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c", size = 775556, upload-time = "2025-09-25T21:32:01.31Z" }, + { url = "https://files.pythonhosted.org/packages/10/cb/16c3f2cf3266edd25aaa00d6c4350381c8b012ed6f5276675b9eba8d9ff4/pyyaml-6.0.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00", size = 882114, upload-time = "2025-09-25T21:32:03.376Z" }, + { url = "https://files.pythonhosted.org/packages/71/60/917329f640924b18ff085ab889a11c763e0b573da888e8404ff486657602/pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d", size = 806638, upload-time = "2025-09-25T21:32:04.553Z" }, + { url = "https://files.pythonhosted.org/packages/dd/6f/529b0f316a9fd167281a6c3826b5583e6192dba792dd55e3203d3f8e655a/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a", size = 767463, upload-time = "2025-09-25T21:32:06.152Z" }, + { url = "https://files.pythonhosted.org/packages/f2/6a/b627b4e0c1dd03718543519ffb2f1deea4a1e6d42fbab8021936a4d22589/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4", size = 794986, upload-time = "2025-09-25T21:32:07.367Z" }, + { url = "https://files.pythonhosted.org/packages/45/91/47a6e1c42d9ee337c4839208f30d9f09caa9f720ec7582917b264defc875/pyyaml-6.0.3-cp311-cp311-win32.whl", hash = "sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b", size = 142543, upload-time = "2025-09-25T21:32:08.95Z" }, + { url = "https://files.pythonhosted.org/packages/da/e3/ea007450a105ae919a72393cb06f122f288ef60bba2dc64b26e2646fa315/pyyaml-6.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf", size = 158763, upload-time = "2025-09-25T21:32:09.96Z" }, + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" }, + { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" }, + { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" }, + { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" }, + { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" }, + { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" }, + { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" }, + { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" }, + { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" }, + { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" }, + { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" }, + { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, +] + +[[package]] +name = "referencing" +version = "0.37.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "rpds-py" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/22/f5/df4e9027acead3ecc63e50fe1e36aca1523e1719559c499951bb4b53188f/referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8", size = 78036, upload-time = "2025-10-13T15:30:48.871Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" }, +] + +[[package]] +name = "regex" +version = "2026.4.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cb/0e/3a246dbf05666918bd3664d9d787f84a9108f6f43cc953a077e4a7dfdb7e/regex-2026.4.4.tar.gz", hash = "sha256:e08270659717f6973523ce3afbafa53515c4dc5dcad637dc215b6fd50f689423", size = 416000, upload-time = "2026-04-03T20:56:28.155Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/7a/617356cbecdb452812a5d42f720d6d5096b360d4a4c1073af700ea140ad2/regex-2026.4.4-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:b4c36a85b00fadb85db9d9e90144af0a980e1a3d2ef9cd0f8a5bef88054657c6", size = 489415, upload-time = "2026-04-03T20:53:11.645Z" }, + { url = "https://files.pythonhosted.org/packages/20/e6/bf057227144d02e3ba758b66649e87531d744dda5f3254f48660f18ae9d8/regex-2026.4.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:dcb5453ecf9cd58b562967badd1edbf092b0588a3af9e32ee3d05c985077ce87", size = 291205, upload-time = "2026-04-03T20:53:13.289Z" }, + { url = "https://files.pythonhosted.org/packages/eb/3b/637181b787dd1a820ba1c712cee2b4144cd84a32dc776ca067b12b2d70c8/regex-2026.4.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:6aa809ed4dc3706cc38594d67e641601bd2f36d5555b2780ff074edfcb136cf8", size = 289225, upload-time = "2026-04-03T20:53:16.002Z" }, + { url = "https://files.pythonhosted.org/packages/05/21/bac05d806ed02cd4b39d9c8e5b5f9a2998c94c3a351b7792e80671fa5315/regex-2026.4.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:33424f5188a7db12958246a54f59a435b6cb62c5cf9c8d71f7cc49475a5fdada", size = 792434, upload-time = "2026-04-03T20:53:17.414Z" }, + { url = "https://files.pythonhosted.org/packages/d9/17/c65d1d8ae90b772d5758eb4014e1e011bb2db353fc4455432e6cc9100df7/regex-2026.4.4-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7d346fccdde28abba117cc9edc696b9518c3307fbfcb689e549d9b5979018c6d", size = 861730, upload-time = "2026-04-03T20:53:18.903Z" }, + { url = "https://files.pythonhosted.org/packages/ad/64/933321aa082a2c6ee2785f22776143ba89840189c20d3b6b1d12b6aae16b/regex-2026.4.4-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:415a994b536440f5011aa77e50a4274d15da3245e876e5c7f19da349caaedd87", size = 906495, upload-time = "2026-04-03T20:53:20.561Z" }, + { url = "https://files.pythonhosted.org/packages/01/ea/4c8d306e9c36ac22417336b1e02e7b358152c34dc379673f2d331143725f/regex-2026.4.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:21e5eb86179b4c67b5759d452ea7c48eb135cd93308e7a260aa489ed2eb423a4", size = 799810, upload-time = "2026-04-03T20:53:22.961Z" }, + { url = "https://files.pythonhosted.org/packages/29/ce/7605048f00e1379eba89d610c7d644d8f695dc9b26d3b6ecfa3132b872ff/regex-2026.4.4-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:312ec9dd1ae7d96abd8c5a36a552b2139931914407d26fba723f9e53c8186f86", size = 774242, upload-time = "2026-04-03T20:53:25.015Z" }, + { url = "https://files.pythonhosted.org/packages/e9/77/283e0d5023fde22cd9e86190d6d9beb21590a452b195ffe00274de470691/regex-2026.4.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:a0d2b28aa1354c7cd7f71b7658c4326f7facac106edd7f40eda984424229fd59", size = 781257, upload-time = "2026-04-03T20:53:26.918Z" }, + { url = "https://files.pythonhosted.org/packages/8b/fb/7f3b772be101373c8626ed34c5d727dcbb8abd42a7b1219bc25fd9a3cc04/regex-2026.4.4-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:349d7310eddff40429a099c08d995c6d4a4bfaf3ff40bd3b5e5cb5a5a3c7d453", size = 854490, upload-time = "2026-04-03T20:53:29.065Z" }, + { url = "https://files.pythonhosted.org/packages/85/30/56547b80f34f4dd2986e1cdd63b1712932f63b6c4ce2f79c50a6cd79d1c2/regex-2026.4.4-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:e7ab63e9fe45a9ec3417509e18116b367e89c9ceb6219222a3396fa30b147f80", size = 763544, upload-time = "2026-04-03T20:53:30.917Z" }, + { url = "https://files.pythonhosted.org/packages/ac/2f/ce060fdfea8eff34a8997603532e44cdb7d1f35e3bc253612a8707a90538/regex-2026.4.4-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:fe896e07a5a2462308297e515c0054e9ec2dd18dfdc9427b19900b37dfe6f40b", size = 844442, upload-time = "2026-04-03T20:53:32.463Z" }, + { url = "https://files.pythonhosted.org/packages/e5/44/810cb113096a1dacbe82789fbfab2823f79d19b7f1271acecb7009ba9b88/regex-2026.4.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:eb59c65069498dbae3c0ef07bbe224e1eaa079825a437fb47a479f0af11f774f", size = 789162, upload-time = "2026-04-03T20:53:34.039Z" }, + { url = "https://files.pythonhosted.org/packages/20/96/9647dd7f2ecf6d9ce1fb04dfdb66910d094e10d8fe53e9c15096d8aa0bd2/regex-2026.4.4-cp311-cp311-win32.whl", hash = "sha256:2a5d273181b560ef8397c8825f2b9d57013de744da9e8257b8467e5da8599351", size = 266227, upload-time = "2026-04-03T20:53:35.601Z" }, + { url = "https://files.pythonhosted.org/packages/33/80/74e13262460530c3097ff343a17de9a34d040a5dc4de9cf3a8241faab51c/regex-2026.4.4-cp311-cp311-win_amd64.whl", hash = "sha256:9542ccc1e689e752594309444081582f7be2fdb2df75acafea8a075108566735", size = 278399, upload-time = "2026-04-03T20:53:37.021Z" }, + { url = "https://files.pythonhosted.org/packages/1c/3c/39f19f47f19dcefa3403f09d13562ca1c0fd07ab54db2bc03148f3f6b46a/regex-2026.4.4-cp311-cp311-win_arm64.whl", hash = "sha256:b5f9fb784824a042be3455b53d0b112655686fdb7a91f88f095f3fee1e2a2a54", size = 270473, upload-time = "2026-04-03T20:53:38.633Z" }, + { url = "https://files.pythonhosted.org/packages/e5/28/b972a4d3df61e1d7bcf1b59fdb3cddef22f88b6be43f161bb41ebc0e4081/regex-2026.4.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:c07ab8794fa929e58d97a0e1796b8b76f70943fa39df225ac9964615cf1f9d52", size = 490434, upload-time = "2026-04-03T20:53:40.219Z" }, + { url = "https://files.pythonhosted.org/packages/84/20/30041446cf6dc3e0eab344fc62770e84c23b6b68a3b657821f9f80cb69b4/regex-2026.4.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2c785939dc023a1ce4ec09599c032cc9933d258a998d16ca6f2b596c010940eb", size = 292061, upload-time = "2026-04-03T20:53:41.862Z" }, + { url = "https://files.pythonhosted.org/packages/62/c8/3baa06d75c98c46d4cc4262b71fd2edb9062b5665e868bca57859dadf93a/regex-2026.4.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1b1ce5c81c9114f1ce2f9288a51a8fd3aeea33a0cc440c415bf02da323aa0a76", size = 289628, upload-time = "2026-04-03T20:53:43.701Z" }, + { url = "https://files.pythonhosted.org/packages/31/87/3accf55634caad8c0acab23f5135ef7d4a21c39f28c55c816ae012931408/regex-2026.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:760ef21c17d8e6a4fe8cf406a97cf2806a4df93416ccc82fc98d25b1c20425be", size = 796651, upload-time = "2026-04-03T20:53:45.379Z" }, + { url = "https://files.pythonhosted.org/packages/f6/0c/aaa2c83f34efedbf06f61cb1942c25f6cf1ee3b200f832c4d05f28306c2e/regex-2026.4.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7088fcdcb604a4417c208e2169715800d28838fefd7455fbe40416231d1d47c1", size = 865916, upload-time = "2026-04-03T20:53:47.064Z" }, + { url = "https://files.pythonhosted.org/packages/d9/f6/8c6924c865124643e8f37823eca845dc27ac509b2ee58123685e71cd0279/regex-2026.4.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:07edca1ba687998968f7db5bc355288d0c6505caa7374f013d27356d93976d13", size = 912287, upload-time = "2026-04-03T20:53:49.422Z" }, + { url = "https://files.pythonhosted.org/packages/11/0e/a9f6f81013e0deaf559b25711623864970fe6a098314e374ccb1540a4152/regex-2026.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:993f657a7c1c6ec51b5e0ba97c9817d06b84ea5fa8d82e43b9405de0defdc2b9", size = 801126, upload-time = "2026-04-03T20:53:51.096Z" }, + { url = "https://files.pythonhosted.org/packages/71/61/3a0cc8af2dc0c8deb48e644dd2521f173f7e6513c6e195aad9aa8dd77ac5/regex-2026.4.4-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:2b69102a743e7569ebee67e634a69c4cb7e59d6fa2e1aa7d3bdbf3f61435f62d", size = 776788, upload-time = "2026-04-03T20:53:52.889Z" }, + { url = "https://files.pythonhosted.org/packages/64/0b/8bb9cbf21ef7dee58e49b0fdb066a7aded146c823202e16494a36777594f/regex-2026.4.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6dac006c8b6dda72d86ea3d1333d45147de79a3a3f26f10c1cf9287ca4ca0ac3", size = 785184, upload-time = "2026-04-03T20:53:55.627Z" }, + { url = "https://files.pythonhosted.org/packages/99/c2/d3e80e8137b25ee06c92627de4e4d98b94830e02b3e6f81f3d2e3f504cf5/regex-2026.4.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:50a766ee2010d504554bfb5f578ed2e066898aa26411d57e6296230627cdefa0", size = 859913, upload-time = "2026-04-03T20:53:57.249Z" }, + { url = "https://files.pythonhosted.org/packages/bc/e6/9d5d876157d969c804622456ef250017ac7a8f83e0e14f903b9e6df5ce95/regex-2026.4.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:9e2f5217648f68e3028c823df58663587c1507a5ba8419f4fdfc8a461be76043", size = 765732, upload-time = "2026-04-03T20:53:59.428Z" }, + { url = "https://files.pythonhosted.org/packages/82/80/b568935b4421388561c8ed42aff77247285d3ae3bb2a6ca22af63bae805e/regex-2026.4.4-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:39d8de85a08e32632974151ba59c6e9140646dcc36c80423962b1c5c0a92e244", size = 852152, upload-time = "2026-04-03T20:54:01.505Z" }, + { url = "https://files.pythonhosted.org/packages/39/29/f0f81217e21cd998245da047405366385d5c6072048038a3d33b37a79dc0/regex-2026.4.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:55d9304e0e7178dfb1e106c33edf834097ddf4a890e2f676f6c5118f84390f73", size = 789076, upload-time = "2026-04-03T20:54:03.323Z" }, + { url = "https://files.pythonhosted.org/packages/49/1d/1d957a61976ab9d4e767dd4f9d04b66cc0c41c5e36cf40e2d43688b5ae6f/regex-2026.4.4-cp312-cp312-win32.whl", hash = "sha256:04bb679bc0bde8a7bfb71e991493d47314e7b98380b083df2447cda4b6edb60f", size = 266700, upload-time = "2026-04-03T20:54:05.639Z" }, + { url = "https://files.pythonhosted.org/packages/c5/5c/bf575d396aeb58ea13b06ef2adf624f65b70fafef6950a80fc3da9cae3bc/regex-2026.4.4-cp312-cp312-win_amd64.whl", hash = "sha256:db0ac18435a40a2543dbb3d21e161a6c78e33e8159bd2e009343d224bb03bb1b", size = 277768, upload-time = "2026-04-03T20:54:07.312Z" }, + { url = "https://files.pythonhosted.org/packages/c9/27/049df16ec6a6828ccd72add3c7f54b4df029669bea8e9817df6fff58be90/regex-2026.4.4-cp312-cp312-win_arm64.whl", hash = "sha256:4ce255cc05c1947a12989c6db801c96461947adb7a59990f1360b5983fab4983", size = 270568, upload-time = "2026-04-03T20:54:09.484Z" }, + { url = "https://files.pythonhosted.org/packages/9d/83/c4373bc5f31f2cf4b66f9b7c31005bd87fe66f0dce17701f7db4ee79ee29/regex-2026.4.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:62f5519042c101762509b1d717b45a69c0139d60414b3c604b81328c01bd1943", size = 490273, upload-time = "2026-04-03T20:54:11.202Z" }, + { url = "https://files.pythonhosted.org/packages/46/f8/fe62afbcc3cf4ad4ac9adeaafd98aa747869ae12d3e8e2ac293d0593c435/regex-2026.4.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:3790ba9fb5dd76715a7afe34dbe603ba03f8820764b1dc929dd08106214ed031", size = 291954, upload-time = "2026-04-03T20:54:13.412Z" }, + { url = "https://files.pythonhosted.org/packages/5a/92/4712b9fe6a33d232eeb1c189484b80c6c4b8422b90e766e1195d6e758207/regex-2026.4.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8fae3c6e795d7678963f2170152b0d892cf6aee9ee8afc8c45e6be38d5107fe7", size = 289487, upload-time = "2026-04-03T20:54:15.824Z" }, + { url = "https://files.pythonhosted.org/packages/88/2c/f83b93f85e01168f1070f045a42d4c937b69fdb8dd7ae82d307253f7e36e/regex-2026.4.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:298c3ec2d53225b3bf91142eb9691025bab610e0c0c51592dde149db679b3d17", size = 796646, upload-time = "2026-04-03T20:54:18.229Z" }, + { url = "https://files.pythonhosted.org/packages/df/55/61a2e17bf0c4dc57e11caf8dd11771280d8aaa361785f9e3bc40d653f4a7/regex-2026.4.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e9638791082eaf5b3ac112c587518ee78e083a11c4b28012d8fe2a0f536dfb17", size = 865904, upload-time = "2026-04-03T20:54:20.019Z" }, + { url = "https://files.pythonhosted.org/packages/45/32/1ac8ed1b5a346b5993a3d256abe0a0f03b0b73c8cc88d928537368ac65b6/regex-2026.4.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ae3e764bd4c5ff55035dc82a8d49acceb42a5298edf6eb2fc4d328ee5dd7afae", size = 912304, upload-time = "2026-04-03T20:54:22.403Z" }, + { url = "https://files.pythonhosted.org/packages/26/47/2ee5c613ab546f0eddebf9905d23e07beb933416b1246c2d8791d01979b4/regex-2026.4.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ffa81f81b80047ba89a3c69ae6a0f78d06f4a42ce5126b0eb2a0a10ad44e0b2e", size = 801126, upload-time = "2026-04-03T20:54:24.308Z" }, + { url = "https://files.pythonhosted.org/packages/75/cd/41dacd129ca9fd20bd7d02f83e0fad83e034ac8a084ec369c90f55ef37e2/regex-2026.4.4-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f56ebf9d70305307a707911b88469213630aba821e77de7d603f9d2f0730687d", size = 776772, upload-time = "2026-04-03T20:54:26.319Z" }, + { url = "https://files.pythonhosted.org/packages/89/6d/5af0b588174cb5f46041fa7dd64d3fd5cd2fe51f18766703d1edc387f324/regex-2026.4.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:773d1dfd652bbffb09336abf890bfd64785c7463716bf766d0eb3bc19c8b7f27", size = 785228, upload-time = "2026-04-03T20:54:28.387Z" }, + { url = "https://files.pythonhosted.org/packages/b7/3b/f5a72b7045bd59575fc33bf1345f156fcfd5a8484aea6ad84b12c5a82114/regex-2026.4.4-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:d51d20befd5275d092cdffba57ded05f3c436317ee56466c8928ac32d960edaf", size = 860032, upload-time = "2026-04-03T20:54:30.641Z" }, + { url = "https://files.pythonhosted.org/packages/39/a4/72a317003d6fcd7a573584a85f59f525dfe8f67e355ca74eb6b53d66a5e2/regex-2026.4.4-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:0a51cdb3c1e9161154f976cb2bef9894bc063ac82f31b733087ffb8e880137d0", size = 765714, upload-time = "2026-04-03T20:54:32.789Z" }, + { url = "https://files.pythonhosted.org/packages/25/1e/5672e16f34dbbcb2560cc7e6a2fbb26dfa8b270711e730101da4423d3973/regex-2026.4.4-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:ae5266a82596114e41fb5302140e9630204c1b5f325c770bec654b95dd54b0aa", size = 852078, upload-time = "2026-04-03T20:54:34.546Z" }, + { url = "https://files.pythonhosted.org/packages/f7/0d/c813f0af7c6cc7ed7b9558bac2e5120b60ad0fa48f813e4d4bd55446f214/regex-2026.4.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:c882cd92ec68585e9c1cf36c447ec846c0d94edd706fe59e0c198e65822fd23b", size = 789181, upload-time = "2026-04-03T20:54:36.642Z" }, + { url = "https://files.pythonhosted.org/packages/ea/6d/a344608d1adbd2a95090ddd906cec09a11be0e6517e878d02a5123e0917f/regex-2026.4.4-cp313-cp313-win32.whl", hash = "sha256:05568c4fbf3cb4fa9e28e3af198c40d3237cf6041608a9022285fe567ec3ad62", size = 266690, upload-time = "2026-04-03T20:54:38.343Z" }, + { url = "https://files.pythonhosted.org/packages/31/07/54049f89b46235ca6f45cd6c88668a7050e77d4a15555e47dd40fde75263/regex-2026.4.4-cp313-cp313-win_amd64.whl", hash = "sha256:3384df51ed52db0bea967e21458ab0a414f67cdddfd94401688274e55147bb81", size = 277733, upload-time = "2026-04-03T20:54:40.11Z" }, + { url = "https://files.pythonhosted.org/packages/0e/21/61366a8e20f4d43fb597708cac7f0e2baadb491ecc9549b4980b2be27d16/regex-2026.4.4-cp313-cp313-win_arm64.whl", hash = "sha256:acd38177bd2c8e69a411d6521760806042e244d0ef94e2dd03ecdaa8a3c99427", size = 270565, upload-time = "2026-04-03T20:54:41.883Z" }, + { url = "https://files.pythonhosted.org/packages/f1/1e/3a2b9672433bef02f5d39aa1143ca2c08f311c1d041c464a42be9ae648dc/regex-2026.4.4-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:f94a11a9d05afcfcfa640e096319720a19cc0c9f7768e1a61fceee6a3afc6c7c", size = 494126, upload-time = "2026-04-03T20:54:43.602Z" }, + { url = "https://files.pythonhosted.org/packages/4e/4b/c132a4f4fe18ad3340d89fcb56235132b69559136036b845be3c073142ed/regex-2026.4.4-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:36bcb9d6d1307ab629edc553775baada2aefa5c50ccc0215fbfd2afcfff43141", size = 293882, upload-time = "2026-04-03T20:54:45.41Z" }, + { url = "https://files.pythonhosted.org/packages/f4/5f/eaa38092ce7a023656280f2341dbbd4ad5f05d780a70abba7bb4f4bea54c/regex-2026.4.4-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:261c015b3e2ed0919157046d768774ecde57f03d8fa4ba78d29793447f70e717", size = 292334, upload-time = "2026-04-03T20:54:47.051Z" }, + { url = "https://files.pythonhosted.org/packages/5f/f6/dd38146af1392dac33db7074ab331cec23cced3759167735c42c5460a243/regex-2026.4.4-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c228cf65b4a54583763645dcd73819b3b381ca8b4bb1b349dee1c135f4112c07", size = 811691, upload-time = "2026-04-03T20:54:49.074Z" }, + { url = "https://files.pythonhosted.org/packages/7a/f0/dc54c2e69f5eeec50601054998ec3690d5344277e782bd717e49867c1d29/regex-2026.4.4-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:dd2630faeb6876fb0c287f664d93ddce4d50cd46c6e88e60378c05c9047e08ca", size = 871227, upload-time = "2026-04-03T20:54:51.035Z" }, + { url = "https://files.pythonhosted.org/packages/a1/af/cb16bd5dc61621e27df919a4449bbb7e5a1034c34d307e0a706e9cc0f3e3/regex-2026.4.4-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6a50ab11b7779b849472337191f3a043e27e17f71555f98d0092fa6d73364520", size = 917435, upload-time = "2026-04-03T20:54:52.994Z" }, + { url = "https://files.pythonhosted.org/packages/5c/71/8b260897f22996b666edd9402861668f45a2ca259f665ac029e6104a2d7d/regex-2026.4.4-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0734f63afe785138549fbe822a8cfeaccd1bae814c5057cc0ed5b9f2de4fc883", size = 816358, upload-time = "2026-04-03T20:54:54.884Z" }, + { url = "https://files.pythonhosted.org/packages/1c/60/775f7f72a510ef238254906c2f3d737fc80b16ca85f07d20e318d2eea894/regex-2026.4.4-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4ee50606cb1967db7e523224e05f32089101945f859928e65657a2cbb3d278b", size = 785549, upload-time = "2026-04-03T20:54:57.01Z" }, + { url = "https://files.pythonhosted.org/packages/58/42/34d289b3627c03cf381e44da534a0021664188fa49ba41513da0b4ec6776/regex-2026.4.4-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:6c1818f37be3ca02dcb76d63f2c7aaba4b0dc171b579796c6fbe00148dfec6b1", size = 801364, upload-time = "2026-04-03T20:54:58.981Z" }, + { url = "https://files.pythonhosted.org/packages/fc/20/f6ecf319b382a8f1ab529e898b222c3f30600fcede7834733c26279e7465/regex-2026.4.4-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:f5bfc2741d150d0be3e4a0401a5c22b06e60acb9aa4daa46d9e79a6dcd0f135b", size = 866221, upload-time = "2026-04-03T20:55:00.88Z" }, + { url = "https://files.pythonhosted.org/packages/92/6a/9f16d3609d549bd96d7a0b2aee1625d7512ba6a03efc01652149ef88e74d/regex-2026.4.4-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:504ffa8a03609a087cad81277a629b6ce884b51a24bd388a7980ad61748618ff", size = 772530, upload-time = "2026-04-03T20:55:03.213Z" }, + { url = "https://files.pythonhosted.org/packages/fa/f6/aa9768bc96a4c361ac96419fbaf2dcdc33970bb813df3ba9b09d5d7b6d96/regex-2026.4.4-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:70aadc6ff12e4b444586e57fc30771f86253f9f0045b29016b9605b4be5f7dfb", size = 856989, upload-time = "2026-04-03T20:55:05.087Z" }, + { url = "https://files.pythonhosted.org/packages/4d/b4/c671db3556be2473ae3e4bb7a297c518d281452871501221251ea4ecba57/regex-2026.4.4-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f4f83781191007b6ef43b03debc35435f10cad9b96e16d147efe84a1d48bdde4", size = 803241, upload-time = "2026-04-03T20:55:07.162Z" }, + { url = "https://files.pythonhosted.org/packages/2a/5c/83e3b1d89fa4f6e5a1bc97b4abd4a9a97b3c1ac7854164f694f5f0ba98a0/regex-2026.4.4-cp313-cp313t-win32.whl", hash = "sha256:e014a797de43d1847df957c0a2a8e861d1c17547ee08467d1db2c370b7568baa", size = 269921, upload-time = "2026-04-03T20:55:09.62Z" }, + { url = "https://files.pythonhosted.org/packages/28/07/077c387121f42cdb4d92b1301133c0d93b5709d096d1669ab847dda9fe2e/regex-2026.4.4-cp313-cp313t-win_amd64.whl", hash = "sha256:b15b88b0d52b179712632832c1d6e58e5774f93717849a41096880442da41ab0", size = 281240, upload-time = "2026-04-03T20:55:11.521Z" }, + { url = "https://files.pythonhosted.org/packages/9d/22/ead4a4abc7c59a4d882662aa292ca02c8b617f30b6e163bc1728879e9353/regex-2026.4.4-cp313-cp313t-win_arm64.whl", hash = "sha256:586b89cdadf7d67bf86ae3342a4dcd2b8d70a832d90c18a0ae955105caf34dbe", size = 272440, upload-time = "2026-04-03T20:55:13.365Z" }, + { url = "https://files.pythonhosted.org/packages/f0/f5/ed97c2dc47b5fbd4b73c0d7d75f9ebc8eca139f2bbef476bba35f28c0a77/regex-2026.4.4-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:2da82d643fa698e5e5210e54af90181603d5853cf469f5eedf9bfc8f59b4b8c7", size = 490343, upload-time = "2026-04-03T20:55:15.241Z" }, + { url = "https://files.pythonhosted.org/packages/80/e9/de4828a7385ec166d673a5790ad06ac48cdaa98bc0960108dd4b9cc1aef7/regex-2026.4.4-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:54a1189ad9d9357760557c91103d5e421f0a2dabe68a5cdf9103d0dcf4e00752", size = 291909, upload-time = "2026-04-03T20:55:17.558Z" }, + { url = "https://files.pythonhosted.org/packages/b4/d6/5cfbfc97f3201a4d24b596a77957e092030dcc4205894bc035cedcfce62f/regex-2026.4.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:76d67d5afb1fe402d10a6403bae668d000441e2ab115191a804287d53b772951", size = 289692, upload-time = "2026-04-03T20:55:20.561Z" }, + { url = "https://files.pythonhosted.org/packages/8e/ac/f2212d9fd56fe897e36d0110ba30ba2d247bd6410c5bd98499c7e5a1e1f2/regex-2026.4.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e7cd3e4ee8d80447a83bbc9ab0c8459781fa77087f856c3e740d7763be0df27f", size = 796979, upload-time = "2026-04-03T20:55:22.56Z" }, + { url = "https://files.pythonhosted.org/packages/c9/e3/a016c12675fbac988a60c7e1c16e67823ff0bc016beb27bd7a001dbdabc6/regex-2026.4.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e19e18c568d2866d8b6a6dfad823db86193503f90823a8f66689315ba28fbe8", size = 866744, upload-time = "2026-04-03T20:55:24.646Z" }, + { url = "https://files.pythonhosted.org/packages/af/a4/0b90ca4cf17adc3cb43de80ec71018c37c88ad64987e8d0d481a95ca60b5/regex-2026.4.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7698a6f38730fd1385d390d1ed07bb13dce39aa616aca6a6d89bea178464b9a4", size = 911613, upload-time = "2026-04-03T20:55:27.033Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3b/2b3dac0b82d41ab43aa87c6ecde63d71189d03fe8854b8ca455a315edac3/regex-2026.4.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:173a66f3651cdb761018078e2d9487f4cf971232c990035ec0eb1cdc6bf929a9", size = 800551, upload-time = "2026-04-03T20:55:29.532Z" }, + { url = "https://files.pythonhosted.org/packages/25/fe/5365eb7aa0e753c4b5957815c321519ecab033c279c60e1b1ae2367fa810/regex-2026.4.4-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fa7922bbb2cc84fa062d37723f199d4c0cd200245ce269c05db82d904db66b83", size = 776911, upload-time = "2026-04-03T20:55:31.526Z" }, + { url = "https://files.pythonhosted.org/packages/aa/b3/7fb0072156bba065e3b778a7bc7b0a6328212be5dd6a86fd207e0c4f2dab/regex-2026.4.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:59f67cd0a0acaf0e564c20bbd7f767286f23e91e2572c5703bf3e56ea7557edb", size = 785751, upload-time = "2026-04-03T20:55:33.797Z" }, + { url = "https://files.pythonhosted.org/packages/02/1a/9f83677eb699273e56e858f7bd95acdbee376d42f59e8bfca2fd80d79df3/regex-2026.4.4-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:475e50f3f73f73614f7cba5524d6de49dee269df00272a1b85e3d19f6d498465", size = 860484, upload-time = "2026-04-03T20:55:35.745Z" }, + { url = "https://files.pythonhosted.org/packages/3b/7a/93937507b61cfcff8b4c5857f1b452852b09f741daa9acae15c971d8554e/regex-2026.4.4-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:a1c0c7d67b64d85ac2e1879923bad2f08a08f3004055f2f406ef73c850114bd4", size = 765939, upload-time = "2026-04-03T20:55:37.972Z" }, + { url = "https://files.pythonhosted.org/packages/86/ea/81a7f968a351c6552b1670ead861e2a385be730ee28402233020c67f9e0f/regex-2026.4.4-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:1371c2ccbb744d66ee63631cc9ca12aa233d5749972626b68fe1a649dd98e566", size = 851417, upload-time = "2026-04-03T20:55:39.92Z" }, + { url = "https://files.pythonhosted.org/packages/4c/7e/323c18ce4b5b8f44517a36342961a0306e931e499febbd876bb149d900f0/regex-2026.4.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:59968142787042db793348a3f5b918cf24ced1f23247328530e063f89c128a95", size = 789056, upload-time = "2026-04-03T20:55:42.303Z" }, + { url = "https://files.pythonhosted.org/packages/c0/af/e7510f9b11b1913b0cd44eddb784b2d650b2af6515bfce4cffcc5bfd1d38/regex-2026.4.4-cp314-cp314-win32.whl", hash = "sha256:59efe72d37fd5a91e373e5146f187f921f365f4abc1249a5ab446a60f30dd5f8", size = 272130, upload-time = "2026-04-03T20:55:44.995Z" }, + { url = "https://files.pythonhosted.org/packages/9a/51/57dae534c915e2d3a21490e88836fa2ae79dde3b66255ecc0c0a155d2c10/regex-2026.4.4-cp314-cp314-win_amd64.whl", hash = "sha256:e0aab3ff447845049d676827d2ff714aab4f73f340e155b7de7458cf53baa5a4", size = 280992, upload-time = "2026-04-03T20:55:47.316Z" }, + { url = "https://files.pythonhosted.org/packages/0a/5e/abaf9f4c3792e34edb1434f06717fae2b07888d85cb5cec29f9204931bf8/regex-2026.4.4-cp314-cp314-win_arm64.whl", hash = "sha256:a7a5bb6aa0cf62208bb4fa079b0c756734f8ad0e333b425732e8609bd51ee22f", size = 273563, upload-time = "2026-04-03T20:55:49.273Z" }, + { url = "https://files.pythonhosted.org/packages/ff/06/35da85f9f217b9538b99cbb170738993bcc3b23784322decb77619f11502/regex-2026.4.4-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:97850d0638391bdc7d35dc1c1039974dcb921eaafa8cc935ae4d7f272b1d60b3", size = 494191, upload-time = "2026-04-03T20:55:51.258Z" }, + { url = "https://files.pythonhosted.org/packages/54/5b/1bc35f479eef8285c4baf88d8c002023efdeebb7b44a8735b36195486ae7/regex-2026.4.4-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:ee7337f88f2a580679f7bbfe69dc86c043954f9f9c541012f49abc554a962f2e", size = 293877, upload-time = "2026-04-03T20:55:53.214Z" }, + { url = "https://files.pythonhosted.org/packages/39/5b/f53b9ad17480b3ddd14c90da04bfb55ac6894b129e5dea87bcaf7d00e336/regex-2026.4.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7429f4e6192c11d659900c0648ba8776243bf396ab95558b8c51a345afeddde6", size = 292410, upload-time = "2026-04-03T20:55:55.736Z" }, + { url = "https://files.pythonhosted.org/packages/bb/56/52377f59f60a7c51aa4161eecf0b6032c20b461805aca051250da435ffc9/regex-2026.4.4-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dc4f10fbd5dd13dcf4265b4cc07d69ca70280742870c97ae10093e3d66000359", size = 811831, upload-time = "2026-04-03T20:55:57.802Z" }, + { url = "https://files.pythonhosted.org/packages/dd/63/8026310bf066f702a9c361f83a8c9658f3fe4edb349f9c1e5d5273b7c40c/regex-2026.4.4-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a152560af4f9742b96f3827090f866eeec5becd4765c8e0d3473d9d280e76a5a", size = 871199, upload-time = "2026-04-03T20:56:00.333Z" }, + { url = "https://files.pythonhosted.org/packages/20/9f/a514bbb00a466dbb506d43f187a04047f7be1505f10a9a15615ead5080ee/regex-2026.4.4-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:54170b3e95339f415d54651f97df3bff7434a663912f9358237941bbf9143f55", size = 917649, upload-time = "2026-04-03T20:56:02.445Z" }, + { url = "https://files.pythonhosted.org/packages/cb/6b/8399f68dd41a2030218839b9b18360d79b86d22b9fab5ef477c7f23ca67c/regex-2026.4.4-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:07f190d65f5a72dcb9cf7106bfc3d21e7a49dd2879eda2207b683f32165e4d99", size = 816388, upload-time = "2026-04-03T20:56:04.595Z" }, + { url = "https://files.pythonhosted.org/packages/1e/9c/103963f47c24339a483b05edd568594c2be486188f688c0170fd504b2948/regex-2026.4.4-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9a2741ce5a29d3c84b0b94261ba630ab459a1b847a0d6beca7d62d188175c790", size = 785746, upload-time = "2026-04-03T20:56:07.13Z" }, + { url = "https://files.pythonhosted.org/packages/fa/ee/7f6054c0dec0cee3463c304405e4ff42e27cff05bf36fcb34be549ab17bd/regex-2026.4.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:b26c30df3a28fd9793113dac7385a4deb7294a06c0f760dd2b008bd49a9139bc", size = 801483, upload-time = "2026-04-03T20:56:09.365Z" }, + { url = "https://files.pythonhosted.org/packages/30/c2/51d3d941cf6070dc00c3338ecf138615fc3cce0421c3df6abe97a08af61a/regex-2026.4.4-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:421439d1bee44b19f4583ccf42670ca464ffb90e9fdc38d37f39d1ddd1e44f1f", size = 866331, upload-time = "2026-04-03T20:56:12.039Z" }, + { url = "https://files.pythonhosted.org/packages/16/e8/76d50dcc122ac33927d939f350eebcfe3dbcbda96913e03433fc36de5e63/regex-2026.4.4-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:b40379b53ecbc747fd9bdf4a0ea14eb8188ca1bd0f54f78893a39024b28f4863", size = 772673, upload-time = "2026-04-03T20:56:14.558Z" }, + { url = "https://files.pythonhosted.org/packages/a5/6e/5f6bf75e20ea6873d05ba4ec78378c375cbe08cdec571c83fbb01606e563/regex-2026.4.4-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:08c55c13d2eef54f73eeadc33146fb0baaa49e7335eb1aff6ae1324bf0ddbe4a", size = 857146, upload-time = "2026-04-03T20:56:16.663Z" }, + { url = "https://files.pythonhosted.org/packages/0b/33/3c76d9962949e487ebba353a18e89399f292287204ac8f2f4cfc3a51c233/regex-2026.4.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9776b85f510062f5a75ef112afe5f494ef1635607bf1cc220c1391e9ac2f5e81", size = 803463, upload-time = "2026-04-03T20:56:18.923Z" }, + { url = "https://files.pythonhosted.org/packages/19/eb/ef32dcd2cb69b69bc0c3e55205bce94a7def48d495358946bc42186dcccc/regex-2026.4.4-cp314-cp314t-win32.whl", hash = "sha256:385edaebde5db5be103577afc8699fea73a0e36a734ba24870be7ffa61119d74", size = 275709, upload-time = "2026-04-03T20:56:20.996Z" }, + { url = "https://files.pythonhosted.org/packages/a0/86/c291bf740945acbf35ed7dbebf8e2eea2f3f78041f6bd7cdab80cb274dc0/regex-2026.4.4-cp314-cp314t-win_amd64.whl", hash = "sha256:5d354b18839328927832e2fa5f7c95b7a3ccc39e7a681529e1685898e6436d45", size = 285622, upload-time = "2026-04-03T20:56:23.641Z" }, + { url = "https://files.pythonhosted.org/packages/d5/e7/ec846d560ae6a597115153c02ca6138a7877a1748b2072d9521c10a93e58/regex-2026.4.4-cp314-cp314t-win_arm64.whl", hash = "sha256:af0384cb01a33600c49505c27c6c57ab0b27bf84a74e28524c92ca897ebdac9d", size = 275773, upload-time = "2026-04-03T20:56:26.07Z" }, +] + +[[package]] +name = "requests" +version = "2.33.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5f/a4/98b9c7c6428a668bf7e42ebb7c79d576a1c3c1e3ae2d47e674b468388871/requests-2.33.1.tar.gz", hash = "sha256:18817f8c57c6263968bc123d237e3b8b08ac046f5456bd1e307ee8f4250d3517", size = 134120, upload-time = "2026-03-30T16:09:15.531Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d7/8e/7540e8a2036f79a125c1d2ebadf69ed7901608859186c856fa0388ef4197/requests-2.33.1-py3-none-any.whl", hash = "sha256:4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a", size = 64947, upload-time = "2026-03-30T16:09:13.83Z" }, +] + +[[package]] +name = "rich" +version = "14.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/c6/f3b320c27991c46f43ee9d856302c70dc2d0fb2dba4842ff739d5f46b393/rich-14.3.3.tar.gz", hash = "sha256:b8daa0b9e4eef54dd8cf7c86c03713f53241884e814f4e2f5fb342fe520f639b", size = 230582, upload-time = "2026-02-19T17:23:12.474Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/14/25/b208c5683343959b670dc001595f2f3737e051da617f66c31f7c4fa93abc/rich-14.3.3-py3-none-any.whl", hash = "sha256:793431c1f8619afa7d3b52b2cdec859562b950ea0d4b6b505397612db8d5362d", size = 310458, upload-time = "2026-02-19T17:23:13.732Z" }, +] + +[[package]] +name = "rpds-py" +version = "0.30.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/20/af/3f2f423103f1113b36230496629986e0ef7e199d2aa8392452b484b38ced/rpds_py-0.30.0.tar.gz", hash = "sha256:dd8ff7cf90014af0c0f787eea34794ebf6415242ee1d6fa91eaba725cc441e84", size = 69469, upload-time = "2025-11-30T20:24:38.837Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4d/6e/f964e88b3d2abee2a82c1ac8366da848fce1c6d834dc2132c3fda3970290/rpds_py-0.30.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:a2bffea6a4ca9f01b3f8e548302470306689684e61602aa3d141e34da06cf425", size = 370157, upload-time = "2025-11-30T20:21:53.789Z" }, + { url = "https://files.pythonhosted.org/packages/94/ba/24e5ebb7c1c82e74c4e4f33b2112a5573ddc703915b13a073737b59b86e0/rpds_py-0.30.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:dc4f992dfe1e2bc3ebc7444f6c7051b4bc13cd8e33e43511e8ffd13bf407010d", size = 359676, upload-time = "2025-11-30T20:21:55.475Z" }, + { url = "https://files.pythonhosted.org/packages/84/86/04dbba1b087227747d64d80c3b74df946b986c57af0a9f0c98726d4d7a3b/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:422c3cb9856d80b09d30d2eb255d0754b23e090034e1deb4083f8004bd0761e4", size = 389938, upload-time = "2025-11-30T20:21:57.079Z" }, + { url = "https://files.pythonhosted.org/packages/42/bb/1463f0b1722b7f45431bdd468301991d1328b16cffe0b1c2918eba2c4eee/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:07ae8a593e1c3c6b82ca3292efbe73c30b61332fd612e05abee07c79359f292f", size = 402932, upload-time = "2025-11-30T20:21:58.47Z" }, + { url = "https://files.pythonhosted.org/packages/99/ee/2520700a5c1f2d76631f948b0736cdf9b0acb25abd0ca8e889b5c62ac2e3/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:12f90dd7557b6bd57f40abe7747e81e0c0b119bef015ea7726e69fe550e394a4", size = 525830, upload-time = "2025-11-30T20:21:59.699Z" }, + { url = "https://files.pythonhosted.org/packages/e0/ad/bd0331f740f5705cc555a5e17fdf334671262160270962e69a2bdef3bf76/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:99b47d6ad9a6da00bec6aabe5a6279ecd3c06a329d4aa4771034a21e335c3a97", size = 412033, upload-time = "2025-11-30T20:22:00.991Z" }, + { url = "https://files.pythonhosted.org/packages/f8/1e/372195d326549bb51f0ba0f2ecb9874579906b97e08880e7a65c3bef1a99/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:33f559f3104504506a44bb666b93a33f5d33133765b0c216a5bf2f1e1503af89", size = 390828, upload-time = "2025-11-30T20:22:02.723Z" }, + { url = "https://files.pythonhosted.org/packages/ab/2b/d88bb33294e3e0c76bc8f351a3721212713629ffca1700fa94979cb3eae8/rpds_py-0.30.0-cp311-cp311-manylinux_2_31_riscv64.whl", hash = "sha256:946fe926af6e44f3697abbc305ea168c2c31d3e3ef1058cf68f379bf0335a78d", size = 404683, upload-time = "2025-11-30T20:22:04.367Z" }, + { url = "https://files.pythonhosted.org/packages/50/32/c759a8d42bcb5289c1fac697cd92f6fe01a018dd937e62ae77e0e7f15702/rpds_py-0.30.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:495aeca4b93d465efde585977365187149e75383ad2684f81519f504f5c13038", size = 421583, upload-time = "2025-11-30T20:22:05.814Z" }, + { url = "https://files.pythonhosted.org/packages/2b/81/e729761dbd55ddf5d84ec4ff1f47857f4374b0f19bdabfcf929164da3e24/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d9a0ca5da0386dee0655b4ccdf46119df60e0f10da268d04fe7cc87886872ba7", size = 572496, upload-time = "2025-11-30T20:22:07.713Z" }, + { url = "https://files.pythonhosted.org/packages/14/f6/69066a924c3557c9c30baa6ec3a0aa07526305684c6f86c696b08860726c/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:8d6d1cc13664ec13c1b84241204ff3b12f9bb82464b8ad6e7a5d3486975c2eed", size = 598669, upload-time = "2025-11-30T20:22:09.312Z" }, + { url = "https://files.pythonhosted.org/packages/5f/48/905896b1eb8a05630d20333d1d8ffd162394127b74ce0b0784ae04498d32/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3896fa1be39912cf0757753826bc8bdc8ca331a28a7c4ae46b7a21280b06bb85", size = 561011, upload-time = "2025-11-30T20:22:11.309Z" }, + { url = "https://files.pythonhosted.org/packages/22/16/cd3027c7e279d22e5eb431dd3c0fbc677bed58797fe7581e148f3f68818b/rpds_py-0.30.0-cp311-cp311-win32.whl", hash = "sha256:55f66022632205940f1827effeff17c4fa7ae1953d2b74a8581baaefb7d16f8c", size = 221406, upload-time = "2025-11-30T20:22:13.101Z" }, + { url = "https://files.pythonhosted.org/packages/fa/5b/e7b7aa136f28462b344e652ee010d4de26ee9fd16f1bfd5811f5153ccf89/rpds_py-0.30.0-cp311-cp311-win_amd64.whl", hash = "sha256:a51033ff701fca756439d641c0ad09a41d9242fa69121c7d8769604a0a629825", size = 236024, upload-time = "2025-11-30T20:22:14.853Z" }, + { url = "https://files.pythonhosted.org/packages/14/a6/364bba985e4c13658edb156640608f2c9e1d3ea3c81b27aa9d889fff0e31/rpds_py-0.30.0-cp311-cp311-win_arm64.whl", hash = "sha256:47b0ef6231c58f506ef0b74d44e330405caa8428e770fec25329ed2cb971a229", size = 229069, upload-time = "2025-11-30T20:22:16.577Z" }, + { url = "https://files.pythonhosted.org/packages/03/e7/98a2f4ac921d82f33e03f3835f5bf3a4a40aa1bfdc57975e74a97b2b4bdd/rpds_py-0.30.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a161f20d9a43006833cd7068375a94d035714d73a172b681d8881820600abfad", size = 375086, upload-time = "2025-11-30T20:22:17.93Z" }, + { url = "https://files.pythonhosted.org/packages/4d/a1/bca7fd3d452b272e13335db8d6b0b3ecde0f90ad6f16f3328c6fb150c889/rpds_py-0.30.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6abc8880d9d036ecaafe709079969f56e876fcf107f7a8e9920ba6d5a3878d05", size = 359053, upload-time = "2025-11-30T20:22:19.297Z" }, + { url = "https://files.pythonhosted.org/packages/65/1c/ae157e83a6357eceff62ba7e52113e3ec4834a84cfe07fa4b0757a7d105f/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ca28829ae5f5d569bb62a79512c842a03a12576375d5ece7d2cadf8abe96ec28", size = 390763, upload-time = "2025-11-30T20:22:21.661Z" }, + { url = "https://files.pythonhosted.org/packages/d4/36/eb2eb8515e2ad24c0bd43c3ee9cd74c33f7ca6430755ccdb240fd3144c44/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a1010ed9524c73b94d15919ca4d41d8780980e1765babf85f9a2f90d247153dd", size = 408951, upload-time = "2025-11-30T20:22:23.408Z" }, + { url = "https://files.pythonhosted.org/packages/d6/65/ad8dc1784a331fabbd740ef6f71ce2198c7ed0890dab595adb9ea2d775a1/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f8d1736cfb49381ba528cd5baa46f82fdc65c06e843dab24dd70b63d09121b3f", size = 514622, upload-time = "2025-11-30T20:22:25.16Z" }, + { url = "https://files.pythonhosted.org/packages/63/8e/0cfa7ae158e15e143fe03993b5bcd743a59f541f5952e1546b1ac1b5fd45/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d948b135c4693daff7bc2dcfc4ec57237a29bd37e60c2fabf5aff2bbacf3e2f1", size = 414492, upload-time = "2025-11-30T20:22:26.505Z" }, + { url = "https://files.pythonhosted.org/packages/60/1b/6f8f29f3f995c7ffdde46a626ddccd7c63aefc0efae881dc13b6e5d5bb16/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47f236970bccb2233267d89173d3ad2703cd36a0e2a6e92d0560d333871a3d23", size = 394080, upload-time = "2025-11-30T20:22:27.934Z" }, + { url = "https://files.pythonhosted.org/packages/6d/d5/a266341051a7a3ca2f4b750a3aa4abc986378431fc2da508c5034d081b70/rpds_py-0.30.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:2e6ecb5a5bcacf59c3f912155044479af1d0b6681280048b338b28e364aca1f6", size = 408680, upload-time = "2025-11-30T20:22:29.341Z" }, + { url = "https://files.pythonhosted.org/packages/10/3b/71b725851df9ab7a7a4e33cf36d241933da66040d195a84781f49c50490c/rpds_py-0.30.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a8fa71a2e078c527c3e9dc9fc5a98c9db40bcc8a92b4e8858e36d329f8684b51", size = 423589, upload-time = "2025-11-30T20:22:31.469Z" }, + { url = "https://files.pythonhosted.org/packages/00/2b/e59e58c544dc9bd8bd8384ecdb8ea91f6727f0e37a7131baeff8d6f51661/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:73c67f2db7bc334e518d097c6d1e6fed021bbc9b7d678d6cc433478365d1d5f5", size = 573289, upload-time = "2025-11-30T20:22:32.997Z" }, + { url = "https://files.pythonhosted.org/packages/da/3e/a18e6f5b460893172a7d6a680e86d3b6bc87a54c1f0b03446a3c8c7b588f/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:5ba103fb455be00f3b1c2076c9d4264bfcb037c976167a6047ed82f23153f02e", size = 599737, upload-time = "2025-11-30T20:22:34.419Z" }, + { url = "https://files.pythonhosted.org/packages/5c/e2/714694e4b87b85a18e2c243614974413c60aa107fd815b8cbc42b873d1d7/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7cee9c752c0364588353e627da8a7e808a66873672bcb5f52890c33fd965b394", size = 563120, upload-time = "2025-11-30T20:22:35.903Z" }, + { url = "https://files.pythonhosted.org/packages/6f/ab/d5d5e3bcedb0a77f4f613706b750e50a5a3ba1c15ccd3665ecc636c968fd/rpds_py-0.30.0-cp312-cp312-win32.whl", hash = "sha256:1ab5b83dbcf55acc8b08fc62b796ef672c457b17dbd7820a11d6c52c06839bdf", size = 223782, upload-time = "2025-11-30T20:22:37.271Z" }, + { url = "https://files.pythonhosted.org/packages/39/3b/f786af9957306fdc38a74cef405b7b93180f481fb48453a114bb6465744a/rpds_py-0.30.0-cp312-cp312-win_amd64.whl", hash = "sha256:a090322ca841abd453d43456ac34db46e8b05fd9b3b4ac0c78bcde8b089f959b", size = 240463, upload-time = "2025-11-30T20:22:39.021Z" }, + { url = "https://files.pythonhosted.org/packages/f3/d2/b91dc748126c1559042cfe41990deb92c4ee3e2b415f6b5234969ffaf0cc/rpds_py-0.30.0-cp312-cp312-win_arm64.whl", hash = "sha256:669b1805bd639dd2989b281be2cfd951c6121b65e729d9b843e9639ef1fd555e", size = 230868, upload-time = "2025-11-30T20:22:40.493Z" }, + { url = "https://files.pythonhosted.org/packages/ed/dc/d61221eb88ff410de3c49143407f6f3147acf2538c86f2ab7ce65ae7d5f9/rpds_py-0.30.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:f83424d738204d9770830d35290ff3273fbb02b41f919870479fab14b9d303b2", size = 374887, upload-time = "2025-11-30T20:22:41.812Z" }, + { url = "https://files.pythonhosted.org/packages/fd/32/55fb50ae104061dbc564ef15cc43c013dc4a9f4527a1f4d99baddf56fe5f/rpds_py-0.30.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e7536cd91353c5273434b4e003cbda89034d67e7710eab8761fd918ec6c69cf8", size = 358904, upload-time = "2025-11-30T20:22:43.479Z" }, + { url = "https://files.pythonhosted.org/packages/58/70/faed8186300e3b9bdd138d0273109784eea2396c68458ed580f885dfe7ad/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2771c6c15973347f50fece41fc447c054b7ac2ae0502388ce3b6738cd366e3d4", size = 389945, upload-time = "2025-11-30T20:22:44.819Z" }, + { url = "https://files.pythonhosted.org/packages/bd/a8/073cac3ed2c6387df38f71296d002ab43496a96b92c823e76f46b8af0543/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0a59119fc6e3f460315fe9d08149f8102aa322299deaa5cab5b40092345c2136", size = 407783, upload-time = "2025-11-30T20:22:46.103Z" }, + { url = "https://files.pythonhosted.org/packages/77/57/5999eb8c58671f1c11eba084115e77a8899d6e694d2a18f69f0ba471ec8b/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:76fec018282b4ead0364022e3c54b60bf368b9d926877957a8624b58419169b7", size = 515021, upload-time = "2025-11-30T20:22:47.458Z" }, + { url = "https://files.pythonhosted.org/packages/e0/af/5ab4833eadc36c0a8ed2bc5c0de0493c04f6c06de223170bd0798ff98ced/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:692bef75a5525db97318e8cd061542b5a79812d711ea03dbc1f6f8dbb0c5f0d2", size = 414589, upload-time = "2025-11-30T20:22:48.872Z" }, + { url = "https://files.pythonhosted.org/packages/b7/de/f7192e12b21b9e9a68a6d0f249b4af3fdcdff8418be0767a627564afa1f1/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9027da1ce107104c50c81383cae773ef5c24d296dd11c99e2629dbd7967a20c6", size = 394025, upload-time = "2025-11-30T20:22:50.196Z" }, + { url = "https://files.pythonhosted.org/packages/91/c4/fc70cd0249496493500e7cc2de87504f5aa6509de1e88623431fec76d4b6/rpds_py-0.30.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:9cf69cdda1f5968a30a359aba2f7f9aa648a9ce4b580d6826437f2b291cfc86e", size = 408895, upload-time = "2025-11-30T20:22:51.87Z" }, + { url = "https://files.pythonhosted.org/packages/58/95/d9275b05ab96556fefff73a385813eb66032e4c99f411d0795372d9abcea/rpds_py-0.30.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a4796a717bf12b9da9d3ad002519a86063dcac8988b030e405704ef7d74d2d9d", size = 422799, upload-time = "2025-11-30T20:22:53.341Z" }, + { url = "https://files.pythonhosted.org/packages/06/c1/3088fc04b6624eb12a57eb814f0d4997a44b0d208d6cace713033ff1a6ba/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5d4c2aa7c50ad4728a094ebd5eb46c452e9cb7edbfdb18f9e1221f597a73e1e7", size = 572731, upload-time = "2025-11-30T20:22:54.778Z" }, + { url = "https://files.pythonhosted.org/packages/d8/42/c612a833183b39774e8ac8fecae81263a68b9583ee343db33ab571a7ce55/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ba81a9203d07805435eb06f536d95a266c21e5b2dfbf6517748ca40c98d19e31", size = 599027, upload-time = "2025-11-30T20:22:56.212Z" }, + { url = "https://files.pythonhosted.org/packages/5f/60/525a50f45b01d70005403ae0e25f43c0384369ad24ffe46e8d9068b50086/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:945dccface01af02675628334f7cf49c2af4c1c904748efc5cf7bbdf0b579f95", size = 563020, upload-time = "2025-11-30T20:22:58.2Z" }, + { url = "https://files.pythonhosted.org/packages/0b/5d/47c4655e9bcd5ca907148535c10e7d489044243cc9941c16ed7cd53be91d/rpds_py-0.30.0-cp313-cp313-win32.whl", hash = "sha256:b40fb160a2db369a194cb27943582b38f79fc4887291417685f3ad693c5a1d5d", size = 223139, upload-time = "2025-11-30T20:23:00.209Z" }, + { url = "https://files.pythonhosted.org/packages/f2/e1/485132437d20aa4d3e1d8b3fb5a5e65aa8139f1e097080c2a8443201742c/rpds_py-0.30.0-cp313-cp313-win_amd64.whl", hash = "sha256:806f36b1b605e2d6a72716f321f20036b9489d29c51c91f4dd29a3e3afb73b15", size = 240224, upload-time = "2025-11-30T20:23:02.008Z" }, + { url = "https://files.pythonhosted.org/packages/24/95/ffd128ed1146a153d928617b0ef673960130be0009c77d8fbf0abe306713/rpds_py-0.30.0-cp313-cp313-win_arm64.whl", hash = "sha256:d96c2086587c7c30d44f31f42eae4eac89b60dabbac18c7669be3700f13c3ce1", size = 230645, upload-time = "2025-11-30T20:23:03.43Z" }, + { url = "https://files.pythonhosted.org/packages/ff/1b/b10de890a0def2a319a2626334a7f0ae388215eb60914dbac8a3bae54435/rpds_py-0.30.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:eb0b93f2e5c2189ee831ee43f156ed34e2a89a78a66b98cadad955972548be5a", size = 364443, upload-time = "2025-11-30T20:23:04.878Z" }, + { url = "https://files.pythonhosted.org/packages/0d/bf/27e39f5971dc4f305a4fb9c672ca06f290f7c4e261c568f3dea16a410d47/rpds_py-0.30.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:922e10f31f303c7c920da8981051ff6d8c1a56207dbdf330d9047f6d30b70e5e", size = 353375, upload-time = "2025-11-30T20:23:06.342Z" }, + { url = "https://files.pythonhosted.org/packages/40/58/442ada3bba6e8e6615fc00483135c14a7538d2ffac30e2d933ccf6852232/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cdc62c8286ba9bf7f47befdcea13ea0e26bf294bda99758fd90535cbaf408000", size = 383850, upload-time = "2025-11-30T20:23:07.825Z" }, + { url = "https://files.pythonhosted.org/packages/14/14/f59b0127409a33c6ef6f5c1ebd5ad8e32d7861c9c7adfa9a624fc3889f6c/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:47f9a91efc418b54fb8190a6b4aa7813a23fb79c51f4bb84e418f5476c38b8db", size = 392812, upload-time = "2025-11-30T20:23:09.228Z" }, + { url = "https://files.pythonhosted.org/packages/b3/66/e0be3e162ac299b3a22527e8913767d869e6cc75c46bd844aa43fb81ab62/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1f3587eb9b17f3789ad50824084fa6f81921bbf9a795826570bda82cb3ed91f2", size = 517841, upload-time = "2025-11-30T20:23:11.186Z" }, + { url = "https://files.pythonhosted.org/packages/3d/55/fa3b9cf31d0c963ecf1ba777f7cf4b2a2c976795ac430d24a1f43d25a6ba/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:39c02563fc592411c2c61d26b6c5fe1e51eaa44a75aa2c8735ca88b0d9599daa", size = 408149, upload-time = "2025-11-30T20:23:12.864Z" }, + { url = "https://files.pythonhosted.org/packages/60/ca/780cf3b1a32b18c0f05c441958d3758f02544f1d613abf9488cd78876378/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:51a1234d8febafdfd33a42d97da7a43f5dcb120c1060e352a3fbc0c6d36e2083", size = 383843, upload-time = "2025-11-30T20:23:14.638Z" }, + { url = "https://files.pythonhosted.org/packages/82/86/d5f2e04f2aa6247c613da0c1dd87fcd08fa17107e858193566048a1e2f0a/rpds_py-0.30.0-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:eb2c4071ab598733724c08221091e8d80e89064cd472819285a9ab0f24bcedb9", size = 396507, upload-time = "2025-11-30T20:23:16.105Z" }, + { url = "https://files.pythonhosted.org/packages/4b/9a/453255d2f769fe44e07ea9785c8347edaf867f7026872e76c1ad9f7bed92/rpds_py-0.30.0-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6bdfdb946967d816e6adf9a3d8201bfad269c67efe6cefd7093ef959683c8de0", size = 414949, upload-time = "2025-11-30T20:23:17.539Z" }, + { url = "https://files.pythonhosted.org/packages/a3/31/622a86cdc0c45d6df0e9ccb6becdba5074735e7033c20e401a6d9d0e2ca0/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c77afbd5f5250bf27bf516c7c4a016813eb2d3e116139aed0096940c5982da94", size = 565790, upload-time = "2025-11-30T20:23:19.029Z" }, + { url = "https://files.pythonhosted.org/packages/1c/5d/15bbf0fb4a3f58a3b1c67855ec1efcc4ceaef4e86644665fff03e1b66d8d/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:61046904275472a76c8c90c9ccee9013d70a6d0f73eecefd38c1ae7c39045a08", size = 590217, upload-time = "2025-11-30T20:23:20.885Z" }, + { url = "https://files.pythonhosted.org/packages/6d/61/21b8c41f68e60c8cc3b2e25644f0e3681926020f11d06ab0b78e3c6bbff1/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:4c5f36a861bc4b7da6516dbdf302c55313afa09b81931e8280361a4f6c9a2d27", size = 555806, upload-time = "2025-11-30T20:23:22.488Z" }, + { url = "https://files.pythonhosted.org/packages/f9/39/7e067bb06c31de48de3eb200f9fc7c58982a4d3db44b07e73963e10d3be9/rpds_py-0.30.0-cp313-cp313t-win32.whl", hash = "sha256:3d4a69de7a3e50ffc214ae16d79d8fbb0922972da0356dcf4d0fdca2878559c6", size = 211341, upload-time = "2025-11-30T20:23:24.449Z" }, + { url = "https://files.pythonhosted.org/packages/0a/4d/222ef0b46443cf4cf46764d9c630f3fe4abaa7245be9417e56e9f52b8f65/rpds_py-0.30.0-cp313-cp313t-win_amd64.whl", hash = "sha256:f14fc5df50a716f7ece6a80b6c78bb35ea2ca47c499e422aa4463455dd96d56d", size = 225768, upload-time = "2025-11-30T20:23:25.908Z" }, + { url = "https://files.pythonhosted.org/packages/86/81/dad16382ebbd3d0e0328776d8fd7ca94220e4fa0798d1dc5e7da48cb3201/rpds_py-0.30.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:68f19c879420aa08f61203801423f6cd5ac5f0ac4ac82a2368a9fcd6a9a075e0", size = 362099, upload-time = "2025-11-30T20:23:27.316Z" }, + { url = "https://files.pythonhosted.org/packages/2b/60/19f7884db5d5603edf3c6bce35408f45ad3e97e10007df0e17dd57af18f8/rpds_py-0.30.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ec7c4490c672c1a0389d319b3a9cfcd098dcdc4783991553c332a15acf7249be", size = 353192, upload-time = "2025-11-30T20:23:29.151Z" }, + { url = "https://files.pythonhosted.org/packages/bf/c4/76eb0e1e72d1a9c4703c69607cec123c29028bff28ce41588792417098ac/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f251c812357a3fed308d684a5079ddfb9d933860fc6de89f2b7ab00da481e65f", size = 384080, upload-time = "2025-11-30T20:23:30.785Z" }, + { url = "https://files.pythonhosted.org/packages/72/87/87ea665e92f3298d1b26d78814721dc39ed8d2c74b86e83348d6b48a6f31/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ac98b175585ecf4c0348fd7b29c3864bda53b805c773cbf7bfdaffc8070c976f", size = 394841, upload-time = "2025-11-30T20:23:32.209Z" }, + { url = "https://files.pythonhosted.org/packages/77/ad/7783a89ca0587c15dcbf139b4a8364a872a25f861bdb88ed99f9b0dec985/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3e62880792319dbeb7eb866547f2e35973289e7d5696c6e295476448f5b63c87", size = 516670, upload-time = "2025-11-30T20:23:33.742Z" }, + { url = "https://files.pythonhosted.org/packages/5b/3c/2882bdac942bd2172f3da574eab16f309ae10a3925644e969536553cb4ee/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4e7fc54e0900ab35d041b0601431b0a0eb495f0851a0639b6ef90f7741b39a18", size = 408005, upload-time = "2025-11-30T20:23:35.253Z" }, + { url = "https://files.pythonhosted.org/packages/ce/81/9a91c0111ce1758c92516a3e44776920b579d9a7c09b2b06b642d4de3f0f/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47e77dc9822d3ad616c3d5759ea5631a75e5809d5a28707744ef79d7a1bcfcad", size = 382112, upload-time = "2025-11-30T20:23:36.842Z" }, + { url = "https://files.pythonhosted.org/packages/cf/8e/1da49d4a107027e5fbc64daeab96a0706361a2918da10cb41769244b805d/rpds_py-0.30.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:b4dc1a6ff022ff85ecafef7979a2c6eb423430e05f1165d6688234e62ba99a07", size = 399049, upload-time = "2025-11-30T20:23:38.343Z" }, + { url = "https://files.pythonhosted.org/packages/df/5a/7ee239b1aa48a127570ec03becbb29c9d5a9eb092febbd1699d567cae859/rpds_py-0.30.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:4559c972db3a360808309e06a74628b95eaccbf961c335c8fe0d590cf587456f", size = 415661, upload-time = "2025-11-30T20:23:40.263Z" }, + { url = "https://files.pythonhosted.org/packages/70/ea/caa143cf6b772f823bc7929a45da1fa83569ee49b11d18d0ada7f5ee6fd6/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:0ed177ed9bded28f8deb6ab40c183cd1192aa0de40c12f38be4d59cd33cb5c65", size = 565606, upload-time = "2025-11-30T20:23:42.186Z" }, + { url = "https://files.pythonhosted.org/packages/64/91/ac20ba2d69303f961ad8cf55bf7dbdb4763f627291ba3d0d7d67333cced9/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:ad1fa8db769b76ea911cb4e10f049d80bf518c104f15b3edb2371cc65375c46f", size = 591126, upload-time = "2025-11-30T20:23:44.086Z" }, + { url = "https://files.pythonhosted.org/packages/21/20/7ff5f3c8b00c8a95f75985128c26ba44503fb35b8e0259d812766ea966c7/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:46e83c697b1f1c72b50e5ee5adb4353eef7406fb3f2043d64c33f20ad1c2fc53", size = 553371, upload-time = "2025-11-30T20:23:46.004Z" }, + { url = "https://files.pythonhosted.org/packages/72/c7/81dadd7b27c8ee391c132a6b192111ca58d866577ce2d9b0ca157552cce0/rpds_py-0.30.0-cp314-cp314-win32.whl", hash = "sha256:ee454b2a007d57363c2dfd5b6ca4a5d7e2c518938f8ed3b706e37e5d470801ed", size = 215298, upload-time = "2025-11-30T20:23:47.696Z" }, + { url = "https://files.pythonhosted.org/packages/3e/d2/1aaac33287e8cfb07aab2e6b8ac1deca62f6f65411344f1433c55e6f3eb8/rpds_py-0.30.0-cp314-cp314-win_amd64.whl", hash = "sha256:95f0802447ac2d10bcc69f6dc28fe95fdf17940367b21d34e34c737870758950", size = 228604, upload-time = "2025-11-30T20:23:49.501Z" }, + { url = "https://files.pythonhosted.org/packages/e8/95/ab005315818cc519ad074cb7784dae60d939163108bd2b394e60dc7b5461/rpds_py-0.30.0-cp314-cp314-win_arm64.whl", hash = "sha256:613aa4771c99f03346e54c3f038e4cc574ac09a3ddfb0e8878487335e96dead6", size = 222391, upload-time = "2025-11-30T20:23:50.96Z" }, + { url = "https://files.pythonhosted.org/packages/9e/68/154fe0194d83b973cdedcdcc88947a2752411165930182ae41d983dcefa6/rpds_py-0.30.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:7e6ecfcb62edfd632e56983964e6884851786443739dbfe3582947e87274f7cb", size = 364868, upload-time = "2025-11-30T20:23:52.494Z" }, + { url = "https://files.pythonhosted.org/packages/83/69/8bbc8b07ec854d92a8b75668c24d2abcb1719ebf890f5604c61c9369a16f/rpds_py-0.30.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a1d0bc22a7cdc173fedebb73ef81e07faef93692b8c1ad3733b67e31e1b6e1b8", size = 353747, upload-time = "2025-11-30T20:23:54.036Z" }, + { url = "https://files.pythonhosted.org/packages/ab/00/ba2e50183dbd9abcce9497fa5149c62b4ff3e22d338a30d690f9af970561/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0d08f00679177226c4cb8c5265012eea897c8ca3b93f429e546600c971bcbae7", size = 383795, upload-time = "2025-11-30T20:23:55.556Z" }, + { url = "https://files.pythonhosted.org/packages/05/6f/86f0272b84926bcb0e4c972262f54223e8ecc556b3224d281e6598fc9268/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5965af57d5848192c13534f90f9dd16464f3c37aaf166cc1da1cae1fd5a34898", size = 393330, upload-time = "2025-11-30T20:23:57.033Z" }, + { url = "https://files.pythonhosted.org/packages/cb/e9/0e02bb2e6dc63d212641da45df2b0bf29699d01715913e0d0f017ee29438/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9a4e86e34e9ab6b667c27f3211ca48f73dba7cd3d90f8d5b11be56e5dbc3fb4e", size = 518194, upload-time = "2025-11-30T20:23:58.637Z" }, + { url = "https://files.pythonhosted.org/packages/ee/ca/be7bca14cf21513bdf9c0606aba17d1f389ea2b6987035eb4f62bd923f25/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e5d3e6b26f2c785d65cc25ef1e5267ccbe1b069c5c21b8cc724efee290554419", size = 408340, upload-time = "2025-11-30T20:24:00.2Z" }, + { url = "https://files.pythonhosted.org/packages/c2/c7/736e00ebf39ed81d75544c0da6ef7b0998f8201b369acf842f9a90dc8fce/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:626a7433c34566535b6e56a1b39a7b17ba961e97ce3b80ec62e6f1312c025551", size = 383765, upload-time = "2025-11-30T20:24:01.759Z" }, + { url = "https://files.pythonhosted.org/packages/4a/3f/da50dfde9956aaf365c4adc9533b100008ed31aea635f2b8d7b627e25b49/rpds_py-0.30.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:acd7eb3f4471577b9b5a41baf02a978e8bdeb08b4b355273994f8b87032000a8", size = 396834, upload-time = "2025-11-30T20:24:03.687Z" }, + { url = "https://files.pythonhosted.org/packages/4e/00/34bcc2565b6020eab2623349efbdec810676ad571995911f1abdae62a3a0/rpds_py-0.30.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fe5fa731a1fa8a0a56b0977413f8cacac1768dad38d16b3a296712709476fbd5", size = 415470, upload-time = "2025-11-30T20:24:05.232Z" }, + { url = "https://files.pythonhosted.org/packages/8c/28/882e72b5b3e6f718d5453bd4d0d9cf8df36fddeb4ddbbab17869d5868616/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:74a3243a411126362712ee1524dfc90c650a503502f135d54d1b352bd01f2404", size = 565630, upload-time = "2025-11-30T20:24:06.878Z" }, + { url = "https://files.pythonhosted.org/packages/3b/97/04a65539c17692de5b85c6e293520fd01317fd878ea1995f0367d4532fb1/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:3e8eeb0544f2eb0d2581774be4c3410356eba189529a6b3e36bbbf9696175856", size = 591148, upload-time = "2025-11-30T20:24:08.445Z" }, + { url = "https://files.pythonhosted.org/packages/85/70/92482ccffb96f5441aab93e26c4d66489eb599efdcf96fad90c14bbfb976/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:dbd936cde57abfee19ab3213cf9c26be06d60750e60a8e4dd85d1ab12c8b1f40", size = 556030, upload-time = "2025-11-30T20:24:10.956Z" }, + { url = "https://files.pythonhosted.org/packages/20/53/7c7e784abfa500a2b6b583b147ee4bb5a2b3747a9166bab52fec4b5b5e7d/rpds_py-0.30.0-cp314-cp314t-win32.whl", hash = "sha256:dc824125c72246d924f7f796b4f63c1e9dc810c7d9e2355864b3c3a73d59ade0", size = 211570, upload-time = "2025-11-30T20:24:12.735Z" }, + { url = "https://files.pythonhosted.org/packages/d0/02/fa464cdfbe6b26e0600b62c528b72d8608f5cc49f96b8d6e38c95d60c676/rpds_py-0.30.0-cp314-cp314t-win_amd64.whl", hash = "sha256:27f4b0e92de5bfbc6f86e43959e6edd1425c33b5e69aab0984a72047f2bcf1e3", size = 226532, upload-time = "2025-11-30T20:24:14.634Z" }, + { url = "https://files.pythonhosted.org/packages/69/71/3f34339ee70521864411f8b6992e7ab13ac30d8e4e3309e07c7361767d91/rpds_py-0.30.0-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:c2262bdba0ad4fc6fb5545660673925c2d2a5d9e2e0fb603aad545427be0fc58", size = 372292, upload-time = "2025-11-30T20:24:16.537Z" }, + { url = "https://files.pythonhosted.org/packages/57/09/f183df9b8f2d66720d2ef71075c59f7e1b336bec7ee4c48f0a2b06857653/rpds_py-0.30.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:ee6af14263f25eedc3bb918a3c04245106a42dfd4f5c2285ea6f997b1fc3f89a", size = 362128, upload-time = "2025-11-30T20:24:18.086Z" }, + { url = "https://files.pythonhosted.org/packages/7a/68/5c2594e937253457342e078f0cc1ded3dd7b2ad59afdbf2d354869110a02/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3adbb8179ce342d235c31ab8ec511e66c73faa27a47e076ccc92421add53e2bb", size = 391542, upload-time = "2025-11-30T20:24:20.092Z" }, + { url = "https://files.pythonhosted.org/packages/49/5c/31ef1afd70b4b4fbdb2800249f34c57c64beb687495b10aec0365f53dfc4/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:250fa00e9543ac9b97ac258bd37367ff5256666122c2d0f2bc97577c60a1818c", size = 404004, upload-time = "2025-11-30T20:24:22.231Z" }, + { url = "https://files.pythonhosted.org/packages/e3/63/0cfbea38d05756f3440ce6534d51a491d26176ac045e2707adc99bb6e60a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9854cf4f488b3d57b9aaeb105f06d78e5529d3145b1e4a41750167e8c213c6d3", size = 527063, upload-time = "2025-11-30T20:24:24.302Z" }, + { url = "https://files.pythonhosted.org/packages/42/e6/01e1f72a2456678b0f618fc9a1a13f882061690893c192fcad9f2926553a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:993914b8e560023bc0a8bf742c5f303551992dcb85e247b1e5c7f4a7d145bda5", size = 413099, upload-time = "2025-11-30T20:24:25.916Z" }, + { url = "https://files.pythonhosted.org/packages/b8/25/8df56677f209003dcbb180765520c544525e3ef21ea72279c98b9aa7c7fb/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58edca431fb9b29950807e301826586e5bbf24163677732429770a697ffe6738", size = 392177, upload-time = "2025-11-30T20:24:27.834Z" }, + { url = "https://files.pythonhosted.org/packages/4a/b4/0a771378c5f16f8115f796d1f437950158679bcd2a7c68cf251cfb00ed5b/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_31_riscv64.whl", hash = "sha256:dea5b552272a944763b34394d04577cf0f9bd013207bc32323b5a89a53cf9c2f", size = 406015, upload-time = "2025-11-30T20:24:29.457Z" }, + { url = "https://files.pythonhosted.org/packages/36/d8/456dbba0af75049dc6f63ff295a2f92766b9d521fa00de67a2bd6427d57a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ba3af48635eb83d03f6c9735dfb21785303e73d22ad03d489e88adae6eab8877", size = 423736, upload-time = "2025-11-30T20:24:31.22Z" }, + { url = "https://files.pythonhosted.org/packages/13/64/b4d76f227d5c45a7e0b796c674fd81b0a6c4fbd48dc29271857d8219571c/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:dff13836529b921e22f15cb099751209a60009731a68519630a24d61f0b1b30a", size = 573981, upload-time = "2025-11-30T20:24:32.934Z" }, + { url = "https://files.pythonhosted.org/packages/20/91/092bacadeda3edf92bf743cc96a7be133e13a39cdbfd7b5082e7ab638406/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:1b151685b23929ab7beec71080a8889d4d6d9fa9a983d213f07121205d48e2c4", size = 599782, upload-time = "2025-11-30T20:24:35.169Z" }, + { url = "https://files.pythonhosted.org/packages/d1/b7/b95708304cd49b7b6f82fdd039f1748b66ec2b21d6a45180910802f1abf1/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:ac37f9f516c51e5753f27dfdef11a88330f04de2d564be3991384b2f3535d02e", size = 562191, upload-time = "2025-11-30T20:24:36.853Z" }, +] + +[[package]] +name = "scale-vero" +version = "0.5.0" +source = { editable = "." } +dependencies = [ + { name = "click" }, + { name = "pydantic" }, + { name = "wcmatch" }, +] + +[package.optional-dependencies] +claude = [ + { name = "claude-agent-sdk" }, +] +harbor = [ + { name = "fastapi" }, + { name = "httpx" }, + { name = "jinja2" }, + { name = "pyyaml" }, + { name = "uvicorn" }, +] +optimize = [ + { name = "async-lru" }, + { name = "beautifulsoup4" }, + { name = "httpx" }, + { name = "lxml" }, + { name = "openai-agents", extra = ["litellm"] }, + { name = "orjson" }, + { name = "pypdf" }, + { name = "trafilatura" }, +] +wandb = [ + { name = "wandb" }, +] + +[package.dev-dependencies] +dev = [ + { name = "gitpython" }, + { name = "pytest" }, + { name = "pytest-asyncio" }, + { name = "pytest-json-report" }, + { name = "scale-vero", extra = ["claude", "harbor", "optimize"] }, +] + +[package.metadata] +requires-dist = [ + { name = "async-lru", marker = "extra == 'optimize'", specifier = ">=2.0.5" }, + { name = "beautifulsoup4", marker = "extra == 'optimize'", specifier = ">=4.14.2" }, + { name = "claude-agent-sdk", marker = "extra == 'claude'", specifier = ">=0.1.56" }, + { name = "click", specifier = ">=8.0.0" }, + { name = "fastapi", marker = "extra == 'harbor'", specifier = ">=0.110" }, + { name = "httpx", marker = "extra == 'harbor'", specifier = ">=0.28.1" }, + { name = "httpx", marker = "extra == 'optimize'", specifier = ">=0.28.1" }, + { name = "jinja2", marker = "extra == 'harbor'", specifier = ">=3.1.6" }, + { name = "lxml", marker = "extra == 'optimize'", specifier = ">=6.0.2" }, + { name = "openai-agents", extras = ["litellm"], marker = "extra == 'optimize'", specifier = ">=0.18.3,<0.19" }, + { name = "orjson", marker = "extra == 'optimize'", specifier = ">=3.10" }, + { name = "pydantic", specifier = ">=2.11.7" }, + { name = "pypdf", marker = "extra == 'optimize'", specifier = ">=6.2.0" }, + { name = "pyyaml", marker = "extra == 'harbor'", specifier = ">=6.0.2" }, + { name = "trafilatura", marker = "extra == 'optimize'", specifier = ">=2.0.0" }, + { name = "uvicorn", marker = "extra == 'harbor'", specifier = ">=0.27" }, + { name = "wandb", marker = "extra == 'wandb'", specifier = ">=0.19.10" }, + { name = "wcmatch", specifier = ">=10.1" }, +] +provides-extras = ["harbor", "claude", "optimize", "wandb"] + +[package.metadata.requires-dev] +dev = [ + { name = "gitpython", specifier = ">=3.1.46" }, + { name = "pytest", specifier = ">=9.0.2" }, + { name = "pytest-asyncio", specifier = ">=1.3.0" }, + { name = "pytest-json-report", specifier = ">=1.5.0" }, + { name = "scale-vero", extras = ["optimize", "harbor", "claude"] }, +] + +[[package]] +name = "sentry-sdk" +version = "2.65.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f1/1f/ed17a390348156ca99fe622b97cd7d2f1969b5f49df89084b0f28e7953e9/sentry_sdk-2.65.0.tar.gz", hash = "sha256:c94dc945d54bad49d4f20448b1e6b217ca2f92f46d05c3e83d41764af685c3d1", size = 932133, upload-time = "2026-07-13T11:33:19.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/21/3b/326ad4c03b5da89b5124c8890af66e8119c4d2e10abc0619e0d67d9f7c7f/sentry_sdk-2.65.0-py3-none-any.whl", hash = "sha256:3595169677a808e4d0e1ea6ffb89443459549c7a98392ed71c77c847182ab6bf", size = 503869, upload-time = "2026-07-13T11:33:17.71Z" }, +] + +[[package]] +name = "shellingham" +version = "1.5.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/15/8b3609fd3830ef7b27b655beb4b4e9c62313a4e8da8c676e142cc210d58e/shellingham-1.5.4.tar.gz", hash = "sha256:8dbca0739d487e5bd35ab3ca4b36e11c4078f3a234bfce294b0a0291363404de", size = 10310, upload-time = "2023-10-24T04:13:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl", hash = "sha256:7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686", size = 9755, upload-time = "2023-10-24T04:13:38.866Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "smmap" +version = "5.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1f/ea/49c993d6dfdd7338c9b1000a0f36817ed7ec84577ae2e52f890d1a4ff909/smmap-5.0.3.tar.gz", hash = "sha256:4d9debb8b99007ae47165abc08670bd74cb74b5227dda7f643eccc4e9eb5642c", size = 22506, upload-time = "2026-03-09T03:43:26.1Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl", hash = "sha256:c106e05d5a61449cf6ba9a1e650227ecfb141590d2a98412103ff35d89fc7b2f", size = 24390, upload-time = "2026-03-09T03:43:24.361Z" }, +] + +[[package]] +name = "sniffio" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372, upload-time = "2024-02-25T23:20:04.057Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, +] + +[[package]] +name = "soupsieve" +version = "2.8.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7b/ae/2d9c981590ed9999a0d91755b47fc74f74de286b0f5cee14c9269041e6c4/soupsieve-2.8.3.tar.gz", hash = "sha256:3267f1eeea4251fb42728b6dfb746edc9acaffc4a45b27e19450b676586e8349", size = 118627, upload-time = "2026-01-20T04:27:02.457Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/46/2c/1462b1d0a634697ae9e55b3cecdcb64788e8b7d63f54d923fcd0bb140aed/soupsieve-2.8.3-py3-none-any.whl", hash = "sha256:ed64f2ba4eebeab06cc4962affce381647455978ffc1e36bb79a545b91f45a95", size = 37016, upload-time = "2026-01-20T04:27:01.012Z" }, +] + +[[package]] +name = "sse-starlette" +version = "3.3.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "starlette" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/26/8c/f9290339ef6d79badbc010f067cd769d6601ec11a57d78569c683fb4dd87/sse_starlette-3.3.4.tar.gz", hash = "sha256:aaf92fc067af8a5427192895ac028e947b484ac01edbc3caf00e7e7137c7bef1", size = 32427, upload-time = "2026-03-29T09:00:23.307Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f8/7f/3de5402f39890ac5660b86bcf5c03f9d855dad5c4ed764866d7b592b46fd/sse_starlette-3.3.4-py3-none-any.whl", hash = "sha256:84bb06e58939a8b38d8341f1bc9792f06c2b53f48c608dd207582b664fc8f3c1", size = 14330, upload-time = "2026-03-29T09:00:21.846Z" }, +] + +[[package]] +name = "starlette" +version = "1.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/81/69/17425771797c36cded50b7fe44e850315d039f28b15901ab44839e70b593/starlette-1.0.0.tar.gz", hash = "sha256:6a4beaf1f81bb472fd19ea9b918b50dc3a77a6f2e190a12954b25e6ed5eea149", size = 2655289, upload-time = "2026-03-22T18:29:46.779Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/c9/584bc9651441b4ba60cc4d557d8a547b5aff901af35bda3a4ee30c819b82/starlette-1.0.0-py3-none-any.whl", hash = "sha256:d3ec55e0bb321692d275455ddfd3df75fff145d009685eb40dc91fc66b03d38b", size = 72651, upload-time = "2026-03-22T18:29:45.111Z" }, +] + +[[package]] +name = "tiktoken" +version = "0.12.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "regex" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7d/ab/4d017d0f76ec3171d469d80fc03dfbb4e48a4bcaddaa831b31d526f05edc/tiktoken-0.12.0.tar.gz", hash = "sha256:b18ba7ee2b093863978fcb14f74b3707cdc8d4d4d3836853ce7ec60772139931", size = 37806, upload-time = "2025-10-06T20:22:45.419Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/de/46/21ea696b21f1d6d1efec8639c204bdf20fde8bafb351e1355c72c5d7de52/tiktoken-0.12.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:6e227c7f96925003487c33b1b32265fad2fbcec2b7cf4817afb76d416f40f6bb", size = 1051565, upload-time = "2025-10-06T20:21:44.566Z" }, + { url = "https://files.pythonhosted.org/packages/c9/d9/35c5d2d9e22bb2a5f74ba48266fb56c63d76ae6f66e02feb628671c0283e/tiktoken-0.12.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c06cf0fcc24c2cb2adb5e185c7082a82cba29c17575e828518c2f11a01f445aa", size = 995284, upload-time = "2025-10-06T20:21:45.622Z" }, + { url = "https://files.pythonhosted.org/packages/01/84/961106c37b8e49b9fdcf33fe007bb3a8fdcc380c528b20cc7fbba80578b8/tiktoken-0.12.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:f18f249b041851954217e9fd8e5c00b024ab2315ffda5ed77665a05fa91f42dc", size = 1129201, upload-time = "2025-10-06T20:21:47.074Z" }, + { url = "https://files.pythonhosted.org/packages/6a/d0/3d9275198e067f8b65076a68894bb52fd253875f3644f0a321a720277b8a/tiktoken-0.12.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:47a5bc270b8c3db00bb46ece01ef34ad050e364b51d406b6f9730b64ac28eded", size = 1152444, upload-time = "2025-10-06T20:21:48.139Z" }, + { url = "https://files.pythonhosted.org/packages/78/db/a58e09687c1698a7c592e1038e01c206569b86a0377828d51635561f8ebf/tiktoken-0.12.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:508fa71810c0efdcd1b898fda574889ee62852989f7c1667414736bcb2b9a4bd", size = 1195080, upload-time = "2025-10-06T20:21:49.246Z" }, + { url = "https://files.pythonhosted.org/packages/9e/1b/a9e4d2bf91d515c0f74afc526fd773a812232dd6cda33ebea7f531202325/tiktoken-0.12.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:a1af81a6c44f008cba48494089dd98cccb8b313f55e961a52f5b222d1e507967", size = 1255240, upload-time = "2025-10-06T20:21:50.274Z" }, + { url = "https://files.pythonhosted.org/packages/9d/15/963819345f1b1fb0809070a79e9dd96938d4ca41297367d471733e79c76c/tiktoken-0.12.0-cp311-cp311-win_amd64.whl", hash = "sha256:3e68e3e593637b53e56f7237be560f7a394451cb8c11079755e80ae64b9e6def", size = 879422, upload-time = "2025-10-06T20:21:51.734Z" }, + { url = "https://files.pythonhosted.org/packages/a4/85/be65d39d6b647c79800fd9d29241d081d4eeb06271f383bb87200d74cf76/tiktoken-0.12.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b97f74aca0d78a1ff21b8cd9e9925714c15a9236d6ceacf5c7327c117e6e21e8", size = 1050728, upload-time = "2025-10-06T20:21:52.756Z" }, + { url = "https://files.pythonhosted.org/packages/4a/42/6573e9129bc55c9bf7300b3a35bef2c6b9117018acca0dc760ac2d93dffe/tiktoken-0.12.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:2b90f5ad190a4bb7c3eb30c5fa32e1e182ca1ca79f05e49b448438c3e225a49b", size = 994049, upload-time = "2025-10-06T20:21:53.782Z" }, + { url = "https://files.pythonhosted.org/packages/66/c5/ed88504d2f4a5fd6856990b230b56d85a777feab84e6129af0822f5d0f70/tiktoken-0.12.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:65b26c7a780e2139e73acc193e5c63ac754021f160df919add909c1492c0fb37", size = 1129008, upload-time = "2025-10-06T20:21:54.832Z" }, + { url = "https://files.pythonhosted.org/packages/f4/90/3dae6cc5436137ebd38944d396b5849e167896fc2073da643a49f372dc4f/tiktoken-0.12.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:edde1ec917dfd21c1f2f8046b86348b0f54a2c0547f68149d8600859598769ad", size = 1152665, upload-time = "2025-10-06T20:21:56.129Z" }, + { url = "https://files.pythonhosted.org/packages/a3/fe/26df24ce53ffde419a42f5f53d755b995c9318908288c17ec3f3448313a3/tiktoken-0.12.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:35a2f8ddd3824608b3d650a000c1ef71f730d0c56486845705a8248da00f9fe5", size = 1194230, upload-time = "2025-10-06T20:21:57.546Z" }, + { url = "https://files.pythonhosted.org/packages/20/cc/b064cae1a0e9fac84b0d2c46b89f4e57051a5f41324e385d10225a984c24/tiktoken-0.12.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:83d16643edb7fa2c99eff2ab7733508aae1eebb03d5dfc46f5565862810f24e3", size = 1254688, upload-time = "2025-10-06T20:21:58.619Z" }, + { url = "https://files.pythonhosted.org/packages/81/10/b8523105c590c5b8349f2587e2fdfe51a69544bd5a76295fc20f2374f470/tiktoken-0.12.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffc5288f34a8bc02e1ea7047b8d041104791d2ddbf42d1e5fa07822cbffe16bd", size = 878694, upload-time = "2025-10-06T20:21:59.876Z" }, + { url = "https://files.pythonhosted.org/packages/00/61/441588ee21e6b5cdf59d6870f86beb9789e532ee9718c251b391b70c68d6/tiktoken-0.12.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:775c2c55de2310cc1bc9a3ad8826761cbdc87770e586fd7b6da7d4589e13dab3", size = 1050802, upload-time = "2025-10-06T20:22:00.96Z" }, + { url = "https://files.pythonhosted.org/packages/1f/05/dcf94486d5c5c8d34496abe271ac76c5b785507c8eae71b3708f1ad9b45a/tiktoken-0.12.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a01b12f69052fbe4b080a2cfb867c4de12c704b56178edf1d1d7b273561db160", size = 993995, upload-time = "2025-10-06T20:22:02.788Z" }, + { url = "https://files.pythonhosted.org/packages/a0/70/5163fe5359b943f8db9946b62f19be2305de8c3d78a16f629d4165e2f40e/tiktoken-0.12.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:01d99484dc93b129cd0964f9d34eee953f2737301f18b3c7257bf368d7615baa", size = 1128948, upload-time = "2025-10-06T20:22:03.814Z" }, + { url = "https://files.pythonhosted.org/packages/0c/da/c028aa0babf77315e1cef357d4d768800c5f8a6de04d0eac0f377cb619fa/tiktoken-0.12.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:4a1a4fcd021f022bfc81904a911d3df0f6543b9e7627b51411da75ff2fe7a1be", size = 1151986, upload-time = "2025-10-06T20:22:05.173Z" }, + { url = "https://files.pythonhosted.org/packages/a0/5a/886b108b766aa53e295f7216b509be95eb7d60b166049ce2c58416b25f2a/tiktoken-0.12.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:981a81e39812d57031efdc9ec59fa32b2a5a5524d20d4776574c4b4bd2e9014a", size = 1194222, upload-time = "2025-10-06T20:22:06.265Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f8/4db272048397636ac7a078d22773dd2795b1becee7bc4922fe6207288d57/tiktoken-0.12.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9baf52f84a3f42eef3ff4e754a0db79a13a27921b457ca9832cf944c6be4f8f3", size = 1255097, upload-time = "2025-10-06T20:22:07.403Z" }, + { url = "https://files.pythonhosted.org/packages/8e/32/45d02e2e0ea2be3a9ed22afc47d93741247e75018aac967b713b2941f8ea/tiktoken-0.12.0-cp313-cp313-win_amd64.whl", hash = "sha256:b8a0cd0c789a61f31bf44851defbd609e8dd1e2c8589c614cc1060940ef1f697", size = 879117, upload-time = "2025-10-06T20:22:08.418Z" }, + { url = "https://files.pythonhosted.org/packages/ce/76/994fc868f88e016e6d05b0da5ac24582a14c47893f4474c3e9744283f1d5/tiktoken-0.12.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:d5f89ea5680066b68bcb797ae85219c72916c922ef0fcdd3480c7d2315ffff16", size = 1050309, upload-time = "2025-10-06T20:22:10.939Z" }, + { url = "https://files.pythonhosted.org/packages/f6/b8/57ef1456504c43a849821920d582a738a461b76a047f352f18c0b26c6516/tiktoken-0.12.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b4e7ed1c6a7a8a60a3230965bdedba8cc58f68926b835e519341413370e0399a", size = 993712, upload-time = "2025-10-06T20:22:12.115Z" }, + { url = "https://files.pythonhosted.org/packages/72/90/13da56f664286ffbae9dbcfadcc625439142675845baa62715e49b87b68b/tiktoken-0.12.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:fc530a28591a2d74bce821d10b418b26a094bf33839e69042a6e86ddb7a7fb27", size = 1128725, upload-time = "2025-10-06T20:22:13.541Z" }, + { url = "https://files.pythonhosted.org/packages/05/df/4f80030d44682235bdaecd7346c90f67ae87ec8f3df4a3442cb53834f7e4/tiktoken-0.12.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:06a9f4f49884139013b138920a4c393aa6556b2f8f536345f11819389c703ebb", size = 1151875, upload-time = "2025-10-06T20:22:14.559Z" }, + { url = "https://files.pythonhosted.org/packages/22/1f/ae535223a8c4ef4c0c1192e3f9b82da660be9eb66b9279e95c99288e9dab/tiktoken-0.12.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:04f0e6a985d95913cabc96a741c5ffec525a2c72e9df086ff17ebe35985c800e", size = 1194451, upload-time = "2025-10-06T20:22:15.545Z" }, + { url = "https://files.pythonhosted.org/packages/78/a7/f8ead382fce0243cb625c4f266e66c27f65ae65ee9e77f59ea1653b6d730/tiktoken-0.12.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:0ee8f9ae00c41770b5f9b0bb1235474768884ae157de3beb5439ca0fd70f3e25", size = 1253794, upload-time = "2025-10-06T20:22:16.624Z" }, + { url = "https://files.pythonhosted.org/packages/93/e0/6cc82a562bc6365785a3ff0af27a2a092d57c47d7a81d9e2295d8c36f011/tiktoken-0.12.0-cp313-cp313t-win_amd64.whl", hash = "sha256:dc2dd125a62cb2b3d858484d6c614d136b5b848976794edfb63688d539b8b93f", size = 878777, upload-time = "2025-10-06T20:22:18.036Z" }, + { url = "https://files.pythonhosted.org/packages/72/05/3abc1db5d2c9aadc4d2c76fa5640134e475e58d9fbb82b5c535dc0de9b01/tiktoken-0.12.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:a90388128df3b3abeb2bfd1895b0681412a8d7dc644142519e6f0a97c2111646", size = 1050188, upload-time = "2025-10-06T20:22:19.563Z" }, + { url = "https://files.pythonhosted.org/packages/e3/7b/50c2f060412202d6c95f32b20755c7a6273543b125c0985d6fa9465105af/tiktoken-0.12.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:da900aa0ad52247d8794e307d6446bd3cdea8e192769b56276695d34d2c9aa88", size = 993978, upload-time = "2025-10-06T20:22:20.702Z" }, + { url = "https://files.pythonhosted.org/packages/14/27/bf795595a2b897e271771cd31cb847d479073497344c637966bdf2853da1/tiktoken-0.12.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:285ba9d73ea0d6171e7f9407039a290ca77efcdb026be7769dccc01d2c8d7fff", size = 1129271, upload-time = "2025-10-06T20:22:22.06Z" }, + { url = "https://files.pythonhosted.org/packages/f5/de/9341a6d7a8f1b448573bbf3425fa57669ac58258a667eb48a25dfe916d70/tiktoken-0.12.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:d186a5c60c6a0213f04a7a802264083dea1bbde92a2d4c7069e1a56630aef830", size = 1151216, upload-time = "2025-10-06T20:22:23.085Z" }, + { url = "https://files.pythonhosted.org/packages/75/0d/881866647b8d1be4d67cb24e50d0c26f9f807f994aa1510cb9ba2fe5f612/tiktoken-0.12.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:604831189bd05480f2b885ecd2d1986dc7686f609de48208ebbbddeea071fc0b", size = 1194860, upload-time = "2025-10-06T20:22:24.602Z" }, + { url = "https://files.pythonhosted.org/packages/b3/1e/b651ec3059474dab649b8d5b69f5c65cd8fcd8918568c1935bd4136c9392/tiktoken-0.12.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:8f317e8530bb3a222547b85a58583238c8f74fd7a7408305f9f63246d1a0958b", size = 1254567, upload-time = "2025-10-06T20:22:25.671Z" }, + { url = "https://files.pythonhosted.org/packages/80/57/ce64fd16ac390fafde001268c364d559447ba09b509181b2808622420eec/tiktoken-0.12.0-cp314-cp314-win_amd64.whl", hash = "sha256:399c3dd672a6406719d84442299a490420b458c44d3ae65516302a99675888f3", size = 921067, upload-time = "2025-10-06T20:22:26.753Z" }, + { url = "https://files.pythonhosted.org/packages/ac/a4/72eed53e8976a099539cdd5eb36f241987212c29629d0a52c305173e0a68/tiktoken-0.12.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:c2c714c72bc00a38ca969dae79e8266ddec999c7ceccd603cc4f0d04ccd76365", size = 1050473, upload-time = "2025-10-06T20:22:27.775Z" }, + { url = "https://files.pythonhosted.org/packages/e6/d7/0110b8f54c008466b19672c615f2168896b83706a6611ba6e47313dbc6e9/tiktoken-0.12.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:cbb9a3ba275165a2cb0f9a83f5d7025afe6b9d0ab01a22b50f0e74fee2ad253e", size = 993855, upload-time = "2025-10-06T20:22:28.799Z" }, + { url = "https://files.pythonhosted.org/packages/5f/77/4f268c41a3957c418b084dd576ea2fad2e95da0d8e1ab705372892c2ca22/tiktoken-0.12.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:dfdfaa5ffff8993a3af94d1125870b1d27aed7cb97aa7eb8c1cefdbc87dbee63", size = 1129022, upload-time = "2025-10-06T20:22:29.981Z" }, + { url = "https://files.pythonhosted.org/packages/4e/2b/fc46c90fe5028bd094cd6ee25a7db321cb91d45dc87531e2bdbb26b4867a/tiktoken-0.12.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:584c3ad3d0c74f5269906eb8a659c8bfc6144a52895d9261cdaf90a0ae5f4de0", size = 1150736, upload-time = "2025-10-06T20:22:30.996Z" }, + { url = "https://files.pythonhosted.org/packages/28/c0/3c7a39ff68022ddfd7d93f3337ad90389a342f761c4d71de99a3ccc57857/tiktoken-0.12.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:54c891b416a0e36b8e2045b12b33dd66fb34a4fe7965565f1b482da50da3e86a", size = 1194908, upload-time = "2025-10-06T20:22:32.073Z" }, + { url = "https://files.pythonhosted.org/packages/ab/0d/c1ad6f4016a3968c048545f5d9b8ffebf577774b2ede3e2e352553b685fe/tiktoken-0.12.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5edb8743b88d5be814b1a8a8854494719080c28faaa1ccbef02e87354fe71ef0", size = 1253706, upload-time = "2025-10-06T20:22:33.385Z" }, + { url = "https://files.pythonhosted.org/packages/af/df/c7891ef9d2712ad774777271d39fdef63941ffba0a9d59b7ad1fd2765e57/tiktoken-0.12.0-cp314-cp314t-win_amd64.whl", hash = "sha256:f61c0aea5565ac82e2ec50a05e02a6c44734e91b51c10510b084ea1b8e633a71", size = 920667, upload-time = "2025-10-06T20:22:34.444Z" }, +] + +[[package]] +name = "tld" +version = "0.13.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5c/5d/76b4383ac4e5b5e254e50c09807b3e13820bed6d6c11cd540264988d6802/tld-0.13.2.tar.gz", hash = "sha256:d983fa92b9d717400742fca844e29d5e18271079c7bcfabf66d01b39b4a14345", size = 467175, upload-time = "2026-03-06T23:50:34.498Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/90/39a85a4b63c84213e78b3c17d22e1bf45328acf8ebb33ef93be30d0a3911/tld-0.13.2-py2.py3-none-any.whl", hash = "sha256:9b8fdbdb880e7ba65b216a4937f2c94c49a7226723783d5838fc958ac76f4e0c", size = 296743, upload-time = "2026-03-06T23:50:32.465Z" }, +] + +[[package]] +name = "tokenizers" +version = "0.22.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/73/6f/f80cfef4a312e1fb34baf7d85c72d4411afde10978d4657f8cdd811d3ccc/tokenizers-0.22.2.tar.gz", hash = "sha256:473b83b915e547aa366d1eee11806deaf419e17be16310ac0a14077f1e28f917", size = 372115, upload-time = "2026-01-05T10:45:15.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/92/97/5dbfabf04c7e348e655e907ed27913e03db0923abb5dfdd120d7b25630e1/tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:544dd704ae7238755d790de45ba8da072e9af3eea688f698b137915ae959281c", size = 3100275, upload-time = "2026-01-05T10:41:02.158Z" }, + { url = "https://files.pythonhosted.org/packages/2e/47/174dca0502ef88b28f1c9e06b73ce33500eedfac7a7692108aec220464e7/tokenizers-0.22.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1e418a55456beedca4621dbab65a318981467a2b188e982a23e117f115ce5001", size = 2981472, upload-time = "2026-01-05T10:41:00.276Z" }, + { url = "https://files.pythonhosted.org/packages/d6/84/7990e799f1309a8b87af6b948f31edaa12a3ed22d11b352eaf4f4b2e5753/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2249487018adec45d6e3554c71d46eb39fa8ea67156c640f7513eb26f318cec7", size = 3290736, upload-time = "2026-01-05T10:40:32.165Z" }, + { url = "https://files.pythonhosted.org/packages/78/59/09d0d9ba94dcd5f4f1368d4858d24546b4bdc0231c2354aa31d6199f0399/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:25b85325d0815e86e0bac263506dd114578953b7b53d7de09a6485e4a160a7dd", size = 3168835, upload-time = "2026-01-05T10:40:38.847Z" }, + { url = "https://files.pythonhosted.org/packages/47/50/b3ebb4243e7160bda8d34b731e54dd8ab8b133e50775872e7a434e524c28/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bfb88f22a209ff7b40a576d5324bf8286b519d7358663db21d6246fb17eea2d5", size = 3521673, upload-time = "2026-01-05T10:40:56.614Z" }, + { url = "https://files.pythonhosted.org/packages/e0/fa/89f4cb9e08df770b57adb96f8cbb7e22695a4cb6c2bd5f0c4f0ebcf33b66/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1c774b1276f71e1ef716e5486f21e76333464f47bece56bbd554485982a9e03e", size = 3724818, upload-time = "2026-01-05T10:40:44.507Z" }, + { url = "https://files.pythonhosted.org/packages/64/04/ca2363f0bfbe3b3d36e95bf67e56a4c88c8e3362b658e616d1ac185d47f2/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df6c4265b289083bf710dff49bc51ef252f9d5be33a45ee2bed151114a56207b", size = 3379195, upload-time = "2026-01-05T10:40:51.139Z" }, + { url = "https://files.pythonhosted.org/packages/2e/76/932be4b50ef6ccedf9d3c6639b056a967a86258c6d9200643f01269211ca/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:369cc9fc8cc10cb24143873a0d95438bb8ee257bb80c71989e3ee290e8d72c67", size = 3274982, upload-time = "2026-01-05T10:40:58.331Z" }, + { url = "https://files.pythonhosted.org/packages/1d/28/5f9f5a4cc211b69e89420980e483831bcc29dade307955cc9dc858a40f01/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:29c30b83d8dcd061078b05ae0cb94d3c710555fbb44861139f9f83dcca3dc3e4", size = 9478245, upload-time = "2026-01-05T10:41:04.053Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fb/66e2da4704d6aadebf8cb39f1d6d1957df667ab24cff2326b77cda0dcb85/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:37ae80a28c1d3265bb1f22464c856bd23c02a05bb211e56d0c5301a435be6c1a", size = 9560069, upload-time = "2026-01-05T10:45:10.673Z" }, + { url = "https://files.pythonhosted.org/packages/16/04/fed398b05caa87ce9b1a1bb5166645e38196081b225059a6edaff6440fac/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:791135ee325f2336f498590eb2f11dc5c295232f288e75c99a36c5dbce63088a", size = 9899263, upload-time = "2026-01-05T10:45:12.559Z" }, + { url = "https://files.pythonhosted.org/packages/05/a1/d62dfe7376beaaf1394917e0f8e93ee5f67fea8fcf4107501db35996586b/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:38337540fbbddff8e999d59970f3c6f35a82de10053206a7562f1ea02d046fa5", size = 10033429, upload-time = "2026-01-05T10:45:14.333Z" }, + { url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363, upload-time = "2026-01-05T10:45:20.593Z" }, + { url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786, upload-time = "2026-01-05T10:45:18.411Z" }, + { url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133, upload-time = "2026-01-05T10:45:17.232Z" }, +] + +[[package]] +name = "tqdm" +version = "4.67.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374, upload-time = "2026-02-03T17:35:50.982Z" }, +] + +[[package]] +name = "trafilatura" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "courlan" }, + { name = "htmldate" }, + { name = "justext" }, + { name = "lxml" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/25/e3ebeefdebfdfae8c4a4396f5a6ea51fc6fa0831d63ce338e5090a8003dc/trafilatura-2.0.0.tar.gz", hash = "sha256:ceb7094a6ecc97e72fea73c7dba36714c5c5b577b6470e4520dca893706d6247", size = 253404, upload-time = "2024-12-03T15:23:24.16Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8a/b6/097367f180b6383a3581ca1b86fcae284e52075fa941d1232df35293363c/trafilatura-2.0.0-py3-none-any.whl", hash = "sha256:77eb5d1e993747f6f20938e1de2d840020719735690c840b9a1024803a4cd51d", size = 132557, upload-time = "2024-12-03T15:23:21.41Z" }, +] + +[[package]] +name = "typer" +version = "0.24.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "click" }, + { name = "rich" }, + { name = "shellingham" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f5/24/cb09efec5cc954f7f9b930bf8279447d24618bb6758d4f6adf2574c41780/typer-0.24.1.tar.gz", hash = "sha256:e39b4732d65fbdcde189ae76cf7cd48aeae72919dea1fdfc16593be016256b45", size = 118613, upload-time = "2026-02-21T16:54:40.609Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4a/91/48db081e7a63bb37284f9fbcefda7c44c277b18b0e13fbc36ea2335b71e6/typer-0.24.1-py3-none-any.whl", hash = "sha256:112c1f0ce578bfb4cab9ffdabc68f031416ebcc216536611ba21f04e9aa84c9e", size = 56085, upload-time = "2026-02-21T16:54:41.616Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/19/f5/cd531b2d15a671a40c0f66cf06bc3570a12cd56eef98960068ebbad1bf5a/tzdata-2026.1.tar.gz", hash = "sha256:67658a1903c75917309e753fdc349ac0efd8c27db7a0cb406a25be4840f87f98", size = 197639, upload-time = "2026-04-03T11:25:22.002Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b0/70/d460bd685a170790ec89317e9bd33047988e4bce507b831f5db771e142de/tzdata-2026.1-py2.py3-none-any.whl", hash = "sha256:4b1d2be7ac37ceafd7327b961aa3a54e467efbdb563a23655fbfe0d39cfc42a9", size = 348952, upload-time = "2026-04-03T11:25:20.313Z" }, +] + +[[package]] +name = "tzlocal" +version = "5.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "tzdata", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8b/2e/c14812d3d4d9cd1773c6be938f89e5735a1f11a9f184ac3639b93cef35d5/tzlocal-5.3.1.tar.gz", hash = "sha256:cceffc7edecefea1f595541dbd6e990cb1ea3d19bf01b2809f362a03dd7921fd", size = 30761, upload-time = "2025-03-05T21:17:41.549Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/14/e2a54fabd4f08cd7af1c07030603c3356b74da07f7cc056e600436edfa17/tzlocal-5.3.1-py3-none-any.whl", hash = "sha256:eb1a66c3ef5847adf7a834f1be0800581b683b5608e74f86ecbcef8ab91bb85d", size = 18026, upload-time = "2025-03-05T21:17:39.857Z" }, +] + +[[package]] +name = "urllib3" +version = "2.6.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/24/5f1b3bdffd70275f6661c76461e25f024d5a38a46f04aaca912426a2b1d3/urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed", size = 435556, upload-time = "2026-01-07T16:24:43.925Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4", size = 131584, upload-time = "2026-01-07T16:24:42.685Z" }, +] + +[[package]] +name = "uvicorn" +version = "0.43.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/62/f2/368268300fb8af33743508d738ef7bb4d56afdb46c6d9c0fa3dd515df171/uvicorn-0.43.0.tar.gz", hash = "sha256:ab1652d2fb23abf124f36ccc399828558880def222c3cb3d98d24021520dc6e8", size = 85686, upload-time = "2026-04-03T18:37:48.984Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/55/df/0cf5b0c451602748fdc7a702d4667f6e209bf96aa6e3160d754234445f2a/uvicorn-0.43.0-py3-none-any.whl", hash = "sha256:46fac64f487fd968cd999e5e49efbbe64bd231b5bd8b4a0b482a23ebce499620", size = 68591, upload-time = "2026-04-03T18:37:47.64Z" }, +] + +[[package]] +name = "wandb" +version = "0.28.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "gitpython" }, + { name = "packaging" }, + { name = "platformdirs" }, + { name = "protobuf" }, + { name = "pydantic" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "sentry-sdk" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5f/a7/683bfbd6cbade3012bc90d3e9c4cfc72dd62566195bf4c30321946d64b77/wandb-0.28.0.tar.gz", hash = "sha256:b20e5af0fe80e2e2a466b0466a1d60cedcc578dce0f036eca04f4a0adcad95b6", size = 40558332, upload-time = "2026-06-23T00:38:50.115Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c0/47/1723605f76c5d6446b6d0db65b83eda1599721bc8c1e65bd76cc1682b1a7/wandb-0.28.0-py3-none-macosx_12_0_arm64.whl", hash = "sha256:c3dab1205a5aca4abbad1eca08902cdba86add0edfa83d8d61b4429d0e79fa87", size = 24335272, upload-time = "2026-06-23T00:38:26.002Z" }, + { url = "https://files.pythonhosted.org/packages/81/ff/42b539bc75bc48fc86981dccde89327ba9b71504b805b9ba42cba7c26de9/wandb-0.28.0-py3-none-macosx_12_0_x86_64.whl", hash = "sha256:ae255da18726ee8e731ef82cbc85035b901a28ae14cf91604c361b44b8d44ce0", size = 25557959, upload-time = "2026-06-23T00:38:28.993Z" }, + { url = "https://files.pythonhosted.org/packages/15/55/c3db03d04aeab3726066a418b2ef6a1f8119774ee510f4fbe992f52b7472/wandb-0.28.0-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:6dbcba12ab168aa37561f2f32dcdef8713495fc25fa7d30fdc9bfb37989694dd", size = 24878557, upload-time = "2026-06-23T00:38:31.417Z" }, + { url = "https://files.pythonhosted.org/packages/d8/5d/1385ce3c219cb5bd30d4027687e3f8d25969c7dfd09adad1cbd5080e1a72/wandb-0.28.0-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:325b2d0bd88be6eda5db10542499bad3710927f2569c81a84dc5eeaffc76825c", size = 26764727, upload-time = "2026-06-23T00:38:33.775Z" }, + { url = "https://files.pythonhosted.org/packages/00/58/23b6c17a6d3d5422b007707961c4496b2f6f892624d2910c9f7742fcc202/wandb-0.28.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:8954bc1c62ae43914dce2bebfd1d9957f72350f8fbb78e5cdfe2ca9b6be8a7b8", size = 25051656, upload-time = "2026-06-23T00:38:36.281Z" }, + { url = "https://files.pythonhosted.org/packages/89/67/9be00fb2db2281063af24a148636d2dd363d337317642ab5d8e93572c794/wandb-0.28.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:9fec6c908554c2dad33110c1312bc3028cc2e430f0679f16b84f82c8ea801e3b", size = 27074113, upload-time = "2026-06-23T00:38:38.737Z" }, + { url = "https://files.pythonhosted.org/packages/59/b1/f7a96c09cab0c5131b1e6466659b093b401e1653cbe6bb77b462fc1c361d/wandb-0.28.0-py3-none-win32.whl", hash = "sha256:8834ef3a7c8c43b701654162783caa7ad37af48a0ff06fc35d0d65a411f76ccd", size = 24525206, upload-time = "2026-06-23T00:38:42.041Z" }, + { url = "https://files.pythonhosted.org/packages/c6/c4/c7bed5e981679c74e9fbb22c03ff31c42e95f266199d03d8d325f4d0e6df/wandb-0.28.0-py3-none-win_amd64.whl", hash = "sha256:ac1f82292e2da4f98297b78c3a46726b3a6c5734ecb75fc39b8db2c8a4989159", size = 24525214, upload-time = "2026-06-23T00:38:44.549Z" }, + { url = "https://files.pythonhosted.org/packages/f0/77/b5ce9696c8cb955521a7941fbc443e78b2f504894c6ae1a2d0b1de6e12ae/wandb-0.28.0-py3-none-win_arm64.whl", hash = "sha256:c5b0faf1b84cf79ebabed77538c1940a4c6053e815f767a4004e877a1354bed1", size = 22378208, upload-time = "2026-06-23T00:38:47.148Z" }, +] + +[[package]] +name = "wcmatch" +version = "10.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "bracex" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/79/3e/c0bdc27cf06f4e47680bd5803a07cb3dfd17de84cde92dd217dcb9e05253/wcmatch-10.1.tar.gz", hash = "sha256:f11f94208c8c8484a16f4f48638a85d771d9513f4ab3f37595978801cb9465af", size = 117421, upload-time = "2025-06-22T19:14:02.49Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/eb/d8/0d1d2e9d3fabcf5d6840362adcf05f8cf3cd06a73358140c3a97189238ae/wcmatch-10.1-py3-none-any.whl", hash = "sha256:5848ace7dbb0476e5e55ab63c6bbd529745089343427caa5537f230cc01beb8a", size = 39854, upload-time = "2025-06-22T19:14:00.978Z" }, +] + +[[package]] +name = "websockets" +version = "16.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/21/f7/bc3a25c5ec26ce62ce487690becc2f3710bbc7b33338f005ad390db0b986/websockets-16.1.1.tar.gz", hash = "sha256:db234eda965dcce15df96bb9709f587cd87d4d52aaf0e80e2f34ec04c7670c57", size = 182204, upload-time = "2026-07-17T22:51:05.858Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2b/03/47debfe28e9d6d354be5d777b67fd44c359b9eb299a5d103500bd7cc3e37/websockets-16.1.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:d0fcf657e9f13ff4b177960ab2200237b12994232dfb6df16f1cfe1d4339f93c", size = 179566, upload-time = "2026-07-17T22:48:49.596Z" }, + { url = "https://files.pythonhosted.org/packages/72/93/31efa1ed78c17e5cfc229fd449e3966e1b9cc15753204cd585cc8dd01f4a/websockets-16.1.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:b852788aa51764e2d8e4cf5493d559326bcae5e38d16ba25ffa322b034df272a", size = 177250, upload-time = "2026-07-17T22:48:50.942Z" }, + { url = "https://files.pythonhosted.org/packages/01/4a/542378ab3972b0c1cf1df3df3eff9591cea0d30c58c3aa3c4ddbc244e787/websockets-16.1.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:1427fb4cf0d72f66333e2cacc3ff5f575bf2d7008166ce991a4a470b21d51a22", size = 177528, upload-time = "2026-07-17T22:48:52.59Z" }, + { url = "https://files.pythonhosted.org/packages/33/d9/162321f63c7eed558e9e1798ed7a1e34a4f6dab51f35419e4ed7a4907979/websockets-16.1.1-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:da4ca1a9d72f9030b3146b8d7022719a9f3d478f61efe6f7dd51d243f61c51b2", size = 186859, upload-time = "2026-07-17T22:48:53.915Z" }, + { url = "https://files.pythonhosted.org/packages/de/09/87df740f7430ce564bd52402e9c9458d4d0459cc7d2ee29e530c8204851b/websockets-16.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:86d7f0f8bdb25d2c632b72527325e4776430fd5bc61b9118de4e2b8ddb5f5b01", size = 188095, upload-time = "2026-07-17T22:48:55.384Z" }, + { url = "https://files.pythonhosted.org/packages/d2/12/3d2703af7cc095f3c81904c92208cc1ae79affbc67376944b50ee9301f73/websockets-16.1.1-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:7dfcad78ea1492ee3a9ec765cb7f51bbc17d477107aaf6b22abf7b2558d1c5a0", size = 191385, upload-time = "2026-07-17T22:48:56.742Z" }, + { url = "https://files.pythonhosted.org/packages/1d/69/986aa0234a964a00f5149cfc46e136e96c8faad1c783474550f40d31aef4/websockets-16.1.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:fb9a0a6dc3d1b3986cb88091b6899f0396651e0f74e2c9766ab8d6ffc3842e29", size = 188653, upload-time = "2026-07-17T22:48:58.134Z" }, + { url = "https://files.pythonhosted.org/packages/35/6b/10f9d03e3970a69ba67bd3b46b87a929b586d0300fadbfe14f57c1f85490/websockets-16.1.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:29dfa8114c4a620c69591c5973860f768eac29d3fd6904f37f34266cb219c512", size = 187426, upload-time = "2026-07-17T22:48:59.515Z" }, + { url = "https://files.pythonhosted.org/packages/56/db/bb3aad62bf63d8bb3f0634b2eabffcfb3677a34bd19492110ff6869cf703/websockets-16.1.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:6ff9417c0ada4d0f7d212f928303e5579bdf3ace4c802fa4afabb30995da58c3", size = 184882, upload-time = "2026-07-17T22:49:00.916Z" }, + { url = "https://files.pythonhosted.org/packages/6c/4c/c09a2ea9bfbeccce52fdc383e5f28af4bc8843338aabac28c81489af6120/websockets-16.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8fe0b50da2d84535fb4f7b4bfa951280f97ce3d558a0443b541166d609e67b57", size = 187584, upload-time = "2026-07-17T22:49:02.283Z" }, + { url = "https://files.pythonhosted.org/packages/c7/8b/31bb4eb4d9eaacf1fdd39d115772a8aeaedfc19b5dc262e57ffbc8a9d42c/websockets-16.1.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:34420aaa64440ebd51ac72ca8a45ef4626429438c9b02e633ae412ed43f925d3", size = 186174, upload-time = "2026-07-17T22:49:03.973Z" }, + { url = "https://files.pythonhosted.org/packages/2f/e4/dc02d725610a1ad49e193ef91a548194d71bdc6cdf27da83067dd1f73995/websockets-16.1.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:a6a61aff018180c9c50b7b0da33bfd29d378af3497429c95006c589a23a11648", size = 187986, upload-time = "2026-07-17T22:49:05.553Z" }, + { url = "https://files.pythonhosted.org/packages/e0/73/30ed84c8bfd14c73d4af29d5ed9323c3073b48e0b7b23b67070f4e7fd59b/websockets-16.1.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:04fd29a0e2fe9414a95b00e92c67ae51bf900c50c0f8a4b2dafdad621f49ea1d", size = 185565, upload-time = "2026-07-17T22:49:06.959Z" }, + { url = "https://files.pythonhosted.org/packages/7d/d3/4be8d4959f51e31b4f8fc0ece12b45bd3b6c0d15ea23b9990d9c11fc805f/websockets-16.1.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:5c31aa7e39ee3e8a358573257f1c0bb5c52430d1b637030dd9c8cc2c282926be", size = 186598, upload-time = "2026-07-17T22:49:08.293Z" }, + { url = "https://files.pythonhosted.org/packages/26/fa/abb38597a52d84ed9cfacadc7a0c6f2db282c0ab23cdf72b58a666a21227/websockets-16.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d14bfb217eb4701e850f1525c9d29d79c44794cdf1c299ead25f39f8c78dea81", size = 186834, upload-time = "2026-07-17T22:49:09.766Z" }, + { url = "https://files.pythonhosted.org/packages/59/80/1119ad08a228b90c4eb77fbe48df7836731a605f5f881ba701ca826a4a65/websockets-16.1.1-cp311-cp311-win32.whl", hash = "sha256:2e28e602bb13da44fbe518c1781a88e3b9d4c3d48d02c9bad83e546164336f57", size = 179940, upload-time = "2026-07-17T22:49:11.196Z" }, + { url = "https://files.pythonhosted.org/packages/71/b2/e511c1c6f64a95c2f3fc54bffda0e14eaa7e9442be605c29270f7589b918/websockets-16.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:7421fad442de870a8cbf2287d1cad7e706ece0dbfeba5e911df132cbdc1cb56a", size = 180239, upload-time = "2026-07-17T22:49:12.519Z" }, + { url = "https://files.pythonhosted.org/packages/17/9d/681cda21c9eee743203a6cb79b9d3d05adad9aa60ec660c6c9bf4dd619ca/websockets-16.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:cc97814dfb786a83b6e2dc2e79351e1b83e6d715647d6887fcabd83026417a00", size = 179600, upload-time = "2026-07-17T22:49:13.92Z" }, + { url = "https://files.pythonhosted.org/packages/fb/8d/6195a88b45e8d2a8f745fc2046e36f885a3c9763e6767d2c46229bf9510c/websockets-16.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:e047dc87ef7ca50f4d309bf775ad4a71711c58556d75d7bd0604b2317f43e94b", size = 177272, upload-time = "2026-07-17T22:49:15.453Z" }, + { url = "https://files.pythonhosted.org/packages/73/e3/fe2d498c64dea0095c9a9f9a351af4cd6eef31b618395582bc1f38ba45ff/websockets-16.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:01fbdcbac298efe19360b94bc0039c8f746f0220ba570f327577bfee81059175", size = 177542, upload-time = "2026-07-17T22:49:16.875Z" }, + { url = "https://files.pythonhosted.org/packages/fe/ed/f1831681fce0e3242346e5458486003c5f124ed69e5e0b847fd029db4973/websockets-16.1.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0f62863e8a00a6d33c3d6566ec0b89f23787b747ffe0c3bc71ec0e76b82c94b1", size = 187137, upload-time = "2026-07-17T22:49:18.323Z" }, + { url = "https://files.pythonhosted.org/packages/6f/79/4ff9dcc1bb46f6b4c536936dde1fd60f9b564f3304307274db97f4c9496d/websockets-16.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8087e82f842609734c9b5a1330464f8e94e346ba0e18c832c08bafa4b0d63c15", size = 188374, upload-time = "2026-07-17T22:49:19.65Z" }, + { url = "https://files.pythonhosted.org/packages/62/c3/5c49b6efb36cab733d23773f6de575e1dba65736ead17d5d2b2a1daef779/websockets-16.1.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2bb5d041a8307d2e18782e7ce777f6fdb1e8c2f5d09291484b18c294b789d9aa", size = 191155, upload-time = "2026-07-17T22:49:21.331Z" }, + { url = "https://files.pythonhosted.org/packages/6e/f6/56ccceda3a4838d18f1d40821480da4775397e8b1eecf4031e20c50e2e90/websockets-16.1.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1db4de4a0e95673f7545d393c49eeb0c2f18ac1ef93073218c79d5cdb2ee75ab", size = 189011, upload-time = "2026-07-17T22:49:22.889Z" }, + { url = "https://files.pythonhosted.org/packages/86/d6/ad5286241a2bce1107e2798d3bfbd62cf79aee167bdb654f8cb1e9dbf949/websockets-16.1.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f17dbe07eb3ea7f99e4df9b7e0efefe80fbf30d37a8cc4d561a0aed310bc8847", size = 187766, upload-time = "2026-07-17T22:49:24.339Z" }, + { url = "https://files.pythonhosted.org/packages/bc/67/d65c970b7e347fdca69479beb7811c2060529956730a7a4e3ae7c66b0e31/websockets-16.1.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4b57693728576d84ede0a77987ab16881b783d2cd9f1dc180a8fbbc3f79c4428", size = 185173, upload-time = "2026-07-17T22:49:25.743Z" }, + { url = "https://files.pythonhosted.org/packages/1d/5b/14af3cd4ee69d8ea9baca58f3dc3cfb1ba78332a347fd478cb096549d60e/websockets-16.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2a636ff1e7a5c4edf71ef0e79adae7f25dba93b4fcbe3dc958733477ffeb0eaf", size = 187809, upload-time = "2026-07-17T22:49:27.147Z" }, + { url = "https://files.pythonhosted.org/packages/7b/11/be301710d70de97e3e7b3586e6d492c9c06d6a61bf1c2202c36cf0c75607/websockets-16.1.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:d6bec75c290fe484a8ba4cacdf838501e17c06ecfbbf31eede81a9e431bd7751", size = 186412, upload-time = "2026-07-17T22:49:28.611Z" }, + { url = "https://files.pythonhosted.org/packages/db/07/fe1435bf6fe738a3d3b54dbe0c18dabf12cba4d909ac8b58b539ce27c1f4/websockets-16.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:54509b8e92fee4453e152b7558ddef37ce9705a044922f2095a6105e3f80c96f", size = 188290, upload-time = "2026-07-17T22:49:29.965Z" }, + { url = "https://files.pythonhosted.org/packages/8a/0a/81f394aff8efcbb01208c1ced77df0a3c7fcce584a88c7273663697946c2/websockets-16.1.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:f0aa4aad3b1b69ad3fd85a0fd0952ec64331c762bd77ec51cc814170873890b2", size = 185844, upload-time = "2026-07-17T22:49:31.447Z" }, + { url = "https://files.pythonhosted.org/packages/39/5c/dd485b995473f415510251fe9bd708f2d24458f439fce958daf8d66dc7c6/websockets-16.1.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:42290eb6db4ccaca7012656738214f8514082fb6fa40cdeb61bb9a471b52e383", size = 186823, upload-time = "2026-07-17T22:49:33.104Z" }, + { url = "https://files.pythonhosted.org/packages/9d/0b/f78de76ff446f1e66af12b43c48a35f31744de93cfdec2f4ea67d5d7bbf1/websockets-16.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:53260c8930da5771cec89439bff99c20c8cb03ddb9588b980697355a83cd4bd3", size = 187102, upload-time = "2026-07-17T22:49:34.616Z" }, + { url = "https://files.pythonhosted.org/packages/37/a1/4cf892007778eaf84ad162bfc98046e0ed89b63ac55949e3236626b2a23f/websockets-16.1.1-cp312-cp312-win32.whl", hash = "sha256:1d27fa8462ad6a1cb36206a3d0640b2333340def181fae11ed7f9adeaa5c0747", size = 179943, upload-time = "2026-07-17T22:49:36.213Z" }, + { url = "https://files.pythonhosted.org/packages/d9/de/6abe251d28c3a3f217096575400b27750b18e0b1d2fff3a2a239960fea07/websockets-16.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:b436f6ec4fc3a6b4237c84d3f83170ed2b40bb584222f0ac47a0c8a5921980c7", size = 180243, upload-time = "2026-07-17T22:49:37.626Z" }, + { url = "https://files.pythonhosted.org/packages/ce/fd/6ec6c6d2850aea25b1b2aa9901a016980bb87d01e89b3eb00470b1b5d471/websockets-16.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ab59169ace05dcb49a1d4118f0bde139557adf45091bd85747e36bf5de984dd1", size = 179587, upload-time = "2026-07-17T22:49:38.959Z" }, + { url = "https://files.pythonhosted.org/packages/5f/d8/1d299d2dd34087db39831a34cc645ef8a6f89d78efada6983093513cd81c/websockets-16.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5e3b7d601f6f84156b08cc4a5e541c2b50ad7b36cfc302b657a12477c904a5df", size = 177272, upload-time = "2026-07-17T22:49:40.293Z" }, + { url = "https://files.pythonhosted.org/packages/3d/86/0a70d3ae2f0f2256bb41302d9804dbca65d4360281e7feb3e1f94102ac46/websockets-16.1.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cd2ca96a082a36964aca83e992f72abeb61b7306c1a6cba4c7d06a7b93750cac", size = 177530, upload-time = "2026-07-17T22:49:41.786Z" }, + { url = "https://files.pythonhosted.org/packages/b5/c2/c676c69444d9db448b3f0a55a98dcc534affce0bce961d9d2f0b8499b10a/websockets-16.1.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f5d497865f05bb222cab7016c6034542e84e5f29f49c6fd3f4939cda7197b5b8", size = 187197, upload-time = "2026-07-17T22:49:43.658Z" }, + { url = "https://files.pythonhosted.org/packages/0b/13/88137fbaf726ebe29d62c1117fa11fa2bbb6209dc79d4ad738efbe36a2aa/websockets-16.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bae954c382e013d5ea5b190d2830526bfa45ad121c326da0049b8c769f185db6", size = 188433, upload-time = "2026-07-17T22:49:45.147Z" }, + { url = "https://files.pythonhosted.org/packages/01/6d/46c2f2ce6751cb26f39293e1ecbf8544cb01321397cd476c2756b98c216d/websockets-16.1.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e09f753a169951eb4f28c2c774f71069304f66e7277e0f5a2892423599cfa854", size = 189868, upload-time = "2026-07-17T22:49:46.581Z" }, + { url = "https://files.pythonhosted.org/packages/29/2b/170a9e8097636cfde4dc3c592b6e00b18a44a2f5407606d96ca542dd5838/websockets-16.1.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:024193f8551a2b0eafbdd160911012c4e6c228c28430c84433253299a9e42d6a", size = 189059, upload-time = "2026-07-17T22:49:47.972Z" }, + { url = "https://files.pythonhosted.org/packages/a7/48/f0d4ebc9ab4b473b8861b9e20fdb663d515d42f7befdf62cdb60fee7a1ec/websockets-16.1.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:aabe464bfd13bd25f4821faf111da6fefdc389f870265a53105580e45b0a2e49", size = 187814, upload-time = "2026-07-17T22:49:49.344Z" }, + { url = "https://files.pythonhosted.org/packages/d5/ba/39a41d3ae8e72696a9492581900611c5a91e2b07563b0bcd2523adea9854/websockets-16.1.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a28fcbc9b6baf54a2e23f8655f308e4ccc6afdd7266f8fe7954f320dcda0f785", size = 185229, upload-time = "2026-07-17T22:49:50.787Z" }, + { url = "https://files.pythonhosted.org/packages/3c/36/ac15b604f850d1907f0a85ed721cefe47cd45034b3620069b829746cccbe/websockets-16.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:79eace538c6a97e96d0d03d4f9d314f9677f5ed85a8a984992ffd90b13cb8a56", size = 187874, upload-time = "2026-07-17T22:49:52.228Z" }, + { url = "https://files.pythonhosted.org/packages/a8/f3/3fbd5d71d59299c3770faa5884d4f45070236ca5a35ab3a61830812c409a/websockets-16.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:496af849a472b531f758dbd4d61338f5000538cb1a7b3d20d9d32a264517f509", size = 186469, upload-time = "2026-07-17T22:49:53.776Z" }, + { url = "https://files.pythonhosted.org/packages/b4/fc/dd90349bba58af2a53ef2ddd9c32716c81eb6d59a0687939fff561860878/websockets-16.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5283810d2646741a0d8da2aa733d6aefa0545809afccb2a5d105a26bc45125f1", size = 188347, upload-time = "2026-07-17T22:49:55.202Z" }, + { url = "https://files.pythonhosted.org/packages/4c/f3/f73ba86427682da59b78c11d77ba56d5b801c32e84afe79b274bbd6a9bb2/websockets-16.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:4e3b680b1e0a27457e727a0d572fd81dffa87b6dbf8b228ab57da64f7d85aead", size = 185903, upload-time = "2026-07-17T22:49:56.75Z" }, + { url = "https://files.pythonhosted.org/packages/34/7c/f95eb20e80104173b3a0a092291f89ea4047ef6e608e0a57ca06eb14eecb/websockets-16.1.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:69159730a823dde3ea8d08783e8d47ef135a6d7e8d44eb127e32b321c9db8e3e", size = 186855, upload-time = "2026-07-17T22:49:58.467Z" }, + { url = "https://files.pythonhosted.org/packages/b0/35/dd875b3e050ff232d60fa377707f890e369f74d134f1be32e8f68879747c/websockets-16.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ed5bb271084b46530ee2ddc0410537a9961152c5ccba2fc98c5276d992ccba87", size = 187140, upload-time = "2026-07-17T22:50:00.016Z" }, + { url = "https://files.pythonhosted.org/packages/e8/dc/5cbfcb41824502f6af93b8f3943a4d06c67c23c7d2e31eb18748c4a5b2a7/websockets-16.1.1-cp313-cp313-win32.whl", hash = "sha256:cfb70b4eb56cac4da0a83588f3ad50d46beb0690391082f3d4e2d488c70b68ea", size = 179928, upload-time = "2026-07-17T22:50:01.685Z" }, + { url = "https://files.pythonhosted.org/packages/b0/c1/71e5deb5b7f8f226997ab64908c184ac3105c0155ce2d486f318e5dd08a8/websockets-16.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:d9531d9cbeac99af6f038fb1bc351403531f7d634a2c2e10e2f7c854c6ed5b68", size = 180242, upload-time = "2026-07-17T22:50:03.117Z" }, + { url = "https://files.pythonhosted.org/packages/73/a2/ba78a164eeea4620df4a4df4bd2ed6017438c4655cc0f36f2c0bc0432355/websockets-16.1.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:443aefe96b7fdb132e2a70806cca1f2af49bb3f28e47abcd7c2e9dcf4d8fa1b8", size = 179635, upload-time = "2026-07-17T22:50:05.001Z" }, + { url = "https://files.pythonhosted.org/packages/b9/08/d26d7a7628cd4ac34cbbdb63ac80914ca842ed8e42938c40a53567806df3/websockets-16.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:6456ff333092d509127d75a638cb411afae8ff17f092635015d1902efec8a293", size = 177320, upload-time = "2026-07-17T22:50:06.427Z" }, + { url = "https://files.pythonhosted.org/packages/0f/45/ebec83e6269536aa5932533c67b0af5c781f3e73fdbcd68672dcf43f4f44/websockets-16.1.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:fce6c48559c86d1ac3632ccb1bebc7d5442fbe79bd9bb0e40379ee54be2a4051", size = 177544, upload-time = "2026-07-17T22:50:07.834Z" }, + { url = "https://files.pythonhosted.org/packages/c9/d5/abc614d2297f6c1c3e01e61260364457a47c25cc1cf6a879038902bc6aa8/websockets-16.1.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:92b820d345f7a3fc7b8163949ee92df910f290c3fc517b3d5301c78065adafe1", size = 187270, upload-time = "2026-07-17T22:50:09.275Z" }, + { url = "https://files.pythonhosted.org/packages/52/71/4c99af3b87dff1b2927981f6876607d4acb45338c665242168d3982f7758/websockets-16.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2a606d9c24035242a3e256e9d5b77ed9cd6bccfcb7cf993e5ca3c0f6f68fb6a7", size = 188509, upload-time = "2026-07-17T22:50:10.722Z" }, + { url = "https://files.pythonhosted.org/packages/9b/b4/5c8ca14b0df7eb84ed0524165c5359150210140817a3312aee57bf62a1cf/websockets-16.1.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:414e596c75f74e0994084694189d7dc9229fb278e33064d6784b73ffbba3ca31", size = 189882, upload-time = "2026-07-17T22:50:12.293Z" }, + { url = "https://files.pythonhosted.org/packages/25/c1/bedfba9e70557129cb8083748d167bdcc01483dedf0f0df143676df05cbe/websockets-16.1.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:536676848fc5961aca9d20389951f59169508f765637a172403dc5434d722fa0", size = 189114, upload-time = "2026-07-17T22:50:13.789Z" }, + { url = "https://files.pythonhosted.org/packages/df/09/aa835b2787835aebd839114be5de51b797cb480b63ba42b26d34dfe147cb/websockets-16.1.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:97fd3a0e8b53efa41970ac1dff3d8cf0d2884cadeb4caaf95db7ad1526926ee3", size = 187861, upload-time = "2026-07-17T22:50:15.179Z" }, + { url = "https://files.pythonhosted.org/packages/20/26/f6408330694dbc9830857d9d23bc14ac4f6875127a480cfdda8d5ca21198/websockets-16.1.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7b1b19636af86a3c7995d4d028dbe376f39b4bf31541146f9c123582a6c94562", size = 185286, upload-time = "2026-07-17T22:50:16.741Z" }, + { url = "https://files.pythonhosted.org/packages/17/9a/e0675e70dd8a80762cf35bb18799d3f290a4890ffe6439bc51d222796083/websockets-16.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:41c8e77f17294c0ac18008a7309b99b34ee72247ef10b6dff4c3f8b5ac29896b", size = 187935, upload-time = "2026-07-17T22:50:18.213Z" }, + { url = "https://files.pythonhosted.org/packages/33/c1/3234cfb86afde01b81e9bddcc6e534c440975d60a13991259e833069ab3e/websockets-16.1.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:9f63bcef7f4b02b06b35fc01c93b96c43b5e88e1e8868676caacf493d5a31f3a", size = 186444, upload-time = "2026-07-17T22:50:19.67Z" }, + { url = "https://files.pythonhosted.org/packages/89/87/9c15206e1d778923d8daa9657de07aa62ea815e13448319c98458c37b281/websockets-16.1.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:dab9eb87869da2d6ed3af3f3adf28414baae6ec9d4df355ffc18889132f3436c", size = 188409, upload-time = "2026-07-17T22:50:21.28Z" }, + { url = "https://files.pythonhosted.org/packages/f2/00/cf5de5c67676de2d3eef8b2a518f168f6796595447a5b7161ba0d012915c/websockets-16.1.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:43e3a9fdd7cbf7ba6040c31fae0faf84ca1474fef777c4e37912f1540f854499", size = 185958, upload-time = "2026-07-17T22:50:22.719Z" }, + { url = "https://files.pythonhosted.org/packages/62/c0/731b6ddede2e4136912ec4cff2cffbda35af73546be4762c3d7bd3bd79af/websockets-16.1.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:056ae37939ed7e9974f364f5864e76e49182622d8f9751ac1903c0d09b013985", size = 186911, upload-time = "2026-07-17T22:50:24.108Z" }, + { url = "https://files.pythonhosted.org/packages/8c/7f/39c634472c4469a24a7c09cecddffb08fac6d0e74f73881a94ee8a40a196/websockets-16.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a0eadbbf2c30f01efa58e1f110eb6fa293261f6b0b1aa38f7f48707107690af9", size = 187204, upload-time = "2026-07-17T22:50:25.548Z" }, + { url = "https://files.pythonhosted.org/packages/26/89/9667c256c256dafcc62d21328ce7a40067da857969b68ee9af375b0aaf72/websockets-16.1.1-cp314-cp314-win32.whl", hash = "sha256:195c978b065fa40910582464f99d6b15c8b314c68e0546549a55ed83f4735328", size = 179603, upload-time = "2026-07-17T22:50:27.086Z" }, + { url = "https://files.pythonhosted.org/packages/bd/dd/1c099d6c0fc5deb6b46ccdbb6981fdb4b12c917869cb3952408409dc18db/websockets-16.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:4e8d01cc3bcae7bbf8167f944aeafefed590fae5693552bba9794a9df68371cc", size = 179948, upload-time = "2026-07-17T22:50:28.521Z" }, + { url = "https://files.pythonhosted.org/packages/35/25/9956b2d5e0529d5d23924f21bba1440d4c5c88a562e4f08550871ffa97a7/websockets-16.1.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:0ffd3031ea8bda8d61762e84220186105ba3b748b3c8da2ae4f7816fac03e573", size = 179963, upload-time = "2026-07-17T22:50:29.982Z" }, + { url = "https://files.pythonhosted.org/packages/17/06/55ffc976c488b6aee9ea05761ff7c4e88e7c1fd82818c8ca7b556ad2f90c/websockets-16.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:84a2cef8deffbd9ab8ee0ea546a2a6a7030c28f44e6cdd4547dbfeb489eb8999", size = 177497, upload-time = "2026-07-17T22:50:31.396Z" }, + { url = "https://files.pythonhosted.org/packages/0c/e8/f7dac2e980bacc92bdc26cebae4ae4d50cae5380732c50980598fc0bbae4/websockets-16.1.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:3df13f73af9b3b38ab1195eb299ecb67a4330c911c97ae04043ff74085728abe", size = 177698, upload-time = "2026-07-17T22:50:32.829Z" }, + { url = "https://files.pythonhosted.org/packages/b2/39/26762f734113e22da2b942c3aca85798e0c0405d64c256549540ff31e5a1/websockets-16.1.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:23253dd5bcae3f9aaee0a1d30967a8dbd52e5d3cff93a2e5b84df57b77d4750d", size = 187561, upload-time = "2026-07-17T22:50:34.24Z" }, + { url = "https://files.pythonhosted.org/packages/11/94/c3f330851806b9b02138b774d593478323e73c99238681b4b93efe64e02d/websockets-16.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9c1c5705e314449e3308872fe084b8571ce078ee4fc55a98a769bdefe5917392", size = 188732, upload-time = "2026-07-17T22:50:36.088Z" }, + { url = "https://files.pythonhosted.org/packages/d1/f2/eb2c450f052de334ae33cf200ece6e87b0e14d186807074e4eb1cd2cdea2/websockets-16.1.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:69e52d175a0a7d1e13b4b67ad41c560b7d98e8c6f6126eb0bda496c784faf8c7", size = 190872, upload-time = "2026-07-17T22:50:38.008Z" }, + { url = "https://files.pythonhosted.org/packages/70/31/2ac8cecf3a74f7fed9132129fc3d90b3998a1554570c11a69b2a8c20332d/websockets-16.1.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1f79c89b5eb034d1722938a891916582f8f7f503f58ca22518a63c3f2cd18499", size = 189305, upload-time = "2026-07-17T22:50:39.53Z" }, + { url = "https://files.pythonhosted.org/packages/6a/cf/8ab19650d3c0d4562c92e70ab47c257c4aa5c6a713ed87fe63766b31fefc/websockets-16.1.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:39f2a024af5c345ffe8fcf1ee18c049c024c94df393bb09b044a6917c77bde43", size = 188033, upload-time = "2026-07-17T22:50:40.912Z" }, + { url = "https://files.pythonhosted.org/packages/66/d7/a49a38a6127a4acb134fb1912b215d900cc657605cff32445bf519f3acc4/websockets-16.1.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:952303a7318d4cbe1011400839bb2051c9f84fa0a35923267f5daba34b15d458", size = 185748, upload-time = "2026-07-17T22:50:42.559Z" }, + { url = "https://files.pythonhosted.org/packages/95/3e/ad1fa40388c7f2e0bb2c7930d0090b6c5498594bd1cdaec18864df3d9e97/websockets-16.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:249116b4a76063d930a46391ad56e135c286e4562a18309029fc2c73f4ed4c62", size = 188285, upload-time = "2026-07-17T22:50:43.974Z" }, + { url = "https://files.pythonhosted.org/packages/35/b8/d5db28ca264b9104f82196f92dc8843e35fd391f763d42e4ad358f5bc97e/websockets-16.1.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:61922544a0587a13fd3f53e4c0e5e606510c7b0d9d22c8444e5fae22a06b38cb", size = 186777, upload-time = "2026-07-17T22:50:45.474Z" }, + { url = "https://files.pythonhosted.org/packages/42/9c/726cb39d0cc43ae848dce4aa2acb04eecc6738b1264ec6d700bf6bcfb9f8/websockets-16.1.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:46dcaa042cd1de6c59e7d9269fa63ff7572b6df40510600b678f0826b3c7af51", size = 188682, upload-time = "2026-07-17T22:50:46.973Z" }, + { url = "https://files.pythonhosted.org/packages/be/c7/1168704de8c2dd483edabe4a22cbe4465dd8be8dd95561d214f9fe092871/websockets-16.1.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:38565aca3e01ea8734e578fb2118dade0ecb0250533f29e22b8d1a7a196cf4d0", size = 186377, upload-time = "2026-07-17T22:50:48.413Z" }, + { url = "https://files.pythonhosted.org/packages/ca/40/f9ff2d630ffce4e7dfea0b2288e1caf9ebbf9ff8a9ec9396136ce8b94935/websockets-16.1.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:42f599f4d48c7e1a3338fdaac3acd075be3b3cf02d4b274f3bf2767aedd3d217", size = 187148, upload-time = "2026-07-17T22:50:49.845Z" }, + { url = "https://files.pythonhosted.org/packages/b5/71/e177c8299f78d7cbe2d14df228643c10c70c0e86e108e092056bbcc16e46/websockets-16.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:dcc04fedf83effaeb9cce98abc9469bb1b42ef85f03e01c8c1f4438ef7555737", size = 187578, upload-time = "2026-07-17T22:50:51.619Z" }, + { url = "https://files.pythonhosted.org/packages/49/b2/b6987faf330f5af5c787a2610124c2e8403d51724f9001ec4fff6311fe7a/websockets-16.1.1-cp314-cp314t-win32.whl", hash = "sha256:8483c2096363120eea8b07c06ae7304d520f686665fffd4811fad423930a65d7", size = 179729, upload-time = "2026-07-17T22:50:53.269Z" }, + { url = "https://files.pythonhosted.org/packages/a2/6e/fbac6ed878dd362fbad7d415fa4f84d38e3e33fed8cde45c64e783acf826/websockets-16.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:bcce07e23e5769375158f5efdcdafa8d5cd014b93c6683865b840ed65b96f231", size = 180072, upload-time = "2026-07-17T22:50:54.969Z" }, + { url = "https://files.pythonhosted.org/packages/e1/ed/71fea6e141590cafc40b14dc5943b0845606bee87bdb52a21b6a73eb4311/websockets-16.1.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:820fb8450edddae3812fd58cbc08e2bf22812cb248ecb5f06dbb82119a56e869", size = 177185, upload-time = "2026-07-17T22:50:56.665Z" }, + { url = "https://files.pythonhosted.org/packages/01/ec/00e7eeca200facf9266a83e4cbbf1bed0e67fba1d4d45031d3e5b3d81b5c/websockets-16.1.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:125f22dbefaf1554fea66fc83851490edb284ce4f501d37ffed2752f418332d9", size = 177459, upload-time = "2026-07-17T22:50:58.197Z" }, + { url = "https://files.pythonhosted.org/packages/75/fd/5774c4b33f7c0d8f0c51809c8b3a93456c48e3543579262cfa64eb5f522e/websockets-16.1.1-pp311-pypy311_pp73-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:30bbe120437b5648a77d3519b7024ea09530e0b5b18d3698c5a0ae536fe0cc2e", size = 178294, upload-time = "2026-07-17T22:50:59.641Z" }, + { url = "https://files.pythonhosted.org/packages/37/c3/48e2c03d2bd79bb45948841c592d24156312dd5f58cdf8f549febe652fb6/websockets-16.1.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b6b9dadbef0cccd9f4c4ee96b08898afa73e26803bbe0f6aeb5bb12b0074206d", size = 179190, upload-time = "2026-07-17T22:51:01.129Z" }, + { url = "https://files.pythonhosted.org/packages/2d/3f/73e511ecf2496ceac57dd4ed8388efe2bcf0769338a2dbf242c8366ae87e/websockets-16.1.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:56cd5fc4f10a9ea8aa0804bddb7b42506cf9e136046f3b4c27de8fec9e2ecba5", size = 180330, upload-time = "2026-07-17T22:51:02.603Z" }, + { url = "https://files.pythonhosted.org/packages/be/4d/2d0d67834092e354d2b0498f014a41249a89556bc406cf86f3e1557bb463/websockets-16.1.1-py3-none-any.whl", hash = "sha256:6abbd3e82c731c8e531714466acd5d87b5e88ac3243465337ba71d68e23ae7e3", size = 173814, upload-time = "2026-07-17T22:51:04.184Z" }, +] + +[[package]] +name = "yarl" +version = "1.23.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "multidict" }, + { name = "propcache" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/23/6e/beb1beec874a72f23815c1434518bfc4ed2175065173fb138c3705f658d4/yarl-1.23.0.tar.gz", hash = "sha256:53b1ea6ca88ebd4420379c330aea57e258408dd0df9af0992e5de2078dc9f5d5", size = 194676, upload-time = "2026-03-01T22:07:53.373Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/aa/60da938b8f0997ba3a911263c40d82b6f645a67902a490b46f3355e10fae/yarl-1.23.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:b35d13d549077713e4414f927cdc388d62e543987c572baee613bf82f11a4b99", size = 123641, upload-time = "2026-03-01T22:04:42.841Z" }, + { url = "https://files.pythonhosted.org/packages/24/84/e237607faf4e099dbb8a4f511cfd5efcb5f75918baad200ff7380635631b/yarl-1.23.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:cbb0fef01f0c6b38cb0f39b1f78fc90b807e0e3c86a7ff3ce74ad77ce5c7880c", size = 86248, upload-time = "2026-03-01T22:04:44.757Z" }, + { url = "https://files.pythonhosted.org/packages/b2/0d/71ceabc14c146ba8ee3804ca7b3d42b1664c8440439de5214d366fec7d3a/yarl-1.23.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:dc52310451fc7c629e13c4e061cbe2dd01684d91f2f8ee2821b083c58bd72432", size = 85988, upload-time = "2026-03-01T22:04:46.365Z" }, + { url = "https://files.pythonhosted.org/packages/8c/6c/4a90d59c572e46b270ca132aca66954f1175abd691f74c1ef4c6711828e2/yarl-1.23.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b2c6b50c7b0464165472b56b42d4c76a7b864597007d9c085e8b63e185cf4a7a", size = 100566, upload-time = "2026-03-01T22:04:47.639Z" }, + { url = "https://files.pythonhosted.org/packages/49/fb/c438fb5108047e629f6282a371e6e91cf3f97ee087c4fb748a1f32ceef55/yarl-1.23.0-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:aafe5dcfda86c8af00386d7781d4c2181b5011b7be3f2add5e99899ea925df05", size = 92079, upload-time = "2026-03-01T22:04:48.925Z" }, + { url = "https://files.pythonhosted.org/packages/d9/13/d269aa1aed3e4f50a5a103f96327210cc5fa5dd2d50882778f13c7a14606/yarl-1.23.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9ee33b875f0b390564c1fb7bc528abf18c8ee6073b201c6ae8524aca778e2d83", size = 108741, upload-time = "2026-03-01T22:04:50.838Z" }, + { url = "https://files.pythonhosted.org/packages/85/fb/115b16f22c37ea4437d323e472945bea97301c8ec6089868fa560abab590/yarl-1.23.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4c41e021bc6d7affb3364dc1e1e5fa9582b470f283748784bd6ea0558f87f42c", size = 108099, upload-time = "2026-03-01T22:04:52.499Z" }, + { url = "https://files.pythonhosted.org/packages/9a/64/c53487d9f4968045b8afa51aed7ca44f58b2589e772f32745f3744476c82/yarl-1.23.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:99c8a9ed30f4164bc4c14b37a90208836cbf50d4ce2a57c71d0f52c7fb4f7598", size = 102678, upload-time = "2026-03-01T22:04:55.176Z" }, + { url = "https://files.pythonhosted.org/packages/85/59/cd98e556fbb2bf8fab29c1a722f67ad45c5f3447cac798ab85620d1e70af/yarl-1.23.0-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f2af5c81a1f124609d5f33507082fc3f739959d4719b56877ab1ee7e7b3d602b", size = 100803, upload-time = "2026-03-01T22:04:56.588Z" }, + { url = "https://files.pythonhosted.org/packages/9e/c0/b39770b56d4a9f0bb5f77e2f1763cd2d75cc2f6c0131e3b4c360348fcd65/yarl-1.23.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6b41389c19b07c760c7e427a3462e8ab83c4bb087d127f0e854c706ce1b9215c", size = 100163, upload-time = "2026-03-01T22:04:58.492Z" }, + { url = "https://files.pythonhosted.org/packages/e7/64/6980f99ab00e1f0ff67cb84766c93d595b067eed07439cfccfc8fb28c1a6/yarl-1.23.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:1dc702e42d0684f42d6519c8d581e49c96cefaaab16691f03566d30658ee8788", size = 93859, upload-time = "2026-03-01T22:05:00.268Z" }, + { url = "https://files.pythonhosted.org/packages/38/69/912e6c5e146793e5d4b5fe39ff5b00f4d22463dfd5a162bec565ac757673/yarl-1.23.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:0e40111274f340d32ebcc0a5668d54d2b552a6cca84c9475859d364b380e3222", size = 108202, upload-time = "2026-03-01T22:05:02.273Z" }, + { url = "https://files.pythonhosted.org/packages/59/97/35ca6767524687ad64e5f5c31ad54bc76d585585a9fcb40f649e7e82ffed/yarl-1.23.0-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:4764a6a7588561a9aef92f65bda2c4fb58fe7c675c0883862e6df97559de0bfb", size = 99866, upload-time = "2026-03-01T22:05:03.597Z" }, + { url = "https://files.pythonhosted.org/packages/d3/1c/1a3387ee6d73589f6f2a220ae06f2984f6c20b40c734989b0a44f5987308/yarl-1.23.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:03214408cfa590df47728b84c679ae4ef00be2428e11630277be0727eba2d7cc", size = 107852, upload-time = "2026-03-01T22:05:04.986Z" }, + { url = "https://files.pythonhosted.org/packages/a4/b8/35c0750fcd5a3f781058bfd954515dd4b1eab45e218cbb85cf11132215f1/yarl-1.23.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:170e26584b060879e29fac213e4228ef063f39128723807a312e5c7fec28eff2", size = 102919, upload-time = "2026-03-01T22:05:06.397Z" }, + { url = "https://files.pythonhosted.org/packages/e5/1c/9a1979aec4a81896d597bcb2177827f2dbee3f5b7cc48b2d0dadb644b41d/yarl-1.23.0-cp311-cp311-win32.whl", hash = "sha256:51430653db848d258336cfa0244427b17d12db63d42603a55f0d4546f50f25b5", size = 82602, upload-time = "2026-03-01T22:05:08.444Z" }, + { url = "https://files.pythonhosted.org/packages/93/22/b85eca6fa2ad9491af48c973e4c8cf6b103a73dbb271fe3346949449fca0/yarl-1.23.0-cp311-cp311-win_amd64.whl", hash = "sha256:bf49a3ae946a87083ef3a34c8f677ae4243f5b824bfc4c69672e72b3d6719d46", size = 87461, upload-time = "2026-03-01T22:05:10.145Z" }, + { url = "https://files.pythonhosted.org/packages/93/95/07e3553fe6f113e6864a20bdc53a78113cda3b9ced8784ee52a52c9f80d8/yarl-1.23.0-cp311-cp311-win_arm64.whl", hash = "sha256:b39cb32a6582750b6cc77bfb3c49c0f8760dc18dc96ec9fb55fbb0f04e08b928", size = 82336, upload-time = "2026-03-01T22:05:11.554Z" }, + { url = "https://files.pythonhosted.org/packages/88/8a/94615bc31022f711add374097ad4144d569e95ff3c38d39215d07ac153a0/yarl-1.23.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:1932b6b8bba8d0160a9d1078aae5838a66039e8832d41d2992daa9a3a08f7860", size = 124737, upload-time = "2026-03-01T22:05:12.897Z" }, + { url = "https://files.pythonhosted.org/packages/e3/6f/c6554045d59d64052698add01226bc867b52fe4a12373415d7991fdca95d/yarl-1.23.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:411225bae281f114067578891bc75534cfb3d92a3b4dfef7a6ca78ba354e6069", size = 87029, upload-time = "2026-03-01T22:05:14.376Z" }, + { url = "https://files.pythonhosted.org/packages/19/2a/725ecc166d53438bc88f76822ed4b1e3b10756e790bafd7b523fe97c322d/yarl-1.23.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:13a563739ae600a631c36ce096615fe307f131344588b0bc0daec108cdb47b25", size = 86310, upload-time = "2026-03-01T22:05:15.71Z" }, + { url = "https://files.pythonhosted.org/packages/99/30/58260ed98e6ff7f90ba84442c1ddd758c9170d70327394a6227b310cd60f/yarl-1.23.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9cbf44c5cb4a7633d078788e1b56387e3d3cf2b8139a3be38040b22d6c3221c8", size = 97587, upload-time = "2026-03-01T22:05:17.384Z" }, + { url = "https://files.pythonhosted.org/packages/76/0a/8b08aac08b50682e65759f7f8dde98ae8168f72487e7357a5d684c581ef9/yarl-1.23.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:53ad387048f6f09a8969631e4de3f1bf70c50e93545d64af4f751b2498755072", size = 92528, upload-time = "2026-03-01T22:05:18.804Z" }, + { url = "https://files.pythonhosted.org/packages/52/07/0b7179101fe5f8385ec6c6bb5d0cb9f76bd9fb4a769591ab6fb5cdbfc69a/yarl-1.23.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4a59ba56f340334766f3a4442e0efd0af895fae9e2b204741ef885c446b3a1a8", size = 105339, upload-time = "2026-03-01T22:05:20.235Z" }, + { url = "https://files.pythonhosted.org/packages/d3/8a/36d82869ab5ec829ca8574dfcb92b51286fcfb1e9c7a73659616362dc880/yarl-1.23.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:803a3c3ce4acc62eaf01eaca1208dcf0783025ef27572c3336502b9c232005e7", size = 105061, upload-time = "2026-03-01T22:05:22.268Z" }, + { url = "https://files.pythonhosted.org/packages/66/3e/868e5c3364b6cee19ff3e1a122194fa4ce51def02c61023970442162859e/yarl-1.23.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a3d2bff8f37f8d0f96c7ec554d16945050d54462d6e95414babaa18bfafc7f51", size = 100132, upload-time = "2026-03-01T22:05:23.638Z" }, + { url = "https://files.pythonhosted.org/packages/cf/26/9c89acf82f08a52cb52d6d39454f8d18af15f9d386a23795389d1d423823/yarl-1.23.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c75eb09e8d55bceb4367e83496ff8ef2bc7ea6960efb38e978e8073ea59ecb67", size = 99289, upload-time = "2026-03-01T22:05:25.749Z" }, + { url = "https://files.pythonhosted.org/packages/6f/54/5b0db00d2cb056922356104468019c0a132e89c8d3ab67d8ede9f4483d2a/yarl-1.23.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:877b0738624280e34c55680d6054a307aa94f7d52fa0e3034a9cc6e790871da7", size = 96950, upload-time = "2026-03-01T22:05:27.318Z" }, + { url = "https://files.pythonhosted.org/packages/f6/40/10fa93811fd439341fad7e0718a86aca0de9548023bbb403668d6555acab/yarl-1.23.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b5405bb8f0e783a988172993cfc627e4d9d00432d6bbac65a923041edacf997d", size = 93960, upload-time = "2026-03-01T22:05:28.738Z" }, + { url = "https://files.pythonhosted.org/packages/bc/d2/8ae2e6cd77d0805f4526e30ec43b6f9a3dfc542d401ac4990d178e4bf0cf/yarl-1.23.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1c3a3598a832590c5a3ce56ab5576361b5688c12cb1d39429cf5dba30b510760", size = 104703, upload-time = "2026-03-01T22:05:30.438Z" }, + { url = "https://files.pythonhosted.org/packages/2f/0c/b3ceacf82c3fe21183ce35fa2acf5320af003d52bc1fcf5915077681142e/yarl-1.23.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:8419ebd326430d1cbb7efb5292330a2cf39114e82df5cc3d83c9a0d5ebeaf2f2", size = 98325, upload-time = "2026-03-01T22:05:31.835Z" }, + { url = "https://files.pythonhosted.org/packages/9d/e0/12900edd28bdab91a69bd2554b85ad7b151f64e8b521fe16f9ad2f56477a/yarl-1.23.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:be61f6fff406ca40e3b1d84716fde398fc08bc63dd96d15f3a14230a0973ed86", size = 105067, upload-time = "2026-03-01T22:05:33.358Z" }, + { url = "https://files.pythonhosted.org/packages/15/61/74bb1182cf79c9bbe4eb6b1f14a57a22d7a0be5e9cedf8e2d5c2086474c3/yarl-1.23.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3ceb13c5c858d01321b5d9bb65e4cf37a92169ea470b70fec6f236b2c9dd7e34", size = 100285, upload-time = "2026-03-01T22:05:35.4Z" }, + { url = "https://files.pythonhosted.org/packages/69/7f/cd5ef733f2550de6241bd8bd8c3febc78158b9d75f197d9c7baa113436af/yarl-1.23.0-cp312-cp312-win32.whl", hash = "sha256:fffc45637bcd6538de8b85f51e3df3223e4ad89bccbfca0481c08c7fc8b7ed7d", size = 82359, upload-time = "2026-03-01T22:05:36.811Z" }, + { url = "https://files.pythonhosted.org/packages/f5/be/25216a49daeeb7af2bec0db22d5e7df08ed1d7c9f65d78b14f3b74fd72fc/yarl-1.23.0-cp312-cp312-win_amd64.whl", hash = "sha256:f69f57305656a4852f2a7203efc661d8c042e6cc67f7acd97d8667fb448a426e", size = 87674, upload-time = "2026-03-01T22:05:38.171Z" }, + { url = "https://files.pythonhosted.org/packages/d2/35/aeab955d6c425b227d5b7247eafb24f2653fedc32f95373a001af5dfeb9e/yarl-1.23.0-cp312-cp312-win_arm64.whl", hash = "sha256:6e87a6e8735b44816e7db0b2fbc9686932df473c826b0d9743148432e10bb9b9", size = 81879, upload-time = "2026-03-01T22:05:40.006Z" }, + { url = "https://files.pythonhosted.org/packages/9a/4b/a0a6e5d0ee8a2f3a373ddef8a4097d74ac901ac363eea1440464ccbe0898/yarl-1.23.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:16c6994ac35c3e74fb0ae93323bf8b9c2a9088d55946109489667c510a7d010e", size = 123796, upload-time = "2026-03-01T22:05:41.412Z" }, + { url = "https://files.pythonhosted.org/packages/67/b6/8925d68af039b835ae876db5838e82e76ec87b9782ecc97e192b809c4831/yarl-1.23.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4a42e651629dafb64fd5b0286a3580613702b5809ad3f24934ea87595804f2c5", size = 86547, upload-time = "2026-03-01T22:05:42.841Z" }, + { url = "https://files.pythonhosted.org/packages/ae/50/06d511cc4b8e0360d3c94af051a768e84b755c5eb031b12adaaab6dec6e5/yarl-1.23.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7c6b9461a2a8b47c65eef63bb1c76a4f1c119618ffa99ea79bc5bb1e46c5821b", size = 85854, upload-time = "2026-03-01T22:05:44.85Z" }, + { url = "https://files.pythonhosted.org/packages/c4/f4/4e30b250927ffdab4db70da08b9b8d2194d7c7b400167b8fbeca1e4701ca/yarl-1.23.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2569b67d616eab450d262ca7cb9f9e19d2f718c70a8b88712859359d0ab17035", size = 98351, upload-time = "2026-03-01T22:05:46.836Z" }, + { url = "https://files.pythonhosted.org/packages/86/fc/4118c5671ea948208bdb1492d8b76bdf1453d3e73df051f939f563e7dcc5/yarl-1.23.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e9d9a4d06d3481eab79803beb4d9bd6f6a8e781ec078ac70d7ef2dcc29d1bea5", size = 92711, upload-time = "2026-03-01T22:05:48.316Z" }, + { url = "https://files.pythonhosted.org/packages/56/11/1ed91d42bd9e73c13dc9e7eb0dd92298d75e7ac4dd7f046ad0c472e231cd/yarl-1.23.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f514f6474e04179d3d33175ed3f3e31434d3130d42ec153540d5b157deefd735", size = 106014, upload-time = "2026-03-01T22:05:50.028Z" }, + { url = "https://files.pythonhosted.org/packages/ce/c9/74e44e056a23fbc33aca71779ef450ca648a5bc472bdad7a82339918f818/yarl-1.23.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fda207c815b253e34f7e1909840fd14299567b1c0eb4908f8c2ce01a41265401", size = 105557, upload-time = "2026-03-01T22:05:51.416Z" }, + { url = "https://files.pythonhosted.org/packages/66/fe/b1e10b08d287f518994f1e2ff9b6d26f0adeecd8dd7d533b01bab29a3eda/yarl-1.23.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:34b6cf500e61c90f305094911f9acc9c86da1a05a7a3f5be9f68817043f486e4", size = 101559, upload-time = "2026-03-01T22:05:52.872Z" }, + { url = "https://files.pythonhosted.org/packages/72/59/c5b8d94b14e3d3c2a9c20cb100119fd534ab5a14b93673ab4cc4a4141ea5/yarl-1.23.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d7504f2b476d21653e4d143f44a175f7f751cd41233525312696c76aa3dbb23f", size = 100502, upload-time = "2026-03-01T22:05:54.954Z" }, + { url = "https://files.pythonhosted.org/packages/77/4f/96976cb54cbfc5c9fd73ed4c51804f92f209481d1fb190981c0f8a07a1d7/yarl-1.23.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:578110dd426f0d209d1509244e6d4a3f1a3e9077655d98c5f22583d63252a08a", size = 98027, upload-time = "2026-03-01T22:05:56.409Z" }, + { url = "https://files.pythonhosted.org/packages/63/6e/904c4f476471afdbad6b7e5b70362fb5810e35cd7466529a97322b6f5556/yarl-1.23.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:609d3614d78d74ebe35f54953c5bbd2ac647a7ddb9c30a5d877580f5e86b22f2", size = 95369, upload-time = "2026-03-01T22:05:58.141Z" }, + { url = "https://files.pythonhosted.org/packages/9d/40/acfcdb3b5f9d68ef499e39e04d25e141fe90661f9d54114556cf83be8353/yarl-1.23.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4966242ec68afc74c122f8459abd597afd7d8a60dc93d695c1334c5fd25f762f", size = 105565, upload-time = "2026-03-01T22:06:00.286Z" }, + { url = "https://files.pythonhosted.org/packages/5e/c6/31e28f3a6ba2869c43d124f37ea5260cac9c9281df803c354b31f4dd1f3c/yarl-1.23.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:e0fd068364a6759bc794459f0a735ab151d11304346332489c7972bacbe9e72b", size = 99813, upload-time = "2026-03-01T22:06:01.712Z" }, + { url = "https://files.pythonhosted.org/packages/08/1f/6f65f59e72d54aa467119b63fc0b0b1762eff0232db1f4720cd89e2f4a17/yarl-1.23.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:39004f0ad156da43e86aa71f44e033de68a44e5a31fc53507b36dd253970054a", size = 105632, upload-time = "2026-03-01T22:06:03.188Z" }, + { url = "https://files.pythonhosted.org/packages/a3/c4/18b178a69935f9e7a338127d5b77d868fdc0f0e49becd286d51b3a18c61d/yarl-1.23.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e5723c01a56c5028c807c701aa66722916d2747ad737a046853f6c46f4875543", size = 101895, upload-time = "2026-03-01T22:06:04.651Z" }, + { url = "https://files.pythonhosted.org/packages/8f/54/f5b870b5505663911dba950a8e4776a0dbd51c9c54c0ae88e823e4b874a0/yarl-1.23.0-cp313-cp313-win32.whl", hash = "sha256:1b6b572edd95b4fa8df75de10b04bc81acc87c1c7d16bcdd2035b09d30acc957", size = 82356, upload-time = "2026-03-01T22:06:06.04Z" }, + { url = "https://files.pythonhosted.org/packages/7a/84/266e8da36879c6edcd37b02b547e2d9ecdfea776be49598e75696e3316e1/yarl-1.23.0-cp313-cp313-win_amd64.whl", hash = "sha256:baaf55442359053c7d62f6f8413a62adba3205119bcb6f49594894d8be47e5e3", size = 87515, upload-time = "2026-03-01T22:06:08.107Z" }, + { url = "https://files.pythonhosted.org/packages/00/fd/7e1c66efad35e1649114fa13f17485f62881ad58edeeb7f49f8c5e748bf9/yarl-1.23.0-cp313-cp313-win_arm64.whl", hash = "sha256:fb4948814a2a98e3912505f09c9e7493b1506226afb1f881825368d6fb776ee3", size = 81785, upload-time = "2026-03-01T22:06:10.181Z" }, + { url = "https://files.pythonhosted.org/packages/9c/fc/119dd07004f17ea43bb91e3ece6587759edd7519d6b086d16bfbd3319982/yarl-1.23.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:aecfed0b41aa72b7881712c65cf764e39ce2ec352324f5e0837c7048d9e6daaa", size = 130719, upload-time = "2026-03-01T22:06:11.708Z" }, + { url = "https://files.pythonhosted.org/packages/e6/0d/9f2348502fbb3af409e8f47730282cd6bc80dec6630c1e06374d882d6eb2/yarl-1.23.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:a41bcf68efd19073376eb8cf948b8d9be0af26256403e512bb18f3966f1f9120", size = 89690, upload-time = "2026-03-01T22:06:13.429Z" }, + { url = "https://files.pythonhosted.org/packages/50/93/e88f3c80971b42cfc83f50a51b9d165a1dbf154b97005f2994a79f212a07/yarl-1.23.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:cde9a2ecd91668bcb7f077c4966d8ceddb60af01b52e6e3e2680e4cf00ad1a59", size = 89851, upload-time = "2026-03-01T22:06:15.53Z" }, + { url = "https://files.pythonhosted.org/packages/1c/07/61c9dd8ba8f86473263b4036f70fb594c09e99c0d9737a799dfd8bc85651/yarl-1.23.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5023346c4ee7992febc0068e7593de5fa2bf611848c08404b35ebbb76b1b0512", size = 95874, upload-time = "2026-03-01T22:06:17.553Z" }, + { url = "https://files.pythonhosted.org/packages/9e/e9/f9ff8ceefba599eac6abddcfb0b3bee9b9e636e96dbf54342a8577252379/yarl-1.23.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:d1009abedb49ae95b136a8904a3f71b342f849ffeced2d3747bf29caeda218c4", size = 88710, upload-time = "2026-03-01T22:06:19.004Z" }, + { url = "https://files.pythonhosted.org/packages/eb/78/0231bfcc5d4c8eec220bc2f9ef82cb4566192ea867a7c5b4148f44f6cbcd/yarl-1.23.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a8d00f29b42f534cc8aa3931cfe773b13b23e561e10d2b26f27a8d309b0e82a1", size = 101033, upload-time = "2026-03-01T22:06:21.203Z" }, + { url = "https://files.pythonhosted.org/packages/cd/9b/30ea5239a61786f18fd25797151a17fbb3be176977187a48d541b5447dd4/yarl-1.23.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:95451e6ce06c3e104556d73b559f5da6c34a069b6b62946d3ad66afcd51642ea", size = 100817, upload-time = "2026-03-01T22:06:22.738Z" }, + { url = "https://files.pythonhosted.org/packages/62/e2/a4980481071791bc83bce2b7a1a1f7adcabfa366007518b4b845e92eeee3/yarl-1.23.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:531ef597132086b6cf96faa7c6c1dcd0361dd5f1694e5cc30375907b9b7d3ea9", size = 97482, upload-time = "2026-03-01T22:06:24.21Z" }, + { url = "https://files.pythonhosted.org/packages/e5/1e/304a00cf5f6100414c4b5a01fc7ff9ee724b62158a08df2f8170dfc72a2d/yarl-1.23.0-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:88f9fb0116fbfcefcab70f85cf4b74a2b6ce5d199c41345296f49d974ddb4123", size = 95949, upload-time = "2026-03-01T22:06:25.697Z" }, + { url = "https://files.pythonhosted.org/packages/68/03/093f4055ed4cae649ac53bca3d180bd37102e9e11d048588e9ab0c0108d0/yarl-1.23.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:e7b0460976dc75cb87ad9cc1f9899a4b97751e7d4e77ab840fc9b6d377b8fd24", size = 95839, upload-time = "2026-03-01T22:06:27.309Z" }, + { url = "https://files.pythonhosted.org/packages/b9/28/4c75ebb108f322aa8f917ae10a8ffa4f07cae10a8a627b64e578617df6a0/yarl-1.23.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:115136c4a426f9da976187d238e84139ff6b51a20839aa6e3720cd1026d768de", size = 90696, upload-time = "2026-03-01T22:06:29.048Z" }, + { url = "https://files.pythonhosted.org/packages/23/9c/42c2e2dd91c1a570402f51bdf066bfdb1241c2240ba001967bad778e77b7/yarl-1.23.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:ead11956716a940c1abc816b7df3fa2b84d06eaed8832ca32f5c5e058c65506b", size = 100865, upload-time = "2026-03-01T22:06:30.525Z" }, + { url = "https://files.pythonhosted.org/packages/74/05/1bcd60a8a0a914d462c305137246b6f9d167628d73568505fce3f1cb2e65/yarl-1.23.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:fe8f8f5e70e6dbdfca9882cd9deaac058729bcf323cf7a58660901e55c9c94f6", size = 96234, upload-time = "2026-03-01T22:06:32.692Z" }, + { url = "https://files.pythonhosted.org/packages/90/b2/f52381aac396d6778ce516b7bc149c79e65bfc068b5de2857ab69eeea3b7/yarl-1.23.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:a0e317df055958a0c1e79e5d2aa5a5eaa4a6d05a20d4b0c9c3f48918139c9fc6", size = 100295, upload-time = "2026-03-01T22:06:34.268Z" }, + { url = "https://files.pythonhosted.org/packages/e5/e8/638bae5bbf1113a659b2435d8895474598afe38b4a837103764f603aba56/yarl-1.23.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6f0fd84de0c957b2d280143522c4f91a73aada1923caee763e24a2b3fda9f8a5", size = 97784, upload-time = "2026-03-01T22:06:35.864Z" }, + { url = "https://files.pythonhosted.org/packages/80/25/a3892b46182c586c202629fc2159aa13975d3741d52ebd7347fd501d48d5/yarl-1.23.0-cp313-cp313t-win32.whl", hash = "sha256:93a784271881035ab4406a172edb0faecb6e7d00f4b53dc2f55919d6c9688595", size = 88313, upload-time = "2026-03-01T22:06:37.39Z" }, + { url = "https://files.pythonhosted.org/packages/43/68/8c5b36aa5178900b37387937bc2c2fe0e9505537f713495472dcf6f6fccc/yarl-1.23.0-cp313-cp313t-win_amd64.whl", hash = "sha256:dd00607bffbf30250fe108065f07453ec124dbf223420f57f5e749b04295e090", size = 94932, upload-time = "2026-03-01T22:06:39.579Z" }, + { url = "https://files.pythonhosted.org/packages/c6/cc/d79ba8292f51f81f4dc533a8ccfb9fc6992cabf0998ed3245de7589dc07c/yarl-1.23.0-cp313-cp313t-win_arm64.whl", hash = "sha256:ac09d42f48f80c9ee1635b2fcaa819496a44502737660d3c0f2ade7526d29144", size = 84786, upload-time = "2026-03-01T22:06:41.988Z" }, + { url = "https://files.pythonhosted.org/packages/90/98/b85a038d65d1b92c3903ab89444f48d3cee490a883477b716d7a24b1a78c/yarl-1.23.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:21d1b7305a71a15b4794b5ff22e8eef96ff4a6d7f9657155e5aa419444b28912", size = 124455, upload-time = "2026-03-01T22:06:43.615Z" }, + { url = "https://files.pythonhosted.org/packages/39/54/bc2b45559f86543d163b6e294417a107bb87557609007c007ad889afec18/yarl-1.23.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:85610b4f27f69984932a7abbe52703688de3724d9f72bceb1cca667deff27474", size = 86752, upload-time = "2026-03-01T22:06:45.425Z" }, + { url = "https://files.pythonhosted.org/packages/24/f9/e8242b68362bffe6fb536c8db5076861466fc780f0f1b479fc4ffbebb128/yarl-1.23.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:23f371bd662cf44a7630d4d113101eafc0cfa7518a2760d20760b26021454719", size = 86291, upload-time = "2026-03-01T22:06:46.974Z" }, + { url = "https://files.pythonhosted.org/packages/ea/d8/d1cb2378c81dd729e98c716582b1ccb08357e8488e4c24714658cc6630e8/yarl-1.23.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c4a80f77dc1acaaa61f0934176fccca7096d9b1ff08c8ba9cddf5ae034a24319", size = 99026, upload-time = "2026-03-01T22:06:48.459Z" }, + { url = "https://files.pythonhosted.org/packages/0a/ff/7196790538f31debe3341283b5b0707e7feb947620fc5e8236ef28d44f72/yarl-1.23.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:bd654fad46d8d9e823afbb4f87c79160b5a374ed1ff5bde24e542e6ba8f41434", size = 92355, upload-time = "2026-03-01T22:06:50.306Z" }, + { url = "https://files.pythonhosted.org/packages/c1/56/25d58c3eddde825890a5fe6aa1866228377354a3c39262235234ab5f616b/yarl-1.23.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:682bae25f0a0dd23a056739f23a134db9f52a63e2afd6bfb37ddc76292bbd723", size = 106417, upload-time = "2026-03-01T22:06:52.1Z" }, + { url = "https://files.pythonhosted.org/packages/51/8a/882c0e7bc8277eb895b31bce0138f51a1ba551fc2e1ec6753ffc1e7c1377/yarl-1.23.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a82836cab5f197a0514235aaf7ffccdc886ccdaa2324bc0aafdd4ae898103039", size = 106422, upload-time = "2026-03-01T22:06:54.424Z" }, + { url = "https://files.pythonhosted.org/packages/42/2b/fef67d616931055bf3d6764885990a3ac647d68734a2d6a9e1d13de437a2/yarl-1.23.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1c57676bdedc94cd3bc37724cf6f8cd2779f02f6aba48de45feca073e714fe52", size = 101915, upload-time = "2026-03-01T22:06:55.895Z" }, + { url = "https://files.pythonhosted.org/packages/18/6a/530e16aebce27c5937920f3431c628a29a4b6b430fab3fd1c117b26ff3f6/yarl-1.23.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c7f8dc16c498ff06497c015642333219871effba93e4a2e8604a06264aca5c5c", size = 100690, upload-time = "2026-03-01T22:06:58.21Z" }, + { url = "https://files.pythonhosted.org/packages/88/08/93749219179a45e27b036e03260fda05190b911de8e18225c294ac95bbc9/yarl-1.23.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:5ee586fb17ff8f90c91cf73c6108a434b02d69925f44f5f8e0d7f2f260607eae", size = 98750, upload-time = "2026-03-01T22:06:59.794Z" }, + { url = "https://files.pythonhosted.org/packages/d9/cf/ea424a004969f5d81a362110a6ac1496d79efdc6d50c2c4b2e3ea0fc2519/yarl-1.23.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:17235362f580149742739cc3828b80e24029d08cbb9c4bda0242c7b5bc610a8e", size = 94685, upload-time = "2026-03-01T22:07:01.375Z" }, + { url = "https://files.pythonhosted.org/packages/e2/b7/14341481fe568e2b0408bcf1484c652accafe06a0ade9387b5d3fd9df446/yarl-1.23.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:0793e2bd0cf14234983bbb371591e6bea9e876ddf6896cdcc93450996b0b5c85", size = 106009, upload-time = "2026-03-01T22:07:03.151Z" }, + { url = "https://files.pythonhosted.org/packages/0a/e6/5c744a9b54f4e8007ad35bce96fbc9218338e84812d36f3390cea616881a/yarl-1.23.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:3650dc2480f94f7116c364096bc84b1d602f44224ef7d5c7208425915c0475dd", size = 100033, upload-time = "2026-03-01T22:07:04.701Z" }, + { url = "https://files.pythonhosted.org/packages/0c/23/e3bfc188d0b400f025bc49d99793d02c9abe15752138dcc27e4eaf0c4a9e/yarl-1.23.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:f40e782d49630ad384db66d4d8b73ff4f1b8955dc12e26b09a3e3af064b3b9d6", size = 106483, upload-time = "2026-03-01T22:07:06.231Z" }, + { url = "https://files.pythonhosted.org/packages/72/42/f0505f949a90b3f8b7a363d6cbdf398f6e6c58946d85c6d3a3bc70595b26/yarl-1.23.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:94f8575fbdf81749008d980c17796097e645574a3b8c28ee313931068dad14fe", size = 102175, upload-time = "2026-03-01T22:07:08.4Z" }, + { url = "https://files.pythonhosted.org/packages/aa/65/b39290f1d892a9dd671d1c722014ca062a9c35d60885d57e5375db0404b5/yarl-1.23.0-cp314-cp314-win32.whl", hash = "sha256:c8aa34a5c864db1087d911a0b902d60d203ea3607d91f615acd3f3108ac32169", size = 83871, upload-time = "2026-03-01T22:07:09.968Z" }, + { url = "https://files.pythonhosted.org/packages/a9/5b/9b92f54c784c26e2a422e55a8d2607ab15b7ea3349e28359282f84f01d43/yarl-1.23.0-cp314-cp314-win_amd64.whl", hash = "sha256:63e92247f383c85ab00dd0091e8c3fa331a96e865459f5ee80353c70a4a42d70", size = 89093, upload-time = "2026-03-01T22:07:11.501Z" }, + { url = "https://files.pythonhosted.org/packages/e0/7d/8a84dc9381fd4412d5e7ff04926f9865f6372b4c2fd91e10092e65d29eb8/yarl-1.23.0-cp314-cp314-win_arm64.whl", hash = "sha256:70efd20be968c76ece7baa8dafe04c5be06abc57f754d6f36f3741f7aa7a208e", size = 83384, upload-time = "2026-03-01T22:07:13.069Z" }, + { url = "https://files.pythonhosted.org/packages/dd/8d/d2fad34b1c08aa161b74394183daa7d800141aaaee207317e82c790b418d/yarl-1.23.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:9a18d6f9359e45722c064c97464ec883eb0e0366d33eda61cb19a244bf222679", size = 131019, upload-time = "2026-03-01T22:07:14.903Z" }, + { url = "https://files.pythonhosted.org/packages/19/ff/33009a39d3ccf4b94d7d7880dfe17fb5816c5a4fe0096d9b56abceea9ac7/yarl-1.23.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2803ed8b21ca47a43da80a6fd1ed3019d30061f7061daa35ac54f63933409412", size = 89894, upload-time = "2026-03-01T22:07:17.372Z" }, + { url = "https://files.pythonhosted.org/packages/0c/f1/dab7ac5e7306fb79c0190766a3c00b4cb8d09a1f390ded68c85a5934faf5/yarl-1.23.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:394906945aa8b19fc14a61cf69743a868bb8c465efe85eee687109cc540b98f4", size = 89979, upload-time = "2026-03-01T22:07:19.361Z" }, + { url = "https://files.pythonhosted.org/packages/aa/b1/08e95f3caee1fad6e65017b9f26c1d79877b502622d60e517de01e72f95d/yarl-1.23.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:71d006bee8397a4a89f469b8deb22469fe7508132d3c17fa6ed871e79832691c", size = 95943, upload-time = "2026-03-01T22:07:21.266Z" }, + { url = "https://files.pythonhosted.org/packages/c0/cc/6409f9018864a6aa186c61175b977131f373f1988e198e031236916e87e4/yarl-1.23.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:62694e275c93d54f7ccedcfef57d42761b2aad5234b6be1f3e3026cae4001cd4", size = 88786, upload-time = "2026-03-01T22:07:23.129Z" }, + { url = "https://files.pythonhosted.org/packages/76/40/cc22d1d7714b717fde2006fad2ced5efe5580606cb059ae42117542122f3/yarl-1.23.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a31de1613658308efdb21ada98cbc86a97c181aa050ba22a808120bb5be3ab94", size = 101307, upload-time = "2026-03-01T22:07:24.689Z" }, + { url = "https://files.pythonhosted.org/packages/8f/0d/476c38e85ddb4c6ec6b20b815bdd779aa386a013f3d8b85516feee55c8dc/yarl-1.23.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fb1e8b8d66c278b21d13b0a7ca22c41dd757a7c209c6b12c313e445c31dd3b28", size = 100904, upload-time = "2026-03-01T22:07:26.287Z" }, + { url = "https://files.pythonhosted.org/packages/72/32/0abe4a76d59adf2081dcb0397168553ece4616ada1c54d1c49d8936c74f8/yarl-1.23.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50f9d8d531dfb767c565f348f33dd5139a6c43f5cbdf3f67da40d54241df93f6", size = 97728, upload-time = "2026-03-01T22:07:27.906Z" }, + { url = "https://files.pythonhosted.org/packages/b7/35/7b30f4810fba112f60f5a43237545867504e15b1c7647a785fbaf588fac2/yarl-1.23.0-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:575aa4405a656e61a540f4a80eaa5260f2a38fff7bfdc4b5f611840d76e9e277", size = 95964, upload-time = "2026-03-01T22:07:30.198Z" }, + { url = "https://files.pythonhosted.org/packages/2d/86/ed7a73ab85ef00e8bb70b0cb5421d8a2a625b81a333941a469a6f4022828/yarl-1.23.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:041b1a4cefacf65840b4e295c6985f334ba83c30607441ae3cf206a0eed1a2e4", size = 95882, upload-time = "2026-03-01T22:07:32.132Z" }, + { url = "https://files.pythonhosted.org/packages/19/90/d56967f61a29d8498efb7afb651e0b2b422a1e9b47b0ab5f4e40a19b699b/yarl-1.23.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:d38c1e8231722c4ce40d7593f28d92b5fc72f3e9774fe73d7e800ec32299f63a", size = 90797, upload-time = "2026-03-01T22:07:34.404Z" }, + { url = "https://files.pythonhosted.org/packages/72/00/8b8f76909259f56647adb1011d7ed8b321bcf97e464515c65016a47ecdf0/yarl-1.23.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:d53834e23c015ee83a99377db6e5e37d8484f333edb03bd15b4bc312cc7254fb", size = 101023, upload-time = "2026-03-01T22:07:35.953Z" }, + { url = "https://files.pythonhosted.org/packages/ac/e2/cab11b126fb7d440281b7df8e9ddbe4851e70a4dde47a202b6642586b8d9/yarl-1.23.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:2e27c8841126e017dd2a054a95771569e6070b9ee1b133366d8b31beb5018a41", size = 96227, upload-time = "2026-03-01T22:07:37.594Z" }, + { url = "https://files.pythonhosted.org/packages/c2/9b/2c893e16bfc50e6b2edf76c1a9eb6cb0c744346197e74c65e99ad8d634d0/yarl-1.23.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:76855800ac56f878847a09ce6dba727c93ca2d89c9e9d63002d26b916810b0a2", size = 100302, upload-time = "2026-03-01T22:07:39.334Z" }, + { url = "https://files.pythonhosted.org/packages/28/ec/5498c4e3a6d5f1003beb23405671c2eb9cdbf3067d1c80f15eeafe301010/yarl-1.23.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e09fd068c2e169a7070d83d3bde728a4d48de0549f975290be3c108c02e499b4", size = 98202, upload-time = "2026-03-01T22:07:41.717Z" }, + { url = "https://files.pythonhosted.org/packages/fe/c3/cd737e2d45e70717907f83e146f6949f20cc23cd4bf7b2688727763aa458/yarl-1.23.0-cp314-cp314t-win32.whl", hash = "sha256:73309162a6a571d4cbd3b6a1dcc703c7311843ae0d1578df6f09be4e98df38d4", size = 90558, upload-time = "2026-03-01T22:07:43.433Z" }, + { url = "https://files.pythonhosted.org/packages/e1/19/3774d162f6732d1cfb0b47b4140a942a35ca82bb19b6db1f80e9e7bdc8f8/yarl-1.23.0-cp314-cp314t-win_amd64.whl", hash = "sha256:4503053d296bc6e4cbd1fad61cf3b6e33b939886c4f249ba7c78b602214fabe2", size = 97610, upload-time = "2026-03-01T22:07:45.773Z" }, + { url = "https://files.pythonhosted.org/packages/51/47/3fa2286c3cb162c71cdb34c4224d5745a1ceceb391b2bd9b19b668a8d724/yarl-1.23.0-cp314-cp314t-win_arm64.whl", hash = "sha256:44bb7bef4ea409384e3f8bc36c063d77ea1b8d4a5b2706956c0d6695f07dcc25", size = 86041, upload-time = "2026-03-01T22:07:49.026Z" }, + { url = "https://files.pythonhosted.org/packages/69/68/c8739671f5699c7dc470580a4f821ef37c32c4cb0b047ce223a7f115757f/yarl-1.23.0-py3-none-any.whl", hash = "sha256:a2df6afe50dea8ae15fa34c9f824a3ee958d785fd5d089063d960bae1daa0a3f", size = 48288, upload-time = "2026-03-01T22:07:51.388Z" }, +] + +[[package]] +name = "zipp" +version = "3.23.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/02/0f2892c661036d50ede074e376733dca2ae7c6eb617489437771209d4180/zipp-3.23.0.tar.gz", hash = "sha256:a07157588a12518c9d4034df3fbbee09c814741a33ff63c05fa29d26a2404166", size = 25547, upload-time = "2025-06-08T17:06:39.4Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2e/54/647ade08bf0db230bfea292f893923872fd20be6ac6f53b2b936ba839d75/zipp-3.23.0-py3-none-any.whl", hash = "sha256:071652d6115ed432f5ce1d34c336c0adfd6a884660d1e9712a256d3d3bd4b14e", size = 10276, upload-time = "2025-06-08T17:06:38.034Z" }, +]