Add azd ai eval extension for Foundry evaluations - #9339
Draft
m7md7sien wants to merge 62 commits into
Draft
Conversation
New azd extension exposing azd ai eval, registering the azure.ai.eval service-target provider. Scaffold only: manifest, entrypoint, root command.
Lifted eval_api (models, operations, poller, generation, portal_urls) and dataset_api from azure.ai.agents, de-agent-scoped. Added evalcore with IsTransientError and an EvaluatorList that accepts either a bare string or a mapping with a threshold. Skipped artifacts.go and eval_config.go since the config model differs.
… commands - EvalConfig and GenerateConfig types with validation covering cross-references, duplicate names, unsupported target types, and evaluation levels - ResolveGroup picks the only group or errors with the available names - ArtifactPath accepts a directory or an explicit file path for local_dir - evalContext resolves the project endpoint (flag, azd env, host env) and builds both clients against the azd developer CLI credential - dataset create/update/list/show/delete with -o json - Tier-0 tests for parsing, validation, group resolution, and path handling
- buildEvalGroupRequest maps evaluators to testing criteria, keeping the builtin prefix on evaluator_name while stripping it from name, and carries per-evaluator thresholds in initialization_parameters - run resolves the group from --eval-id, a pinned id, or the azd environment, creating it when absent, then binds the dataset to the run since the group has no dataset binding today - Local datasets are sent inline with optional truncation; registered datasets are referenced by id
- evaluator upload/update/list/show/builtins/delete; rubric evaluators only in M1, code evaluators deferred to M2 with the folder walk and RBAC they require - normalizeRubricBody accepts a bare definition or a full document - results show/export with per-criteria pass and fail counts, --failed-only, and JSON or CSV output, replacing the counts-only view - Added ListEvaluators, ListEvaluatorVersions, DeleteEvaluatorVersion, and CancelOpenAIEvalRun to the eval client
init scaffolds both YAML files and the artifact directories without any service call, so it works offline and unauthenticated. Built-ins are referenced from the group but never declared as custom evaluators. A dataset flag containing a path becomes a local source; a bare name references a registered dataset. Tests assert the scaffold loads and validates, and that paths are used verbatim rather than re-rooted.
…tion Registers the azure.ai.eval service target so azd up and azd deploy reach this extension; the extension ships no deploy command of its own. - Reads the eval config from the service entry's inline properties, the same AdditionalProperties channel the agents extension uses - Deploy reconciles datasets, then evaluators, then eval groups, since a group references the versions the first two resolve to; it fails fast and the next deploy resumes - Datasets are change-detected with a local SHA-256 digest kept in the azd environment, because the dataset API returns no content hash and comparing against the service would mean downloading the blob every deploy - Evaluator definitions come back inline, so those are compared directly - Package and Publish are no-ops; eval artifacts are plain files already on disk
- generate submits the rubric and dataset generation jobs, downloads the artifacts locally, and writes source references into the deployment spec - MergeArtifactRefs edits through the yaml Node API so comments, key order, and hand-edited sibling keys survive; matching is by name and merging is idempotent - Raised the client poll budget from 2s x 300 to 5s x 720. The old 10 minute limit gave up while the service was still working, which is the timeout that forced a second command - A supplied --evaluator or a local --dataset is honored and its generation is skipped - Tests cover comment preservation, section creation, idempotence, and fingerprinting
Adds the templated release pipeline for the new extension and lists it as a dependency of the microsoft.foundry meta-package. The registry.json artifact entries are generated by the release, so they are not hand-authored here.
… tests Live testing against a real project found two issues. 1. The dataset model only bound snake_case URIs (data_uri, blob_uri), but the project endpoint returns camelCase (dataUri). ResolvedBlobURI therefore returned empty, which would have failed the generate download much later with no useful error. Both spellings are now accepted. 2. Built-in evaluators do not share one input contract. builtin.ifeval requires instruction_id_list and is rejected under the agent-target data mapping with MissingRequiredDataMapping. The live tests now select an evaluator whose inputs match, and the helper documents why. Live tests are gated behind the live build tag and AZURE_AI_EVAL_E2E_LIVE, and clean up every resource they create. Verified: builtin listing, the full dataset pending-upload lifecycle with version auto-increment, and eval group creation returning 201.
…published contract The builder sent one fixed data mapping and one fixed set of initialization parameters to every evaluator. That only suited agent-target quality evaluators, and the service rejected the rest. The evaluator listing publishes a full contract per evaluator: definition.data_schema (accepted and required inputs), definition.init_parameters, and supported_evaluation_levels. The builder now reads it and shapes each testing criterion accordingly. This fixes four concrete defects. - Required inputs were never honoured, so builtin.ifeval (instruction_id_list), builtin.similarity (ground_truth) and builtin.retrieval (context) all failed with MissingRequiredDataMapping. Fields not supplied by the agent target are now bound to dataset columns, and the item schema declares them. - Inputs an evaluator does not accept were sent anyway. - initialization_parameters always carried model, deployment_name and threshold. No evaluator accepts 'model', and builtin.ifeval accepts nothing at all. Parameters are now filtered to the declared properties, and a required one that is missing is reported locally. - evaluation_level was sent as run metadata, where it has no effect. It is an initialization parameter on the evaluators that declare it. It also encodes an exclusivity rule the service enforces: 'messages' and 'query'/'response' cannot both appear in a mapping, so the evaluation level selects between the conversation and turn shapes. A missing dataset column is now caught before the request is sent and names the column, rather than surfacing as a 400 pointing at testing_criteria[0].data_mapping. An evaluator with no published contract keeps the previous agent-target shape, so custom evaluators are unaffected. Verified against a live project: all ten built-ins are accepted, where two previously failed. The live test exercises the shipping builder rather than a hand-rolled request, so a regression in this logic fails the suite. Adds DeleteOpenAIEval so those tests clean up after themselves.
…invokes Without build.ps1 and build.sh the dev kit reported a successful build in under a second and produced no binary, so azd x pack had nothing to package and azd x publish failed with 'Artifacts not found'. Copied from azure.ai.agents with the version package path retargeted, plus its golangci config. Verified end to end: build, pack, publish, install from the local registry, and 'azd ai eval --help' listing every command.
…ly returns The listing spells it evaluator_type, so the TYPE column in 'evaluator list' and 'evaluator builtins' was always blank. Both spellings are now accepted.
Running a real 'azd deploy' against the service-target provider surfaced two failures that no unit test covered. Evaluator references only decoded from YAML. azd hands the service entry to the extension as JSON, so a group written as '- builtin.task_adherence' -- the form the CLI's own init command writes -- failed with 'cannot unmarshal string into EvaluatorRef'. EvaluatorList now decodes and encodes the mixed string-or-mapping form through JSON as well, and a test asserts the two decoders agree. The dataset reconciler passed the declared version straight to UploadNewVersion, which derives the next version from it. A declaration without an explicit version passed empty, so every deploy retried 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset once that version existed. It now looks up the latest registered version first. Verified against a live project: first deploy publishes the dataset at 1.0 and creates the group; an unchanged redeploy reports 'unchanged at version 1.0' and uploads nothing; and editing the dataset publishes 2.0 and recreates the group, since groups are immutable.
azd core does not resolve $ref for extensions. It strips the ServiceConfig fields it owns and leaves $ref at the top of the map for the owning extension to resolve, so a service authored the way the spec documents it -- host: azure.ai.eval plus $ref: ./evals/azure.yaml -- parsed to an empty config. azd deploy then reported success in three seconds having created nothing, which is worse than failing. The provider now calls foundry.ResolveFileRefs with the project root from the azd project client. Relative source paths inside an included file are written against that file, but ResolveFileRefs inlines content without rebasing them, so the include's own directory is now the base for source resolution. Verified against a live project: the $ref form deploys, and the dataset fingerprint matches the one from the equivalent inline config, confirming both forms resolve to the same file.
…ation changes Change detection only covered upstream artifacts, so retargeting a group at a different agent, swapping an evaluator, or changing the judge model left the old group in place. Groups are immutable, so the edit silently had no effect and later runs kept evaluating the previous definition. The group's declaration is now fingerprinted alongside the dataset and evaluator artifacts. The id and description are excluded: one is server assigned and the other is cosmetic, so neither should force a recreate. The digest is recorded when an existing group is reused as well as when one is created. Recording it only on create meant a group deployed before this change never established a baseline, and the first edit after it would still go undetected. Verified against a live project: changing the target produced a new group id, and two further deploys with no change reused it.
The data-plane clients trace every request and response through log.Printf, which Go writes to stderr by default, so a plain command interleaved raw URLs and status lines with its own output. A long generate run was mostly HTTP traces. Ports the debug setup from the agents extension: the standard logger is discarded unless --debug or AZD_EXT_DEBUG is set, and debug output goes to a dated file rather than the terminal. The hook chains the SDK PersistentPreRunE instead of replacing it. Assigning PersistentPreRun has no effect once the E variant is set, and overwriting the E variant would drop the SDK own setup. Also reports jobs as submitted when generate is given --no-wait, which is a successful submission rather than an empty result.
…fails Data generation with an agent source is accepted and then fails within seconds with DataGenerationJobSystemError, whose message says only that something went wrong and to try again. It is not transient: it reproduces for every agent tried, while the identical request without the agent source runs normally. The CLI now names the agent, says a retry will not help, and points at the two workarounds, instead of relaying advice that cannot succeed.
The spec lists run start, list, show and cancel, and M1 requires every operation to be reachable atomically, but run was a single composite command with no subcommands. Listing runs, inspecting one, and cancelling an in-flight run were unreachable, even though the client already had the calls. Adds run list, run show and run cancel. Each takes the eval group id as an optional argument and otherwise falls back to the id recorded in the azd environment, matching results show. Cancelling a run that already reached a terminal state is refused locally, because the service reports success either way and the CLI would otherwise claim to have cancelled a finished run. Two related fixes. Passing --project-endpoint disabled the azd environment cache entirely: the environment name was only resolved when the endpoint came from azd, so every cached eval group and run id lookup returned empty. The name is now resolved independently of where the endpoint came from. The spec documents --wait and --no-wait, but cobra does not derive the negative form from a bool, so --no-wait was rejected as an unknown flag. Verified live: start with --no-wait, list, show, cancel, and the terminal-state guard on a second cancel. JSON output checked on the new subcommands.
The spec lists run start alongside list, show and cancel. The behaviour existed only as the composite `azd ai eval run`, so the atomic name in the spec did not resolve. Both forms are now built by one constructor, so their flags cannot drift apart, and a test asserts that.
… work Exercising the atomic write commands against a live project found three failures. None were covered by tests, because none of these paths had been run end to end. dataset update always collided. It passed the --version flag straight to UploadNewVersion, which derives the next version from what it is given, so an omitted flag restarted at 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset. The flag help promised the opposite, that omitting it would take the next version. This is the same defect that was fixed in the deploy reconciler earlier, so the discovery is now centralised in DatasetClient.UploadNextVersion and both callers use it, rather than being fixed twice and available to be missed a third time. evaluator upload rejected every hand-authored rubric. The service needs a type discriminator on the definition, and without it fails the whole request with "The request field is required", which names a field that is present. Generated rubrics carry the type, so only the hand-authored path documented in the spec was affected. The type is now filled in when absent and left alone when set. evaluator show returned 404. It omitted the version segment from the path, but the service has no route for an unversioned evaluator, despite the doc comment claiming the latest would be fetched. The latest version is now resolved first, comparing numerically because versions are integers as strings and a lexical compare ranks "9" above "15" -- the service already publishes evaluators at version 15 and 17. Verified live: dataset create, show, update to 2.0, list and delete; evaluator upload, show resolving the latest, update to version 2, list and delete. Both suites leave nothing behind.
…nfig Deploying a config that declares a custom evaluator, rather than only built-in ones, failed in two ways. Every earlier test used built-ins, so neither showed up. The evaluator was republished on every deploy. The service enriches a definition when it stores it, so a rubric consisting of nothing but type and dimensions comes back carrying data_schema, init_parameters and metrics it was never given. Comparing whole documents therefore never matched. Only the keys the author actually wrote are compared now, structurally, so key order and formatting are not changes either. This is what the spec means by repeated azd up creating no redundant versions. The eval group was then rejected with a request for a model that had been set. Evaluators disagree on what the judge model is called: built-ins declare deployment_name, and a custom rubric declares model. The builder sent only deployment_name, so the custom evaluator saw its required parameter missing. The judge model is now bound under whichever name the evaluator declares. Verified live: first deploy publishes the evaluator and creates the group, two redeploys report it unchanged and publish nothing, editing the rubric publishes the next version and recreates the group, and a further redeploy is a no-op again.
…d dataset The flag is documented as taking a path or the name of a registered dataset, and means use this one instead of generating. It only suppressed generation when the value looked like a local path, so passing the name of an existing dataset still submitted a generation job and, since agent-seeded generation is currently broken server-side, failed the whole command. --evaluator already skipped unconditionally, so the two flags disagreed. Both the skip and the default-spec synthesis now key off whether the flag was supplied at all. This was the last thing standing between a generated config and the documented end-to-end flow. Verified live: init scaffolds a group referencing its own rubric, generate writes that rubric and merges the reference into the same file while preserving comments and ordering, azd up registers the dataset and evaluator and creates the group, and the run completes and scores against the generated rubric.
…listing
GET /datasets/{name}/versions returns nothing for a second or two after a
version is created, even though the version itself reads back immediately.
Measured: empty at 0s, populated at 2s.
That undermines the version discovery added for dataset update, which reads the
listing to decide what to increment from. An empty listing is ambiguous -- it
means either a new dataset or a stale read -- so back-to-back create and update
could still restart at 1.0 and take a 409.
Rather than delaying every first upload to wait for the index, a conflict is
now treated as the stale read it is: re-read the listing, which by then
reflects reality, and retry once. The common path is unchanged.
The live test asserted on the first listing response and was failing for the
same reason. It now polls, and says why.
Verified: create immediately followed by update produces 2.0 rather than a
conflict, and the full live suite passes.
…requires The shared extension build template invokes ci-build.ps1 and ci-test.ps1 from the extension directory. Neither existed, so the release pipeline added alongside this extension would have failed on its first run. Both are modelled on the agents extension with two deliberate differences. ci-build.ps1 reads version.txt from the extension directory rather than its parent, where no such file exists, so the default works when the pipeline is not supplying -Version. It accepts -BuildRecordMode, which the template always passes, but builds nothing extra: this extension has no record/playback mode and no pipeline step consumes a record binary. ci-test.ps1 passes --junitfile explicitly. The pipeline publishes **/junitTestReport.xml from the extension directory, and the extension template does not set GOTESTSUM_JUNITFILE the way the CLI build does, so without this no test results would surface in the build. Verified locally with gotestsum installed: 97 tests across 7 suites reported. Also adds the README and CHANGELOG that 17 of the 21 extensions ship. The README documents the deployed shape, the command surface, the rubric weight constraint, and how to run the live tests.
The manifest declared two capabilities the extension did not back. metadata was declared but the command was never registered, so azd could not discover the command tree: azd ai eval metadata failed with unknown command while the same call against a peer extension returned its full tree. azd uses this for discovery, so the declaration was actively misleading. The command is now registered and reports nine commands. lifecycle-events was declared but no event handlers exist. The SDK only starts its event manager when handlers are registered, so the capability was an unused permission rather than a broken promise. It is removed; the listen command is still invoked because the service-target-provider capability triggers it, which a deploy after the change confirms. Adds tests over the manifest so neither can drift again: every declared capability must be backed by the command that implements it, the declared provider name must match the host the code registers, and version.txt must agree with the manifest version, which until now was only a comment asking for it.
…ction from a file Auditing every flag and API sequence the spec documents against the running extension turned up two gaps. The spec describes a drift check that was never implemented. It matters because of how change detection works: when local content is unchanged, the version recorded at the last deploy is reused, so a version published outside the repo would be silently ignored and the eval group pinned to older data. A deploy now fails when the service holds a newer version than the recorded one, naming both versions. An explicit version: on the declaration skips the check, because that is the author stating which version they want. This was added after testing the remedy the error message suggests and finding it did not work -- the message now describes something that does. --gen-instruction-file was documented but absent. A useful generation instruction is usually longer than fits on a command line, and putting it in a file makes it reviewable with the rest of the config. Verified live: publishing a version out-of-band fails the next deploy, and pinning that version lets it through.
… edits M1 exits on all the spec examples running end to end, so I ran them verbatim. Two did not. --eval-id could never work. It is meant to run an existing group ignoring the config, and appears in both the CI/CD example and the recovery advice, but a run needs a target and a dataset and an eval group carries neither: the group holds only its testing criteria, and the dataset travels on the run. Every --eval-id invocation failed asking for a target. The pairing survives in the group's previous run, so re-running a group now repeats what it last ran, and a group that has never run says so and points at the config-based path. The failure-and-recovery example promised an error that did not exist. A run sends a local dataset inline, so unregistered local edits were evaluated silently and the results could not be traced to any dataset version. That now fails with the message the spec documents, once a deploy has recorded a fingerprint to compare against. Before that there is nothing to have drifted from, and running is how a group first comes into existence. Verified live: the CI/CD example returns JSON with a run id, and with unregistered edits the config-based run fails while --eval-id succeeds, which is exactly the recovery the spec describes.
…ploy spec init --dataset ./tests/golden.jsonl wrote that path into evals/azure.yaml unchanged, but source: is resolved relative to the file it appears in, so the deploy looked for evals/tests/golden.jsonl and failed on a file the user had just pointed at. This is the spec's bring-your-own-data example exactly as written, so that example could never have worked. The path is now rebased onto the output directory, with forward slashes so the config reads the same on every platform, and absolute paths left alone. With this the documented examples all run end to end, which is what M1 exits on: bring-your-own-data through init, azd up and run; results show --failed-only -O writing its file; and the CI/CD sequence of dataset create, run start -o json and results export --format csv.
agent.context.traces accepts source, window and sample, but the generation API takes a day count and nothing else, so source and sample were parsed and dropped without a word. An author who set sample: 500 believed they had narrowed the trace selection when nothing had changed. Both fields are documented in the spec, so this was reachable by following it. They are now reported as having no effect, naming each one, with the verb agreeing so one field reads "has" and two read "have". The warning goes to stdout rather than stderr because azd does not surface an extension's stderr -- written to stderr it was invisible in a real run even though the unit test passed -- and is suppressed under -o json so the output stays parseable.
… flow M1 exits on the spec examples running end to end, and the first one could not: its generate step failed. Two separate causes, the second only reachable once the first was out of the way. Agent-seeded data generation fails server-side for every agent, which no user can act on. The same request carrying only the prompt succeeds and produces a usable dataset, so a failure now retries without the agent and says that it did. The rubric still uses the agent's context; only the dataset falls back. That exposed the real defect underneath. A dataset's URI points at either the blob or the container holding it, and nothing in the payload distinguishes them: isSingleFile is true either way. Uploads end in the file name, generated datasets end in the container, and downloading a container returns 409. The download path also sent no SAS token, so it could not have authenticated even against the right URL. It had never run successfully, because generation always failed before reaching it. Downloads now fetch a credential, use the blob directly when the URI names one, and otherwise list the container and take the JSONL inside. This retires an assumption recorded earlier as holding: that a dataset blob URI can be downloaded directly. It is true only for uploads.
… source that never works Agent-seeded data generation fails server-side for every agent. Following it through the stack, the request reaches AOAI unchanged: RAISvc forwards it and FineTuning rewrites the path, and neither resolves the agent. Nothing in this repo can fix it, and nothing the client sends changes the outcome — a nonexistent agent name fails exactly like a real one, so the agent is never looked up at all. But the contract says what that source was for: it "references an agent to fetch instructions / metadata from". That is a read this client can do itself. So `generate` now resolves the agent's context locally, preferring what the author supplied: an explicit instruction, else the file named by agent.context.instructions, else the agent's published instructions. That last step is what makes the scaffolded flow work. `init` writes agent.context.instructions pointing at a conventional path, and nothing read it — the field, and its `tools` sibling, were accepted and dropped. With no instruction anywhere, generation had nothing to fall back to when the agent source failed, so `init` followed by `generate` could not produce a dataset. It now generates one from the agent's own instructions, on topic and scored by a rubric drawn from the same source. `tools` is still unread, so it is called out rather than silently dropped, and init no longer scaffolds it — warning about a field the user never chose is just noise. Also stops relying on the service to reject a missing model. Generation is billed against a deployment, and without one the request failed halfway through the command with a service error naming nothing the caller controls. It now fails before any network call, naming both ways to supply it.
Closes the scheduling half of M2. A schedule pairs a trigger with an eval group and the run to repeat, since the group holds only its testing criteria and the target and dataset travel with the run. `set` builds that run the same way `run` does, so a scheduled evaluation is the one already being run by hand. Triggers come from flags rather than raw JSON: --cron, or --every with the qualifier that period actually reads. Passing a qualifier from another period is rejected instead of dropped, because the service ignores what it does not recognise and the author would never learn the selection did not narrow. The route took finding. GET answers on both /schedules and /evaluations/schedules, but neither accepts a POST; creation is a PUT on a named resource, and everything else 404s. Three service behaviours shape the command, all measured rather than assumed: A project holds one schedule at a time. A second is refused with a 400 that carries no body at all, so the count is what explains it — the error now names the schedule already there and how to remove it. This also corrected an earlier reading of mine: a run of "required field" probes all failed for this reason, not the one I first recorded. A PUT over an existing schedule is accepted, echoes the new body, and changes nothing. Neither a new cron expression nor a new trigger type takes effect. So `set` refuses an existing name rather than reporting an edit that did not happen. Recreating under a name that was used before never leaves Creating and then cannot be deleted, so the refusal points at a different name rather than offering to replace in place. Deleting also needed care: a schedule mid-provision refuses, as 409 while busy or 404 because its trigger does not exist yet, so delete waits for it to settle. Also fixes a latent client bug this surfaced: 204 was not in the set of accepted statuses, so every successful delete was reported as a failure.
… run failed Traces as a run data source were recorded as awaiting service support. They are not: azure_ai_traces is in the run data-source discriminator and the service accepts and executes it. The note was an assumption I had not re-tested, so `run --from-traces` now evaluates what the agent already did rather than asking it fresh questions from a dataset. The agent comes from the group's target, or from whatever the group ran against last when --eval-id bypasses the config, so the flag needs no argument. A window is only sent when asked for: the service defaults it, and an unset bound sent as an epoch would quietly mean 1970. Verified against the live service, which stored the payload and normalised 7d into lookback_hours 168 while honouring max_traces. The other half of this was finding the run had failed and not being told why. A failed run carries the reason, and it is usually the only actionable thing in the response, but it was dropped: the trace case explains that the agent emitted no GenAI content and what to do about it, and all the caller saw was the word "failed". Every failure path was losing this, not just traces.
…vice reads --trace-window was being dropped. The traces data source has no start bound: its window is lookback_hours, and a start_time is accepted and discarded, leaving the default seven days in place. It looked like it worked because the value I first tested with, 7d, is the default. Asking for 30d stored lookback_hours 168 and queried a week, which is the failure mode this extension has been fixing everywhere else: a field accepted, ignored, and never reported. Now sends lookback_hours, verified against the service, which stored 720 and queried 2026-06-28 to 2026-07-28. Found by reading the service's own contract rather than the response echo. The echo was no help: it reflected a well-formed request whose window had already been replaced by the default.
The second M4 scenario listed as awaiting service support and reachable today. `run --response-id` evaluates responses the project already holds, so a run that already happened can be scored without replaying it. The shape is not the obvious one. There is no list of ids on the data source: the ids travel as ordinary JSONL rows and a data_mapping points the service at the field holding each one, which is what lets it retrieve the chat history behind the response. Every guess at a plainer shape was rejected with the same unhelpful "Item generation source content is empty", so this came from reading the service contract instead. Verified live: three stored responses evaluated, three passed, and the stored payload matched what was sent field for field.
target.type: model evaluates a deployment with no agent in front of it. The
config already rejected it by name, and the test asserting that used "model"
as its example of an unsupported type, so the gap was recorded twice as
deliberate. The service supports it.
What makes it more than a new enum value is that the criteria have to change
with it. An agent returns output_items and tool calls; a model answers as
plain text under output_text. Binding the agent shape to a model run does not
fail at validation, it fails per row at execution with "Missing inputs for
line 1: data.sample.tool_definitions, data.sample.output_items" — the group
looks well formed and every row errors.
So the sample bindings are chosen from the target kind rather than from
whether a target exists at all, and a model target binds response alone.
Verified end to end: a model-target group deployed with response bound to
{{sample.output_text}}, ran, and scored 2 passed / 1 failed / 0 errored across
coherence and fluency.
A group whose dataset has no local source could not run at all. The run sent the dataset name as a file_id, and a file_id means an uploaded file, so the service answered "invalid data source file ids". Every test until now used a local source, which is why the path stayed unexercised. Registered datasets are now fetched and their rows sent inline, the same as a local file. That also gives --max-samples the meaning it always claimed: it was documented as ignored for registered datasets, because a file reference carries no row limit and the service cannot narrow one. Fetching the rows is what makes narrowing possible, so the flag now behaves the same either way and the caveat is gone from its help. Found while implementing the last M4 scenario, subsetting a registered dataset. The subsetting worked on the first try; the run underneath it did not, which is the part worth having found. Verified live against a published dataset: the whole set scores 2 passed / 1 failed, and --max-samples 2 scores 2 rows.
…irst one's id A config with two eval groups aliased them. Resolved ids were kept under one shared EVAL_GROUP_ID, so on the second deploy the first group read the id the second had left there, confirmed that group existed, and adopted it. Both declarations then pointed at the same group, and running one scored the other's criteria against its own dataset. Nothing failed; the results were just wrong. The first deploy hid it, because the env is not readable back within the same deploy, so both groups were created correctly and only the cache was left crossed. It needed a second deploy to surface, which is why the single-group testing everything else used never reached it. Ids are now keyed by declaration name, as the fingerprints already were. EVAL_GROUP_ID is still written as the last-deployed group, which is what the commands fall back to when a config names only one. Same fix for the dataset version read while narrowing a registered dataset: it was reading the shared EVAL_DATASET_VERSION, which with two datasets is whichever was published last. Verified live: two groups keep distinct ids across repeated deploys, and each runs its own criteria.
…un separately Finishes what the group-id fix started. Two more places assumed one group. Every command taking an eval-id could only be pointed at a group by its service id. With one group the cached id made that invisible; with two there was no way to ask for the other except by looking its id up. They now accept --eval-group, naming the group the way the config does. That exposed the next layer: the remembered run was also a single key, so it belonged to whichever group ran last. Asking group A for its latest results fetched group B's run id inside group A and returned 404 — the group resolved correctly and the run did not. Runs are now remembered per group. A remembered run that no longer resolves also stops being fatal. It is a convenience, so a stale one falls through to the group's current latest rather than failing; an explicit --run-id still reports what went wrong. Verified live with two groups: each resolves to its own run and its own criteria, and an undeployed name is refused by name.
…lares `version:` on a dataset was being passed to the upload helper whose argument is the version to count from, not the one to write. So a config declaring version "1.0" published 2.0, and every later edit climbed again. The author never got the version they asked for and nothing said so. The field also meant two different things depending on the branch taken: unchanged content resolved to the declared version, changed content published one above it. A declared version is now the version published. If it already exists and the local file differs, that is the author's decision to make, so the deploy stops and says to raise the pin or drop it. Without a pin nothing changes: each deploy still takes the next version. Verified live: version "1.0" on a new dataset publishes 1.0, and editing the file while still pinned to 1.0 fails with that instruction instead of quietly publishing 2.0.
…ll not honour An evaluator declaring both a source and a version published at whatever version came next and the group bound that, so the pin described a version nothing used. A config asking for version 7 deployed version 1 and said nothing. Unlike a dataset, the version here is not ours to choose: the service assigns it on publish. So there is nothing to honour, and the field is refused alongside a source rather than accepted and ignored. The message names both ways out, since either is a coherent thing to have meant: drop version to publish the file, or drop source to reference a version already published. Both remaining forms verified live: source alone publishes and then reports unchanged, version alone references what is on the project.
…hemas
Criteria are meant to be shaped from each evaluator's own contract. For
built-ins they never were: the schemas were fetched with an unfiltered list,
which returns only the project's own evaluators. Built-ins have to be asked
for by type, so every one of them fell back to the legacy field list instead.
Nothing looked wrong because that list is query, response, tool_calls,
tool_definitions, which is exactly what the common evaluators want. It fails
where an evaluator wants anything else. builtin.task_completion at
conversation level wants messages, so its criterion was published with an
empty data_mapping and no evaluation_level: a group that could not score a
single row, and no error anywhere.
Also drops the requirement that a group declare a target. A dataset holding
both sides of an exchange has nothing to invoke, and the service runs it
happily; refusing it was ours alone.
Verified live: the same group now publishes messages bound to {{item.messages}}
with evaluation_level conversation, and runs to 1 passed / 1 failed / 0
errored.
Note for anyone reading the earlier verification: groups built from these
schemas were confirmed accepted by the service, and that is still true. The
service accepts an empty mapping. Accepted was never the same as correct.
…schemas The built-in schemas were never reaching the builder in the shipping path, and the test that walks every built-in could not see it: it fetched the schemas itself with the Builtin filter and handed them in. It was building the input the product was failing to build, so the builder was verified against every published contract while the code that supplies them resolved none. That test now takes its schemas from the production lookup, so a break in supply breaks the test. Two more cover the lookup directly: one asserts built-ins resolve at all and reports how many the unfiltered listing returns, the other that a conversation-level evaluator binds its conversation field rather than producing the empty mapping that scores nothing. Checked by reverting the fix: the new test fails with "the unfiltered listing returns 0 of them", and all ten built-ins pass again with it restored.
Coverage across the decision-making code was thin in exactly the places this extension has been getting wrong: precedence rules and URI resolution, where a wrong answer produces no error at all. Precedence: which of a flag and a config option wins, for the sample cap and the evaluation level. options.max_samples was already parsed and dropped once, and nothing failed when it was; a test would have said so. Level filtering: that a conversation evaluator is sent messages and a turn one query and response, in both directions, and that a required field survives the filter so a real conflict still surfaces as a missing field rather than being reshaped away. URI resolution: the service spells these fields several ways and a URI read from the wrong spelling comes back empty rather than wrong, which is how the dataset URI went unbound the first time. Also that an upload's SAS-bearing URI and its plain finalize URI are read from their own places, since confusing them fails at different stages.
Giving each eval group its own env entry stopped a second group from adopting the first one's id, but it also removed the only read of EVAL_GROUP_ID on the run path. Setting that key by hand is the documented way to point a config at a group created in the portal or by another tool, so the documented behaviour quietly stopped working: the override was ignored and a new group created. The shared key is read again, but only when the config declares a single group. With more than one there is no way to tell which group it refers to, which is what caused the aliasing in the first place.
The list commands passed the service's own envelope straight through, and the
two services behind them do not agree on one: runs came back as {"data": [..]}
and datasets, evaluators and schedules as {"value": [..]}. A script had to know
which API happened to back each command to read its output, and the envelopes
also carry paging fields the extension never follows, implying there is more to
fetch when there is not.
Each list now emits a bare array. A nil slice is normalized to [] so that an
empty listing is still something a caller can iterate rather than null.
`run start --eval-id <id>` is the form the CI example uses, but the sibling commands accepted the id only as a positional argument, so `run list --eval-id` failed with "unknown flag". Writing the verification script for the spec examples is what surfaced it: the flag learned in one command does not work in the next one a script reaches for. --eval-id now sits alongside --eval-group on run list/show/cancel and results show/export/compare, resolved with the same precedence as before. The positional argument keeps working and still wins.
A run with a single sample has no standard deviation, and the service reports it as the string "NaN" because JSON has no NaN literal. The model declared float64, so decoding failed and the whole comparison was thrown away with "cannot unmarshal string into Go struct field ... of type float64" — including the TooFewSamples verdict that exists to explain that exact situation. Comparing a one-sample gate, which is the cheapest thing a pipeline does, could not work. The statistics are now a LenientFloat that decodes a number or any of the quoted forms, and marshals non-finite values as null, because encoding/json refuses NaN outright and would have broken -o json instead. The table prints an undefined statistic as a dash rather than the literal NaN. The earlier live check passed only because every run it compared had enough samples for the deviation to be a real number.
A schedule repeats the eval group's most recent run, so scheduling a group whose last run came from --from-traces makes it a trace evaluation, and the service then allows only an hourly trigger. Asking for a daily one produced a raw 400 saying "Scheduled trace evaluations only support hourly recurrence triggers", which is baffling when a trigger was the only thing asked for and traces were never mentioned. Confirmed by experiment: daily was accepted after an agent run, refused after a traces run on the same group, and hourly accepted for that same traces run. The error now names the cause and both ways out. Deleting a schedule that does not exist printed 2364 characters of service error document wrapping an inner 404 from the trigger service. A missing name is the ordinary typo, so it now says so in one line and points at schedule list. The live run test also asserted only that the run reached a terminal state. A run whose every sample errors still reports "completed", so it now requires no errored samples and at least one scored one - otherwise a broken target or evaluator would leave the suite green.
init scaffolded evals/azure.yaml and then printed a services block for the reader to paste into the root azure.yaml. azd acts on nothing until that reference exists, so the documented flow stopped between init and azd up, and every scenario run so far had passed only because the harness wrote that file itself before calling init. The reference is mechanical, so init writes it: creating the project file when the folder has none, and otherwise adding the service through the YAML node tree so the project keeps its name, comments, key order and other services. A project already declaring an azure.ai.eval service is left alone - matched on the host rather than the service name, so running init twice cannot deploy the same evals twice. Anything that cannot be edited safely still prints the block. The next steps now suggest azd deploy rather than azd up when the project has no infra to provision, since azd up stops at a missing infra/main.bicep without saying that provisioning is what it wanted.
…es found
init hand-edited the root azure.yaml through the YAML node tree. No other
extension does that - the agents extension adds its service with azd's own
Project().AddService, and the only azure.yaml any extension writes is a staged
copy in a temp directory. So the eval service goes through AddService too, and
azd owns the edit. A folder with no project is no longer given one: evals attach
to a project, so init says to run azd init.
That also removes the azd up divergence. init suggested azd deploy when the
project had no infra, which quietly moved people off the flow the spec
describes; it says azd up again, and a project without infra fails in
provisioning where the reason belongs.
Two defects the corner cases turned up:
A malformed dataset row was published without complaint. {not json at all}
became version 1.0 with an eval group bound to it, and would have failed much
later on a row nobody had looked at. Every row is now checked before upload and
the offending line is named. The scan buffer is 8MB because a conversation-level
row runs well past bufio's 64KB default.
Asking for runs of an eval group that does not exist printed 1682 characters of
HTTP error. A mistyped or undeployed id is ordinary, so it now says so in one
line.
…es one The service entry named no dependencies, so azd had nothing to order it against. The agents extension wires uses: to the ai-project service it created, and the eval service wants the same: it evaluates against that project, so the project should be provisioned first. It is conditional for the reason agents makes it conditional - naming a service the project does not declare is a broken reference, and an eval config can sit in a repo that reaches an existing Foundry project by endpoint rather than declaring one.
|
Azure Pipelines: 7 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
… items Renames, all user-facing, from the spec review: - "eval group" is not a service concept. The object is an eval, so evalGroups: becomes evals:, --eval-group becomes --eval, EVAL_GROUP_ID becomes EVAL_ID, and the prose follows. Breaking, and cheapest now. - --eval-model meant the judge in run and the generator in generate, which is two different things wearing one name. Split into --judge-model and --generation-model. - --gen-instruction is the agent's instruction, so --agent-instruction. - evaluator upload becomes evaluator create, matching dataset create. update is gone: both published a new immutable version, so it was a synonym. - evaluator builtins folds into evaluator list --builtin. results show only ever fetched the run, which carries totals and a per-criterion breakdown - so it could say how many rows failed but never which or why, and --failed-only filtered criteria rather than rows. It now reads the run's output_items, which carry the evaluated row, each evaluator's score and verdict, and the judge's reason. --failed-only filters rows, and -o json carries the whole thing. run delete is added; the API supported it all along.
Nothing read agent.context.tools. Tool definitions reach an evaluator from the
live agent as {{sample.tool_definitions}}, which is what the tool evaluators
score, and the agents extension's own eval and eval-generate paths do not read a
local tools file either - only optimize does, and optimize stays there. The
generation API has no field for tools, so the only way to pass them would have
been prose folded into the prompt. A key with no consumer is worse than no key,
so it is gone rather than warned about.
dataset update goes the same way evaluator update did: both published a new
immutable version, so update was a synonym for create.
… --limit The evalGroups -> evals rename covered the config key and the flags but stopped there. Ten user-facing strings still said "group", so the CLI contradicted the spec and read as though the rename had been abandoned half-way: --eval help "Name a group from the config" run --name help "Defaults to the group name plus a timestamp" schedule --name help "Defaults to the group name" schedule list header "EVAL GROUP" schedule show field "group:" run no-previous-run "pass a config that declares the group" run from-traces "Declare target.type: agent on the group" schedule traces-only "this group's most recent run", "run the group once" Also rename the metadata written onto the eval, azd_eval_group -> azd_eval. Nothing reads it back, so this is safe. Separately, run list called ListOpenAIEvalRuns with a hardcoded limit of 0 and exposed no way to change it, even though the client already took the parameter. Add --limit.
`evaluator create --folder <dir>` publishes a folder of Python as an
evaluator version, and an `evaluators:` entry whose `source:` is a
directory is reconciled by `azd up` the same way a dataset is.
The wire shape was ambiguous. Two documents disagreed, and both were
partly wrong. The published OpenAPI document that RAISvc vendors and
contract-tests against settles it:
{"type":"code", "code_text", "entry_point", "image_tag", "blob_uri"}
plus inherited init_parameters, data_schema and metrics, all snake_case.
The sibling custom_evaluator_upload spec is stale on four counts - the
route is POST /evaluators/{name}/versions not PUT .../versions/{version},
credentials is POST not GET, the body is snake_case not camelCase, and
the discriminator is "code" not "CodeBased". There is no version-based
shape negotiation; one contract serves every api-version.
blob_uri is a container prefix rather than an archive: the consumer
enumerates the container, so files upload at their relative paths.
Two deliberate choices where the evidence did not reach:
Only blob_uri is sent, never code_text. The contract allows either, and
inline would save a round trip for a single-file evaluator, but nothing
observable confirms the executor runs it - the hand-off converter drops
both fields and refetches from the catalog. blob_uri is the alternative
with a demonstrated consumer. Sending the unproven one would trade a
saved upload for an evaluator that registers cleanly and then fails when
it is run, which is much harder to diagnose.
Dot-prefixed files are excluded from the upload, not just dot-prefixed
directories. An evaluator folder kept in a repo collects .env, .netrc
and .pypirc, and publishing would copy those secrets into blob storage.
Nothing a Python evaluator needs at runtime starts with a dot.
entry_point is sent although RAISvc has no such property yet; it is
spec-declared, and the create path ignores unknown members, so it starts
persisting when the service catches up.
The live round-trip test is written but unrun - it needs a project
endpoint. It is the only thing that will confirm the service accepts
this body; everything here is read from source and spec, not observed.
The live round-trip had never been executed. Running it against a real project found three defects that reading the contract could not. pendingUploadType was sent as "BlobReference". The enum value is "TemporaryBlobReference", so the body failed to bind and the service answered 400 "The request field is required" - a null model, reported as a missing field rather than an invalid one. startPendingUpload answers with TemporaryDataReferenceResponseDto, which carries the location under blobReferenceForConsumption. Only blobReference was modelled, which datasets use, so the reply parsed into an empty struct and the upload failed for want of a credential. Both names are now accepted. Storage is reserved under a version guessed from the registered ones, and that list is eventually consistent: immediately after a publish it still reports the evaluator as unknown. The guess then collided with an existing version, which the service refuses with a 409 wrapped in a 500. A collision now advances the guess and retries instead of failing. Also relax the version assertion, which required the created version to equal the one storage was reserved under. There is no create-at-version route - the create assigns its own number and the reservation can take one too, so the two legitimately differ. Probing the service shows a reservation returns a container named by GUID, not by version, so a definition always points at exactly the bytes uploaded for it and the divergence is cosmetic. The test now asserts what actually matters: a second publish creates a new version, and the two versions keep separate storage. Verified against a live Foundry project: all three code evaluator round trips pass, and the full live suite is green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
azd ai eval, a new azd extension for Foundry evaluations, implementing the design spec.Status: draft. The spec is still in review (foundrysdk_specs#251) and open comments there may still move the command surface.
What it does
azd ai eval initscaffoldsevals/eval_generate.yamlandevals/azure.yaml, and declares the eval service in the project'sazure.yamlthrough azd's ownProject().AddService— the same call the agents extension uses, so azd owns that edit.azd ai eval generatesubmits the data- and rubric-generation jobs, downloads the artifacts locally, and writessource:references back intoevals/azure.yaml.azd up/azd deployreconcile datasets, evaluators and eval groups through aservice-target-providerregistered forhost: azure.ai.eval.azd ai eval runensures the eval group exists, starts a run, waits and summarizes.Atomic commands cover
dataset,evaluator,run,resultsandschedule. Every read command supports-o json; every command supports--no-prompt.Verification
-tags live.Notes for reviewers
$refis resolved by the extension: azd core strips the fields it owns and leaves$refinAdditionalPropertiesfor the owning extension.