Skip to content

Refresh Azure CLI options from current output - #4546

Merged
thomhurst merged 20 commits into
mainfrom
issue-4331-azure-current-output
Sep 14, 2026
Merged

thomhurst merged 20 commits into
mainfrom
issue-4331-azure-current-output

Conversation

@thomhurst

@thomhurst thomhurst commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Refreshes Azure options from Azure CLI 2.90.0 help, covering 5,131 commands. The scraper preserves description-only values, credentials, datetime values, optional names and IDs, repeated image tags, boolean actions, and delimited values. Password identifier segments receive secret metadata while paths, names, and IDs remain visible. VM/VMSS usernames stay scalar; actual NetApp, SignalR, and authentication-audience list definitions produce grouped values, including the CLI's documented Space-separeted list spelling.

Generated global-option documentation follows each tool's argument order. Public API synchronization discovers additions before checking removals, restores baselines on failure, and handles case-only filename changes. Compiler diagnostics track PID plus process start time and reset sampling delay for replacement processes.

Final-source generation run 34832142558 produced the code, documentation, API baselines, command coverage, and provenance incorporated through #4955. Its source fingerprint is fa293cbb7c3611630fc4377b1c0a68bcb5d200e2a13b2c626592ac72cad5ac7d, matching the final Azure generator source. No generated implementation was hand-edited.

Validation:

  • All 70 Azure scraper tests and all 1,494 generator tests passed locally. Release generator build passed with zero warnings/errors; scoped severity-info formatting and diff checks passed.
  • Final CI passes the full Linux pipeline, Windows/macOS builds, analyzers, trim/Native AOT, and documentation deployment. All 28 Azure integration tests pass, including the 15 regressions for password metadata, usernames, NetApp/SignalR lists, and authentication audiences.
  • All seven compiler-trace tests passed on Linux. The earlier local Azure compilation 2 GB limit remains recorded; validation used CI without raising or retrying that local limit.
  • The current-head review confirms the old Azure-output blockers are resolved. Its enum-import concern is already addressed by fix(generator): parse pnpm 12 clap-style help #4713 on main: the conflict-free merge retains the conditional enum namespace import and its regression test. No PR change removes either.

Refs #4331

@thomhurst thomhurst added dependencies Pull requests that update a dependency file automated breaking Breaking API change .NET Pull requests that update .net code labels Sep 3, 2026
@thomhurst

Copy link
Copy Markdown
Owner Author

@claude Please review this generated Azure CLI synchronization.

@greptile-apps

greptile-apps Bot commented Sep 3, 2026

Copy link
Copy Markdown

Too many files changed for review (3000 files, 100 file limit).

@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Caution

CodeRabbit couldn't post its review summary.

Error details
Validation Failed: {"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"} - https://docs.github.com/rest/issues/comments#create-an-issue-comment

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Refresh Azure CLI options from current output (#4546)

Scope note: GitHub's diff view truncates at 300 files; this PR touches ~6,260 files. I reconstructed the full diff locally via git diff be8fc008^ be8fc008 (single commit, chore(azure): refresh generated options). It consists of ~6,260 auto-generated *.Generated.cs files rebuilt from current az CLI help output, one generated coverage JSON, and hand-written updates to test/ModularPipelines.Azure.UnitTests/AzureCommandTests.cs for the new required positional constructors and renamed ACR properties (ResourceGroupValueResourceGroup, TargetValueTarget, plus removal of the old [Obsolete] bool shims).

Verified clean:

  • No duplicate property/constructor-parameter names introduced (no compile breakage from the new required-positional-parameter constructors).
  • No other source in the repo still references the old ACR property names or constructs the newly-required-constructor types via object initializers that would now fail to compile.
  • AzureCommandTests.cs is internally consistent with the regenerated options it exercises.
  • The regenerated coverage JSON is well-formed.
  • This aligns with CLAUDE.md's guidance that "current tool output is the sole source of truth for generated options" — no hand-preserved shims were kept, which is correct per project convention.

Finding (BLOCKING): the refresh is inconsistent within the very same generated records — some value-taking options were correctly promoted from CliFlag bool? to CliOption, while sibling options with identically-worded doc comments were left as broken boolean flags.

  1. src/ModularPipelines.Azure/Options/AzAcrBuildOptions.Generated.cs (lines ~34, ~91) — BuildArg and SecretBuildArg remain [CliFlag("--build-arg")] public bool? BuildArg even though their doc comments explicitly describe a name[=value] argument ("Build argument in '--build-arg name[=value]' format..."). In this exact same file/commit, AuthMode, Image, and SourceAcrAuthId were correctly reclassified from CliFlag/bool? to CliOption/string? for the same reason. Setting BuildArg = true emits the bare --build-arg switch with no value — a silently broken az acr build invocation.

  2. src/ModularPipelines.Azure/Options/AzSynapseSparkJobSubmitOptions.Generated.cs (lines ~55-65) — MainClassName, Configuration, and ReferenceFiles remain CliFlag/bool? despite doc comments describing value-carrying arguments (a class identifier, a job configuration, a file list). This same commit correctly fixed Archives, Language, PythonFiles, and Tags in the identical record from CliFlag/bool? to CliOption/string? or IEnumerable<string>?.

Why this matters / suggested approach: Per CLAUDE.md, the fix belongs in the scraper/generator (tools/ModularPipelines.OptionsGenerator/), not in hand-edits to the generated files. The generator's flag-vs-option classification heuristic appears to key off some signal (likely presence of an explicit value placeholder or specific phrasing) that misses arguments described via name[=value]-style syntax or multi-word value descriptions — worth checking whether the classifier is keying off the CLI help's argument-metavar pattern and failing for these non-standard forms. Since this PR's diff shows the generator getting some properties in a record right and others in the same record wrong, this looks like a narrow gap in that heuristic rather than a systemic breakage, but it should be fixed in the generator so it doesn't need to be manually re-caught on every future refresh. A heuristic scan for the same doc-text-vs-attribute mismatch pattern across the diff surfaced this as widespread (order of 100+ occurrences), so an isolated manual patch of just these two files would not be sufficient — the generator itself needs the fix, then a re-run of the refresh.

@thomhurst

Copy link
Copy Markdown
Owner Author

Addressed the exact-head Claude blocker at 01f8860. Extended the Azure scraper—not generated files—to recognize description-only value families, including build/secret arguments, paths, platforms, timeouts, configurations, class identifiers, commands, and file collections; added numeric/list cardinality detection and exact ACR/Synapse regression tests. Validation: Azure scraper tests 26/26; full generator tests 1,265/1,265. A full local Azure regeneration reached command 413 but hit the repository-mandated 2-GB process-tree guard, so I did not raise/retry it. Dispatched the 60-minute CI generator on this exact branch to produce the complete regenerated artifact set; auto-merge is disabled.

@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Caution

CodeRabbit couldn't post its review summary.

Error details
Validation Failed: {"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"} - https://docs.github.com/rest/issues/comments#create-an-issue-comment

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Refresh Azure CLI options from current output (#4546)

Re-review scope: since the last Claude review (head be8fc008), this branch adds exactly one commit, 01f886096b ("fix(azure): detect description-only values"), touching only AzCliScraper.cs and its test file — no *.Generated.cs files changed. I diffed be8fc008..01f886096b directly to confirm this.

The prior BLOCKING finding is not actually resolved yet.

The PR comment states the scraper was extended to "recognize description-only value families" and that CI was dispatched to "produce the complete regenerated artifact set." That scraper fix is real, narrowly scoped, and well covered by two new tests (Nonstandard_Description_Only_Values_Are_Not_Flags, Synapse_Description_Only_Values_Are_Not_Flags) — this is the right place to fix it per CLAUDE.md ("update the scraper or generator", not hand-edit generated files). However, the regenerated output implied by those tests has not actually landed on this branch. At the current head (01f886096b8df9b4ae49d6283a91f23c1d5f8fe0), the exact same generated files from the original finding are unchanged and still broken:

  • AzAcrBuildOptions.Generated.cs: BuildArg, SecretBuildArg, File, Platform, and Timeout are all still [CliFlag] public bool? despite value-carrying descriptions (the new scraper test literally asserts these should now be IEnumerable<string>?/string?/int?):
    /// <summary>
    /// Build argument in '--build-arg name[=value]' format. Multiples are supported by passing '--build-arg name[=value]' multiple times. IMPORTANT: This parameter should not include passwords, access tokens, or sensitive information of any kind. This parameter value will be visible to the ACR team for debugging purposes.
    /// </summary>
    [CliFlag("--build-arg")]
    public bool? BuildArg { get; set; }
    /// <summary>
    /// The relative path of the the docker file to the source code root folder. Default to 'Dockerfile'.
    /// </summary>
    [CliFlag("--file", ShortForm = "-f")]

    /// <summary>
    /// The platform where build/task is run, Eg, 'windows' and 'linux'. When it's used in build commands, it also can be specified in 'os/arch/variant' format for the resulting image. Eg, linux/arm/v7. The 'arch' and 'variant' parts are optional.
    /// </summary>
    [CliFlag("--platform")]
    public bool? Platform { get; set; }

    /// <summary>
    /// The timeout in seconds.
    /// </summary>
    [CliFlag("--timeout")]
    public bool? Timeout { get; set; }
  • AzSynapseSparkJobSubmitOptions.Generated.cs: Configuration, MainClassName, and ReferenceFiles are likewise still [CliFlag] public bool?:
    /// <summary>
    /// The configuration of Spark job.
    /// </summary>
    [CliFlag("--configuration")]
    public bool? Configuration { get; set; }
    /// <summary>
    /// The Spark job language. Allowed values: CSharp, PySpark, Python, Scala, Spark, SparkDotNet. Default: Scala.
    /// </summary>
    [CliOption("--language")]
    public string? Language { get; set; }
    /// <summary>
    /// The fully-qualified identifier or the main class that is in the main definition file.
    /// </summary>
    [CliFlag("--main-class-name")]
    public bool? MainClassName { get; set; }

    /// <summary>
    /// Additional files used for reference in the main definition file.
    /// </summary>
    [CliFlag("--reference-files")]
    public bool? ReferenceFiles { get; set; }

Setting BuildArg = true (or Configuration = true, etc.) still emits a bare switch with no value — the same silently-broken CLI invocation as before. Also worth noting: File, which shares the same bug pattern, wasn't called out in the original finding but is covered by the new scraper test, and it's still broken in the committed output too — reinforcing that this is a mechanical "generator changed, artifacts didn't" gap rather than something that needs more heuristic work.

Why this matters / suggested approach: the fix isn't done until the regenerated *.Generated.cs files are actually committed to this branch — a scraper change with no corresponding regeneration leaves the shipped bug in place regardless of how correct the scraper logic now is. Since local regeneration hits the repo's 2GB agent memory guard, this has to come from the dispatched CI run's artifact being committed back onto the branch before this can be considered resolved; please push that regenerated output (or confirm the CI run failed/is still pending) rather than merging on the strength of the scraper-only fix.

As a structural improvement worth considering separately from this PR: a lightweight CI check that fails when tools/ModularPipelines.OptionsGenerator/** changes without a corresponding diff in the generated packages (or that just runs the generator and diffs its output against what's committed) would catch this "scraper fixed, artifacts stale" gap automatically, instead of relying on a reviewer to manually diff two heads of a 6,000+ file PR to notice regeneration didn't happen.

@thomhurst

Copy link
Copy Markdown
Owner Author

Ubuntu failure is outside this PR's Azure-generated diff: RunCoreUnitTestsModule hit its 20-minute ceiling after RunReportEnricherTimeoutDoesNotSkipLaterEnrichers stalled. A concurrent unrelated PR (#4548) hit the same 20-minute core-test ceiling on a different timer/cancellation test, so this is shared runner/test-timeout behavior rather than Azure output. I did not rerun merely for green. The workflow is still waiting for its queued macOS job; full logs were inspected through the completed-job API.

@thomhurst

Copy link
Copy Markdown
Owner Author

Started an Azure-only authoritative regeneration from exact head f99a80d with a 90-minute CI allowance and staged patch artifact. The temporary workflow instrumentation will be reverted after applying the exact generated output. @claude review will be requested on the final head.

@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Caution

CodeRabbit couldn't post its review summary.

Error details
Validation Failed: {"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"} - https://docs.github.com/rest/issues/comments#create-an-issue-comment

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

This PR adds new scraper detection logic (AzCliScraper.cs) for description-only Azure CLI options that don't declare an explicit value placeholder, so they're correctly typed instead of misclassified as boolean flags. The scraper-logic tests (AzCliScraperTests.cs) are well-targeted and pass.

Blocking issue: the fix isn't actually applied to the shipped generated code.

The PR's commits land in this order:

  1. be8fc008chore(azure): refresh generated options (bulk regeneration, 6263 files)
  2. 01f88609fix(azure): detect description-only values (the actual scraper regex fix, touches only AzCliScraper.cs + its tests)
  3. f99a80dbci(options): retain Azure generation patch (CI workflow only)

The regeneration happened before the scraper fix, and nothing in commit 2 or 3 re-runs generation. So the checked-in .Generated.cs files still contain the exact bug this PR claims to fix.

Concrete, verifiable contradiction: the new test Nonstandard_Description_Only_Values_Are_Not_Flags asserts that for az acr build, --build-arg, --secret-build-arg should be IEnumerable<string>?, --platform/--file should be string?, and --timeout should be int?. But src/ModularPipelines.Azure/Options/AzAcrBuildOptions.Generated.cs (shipped in this same PR, via commit 1) still has all four of BuildArg, SecretBuildArg, Platform, Timeout as [CliFlag] bool? presence-only flags:

[CliFlag("--build-arg")]
public bool? BuildArg { get; set; }
...
[CliFlag("--platform")]
public bool? Platform { get; set; }
...
[CliFlag("--secret-build-arg")]
public bool? SecretBuildArg { get; set; }
...
[CliFlag("--timeout")]
public bool? Timeout { get; set; }

Notably, other properties in that same file (AuthMode, Image, SourceAcrAuthId, ResourceGroup, Target) were correctly fixed by commit 1 (from an earlier scraper improvement) — so this isn't a case where the file was untouched; it's specifically the new description-only detection pattern from commit 2 that never got applied. Same mismatch pattern applies to AzSynapseSparkJobSubmitOptions.Generated.cs vs. the new Synapse_Description_Only_Values_Are_Not_Flags test (Configuration, MainClassName, ReferenceFiles).

Impact: A consumer calling new AzAcrBuildOptions(...) { Platform = true } cannot actually pass a platform value (e.g. "linux/amd64") to az acr build --platform — the property is still boolean presence-only. The PR's own tests prove the shipped API is wrong; CI is green only because the scraper-level unit tests check the scraper's output in isolation, not that the checked-in generated corpus matches current scraper output.

Suggested fix: re-run the Azure options generation after the scraper fix (i.e., squash/reorder so generation happens after 01f88609, or add a follow-up regeneration commit) so the .Generated.cs files reflect the new detection logic. Given AzDescriptionOnlyValuePattern/AzNumericValueDescriptionPattern/AzCollectionValueDescriptionPattern look like broadly-applicable patterns, it's worth doing a full diff of the regenerated Azure corpus against what's currently checked in — there are likely more than the two files above affected (a substantial number of other [CliFlag] properties elsewhere in the Azure package could match these same patterns).

As a process improvement: consider adding a CI check that fails if the checked-in generated corpus differs from a fresh run of the current generator/scraper — this class of "logic fixed, output not regenerated" bug would be easy to detect automatically before merge rather than requiring a manual cross-reference like this review did.

@thomhurst

Copy link
Copy Markdown
Owner Author

Azure generation completed 6,422 files at run 33752281684, then PublicAPI snapshot collection failed before the recovery artifact step. Moved temporary patch staging/upload ahead of that sync and merged current main/provenance changes at exact head 5ef57b6. Replacement authoritative run: https://github.com/thomhurst/ModularPipelines/actions/runs/33766998203

@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Caution

CodeRabbit couldn't post its review summary.

Error details
Validation Failed: {"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"} - https://docs.github.com/rest/issues/comments#create-an-issue-comment

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Refresh Azure CLI options from current output (#4546)

Re-review scope: this is the fourth Claude review round on this PR. Three prior reviews (heads be8fc008, 01f886096b, f99a80dba) all landed the same BLOCKING verdict for the same root cause. I re-checked whether it's resolved at the current head, 5ef57b68, and it is not — plus a targeted spot-check across ~20 other generated files surfaced two more instances of the same failure class, including one outright regression.

The previously-flagged blocker is still unresolved

Across three rounds, this PR has claimed the scraper fix (AzCliScraper.cs, commit 01f886096) for "description-only value" options was regenerated into the shipped corpus. It still hasn't been. At the current head:

  • AzAcrBuildOptions.Generated.csBuildArg, File, Platform, SecretBuildArg, and Timeout are all still [CliFlag] public bool?, even though their own doc comments describe value-carrying arguments ("Build argument in '--build-arg name[=value]' format...", "The timeout in seconds.") and the PR's own new scraper test (Nonstandard_Description_Only_Values_Are_Not_Flags) asserts these should be IEnumerable<string>?/string?/int?.
  • AzSynapseSparkJobSubmitOptions.Generated.csConfiguration, MainClassName, ReferenceFiles are likewise still [CliFlag] bool?, contradicted by the PR's own Synapse_Description_Only_Values_Are_Not_Flags test.

Setting e.g. BuildArg = true or Configuration = true still emits a bare switch with no value — the same silently-broken CLI invocation flagged three reviews ago. The comment thread describes several CI regeneration attempts (a run reaching 6,422 files, then failing "before the recovery artifact step") that were apparently never actually applied back onto this branch — the merge to the current head brought in main, not the regenerated Azure output.

Two more instances of the same failure class, one a regression

Spot-checking beyond the two files above (Keyvault, Network, Cosmosdb, Functionapp, Monitor, Storage, Eventhubs):

  • AzEventhubsNamespaceAuthorizationRuleShowOptions.Generated.cs (newly added) — AuthorizationRuleName ("The authorization rule name.") and NamespaceName ("The Namespace name.") are generated as [CliFlag] bool? instead of [CliOption] string?, and neither was promoted to the constructor. As written, this record cannot specify which namespace or rule to target at all — worse than "drops a value," this command is unusable. The sibling AzEventhubsEventhubConsumerGroupDeleteOptions.Generated.cs has the identical pattern for ConsumerGroupName/EventhubName/NamespaceName, and AzMonitorAccountCreateOptions.Generated.cs's Location shows the same misclassification — this looks systemic for at least the eventhubs extension command family, not isolated to the two files already reported.
  • AzAcrTaskCreateOptions.Generated.cs — this is an actual regression, not just an unresolved gap: on main, Image is ImageValues : IEnumerable<string>? (correctly multi-valued, with a deprecated bool shim for back-compat). On this branch it was narrowed to Image : string? — single-valued — while the doc comment is unchanged and still reads "Multiple tags are supported by passing -t multiple times." This refresh makes an already-correct property worse.

Why this matters / suggested approach

Per CLAUDE.md, fixes belong in the scraper/generator, and "current tool output is the sole source of truth" — so a breaking regen is expected and fine on its own. The problem here isn't the scraper logic (which is narrowly scoped and well-tested in isolation); it's that the shipped .Generated.cs corpus in this PR doesn't reflect that scraper's own output, in both directions (some fixes never applied, and at least one prior-correct property got worse). Given three review rounds have now independently hand-verified the same "generator changed, artifacts stale" gap on different file pairs, this isn't something a reviewer should need to keep manually spot-checking — as suggested in the prior review, a CI gate that runs the generator and fails if its output differs from what's committed (or a required "generator diff is empty" check gating merge) would catch this mechanically before every future refresh, rather than relying on someone diffing two heads of a 6,000+ file PR by hand each time.

Minor, non-blocking

.github/workflows/generate-cli-options.yml still carries the "Stage generated Azure output for recovery" / "Preserve generated Azure patch" / "Upload generated Azure patch" steps added purely to recover from this PR's own troubled regeneration attempts, hardcoded to matrix.tool == 'az' only (no other of the ~40 tools in this matrix has an equivalent). The PR's own comment thread says "the temporary workflow instrumentation will be reverted after applying the exact generated output" — worth reverting now that generation is expected to land, or generalizing into the matrix if it's meant to be permanent, rather than leaving a one-off special case for az.

Looked clean

Required-parameter promotion into record constructors (e.g. AzAcrCreateOptions), removal of obsolete bool shims, the AzureCommandTests.cs/AzureCommandTests updates for renamed/positional ACR properties, AzExtensions.Generated.cs's service registrations (renames like AzComputefleetAzComputeFleet are consistent, additions match new commands), and Az.CommandCoverage.json (valid, commandCount/commands.length both 5131, matches the PR's claimed figures).

@thomhurst

Copy link
Copy Markdown
Owner Author

#4609 (empty-baseline handling in the shared sync script) has merged, so the Azure regeneration that previously died in Merge-PublicApiBaselineSnapshot.ps1 can run again. Dispatched an authoritative az regeneration on this branch (create-pr, no auto-merge): https://github.com/thomhurst/ModularPipelines/actions/runs/34016382524. Its child PR (base = this branch) carries the current-output refresh; merge it here, then re-request review. The earlier CI failures here were the distributed timing flakes since stabilised on main.

@thomhurst

Copy link
Copy Markdown
Owner Author

Cancelled the just-dispatched az regeneration (run 34016382524): #4658 merged generator changes minutes later and #4663 (deterministic enum ordering) is about to merge, so any snapshot produced now would fail Assert-GeneratedOptionsFreshness against main once this PR is rebased. It will be re-dispatched after those generator PRs land.

@thomhurst
thomhurst force-pushed the issue-4331-azure-current-output branch from 5ef57b6 to e399168 Compare September 13, 2026 15:26
@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown

Caution

CodeRabbit couldn't post its review summary.

Error details
Validation Failed: {"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"} - https://docs.github.com/rest/issues/comments#create-an-issue-comment

@thomhurst

Copy link
Copy Markdown
Owner Author

Pushed e399168 after rebasing onto current main. This addresses the remaining scraper gaps behind the latest review: resource-name/ID descriptions, locations, and repeated ACR image tags. Live Azure help confirms the resource-ID alternatives are optional value options, so they remain optional rather than becoming mandatory constructor inputs. Existing description-only tests cover the BuildArg/SecretBuildArg and Synapse findings. All 1,360 generator tests pass; Release generator build and scoped formatting pass.

The temporary Azure recovery workflow changes are removed. Full regeneration is running at https://github.com/thomhurst/ModularPipelines/actions/runs/34765636116 from this exact branch, with auto-merge and shrinkage approval disabled. The generated-output review blocker remains open until that output lands and is validated. Full local Azure validation previously hit the repository's 2 GB guard and was not retried.

@thomhurst
thomhurst deployed to Pull Requests September 13, 2026 15:29 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Review summary

Scraper/generator changes (the actual hand-written logic in this PR) look correct and well-tested.

  • tools/.../Scrapers/Cli/AzCliScraper.cs: the new DescriptionNamesValueOption heuristic (for -name/-id switches whose description paraphrases the switch name, e.g. "The Event Hub name." for --eventhub-name) is correctly guarded — it only fires when the description opens with an article (The/An/A ), so unrelated flag descriptions like "Show name instead of identifier." (--show-name) correctly stay bool? flags, per the new Name_Display_Action_Remains_A_Flag test. I traced all the new [Arguments(...)] cases in Named_Values_Are_Not_Flags through the code by hand and they resolve correctly.
  • The new AzDescriptionOnlyValuePattern / AzNumericValueDescriptionPattern / AzCollectionValueDescriptionPattern regexes and the repeated-image-tag fix in CliScraperBase.cs are narrowly scoped to the observed CLI text and backed by matching regression tests (Nonstandard_Description_Only_Values_Are_Not_Flags, Synapse_Description_Only_Values_Are_Not_Flags, Repeated_Image_Tags_Remain_Collections, the new RepeatableOptionAdapterTests cases). This is the right place to fix it per repo convention (generic scraper logic, not hand-edited generated output).

Blocking issue: the committed generated corpus in this PR does not yet reflect the scraper fixes it contains, and the PR is self-described as not ready to merge.

I verified this concretely rather than just taking the PR description at its word: the new Nonstandard_Description_Only_Values_Are_Not_Flags test exercises az acr build and asserts BuildArg/SecretBuildArgIEnumerable<string>?, Platformstring?, Timeoutint?. But the actual committed src/ModularPipelines.Azure/Options/AzAcrBuildOptions.Generated.cs in this diff still has all four as [CliFlag(...)] public bool? ... — the pre-fix shape. So the ~6,267-file generated corpus in this PR was produced by the old scraper, not the one being added here.

This matches what the author has already stated across the PR thread: full regeneration (run 34765636116) is still pending, and the PR "must remain unmerged until the regenerated output is incorporated, reviewed, and passes Azure validation." As of the reviewed head (e3991684), that regeneration hasn't landed yet, so this remains an open blocker — not a defect in the code shown, but the PR isn't in a mergeable state.

Suggestion: rather than relying on comment-thread discipline to keep the generated output and scraper in sync across a multi-day PR, consider making the Assert-GeneratedOptionsFreshness-style check (mentioned in the PR thread) a required CI status check gating merge on this PR, rather than a manual step run ad hoc. That would make "generated output is stale relative to scraper" a hard CI failure instead of something a reviewer has to spot by diffing a specific file, and would prevent exactly the state this PR is currently in.

No other architectural concerns in the reviewable (non-generated) code.

@thomhurst

Copy link
Copy Markdown
Owner Author

Pushed 28799c7 with recovered authoritative Azure output and the next two fixes.

Run 34805875507 passed API synchronization (591.3s snapshot, 122.9s removals), reconstructed 150,041 APIs, and passed exact staging. Its final build failed only at AzureCommandTests.cs:60 because Executors is now int. The handwritten test now supplies 1 and retains its command-rendering expectation.

Recovered all 6,422 generated files byte-for-byte from that run, including 62 exact filename corrections. Reconstructed the baselines with Merge-PublicApiBaselineSnapshot.ps1 from the run's original baselines, current API snapshot, and confirmed removals. The originals match this branch after line-ending normalization. Verified 4,987 staged generated/API paths against the CI manifest and baseline allowlist. API impact: 88,017 additions, 16,183 removals/changes, and 9,067 same-name signature changes. Generated files and baselines were not hand-edited.

Manual review also found that actual help says "Multiples are supported by passing '--build-arg name[=value]' multiple times", while the earlier fixture omitted quotes and the value example. Shared repeat detection now recognizes matching single/double/backtick quotes; all three added cases failed before the fix. The old unquoted case remains covered, and repeated options remain ungrouped. BuildArg and SecretBuildArg are still scalar in the recovered d9b6 output, so fresh generation remains a merge prerequisite.

All 1,451 generator tests pass after the fix and formatting cleanup; Release generator build has zero warnings/errors. Scoped severity-info generator verification, scoped Azure test formatting, all 61 API baseline checks, artifact hash/name checks, and diff checks pass. Local Azure compilation/test remains deferred after the earlier 2-GB limit, without a retry or increased limit.

Fresh authoritative Azure generation from this exact head: https://github.com/thomhurst/ModularPipelines/actions/runs/34810397663. Auto-merge and coverage-shrinkage approval are disabled. This run must update the repeatable properties, provenance, and generated documentation before this PR can merge.

@thomhurst
thomhurst deployed to Pull Requests September 14, 2026 05:41 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Review summary

Scope note: this PR regenerates ~6,400 Azure CLI option files. Per this repo's CLAUDE.md, generated *.Generated.cs/PublicAPI.*.txt output is authoritative and out of scope for hand-editing/second-guessing at the line level — but it must actually match what the current scraper produces, and that's where the previously-reported blocker lives.

Blocking — unchanged from the prior review (originally flagged at 852322f9): the committed generated corpus still doesn't match this PR's own scraper logic.

I re-verified this at the current head (28799c79). src/ModularPipelines.Azure/Options/AzAcrBuildOptions.Generated.cs:36-37 and :90-92 still declare:

[CliOption("--build-arg")]
public string? BuildArg { get; set; }
...
[SecretValue]
[CliOption("--secret-build-arg")]
public string? SecretBuildArg { get; set; }

Their doc comments (lines 34 and 88) contain the exact real Azure CLI text: Multiples are supported by passing '--build-arg name[=value]' multiple times. / ...'--secret-build-arg name[=value]' multiple times.. I traced this against the scraper's own RepeatableValuePattern regex in CliScraperBase.cs:1493-1512 (the multiples? ... supported by passing (?:...|quoted example) multiple times alternative) — it matches this description text. I also confirmed tools/.../AzCliScraperTests.cs:327-355 (Nonstandard_Description_Only_Values_Are_Not_Flags, parameterized with quote = "'") feeds this exact description string through TestAzCliScraper().Parse(...) and asserts BuildArg and SecretBuildArg both resolve to CSharpType == "IEnumerable<string>?". There's no per-command override for acr.build in TypeOverrides/az.json that would force these scalar. So the scraper this PR ships would regenerate these two properties as IEnumerable<string>?, but the committed .Generated.cs still has string? — the corpus predates this PR's own fix and hasn't been refreshed.

This is the same defect the prior review blocked on, just re-verified against the current head rather than assumed carried-over.

Architectural point (repeated from the prior review, since it's still what's letting this recur): Assert-GeneratedOptionsFreshness.ps1 / dotnet.yml only fingerprint the generator against the PR's base branch, never the PR's own head, so CI structurally can't catch "this PR's own scraper changes aren't reflected in this PR's own committed generated output." That's why this has to be caught by manually spot-checking a single file in a 6,000-file diff instead of failing CI automatically. Fingerprinting against the PR head's own generator source (or running an actual generate-and-git diff --exit-code step) would turn this into a hard, automatic CI failure instead of a manual-review catch.

No other concerns in the reviewable (non-generated) code — the scraper heuristics, the two-pass Sync-PublicApiBaselines.ps1 rewrite, the case-only rename tracking in CodeGeneratorOrchestrator.cs, and the new trace-public-api-compiler.py diagnostics tool all look correct and are covered by their new tests.

Co-authored-by: thomhurst <9139608+thomhurst@users.noreply.github.com>
@thomhurst
thomhurst deployed to Pull Requests September 14, 2026 06:49 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Review summary

The previously-blocking issue is resolved. Every prior review round on this PR (7 rounds, heads 852322f9 through 28799c79) blocked on the same defect: the committed ~6,400-file generated Azure corpus predated this PR own scraper fixes, most concretely src/ModularPipelines.Azure/Options/AzAcrBuildOptions.Generated.cs still declaring BuildArg/SecretBuildArg/Platform/Timeout as bool?/string? instead of the IEnumerable<string>?/string?/int? shape the new Nonstandard_Description_Only_Values_Are_Not_Flags test asserted.

I re-verified this directly at the current head (6170285, landed via the final chore: Update az CLI options (#4951) merge commit):

  • AzAcrBuildOptions.Generated.cs:37,79,92,110 now correctly declare IEnumerable<string>? BuildArg, string? Platform, IEnumerable<string>? SecretBuildArg, and int? Timeout, matching the scraper repeatable-value and description-only-value fixes and their regression tests.
  • AzSynapseSparkJobSubmitOptions/AzSynapseSparkSessionCreateOptions now declare Executors as required int (not bool?), matching PublicAPI.Unshipped.txt and the corrected regression tests.
  • test/ModularPipelines.Azure.UnitTests/AzureCommandTests.cs:60 was updated in lockstep, passing Executors: 1 to match the new int signature.

The two commits since the last review (28799c793d6c through 6170285) are the authoritative Azure regeneration itself (chore: Update az CLI options) plus its immediate predecessor, both already confirmed data-only against the already-reviewed scraper/generator logic. No new hand-written logic shipped in that window.

Hand-written logic (unchanged from prior rounds, still solid): the scraper heuristics in AzCliScraper.cs/CliScraperBase.cs (article-led -name/-id description matching, the quote-backreference fix for quoted repeat examples, description-only numeric/collection detection), the two-pass Sync-PublicApiBaselines.ps1 rewrite, the case-only rename tracking through CodeGeneratorOrchestrator.cs, and the new trace-public-api-compiler.py diagnostics tool all remain correctly scoped and covered by dedicated tests, consistent with the last several review rounds.

Non-blocking, for awareness (low confidence, no confirmed real-world trigger across the 6,422 regenerated commands):

  1. The RepeatableValueRegex quoted-example alternative in CliScraperBase.cs excludes all three quote characters from the middle of the example rather than just the one that opened it, relying on a named-group backreference for the closing quote. If a real CLI description ever mixed quote styles inside a single example, the match could stop short of the true closing quote and the option would fall back to scalar instead of a collection type. Every example in this corpus uses one consistent quote style per description, so this has not fired; worth a regression test if a future CLI surface mixes styles.
  2. The DescriptionNamesValueOption heuristic in AzCliScraper.cs (flags named like *-name/*-id whose description restates the switch name) is guarded by requiring an article-led description, but does not otherwise constrain the switch semantics, so a hypothetical boolean flag documented in that shape could misclassify. Not observed in the current corpus; noting only so the pattern is on record.

Architectural point worth reconsidering (repeated from prior rounds, since it is what let the corpus/scraper mismatch recur across five-plus push cycles): Assert-GeneratedOptionsFreshness.ps1 and dotnet.yml fingerprint the generator against the base branch, never the head of this PR, so CI structurally cannot catch a case where changes to the scraper in this PR are not reflected in the generated output committed in this same PR. That gap had to be caught by manual review of a single file in a 6,000-file diff, which is exactly what happened here across a week of iteration. Now that the corpus is finally in sync, a follow-up CI check that fingerprints against the generator source at the PR head itself (or runs generate-and-diff) would turn this class of staleness into an automatic CI failure instead of requiring another multi-round manual catch.

No other concerns in the reviewable, non-generated code. This PR is in a mergeable state as far as the diff content goes.

@thomhurst
thomhurst force-pushed the issue-4331-azure-current-output branch from 27b7166 to 6c637dd Compare September 14, 2026 07:20
@thomhurst

Copy link
Copy Markdown
Owner Author

Pushed 6c637dd after manual review of the regenerated Azure corpus found two additional defects.

  1. AdminUser/AdminPassword/RestoreTime/Expiry still used CliFlag/bool? despite descriptions and CLI examples requiring values. The generic Azure description parser now recognizes credential and datetime value descriptions. Password options retain secret masking; action descriptions such as Reset the administrator password and Use the current time remain presence-only flags. Seven new cases failed before the fix, including captured Azure CLI 2.84.0 MySQL/Postgres help matching the descriptions in the generated 2.90.0 corpus.
  2. Generated documentation always said global options precede subcommands, even though Azure's metadata and generated base class explicitly put them afterward. MarkdownDocumentationGenerator now uses GlobalOptionsBeforeSubcommands. The new after-subcommand regression failed before the fix; the existing before-subcommand path remains covered.

Validation: all 1,459 generator tests pass; generator Release solution build has zero warnings/errors; scoped severity-info verification passes. Fixture normalization only removes trailing blank lines. No generated option or API baseline was hand-edited. The existing local Azure 2-GB compilation limit remains deferred to CI.

Authoritative regeneration for this exact source: https://github.com/thomhurst/ModularPipelines/actions/runs/34817378641. Generator fingerprint: 99abaa76e0a93723e9a40cff3069b062b5876c8fc306e94e283120506110a1bc. Auto-merge and coverage-shrinkage approval are disabled. The parent must remain unmerged until the new value-bearing options and corrected documentation are incorporated and reviewed.

@github-actions

Copy link
Copy Markdown
Contributor

Review summary

Blocking — the committed generated corpus still doesn't reflect this PR's own final scraper fixes. This is the same defect class every prior round on this PR blocked on (7+ rounds through 28799c79, resolved at 6170285), now recurring for the two additional fixes introduced in the latest commit (6c637ddd826f868d42940d6a7463b7d48abb90fd, fix(generator): preserve Azure credentials and datetime values).

That commit adds AzCliScraperTests.Captured_Database_Help_Preserves_Scalar_Values and Credential_And_DateTime_Descriptions_Declare_Values, asserting that credential/datetime-shaped descriptions (--admin-user, --admin-password, --restore-time, --expiry) now parse as IsFlag == false, CSharpType == "string?". Re-verified directly against the committed output at this exact head:

  • src/ModularPipelines.Azure/Options/AzMysqlFlexibleServerCreateOptions.Generated.cs:218-225AdminPassword and AdminUser are still [CliFlag(...)] public bool?, not string?.
  • src/ModularPipelines.Azure/Options/AzPostgresFlexibleServerRestoreOptions.Generated.cs:83-84RestoreTime is still [CliFlag("--restore-time")] public bool?.
  • src/ModularPipelines.Azure/Options/AzAmsAssetGetSasUrlsOptions.Generated.cs:27 and AzStorageContainerPolicyCreateOptions.Generated.cs:38Expiry is still bool?.

The second fix in the same commit (AzCliScraper.GlobalOptionsBeforeSubcommands flipped to false, MarkdownDocumentationGenerator now renders "before"/"after" based on it) is equally unregenerated:

  • docs/docs/mp-packages/cli/az.md:46 still reads "Global options are rendered before the selected subcommand," even though the scraper now declares global options render after the subcommand for az.

Both the PR description and the commit message acknowledge this directly — "This PR remains unmerged until the new output is incorporated and reviewed" — so this isn't a surprise, but it does mean the PR is not yet in a mergeable state: the 6,422-file Azure corpus and the docs need one more authoritative regeneration pass before these two fixes actually take effect anywhere outside the generator's own test suite.

Architectural point (repeated from every prior round, since it's what lets this recur every time): Assert-GeneratedOptionsFreshness.ps1 / dotnet.yml fingerprint the generator against the base branch, never against this PR's own head. That means CI structurally cannot fail when a PR's own scraper/generator change isn't reflected in that same PR's own committed output — it has to be caught by manually spot-checking individual files in a 6,000+ file diff, which is exactly what's happened across nine review rounds now. A CI step that regenerates from the PR head and runs git diff --exit-code against the committed corpus would turn this whole class of staleness into an automatic, immediate CI failure instead of a recurring manual catch.

Hand-written logic (unchanged from prior rounds, still solid): the scraper heuristics in AzCliScraper.cs/CliScraperBase.cs, the two-pass Sync-PublicApiBaselines.ps1 rewrite, and the case-only rename tracking through CodeGeneratorOrchestrator.cs remain correctly scoped and covered by dedicated tests.

@thomhurst

Copy link
Copy Markdown
Owner Author

Pushed aecd27d with both #4955 defects fixed in source. Pwd identifier segments now receive secret metadata while paths, names, IDs, and unrelated segments remain visible. Azure list inference uses the input definition instead of later references to reserved/allowed-value lists, keeping VM/VMSS AdminUsername scalar.

All 1,479 generator tests pass; eleven new cases fail before the fixes. Full captured Azure CLI 2.84.0 VM, VMSS, and SQL VM help reproduces the affected descriptions; only trailing blank lines were trimmed. Seven new Azure integration regressions cover the generated password annotations and actual scalar username rendering. Their execution remains deferred until generated output is incorporated, alongside the previously recorded Azure compilation 2-GB limit.

Release generator solution build has zero warnings/errors. Scoped information-level generator and Azure test formatting verification pass. Full authoritative generation from this exact commit: https://github.com/thomhurst/ModularPipelines/actions/runs/34823681275, with auto-merge and coverage-shrinkage approval disabled. Fingerprint: 44ca29f5f7a70bca3524d42172fd82be9545fbcf615b95d441cd8609a513bad8.

The documentation-order concern on #4955 is already fixed in this parent: MarkdownDocumentationGenerator uses tool.GlobalOptionsBeforeSubcommands and both before/after cases are tested. Keeping the generated child targeted at this source branch preserves that fix when #4546 merges into main; retargeting the child alone would omit its generator prerequisite. Current #4955 remains blocked pending final-source regeneration.

@github-actions

Copy link
Copy Markdown
Contributor

Review

This PR regenerates Azure CLI options from a fresh, larger CLI corpus (about 6,422 commands) and includes real fixes to the OptionsGenerator scraping/documentation logic. Per repo guidance, generated *.Generated.cs files are authoritative and were not second-guessed; the review focused on the hand-authored scraper/generator/tooling code.

Blocking

1. docs/docs/mp-packages/cli/az.md:46 -- stale global-options ordering text

MarkdownDocumentationGenerator.cs now derives the "rendered before/after" sentence from tool.GlobalOptionsBeforeSubcommands instead of hardcoding "before" (Generators/MarkdownDocumentationGenerator.cs:98). AzCliScraper.GlobalOptionsBeforeSubcommands overrides to false, and AzCliScraperTests.cs:521 asserts this. So the regenerated az.md should now say "after" -- but the committed file still reads:

Global options are rendered before the selected subcommand.

This contradicts the feature described in this PR ("Generated global-option documentation follows each tool declared argument order") and the generator own logic/tests. It suggests az.md was not regenerated against the final generator build, or the doc-generation step in generate-cli-options.yml ran out of order relative to this fix. Since this doc file is generated output that ships to users, it is worth confirming the generation pipeline actually re-ran end-to-end before merge, not just the .Generated.cs option files.

2. AzCliScraper.cs:556-557 -- anchored regex narrows "list of" detection, likely a type-inference regression

The prior check was a plain substring test, description.Contains("list of") (case-insensitive, anywhere in the text) -- used both in GetOptionType and HelpDeclaresGroupedValues. This PR replaces it with AzListValueDescriptionPattern, an anchored regex (^(?:specify|...)?(?:a|an|the)?(?:json|ordered|comma-separated|space-separated)?\s*list\s+of\b) that only matches when "list of" appears at, or very near, the start of the description.

Any option whose help text places "list of" mid-sentence -- for example a second sentence such as "Resource tags. A list of key=value pairs." -- will no longer be recognized as a list, silently narrowing it from IEnumerable<string> to a scalar string in the generated options. Since this scraper is meant to broaden/refine list detection (per the PR description), this specific change narrows it in a way that could regress real options across the larger catalog. Worth checking the regenerated output for any option that previously came out as IEnumerable<string> and now comes out as string, or relaxing the anchor so it also matches after sentence boundaries rather than only at the very start of the description.

Non-blocking, worth a look

3. scripts/trace-public-api-compiler.py:15-27 (is_descendant) -- PID-reuse race

find_compilers reads /proc/<pid>/cmdline for a candidate PID, then separately calls is_descendant, which re-reads /proc/<pid>/status (and walks up via PPid) afterward. Between the two reads, the process can exit and the PID can be recycled by an unrelated process on a busy CI runner, so is_descendant could end up sampling or reporting against the wrong process. This is a narrow race and the tool is diagnostic only (compiler tracing for API-sync tooling), so severity is low -- but if it proves flaky in CI, pinning identity via the start time in /proc/<pid>/stat in addition to the PID would close the gap.

Not reviewed in depth

The *.Generated.cs option files and the PublicAPI.Unshipped.txt / PublicAPI.Shipped.txt changes are treated as authoritative generator output per CLAUDE.md and were not hand-checked line by line; the concerns above (particularly #2) are the more useful signal on whether that output is currently correct.

@thomhurst

Copy link
Copy Markdown
Owner Author

Addressed comment 5661442109 in 9eaa3b9.

  1. The committed az.md is still from earlier output, as recorded by the pending-generation gate. MarkdownDocumentationGenerator is registered alongside the option generators and uses the tool's declared global-option order; Azure defaults to the generated docs directory. Replacement full generation includes documentation: https://github.com/thomhurst/ModularPipelines/actions/runs/34826041981. We must verify the resulting az.md says "after" and its provenance matches this source before merging. No generated document was manually edited.
  2. Confirmed the list concern against full Azure CLI 2.84.0 netappfiles account ad add help. Administrators, BackupOperators, and SecurityOperators define their username lists after introductory sentences. All three captured cases and two additional sentence-boundary cases fail before the fix. The shared Azure list predicate now recognizes these definitions for both value detection and cardinality, while VM/VMSS usernames and the SQL time-zone description's externally exposed list remain scalar. All 263 relevant generator tests pass, including all 61 Azure tests. Three Azure rendering tests were added and await replacement output, alongside the existing seven credential/username regressions.
  3. Reproduced PID reuse between reading cmdline and checking ancestry. Compiler selection now verifies start time before and after inspection; sampling tracks PID plus start time and rechecks identity before launching the trace. Replacement processes cannot inherit the previous compiler's 120-second delay. All seven diagnostic tests pass in Linux, including the replacement and delay regressions.

Release generator build has zero warnings/errors; scoped information-level formatting and diff checks pass. Existing 2-GB validation limits remain unchanged. Current Linux Azure integration failures are the seven expected old-output contract failures; final-source regeneration remains required. Fingerprint: a6c1ce44cc70d83e8347ded58b42d94a37c0ba27ca672676467daf26a2fc6bb6.

@github-actions

Copy link
Copy Markdown
Contributor

Review: Refresh Azure CLI options from current output (#4546)

Re-review scope: this PR has been through 9+ prior rounds, all blocked on the same defect class — the committed ~6,400-file generated Azure corpus not reflecting this PR's own scraper fixes. The latest push (head 9eaa3b909941d1bea6def90b3ab1236edac28130) claims to address the previous round's three findings (stale az.md, an over-anchored "list of" regex, and a PID-reuse race in the tracing script) via "replacement full generation" run 34826041981. I verified each claim directly against the committed files at this exact head rather than trusting the PR description.

Still blocking — the generated corpus still doesn't reflect this PR's own fixes (recurring for the 10th round)

1. docs/docs/mp-packages/cli/az.md:46 is still stale. AzCliScraper.GlobalOptionsBeforeSubcommands is false (AzCliScraper.cs:74), so MarkdownDocumentationGenerator.cs:98 should render "after". The committed file still reads "Global options are rendered before the selected subcommand." — unchanged from the previous blocking round. The author's own comment acknowledges this needs verification post-regeneration ("We must verify the resulting az.md says 'after'... before merging") but the current head still has the old text.

2. The "list of after an introductory sentence" fix is not reflected in AzNetappfilesAccountAdAddOptions.Generated.cs. This PR adds Captured_Netapp_Help_Preserves_Lists_Defined_After_An_Introduction specifically asserting Administrators/BackupOperators/SecurityOperators become IEnumerable<string>? via [CliOption]. The committed file still declares all three as [CliFlag] public bool? (lines 56, 74, 134) — the exact pre-fix shape the new test exists to rule out. Note the file was regenerated in this PR (obsolete shim properties were removed, several other CliFlagCliOption corrections landed, positional constructor params were added, descriptions were completed) — it just didn't pick up this specific fix.

3. The VM/VMSS admin-username and SQL VM password fixes (the headline fixes of the latest commit, aecd27d4) are also not reflected. AzVmCreateOptions.Generated.cs:385-386 still has [CliFlag] public bool? AdminUsername, not the scalar [CliOption] string? that Captured_Vm_Help_Keeps_Admin_Username_Scalar asserts. AzSqlVmAddToGroupOptions.Generated.cs:47-60 still has BootstrapAccPwd/OperatorAccPwd/ServiceAccPwd as [CliFlag] public bool? rather than secret-masked [CliOption] string?, contradicting Captured_Sql_Vm_Help_Classifies_Account_Passwords_As_Secrets.

Taken together, none of the three concrete scenarios called out across the last two review rounds (list-after-introduction, VM/VMSS username, SQL VM passwords) actually took effect in the committed output at this head, despite the PR description and commit messages asserting the "final-source" regeneration incorporated them. This is the same root cause as every prior round: whatever regeneration pipeline run produced the currently-committed .Generated.cs/.md files predates (or didn't correctly pick up) this branch's final scraper commit aecd27d4b1c49aa5d3504948e3d4a35dbc6f4383.

Hand-written logic (unchanged from prior rounds — still solid)

  • AzCliScraper.cs: DescriptionNamesValueOption (article-led -name/-id paraphrase detection), the new description-only/credential/datetime/numeric/collection regexes, and the updated AzListValueDescriptionPattern (now correctly matching "list of" at a sentence boundary via (?:^|(?<=[.!?])\s+), addressing the previously-flagged over-anchoring) are all well-scoped and covered by targeted tests (AzCliScraperTests.cs), including fixture-backed regressions against captured Azure CLI 2.84.0 help text.
  • GeneratorUtils.cs: the Pwd secret-keyword addition uses the existing boundary-aware ContainsIdentifierSegment helper, correctly excluding PwdFile/PwdPath (via IsFilePathOption), PwdName (via the Name suffix check), PwdId (via the exact-match IsSecretIdentifier allowlist), and SpwdEntry (lowercase p fails the boundary check) — verified by GeneratorUtilsTests.cs.
  • Sync-PublicApiBaselines.ps1: the two-pass rewrite (discover additions against emptied baselines, then check removals against original + additions, restoring originals in a finally on any failure) is correct — $originalShipped/$originalUnshipped are copied to a separate temp path before any mutation, so the failure-path restore is safe.
  • CodeGeneratorOrchestrator.cs: case-only rename tracking (EnsureExactFileNameCasing returning the renamed path, threaded into FilesDeleted) is coherent, and the complexity-reduction refactors (GetEmptyCliGenerationFailureAsync, WriteGeneratedFilesAsync) preserve behavior.
  • trace-public-api-compiler.py: the previously-flagged PID-reuse race is fixed — find_compilers now captures process_start_time before inspecting a candidate and re-checks it hasn't changed before yielding, closing the window where a recycled PID could be misattributed.

Architectural point (repeated from every prior round)

Assert-GeneratedOptionsFreshness.ps1 / dotnet.yml fingerprint the generator against the base branch, never this PR's own head. That is structurally why this exact defect (scraper fix present, generated output absent) has now recurred across 10 review rounds and 3 different concrete fixes within the same PR — nothing in CI can catch "this PR's own generated corpus doesn't match this PR's own generator." A CI step that regenerates from the PR head and diffs against the committed corpus would turn this into an immediate, automatic failure instead of requiring another manual spot-check every round. Given how consistently this has bitten the PR, it's worth prioritizing over further manual regeneration attempts.

Not reviewed in depth: the remaining ~6,400 generated .Generated.cs/PublicAPI.*.txt files, per CLAUDE.md — these are machine output and authoritative when correct, but the three spot-checks above show the current commit isn't yet the correct output.

@thomhurst

Copy link
Copy Markdown
Owner Author

Pushed 6712adedebc2a48752a311c2bdea8879a6c1fa2a for child #4955 findings 4003756695 and 4004086711. Captured Azure CLI 2.84.0 help confirms the misspelled Space-separeted list wording for SignalR allow/deny/connection-name and the qualified allowed-list wording for Google/Microsoft authentication audiences. The generic Azure list detector now recognizes those input definitions without treating reserved-value documentation as multiple inputs.

Eight regressions failed before the change. All 70 Azure scraper tests and the full 1,494-test generator suite pass. Release generator build has zero warnings/errors; scoped severity-info generator and Azure test formatting/verification passes. Five new Azure rendering contracts cover all affected properties; their execution remains deferred to CI under the prior Azure compilation 2 GB limit.

Fresh full Azure regeneration is https://github.com/thomhurst/ModularPipelines/actions/runs/34832142558 from this exact source, with auto-merge and coverage-shrinkage approval disabled. #4955 remains blocked until replacement generated options/API/provenance incorporate these corrections. Existing parent output findings also remain held until generation and integration validation complete.

@github-actions

Copy link
Copy Markdown
Contributor

Review: Refresh Azure CLI options from current output (#4546)

Re-review scope: this PR has now been through more than a dozen review rounds, essentially all blocked on the same defect class — the committed ~6,400-file generated Azure corpus not reflecting this PR's own scraper fixes. Since the last review (head 9eaa3b90), one commit landed: 6712aded, which extends AzListValueDescriptionPattern to recognize the misspelled Space-separeted list wording (SignalR network-rule update --allow/--deny/--connection-name) and the "configuration settings of the allowed list" wording (Container Apps --allowed-audiences). I checked the three previously-blocking findings plus the new commit's own regeneration directly against the committed files at this exact head.

Still blocking — the generated corpus still doesn't reflect this PR's own fixes

1. The three previously-flagged stale files are unchanged.

  • docs/docs/mp-packages/cli/az.md:46 still reads "Global options are rendered before the selected subcommand," even though AzCliScraper.GlobalOptionsBeforeSubcommands is false.
  • AzNetappfilesAccountAdAddOptions.Generated.cs:56,74,134 still declares Administrators/BackupOperators/SecurityOperators as [CliFlag] public bool?, not the IEnumerable<string>? the PR's own Captured_Netapp_Help_Preserves_Lists_Defined_After_An_Introduction test requires.
  • AzVmCreateOptions.Generated.cs:386 still has [CliFlag] public bool? AdminUsername, and AzSqlVmAddToGroupOptions.Generated.cs:48,54,60 still has BootstrapAccPwd/OperatorAccPwd/ServiceAccPwd as [CliFlag] public bool? rather than secret-masked [CliOption] string?.

2. New evidence from this round's own commit, in a single file. AzSignalrNetworkRuleUpdateOptions.Generated.cs is a good illustration of why this keeps happening: it was partially regenerated for 6712aded's fix — Allow (line 26-27) and Deny (line 32-33) are now correctly [CliOption(..., GroupValues = true)] IEnumerable<string>?. But the third property the same fix and the same new test (Captured_Help_Preserves_Qualified_List_Definitions, parameterized over Allow/Deny/ConnectionName from the same fixture) explicitly covers, ConnectionName (line 38-39), is still [CliFlag] public bool? ConnectionName, even though its description ("Space-separeted list of private endpoint connection name.") matches the exact same AzListValueDescriptionPattern regex as Allow/Deny. This isn't just "regeneration hasn't run since the last commit" (which would leave all three stale) — it's a regeneration that landed inconsistently within the same file, for properties covered by the same test case.

Given the two ContainerApp AllowedAudiences properties from the same commit did land correctly (AzContainerappAuthGoogleUpdateOptions.Generated.cs:26-27, AzContainerappAuthMicrosoftUpdateOptions.Generated.cs:26-27), the regeneration pipeline is clearly capable of producing the right output — it's just not reliably producing it for every option the PR's own tests assert on, across many runs now. That's a stronger case than before for treating the generated corpus as unverified until it's checked mechanically rather than by another round of manual spot-checks.

Hand-written logic (unchanged from the last-confirmed-good round)

  • AzCliScraper.cs's AzListValueDescriptionPattern update (adding space-separ[ae]ted and the configuration settings of ... allowed list wording to the introductory-sentence match) is correctly scoped and covered by Captured_Help_Preserves_Qualified_List_Definitions in AzCliScraperTests.cs, which is the right way to add this case — it doesn't weaken the existing negative lookahead that excludes "list is documented elsewhere" phrasing.
  • No other hand-written files changed since 9eaa3b90 (GeneratorUtils.cs, Sync-PublicApiBaselines.ps1, CodeGeneratorOrchestrator.cs, trace-public-api-compiler.py are all identical to the round already confirmed correct).

Architectural point (repeated from every prior round)

Assert-GeneratedOptionsFreshness.ps1 is still byte-identical to main in this PR — it still fingerprints the generator against the base branch, never this PR's own head, and there's still no CI step that regenerates from the PR head and diffs against the committed corpus. That gap is the root cause of every round of this review, including the new same-file partial-regeneration example above, which no existing check would catch. At this point, with the defect recurring even within a single file for a single test case, this seems worth landing before attempting another full regeneration — otherwise there's no way to know the next attempt didn't silently drop something else.

Not reviewed in depth: the remaining ~6,400 generated .Generated.cs/PublicAPI.*.txt files, per CLAUDE.md — these are machine output and authoritative when correct, but the spot-checks above (four now, including one intra-file inconsistency) show the current commit still isn't the correct output.

Co-authored-by: thomhurst <9139608+thomhurst@users.noreply.github.com>
@thomhurst
thomhurst deployed to Pull Requests September 14, 2026 11:56 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Review (head 14b65bbdc)

This PR has been through many review rounds already, essentially all blocked on the generated Azure corpus lagging behind the scraper fixes it contains. I re-verified the previously-flagged blockers against the current head and checked for anything new.

Previously-flagged issues — confirmed resolved at this head:

  • AzAcrBuildOptions.Generated.cs: BuildArg/SecretBuildArg are now IEnumerable<string>?, Platform is string?, Timeout is int? — matches the Nonstandard_Description_Only_Values_Are_Not_Flags test.
  • AzSignalrNetworkRuleUpdateOptions.Generated.cs: Allow, Deny, and ConnectionName are now all correctly IEnumerable<string>? (previously ConnectionName was stale while its siblings were fixed).
  • docs/docs/mp-packages/cli/az.md: now correctly reads "rendered after the selected subcommand", matching AzCliScraper.GlobalOptionsBeforeSubcommands = false.
  • The 20-minute timeout-minutes on "Synchronize generated public API baselines": an earlier full-scale run did hit this ceiling, which is why the dotnet-trace diagnostics were added; a subsequent full-scale run completed the snapshot+removal builds in ~616s total, comfortably within budget. I don't see a live regression here.

New finding (not previously raised):

tools/ModularPipelines.OptionsGenerator/src/ModularPipelines.OptionsGenerator/Generators/MarkdownDocumentationGenerator.cs:214-217 (AppendExample) no longer emits using {tool.TargetNamespace}.Enums; for the generated "Module example" snippet. Compare with the sibling generator, OptionsClassGenerator.cs:94-96, which still gates on GeneratorUtils.UsesGeneratedEnums(tool, command) and emits the enum usingMarkdownDocumentationGenerator dropped that same check entirely, so it only ever emits using ModularPipelines; and using {tool.TargetNamespace}.Options;.

If any tool's curated DocumentationExampleCatalog entry ever selects a command with an enum-typed option (the pattern already exists elsewhere in this codebase, e.g. pnpm's AuditLevel), the generated docs page's C# example will reference SomeEnum.Value without importing {Namespace}.Enums, producing a sample that doesn't compile if copied verbatim.

This regression is undetectable by the existing test suite because the same commit that removed the check also stripped the enum scenario out of MarkdownDocumentationGeneratorTests.cs (GenerateAsync_UsesCuratedExamplesForPopularTools) — the PnpmAuditAuditLevel enum, Enums = [auditLevel], and the expected using {{tool.TargetNamespace}}.Enums; line were all deleted from the test's expected output alongside the production code change. So there's no regression test standing guard here, unlike OptionsClassGenerator's equivalent path.

Suggested fix: restore the GeneratorUtils.UsesGeneratedEnums(tool, command) gate (and the using {tool.TargetNamespace}.Enums; line) in AppendExample, mirroring OptionsClassGenerator, and restore (or re-add) a test case that exercises a curated example command with an enum-typed option so this can't silently regress again.

No other architectural concerns — the hand-written scraper/generator logic elsewhere is well-scoped and test-backed, per the extensive prior review history on this PR.

@thomhurst

Copy link
Copy Markdown
Owner Author

Regarding review 5663839708: the reported enum-import removal is not part of this PR's changes. The PR's three-dot diff changes global-option ordering and equivalent formatting in MarkdownDocumentationGenerator; it does not remove the enum import or the enum test scenario.

Those additions landed separately on main in 45b3ba1 (#4713), after this branch diverged. I computed the conflict-free merge of current main 4f9c42c and this PR's exact head 14b65bb. The resulting tree 22f3652c406c49bf8525ddd1e85021928c7026cd retains both:

  • AppendExample's GeneratorUtils.UsesGeneratedEnums(tool, command) gate and generated namespace import.
  • MarkdownDocumentationGeneratorTests' PnpmAuditAuditLevel enum, EnumDefinition/Enums metadata, and expected enum namespace import.

The proposed fix is therefore already present in the merged result; no additional source change is needed. The latest review confirms the previous Azure-output blockers are resolved. All 17 current checks are terminal and successful/neutral/expected-skipped, including Linux pipeline, Windows/macOS builds, and documentation deployment. There are no unresolved review threads.

@thomhurst
thomhurst merged commit b17fe5d into main Sep 14, 2026
17 checks passed
@thomhurst
thomhurst deleted the issue-4331-azure-current-output branch September 14, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automated breaking Breaking API change dependencies Pull requests that update a dependency file .NET Pull requests that update .net code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant