Skip to content

az containerapp job start --image replaces the requested image with the quickstart image when --registry-identity system is used #33548

@eric15342335

Description

@eric15342335

When starting an existing Azure Container Apps job with both:

--image <private-acr-image>
--registry-identity system

the execution does not use the image supplied through --image.

Instead, Azure CLI submits the following public quickstart image in the execution template:

mcr.microsoft.com/k8se/quickstart:latest

The existing job is already configured with:

  • A system-assigned managed identity
  • A private Azure Container Registry
  • The registry identity set to system
  • The requested private image in the stored job template

Therefore, no identity bootstrap should be required when starting an execution.

Related command

az containerapp job start

Errors

No CLI exception or non-zero exit code is returned.

az containerapp job start completes successfully with exit code 0 and creates an execution. However, inspecting the resulting execution shows that the image supplied through --image was replaced.

The execution was started using:

$execution = az containerapp job start `
  --resource-group "<RESOURCE_GROUP>" `
  --name "<JOB_NAME>" `
  --image "<ACR_NAME>.azurecr.io/<REPOSITORY>:<TAG>" `
  --container-name "<CONTAINER_NAME>" `
  --cpu 0.25 `
  --memory 0.5Gi `
  --registry-identity system `
  --query name `
  --output tsv

The resulting execution was inspected using:

az containerapp job execution show `
  --resource-group "<RESOURCE_GROUP>" `
  --name "<JOB_NAME>" `
  --job-execution-name $execution `
  --query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
  --output jsonc

The inspection command returned:

[
  {
    "cpu": 0.25,
    "image": "mcr.microsoft.com/k8se/quickstart:latest",
    "memory": "0.5Gi",
    "name": "<CONTAINER_NAME>"
  }
]

The command therefore reports success, but the execution uses mcr.microsoft.com/k8se/quickstart:latest instead of the private image explicitly supplied through --image.

Issue script & Debug output

$ErrorActionPreference = "Stop"

$SubscriptionId = "<SUBSCRIPTION_ID>"
$ResourceGroup = "<RESOURCE_GROUP>"
$EnvironmentName = "<CONTAINER_APPS_ENVIRONMENT>"
$AcrName = "<ACR_NAME>"
$WorkloadProfileName = "Consumption"

$Suffix = Get-Date -Format "MMddHHmmss"
$JobName = "cli-job-start-repro-$Suffix"
$ContainerName = "repro"
$Repository = "cli-job-start-repro"
$Tag = $Suffix
$SourceImage = "mcr.microsoft.com/k8se/quickstart-jobs:latest"

$ExecutionName = $null
$JobCreated = $false
$ImageImported = $false

function Assert-AzSuccess {
    param([string]$Operation)

    if ($LASTEXITCODE -ne 0) {
        throw "$Operation failed with exit code $LASTEXITCODE."
    }
}

try {
    Write-Host "Selecting the test subscription"

    az account set `
      --subscription $SubscriptionId

    Assert-AzSuccess "Selecting the subscription"

    $LoginServerOutput = az acr show `
      --name $AcrName `
      --query loginServer `
      --output tsv

    Assert-AzSuccess "Reading the ACR login server"

    $LoginServer = ($LoginServerOutput | Out-String).Trim()
    $PrivateImage = "${LoginServer}/${Repository}:${Tag}"

    Write-Host "Importing a small test image into the private ACR"

    az acr import `
      --name $AcrName `
      --source $SourceImage `
      --image "${Repository}:${Tag}" `
      --force `
      --output none

    Assert-AzSuccess "Importing the test image"
    $ImageImported = $true

    Write-Host "Creating a temporary job on the Consumption workload profile"

    az containerapp job create `
      --resource-group $ResourceGroup `
      --name $JobName `
      --environment $EnvironmentName `
      --trigger-type Manual `
      --replica-timeout 300 `
      --replica-retry-limit 0 `
      --replica-completion-count 1 `
      --parallelism 1 `
      --image $PrivateImage `
      --container-name $ContainerName `
      --cpu 0.25 `
      --memory 0.5Gi `
      --workload-profile-name $WorkloadProfileName `
      --mi-system-assigned `
      --registry-server $LoginServer `
      --registry-identity system `
      --output none

    Assert-AzSuccess "Creating the temporary job"
    $JobCreated = $true

    Write-Host ""
    Write-Host "Stored job configuration"
    Write-Host ""

    az containerapp job show `
      --resource-group $ResourceGroup `
      --name $JobName `
      --query '{identityType:identity.type,registries:properties.configuration.registries,workloadProfile:properties.workloadProfileName,containers:properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}}' `
      --output jsonc `
      --debug

    Assert-AzSuccess "Reading the stored job configuration"

    Write-Host ""
    Write-Host "Starting the execution with --image and --registry-identity system"
    Write-Host ""

    $ExecutionOutput = az containerapp job start `
      --resource-group $ResourceGroup `
      --name $JobName `
      --image $PrivateImage `
      --container-name $ContainerName `
      --cpu 0.25 `
      --memory 0.5Gi `
      --registry-identity system `
      --query name `
      --output tsv `
      --debug

    Assert-AzSuccess "Starting the job execution"

    $ExecutionName = ($ExecutionOutput | Out-String).Trim()

    Write-Host ""
    Write-Host "Execution name: $ExecutionName"
    Write-Host ""
    Write-Host "Resulting execution template"
    Write-Host ""

    az containerapp job execution show `
      --resource-group $ResourceGroup `
      --name $JobName `
      --job-execution-name $ExecutionName `
      --query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
      --output jsonc `
      --debug

    Assert-AzSuccess "Reading the execution template"
}
finally {
    if ($ExecutionName) {
        Write-Host ""
        Write-Host "Stopping the temporary execution"

        az containerapp job stop `
          --resource-group $ResourceGroup `
          --name $JobName `
          --job-execution-name $ExecutionName `
          --only-show-errors `
          --output none
    }

    if ($JobCreated) {
        Write-Host "Deleting the temporary job"

        az containerapp job delete `
          --resource-group $ResourceGroup `
          --name $JobName `
          --yes `
          --only-show-errors `
          --output none
    }

    if ($ImageImported) {
        Write-Host "Deleting the temporary ACR image"

        az acr repository delete `
          --name $AcrName `
          --image "${Repository}:${Tag}" `
          --yes `
          --only-show-errors `
          --output none
    }
}

The complete reproduction script above was run with Azure CLI 2.87.0 and the containerapp extension 1.3.0b4.

The existing job was successfully created with the requested private image:

{
  "containers": [
    {
      "cpu": 0.25,
      "image": "<ACR_NAME>.azurecr.io/cli-job-start-repro:<TAG>",
      "memory": "0.5Gi",
      "name": "repro"
    }
  ],
  "identityType": "SystemAssigned",
  "registries": [
    {
      "identity": "system",
      "server": "<ACR_NAME>.azurecr.io"
    }
  ],
  "workloadProfile": "Consumption"
}

The following command was then run:

$ExecutionOutput = az containerapp job start `
  --resource-group "<RESOURCE_GROUP>" `
  --name "<JOB_NAME>" `
  --image "<ACR_NAME>.azurecr.io/cli-job-start-repro:<TAG>" `
  --container-name "repro" `
  --cpu 0.25 `
  --memory 0.5Gi `
  --registry-identity system `
  --query name `
  --output tsv `
  --debug

Relevant sanitized debug output:

CommandName: containerapp job start
ParameterSetName: --resource-group --name --image --container-name --cpu --memory --registry-identity --query --output --debug

Request URL:
https://management.azure.com/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.App/jobs/<JOB_NAME>/start?api-version=<API_VERSION>

Request method:
POST

Authorization:
<REDACTED>

Request body:
{
  "containers": [
    {
      "image": "mcr.microsoft.com/k8se/quickstart:latest",
      "name": "repro",
      "resources": {
        "cpu": 0.25,
        "memory": "0.5Gi"
      }
    }
  ]
}

The request body shows that Azure CLI replaced the image supplied through --image before sending the request to the Container Apps service.

The resulting execution was inspected using:

az containerapp job execution show `
  --resource-group "<RESOURCE_GROUP>" `
  --name "<JOB_NAME>" `
  --job-execution-name "<EXECUTION_NAME>" `
  --query 'properties.template.containers[].{name:name,image:image,cpu:resources.cpu,memory:resources.memory}' `
  --output jsonc `
  --debug

The execution contained:

[
  {
    "cpu": 0.25,
    "image": "mcr.microsoft.com/k8se/quickstart:latest",
    "memory": "0.5Gi",
    "name": "repro"
  }
]

The command exited successfully with exit code 0, but the wrong image was submitted and executed.

Expected behavior

The execution should use the image explicitly supplied through --image:

<ACR_NAME>.azurecr.io/<REPOSITORY>:<TAG>

The job's existing registry configuration and system-assigned identity should be used to authenticate the image pull.

At minimum, --registry-identity system must not silently replace the requested image.

Environment Summary

PS> az version
{
  "azure-cli": "2.87.0",
  "azure-cli-core": "2.87.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "containerapp": "1.3.0b4"
  }
}
PS> az extension show `
  --name containerapp `
  --query '{name:name, version:version, path:path}' `
  --output json
{
  "name": "containerapp",
  "path": "C:\\Users\\<user>\\.azure\\cliextensions\\containerapp",
  "version": "1.3.0b4"
}

Additional context

Suspected cause

The current implementation in

# If no image is provided, fetch the existing job's image
if image is not None:
container_def["image"] = image if not is_registry_msi_system(registry_identity) else HELLO_WORLD_IMAGE

contains this logic in the job-start path:

        # If no image is provided, fetch the existing job's image
        if image is not None:
            container_def["image"] = image if not is_registry_msi_system(registry_identity) else HELLO_WORLD_IMAGE

This appears to reuse the public-placeholder-image logic needed when initially creating a resource with a new system-assigned identity.

That bootstrap can make sense during job creation:

  1. Create the job with a public image.
  2. Obtain the newly created system identity's principal ID.
  3. Grant the identity permission to pull from ACR.
  4. Update the job to use the requested private image.

However, job start operates on an existing job whose identity and registry permissions already exist. It also does not perform a subsequent update that restores the requested image. The quickstart image therefore becomes the final execution image.

Suggested fix

The immediate fix appears to be to preserve the requested image unconditionally:

if image is not None:
    container_def["image"] = image

Additionally, please consider removing or deprecating --registry-identity from az containerapp job start.

Registry authentication is persistent job configuration and can already be configured through job creation or the job registry commands. If execution-level registry identity overrides are not supported by the service API, the CLI should reject this option with a clear error rather than silently changing the image.

For backward compatibility, a possible migration would be:

  1. Stop replacing the image with HELLO_WORLD_IMAGE.
  2. Emit a warning or validation error when --registry-identity is passed to job start.
  3. Deprecate and later remove the argument from this command.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Auto-AssignAuto assign by botContainerAppService AttentionThis issue is responsible by Azure service team.act-observability-squadbugThis issue requires a change to an existing behavior in the product in order to be resolved.customer-reportedIssues that are reported by GitHub users external to the Azure organization.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions