sagemaker-core 2.15.0: role validation raises false-positive `RoleValidationError` under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)

[akwoprosper]

PySDK Version

• PySDK V2 (2.x)
• PySDK V3 (3.x)

Reported against the sagemaker-core distribution, version 2.15.0 (repo tag v3.15.0).

Describe the bug

sagemaker-core 2.15.0 added a client-side permission pre-check that runs during high-level construction (e.g. ModelTrainer(...) → TrainDefaults.get_role) before any training job is submitted: resolve_and_validate_role → _evaluate_permissions → iam:SimulatePrincipalPolicy. It raises RoleValidationError on any non-allowed simulate verdict — and that verdict includes the AWS Organizations / SCP layer (OrganizationsDecisionDetail.AllowedByOrganizations).

Per AWS docs, the IAM policy simulator does not evaluate SCPs that have any conditions. So in an account whose organization uses condition-based SCPs, SimulatePrincipalPolicy returns AllowedByOrganizations: false (with EvalDecision: implicitDeny, MatchedStatements: []) for actions that are actually permitted at run time. The pre-check treats this as a definitive denial and raises — a false positive — even though the execution role is correctly configured and the real API call would succeed.

Two observable consequences:

1. Creating a brand-new, fully-permissioned role does not help — the simulate is denied at the org layer regardless of the role's own policies.
2. The same role works fine from a notebook / via a direct create_training_job call, because those paths don't run this client-side pre-check.

2.15.0 is currently the latest published release, so there is no fixed version to upgrade to.

To reproduce

Prerequisites: an AWS account under an organization with at least one condition-based SCP; a training execution role that trusts sagemaker.amazonaws.com and grants the training smoke-test actions at Resource: *; a calling identity that can call iam:SimulatePrincipalPolicy.

pip install 'sagemaker-core==2.15.0'

from sagemaker.core.helper.iam_role_resolver import IamRoleResolver

# Also reproducible via ModelTrainer(...) construction with role_arn set to the same role.
IamRoleResolver().resolve_and_validate_role(
    role_arn="arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>",
    role_type="training",
)

Result:

RoleValidationError: IAM role 'arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>' cannot be used for 'training' workloads.
Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions, ec2:DescribeNetworkInterfaces,
ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs,
ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetAuthorizationToken,
ecr:GetDownloadUrlForLayer

Confirm the verdict is an org-layer artifact rather than a real permission gap:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role> \
  --action-names cloudwatch:PutMetricData ec2:CreateNetworkInterface sagemaker:CreateTrainingJob

{
  "EvalActionName": "cloudwatch:PutMetricData",
  "EvalDecision": "implicitDeny",
  "MatchedStatements": [],
  "OrganizationsDecisionDetail": { "AllowedByOrganizations": "false" }
  // ...same for the other actions, including sagemaker:CreateTrainingJob —
  // yet CreateTrainingJob calls from this role succeed at run time (visible in CloudTrail).
}

The identical role runs the same workload successfully from a notebook / via direct API, so the real run-time evaluation permits these actions.

Expected behavior

The pre-check should not hard-fail on an Organizations/SCP-layer denial that the IAM policy simulator cannot faithfully evaluate. Because the simulator ignores condition-based SCPs, an AllowedByOrganizations: false result (with no matched explicit identity Deny) is unverifiable, not authoritative — it should be treated the same as the existing "caller can't call simulate → warn and proceed" path, letting the real API call be the source of truth. An explicit opt-out (e.g. validate_role=False or an env var) would also let users bypass the client-side check without modifying IAM.

Screenshots or logs

.../site-packages/sagemaker/core/helper/iam_role_resolver.py:573 in resolve_and_validate_role
570   # Permission check (definitive denial blocks; unverifiable warns)
571   verdict, denied = _evaluate_permissions(iam_client, role_arn, rol...
572   if verdict is False:
> 573       raise RoleValidationError(
574           _build_validation_error_message(role_arn, role_type, miss...

System information

• SageMaker Python SDK version: sagemaker-core 2.15.0 (repo tag v3.15.0)
• Framework name or algorithm: N/A — fails during role validation, before framework/job selection (framework-agnostic)
• Framework version: N/A
• Python version: 3.12
• CPU or GPU: N/A (fails before job submission)
• Custom Docker image (Y/N): N/A

Additional context

Introduced in sagemaker-core 2.15.0 — file sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, added in commit dba1127a ("New release (#5969)"), first tag v3.15.0. Authoring PRs: #2041 (added SimulatePrincipalPolicy-based resolve_or_create_role) → #2080 (replaced it with the raising resolve_and_validate_role). #2080 notes it gates only on *-resource "smoke test" actions to avoid false denials on resource-scoped actions, but does not account for the Organizations/SCP layer the simulate call implicitly evaluates — which is the source of this false positive.

Suggested fix: in _evaluate_permissions, when an action is implicitDeny with no matched identity/SCP statement and OrganizationsDecisionDetail.AllowedByOrganizations == false, treat it as unverifiable (warn + proceed) rather than a missing permission; and/or add an explicit opt-out.

Relevant docs:

• IAM policy simulator can't test SCPs with conditions
• IAM policy evaluation logic (explicit Deny overrides Allow)

Workarounds (both verified):

1. Attach an explicit Deny on iam:SimulatePrincipalPolicy to the identity running the SDK — it then skips the pre-check, warns, and proceeds (an explicit Deny is needed to override any existing Allow).
2. Pin sagemaker-core<2.15.0, which predates the pre-check.