EC2CreateInstanceOperator could leave EC2 instances running when failing #60904
+116
−24
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ures occurred after successful instance creation (e.g. waiter failures due to missing DescribeInstances permissions). This change adds best-effort cleanup when post-creation steps fail by attempting to terminate created instances. Cleanup errors are logged but do not mask the original exception. Tests cover successful cleanup on failure and ensure cleanup failures do not override the original error.
boring-cyborg
bot
added
area:providers
provider:amazon
SameerMesiah97
changed the title
This was referenced Jan 24, 2026
suii2210
pushed a commit
to suii2210/airflow
that referenced
this pull request
Jan 26, 2026
…ures (apache#60904) occurred after successful instance creation (e.g. waiter failures due to missing DescribeInstances permissions). This change adds best-effort cleanup when post-creation steps fail by attempting to terminate created instances. Cleanup errors are logged but do not mask the original exception. Tests cover successful cleanup on failure and ensure cleanup failures do not override the original error. Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>
2 tasks
85 tasks
shreyas-dev
pushed a commit
to shreyas-dev/airflow
that referenced
this pull request
Jan 29, 2026
…ures (apache#60904) occurred after successful instance creation (e.g. waiter failures due to missing DescribeInstances permissions). This change adds best-effort cleanup when post-creation steps fail by attempting to terminate created instances. Cleanup errors are logged but do not mask the original exception. Tests cover successful cleanup on failure and ensure cleanup failures do not override the original error. Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Added best-effort cleanup to
EC2CreateInstanceOperatorto ensure EC2 instances are terminated when failures occur after successful instance creation.Previously, the operator could create an EC2 instance via
RunInstancesand then fail during post-creation steps (for example, when waiting for the instance withwait_for_completion=TrueandDescribeInstancespermissions are missing). In these cases, the task failed while leaving the EC2 instance running.The operator now attempts to terminate any created instances if an exception is raised after instance creation. Cleanup is performed opportunistically and does not mask or replace the original exception if termination fails.
Rationale
EC2CreateInstanceOperatormanages the lifecycle of an external infrastructure resource. If instance creation succeeds but subsequent steps fail, leaving the instance running is both surprising and potentially costly.Failures after instance creation can occur for multiple reasons, including IAM permission errors (for example, missing
DescribeInstances) as well as loss of access to observability or metadata systems used during task execution. In all of these cases, the operator has failed to complete successfully from Airflow’s perspective, and execution state may no longer be reliable.Attempting best-effort cleanup in these scenarios avoids leaving unmanaged EC2 instances running when the task itself has failed, and aligns with the behavior of other Airflow operators that manage external resources. Cleanup failures are logged and do not mask the original exception, preserving existing failure semantics while improving safety.
Tests
Backwards Compatibility
No changes to the public API or operator parameters.
Closes: #60903