diff --git a/modules/orchestration-providers/webhook/README.md b/modules/orchestration-providers/webhook/README.md new file mode 100644 index 0000000000..53d0115ebb --- /dev/null +++ b/modules/orchestration-providers/webhook/README.md @@ -0,0 +1,61 @@ +# Webhook orchestration provider + +This internal module owns the event-driven runner demand controls used by `runner-config`: scale-up, scale-down, scheduled pool reconciliation, and optional queued-job retry. It receives the common GitHub, Lambda, runner-registration, SSM, observability, and selected compute-provider contracts from the parent configuration module, then resolves webhook-specific defaults and tag precedence before invoking its leaf modules. Lifecycle, boot time, and capacity are provider-owned under `config.runner`; the provider resolves the lifecycle contract for runner bootstrap, forwards capacity to scale-up and pool, and forwards boot time to scale-down and pool. It also combines the shared Lambda artifact bucket with its own `config.lambda.artifact` zip or S3 key/version shared by scale, pool, and job-retry; provider-specific artifact fields do not leak into the common Lambda contract. + +`runner-config` selects this provider when `orchestration_provider.webhook` is the one populated orchestration block. The parent continues to own common runner resources, shared SSM configuration, and compute-provider selection. A future orchestration provider should be implemented as a sibling module with the same parent-facing resource boundary; it should not add its stateful resources to this webhook module. + +The scale-down lifecycle is documented in the [scale-down state diagram](./scale-down-state-diagram.md). + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.4.0 | +| [aws](#requirement\_aws) | >= 6.33 | + +## Providers + +| Name | Version | +|------|---------| +| [terraform](#provider\_terraform) | n/a | + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [job\_retry](#module\_job\_retry) | ./job-retry | n/a | +| [pool](#module\_pool) | ./pool | n/a | +| [scale\_runners](#module\_scale\_runners) | ./scale-runners | n/a | + +## Resources + +| Name | Type | +|------|------| +| [terraform_data.validate_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [aws\_partition](#input\_aws\_partition) | AWS partition used to construct ARNs. | `string` | `"aws"` | no | +| [config](#input\_config) | Provider-owned webhook values supplied from `orchestration_provider.webhook`. The parent resolves inherited input values before calling this module; this provider still resolves the documented JIT, artifact, and tag-precedence fallbacks.

- `runner`: Runner lifecycle, boot timeout, and capacity settings owned by webhook orchestration.
- `runner.boot_time_in_minutes`: Expected runner boot duration used by scale-down and pool controls.
- `runner.ephemeral`: Registers runners in ephemeral mode.
- `runner.jit_config_enabled`: Explicitly enables or disables just-in-time configuration. Null follows `runner.ephemeral`.
- `runner.maximum_count`: Maximum number of runners managed for this runner configuration.
- `github.organization_runners`: Registers runners at organization scope when true; otherwise registration is repository-scoped.
- `queue.build.arn`: ARN of the runner configuration's build queue.
- `queue.build.url`: URL of the runner configuration's build queue.
- `queue.kms_key_id`: Optional KMS key ARN encrypting the build queue. This is independent from the Parameter Store KMS key.
- `queue.tags`: Tags inherited by queue-related provider resources before component-specific overrides.
- `lambda.artifact`: Runner-control artifact shared by scale, pool, and job-retry components. At most one of `zip` or `s3` may be selected; no selection uses the packaged runner archive.
- `lambda.artifact.zip`: Optional local path to the runner-control Lambda archive.
- `lambda.artifact.s3`: Optional S3 object selector in the common `lambda.artifact.s3.bucket`. Wrapper presence must be known during planning and selecting it requires a non-null common bucket.
- `lambda.artifact.s3.key`: Object key of the runner-control Lambda archive.
- `lambda.artifact.s3.object_version`: Optional object version of the runner-control Lambda archive.
- `lambda.scale.up.memory_size`: Memory allocated to the scale-up Lambda in MB.
- `lambda.scale.up.timeout`: Scale-up Lambda timeout in seconds.
- `lambda.scale.up.reserved_concurrent_executions`: Reserved concurrency for scale-up. Use `-1` for unreserved concurrency.
- `lambda.scale.up.job_queued_check_enabled`: Enables queued-job verification before scaling. Null follows the resolved runner mode.
- `lambda.scale.up.event_source_mapping.batch_size`: Maximum build-queue records delivered per scale-up invocation.
- `lambda.scale.up.event_source_mapping.maximum_batching_window_in_seconds`: Maximum batching window for build-queue records.
- `lambda.scale.up.tags`: Tags applied within scale-up resource scopes after common provider tags.
- `lambda.scale.down.memory_size`: Memory allocated to the scale-down Lambda in MB.
- `lambda.scale.down.timeout`: Scale-down Lambda timeout in seconds.
- `lambda.scale.down.schedule_expression`: EventBridge schedule expression that invokes scale-down.
- `lambda.scale.down.minimum_running_time_in_minutes`: Optional minimum runner age before scale-down may terminate it. Null selects the operating-system default.
- `lambda.scale.down.idle_config`: Time-based desired idle-runner configurations.
- `lambda.scale.down.idle_config[].cron`: Cron expression identifying when the idle configuration applies.
- `lambda.scale.down.idle_config[].timeZone`: IANA time zone used to evaluate the cron expression.
- `lambda.scale.down.idle_config[].idleCount`: Number of idle runners retained during the matching period.
- `lambda.scale.down.idle_config[].evictionStrategy`: Selection strategy used when excess idle runners are removed.
- `lambda.scale.down.tags`: Tags applied within scale-down resource scopes after common provider tags.
- `lambda.pool.memory_size`: Memory allocated to the pool Lambda in MB.
- `lambda.pool.timeout`: Pool Lambda timeout in seconds.
- `lambda.pool.reserved_concurrent_executions`: Reserved concurrency for the pool Lambda. Use `-1` for unreserved concurrency.
- `lambda.pool.config`: Scheduled target pool sizes. An empty list disables the pool component.
- `lambda.pool.config[].schedule_expression`: Scheduler expression that activates the target size.
- `lambda.pool.config[].schedule_expression_timezone`: Optional IANA time zone used to evaluate the schedule.
- `lambda.pool.config[].size`: Desired number of runners for the schedule.
- `lambda.pool.include_busy_runners`: Includes busy runners when reconciling scheduled pool capacity.
- `lambda.pool.runner_owner`: Optional GitHub organization or repository owner used for pooled runners.
- `lambda.pool.tags`: Tags applied within pool resource scopes after common provider tags.
- `job_retry.enabled`: Creates the retry queue, Lambda function, event-source mapping, and related IAM resources.
- `job_retry.delay_in_seconds`: Initial delay before a queued-job retry check.
- `job_retry.delay_backoff`: Multiplier applied to the delay after each unsuccessful check.
- `job_retry.max_attempts`: Maximum retry-check attempts before the message is no longer republished.
- `job_retry.tags`: Tags applied within job-retry resource scopes after common provider tags.
- `job_retry.lambda.memory_size`: Memory allocated to the job-retry Lambda in MB.
- `job_retry.lambda.reserved_concurrent_executions`: Reserved concurrency for job retry. Use `-1` for unreserved concurrency.
- `job_retry.lambda.timeout`: Job-retry Lambda timeout in seconds and visibility timeout for its retry queue. | 
object({
    runner = object({
      boot_time_in_minutes = number
      ephemeral            = bool
      jit_config_enabled   = optional(bool, null)
      maximum_count        = number
    })
    github = object({
      organization_runners = bool
    })
    queue = object({
      build = object({
        arn = string
        url = string
      })
      kms_key_id = optional(string, null)
      tags       = optional(map(string), {})
    })
    lambda = object({
      artifact = object({
        zip = optional(string, null)
        s3 = optional(object({
          key            = string
          object_version = optional(string, null)
        }), null)
      })
      scale = object({
        up = object({
          memory_size                    = number
          timeout                        = number
          reserved_concurrent_executions = number
          job_queued_check_enabled       = optional(bool, null)
          event_source_mapping = object({
            batch_size                         = number
            maximum_batching_window_in_seconds = number
          })
          tags = optional(map(string), {})
        })
        down = object({
          memory_size                     = number
          timeout                         = number
          schedule_expression             = string
          minimum_running_time_in_minutes = optional(number, null)
          idle_config = list(object({
            cron             = string
            timeZone         = string
            idleCount        = number
            evictionStrategy = string
          }))
          tags = optional(map(string), {})
        })
      })
      pool = object({
        memory_size                    = number
        timeout                        = number
        reserved_concurrent_executions = number
        config = list(object({
          schedule_expression          = string
          schedule_expression_timezone = optional(string)
          size                         = number
        }))
        include_busy_runners = bool
        runner_owner         = optional(string, null)
        tags                 = optional(map(string), {})
      })
    })
    job_retry = object({
      enabled          = bool
      delay_in_seconds = number
      delay_backoff    = number
      max_attempts     = number
      tags             = optional(map(string), {})
      lambda = object({
        memory_size                    = number
        reserved_concurrent_executions = number
        timeout                        = number
      })
    })
  })
| n/a | yes | +| [github](#input\_github) | Common GitHub API client and GitHub App Parameter Store references. | 
object({
    app_parameters = object({
      key_base64      = list(map(string))
      id              = list(map(string))
      installation_id = list(object({ name = string, arn = string }))
    })
    enterprise_server = object({
      url        = optional(string, null)
      ssl_verify = bool
    })
    user_agent = optional(string, null)
  })
| n/a | yes | +| [lambda](#input\_lambda) | Common Lambda substrate. Only the shared artifact bucket crosses this boundary; the webhook provider owns its archive key, version, and local zip selection. | 
object({
    artifact = object({
      s3 = object({
        bucket = optional(string, null)
      })
    })
    runtime            = string
    architecture       = string
    subnet_ids         = list(string)
    security_group_ids = list(string)
    tags               = optional(map(string), {})
    role = object({
      path                 = string
      permissions_boundary = optional(string, null)
      principals = optional(list(object({
        type        = string
        identifiers = list(string)
      })), [])
    })
  })
| n/a | yes | +| [observability](#input\_observability) | Common logging, tracing, and metrics configuration consumed by webhook controls. | 
object({
    logs = object({
      level             = string
      retention_in_days = number
      kms_key_id        = optional(string, null)
      class             = string
      tags              = optional(map(string), {})
    })
    tracing = object({
      mode                  = optional(string, null)
      capture_http_requests = bool
      capture_error         = bool
    })
    metrics = object({
      enabled   = bool
      namespace = string
      metric = object({
        github_app_rate_limit = object({
          enabled = bool
        })
        job_retry = object({
          enabled = bool
        })
      })
    })
  })
| n/a | yes | +| [prefix](#input\_prefix) | Prefix used to identify resources created for this webhook orchestration provider. | `string` | n/a | yes | +| [runner](#input\_runner) | Common runner registration values consumed by webhook demand controls. Lifecycle, boot timeout, and capacity remain provider-owned under config.runner. | 
object({
    os                   = string
    auto_update_disabled = bool
    labels               = list(string)
    group_name           = string
    name_prefix          = string
  })
| n/a | yes | +| [runner\_provider](#input\_runner\_provider) | Selected compute-provider capabilities consumed by webhook scale-up, scale-down, and pool controls. | 
object({
    type = string
    scale_up = object({
      environment_variables      = map(string)
      iam_policy_json            = string
      additional_iam_policy_json = optional(string, null)
      managed_policy = optional(object({
        arn = string
      }), null)
    })
    scale_down = object({
      environment_variables = map(string)
      iam_policy_json       = string
    })
    pool = object({
      environment_variables  = map(string)
      iam_policy_json        = string
      managed_policy_enabled = bool
      managed_policy_arn     = optional(string, null)
    })
  })
| n/a | yes | +| [ssm](#input\_ssm) | Resolved Parameter Store paths, optional decrypt key, and runtime parameter tags. | 
object({
    token_path           = string
    token_path_arn       = string
    config_path          = string
    config_path_arn      = string
    kms_key_id           = optional(string, null)
    parameter_store_tags = string
  })
| n/a | yes | +| [tags](#input\_tags) | Base tags available to webhook-provider resources. Component-specific tags override this map within their documented scopes. | `map(string)` | `{}` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [job\_retry](#output\_job\_retry) | Job-retry resources. Null when job retry is disabled. | +| [pool](#output\_pool) | Scheduled pool resources. Null when no pool schedule is configured. | +| [runner\_lifecycle](#output\_runner\_lifecycle) | Effective webhook-owned runner lifecycle consumed by runner-config bootstrap parameters. | +| [scale\_down](#output\_scale\_down) | Scale-down control-plane resources. | +| [scale\_up](#output\_scale\_up) | Scale-up control-plane resources. | + diff --git a/modules/orchestration-providers/webhook/job-retry.tf b/modules/orchestration-providers/webhook/job-retry.tf new file mode 100644 index 0000000000..651e12c9ea --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry.tf @@ -0,0 +1,48 @@ +module "job_retry" { + source = "./job-retry" + count = local.job_retry_enabled ? 1 : 0 + + config = { + prefix = local.resolved_config.prefix + aws_partition = var.aws_partition + lambda = { + artifact = local.resolved_config.lambda.artifact + runtime = local.resolved_config.lambda.runtime + architecture = local.resolved_config.lambda.architecture + memory_size = local.resolved_config.job_retry.lambda.memory_size + timeout = local.resolved_config.job_retry.lambda.timeout + reserved_concurrent_executions = local.resolved_config.job_retry.lambda.reserved_concurrent_executions + environment_variables = {} + vpc = { + subnet_ids = local.resolved_config.lambda.subnet_ids + security_group_ids = local.resolved_config.lambda.security_group_ids + } + role = local.resolved_config.lambda.role + } + runner = { + name_prefix = local.resolved_config.runner.name_prefix + } + github = local.resolved_config.github + queue = { + build = local.resolved_config.queue.build + kms_key_id = local.resolved_config.queue.kms_key_id + event_source_mapping = local.resolved_config.queue.event_source_mapping + encryption = { + sqs_managed_sse_enabled = true + kms_master_key_id = null + kms_data_key_reuse_period_seconds = null + } + } + ssm = { + kms_key_id = local.resolved_config.ssm.kms_key_id + } + observability = local.resolved_config.observability + tags = { + resources = local.job_retry_tags + lambda = local.job_retry_lambda_tags + log_group = local.job_retry_log_tags + queue = local.job_retry_queue_tags + event_source_mapping = local.job_retry_queue_tags + } + } +} diff --git a/modules/orchestration-providers/webhook/job-retry/README.md b/modules/orchestration-providers/webhook/job-retry/README.md new file mode 100644 index 0000000000..9c6e4e0f52 --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/README.md @@ -0,0 +1,63 @@ +# Module - Job Retry + +This module is listening to a SQS queue where the scale-up lambda publishes messages for jobs that needs to trigger a retry if still queued. The job retry module lambda function is handling the messages, checking if the job is queued. Next for queued jobs a message is published to the build queue for the scale-up lambda. The scale-up lambda will handle the message as any other workflow job event. + +## Usages + +The module is an inner module used by the webhook orchestration provider when the opt-in feature for job retry is enabled. The module is not intended to be used standalone. + + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.4.0 | +| [aws](#requirement\_aws) | >= 6.21 | + +## Providers + +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 6.21 | +| [terraform](#provider\_terraform) | n/a | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [aws_cloudwatch_log_group.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | +| [aws_iam_role.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role_policy.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.job_retry_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.job_retry_xray](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy_attachment.job_retry_vpc_execution_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | +| [aws_lambda_event_source_mapping.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_event_source_mapping) | resource | +| [aws_lambda_function.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function) | resource | +| [aws_lambda_permission.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) | resource | +| [aws_sqs_queue.job_retry_check_queue](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | +| [aws_sqs_queue_policy.job_retry_check_queue_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_policy) | resource | +| [terraform_data.validate_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource | +| [aws_iam_policy_document.deny_insecure_transport](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.job_retry_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.lambda_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.lambda_xray](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [config](#input\_config) | Provider-neutral job-retry configuration assembled by runner-config.

- `prefix`: Prefix used to name job-retry resources.
- `aws_partition`: AWS partition used to construct the Lambda VPC managed-policy ARN.
- `lambda.artifact.zip`: Resolved local control-plane archive.
- `lambda.artifact.s3.bucket`: Optional S3 bucket containing the Lambda archive.
- `lambda.artifact.s3.key`: Object key of the Lambda archive.
- `lambda.artifact.s3.object_version`: Optional object version of the Lambda archive.
- `lambda.runtime`: Runtime used by the job-retry Lambda.
- `lambda.architecture`: Instruction-set architecture used by the job-retry Lambda.
- `lambda.memory_size`: Memory allocated to the job-retry Lambda.
- `lambda.timeout`: Lambda timeout and retry-queue visibility timeout in seconds.
- `lambda.reserved_concurrent_executions`: Reserved concurrency for the Lambda. Use `-1` for unreserved concurrency.
- `lambda.environment_variables`: Additional Lambda environment variables. Required job-retry variables override matching keys.
- `lambda.vpc.subnet_ids`: Subnets used for Lambda VPC configuration.
- `lambda.vpc.security_group_ids`: Security groups used for Lambda VPC configuration.
- `lambda.role.path`: IAM path used for the job-retry Lambda role.
- `lambda.role.permissions_boundary`: Optional permissions boundary for the Lambda role.
- `lambda.role.principals`: Extra principals allowed to assume the Lambda role, for example during local testing.
- `runner.name_prefix`: Prefix used to identify runners belonging to this runner configuration.
- `github.organization_runners`: Enables organization runners.
- `github.enterprise_server.url`: Optional GitHub Enterprise Server URL.
- `github.enterprise_server.ssl_verify`: Enables TLS certificate verification for GitHub Enterprise Server requests.
- `github.user_agent`: Optional User-Agent sent to GitHub.
- `github.app_parameters.key_base64`: Ordered Parameter Store references for GitHub App private keys.
- `github.app_parameters.id`: Ordered Parameter Store references for GitHub App IDs.
- `github.app_parameters.installation_id`: Ordered optional Parameter Store references for GitHub App installation IDs.
- `queue.build`: URL and ARN of the build queue to which retry messages are published.
- `queue.kms_key_id`: Optional KMS key ARN used to encrypt the build queue. This is distinct from the Parameter Store key.
- `queue.event_source_mapping.batch_size`: Maximum records delivered per job-retry invocation.
- `queue.event_source_mapping.maximum_batching_window_in_seconds`: Maximum event batching window.
- `queue.encryption`: Server-side encryption configuration for the retry queue.
- `ssm.kms_key_id`: Optional KMS key ARN used by the job-retry IAM policy. Its value may be unknown until apply.
- `observability.logs`: Logging level, retention, encryption, and log-class configuration.
- `observability.tracing`: Lambda X-Ray and tracing-helper configuration.
- `observability.metrics`: Metrics enablement, namespace, and job-retry metric configuration.
- `tags.resources`: Tags for the job-retry Lambda role and component resources.
- `tags.lambda`: Tags for the job-retry Lambda function.
- `tags.log_group`: Tags for the job-retry log group.
- `tags.queue`: Tags for the retry queue.
- `tags.event_source_mapping`: Tags for the retry-queue event-source mapping. | 
object({
    prefix        = string
    aws_partition = string
    lambda = object({
      artifact = object({
        zip = string
        s3 = object({
          bucket         = optional(string, null)
          key            = optional(string, null)
          object_version = optional(string, null)
        })
      })
      runtime                        = string
      architecture                   = string
      memory_size                    = number
      timeout                        = number
      reserved_concurrent_executions = number
      environment_variables          = map(string)
      vpc = object({
        subnet_ids         = list(string)
        security_group_ids = list(string)
      })
      role = object({
        path                 = string
        permissions_boundary = optional(string, null)
        principals = list(object({
          type        = string
          identifiers = list(string)
        }))
      })
    })
    runner = object({
      name_prefix = string
    })
    github = object({
      organization_runners = bool
      enterprise_server = object({
        url        = optional(string, null)
        ssl_verify = optional(bool, true)
      })
      user_agent = optional(string, null)
      app_parameters = object({
        key_base64      = list(map(string))
        id              = list(map(string))
        installation_id = list(object({ name = string, arn = string }))
      })
    })
    queue = object({
      build = object({
        url = string
        arn = string
      })
      kms_key_id = optional(string, null)
      event_source_mapping = object({
        batch_size                         = number
        maximum_batching_window_in_seconds = number
      })
      encryption = object({
        sqs_managed_sse_enabled           = bool
        kms_master_key_id                 = optional(string, null)
        kms_data_key_reuse_period_seconds = optional(number, null)
      })
    })
    ssm = object({
      kms_key_id = optional(string, null)
    })
    observability = object({
      logs = object({
        level             = string
        retention_in_days = number
        kms_key_id        = optional(string, null)
        class             = string
      })
      tracing = object({
        mode                  = optional(string, null)
        capture_http_requests = bool
        capture_error         = bool
      })
      metrics = object({
        enabled   = bool
        namespace = string
        metric = object({
          github_app_rate_limit = object({
            enabled = bool
          })
          job_retry = object({
            enabled = bool
          })
        })
      })
    })
    tags = object({
      resources            = map(string)
      lambda               = map(string)
      log_group            = map(string)
      queue                = map(string)
      event_source_mapping = map(string)
    })
  })
| n/a | yes | + +## Outputs + +| Name | Description | +|------|-------------| +| [job\_retry\_check\_queue](#output\_job\_retry\_check\_queue) | Queue consumed by the job-retry Lambda. | +| [lambda](#output\_lambda) | Job-retry Lambda resources. | + diff --git a/modules/orchestration-providers/webhook/job-retry/iam-policies.tf b/modules/orchestration-providers/webhook/job-retry/iam-policies.tf new file mode 100644 index 0000000000..0e79e8a265 --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/iam-policies.tf @@ -0,0 +1,122 @@ +# IAM policies attached to the job-retry Lambda role. +data "aws_iam_policy_document" "lambda_assume_role" { + statement { + sid = "WebhookJobRetryAssumeRole" + actions = ["sts:AssumeRole"] + + principals { + type = "Service" + identifiers = ["lambda.amazonaws.com"] + } + + dynamic "principals" { + for_each = var.config.lambda.role.principals + + content { + type = principals.value.type + identifiers = principals.value.identifiers + } + } + } +} + +data "aws_iam_policy_document" "job_retry_logging" { + statement { + sid = "WebhookJobRetryWriteLogs" + effect = "Allow" + + actions = [ + "logs:CreateLogStream", + "logs:PutLogEvents", + ] + + resources = ["${aws_cloudwatch_log_group.job_retry.arn}*"] + } +} + +data "aws_iam_policy_document" "lambda_xray" { + count = var.config.observability.tracing.mode != null ? 1 : 0 + + # AWS X-Ray write/read trace APIs do not support resource-level permissions. + statement { + sid = "AllowXRay" + effect = "Allow" + actions = [ + "xray:BatchGetTraces", + "xray:GetTraceSummaries", + "xray:PutTelemetryRecords", + "xray:PutTraceSegments", + ] + resources = ["*"] + } +} + +data "aws_iam_policy_document" "job_retry" { + statement { + sid = "WebhookJobRetryReadGitHubAppParameters" + effect = "Allow" + + actions = [ + "ssm:GetParameter", + "ssm:GetParameters", + ] + + resources = concat( + [for p in var.config.github.app_parameters.id : p.arn], + [for p in var.config.github.app_parameters.key_base64 : p.arn], + [for p in var.config.github.app_parameters.installation_id : p.arn if p != null], + ) + } + + statement { + sid = "WebhookJobRetryConsumeRetryQueue" + effect = "Allow" + + actions = [ + "sqs:ReceiveMessage", + "sqs:GetQueueAttributes", + "sqs:DeleteMessage", + ] + + resources = [aws_sqs_queue.job_retry_check_queue.arn] + } + + statement { + sid = "WebhookJobRetryPublishBuildQueue" + effect = "Allow" + + actions = [ + "sqs:SendMessage", + "sqs:GetQueueAttributes", + ] + + resources = [var.config.queue.build.arn] + } + + dynamic "statement" { + for_each = var.config.ssm.kms_key_id == null ? [] : [var.config.ssm.kms_key_id] + iterator = kms_key + + content { + sid = "WebhookJobRetryDecryptParameterStore" + effect = "Allow" + actions = ["kms:Decrypt"] + resources = [kms_key.value] + } + } + + dynamic "statement" { + for_each = var.config.queue.kms_key_id == null ? [] : [var.config.queue.kms_key_id] + iterator = kms_key + + content { + sid = "WebhookJobRetryEncryptBuildQueueMessage" + effect = "Allow" + actions = [ + "kms:Decrypt", + "kms:GenerateDataKey", + ] + resources = [kms_key.value] + } + } +} diff --git a/modules/orchestration-providers/webhook/job-retry/job-retry.tf b/modules/orchestration-providers/webhook/job-retry/job-retry.tf new file mode 100644 index 0000000000..a536cfe196 --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/job-retry.tf @@ -0,0 +1,179 @@ +# Provider-neutral job-retry queue and Lambda resources. +locals { + name = "job-retry" + vpc_enabled = ( + length(var.config.lambda.vpc.subnet_ids) > 0 && + length(var.config.lambda.vpc.security_group_ids) > 0 + ) + + lambda_environment_variables = { + ENVIRONMENT = var.config.prefix + LOG_LEVEL = var.config.observability.logs.level + PREFIX = var.config.prefix + POWERTOOLS_LOGGER_LOG_EVENT = var.config.observability.logs.level == "debug" ? "true" : "false" + POWERTOOLS_SERVICE_NAME = local.name + POWERTOOLS_TRACE_ENABLED = var.config.observability.tracing.mode != null + POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.config.observability.tracing.capture_http_requests + POWERTOOLS_TRACER_CAPTURE_ERROR = var.config.observability.tracing.capture_error + POWERTOOLS_METRICS_NAMESPACE = var.config.observability.metrics.namespace + } + + job_retry_environment_variables = { + ENABLE_ORGANIZATION_RUNNERS = var.config.github.organization_runners + ENABLE_METRIC_JOB_RETRY = var.config.observability.metrics.enabled && var.config.observability.metrics.metric.job_retry.enabled + ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = var.config.observability.metrics.enabled && var.config.observability.metrics.metric.github_app_rate_limit.enabled + GHES_URL = var.config.github.enterprise_server.url + NODE_TLS_REJECT_UNAUTHORIZED = var.config.github.enterprise_server.url != null && !var.config.github.enterprise_server.ssl_verify ? 0 : 1 + USER_AGENT = var.config.github.user_agent + JOB_QUEUE_SCALE_UP_URL = var.config.queue.build.url + PARAMETER_GITHUB_APP_ID_NAME = join(":", [for p in var.config.github.app_parameters.id : p.name]) + PARAMETER_GITHUB_APP_KEY_BASE64_NAME = join(":", [for p in var.config.github.app_parameters.key_base64 : p.name]) + PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = join(":", [for p in var.config.github.app_parameters.installation_id : p != null ? p.name : ""]) + RUNNER_NAME_PREFIX = var.config.runner.name_prefix + } + + environment_variables = merge( + local.lambda_environment_variables, + var.config.lambda.environment_variables, + local.job_retry_environment_variables, + ) +} + +resource "aws_sqs_queue_policy" "job_retry_check_queue_policy" { + queue_url = aws_sqs_queue.job_retry_check_queue.id + policy = data.aws_iam_policy_document.deny_insecure_transport.json +} + +resource "aws_sqs_queue" "job_retry_check_queue" { + name = "${var.config.prefix}-job-retry" + visibility_timeout_seconds = var.config.lambda.timeout + + sqs_managed_sse_enabled = var.config.queue.encryption.sqs_managed_sse_enabled + kms_master_key_id = var.config.queue.encryption.kms_master_key_id + kms_data_key_reuse_period_seconds = var.config.queue.encryption.kms_data_key_reuse_period_seconds + + tags = var.config.tags.queue +} + +resource "aws_lambda_function" "job_retry" { + s3_bucket = var.config.lambda.artifact.s3.bucket + s3_key = var.config.lambda.artifact.s3.key + s3_object_version = var.config.lambda.artifact.s3.object_version + filename = var.config.lambda.artifact.s3.bucket == null ? var.config.lambda.artifact.zip : null + source_code_hash = var.config.lambda.artifact.s3.bucket == null ? filebase64sha256(var.config.lambda.artifact.zip) : null + function_name = "${var.config.prefix}-${local.name}" + role = aws_iam_role.job_retry.arn + handler = "index.jobRetryCheck" + runtime = var.config.lambda.runtime + timeout = var.config.lambda.timeout + memory_size = var.config.lambda.memory_size + reserved_concurrent_executions = var.config.lambda.reserved_concurrent_executions + architectures = [var.config.lambda.architecture] + + environment { + variables = local.environment_variables + } + + dynamic "vpc_config" { + for_each = local.vpc_enabled ? [true] : [] + + content { + security_group_ids = var.config.lambda.vpc.security_group_ids + subnet_ids = var.config.lambda.vpc.subnet_ids + } + } + + dynamic "tracing_config" { + for_each = var.config.observability.tracing.mode != null ? [true] : [] + + content { + mode = var.config.observability.tracing.mode + } + } + + tags = var.config.tags.lambda +} + +resource "aws_cloudwatch_log_group" "job_retry" { + name = "/aws/lambda/${aws_lambda_function.job_retry.function_name}" + retention_in_days = var.config.observability.logs.retention_in_days + kms_key_id = var.config.observability.logs.kms_key_id + log_group_class = var.config.observability.logs.class + tags = var.config.tags.log_group +} + +resource "aws_iam_role" "job_retry" { + name = "${substr("${var.config.prefix}-${local.name}", 0, 54)}-${substr(md5("${var.config.prefix}-${local.name}"), 0, 8)}" + assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json + path = var.config.lambda.role.path + permissions_boundary = var.config.lambda.role.permissions_boundary + tags = var.config.tags.resources +} + +resource "aws_iam_role_policy" "job_retry_logging" { + name = "logging-policy" + role = aws_iam_role.job_retry.name + policy = data.aws_iam_policy_document.job_retry_logging.json +} + +resource "aws_iam_role_policy_attachment" "job_retry_vpc_execution_role" { + count = local.vpc_enabled ? 1 : 0 + role = aws_iam_role.job_retry.name + policy_arn = "arn:${var.config.aws_partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole" +} + +resource "aws_iam_role_policy" "job_retry_xray" { + count = var.config.observability.tracing.mode != null ? 1 : 0 + name = "xray-policy" + policy = data.aws_iam_policy_document.lambda_xray[0].json + role = aws_iam_role.job_retry.name +} + +resource "aws_lambda_event_source_mapping" "job_retry" { + event_source_arn = aws_sqs_queue.job_retry_check_queue.arn + function_name = aws_lambda_function.job_retry.arn + batch_size = var.config.queue.event_source_mapping.batch_size + maximum_batching_window_in_seconds = var.config.queue.event_source_mapping.maximum_batching_window_in_seconds + tags = var.config.tags.event_source_mapping +} + +resource "aws_lambda_permission" "job_retry" { + statement_id = "AllowExecutionFromSQS" + action = "lambda:InvokeFunction" + function_name = aws_lambda_function.job_retry.function_name + principal = "sqs.amazonaws.com" + source_arn = aws_sqs_queue.job_retry_check_queue.arn +} + +resource "aws_iam_role_policy" "job_retry" { + name = "job_retry-policy" + role = aws_iam_role.job_retry.name + policy = data.aws_iam_policy_document.job_retry.json +} + +data "aws_iam_policy_document" "deny_insecure_transport" { + statement { + sid = "DenyInsecureTransport" + + effect = "Deny" + + principals { + type = "AWS" + identifiers = ["*"] + } + + actions = [ + "sqs:*" + ] + + resources = [ + aws_sqs_queue.job_retry_check_queue.arn + ] + + condition { + test = "Bool" + variable = "aws:SecureTransport" + values = ["false"] + } + } +} diff --git a/modules/orchestration-providers/webhook/job-retry/outputs.tf b/modules/orchestration-providers/webhook/job-retry/outputs.tf new file mode 100644 index 0000000000..4f08cc4498 --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/outputs.tf @@ -0,0 +1,13 @@ +output "lambda" { + description = "Job-retry Lambda resources." + value = { + function = aws_lambda_function.job_retry + log_group = aws_cloudwatch_log_group.job_retry + role = aws_iam_role.job_retry + } +} + +output "job_retry_check_queue" { + description = "Queue consumed by the job-retry Lambda." + value = aws_sqs_queue.job_retry_check_queue +} diff --git a/modules/orchestration-providers/webhook/job-retry/tests/job-retry.tftest.hcl b/modules/orchestration-providers/webhook/job-retry/tests/job-retry.tftest.hcl new file mode 100644 index 0000000000..25d1dfaafc --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/tests/job-retry.tftest.hcl @@ -0,0 +1,394 @@ +mock_provider "aws" { + mock_data "aws_iam_policy_document" { + defaults = { + json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}" + } + } + + mock_resource "aws_iam_role" { + defaults = { + arn = "arn:aws:iam::123456789012:role/job-retry-test" + } + } + +} + +variables { + config = { + prefix = "job-retry-test" + aws_partition = "aws" + lambda = { + artifact = { + zip = "unused.zip" + s3 = { + bucket = "lambda-artifacts" + key = "job-retry.zip" + } + } + architecture = "arm64" + runtime = "nodejs24.x" + memory_size = 256 + timeout = 30 + reserved_concurrent_executions = 1 + environment_variables = { + CUSTOM_ENV = "preserved" + RUNNER_NAME_PREFIX = "caller-prefix-" + } + vpc = { + security_group_ids = ["sg-12345678"] + subnet_ids = ["subnet-12345678"] + } + role = { + path = "/job-retry-test/" + principals = [{ + type = "AWS" + identifiers = ["arn:aws:iam::123456789012:root"] + }] + } + } + runner = { + name_prefix = "required-prefix-" + } + github = { + organization_runners = false + enterprise_server = { + url = "https://experimental-job-retry.example.com" + ssl_verify = false + } + user_agent = "experimental-job-retry-user-agent" + app_parameters = { + key_base64 = [ + { + name = "/github-runner/key-base64" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64" + }, + { + name = "/github-runner/key-base64-2" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64-2" + }, + ] + id = [ + { + name = "/github-runner/app-id" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id" + }, + { + name = "/github-runner/app-id-2" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id-2" + }, + ] + installation_id = [ + null, + { + name = "/github-runner/installation-id-2" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/installation-id-2" + }, + ] + } + } + queue = { + build = { + url = "https://sqs.eu-west-1.amazonaws.com/123456789012/build-queue" + arn = "arn:aws:sqs:eu-west-1:123456789012:build-queue" + } + kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/build-queue-test" + event_source_mapping = { + batch_size = 10 + maximum_batching_window_in_seconds = 0 + } + encryption = { + sqs_managed_sse_enabled = true + } + } + ssm = { + kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/job-retry-test" + } + observability = { + logs = { + level = "trace" + class = "INFREQUENT_ACCESS" + retention_in_days = 180 + } + tracing = { + mode = "Active" + capture_http_requests = false + capture_error = false + } + metrics = { + enabled = false + namespace = "JobRetryTest" + metric = { + github_app_rate_limit = { + enabled = true + } + job_retry = { + enabled = true + } + } + } + } + tags = { + resources = { scope = "resources" } + lambda = { scope = "lambda" } + log_group = { scope = "log-group" } + queue = { scope = "queue" } + event_source_mapping = { scope = "event-source-mapping" } + } + } +} + +run "preserves_nested_job_retry_configuration" { + command = plan + + assert { + condition = output.lambda.function.environment[0].variables["CUSTOM_ENV"] == "preserved" + error_message = "Caller-provided job-retry environment variables must be preserved." + } + + assert { + condition = output.lambda.function.environment[0].variables["RUNNER_NAME_PREFIX"] == "required-prefix-" + error_message = "Required job-retry environment variables must override caller-provided values." + } + + assert { + condition = ( + output.lambda.function.environment[0].variables["GHES_URL"] == "https://experimental-job-retry.example.com" + && output.lambda.function.environment[0].variables["NODE_TLS_REJECT_UNAUTHORIZED"] == "0" + && output.lambda.function.environment[0].variables["USER_AGENT"] == "experimental-job-retry-user-agent" + && output.lambda.function.environment[0].variables["PARAMETER_GITHUB_APP_ID_NAME"] == "/github-runner/app-id:/github-runner/app-id-2" + && output.lambda.function.environment[0].variables["PARAMETER_GITHUB_APP_KEY_BASE64_NAME"] == "/github-runner/key-base64:/github-runner/key-base64-2" + && output.lambda.function.environment[0].variables["PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME"] == ":/github-runner/installation-id-2" + && contains(data.aws_iam_policy_document.job_retry.statement[0].resources, "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id-2") + && contains(data.aws_iam_policy_document.job_retry.statement[0].resources, "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64-2") + && contains(data.aws_iam_policy_document.job_retry.statement[0].resources, "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/installation-id-2") + ) + error_message = "Job retry must receive the nested GitHub connection settings, pass every app parameter, and grant access to every corresponding SSM ARN." + } + + assert { + condition = ( + toset(keys(output.lambda)) == toset(["function", "log_group", "role"]) + && output.lambda.function.s3_bucket == "lambda-artifacts" + && output.lambda.function.s3_key == "job-retry.zip" + && output.lambda.function.reserved_concurrent_executions == 1 + ) + error_message = "The nested Lambda configuration and direct resource output contract must be preserved." + } + + assert { + condition = ( + output.lambda.function.tags == tomap({ scope = "lambda" }) + && output.lambda.log_group.tags == tomap({ scope = "log-group" }) + && output.lambda.role.tags == tomap({ scope = "resources" }) + && output.job_retry_check_queue.tags == tomap({ scope = "queue" }) + && aws_lambda_event_source_mapping.job_retry.tags == tomap({ scope = "event-source-mapping" }) + ) + error_message = "Resolved nested tag maps must be applied to their owned resources." + } + + assert { + condition = ( + output.lambda.log_group.log_group_class == "INFREQUENT_ACCESS" + && length(data.aws_iam_policy_document.job_retry.statement) == 5 + && one([ + for statement in data.aws_iam_policy_document.job_retry.statement : statement + if statement.sid == "WebhookJobRetryDecryptParameterStore" + ]).resources == toset(["arn:aws:kms:eu-west-1:123456789012:key/job-retry-test"]) + && one([ + for statement in data.aws_iam_policy_document.job_retry.statement : statement + if statement.sid == "WebhookJobRetryDecryptParameterStore" + ]).actions == toset(["kms:Decrypt"]) + && one([ + for statement in data.aws_iam_policy_document.job_retry.statement : statement + if statement.sid == "WebhookJobRetryEncryptBuildQueueMessage" + ]).resources == toset(["arn:aws:kms:eu-west-1:123456789012:key/build-queue-test"]) + && one([ + for statement in data.aws_iam_policy_document.job_retry.statement : statement + if statement.sid == "WebhookJobRetryEncryptBuildQueueMessage" + ]).actions == toset(["kms:Decrypt", "kms:GenerateDataKey"]) + && length(aws_lambda_function.job_retry.vpc_config) == 1 + && length(aws_iam_role_policy_attachment.job_retry_vpc_execution_role) == 1 + && length(aws_iam_role_policy.job_retry_xray) == 1 + && length(data.aws_iam_policy_document.lambda_assume_role.statement[0].principals) == 2 + ) + error_message = "Logging, distinct Parameter Store/build-queue KMS grants, complete VPC, tracing, and extra role-principal configuration must be preserved." + } + + assert { + condition = ( + data.aws_iam_policy_document.lambda_xray[0].statement[0].sid == "AllowXRay" + && data.aws_iam_policy_document.lambda_xray[0].statement[0].resources == toset(["*"]) + && toset(data.aws_iam_policy_document.lambda_xray[0].statement[0].actions) == toset([ + "xray:BatchGetTraces", + "xray:GetTraceSummaries", + "xray:PutTelemetryRecords", + "xray:PutTraceSegments", + ]) + ) + error_message = "Only the resource-agnostic X-Ray APIs may retain a wildcard resource in the job-retry policies." + } + +} + +run "does_not_enable_partial_vpc_configuration" { + command = plan + + variables { + config = { + prefix = "job-retry-test" + aws_partition = "aws" + lambda = { + artifact = { + zip = "unused.zip" + s3 = { + bucket = "lambda-artifacts" + key = "job-retry.zip" + } + } + architecture = "arm64" + runtime = "nodejs24.x" + memory_size = 256 + timeout = 30 + reserved_concurrent_executions = 1 + environment_variables = {} + vpc = { + security_group_ids = [] + subnet_ids = ["subnet-12345678"] + } + role = { + path = "/job-retry-test/" + principals = [] + } + } + runner = { + name_prefix = "" + } + github = { + organization_runners = false + enterprise_server = {} + app_parameters = { + key_base64 = [{ + name = "/github-runner/key-base64" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64" + }] + id = [{ + name = "/github-runner/app-id" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id" + }] + installation_id = [null] + } + } + queue = { + build = { + url = "https://sqs.eu-west-1.amazonaws.com/123456789012/build-queue" + arn = "arn:aws:sqs:eu-west-1:123456789012:build-queue" + } + event_source_mapping = { + batch_size = 10 + maximum_batching_window_in_seconds = 0 + } + encryption = { + sqs_managed_sse_enabled = true + } + } + ssm = {} + observability = { + logs = { + level = "info" + class = "STANDARD" + retention_in_days = 180 + } + tracing = { + capture_http_requests = false + capture_error = false + } + metrics = { + enabled = false + namespace = "GitHub Runners" + metric = { + github_app_rate_limit = { + enabled = true + } + job_retry = { + enabled = true + } + } + } + } + tags = { + resources = {} + lambda = {} + log_group = {} + queue = {} + event_source_mapping = {} + } + } + } + + assert { + condition = ( + length(aws_lambda_function.job_retry.vpc_config) == 0 + && length(aws_iam_role_policy_attachment.job_retry_vpc_execution_role) == 0 + && length(data.aws_iam_policy_document.job_retry.statement) == 3 + && length([ + for statement in data.aws_iam_policy_document.job_retry.statement : statement + if contains(statement.actions, "kms:Decrypt") + ]) == 0 + ) + error_message = "Partial VPC inputs must stay disabled and a null KMS key must omit the KMS statement entirely." + } +} + +run "rejects_unsupported_lambda_architecture" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + config = merge(var.config, { + lambda = merge(var.config.lambda, { + architecture = "unsupported" + }) + }) + } + + expect_failures = [terraform_data.validate_config] +} + +run "rejects_unsupported_log_level" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + config = merge(var.config, { + observability = merge(var.config.observability, { + logs = merge(var.config.observability.logs, { + level = "verbose" + }) + }) + }) + } + + expect_failures = [terraform_data.validate_config] +} + +run "rejects_resource_prefix_longer_than_aws_limit" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + config = merge(var.config, { + prefix = "1234567890123456789012345678901234567890123456789012345" + }) + } + + expect_failures = [terraform_data.validate_config] +} diff --git a/modules/orchestration-providers/webhook/job-retry/validations.tf b/modules/orchestration-providers/webhook/job-retry/validations.tf new file mode 100644 index 0000000000..832f53ddd5 --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/validations.tf @@ -0,0 +1,26 @@ +resource "terraform_data" "validate_config" { + lifecycle { + precondition { + condition = contains(["arm64", "x86_64"], var.config.lambda.architecture) + error_message = "config.lambda.architecture must be arm64 or x86_64." + } + + precondition { + condition = contains([ + "silly", + "trace", + "debug", + "info", + "warn", + "error", + "fatal", + ], var.config.observability.logs.level) + error_message = "config.observability.logs.level must be one of silly, trace, debug, info, warn, error, or fatal." + } + + precondition { + condition = length(var.config.prefix) + length("job-retry") <= 63 + error_message = "The length of config.prefix plus job-retry must be less than or equal to 63." + } + } +} diff --git a/modules/orchestration-providers/webhook/job-retry/variables.tf b/modules/orchestration-providers/webhook/job-retry/variables.tf new file mode 100644 index 0000000000..e8235265f8 --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/variables.tf @@ -0,0 +1,147 @@ +variable "config" { + description = <<-EOT + Provider-neutral job-retry configuration assembled by runner-config. + + - `prefix`: Prefix used to name job-retry resources. + - `aws_partition`: AWS partition used to construct the Lambda VPC managed-policy ARN. + - `lambda.artifact.zip`: Resolved local control-plane archive. + - `lambda.artifact.s3.bucket`: Optional S3 bucket containing the Lambda archive. + - `lambda.artifact.s3.key`: Object key of the Lambda archive. + - `lambda.artifact.s3.object_version`: Optional object version of the Lambda archive. + - `lambda.runtime`: Runtime used by the job-retry Lambda. + - `lambda.architecture`: Instruction-set architecture used by the job-retry Lambda. + - `lambda.memory_size`: Memory allocated to the job-retry Lambda. + - `lambda.timeout`: Lambda timeout and retry-queue visibility timeout in seconds. + - `lambda.reserved_concurrent_executions`: Reserved concurrency for the Lambda. Use `-1` for unreserved concurrency. + - `lambda.environment_variables`: Additional Lambda environment variables. Required job-retry variables override matching keys. + - `lambda.vpc.subnet_ids`: Subnets used for Lambda VPC configuration. + - `lambda.vpc.security_group_ids`: Security groups used for Lambda VPC configuration. + - `lambda.role.path`: IAM path used for the job-retry Lambda role. + - `lambda.role.permissions_boundary`: Optional permissions boundary for the Lambda role. + - `lambda.role.principals`: Extra principals allowed to assume the Lambda role, for example during local testing. + - `runner.name_prefix`: Prefix used to identify runners belonging to this runner configuration. + - `github.organization_runners`: Enables organization runners. + - `github.enterprise_server.url`: Optional GitHub Enterprise Server URL. + - `github.enterprise_server.ssl_verify`: Enables TLS certificate verification for GitHub Enterprise Server requests. + - `github.user_agent`: Optional User-Agent sent to GitHub. + - `github.app_parameters.key_base64`: Ordered Parameter Store references for GitHub App private keys. + - `github.app_parameters.id`: Ordered Parameter Store references for GitHub App IDs. + - `github.app_parameters.installation_id`: Ordered optional Parameter Store references for GitHub App installation IDs. + - `queue.build`: URL and ARN of the build queue to which retry messages are published. + - `queue.kms_key_id`: Optional KMS key ARN used to encrypt the build queue. This is distinct from the Parameter Store key. + - `queue.event_source_mapping.batch_size`: Maximum records delivered per job-retry invocation. + - `queue.event_source_mapping.maximum_batching_window_in_seconds`: Maximum event batching window. + - `queue.encryption`: Server-side encryption configuration for the retry queue. + - `ssm.kms_key_id`: Optional KMS key ARN used by the job-retry IAM policy. Its value may be unknown until apply. + - `observability.logs`: Logging level, retention, encryption, and log-class configuration. + - `observability.tracing`: Lambda X-Ray and tracing-helper configuration. + - `observability.metrics`: Metrics enablement, namespace, and job-retry metric configuration. + - `tags.resources`: Tags for the job-retry Lambda role and component resources. + - `tags.lambda`: Tags for the job-retry Lambda function. + - `tags.log_group`: Tags for the job-retry log group. + - `tags.queue`: Tags for the retry queue. + - `tags.event_source_mapping`: Tags for the retry-queue event-source mapping. + EOT + + type = object({ + prefix = string + aws_partition = string + lambda = object({ + artifact = object({ + zip = string + s3 = object({ + bucket = optional(string, null) + key = optional(string, null) + object_version = optional(string, null) + }) + }) + runtime = string + architecture = string + memory_size = number + timeout = number + reserved_concurrent_executions = number + environment_variables = map(string) + vpc = object({ + subnet_ids = list(string) + security_group_ids = list(string) + }) + role = object({ + path = string + permissions_boundary = optional(string, null) + principals = list(object({ + type = string + identifiers = list(string) + })) + }) + }) + runner = object({ + name_prefix = string + }) + github = object({ + organization_runners = bool + enterprise_server = object({ + url = optional(string, null) + ssl_verify = optional(bool, true) + }) + user_agent = optional(string, null) + app_parameters = object({ + key_base64 = list(map(string)) + id = list(map(string)) + installation_id = list(object({ name = string, arn = string })) + }) + }) + queue = object({ + build = object({ + url = string + arn = string + }) + kms_key_id = optional(string, null) + event_source_mapping = object({ + batch_size = number + maximum_batching_window_in_seconds = number + }) + encryption = object({ + sqs_managed_sse_enabled = bool + kms_master_key_id = optional(string, null) + kms_data_key_reuse_period_seconds = optional(number, null) + }) + }) + ssm = object({ + kms_key_id = optional(string, null) + }) + observability = object({ + logs = object({ + level = string + retention_in_days = number + kms_key_id = optional(string, null) + class = string + }) + tracing = object({ + mode = optional(string, null) + capture_http_requests = bool + capture_error = bool + }) + metrics = object({ + enabled = bool + namespace = string + metric = object({ + github_app_rate_limit = object({ + enabled = bool + }) + job_retry = object({ + enabled = bool + }) + }) + }) + }) + tags = object({ + resources = map(string) + lambda = map(string) + log_group = map(string) + queue = map(string) + event_source_mapping = map(string) + }) + }) + + nullable = false +} diff --git a/modules/orchestration-providers/webhook/job-retry/versions.tf b/modules/orchestration-providers/webhook/job-retry/versions.tf new file mode 100644 index 0000000000..fcec7c620d --- /dev/null +++ b/modules/orchestration-providers/webhook/job-retry/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.4.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 6.21" + } + } +} diff --git a/modules/orchestration-providers/webhook/main.tf b/modules/orchestration-providers/webhook/main.tf new file mode 100644 index 0000000000..d9b1722d98 --- /dev/null +++ b/modules/orchestration-providers/webhook/main.tf @@ -0,0 +1,66 @@ +locals { + packaged_runners_lambda_zip = "${path.module}/../../../lambdas/functions/control-plane/runners.zip" + runner_control_artifact_s3_selected = var.config.lambda.artifact.s3 != null + runner_control_artifact = { + zip = local.runner_control_artifact_s3_selected ? null : coalesce( + var.config.lambda.artifact.zip, + local.packaged_runners_lambda_zip, + ) + s3 = { + bucket = local.runner_control_artifact_s3_selected ? var.lambda.artifact.s3.bucket : null + key = try(var.config.lambda.artifact.s3.key, null) + object_version = try(var.config.lambda.artifact.s3.object_version, null) + } + } + + resolved_config = { + prefix = var.prefix + tags = var.tags + runner = merge(var.runner, var.config.runner, { + jit_config_enabled = ( + var.config.runner.jit_config_enabled == null + ? var.config.runner.ephemeral + : var.config.runner.jit_config_enabled + ) + }) + github = merge(var.github, var.config.github) + lambda = merge(var.lambda, { + artifact = local.runner_control_artifact + }) + queue = merge(var.config.queue, { + event_source_mapping = var.config.lambda.scale.up.event_source_mapping + }) + scale_up = var.config.lambda.scale.up + scale_down = var.config.lambda.scale.down + pool = var.config.lambda.pool + job_retry = var.config.job_retry + ssm = var.ssm + observability = var.observability + } + + common_tags = local.resolved_config.tags + lambda_tags = merge(local.common_tags, local.resolved_config.lambda.tags) + queue_tags = merge(local.common_tags, local.resolved_config.queue.tags) + observability_log_tags = merge(local.common_tags, local.resolved_config.observability.logs.tags) + + scale_up_tags = merge(local.common_tags, local.resolved_config.scale_up.tags) + scale_up_lambda_tags = merge(local.lambda_tags, local.resolved_config.scale_up.tags) + scale_up_log_tags = merge(local.observability_log_tags, local.resolved_config.scale_up.tags) + scale_up_queue_tags = merge(local.queue_tags, local.resolved_config.scale_up.tags) + + scale_down_tags = merge(local.common_tags, local.resolved_config.scale_down.tags) + scale_down_lambda_tags = merge(local.lambda_tags, local.resolved_config.scale_down.tags) + scale_down_log_tags = merge(local.observability_log_tags, local.resolved_config.scale_down.tags) + + pool_tags = merge(local.common_tags, local.resolved_config.pool.tags) + pool_lambda_tags = merge(local.lambda_tags, local.resolved_config.pool.tags) + pool_log_tags = merge(local.observability_log_tags, local.resolved_config.pool.tags) + + job_retry_enabled = local.resolved_config.job_retry.enabled + job_retry_tags = merge(local.common_tags, local.resolved_config.job_retry.tags) + job_retry_lambda_tags = merge(local.lambda_tags, local.resolved_config.job_retry.tags) + job_retry_log_tags = merge(local.observability_log_tags, local.resolved_config.job_retry.tags) + job_retry_queue_tags = merge(local.queue_tags, local.resolved_config.job_retry.tags) + + enable_job_queued_check = local.resolved_config.scale_up.job_queued_check_enabled == null ? !local.resolved_config.runner.ephemeral : local.resolved_config.scale_up.job_queued_check_enabled +} diff --git a/modules/orchestration-providers/webhook/outputs.tf b/modules/orchestration-providers/webhook/outputs.tf new file mode 100644 index 0000000000..0fb477ba7d --- /dev/null +++ b/modules/orchestration-providers/webhook/outputs.tf @@ -0,0 +1,30 @@ +output "scale_up" { + description = "Scale-up control-plane resources." + value = module.scale_runners.scale_up +} + +output "scale_down" { + description = "Scale-down control-plane resources." + value = module.scale_runners.scale_down +} + +output "pool" { + description = "Scheduled pool resources. Null when no pool schedule is configured." + value = one(module.pool[*].pool) +} + +output "job_retry" { + description = "Job-retry resources. Null when job retry is disabled." + value = local.job_retry_enabled ? { + lambda = one(module.job_retry[*].lambda) + queue = one(module.job_retry[*].job_retry_check_queue) + } : null +} + +output "runner_lifecycle" { + description = "Effective webhook-owned runner lifecycle consumed by runner-config bootstrap parameters." + value = { + ephemeral = local.resolved_config.runner.ephemeral + jit_config_enabled = local.resolved_config.runner.jit_config_enabled + } +} diff --git a/modules/orchestration-providers/webhook/pool.tf b/modules/orchestration-providers/webhook/pool.tf new file mode 100644 index 0000000000..6fb9ad3d34 --- /dev/null +++ b/modules/orchestration-providers/webhook/pool.tf @@ -0,0 +1,66 @@ +module "pool" { + count = length(local.resolved_config.pool.config) > 0 ? 1 : 0 + source = "./pool" + + config = { + prefix = local.resolved_config.prefix + ghes = { + ssl_verify = local.resolved_config.github.enterprise_server.ssl_verify + url = local.resolved_config.github.enterprise_server.url + } + user_agent = local.resolved_config.github.user_agent + github_app_parameters = local.resolved_config.github.app_parameters + runners_maximum_count = local.resolved_config.runner.maximum_count + kms_key_id = local.resolved_config.ssm.kms_key_id + lambda = { + log_level = local.resolved_config.observability.logs.level + logging_retention_in_days = local.resolved_config.observability.logs.retention_in_days + logging_kms_key_id = local.resolved_config.observability.logs.kms_key_id + log_class = local.resolved_config.observability.logs.class + reserved_concurrent_executions = local.resolved_config.pool.reserved_concurrent_executions + s3_bucket = local.resolved_config.lambda.artifact.s3.bucket + s3_key = local.resolved_config.lambda.artifact.s3.key + s3_object_version = local.resolved_config.lambda.artifact.s3.object_version + security_group_ids = local.resolved_config.lambda.security_group_ids + subnet_ids = local.resolved_config.lambda.subnet_ids + architecture = local.resolved_config.lambda.architecture + memory_size = local.resolved_config.pool.memory_size + runtime = local.resolved_config.lambda.runtime + timeout = local.resolved_config.pool.timeout + zip = local.resolved_config.lambda.artifact.zip + parameter_store_tags = local.resolved_config.ssm.parameter_store_tags + principals = local.resolved_config.lambda.role.principals + } + pool = local.resolved_config.pool.config + include_busy_runners = local.resolved_config.pool.include_busy_runners + role_path = local.resolved_config.lambda.role.path + role_permissions_boundary = local.resolved_config.lambda.role.permissions_boundary + runner = { + disable_runner_autoupdate = local.resolved_config.runner.auto_update_disabled + ephemeral = local.resolved_config.runner.ephemeral + enable_jit_config = local.resolved_config.runner.jit_config_enabled + labels = local.resolved_config.runner.labels + group_name = local.resolved_config.runner.group_name + name_prefix = local.resolved_config.runner.name_prefix + pool_owner = local.resolved_config.pool.runner_owner + boot_time_in_minutes = local.resolved_config.runner.boot_time_in_minutes + } + ssm_token_path = local.resolved_config.ssm.token_path + ssm_token_path_arn = local.resolved_config.ssm.token_path_arn + ssm_config_path = local.resolved_config.ssm.config_path + tags = local.pool_tags + lambda_tags = local.pool_lambda_tags + log_group_tags = local.pool_log_tags + arn_ssm_parameters_path_config = local.resolved_config.ssm.config_path_arn + } + + aws_partition = var.aws_partition + tracing_config = local.resolved_config.observability.tracing + runner_provider = { + type = var.runner_provider.type + environment_variables = var.runner_provider.pool.environment_variables + iam_policy_json = var.runner_provider.pool.iam_policy_json + managed_policy_enabled = var.runner_provider.pool.managed_policy_enabled + managed_policy_arn = var.runner_provider.pool.managed_policy_arn + } +} diff --git a/modules/orchestration-providers/webhook/pool/README.md b/modules/orchestration-providers/webhook/pool/README.md new file mode 100644 index 0000000000..877eec8039 --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/README.md @@ -0,0 +1,66 @@ +# Pool module + +This module creates the AWS resources required to maintain a pool of runners. However terraform modules are always exposed and theoretically can be used anywhere. This module is seen as a strict inner module. + +## Why a submodule for the pool + +The pool is an opt-in feature. To be able to use the count on a module level to avoid counts per resources a module is created. All inputs of the module are already defined on a higher level. See the mapping of the variables in [`pool.tf`](../pool.tf) + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.4.0 | +| [aws](#requirement\_aws) | >= 6.21 | + +## Providers + +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 6.21 | +| [terraform](#provider\_terraform) | n/a | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [aws_cloudwatch_log_group.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | +| [aws_iam_role.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role.scheduler](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role_policy.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.pool_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.pool_xray](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.scheduler](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy_attachment.pool_vpc_execution_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | +| [aws_iam_role_policy_attachment.provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | +| [aws_lambda_function.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function) | resource | +| [aws_scheduler_schedule.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/scheduler_schedule) | resource | +| [aws_scheduler_schedule_group.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/scheduler_schedule_group) | resource | +| [terraform_data.validate_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource | +| [aws_iam_policy_document.lambda_assume_role_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.lambda_xray](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.pool_common](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.pool_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scheduler](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scheduler_assume](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [aws\_partition](#input\_aws\_partition) | (optional) partition for the arn if not 'aws' | `string` | `"aws"` | no | +| [config](#input\_config) | Configuration passed from the webhook orchestration provider to the pool Lambda and scheduler.

- `lambda`: Pool Lambda runtime and deployment configuration.
- `lambda.log_level`: Logging level used by the pool Lambda.
- `lambda.logging_retention_in_days`: Number of days to retain events in the pool Lambda log group.
- `lambda.logging_kms_key_id`: KMS key ID used to encrypt the pool Lambda log group.
- `lambda.log_class`: CloudWatch Logs class for the pool Lambda log group.
- `lambda.reserved_concurrent_executions`: Reserved concurrency for the pool Lambda. Use -1 for no reservation.
- `lambda.s3_bucket`: S3 bucket containing the pool Lambda deployment package.
- `lambda.s3_key`: S3 key of the pool Lambda deployment package.
- `lambda.s3_object_version`: S3 object version of the pool Lambda deployment package.
- `lambda.security_group_ids`: Security group IDs associated with the pool Lambda.
- `lambda.runtime`: AWS Lambda runtime used by the pool Lambda.
- `lambda.architecture`: AWS Lambda architecture used by the pool Lambda.
- `lambda.memory_size`: Memory allocated to the pool Lambda in MB.
- `lambda.timeout`: Pool Lambda timeout in seconds.
- `lambda.zip`: Local path to the pool Lambda deployment package when S3 is not used.
- `lambda.subnet_ids`: Subnet IDs in which the pool Lambda runs.
- `lambda.parameter_store_tags`: JSON-encoded tags supplied to the pool Lambda for SSM parameters it creates.
- `lambda.principals`: Additional principals allowed to assume the pool Lambda role.
- `tags`: Common tags added to pool resources.
- `ghes`: GitHub Enterprise Server connection configuration.
- `ghes.url`: GitHub Enterprise Server URL; null when using public GitHub.
- `ghes.ssl_verify`: Whether the pool Lambda verifies the GitHub Enterprise Server TLS certificate.
- `github_app_parameters`: Ordered SSM parameter metadata for GitHub App credentials.
- `github_app_parameters.key_base64`: Ordered Parameter Store references for GitHub App private keys.
- `github_app_parameters.id`: Ordered Parameter Store references for GitHub App IDs.
- `github_app_parameters.installation_id`: Ordered optional Parameter Store references for GitHub App installation IDs.
- `runner`: Runner registration configuration used by the pool Lambda.
- `runner.disable_runner_autoupdate`: Whether GitHub runner automatic updates are disabled.
- `runner.ephemeral`: Whether runners register as ephemeral runners.
- `runner.enable_jit_config`: Whether runners use just-in-time registration configuration.
- `runner.labels`: Labels assigned to runners created by the pool Lambda.
- `runner.group_name`: GitHub runner group assigned to runners created by the pool Lambda.
- `runner.name_prefix`: Prefix used for runner names.
- `runner.pool_owner`: GitHub organization or repository that owns the runner pool.
- `runner.boot_time_in_minutes`: Webhook-provider runner boot timeout used by pool reconciliation.
- `runners_maximum_count`: Webhook-provider runner capacity limit enforced by the pool Lambda.
- `prefix`: Prefix used to name pool resources.
- `pool`: Scheduled pool targets.
- `pool[*].schedule_expression`: EventBridge Scheduler expression for a pool target.
- `pool[*].schedule_expression_timezone`: Time zone used to evaluate the schedule expression.
- `pool[*].size`: Desired runner count for the scheduled pool target.
- `include_busy_runners`: Whether busy runners count toward the desired pool size.
- `role_permissions_boundary`: Permissions boundary applied to IAM roles created for the pool.
- `kms_key_id`: Optional customer-managed KMS key ARN that the pool Lambda may use to decrypt encrypted parameters.
- `role_path`: IAM path applied to roles created for the pool.
- `ssm_token_path`: SSM path under which runner registration tokens are stored.
- `ssm_token_path_arn`: ARN matching the runner registration-token SSM path.
- `ssm_config_path`: SSM path under which runner configuration is stored.
- `arn_ssm_parameters_path_config`: ARN matching the runner configuration SSM path.
- `lambda_tags`: Tags added specifically to the pool Lambda function, overriding common tags with the same key.
- `log_group_tags`: Tags added specifically to the pool Lambda log group, overriding common tags with the same key.
- `user_agent`: User-Agent header used for GitHub API requests. | 
object({
    lambda = object({
      log_level                      = string
      logging_retention_in_days      = number
      logging_kms_key_id             = string
      log_class                      = string
      reserved_concurrent_executions = number
      s3_bucket                      = string
      s3_key                         = string
      s3_object_version              = string
      security_group_ids             = list(string)
      runtime                        = string
      architecture                   = string
      memory_size                    = number
      timeout                        = number
      zip                            = string
      subnet_ids                     = list(string)
      parameter_store_tags           = string
      principals = optional(list(object({
        type        = string
        identifiers = list(string)
      })), [])
    })
    tags = map(string)
    ghes = object({
      url        = string
      ssl_verify = string
    })
    github_app_parameters = object({
      key_base64      = list(map(string))
      id              = list(map(string))
      installation_id = list(object({ name = string, arn = string }))
    })
    runner = object({
      disable_runner_autoupdate = bool
      ephemeral                 = bool
      enable_jit_config         = bool
      labels                    = list(string)
      group_name                = string
      name_prefix               = string
      pool_owner                = string
      boot_time_in_minutes      = number
    })
    runners_maximum_count = number
    prefix                = string
    pool = list(object({
      schedule_expression          = string
      schedule_expression_timezone = string
      size                         = number
    }))
    include_busy_runners           = bool
    role_permissions_boundary      = string
    kms_key_id                     = optional(string, null)
    role_path                      = string
    ssm_token_path                 = string
    ssm_token_path_arn             = string
    ssm_config_path                = string
    arn_ssm_parameters_path_config = string
    lambda_tags                    = map(string)
    log_group_tags                 = optional(map(string), {})
    user_agent                     = string
  })
| n/a | yes | +| [runner\_provider](#input\_runner\_provider) | Compute provider integration used by the pool Lambda.

- `type`: Compute provider type passed to scheduled pool invocations.
- `environment_variables`: Provider-specific environment variables added to the pool Lambda.
- `iam_policy_json`: Provider-specific IAM policy document merged into the pool Lambda policy.
- `managed_policy_enabled`: Whether to attach a provider-specific managed IAM policy to the pool Lambda role.
- `managed_policy_arn`: ARN of the provider-specific managed IAM policy to attach when enabled. | 
object({
    type                   = string
    environment_variables  = map(string)
    iam_policy_json        = string
    managed_policy_enabled = bool
    managed_policy_arn     = optional(string, null)
  })
| n/a | yes | +| [tracing\_config](#input\_tracing\_config) | Tracing configuration for the pool Lambda.

- `mode`: AWS X-Ray tracing mode. A null value disables tracing.
- `capture_http_requests`: Whether Powertools tracing captures outgoing HTTP requests.
- `capture_error`: Whether Powertools tracing captures errors as tracing metadata. | 
object({
    mode                  = optional(string, null)
    capture_http_requests = optional(bool, false)
    capture_error         = optional(bool, false)
  })
| `{}` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [pool](#output\_pool) | Scheduled pool Lambda resources. | + diff --git a/modules/orchestration-providers/webhook/pool/iam-policies.tf b/modules/orchestration-providers/webhook/pool/iam-policies.tf new file mode 100644 index 0000000000..f5a9285bce --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/iam-policies.tf @@ -0,0 +1,77 @@ +# IAM policies attached to the pool Lambda role. +data "aws_iam_policy_document" "pool_common" { + statement { + sid = "WebhookPoolWriteRuntimeParameters" + effect = "Allow" + + actions = [ + "ssm:AddTagsToResource", + "ssm:PutParameter", + ] + + resources = [ + var.config.ssm_token_path_arn, + "${var.config.ssm_token_path_arn}/*", + var.config.arn_ssm_parameters_path_config, + "${var.config.arn_ssm_parameters_path_config}/*", + ] + } + + statement { + sid = "WebhookPoolReadRunnerConfigParameters" + effect = "Allow" + + actions = [ + "ssm:GetParameter", + "ssm:GetParameters", + "ssm:GetParametersByPath", + ] + + resources = [ + var.config.arn_ssm_parameters_path_config, + "${var.config.arn_ssm_parameters_path_config}/*", + ] + } + + statement { + sid = "WebhookPoolReadGitHubAppParameters" + effect = "Allow" + + actions = [ + "ssm:GetParameter", + "ssm:GetParameters", + ] + + resources = concat( + [for p in var.config.github_app_parameters.id : p.arn], + [for p in var.config.github_app_parameters.key_base64 : p.arn], + [for p in var.config.github_app_parameters.installation_id : p.arn if p != null], + ) + } + + dynamic "statement" { + for_each = var.config.kms_key_id == null ? [] : [var.config.kms_key_id] + iterator = kms_key + + content { + sid = "WebhookPoolDecryptParameterStore" + effect = "Allow" + actions = ["kms:Decrypt"] + resources = [kms_key.value] + } + } +} + +data "aws_iam_policy_document" "pool_logging" { + statement { + sid = "WebhookPoolWriteLogs" + effect = "Allow" + + actions = [ + "logs:CreateLogStream", + "logs:PutLogEvents", + ] + + resources = ["${aws_cloudwatch_log_group.pool.arn}*"] + } +} diff --git a/modules/orchestration-providers/webhook/pool/outputs.tf b/modules/orchestration-providers/webhook/pool/outputs.tf new file mode 100644 index 0000000000..cfc429ecce --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/outputs.tf @@ -0,0 +1,8 @@ +output "pool" { + description = "Scheduled pool Lambda resources." + value = { + lambda = aws_lambda_function.pool + log_group = aws_cloudwatch_log_group.pool + role = aws_iam_role.pool + } +} diff --git a/modules/orchestration-providers/webhook/pool/pool.tf b/modules/orchestration-providers/webhook/pool/pool.tf new file mode 100644 index 0000000000..cff2776e90 --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/pool.tf @@ -0,0 +1,236 @@ +# Provider-neutral pool Lambda and scheduler wiring. +locals { + pool_name_prefix = ( + length("${var.config.prefix}-pool") <= 38 + ? "${var.config.prefix}-pool" + : "${substr("${var.config.prefix}-pool", 0, 29)}-${substr(md5("${var.config.prefix}-pool"), 0, 8)}" + ) + + common_environment_variables = { + DISABLE_RUNNER_AUTOUPDATE = var.config.runner.disable_runner_autoupdate + ENABLE_EPHEMERAL_RUNNERS = var.config.runner.ephemeral + ENABLE_JIT_CONFIG = var.config.runner.enable_jit_config + ENVIRONMENT = var.config.prefix + GHES_URL = var.config.ghes.url + USER_AGENT = var.config.user_agent + LOG_LEVEL = upper(var.config.lambda.log_level) + NODE_TLS_REJECT_UNAUTHORIZED = var.config.ghes.url != null && !var.config.ghes.ssl_verify ? 0 : 1 + PARAMETER_GITHUB_APP_ID_NAME = join(":", [for p in var.config.github_app_parameters.id : p.name]) + PARAMETER_GITHUB_APP_KEY_BASE64_NAME = join(":", [for p in var.config.github_app_parameters.key_base64 : p.name]) + PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = join(":", [for p in var.config.github_app_parameters.installation_id : p != null ? p.name : ""]) + POWERTOOLS_LOGGER_LOG_EVENT = var.config.lambda.log_level == "debug" ? "true" : "false" + RUNNER_LABELS = lower(join(",", var.config.runner.labels)) + RUNNER_GROUP_NAME = var.config.runner.group_name + RUNNER_NAME_PREFIX = var.config.runner.name_prefix + RUNNER_OWNER = var.config.runner.pool_owner + RUNNER_BOOT_TIME_IN_MINUTES = var.config.runner.boot_time_in_minutes + RUNNERS_MAXIMUM_COUNT = var.config.runners_maximum_count + SSM_TOKEN_PATH = var.config.ssm_token_path + SSM_CONFIG_PATH = var.config.ssm_config_path + POWERTOOLS_SERVICE_NAME = "${var.config.prefix}-pool" + POWERTOOLS_TRACE_ENABLED = var.tracing_config.mode != null ? true : false + POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.tracing_config.capture_http_requests + POWERTOOLS_TRACER_CAPTURE_ERROR = var.tracing_config.capture_error + SSM_PARAMETER_STORE_TAGS = var.config.lambda.parameter_store_tags + INCLUDE_BUSY_RUNNERS = var.config.include_busy_runners + } +} + +resource "aws_lambda_function" "pool" { + + s3_bucket = var.config.lambda.s3_bucket != null ? var.config.lambda.s3_bucket : null + s3_key = var.config.lambda.s3_key != null ? var.config.lambda.s3_key : null + s3_object_version = var.config.lambda.s3_object_version != null ? var.config.lambda.s3_object_version : null + filename = var.config.lambda.s3_bucket == null ? var.config.lambda.zip : null + source_code_hash = var.config.lambda.s3_bucket == null ? filebase64sha256(var.config.lambda.zip) : null + function_name = "${var.config.prefix}-pool" + role = aws_iam_role.pool.arn + handler = "index.adjustPool" + architectures = [var.config.lambda.architecture] + runtime = var.config.lambda.runtime + timeout = var.config.lambda.timeout + reserved_concurrent_executions = var.config.lambda.reserved_concurrent_executions + memory_size = var.config.lambda.memory_size + tags = merge(var.config.tags, var.config.lambda_tags) + + environment { + variables = merge(var.runner_provider.environment_variables, local.common_environment_variables) + } + + dynamic "vpc_config" { + for_each = var.config.lambda.subnet_ids != null && var.config.lambda.security_group_ids != null ? [true] : [] + content { + security_group_ids = var.config.lambda.security_group_ids + subnet_ids = var.config.lambda.subnet_ids + } + } + + dynamic "tracing_config" { + for_each = var.tracing_config.mode != null ? [true] : [] + content { + mode = var.tracing_config.mode + } + } +} + +resource "aws_cloudwatch_log_group" "pool" { + name = "/aws/lambda/${aws_lambda_function.pool.function_name}" + retention_in_days = var.config.lambda.logging_retention_in_days + kms_key_id = var.config.lambda.logging_kms_key_id + log_group_class = var.config.lambda.log_class + tags = merge(var.config.tags, var.config.log_group_tags) +} + +resource "aws_iam_role" "pool" { + name = "${substr("${var.config.prefix}-pool-lambda", 0, 54)}-${substr(md5("${var.config.prefix}-pool-lambda"), 0, 8)}" + assume_role_policy = data.aws_iam_policy_document.lambda_assume_role_policy.json + path = var.config.role_path + permissions_boundary = var.config.role_permissions_boundary + tags = var.config.tags +} + +resource "aws_iam_role_policy" "pool" { + name = "pool-policy" + role = aws_iam_role.pool.name + policy = data.aws_iam_policy_document.pool.json +} + +data "aws_iam_policy_document" "pool" { + source_policy_documents = [ + data.aws_iam_policy_document.pool_common.json, + var.runner_provider.iam_policy_json, + ] +} + +resource "aws_iam_role_policy" "pool_logging" { + name = "logging-policy" + role = aws_iam_role.pool.name + policy = data.aws_iam_policy_document.pool_logging.json +} + +resource "aws_iam_role_policy_attachment" "pool_vpc_execution_role" { + count = length(var.config.lambda.subnet_ids) > 0 ? 1 : 0 + role = aws_iam_role.pool.name + policy_arn = "arn:${var.aws_partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole" +} + +data "aws_iam_policy_document" "lambda_assume_role_policy" { + statement { + actions = ["sts:AssumeRole"] + + principals { + type = "Service" + identifiers = ["lambda.amazonaws.com"] + } + + dynamic "principals" { + for_each = var.config.lambda.principals + + content { + type = principals.value.type + identifiers = principals.value.identifiers + } + } + } +} + +resource "aws_iam_role_policy_attachment" "provider" { + count = var.runner_provider.managed_policy_enabled ? 1 : 0 + role = aws_iam_role.pool.name + policy_arn = var.runner_provider.managed_policy_arn +} + +# AWS X-Ray write/read trace APIs do not support resource-level permissions. +data "aws_iam_policy_document" "lambda_xray" { + count = var.tracing_config.mode != null ? 1 : 0 + statement { + actions = [ + "xray:BatchGetTraces", + "xray:GetTraceSummaries", + "xray:PutTelemetryRecords", + "xray:PutTraceSegments" + ] + effect = "Allow" + resources = [ + "*" + ] + sid = "AllowXRay" + } +} + +resource "aws_iam_role_policy" "pool_xray" { + count = var.tracing_config.mode != null ? 1 : 0 + name = "xray-policy" + policy = data.aws_iam_policy_document.lambda_xray[0].json + role = aws_iam_role.pool.name +} + +resource "aws_scheduler_schedule_group" "pool" { + name_prefix = local.pool_name_prefix + + tags = var.config.tags +} + +data "aws_iam_policy_document" "scheduler_assume" { + statement { + sid = "ScheduleGroupAssumeRole" + actions = ["sts:AssumeRole"] + principals { + type = "Service" + identifiers = ["scheduler.amazonaws.com"] + } + + condition { + test = "StringEquals" + variable = "aws:SourceArn" + values = [aws_scheduler_schedule_group.pool.arn] + } + } +} + +data "aws_iam_policy_document" "scheduler" { + statement { + sid = "InvokePoolLambda" + actions = ["lambda:InvokeFunction"] + resources = [aws_lambda_function.pool.arn] + } +} + +resource "aws_iam_role" "scheduler" { + name_prefix = local.pool_name_prefix + + path = var.config.role_path + permissions_boundary = var.config.role_permissions_boundary + + assume_role_policy = data.aws_iam_policy_document.scheduler_assume.json + tags = var.config.tags +} + +resource "aws_iam_role_policy" "scheduler" { + name = "terraform" + role = aws_iam_role.scheduler.name + policy = data.aws_iam_policy_document.scheduler.json +} + +resource "aws_scheduler_schedule" "pool" { + for_each = { for i, v in var.config.pool : i => v } + + name = "${var.config.prefix}-pool-${each.key}-rule" + group_name = aws_scheduler_schedule_group.pool.name + + flexible_time_window { + mode = "OFF" + } + + schedule_expression = each.value.schedule_expression + schedule_expression_timezone = each.value.schedule_expression_timezone + + target { + arn = aws_lambda_function.pool.arn + role_arn = aws_iam_role.scheduler.arn + input = jsonencode({ + poolSize = each.value.size + type = var.runner_provider.type + }) + } +} diff --git a/modules/orchestration-providers/webhook/pool/tests/provider.tftest.hcl b/modules/orchestration-providers/webhook/pool/tests/provider.tftest.hcl new file mode 100644 index 0000000000..c04f435024 --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/tests/provider.tftest.hcl @@ -0,0 +1,297 @@ +mock_provider "aws" { + mock_data "aws_iam_policy_document" { + defaults = { + json = "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"logs:CreateLogStream\",\"Resource\":\"*\"}]}" + } + } +} + +variables { + config = { + lambda = { + log_level = "info" + logging_retention_in_days = 14 + logging_kms_key_id = null + log_class = "STANDARD" + reserved_concurrent_executions = 1 + s3_bucket = "lambda-artifacts" + s3_key = "runners.zip" + s3_object_version = null + security_group_ids = [] + runtime = "nodejs24.x" + architecture = "arm64" + memory_size = 256 + timeout = 60 + zip = "runners.zip" + subnet_ids = [] + parameter_store_tags = "{}" + principals = [{ + type = "AWS" + identifiers = ["arn:aws:iam::123456789012:role/local-testing"] + }] + } + tags = { + Environment = "pool-test" + } + ghes = { + url = null + ssl_verify = true + } + github_app_parameters = { + key_base64 = [ + { + name = "/github-runner/key-base64" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64" + }, + { + name = "/github-runner/key-base64-2" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64-2" + }, + ] + id = [ + { + name = "/github-runner/app-id" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id" + }, + { + name = "/github-runner/app-id-2" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id-2" + }, + ] + installation_id = [ + null, + { + name = "/github-runner/installation-id-2" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/installation-id-2" + }, + ] + } + runner = { + disable_runner_autoupdate = false + ephemeral = true + enable_jit_config = true + labels = ["self-hosted", "microvm"] + group_name = "default" + name_prefix = "microvm" + pool_owner = "example" + boot_time_in_minutes = 13 + } + runners_maximum_count = 10 + prefix = "pool-test" + pool = [{ + schedule_expression = "cron(0 8 * * ? *)" + schedule_expression_timezone = "UTC" + size = 2 + }] + include_busy_runners = false + role_permissions_boundary = null + kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/pool-test" + role_path = "/" + ssm_token_path = "/github-runner/tokens" + ssm_token_path_arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/tokens" + ssm_config_path = "/github-runner/config" + arn_ssm_parameters_path_config = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/config" + lambda_tags = {} + user_agent = "terraform-aws-github-runner" + } + + runner_provider = { + type = "microvm" + environment_variables = { + MICROVM_CLUSTER = "runner-cluster" + } + iam_policy_json = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Effect = "Allow" + Action = ["microvm:CreateRunner"] + Resource = ["*"] + }] + }) + managed_policy_enabled = true + managed_policy_arn = "arn:aws:iam::123456789012:policy/microvm-pool" + } + + tracing_config = { + mode = "Active" + capture_http_requests = true + capture_error = true + } +} + +run "provider_supplies_only_compute_specific_pool_configuration" { + command = plan + + assert { + condition = ( + length(data.aws_iam_policy_document.lambda_assume_role_policy.statement[0].principals) == 2 && + contains(data.aws_iam_policy_document.lambda_assume_role_policy.statement[0].principals[*].type, "AWS") + ) + error_message = "The pool Lambda trust policy must include configured additional principals." + } + + assert { + condition = toset(keys(output.pool)) == toset(["lambda", "log_group", "role"]) + error_message = "The pool module must expose its resources through one nested output." + } + + assert { + condition = ( + aws_lambda_function.pool.environment[0].variables["RUNNER_OWNER"] == "example" + && aws_lambda_function.pool.environment[0].variables["RUNNERS_MAXIMUM_COUNT"] == "10" + && aws_lambda_function.pool.environment[0].variables["RUNNER_BOOT_TIME_IN_MINUTES"] == "13" + ) + error_message = "The pool module must assemble common runner registration values and webhook-provider capacity and boot-time settings." + } + + assert { + condition = ( + aws_lambda_function.pool.environment[0].variables["PARAMETER_GITHUB_APP_ID_NAME"] == "/github-runner/app-id:/github-runner/app-id-2" + && aws_lambda_function.pool.environment[0].variables["PARAMETER_GITHUB_APP_KEY_BASE64_NAME"] == "/github-runner/key-base64:/github-runner/key-base64-2" + && aws_lambda_function.pool.environment[0].variables["PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME"] == ":/github-runner/installation-id-2" + && contains(data.aws_iam_policy_document.pool_common.statement[2].resources, "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id-2") + && contains(data.aws_iam_policy_document.pool_common.statement[2].resources, "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64-2") + && contains(data.aws_iam_policy_document.pool_common.statement[2].resources, "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/installation-id-2") + ) + error_message = "Pool must pass every GitHub App parameter and grant access to every corresponding SSM ARN." + } + + assert { + condition = aws_lambda_function.pool.environment[0].variables["MICROVM_CLUSTER"] == "runner-cluster" + error_message = "The pool module must merge compute-provider environment variables into the Lambda environment." + } + + assert { + condition = !contains(keys(aws_lambda_function.pool.environment[0].variables), "AMI_ID_SSM_PARAMETER_NAME") + error_message = "The common pool module must not add EC2-specific environment variables." + } + + assert { + condition = jsondecode(aws_scheduler_schedule.pool["0"].target[0].input).type == "microvm" + error_message = "The pool scheduler payload must select the configured compute provider." + } + + assert { + condition = length(data.aws_iam_policy_document.pool.source_policy_documents) == 2 + error_message = "The pool role policy must merge the common and compute-provider policy documents." + } + + assert { + condition = ( + length(data.aws_iam_policy_document.pool_common.statement) == 4 + && one([ + for statement in data.aws_iam_policy_document.pool_common.statement : statement + if statement.sid == "WebhookPoolDecryptParameterStore" + ]).resources == toset(["arn:aws:kms:eu-west-1:123456789012:key/pool-test"]) + ) + error_message = "The pool KMS policy statement must consume the scalar key ARN." + } + + assert { + condition = ( + one([ + for statement in data.aws_iam_policy_document.pool_common.statement : statement + if statement.sid == "WebhookPoolWriteRuntimeParameters" + ]).resources == toset([ + "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/tokens", + "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/tokens/*", + "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/config", + "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/config/*", + ]) + && !contains(one([ + for statement in data.aws_iam_policy_document.pool_common.statement : statement + if statement.sid == "WebhookPoolWriteRuntimeParameters" + ]).resources, "*") + ) + error_message = "The pool Lambda must scope runtime SSM writes to the token and runner-config parameter paths." + } + + assert { + condition = ( + data.aws_iam_policy_document.lambda_xray[0].statement[0].sid == "AllowXRay" + && data.aws_iam_policy_document.lambda_xray[0].statement[0].resources == toset(["*"]) + && toset(data.aws_iam_policy_document.lambda_xray[0].statement[0].actions) == toset([ + "xray:BatchGetTraces", + "xray:GetTraceSummaries", + "xray:PutTelemetryRecords", + "xray:PutTraceSegments", + ]) + ) + error_message = "Only the resource-agnostic X-Ray APIs may retain a wildcard resource in the pool policies." + } + + assert { + condition = length(aws_iam_role_policy_attachment.provider) == 1 + error_message = "The optional compute-provider managed policy must be attached to the pool role." + } +} + +run "omits_optional_kms_statement" { + command = plan + + variables { + config = merge(var.config, { + kms_key_id = null + }) + } + + assert { + condition = ( + length(data.aws_iam_policy_document.pool_common.statement) == 3 + && length([ + for statement in data.aws_iam_policy_document.pool_common.statement : statement + if anytrue([for action in statement.actions : startswith(action, "kms:")]) + ]) == 0 + ) + error_message = "A null Parameter Store key must omit the optional pool KMS statement." + } +} + +run "rejects_empty_compute_provider_type" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + runner_provider = merge(var.runner_provider, { + type = " " + }) + } + + expect_failures = [terraform_data.validate_config] +} + +run "rejects_invalid_compute_provider_policy" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + runner_provider = merge(var.runner_provider, { + iam_policy_json = "not-json" + }) + } + + expect_failures = [terraform_data.validate_config] +} + +run "requires_enabled_compute_provider_managed_policy_arn" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + runner_provider = merge(var.runner_provider, { + managed_policy_enabled = true + managed_policy_arn = null + }) + } + + expect_failures = [terraform_data.validate_config] +} diff --git a/modules/orchestration-providers/webhook/pool/validations.tf b/modules/orchestration-providers/webhook/pool/validations.tf new file mode 100644 index 0000000000..f18d251c23 --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/validations.tf @@ -0,0 +1,18 @@ +resource "terraform_data" "validate_config" { + lifecycle { + precondition { + condition = trimspace(var.runner_provider.type) != "" + error_message = "The compute provider type must not be empty." + } + + precondition { + condition = can(jsondecode(var.runner_provider.iam_policy_json)) + error_message = "The compute provider IAM policy must be valid JSON." + } + + precondition { + condition = !var.runner_provider.managed_policy_enabled || var.runner_provider.managed_policy_arn != null + error_message = "The compute provider managed policy ARN must be set when its attachment is enabled." + } + } +} diff --git a/modules/orchestration-providers/webhook/pool/variables.tf b/modules/orchestration-providers/webhook/pool/variables.tf new file mode 100644 index 0000000000..e1f516c8ad --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/variables.tf @@ -0,0 +1,161 @@ +variable "config" { + description = <<-EOF + Configuration passed from the webhook orchestration provider to the pool Lambda and scheduler. + + - `lambda`: Pool Lambda runtime and deployment configuration. + - `lambda.log_level`: Logging level used by the pool Lambda. + - `lambda.logging_retention_in_days`: Number of days to retain events in the pool Lambda log group. + - `lambda.logging_kms_key_id`: KMS key ID used to encrypt the pool Lambda log group. + - `lambda.log_class`: CloudWatch Logs class for the pool Lambda log group. + - `lambda.reserved_concurrent_executions`: Reserved concurrency for the pool Lambda. Use -1 for no reservation. + - `lambda.s3_bucket`: S3 bucket containing the pool Lambda deployment package. + - `lambda.s3_key`: S3 key of the pool Lambda deployment package. + - `lambda.s3_object_version`: S3 object version of the pool Lambda deployment package. + - `lambda.security_group_ids`: Security group IDs associated with the pool Lambda. + - `lambda.runtime`: AWS Lambda runtime used by the pool Lambda. + - `lambda.architecture`: AWS Lambda architecture used by the pool Lambda. + - `lambda.memory_size`: Memory allocated to the pool Lambda in MB. + - `lambda.timeout`: Pool Lambda timeout in seconds. + - `lambda.zip`: Local path to the pool Lambda deployment package when S3 is not used. + - `lambda.subnet_ids`: Subnet IDs in which the pool Lambda runs. + - `lambda.parameter_store_tags`: JSON-encoded tags supplied to the pool Lambda for SSM parameters it creates. + - `lambda.principals`: Additional principals allowed to assume the pool Lambda role. + - `tags`: Common tags added to pool resources. + - `ghes`: GitHub Enterprise Server connection configuration. + - `ghes.url`: GitHub Enterprise Server URL; null when using public GitHub. + - `ghes.ssl_verify`: Whether the pool Lambda verifies the GitHub Enterprise Server TLS certificate. + - `github_app_parameters`: Ordered SSM parameter metadata for GitHub App credentials. + - `github_app_parameters.key_base64`: Ordered Parameter Store references for GitHub App private keys. + - `github_app_parameters.id`: Ordered Parameter Store references for GitHub App IDs. + - `github_app_parameters.installation_id`: Ordered optional Parameter Store references for GitHub App installation IDs. + - `runner`: Runner registration configuration used by the pool Lambda. + - `runner.disable_runner_autoupdate`: Whether GitHub runner automatic updates are disabled. + - `runner.ephemeral`: Whether runners register as ephemeral runners. + - `runner.enable_jit_config`: Whether runners use just-in-time registration configuration. + - `runner.labels`: Labels assigned to runners created by the pool Lambda. + - `runner.group_name`: GitHub runner group assigned to runners created by the pool Lambda. + - `runner.name_prefix`: Prefix used for runner names. + - `runner.pool_owner`: GitHub organization or repository that owns the runner pool. + - `runner.boot_time_in_minutes`: Webhook-provider runner boot timeout used by pool reconciliation. + - `runners_maximum_count`: Webhook-provider runner capacity limit enforced by the pool Lambda. + - `prefix`: Prefix used to name pool resources. + - `pool`: Scheduled pool targets. + - `pool[*].schedule_expression`: EventBridge Scheduler expression for a pool target. + - `pool[*].schedule_expression_timezone`: Time zone used to evaluate the schedule expression. + - `pool[*].size`: Desired runner count for the scheduled pool target. + - `include_busy_runners`: Whether busy runners count toward the desired pool size. + - `role_permissions_boundary`: Permissions boundary applied to IAM roles created for the pool. + - `kms_key_id`: Optional customer-managed KMS key ARN that the pool Lambda may use to decrypt encrypted parameters. + - `role_path`: IAM path applied to roles created for the pool. + - `ssm_token_path`: SSM path under which runner registration tokens are stored. + - `ssm_token_path_arn`: ARN matching the runner registration-token SSM path. + - `ssm_config_path`: SSM path under which runner configuration is stored. + - `arn_ssm_parameters_path_config`: ARN matching the runner configuration SSM path. + - `lambda_tags`: Tags added specifically to the pool Lambda function, overriding common tags with the same key. + - `log_group_tags`: Tags added specifically to the pool Lambda log group, overriding common tags with the same key. + - `user_agent`: User-Agent header used for GitHub API requests. + EOF + type = object({ + lambda = object({ + log_level = string + logging_retention_in_days = number + logging_kms_key_id = string + log_class = string + reserved_concurrent_executions = number + s3_bucket = string + s3_key = string + s3_object_version = string + security_group_ids = list(string) + runtime = string + architecture = string + memory_size = number + timeout = number + zip = string + subnet_ids = list(string) + parameter_store_tags = string + principals = optional(list(object({ + type = string + identifiers = list(string) + })), []) + }) + tags = map(string) + ghes = object({ + url = string + ssl_verify = string + }) + github_app_parameters = object({ + key_base64 = list(map(string)) + id = list(map(string)) + installation_id = list(object({ name = string, arn = string })) + }) + runner = object({ + disable_runner_autoupdate = bool + ephemeral = bool + enable_jit_config = bool + labels = list(string) + group_name = string + name_prefix = string + pool_owner = string + boot_time_in_minutes = number + }) + runners_maximum_count = number + prefix = string + pool = list(object({ + schedule_expression = string + schedule_expression_timezone = string + size = number + })) + include_busy_runners = bool + role_permissions_boundary = string + kms_key_id = optional(string, null) + role_path = string + ssm_token_path = string + ssm_token_path_arn = string + ssm_config_path = string + arn_ssm_parameters_path_config = string + lambda_tags = map(string) + log_group_tags = optional(map(string), {}) + user_agent = string + }) +} + +variable "runner_provider" { + description = <<-EOF + Compute provider integration used by the pool Lambda. + + - `type`: Compute provider type passed to scheduled pool invocations. + - `environment_variables`: Provider-specific environment variables added to the pool Lambda. + - `iam_policy_json`: Provider-specific IAM policy document merged into the pool Lambda policy. + - `managed_policy_enabled`: Whether to attach a provider-specific managed IAM policy to the pool Lambda role. + - `managed_policy_arn`: ARN of the provider-specific managed IAM policy to attach when enabled. + EOF + type = object({ + type = string + environment_variables = map(string) + iam_policy_json = string + managed_policy_enabled = bool + managed_policy_arn = optional(string, null) + }) +} + +variable "aws_partition" { + description = "(optional) partition for the arn if not 'aws'" + type = string + default = "aws" +} + +variable "tracing_config" { + description = <<-EOF + Tracing configuration for the pool Lambda. + + - `mode`: AWS X-Ray tracing mode. A null value disables tracing. + - `capture_http_requests`: Whether Powertools tracing captures outgoing HTTP requests. + - `capture_error`: Whether Powertools tracing captures errors as tracing metadata. + EOF + type = object({ + mode = optional(string, null) + capture_http_requests = optional(bool, false) + capture_error = optional(bool, false) + }) + default = {} +} diff --git a/modules/orchestration-providers/webhook/pool/versions.tf b/modules/orchestration-providers/webhook/pool/versions.tf new file mode 100644 index 0000000000..fcec7c620d --- /dev/null +++ b/modules/orchestration-providers/webhook/pool/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.4.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 6.21" + } + } +} diff --git a/modules/orchestration-providers/webhook/scale-down-state-diagram.md b/modules/orchestration-providers/webhook/scale-down-state-diagram.md new file mode 100644 index 0000000000..2e94d9ba79 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-down-state-diagram.md @@ -0,0 +1,150 @@ +# GitHub Actions Runner Scale-Down State Diagram + + + +The scale-down Lambda function runs on a scheduled basis (every 5 minutes by default) to manage GitHub Actions runner instances. It performs a two-phase cleanup process: first terminating confirmed orphaned instances, then evaluating active runners to maintain the desired idle capacity while removing unnecessary instances. + +```mermaid +stateDiagram-v2 + [*] --> ScheduledExecution : Cron Trigger every 5 min + + ScheduledExecution --> Phase1_OrphanTermination : Start Phase 1 + + state Phase1_OrphanTermination { + [*] --> ListOrphanInstances : Query EC2 for ghr orphan true + + ListOrphanInstances --> CheckOrphanType : For each orphan + + state CheckOrphanType <> + CheckOrphanType --> HasRunnerIdTag : Has ghr github runner id + CheckOrphanType --> TerminateOrphan : No runner ID tag + + HasRunnerIdTag --> LastChanceCheck : Query GitHub API + + state LastChanceCheck <> + LastChanceCheck --> ConfirmedOrphan : Offline and busy + LastChanceCheck --> FalsePositive : Exists and not problematic + + ConfirmedOrphan --> TerminateOrphan + FalsePositive --> RemoveOrphanTag + + TerminateOrphan --> NextOrphan : Continue processing + RemoveOrphanTag --> NextOrphan + + NextOrphan --> CheckOrphanType : More orphans? + NextOrphan --> Phase2_ActiveRunners : All processed + } + + Phase1_OrphanTermination --> Phase2_ActiveRunners : Phase 1 Complete + + state Phase2_ActiveRunners { + [*] --> ListActiveRunners : Query non-orphan EC2 instances + + ListActiveRunners --> GroupByOwner : Sort by owner and repo + + GroupByOwner --> ProcessOwnerGroup : For each owner + + state ProcessOwnerGroup { + [*] --> SortByStrategy : Apply eviction strategy + SortByStrategy --> ProcessRunner : Oldest first or newest first + + ProcessRunner --> QueryGitHub : Get GitHub runners for owner + + QueryGitHub --> MatchRunner : Find runner by instance ID suffix + + state MatchRunner <> + MatchRunner --> FoundInGitHub : Runner exists in GitHub + MatchRunner --> NotFoundInGitHub : Runner not in GitHub + + state FoundInGitHub { + [*] --> CheckMinimumTime : Has minimum runtime passed? + + state CheckMinimumTime <> + CheckMinimumTime --> TooYoung : Runtime less than minimum + CheckMinimumTime --> CheckIdleQuota : Runtime greater than or equal to minimum + + TooYoung --> NextRunner + + state CheckIdleQuota <> + CheckIdleQuota --> KeepIdle : Idle quota available + CheckIdleQuota --> CheckBusyState : Quota full + + KeepIdle --> NextRunner + + state CheckBusyState <> + CheckBusyState --> KeepBusy : Runner busy + CheckBusyState --> TerminateIdle : Runner idle + + KeepBusy --> NextRunner + TerminateIdle --> DeregisterFromGitHub + DeregisterFromGitHub --> TerminateInstance + TerminateInstance --> NextRunner + } + + state NotFoundInGitHub { + [*] --> CheckBootTime : Has boot time exceeded? + + state CheckBootTime <> + CheckBootTime --> StillBooting : Boot time less than threshold + CheckBootTime --> MarkOrphan : Boot time greater than or equal to threshold + + StillBooting --> NextRunner + MarkOrphan --> TagAsOrphan : Set ghr orphan true + TagAsOrphan --> NextRunner + } + + NextRunner --> ProcessRunner : More runners in group? + NextRunner --> NextOwnerGroup : Group complete + } + + NextOwnerGroup --> ProcessOwnerGroup : More owner groups? + NextOwnerGroup --> ExecutionComplete : All groups processed + } + + Phase2_ActiveRunners --> ExecutionComplete : Phase 2 Complete + + ExecutionComplete --> [*] : Wait for next cron trigger + + note right of LastChanceCheck + Uses ghr github runner id tag + for precise GitHub API lookup + end note + + note right of MatchRunner + Matches GitHub runner name + ending with EC2 instance ID + end note + + note right of CheckMinimumTime + Minimum running time in minutes + (Linux: 5min, Windows: 15min, OSX: 20min) + end note + + note right of CheckBootTime + Runner boot time in minutes + Default configuration value + end note +``` + + + +## Key Decision Points + +| State | Condition | Action | +|-------|-----------|--------| +| **Orphan w/ Runner ID** | GitHub: offline + busy | Terminate (confirmed orphan) | +| **Orphan w/ Runner ID** | GitHub: exists + healthy | Remove orphan tag (false positive) | +| **Orphan w/o Runner ID** | Always | Terminate (no way to verify) | +| **Active Runner Found** | Runtime < minimum | Keep (too young) | +| **Active Runner Found** | Idle quota available | Keep as idle | +| **Active Runner Found** | Quota full + idle | Terminate + deregister | +| **Active Runner Found** | Quota full + busy | Keep running | +| **Active Runner Missing** | Boot time exceeded | Mark as orphan | +| **Active Runner Missing** | Still booting | Wait | + +## Configuration Parameters + +- **Cron Schedule**: `cron(*/5 * * * ? *)` (every 5 minutes) +- **Minimum Runtime**: Linux 5min, Windows 15min, OSX 20min +- **Boot Timeout**: Configurable via `orchestration_provider.webhook.runner.boot_time_in_minutes`; stable-v1 inputs are translated from `runner_boot_time_in_minutes`. +- **Idle Config**: Per-environment configuration for desired idle runners diff --git a/modules/orchestration-providers/webhook/scale-runners.tf b/modules/orchestration-providers/webhook/scale-runners.tf new file mode 100644 index 0000000000..caf79eefb5 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners.tf @@ -0,0 +1,61 @@ +module "scale_runners" { + source = "./scale-runners" + + aws_partition = var.aws_partition + + config = { + prefix = local.resolved_config.prefix + lambda = { + artifact = local.resolved_config.lambda.artifact + runtime = local.resolved_config.lambda.runtime + architecture = local.resolved_config.lambda.architecture + vpc = { + subnet_ids = local.resolved_config.lambda.subnet_ids + security_group_ids = local.resolved_config.lambda.security_group_ids + } + role = local.resolved_config.lambda.role + } + runner = local.resolved_config.runner + github = local.resolved_config.github + queue = { + build = local.resolved_config.queue.build + kms_key_id = local.resolved_config.queue.kms_key_id + event_source_mapping = local.resolved_config.queue.event_source_mapping + } + ssm = local.resolved_config.ssm + observability = { + logs = local.resolved_config.observability.logs + tracing = local.resolved_config.observability.tracing + metrics = local.resolved_config.observability.metrics + } + scale_up = merge(local.resolved_config.scale_up, { + job_queued_check_enabled = local.enable_job_queued_check + tags = { + resources = local.scale_up_tags + lambda = local.scale_up_lambda_tags + log_group = local.scale_up_log_tags + event_source_mapping = local.scale_up_queue_tags + } + }) + scale_down = merge(local.resolved_config.scale_down, { + tags = { + resources = local.scale_down_tags + lambda = local.scale_down_lambda_tags + log_group = local.scale_down_log_tags + } + }) + job_retry = { + enabled = local.job_retry_enabled + max_attempts = local.resolved_config.job_retry.max_attempts + delay_in_seconds = local.resolved_config.job_retry.delay_in_seconds + delay_backoff = local.resolved_config.job_retry.delay_backoff + queue = one(module.job_retry[*].job_retry_check_queue) + } + } + + runner_provider = { + type = var.runner_provider.type + scale_up = var.runner_provider.scale_up + scale_down = var.runner_provider.scale_down + } +} diff --git a/modules/orchestration-providers/webhook/scale-runners/README.md b/modules/orchestration-providers/webhook/scale-runners/README.md new file mode 100644 index 0000000000..3b096f9b85 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/README.md @@ -0,0 +1,79 @@ +# Scale runners module + +> This module is treated as an internal module; breaking changes do not trigger a major release bump. + +This provider-neutral child module owns the scale-up and scale-down Lambda functions, their event sources and schedules, and their IAM and logging resources. `runner-config` supplies common configuration through the webhook orchestration provider together with the selected compute provider's environment and IAM fragments. + +The module is an implementation detail of the experimental runner configuration. It is composed by the webhook orchestration provider and is not intended to be called directly. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.4.0 | +| [aws](#requirement\_aws) | >= 6.33 | + +## Providers + +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | >= 6.33 | +| [terraform](#provider\_terraform) | n/a | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [aws_cloudwatch_event_rule.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule) | resource | +| [aws_cloudwatch_event_target.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_target) | resource | +| [aws_cloudwatch_log_group.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | +| [aws_cloudwatch_log_group.scale_up](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | +| [aws_iam_role.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role.scale_up](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | +| [aws_iam_role_policy.job_retry_sqs_publish](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.scale_down_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.scale_down_xray](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.scale_up](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.scale_up_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.scale_up_xray](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy.service_linked_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | +| [aws_iam_role_policy_attachment.provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | +| [aws_iam_role_policy_attachment.scale_down_vpc_execution_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | +| [aws_iam_role_policy_attachment.scale_up_vpc_execution_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | +| [aws_lambda_event_source_mapping.scale_up](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_event_source_mapping) | resource | +| [aws_lambda_function.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function) | resource | +| [aws_lambda_function.scale_up](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function) | resource | +| [aws_lambda_permission.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) | resource | +| [aws_lambda_permission.scale_runners_lambda](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) | resource | +| [terraform_data.validate_config](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource | +| [aws_iam_policy_document.lambda_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.lambda_xray](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scale_down_common](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scale_down_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scale_up](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scale_up_common](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scale_up_job_retry_publish](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | +| [aws_iam_policy_document.scale_up_logging](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [aws\_partition](#input\_aws\_partition) | AWS partition used to construct IAM policy ARNs. | `string` | `"aws"` | no | +| [config](#input\_config) | Provider-neutral scale-up and scale-down configuration assembled by runner-config.

- `prefix`: Prefix used to name scaling resources.
- `lambda.artifact.zip`: Resolved local control-plane archive.
- `lambda.artifact.s3.bucket`: Optional S3 bucket containing the Lambda archive.
- `lambda.artifact.s3.key`: Object key of the Lambda archive.
- `lambda.artifact.s3.object_version`: Optional object version of the Lambda archive.
- `lambda.runtime`: Runtime used by both scaling Lambdas.
- `lambda.architecture`: Instruction-set architecture used by both scaling Lambdas.
- `lambda.vpc.subnet_ids`: Subnets used for Lambda VPC configuration.
- `lambda.vpc.security_group_ids`: Security groups used for Lambda VPC configuration.
- `lambda.role.path`: IAM path used for the scaling Lambda roles.
- `lambda.role.permissions_boundary`: Optional permissions boundary for the scaling Lambda roles.
- `lambda.role.principals`: Additional principals allowed to assume the scaling Lambda roles.
- `runner.os`: Runner operating system used for the minimum-runtime default.
- `runner.auto_update_disabled`: Disables the GitHub runner application's built-in updater.
- `runner.ephemeral`: Registers runners in ephemeral mode.
- `runner.jit_config_enabled`: Enables or disables just-in-time runner configuration.
- `runner.labels`: Labels supplied when a runner is registered.
- `runner.group_name`: GitHub runner group used during registration.
- `runner.name_prefix`: Prefix added to registered runner names.
- `runner.boot_time_in_minutes`: Webhook-provider runner boot timeout used by scale-down.
- `runner.maximum_count`: Webhook-provider runner capacity limit for this runner configuration.
- `github.organization_runners`: Registers organization runners when true.
- `github.enterprise_server.url`: Optional GitHub Enterprise Server URL.
- `github.enterprise_server.ssl_verify`: Enables TLS verification for GitHub Enterprise Server.
- `github.user_agent`: Optional User-Agent sent to GitHub.
- `github.app_parameters.key_base64`: Ordered Parameter Store references for GitHub App private keys.
- `github.app_parameters.id`: Ordered Parameter Store references for GitHub App IDs.
- `github.app_parameters.installation_id`: Ordered optional Parameter Store references for GitHub App installation IDs.
- `queue.build.arn`: ARN of the build queue consumed by scale-up.
- `queue.kms_key_id`: Optional KMS key ARN used to encrypt the build queue. This is distinct from the Parameter Store key.
- `queue.event_source_mapping.batch_size`: Maximum records delivered per scale-up invocation.
- `queue.event_source_mapping.maximum_batching_window_in_seconds`: Maximum event batching window.
- `ssm.token_path`: Parameter Store path used for registration tokens.
- `ssm.token_path_arn`: ARN of the Parameter Store path used for registration tokens.
- `ssm.config_path`: Parameter Store path used for persistent runner configuration.
- `ssm.config_path_arn`: ARN of the persistent runner configuration path.
- `ssm.kms_key_id`: Optional KMS key ARN used to decrypt shared parameters. Its value may be unknown until apply.
- `ssm.parameter_store_tags`: JSON-encoded tags applied to parameters created at runtime.
- `observability.logs`: Shared logging level, retention, encryption, and log-class configuration.
- `observability.tracing`: Lambda X-Ray and tracing-helper configuration.
- `observability.metrics`: Metrics enablement, namespace, and GitHub rate-limit metric configuration.
- `scale_up`: Scale-up Lambda sizing, concurrency, queued-job behavior, and resolved resource tag maps.
- `scale_up.tags.resources`: Tags for the scale-up IAM role and other component resources.
- `scale_up.tags.lambda`: Tags for the scale-up Lambda function.
- `scale_up.tags.log_group`: Tags for the scale-up log group.
- `scale_up.tags.event_source_mapping`: Tags for the build-queue event-source mapping.
- `scale_down`: Scale-down Lambda sizing, schedule, idle configuration, minimum runtime, and resolved resource tag maps.
- `scale_down.tags.resources`: Tags for the scale-down IAM role and EventBridge rule.
- `scale_down.tags.lambda`: Tags for the scale-down Lambda function.
- `scale_down.tags.log_group`: Tags for the scale-down log group.
- `job_retry.enabled`: Enables publishing retry checks from scale-up.
- `job_retry.queue`: Retry queue ARN and URL. Required when job retry is enabled.
- `job_retry.max_attempts`: Maximum queued-job retry attempts.
- `job_retry.delay_in_seconds`: Initial delay before checking the queued job.
- `job_retry.delay_backoff`: Multiplier applied to subsequent delays. | 
object({
    prefix = string
    lambda = object({
      artifact = object({
        zip = string
        s3 = object({
          bucket         = optional(string, null)
          key            = optional(string, null)
          object_version = optional(string, null)
        })
      })
      runtime      = string
      architecture = string
      vpc = object({
        subnet_ids         = list(string)
        security_group_ids = list(string)
      })
      role = object({
        path                 = string
        permissions_boundary = optional(string, null)
        principals = optional(list(object({
          type        = string
          identifiers = list(string)
        })), [])
      })
    })
    runner = object({
      os                   = string
      auto_update_disabled = bool
      ephemeral            = bool
      jit_config_enabled   = optional(bool, null)
      labels               = list(string)
      group_name           = string
      name_prefix          = string
      boot_time_in_minutes = number
      maximum_count        = number
    })
    github = object({
      organization_runners = bool
      enterprise_server = object({
        url        = optional(string, null)
        ssl_verify = bool
      })
      user_agent = optional(string, null)
      app_parameters = object({
        key_base64      = list(map(string))
        id              = list(map(string))
        installation_id = list(object({ name = string, arn = string }))
      })
    })
    queue = object({
      build = object({
        arn = string
      })
      kms_key_id = optional(string, null)
      event_source_mapping = object({
        batch_size                         = number
        maximum_batching_window_in_seconds = number
      })
    })
    ssm = object({
      token_path           = string
      token_path_arn       = string
      config_path          = string
      config_path_arn      = string
      parameter_store_tags = string
      kms_key_id           = optional(string, null)
    })
    observability = object({
      logs = object({
        level             = string
        retention_in_days = number
        kms_key_id        = optional(string, null)
        class             = string
      })
      tracing = object({
        mode                  = optional(string, null)
        capture_http_requests = bool
        capture_error         = bool
      })
      metrics = object({
        enabled   = bool
        namespace = string
        metric = object({
          github_app_rate_limit = object({
            enabled = bool
          })
        })
      })
    })
    scale_up = object({
      memory_size                    = number
      timeout                        = number
      reserved_concurrent_executions = number
      job_queued_check_enabled       = bool
      tags = object({
        resources            = map(string)
        lambda               = map(string)
        log_group            = map(string)
        event_source_mapping = map(string)
      })
    })
    scale_down = object({
      memory_size                     = number
      timeout                         = number
      schedule_expression             = string
      minimum_running_time_in_minutes = optional(number, null)
      idle_config = list(object({
        cron             = string
        timeZone         = string
        idleCount        = number
        evictionStrategy = string
      }))
      tags = object({
        resources = map(string)
        lambda    = map(string)
        log_group = map(string)
      })
    })
    job_retry = object({
      enabled          = bool
      max_attempts     = number
      delay_in_seconds = number
      delay_backoff    = number
      queue = optional(object({
        arn = string
        url = string
      }), null)
    })
  })
| n/a | yes | +| [runner\_provider](#input\_runner\_provider) | Selected compute-provider integration for the scaling control plane.

- `type`: Compute-provider discriminator supplied to both Lambdas.
- `scale_up.environment_variables`: Provider-specific scale-up environment variables.
- `scale_up.iam_policy_json`: Provider-specific IAM policy merged into the common scale-up policy.
- `scale_up.additional_iam_policy_json`: Optional additional provider policy attached separately to the scale-up role.
- `scale_up.managed_policy`: Optional provider-managed policy attachment. Object presence controls attachment creation.
- `scale_up.managed_policy.arn`: ARN of the provider-managed policy. The ARN may remain unknown until apply.
- `scale_down.environment_variables`: Provider-specific scale-down environment variables.
- `scale_down.iam_policy_json`: Provider-specific IAM policy merged into the common scale-down policy. | 
object({
    type = string
    scale_up = object({
      environment_variables      = map(string)
      iam_policy_json            = string
      additional_iam_policy_json = optional(string, null)
      managed_policy = optional(object({
        arn = string
      }), null)
    })
    scale_down = object({
      environment_variables = map(string)
      iam_policy_json       = string
    })
  })
| n/a | yes | + +## Outputs + +| Name | Description | +|------|-------------| +| [scale\_down](#output\_scale\_down) | Scale-down Lambda resources. | +| [scale\_up](#output\_scale\_up) | Scale-up Lambda resources. | + diff --git a/modules/orchestration-providers/webhook/scale-runners/common-config.tf b/modules/orchestration-providers/webhook/scale-runners/common-config.tf new file mode 100644 index 0000000000..7c8a04d095 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/common-config.tf @@ -0,0 +1,20 @@ +locals { + vpc_enabled = ( + length(var.config.lambda.vpc.subnet_ids) > 0 && + length(var.config.lambda.vpc.security_group_ids) > 0 + ) + + job_retry_config = var.config.job_retry.enabled ? { + enable = true + maxAttempts = var.config.job_retry.max_attempts + delayInSeconds = var.config.job_retry.delay_in_seconds + delayBackoff = var.config.job_retry.delay_backoff + queueUrl = var.config.job_retry.queue.url + } : {} + + min_runtime_defaults = { + windows = 15 + linux = 5 + osx = 20 + } +} diff --git a/modules/orchestration-providers/webhook/scale-runners/lambda-iam-policies.tf b/modules/orchestration-providers/webhook/scale-runners/lambda-iam-policies.tf new file mode 100644 index 0000000000..0c9214d0d3 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/lambda-iam-policies.tf @@ -0,0 +1,36 @@ +data "aws_iam_policy_document" "lambda_assume_role" { + statement { + actions = ["sts:AssumeRole"] + + principals { + type = "Service" + identifiers = ["lambda.amazonaws.com"] + } + + dynamic "principals" { + for_each = var.config.lambda.role.principals + + content { + type = principals.value.type + identifiers = principals.value.identifiers + } + } + } +} + +data "aws_iam_policy_document" "lambda_xray" { + count = var.config.observability.tracing.mode != null ? 1 : 0 + + # AWS X-Ray write/read trace APIs do not support resource-level permissions. + statement { + sid = "AllowXRay" + effect = "Allow" + actions = [ + "xray:BatchGetTraces", + "xray:GetTraceSummaries", + "xray:PutTelemetryRecords", + "xray:PutTraceSegments", + ] + resources = ["*"] + } +} diff --git a/modules/orchestration-providers/webhook/scale-runners/outputs.tf b/modules/orchestration-providers/webhook/scale-runners/outputs.tf new file mode 100644 index 0000000000..74d54d2101 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/outputs.tf @@ -0,0 +1,17 @@ +output "scale_up" { + description = "Scale-up Lambda resources." + value = { + lambda = aws_lambda_function.scale_up + log_group = aws_cloudwatch_log_group.scale_up + role = aws_iam_role.scale_up + } +} + +output "scale_down" { + description = "Scale-down Lambda resources." + value = { + lambda = aws_lambda_function.scale_down + log_group = aws_cloudwatch_log_group.scale_down + role = aws_iam_role.scale_down + } +} diff --git a/modules/orchestration-providers/webhook/scale-runners/scale-down-iam-policies.tf b/modules/orchestration-providers/webhook/scale-runners/scale-down-iam-policies.tf new file mode 100644 index 0000000000..b95cb9e686 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/scale-down-iam-policies.tf @@ -0,0 +1,46 @@ +data "aws_iam_policy_document" "scale_down_common" { + statement { + sid = "WebhookScaleDownReadGitHubAppParameters" + effect = "Allow" + actions = [ + "ssm:GetParameter", + "ssm:GetParameters", + ] + resources = concat( + [for p in var.config.github.app_parameters.id : p.arn], + [for p in var.config.github.app_parameters.key_base64 : p.arn], + [for p in var.config.github.app_parameters.installation_id : p.arn if p != null], + ) + } + + dynamic "statement" { + for_each = var.config.ssm.kms_key_id == null ? [] : [var.config.ssm.kms_key_id] + iterator = kms_key + + content { + sid = "WebhookScaleDownDecryptParameterStore" + effect = "Allow" + actions = ["kms:Decrypt"] + resources = [kms_key.value] + } + } +} + +data "aws_iam_policy_document" "scale_down" { + source_policy_documents = [ + data.aws_iam_policy_document.scale_down_common.json, + var.runner_provider.scale_down.iam_policy_json, + ] +} + +data "aws_iam_policy_document" "scale_down_logging" { + statement { + sid = "WebhookScaleDownWriteLogs" + effect = "Allow" + actions = [ + "logs:CreateLogStream", + "logs:PutLogEvents", + ] + resources = ["${aws_cloudwatch_log_group.scale_down.arn}*"] + } +} diff --git a/modules/orchestration-providers/webhook/scale-runners/scale-down.tf b/modules/orchestration-providers/webhook/scale-runners/scale-down.tf new file mode 100644 index 0000000000..44e651d78c --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/scale-down.tf @@ -0,0 +1,116 @@ +resource "aws_lambda_function" "scale_down" { + s3_bucket = var.config.lambda.artifact.s3.bucket + s3_key = var.config.lambda.artifact.s3.key + s3_object_version = var.config.lambda.artifact.s3.object_version + filename = var.config.lambda.artifact.s3.bucket == null ? var.config.lambda.artifact.zip : null + source_code_hash = var.config.lambda.artifact.s3.bucket == null ? filebase64sha256(var.config.lambda.artifact.zip) : null + function_name = "${var.config.prefix}-scale-down" + role = aws_iam_role.scale_down.arn + handler = "index.scaleDownHandler" + runtime = var.config.lambda.runtime + timeout = var.config.scale_down.timeout + tags = var.config.scale_down.tags.lambda + memory_size = var.config.scale_down.memory_size + architectures = [var.config.lambda.architecture] + + environment { + variables = merge(var.runner_provider.scale_down.environment_variables, { + ENVIRONMENT = var.config.prefix + ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = var.config.observability.metrics.enabled && var.config.observability.metrics.metric.github_app_rate_limit.enabled + GHES_URL = var.config.github.enterprise_server.url + USER_AGENT = var.config.github.user_agent + LOG_LEVEL = upper(var.config.observability.logs.level) + MINIMUM_RUNNING_TIME_IN_MINUTES = coalesce(var.config.scale_down.minimum_running_time_in_minutes, local.min_runtime_defaults[var.config.runner.os]) + NODE_TLS_REJECT_UNAUTHORIZED = var.config.github.enterprise_server.url != null && !var.config.github.enterprise_server.ssl_verify ? 0 : 1 + PARAMETER_GITHUB_APP_ID_NAME = join(":", [for p in var.config.github.app_parameters.id : p.name]) + PARAMETER_GITHUB_APP_KEY_BASE64_NAME = join(":", [for p in var.config.github.app_parameters.key_base64 : p.name]) + PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = join(":", [for p in var.config.github.app_parameters.installation_id : p != null ? p.name : ""]) + POWERTOOLS_LOGGER_LOG_EVENT = var.config.observability.logs.level == "debug" ? "true" : "false" + SCALE_DOWN_CONFIG = jsonencode(var.config.scale_down.idle_config) + POWERTOOLS_SERVICE_NAME = "${var.config.prefix}-scale-down" + POWERTOOLS_METRICS_NAMESPACE = var.config.observability.metrics.namespace + POWERTOOLS_TRACE_ENABLED = var.config.observability.tracing.mode != null + POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.config.observability.tracing.capture_http_requests + POWERTOOLS_TRACER_CAPTURE_ERROR = var.config.observability.tracing.capture_error + COMPUTE_PROVIDER_TYPE = var.runner_provider.type + RUNNER_BOOT_TIME_IN_MINUTES = var.config.runner.boot_time_in_minutes + }) + } + + dynamic "vpc_config" { + for_each = local.vpc_enabled ? [true] : [] + + content { + security_group_ids = var.config.lambda.vpc.security_group_ids + subnet_ids = var.config.lambda.vpc.subnet_ids + } + } + + dynamic "tracing_config" { + for_each = var.config.observability.tracing.mode != null ? [true] : [] + + content { + mode = var.config.observability.tracing.mode + } + } +} + +resource "aws_cloudwatch_log_group" "scale_down" { + name = "/aws/lambda/${aws_lambda_function.scale_down.function_name}" + retention_in_days = var.config.observability.logs.retention_in_days + kms_key_id = var.config.observability.logs.kms_key_id + log_group_class = var.config.observability.logs.class + tags = var.config.scale_down.tags.log_group +} + +resource "aws_cloudwatch_event_rule" "scale_down" { + name = "${var.config.prefix}-scale-down-rule" + schedule_expression = var.config.scale_down.schedule_expression + tags = var.config.scale_down.tags.resources +} + +resource "aws_cloudwatch_event_target" "scale_down" { + rule = aws_cloudwatch_event_rule.scale_down.name + arn = aws_lambda_function.scale_down.arn +} + +resource "aws_lambda_permission" "scale_down" { + statement_id = "AllowExecutionFromCloudWatch" + action = "lambda:InvokeFunction" + function_name = aws_lambda_function.scale_down.function_name + principal = "events.amazonaws.com" + source_arn = aws_cloudwatch_event_rule.scale_down.arn +} + +resource "aws_iam_role" "scale_down" { + name = "${substr("${var.config.prefix}-scale-down-lambda", 0, 54)}-${substr(md5("${var.config.prefix}-scale-down-lambda"), 0, 8)}" + assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json + path = var.config.lambda.role.path + permissions_boundary = var.config.lambda.role.permissions_boundary + tags = var.config.scale_down.tags.resources +} + +resource "aws_iam_role_policy" "scale_down" { + name = "scale-down-policy" + role = aws_iam_role.scale_down.name + policy = data.aws_iam_policy_document.scale_down.json +} + +resource "aws_iam_role_policy" "scale_down_logging" { + name = "logging-policy" + role = aws_iam_role.scale_down.name + policy = data.aws_iam_policy_document.scale_down_logging.json +} + +resource "aws_iam_role_policy_attachment" "scale_down_vpc_execution_role" { + count = local.vpc_enabled ? 1 : 0 + role = aws_iam_role.scale_down.name + policy_arn = "arn:${var.aws_partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole" +} + +resource "aws_iam_role_policy" "scale_down_xray" { + count = var.config.observability.tracing.mode != null ? 1 : 0 + name = "xray-policy" + policy = data.aws_iam_policy_document.lambda_xray[0].json + role = aws_iam_role.scale_down.name +} diff --git a/modules/orchestration-providers/webhook/scale-runners/scale-up-iam-policies.tf b/modules/orchestration-providers/webhook/scale-runners/scale-up-iam-policies.tf new file mode 100644 index 0000000000..b3c87b8ad7 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/scale-up-iam-policies.tf @@ -0,0 +1,102 @@ +data "aws_iam_policy_document" "scale_up_common" { + statement { + sid = "WebhookScaleUpWriteRuntimeParameters" + effect = "Allow" + actions = [ + "ssm:PutParameter", + "ssm:AddTagsToResource", + ] + resources = [ + var.config.ssm.token_path_arn, + "${var.config.ssm.token_path_arn}/*", + var.config.ssm.config_path_arn, + "${var.config.ssm.config_path_arn}/*", + ] + } + + statement { + sid = "WebhookScaleUpReadGitHubAppAndRunnerConfigParameters" + effect = "Allow" + actions = [ + "ssm:GetParameter", + "ssm:GetParameters", + ] + resources = concat( + [for p in var.config.github.app_parameters.id : p.arn], + [for p in var.config.github.app_parameters.key_base64 : p.arn], + [for p in var.config.github.app_parameters.installation_id : p.arn if p != null], + [ + var.config.ssm.config_path_arn, + "${var.config.ssm.config_path_arn}/*", + ], + ) + } + + statement { + sid = "WebhookScaleUpConsumeBuildQueue" + effect = "Allow" + actions = [ + "sqs:ReceiveMessage", + "sqs:GetQueueAttributes", + "sqs:DeleteMessage", + ] + resources = [var.config.queue.build.arn] + } + + dynamic "statement" { + for_each = var.config.ssm.kms_key_id == null ? [] : [var.config.ssm.kms_key_id] + iterator = kms_key + + content { + sid = "WebhookScaleUpDecryptParameterStore" + effect = "Allow" + actions = ["kms:Decrypt"] + resources = [kms_key.value] + } + } + + dynamic "statement" { + for_each = var.config.queue.kms_key_id == null ? [] : [var.config.queue.kms_key_id] + iterator = kms_key + + content { + sid = "WebhookScaleUpDecryptBuildQueue" + effect = "Allow" + actions = ["kms:Decrypt"] + resources = [kms_key.value] + } + } +} + +data "aws_iam_policy_document" "scale_up" { + source_policy_documents = [ + data.aws_iam_policy_document.scale_up_common.json, + var.runner_provider.scale_up.iam_policy_json, + ] +} + +data "aws_iam_policy_document" "scale_up_logging" { + statement { + sid = "WebhookScaleUpWriteLogs" + effect = "Allow" + actions = [ + "logs:CreateLogStream", + "logs:PutLogEvents", + ] + resources = ["${aws_cloudwatch_log_group.scale_up.arn}*"] + } +} + +data "aws_iam_policy_document" "scale_up_job_retry_publish" { + count = var.config.job_retry.enabled ? 1 : 0 + + statement { + sid = "WebhookScaleUpPublishJobRetryQueue" + effect = "Allow" + actions = [ + "sqs:SendMessage", + "sqs:GetQueueAttributes", + ] + resources = [var.config.job_retry.queue.arn] + } +} diff --git a/modules/orchestration-providers/webhook/scale-runners/scale-up.tf b/modules/orchestration-providers/webhook/scale-runners/scale-up.tf new file mode 100644 index 0000000000..2997aeac21 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/scale-up.tf @@ -0,0 +1,146 @@ +resource "aws_lambda_function" "scale_up" { + s3_bucket = var.config.lambda.artifact.s3.bucket + s3_key = var.config.lambda.artifact.s3.key + s3_object_version = var.config.lambda.artifact.s3.object_version + filename = var.config.lambda.artifact.s3.bucket == null ? var.config.lambda.artifact.zip : null + source_code_hash = var.config.lambda.artifact.s3.bucket == null ? filebase64sha256(var.config.lambda.artifact.zip) : null + function_name = "${var.config.prefix}-scale-up" + role = aws_iam_role.scale_up.arn + handler = "index.scaleUpHandler" + runtime = var.config.lambda.runtime + timeout = var.config.scale_up.timeout + reserved_concurrent_executions = var.config.scale_up.reserved_concurrent_executions + memory_size = var.config.scale_up.memory_size + tags = var.config.scale_up.tags.lambda + architectures = [var.config.lambda.architecture] + + environment { + variables = merge(var.runner_provider.scale_up.environment_variables, { + DISABLE_RUNNER_AUTOUPDATE = var.config.runner.auto_update_disabled + ENABLE_EPHEMERAL_RUNNERS = var.config.runner.ephemeral + ENABLE_JIT_CONFIG = var.config.runner.jit_config_enabled + ENABLE_JOB_QUEUED_CHECK = var.config.scale_up.job_queued_check_enabled + ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = var.config.observability.metrics.enabled && var.config.observability.metrics.metric.github_app_rate_limit.enabled + ENABLE_ORGANIZATION_RUNNERS = var.config.github.organization_runners + ENVIRONMENT = var.config.prefix + GHES_URL = var.config.github.enterprise_server.url + USER_AGENT = var.config.github.user_agent + LOG_LEVEL = upper(var.config.observability.logs.level) + MINIMUM_RUNNING_TIME_IN_MINUTES = coalesce(var.config.scale_down.minimum_running_time_in_minutes, local.min_runtime_defaults[var.config.runner.os]) + NODE_TLS_REJECT_UNAUTHORIZED = var.config.github.enterprise_server.url != null && !var.config.github.enterprise_server.ssl_verify ? 0 : 1 + PARAMETER_GITHUB_APP_ID_NAME = join(":", [for p in var.config.github.app_parameters.id : p.name]) + PARAMETER_GITHUB_APP_KEY_BASE64_NAME = join(":", [for p in var.config.github.app_parameters.key_base64 : p.name]) + PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = join(":", [for p in var.config.github.app_parameters.installation_id : p != null ? p.name : ""]) + POWERTOOLS_LOGGER_LOG_EVENT = var.config.observability.logs.level == "debug" ? "true" : "false" + POWERTOOLS_METRICS_NAMESPACE = var.config.observability.metrics.namespace + POWERTOOLS_TRACE_ENABLED = var.config.observability.tracing.mode != null + POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.config.observability.tracing.capture_http_requests + POWERTOOLS_TRACER_CAPTURE_ERROR = var.config.observability.tracing.capture_error + RUNNER_LABELS = lower(join(",", var.config.runner.labels)) + RUNNER_GROUP_NAME = var.config.runner.group_name + RUNNER_NAME_PREFIX = var.config.runner.name_prefix + COMPUTE_PROVIDER_TYPE = var.runner_provider.type + RUNNERS_MAXIMUM_COUNT = var.config.runner.maximum_count + POWERTOOLS_SERVICE_NAME = "${var.config.prefix}-scale-up" + SSM_TOKEN_PATH = var.config.ssm.token_path + SSM_CONFIG_PATH = var.config.ssm.config_path + SSM_PARAMETER_STORE_TAGS = var.config.ssm.parameter_store_tags + JOB_RETRY_CONFIG = jsonencode(local.job_retry_config) + }) + } + + dynamic "vpc_config" { + for_each = local.vpc_enabled ? [true] : [] + + content { + security_group_ids = var.config.lambda.vpc.security_group_ids + subnet_ids = var.config.lambda.vpc.subnet_ids + } + } + + dynamic "tracing_config" { + for_each = var.config.observability.tracing.mode != null ? [true] : [] + + content { + mode = var.config.observability.tracing.mode + } + } +} + +resource "aws_cloudwatch_log_group" "scale_up" { + name = "/aws/lambda/${aws_lambda_function.scale_up.function_name}" + retention_in_days = var.config.observability.logs.retention_in_days + kms_key_id = var.config.observability.logs.kms_key_id + log_group_class = var.config.observability.logs.class + tags = var.config.scale_up.tags.log_group +} + +resource "aws_lambda_event_source_mapping" "scale_up" { + event_source_arn = var.config.queue.build.arn + function_name = aws_lambda_function.scale_up.arn + function_response_types = ["ReportBatchItemFailures"] + batch_size = var.config.queue.event_source_mapping.batch_size + maximum_batching_window_in_seconds = var.config.queue.event_source_mapping.maximum_batching_window_in_seconds + tags = var.config.scale_up.tags.event_source_mapping +} + +resource "aws_lambda_permission" "scale_runners_lambda" { + statement_id = "AllowExecutionFromSQS" + action = "lambda:InvokeFunction" + function_name = aws_lambda_function.scale_up.function_name + principal = "sqs.amazonaws.com" + source_arn = var.config.queue.build.arn +} + +resource "aws_iam_role" "scale_up" { + name = "${substr("${var.config.prefix}-scale-up-lambda", 0, 54)}-${substr(md5("${var.config.prefix}-scale-up-lambda"), 0, 8)}" + assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json + path = var.config.lambda.role.path + permissions_boundary = var.config.lambda.role.permissions_boundary + tags = var.config.scale_up.tags.resources +} + +resource "aws_iam_role_policy" "scale_up" { + name = "scale-up-policy" + role = aws_iam_role.scale_up.name + policy = data.aws_iam_policy_document.scale_up.json +} + +resource "aws_iam_role_policy" "scale_up_logging" { + name = "logging-policy" + role = aws_iam_role.scale_up.name + policy = data.aws_iam_policy_document.scale_up_logging.json +} + +resource "aws_iam_role_policy" "service_linked_role" { + count = var.runner_provider.scale_up.additional_iam_policy_json != null ? 1 : 0 + name = "service_linked_role" + role = aws_iam_role.scale_up.name + policy = var.runner_provider.scale_up.additional_iam_policy_json +} + +resource "aws_iam_role_policy_attachment" "scale_up_vpc_execution_role" { + count = local.vpc_enabled ? 1 : 0 + role = aws_iam_role.scale_up.name + policy_arn = "arn:${var.aws_partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole" +} + +resource "aws_iam_role_policy_attachment" "provider" { + count = var.runner_provider.scale_up.managed_policy != null ? 1 : 0 + role = aws_iam_role.scale_up.name + policy_arn = var.runner_provider.scale_up.managed_policy.arn +} + +resource "aws_iam_role_policy" "scale_up_xray" { + count = var.config.observability.tracing.mode != null ? 1 : 0 + name = "xray-policy" + policy = data.aws_iam_policy_document.lambda_xray[0].json + role = aws_iam_role.scale_up.name +} + +resource "aws_iam_role_policy" "job_retry_sqs_publish" { + count = var.config.job_retry.enabled ? 1 : 0 + name = "publish-retry-check-sqs-policy" + role = aws_iam_role.scale_up.name + policy = data.aws_iam_policy_document.scale_up_job_retry_publish[0].json +} diff --git a/modules/orchestration-providers/webhook/scale-runners/tests/scale-runners.tftest.hcl b/modules/orchestration-providers/webhook/scale-runners/tests/scale-runners.tftest.hcl new file mode 100644 index 0000000000..acca05980b --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/tests/scale-runners.tftest.hcl @@ -0,0 +1,460 @@ +mock_provider "aws" { + mock_data "aws_iam_policy_document" { + defaults = { + json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}" + } + } + + mock_resource "aws_iam_role" { + defaults = { + arn = "arn:aws:iam::123456789012:role/scale-runners-test" + } + } +} + +variables { + aws_partition = "aws-us-gov" + + config = { + prefix = "scale-runners-test" + lambda = { + artifact = { + zip = "runners.zip" + s3 = { + bucket = "lambda-artifacts" + key = "runners.zip" + object_version = "test-version" + } + } + runtime = "nodejs24.x" + architecture = "arm64" + vpc = { + subnet_ids = ["subnet-12345678"] + security_group_ids = ["sg-12345678"] + } + role = { + path = "/scale-runners-test/" + permissions_boundary = "arn:aws-us-gov:iam::123456789012:policy/permissions-boundary" + principals = [{ + type = "AWS" + identifiers = ["arn:aws-us-gov:iam::123456789012:role/local-testing"] + }] + } + } + runner = { + os = "windows" + auto_update_disabled = true + ephemeral = true + jit_config_enabled = true + labels = ["Self-Hosted", "MicroVM"] + group_name = "test-group" + name_prefix = "test-runner-" + boot_time_in_minutes = 12 + maximum_count = 7 + } + github = { + organization_runners = true + enterprise_server = { + url = "https://github.example.com" + ssl_verify = false + } + user_agent = "scale-runners-test" + app_parameters = { + key_base64 = [ + { + name = "/github-runner/key-base64" + arn = "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/key-base64" + }, + { + name = "/github-runner/key-base64-2" + arn = "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/key-base64-2" + }, + ] + id = [ + { + name = "/github-runner/app-id" + arn = "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/app-id" + }, + { + name = "/github-runner/app-id-2" + arn = "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/app-id-2" + }, + ] + installation_id = [ + null, + { + name = "/github-runner/installation-id-2" + arn = "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/installation-id-2" + }, + ] + } + } + queue = { + build = { + arn = "arn:aws-us-gov:sqs:us-gov-west-1:123456789012:build-queue" + } + kms_key_id = "arn:aws-us-gov:kms:us-gov-west-1:123456789012:key/build-queue-test" + event_source_mapping = { + batch_size = 25 + maximum_batching_window_in_seconds = 5 + } + } + ssm = { + token_path = "/github-runner/tokens" + token_path_arn = "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/tokens" + config_path = "/github-runner/config" + config_path_arn = "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/config" + parameter_store_tags = jsonencode([{ + Key = "Environment" + Value = "test" + }]) + kms_key_id = "arn:aws-us-gov:kms:us-gov-west-1:123456789012:key/scale-runners-test" + } + observability = { + logs = { + level = "debug" + retention_in_days = 14 + kms_key_id = "arn:aws-us-gov:kms:us-gov-west-1:123456789012:key/logs" + class = "INFREQUENT_ACCESS" + } + tracing = { + mode = "Active" + capture_http_requests = true + capture_error = true + } + metrics = { + enabled = true + namespace = "ScaleRunnersTest" + metric = { + github_app_rate_limit = { + enabled = true + } + } + } + } + scale_up = { + memory_size = 768 + timeout = 90 + reserved_concurrent_executions = 2 + job_queued_check_enabled = true + tags = { + resources = { Scope = "scale-up" } + lambda = { Scope = "scale-up-lambda" } + log_group = { Scope = "scale-up-log" } + event_source_mapping = { Scope = "scale-up-queue" } + } + } + scale_down = { + memory_size = 640 + timeout = 75 + schedule_expression = "rate(10 minutes)" + minimum_running_time_in_minutes = null + idle_config = [{ + cron = "* * * * *" + timeZone = "UTC" + idleCount = 2 + evictionStrategy = "oldest_first" + }] + tags = { + resources = { Scope = "scale-down" } + lambda = { Scope = "scale-down-lambda" } + log_group = { Scope = "scale-down-log" } + } + } + job_retry = { + enabled = true + max_attempts = 4 + delay_in_seconds = 120 + delay_backoff = 3 + queue = { + arn = "arn:aws-us-gov:sqs:us-gov-west-1:123456789012:job-retry" + url = "https://sqs.us-gov-west-1.amazonaws.com/123456789012/job-retry" + } + } + } + + runner_provider = { + type = "microvm" + scale_up = { + environment_variables = { + MICROVM_CLUSTER = "runner-cluster" + } + iam_policy_json = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Effect = "Allow" + Action = ["microvm:CreateRunner"] + Resource = ["*"] + }] + }) + additional_iam_policy_json = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Effect = "Allow" + Action = ["iam:CreateServiceLinkedRole"] + Resource = ["*"] + }] + }) + managed_policy = { + arn = "arn:aws-us-gov:iam::123456789012:policy/microvm-scale-up" + } + } + scale_down = { + environment_variables = { + MICROVM_CLUSTER = "runner-cluster" + } + iam_policy_json = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Effect = "Allow" + Action = ["microvm:DeleteRunner"] + Resource = ["*"] + }] + }) + } + } +} + +run "assembles_provider_neutral_scaling_control_plane" { + command = plan + + assert { + condition = ( + length(data.aws_iam_policy_document.lambda_assume_role.statement[0].principals) == 2 && + contains(data.aws_iam_policy_document.lambda_assume_role.statement[0].principals[*].type, "AWS") + ) + error_message = "The scaling Lambda trust policy must include configured additional principals." + } + + assert { + condition = ( + toset(keys(output.scale_up)) == toset(["lambda", "log_group", "role"]) + && toset(keys(output.scale_down)) == toset(["lambda", "log_group", "role"]) + ) + error_message = "Scale runners must expose nested scale-up and scale-down Lambda resource contracts." + } + + assert { + condition = ( + aws_lambda_function.scale_up.environment[0].variables["COMPUTE_PROVIDER_TYPE"] == "microvm" + && aws_lambda_function.scale_down.environment[0].variables["COMPUTE_PROVIDER_TYPE"] == "microvm" + && aws_lambda_function.scale_up.environment[0].variables["RUNNERS_MAXIMUM_COUNT"] == "7" + && aws_lambda_function.scale_down.environment[0].variables["RUNNER_BOOT_TIME_IN_MINUTES"] == "12" + && aws_lambda_function.scale_up.environment[0].variables["MICROVM_CLUSTER"] == "runner-cluster" + && aws_lambda_function.scale_down.environment[0].variables["MICROVM_CLUSTER"] == "runner-cluster" + && !contains(keys(aws_lambda_function.scale_up.environment[0].variables), "INSTANCE_TYPES") + ) + error_message = "The common scaling Lambdas must select the compute provider while injecting webhook-owned capacity and boot-time settings." + } + + assert { + condition = ( + aws_lambda_function.scale_up.environment[0].variables["LOG_LEVEL"] == "DEBUG" + && aws_lambda_function.scale_up.environment[0].variables["RUNNER_LABELS"] == "self-hosted,microvm" + && aws_lambda_function.scale_up.environment[0].variables["MINIMUM_RUNNING_TIME_IN_MINUTES"] == "15" + && aws_lambda_function.scale_down.environment[0].variables["MINIMUM_RUNNING_TIME_IN_MINUTES"] == "15" + && aws_lambda_function.scale_up.environment[0].variables["NODE_TLS_REJECT_UNAUTHORIZED"] == "0" + && jsondecode(aws_lambda_function.scale_up.environment[0].variables["SSM_PARAMETER_STORE_TAGS"])[0].Value == "test" + ) + error_message = "Scale runners must assemble shared runner, logging, TLS, lifetime, and Parameter Store environment variables." + } + + assert { + condition = ( + aws_lambda_function.scale_up.environment[0].variables["PARAMETER_GITHUB_APP_ID_NAME"] == "/github-runner/app-id:/github-runner/app-id-2" + && aws_lambda_function.scale_down.environment[0].variables["PARAMETER_GITHUB_APP_KEY_BASE64_NAME"] == "/github-runner/key-base64:/github-runner/key-base64-2" + && aws_lambda_function.scale_up.environment[0].variables["PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME"] == ":/github-runner/installation-id-2" + && contains(data.aws_iam_policy_document.scale_up_common.statement[1].resources, "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/app-id-2") + && contains(data.aws_iam_policy_document.scale_down_common.statement[0].resources, "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/key-base64-2") + && contains(data.aws_iam_policy_document.scale_down_common.statement[0].resources, "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/installation-id-2") + ) + error_message = "Scale-up and scale-down must pass every GitHub App parameter and grant access to every corresponding SSM ARN." + } + + assert { + condition = ( + jsondecode(aws_lambda_function.scale_up.environment[0].variables["JOB_RETRY_CONFIG"]).queueUrl == "https://sqs.us-gov-west-1.amazonaws.com/123456789012/job-retry" + && jsondecode(aws_lambda_function.scale_up.environment[0].variables["JOB_RETRY_CONFIG"]).maxAttempts == "4" + && jsondecode(aws_lambda_function.scale_down.environment[0].variables["SCALE_DOWN_CONFIG"])[0].idleCount == 2 + ) + error_message = "Scale runners must preserve job-retry and idle-runner configuration at the Lambda boundary." + } + + assert { + condition = ( + aws_lambda_function.scale_up.memory_size == 768 + && aws_lambda_function.scale_up.timeout == 90 + && aws_lambda_function.scale_up.reserved_concurrent_executions == 2 + && aws_lambda_function.scale_down.memory_size == 640 + && aws_lambda_function.scale_down.timeout == 75 + && aws_cloudwatch_log_group.scale_up.log_group_class == "INFREQUENT_ACCESS" + && aws_cloudwatch_log_group.scale_down.retention_in_days == 14 + ) + error_message = "The child module must preserve Lambda sizing and log-group configuration." + } + + assert { + condition = ( + aws_lambda_event_source_mapping.scale_up.event_source_arn == "arn:aws-us-gov:sqs:us-gov-west-1:123456789012:build-queue" + && aws_lambda_event_source_mapping.scale_up.batch_size == 25 + && aws_lambda_event_source_mapping.scale_up.maximum_batching_window_in_seconds == 5 + && aws_lambda_event_source_mapping.scale_up.tags["Scope"] == "scale-up-queue" + && aws_cloudwatch_event_rule.scale_down.schedule_expression == "rate(10 minutes)" + && aws_cloudwatch_event_rule.scale_down.tags["Scope"] == "scale-down" + ) + error_message = "Scale-up queue and scale-down schedule triggers must remain owned by the child module." + } + + assert { + condition = ( + aws_lambda_function.scale_up.tags["Scope"] == "scale-up-lambda" + && aws_cloudwatch_log_group.scale_up.tags["Scope"] == "scale-up-log" + && aws_iam_role.scale_up.tags["Scope"] == "scale-up" + && aws_lambda_function.scale_down.tags["Scope"] == "scale-down-lambda" + && aws_cloudwatch_log_group.scale_down.tags["Scope"] == "scale-down-log" + && aws_iam_role.scale_down.tags["Scope"] == "scale-down" + ) + error_message = "Resolved component tag maps must reach the resources owned by scale runners." + } + + assert { + condition = ( + length(aws_lambda_function.scale_up.vpc_config) == 1 + && length(aws_lambda_function.scale_down.vpc_config) == 1 + && length(aws_iam_role_policy_attachment.scale_up_vpc_execution_role) == 1 + && length(aws_iam_role_policy_attachment.scale_down_vpc_execution_role) == 1 + && aws_iam_role_policy_attachment.scale_up_vpc_execution_role[0].policy_arn == "arn:aws-us-gov:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole" + ) + error_message = "A complete Lambda VPC configuration must configure both Lambdas and their partition-aware execution policies." + } + + assert { + condition = ( + length(aws_lambda_function.scale_up.tracing_config) == 1 + && length(aws_lambda_function.scale_down.tracing_config) == 1 + && length(aws_iam_role_policy.scale_up_xray) == 1 + && length(aws_iam_role_policy.scale_down_xray) == 1 + ) + error_message = "Active tracing must configure both Lambdas and attach their X-Ray policies." + } + + assert { + condition = ( + length(aws_iam_role_policy.service_linked_role) == 1 + && length(aws_iam_role_policy_attachment.provider) == 1 + && aws_iam_role_policy_attachment.provider[0].policy_arn == "arn:aws-us-gov:iam::123456789012:policy/microvm-scale-up" + && length(aws_iam_role_policy.job_retry_sqs_publish) == 1 + ) + error_message = "Optional compute-provider and job-retry IAM integrations must be attached to the scale-up role." + } + + assert { + condition = ( + length(data.aws_iam_policy_document.scale_up.source_policy_documents) == 2 + && length(data.aws_iam_policy_document.scale_down.source_policy_documents) == 2 + && length(data.aws_iam_policy_document.scale_up_common.statement) == 5 + && length(data.aws_iam_policy_document.scale_down_common.statement) == 2 + && one([ + for statement in data.aws_iam_policy_document.scale_up_common.statement : statement + if statement.sid == "WebhookScaleUpDecryptParameterStore" + ]).resources == toset(["arn:aws-us-gov:kms:us-gov-west-1:123456789012:key/scale-runners-test"]) + && one([ + for statement in data.aws_iam_policy_document.scale_up_common.statement : statement + if statement.sid == "WebhookScaleUpDecryptBuildQueue" + ]).resources == toset(["arn:aws-us-gov:kms:us-gov-west-1:123456789012:key/build-queue-test"]) + && one([ + for statement in data.aws_iam_policy_document.scale_up_common.statement : statement + if statement.sid == "WebhookScaleUpDecryptBuildQueue" + ]).actions == toset(["kms:Decrypt"]) + && one([ + for statement in data.aws_iam_policy_document.scale_down_common.statement : statement + if statement.sid == "WebhookScaleDownDecryptParameterStore" + ]).resources == toset(["arn:aws-us-gov:kms:us-gov-west-1:123456789012:key/scale-runners-test"]) + && length(data.aws_iam_policy_document.scale_up_job_retry_publish) == 1 + ) + error_message = "Common, provider, distinct Parameter Store/build-queue KMS, and retry IAM fragments must retain their conditional plan shape." + } + + assert { + condition = ( + one([ + for statement in data.aws_iam_policy_document.scale_up_common.statement : statement + if statement.sid == "WebhookScaleUpWriteRuntimeParameters" + ]).resources == toset([ + "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/tokens", + "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/tokens/*", + "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/config", + "arn:aws-us-gov:ssm:us-gov-west-1:123456789012:parameter/github-runner/config/*", + ]) + && !contains(one([ + for statement in data.aws_iam_policy_document.scale_up_common.statement : statement + if statement.sid == "WebhookScaleUpWriteRuntimeParameters" + ]).resources, "*") + ) + error_message = "Scale-up must scope runtime SSM writes to the token and runner-config parameter paths." + } + + assert { + condition = ( + data.aws_iam_policy_document.lambda_xray[0].statement[0].sid == "AllowXRay" + && data.aws_iam_policy_document.lambda_xray[0].statement[0].resources == toset(["*"]) + && toset(data.aws_iam_policy_document.lambda_xray[0].statement[0].actions) == toset([ + "xray:BatchGetTraces", + "xray:GetTraceSummaries", + "xray:PutTelemetryRecords", + "xray:PutTraceSegments", + ]) + ) + error_message = "Only the resource-agnostic X-Ray APIs may retain a wildcard resource in the scale-runner common policies." + } +} + +run "omits_optional_kms_statements" { + command = plan + + variables { + config = merge(var.config, { + queue = merge(var.config.queue, { + kms_key_id = null + }) + ssm = merge(var.config.ssm, { + kms_key_id = null + }) + }) + } + + assert { + condition = ( + length(data.aws_iam_policy_document.scale_up_common.statement) == 3 + && length(data.aws_iam_policy_document.scale_down_common.statement) == 1 + && length([ + for statement in data.aws_iam_policy_document.scale_up_common.statement : statement + if anytrue([for action in statement.actions : startswith(action, "kms:")]) + ]) == 0 + && length([ + for statement in data.aws_iam_policy_document.scale_down_common.statement : statement + if anytrue([for action in statement.actions : startswith(action, "kms:")]) + ]) == 0 + ) + error_message = "Null Parameter Store and build-queue keys must omit every optional scale-runner KMS statement." + } +} + +run "requires_job_retry_queue_when_enabled" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + config = merge(var.config, { + job_retry = merge(var.config.job_retry, { + enabled = true + queue = null + }) + }) + } + + expect_failures = [terraform_data.validate_config] +} diff --git a/modules/orchestration-providers/webhook/scale-runners/validations.tf b/modules/orchestration-providers/webhook/scale-runners/validations.tf new file mode 100644 index 0000000000..50cdc1a136 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/validations.tf @@ -0,0 +1,8 @@ +resource "terraform_data" "validate_config" { + lifecycle { + precondition { + condition = !var.config.job_retry.enabled || var.config.job_retry.queue != null + error_message = "config.job_retry.queue must be set when config.job_retry.enabled is true." + } + } +} diff --git a/modules/orchestration-providers/webhook/scale-runners/variables.tf b/modules/orchestration-providers/webhook/scale-runners/variables.tf new file mode 100644 index 0000000000..e191e303d1 --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/variables.tf @@ -0,0 +1,233 @@ +variable "aws_partition" { + description = "AWS partition used to construct IAM policy ARNs." + type = string + default = "aws" +} + +variable "config" { + description = <<-EOT + Provider-neutral scale-up and scale-down configuration assembled by runner-config. + + - `prefix`: Prefix used to name scaling resources. + - `lambda.artifact.zip`: Resolved local control-plane archive. + - `lambda.artifact.s3.bucket`: Optional S3 bucket containing the Lambda archive. + - `lambda.artifact.s3.key`: Object key of the Lambda archive. + - `lambda.artifact.s3.object_version`: Optional object version of the Lambda archive. + - `lambda.runtime`: Runtime used by both scaling Lambdas. + - `lambda.architecture`: Instruction-set architecture used by both scaling Lambdas. + - `lambda.vpc.subnet_ids`: Subnets used for Lambda VPC configuration. + - `lambda.vpc.security_group_ids`: Security groups used for Lambda VPC configuration. + - `lambda.role.path`: IAM path used for the scaling Lambda roles. + - `lambda.role.permissions_boundary`: Optional permissions boundary for the scaling Lambda roles. + - `lambda.role.principals`: Additional principals allowed to assume the scaling Lambda roles. + - `runner.os`: Runner operating system used for the minimum-runtime default. + - `runner.auto_update_disabled`: Disables the GitHub runner application's built-in updater. + - `runner.ephemeral`: Registers runners in ephemeral mode. + - `runner.jit_config_enabled`: Enables or disables just-in-time runner configuration. + - `runner.labels`: Labels supplied when a runner is registered. + - `runner.group_name`: GitHub runner group used during registration. + - `runner.name_prefix`: Prefix added to registered runner names. + - `runner.boot_time_in_minutes`: Webhook-provider runner boot timeout used by scale-down. + - `runner.maximum_count`: Webhook-provider runner capacity limit for this runner configuration. + - `github.organization_runners`: Registers organization runners when true. + - `github.enterprise_server.url`: Optional GitHub Enterprise Server URL. + - `github.enterprise_server.ssl_verify`: Enables TLS verification for GitHub Enterprise Server. + - `github.user_agent`: Optional User-Agent sent to GitHub. + - `github.app_parameters.key_base64`: Ordered Parameter Store references for GitHub App private keys. + - `github.app_parameters.id`: Ordered Parameter Store references for GitHub App IDs. + - `github.app_parameters.installation_id`: Ordered optional Parameter Store references for GitHub App installation IDs. + - `queue.build.arn`: ARN of the build queue consumed by scale-up. + - `queue.kms_key_id`: Optional KMS key ARN used to encrypt the build queue. This is distinct from the Parameter Store key. + - `queue.event_source_mapping.batch_size`: Maximum records delivered per scale-up invocation. + - `queue.event_source_mapping.maximum_batching_window_in_seconds`: Maximum event batching window. + - `ssm.token_path`: Parameter Store path used for registration tokens. + - `ssm.token_path_arn`: ARN of the Parameter Store path used for registration tokens. + - `ssm.config_path`: Parameter Store path used for persistent runner configuration. + - `ssm.config_path_arn`: ARN of the persistent runner configuration path. + - `ssm.kms_key_id`: Optional KMS key ARN used to decrypt shared parameters. Its value may be unknown until apply. + - `ssm.parameter_store_tags`: JSON-encoded tags applied to parameters created at runtime. + - `observability.logs`: Shared logging level, retention, encryption, and log-class configuration. + - `observability.tracing`: Lambda X-Ray and tracing-helper configuration. + - `observability.metrics`: Metrics enablement, namespace, and GitHub rate-limit metric configuration. + - `scale_up`: Scale-up Lambda sizing, concurrency, queued-job behavior, and resolved resource tag maps. + - `scale_up.tags.resources`: Tags for the scale-up IAM role and other component resources. + - `scale_up.tags.lambda`: Tags for the scale-up Lambda function. + - `scale_up.tags.log_group`: Tags for the scale-up log group. + - `scale_up.tags.event_source_mapping`: Tags for the build-queue event-source mapping. + - `scale_down`: Scale-down Lambda sizing, schedule, idle configuration, minimum runtime, and resolved resource tag maps. + - `scale_down.tags.resources`: Tags for the scale-down IAM role and EventBridge rule. + - `scale_down.tags.lambda`: Tags for the scale-down Lambda function. + - `scale_down.tags.log_group`: Tags for the scale-down log group. + - `job_retry.enabled`: Enables publishing retry checks from scale-up. + - `job_retry.queue`: Retry queue ARN and URL. Required when job retry is enabled. + - `job_retry.max_attempts`: Maximum queued-job retry attempts. + - `job_retry.delay_in_seconds`: Initial delay before checking the queued job. + - `job_retry.delay_backoff`: Multiplier applied to subsequent delays. + EOT + + type = object({ + prefix = string + lambda = object({ + artifact = object({ + zip = string + s3 = object({ + bucket = optional(string, null) + key = optional(string, null) + object_version = optional(string, null) + }) + }) + runtime = string + architecture = string + vpc = object({ + subnet_ids = list(string) + security_group_ids = list(string) + }) + role = object({ + path = string + permissions_boundary = optional(string, null) + principals = optional(list(object({ + type = string + identifiers = list(string) + })), []) + }) + }) + runner = object({ + os = string + auto_update_disabled = bool + ephemeral = bool + jit_config_enabled = optional(bool, null) + labels = list(string) + group_name = string + name_prefix = string + boot_time_in_minutes = number + maximum_count = number + }) + github = object({ + organization_runners = bool + enterprise_server = object({ + url = optional(string, null) + ssl_verify = bool + }) + user_agent = optional(string, null) + app_parameters = object({ + key_base64 = list(map(string)) + id = list(map(string)) + installation_id = list(object({ name = string, arn = string })) + }) + }) + queue = object({ + build = object({ + arn = string + }) + kms_key_id = optional(string, null) + event_source_mapping = object({ + batch_size = number + maximum_batching_window_in_seconds = number + }) + }) + ssm = object({ + token_path = string + token_path_arn = string + config_path = string + config_path_arn = string + parameter_store_tags = string + kms_key_id = optional(string, null) + }) + observability = object({ + logs = object({ + level = string + retention_in_days = number + kms_key_id = optional(string, null) + class = string + }) + tracing = object({ + mode = optional(string, null) + capture_http_requests = bool + capture_error = bool + }) + metrics = object({ + enabled = bool + namespace = string + metric = object({ + github_app_rate_limit = object({ + enabled = bool + }) + }) + }) + }) + scale_up = object({ + memory_size = number + timeout = number + reserved_concurrent_executions = number + job_queued_check_enabled = bool + tags = object({ + resources = map(string) + lambda = map(string) + log_group = map(string) + event_source_mapping = map(string) + }) + }) + scale_down = object({ + memory_size = number + timeout = number + schedule_expression = string + minimum_running_time_in_minutes = optional(number, null) + idle_config = list(object({ + cron = string + timeZone = string + idleCount = number + evictionStrategy = string + })) + tags = object({ + resources = map(string) + lambda = map(string) + log_group = map(string) + }) + }) + job_retry = object({ + enabled = bool + max_attempts = number + delay_in_seconds = number + delay_backoff = number + queue = optional(object({ + arn = string + url = string + }), null) + }) + }) + + nullable = false +} + +variable "runner_provider" { + description = <<-EOT + Selected compute-provider integration for the scaling control plane. + + - `type`: Compute-provider discriminator supplied to both Lambdas. + - `scale_up.environment_variables`: Provider-specific scale-up environment variables. + - `scale_up.iam_policy_json`: Provider-specific IAM policy merged into the common scale-up policy. + - `scale_up.additional_iam_policy_json`: Optional additional provider policy attached separately to the scale-up role. + - `scale_up.managed_policy`: Optional provider-managed policy attachment. Object presence controls attachment creation. + - `scale_up.managed_policy.arn`: ARN of the provider-managed policy. The ARN may remain unknown until apply. + - `scale_down.environment_variables`: Provider-specific scale-down environment variables. + - `scale_down.iam_policy_json`: Provider-specific IAM policy merged into the common scale-down policy. + EOT + + type = object({ + type = string + scale_up = object({ + environment_variables = map(string) + iam_policy_json = string + additional_iam_policy_json = optional(string, null) + managed_policy = optional(object({ + arn = string + }), null) + }) + scale_down = object({ + environment_variables = map(string) + iam_policy_json = string + }) + }) + + nullable = false +} diff --git a/modules/orchestration-providers/webhook/scale-runners/versions.tf b/modules/orchestration-providers/webhook/scale-runners/versions.tf new file mode 100644 index 0000000000..3ef011ea0a --- /dev/null +++ b/modules/orchestration-providers/webhook/scale-runners/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.4.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 6.33" + } + } +} diff --git a/modules/orchestration-providers/webhook/tests/webhook.tftest.hcl b/modules/orchestration-providers/webhook/tests/webhook.tftest.hcl new file mode 100644 index 0000000000..ac92fb907b --- /dev/null +++ b/modules/orchestration-providers/webhook/tests/webhook.tftest.hcl @@ -0,0 +1,315 @@ +mock_provider "aws" { + mock_data "aws_iam_policy_document" { + defaults = { + json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}" + } + } + + mock_resource "aws_iam_role" { + defaults = { + arn = "arn:aws:iam::123456789012:role/webhook-orchestration-test" + } + } +} + +variables { + prefix = "webhook-test" + + tags = { + Scope = "common" + Precedence = "common" + } + + runner = { + os = "linux" + auto_update_disabled = false + labels = ["self-hosted", "linux"] + group_name = "default" + name_prefix = "webhook-test-" + } + + github = { + app_parameters = { + key_base64 = [{ + name = "/github-runner/key-base64" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/key-base64" + }] + id = [{ + name = "/github-runner/app-id" + arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id" + }] + installation_id = [null] + } + enterprise_server = { + url = null + ssl_verify = true + } + user_agent = "webhook-orchestration-test" + } + + lambda = { + artifact = { + s3 = { + bucket = "lambda-artifacts" + } + } + runtime = "nodejs24.x" + architecture = "arm64" + subnet_ids = [] + security_group_ids = [] + tags = { + Lambda = "yes" + Precedence = "lambda" + } + role = { + path = "/webhook-test/" + } + } + + ssm = { + token_path = "/github-runner/tokens" + token_path_arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/tokens" + config_path = "/github-runner/config" + config_path_arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/config" + kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/webhook-test" + parameter_store_tags = "[]" + } + + observability = { + logs = { + level = "info" + retention_in_days = 14 + kms_key_id = null + class = "STANDARD" + } + tracing = { + mode = null + capture_http_requests = false + capture_error = false + } + metrics = { + enabled = true + namespace = "WebhookTest" + metric = { + github_app_rate_limit = { + enabled = true + } + job_retry = { + enabled = true + } + } + } + } + + config = { + runner = { + boot_time_in_minutes = 11 + ephemeral = true + jit_config_enabled = null + maximum_count = 10 + } + github = { + organization_runners = true + } + queue = { + build = { + arn = "arn:aws:sqs:eu-west-1:123456789012:build-queue" + url = "https://sqs.eu-west-1.amazonaws.com/123456789012/build-queue" + } + kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/build-queue-test" + tags = { + Queue = "yes" + } + } + lambda = { + artifact = { + s3 = { + key = "runners.zip" + } + } + scale = { + up = { + memory_size = 512 + timeout = 60 + reserved_concurrent_executions = 1 + job_queued_check_enabled = null + event_source_mapping = { + batch_size = 10 + maximum_batching_window_in_seconds = 0 + } + tags = { + ScaleUp = "yes" + Precedence = "scale-up" + } + } + down = { + memory_size = 512 + timeout = 60 + schedule_expression = "cron(*/5 * * * ? *)" + minimum_running_time_in_minutes = null + idle_config = [] + tags = { + ScaleDown = "yes" + } + } + } + pool = { + memory_size = 512 + timeout = 60 + reserved_concurrent_executions = 1 + config = [{ + schedule_expression = "cron(0 8 * * ? *)" + schedule_expression_timezone = "UTC" + size = 1 + }] + include_busy_runners = false + runner_owner = "example" + tags = { + Pool = "yes" + } + } + } + job_retry = { + enabled = true + delay_in_seconds = 300 + delay_backoff = 2 + max_attempts = 2 + tags = { + JobRetry = "yes" + } + lambda = { + memory_size = 256 + reserved_concurrent_executions = 1 + timeout = 30 + } + } + } + + runner_provider = { + type = "test-provider" + scale_up = { + environment_variables = { + TEST_SCALE_UP = "yes" + } + iam_policy_json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}" + additional_iam_policy_json = null + managed_policy = null + } + scale_down = { + environment_variables = { + TEST_SCALE_DOWN = "yes" + } + iam_policy_json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}" + } + pool = { + environment_variables = { + TEST_POOL = "yes" + } + iam_policy_json = "{\"Version\":\"2012-10-17\",\"Statement\":[]}" + managed_policy_enabled = false + managed_policy_arn = null + } + } +} + +run "owns_webhook_control_plane" { + command = plan + + assert { + condition = ( + toset(keys(output.scale_up)) == toset(["lambda", "log_group", "role"]) + && toset(keys(output.scale_down)) == toset(["lambda", "log_group", "role"]) + ) + error_message = "The webhook provider must own and expose both scaling functions." + } + + assert { + condition = ( + output.pool != null + && output.job_retry != null + && output.job_retry.lambda != null + && output.job_retry.queue != null + ) + error_message = "The webhook provider must own the optional pool and job-retry resources when enabled." + } + + assert { + condition = ( + output.scale_up.lambda.environment[0].variables["COMPUTE_PROVIDER_TYPE"] == "test-provider" + && output.scale_up.lambda.environment[0].variables["TEST_SCALE_UP"] == "yes" + && output.scale_down.lambda.environment[0].variables["TEST_SCALE_DOWN"] == "yes" + && output.pool.lambda.environment[0].variables["TEST_POOL"] == "yes" + ) + error_message = "The webhook provider must forward each compute-provider capability to the matching leaf." + } + + assert { + condition = ( + output.scale_up.lambda.environment[0].variables["RUNNERS_MAXIMUM_COUNT"] == "10" + && output.pool.lambda.environment[0].variables["RUNNERS_MAXIMUM_COUNT"] == "10" + && output.scale_down.lambda.environment[0].variables["RUNNER_BOOT_TIME_IN_MINUTES"] == "11" + && output.pool.lambda.environment[0].variables["RUNNER_BOOT_TIME_IN_MINUTES"] == "11" + && output.scale_up.lambda.environment[0].variables["ENABLE_JIT_CONFIG"] == "true" + && output.pool.lambda.environment[0].variables["ENABLE_JIT_CONFIG"] == "true" + ) + error_message = "The webhook provider must route its provider-owned runner lifecycle, capacity, and boot-time values without reading them from common runner values." + } + + assert { + condition = ( + output.runner_lifecycle.ephemeral + && output.runner_lifecycle.jit_config_enabled + ) + error_message = "The webhook provider must expose its resolved lifecycle contract and default JIT configuration to the effective ephemeral mode." + } + + assert { + condition = ( + output.scale_up.lambda.s3_bucket == "lambda-artifacts" + && output.scale_up.lambda.s3_key == "runners.zip" + && output.scale_down.lambda.s3_bucket == "lambda-artifacts" + && output.pool.lambda.s3_key == "runners.zip" + ) + error_message = "The webhook provider must combine the common artifact bucket with its shared runner-control artifact key for scale, pool, and job retry." + } + + assert { + condition = ( + output.scale_up.lambda.tags["Scope"] == "common" + && output.scale_up.lambda.tags["Lambda"] == "yes" + && output.scale_up.lambda.tags["ScaleUp"] == "yes" + && output.scale_up.lambda.tags["Precedence"] == "scale-up" + && output.job_retry.queue.tags["JobRetry"] == "yes" + && output.job_retry.queue.tags["Queue"] == "yes" + ) + error_message = "Provider-owned normalization must preserve common, substrate, and webhook component tag precedence." + } + + assert { + condition = ( + length(module.pool) == 1 + && length(module.job_retry) == 1 + ) + error_message = "Pool and job-retry leaf ownership must remain inside the webhook provider." + } +} + +run "rejects_conflicting_artifact_sources" { + command = plan + + plan_options { + target = [terraform_data.validate_config] + } + + variables { + config = merge(var.config, { + lambda = merge(var.config.lambda, { + artifact = merge(var.config.lambda.artifact, { + zip = "runners.zip" + }) + }) + }) + } + + expect_failures = [terraform_data.validate_config] +} diff --git a/modules/orchestration-providers/webhook/validations.tf b/modules/orchestration-providers/webhook/validations.tf new file mode 100644 index 0000000000..0f4d5213f9 --- /dev/null +++ b/modules/orchestration-providers/webhook/validations.tf @@ -0,0 +1,11 @@ +resource "terraform_data" "validate_config" { + lifecycle { + precondition { + condition = !( + var.config.lambda.artifact.zip != null && + var.config.lambda.artifact.s3 != null + ) + error_message = "config.lambda.artifact must select at most one of zip or s3." + } + } +} diff --git a/modules/orchestration-providers/webhook/variables.tf b/modules/orchestration-providers/webhook/variables.tf new file mode 100644 index 0000000000..5dfecdbd6c --- /dev/null +++ b/modules/orchestration-providers/webhook/variables.tf @@ -0,0 +1,272 @@ +variable "aws_partition" { + description = "AWS partition used to construct ARNs." + type = string + default = "aws" +} + +variable "prefix" { + description = "Prefix used to identify resources created for this webhook orchestration provider." + type = string +} + +variable "tags" { + description = "Base tags available to webhook-provider resources. Component-specific tags override this map within their documented scopes." + type = map(string) + default = {} +} + +variable "config" { + description = <<-EOT + Provider-owned webhook values supplied from `orchestration_provider.webhook`. The parent resolves inherited input values before calling this module; this provider still resolves the documented JIT, artifact, and tag-precedence fallbacks. + + - `runner`: Runner lifecycle, boot timeout, and capacity settings owned by webhook orchestration. + - `runner.boot_time_in_minutes`: Expected runner boot duration used by scale-down and pool controls. + - `runner.ephemeral`: Registers runners in ephemeral mode. + - `runner.jit_config_enabled`: Explicitly enables or disables just-in-time configuration. Null follows `runner.ephemeral`. + - `runner.maximum_count`: Maximum number of runners managed for this runner configuration. + - `github.organization_runners`: Registers runners at organization scope when true; otherwise registration is repository-scoped. + - `queue.build.arn`: ARN of the runner configuration's build queue. + - `queue.build.url`: URL of the runner configuration's build queue. + - `queue.kms_key_id`: Optional KMS key ARN encrypting the build queue. This is independent from the Parameter Store KMS key. + - `queue.tags`: Tags inherited by queue-related provider resources before component-specific overrides. + - `lambda.artifact`: Runner-control artifact shared by scale, pool, and job-retry components. At most one of `zip` or `s3` may be selected; no selection uses the packaged runner archive. + - `lambda.artifact.zip`: Optional local path to the runner-control Lambda archive. + - `lambda.artifact.s3`: Optional S3 object selector in the common `lambda.artifact.s3.bucket`. Wrapper presence must be known during planning and selecting it requires a non-null common bucket. + - `lambda.artifact.s3.key`: Object key of the runner-control Lambda archive. + - `lambda.artifact.s3.object_version`: Optional object version of the runner-control Lambda archive. + - `lambda.scale.up.memory_size`: Memory allocated to the scale-up Lambda in MB. + - `lambda.scale.up.timeout`: Scale-up Lambda timeout in seconds. + - `lambda.scale.up.reserved_concurrent_executions`: Reserved concurrency for scale-up. Use `-1` for unreserved concurrency. + - `lambda.scale.up.job_queued_check_enabled`: Enables queued-job verification before scaling. Null follows the resolved runner mode. + - `lambda.scale.up.event_source_mapping.batch_size`: Maximum build-queue records delivered per scale-up invocation. + - `lambda.scale.up.event_source_mapping.maximum_batching_window_in_seconds`: Maximum batching window for build-queue records. + - `lambda.scale.up.tags`: Tags applied within scale-up resource scopes after common provider tags. + - `lambda.scale.down.memory_size`: Memory allocated to the scale-down Lambda in MB. + - `lambda.scale.down.timeout`: Scale-down Lambda timeout in seconds. + - `lambda.scale.down.schedule_expression`: EventBridge schedule expression that invokes scale-down. + - `lambda.scale.down.minimum_running_time_in_minutes`: Optional minimum runner age before scale-down may terminate it. Null selects the operating-system default. + - `lambda.scale.down.idle_config`: Time-based desired idle-runner configurations. + - `lambda.scale.down.idle_config[].cron`: Cron expression identifying when the idle configuration applies. + - `lambda.scale.down.idle_config[].timeZone`: IANA time zone used to evaluate the cron expression. + - `lambda.scale.down.idle_config[].idleCount`: Number of idle runners retained during the matching period. + - `lambda.scale.down.idle_config[].evictionStrategy`: Selection strategy used when excess idle runners are removed. + - `lambda.scale.down.tags`: Tags applied within scale-down resource scopes after common provider tags. + - `lambda.pool.memory_size`: Memory allocated to the pool Lambda in MB. + - `lambda.pool.timeout`: Pool Lambda timeout in seconds. + - `lambda.pool.reserved_concurrent_executions`: Reserved concurrency for the pool Lambda. Use `-1` for unreserved concurrency. + - `lambda.pool.config`: Scheduled target pool sizes. An empty list disables the pool component. + - `lambda.pool.config[].schedule_expression`: Scheduler expression that activates the target size. + - `lambda.pool.config[].schedule_expression_timezone`: Optional IANA time zone used to evaluate the schedule. + - `lambda.pool.config[].size`: Desired number of runners for the schedule. + - `lambda.pool.include_busy_runners`: Includes busy runners when reconciling scheduled pool capacity. + - `lambda.pool.runner_owner`: Optional GitHub organization or repository owner used for pooled runners. + - `lambda.pool.tags`: Tags applied within pool resource scopes after common provider tags. + - `job_retry.enabled`: Creates the retry queue, Lambda function, event-source mapping, and related IAM resources. + - `job_retry.delay_in_seconds`: Initial delay before a queued-job retry check. + - `job_retry.delay_backoff`: Multiplier applied to the delay after each unsuccessful check. + - `job_retry.max_attempts`: Maximum retry-check attempts before the message is no longer republished. + - `job_retry.tags`: Tags applied within job-retry resource scopes after common provider tags. + - `job_retry.lambda.memory_size`: Memory allocated to the job-retry Lambda in MB. + - `job_retry.lambda.reserved_concurrent_executions`: Reserved concurrency for job retry. Use `-1` for unreserved concurrency. + - `job_retry.lambda.timeout`: Job-retry Lambda timeout in seconds and visibility timeout for its retry queue. + EOT + type = object({ + runner = object({ + boot_time_in_minutes = number + ephemeral = bool + jit_config_enabled = optional(bool, null) + maximum_count = number + }) + github = object({ + organization_runners = bool + }) + queue = object({ + build = object({ + arn = string + url = string + }) + kms_key_id = optional(string, null) + tags = optional(map(string), {}) + }) + lambda = object({ + artifact = object({ + zip = optional(string, null) + s3 = optional(object({ + key = string + object_version = optional(string, null) + }), null) + }) + scale = object({ + up = object({ + memory_size = number + timeout = number + reserved_concurrent_executions = number + job_queued_check_enabled = optional(bool, null) + event_source_mapping = object({ + batch_size = number + maximum_batching_window_in_seconds = number + }) + tags = optional(map(string), {}) + }) + down = object({ + memory_size = number + timeout = number + schedule_expression = string + minimum_running_time_in_minutes = optional(number, null) + idle_config = list(object({ + cron = string + timeZone = string + idleCount = number + evictionStrategy = string + })) + tags = optional(map(string), {}) + }) + }) + pool = object({ + memory_size = number + timeout = number + reserved_concurrent_executions = number + config = list(object({ + schedule_expression = string + schedule_expression_timezone = optional(string) + size = number + })) + include_busy_runners = bool + runner_owner = optional(string, null) + tags = optional(map(string), {}) + }) + }) + job_retry = object({ + enabled = bool + delay_in_seconds = number + delay_backoff = number + max_attempts = number + tags = optional(map(string), {}) + lambda = object({ + memory_size = number + reserved_concurrent_executions = number + timeout = number + }) + }) + }) + nullable = false +} + +variable "runner" { + description = "Common runner registration values consumed by webhook demand controls. Lifecycle, boot timeout, and capacity remain provider-owned under config.runner." + type = object({ + os = string + auto_update_disabled = bool + labels = list(string) + group_name = string + name_prefix = string + }) +} + +variable "github" { + description = "Common GitHub API client and GitHub App Parameter Store references." + type = object({ + app_parameters = object({ + key_base64 = list(map(string)) + id = list(map(string)) + installation_id = list(object({ name = string, arn = string })) + }) + enterprise_server = object({ + url = optional(string, null) + ssl_verify = bool + }) + user_agent = optional(string, null) + }) +} + +variable "lambda" { + description = "Common Lambda substrate. Only the shared artifact bucket crosses this boundary; the webhook provider owns its archive key, version, and local zip selection." + type = object({ + artifact = object({ + s3 = object({ + bucket = optional(string, null) + }) + }) + runtime = string + architecture = string + subnet_ids = list(string) + security_group_ids = list(string) + tags = optional(map(string), {}) + role = object({ + path = string + permissions_boundary = optional(string, null) + principals = optional(list(object({ + type = string + identifiers = list(string) + })), []) + }) + }) +} + +variable "ssm" { + description = "Resolved Parameter Store paths, optional decrypt key, and runtime parameter tags." + type = object({ + token_path = string + token_path_arn = string + config_path = string + config_path_arn = string + kms_key_id = optional(string, null) + parameter_store_tags = string + }) +} + +variable "observability" { + description = "Common logging, tracing, and metrics configuration consumed by webhook controls." + type = object({ + logs = object({ + level = string + retention_in_days = number + kms_key_id = optional(string, null) + class = string + tags = optional(map(string), {}) + }) + tracing = object({ + mode = optional(string, null) + capture_http_requests = bool + capture_error = bool + }) + metrics = object({ + enabled = bool + namespace = string + metric = object({ + github_app_rate_limit = object({ + enabled = bool + }) + job_retry = object({ + enabled = bool + }) + }) + }) + }) +} + +variable "runner_provider" { + description = "Selected compute-provider capabilities consumed by webhook scale-up, scale-down, and pool controls." + type = object({ + type = string + scale_up = object({ + environment_variables = map(string) + iam_policy_json = string + additional_iam_policy_json = optional(string, null) + managed_policy = optional(object({ + arn = string + }), null) + }) + scale_down = object({ + environment_variables = map(string) + iam_policy_json = string + }) + pool = object({ + environment_variables = map(string) + iam_policy_json = string + managed_policy_enabled = bool + managed_policy_arn = optional(string, null) + }) + }) + nullable = false +} diff --git a/modules/orchestration-providers/webhook/versions.tf b/modules/orchestration-providers/webhook/versions.tf new file mode 100644 index 0000000000..3ef011ea0a --- /dev/null +++ b/modules/orchestration-providers/webhook/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.4.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = ">= 6.33" + } + } +}