bug: CloudFormation custom resources: ServiceTimeout ignored, Update requests lack OldResourceProperties/PhysicalResourceId, and a Lambda provider running over 60 s is re-invoked every minute

[k-rooot]

Current behavior

Four differences from AWS show up with Lambda-backed custom resources (Custom::* / AWS::CloudFormation::CustomResource):

	What LocalStack does	Observed
A	ServiceTimeout is ignored. A custom resource whose provider never replies stays CREATE_IN_PROGRESS.	With ServiceTimeout: 30, the stack was still CREATE_IN_PROGRESS after 230 s. An earlier run with ServiceTimeout: 60 was still CREATE_IN_PROGRESS after 300 s. A provider that raises an exception without replying behaves the same way.
B	Update requests carry no OldResourceProperties and no PhysicalResourceId.	"RequestType": "Update", "PhysicalResourceId": null, "OldResourceProperties": null
B2	The Delete request's ResourceProperties also contain the keys from the provider's earlier response Data.	The template declares Mode, ServiceToken and Value. The Delete request gets Echo, Mode, ServiceToken and Value, where Echo is a key the provider returned in Data.
C	A provider that runs longer than about 60 s is invoked again with the same request about every 60 s, even though it succeeds and replies.	A provider that takes 70 s ran, and replied, three times in 150 s, each time with the same Lambda request id.

Item C comes from Lambda asynchronous invocation, not from CloudFormation. A single Invoke with InvocationType=Event of a plain function that succeeds after 75 s runs about once a minute with the same request id. That was 5 runs in 300 s, and in a longer run 9 runs in 400 s with no sign of stopping. The LocalStack log shows no function error. Each re-run starts while the previous one is still running.

Practical consequences:

• A stuck provider hangs the stack until the client gives up, with no failure reason.
• Providers that decide between "modify" and "replace" by comparing OldResourceProperties with ResourceProperties always choose "replace". An example is the AWS CDK DNS-validated certificate requestor, which requests a new certificate on every update.
• Long-running providers repeat their side effects. Examples are certificate requests that wait for validation and resources that are polled until they are ready.

Expected behavior

As on AWS:

• A. The resource fails once ServiceTimeout seconds pass without a response. "The maximum time, in seconds, that can elapse before a custom resource operation times out. The value must be an integer from 1 to 3600. The default value is 3600 seconds (1 hour)." (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudformation-customresource.html). See also "Response timeout" in https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html: "If no response is received during this time, the stack operation fails."
• B. Update requests contain PhysicalResourceId (Required: Yes) and OldResourceProperties ("The resource property values that were previously declared by the template developer in the CloudFormation template. Required: Yes"). ResourceProperties "contains the contents of the Properties object sent by the template developer". Response Data is not merged into it (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-ref.html).
• C. An asynchronous invocation that succeeds runs once. Lambda retries only "if the function returns an error", at most twice, waiting 1 and then 2 minutes (https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-error-handling.html). The same page warns that an occasional duplicate delivery is possible because the queue is eventually consistent. A re-run every minute for as long as the function takes over 60 s is a different thing.

How are you starting LocalStack?

With a docker run command:

docker run --rm -p 127.0.0.1:4566:4566 -e LOCALSTACK_AUTH_TOKEN -v /var/run/docker.sock:/var/run/docker.sock localstack/localstack-pro:2026.8.1

Steps to reproduce

Self-contained script (boto3 only, about 5 minutes). It creates three small stacks, each with an inline Python provider: one that never replies (with ServiceTimeout: 30), one that takes 70 s, and one that is updated. It also makes a single asynchronous invoke of a plain function that takes 75 s. Script:

"""CloudFormation custom resources on LocalStack: three differences from AWS.

  A. ServiceTimeout is ignored: a resource whose provider never answers stays CREATE_IN_PROGRESS.
  B. Update requests carry no OldResourceProperties and no PhysicalResourceId, and the Delete request's
     ResourceProperties also contain the keys of the provider's response Data ("Echo" below).
  C. A Lambda provider that runs longer than about 60 s is invoked again with the same request
     every ~60 s, although it succeeds. The root cause is in Lambda asynchronous (Event) invocation:
     part D shows the same with a plain Lambda function and no CloudFormation.

Run against LocalStack on http://localhost:4566 (pip install boto3). Takes about 5 minutes.
"""
import io
import json
import os
import time
import zipfile

import boto3

KW = dict(endpoint_url=os.environ.get("ENDPOINT", "http://localhost:4566"), region_name="us-east-1",
          aws_access_key_id="test", aws_secret_access_key="test")
cfn = boto3.client("cloudformation", **KW)
logs = boto3.client("logs", **KW)
lam = boto3.client("lambda", **KW)

# Minimal provider: logs each request, then (depending on Mode) replies at once, never, or after 70 s.
PROVIDER = r'''
import json, time, urllib.request
def handler(event, context):
    print(time.strftime("%H:%M:%S") + " REQUEST " + json.dumps({"RequestType": event["RequestType"], "LambdaRequestId": context.aws_request_id,
          "PhysicalResourceId": event.get("PhysicalResourceId"), "OldResourceProperties": event.get("OldResourceProperties"),
          "ResourcePropertiesKeys": sorted(event["ResourceProperties"])}))
    mode = event["ResourceProperties"]["Mode"]
    if mode == "never-reply" and event["RequestType"] != "Delete":
        return
    if mode == "slow" and event["RequestType"] != "Delete":
        time.sleep(70)
    body = json.dumps({"Status": "SUCCESS", "PhysicalResourceId": "my-physical-id", "StackId": event["StackId"],
                       "RequestId": event["RequestId"], "LogicalResourceId": event["LogicalResourceId"],
                       "Data": {"Echo": event["ResourceProperties"]["Value"]}}).encode()
    urllib.request.urlopen(urllib.request.Request(event["ResponseURL"], data=body, method="PUT"))
    print(time.strftime("%H:%M:%S") + " REPLIED")
'''


def template(mode, value="v1", service_timeout=None):
    props = {"ServiceToken": {"Fn::GetAtt": ["Provider", "Arn"]}, "Mode": mode, "Value": value}
    if service_timeout:
        props["ServiceTimeout"] = service_timeout
    return json.dumps({"Resources": {
        "Provider": {"Type": "AWS::Lambda::Function", "Properties": {
            "Runtime": "python3.12", "Handler": "index.handler", "Timeout": 120, "Code": {"ZipFile": PROVIDER},
            "Role": "arn:aws:iam::000000000000:role/any"}},
        "Thing": {"Type": "Custom::Thing", "Properties": props}},
        "Outputs": {"Value": {"Value": {"Fn::GetAtt": ["Thing", "Echo"]}}}})


def wait(stack, limit):
    start = time.time()
    while True:
        status = cfn.describe_stacks(StackName=stack)["Stacks"][0]["StackStatus"]
        if not status.endswith("_IN_PROGRESS") or time.time() - start > limit:
            return status, round(time.time() - start)
        time.sleep(5)


def log_group(stack):
    return "/aws/lambda/" + cfn.describe_stack_resource(
        StackName=stack, LogicalResourceId="Provider")["StackResourceDetail"]["PhysicalResourceId"]


def provider_log(group):
    lines = []
    for stream in logs.describe_log_streams(logGroupName=group)["logStreams"]:
        for e in logs.get_log_events(logGroupName=group, logStreamName=stream["logStreamName"])["events"]:
            if " REQUEST " in e["message"] or " REPLIED" in e["message"]:
                lines.append("  " + e["message"].strip())
    return "\n".join(sorted(lines))


# D. plain Lambda: one asynchronous invocation of a function that succeeds after 75 s.
SLOW = "import time\ndef handler(event, context):\n    print(time.strftime('%H:%M:%S') + ' INVOKED ' + context.aws_request_id)\n    time.sleep(75)\n    print(time.strftime('%H:%M:%S') + ' SUCCEEDED')\n"
zipped = io.BytesIO()
with zipfile.ZipFile(zipped, "w") as z:
    z.writestr("index.py", SLOW)
lam.create_function(FunctionName="slow-async", Runtime="python3.12", Handler="index.handler", Timeout=120,
                    Role="arn:aws:iam::000000000000:role/any", Code={"ZipFile": zipped.getvalue()})
lam.get_waiter("function_active_v2").wait(FunctionName="slow-async")
lam.invoke(FunctionName="slow-async", InvocationType="Event", Payload=b"{}")
started = time.time()

cfn.create_stack(StackName="cr-timeout", TemplateBody=template("never-reply", service_timeout=30))
cfn.create_stack(StackName="cr-slow", TemplateBody=template("slow"))
cfn.create_stack(StackName="cr-update", TemplateBody=template("reply"))
try:
    print("B. create ->", wait("cr-update", 120))
    cfn.update_stack(StackName="cr-update", TemplateBody=template("reply", value="v2"))
    print("B. update (Value v1 -> v2) ->", wait("cr-update", 120))
    group = log_group("cr-update")
    cfn.delete_stack(StackName="cr-update")
    time.sleep(15)
    print(provider_log(group))

    print("C. provider that takes 70 s and replies once per invocation: create ->", wait("cr-slow", 180))
    time.sleep(150)  # let any further invocations finish
    print(provider_log(log_group("cr-slow")))

    stack = cfn.describe_stacks(StackName="cr-timeout")["Stacks"][0]
    print("A. ServiceTimeout 30, provider never replies: %ds after create -> %s"
          % (time.time() - stack["CreationTime"].timestamp(), stack["StackStatus"]))

    time.sleep(max(0, 300 - (time.time() - started)))
    lines = []
    for stream in logs.describe_log_streams(logGroupName="/aws/lambda/slow-async")["logStreams"]:
        lines += [e["message"].strip() for e in logs.get_log_events(
            logGroupName="/aws/lambda/slow-async", logStreamName=stream["logStreamName"])["events"]
            if " INVOKED " in e["message"] or " SUCCEEDED" in e["message"]]
    print("D. one Event invoke of a function that succeeds after 75 s, %ds later:" % (time.time() - started))
    print("\n".join("  " + line for line in sorted(lines)))
finally:
    lam.delete_function(FunctionName="slow-async")
    for stack in ("cr-timeout", "cr-slow", "cr-update"):
        cfn.delete_stack(StackName=stack)

Output (localstack/localstack-pro:2026.8.1; 2026.7.5 gives the same result):

B. create -> ('CREATE_COMPLETE', 10)
B. update (Value v1 -> v2) -> ('UPDATE_COMPLETE', 10)
  09:39:28 REPLIED
  09:39:28 REQUEST {"RequestType": "Create", "LambdaRequestId": "b0301ff7-d8f2-465e-85be-5cf583d61f36", "PhysicalResourceId": null, "OldResourceProperties": null, "ResourcePropertiesKeys": ["Mode", "ServiceToken", "Value"]}
  09:39:37 REPLIED
  09:39:37 REQUEST {"RequestType": "Update", "LambdaRequestId": "0e45d4c0-fc50-4f15-be06-776b15180e24", "PhysicalResourceId": null, "OldResourceProperties": null, "ResourcePropertiesKeys": ["Mode", "ServiceToken", "Value"]}
  09:39:47 REPLIED
  09:39:47 REQUEST {"RequestType": "Delete", "LambdaRequestId": "ecbbf99c-ae0a-4d7d-be44-2ffd35e61c3d", "PhysicalResourceId": "my-physical-id", "OldResourceProperties": null, "ResourcePropertiesKeys": ["Echo", "Mode", "ServiceToken", "Value"]}
C. provider that takes 70 s and replies once per invocation: create -> ('CREATE_COMPLETE', 45)
  09:39:29 REQUEST {"RequestType": "Create", "LambdaRequestId": "6111e314-c44b-42d1-86c3-ba65b0f79d2c", "PhysicalResourceId": null, "OldResourceProperties": null, "ResourcePropertiesKeys": ["Mode", "ServiceToken", "Value"]}
  09:40:29 REQUEST {"RequestType": "Create", "LambdaRequestId": "6111e314-c44b-42d1-86c3-ba65b0f79d2c", "PhysicalResourceId": null, "OldResourceProperties": null, "ResourcePropertiesKeys": ["Mode", "ServiceToken", "Value"]}
  09:40:39 REPLIED
  09:41:29 REQUEST {"RequestType": "Create", "LambdaRequestId": "6111e314-c44b-42d1-86c3-ba65b0f79d2c", "PhysicalResourceId": null, "OldResourceProperties": null, "ResourcePropertiesKeys": ["Mode", "ServiceToken", "Value"]}
  09:41:39 REPLIED
  09:42:39 REPLIED
A. ServiceTimeout 30, provider never replies: 230s after create -> CREATE_IN_PROGRESS
D. one Event invoke of a function that succeeds after 75 s, 300s later:
  09:39:23 INVOKED dd7e7dda-c4a6-420c-bcd7-16ee205b085a
  09:40:23 INVOKED dd7e7dda-c4a6-420c-bcd7-16ee205b085a
  09:40:38 SUCCEEDED
  09:41:24 INVOKED dd7e7dda-c4a6-420c-bcd7-16ee205b085a
  09:41:38 SUCCEEDED
  09:42:22 INVOKED dd7e7dda-c4a6-420c-bcd7-16ee205b085a
  09:42:25 INVOKED dd7e7dda-c4a6-420c-bcd7-16ee205b085a
  09:42:39 SUCCEEDED
  09:43:37 SUCCEEDED
  09:43:40 SUCCEEDED

The provider sends one reply per invocation (REPLIED). In part C the first invocation is still running when the second and third start.

Environment

• OS: macOS 26 (Apple silicon), Docker Desktop
• LocalStack: localstack/localstack-pro:2026.7.5 and localstack/localstack-pro:2026.8.1

Anything else?

Workarounds:

• A: none in the template. A client-side deadline on the stack operation is the only guard.
• B: the provider has to keep its own record of the previous properties and physical id.
• C: keep the provider's work under about 60 s, or make it fully idempotent.

Replies to the ResponseURL (http://s3.localhost.localstack.cloud:4566/...) work from Lambda containers with default settings. That includes providers that force https on port 443, as the AWS CDK providers do. So the hang in A is not a connectivity problem.

[EddEcho]

Hi @k-rooot, thanks for flagging this. We've reproduced all four, confirmed them against real AWS, and handed them to engineering. We'll follow up here once a fix lands so you can verify it.