diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 022d8b575..43dc473e3 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -19,7 +19,7 @@ jobs: - name: Set up Ruby uses: ruby/setup-ruby@v1 with: - ruby-version: 3.0.3 + ruby-version: '3.4' bundler-cache: true - name: Run tests @@ -44,7 +44,7 @@ jobs: - name: Set up Ruby uses: ruby/setup-ruby@v1 with: - ruby-version: 3.0.3 + ruby-version: '3.4' - name: Bundle install run: | diff --git a/Makefile b/Makefile index 6967cb5bb..6308949d0 100644 --- a/Makefile +++ b/Makefile @@ -4,6 +4,7 @@ PROTO_DIRS = $(sort $(dir $(PROTO_FILES))) PROTO_OUT := lib/gen proto: - $(foreach PROTO_DIR,$(PROTO_DIRS),bundle exec grpc_tools_ruby_protoc -Iproto --ruby_out=$(PROTO_OUT) --grpc_out=$(PROTO_OUT) $(PROTO_DIR)*.proto;) + @set -e; $(foreach PROTO_DIR,$(PROTO_DIRS),bundle exec grpc_tools_ruby_protoc -Iproto --ruby_out=$(PROTO_OUT) --grpc_out=$(PROTO_OUT) $(PROTO_DIR)*.proto;) + @for script in scripts/backport_*.rb; do bundle exec ruby "$$script" || exit $$?; done .PHONY: proto diff --git a/README.md b/README.md index a64c11590..2d6970fb3 100644 --- a/README.md +++ b/README.md @@ -322,6 +322,30 @@ To achieve this there are two methods (returning a UUID token) available from yo Both tokens will remain the same across multiple retry attempts of the activity. +When an Activity error carries a service-provided retry interval, include +`Temporal::RetryDelay` in that error class and set `next_retry_delay` in seconds +before raising it. The value must be positive, at least one nanosecond, and may be fractional. For example: + +```ruby +class RateLimited < StandardError + include Temporal::RetryDelay + + attr_reader :retry_after_seconds +end + +def execute + ExternalService.call +rescue RateLimited => error + error.next_retry_delay = error.retry_after_seconds + raise +end +``` + +Only errors that include this module use the override. The retry policy's maximum attempts and +the Activity's timeouts still limit retries. Use `nil` for no override; zero and other invalid +delay values raise `ArgumentError` when set. The server versions used by this SDK can treat a zero +override as a retry timeout. + ### Asynchronous completion When dealing with asynchronous business logic in your activities, you might need to wait for an diff --git a/docs/retry-delay-parity.md b/docs/retry-delay-parity.md new file mode 100644 index 000000000..0dce332f5 --- /dev/null +++ b/docs/retry-delay-parity.md @@ -0,0 +1,18 @@ +# Per-failure retry delay parity + +The first-party Ruby SDK reference is [`temporalio/sdk-ruby` at `2169e3e0dc2b4f59dbf9cc93205e19c2c01a037e`](https://github.com/temporalio/sdk-ruby/tree/2169e3e0dc2b4f59dbf9cc93205e19c2c01a037e). Its [Activity failure test](https://github.com/temporalio/sdk-ruby/blob/2169e3e0dc2b4f59dbf9cc93205e19c2c01a037e/temporalio/test/worker_activity_test.rb#L200-L251) sets a fractional `next_retry_delay: 1.23` on a non-retryable `ApplicationError` and checks that the client receives its message, type, details, non-retryable flag, and retry delay. That test proves metadata round-trip; its non-retryable setting means it does not verify a scheduled retry. The [first-party failure converter](https://github.com/temporalio/sdk-ruby/blob/2169e3e0dc2b4f59dbf9cc93205e19c2c01a037e/temporalio/lib/temporalio/converters/failure_converter.rb#L40-L54) writes the delay to `ApplicationFailureInfo`; [deserialization](https://github.com/temporalio/sdk-ruby/blob/2169e3e0dc2b4f59dbf9cc93205e19c2c01a037e/temporalio/lib/temporalio/converters/failure_converter.rb#L140-L154) restores it. The [wire field](https://github.com/temporalio/api/blob/v1.44.0/temporal/api/failure/v1/message.proto) is field 4, a protobuf `Duration`. + +| First-party behavior or boundary | This fork's coverage | +| --- | --- | +| Fractional delay survives serialization and deserialization | `failure_spec.rb`: `preserves a fractional retry delay...` runs for both legacy and v2 exception serialization, checks seconds/nanos, original class, message, and reconstructed delay. | +| Original error type, details, and stack trace survive as appropriate to converter mode | Existing `failure_spec.rb` full/legacy/fallback cases and `workflow/errors_spec.rb` reconstruction cases remain in the focused suite. New fractional cases assert class and message for both modes. | +| Retry-delay metadata remains available when the original class is not installed on the reader | `workflow/errors_spec.rb`: `retains a retry delay when the original error class is unavailable`. This is specific to the fork's class-based reconstruction. | +| Server schedules the next remote Activity attempt using the override, instead of the retry policy interval | `spec/integration/retry_delay_spec.rb` inspects `DescribeWorkflowExecution` pending failure for the fractional field and waits for the second actual Activity attempt; policy interval and maximum interval are 1 second, while the override is 2.3 seconds. Run with `TEMPORAL_TEST_PORT` against an isolated Temporal server. | +| Inbound delay `0` is distinct from an absent override | `failure_spec.rb`: `preserves an inbound zero delay without emitting one by default` also reserializes the restored error without an invalid override. The [first-party converter](https://github.com/temporalio/sdk-ruby/blob/2169e3e0dc2b4f59dbf9cc93205e19c2c01a037e/temporalio/lib/temporalio/internal/proto_utils.rb#L8-L20) preserves zero. This fork reads it for interoperability but does not emit it on re-raise because Temporal Server 1.32 treats a zero custom delay as a retry timeout. | +| Ordinary errors remain retryable without an opt-in delay | `failure_spec.rb`: `does not infer an override...`; existing ordinary error tests. Only exceptions including `Temporal::RetryDelay` emit the field. | +| Invalid inputs cannot create malformed duration or silently lose Activity failure reports | `failure_spec.rb`: `rejects invalid delay values before an error can be raised`. Setter rejects zero, sub-nanosecond, negative, non-finite, non-real, non-numeric, and out-of-range values; valid fractional values serialize at nanosecond precision. | +| Retry caps and total timeouts still apply | The [Temporal retry-policy contract](https://docs.temporal.io/encyclopedia/retry-policies#per-error-next-retry-delay) states the override replaces the interval while limits remain in force. The SDK sends an ordinary `ApplicationFailureInfo` with the additional duration; server owns enforcement. The integration test uses `max_attempts: 2` and checks exactly two attempts. | +| First-party non-retryable flag, arbitrary `ApplicationError` details, and cause chain | This fork keeps its existing class-based non-retriable policy and legacy/v2 exception payload behavior. Adding a separate ApplicationError wrapper or per-instance non-retryable API is outside this retry-delay change. The first-party test's non-retryable flag is used to terminate promptly, not to test retry scheduling. | +| First-party Go local Activity delay tests | [Go `ErrorWithNextDelay`](https://github.com/temporalio/sdk-go/blob/733b222690ef005e9cb09e684a11372fe7241a68/test/activity_test.go#L70) and [local Activity test](https://github.com/temporalio/sdk-go/blob/733b222690ef005e9cb09e684a11372fe7241a68/test/workflow_test.go#L90) exercise local Activities, which this fork does not implement. They are not a direct remote Activity parity test. | + +This table maps behaviors, not raw test counts. The live integration assertion is required to claim server scheduling behavior; a skipped test only establishes that the test harness is present. diff --git a/lib/gen/temporal/api/failure/v1/message_pb.rb b/lib/gen/temporal/api/failure/v1/message_pb.rb index 664755cd7..c69a6f9c3 100644 --- a/lib/gen/temporal/api/failure/v1/message_pb.rb +++ b/lib/gen/temporal/api/failure/v1/message_pb.rb @@ -6,9 +6,10 @@ require 'temporal/api/common/v1/message_pb' require 'temporal/api/enums/v1/workflow_pb' +require 'google/protobuf/duration_pb' -descriptor_data = "\n%temporal/api/failure/v1/message.proto\x12\x17temporal.api.failure.v1\x1a$temporal/api/common/v1/message.proto\x1a$temporal/api/enums/v1/workflow.proto\"p\n\x16\x41pplicationFailureInfo\x12\x0c\n\x04type\x18\x01 \x01(\t\x12\x15\n\rnon_retryable\x18\x02 \x01(\x08\x12\x31\n\x07\x64\x65tails\x18\x03 \x01(\x0b\x32 .temporal.api.common.v1.Payloads\"\x90\x01\n\x12TimeoutFailureInfo\x12\x38\n\x0ctimeout_type\x18\x01 \x01(\x0e\x32\".temporal.api.enums.v1.TimeoutType\x12@\n\x16last_heartbeat_details\x18\x02 \x01(\x0b\x32 .temporal.api.common.v1.Payloads\"H\n\x13\x43\x61nceledFailureInfo\x12\x31\n\x07\x64\x65tails\x18\x01 \x01(\x0b\x32 .temporal.api.common.v1.Payloads\"\x17\n\x15TerminatedFailureInfo\"*\n\x11ServerFailureInfo\x12\x15\n\rnon_retryable\x18\x01 \x01(\x08\"\\\n\x18ResetWorkflowFailureInfo\x12@\n\x16last_heartbeat_details\x18\x01 \x01(\x0b\x32 .temporal.api.common.v1.Payloads\"\xe7\x01\n\x13\x41\x63tivityFailureInfo\x12\x1a\n\x12scheduled_event_id\x18\x01 \x01(\x03\x12\x18\n\x10started_event_id\x18\x02 \x01(\x03\x12\x10\n\x08identity\x18\x03 \x01(\t\x12;\n\ractivity_type\x18\x04 \x01(\x0b\x32$.temporal.api.common.v1.ActivityType\x12\x13\n\x0b\x61\x63tivity_id\x18\x05 \x01(\t\x12\x36\n\x0bretry_state\x18\x06 \x01(\x0e\x32!.temporal.api.enums.v1.RetryState\"\xa8\x02\n!ChildWorkflowExecutionFailureInfo\x12\x11\n\tnamespace\x18\x01 \x01(\t\x12\x45\n\x12workflow_execution\x18\x02 \x01(\x0b\x32).temporal.api.common.v1.WorkflowExecution\x12;\n\rworkflow_type\x18\x03 \x01(\x0b\x32$.temporal.api.common.v1.WorkflowType\x12\x1a\n\x12initiated_event_id\x18\x04 \x01(\x03\x12\x18\n\x10started_event_id\x18\x05 \x01(\x03\x12\x36\n\x0bretry_state\x18\x06 \x01(\x0e\x32!.temporal.api.enums.v1.RetryState\"\xe2\x06\n\x07\x46\x61ilure\x12\x0f\n\x07message\x18\x01 \x01(\t\x12\x0e\n\x06source\x18\x02 \x01(\t\x12\x13\n\x0bstack_trace\x18\x03 \x01(\t\x12;\n\x12\x65ncoded_attributes\x18\x14 \x01(\x0b\x32\x1f.temporal.api.common.v1.Payload\x12/\n\x05\x63\x61use\x18\x04 \x01(\x0b\x32 .temporal.api.failure.v1.Failure\x12S\n\x18\x61pplication_failure_info\x18\x05 \x01(\x0b\x32/.temporal.api.failure.v1.ApplicationFailureInfoH\x00\x12K\n\x14timeout_failure_info\x18\x06 \x01(\x0b\x32+.temporal.api.failure.v1.TimeoutFailureInfoH\x00\x12M\n\x15\x63\x61nceled_failure_info\x18\x07 \x01(\x0b\x32,.temporal.api.failure.v1.CanceledFailureInfoH\x00\x12Q\n\x17terminated_failure_info\x18\x08 \x01(\x0b\x32..temporal.api.failure.v1.TerminatedFailureInfoH\x00\x12I\n\x13server_failure_info\x18\t \x01(\x0b\x32*.temporal.api.failure.v1.ServerFailureInfoH\x00\x12X\n\x1breset_workflow_failure_info\x18\n \x01(\x0b\x32\x31.temporal.api.failure.v1.ResetWorkflowFailureInfoH\x00\x12M\n\x15\x61\x63tivity_failure_info\x18\x0b \x01(\x0b\x32,.temporal.api.failure.v1.ActivityFailureInfoH\x00\x12k\n%child_workflow_execution_failure_info\x18\x0c \x01(\x0b\x32:.temporal.api.failure.v1.ChildWorkflowExecutionFailureInfoH\x00\x42\x0e\n\x0c\x66\x61ilure_infoB\x8e\x01\n\x1aio.temporal.api.failure.v1B\x0cMessageProtoP\x01Z%go.temporal.io/api/failure/v1;failure\xaa\x02\x19Temporalio.Api.Failure.V1\xea\x02\x1cTemporalio::Api::Failure::V1b\x06proto3" +descriptor_data = "\n%temporal/api/failure/v1/message.proto\x12\x17temporal.api.failure.v1\x1A$temporal/api/common/v1/message.proto\x1A$temporal/api/enums/v1/workflow.proto\x1A\x1Egoogle/protobuf/duration.proto\"\xA5\x01\n\x16ApplicationFailureInfo\x12\f\n\x04type\x18\x01 \x01(\t\x12\x15\n\rnon_retryable\x18\x02 \x01(\b\x121\n\adetails\x18\x03 \x01(\v2 .temporal.api.common.v1.Payloads\x123\n\x10next_retry_delay\x18\x04 \x01(\v2\x19.google.protobuf.Duration\"\x90\x01\n\x12TimeoutFailureInfo\x128\n\ftimeout_type\x18\x01 \x01(\x0E2\".temporal.api.enums.v1.TimeoutType\x12@\n\x16last_heartbeat_details\x18\x02 \x01(\v2 .temporal.api.common.v1.Payloads\"H\n\x13CanceledFailureInfo\x121\n\adetails\x18\x01 \x01(\v2 .temporal.api.common.v1.Payloads\"\x17\n\x15TerminatedFailureInfo\"*\n\x11ServerFailureInfo\x12\x15\n\rnon_retryable\x18\x01 \x01(\b\"\\\n\x18ResetWorkflowFailureInfo\x12@\n\x16last_heartbeat_details\x18\x01 \x01(\v2 .temporal.api.common.v1.Payloads\"\xE7\x01\n\x13ActivityFailureInfo\x12\x1A\n\x12scheduled_event_id\x18\x01 \x01(\x03\x12\x18\n\x10started_event_id\x18\x02 \x01(\x03\x12\x10\n\bidentity\x18\x03 \x01(\t\x12;\n\ractivity_type\x18\x04 \x01(\v2$.temporal.api.common.v1.ActivityType\x12\x13\n\vactivity_id\x18\x05 \x01(\t\x126\n\vretry_state\x18\x06 \x01(\x0E2!.temporal.api.enums.v1.RetryState\"\xA8\x02\n!ChildWorkflowExecutionFailureInfo\x12\x11\n\tnamespace\x18\x01 \x01(\t\x12E\n\x12workflow_execution\x18\x02 \x01(\v2).temporal.api.common.v1.WorkflowExecution\x12;\n\rworkflow_type\x18\x03 \x01(\v2$.temporal.api.common.v1.WorkflowType\x12\x1A\n\x12initiated_event_id\x18\x04 \x01(\x03\x12\x18\n\x10started_event_id\x18\x05 \x01(\x03\x126\n\vretry_state\x18\x06 \x01(\x0E2!.temporal.api.enums.v1.RetryState\"\xE2\x06\n\aFailure\x12\x0F\n\amessage\x18\x01 \x01(\t\x12\x0E\n\x06source\x18\x02 \x01(\t\x12\x13\n\vstack_trace\x18\x03 \x01(\t\x12;\n\x12encoded_attributes\x18\x14 \x01(\v2\x1F.temporal.api.common.v1.Payload\x12/\n\x05cause\x18\x04 \x01(\v2 .temporal.api.failure.v1.Failure\x12S\n\x18application_failure_info\x18\x05 \x01(\v2/.temporal.api.failure.v1.ApplicationFailureInfoH\x00\x12K\n\x14timeout_failure_info\x18\x06 \x01(\v2+.temporal.api.failure.v1.TimeoutFailureInfoH\x00\x12M\n\x15canceled_failure_info\x18\a \x01(\v2,.temporal.api.failure.v1.CanceledFailureInfoH\x00\x12Q\n\x17terminated_failure_info\x18\b \x01(\v2..temporal.api.failure.v1.TerminatedFailureInfoH\x00\x12I\n\x13server_failure_info\x18\t \x01(\v2*.temporal.api.failure.v1.ServerFailureInfoH\x00\x12X\n\ereset_workflow_failure_info\x18\n \x01(\v21.temporal.api.failure.v1.ResetWorkflowFailureInfoH\x00\x12M\n\x15activity_failure_info\x18\v \x01(\v2,.temporal.api.failure.v1.ActivityFailureInfoH\x00\x12k\n%child_workflow_execution_failure_info\x18\f \x01(\v2:.temporal.api.failure.v1.ChildWorkflowExecutionFailureInfoH\x00B\x0E\n\ffailure_infoB\x8E\x01\n\x1Aio.temporal.api.failure.v1B\fMessageProtoP\x01Z%go.temporal.io/api/failure/v1;failure\xAA\x02\x19Temporalio.Api.Failure.V1\xEA\x02\x1CTemporalio::Api::Failure::V1b\x06proto3" pool = ::Google::Protobuf::DescriptorPool.generated_pool pool.add_serialized_file(descriptor_data) diff --git a/lib/temporal/connection/serializer/failure.rb b/lib/temporal/connection/serializer/failure.rb index 2d17e949a..5049a34aa 100644 --- a/lib/temporal/connection/serializer/failure.rb +++ b/lib/temporal/connection/serializer/failure.rb @@ -1,4 +1,6 @@ require 'temporal/connection/serializer/base' +require 'temporal/errors' +require 'google/protobuf/duration_pb' module Temporal module Connection @@ -27,18 +29,29 @@ def to_proto else details = converter.to_details_payloads(object.message) end + next_retry_delay = duration_from(object.next_retry_delay) if object.is_a?(Temporal::RetryDelay) Temporalio::Api::Failure::V1::Failure.new( message: object.message, stack_trace: stack_trace_from(object.backtrace), application_failure_info: Temporalio::Api::Failure::V1::ApplicationFailureInfo.new( type: object.class.name, - details: details + details: details, + next_retry_delay: next_retry_delay ) ) end private + def duration_from(value) + # A failure read from history can carry a delay rejected by this server. Keep its error intact on re-raise. + return unless Temporal::RetryDelay.valid_seconds?(value) + + nanoseconds = (value.to_r * 1_000_000_000).round + seconds, nanos = nanoseconds.divmod(1_000_000_000) + Google::Protobuf::Duration.new(seconds: seconds, nanos: nanos) + end + def stack_trace_from(backtrace) return unless backtrace diff --git a/lib/temporal/errors.rb b/lib/temporal/errors.rb index 1c423a6ce..c432ba5e5 100644 --- a/lib/temporal/errors.rb +++ b/lib/temporal/errors.rb @@ -1,4 +1,29 @@ module Temporal + # Include this in an exception class to opt in to a per-failure retry delay. + # The value is seconds and may be fractional. It only overrides the next + # interval; the retry policy's attempt limit and timeouts still apply. + module RetryDelay + attr_reader :next_retry_delay + + def next_retry_delay=(seconds) + unless seconds.nil? || Temporal::RetryDelay.valid_seconds?(seconds) + raise ArgumentError, 'next_retry_delay must be a finite number of seconds between 1 nanosecond and 315576000000 seconds' + end + + @next_retry_delay = seconds + end + + def self.valid_seconds?(seconds) + seconds.is_a?(Numeric) && seconds.real? && seconds.finite? && + seconds.to_r >= Rational(1, 1_000_000_000) && seconds <= 315_576_000_000 + end + + def self.restore(error, seconds) + error.extend(self) unless error.is_a?(self) + error.instance_variable_set(:@next_retry_delay, seconds) + end + end + # Superclass for all Temporal errors class Error < StandardError; end diff --git a/lib/temporal/workflow/errors.rb b/lib/temporal/workflow/errors.rb index f13f03bfa..7a43ed8a5 100644 --- a/lib/temporal/workflow/errors.rb +++ b/lib/temporal/workflow/errors.rb @@ -50,6 +50,12 @@ def self.generate_error(failure, converter, default_exception_class = StandardEr ) end exception.tap do |exception| + retry_delay = failure.application_failure_info.next_retry_delay + if retry_delay + # Older failures may carry zero; preserve their metadata without applying outbound validation. + Temporal::RetryDelay.restore(exception, + (retry_delay.seconds + retry_delay.nanos.quo(1_000_000_000)).to_f) + end backtrace = failure.stack_trace.split("\n") exception.set_backtrace(backtrace) if !backtrace.empty? end diff --git a/rbi/temporal-ruby.rbi b/rbi/temporal-ruby.rbi index cdcda0780..e0ffd17b7 100644 --- a/rbi/temporal-ruby.rbi +++ b/rbi/temporal-ruby.rbi @@ -1,4 +1,8 @@ module Temporal + module RetryDelay + def next_retry_delay; end + def next_retry_delay=(seconds); end + end class Activity; end class ActivityException; end module Client diff --git a/scripts/backport_next_retry_delay.rb b/scripts/backport_next_retry_delay.rb new file mode 100644 index 000000000..3ee1a3533 --- /dev/null +++ b/scripts/backport_next_retry_delay.rb @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# Restore the additive next_retry_delay field after regenerating the pinned API proto. + +require 'google/protobuf/descriptor_pb' + +path = File.expand_path('../lib/gen/temporal/api/failure/v1/message_pb.rb', __dir__) +source = File.read(path) +match = source.match(/^descriptor_data = (".*")$/) +raise 'Unexpected generated failure descriptor' unless match + +descriptor = Google::Protobuf::FileDescriptorProto.decode(match[1].undump) +raise 'Unexpected failure proto' unless descriptor.name == 'temporal/api/failure/v1/message.proto' +application = descriptor.message_type.find { |message| message.name == 'ApplicationFailureInfo' } +raise 'Missing ApplicationFailureInfo' unless application +field = application.field.find { |item| item.name == 'next_retry_delay' } +if field + raise 'Unexpected next_retry_delay field' unless field.number == 4 && field.type_name == '.google.protobuf.Duration' + exit 0 +end +raise 'Field 4 is occupied' if application.field.any? { |item| item.number == 4 } +raise 'Duration already imported' if descriptor.dependency.include?('google/protobuf/duration.proto') + +application.field << Google::Protobuf::FieldDescriptorProto.new( + name: 'next_retry_delay', number: 4, + label: :LABEL_OPTIONAL, type: :TYPE_MESSAGE, type_name: '.google.protobuf.Duration' +) +descriptor.dependency << 'google/protobuf/duration.proto' +source.sub!(match[0]) { "descriptor_data = #{Google::Protobuf::FileDescriptorProto.encode(descriptor).dump}" } +old_require = "require 'temporal/api/enums/v1/workflow_pb'" +raise 'Unexpected generated failure imports' unless source.scan(old_require).size == 1 +source.sub!(old_require, "#{old_require}\nrequire 'google/protobuf/duration_pb'") +File.write(path, source) diff --git a/spec/config/fabrication.rb b/spec/config/fabrication.rb index d4afe3e3e..cd0f15755 100644 --- a/spec/config/fabrication.rb +++ b/spec/config/fabrication.rb @@ -1,3 +1,4 @@ +require 'ostruct' require 'fabrication' Fabrication.configure do |config| diff --git a/spec/integration/retry_delay_spec.rb b/spec/integration/retry_delay_spec.rb new file mode 100644 index 000000000..a122e16ce --- /dev/null +++ b/spec/integration/retry_delay_spec.rb @@ -0,0 +1,78 @@ +require 'temporal/worker' +require 'tempfile' + +class RetryDelayIntegrationError < StandardError + include Temporal::RetryDelay +end + +class RetryDelayIntegrationActivity < Temporal::Activity + retry_policy interval: 1, backoff: 1, max_interval: 1, max_attempts: 2 + + def execute(path) + File.open(path, 'a') { |file| file.puts(Process.clock_gettime(Process::CLOCK_REALTIME)) } + error = RetryDelayIntegrationError.new('rate limited') + error.next_retry_delay = 2.3 + raise error + end +end + +class RetryDelayIntegrationWorkflow < Temporal::Workflow + def execute(path) + RetryDelayIntegrationActivity.execute!(path) + rescue RetryDelayIntegrationError => error + [error.class.name, error.message, error.next_retry_delay] + end +end + +describe 'Activity next_retry_delay against Temporal Server' do + it 'uses the failure delay instead of the retry policy interval' do + skip 'Set TEMPORAL_TEST_PORT to run against a local Temporal test server' unless ENV['TEMPORAL_TEST_PORT'] + + config = Temporal::Configuration.new + config.host = '127.0.0.1' + config.port = ENV.fetch('TEMPORAL_TEST_PORT').to_i + config.namespace = 'default' + config.task_queue = "retry-delay-#{SecureRandom.uuid}" + config.logger.level = Logger::FATAL + client = Temporal::Client.new(config) + + Tempfile.create('temporal-retry-delay-') do |attempt_file| + worker_pid = fork do + worker = Temporal::Worker.new(config) + worker.register_workflow(RetryDelayIntegrationWorkflow) + worker.register_activity(RetryDelayIntegrationActivity) + worker.start + end + begin + workflow_id = SecureRandom.uuid + run_id = client.start_workflow(RetryDelayIntegrationWorkflow, attempt_file.path, + options: { workflow_id: workflow_id }) + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 10 + pending_failure = nil + until pending_failure || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + response = client.connection.describe_workflow_execution(namespace: 'default', workflow_id: workflow_id, + run_id: run_id) + pending_failure = response.pending_activities.first&.last_failure + sleep 0.05 unless pending_failure + end + expect(pending_failure).not_to be_nil + expect(pending_failure.application_failure_info.next_retry_delay.seconds).to eq(2) + expect(pending_failure.application_failure_info.next_retry_delay.nanos).to eq(300_000_000) + + expect(client.await_workflow_result(RetryDelayIntegrationWorkflow, workflow_id: workflow_id, + run_id: run_id, timeout: 10)) + .to eq(['RetryDelayIntegrationError', 'rate limited', 2.3]) + attempts = File.readlines(attempt_file.path).map(&:to_f) + expect(attempts.size).to eq(2) + expect(attempts.last - attempts.first).to be >= 2.0 + ensure + begin + Process.kill('TERM', worker_pid) + rescue Errno::ESRCH + # Preserve the original test failure if the worker already exited. + end + Process.wait(worker_pid) + end + end + end +end diff --git a/spec/unit/lib/temporal/connection/serializer/failure_spec.rb b/spec/unit/lib/temporal/connection/serializer/failure_spec.rb index 2bde03376..56c37895c 100644 --- a/spec/unit/lib/temporal/connection/serializer/failure_spec.rb +++ b/spec/unit/lib/temporal/connection/serializer/failure_spec.rb @@ -106,5 +106,54 @@ def initialize; end expect(failure_proto.application_failure_info.type).to eq('MyArglessError') end + class DelayedError < StandardError + include Temporal::RetryDelay + end + + [false, true].each do |serialize_whole_error| + it "preserves a fractional retry delay and error details with v2=#{serialize_whole_error}" do + error = DelayedError.new('rate limited') + error.next_retry_delay = 1.23 + failure = described_class.new(error, converter, serialize_whole_error: serialize_whole_error).to_proto + + expect(failure.application_failure_info.type).to eq('DelayedError') + expect(failure.application_failure_info.next_retry_delay.seconds).to eq(1) + expect(failure.application_failure_info.next_retry_delay.nanos).to eq(230_000_000) + restored = Temporal::Workflow::Errors.generate_error(failure, converter) + expect(restored).to be_a(DelayedError) + expect(restored.message).to eq('rate limited') + expect(restored.next_retry_delay).to eq(1.23) + end + end + + it 'preserves an inbound zero delay without emitting one by default' do + failure = described_class.new(DelayedError.new('old failure'), converter).to_proto + failure.application_failure_info.next_retry_delay = Google::Protobuf::Duration.new + + expect(failure.application_failure_info.next_retry_delay).to eq(Google::Protobuf::Duration.new) + restored = Temporal::Workflow::Errors.generate_error(failure, converter) + expect(restored.next_retry_delay).to eq(0) + expect(described_class.new(restored, converter).to_proto.application_failure_info.next_retry_delay).to be_nil + expect(described_class.new(StandardError.new('ordinary'), converter).to_proto.application_failure_info.next_retry_delay) + .to be_nil + end + + it 'does not infer an override from similarly named fields on ordinary errors' do + error = StandardError.new('rate limited') + error.define_singleton_method(:retry_after_seconds) { 5 } + error.define_singleton_method(:next_retry_delay) { 5 } + + expect(described_class.new(error, converter).to_proto.application_failure_info.next_retry_delay).to be_nil + end + + it 'rejects invalid delay values before an error can be raised' do + error = DelayedError.new('invalid') + [-1, 0, 0.5e-9, Float::INFINITY, Float::NAN, '1.5', Complex(1, 2), 315_576_000_001].each do |value| + expect { error.next_retry_delay = value }.to raise_error(ArgumentError) + end + error.next_retry_delay = 1e-9 + expect(described_class.new(error, converter).to_proto.application_failure_info.next_retry_delay.nanos).to eq(1) + end + end end diff --git a/spec/unit/lib/temporal/testing/replay_tester_spec.rb b/spec/unit/lib/temporal/testing/replay_tester_spec.rb index 861a58134..31bdc0497 100644 --- a/spec/unit/lib/temporal/testing/replay_tester_spec.rb +++ b/spec/unit/lib/temporal/testing/replay_tester_spec.rb @@ -12,7 +12,10 @@ def execute end class TestReplayWorkflow < Temporal::Workflow - def execute(run_activity: false, run_sleep: false, result: "success") + def execute(args = {}) + run_activity = args.fetch(:run_activity, false) + run_sleep = args.fetch(:run_sleep, false) + result = args.fetch(:result, "success") TestReplayActivity.execute! if run_activity workflow.sleep(1) if run_sleep diff --git a/spec/unit/lib/temporal/workflow/context_spec.rb b/spec/unit/lib/temporal/workflow/context_spec.rb index 05a612824..90e1ca87f 100644 --- a/spec/unit/lib/temporal/workflow/context_spec.rb +++ b/spec/unit/lib/temporal/workflow/context_spec.rb @@ -419,7 +419,7 @@ def wait_for_any stack_trace = query_registry.handle(Temporal::Workflow::StackTraceTracker::STACK_TRACE_QUERY_NAME) expect(stack_trace).to start_with('Fiber count: 1') - expect(stack_trace).to include('block in wait_for_any') + expect(stack_trace).to match(/block in (?:[^'\n]*#)?wait_for_any/) end it 'cleared after unblocked' do @@ -488,7 +488,7 @@ def wait_until(&blk) stack_trace = query_registry.handle(Temporal::Workflow::StackTraceTracker::STACK_TRACE_QUERY_NAME) expect(stack_trace).to start_with('Fiber count: 1') - expect(stack_trace).to include('block in wait_until') + expect(stack_trace).to match(/block in (?:[^'\n]*#)?wait_until/) end it 'cleared after unblocked' do diff --git a/spec/unit/lib/temporal/workflow/errors_spec.rb b/spec/unit/lib/temporal/workflow/errors_spec.rb index bce9d477d..cfda5197e 100644 --- a/spec/unit/lib/temporal/workflow/errors_spec.rb +++ b/spec/unit/lib/temporal/workflow/errors_spec.rb @@ -88,6 +88,30 @@ def initialize(foo, bar) end + it 'rounds retry delay seconds and nanoseconds together when restoring an error' do + failure = Fabricate(:api_application_failure, message: 'rate limited', backtrace: [], + error_class: SomeError.to_s) + failure.application_failure_info.next_retry_delay = Google::Protobuf::Duration.new(seconds: 1, nanos: 118_000_000) + + error = described_class.generate_error(failure, converter) + + expect(error).to be_a(SomeError) + expect(error.next_retry_delay).to eq(1.118) + end + + it 'retains a retry delay when the original error class is unavailable' do + failure = Fabricate(:api_application_failure, message: 'rate limited', backtrace: [], + error_class: 'MissingRateLimitError') + failure.application_failure_info.next_retry_delay = Google::Protobuf::Duration.new(seconds: 2, nanos: 300_000_000) + allow(Temporal.logger).to receive(:error) + + error = described_class.generate_error(failure, converter) + + expect(error).to be_a(StandardError) + expect(error.message).to eq('MissingRateLimitError: rate limited') + expect(error.next_retry_delay).to eq(2.3) + end + it "falls back to StandardError when the client can't initialize the error class due to arity" do allow(Temporal.logger).to receive(:error) diff --git a/spec/unit/lib/temporal/workflow/stack_trace_tracker_spec.rb b/spec/unit/lib/temporal/workflow/stack_trace_tracker_spec.rb index 5db23fb70..efac9f13a 100644 --- a/spec/unit/lib/temporal/workflow/stack_trace_tracker_spec.rb +++ b/spec/unit/lib/temporal/workflow/stack_trace_tracker_spec.rb @@ -50,7 +50,7 @@ def record_two_function expect(first_stack).to include("record_two_function") second_stack = stack_traces[2] - expect(second_stack).to include("block in record_two_function") + expect(second_stack).to match(/block in (?:[^'\n]*#)?record_two_function/) end end end \ No newline at end of file