Skip to content

fix(server): always stamp status.timestamp on terminal task states - #1346

Open
Linux2010 wants to merge 1 commit into
a2aproject:mainfrom
Linux2010:fix/1331-failed-timestamp
Open

Linux2010 wants to merge 1 commit into
a2aproject:mainfrom
Linux2010:fix/1331-failed-timestamp

Conversation

@Linux2010

Copy link
Copy Markdown
Contributor

Description

Framework failure paths could persist a FAILED task without status.timestamp, and DatabaseTaskStore derives last_updated from status.timestamp (database_task_store.py:152-156) — so the task was persisted with last_updated = NULL, breaking ListTasks ordering and status_timestamp_after filtering. Root-cause analysis and a runtime reproduction are in #1331.

Per the two-layer approach endorsed on the issue:

  • Failure sites stamp explicitly (ActiveTask): _write_failed_status, _mark_task_as_failed, and the producer failure path now stamp the current time on the FAILED status.
  • Defensive backstop in TaskManager.save_task_event: a terminal state arriving without a timestamp (raw Task path or status-event path) is auto-stamped with the current time, so future call sites cannot reintroduce the NULL-last_updated hole.

Semantics: non-terminal states are never touched, and explicitly provided timestamps are never overwritten.

  • Follow the CONTRIBUTING Guide
  • PR title follows Conventional Commits (fix: → SemVer patch)
  • Tests and linter pass
  • Appropriate docs were updated (guard docstring states the contract)

Fixes #1331 🦕

Validation

  • Claim: an executor crashing mid-flight now persists TASK_STATE_FAILED with status.timestamp set; non-terminal statuses and explicitly provided timestamps are untouched.
  • Exact head: c34c53f
  • BEFORE: persisted task = state=TASK_STATE_FAILED, timestamp set=False (repro in [Bug]: Framework-written FAILED status drops status.timestamp, persisting last_updated as NULL #1331). AFTER: timestamp set=True.
  • Tests: 4 new tests in tests/server/tasks/test_task_manager.py (terminal-unstamped auto-stamp, provided-timestamp preserved, non-terminal untouched, raw-Task path guarded) + 1 end-to-end in test_default_request_handler_v2.py.
  • Full suite: 2302 passed, coverage 93% (gate 88%).
  • Environment: Python 3.10.20, macOS.
  • Limits / not tested: DatabaseTaskStore integration (covered indirectly via the same status pipeline; the e2e asserts the in-memory store the same way CI's coverage report measures it).

Prepared with AI assistance (WorkBuddy); reviewed and verified by Linux2010

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🧪 Code Coverage (vs main)

⬇️ Download Full Report

Base PR Delta
src/a2a/server/agent_execution/active_task.py 95.40% 95.45% 🟢 +0.05%
src/a2a/server/cluster/database_event_stream.py 92.86% 96.94% 🟢 +4.08%
src/a2a/server/tasks/task_manager.py 98.86% 98.92% 🟢 +0.06%
Total 92.89% 92.95% 🟢 +0.05%

Generated by coverage-comment.yml

@mykytanetipa mykytanetipa self-assigned this Oct 19, 2026
Framework failure paths could persist a FAILED task without
status.timestamp: the consumer failure path built a
TaskStatusUpdateEvent with an unstamped TaskStatus (which
TaskManager.save_task_event then CopyFrom'd over the task, clearing
any prior timestamp), and the producer failure path mutated
task.status.state directly before saving. DatabaseTaskStore derives
last_updated from status.timestamp, so the task was persisted with
last_updated = NULL, breaking ListTasks ordering and time-based
filters (issue a2aproject#1331).

Per the approach endorsed on the issue, this lands both layers:

- failure sites in ActiveTask (_write_failed_status,
  _mark_task_as_failed, and the producer failure path) stamp the
  current time explicitly;
- TaskManager.save_task_event gains a defensive backstop that
  auto-populates the timestamp whenever a terminal state arrives
  unstamped, covering raw Task and status-event paths alike.

Non-terminal states are left untouched, and explicitly provided
timestamps are never overwritten.

Signed-off-by: andy <linux2011@qq.com>
@Linux2010
Linux2010 force-pushed the fix/1331-failed-timestamp branch from c34c53f to 6be10db Compare October 21, 2026 02:00
@Linux2010

Copy link
Copy Markdown
Contributor Author

Friendly ping — this is ready for review whenever anyone has bandwidth (both branches just rebased onto latest main). Happy to address any feedback. Thanks!

@mykytanetipa mykytanetipa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only stamp terminal states in TaskManager?

  • If an agent sends an unstamped WORKING or INPUT_REQUIRED update (for example via new_text_status_update_event, which doesn't set a timestamp), task.status.CopyFrom(event.status) still wipes out the existing timestamp and writes last_updated = NULL to the DB.

  • New SUBMITTED tasks created via new_task_from_user_message or TaskManager._init_task_obj (which ensure_task_id saves directly through _save_task) also start without a timestamp, so tasks created with return_immediately=True or artifact-first flows end up with last_updated = NULL too.

I think we should stamp any status that lacks a timestamp (in save_task_event and _init_task_obj / _save_task), not just terminal ones

status=TaskStatus(state=TaskState.TASK_STATE_FAILED),
status=TaskStatus(
state=TaskState.TASK_STATE_FAILED,
timestamp=_now_timestamp(),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a rebase onto main. #1366 already removed the failure write in _run_producer and updated _write_failed_status to set status.timestamp.FromDatetime(datetime.now(timezone.utc)) directly.

After rebasing, only _mark_task_as_failed (and cancel, see below) will need stamping here, and we can just call .FromDatetime(datetime.now(timezone.utc)) directly instead of adding _now_timestamp().

Comment on lines 928 to 930
updated = Task()
updated.CopyFrom(task)
updated.status.state = TaskState.TASK_STATE_CANCELED

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are still missing timestamps on the CANCELED paths:

  • In ActiveTask.cancel(), updated.CopyFrom(task) copies the old timestamp from the previous state (like WORKING or INPUT_REQUIRED) and only changes state to CANCELED. Since timestamp is already set on the copy, _stamp_terminal_timestamp in TaskManager won't update it, so the task keeps the stale timestamp.

  • In DefaultRequestHandlerV2._write_cancel(), we call _versioned_store.save() directly without going through TaskManager, so cancelled.status.timestamp is either left empty or keeps the old timestamp.

Both places should explicitly set status.timestamp.FromDatetime(datetime.now(timezone.utc)) when transitioning to CANCELED.

Comment on lines 293 to +294
task.status.CopyFrom(event.status)
self._stamp_terminal_timestamp(task)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we stamp task after task.status.CopyFrom(event.status), event.status.timestamp stays empty. That same event object is then sent to streaming subscribers, push notifications, and the task_events table without a timestamp. If we stamp event.status before CopyFrom, both the event and the task get the same timestamp.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Framework-written FAILED status drops status.timestamp, persisting last_updated as NULL

2 participants