fix(telemetry): record task and crew failures instead of reporting them as OK

[joaomdmoura]

Why

error_count is zero for every month in crew_task_executions_daily_target — going back to at least Sept 2025, across ~230M task executions. That isn't a low error rate, it's broken instrumentation.

Root cause: failures were being recorded as successes.

Three separate defects

1. Task failures were recorded as OK.

TaskFailedEvent routed to Telemetry.task_ended, which calls close_span():

def close_span(span: Span) -> None:
    span.set_status(Status(StatusCode.OK))   # unconditional
    span.end()

Every failed task was exported with otel.status_code: OK. No downstream query could ever count one — which is exactly what the zeros show.

2. Crew failures weren't recorded at all, and leaked their span.

on_crew_failed never touched telemetry. A crew that raised left _execution_span open — never ended, never exported. The failure was invisible and the span was lost.

3. Some task failures leaked their span too.

on_task_failed only ended the span when source.agent.crew was present. A task failing without one got popped from the span map and never closed.

Changes

• close_span_with_error() — sets StatusCode.ERROR, optionally records error_type.
• Telemetry.task_failed() and Telemetry.crew_failed(). crew_failed clears _execution_span so it can't be double-closed, and no-ops safely when no span exists (share_crew=False).
• Wire both failure events to them, closing spans unconditionally so neither path can leak.
• Optional error_type on TaskFailedEvent and CrewKickoffFailedEvent, populated with type(e).__name__ at the four emit sites. Defaults to None — existing callers unaffected.

Privacy

Only the exception class name is recorded, never the message. TaskFailedEvent.error already carries str(e), which routinely contains prompts, model output, and credentials — that field is untouched and never reaches telemetry.

close_span_with_error drops any value failing str.isidentifier(), so a message can't be recorded even if passed by mistake. Every builtin exception name is a valid identifier, so nothing legitimate is lost. Tested against six message-shaped inputs including one containing sk-live-1234.

Verification

End-to-end through the real telemetry path — two successful tasks and one TimeoutError whose message contained a fake API key:

total=3 errors=1  <- error_count is now countable
recorded error_type: TimeoutError
secret leaked anywhere? False

Testing

New tests/telemetry/test_failure_instrumentation.py — 16 tests covering error status, the success/failure distinction, the PII guard, span-leak regressions for both task and crew, and event backwards compatibility.

• tests/telemetry/: 47 passed, 1 skipped, stable across repeated randomized runs.
• tests/events/: 184 passed, 2 failed — both pre-existing on main (verified by stashing).
• tests/utilities/test_events.py: 46 passed.
• tests/a2a/: collection error from a missing optional a2a module, pre-existing.
• Ruff and mypy clean.

One thing worth knowing for future telemetry tests: the suite runs with OTEL_SDK_DISABLED=true, and the root conftest pops it on teardown. So only the first test in a session sees a disabled SDK and silently gets non-recording spans. This module sets it explicitly rather than depending on that leak.

Not included: total_duration_ms

Also zero, but it's a pipeline-side bug, not an SDK one. The data is there — the raw duration column is a Go-style string:

duration: "2.026641s"   (99.99% of rows use the `s` suffix; 21 in 200k are empty)

toInt64OrZero("2.026641s") returns 0, so the MV sums to nothing. The fix belongs in the ClickHouse materialized views:

toFloat64OrZero(replaceRegexpOne(duration, 's$', '')) * 1000  AS duration_ms

I can't deploy that from here — flagging it so it gets picked up alongside this.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UNumDnNbiyw3pv1WakAe6t

---

Note

Medium Risk
Changes how execution spans are closed on failure paths in telemetry and the event listener; behavior is well-tested but affects observability pipelines that depend on span status and export.

Overview
Fixes broken failure metrics by closing OpenTelemetry spans with ERROR instead of OK, and by ending spans that were previously leaked or never touched on crew kickoff failure.

Adds close_span_with_error(), which sets StatusCode.ERROR and optionally records error_type only when the value passes isidentifier() (so error messages with prompts or secrets never reach telemetry). Telemetry.task_failed() and Telemetry.crew_failed() use this helper; crew failure also clears _execution_span after close.

TaskFailedEvent and CrewKickoffFailedEvent gain optional error_type (defaults None); kickoff and task emit sites set type(e).__name__. The event listener routes task failures to task_failed unconditionally (no longer gated on source.agent.crew) and calls crew_failed on CrewKickoffFailedEvent.

New tests/telemetry/test_failure_instrumentation.py covers error vs success spans, PII guard, span-leak regressions, and backwards compatibility.

^{Reviewed by Cursor Bugbot for commit 328e0e3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Crew and task failure events now include exception class names. Telemetry marks failed spans with error status, records valid error types, clears crew spans, and avoids exception messages. Regression tests cover propagation, filtering, cleanup, and compatibility.

Changes

Failure telemetry instrumentation

Layer / File(s)	Summary
Failure event contracts and emission lib/crewai/src/crewai/events/types/*, lib/crewai/src/crewai/crew.py, lib/crewai/src/crewai/task.py	Failure events now define optional error_type fields. Synchronous and asynchronous crew and task execution populate these fields with exception class names.
Failed span handling lib/crewai/src/crewai/telemetry/utils.py, lib/crewai/src/crewai/telemetry/telemetry.py, lib/crewai/src/crewai/events/event_listener.py	Telemetry closes failed task and crew spans with error status, records valid exception types, clears crew execution spans, and handles missing spans safely.
Failure instrumentation validation lib/crewai/tests/telemetry/test_failure_instrumentation.py	Tests validate span status, error-type filtering, span cleanup, failure event propagation, sensitive message exclusion, and None defaults.

Sequence Diagram(s)

sequenceDiagram
  participant CrewOrTaskExecution
  participant FailureEvent
  participant EventListener
  participant Telemetry
  participant Span
  CrewOrTaskExecution->>FailureEvent: emit failure with error_type
  FailureEvent->>EventListener: deliver failure event
  EventListener->>Telemetry: call crew_failed or task_failed
  Telemetry->>Span: set error status and record valid error_type
  Telemetry->>Span: end failed span

Loading

Suggested reviewers: copilot, greysonlalonde

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 69.57% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the primary telemetry fix for task and crew failures.
Description check	✅ Passed	The description directly explains the telemetry defects, implemented fixes, privacy safeguards, and tests.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/task-crew-failure-telemetry

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Fixes telemetry instrumentation so task and crew failures are exported as failures (ERROR) instead of being reported as OK or not exported due to leaked spans, enabling downstream error counting and safer failure attribution via error_type.

Changes:

• Add close_span_with_error() and route task/crew failure paths to close spans with StatusCode.ERROR, optionally recording a sanitized error_type.
• Extend failure events (TaskFailedEvent, CrewKickoffFailedEvent) with optional error_type and populate it at emit sites.
• Add regression tests covering failure status, PII guard behavior, and span-leak prevention.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
lib/crewai/src/crewai/telemetry/utils.py	Adds close_span_with_error() to end spans with ERROR and optionally record sanitized error_type.
lib/crewai/src/crewai/telemetry/telemetry.py	Adds Telemetry.task_failed() / Telemetry.crew_failed() to close spans correctly on failures and prevent crew span leaks.
lib/crewai/src/crewai/events/event_listener.py	Wires crew/task failure events to the new telemetry failure methods and closes task spans unconditionally to avoid leaks.
lib/crewai/src/crewai/events/types/task_events.py	Adds optional error_type field to TaskFailedEvent for safe telemetry categorization.
lib/crewai/src/crewai/events/types/crew_events.py	Adds optional error_type field to CrewKickoffFailedEvent for safe telemetry categorization.
lib/crewai/src/crewai/task.py	Populates error_type when emitting TaskFailedEvent from sync/async execution paths.
lib/crewai/src/crewai/crew.py	Populates error_type when emitting CrewKickoffFailedEvent from sync/async kickoff paths.
lib/crewai/tests/telemetry/test_failure_instrumentation.py	Adds regression coverage for failure vs success spans, PII guard, and span-leak scenarios.

Suppressed comments (1)

lib/crewai/src/crewai/task.py:963

• Using raise e re-raises the exception with a new traceback starting at this line, which can hide the original failure location. Prefer a bare raise to preserve the original traceback when propagating the caught exception.

            raise e

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

🧹 Nitpick comments (1)

lib/crewai/tests/telemetry/test_failure_instrumentation.py (1)

1-211: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a wiring-level test for on_task_failed without an agent crew.

The suite tests Telemetry.task_failed in isolation with a Mock task, and it tests close_span_with_error in isolation. It does not test EventListener.on_task_failed itself. This handler is the exact place where the PR removes the source.agent.crew gate, so add a test that emits a TaskFailedEvent for a task whose agent.crew is None (or falsy) through the event bus and confirms EventListener.execution_spans no longer holds the span, and that the span is closed. This closes the coverage gap for the specific regression this PR fixes.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/crewai/tests/telemetry/test_failure_instrumentation.py` around lines 1 -
211, Add a wiring-level test for EventListener.on_task_failed that publishes a
TaskFailedEvent through the event bus for a task whose agent.crew is None or
falsy. Assert the associated span is removed from EventListener.execution_spans
and finished with error status, covering the handler path without relying on
Telemetry.task_failed directly.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@lib/crewai/tests/telemetry/test_failure_instrumentation.py`:
- Around line 1-211: Add a wiring-level test for EventListener.on_task_failed
that publishes a TaskFailedEvent through the event bus for a task whose
agent.crew is None or falsy. Assert the associated span is removed from
EventListener.execution_spans and finished with error status, covering the
handler path without relying on Telemetry.task_failed directly.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2e8ff7c9-33c8-47ef-8d75-b6e1a3d291b4

📥 Commits

Reviewing files that changed from the base of the PR and between c8f441c and 328e0e3.

📒 Files selected for processing (8)

• lib/crewai/src/crewai/crew.py
• lib/crewai/src/crewai/events/event_listener.py
• lib/crewai/src/crewai/events/types/crew_events.py
• lib/crewai/src/crewai/events/types/task_events.py
• lib/crewai/src/crewai/task.py
• lib/crewai/src/crewai/telemetry/telemetry.py
• lib/crewai/src/crewai/telemetry/utils.py
• lib/crewai/tests/telemetry/test_failure_instrumentation.py