Repository navigation
Conversation
On resume the runtime re-entered both spans as NonRecordingSpans, so the scores set on them were dropped and the evaluator span lost execution.id. Re-open them as recording spans under the saved ids and start time so the trace upsert replaces the suspended job's empty output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
github-actions
Bot
added
test:uipath-langchain
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 506ade2500
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| cast(Tracer, tracer).id_generator = _SavedIdGenerator( | ||
| int(saved["trace_id"], 16), int(saved["span_id"], 16) | ||
| ) |
There was a problem hiding this comment.
In a long-lived worker, UiPathTraceManager reuses the global tracer provider across jobs, so assigning a fixed generator to this cached tracer persists after the resumed evaluation completes. A later non-resume job does not replace the generator because saved is false, causing its Eval Set Run and Evaluation spans to reuse the prior evaluation's trace/span IDs; this can overwrite prior trace data and even give a child the same span ID as its parent. Avoid mutating the shared tracer or restore its original generator after creating the resumed span.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🟡 Changes recommended
Resuming beneath a different active caller span can change the restored trace ID and parent relationship.
1 open issue
What changed in this PR
Restores recording evaluation spans after suspension so resumed scores reach the trace backend.
Changes:
- Adds shared span persistence and restoration, including saved start times.
- Adds resume tracing tests and updates the test tracer.
- Bumps
uipathto2.14.42.
| File | Description |
|---|---|
packages/uipath/uv.lock |
Updates locked package version. |
packages/uipath/tests/cli/eval/test_eval_tracing_integration.py |
Allows additional tracer arguments. |
packages/uipath/tests/cli/eval/test_eval_resume_spans.py |
Tests resumed scores, span identity, and execution IDs. |
packages/uipath/src/uipath/eval/runtime/runtime.py |
Uses the shared resumable-span helper. |
packages/uipath/src/uipath/eval/runtime/_resumable_spans.py |
Persists and reopens recording spans. |
packages/uipath/pyproject.toml |
Bumps the package version. |
Review effort: Balanced
| with tracer.start_as_current_span( | ||
| name, attributes=attributes, start_time=start_time |
|
When an eval item suspends on a process tool or API workflow tool call and resumes in a second job, the evaluator runs and scores but the scores never reach the trace. The
Evaluationspan has nooutput.scores, theEvaluation Set Runspan stays at{"scores": {}}, and theEvaluator:span is missingexecution.id. Teodor hit this in cloud: items bed87113 and c3f157bb in the shared trace.Root cause
On resume
_restore_parent_spanrebuilt both spans as aNonRecordingSpanfrom the saved ids and entered them withuse_span(..., end_on_exit=False). Because of that:configure_evaluation_spanandconfigure_eval_set_run_spancallset_attributeon a non-recording span, which is a no-op, so the scores are droppedUiPathExecutionTraceProcessorMixinonly copiesexecution.idfrom a recording parent, so the evaluator span gets noneFix
New
_resumable_spans.pywithstart_resumable_span. On resume it starts a real recording span through a tracer whoseid_generatorreturns the saved trace and span id, with the savedstart_time. The LlmOps upsert is keyed on span id, so the resumed job overwrites the suspended job's row with the scored one. The agent spans already rely on this, which is why the agent run span in the trace has its start before the suspend and its end after the resume. With theEvaluationspan recording again,execution.idreaches the evaluator span the normal way.The saved context now stores
start_timeas well. A context saved by an older version still works, its start time just becomes the resume time.The four storage helpers on
UiPathEvalRuntimemoved into the new module, andruntime.pyuses one_resumable_spancall for both spans.Tests
test_eval_resume_spans.pyruns a suspend job and then a resume job against shared in-memory storage and anInMemorySpanExporter. It checks that both spans report scores, keep their original span id, trace id, parent and start time, and that the evaluator span carries the evaluation'sexecution.id. All four failed before the fix. The fake tracer intest_eval_tracing_integration.pynow acceptsstart_time, matching the SDK signature.Not in this PR
rootspan with a new id. Cosmetic, separate change.UPDATE_EVAL_SET_RUNwith empty scores, buthandle_update_eval_set_runskips it because the eval set run id is user provided, so I left it alone.Related: #1949 (empty
AgentRunHistoryfrom the same trace).🤖 Generated with Claude Code