fix: Feature Idea: Deterministic arbitration/escrow for agent-to-agent transactions - #6792
Open
ojassharma7 wants to merge 2 commits into
Open
fix: Feature Idea: Deterministic arbitration/escrow for agent-to-agent transactions#6792ojassharma7 wants to merge 2 commits into
ojassharma7 wants to merge 2 commits into
Conversation
…t transactions Closes crewAIInc#6782
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughChangesDeterministic arbitration validates task deliverables against Pydantic contracts, deadlines, and optional CEL rules. It returns structured approval or dispute results, integrates with task guardrails and events, exposes public APIs, and includes tests and documentation. Deterministic arbitration
Sequence Diagram(s)sequenceDiagram
participant Task
participant ArbitrationGuardrail
participant ArbitrationEngine
participant Contract
Task->>ArbitrationGuardrail: submit TaskOutput
ArbitrationGuardrail->>ArbitrationEngine: evaluate payload
ArbitrationEngine->>Contract: validate payload
Contract-->>ArbitrationEngine: validated payload or violations
ArbitrationEngine-->>ArbitrationGuardrail: approved or disputed result
ArbitrationGuardrail-->>Task: serialized output or retry instructions
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a deterministic “arbitration/escrow” mechanism for agent deliverables, allowing task outputs to be validated against hard constraints (Pydantic contracts, optional CEL rules, and deadlines) before a task is considered resolved.
Changes:
- Adds a new
ArbitrationEngine+ArbitrationGuardrailimplementation for deterministic validation/dispute outcomes. - Exposes the new arbitration types at the
crewaitop-level API and integrates them into guardrail-start event classification. - Documents the new guardrail type and adds a dedicated test suite covering approval/dispute cases, CEL rule evaluation, and retry behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/edge/en/concepts/tasks.mdx | Documents deterministic arbitration guardrails as a new guardrail type with an example. |
| lib/crewai/src/crewai/init.py | Re-exports arbitration engine/guardrail/result types as part of the public API. |
| lib/crewai/src/crewai/events/types/llm_guardrail_events.py | Classifies ArbitrationGuardrail in guardrail-start events (guardrail_type="arbitration"). |
| lib/crewai/src/crewai/tasks/arbitration.py | Implements deterministic arbitration evaluation (Pydantic validation, optional CEL rules, deadlines) and a Task guardrail wrapper. |
| lib/crewai/tests/test_arbitration.py | Adds tests for approved/disputed outcomes, invalid JSON handling, deadline disputes, CEL rule pass/fail, and retry integration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+255
to
+259
| from typing import cast | ||
|
|
||
| from celpy import Environment | ||
| from celpy.adapter import CELJSONEncoder, json_to_cel | ||
| from celpy.evaluation import Context |
| parts.append(f"deadline={self.deadline.isoformat()}") | ||
| return " ".join(parts) | ||
|
|
||
| def __call__(self, task_output: TaskOutput): |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (3)
lib/crewai/src/crewai/tasks/arbitration.py (2)
262-271: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueOptional: cache the compiled CEL programs.
_evaluate_rulesbuilds a newEnvironmentand recompiles every rule on eachevaluatecall. A guardrail with fixed rules recompiles on every task retry. Consider compiling once per rule set and caching the programs onArbitrationEngine.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/tasks/arbitration.py` around lines 262 - 271, Update ArbitrationEngine._evaluate_rules to cache compiled CEL programs for each fixed rule set instead of creating a new Environment and recompiling rules on every evaluation. Store and reuse the compiled programs on the ArbitrationEngine instance, while preserving rule trimming, empty-rule skipping, and evaluation behavior.
347-347: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAnnotate the
__call__return type for the guardrail callable contract.
ArbitrationGuardrail.__call__is the callable thatTask(guardrail=...)passes as the guardrail, and it returns a(bool, str)tuple.GuardrailCallablehas the same signature, so useGuardrailCallableortuple[bool, str | TaskOutput]to express the contract.♻️ Proposed change
- def __call__(self, task_output: TaskOutput): + def __call__(self, task_output: TaskOutput) -> tuple[bool, str | TaskOutput]:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/tasks/arbitration.py` at line 347, Update ArbitrationGuardrail.__call__ to annotate its return type with the existing GuardrailCallable contract, or the equivalent tuple[bool, str | TaskOutput] type, while preserving its current behavior.Source: Coding guidelines
docs/edge/en/concepts/tasks.mdx (1)
374-377: 🗄️ Data Integrity & Integration | 🔵 TrivialClarify the external payment boundary.
ArbitrationGuardrail.__call__only validates the task output and returns a result. It does not bind approval to a transaction or release funds. Document that the caller must persist the decision, bind it to a transaction or version, and use replay-safe, idempotent payment handling.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/edge/en/concepts/tasks.mdx` around lines 374 - 377, Update the outcomes and release-gate documentation near ArbitrationGuardrail.__call__ to state that the method only validates task output and returns a result; the caller must persist the decision, bind it to the relevant transaction or task version, and use replay-safe, idempotent handling when releasing funds through an external payment rail.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/edge/en/concepts/tasks.mdx`:
- Around line 374-376: Update the Outcomes description in the task concepts
documentation to replace “field-level violations” with “structured violations,”
while preserving the listed payload-level, deadline, and validation error
examples.
- Around line 356-367: Align the HotelBooking price constraint with the task
wording by enforcing a strict upper bound: change the price_per_night field in
HotelBooking to use lt=180 and update the ArbitrationGuardrail rule to
price_per_night < 180.0. Preserve the existing “under $180/night” description.
- Around line 355-370: Update the HotelBooking contract and escrow_guardrail so
the non-smoking requirement is enforceable: add a smoking-status field with an
explicit non-smoking constraint and include the corresponding guardrail rule, or
remove “non-smoking” from booking_task.description. Keep the task description,
output schema, and guardrail rules consistent.
- Around line 360-364: Update the escrow_guardrail setup around
ArbitrationGuardrail so its deadline is calculated at handoff or execution start
rather than when the task configuration is constructed. Create the guardrail
through a function that receives the handoff timestamp, or reuse the task
transaction’s execution timestamp, and preserve the one-hour deadline window.
- Around line 355-363: Update HotelBooking.price_per_night to enforce a
non-negative value with a Pydantic lower bound of zero, and extend
escrow_guardrail’s CEL rules with the corresponding price_per_night >= 0
constraint while preserving the existing upper-bound validation.
In `@lib/crewai/tests/test_arbitration.py`:
- Around line 139-167: Extend the arbitration tests around
`test_evaluate_accepts_task_output_json_dict` with approval cases for
`TaskOutput.pydantic` and valid JSON supplied through `TaskOutput.raw`,
preserving the same `CodeReviewOutputContract` behavior. In
`test_guardrail_call_approves_and_returns_json`, deserialize `value` with
`json.loads` and assert it equals `valid_payload` instead of checking for a
serialized substring.
- Around line 66-84: Update test_evaluate_disputes_invalid_payload to verify
each invalid field produces its expected structured contract violation rather
than only asserting that at least one violation exists. Assert violations for
the invalid task_id, summary, confidence_score, blocking_issues_found, status,
and unexpected_field while preserving the existing disputed, approval, and
retry-instruction assertions; alternatively, parameterize isolated invalid
payloads to test each constraint independently.
---
Nitpick comments:
In `@docs/edge/en/concepts/tasks.mdx`:
- Around line 374-377: Update the outcomes and release-gate documentation near
ArbitrationGuardrail.__call__ to state that the method only validates task
output and returns a result; the caller must persist the decision, bind it to
the relevant transaction or task version, and use replay-safe, idempotent
handling when releasing funds through an external payment rail.
In `@lib/crewai/src/crewai/tasks/arbitration.py`:
- Around line 262-271: Update ArbitrationEngine._evaluate_rules to cache
compiled CEL programs for each fixed rule set instead of creating a new
Environment and recompiling rules on every evaluation. Store and reuse the
compiled programs on the ArbitrationEngine instance, while preserving rule
trimming, empty-rule skipping, and evaluation behavior.
- Line 347: Update ArbitrationGuardrail.__call__ to annotate its return type
with the existing GuardrailCallable contract, or the equivalent tuple[bool, str
| TaskOutput] type, while preserving its current behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e52d0b79-26a4-48c8-8c4f-98b59ee6ed66
📒 Files selected for processing (5)
docs/edge/en/concepts/tasks.mdxlib/crewai/src/crewai/__init__.pylib/crewai/src/crewai/events/types/llm_guardrail_events.pylib/crewai/src/crewai/tasks/arbitration.pylib/crewai/tests/test_arbitration.py
Comment on lines
+360
to
+364
| escrow_guardrail = ArbitrationGuardrail( | ||
| HotelBooking, | ||
| rules=["output.price_per_night <= 180.0"], | ||
| deadline=datetime.now(timezone.utc) + timedelta(hours=1), | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate file and surrounding context =="
if [ -f docs/edge/en/concepts/tasks.mdx ]; then
sed -n '320,390p' docs/edge/en/concepts/tasks.mdx | cat -n
else
fd -i 'tasks.mdx' .
fi
echo "== Search for ArbitrationGuardrail usage and deadline semantics =="
rg -n "ArbitrationGuardrail|deadline|handoff|task execution|execution" docs/edge/en/concepts/tasks.mdx . --glob '!dist/**' --glob '!build/**' | sed -n '1,220p'
echo "== Search for source definition if present in repo =="
fd -i "guardrail|arbitration" . | sed -n '1,80p'
rg -n "class ArbitrationGuardrail|def .*deadline|deadline" . --glob '!dist/**' --glob '!build/**' | sed -n '1,220p'Repository: crewAIInc/crewAI
Length of output: 36553
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Arbitration implementation =="
sed -n '288,355p' lib/crewai/src/crewai/tasks/arbitration.py | cat -n
sed -n '60,132p' lib/crewai/src/crewai/tasks/arbitration.py | cat -n
echo "== Guardrail evaluation call sites =="
rg -n "guardrail\s*=|if guardrail|guardrail\(" lib/crewai/src/crewai -g '*.py' | sed -n '1,220p'
rg -n "ArbitrationGuardrail|evaluate\(" lib/crewai/src/crewai/src lib/crewai/src/crewai -g '*.py' | sed -n '1,220p'
echo "== Guardrail tests around deadline semantics =="
sed -n '80,115p' lib/crewai/tests/test_arbitration.py | cat -n
echo "== Deterministic probe: guardrail configuration vs evaluation time =="
python3 - <<'PY'
from datetime import datetime, timedelta, timezone
deadline_configured_at = datetime(2020_01_01, 12, 0, tzinfo=timezone.utc)
deadline = deadline_configured_at + timedelta(hours=1)
task_executes_at = deadline + timedelta(minutes=10)
print("constructed deadline:", deadline == deadline_configured_at + timedelta(hours=1))
print("evaluation after deadline:", task_executes_at > deadline)
print("fixed deadline allows 70 minutes after construction:", task_executes_at - deadline_configured_at)
print("relative handoff deadline from execution would be:", timedelta(hours=1), "<", (task_executes_at - task_executes_at).total_seconds())
PYRepository: crewAIInc/crewAI
Length of output: 13919
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Guardrail dispatcher and arbitration callable =="
sed -n '115,185p' lib/crewai/src/crewai/utilities/guardrail.py | cat -n
sed -n '355,385p' lib/crewai/src/crewai/tasks/arbitration.py | cat -n
echo "== Guardrail tests about miss timing =="
sed -n '88,110p' lib/crewai/tests/test_arbitration.py | cat -n
echo "== Deterministic probe of construction-bound deadline behavior =="
python3 - <<'PY'
from datetime import datetime, timedelta, timezone
constructed_at = datetime(2026, 1, 1, 0, 0, tzinfo=timezone.utc)
fixed_deadline = constructed_at + timedelta(hours=1)
handoff_and_submission = fixed_deadline + timedelta(minutes=10)
print({
"fixed_deadline": fixed_deadline.isoformat(),
"handoff_and_submission": handoff_and_submission.isoformat(),
"elapsed_between_configuration_and_handoff_minutes": (handoff_and_submission - constructed_at).total_seconds() / 60,
"deadline_missed_by_runtime_logic": handoff_and_submission > fixed_deadline,
})
PYRepository: crewAIInc/crewAI
Length of output: 4813
Compute the deadline from handoff time.
datetime.now(timezone.utc) + timedelta(hours=1) is stored when escrow_guardrail is constructed. ArbitrationGuardrail passes that fixed deadline to evaluate, so a 10-minute delay between task configuration and handoff submission triggers deadline_missed. Pass the handoff/execution start timestamp into a function that creates the guardrail, or use a timestamp from the task transaction instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/edge/en/concepts/tasks.mdx` around lines 360 - 364, Update the
escrow_guardrail setup around ArbitrationGuardrail so its deadline is calculated
at handoff or execution start rather than when the task configuration is
constructed. Create the guardrail through a function that receives the handoff
timestamp, or reuse the task transaction’s execution timestamp, and preserve the
one-hour deadline window.
Comment on lines
+66
to
+84
| def test_evaluate_disputes_invalid_payload(engine: ArbitrationEngine) -> None: | ||
| bad_payload = { | ||
| "task_id": "not-a-uuid", | ||
| "status": "done", | ||
| "summary": "too short", | ||
| "confidence_score": 1.4, | ||
| "files_reviewed": [], | ||
| "blocking_issues_found": -3, | ||
| "unexpected_field": "hack attempt", | ||
| } | ||
|
|
||
| result = engine.evaluate(CodeReviewOutputContract, bad_payload) | ||
|
|
||
| assert result.status is ArbitrationStatus.DISPUTED | ||
| assert not result.is_approved | ||
| assert len(result.violations) >= 1 | ||
| instructions = result.to_retry_instructions() | ||
| assert "DISPUTED" in instructions | ||
| assert "constraint=" in instructions |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Assert each contract violation.
Line 81 passes if any one invalid field still causes a dispute. It does not verify the required-field, range, custom-status, or extra-field contract checks independently.
Assert the expected violation fields, or parameterize isolated invalid payloads. This verifies each hard constraint and its structured dispute output.
Proposed assertion
assert result.status is ArbitrationStatus.DISPUTED
assert not result.is_approved
- assert len(result.violations) >= 1
+ assert {
+ violation.field for violation in result.violations
+ } >= {
+ "status",
+ "summary",
+ "confidence_score",
+ "files_reviewed",
+ "blocking_issues_found",
+ "unexpected_field",
+ }As per coding guidelines, unit tests for new functionality must focus on behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/tests/test_arbitration.py` around lines 66 - 84, Update
test_evaluate_disputes_invalid_payload to verify each invalid field produces its
expected structured contract violation rather than only asserting that at least
one violation exists. Assert violations for the invalid task_id, summary,
confidence_score, blocking_issues_found, status, and unexpected_field while
preserving the existing disputed, approval, and retry-instruction assertions;
alternatively, parameterize isolated invalid payloads to test each constraint
independently.
Source: Coding guidelines
Comment on lines
+139
to
+167
| def test_evaluate_accepts_task_output_json_dict( | ||
| engine: ArbitrationEngine, valid_payload: dict | ||
| ) -> None: | ||
| task_output = TaskOutput( | ||
| description="Review code", | ||
| agent="reviewer", | ||
| raw="ignored when json_dict is set", | ||
| json_dict=valid_payload, | ||
| ) | ||
|
|
||
| result = engine.evaluate(CodeReviewOutputContract, task_output) | ||
|
|
||
| assert result.is_approved | ||
|
|
||
|
|
||
| def test_guardrail_call_approves_and_returns_json(valid_payload: dict) -> None: | ||
| guardrail = ArbitrationGuardrail(CodeReviewOutputContract) | ||
| task_output = TaskOutput( | ||
| description="Review code", | ||
| agent="reviewer", | ||
| raw=str(valid_payload), | ||
| json_dict=valid_payload, | ||
| ) | ||
|
|
||
| ok, value = guardrail(task_output) | ||
|
|
||
| assert ok is True | ||
| assert '"status": "completed"' in value | ||
| assert "ArbitrationGuardrail(CodeReviewOutputContract)" in guardrail.description |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Cover all supported TaskOutput paths and validate serialized JSON.
This test covers only TaskOutput.json_dict. The arbitration engine also accepts TaskOutput.pydantic and valid JSON in TaskOutput.raw.
Add approval tests for those inputs. Replace the substring check at Line 166 with json.loads(value) == valid_payload to verify the guardrail output contract.
As per coding guidelines, unit tests for new functionality must focus on behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/tests/test_arbitration.py` around lines 139 - 167, Extend the
arbitration tests around `test_evaluate_accepts_task_output_json_dict` with
approval cases for `TaskOutput.pydantic` and valid JSON supplied through
`TaskOutput.raw`, preserving the same `CodeReviewOutputContract` behavior. In
`test_guardrail_call_approves_and_returns_json`, deserialize `value` with
`json.loads` and assert it equals `valid_payload` instead of checking for a
serialized substring.
Source: Coding guidelines
Handle missing celpy as a DISPUTED violation, annotate the guardrail callable, and tighten the docs example constraints. Co-authored-by: Cursor <cursoragent@cursor.com>
Author
|
Addressed the actionable Copilot/CodeRabbit notes:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Suppressed comments (3)
docs/edge/en/concepts/tasks.mdx:348
- The docs introduce optional CEL rules, but don’t mention that CEL evaluation requires an extra dependency (
cel-pythonprovidingcelpy). Without that, any task usingrules=[...]will always dispute with a rule evaluation error. Adding a short install note here would prevent confusing retries for users.
For agent-to-agent handoffs where you need machine-speed, auditable decisions
(schema shape, numeric bounds, deadlines) without LLM hallucination risk, use
`ArbitrationGuardrail`. It evaluates the task output against a Pydantic contract
and optional CEL boolean rules, then returns either an approved payload or
precise dispute feedback the agent can retry against.
lib/crewai/src/crewai/tasks/arbitration.py:71
to_retry_instructions()only appends the "expected" detail whenviolation.expectedis truthy. Ifexpectedis a valid but falsy value (e.g., "0"), it will be omitted from the retry instructions, which makes disputes less actionable.
f"constraint='{violation.constraint}' -> {violation.message}"
)
if violation.expected:
detail += f" (expected: {violation.expected})"
lines.append(detail)
lib/crewai/src/crewai/tasks/arbitration.py:346
ArbitrationEngine._evaluate_rules()treats a missing CEL runtime as a normal deliverable violation. In a Task guardrail context this triggers retries even though the agent can’t fix a missing dependency, wasting attempts/tokens. Consider failing fast duringArbitrationGuardrailinitialization whenrulesare configured butcel-python/celpyis unavailable.
self.contract = contract
self.rules = list(rules) if rules else None
self.deadline = deadline
self.engine = engine or ArbitrationEngine()
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6782.
What changed
docs/edge/en/concepts/tasks.mdxlib/crewai/src/crewai/__init__.pylib/crewai/src/crewai/events/types/llm_guardrail_events.pylib/crewai/src/crewai/tasks/arbitration.pylib/crewai/tests/test_arbitration.pyVerification
The project's own test suite was run before and after this change; it introduces no new test failures or lint violations.