Implement multi-agent composition and sub-workflow invocation

[rogue-socket]

Summary

Multi-agent composition is the 0.3.0 flagship feature. A workflow step should be able to invoke a sub-workflow or delegate to another agent, enabling orchestrator-specialist patterns. The registries and executor loop are ready; state isolation, storage schema, and event hierarchy are not.

Current Agent Execution Flow

Executor.__execute_steps_loop()  (core.py:810)
  → step_type == "agent"
  → _dispatch_agent()  (core.py:449)
    → AgentRegistry.get(agent_id, version)  (core.py:467)
    → AgentExecutor(definition, llm_client, tool_registry, prompt_registry)  (core.py:472)
    → AgentContext(run_id, step_id, state, emit)  (core.py:473-481)
    → agent_executor.execute(context)  (core.py:484-490)
      → resolve_strategy()  (strategies.py:830-843)
      → strategy.run()  →  _run_pipeline()  (strategies.py:436-597)
        → model steps: LLM call + tool dispatch
        → tool steps: direct tool call
      → AgentResult(outputs, trace, token_usage)

Existing TODOs

Composition: core.py:1267-1269

# TODO(roadmap): Multi-agent composition: allow a step to invoke
#   a sub-workflow or delegate to another agent definition.

Pipeline-level nesting: definition.py:57-58

# TODO(roadmap): Consider adding "agent" pipeline step type for nested agent calls

Fan-out: core.py:1255-1266 — DAG scheduler design outlined in comments

Readiness Assessment

Component	Ready?	Details
WorkflowRegistry	✅	Resolves by id+version, from_directory() scans
AgentRegistry	✅	Full bidirectional lookup
Executor loop	⚠️	Async-safe, but re-entrancy untested
Step dispatch	⚠️	Can add _dispatch_workflow() alongside existing dispatchers
State namespace	❌	Flat steps.<step_id> — needs hierarchical prefixing
Storage schema	❌	No parent_run_id column on runs table
Event hierarchy	⚠️	Events work but lack nesting context
Replay	❌	RunReplayer handles single runs only
Tests	❌	Zero composition tests

Re-Entrancy Risks

Concern	Status	Risk
Event loop	SAFE	asyncio.run() not used internally
SQLite connection	NEEDS DESIGN	_in_transaction flag is instance-level; nested calls absorbed
LLM client	SAFE BUT SCOPED	_run_usage[run_id] keyed by run_id
Event callbacks	AT RISK	Single on_event — no hierarchy context in payloads

State Isolation Problem

Concern	Current	Needed
Output namespace	Both write to steps.<step_id>	Prefix: steps.<parent_step>.<child_step>
Input passing	Sub-agent sees full parent state	Explicit input contract (declared in YAML)
Run records	No parent-child relationship	parent_run_id FK on runs table
State versions	Single chain per run	Per-sub-run version chains
Resume safety	Flat step scanning	Recursive sub-run scanning

Storage Schema Changes Needed

ALTER TABLE runs ADD COLUMN parent_run_id TEXT REFERENCES runs(id);
ALTER TABLE runs ADD COLUMN nesting_depth INTEGER DEFAULT 0;
ALTER TABLE runs ADD COLUMN execution_path TEXT;

Proposed YAML Syntax

steps:
  - id: prepare
    type: function
    handler: prepare_data
    outputs: [extracted_data]

  - id: analyze
    type: workflow
    workflow: specialist_workflow@v2
    inputs:
      data: steps.prepare.extracted_data
    outputs: [results, metrics]

  - id: summarize
    type: agent
    agent: summarizer
    inputs:
      analysis: steps.analyze.results

Implementation Phases

Phase 1: MVP (~3-4 weeks)

1. Add workflow to VALID_STEP_TYPES
2. New _dispatch_workflow() method
3. parent_run_id column + schema migration
4. Flat output nesting under parent step
5. Basic integration tests

Phase 2: Production-Ready (~2-3 weeks)

6. Input contracts for sub-workflows
7. Nesting depth guard (max depth = 5)
8. Event hierarchy enrichment
9. Replay support for nested runs
10. CLI inspect rendering of sub-runs
11. Reference example workflow

Phase 3: Fan-Out Integration (Future)

12. Parallel sub-workflow invocation
13. Partial failure handling
14. Atomic result merging

Priority

P2 — 0.3.0 flagship. ~5-6 weeks total. Depends on storage schema migration and state isolation design.

---

🤖 Generated with Claude Code