New Worker Design

[nvdtf]

🚀 Dual-Worker Architecture

Architecture Upgrade: Replaces single interval-based worker with dual-worker system (SchedulerHandler + WorkerHandler) enabling parallel request processing and crash recovery

---

✅ Issues Resolved

• FLOW-2: No recovery mechanism for stuck PROCESSING requests #17
• FLOW-3: Cross-VM state desync when completeProcessing fails after successful operation #18
• FLOW-4: ERC20 transfer failure creates permanently failing request #19
• FLOW-6: Unfinalized Processing Requests Can Wedge the Global Queue #21
• S2: Scheduler handler selection assumes single handler type #33
• [FEATURE] Implement Parallel Execution for Flow Scheduler Transactions #12
• [BUG] Unstuck the worker in the supervisor and worker setup #10
• Optimize Scheduled Transactions Computation Limit #55

---

📋 Summary

This PR migrates the FlowYieldVaultsEVM worker design from a single interval-based worker that processed one request at a time to a dual-worker architecture. The new system enables parallel processing of multiple requests with automatic scheduling and crash recovery.

---

🔄 What Changed

Before

• Single interval-based worker: Processed requests sequentially
  • One PENDING request processed per execution
  • Fixed interval between executions
  • Sequential, one-at-a-time processing
  • No panic recovery

After

• Dual-worker architecture:
  • SchedulerHandler: Recurrent worker that runs at fixed intervals (1.0s)
    • Crash recovery for failed WorkerHandlers
    • Checks for pending requests
    • Preprocesses requests (PENDING → PROCESSING)
    • Spawns 0 to multiple WorkerHandlers based on available capacity and pending requests
    • Passes request ID to each worker as assigned work
  • WorkerHandler: Per-request worker that processes individual requests
    • Each request gets its own scheduled WorkerHandler
    • Processes request and finalizes status on EVM
    • Removes itself from tracking after completion
    • Safe to panic, will be recovered by SchedulerHandler
• Parallel processing: Up to maxProcessingRequests (default: 3) requests processed concurrently
• FlowTransactionScheduler: Native Flow scheduling infrastructure for automatic execution

---

✨ New Features

Feature	Description
⚡ Parallel Processing	Multiple requests processed concurrently (up to maxProcessingRequests)
🔄 Dual-Worker System	SchedulerHandler orchestrates, WorkerHandler executes
🔍 Preprocessing	SchedulerHandler validates requests before scheduling (PENDING → PROCESSING)
🔁 Crash Recovery	Detects panicked WorkerHandlers and marks requests as FAILED
📊 Capacity Control	Configurable concurrent processing limit (maxProcessingRequests: 3)
🎯 Sequential Offsetting	Multiple requests from same EVM address offset sequentially to avoid randomization

---

🏗️ Architecture Changes

Before: Single Worker

┌─────────────────────────────────────┐
│   Single Interval-Based Worker      │
│   - Runs at fixed interval          │
│   - Processes 1 request at a time   │
│   - Sequential execution            │
└─────────────────────────────────────┘

After: Dual-Worker System

┌─────────────────────────────────────────────────────────────────────────────┐
│                              Flow EVM                                       │
│  ┌──────────────┐         ┌───────────────────────────┐                     │
│  │   EVM User   │────────▶│  FlowYieldVaultsRequests  │                     │
│  │              │◀────────│   (Request Queue + Escrow)│                     │
│  └──────────────┘         └─────────────┬─────────────┘                     │
└─────────────────────────────────────────┼───────────────────────────────────┘
                                          │ COA Bridge
┌─────────────────────────────────────────┼───────────────────────────────────┐
│                              Flow Cadence                                   │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │                       FlowYieldVaultsEVM (Worker)                     │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │     FlowYieldVaultsEVMWorkerOps (NEW)                                 │  │
│  │  ┌─────────────────────┐    ┌─────────────────────────────────────┐   │  │
│  │  │   SchedulerHandler  │───▶│         WorkerHandler               │   │  │
│  │  │   (Recurrent)       │    │   (Per-request, scheduled)          │   │  │
│  │  │   - Crash recovery  │    │   - Processes 1 request             │   │  │
│  │  │   - Checks queue    │    │   - Finalizes status                │   │  │
│  │  │   - Preprocesses    │    │                                     │   │  │
│  │  │   - Spawns 0-N      │    │                                     │   │  │
│  │  │     WorkerHandlers  │    │                                     │   │  │
│  │  └─────────────────────┘    └─────────────────────────────────────┘   │  │
│  │                                                                       │  │
│  │  State: scheduledRequests, isSchedulerPaused                          │  │
│  │  Config: schedulerWakeupInterval (1s), maxProcessingRequests (3)      │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────┘

Request Processing Flow:

1. User submits request → FlowYieldVaultsRequests escrows funds (status: PENDING)
2. SchedulerHandler (runs every 1.0s):
   • Perform crash recovery for failed WorkerHandlers
   • Calculates available capacity: maxProcessingRequests - current_in_flight
   • Fetches pending requests from EVM
   • Calls preprocessRequests() to validate and transition PENDING → PROCESSING
   • Spawns 0 to N WorkerHandlers (one per request, up to capacity limit)
3. WorkerHandler (scheduled per request):
   • Processes single request via processRequest()
   • Executes Cadence operation (create/deposit/withdraw/close YieldVault)
   • Calls completeProcessing() to mark COMPLETED/FAILED on EVM
   • Removes itself from scheduledRequests tracking

---

🧪 Testing

New Test Suite: run_worker_tests.sh

Comprehensive E2E tests covering:

• ✅ SchedulerHandler initialization and auto-scheduling
• ✅ Automated request processing via FlowTransactionScheduler
• ✅ Parallel processing (multiple WorkerHandlers running concurrently)
• ✅ Capacity limits (maxProcessingRequests respected)
• ✅ Pause/unpause scheduler functionality
• ✅ Stop all scheduled transactions
• ✅ Multi-user automated processing
• ✅ Crash recovery for panicked WorkerHandlers
• ✅ Sequential offsetting for same-user requests

Run Tests:

./local/setup_and_run_emulator.sh && ./local/deploy_full_stack.sh
./local/run_worker_tests.sh

• ✅ New CI test added that executes ./local/run_worker_tests.sh.

---

📖 Documentation Updates

• Updated README with dual-worker architecture diagram
• Updated design document with SchedulerHandler/WorkerHandler flow
• Added worker tests documentation
• Updated configuration tables

[liobrasil]

LGTM