Enhancement: Unify Primary/Fallback LLM Failover Policy Across Execution Paths

[Congregalis]

Summary

The platform defines both primary_model_id and fallback_model_id, but failover behavior is inconsistent across chat/channel/background paths. Some paths only do config-level fallback, some attempt runtime fallback, and some ignore fallback entirely.
This issue proposes one shared failover policy and one shared executor to make behavior consistent and observable.

Current Behavior

1. Model schema supports primary + fallback

primary_model_id: uuid.UUID | None = None
fallback_model_id: uuid.UUID | None = None

2. Web chat does primary-first + config-level fallback

if agent.primary_model_id:
... load primary ...
if agent.fallback_model_id:
... load fallback ...
if not llm_model and fallback_llm_model:
llm_model = fallback_llm_model

3. Runtime fallback exists, but lower layer often returns error text instead of raising

## call_llm catches and returns string
except LLMError as e:
return f"[LLM Error] {e}"
except Exception as e:
return f"[LLM call error] ..."

This weakens outer except-driven fallback logic.

4. Slack/Teams/Discord/WeCom/DingTalk reuse Feishu _call_agent_llm

These paths share the same failover characteristics as Feishu.

5. Background services mostly do one-shot selection (primary or fallback)

model_id = agent.primary_model_id or agent.fallback_model_id

No runtime failover retry after selected model fails.

6. Some paths are primary-only
   Files: backend/app/services/trigger_daemon.py, backend/app/api/gateway.py, backend/app/services/agent_manager.py

Primary is used directly; fallback is not part of the path.

Proposed Solution

A) Add one shared failover executor
Create llm_failover and route all LLM entrypoints through it.

e.g.

async def invoke_with_failover(primary_model, fallback_model, invoke_once, context):
...

B) Unified switching rules

1. Try primary if available.
2. If primary missing/unavailable, use fallback directly.
3. If primary fails with retryable error, retry once on fallback.
4. If error is non-retryable (auth/validation/schema), do not switch.
5. Max attempts per request: 2 (primary + fallback).

C) Retryable error scope (To be discussed)

• Network timeout / connection errors
• Provider 429
• Provider 5xx
• Explicit transient provider errors

[wisdomqin]

Thanks for the thorough analysis — the problem statement is accurate. Here's our assessment with two possible approaches for your consideration:

Option A: Full Refactor (as proposed)

Create a unified invoke_with_failover executor and route all LLM entrypoints through it.

Pros: Cleanest architecture, single source of truth for failover logic.
Cons: High blast radius — requires changing WebSocket handler, all channel handlers (Feishu/Slack/Discord/Teams/DingTalk/WeCom), trigger daemon, gateway, and agent manager simultaneously. Also requires refactoring call_llm to raise exceptions instead of returning error strings, which touches every existing caller.

Option B: Wrapper-based incremental approach (recommended)

Instead of refactoring call_llm internals, add a thin call_llm_with_failover(agent, messages, ...) wrapper that:

1. Calls call_llm with the primary model
2. Inspects the return value — if it starts with [LLM Error] / [LLM call error] and matches retryable patterns (timeout, 429, 5xx), retry with the fallback model
3. Returns the result as-is

Key advantage: call_llm stays untouched, so all existing callers are unaffected. Migration can happen one path at a time:

• PR 1: Add the call_llm_with_failover function (zero risk, no caller changes)
• PR 2: Migrate Web Chat handler (highest user impact)
• PR 3: Migrate channel handlers
• PR 4: Migrate background/trigger paths

Each PR is small, reviewable, and independently rollback-safe.

Additional note

The A2A retry logic recently added in send_message_to_agent (PR #155 follow-up) already implements its own retry + backoff at the call site. Whichever approach we choose should eventually consolidate that logic as well to avoid drift.

Looking forward to your thoughts on which direction to go.

[wisdomqin]

@yaojin3616

[Congregalis]

Thanks for the thorough analysis — the problem statement is accurate. Here's our assessment with two possible approaches for your consideration:

Option A: Full Refactor (as proposed)

Create a unified invoke_with_failover executor and route all LLM entrypoints through it.

Pros: Cleanest architecture, single source of truth for failover logic. Cons: High blast radius — requires changing WebSocket handler, all channel handlers (Feishu/Slack/Discord/Teams/DingTalk/WeCom), trigger daemon, gateway, and agent manager simultaneously. Also requires refactoring call_llm to raise exceptions instead of returning error strings, which touches every existing caller.

Option B: Wrapper-based incremental approach (recommended)

Instead of refactoring call_llm internals, add a thin call_llm_with_failover(agent, messages, ...) wrapper that:

1. Calls call_llm with the primary model
2. Inspects the return value — if it starts with [LLM Error] / [LLM call error] and matches retryable patterns (timeout, 429, 5xx), retry with the fallback model
3. Returns the result as-is

Key advantage: call_llm stays untouched, so all existing callers are unaffected. Migration can happen one path at a time:

• PR 1: Add the call_llm_with_failover function (zero risk, no caller changes)
• PR 2: Migrate Web Chat handler (highest user impact)
• PR 3: Migrate channel handlers
• PR 4: Migrate background/trigger paths

Each PR is small, reviewable, and independently rollback-safe.

Additional note

The A2A retry logic recently added in send_message_to_agent (PR #155 follow-up) already implements its own retry + backoff at the call site. Whichever approach we choose should eventually consolidate that logic as well to avoid drift.

Looking forward to your thoughts on which direction to go.

Totally agree with Option B.

Two additions for consideration:

1. Before switching to fallback, apply guard checks for idempotency and consistency. For example:

• block if a side-effecting tool has already been executed
• block if a fallback switch has already happened once
• block by default if streaming output has already started

2. Add a user-visible flag/event so the caller can tell when failover is happening.

On the additional note:
PR #155 did not change the logic of send_message_to_agent; the retry/backoff there is for send_file_to_agent, which I believe is separate from the LLM failover scope in this issue. That said, I agree the same retry/backoff pattern can eventually be consolidated into call_llm_with_failover.

Thanks for the review.