Skip to content

Context Agent poller resilience PR - #267

Open
dberardo-com wants to merge 1 commit into
nextcloud:mainfrom
dberardo-com:main
Open

dberardo-com wants to merge 1 commit into
nextcloud:mainfrom
dberardo-com:main

Conversation

@dberardo-com

Copy link
Copy Markdown

DISCLAIMER: this is PR is vibe coded !!

Relates to #266

Summary

  • Keep and cancel the poller as part of the FastAPI application lifespan.
  • Restart its coroutine after unexpected termination (5s backoff).
  • Bound next_task() with CONTEXT_AGENT_CLAIM_TIMEOUT_SEC (default 60s).
  • Wake immediately on /trigger, with configurable idle fallback
    CONTEXT_AGENT_IDLE_POLL_SEC (default 30s).
  • Isolate unexpected task-handler exceptions so they do not terminate the poller.
  • Log local claim errors, unexpected handler failures, and task durations,
    without logging user prompts or task payloads.
  • Do not log empty 2-second polls.

Scope

Only ex_app/lib/main.py is needed for this source change.
Do not include site-specific ConfigMap patches (agent.py, context_chat.py,
nc_model.py), credentials, or Kubernetes cleanup logic in the upstream PR.

Verification

  • python -m py_compile ex_app/lib/main.py
  • An isolated asyncio smoke test passed: trigger wake, handler-exception
    containment, supervisor restart, claim timeout/recovery, cancellation.
  • Full integration tests against the upstream dependency environment are still
    required; the isolated test is not equivalent to a full application test.
  • User's local patched deployment demonstrated LocalAI failure followed by
    successful processing of a new task without pod restart. This does not prove
    unpatched upstream would fail the same test.

Limitations

  • A failed LLM inference is not automatically retried.
  • asyncio.wait_for depends on cooperative cancellation and can leave a
    server-side claimed task in RUNNING if the response is lost or arrives late.
  • Pod crashes can still leave TaskProcessing RUNNING records; no reconciliation
    is provided here.
  • Unexpected poller failure while a TaskGroup is active may cancel its child
    handlers; this is safer than leaving the poller permanently dead but does
    not guarantee recovery of in-progress tasks.
  • An active task that gets stuck inside handle_task() is not forcibly timed
    out by the task-acquisition timeout.
  • The existing 2-second busy polling is retained but logged only on errors;
    it still generates Nextcloud API traffic.

Refactor timeout handling and logging in main.py. Introduce positive_timeout function to read environment variables safely. Update polling intervals and error handling in background tasks.

Signed-off-by: dberardo-com <65530457+dberardo-com@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant