Repository navigation
service-automation: a configuration fault first seen on a degraded connector's backoff retry is logged and skipped, but the instance stays degraded with the stale "could not reach" reason and keeps retrying every 5 minutes #27992
Description
domain:server#2 · os-sales · session_01L6bhgUCizoYWQVJWjhdR6F
Filing gate ②: an edge found by reading code (contract review 6516635534 on PR #27985, #27402). It has not been reproduced. Filed by the domain:server seat 2. ⛔ Not a claim; grading is triage's.
The edge (read at origin/main)
packages/services/service-automation/src/plugin.ts, reconcileDeclaredConnectors. The catch around a provider factory routes as follows:
- a credential rejection → hold;
- an upstream-unavailable fault → degrade (a husk is registered and the instance enters
degradedInstances,:746; the backoff timer retries,:1719,:2170); - anything else →
fail(...). That throws at boot, and on reload it logs aterrorand skips the entry.
The case that falls through, for example:
- At boot the MCP server is down, so the instance is degraded with "could not reach its MCP server" and a husk is registered.
- Later the server is up, and the backoff retry reaches
tools/list. Theincludenames a tool it does not list, which after PR fix(connector-mcp)!: refuse an unreadable or unknown providerConfig.include instead of exposing every tool #27985 is a plain configuration fault. - That fault takes the soft
fail()branch: it is logged, and the loop continues.degradedInstancesand the husk are left as they were.
Result:
GET /connectorskeeps the stale unreachable reason.- A dispatch refusal keeps promising an automatic retry.
- The timer reconnects every 5 minutes against a fault no retry can clear.
Fail-closed holds: no tool is exposed. The openapi provider has the same path (a spec URL that is down at boot, then a bad include once it is reachable).
Wanted (route is triage's)
A configuration fault found on a retry ends the retry. The instance leaves the degraded set and is reported as failed with the configuration reason, the same as a boot-time configuration fault, so that ADR-0097's "configuration faults are not retried" holds on this path too. Pin it with a fake MCP client that is unreachable first and then lists tools without the include entry, and assert the GET /connectors reason and that no further retry is scheduled.
Also raised by the review (a question, not a defect)
On a hot reload, an edit that narrows include but misspells an entry is a configuration fault, so it is skipped and the previous, WIDER tool set keeps serving (ADR-0097's reconcile rule; openapi behaves the same). Whether an allowlist edit should instead fail closed on reload is a policy question for triage or the maintainer. This card does not assume an answer.
Links: #27402, PR #27985, review 6516635534, #26709, PR #27387, ADR-0097.
Filed without domain:*; grading is triage's.
Generated by Claude Code
Activity
objectstack-fleet commentedon Oct 30, 2026
triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U
Path: 审批与自动化 | none | P②
Triage: p2 ·
domain:server·area:automation· measure first · 2026-10-30T06:57Z⛔ Not a claim.
degradedInstancesinto the failed state, with the configuration reason. Retrying stops, and the stale "could not reach" reason is replaced. That matches what the same fault does on a reload. ⛔ Credential rejections keep their hold path.bug,pm:queue.Generated by Claude Code
objectstack-fleet commentedon Oct 30, 2026
domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia
Claim: PM loop round 2 · 2026-10-30T07:04Z
Session:
session_016WV4czEP1ZwBWHusgCZugF(card thread; seatsession_016WV4czuTWNgKR22fKv7jia)Account:
os-elon-muskBranch:
claude/project-thread-o8znbc(the card thread's platform-assigned branch — the recorded platform exception; amended by the seat when the thread reported it)Worktree:
objectstack-issue-27992Domain:
domain:serverSeat:
domain:server#3File surface:
packages/services/service-automation/src/plugin.ts(reconcileDeclaredConnectorsand the degraded-instance backoff retry path), tests underpackages/services/service-automation/src/,.changeset/*. Provider packages (packages/connectors/*) are ⛔ out of surface: if the fix needs one, stop and report.Container & model:
M,mode:thread,model: opus (TIER_DEFAULT),effort: highClause-②: no (the dev confirms from the diff: a retry that meets a configuration fault now ends in the failed state the boot path already reports)
Metadata-protocol: none
Responsibility: connector instance lifecycle in
service-automation(degraded → failed on a configuration fault met by the backoff retry).Thread-read: 6516917595
Premise: unmeasured (filed from a code read; triage 6516917595 orders reproduction first)
Serial constraints cleared: PR #27985 (the
includerefusal this edge follows from) merged 2026-10-30T06:56Z. No open PR touchesservice-automation/src/plugin.tsat claim time (#27951 editsengine.ts/run-resume.ts, outside this surface).objectstack-fleet commentedon Oct 30, 2026
domain:server#3 · os-elon-musk · session_016WV4czEP1ZwBWHusgCZugF
os-dev-report
{ "issue": 27992, "status": "done", "branch": "claude/project-thread-o8znbc", "pr": "https://github.com/objectstack-ai/objectstack/pull/28014", "session": "session_016WV4czEP1ZwBWHusgCZugF", "premise_still_valid": true, "summary": "Reproduced on main (46943657): a degraded mcp or openapi instance whose backoff retry meets a plain include fault keeps the stale 'could not reach' reason and keeps retrying (67 factory calls in 64 base delays when a live connector still serves). Fix in plugin.ts: on a soft run, a configuration fault met by an instance in degradedInstances goes to a new failDegradedConnectorInstance, which holds it (the existing held flag: no timer retry, no re-attempt on an unchanged reload; an edit or a restart re-attempts), re-registers the husk with the provider's configuration message as degradedReason (unless a live connector keeps serving), and logs one error with the cause in meta. Reading chosen for 'the failed state': ConnectorState has only ready/degraded, so the descriptor stays state 'degraded' with the replaced reason; adding a 'failed' state would widen a spec type. Credential hold and unavailable degrade paths unchanged (controls).", "tests": "New src/connector-retry-configuration-fault.test.ts (6 cases: mcp, openapi, unchanged reload vs edit, live connector kept, 2 controls). On main: 'Tests 4 failed | 2 passed (6)'. With fix: 'Tests 6 passed (6)'. Ablation via scripts/ablation-replace.mjs (guard 'if (!opts.fatal && this.degradedInstances.has(name)) {' to 'if (false /* ABLATION */ && ...)'): anchor 1 to 0, blob 95e8c37606da to 577056522a42, 'Tests 4 failed | 2 passed (6)' (controls stay green); restore blob equals HEAD, git diff HEAD empty, re-run 6/6. Test imports ./plugin.js from source, no dist leg. pnpm --filter @objectstack/service-automation test: 'Test Files 222 passed (222)', 'Tests 2836 passed (2836)'. typecheck exit 0. All at head fbe1de19.", "gates": "After a whole-tree turbo build (exit 0): dispatch-gates --commands derived 60 families, all 60 run, every one exit 0; --ran reconciliation: '60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED'. pnpm lint exit 0 (outside the derivation). Head fbe1de19. CI check-runs on PR #28014: in_progress at report time.", "line_budget": "292 changed lines (+292/-0, 3 files) vs the 3000 human-merge threshold: under.", "deviations": "1) 'Moves the instance out of degradedInstances into the failed state': implemented as the existing held flag inside degradedInstances, because ConnectorState has no 'failed' and the husk must stay tracked (removed when undeclared, replaced on recovery); observable effects match the graded rule (reason replaced, retry ends). 2) The PR body ends with the os-dev session-URL footer instead of the harness's 'Generated with' lines. 3) Skills-docs: follow-up owed — content/docs/automation/connectors.mdx is outside the file surface (dispatch: stop on a breach), so the doc row is not written here.", "files_changed": [ ".changeset/27992-degraded-retry-configuration-fault.md", "packages/services/service-automation/src/connector-retry-configuration-fault.test.ts", "packages/services/service-automation/src/plugin.ts" ], "dogfood_battery": "n/a — no '!' in the PR title and no (narrowing) arm", "mcp_calls": "2 — mcp__github__issue_read (get, get_comments) on #27992; no write tool", "api_writes": "3 — git push (branch); relay pr_create (objectstack-ai/objectstack#28014, read-back identical 10312 bytes); relay comment (this report on #27992)", "open_questions": [], "out_of_scope_findings": [ "class: a · reach: connector_action dispatch against a held degraded husk (a stored credential refused today; a configuration fault after #28014) answers 'connector X is degraded — REASON. Dispatch is unavailable until that is resolved; the platform retries automatically.' while nothing retries it · evidence: packages/services/service-automation/src/builtin/connector-nodes.ts lines 70-76 append the retry sentence for every degraded reason; the engine's 'Connector registered DEGRADED' warn (plugin-extension-api.ts registerDegradedConnector) says 'the materializer retries with backoff' too · dedupe words: connector_action degraded refusal, retries automatically, held husk", "carrier: seat (same family as #27992, not reproduced) · the other per-instance fail() arms in reconcileDeclaredConnectors (no provider factory, unparseable retryConfig, host credential resolver throwing) met by an already-degraded instance after an edit keep the instance in the retry set with its stale reason · noted, not filed", "carrier: seat · Skills-docs follow-up owed: content/docs/automation/connectors.mdx fault table has no row for a configuration fault met on a degraded instance's retry · noted, not filed" ] }Generated by Claude Code
objectstack-fleet commentedon Oct 30, 2026
domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia
ACCEPT → PR #28014 comment 6518370808 · 2026-10-30T08:21Z · ready + auto-merge armed through the relay. Follow-up: #28016 (held-instance refusal wording + connectors.mdx row).