Skip to content

Catch up on mentions made while the connector was down, and restart it after a memory stop - #928

Open
robzolkos wants to merge 3 commits into
connect-one-shot-watcherfrom
connect-reap-catch-up
Open

robzolkos wants to merge 3 commits into
connect-one-shot-watcherfrom
connect-reap-catch-up

Conversation

@robzolkos

@robzolkos robzolkos commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

When Claude Code stops basecamp connect because memory is low, the agent stops answering. Mentions made while it's down are also lost when it starts again. This PR makes two changes:

  • A restarted connector catches up. It hands off requests made up to 4 hours before it started, as long as this run admitted them. Before, the limit was one minute.
  • The skill starts the connector again at once after a memory stop, and the watcher too.

Stacked on #902 (connect-one-shot-watcher). The skill text changes #902's watcher steps, and #902's handled-ids file is what stops a request from being handled twice across restarts. Merge #902 first.

Card: Claude Code kills the connector for low memory, and the agent stops answering. These are options 2 and 3 of the proposal on the card.

Why

Claude Code 2.1.303 stops an idle session's background Bash tasks under critical memory pressure. All of these have to hold:

  • the system reports critical pressure
  • on Linux, less than 10% and less than 4 GiB of memory is available
  • nobody has typed for 30 minutes
  • no turn or delegated agent is running

Its notice says "stopped because the system is running low on memory … Do not start it again on your own". The session then waits for a person, and the agent stays deaf. Claude Code ends the task's process group and descendants, and it doesn't look at the task's own memory, so a small connector gets stopped too. A task started again within 30 minutes of such a stop isn't stopped again until someone types in the session.

On restart, handOff closed every admitted request made more than HandoffGrace (one minute) before the run started as before_this_run. The feed resumes from its checkpoint, so intake does see the mentions made during the outage. Handoff then dropped them.

Change

internal/connector/handoff.go

  • New rule in handoffCutoff. If this run wrote a record's admission decision (Decision.DecidedAt at or after Started), the record may have been made up to handoffCatchUp (4 hours) before the run.
  • A record an earlier run admitted keeps HandoffGrace (one minute). An earlier run's decision was made under whatever trust held then. Replaying it hours later could hand off a request from someone who was removed from --allow while the connector was down. Requests made during an outage are admitted by the new run, under current trust.
  • The before_this_run log line now carries the cutoff.

internal/commands/connect_run.go: the run's start is taken before intake and admission start, not after. Otherwise something this run admits could be stamped before Started and get the one-minute rule.

skills/basecamp-connect/SKILL.md

  • When the connector stops: on a low-memory notice, start the connector again at once as in step 1. Then drain the old output file and start the watcher on the new one, and tell the person. The person asked for it to run when they started it.
  • A watcher-only notice restarts the watcher on the running connector's current file, never on an old one.
  • When the connector stops, Checking on it and Stopping now state the 4-hour catch-up instead of "not picked up later" and "starts from the present".
  • The per-request steps are unchanged, so the compaction budget isn't touched.

Tests

  • handoff_test.go:
    • A request this run admitted is handed off when made 2 minutes, 1 hour or exactly 4 hours before the run.
    • It is closed as before_this_run at 4 hours and 1 second.
    • A request an earlier run admitted is handed off at exactly one minute, and closed at one minute and one second and at one hour.
  • skill_layout_test.go: the memory-stop steps exist, including the watcher rule. The skill states the catch-up in the same number of hours as handoffCatchUp, so changing one without the other fails.
  • Mutations. Each of these fails tests:
    • using the one-minute rule for every record (the this-run test, at 2m, 1h and 4h)
    • using the 4-hour rule for every record (the earlier-run test, at 1m1s and 1h)
    • reverting the skill (both skill tests)
  • Not covered by a test: moving started earlier in runConnect. It removes an ordering race between the handoff's start time and admission's first decision. I found no seam to test that without a real run.
  • bin/ci is green locally (MISE_BATS_VERSION=1.13.0, as in Watch for connector requests with a one-shot background watcher, not Monitor #902).

Trade-offs

  • Late answers. After a stop of up to 4 hours, people get answers to what they asked while the agent was down, possibly after someone else has dealt with it. That's the point of the change. The 4-hour window is a judgment call, set in one constant.
  • Overlap with open PRs.

Summary by cubic

A restarted Basecamp connector now answers requests made while it was down (up to 4 hours before the run), instead of dropping anything older than a minute. The skill also tells the agent to restart the connector at once after Claude Code stops it for low memory.

  • Requests the current run admitted get the 4-hour catch-up window; requests an earlier run admitted keep the one-minute grace, so a restart never replays a decision made under trust that may have changed while the connector was down.
  • The run's start time is recorded before intake and admission begin, so requests admitted by this run are never stamped before the run.
  • The memory-stop steps restart the connector and the watcher. If a notice names only the watcher, check the connector first: if it was stopped too, restart it (which starts a new watcher) and never start one on the old file; otherwise start the watcher on its current output file.

Written for commit 2f62103. Summary will update on new commits.

View guided diff Turn on auto-fix

…r a memory stop

A restarted connector now hands off requests made up to four hours before
it started, instead of one minute, so a mention made while it was down is
still answered. The skill tells the session to start the connector, and the
request watcher, again at once when Claude Code stops them for low memory.
…rent watcher

A request an earlier run admitted keeps the one-minute grace, so a restart
never hands off a decision made under trust that may have changed while the
connector was down. The run's start is taken before intake and admission
begin, so nothing this run admits is stamped before it. A memory-stop notice
for a watcher whose connector was also stopped no longer restarts a watcher
on the old output file.
@robzolkos
robzolkos requested a review from a team as a code owner October 20, 2026 17:43
Copilot AI balanced review requested due to automatic review settings October 20, 2026 17:43
@github-actions github-actions Bot added commands CLI command implementations tests Tests (unit and e2e) skills Agent skills labels Oct 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

Source review identified no blocking issues in the recovery logic or guidance; tests were not executed here.

No open issues 🎉

What changed in this PR

Improves connector recovery after downtime, building on #902’s watcher and persistent deduplication.

Changes:

  • Allows four-hour catch-up for requests admitted by the current run, preserving the one-minute grace for earlier admissions.
  • Records startup time before intake and admission begin.
  • Updates memory-stop recovery guidance and adds boundary and skill-consistency tests.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
skills/​basecamp-connect/​SKILL.md Documents automatic recovery and catch-up limits.
internal/​connector/​skill_layout_test.go Checks recovery guidance and catch-up wording.
internal/​connector/​handoff.go Selects catch-up cutoff by admission time.
internal/​connector/​handoff_test.go Tests current-run and earlier-run cutoff boundaries.
internal/​commands/​connect_run.go Captures startup time before admission begins.

Review effort: Balanced

@robzolkos

Copy link
Copy Markdown
Collaborator Author

🤖 Review status on 2f62103

Which commit each reviewer read:

Pi, first pass (35a423e)

  • Applied in 5030403. An earlier run's cached admission could be handed off hours later under trust that changed while the connector was down, for example a requester removed from --allow. Now only records this run admitted get the 4-hour catch-up. An earlier run's records keep the one-minute grace, and a test holds each side.
  • Applied in 5030403. A late memory-stop notice for the old watcher could restart it on the dead output file. The skill now says to restart only on the current file.
  • Pre-existing, not changed here. A hold set after handoffRecords selects a batch doesn't stop that batch from printing. This also happens with the one-minute grace.
  • Pre-existing, not changed here. Agent mentions older than the cutoff count toward the agent caps until handoff discards them. The longer catch-up makes that rarer, not more common.

Pi, second pass (5030403)

  • Applied in 2f62103. If both tasks were stopped and the watcher's notice arrives first, the skill now says to check the connector first.
  • Declined: "DecidedAt vs Started isn't run identity if the wall clock steps backward." It's true in principle. It needs the system clock to step back, either during a run (one request decided just after start would get the one-minute rule) or between runs by more than the downtime (an earlier run's decision would get the 4-hour rule). A persisted run id would close it, but that's a schema change for a case NTP rarely produces, and each miss is bounded to one request. Not worth the machinery here.
  • Noted, not tested. Moving started before intake and admission in runConnect has no test. I found no seam to test it short of a real run.
  • Pre-existing, separate. An operator's redispatch of an old held snapshot can be discarded by the handoff age cutoff (ledger_decisions.go:275-287).

Open threads: none. This needs #902 merged first, and then a human merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commands CLI command implementations skills Agent skills tests Tests (unit and e2e)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants