Skip to content

Report a silent robot as gone instead of latching its last state - #222

Merged
RobVanProd merged 1 commit into
mainfrom
agent/dashboard-stale-robot-state
Aug 11, 2026
Merged

Report a silent robot as gone instead of latching its last state#222
RobVanProd merged 1 commit into
mainfrom
agent/dashboard-stale-robot-state

Conversation

@RobVanProd

@RobVanProd RobVanProd commented Aug 11, 2026

Copy link
Copy Markdown
Owner

What happened

The loopback console reported the reference robot as connected and in mode Thinking on a heartbeat
4.3 days old, while the robot had no socket at the OS level and no ARP entry on the subnet:

"connected": true,  "mode": "Thinking",
"lastHeartbeatAt": "2026-08-07T15:42:41Z",  "heartbeatAgeSeconds": 370159

Why nothing cleared it

All three sources latch, and they cover for each other:

  • note_client_disconnected() clears _robot_connected only when the reported peer host matches, so a
    robot that vanishes without a clean disconnect never clears it.
  • refresh_robot() clears it only when the bridge listener is already down — the one case where
    connectivity was not in question anyway.
  • status() then re-asserts connectivity from a retained _debug snapshot that is never invalidated, so
    even a cleared flag is overridden by a network_state: connected, bridge_state: ready captured days
    earlier.

heartbeatAgeSeconds was computed and displayed the entire time without gating anything.

Fix

Sustained heartbeat silence outranks both latched sources and blanks the reported mode, which is otherwise
whatever the final heartbeat happened to carry.

This is more than cosmetic: the console is what an operator reads to decide the robot is ready before a
supervised run, and it could not distinguish ready from gone since Friday.

Not flapping

The window is 30 s against a heartbeat cadence of a few seconds, so a missed, late, or unreadable
sample cannot be classified as a robot failure — only sustained absence counts, per the evidence rule in
AGENTS.md. A socket that has not yet carried a heartbeat asserts nothing and still reads as connected.

Tests

Three cases in bridge/test_dashboard_service.py: a stale heartbeat reports gone with both latched sources
deliberately left asserting connected; a brief gap does not; and a freshly connected robot reads connected
before its first heartbeat. Defeating the gate fails the first and passes the others.

python -m unittest discover -s bridge -p "test_dashboard_service.py" — 23/23.
Full bridge suite: 539 tests, with only the four pre-existing numpy/opencv environment gaps
(test_rvc_directml_worker_service, test_vision_service) unrelated to this change.

Scope

Host-side only; no firmware change, so it does not disturb any exact-image qualification in flight.

Found while qualifying #221.

I briefly suspected the robot had a mirror-image defect, having seen bridge_state: error persist while
heartbeats flowed. That is withdrawn — it did not reproduce. bridge_state is the BridgeClient
conversational state machine (Offline/Connecting/Ready/Listening/Thinking/Responding/Error), and a later
observation showed it advancing to listening on the next turn unaided. The defect fixed here is
host-side only.

The loopback console reported the reference robot as connected and "Thinking"
on a heartbeat 4.3 days old, with no socket for it at the OS level and no ARP
entry on the subnet. Nothing cleared the state because all three sources latch.

note_client_disconnected() only clears _robot_connected when the reported peer
host matches, so a robot that vanishes without a clean disconnect never clears
it. refresh_robot() only clears it when the bridge listener is already down,
which is the one case where connectivity is not in question. And status() then
re-asserts connectivity from a retained _debug snapshot that is never
invalidated, so even a cleared flag would be overridden by "network_state:
connected, bridge_state: ready" captured days earlier.

heartbeatAgeSeconds was computed and displayed the whole time without gating
anything.

Treat sustained heartbeat silence as absence: it outranks both latched sources
and blanks the reported mode, which is otherwise whatever the final heartbeat
happened to carry. This matters beyond cosmetics -- the console is what an
operator reads to decide the robot is ready before a supervised run, and it
could not distinguish ready from gone-since-Friday.

The window is 30 s against a cadence of a few seconds, so a missed, late, or
unreadable sample cannot be classified as a robot failure; only sustained
absence counts. A socket that has not yet carried a heartbeat asserts nothing
and still reads as connected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@RobVanProd
RobVanProd merged commit 706590e into main Aug 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant