Report a silent robot as gone instead of latching its last state - #222
Merged
Conversation
The loopback console reported the reference robot as connected and "Thinking" on a heartbeat 4.3 days old, with no socket for it at the OS level and no ARP entry on the subnet. Nothing cleared the state because all three sources latch. note_client_disconnected() only clears _robot_connected when the reported peer host matches, so a robot that vanishes without a clean disconnect never clears it. refresh_robot() only clears it when the bridge listener is already down, which is the one case where connectivity is not in question. And status() then re-asserts connectivity from a retained _debug snapshot that is never invalidated, so even a cleared flag would be overridden by "network_state: connected, bridge_state: ready" captured days earlier. heartbeatAgeSeconds was computed and displayed the whole time without gating anything. Treat sustained heartbeat silence as absence: it outranks both latched sources and blanks the reported mode, which is otherwise whatever the final heartbeat happened to carry. This matters beyond cosmetics -- the console is what an operator reads to decide the robot is ready before a supervised run, and it could not distinguish ready from gone-since-Friday. The window is 30 s against a cadence of a few seconds, so a missed, late, or unreadable sample cannot be classified as a robot failure; only sustained absence counts. A socket that has not yet carried a heartbeat asserts nothing and still reads as connected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
The loopback console reported the reference robot as
connectedand in modeThinkingon a heartbeat4.3 days old, while the robot had no socket at the OS level and no ARP entry on the subnet:
Why nothing cleared it
All three sources latch, and they cover for each other:
note_client_disconnected()clears_robot_connectedonly when the reported peer host matches, so arobot that vanishes without a clean disconnect never clears it.
refresh_robot()clears it only when the bridge listener is already down — the one case whereconnectivity was not in question anyway.
status()then re-asserts connectivity from a retained_debugsnapshot that is never invalidated, soeven a cleared flag is overridden by a
network_state: connected, bridge_state: readycaptured daysearlier.
heartbeatAgeSecondswas computed and displayed the entire time without gating anything.Fix
Sustained heartbeat silence outranks both latched sources and blanks the reported mode, which is otherwise
whatever the final heartbeat happened to carry.
This is more than cosmetic: the console is what an operator reads to decide the robot is ready before a
supervised run, and it could not distinguish ready from gone since Friday.
Not flapping
The window is 30 s against a heartbeat cadence of a few seconds, so a missed, late, or unreadable
sample cannot be classified as a robot failure — only sustained absence counts, per the evidence rule in
AGENTS.md. A socket that has not yet carried a heartbeat asserts nothing and still reads as connected.Tests
Three cases in
bridge/test_dashboard_service.py: a stale heartbeat reports gone with both latched sourcesdeliberately left asserting connected; a brief gap does not; and a freshly connected robot reads connected
before its first heartbeat. Defeating the gate fails the first and passes the others.
python -m unittest discover -s bridge -p "test_dashboard_service.py"— 23/23.Full bridge suite: 539 tests, with only the four pre-existing
numpy/opencvenvironment gaps(
test_rvc_directml_worker_service,test_vision_service) unrelated to this change.Scope
Host-side only; no firmware change, so it does not disturb any exact-image qualification in flight.
Found while qualifying #221.
I briefly suspected the robot had a mirror-image defect, having seen
bridge_state: errorpersist whileheartbeats flowed. That is withdrawn — it did not reproduce.
bridge_stateis theBridgeClientconversational state machine (
Offline/Connecting/Ready/Listening/Thinking/Responding/Error), and a laterobservation showed it advancing to
listeningon the next turn unaided. The defect fixed here ishost-side only.