Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ curl http://tiny-engineer.local/settings
| `serial_log` | USB serial debug logging (default **false**). When off, diagnostic `Serial` output is suppressed |
| `push_button` | Extension-port push-button reader (default **false**). When on, publishes immediate `button.down` / `button.up` plus `button.click` / `button.double_click` / `button.long_click` (see [Outbound events](#outbound-events)). Applies **immediately**. Local reactions use [Actions](#get-settingsactions) |
| `push_button_pin` | GPIO on the Extension port for the push-button: **6**, **7**, **20**, or **21** (default **20**). Persists when `push_button` is off. Applies **immediately** when the extension is on |
| `continuous_timeout` | Minutes a continuous animation (`typing` / `reading` / `thinking` / `talking`) may run before the firmware switches to `attention` then idle (default **5**) |
| `continuous_timeout` | Minutes a continuous animation (`typing` / `reading` / `thinking` / `talking` / `waiting`) may run before the firmware switches to `attention` then idle (default **5**) |
| `loading` | Boot loading screen: `progress` (bar + large IP for 3 s) or `sleep_inertia` (eyes wake; no bar/IP). Default **`progress`**. Applies on **next reboot** |
| `eyes_style` | OLED eyes look: `classic` (procedural rounded eyes), `kaomoji` (animated faces), `cover` (full-half bars for an eye-hole mask), `dots` (12×12 circles), or `kawaii` (round eyes with big moving pupils, a talking mouth, and blush). Default **`classic`**. Applies **immediately**. Classic, cover, and dots sequences are described under [`POST /anim`](#post-anim); kaomoji face mapping is under [Kaomoji eyes](#kaomoji-eyes); cover drawing is under [Cover eyes](#cover-eyes); dots drawing is under [Dots eyes](#dots-eyes); kawaii moods are under [Kawaii eyes](#kawaii-eyes) |
| `access_token_set` | `true` when a non-empty access token is stored (secret itself is never returned) |
Expand Down Expand Up @@ -593,15 +593,15 @@ curl http://tiny-engineer.local/anim

| Field | Meaning |
| --- | --- |
| `animation` | `none`, `typing`, `reading`, `thinking`, `talking`, `ring`, `welcome`, `attention`, `error`, `abort`, `dead`, `wakeup`, or `sleep` |
| `animation` | `none`, `typing`, `reading`, `thinking`, `talking`, `waiting`, `ring`, `welcome`, `attention`, `error`, `abort`, `dead`, `wakeup`, or `sleep` |

### `POST /anim`

Request an animation switch. Each animation runs at least **1s**; if the current one is shorter than that, the switch waits until the hold ends. Rapid posts during the hold keep only the **latest** request. Response `animation` is the currently playing name (may still be the old one until the hold ends). Hand moves use speed-limited servo wrappers.

| Param | Type | Values |
| --- | --- | --- |
| `name` | string | `none`, `typing`, `reading`, `thinking`, `talking`, `ring`, `welcome`, `attention`, `error`, `abort`, `dead`, `wakeup`, `sleep` |
| `name` | string | `none`, `typing`, `reading`, `thinking`, `talking`, `waiting`, `ring`, `welcome`, `attention`, `error`, `abort`, `dead`, `wakeup`, `sleep` |

```bash
curl -X POST "http://tiny-engineer.local/anim?name=typing"
Expand Down Expand Up @@ -634,6 +634,7 @@ Durations and quoted lines below are the stock clips in [`assets/`](../assets/).
| `reading` | **Continuous.** Hands/body park as in `none`. Head nods on same lowest 10° band as typing, but slower. Neck sweeps ±10° around mid; right (angle down) slower than left (angle up). Occasional right-hand down-arrow bursts (1–3 presses, same 15° band as typing). Runs until replaced, or until `continuous_timeout` triggers `attention`. Same-name re-`POST` refreshes that timeout without restarting motion. |
| `thinking` | **Continuous.** Hands/body park as in `none`. Head (pitch) and neck (yaw) ease from the current pose into thinking poses (up + slight left/right). Move → pause → optional micro-adjust (sometimes chained) → pause; nearby pose drift with occasional larger shifts after ~2.2 s, more often over time. Axes stagger start/duration; no periodic sway. Runs until replaced, or until `continuous_timeout` triggers `attention`. Same-name re-`POST` refreshes that timeout without restarting motion. |
| `talking` | **Continuous.** Speaking fidget with no sound of its own. Also the default pose for [`POST /play`](#post-play). Via `/anim`, the head faces the user just above mid with soft random nods, slow neck drift, occasional small torso shifts, and occasional right-hand lifts. During `/play` with pose `talking`, motion follows speech energy from the PCM stream (no extra metadata): emphasis triggers nods, silence settles the pose, phrase starts may raise the hand or shift the torso (not both). **Classic eyes:** a little taller than idle; glances away when ambient or quiet, gaze locked on the listener while speech energy is high. Runs until replaced, or until `continuous_timeout` triggers `attention`. Same-name re-`POST` refreshes that timeout without restarting motion. |
| `waiting` | **Continuous.** Impatient wait with no sound. The torso turns to the robot's left (body `+0.8`), the right hand parks at rest, and the left hand taps the desk in groups of three (each tap lifts 0.18–0.28 of its range, groups every 2.5–8 s at random). Every 3–7 s the head eases (1.4–2.3 s) to one of four wide corners, never the same one twice: down at the tapping hand (most often), up-left, up-right, or down-right. **Classic eyes:** glance to the same corner about 180 ms before the head gets there (±5 px sideways, 3 px up / 2 px down), a little wider looking up and a little lower looking down. The Claude Code hooks use it when Claude has waited about a minute for the user (`Notification` with `idle_prompt`). Runs until replaced, or until `continuous_timeout` triggers `attention`. Same-name re-`POST` refreshes that timeout without restarting motion. |
| `ring` | **One-shot** service-bell gesture; does not loop. Wind-up (body `min`, neck mid, head mid+10°, both hands `max`) → fast right-hand strike to `min+5°` with head to `min` → plays `bell.wav` once on strike (LittleFS; same `uploadfs` requirement as `/test/audio/bell`) → slower bounce to `min+20°` → return to `none` pose and stop. After completion, `GET /anim` reports `none`. |
| `welcome` | **One-shot** hello gesture synced to `welcome.wav` (~2.7 s). Right hand raises during "Hello, human.", holds through the pause, wiggles during "What are we building today?", then lowers. Head nods to mid+10° and returns. Plays automatically after successful Wi-Fi connect at boot; also via API. Requires `welcome.wav` on LittleFS (same `uploadfs` flow as `bell.wav`). After completion, `GET /anim` reports `none`. |
| `attention` | Friendly input-request gesture synced to `attention.wav` (~3.0 s, "pst... human.... you might want to take a look"). Moves into a calm prompt pose first (centered body/neck, head slightly up, right hand raised partway), waits until all servos stop, then plays audio with light neck/head/hand motion during playback (whisper hold → lean toward user → glance/point on "take a look"). **Classic eyes:** phased blink/look cues during audio. After audio ends (or if audio fails to start), holds a gentle waiting loop for **1 minute** (soft head/neck drifts plus occasional slight right-hand waves), then returns to `none`. Requires `attention.wav` on LittleFS (same `uploadfs` flow as `bell.wav`). |
Expand All @@ -649,7 +650,7 @@ When `eyes_style=kaomoji`, the OLED shows looping faces from the expression libr

| Animation / phase | Face |
| --- | --- |
| `none` (idle) | Idle |
| `none` (idle), `waiting` | Idle |
| `typing`, `thinking` | Thinking |
| `reading`, `attention` | Curious |
| `ring` | Surprise |
Expand Down Expand Up @@ -688,6 +689,7 @@ While `welcome`, `attention`, `error`, `abort`, or `dead` plays its stock clip,
| `abort` | Panic: `> <` eyes, wavy mouth; says "Fine. I didn't want to finish it anyway." |
| `wakeup` | Sleepy: heavy lids, yawn |
| `talking` | The mouth talks in syllables, shaped as `ah` (tall), `ee` (wide), or `o` (round). With `/play` audio it opens with the speech loudness and changes shape on each syllable onset. Without audio it babbles in short phrases with pauses |
| `waiting` | The idle smile (`ω`). The pupils follow the classic eyes to each corner the head looks at |
| `sleep` / sleep-close | Lids close, small `ω` |
| `dead` | Sad (lids tilted down at the outer corners, frown), then `x x` eyes during the X hold. Says "Insufficient resources... shutting doooowwwnnn" |

Expand All @@ -696,15 +698,15 @@ Wrong params return **400**:
| `error` | When |
| --- | --- |
| `missing name` | Query param `name` absent |
| `unknown animation` | `name` not `none`, `typing`, `reading`, `thinking`, `talking`, `ring`, `welcome`, `attention`, `error`, `abort`, `dead`, `wakeup`, or `sleep` |
| `unknown animation` | `name` not `none`, `typing`, `reading`, `thinking`, `talking`, `waiting`, `ring`, `welcome`, `attention`, `error`, `abort`, `dead`, `wakeup`, or `sleep` |

### `POST /play`

Streams a WAV to the speaker while one animation runs, then returns to the continuous animation (`typing`, `reading`, `thinking`, `talking`) that was running before the request, otherwise to `none`. The body is played as it arrives, so clips of any length need no storage on the robot. The response comes when the clip ends.
Streams a WAV to the speaker while one animation runs, then returns to the continuous animation (`typing`, `reading`, `thinking`, `talking`, `waiting`) that was running before the request, otherwise to `none`. The body is played as it arrives, so clips of any length need no storage on the robot. The response comes when the clip ends.

| Part | Value |
| --- | --- |
| Path | `/play` runs `talking`; `/play/{name}` runs `talking`, `typing`, `reading`, `thinking`, or `none`. Animations with their own sound are rejected so two clips never overlap |
| Path | `/play` runs `talking`; `/play/{name}` runs `talking`, `typing`, `reading`, `thinking`, `waiting`, or `none`. Animations with their own sound are rejected so two clips never overlap |
| Body | WAV, 16-bit mono PCM at 22050 Hz (the firmware `SAMPLE_RATE`). `Content-Length` is required. Chunks other than `fmt ` and `data` are skipped |
| `Authorization` | `Bearer <token>` when `access_token` is set |

Expand All @@ -725,7 +727,7 @@ Errors arrive before any audio plays:

| Status | `error` | When |
| --- | --- | --- |
| `400` | `name must be talking, typing, reading, thinking or none` | `{name}` is unknown or has its own sound |
| `400` | `name must be talking, typing, reading, thinking, waiting or none` | `{name}` is unknown or has its own sound |
| `415` | `expected 16-bit mono PCM at 22050 Hz` | WAV format or sample rate differs |
| `415` | `not a WAV file`, `no audio data` | Body is not a RIFF/WAVE file, or has no `data` chunk |

Expand Down Expand Up @@ -759,7 +761,7 @@ Onboard WS2812 on **GPIO10** (`RGB_LED_PIN`). Byte order is the saved `rgb_order

| Animation | LED color |
| --- | --- |
| `typing`, `reading`, `thinking`, `talking`, `welcome`, `ring`, `wakeup` | White (full intensity) |
| `typing`, `reading`, `thinking`, `talking`, `waiting`, `welcome`, `ring`, `wakeup` | White (full intensity) |
| `attention`, `error`, `dead` | Red pulse: 500 ms 10%→100%, 500 ms hold 100%, 500 ms 100%→10%, repeat |
| `abort` | Red (full intensity, solid) |
| `none`, `sleep` | Off |
Expand Down
2 changes: 1 addition & 1 deletion docs/hardware/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Constants cited below match [`include/pins.h`](../../include/pins.h).
| USB | Native USB on GPIO18/19 — reached via the main control board USB-C (onboard USB-C unused when assembled). Advanced path: Adafruit 5993 — [wiring.md](wiring.md#usb-connector-5993) |
| Flash | 4 MB stacked flash on GPIO12–GPIO17 (those GPIOs are **not exposed**) |
| Onboard RGB | WS2812 on **GPIO10** |
| Firmware notes | Boot: dim green status. Animations: white (typing/reading/thinking/talking/welcome/ring/wakeup), pulsing red (attention/error/dead), solid red (abort), off (idle/sleep). 1 s fade between non-pulse states. Fatal init errors: hang and blink red (1 = PCA9685, 2 = I2S). See [blink codes](testing.md#boot-failure-blink-codes) and [`docs/api.md`](../api.md#rgb-led) |
| Firmware notes | Boot: dim green status. Animations: white (typing/reading/thinking/talking/waiting/welcome/ring/wakeup), pulsing red (attention/error/dead), solid red (abort), off (idle/sleep). 1 s fade between non-pulse states. Fatal init errors: hang and blink red (1 = PCA9685, 2 = I2S). See [blink codes](testing.md#boot-failure-blink-codes) and [`docs/api.md`](../api.md#rgb-led) |
| BOOT | Button / strapping on **GPIO9** |
| RESET | Separate RESET button (CHIP_EN), not a GPIO |
| UART0 silkscreen | TX = GPIO21, RX = GPIO20 |
Expand Down
2 changes: 1 addition & 1 deletion docs/hardware/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ During normal operation the onboard WS2812 tracks the active animation (not boot

| Animation | LED |
| --- | --- |
| `typing`, `reading`, `thinking`, `talking`, `welcome`, `ring`, `wakeup` | White |
| `typing`, `reading`, `thinking`, `talking`, `waiting`, `welcome`, `ring`, `wakeup` | White |
| `attention`, `error`, `dead` | Pulsing red (10%↔100%, 1.5 s cycle) |
| `abort` | Solid red |
| `none`, `sleep` | Off |
Expand Down
2 changes: 1 addition & 1 deletion docs/hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,6 @@ Config: [`.cursor/hooks.json`](../.cursor/hooks.json). Local anim hook: [`.curso

- Robot offline → hook still exits 0; no agent stall.
- Animation API holds each pose ≥1s and keeps only the latest pending switch — see [`api.md`](api.md).
- The onboard RGB LED follows the active animation (white for typing/reading/thinking/talking/welcome/ring/wakeup, pulsing red for attention/error/dead, solid red for abort, off for `none`/`sleep`) with 1 s fades between non-pulse states — see [RGB LED](api.md#rgb-led).
- The onboard RGB LED follows the active animation (white for typing/reading/thinking/talking/waiting/welcome/ring/wakeup, pulsing red for attention/error/dead, solid red for abort, off for `none`/`sleep`) with 1 s fades between non-pulse states — see [RGB LED](api.md#rgb-led).
- To change the map, edit [`packages/tiny-engineer-cli/src/integrations/cursor/map.js`](../packages/tiny-engineer-cli/src/integrations/cursor/map.js).
- Custom agents or non-Cursor IDEs: call the [REST API](integration.md#2-rest-api-advanced--custom).
1 change: 1 addition & 0 deletions docs/integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ curl -X POST "http://tiny-engineer.local/anim?name=typing"
| `welcome` | Session start / greeting |
| `wakeup` | Sleep-inertia wake (eyes/head) |
| `sleep` | Force sleep (eye close + OLED off) |
| `waiting` | Turn finished and the user has not answered for a while (quiet, before `attention`) |
| `attention` | Needs user input |
| `error` / `abort` | Failure / cancel |
| `dead` | Out of power (`dead.wav`, then X X hold) |
Expand Down
5 changes: 3 additions & 2 deletions packages/tiny-engineer-claude-code/src/cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,15 @@ Event map:
PreCompact → thinking
PermissionRequest → attention
PermissionDenied → abort
Notification → attention
Notification + idle_prompt → waiting
other Notification → attention
Stop → ring
StopFailure → error

Play mode:
Streams a 16-bit mono PCM WAV (at the robot's sample rate) to POST /play and
runs one animation while it plays: talking (default), typing, reading,
thinking or none. The robot then returns to the loop it was in.
thinking, waiting or none. The robot then returns to the loop it was in.
Prints the robot's JSON reply when the clip ends; exits 1 on any error.

Examples:
Expand Down
9 changes: 8 additions & 1 deletion packages/tiny-engineer-claude-code/src/map.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ const EVENT_ANIM = new Map([

/**
* Map a Claude Code hook payload to an animation name, or null to skip.
* @param {{ hook_event_name?: string, tool_name?: string, source?: string, reason?: string }} event
* @param {{ hook_event_name?: string, tool_name?: string, source?: string, reason?: string, notification_type?: string }} event
* @returns {string | null}
*/
export function animationForEvent(event) {
Expand All @@ -56,5 +56,12 @@ export function animationForEvent(event) {
return event.reason === "clear" || event.reason === "resume" ? null : "sleep";
}

// Claude finished about a minute ago and the user has not typed: wait
// quietly. The firmware's continuous_timeout turns it into attention later.
// Every other notification asks for action and keeps attention.
if (name === "Notification" && event.notification_type === "idle_prompt") {
return "waiting";
}

return EVENT_ANIM.get(name) ?? null;
}
14 changes: 14 additions & 0 deletions packages/tiny-engineer-claude-code/test/map.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,20 @@ test("lifecycle events map to their poses", () => {
}),
"attention"
);
assert.equal(
animationForEvent({
hook_event_name: "Notification",
notification_type: "idle_prompt",
}),
"waiting"
);
for (const notification_type of ["elicitation_dialog", "auth_success", undefined]) {
assert.equal(
animationForEvent({ hook_event_name: "Notification", notification_type }),
"attention",
String(notification_type)
);
}
assert.equal(animationForEvent({ hook_event_name: "Stop" }), "ring");
assert.equal(animationForEvent({ hook_event_name: "StopFailure" }), "error");
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ const EVENT_ANIM = new Map([

/**
* Map a Claude Code hook payload to an animation name, or null to skip.
* @param {{ hook_event_name?: string, tool_name?: string, source?: string, reason?: string }} event
* @param {{ hook_event_name?: string, tool_name?: string, source?: string, reason?: string, notification_type?: string }} event
* @returns {string | null}
*/
export function animationForEvent(event) {
Expand All @@ -56,5 +56,12 @@ export function animationForEvent(event) {
return event.reason === "clear" || event.reason === "resume" ? null : "sleep";
}

// Claude finished about a minute ago and the user has not typed: wait
// quietly. The firmware's continuous_timeout turns it into attention later.
// Every other notification asks for action and keeps attention.
if (name === "Notification" && event.notification_type === "idle_prompt") {
return "waiting";
}

return EVENT_ANIM.get(name) ?? null;
}
2 changes: 2 additions & 0 deletions packages/tiny-engineer-cli/src/robot/client.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ export const ANIM_NAMES = new Set([
"reading",
"thinking",
"talking",
"waiting",
"ring",
"welcome",
"attention",
Expand All @@ -21,6 +22,7 @@ export const PLAY_ANIM_NAMES = new Set([
"typing",
"reading",
"thinking",
"waiting",
"none",
]);

Expand Down
2 changes: 2 additions & 0 deletions packages/tiny-engineer-cli/test/client.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ test("validateAnimName accepts firmware names", () => {
assert.equal(validateAnimName("ring"), null);
assert.equal(validateAnimName("typing"), null);
assert.ok(ANIM_NAMES.has("sleep"));
assert.ok(ANIM_NAMES.has("waiting"));
});

test("validateAnimName rejects unknown names", () => {
Expand All @@ -50,6 +51,7 @@ test("validatePlayAnimName accepts play path names", () => {
assert.equal(validatePlayAnimName("talking"), null);
assert.equal(validatePlayAnimName("none"), null);
assert.ok(PLAY_ANIM_NAMES.has("thinking"));
assert.ok(PLAY_ANIM_NAMES.has("waiting"));
});

test("validatePlayAnimName rejects ring", () => {
Expand Down
14 changes: 14 additions & 0 deletions packages/tiny-engineer-cli/test/map-claude-code.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,20 @@ test("lifecycle events map to their poses", () => {
}),
"attention"
);
assert.equal(
animationForEvent({
hook_event_name: "Notification",
notification_type: "idle_prompt",
}),
"waiting"
);
for (const notification_type of ["elicitation_dialog", "auth_success", undefined]) {
assert.equal(
animationForEvent({ hook_event_name: "Notification", notification_type }),
"attention",
String(notification_type)
);
}
assert.equal(animationForEvent({ hook_event_name: "Stop" }), "ring");
assert.equal(animationForEvent({ hook_event_name: "StopFailure" }), "error");
});
Expand Down
2 changes: 1 addition & 1 deletion skills/tiny-engineer-play/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,6 @@ It reads `TINY_ENGINEER_URL` (default `http://tiny-engineer.local`) and `TINY_EN
| Reply | Fix |
| --- | --- |
| `415 expected 16-bit mono PCM at N Hz` | Re-run the `ffmpeg` step with `-ar N` |
| `400 name must be ...` | Use `talking`, `typing`, `reading`, `thinking` or `none` |
| `400 name must be ...` | Use `talking`, `typing`, `reading`, `thinking`, `waiting` or `none` |
| `401 unauthorized` | Set `TINY_ENGINEER_TOKEN` to the robot's `access_token` |
| `robot unreachable` | Check `TINY_ENGINEER_URL`; the robot must be on the same network |
Loading