Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ web/print/files/
.pio
.pio-home
compile_commands.json
# Written by the PlatformIO VS Code extension; machine-specific paths
.vscode/
.gitignore.optional

# Prerequisites
*.d
Expand Down
17 changes: 14 additions & 3 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,12 @@ curl -H "Authorization: Bearer YOUR_TOKEN" http://tiny-engineer.local/health
"provisioning": false,
"setup_ap_ssid": "",
"setup_ap_ip": "",
"oled": true
"oled": true,
"i2c": {
"ok": true,
"pca9685": { "ok": true, "faults": 0, "recoveries": 0, "last_fault_ms": 0 },
"oled": { "ok": true, "faults": 0, "recoveries": 0, "last_fault_ms": 0 }
}
}
```

Expand All @@ -140,7 +145,13 @@ curl -H "Authorization: Bearer YOUR_TOKEN" http://tiny-engineer.local/health
| `provisioning` | `true` when setup AP mode is active |
| `setup_ap_ssid` | Setup AP SSID when `provisioning` is true, else `""` |
| `setup_ap_ip` | Setup AP IP when `provisioning` is true (usually `192.168.4.1`), else `""` |
| `oled` | SSD1306 probed and initialized |
| `oled` | SSD1306 probed and initialized. Goes `false` while an I2C fault keeps the display unreachable |
| `i2c.ok` | `false` while either chip on the shared I2C bus cannot be used. The firmware checks each every 500 ms and sets a faulted chip up again, retrying with a growing delay (1 s up to 30 s) |
| `i2c.pca9685.ok` | Servo driver usable: reachable, holding its setup (awake, internal clock, auto-increment, output mode, prescale), and its last write succeeded. Goes `false` at the first failed servo write; servo writes and outputs stay off until the chip is set up again |
| `i2c.oled.ok` | Same value as `oled`. Stays `false`, with zero counts, on a robot that booted without a display |
| `i2c.<chip>.faults` | Faults on that chip since boot. A bus fault that takes both chips down counts once on each |
| `i2c.<chip>.recoveries` | Faults on that chip the firmware recovered from since boot |
| `i2c.<chip>.last_fault_ms` | `uptime_ms` at that chip's latest fault; `0` when there has been none |

### `GET /settings`

Expand Down Expand Up @@ -540,7 +551,7 @@ tiny-engineer park --transport serial
{ "ok": true, "park": "servos", "hold": true }
```

Pose NVS write failure → **400** `{"ok":false,"error":"pose save failed"}` (PWM hold is still armed). Details: [hardware/servos.md](hardware/servos.md#flash-safe-park).
Pose NVS write failure → **400** `{"ok":false,"error":"pose save failed"}` (PWM hold is still armed). Servo driver faulted (`i2c.pca9685.ok` false in [`GET /health`](#get-health)) → **503** `{"ok":false,"error":"servo driver down"}`: the joints did not move, so the pose is not saved and no PWM hold is armed. Details: [hardware/servos.md](hardware/servos.md#flash-safe-park).

### `POST /test/servo`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ Failure table and expected boot sequence: [testing.md](../hardware/testing.md).

Once on Wi-Fi, the robot exposes **integration-test-style endpoints**:

- `/health` — alive, CPU temp
- `/health` — alive, CPU temp, I2C fault and recovery counts
- `/test/servo` — single joint
- Web UI — hardware tests without memorizing curl

Expand Down
19 changes: 17 additions & 2 deletions docs/hardware/interfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,27 @@ Shared bus.
| SDA | GPIO0 | GP0 → PCA9685 SDA, OLED SDA |
| SCL | GPIO1 | GP1 → PCA9685 SCL, Waveshare OLED **SCL** |

| Device | Address | Firmware behaviour if missing |
| Device | Address | Firmware behaviour if missing at boot |
| --- | --- | --- |
| PCA9685 | **`0x40`** | Hard fail: hang, 1 red blink (see [blink codes](testing.md#boot-failure-blink-codes)) |
| SSD1306 OLED | **`0x3C`** | Soft fail: tests continue without display |

Bus start: `Wire.begin(I2C_SDA, I2C_SCL)` with no explicit frequency → Arduino-ESP32 default (typically **100 kHz**). Not set in this repo.
That table is the boot check. After boot the firmware keeps watching both devices (`src/hardware/i2c_health.cpp`):

- **Check:** every 500 ms the OLED must answer, and the PCA9685 must answer, be awake on its internal clock with auto-increment on, and still hold its output mode (`MODE2 = 0x04`) and prescale. Two failed checks in a row are a fault. Each chip is tracked separately, so one that stays down does not stop the checks on the other. A healthy PCA9685 check is two register reads: `MODE1` and `MODE2` together, then the prescale.
- **Failed servo write:** a fault on its own. The first one takes the PCA9685 down at once and switches its outputs off, even inside a blocking move that never reaches the check. So does a boot setup that does not read back correctly.
- **While down:** writes to the faulted chip are skipped, so failed transfers do not stall audio and HTTP. At a PCA9685 fault the servo outputs are switched off (OE), so a corrupted setup cannot keep driving them. That needs OE driven by the firmware, the default; with `PCA9685_OE_WIRED = false` a fault cannot switch the outputs off ([PCA9685 output enable](#pca9685-output-enable)).
- **PCA9685 recovery:** if the chip does not answer, restart `Wire` and clock out a device left holding SDA first. A chip that kept its setup only lost a write or a read. A chip that lost its setup lost power or was corrupted: set it up again, and set the OLED up again too afterwards, since both share a supply. Either way, the outputs come back on with every servo channel off, and then every channel is written and read back. The PCA9685 applies channel values at the end of a LOW cycle, so each servo starts on a whole pulse; turning OE on mid-pulse would cut one short. Servos jump to where the firmware thinks they are. Outputs stay disabled unless the setup and every channel read back correctly.
- **OLED recovery:** send the panel its setup and the last frame, so a static screen survives too. Every transfer is checked, since the SSD1306 driver ignores failed writes and a half-configured panel still answers its address. The current rotation is kept. Never restarts the bus, so an OLED fault alone does not disturb the servos.
- **Audio:** a recovery runs in steps, one per loop pass, and a pass services at most one chip. Transfers time out after 20 ms (Wire's default is 50 ms), so a pass costs about one timeout at worst, well inside the speaker's DMA buffer of about 65 ms.
- **Retry:** a failed attempt is retried after 1 s, doubling up to 30 s.
- **Report:** `[i2c]` lines on the serial log (with SDA / SCL levels and, for a PCA9685 that answered, its registers) and the `i2c` object in [`GET /health`](../api.md#get-health).

The OLED check is an address ACK only. A panel that loses its setup but still answers is not detected by itself; it is set up again whenever the PCA9685 recovers from a lost setup.

The PCA9685 keeps its registers across an ESP32 reset; only its own power cycle clears them. That is why setup writes `MODE2` explicitly instead of trusting the power-on default.

Bus start: the lines are cleared the same way, then `Wire.begin(I2C_SDA, I2C_SCL)` with no explicit frequency → Arduino-ESP32 default (typically **100 kHz**). Not set in this repo.

PCA9685 **VCC** = **3V3**. OLED **VCC** = **3V3**. Pull-ups are on the main control board and on a typical OLED; the Adafruit PCA9685 breakout has its own. No extra pull-ups.

Expand Down
13 changes: 11 additions & 2 deletions docs/hardware/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,15 +140,24 @@ HTTP runs on STA when connected, or on setup AP at `192.168.4.1` during provisio
| --- | --- | --- |
| OLED `ERROR: OLED not found` then rest of boot runs | Nothing ACK’d at `0x3C` | OLED **VCC=3V3**, GND, SDA=GP0, **SCL**=GP1, common ground, address jumper still 0x3C |
| OLED found but `ERROR: OLED initialization failed` | ACK then `display.begin` failed | Wiring/power glitch, wrong size module, I2C noise |
| OLED found but `ERROR: OLED setup failed; retrying after boot` | ACK, but a setup transfer failed | Retries by itself: an `[i2c] oled FAULT` and `recovered` line follow. Repeats point at wiring, power or I2C noise |
| OLED shows `Join this WiFi` / AP name, then `Then open` / `192.168.4.1` | Setup AP mode active | Connect to the shown AP, open `http://192.168.4.1/config`, finish the servo + screen + LED + speaker + WiFi wizard |
| OLED `WiFi failed` then setup AP | No saved network connected | Join setup AP, open `http://192.168.4.1/config`, enter home WiFi again |
| `ERROR: PCA9685 not found` + red RGB + **hang** | Nothing ACK’d at `0x40` | PCA9685 **VCC=3V3** (not V+), GND, SDA/SCL, I2C address pads, +5V not required for the ACK but needed later for motion |
| `ERROR: I2S initialization failed` + red RGB + **hang** | `I2S.begin` failed | GPIO2/3/4 not shorted to 5V/GND; pin constants; USB CDC still alive so you can read the line |
| I2S OK but `POST /test/audio` is silent | Amp or speaker | MAX98357A **Vin**=USB 5V, GND, GP2/3/4 → BCLK/LRC/DIN, speaker on **SPK+ / SPK-** (not on the PNG, not GND) |
| Servos silent / twitch / ESP32 resets during `POST /test/movement` | Power or SIG | **V+** is +5V, SIG on ch 0–4, **common GND**, supply current — see [power.md](power.md) |
| RGB never goes green | GPIO10 LED path | Board is C3-Zero (LED on GPIO10). Do not expect an external NeoPixel |

OLED absence and Wi-Fi failure are **soft** fails. PCA9685 and I2S failures **halt** in `while (true)`.
| `[i2c] pca9685 FAULT #N` and `[i2c] oled FAULT #N` together, then `[i2c] bus restarted` and `recovered` lines | Both chips stopped answering after boot; firmware restarted the bus | `sda=LOW` / `scl=LOW` on the `bus restarted` line means a line was held: supply dip or wiring. Check supply current — see [power.md](power.md) |
| `[i2c] pca9685 FAULT #N … mode1=… mode2=… prescale=…` | PCA9685 answered but lost its setup (reset to defaults, or a corrupted write) | Recovers by itself without a bus restart. Repeats point at supply dips or I2C noise |
| `[i2c] pca9685 FAULT #N … write failed` | A servo write failed; servo writes and outputs stopped at once | Recovers by itself, usually by rewriting the channels only. Repeats point at supply dips or I2C noise |
| `[i2c] pca9685 FAULT #N … setup failed` | The PCA9685 setup did not read back correctly at boot; outputs stayed off | Recovers by itself once the checks start. Repeats point at supply dips or I2C noise |
| `[i2c] oled FAULT #N` alone | Only the OLED stopped answering | The servos are left alone. OLED **VCC**, GND, SDA, SCL, connector |
| `[i2c] <chip> still down … retry in … ms` keeps repeating | Bus or chip not coming back | Wiring, PCA9685 / OLED **VCC**, then a full power cycle |
| `[i2c] Wire.end failed` or `[i2c] Wire.begin failed` | The ESP32's I2C driver did not restart; both chips stay off until a later attempt succeeds | Report it with the serial log; a reset clears it |
| `i2c.<chip>.faults` above `0` in `GET /health` | A fault happened since boot | `recoveries` equal to `faults` means all were recovered; `last_fault_ms` says when |

OLED absence and Wi-Fi failure are **soft** fails. A PCA9685 that does not answer at boot, and an I2S failure, **halt** in `while (true)`. A PCA9685 that answers but whose setup does not read back is a fault instead (`setup failed`): boot continues with the servo outputs off, and the checks set it up again. I2C faults after boot do not halt: the firmware sets the faulted chip up again, restarting the bus when the PCA9685 stops answering ([interfaces.md](interfaces.md#i2c)).

## Power during the servo test

Expand Down
1 change: 1 addition & 0 deletions platformio.ini
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ build_src_filter =
+<serial/protocol.cpp>
+<extensions/push_button_logic.cpp>
+<extensions/pin_claims.cpp>
+<hardware/i2c_fault.cpp>
build_flags = -std=c++17 -Iinclude -Isrc

; Opt-in OLED-only application. Building does not upload or change robot settings.
Expand Down
14 changes: 14 additions & 0 deletions src/api/commands.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#include "audio/audio.h"
#include "display/oled.h"
#include "extensions/push_button.h"
#include "hardware/i2c_health.h"
#include "hardware/pca9685_servos.h"
#include "hardware/rgb.h"
#include "hardware/servo_wrapper.h"
Expand Down Expand Up @@ -119,9 +120,22 @@ ApiResult apiMoveServo(int index, float angle) {
}

ApiResult apiParkServos() {
// A faulted servo driver would skip the move's writes, while the move still
// marks the centred pose for saving. Refuse before anything changes.
if (!i2cServoDriverUp()) {
return {false, 503, "servo driver down"};
}

setAnimationImmediately(AnimationId::None);
ensureAllServoOutputs();
centerAllServos();

// The driver faulted during the move: the joints never got to mid, so
// neither save that pose nor report a park.
if (!i2cServoDriverUp()) {
return {false, 503, "servo driver down"};
Comment thread
jamro marked this conversation as resolved.
Resolved
}

const bool poseSaved = flushServoPoseToNvs();
setServoParkHoldUntilReboot(true);

Expand Down
8 changes: 8 additions & 0 deletions src/display/oled.h
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@ void showTextThenIdle(const char* line1, const char* line2 = nullptr);
void showIdleScreen();

void initOled();
// True when the OLED answered at boot and its driver started, even if its
// setup then failed: the I2C checks watch it and retry that setup.
bool oledDetectedAtBoot();
// Sets the OLED up again after an I2C fault and sends the last frame, so a
// static screen survives too. Checks every transfer; false leaves the OLED
// unavailable. With panelOn false the panel stays dark but holds the frame for
// when it wakes. Keeps the current rotation.
bool restartOled(bool panelOn);
void runOledTest();

void showServoProgress(
Expand Down
119 changes: 113 additions & 6 deletions src/display/oled_init.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,95 @@ bool i2cDeviceConnected(uint8_t address) {
return Wire.endTransmission() == 0;
}

namespace {

// The panel answered at boot and the driver has its frame buffer, whether or
// not the setup then went through. Separate from oledAvailable, which is
// whether it can be drawn on right now.
bool g_oledDetected = false;

constexpr size_t kOledFrameBytes = OLED_WIDTH * ((OLED_HEIGHT + 7) / 8);
// Frame bytes per transfer, well inside Wire's buffer.
constexpr size_t kOledFrameChunk = 32;
// The driver runs its own transfers at 400 kHz and drops back to 100 kHz for
// the PCA9685 (its clkDuring / clkAfter defaults). The checked transfers here
// do the same, or a frame would take four times as long.
constexpr uint32_t kOledClockHz = 400000;
constexpr uint32_t kBusClockHz = 100000;

// What begin() sends a 128x32 panel on its internal charge pump, minus the
// final DISPLAYON. The driver ignores failed writes, and a panel that missed
// part of this still answers its address, so this copy checks each transfer.
constexpr uint8_t kOledSetup[] = {
SSD1306_DISPLAYOFF,
SSD1306_SETDISPLAYCLOCKDIV, 0x80,
SSD1306_SETMULTIPLEX, OLED_HEIGHT - 1,
SSD1306_SETDISPLAYOFFSET, 0x00,
SSD1306_SETSTARTLINE | 0x00,
SSD1306_CHARGEPUMP, 0x14,
SSD1306_MEMORYMODE, 0x00,
SSD1306_SEGREMAP | 0x01,
SSD1306_COMSCANDEC,
SSD1306_SETCOMPINS, 0x02,
SSD1306_SETCONTRAST, 0x8F,
SSD1306_SETPRECHARGE, 0xF1,
SSD1306_SETVCOMDETECT, 0x40,
SSD1306_DISPLAYALLON_RESUME,
SSD1306_NORMALDISPLAY,
SSD1306_DEACTIVATE_SCROLL
};
static_assert(OLED_WIDTH == 128 && OLED_HEIGHT == 32, "kOledSetup is for a 128x32 panel");

constexpr uint8_t kOledFrameWindow[] = {
SSD1306_PAGEADDR, 0x00, 0xFF,
SSD1306_COLUMNADDR, 0x00, OLED_WIDTH - 1
};

// One I2C transfer: a control byte (0x00 commands, 0x40 frame data), then
// the bytes. True when the panel acknowledged all of it.
bool sendOled(uint8_t control, const uint8_t* bytes, size_t count) {
Wire.beginTransmission(OLED_ADDRESS);
Wire.write(control);
Wire.write(bytes, count);
return Wire.endTransmission() == 0;
}

bool sendOledSetupAndFrame(bool panelOn) {
const uint8_t* frame = display.getBuffer();

if (frame == nullptr ||
!sendOled(0x00, kOledSetup, sizeof(kOledSetup)) ||
!sendOled(0x00, kOledFrameWindow, sizeof(kOledFrameWindow))) {
return false;
}

for (size_t offset = 0; offset < kOledFrameBytes; offset += kOledFrameChunk) {
if (!sendOled(0x40, frame + offset, kOledFrameChunk)) {
return false;
}
}

if (!panelOn) {
return true;
}

const uint8_t on = SSD1306_DISPLAYON;
return sendOled(0x00, &on, 1);
}

// Sends the panel its full setup and the frame buffer, then turns it on when
// panelOn. Safe to repeat: a panel that lost power comes back dark until it
// gets this again. The frame goes over while the panel is still off, so it
// wakes showing it. Leaves the frame buffer and the rotation as they are.
bool configureOled(bool panelOn) {
Wire.setClock(kOledClockHz);
const bool ok = sendOledSetupAndFrame(panelOn);
Wire.setClock(kBusClockHz);
return ok;
}

} // namespace

void initOled() {
serialLogPrintln();
serialLogPrintln("Checking OLED at 0x3C...");
Expand All @@ -25,21 +114,39 @@ void initOled() {

serialLogPrintln("OLED found");

if (!display.begin(
SSD1306_SWITCHCAPVCC,
OLED_ADDRESS
)) {
// begin() allocates and clears the frame buffer and sets the driver up;
// configureOled() then sends the setup again with each transfer checked.
if (!display.begin(SSD1306_SWITCHCAPVCC, OLED_ADDRESS)) {
serialLogPrintln("ERROR: OLED initialization failed");
oledAvailable = false;
return;
}

g_oledDetected = true;
display.setRotation(settingsOledRotate180() ? 2 : 0);

display.clearDisplay();
display.display();
// A setup that did not go through leaves the panel detected but not
// available: the I2C checks then fault it and retry the setup.
if (!configureOled(true)) {
serialLogPrintln("ERROR: OLED setup failed; retrying after boot");
oledAvailable = false;
return;
}

oledAvailable = true;

serialLogPrintln("OLED initialized");
}

// Not begin(): it clears the frame buffer and ignores failed writes. The
// rotation is left alone too, since the setup wizard may be previewing one it
// has not saved. The frame buffer still holds the last frame, so a static
// screen comes back too.
bool oledDetectedAtBoot() {
return g_oledDetected;
}

bool restartOled(bool panelOn) {
oledAvailable = i2cDeviceConnected(OLED_ADDRESS) && configureOled(panelOn);
return oledAvailable;
}
71 changes: 71 additions & 0 deletions src/hardware/i2c_fault.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#include "hardware/i2c_fault.h"

I2cFaultAction I2cFaultTracker::due(uint32_t now) const {
// Unsigned subtraction keeps the intervals right across the millis() wrap.
if (down_) {
return now - lastAttemptMs_ >= retryDelayMs_
? I2cFaultAction::Recover
: I2cFaultAction::None;
}

return !checked_ || now - lastCheckMs_ >= kI2cCheckIntervalMs
? I2cFaultAction::Check
: I2cFaultAction::None;
}

bool I2cFaultTracker::noteCheck(bool ok, uint32_t now) {
checked_ = true;
lastCheckMs_ = now;

if (ok) {
failures_ = 0;
return false;
}

if (++failures_ < kI2cFailuresToFault) {
return false;
}

takeDown(now);
return true;
}

bool I2cFaultTracker::noteFault(uint32_t now) {
if (down_) {
return false;
}

takeDown(now);
return true;
}

void I2cFaultTracker::takeDown(uint32_t now) {
down_ = true;
faults_++;
lastFaultMs_ = now;
attempts_ = 0;
// The first recovery attempt runs right away.
retryDelayMs_ = 0;
lastAttemptMs_ = now;
}

void I2cFaultTracker::noteRecovery(bool ok, uint32_t now) {
attempts_++;
lastAttemptMs_ = now;

if (ok) {
down_ = false;
failures_ = 0;
recoveries_++;
lastCheckMs_ = now;
return;
}

if (retryDelayMs_ == 0) {
retryDelayMs_ = kI2cFirstRetryMs;
} else if (retryDelayMs_ >= kI2cMaxRetryMs / 2) {
retryDelayMs_ = kI2cMaxRetryMs;
} else {
retryDelayMs_ *= 2;
}
}
Loading
Loading