Orchestrator spec and reference impl

[rusty1968]

No description provided.

[leongross]

Do you plan to realize your state machine implementation based on https://github.com/typeconstructor/rot-reducer?

[chrysh]

@rusty1968 Should we also add multi-level cascades, concurrent faults, and double-fault recovery tests? Or is this not necessary?

Answer

All three are already covered: multi-level cascade (cascading_runtime_corruption_cascades_transitively), concurrent fault (corruption_while_recovering_retargets_to_new_component, corruption_during_update_discards_staged), and double fault (retry_cap_self_latches_via_emit). Happy to add a specific scenario these miss.

[chrysh]

Are we trying to restore the golden image several times or just once?

Answer

The state machine isn't aware of the source — note the effect is named RecoverComponent, not "restore golden image." Golden vs A/B vs last-known-good is the platform driver's call. The reducer only decides how many attempts: up to max_retry per component (set 1 for once).

[chrysh]

Even if we only ever update one device at a time, §5.3.3 background polling can report corruption of device B while device A is mid-update or mid-recovery. What does the machine do then?

[chrysh]

Maybe each device should have their own state machine?

[rusty1968]

Even if we only ever update one device at a time, §5.3.3 background polling can report corruption of device B while device A is mid-update or mid-recovery. What does the machine do then?

Excellent question - The scenario is covered by tests:

• corruption_in_updating_triggers_recovery / corruption_during_update_discards_staged — if a device is found corrupt while another device is being updated, the update is dropped and the machine switches to fixing the corrupt device, throwing away the half-finished update image.

• corruption_while_recovering_retargets_to_new_component / restored_for_wrong_component_does_not_advance_recovery — if a second device fails while we're still fixing the first, we switch to fixing the second; and if the first one later says "I'm fixed," we ignore it, since we'd already moved on.

• cascading_runtime_corruption_cascades_transitively — when the failed device is one we're allowed to simply switch off (rather than one we must stop everything to fix), we isolate it and everything that depends on it, and keep going instead of interrupting the current work.

[rusty1968]

Maybe each device should have their own state machine?

Your intuition is directionally correct — but what genuinely matters here is per-device state , and the design already carries it (ComponentStatus, retry_budget, isolation markers,etc). What stays singular is the ordering: we only fix one device at a time, and we only update one device at a time. The machine keeps a single "who am I fixing right now" slot (State::Recovering(id)), so it can't be pointed at two devices at once. That one-at-a-time behavior is on purpose, not a limitation we ran into.

[chrysh]

Maybe each device should have their own state machine?

Your intuition is directionally correct — but what genuinely matters here is per-device state , and the design already carries it (ComponentStatus, retry_budget, isolation markers,etc). What stays singular is the ordering: we only fix one device at a time, and we only update one device at a time. The machine keeps a single "who am I fixing right now" slot (State::Recovering(id)), so it can't be pointed at two devices at once. That one-at-a-time behavior is on purpose, not a limitation we ran into.

What happens if we get another update firmware request for device B while we are fixing the firmware if device A?

[rusty1968]

Maybe each device should have their own state machine?

Your intuition is directionally correct — but what genuinely matters here is per-device state , and the design already carries it (ComponentStatus, retry_budget, isolation markers,etc). What stays singular is the ordering: we only fix one device at a time, and we only update one device at a time. The machine keeps a single "who am I fixing right now" slot (State::Recovering(id)), so it can't be pointed at two devices at once. That one-at-a-time behavior is on purpose, not a limitation we ran into.

What happens if we get another update firmware request for device B while we are fixing the firmware if device A?

TL;DR

The UpdateRequest is silently dropped. Recovery is not interrupted, no effect is emitted, and nothing is queued. The requester gets no response at all.

That drop is deliberate as a policy (recovery outranks update — "fix first, then update"), but the silence is the part worth a second look: the requester cannot tell "refused, busy recovering" apart from "lost."

CSA Alignment

The CSA mandates that failures be reported and models updates as answered requests, which is exactly the gap I am fixing with 57fedb0

This is a sketch of how we use the report effect.

impl Platform for PlatformAst  {
    fn execute(&mut self, effect: Effect) -> Result<(), EffectError> {
        match effect {
            // ... device I/O effects (ReadFirmware, ReleaseReset, ...) ...
            Effect::ReportIsolated(id)       => self.mgmt.report_isolated(id),
            Effect::ReportRecoveryFailed(id) => self.mgmt.report_recovery_failed(id),
            // Pure protocol reply — no device I/O. Answer the pending PLDM
            // FW-update request so the requester gets a definite "busy, retry".
            Effect::ReportUpdateDeferred =>
                self.pldm.respond_update(PldmCompletion::RetryLater),
            // ...
        }
    }
}

The CSA does not mandate update-vs-recovery arbitration. Recovery is described as autonomous and separate from update; the interaction of an in-flight recovery with an incoming update is left to the implementation. Our single-flight, recovery-priority choice is therefore a legitimate implementation-defined policy, neither required nor forbidden by the CSA.

[chrysh]

@rusty1968 Should we add those test?:

/// INV9: the iRoT gate must hold even when the eRoT walk finishes first.
/// C0's `ComponentReady` is still outstanding when the last verdict
/// arrives; completing the walk must not bypass the gate.
#[test]
fn chain_exhaustion_does_not_bypass_irot_gate() {
    let (_effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::active_required()),
            (C1, ComponentAttrs::passive_required()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // C0 released; its iRoT still owes ComponentReady
            Event::VerificationPassed(C1), // walk done — but C0 never reported
        ],
    );
    assert_eq!(state, State::AwaitingReady(Some(C0)));
}

/// Trust invariant: a verdict only means something for the component under
/// verification. A replayed `VerificationPassed` for a component that has
/// since been isolated must not re-release it.
#[test]
fn replayed_verdict_does_not_release_isolated_component() {
    let (effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::passive_isolable()),
            (C1, ComponentAttrs::passive_required()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // released; walk moves to C1
            Event::CorruptionDetected(C0), // Isolable → AssertReset + Isolated
            Event::VerificationPassed(C0), // replayed / spurious verdict
        ],
    );
    let releases_c0 = effects
        .iter()
        .filter(|&&e| e == Effect::ReleaseReset(C0))
        .count();
    assert_eq!(releases_c0, 1, "isolated component was re-released");
    assert_ne!(state, State::Ready, "C1's verdict is still outstanding");
}

/// A healthy Active component must not be dragged into recovery by a stale
/// boot-progress watchdog. After a re-walk, C0's `ComponentReady` arrives
/// while the machine is back in `PreSupervision`; the report must clear its
/// watchdog (as `Booted` does there), making the later `Timeout` stale.
#[test]
fn component_ready_during_rewalk_clears_watchdog() {
    let (effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::active_required()),
            (C1, ComponentAttrs::passive_required()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // C0 released, watchdog armed
            Event::VerificationFailed(C1), // C1 fails → Recovering(C1)
            Event::Restored(C1),           // restore ok → re-walk from top
            Event::ComponentReady(C0),     // C0's iRoT reports in — discarded
            Event::Timeout(C0),            // stale watchdog fires
        ],
    );
    assert!(
        !effects.contains(&Effect::RestoreGoldenImage(C0)),
        "healthy, ready C0 was sent to recovery by a stale watchdog"
    );
    assert_ne!(state, State::Recovering(C0));
}

Answers

1. chain_exhaustion_does_not_bypass_irot_gate

Answer: We don't block in AwaitingReady; a finished walk goes straight to Ready, and a released active that never reports is caught by the boot watchdog, not by waiting.

**Covered by: ** single_active_chain_goes_directly_to_ready, timeout_awaited_enters_recovering

2. replayed_verdict_does_not_release_isolated_component

Answer: A gated component is never re-released: only the component at the cursor can be released, and a replayed verdict for an isolated one is dropped, so ReleaseReset fires once.

**Already Covered by: ** isolable_runtime_corruption_holds_across_rewalk, property_verify_before_release_holds_under_random_sequences

3. component_ready_during_rewalk_clears_watchdog

Answer: A re-walk clears each live component's watchdog via quiesce_all on entry, so a stale timeout afterward is dropped and a healthy component is never dragged into recovery.

**Already Covered by: ** passive_booted_clears_watchdog_then_timeout_is_stale, recovery_rewalk_reverifies_live_sibling_at_rest