From 8df0ac145b2345807c2a5d239d46bec666f20842 Mon Sep 17 00:00:00 2001 From: TimLeenaers Date: Wed, 21 Oct 2026 19:28:33 +0200 Subject: [PATCH 1/4] OptiView Live: document the backup ingest of an engine Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- theolive/api/full-example.mdx | 2 +- theolive/architecture.mdx | 2 + theolive/distribution/redundancy.md | 105 ++++++++++++++++++++++++++-- 3 files changed, 103 insertions(+), 6 deletions(-) diff --git a/theolive/api/full-example.mdx b/theolive/api/full-example.mdx index 66275df05d55..1dc5559ea946 100644 --- a/theolive/api/full-example.mdx +++ b/theolive/api/full-example.mdx @@ -256,7 +256,7 @@ The engines shut down gracefully: players stop playback at the end of the stream ## Next steps -- Add [redundancy](../distribution/redundancy.md) with a second ingest and engine +- Add [redundancy](../distribution/redundancy.md) with a second ingest and engine, or a [backup ingest](../distribution/redundancy.md#backup-ingest) on your engine - Configure [security](../distribution/security/geo-blocking.md) on your distribution - Set up [webhooks](../platform/real-time-update-with-webhooks.mdx) for real-time event notifications - Automate start/stop with [schedulers](../platform/scheduler.mdx) diff --git a/theolive/architecture.mdx b/theolive/architecture.mdx index c654c62978cb..5c467c9c11d9 100644 --- a/theolive/architecture.mdx +++ b/theolive/architecture.mdx @@ -35,6 +35,8 @@ A channel can have **multiple engines**, enabling different levels of redundancy In both cases, the engine **priority** setting determines which engine is active. A lower priority number means higher precedence, so the platform automatically falls back to the next engine when the primary one becomes unavailable. +An engine can also be given a **backup ingest**: when its primary ingest stops delivering, the engine switches to the backup source by itself and returns to the primary once it is back. This protects the contribution feed without running a second transcode. See [Redundancy](./distribution/redundancy.md#backup-ingest) for the details and how it compares to multiple engines. + ## Distribution A distribution represents a collection of engines and how they are delivered and secured over a CDN. diff --git a/theolive/distribution/redundancy.md b/theolive/distribution/redundancy.md index 8ad6e7bc5220..0b0f00f127a2 100644 --- a/theolive/distribution/redundancy.md +++ b/theolive/distribution/redundancy.md @@ -8,9 +8,14 @@ description: Configure failover and redundancy for high-availability streaming. # Redundancy -Redundancy ensures that viewers can continue watching even if part of the streaming pipeline fails. By connecting multiple engines to a single distribution, the platform can automatically fail over to a healthy engine when one becomes unavailable. +Redundancy ensures that viewers can continue watching even if part of the streaming pipeline fails. The platform offers two complementary levels of failover: -## How it works +- **Engine redundancy** — connect multiple engines to a single distribution, so the player fails over to a healthy engine when one becomes unavailable. +- **Backup ingest** — give an engine a second ingest to fall back to, so the engine itself keeps running when its primary source drops. + +## Engine redundancy + +### How it works A distribution can have multiple engines attached to it. Each engine processes the stream independently from its own ingest. When the player connects to a distribution, it receives the stream from the active engine. If that engine goes down, the player automatically fails over to the next available engine — without the viewer having to take any action. @@ -19,7 +24,7 @@ This approach protects the entire path from ingest to delivery: - **Ingest failure** — if an ingest source drops, only the engine connected to it is affected. The other engine, fed by a separate ingest, continues to serve the stream. - **Engine failure** — if an engine fails during transcoding or packaging, the player switches to the other engine on the same distribution. -## Setting up redundancy +### Setting up engine redundancy To set up a redundant channel: @@ -29,11 +34,11 @@ To set up a redundant channel: The engine **priority** determines which engine is active. A lower priority number means higher precedence. The platform serves the stream from the highest-priority engine that is available, and automatically falls back to the next one when needed. -## What the viewer experiences +### What the viewer experiences When a failover occurs, the player detects that the current engine is no longer available and switches to the other engine on the distribution. Viewers may see a brief interruption but do not need to reload or take any action. -### Example: geo-redundant setup with priorities +#### Example: geo-redundant setup with priorities Consider a sports broadcaster streaming a live match from a stadium in London. To protect against failures, they set up the following: @@ -54,3 +59,93 @@ Consider a sports broadcaster streaming a live match from a stadium in London. T **Engine A is taken out of service** (e.g. for maintenance): [Force stop](../platform/starting-and-stopping.mdx#force-stop) Engine A. Its playlist requests start failing, so OptiView Player fails over to Engine B just as it would on a failure. A regular stop would instead end the stream cleanly, and players would treat it as finished rather than fall back. This setup provides both **geographic redundancy** — the two engines run in separate locations, protecting against regional outages — and **priority-based failover** — the platform always prefers the highest-precedence engine that is available. + +## Backup ingest + +Engine redundancy protects against engine failures, but it doubles the transcoding cost: every engine runs its own transcode. When your main concern is the **contribution feed** — a flaky encoder, a congested network or a cut connection — a **backup ingest** is a lighter alternative. You give a single engine a second ingest to fall back to, and the engine switches sources by itself while it keeps running. + +Both options can be combined: each engine of a redundant setup can have its own backup ingest. + +### How it works + +An engine always has a **primary ingest** (`ingestId`). Optionally, you can set a **backup ingest** (`backupIngestId`) on it. The engine then behaves as follows: + +- **At start**, the engine pulls from the primary ingest. If the primary cannot be reached or is not delivering a stream, the engine switches to the backup within seconds instead of waiting for the primary. +- **While streaming**, if the active source drops, the engine reconnects. It tries the primary ingest first, so the engine returns to the primary as soon as it is available again, and only stays on the backup for as long as the primary is down. +- If **neither** ingest delivers a stream, the engine keeps alternating between them until one does or the channel's [auto-stop timeout](../platform/starting-and-stopping.mdx#auto-stop-timeout) expires, exactly as it does with a single ingest. + +The switch happens inside the engine, so it is handled like any ingest reconnect: viewers may notice a brief interruption, but they stay on the same engine and the same distribution. Nothing changes for the player, the CDN or your webhooks — the engine remains `playing` and no engine failover takes place. + +:::info +The engine does not switch away from a source that is still delivering media. If you want to move viewers to the backup feed while the primary is still up, stop the primary encoder. +::: + +### Setting up a backup ingest + +1. Create **two ingests** on your channel, both receiving the same live source (e.g. a main and a backup encoder). +2. Create an **engine** connected to the first ingest, and pass the second one as `backupIngestId`. + +The backup ingest can be set when creating an engine or added later with an update. Pass `null` to remove it again. + +`POST https://api.theo.live/v2/channels/{channelId}/engines` + +```bash +curl -X POST https://api.theo.live/v2/channels/ch_abc123/engines \ + -H "Authorization: Basic $AUTH" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "Engine with backup ingest", + "ingestId": "ing_primary", + "backupIngestId": "ing_backup", + "region": "europe-west", + "quality": { + "abrLadderId": "abr_abc123" + }, + "outputs": { + "hls": true + } + }' +``` + +`PATCH https://api.theo.live/v2/engines/{engineId}` + +```bash +curl -X PATCH https://api.theo.live/v2/engines/eng_ghi789 \ + -H "Authorization: Basic $AUTH" \ + -H "Content-Type: application/json" \ + -d '{ + "backupIngestId": "ing_backup" + }' +``` + +The engine response includes `backupIngestId` when one is set. Like the primary ingest, the backup ingest can only be changed while the engine is `stopped`. + +:::note +When you create a channel with nested ingests and engines in a single request, the engines are bound to the ingest they are listed under and cannot reference a backup yet. Set `backupIngestId` with an engine update once the channel exists. +::: + +### Requirements + +The primary and backup ingest form a pair, and the API validates the pair whenever you create or update the engine or either ingest: + +- The backup must be **another ingest of the same channel** — it cannot be the engine's primary ingest. +- Both ingests must be of type `srt-pull`, `srt-push`, `rtmp-pull` or `rtmp-push`. Other ingest types (such as SDI, NDI, RTSP, HLS or DASH) cannot take part in a pair. +- The backup ingest must be in the **same region** as the engine, like the primary. +- The backup ingest cannot have a [speech enhancement](../contribution/enhancements/speech.mdx) configured; live captioning and translation always follow the primary ingest. + +Requests that break one of these rules are rejected with a `400` error (`backup_ingest_not_supported` or `invalid_region`), and a `backupIngestId` that does not exist on the channel yields a `404`. This also applies to ingest updates: for example, changing the type of an ingest that is the backup of an engine to `sdi-pull` fails. + +An ingest that serves as the backup of an engine is treated like a connected ingest: + +- It **cannot be deleted** while an engine still references it (`409 engines_connected`). Unset or change the engine's `backupIngestId` first. +- Its **source settings** (`type`, `url`, `region`, `tracks`, `captions`, `enhancements`, `restream`) cannot be changed while any engine using it — as primary or as backup — is running. + +### Choosing between the two + +| | Engine redundancy | Backup ingest | +| ------------------ | ------------------------------------ | --------------------------------------- | +| Protects against | Ingest, engine and regional failures | Ingest (contribution) failures | +| Transcoding cost | One transcode per engine | One transcode | +| Who switches | The player, to another engine | The engine, to another source | +| Viewer impact | Brief interruption, new engine | Brief interruption, same engine | +| Returns to primary | New sessions only | Automatically, when the primary is back | From e1c7ff7f6d34169048c16cdfa6a91664e7630f00 Mon Sep 17 00:00:00 2001 From: TimLeenaers Date: Fri, 23 Oct 2026 11:36:14 +0200 Subject: [PATCH 2/4] Backup ingest: engine stays on the backup, recommend engine redundancy Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- theolive/architecture.mdx | 2 +- theolive/distribution/redundancy.md | 22 ++++++++++++---------- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/theolive/architecture.mdx b/theolive/architecture.mdx index 5c467c9c11d9..f362540f64f7 100644 --- a/theolive/architecture.mdx +++ b/theolive/architecture.mdx @@ -35,7 +35,7 @@ A channel can have **multiple engines**, enabling different levels of redundancy In both cases, the engine **priority** setting determines which engine is active. A lower priority number means higher precedence, so the platform automatically falls back to the next engine when the primary one becomes unavailable. -An engine can also be given a **backup ingest**: when its primary ingest stops delivering, the engine switches to the backup source by itself and returns to the primary once it is back. This protects the contribution feed without running a second transcode. See [Redundancy](./distribution/redundancy.md#backup-ingest) for the details and how it compares to multiple engines. +An engine can also be given a **backup ingest**: when its primary ingest stops delivering, the engine switches to the backup source by itself and stays on it. This protects the contribution feed without running a second transcode, but not against engine or regional failures, so multiple engines remain the recommended setup. See [Redundancy](./distribution/redundancy.md#backup-ingest) for the details and how it compares to multiple engines. ## Distribution diff --git a/theolive/distribution/redundancy.md b/theolive/distribution/redundancy.md index 0b0f00f127a2..e7542dd4ab70 100644 --- a/theolive/distribution/redundancy.md +++ b/theolive/distribution/redundancy.md @@ -64,15 +64,15 @@ This setup provides both **geographic redundancy** — the two engines run in se Engine redundancy protects against engine failures, but it doubles the transcoding cost: every engine runs its own transcode. When your main concern is the **contribution feed** — a flaky encoder, a congested network or a cut connection — a **backup ingest** is a lighter alternative. You give a single engine a second ingest to fall back to, and the engine switches sources by itself while it keeps running. -Both options can be combined: each engine of a redundant setup can have its own backup ingest. +A backup ingest only protects the contribution feed. It does not help when the engine itself or its region fails, so **engine redundancy remains the recommended setup** for production streams. Both options can be combined: each engine of a redundant setup can have its own backup ingest. ### How it works An engine always has a **primary ingest** (`ingestId`). Optionally, you can set a **backup ingest** (`backupIngestId`) on it. The engine then behaves as follows: - **At start**, the engine pulls from the primary ingest. If the primary cannot be reached or is not delivering a stream, the engine switches to the backup within seconds instead of waiting for the primary. -- **While streaming**, if the active source drops, the engine reconnects. It tries the primary ingest first, so the engine returns to the primary as soon as it is available again, and only stays on the backup for as long as the primary is down. -- If **neither** ingest delivers a stream, the engine keeps alternating between them until one does or the channel's [auto-stop timeout](../platform/starting-and-stopping.mdx#auto-stop-timeout) expires, exactly as it does with a single ingest. +- **While streaming**, if the primary ingest drops, the engine switches to the backup ingest. Once on the backup, it **stays on the backup** — it does not switch back to the primary when the primary comes back, so viewers are not interrupted a second time. To return to the primary ingest, restart the engine. +- If **neither** ingest delivers a stream, the engine keeps retrying until one does or the channel's [auto-stop timeout](../platform/starting-and-stopping.mdx#auto-stop-timeout) expires, exactly as it does with a single ingest. The switch happens inside the engine, so it is handled like any ingest reconnect: viewers may notice a brief interruption, but they stay on the same engine and the same distribution. Nothing changes for the player, the CDN or your webhooks — the engine remains `playing` and no engine failover takes place. @@ -142,10 +142,12 @@ An ingest that serves as the backup of an engine is treated like a connected ing ### Choosing between the two -| | Engine redundancy | Backup ingest | -| ------------------ | ------------------------------------ | --------------------------------------- | -| Protects against | Ingest, engine and regional failures | Ingest (contribution) failures | -| Transcoding cost | One transcode per engine | One transcode | -| Who switches | The player, to another engine | The engine, to another source | -| Viewer impact | Brief interruption, new engine | Brief interruption, same engine | -| Returns to primary | New sessions only | Automatically, when the primary is back | +| | Engine redundancy | Backup ingest | +| ------------------ | ------------------------------------ | ---------------------------------------- | +| Protects against | Ingest, engine and regional failures | Ingest (contribution) failures | +| Transcoding cost | One transcode per engine | One transcode | +| Who switches | The player, to another engine | The engine, to another source | +| Viewer impact | Brief interruption, new engine | Brief interruption, same engine | +| Returns to primary | New sessions only | No — stays on the backup until restarted | + +Engine redundancy is still the recommended way to protect a production stream, because it is the only option that also covers engine and regional failures. Use a backup ingest on its own when you accept that risk in exchange for a single transcode, or add one to each engine of a redundant setup to make the contribution feed more resilient as well. From cd60a705b7df787602d8a2712e9678f262cb39ed Mon Sep 17 00:00:00 2001 From: TimLeenaers Date: Fri, 23 Oct 2026 11:37:09 +0200 Subject: [PATCH 3/4] Backup ingest: engine stays on the backup, recommend engine redundancy Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- theolive/architecture.mdx | 2 +- theolive/distribution/redundancy.md | 20 ++++++++++---------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/theolive/architecture.mdx b/theolive/architecture.mdx index f362540f64f7..81d816a08ff5 100644 --- a/theolive/architecture.mdx +++ b/theolive/architecture.mdx @@ -35,7 +35,7 @@ A channel can have **multiple engines**, enabling different levels of redundancy In both cases, the engine **priority** setting determines which engine is active. A lower priority number means higher precedence, so the platform automatically falls back to the next engine when the primary one becomes unavailable. -An engine can also be given a **backup ingest**: when its primary ingest stops delivering, the engine switches to the backup source by itself and stays on it. This protects the contribution feed without running a second transcode, but not against engine or regional failures, so multiple engines remain the recommended setup. See [Redundancy](./distribution/redundancy.md#backup-ingest) for the details and how it compares to multiple engines. +An engine can also be given a **backup ingest**: when its primary ingest stops delivering, the engine switches to the backup source by itself and stays on it. This protects the contribution feed without running a second transcode; multiple engines remain the recommended setup. See [Redundancy](./distribution/redundancy.md#backup-ingest) for the details and how it compares to multiple engines. ## Distribution diff --git a/theolive/distribution/redundancy.md b/theolive/distribution/redundancy.md index e7542dd4ab70..38324513fda8 100644 --- a/theolive/distribution/redundancy.md +++ b/theolive/distribution/redundancy.md @@ -64,14 +64,14 @@ This setup provides both **geographic redundancy** — the two engines run in se Engine redundancy protects against engine failures, but it doubles the transcoding cost: every engine runs its own transcode. When your main concern is the **contribution feed** — a flaky encoder, a congested network or a cut connection — a **backup ingest** is a lighter alternative. You give a single engine a second ingest to fall back to, and the engine switches sources by itself while it keeps running. -A backup ingest only protects the contribution feed. It does not help when the engine itself or its region fails, so **engine redundancy remains the recommended setup** for production streams. Both options can be combined: each engine of a redundant setup can have its own backup ingest. +Engine redundancy remains the recommended setup. Both options can be combined: each engine of a redundant setup can have its own backup ingest. ### How it works An engine always has a **primary ingest** (`ingestId`). Optionally, you can set a **backup ingest** (`backupIngestId`) on it. The engine then behaves as follows: - **At start**, the engine pulls from the primary ingest. If the primary cannot be reached or is not delivering a stream, the engine switches to the backup within seconds instead of waiting for the primary. -- **While streaming**, if the primary ingest drops, the engine switches to the backup ingest. Once on the backup, it **stays on the backup** — it does not switch back to the primary when the primary comes back, so viewers are not interrupted a second time. To return to the primary ingest, restart the engine. +- **While streaming**, if the primary ingest drops, the engine switches to the backup ingest and stays on it, even when the primary comes back. - If **neither** ingest delivers a stream, the engine keeps retrying until one does or the channel's [auto-stop timeout](../platform/starting-and-stopping.mdx#auto-stop-timeout) expires, exactly as it does with a single ingest. The switch happens inside the engine, so it is handled like any ingest reconnect: viewers may notice a brief interruption, but they stay on the same engine and the same distribution. Nothing changes for the player, the CDN or your webhooks — the engine remains `playing` and no engine failover takes place. @@ -142,12 +142,12 @@ An ingest that serves as the backup of an engine is treated like a connected ing ### Choosing between the two -| | Engine redundancy | Backup ingest | -| ------------------ | ------------------------------------ | ---------------------------------------- | -| Protects against | Ingest, engine and regional failures | Ingest (contribution) failures | -| Transcoding cost | One transcode per engine | One transcode | -| Who switches | The player, to another engine | The engine, to another source | -| Viewer impact | Brief interruption, new engine | Brief interruption, same engine | -| Returns to primary | New sessions only | No — stays on the backup until restarted | +| | Engine redundancy | Backup ingest | +| ------------------ | ------------------------------------ | ------------------------------- | +| Protects against | Ingest, engine and regional failures | Ingest (contribution) failures | +| Transcoding cost | One transcode per engine | One transcode | +| Who switches | The player, to another engine | The engine, to another source | +| Viewer impact | Brief interruption, new engine | Brief interruption, same engine | +| Returns to primary | New sessions only | No | -Engine redundancy is still the recommended way to protect a production stream, because it is the only option that also covers engine and regional failures. Use a backup ingest on its own when you accept that risk in exchange for a single transcode, or add one to each engine of a redundant setup to make the contribution feed more resilient as well. +Engine redundancy is the recommended setup. From 50abe2b1261a171bde14ea53b30c13d4a7dba45b Mon Sep 17 00:00:00 2001 From: TimLeenaers Date: Fri, 23 Oct 2026 11:45:29 +0200 Subject: [PATCH 4/4] Mark engine redundancy as recommended in the overview Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- theolive/distribution/redundancy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/theolive/distribution/redundancy.md b/theolive/distribution/redundancy.md index 38324513fda8..0db4f864662d 100644 --- a/theolive/distribution/redundancy.md +++ b/theolive/distribution/redundancy.md @@ -10,7 +10,7 @@ description: Configure failover and redundancy for high-availability streaming. Redundancy ensures that viewers can continue watching even if part of the streaming pipeline fails. The platform offers two complementary levels of failover: -- **Engine redundancy** — connect multiple engines to a single distribution, so the player fails over to a healthy engine when one becomes unavailable. +- **Engine redundancy** (recommended) — connect multiple engines to a single distribution, so the player fails over to a healthy engine when one becomes unavailable. - **Backup ingest** — give an engine a second ingest to fall back to, so the engine itself keeps running when its primary source drops. ## Engine redundancy