Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@
[submodule "packages/python-server-sdk"]
path = packages/python-server-sdk
url = https://github.com/fishjam-cloud/python-server-sdk.git
[submodule "api/smelter-cloud"]
path = api/smelter-cloud
url = git@github.com:fishjam-cloud/foundry.git
1 change: 1 addition & 0 deletions api/smelter-cloud
Submodule smelter-cloud added at 7d59e1
9 changes: 8 additions & 1 deletion docs/api/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ type: reference

Describes APIs for direct interaction with Fishjam.

Fishjam publishes documentation for the Sandbox API and Fishjam Server APIs.
Fishjam publishes documentation for the Sandbox API, Fishjam Server APIs, and the Smelter Cloud API.

## Sandbox API

Expand Down Expand Up @@ -51,3 +51,10 @@ the first message that must be sent is an `AuthRequest`,
with a valid Management Token.

Next, you can should subscribe to notifications by sending `SubscribeRequest` event with `SERVER_NOTIFICATION` event type.

## Smelter Cloud

[Smelter Cloud](../smelter/what-is-smelter-cloud) (beta) exposes a REST API for managing compositions and a WebSocket stream for engine events.

- [Smelter Cloud API](./smelter-cloud): authentication, the WebSocket event stream, and errors.
- [Smelter Cloud REST API Reference](/api/smelter-cloud/rest)
61 changes: 61 additions & 0 deletions docs/api/smelter-cloud.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
type: reference
title: Smelter Cloud API
description: Smelter Cloud REST API reference, the WebSocket event stream, and the authentication model.
---

# Smelter Cloud API

Smelter Cloud exposes a REST API for managing compositions, plus a WebSocket stream for engine events. All requests go to `https://rtc.fishjam.io`. For an introduction to the service, see [What is Smelter Cloud?](../smelter/what-is-smelter-cloud)

## REST API

The full REST reference, with every endpoint, request body, and response schema, is published as an OpenAPI document:

- [Smelter Cloud REST API Reference](/api/smelter-cloud/rest)
- [OpenAPI specification](https://github.com/fishjam-cloud/documentation/blob/main/static/api/smelter-cloud-openapi.json)

The specification is generated from the service's source code and republished together with documentation updates.

## WebSocket event stream

Some engine events (for example, an output finishing) are delivered over a WebSocket rather than HTTP. Connect to:

```
GET wss://rtc.fishjam.io/api/composition/{composition_id}/ws
```

Because browsers cannot set an `Authorization` header on a WebSocket, authentication rides on the `Sec-WebSocket-Protocol` header, which must carry **two** subprotocols (order does not matter):

- `json.fishjam.io`: selects the JSON wire format.
- `bearer.auth.fishjam.io.<token>`: your token, appended to the literal prefix.

```js
const ws = new WebSocket(
`wss://rtc.fishjam.io/api/composition/${compositionId}/ws`,
["json.fishjam.io", `bearer.auth.fishjam.io.${token}`],
);
```

Messages are JSON text frames, each with a `type` field identifying the event.

## Authentication

| What you're calling | How it authenticates |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Control-plane calls (composition, inputs, outputs, renderers, events, room) | `Authorization: Bearer <token>`: your Fishjam **Management Token**, the same token used across Fishjam. |
| Publishing to an input (`/whip/{input_id}`) | The **input's** own bearer token, returned when you register a `whip_server` input (or the one you supplied). This is distinct from your account token. |
| Playing back an output (`/whep/{output_id}`) | Optional. Public playback is allowed; add a Bearer token only if you want to require one. |
| The WebSocket event stream | The subprotocol scheme described above. |

Get your Management Token from the [**Fishjam developer panel**](https://fishjam.io/app).

## Errors

Every non-2xx response is a JSON object:

```json
{ "message": "Composition not found", "http_status_code": 404 }
```

Common statuses are `400` (bad request), `401` (unauthorized), `404` (not found), `500` (server error), and `503` (no capacity, returned by composition creation).
11 changes: 11 additions & 0 deletions docs/smelter/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"label": "Smelter Cloud",
"position": 1,
"link": {
"type": "generated-index",
"title": "Smelter Cloud",
"description": "Hosted real-time video composition. Combine live streams, templates, and overlays into a single output stream, without running any rendering infrastructure yourself.",
"slug": "/smelter"
},
"collapsible": false
}
11 changes: 11 additions & 0 deletions docs/smelter/how-to/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"label": "How-to Guides",
"position": 3,
"link": {
"type": "generated-index",
"title": "Smelter Cloud How-to Guides",
"description": "Task-oriented guides for composing streams, writing templates, and driving compositions with Smelter Cloud.",
"slug": "/smelter/how-to"
},
"collapsible": false
}
65 changes: 65 additions & 0 deletions docs/smelter/how-to/compose-a-fishjam-room.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
type: how-to
sidebar_position: 2
description: Link a Fishjam room to a Smelter Cloud composition, render its peers with a template, and push the result to a livestream.
---

# Compose a Fishjam room

Smelter Cloud can compose the peers of a [Fishjam room](./../../explanation/rooms) directly. Linking a room forwards each peer's media into the composition as inputs, so a [template](./write-and-deploy-a-template) can render one tile per participant and push the result to a [livestream](./../../explanation/livestreams).

```
Fishjam room (peers) ──forwarded──▶ Smelter Cloud (template) ──WHIP──▶ Fishjam livestream ──WHEP──▶ [viewers]
```

## Prerequisites

- A composition (see [Quick Start](./../quick-start)).
- A Fishjam room and a [livestream](./../../explanation/livestreams) to publish to. Create both with your Fishjam **Management Token**.
- A built template bundle (see [Write and deploy a template](./write-and-deploy-a-template)).

## Step 1: Link the room

Link the room to the composition. Each connected peer's tracks then become available to the template as inputs.

```bash
curl -X POST "$SMELTER_URL/api/composition/$COMPOSITION/room" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"fishjam_id": "<YOUR_FISHJAM_ID>",
"room_id": "<ROOM_ID>"
}'
```

Your template's `usePeers()` now returns the room's peers, and it re-renders as they join, leave, mute, or start speaking.

## Step 2: Register a templated output to the livestream

Register a templated `whip_client` output that pushes to your Fishjam livestream's WHIP endpoint. The output configuration and the template bundle go together in one multipart request:

```bash
curl -X POST "$SMELTER_URL/api/composition/$COMPOSITION/output/main/template" \
-H "Authorization: Bearer $TOKEN" \
-F 'config={
"type": "whip_client",
"endpoint_url": "<LIVESTREAM_WHIP_URL>",
"bearer_token": "<LIVESTREAM_STREAMER_TOKEN>",
"video": { "resolution": { "width": 1280, "height": 720 }, "initial": { "root": { "type": "view" } } },
"audio": { "initial": { "inputs": [] } }
};type=application/json' \
-F "template=@dist/App.js"
```

Viewers can now watch the composed grid through the livestream's WHEP endpoint.

## Step 3: Unlink when done

Unlink the room to stop forwarding its peers into the composition:

```bash
curl -X DELETE "$SMELTER_URL/api/composition/$COMPOSITION/room" \
-H "Authorization: Bearer $TOKEN"
```

Then unregister the output and delete the composition as usual.
68 changes: 68 additions & 0 deletions docs/smelter/how-to/drive-a-template-with-events.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
type: how-to
sidebar_position: 3
description: Send custom events to a running Smelter Cloud composition to update a template's on-screen state at runtime.
---

# Drive a template with events

Once a [template](./write-and-deploy-a-template) is running, you can push **custom events** to it from your backend to change what it shows, such as toggling a live badge or updating a caption, without re-uploading the template.

## Handle events in the template

Inside a template, subscribe to events with the `eventBus`. Each `on` call returns an unsubscribe function. Event names and payloads are entirely defined by your application.

```tsx
// @jsx: react-jsx
// ---cut-before---
import { useEffect, useState } from "react";
import { Text, View } from "@swmansion/smelter";
import { eventBus } from "@fishjam-cloud/composition";

export default function App() {
const [caption, setCaption] = useState<string | null>(null);
const [live, setLive] = useState(false);

useEffect(() => {
const unsubs = [
eventBus.on<{ text: string }>("SET_CAPTION", ({ text }) =>
setCaption(text),
),
eventBus.on("CLEAR_CAPTION", () => setCaption(null)),
eventBus.on<{ live: boolean }>("SET_LIVE", ({ live }) => setLive(live)),
];
return () => unsubs.forEach((off) => off());
}, []);

return (
<View>
{live && <Text>● LIVE</Text>}
{caption && <Text>{caption}</Text>}
</View>
);
}
```

## Send events from your backend

Send an event with `POST /api/composition/{composition_id}/event`. The `event_name` is matched against your `eventBus.on(...)` subscriptions, and `data` is delivered as the handler's argument.

```bash
curl -X POST "$SMELTER_URL/api/composition/$COMPOSITION/event" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{ "event_name": "SET_CAPTION", "data": { "text": "Welcome to the stream" } }'
```

```bash
curl -X POST "$SMELTER_URL/api/composition/$COMPOSITION/event" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{ "event_name": "SET_LIVE", "data": { "live": true } }'
```

The template re-renders as soon as the event arrives.

:::note
Event names and payloads are an application-defined contract between your backend and your template. There is no canonical set of events; `SET_LIVE` and `SET_CAPTION` above are just examples. Pick names and payload shapes that fit your app, and keep them in sync on both sides.
:::
68 changes: 68 additions & 0 deletions docs/smelter/how-to/inputs-and-outputs.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
type: how-to
sidebar_position: 4
description: Choose the right input and output protocols for a Smelter Cloud composition, from WebRTC to RTMP, HLS, and MP4.
---

# Choose inputs and outputs

A composition pulls media in through **inputs** and pushes the composed result out through **outputs**. Each is a tagged object whose `type` selects the protocol. This guide summarizes the available types and when to use each.

## Inputs

Register an input with `POST /api/composition/{composition_id}/input/{input_id}/register` and a body whose `type` is one of:

| `type` | Use it when | Key fields |
| ------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `whip_server` | A WebRTC client should publish **into** the composition. | `bearer_token` (optional; generated and returned if omitted), `video` (optional, default `true`). |
| `whep_client` | The composition should **pull** a WebRTC stream from a WHEP endpoint. | `endpoint_url` (required), `bearer_token` (optional), `video` (optional). |
| `rtmp_server` | An encoder (OBS, hardware) should push RTMP in. | `app` (required), `stream_key` (required). |
| `hls` | The composition should pull an HLS playlist. | `url` (required). |
| `mp4` | You want to compose a file, optionally looped. | `url` (optional), `loop` (optional). |

For `whip_server`, the register response returns the `bearer_token` a publisher uses to authenticate against the input's WHIP endpoint:

```json
{ "bearer_token": "<INPUT_PUBLISH_TOKEN>" }
```

Unregister any input with `POST …/input/{input_id}/unregister`.

## Outputs

Register an output with `POST /api/composition/{composition_id}/output/{output_id}/register` and a body whose `type` is one of:

| `type` | Use it when | Key fields |
| ------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| `whip_client` | Publish the composed result over WebRTC (for example to a Fishjam livestream). | `endpoint_url` (required), `bearer_token` (optional), `video`, `audio`. |
| `rtmp_client` | Publish to an RTMP destination (for example a social platform). | `url` (required), `video`, `audio`. |

An output's `video` carries the resolution and initial [scene](./../scenes), and `audio` carries the initial audio scene:

```json
{
"type": "whip_client",
"endpoint_url": "https://example.com/whip",
"video": {
"resolution": { "width": 1280, "height": 720 },
"initial": { "root": { "type": "tiles", "children": [] } }
},
"audio": { "initial": { "inputs": [] } }
}
```

Other output operations:

- **Register a templated output** with `POST …/output/{output_id}/template` instead of `…/register`: a multipart request carrying the same configuration plus a template bundle (see [Write and deploy a template](./write-and-deploy-a-template)).
- **Update the scene** live with `POST …/output/{output_id}/update` (see [Scenes](./../scenes)).
- **Force a keyframe** with `POST …/output/{output_id}/request_keyframe`, useful when a new subscriber joins.
- **Unregister** with `POST …/output/{output_id}/unregister`.

## Renderers

Shared assets you can place in a scene:

- **Images**: register with `POST …/image/{image_id}/register` and an `ImageSpec` whose `asset_type` is `png`, `jpeg`, `svg`, `gif`, or `auto` (each takes a `url`; `svg` also accepts a `resolution`). Reference the image in a scene with `{ "type": "image", "image_id": "<id>" }`.
- **Fonts**: register with `POST …/font/register` as a multipart request with a single `font` part carrying the font file. The font is then available to `text` components.

See the [Smelter Cloud API](./../../api/smelter-cloud) reference for the complete request and response schemas.
Loading
Loading