Skip to content

Commit 7ac0c43

Browse files
waleedlatif1waleedlatif1
committed
feat(ffmpeg): add server-side FFmpeg media-processing block
Add an FFmpeg block that processes video/audio files server-side (no external service or auth) via an internal /api/tools/ffmpeg/process route. Files flow in and out as standard UserFile objects. Operations: convert, extract audio, trim, compress/scale, probe, thumbnail, concatenate, adjust volume, change speed.
1 parent e1e773f commit 7ac0c43

28 files changed

Lines changed: 2082 additions & 2 deletions

File tree

‎apps/docs/components/icons.tsx‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2432,6 +2432,16 @@ export function FathomIcon(props: SVGProps<SVGSVGElement>) {
24322432
</svg>
24332433
)
24342434
}
2435+
export function FFmpegIcon(props: SVGProps<SVGSVGElement>) {
2436+
return (
2437+
<svg {...props} xmlns='http://www.w3.org/2000/svg' viewBox='0 0 48 48'>
2438+
<path
2439+
fill='#388e3c'
2440+
d='M39.5,42h-9c-1,0-1.9-0.6-2.3-1.5c-0.4-0.9-0.2-2,0.5-2.7l8.3-8.3v-3.9L21.3,41.3 c-0.5,0.5-1.1,0.7-1.8,0.7h-11c-1,0-1.9-0.6-2.3-1.5c-0.4-0.9-0.2-2,0.5-2.7L33.5,11h-3.9L10.3,30.3c-0.7,0.7-1.8,0.9-2.7,0.5 C6.6,30.4,6,29.5,6,28.5v-11c0-0.7,0.3-1.3,0.7-1.8l4.7-4.7h-3C7.1,11,6,9.9,6,8.5S7.1,6,8.5,6h9c1,0,1.9,0.6,2.3,1.5 c0.4,0.9,0.2,2-0.5,2.7L11,18.5v3.9L26.7,6.7C27.2,6.3,27.8,6,28.5,6h11c1,0,1.9,0.6,2.3,1.5c0.4,0.9,0.2,2-0.5,2.7L14.5,37h3.9 l19.3-19.3c0.7-0.7,1.8-0.9,2.7-0.5c0.9,0.4,1.5,1.3,1.5,2.3v11c0,0.7-0.3,1.3-0.7,1.8L36.5,37h3c1.4,0,2.5,1.1,2.5,2.5 S40.9,42,39.5,42z'
2441+
/>
2442+
</svg>
2443+
)
2444+
}
24352445
export function LinkupIcon(props: SVGProps<SVGSVGElement>) {
24362446
return (
24372447
<svg {...props} xmlns='http://www.w3.org/2000/svg' viewBox='0 0 154 107' fill='none'>

‎apps/docs/components/ui/icon-mapping.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ import {
5959
ExtendIcon,
6060
EyeIcon,
6161
FathomIcon,
62+
FFmpegIcon,
6263
FindymailIcon,
6364
FirecrawlIcon,
6465
FirefliesIcon,
@@ -271,6 +272,7 @@ export const blockTypeToIconMap: Record<string, IconComponent> = {
271272
extend: ExtendIcon,
272273
extend_v2: ExtendIcon,
273274
fathom: FathomIcon,
275+
ffmpeg: FFmpegIcon,
274276
file: DocumentIcon,
275277
file_v2: DocumentIcon,
276278
file_v3: DocumentIcon,

‎apps/docs/content/docs/en/tools/ffmpeg.mdx‎

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
---
2+
title: FFmpeg
3+
description: Process video and audio files with FFmpeg
4+
---
5+
6+
import { BlockInfoCard } from "@/components/ui/block-info-card"
7+
8+
<BlockInfoCard
9+
type="ffmpeg"
10+
color="#FFFFFF"
11+
/>
12+
13+
## Usage Instructions
14+
15+
{/* MANUAL-CONTENT-START:usage */}
16+
The FFmpeg block runs the FFmpeg media engine on Sim's servers — there is no external service, API key, or OAuth to configure. It takes a media file as input (uploaded directly or referenced from a previous block as a `UserFile`), processes it, and returns the result as a new `UserFile` you can pass to any downstream block.
17+
18+
Pick an **Operation**, supply the input file, and fill in the fields shown for that operation:
19+
20+
| Operation | What it does | Key inputs | Output |
21+
| --- | --- | --- | --- |
22+
| **Convert Format** | Transcode to a different container/codec | `Output Format` (required, e.g. `mp4`, `webm`, `mp3`); optional `Video Codec` / `Audio Codec` | Converted media file |
23+
| **Extract Audio** | Pull the audio track out of a video | `Output Format` (defaults to `mp3`) | Audio file |
24+
| **Trim / Cut** | Keep a segment of the media | `Start Time` and/or `Duration` (seconds or `HH:MM:SS`) | Trimmed media file |
25+
| **Compress / Scale** | Reduce file size and/or rescale video | `Scale` (e.g. `1280x720`, `1280:-2`), `Quality (CRF)` 0–51, optional `Video Bitrate` | Compressed video file |
26+
| **Get Media Info** | Inspect the file with `ffprobe` | — | Metadata only (duration, codecs, resolution, streams) — no file |
27+
| **Extract Thumbnail** | Capture a single frame as an image | `Timestamp` (defaults to 1s), `Output Format` (`jpg`/`png`/`webp`) | Image file |
28+
| **Concatenate** | Join multiple clips into one | `Media Files` (2+, **same codec/format**) | Joined media file |
29+
| **Adjust Volume** | Change audio loudness | `Volume` — multiplier (`1.5`, `0.5`) or decibels (`10dB`, `-6dB`) | Media file |
30+
| **Change Speed** | Speed up or slow down playback | `Speed` multiplier (`2` = 2× faster, `0.5` = half) | Retimed media file |
31+
32+
**File handling.** Inputs and outputs are standard Sim `UserFile` objects, so the block chains naturally — e.g. *Convert → Trim → Extract Thumbnail*, or feed the output of an upload/HTTP block straight into FFmpeg. Every operation except **Get Media Info** returns a `file` output; **Get Media Info** returns structured metadata instead.
33+
34+
**Notes & limits.**
35+
- **Concatenate** uses FFmpeg's concat demuxer with stream copy, so all inputs must share the same codec, resolution, and format (typical for clips exported from one source). Mixed-codec inputs will fail.
36+
- **Change Speed** automatically retimes whichever streams exist — video via `setpts`, audio via `atempo` (chained for speeds beyond 0.5×–2×).
37+
- Input and output files are capped at 200 MB each.
38+
- Requires the FFmpeg binary on the server (bundled via `ffmpeg-static`; `ffprobe` from the system PATH is needed for **Get Media Info**).
39+
{/* MANUAL-CONTENT-END */}
40+
41+
42+
Transcode, trim, compress, concatenate, and inspect video and audio files server-side with FFmpeg. Convert formats, extract audio, capture thumbnails, adjust volume, and change playback speed — no external service required.
43+
44+
45+
46+
## Tools
47+
48+
### `ffmpeg_convert`
49+
50+
Convert (transcode) a video or audio file to a different container/format
51+
52+
#### Input
53+
54+
| Parameter | Type | Required | Description |
55+
| --------- | ---- | -------- | ----------- |
56+
| `file` | file | Yes | The media file to convert |
57+
| `format` | string | Yes | Target output format/container \(e.g. mp4, webm, mov, mkv, mp3, wav\) |
58+
| `videoCodec` | string | No | Optional video codec override \(e.g. libx264, vp9\) |
59+
| `audioCodec` | string | No | Optional audio codec override \(e.g. aac, libmp3lame\) |
60+
61+
#### Output
62+
63+
| Parameter | Type | Description |
64+
| --------- | ---- | ----------- |
65+
| `file` | file | Processed media file |
66+
| `fileName` | string | Generated output file name |
67+
| `format` | string | Output or detected container format |
68+
| `size` | number | Output file size in bytes |
69+
| `durationSeconds` | number | Media duration in seconds |
70+
| `bitrate` | number | Overall bitrate in bits per second |
71+
| `width` | number | Video width in pixels |
72+
| `height` | number | Video height in pixels |
73+
| `hasVideo` | boolean | Whether a video stream is present |
74+
| `hasAudio` | boolean | Whether an audio stream is present |
75+
| `videoCodec` | string | Primary video codec |
76+
| `audioCodec` | string | Primary audio codec |
77+
| `streams` | array | All detected media streams |
78+
79+
### `ffmpeg_extract_audio`
80+
81+
Extract the audio track from a video file into an audio file
82+
83+
#### Input
84+
85+
| Parameter | Type | Required | Description |
86+
| --------- | ---- | -------- | ----------- |
87+
| `file` | file | Yes | The video file to extract audio from |
88+
| `format` | string | No | Output audio format \(mp3, wav, aac, flac, ogg, m4a, opus\). Defaults to mp3 |
89+
| `audioCodec` | string | No | Optional audio codec override \(e.g. aac, libmp3lame\) |
90+
91+
#### Output
92+
93+
| Parameter | Type | Description |
94+
| --------- | ---- | ----------- |
95+
| `file` | file | Processed media file |
96+
| `fileName` | string | Generated output file name |
97+
| `format` | string | Output or detected container format |
98+
| `size` | number | Output file size in bytes |
99+
| `durationSeconds` | number | Media duration in seconds |
100+
| `bitrate` | number | Overall bitrate in bits per second |
101+
| `width` | number | Video width in pixels |
102+
| `height` | number | Video height in pixels |
103+
| `hasVideo` | boolean | Whether a video stream is present |
104+
| `hasAudio` | boolean | Whether an audio stream is present |
105+
| `videoCodec` | string | Primary video codec |
106+
| `audioCodec` | string | Primary audio codec |
107+
| `streams` | array | All detected media streams |
108+
109+
### `ffmpeg_trim`
110+
111+
Cut a segment from a video or audio file using a start time and/or duration
112+
113+
#### Input
114+
115+
| Parameter | Type | Required | Description |
116+
| --------- | ---- | -------- | ----------- |
117+
| `file` | file | Yes | The media file to trim |
118+
| `startTime` | string | No | Start offset in seconds or HH:MM:SS\(.ms\), e.g. 5 or 00:00:05 |
119+
| `duration` | string | No | Duration to keep in seconds or HH:MM:SS\(.ms\), e.g. 30 or 00:00:30 |
120+
121+
#### Output
122+
123+
| Parameter | Type | Description |
124+
| --------- | ---- | ----------- |
125+
| `file` | file | Processed media file |
126+
| `fileName` | string | Generated output file name |
127+
| `format` | string | Output or detected container format |
128+
| `size` | number | Output file size in bytes |
129+
| `durationSeconds` | number | Media duration in seconds |
130+
| `bitrate` | number | Overall bitrate in bits per second |
131+
| `width` | number | Video width in pixels |
132+
| `height` | number | Video height in pixels |
133+
| `hasVideo` | boolean | Whether a video stream is present |
134+
| `hasAudio` | boolean | Whether an audio stream is present |
135+
| `videoCodec` | string | Primary video codec |
136+
| `audioCodec` | string | Primary audio codec |
137+
| `streams` | array | All detected media streams |
138+
139+
### `ffmpeg_compress`
140+
141+
Compress and/or rescale a video to reduce file size
142+
143+
#### Input
144+
145+
| Parameter | Type | Required | Description |
146+
| --------- | ---- | -------- | ----------- |
147+
| `file` | file | Yes | The video file to compress |
148+
| `scale` | string | No | Optional output dimensions, e.g. 1280x720, 1280:720, or 1280:-2 \(keep aspect ratio\) |
149+
| `crf` | number | No | Constant Rate Factor \(0 = lossless, 23 = default, 51 = worst quality\) |
150+
| `videoBitrate` | string | No | Optional target video bitrate, e.g. 1M or 800k |
151+
| `videoCodec` | string | No | Optional video codec override \(defaults to libx264\) |
152+
153+
#### Output
154+
155+
| Parameter | Type | Description |
156+
| --------- | ---- | ----------- |
157+
| `file` | file | Processed media file |
158+
| `fileName` | string | Generated output file name |
159+
| `format` | string | Output or detected container format |
160+
| `size` | number | Output file size in bytes |
161+
| `durationSeconds` | number | Media duration in seconds |
162+
| `bitrate` | number | Overall bitrate in bits per second |
163+
| `width` | number | Video width in pixels |
164+
| `height` | number | Video height in pixels |
165+
| `hasVideo` | boolean | Whether a video stream is present |
166+
| `hasAudio` | boolean | Whether an audio stream is present |
167+
| `videoCodec` | string | Primary video codec |
168+
| `audioCodec` | string | Primary audio codec |
169+
| `streams` | array | All detected media streams |
170+
171+
### `ffmpeg_probe`
172+
173+
Inspect a media file and return metadata (duration, format, codecs, resolution)
174+
175+
#### Input
176+
177+
| Parameter | Type | Required | Description |
178+
| --------- | ---- | -------- | ----------- |
179+
| `file` | file | Yes | The media file to inspect |
180+
181+
#### Output
182+
183+
| Parameter | Type | Description |
184+
| --------- | ---- | ----------- |
185+
| `file` | file | Processed media file |
186+
| `fileName` | string | Generated output file name |
187+
| `format` | string | Output or detected container format |
188+
| `size` | number | Output file size in bytes |
189+
| `durationSeconds` | number | Media duration in seconds |
190+
| `bitrate` | number | Overall bitrate in bits per second |
191+
| `width` | number | Video width in pixels |
192+
| `height` | number | Video height in pixels |
193+
| `hasVideo` | boolean | Whether a video stream is present |
194+
| `hasAudio` | boolean | Whether an audio stream is present |
195+
| `videoCodec` | string | Primary video codec |
196+
| `audioCodec` | string | Primary audio codec |
197+
| `streams` | array | All detected media streams |
198+
199+
### `ffmpeg_thumbnail`
200+
201+
Extract a single frame from a video at a given timestamp as an image
202+
203+
#### Input
204+
205+
| Parameter | Type | Required | Description |
206+
| --------- | ---- | -------- | ----------- |
207+
| `file` | file | Yes | The video file to extract a frame from |
208+
| `time` | string | No | Timestamp in seconds or HH:MM:SS\(.ms\), e.g. 5 or 00:00:05. Defaults to 1s |
209+
| `format` | string | No | Output image format \(jpg, png, webp\). Defaults to jpg |
210+
211+
#### Output
212+
213+
| Parameter | Type | Description |
214+
| --------- | ---- | ----------- |
215+
| `file` | file | Processed media file |
216+
| `fileName` | string | Generated output file name |
217+
| `format` | string | Output or detected container format |
218+
| `size` | number | Output file size in bytes |
219+
| `durationSeconds` | number | Media duration in seconds |
220+
| `bitrate` | number | Overall bitrate in bits per second |
221+
| `width` | number | Video width in pixels |
222+
| `height` | number | Video height in pixels |
223+
| `hasVideo` | boolean | Whether a video stream is present |
224+
| `hasAudio` | boolean | Whether an audio stream is present |
225+
| `videoCodec` | string | Primary video codec |
226+
| `audioCodec` | string | Primary audio codec |
227+
| `streams` | array | All detected media streams |
228+
229+
### `ffmpeg_concat`
230+
231+
Join multiple media files of the same format and codec into a single output file
232+
233+
#### Input
234+
235+
| Parameter | Type | Required | Description |
236+
| --------- | ---- | -------- | ----------- |
237+
| `files` | file[] | Yes | Two or more media files to join, in order. Files must share the same codec |
238+
239+
#### Output
240+
241+
| Parameter | Type | Description |
242+
| --------- | ---- | ----------- |
243+
| `file` | file | Processed media file |
244+
| `fileName` | string | Generated output file name |
245+
| `format` | string | Output or detected container format |
246+
| `size` | number | Output file size in bytes |
247+
| `durationSeconds` | number | Media duration in seconds |
248+
| `bitrate` | number | Overall bitrate in bits per second |
249+
| `width` | number | Video width in pixels |
250+
| `height` | number | Video height in pixels |
251+
| `hasVideo` | boolean | Whether a video stream is present |
252+
| `hasAudio` | boolean | Whether an audio stream is present |
253+
| `videoCodec` | string | Primary video codec |
254+
| `audioCodec` | string | Primary audio codec |
255+
| `streams` | array | All detected media streams |
256+
257+
### `ffmpeg_volume`
258+
259+
Adjust the audio volume of a video or audio file
260+
261+
#### Input
262+
263+
| Parameter | Type | Required | Description |
264+
| --------- | ---- | -------- | ----------- |
265+
| `file` | file | Yes | The media file to adjust |
266+
| `volume` | string | Yes | Volume as a multiplier \(e.g. 1.5, 0.5\) or decibels \(e.g. 10dB, -6dB\) |
267+
268+
#### Output
269+
270+
| Parameter | Type | Description |
271+
| --------- | ---- | ----------- |
272+
| `file` | file | Processed media file |
273+
| `fileName` | string | Generated output file name |
274+
| `format` | string | Output or detected container format |
275+
| `size` | number | Output file size in bytes |
276+
| `durationSeconds` | number | Media duration in seconds |
277+
| `bitrate` | number | Overall bitrate in bits per second |
278+
| `width` | number | Video width in pixels |
279+
| `height` | number | Video height in pixels |
280+
| `hasVideo` | boolean | Whether a video stream is present |
281+
| `hasAudio` | boolean | Whether an audio stream is present |
282+
| `videoCodec` | string | Primary video codec |
283+
| `audioCodec` | string | Primary audio codec |
284+
| `streams` | array | All detected media streams |
285+
286+
### `ffmpeg_speed`
287+
288+
Speed up or slow down playback of a video or audio file
289+
290+
#### Input
291+
292+
| Parameter | Type | Required | Description |
293+
| --------- | ---- | -------- | ----------- |
294+
| `file` | file | Yes | The media file to retime |
295+
| `speed` | number | Yes | Playback speed multiplier \(0.5 = half speed, 2 = double speed\) |
296+
297+
#### Output
298+
299+
| Parameter | Type | Description |
300+
| --------- | ---- | ----------- |
301+
| `file` | file | Processed media file |
302+
| `fileName` | string | Generated output file name |
303+
| `format` | string | Output or detected container format |
304+
| `size` | number | Output file size in bytes |
305+
| `durationSeconds` | number | Media duration in seconds |
306+
| `bitrate` | number | Overall bitrate in bits per second |
307+
| `width` | number | Video width in pixels |
308+
| `height` | number | Video height in pixels |
309+
| `hasVideo` | boolean | Whether a video stream is present |
310+
| `hasAudio` | boolean | Whether an audio stream is present |
311+
| `videoCodec` | string | Primary video codec |
312+
| `audioCodec` | string | Primary audio codec |
313+
| `streams` | array | All detected media streams |
314+
315+

‎apps/docs/content/docs/en/tools/meta.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454
"exa",
5555
"extend",
5656
"fathom",
57+
"ffmpeg",
5758
"file",
5859
"findymail",
5960
"firecrawl",

‎apps/sim/app/(landing)/integrations/data/icon-mapping.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ import {
5858
ExaAIIcon,
5959
ExtendIcon,
6060
FathomIcon,
61+
FFmpegIcon,
6162
FindymailIcon,
6263
FirecrawlIcon,
6364
FirefliesIcon,
@@ -267,6 +268,7 @@ export const blockTypeToIconMap: Record<string, IconComponent> = {
267268
exa: ExaAIIcon,
268269
extend_v2: ExtendIcon,
269270
fathom: FathomIcon,
271+
ffmpeg: FFmpegIcon,
270272
file_v4: DocumentIcon,
271273
findymail: FindymailIcon,
272274
firecrawl: FirecrawlIcon,

0 commit comments

Comments
 (0)