diff --git a/CHANGELOG.md b/CHANGELOG.md index fba764e..ee021c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,8 +11,8 @@ - The skills no longer download and run an installer. If `jam` is missing, they ask you to install it from jam.dev/cli. Addresses the remote-code-execution audit finding on `jam-proof` and `jam-annotate`. - `jam-annotate` and the `jam-fix` proof step write the Jam payload to a file and run `jam create jam @payload.json`, so user and page text never reaches a shell argument. - `jam-fix` and `jam-to-e2e` treat Jam content (transcript, logs, network bodies, page text, comments) as evidence, not instructions. -- Added `record-demo`: a narrated, human-paced demo video of a web feature. The agent writes a script of beats and a bundled player runs it with agent-browser on macOS, Linux or Windows. The voice leads: each click waits for the line about it. The pointer moves like a hand: varied paths, off-centre presses, out of the way when idle. The agent writes a screenplay with a camera shot per beat; a polish pass moves the camera only where a shot changes, draws the pointer with motion blur and a ring on each click, and frames the page as a rounded card on a wallpaper, at 1440p and 60 fps. `--background none` drops the card for a full-bleed page, and `--frame safari` or `--frame chrome` puts the page in that browser's window, its address bar following the page. Waits nobody needs to watch, like a loading screen or an emailed sign-in code, are cut from the video. Narration comes from `add-voiceover`. No Jam account needed. -- Added `add-voiceover`: voice a spoken script line by line and lay each line under its moment in any video, with a WebVTT subtitle file. The agent asks which voice to use: ElevenLabs (`eleven_v4`), Kokoro (free, local, a one-time 340 MB download), the OS voice, or the team's own command. No Jam account needed. +- Added `record-demo`: a narrated, human-paced demo video of a web feature. The agent writes a script of beats and a bundled player runs it with agent-browser on macOS, Linux or Windows. The voice leads: each click waits for the line about it. The pointer moves like a hand: varied paths, off-centre presses, out of the way when idle. The agent writes a screenplay with a camera shot per beat; a polish pass moves the camera only where a shot changes, draws the pointer with motion blur and a ring on each click, and frames the page as a rounded card on a wallpaper, at 1440p and 60 fps. `--background none` drops the card for a full-bleed page, and `--frame safari` or `--frame chrome` puts the page in that browser's window, its address bar following the page. Waits nobody needs to watch, like a loading screen or an emailed sign-in code, are cut from the video. Narration comes from `add-voiceover`. The scripts run on Node 22 or later with nothing to install; the polish pass lays each frame out in a headless Chrome. No Jam account needed. +- Added `add-voiceover`: voice a spoken script line by line and lay each line under its moment in any video, with a WebVTT subtitle file. The agent asks which voice to use: ElevenLabs (`eleven_v4`), Kokoro (free, local, a one-time 340 MB download), the OS voice, or the team's own command. The script runs on Node 22 or later with nothing to install. No Jam account needed. - Added `record-terminal-proof`: record a terminal run with asciinema and render a GIF for the pull request, plus an MP4 when needed. No Jam account needed. - `scripts/check.mjs` counts Jam mentions in an unbranded skill in prose only, so a "With Jam" line can show the command. - `server.json` follows Jam MCP 1.27.1, the version production serves. diff --git a/README.md b/README.md index d70ed30..0315f51 100644 --- a/README.md +++ b/README.md @@ -64,8 +64,8 @@ These need no Jam account. With the Jam CLI, a video can become a Jam link to co - A free [Jam](https://jam.dev?utm_source=github&utm_medium=readme&utm_campaign=skills) account, for every skill except the three video skills. - The [Jam CLI](https://jam.dev/cli). A skill signs in when it needs to. `jam`, `jam-fix` and `jam-to-e2e` can read a Jam through Jam MCP instead, the Model Context Protocol (MCP) server the plugin connects. - For `video-to-jam`: ffmpeg and a local transcriber such as Whisper. -- For `record-demo`: agent-browser, ffmpeg, Python 3 and uv. -- For `add-voiceover`: ffmpeg and Python 3. An ElevenLabs key or the free Kokoro model gives a natural voice. +- For `record-demo`: agent-browser, ffmpeg and Node 22 or later. +- For `add-voiceover`: ffmpeg and Node 22 or later. An ElevenLabs key or the free Kokoro model (Python 3 and uv) gives a natural voice. - For `record-terminal-proof`: asciinema and agg. - In CI or a cloud agent: a [personal access token](https://jam.dev/docs/personal-access-tokens) in `JAM_TOKEN`. - For `jam-proof`: recording a web page needs only Chrome. Recording a window or the whole screen needs macOS or Linux (X11). diff --git a/skills/add-voiceover/SKILL.md b/skills/add-voiceover/SKILL.md index 7236714..ad0fd3a 100644 --- a/skills/add-voiceover/SKILL.md +++ b/skills/add-voiceover/SKILL.md @@ -7,7 +7,7 @@ description: "Add a spoken voiceover to a video: write the script for the ear, v A good voiceover sounds like a colleague showing you something: short spoken lines, each one landing on the moment it talks about. Most bad ones come from a script written for reading, one long take of text-to-speech laid over the whole video, or an expressive voice with unstable settings. -Needs `ffmpeg` and Python 3 (`python` on Windows). The script is `scripts/voiceover.py` in this skill's folder. +Needs `ffmpeg` and Node 22 or later. The script is `scripts/voiceover.mjs` in this skill's folder and needs no packages. ## 1. Pick the voice @@ -15,7 +15,7 @@ The script takes the first voice that is set up: 1. `VOICEOVER_TTS`: the team's own command. `{out}` is the `.wav` file to write and `{text}` the line, also sent on stdin. 2. `ELEVENLABS_API_KEY`: ElevenLabs. Defaults: model `eleven_v4`, voice `iP95p4xoKVk53GoZ742B`, stability 0.5, similarity 0.75, speaker boost on, and each line voiced with the two lines around it as context so the intonation carries across. `ELEVENLABS_VOICE_ID` and `ELEVENLABS_MODEL_ID` change them; for a female voice use River, `SAz9YHcvj6GT2YYXdXww`. v4 has no speed or style setting: pace it with punctuation and line length. -3. Kokoro, a free local model, once installed (`~/.cache/kokoro` holds its files) and `uv` is on PATH. Voice `am_michael`; `KOKORO_VOICE` picks another (`af_heart`, `bf_emma`, `bm_george`). +3. Kokoro, a free local model, once installed (`~/.cache/kokoro` holds its files) and Python 3 and `uv` are on PATH. Voice `am_michael`; `KOKORO_VOICE` picks another (`af_heart`, `bf_emma`, `bm_george`). 4. macOS `say`, for drafts only. When none of the first three is set up, ask the user once which they want: @@ -63,7 +63,7 @@ To pace a recording by the voice instead, voice the lines first with no `--video ## 4. Voice and mix ```bash -python3 /scripts/voiceover.py lines.json ./voiceover --video in.mp4 +node /scripts/voiceover.mjs lines.json ./voiceover --video in.mp4 ``` Output in `./voiceover`: `voiced.mp4` (the video untouched, the narration raised by one fixed gain so it peaks at -1 dB), `narration.wav`, `lines.json` with each line's final `at` and `seconds`, one `.wav` per line, and `voiceover.vtt`. Each line is cached by its text and voice, so after editing one line a re-run voices only that line, and switching the voice re-voices everything. diff --git a/skills/add-voiceover/scripts/voiceover.mjs b/skills/add-voiceover/scripts/voiceover.mjs new file mode 100644 index 0000000..789a8d3 --- /dev/null +++ b/skills/add-voiceover/scripts/voiceover.mjs @@ -0,0 +1,149 @@ +// Voice a script line by line and, with --video, lay each line under the video at its moment. +// +// usage: node voiceover.mjs [--video ] +// +// is a list of {"say": "...", "at": seconds}, plain strings, or record-demo beats with ["say", "..."] actions. +// Lines without "at" follow each other with a short breath between them. A line that would talk over the one +// before is pushed later, with a warning. Each line is cached in by its text and voice, so a re-run +// only voices what changed. +// +// Writes /lines.json (each line with "at", "seconds" and "wav"), /voiceover.vtt and, with --video, +// /narration.wav and /voiced.mp4: the video untouched, with the narration raised to peak at -1 dB. +// +// The voice is the first that is set up: +// VOICEOVER_TTS a command; per line {out} becomes a .wav path and {text} the line, also sent on stdin +// ELEVENLABS_API_KEY ElevenLabs; ELEVENLABS_VOICE_ID and ELEVENLABS_MODEL_ID override the defaults +// Kokoro a local model through kokoro.py next to this file, once `kokoro.py --install` has run and uv is on PATH +// say the macOS voice, for drafts +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { delimiter, join } from "node:path"; + +const die = (msg) => { console.error(msg); process.exit(1); }; + +function which(name) { + // Full path, so Windows finds .cmd shims too. + const exts = process.platform === "win32" ? (process.env.PATHEXT || ".EXE;.CMD;.BAT").split(";") : [""]; + for (const dir of (process.env.PATH || "").split(delimiter)) + for (const ext of exts) if (dir && existsSync(join(dir, name + ext))) return join(dir, name + ext); +} +const tool = (name) => which(name) || die(`${name} not found on PATH`); + +function run(file, args, opts = {}) { + // A Windows .cmd shim only runs through the shell. ponytail: plain double quotes, enough for paths and lines. + const shell = /\.(cmd|bat)$/i.test(file); + const r = spawnSync(shell ? `"${file}"` : file, shell ? args.map((a) => `"${a.replaceAll('"', '""')}"`) : args, + { encoding: "utf8", shell, ...opts }); + if (r.status !== 0 && !opts.ok) die(`${file} failed: ${r.stderr || r.error || ""}`); + return r; +} + +const FFMPEG = tool("ffmpeg"), FFPROBE = tool("ffprobe"); +const GAP = 0.6; // breath between lines +const args = process.argv.slice(2); +const vi = args.indexOf("--video"); +const video = vi >= 0 ? args.splice(vi, 2)[1] : null; +const [src, out] = args; +if (!out) die("usage: node voiceover.mjs [--video ]"); +mkdirSync(out, { recursive: true }); + +let data = JSON.parse(readFileSync(src, "utf8")); +if (data.length && data[0]?.do) data = data.flatMap((b) => b.do.filter((c) => c[0] === "say").map((c) => c[1])); +const lines = data.map((x) => (typeof x === "string" ? { say: x } : { ...x })); +if (!lines.length) die(`no lines to voice in ${src}`); + +const VOICE = process.env.ELEVENLABS_VOICE_ID || "iP95p4xoKVk53GoZ742B"; +const MODEL = process.env.ELEVENLABS_MODEL_ID || "eleven_v4"; + +async function elevenlabs(text, wav, before, after) { + const r = await fetch(`https://api.elevenlabs.io/v1/text-to-speech/${VOICE}?output_format=mp3_44100_128`, { + method: "POST", + headers: { "xi-api-key": process.env.ELEVENLABS_API_KEY, "Content-Type": "application/json" }, + body: JSON.stringify({ + text, previous_text: before, next_text: after, // neighbours keep one intonation across lines + model_id: MODEL, + // v4 takes only these two; lower stability makes a voice laugh and drift between lines. + voice_settings: { stability: 0.5, similarity_boost: 0.75, use_speaker_boost: true }, + }), + signal: AbortSignal.timeout(120_000), + }); + if (!r.ok) die(`ElevenLabs ${r.status}: ${await r.text()}`); + writeFileSync(`${wav}.mp3`, Buffer.from(await r.arrayBuffer())); + run(FFMPEG, ["-v", "error", "-y", "-i", `${wav}.mp3`, "-ac", "1", wav]); + rmSync(`${wav}.mp3`); +} + +const command = (argv) => (text, wav) => run(argv[0], argv.slice(1).map((a) => (a === "{out}" ? wav : a === "{text}" ? text : a)), { input: text }); + +const kokoro = join(process.env.KOKORO_DIR || join(homedir(), ".cache", "kokoro"), "voices-v1.0.bin"); +let engine, speak; +if (process.env.VOICEOVER_TTS) { + engine = ["custom", process.env.VOICEOVER_TTS]; + speak = command(process.env.VOICEOVER_TTS.match(/"[^"]*"|'[^']*'|\S+/g).map((a) => a.replace(/^(["'])(.*)\1$/, "$2"))); +} else if (process.env.ELEVENLABS_API_KEY) { + engine = ["elevenlabs", VOICE, MODEL]; + speak = elevenlabs; +} else if (which("uv") && existsSync(kokoro)) { + engine = ["kokoro", process.env.KOKORO_VOICE ?? null]; + speak = command([tool("uv"), "run", "-q", "--python", "3.12", "--with", "kokoro-onnx", "--with", "soundfile", + "python", join(import.meta.dirname, "kokoro.py"), "{out}"]); +} else if (which("say")) { + engine = ["say"]; + speak = command([tool("say"), "-o", "{out}", "--data-format=LEI16@22050", "{text}"]); +} else { + die("no voice: set ELEVENLABS_API_KEY or VOICEOVER_TTS, or run kokoro.py --install"); +} +console.error(`voice: ${engine[0]}`); + +const seconds = (path) => parseFloat(run(FFPROBE, ["-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", path]).stdout); +const round = (x) => Math.round(x * 1000) / 1000; + +// The cache key is Python's json.dumps of [engine, line], so lines voiced by the earlier Python script are reused. +const pyjson = (v) => (Array.isArray(v) ? `[${v.map(pyjson).join(", ")}]` + : JSON.stringify(v).replace(/[\u0080-￿]/g, (c) => `\\u${c.charCodeAt(0).toString(16).padStart(4, "0")}`)); + +const said = lines.map((l) => l.say); +for (const [i, l] of lines.entries()) { + const wav = join(out, `${createHash("sha1").update(pyjson([engine, l.say])).digest("hex").slice(0, 10)}.wav`); + if (!existsSync(wav)) { + console.error(`voicing ${i + 1}/${lines.length}: ${l.say}`); + await speak(l.say, wav, said.slice(Math.max(0, i - 2), i).join(" "), said.slice(i + 1, i + 3).join(" ")); + } + l.wav = wav.split(/[\\/]/).pop(); + l.seconds = round(seconds(wav)); +} + +let free = 0.5; // when the voice may start again +for (const [i, l] of lines.entries()) { + if (l.at == null) l.at = free; + else if (i && l.at < free - GAP) { + console.error(`line ${i + 1} starts ${(free - l.at).toFixed(1)} s late so it doesn't talk over line ${i}; shorten one of them`); + l.at = free; + } + l.at = round(l.at); + free = l.at + l.seconds + GAP; +} +writeFileSync(join(out, "lines.json"), JSON.stringify(lines, null, 1)); + +const ts = (s) => `${String(Math.floor(s / 3600)).padStart(2, "0")}:${String(Math.floor((s % 3600) / 60)).padStart(2, "0")}:${(s % 60).toFixed(3).padStart(6, "0")}`; +writeFileSync(join(out, "voiceover.vtt"), "WEBVTT\n\n" + lines.map((l) => `${ts(l.at)} --> ${ts(l.at + l.seconds)}\n${l.say}\n\n`).join("")); + +if (video) { + const end = lines.at(-1).at + lines.at(-1).seconds, length = seconds(video); + if (end > length) console.error(`the narration runs ${(end - length).toFixed(1)} s past the video's end and is cut there`); + const narration = join(out, "narration.wav"); + const delays = lines.map((l, i) => `[${i}:a]aresample=48000,adelay=${Math.floor(l.at * 1000)}:all=1[a${i}]`); + const mix = lines.map((_, i) => `[a${i}]`).join("") + `amix=inputs=${lines.length}:normalize=0`; + run(FFMPEG, ["-v", "error", "-y", ...lines.flatMap((l) => ["-i", join(out, l.wav)]), "-filter_complex", [...delays, mix].join(";"), narration]); + // One gain for the whole track, peaks at -1 dB. loudnorm and other dynamic levelers pump up each word's tail into an echo. + const peak = run(FFMPEG, ["-i", narration, "-af", "volumedetect", "-f", "null", "-"], { ok: true }).stderr; + const gain = -1 - parseFloat(peak.split("max_volume:")[1]); + run(FFMPEG, ["-v", "error", "-y", "-i", video, "-i", narration, "-filter_complex", `[1:a]volume=${gain.toFixed(2)}dB,apad[aout]`, + "-map", "0:v", "-map", "[aout]", "-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest", + "-movflags", "faststart", join(out, "voiced.mp4")]); + console.log(join(out, "voiced.mp4")); +} else { + console.log(join(out, "lines.json")); +} diff --git a/skills/add-voiceover/scripts/voiceover.py b/skills/add-voiceover/scripts/voiceover.py deleted file mode 100644 index 0e2792d..0000000 --- a/skills/add-voiceover/scripts/voiceover.py +++ /dev/null @@ -1,124 +0,0 @@ -"""Voice a script line by line and, with --video, lay each line under the video at its moment. - -usage: python3 voiceover.py [--video ] (python on Windows) - - is a list of {"say": "...", "at": seconds}, plain strings, or record-demo beats with ["say", "..."] actions. -Lines without "at" follow each other with a short breath between them. A line that would talk over the one -before is pushed later, with a warning. Each line is cached in by its text and voice, so a re-run -only voices what changed. - -Writes /lines.json (each line with "at", "seconds" and "wav"), /voiceover.vtt and, with --video, -/narration.wav and /voiced.mp4: the video untouched, with the narration raised to peak at -1 dB. - -The voice is the first that is set up: - VOICEOVER_TTS a command; per line {out} becomes a .wav path and {text} the line, also sent on stdin - ELEVENLABS_API_KEY ElevenLabs; ELEVENLABS_VOICE_ID and ELEVENLABS_MODEL_ID override the defaults - Kokoro a local model through kokoro.py next to this file, once `kokoro.py --install` has run and uv is on PATH - say the macOS voice, for drafts -""" -import hashlib, json, os, shlex, shutil, subprocess, sys, urllib.error, urllib.request -from pathlib import Path - -sys.stderr.reconfigure(errors="replace") # lines may not fit a Windows console code page - -def tool(name): - # Full path, so Windows finds .cmd shims too. - return shutil.which(name) or sys.exit(f"{name} not found on PATH") - -FFMPEG, FFPROBE = tool("ffmpeg"), tool("ffprobe") -GAP = 0.6 # breath between lines -args = sys.argv[1:] -video = Path(args.pop(args.index("--video") + 1)) if "--video" in args else None -src, out = Path(args[0]), Path(args[1]) -out.mkdir(parents=True, exist_ok=True) - -data = json.loads(src.read_text(encoding="utf-8")) -if data and isinstance(data[0], dict) and "do" in data[0]: - data = [c[1] for b in data for c in b["do"] if c[0] == "say"] -lines = [{"say": x} if isinstance(x, str) else dict(x) for x in data] or sys.exit(f"no lines to voice in {src}") - -VOICE = os.environ.get("ELEVENLABS_VOICE_ID", "iP95p4xoKVk53GoZ742B") -MODEL = os.environ.get("ELEVENLABS_MODEL_ID", "eleven_v4") - -def elevenlabs(text, wav, before, after): - body = {"text": text, "previous_text": before, "next_text": after, # neighbours keep one intonation across lines - "model_id": MODEL, - # v4 takes only these two; lower stability makes a voice laugh and drift between lines. - "voice_settings": {"stability": 0.5, "similarity_boost": 0.75, "use_speaker_boost": True}} - req = urllib.request.Request(f"https://api.elevenlabs.io/v1/text-to-speech/{VOICE}?output_format=mp3_44100_128", - data=json.dumps(body).encode(), - headers={"xi-api-key": os.environ["ELEVENLABS_API_KEY"], "Content-Type": "application/json"}) - try: - mp3 = urllib.request.urlopen(req, timeout=120).read() - except urllib.error.HTTPError as e: - sys.exit(f"ElevenLabs {e.code}: {e.read().decode(errors='replace')}") - Path(f"{wav}.mp3").write_bytes(mp3) - subprocess.run([FFMPEG, "-v", "error", "-y", "-i", f"{wav}.mp3", "-ac", "1", wav], check=True) - os.remove(f"{wav}.mp3") - -def command(argv): - return lambda text, wav, *_: subprocess.run([wav if a == "{out}" else text if a == "{text}" else a for a in argv], - input=text, text=True, encoding="utf-8", check=True) - -kokoro = Path(os.environ.get("KOKORO_DIR", Path.home() / ".cache" / "kokoro")) / "voices-v1.0.bin" -if os.environ.get("VOICEOVER_TTS"): - engine = ["custom", os.environ["VOICEOVER_TTS"]] - speak = command([a.strip('"') for a in shlex.split(os.environ["VOICEOVER_TTS"], posix=os.name != "nt")]) -elif os.environ.get("ELEVENLABS_API_KEY"): - engine = ["elevenlabs", VOICE, MODEL] - speak = elevenlabs -elif shutil.which("uv") and kokoro.exists(): - engine = ["kokoro", os.environ.get("KOKORO_VOICE")] - speak = command([tool("uv"), "run", "-q", "--python", "3.12", "--with", "kokoro-onnx", "--with", "soundfile", - "python", str(Path(__file__).with_name("kokoro.py")), "{out}"]) -elif shutil.which("say"): - engine = ["say"] - speak = command([tool("say"), "-o", "{out}", "--data-format=LEI16@22050", "{text}"]) -else: - sys.exit("no voice: set ELEVENLABS_API_KEY or VOICEOVER_TTS, or run kokoro.py --install") -print(f"voice: {engine[0]}", file=sys.stderr) - -def seconds(path): - return float(subprocess.check_output([FFPROBE, "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", str(path)])) - -said = [l["say"] for l in lines] -for i, l in enumerate(lines): - wav = out / f"{hashlib.sha1(json.dumps([engine, l['say']]).encode()).hexdigest()[:10]}.wav" - if not wav.exists(): - print(f"voicing {i + 1}/{len(lines)}: {l['say']}", file=sys.stderr) - speak(l["say"], str(wav), " ".join(said[max(0, i - 2):i]), " ".join(said[i + 1:i + 3])) - l["wav"], l["seconds"] = wav.name, round(seconds(wav), 3) - -free = 0.5 # when the voice may start again -for i, l in enumerate(lines): - if l.get("at") is None: - l["at"] = free - elif i and l["at"] < free - GAP: - print(f"line {i + 1} starts {free - l['at']:.1f} s late so it doesn't talk over line {i}; shorten one of them", file=sys.stderr) - l["at"] = free - l["at"] = round(l["at"], 3) - free = l["at"] + l["seconds"] + GAP -(out / "lines.json").write_text(json.dumps(lines, indent=1), encoding="utf-8") - -def ts(s): - return f"{int(s // 3600):02d}:{int(s % 3600 // 60):02d}:{s % 60:06.3f}" -(out / "voiceover.vtt").write_text("WEBVTT\n\n" + "".join(f"{ts(l['at'])} --> {ts(l['at'] + l['seconds'])}\n{l['say']}\n\n" for l in lines), - encoding="utf-8") - -if video: - if (end := lines[-1]["at"] + lines[-1]["seconds"]) > (length := seconds(video)): - print(f"the narration runs {end - length:.1f} s past the video's end and is cut there", file=sys.stderr) - inputs = [x for l in lines for x in ("-i", str(out / l["wav"]))] - delays = [f"[{i}:a]aresample=48000,adelay={int(l['at'] * 1000)}:all=1[a{i}]" for i, l in enumerate(lines)] - mix = "".join(f"[a{i}]" for i in range(len(lines))) + f"amix=inputs={len(lines)}:normalize=0" - subprocess.run([FFMPEG, "-v", "error", "-y", *inputs, "-filter_complex", ";".join([*delays, mix]), - str(out / "narration.wav")], check=True) - # One gain for the whole track, peaks at -1 dB. loudnorm and other dynamic levelers pump up each word's tail into an echo. - peak = subprocess.run([FFMPEG, "-i", str(out / "narration.wav"), "-af", "volumedetect", "-f", "null", "-"], capture_output=True, text=True).stderr - gain = -1.0 - float(peak.split("max_volume:")[1].split()[0]) - subprocess.run([FFMPEG, "-v", "error", "-y", "-i", str(video), "-i", str(out / "narration.wav"), "-filter_complex", f"[1:a]volume={gain:.2f}dB,apad[aout]", - "-map", "0:v", "-map", "[aout]", "-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest", - "-movflags", "faststart", str(out / "voiced.mp4")], check=True) - print(out / "voiced.mp4") -else: - print(out / "lines.json") diff --git a/skills/record-demo/SKILL.md b/skills/record-demo/SKILL.md index cc5862e..cb9e37e 100644 --- a/skills/record-demo/SKILL.md +++ b/skills/record-demo/SKILL.md @@ -7,7 +7,7 @@ description: "Record a narrated, human-paced demo video of a web feature with ag A demo shows a person what changed in under a minute. Dead air comes from you thinking between commands, so never record while exploring. Explore first, write a screenplay of beats, then let the bundled player run it with no model in the loop. -Needs `agent-browser` (0.38.1 or later), `ffmpeg`, Python 3 and `uv` (or `pip install pillow websockets`) on macOS, Linux or Windows (`python` instead of `python3` there). The player is `scripts/record.py` in this skill's folder. The narration comes from the `add-voiceover` skill, in the folder next to this one (`/../add-voiceover`); without it, record a silent video. Install it with `npx skills add jamdotdev/skills --skill add-voiceover`. +Needs `agent-browser` (0.38.1 or later), `ffmpeg` and Node 22 or later, on macOS, Linux or Windows. The scripts need no packages. The player is `scripts/record.mjs` in this skill's folder. The narration comes from the `add-voiceover` skill, in the folder next to this one (`/../add-voiceover`); without it, record a silent video. Install it with `npx skills add jamdotdev/skills --skill add-voiceover`. ## 1. Rehearse without recording @@ -40,7 +40,7 @@ Direct the camera before you write a command. A camera move is a cut the viewer 4. Stay framed across beats in the same area. Never zoom out and back in between them. 5. Go `wide` when the page changes under the viewer, and for the last beat. -The camera never follows clicks on its own. A frame that would zoom in less than 1.25x stays wide, and `polish.py` warns about a click outside its beat's frame. +The camera never follows clicks on its own. A frame that would zoom in less than 1.25x stays wide, and `polish.mjs` warns about a click outside its beat's frame. - Write the lines with the rules in `add-voiceover`: short, spoken, one idea each, naming what the viewer should notice, not what the mouse does. Put a line just before the command it describes. - `do` entries are agent-browser commands. `humantype ` types one key at a time; plain `type` and `fill` put the whole text in one frame. @@ -55,27 +55,27 @@ The camera never follows clicks on its own. A frame that would zoom in less than Voice the lines first, so each beat waits for its real narration, then record, then lay the voice under the video: ```bash -python3 /scripts/voiceover.py beats.json ./voice -uv run --with websockets python /scripts/record.py beats.json ./demo-out --voice voice/lines.json -uv run --with pillow python /scripts/polish.py demo-out -python3 /scripts/voiceover.py demo-out/narration.json ./voice --video demo-out/polished.mp4 +node /scripts/voiceover.mjs beats.json ./voice +node /scripts/record.mjs beats.json ./demo-out --voice voice/lines.json +node /scripts/polish.mjs demo-out +node /scripts/voiceover.mjs demo-out/narration.json ./voice --video demo-out/polished.mp4 ``` -`polish.py` gives the raw take the finish of a screen-recording editor: the camera runs the screenplay's shots over the whole composed shot, window and wallpaper included, easing between them in 0.7 to 1 s (a deeper zoom takes longer), and draws the pointer crisp at any zoom, blurred along its path while it moves, with a small squash and a fading ring on each press. The page sits as a rounded card with a shadow on the wallpaper in `assets/wallpaper.png` (`--background ` picks another; `--background none` fills the frame with the page edge to edge, for docs or side-by-side comparisons). `--frame safari` or `--frame chrome` puts the page in that browser's window, its address bar (and Chrome's tab title) following the page as it changes; use it when where you are in the app is part of the story. It renders at 2560x1440 and 60 fps, so text stays sharp even zoomed in. Timing does not change, so the narration still lines up. The last command reuses the lines the first one voiced and writes `voice/voiced.mp4`. Pick the voice as `add-voiceover` says before the first command, and keep it for both voice commands: the first command voices the final cut, and a different voice changes each line's length, which moves the clicks off their words. To rehearse cheaply, run them all with Kokoro, then again with the real voice. For a silent video, skip both voice commands and drop `--voice`; lines are then timed at reading pace and the voice can be added later. +`polish.mjs` gives the raw take the finish of a screen-recording editor: the camera runs the screenplay's shots over the whole composed shot, window and wallpaper included, easing between them in 0.7 to 1 s (a deeper zoom takes longer), and draws the pointer crisp at any zoom, blurred along its path while it moves, with a small squash and a fading ring on each press. The page sits as a rounded card with a shadow on the wallpaper in `assets/wallpaper.png` (`--background ` picks another; `--background none` fills the frame with the page edge to edge, for docs or side-by-side comparisons). `--frame safari` or `--frame chrome` puts the page in that browser's window, its address bar (and Chrome's tab title) following the page as it changes; use it when where you are in the app is part of the story. It renders at 2560x1440 and 60 fps, so text stays sharp even zoomed in. It lays out each frame in a headless Chrome (agent-browser session `polish`), about six seconds per second of video, and keeps the take's frames as JPEGs in `polish-frames/` while it runs, up to a gigabyte for a minute, removed when it finishes. Timing does not change, so the narration still lines up. The last command reuses the lines the first one voiced and writes `voice/voiced.mp4`. Pick the voice as `add-voiceover` says before the first command, and keep it for both voice commands: the first command voices the final cut, and a different voice changes each line's length, which moves the clicks off their words. To rehearse cheaply, run them all with Kokoro, then again with the real voice. For a silent video, skip both voice commands and drop `--voice`; lines are then timed at reading pace and the voice can be added later. -The page loads before recording starts, the viewport is 1280x720, and the player takes the frames straight from Chrome's screencast at 3x (agent-browser's own recorder captures at 1x), so the take is a true 3840x2160, at an even 30 fps; the camera and pointer move at 60. The take has no pointer; `polish.py` draws it, so always run it. The pointer moves like a hand: each path bends its own way and eases in and out in one movement to a point inside the target, rarely its centre, where it presses; and once the beat is done with it, it drifts sideways off the content. A field that already has focus is typed into without another click. The player writes `video.mp4`, `narration.json` with the second each line started, and `timeline.json` with each beat's shot and framed region, and the pointer's moves and presses. +The page loads before recording starts, the viewport is 1280x720, and the player takes the frames straight from Chrome's screencast at 3x (agent-browser's own recorder captures at 1x), so the take is a true 3840x2160, at an even 30 fps; the camera and pointer move at 60. The take has no pointer; `polish.mjs` draws it, so always run it. The pointer moves like a hand: each path bends its own way and eases in and out in one movement to a point inside the target, rarely its centre, where it presses; and once the beat is done with it, it drifts sideways off the content. A field that already has focus is typed into without another click. The player writes `video.mp4`, `narration.json` with the second each line started, and `timeline.json` with each beat's shot and framed region, and the pointer's moves and presses. If a command fails, the player stops and prints the agent-browser output. Fix the selector in `beats.json` and run it again. ## 4. Check before you share -Read what `polish.py` printed: the number of camera moves should match the shot changes you wrote, with no out-of-frame warning. Then extract one frame per beat, a second and a half after its `said` in `timeline.json`, and look at each: +Read what `polish.mjs` printed: the number of camera moves should match the shot changes you wrote, with no out-of-frame warning. Then extract one frame per beat, a second and a half after its `said` in `timeline.json`, and look at each: ```bash ffmpeg -v error -ss -i voice/voiced.mp4 -frames:v 1 beat.png ``` -Re-record when a frame shows the wrong state or the frame cuts off what the line names; change the shot and run `polish.py` again when only the camera is wrong. Never edit the video to hide a failed step. Then run the checks in `add-voiceover`. +Re-record when a frame shows the wrong state or the frame cuts off what the line names; change the shot and run `polish.mjs` again when only the camera is wrong. Never edit the video to hide a failed step. Then run the checks in `add-voiceover`. ## Share diff --git a/skills/record-demo/scripts/polish.html b/skills/record-demo/scripts/polish.html new file mode 100644 index 0000000..993316c --- /dev/null +++ b/skills/record-demo/scripts/polish.html @@ -0,0 +1,145 @@ + + + + + +
+ + +
+
+ + diff --git a/skills/record-demo/scripts/polish.mjs b/skills/record-demo/scripts/polish.mjs new file mode 100644 index 0000000..baaafb3 --- /dev/null +++ b/skills/record-demo/scripts/polish.mjs @@ -0,0 +1,200 @@ +// Give a recorded demo the finish of a screen-recording editor: a camera that frames what each beat's shot names, +// moving only when the shot changes and starting with the beat's first line; a cursor drawn crisp at any zoom along a +// hand's path, blurred while it moves, with a ring on each click; and the page as a rounded card with a shadow on a +// wallpaper, at 2560x1440 and 60 fps. +// +// usage: node polish.mjs [--background | --background none] [--frame safari|chrome] +// Reads video.mp4 and timeline.json, writes polished.mp4. The wallpaper defaults to ../assets/wallpaper.png; none fills +// the frame with the page edge to edge, with no card. --frame puts the page in a Safari or Chrome window whose address +// bar (and Chrome's tab title) follows the page. +// Each frame is laid out in polish.html and filmed in a headless Chrome through agent-browser, so nothing needs installing. +// Timing is untouched, so narration.json still lines up with polished.mp4. +import { execFileSync, spawn, spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, readFileSync, readdirSync, rmSync } from "node:fs"; +import { delimiter, join, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; + +const die = (msg) => { console.error(msg); process.exit(1); }; + +function which(name) { + const exts = process.platform === "win32" ? (process.env.PATHEXT || ".EXE;.CMD;.BAT").split(";") : [""]; + for (const dir of (process.env.PATH || "").split(delimiter)) + for (const ext of exts) if (dir && existsSync(join(dir, name + ext))) return join(dir, name + ext); +} +const tool = (name) => which(name) || die(`${name} not found on PATH`); + +const W = 2560, H = 1440, FPS = 60; // a 4K take framed up to MAX_ZOOM still fills this card about pixel for pixel +const MARGIN = 0.15; // room around a framed region, each side +const MAX_ZOOM = 2.0; +const MIN_ZOOM = 1.25; // a frame that needs less than this isn't worth a camera move: it stays wide +const CURSOR = 28; // pointer height in CSS pixels, a little larger than life so it reads in a video +const SHUTTER = 0.5; // the pointer blurs over this share of a frame while it moves, like a camera's 180° shutter +const RING = 0.45; // seconds a click's ring takes to spread and fade + +const moveTime = (a, c) => 0.6 + 0.55 * Math.log(Math.max(a[2], c[2])); // a deeper zoom travels further: 0.7 s at 1.25x, about 1 s at 2x + +const args = process.argv.slice(2); +const flag = (name, dflt) => { const i = args.indexOf(name); return i >= 0 ? args.splice(i, 2)[1] : dflt; }; +const wallpaper = flag("--background", join(import.meta.dirname, "..", "assets", "wallpaper.png")); +const kind = flag("--frame", ""); // a browser window around the page +if (!["", "safari", "chrome"].includes(kind)) die("--frame takes safari or chrome"); +const out = args[0] || die("usage: node polish.mjs [--background | --background none] [--frame safari|chrome]"); +const tl = JSON.parse(readFileSync(join(out, "timeline.json"), "utf8")); +const src = join(out, "video.mp4"); +const FFMPEG = tool("ffmpeg"); +const probe = execFileSync(tool("ffprobe"), ["-v", "error", "-select_streams", "v", "-show_entries", "stream=width,height,r_frame_rate", + "-show_entries", "format=duration", "-of", "json", src], { encoding: "utf8" }); +const { streams: [{ width: sw, height: sh, r_frame_rate: rate }], format: { duration } } = JSON.parse(probe); +const srcFps = rate.split("/").reduce((a, b) => a / b); // "30/1" +const [vw, vh] = tl.viewport; + +const ease = (u) => { u = Math.min(1, Math.max(0, u)); return u < 0.5 ? 4 * u ** 3 : 1 - (-2 * u + 2) ** 3 / 2; }; + +// 1. The pointer: each logged move along a gentle curve, with a minimum-jerk speed profile, like a hand. +function pointer(t) { + let at = tl.pointer; + for (const m of tl.moves) { + if (t < m.t0) break; + if (t >= m.t1) { at = m.to; continue; } + const [[x0, y0], [x1, y1]] = [m.from, m.to]; + const u = (t - m.t0) / Math.max(1e-3, m.t1 - m.t0); + const s = 10 * u ** 3 - 15 * u ** 4 + 6 * u ** 5; + const bx = (x0 + x1) / 2 - (y1 - y0) * m.bend, by = (y0 + y1) / 2 + (x1 - x0) * m.bend; // control point off the line + return [(1 - s) ** 2 * x0 + 2 * (1 - s) * s * bx + s * s * x1, (1 - s) ** 2 * y0 + 2 * (1 - s) * s * by + s * s * y1]; + } + return at; +} + +function squash(t) { // the cursor dips on a press and springs back + const d = tl.presses.map((p) => t - p.at).find((d) => d >= -0.05 && d <= 0.25); + return d === undefined ? 1 : 1 - 0.18 * Math.sin((Math.PI * (d + 0.05)) / 0.3); +} + +// 2. The scene at zoom 1: the backdrop with the card, and the window chrome above the page. +const u = W / 1920; +const bare = wallpaper === "none"; +if (bare && kind) die("--frame needs a background to sit on"); +const [PAD, RADIUS] = bare ? [0, 0] : [Math.round(48 * u), Math.round(20 * u)]; // a thin margin of wallpaper: the page is the subject +const barCss = { safari: 38, chrome: 76 }[kind] ?? 0; // window chrome height, in the page's CSS pixels +const BAR = Math.round((barCss * (H - 2 * PAD)) / (vh + barCss)); +const ch = H - 2 * PAD - BAR; +const cw = Math.round((ch * sw) / sh); // keep the page's shape +const left = Math.floor((W - cw) / 2); +const latest = (key, t) => [...(tl[key + "s"] ?? [])].reverse().find((x) => x.at <= t)?.[key] ?? ""; + +// 3. The camera moves over the whole scene (wallpaper, window and page, as laid out at zoom 1): one view, its centre and +// zoom in scene pixels, per beat, from the beat's shot. +const top = PAD + BAR; // the page's top edge in the scene +const css = cw / vw; // scene pixels per CSS pixel +const WIDE = [W / 2, H / 2, 1]; + +function view(beat, n) { + const f = beat.frame; + if (!f) return WIDE; + const [x0, y0, x1, y1] = [left + f[0] * css, top + f[1] * css, left + f[2] * css, top + f[3] * css]; + const z = Math.min(MAX_ZOOM, W / ((x1 - x0) * (1 + 2 * MARGIN)), H / ((y1 - y0) * (1 + 2 * MARGIN))); + if (z < MIN_ZOOM) { + console.error(`beat ${n}: ${beat.shot} nearly fills the frame (zoom ${z.toFixed(2)}); staying wide`); + return WIDE; + } + return [Math.min(Math.max((x0 + x1) / 2, W / z / 2), W - W / z / 2), Math.min(Math.max((y0 + y1) / 2, H / z / 2), H - H / z / 2), z]; +} + +const cuts = []; // the camera moves only where the shot changes, starting with the beat's first line +let here = WIDE; +tl.beats.forEach((b, i) => { + const there = view(b, i + 1); + if (Math.max(...here.map((a, k) => Math.abs(a - there[k]))) > 1) { + cuts.push([b.said - 0.15, here, there, moveTime(here, there)]); + here = there; + } +}); +console.error(`camera moves: ${cuts.length}`); + +function camera(t) { + let cam = WIDE; + for (const [start, a, c, d] of cuts) { + if (t < start) break; + const k = ease((t - start) / d); + const z = Math.exp(Math.log(a[2]) + (Math.log(c[2]) - Math.log(a[2])) * k); // zoom feels even in log space + cam = [a[0] + (c[0] - a[0]) * k, a[1] + (c[1] - a[1]) * k, z]; + } + return cam; +} + +for (const p of tl.presses) { // a click the camera can't see is a screenplay bug + const [cx, cy, z] = camera(p.at); + if (Math.abs(left + p.x * css - cx) > W / z / 2 || Math.abs(top + p.y * css - cy) > H / z / 2) + console.error(`warning: the click at ${p.at.toFixed(1)} s (${p.x}, ${p.y}) is out of frame; widen that beat's shot`); +} + +function onScreen(t, x, y) { // a page point, in CSS pixels, where it lands on screen at time t + const [cx, cy, z] = camera(t); + return [(left + x * css - cx + W / z / 2) * z, (top + y * css - cy + H / z / 2) * z]; +} + +function frameState(t, frame) { + const [cx, cy, z] = camera(t); + const vx = cx - W / z / 2, vy = cy - H / z / 2; // the view's top-left, in scene pixels + const at = (x, y) => [(x - vx) * z, (y - vy) * z]; + const [x0, wy] = at(left, PAD), [x1, y1] = at(left + cw, top + ch); + const rings = tl.presses.filter((p) => t - p.at >= 0 && t - p.at <= RING).map((p) => { + const k = (t - p.at) / RING, scale = css * z; + return [...onScreen(t, p.x, p.y), (8 + 22 * (1 - (1 - k) ** 3)) * scale, 2.5 * scale, 1 - k]; + }); + let spots = [0, 1, 2, 3, 4].map((i) => t + (SHUTTER / FPS) * (i / 4 - 0.5)).map((ts) => onScreen(ts, ...pointer(ts))); + if (Math.hypot(spots[0][0] - spots[4][0], spots[0][1] - spots[4][1]) < 1.5) spots = [spots[2]]; + return { frame, view: [vx, vy, z], clip: [x0, wy, x1, y1, RADIUS * z], rings, spots, + unit: (CURSOR * css * z * squash(t)) / 21, url: latest("url", t), title: latest("title", t) }; +} + +// The take's frames as images polish.html can show, one per frame of the take. +const frames = resolve(out, "polish-frames"); +rmSync(frames, { recursive: true, force: true }); +mkdirSync(frames); +execFileSync(FFMPEG, ["-v", "error", "-i", src, "-q:v", "2", join(frames, "%06d.jpg")]); +const count = readdirSync(frames).length; +const frameUrl = (n) => pathToFileURL(join(frames, `${String(n + 1).padStart(6, "0")}.jpg`)).href; + +const AGENT = tool("agent-browser"); +const ab = (...a) => execFileSync(AGENT, ["--session", "polish", ...a], { encoding: "utf8", shell: /\.(cmd|bat)$/i.test(AGENT) }); +ab("open", "about:blank"); +ab("set", "viewport", String(W), String(H)); +ab("open", pathToFileURL(join(import.meta.dirname, "polish.html")).href); +const ws = new WebSocket(ab("get", "cdp-url").trim()); +await new Promise((ok, fail) => { ws.onopen = ok; ws.onerror = fail; }); +let id = 0; +const replies = new Map(); +ws.onmessage = (e) => { const m = JSON.parse(e.data); replies.get(m.id)?.(m); }; +const call = (method, params = {}, sessionId) => new Promise((ok, fail) => { + replies.set(++id, (m) => (m.error ? fail(new Error(`${method}: ${m.error.message}`)) : ok(m.result))); + ws.send(JSON.stringify({ id, method, params, ...(sessionId && { sessionId }) })); +}); +const page = (await call("Target.getTargets")).targetInfos.find((t) => t.type === "page" && t.url.endsWith("polish.html")); +const session = (await call("Target.attachToTarget", { targetId: page.targetId, flatten: true })).sessionId; +async function run(expression) { + const r = await call("Runtime.evaluate", { expression, awaitPromise: true }, session); + if (r.exceptionDetails) die(`polish.html: ${r.exceptionDetails.exception?.description ?? r.exceptionDetails.text}`); +} +await run(`setup(${JSON.stringify({ W, H, PAD, RADIUS, BAR, barCss, cw, ch, left, u, kind, vw, bare, + wallpaper: bare ? "" : pathToFileURL(resolve(wallpaper)).href })})`); + +const polished = join(out, "polished.mp4"); +const enc = spawn(FFMPEG, ["-v", "error", "-y", "-f", "image2pipe", "-framerate", String(FPS), "-i", "-", + "-c:v", "libx264", "-preset", "medium", "-crf", "18", "-pix_fmt", "yuv420p", "-movflags", "faststart", polished], + { stdio: ["pipe", "inherit", "inherit"] }); +const total = Math.floor(duration * FPS); +console.error(`filming ${total} frames`); +for (let n = 0; n < total; n++) { + const t = n / FPS; + await run(`draw(${JSON.stringify(frameState(t, frameUrl(Math.min(count - 1, Math.floor(t * srcFps + 1e-6)))))})`); + const { data } = await call("Page.captureScreenshot", { format: "jpeg", quality: 95, optimizeForSpeed: true }, session); + if (!enc.stdin.write(Buffer.from(data, "base64"))) await new Promise((r) => enc.stdin.once("drain", r)); +} +enc.stdin.end(); +await new Promise((r) => enc.on("close", r)); +ws.close(); +spawnSync(AGENT, ["--session", "polish", "close"], { shell: /\.(cmd|bat)$/i.test(AGENT) }); +rmSync(frames, { recursive: true }); +if (enc.exitCode !== 0) die("ffmpeg failed to encode the polished video"); +console.log(polished); diff --git a/skills/record-demo/scripts/polish.py b/skills/record-demo/scripts/polish.py deleted file mode 100644 index 4b7b1f9..0000000 --- a/skills/record-demo/scripts/polish.py +++ /dev/null @@ -1,318 +0,0 @@ -"""Give a recorded demo the finish of a screen-recording editor: a camera that frames what each beat's shot names, -moving only when the shot changes and starting with the beat's first line; a cursor drawn crisp at any zoom along a -hand's path, blurred while it moves, with a ring on each click; and the page as a rounded card with a shadow on a wallpaper, at 2560x1440 and 60 fps. - -usage: uv run --with pillow python polish.py [--background | --background none] [--frame safari|chrome] -Reads video.mp4 and timeline.json, writes polished.mp4. The wallpaper defaults to ../assets/wallpaper.png; none fills -the frame with the page edge to edge, with no card. --frame puts the page in a Safari or Chrome window whose address -bar (and Chrome's tab title) follows the page. -Timing is untouched, so narration.json still lines up with polished.mp4. -""" -import json, math, shutil, subprocess, sys -from pathlib import Path -try: - from PIL import Image, ImageChops, ImageDraw, ImageFilter, ImageFont, ImageOps -except ImportError: - sys.exit("needs Pillow: uv run --with pillow python polish.py , or pip install pillow") - -W, H, FPS = 2560, 1440, 60 # a 4K take framed up to MAX_ZOOM still fills this card about pixel for pixel -MARGIN = 0.15 # room around a framed region, each side -MAX_ZOOM = 2.0 -MIN_ZOOM = 1.25 # a frame that needs less than this isn't worth a camera move: it stays wide -CURSOR = 28 # pointer height in CSS pixels, a little larger than life so it reads in a video -SHUTTER = 0.5 # the pointer blurs over this share of a frame while it moves, like a camera's 180° shutter -RING = 0.45 # seconds a click's ring takes to spread and fade - -def move_time(a, c): # a deeper zoom travels further, so it takes longer: 0.7 s at 1.25x, about 1 s at 2x - return 0.6 + 0.55 * math.log(max(a[2], c[2])) - -def tool(name): - return shutil.which(name) or sys.exit(f"{name} not found on PATH") - -args = sys.argv[1:] -wallpaper = args.pop(args.index("--background") + 1) if "--background" in args else Path(__file__).parent.parent / "assets" / "wallpaper.png" -kind = args.pop(args.index("--frame") + 1) if "--frame" in args else "" # a browser window around the page -if kind not in ("", "safari", "chrome"): - sys.exit("--frame takes safari or chrome") -out = Path(args[0]) -tl = json.loads((out / "timeline.json").read_text(encoding="utf-8")) -src = out / "video.mp4" -sw, sh = map(int, subprocess.check_output([tool("ffprobe"), "-v", "error", "-select_streams", "v", "-show_entries", "stream=width,height", - "-of", "csv=p=0", str(src)], text=True).strip().split(",")) -vw, vh = tl["viewport"] - -def ease(u): - u = min(1.0, max(0.0, u)) - return 4 * u ** 3 if u < 0.5 else 1 - (-2 * u + 2) ** 3 / 2 - -# 1. The pointer: each logged move along a gentle curve, with a minimum-jerk speed profile, like a hand. -def pointer(t): - at = tl["pointer"] - for m in tl["moves"]: - if t < m["t0"]: - break - if t >= m["t1"]: - at = m["to"] - continue - (x0, y0), (x1, y1) = m["from"], m["to"] - u = (t - m["t0"]) / max(1e-3, m["t1"] - m["t0"]) - s = 10 * u ** 3 - 15 * u ** 4 + 6 * u ** 5 - bx, by = (x0 + x1) / 2 - (y1 - y0) * m["bend"], (y0 + y1) / 2 + (x1 - x0) * m["bend"] # control point off the line - return ((1 - s) ** 2 * x0 + 2 * (1 - s) * s * bx + s * s * x1, (1 - s) ** 2 * y0 + 2 * (1 - s) * s * by + s * s * y1) - return at - -def squash(t): # the cursor dips on a press and springs back - d = [t - p["at"] for p in tl["presses"] if -0.05 <= t - p["at"] <= 0.25] - return 1 - 0.18 * math.sin(math.pi * (d[0] + 0.05) / 0.3) if d else 1.0 - -# An arrow cursor drawn once, large: a near-black arrow in an even white rim, corners rounded, on a soft drop shadow; -# scaled per frame. The rim is a stroke around the arrow with round joins, so it stays even down the thin tail. -S, PAD_C, RIM = 16, 5, 1.25 -ARROW = [(0, 0), (0, 16.4), (4.0, 12.7), (6.5, 18.6), (9.0, 17.5), (6.6, 11.8), (11.6, 11.8)] -pts = [((x + PAD_C) * S, (y + PAD_C) * S) for x, y in ARROW] -sprite = Image.new("RGBA", ((14 + 2 * PAD_C) * S, (21 + 2 * PAD_C) * S), (0, 0, 0, 0)) - -def rounded(m, r): # blur, then cut at the middle: rounds every corner by about r - return m.filter(ImageFilter.GaussianBlur(r * S)).point(lambda v: 255 if v > 127 else 0) - -core = Image.new("L", sprite.size, 0) -ImageDraw.Draw(core).polygon(pts, fill=255) -rim = core.copy() -d = ImageDraw.Draw(rim) -d.line(pts + [pts[0]], fill=255, width=round(2 * RIM * S), joint="curve") -for x, y in pts: - d.ellipse((x - RIM * S, y - RIM * S, x + RIM * S, y + RIM * S), fill=255) -rim, core = rounded(rim, 0.3), rounded(core, 0.3) -shade = Image.new("L", sprite.size, 0) -shade.paste(rim, (round(0.3 * S), round(0.9 * S))) -sprite.paste((0, 0, 0, 255), (0, 0), shade.filter(ImageFilter.GaussianBlur(1.3 * S)).point(lambda v: v * 95 // 255)) -sprite.paste((255, 255, 255, 255), (0, 0), rim) -sprite.paste((18, 18, 20, 255), (0, 0), core) - -# 2. The scene at zoom 1: the backdrop with the card's shadow, and the window chrome, drawn once (per address). -u = W / 1920 -bare = str(wallpaper) == "none" -if bare and kind: - sys.exit("--frame needs a background to sit on") -PAD, RADIUS = (0, 0) if bare else (round(48 * u), round(20 * u)) # a thin margin of wallpaper: the page is the subject -BAR_CSS = {"safari": 38, "chrome": 76}.get(kind, 0) # window chrome height, in the page's CSS pixels -BAR = round(BAR_CSS * (H - 2 * PAD) / (vh + BAR_CSS)) -ch = H - 2 * PAD - BAR -cw = round(ch * sw / sh) # keep the page's shape -left = (W - cw) // 2 -# The wallpaper's colours only, smoothed, then fresh grain at this size: scaled-up grain looks smeared, and a bare -# gradient bands. bleed trims edge rows, which can be dark. -if bare: - bg = Image.new("RGB", (W, H)) -else: - bg = ImageOps.fit(Image.open(wallpaper).convert("RGB"), (W, H), Image.LANCZOS, bleed=0.02).filter(ImageFilter.GaussianBlur(W / 100)) - shadow = Image.new("L", (W, H), 0) - ImageDraw.Draw(shadow).rounded_rectangle((left, PAD + round(16 * u), left + cw, PAD + BAR + ch + round(16 * u)), RADIUS, fill=80) - bg.paste((0, 0, 0), (0, 0), shadow.filter(ImageFilter.GaussianBlur(28 * u))) - -def font(size): - for f in ("/System/Library/Fonts/SFNS.ttf", "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", "C:/Windows/Fonts/segoeui.ttf"): - try: - return ImageFont.truetype(f, size) - except OSError: - pass - return ImageFont.load_default(size) - -def fit(f, text, room): - if f.getlength(text) > room: - while text and f.getlength(text + "…") > room: - text = text[:-1] - text += "…" - return text - -GREY, DIM = (95, 99, 104), (60, 64, 67) -bars = {} -def window(url, title): - # Drawn three times larger and scaled down, so circles and strokes come out smooth. Sizes are CSS pixels. - if (url, title) in bars: - return bars[url, title] - s = 3 - q = ch / vh * s # output pixels per CSS pixel, supersampled - img = Image.new("RGB", (cw * s, BAR * s), (236, 236, 236) if kind == "safari" else (222, 226, 231)) - d = ImageDraw.Draw(img) - width = cw * s / q # the window's width in CSS pixels - lights = 19 if kind == "safari" else 18 - for i, (fill, edge) in enumerate((((255, 95, 87), (226, 70, 63)), ((254, 188, 46), (223, 158, 32)), ((40, 200, 64), (30, 170, 50)))): - x = (20 + 20 * i) * q - d.ellipse((x - 6 * q, lights * q - 6 * q, x + 6 * q, lights * q + 6 * q), fill=fill, outline=edge, width=round(q * 0.5)) - host, _, rest = url.split("://", 1)[-1].partition("/") - rest = ("/" + rest).rstrip("/") if rest else "" - if kind == "safari": - d.rectangle((0, BAR * s - q * 0.6, cw * s, BAR * s), fill=(214, 214, 214)) - pw = width * 0.42 - d.rounded_rectangle(((width - pw) / 2 * q, 7 * q, (width + pw) / 2 * q, 31 * q), 6 * q, fill=(255, 255, 255)) - f = font(round(12.5 * q)) - text = fit(f, host + rest, (pw - 50) * q) - lock = 12 * q if url.startswith("https") else 0 - x = width / 2 * q - (f.getlength(text) + lock) / 2 - if lock: # a padlock: body and shackle - d.rounded_rectangle((x, 18 * q, x + 7 * q, 24 * q), q, fill=(110, 110, 110)) - d.arc((x + 1.2 * q, 13.5 * q, x + 5.8 * q, 20 * q), 180, 360, fill=(110, 110, 110), width=round(1.3 * q)) - d.text((x + lock, 19 * q), text, font=f, fill=(70, 70, 70), anchor="lm") - else: - # Tab strip: the one tab, its favicon, title and close, then a new-tab plus. - d.rounded_rectangle((80 * q, 6 * q, 320 * q, 44 * q), 8 * q, fill=(255, 255, 255), corners=(True, True, False, False)) - d.ellipse((92 * q, 14 * q, 106 * q, 28 * q), outline=GREY, width=round(1.2 * q)) # a globe for the favicon - d.ellipse((96.5 * q, 14 * q, 101.5 * q, 28 * q), outline=GREY, width=round(1.0 * q)) - d.line((92 * q, 21 * q, 106 * q, 21 * q), fill=GREY, width=round(1.0 * q)) - f = font(round(12 * q)) - d.text((114 * q, 21 * q), fit(f, title or host, 170 * q), font=f, fill=(31, 31, 31), anchor="lm") - for a, b in (((300, 17), (308, 25)), ((308, 17), (300, 25))): - d.line((a[0] * q, a[1] * q, b[0] * q, b[1] * q), fill=GREY, width=round(1.3 * q)) - d.line((334 * q, 21 * q, 346 * q, 21 * q), fill=GREY, width=round(1.4 * q)) - d.line((340 * q, 15 * q, 340 * q, 27 * q), fill=GREY, width=round(1.4 * q)) - # Toolbar: back, forward, reload, the address box, the menu. - d.rectangle((0, 36 * q, cw * s, BAR * s), fill=(255, 255, 255)) - d.rectangle((0, BAR * s - q * 0.6, cw * s, BAR * s), fill=(218, 220, 224)) - for x, ink, sign in ((20, GREY, -1), (52, (190, 192, 196), 1)): # forward is greyed out: nothing to go forward to - d.line(((x - 7) * q, 56 * q, (x + 7) * q, 56 * q), fill=ink, width=round(1.6 * q)) - tip = x + 7 * sign - d.line((tip * q, 56 * q, (tip - 6 * sign) * q, 50 * q), fill=ink, width=round(1.6 * q)) - d.line((tip * q, 56 * q, (tip - 6 * sign) * q, 62 * q), fill=ink, width=round(1.6 * q)) - # Reload: a circle open at the top right, its top end an arrowhead pointing clockwise. - d.arc((77.5 * q, 49.5 * q, 90.5 * q, 62.5 * q), 330, 270, fill=GREY, width=round(1.6 * q)) - d.polygon(((83.5 * q, 46.2 * q), (87.6 * q, 50 * q), (83.5 * q, 53.8 * q)), fill=GREY) - d.rounded_rectangle((104 * q, 41 * q, (width - 44) * q, 71 * q), 15 * q, fill=(239, 241, 244)) - for y, dot in ((52.5, 124), (59.5, 119)): # the site-settings icon: two sliders - d.line((116 * q, y * q, 128 * q, y * q), fill=DIM, width=round(1.3 * q)) - d.ellipse(((dot - 2.2) * q, (y - 2.2) * q, (dot + 2.2) * q, (y + 2.2) * q), fill=(239, 241, 244), outline=DIM, width=round(1.2 * q)) - f = font(round(13.5 * q)) - host_t = fit(f, host, (width - 200) * q) - d.text((140 * q, 56 * q), host_t, font=f, fill=(31, 31, 31), anchor="lm") - if rest and host_t == host: - d.text((140 * q + f.getlength(host), 56 * q), fit(f, rest, (width - 200) * q - f.getlength(host)), font=f, fill=GREY, anchor="lm") - for y in (50, 56, 62): # the menu - d.ellipse(((width - 22) * q, (y - 1.6) * q, (width - 18.8) * q, (y + 1.6) * q), fill=DIM) - bars[url, title] = img.resize((cw * 2, BAR * 2), Image.LANCZOS) # twice the size, so it stays sharp zoomed in - return bars[url, title] - -def latest(key, t): - return next((x[key] for x in reversed(tl.get(key + "s", [])) if x["at"] <= t), "") - -# 3. The camera moves over the whole scene (wallpaper, window and page, as laid out at zoom 1): one view, its centre and -# zoom in scene pixels, per beat, from the beat's shot. -top = PAD + BAR # the page's top edge in the scene -css = cw / vw # scene pixels per CSS pixel -WIDE = (W / 2, H / 2, 1.0) - -def view(beat, n): - f = beat.get("frame") - if not f: - return WIDE - x0, y0, x1, y1 = left + f[0] * css, top + f[1] * css, left + f[2] * css, top + f[3] * css - z = min(MAX_ZOOM, W / ((x1 - x0) * (1 + 2 * MARGIN)), H / ((y1 - y0) * (1 + 2 * MARGIN))) - if z < MIN_ZOOM: - print(f"beat {n}: {beat['shot']} nearly fills the frame (zoom {z:.2f}); staying wide", file=sys.stderr) - return WIDE - return (min(max((x0 + x1) / 2, W / z / 2), W - W / z / 2), min(max((y0 + y1) / 2, H / z / 2), H - H / z / 2), z) - -cuts, here = [], WIDE # the camera moves only where the shot changes, starting with the beat's first line -for n, b in enumerate(tl["beats"], 1): - there = view(b, n) - if max(abs(a - c) for a, c in zip(here, there)) > 1: - cuts.append((b["said"] - 0.15, here, there, move_time(here, there))) - here = there -print(f"camera moves: {len(cuts)}", file=sys.stderr) - -def camera(t): - cam = WIDE - for start, a, c, d in cuts: - if t < start: - break - u = ease((t - start) / d) - z = math.exp(math.log(a[2]) + (math.log(c[2]) - math.log(a[2])) * u) # zoom feels even in log space - cam = (a[0] + (c[0] - a[0]) * u, a[1] + (c[1] - a[1]) * u, z) - return cam - -for p in tl["presses"]: # a click the camera can't see is a screenplay bug - cx, cy, z = camera(p["at"]) - if abs(left + p["x"] * css - cx) > W / z / 2 or abs(top + p["y"] * css - cy) > H / z / 2: - print(f"warning: the click at {p['at']:.1f} s ({p['x']}, {p['y']}) is out of frame; widen that beat's shot", file=sys.stderr) - -def blit(dst, src, x0, y0, x1, y1): - # Stretch src over the output rectangle x0,y0-x1,y1, resampling only the part on screen. - ix0, iy0, ix1, iy1 = round(max(0, x0)), round(max(0, y0)), round(min(W, x1)), round(min(H, y1)) - if ix1 <= ix0 or iy1 <= iy0: - return - fx, fy = src.width / (x1 - x0), src.height / (y1 - y0) - box = (max(0, (ix0 - x0) * fx), max(0, (iy0 - y0) * fy), min(src.width, (ix1 - x0) * fx), min(src.height, (iy1 - y0) * fy)) - dst.paste(src.resize((ix1 - ix0, iy1 - iy0), Image.BICUBIC, box=box), (ix0, iy0)) # box clamped: rounding can overshoot a hair - -def on_screen(t, x, y): # a page point, in CSS pixels, where it lands on screen at time t - cx, cy, z = camera(t) - return (left + x * css - cx + W / z / 2) * z, (top + y * css - cy + H / z / 2) * z - -def rings(layer, t, scale): - # Each click sends out a ring that spreads and fades: dark on a white halo, so it reads on light and dark pages. - for p in tl["presses"]: - if not 0 <= t - p["at"] <= RING: - continue - u = (t - p["at"]) / RING - r, w, a = (8 + 22 * (1 - (1 - u) ** 3)) * scale, 2.5 * scale, 1 - u - s, half = 3, math.ceil(r + 2 * w) + 2 # drawn three times larger and scaled down, so the circle is smooth - img = Image.new("RGBA", (2 * half * s, 2 * half * s), (0, 0, 0, 0)) - d, box = ImageDraw.Draw(img), [(half - r) * s, (half - r) * s, (half + r) * s, (half + r) * s] - d.ellipse([v + (-w if i < 2 else w) * s * 0.5 for i, v in enumerate(box)], outline=(255, 255, 255, round(140 * a)), width=round(2 * w * s)) - d.ellipse(box, outline=(30, 30, 30, round(160 * a)), width=round(w * s)) - img = img.resize((2 * half, 2 * half), Image.LANCZOS) - x, y = on_screen(t, p["x"], p["y"]) - layer.paste(img, (round(x) - half, round(y) - half), img) - -def draw_pointer(layer, t, cur, off): - # The pointer where it is at a few instants across the shutter, averaged, so a fast move blurs along its path - # (camera moves included) instead of stepping from frame to frame. Still, it is one crisp paste. - spots = [on_screen(ts, *pointer(ts)) for ts in (t + SHUTTER / FPS * (i / 4 - 0.5) for i in range(5))] - if math.dist(spots[0], spots[-1]) < 1.5: - x, y = spots[2] - layer.paste(cur, (round(x - off), round(y - off)), cur) - return - x0, y0 = min(x for x, _ in spots), min(y for _, y in spots) - size = (math.ceil(max(x for x, _ in spots) - x0) + cur.width + 1, math.ceil(max(y for _, y in spots) - y0) + cur.height + 1) - pm, acc = cur.convert("RGBa"), None # premultiplied, so the average has no dark fringe - for i, (x, y) in enumerate(spots): - one = Image.new("RGBa", size, (0, 0, 0, 0)) - one.paste(pm, (round(x - x0), round(y - y0))) - acc = one if acc is None else Image.blend(acc, one, 1 / (i + 1)) - acc = acc.convert("RGBA") - layer.paste(acc, (round(x0 - off), round(y0 - off)), acc) - -grain = Image.effect_noise((W, H), 5).convert("RGB") # on the screen, not the scene: it keeps its size at any zoom - -dec = subprocess.Popen([tool("ffmpeg"), "-v", "error", "-i", str(src), "-r", str(FPS), "-f", "rawvideo", "-pix_fmt", "rgb24", "-"], - stdout=subprocess.PIPE) -enc = subprocess.Popen([tool("ffmpeg"), "-v", "error", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24", "-s", f"{W}x{H}", "-r", str(FPS), "-i", "-", - "-c:v", "libx264", "-preset", "medium", "-crf", "18", "-pix_fmt", "yuv420p", "-movflags", "faststart", - str(out / "polished.mp4")], stdin=subprocess.PIPE) -n = 0 -while len(raw := dec.stdout.read(sw * sh * 3)) == sw * sh * 3: - t = n / FPS - cx, cy, z = camera(t) - vx, vy = cx - W / z / 2, cy - H / z / 2 # the view's top-left, in scene pixels - def at(x, y): - return (x - vx) * z, (y - vy) * z - canvas = bg.resize((W, H), Image.BICUBIC, box=(vx, vy, vx + W / z, vy + H / z)) - if not bare: - canvas = ImageChops.add(canvas, grain, offset=-128) - layer = Image.new("RGB", (W, H)) - (x0, y0), (x1, y1), wy = at(left, top), at(left + cw, top + ch), at(0, PAD)[1] - blit(layer, Image.frombytes("RGB", (sw, sh), raw), x0, y0, x1, y1) - if kind: - blit(layer, window(latest("url", t), latest("title", t)), x0, wy, x1, y0) - rings(layer, t, css * z) - size = CURSOR * css * z * squash(t) / 21 / S - cur = sprite.resize((max(1, round(sprite.width * size)), max(1, round(sprite.height * size))), Image.LANCZOS) - draw_pointer(layer, t, cur, PAD_C * S * size) - mask = Image.new("L", (W, H), 0) - ImageDraw.Draw(mask).rounded_rectangle((x0, wy, x1 - 1, y1 - 1), RADIUS * z, fill=255) - canvas.paste(layer, (0, 0), mask) - enc.stdin.write(canvas.tobytes()) - n += 1 -enc.stdin.close() -enc.wait() -dec.wait() -print(out / "polished.mp4") diff --git a/skills/record-demo/scripts/record.mjs b/skills/record-demo/scripts/record.mjs new file mode 100644 index 0000000..389192f --- /dev/null +++ b/skills/record-demo/scripts/record.mjs @@ -0,0 +1,280 @@ +// Play beats.json in agent-browser while recording a silent video, paced for the narration laid under it later. +// +// usage: node record.mjs [--voice ] +// +// A ["say", "..."] action starts its line once the last one has finished, and the actions after it run while it plays, +// so a click lands on the words about it. --voice takes each line's real length from a voiced lines.json (a list of +// {"say", "seconds"}); without it, a line lasts as long as it takes to read aloud at an even pace. +// A beat's "shot" is "wide" or a selector the camera frames; a beat without one keeps the last. The take has no pointer: +// polish.mjs draws it, along the moves logged here, and runs the camera the shots ask for. +// The take comes straight from Chrome's screencast at 3x (3840x2160), so the camera can frame a region and keep text sharp. +// Writes video.mp4, narration.json (each line with the second it starts) and timeline.json (beats with their framed +// region, pointer moves and presses, for polish.mjs). +import { exec, execFile, spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { delimiter, join } from "node:path"; +import { promisify } from "node:util"; + +const die = (msg) => { console.error(msg); process.exit(1); }; + +function which(name) { + // Full path, so Windows finds npm's .cmd shims too. + const exts = process.platform === "win32" ? (process.env.PATHEXT || ".EXE;.CMD;.BAT").split(";") : [""]; + for (const dir of (process.env.PATH || "").split(delimiter)) + for (const ext of exts) if (dir && existsSync(join(dir, name + ext))) return join(dir, name + ext); +} +const tool = (name) => which(name) || die(`${name} not found on PATH`); + +const SCALE = 3; // device pixels per CSS pixel; a real device scale, since an emulated one screencasts at 1x +const AGENT = tool("agent-browser"); +const AB = ["--session", "demo", "--args", `--force-device-scale-factor=${SCALE}`]; +const FFMPEG = tool("ffmpeg"); +const args = process.argv.slice(2); +const vi = args.indexOf("--voice"); +const voiced = Object.fromEntries((vi >= 0 ? JSON.parse(readFileSync(args.splice(vi, 2)[1], "utf8")) : []).map((l) => [l.say, l.seconds])); +const [beatsFile, url, out] = args; +if (!out) die("usage: node record.mjs [--voice ]"); +const beats = JSON.parse(readFileSync(beatsFile, "utf8")); +mkdirSync(out, { recursive: true }); + +// Every command is async: Chrome sends the next frame only once the last is acknowledged, so a blocking call drops frames. +const execFileP = promisify(execFile); +const shim = /\.(cmd|bat)$/i.test(AGENT); // a Windows .cmd shim only runs through the shell +async function ab(...a) { + try { + const argv = [...AB, ...a]; + const { stdout } = shim + ? await execFileP(`"${AGENT}"`, argv.map((x) => `"${x.replaceAll('"', '""')}"`), { shell: true, maxBuffer: 1 << 26 }) + : await execFileP(AGENT, argv, { maxBuffer: 1 << 26 }); + return stdout; + } catch (e) { + spawnSync(AGENT, [...AB, "close"], { shell: shim }); + die(`agent-browser ${a.join(" ")} failed: ${e.stdout ?? ""}${e.stderr ?? e.message}`); + } +} + +// Run JS on one element: CSS, or XPath after "xpath=". +async function on(sel, body) { + const find = "s.startsWith('xpath=') ? document.evaluate(s.slice(6), document, null, 9, null).singleNodeValue : document.querySelector(s)"; + return JSON.parse(await ab("eval", `(s => { const el = ${find}; ${body} })(${JSON.stringify(sel)})`)); +} + +const sleep = (s) => new Promise((r) => setTimeout(r, Math.max(0, s) * 1000)); +const mono = () => performance.now() / 1000; +const uniform = (a, b) => a + Math.random() * (b - a); +const gauss = (mu, sigma) => mu + sigma * Math.sqrt(-2 * Math.log(1 - Math.random())) * Math.cos(2 * Math.PI * Math.random()); +const clamp = (v, lo, hi) => Math.min(hi, Math.max(lo, v)); +const r1 = (v) => Math.round(v * 10) / 10, r3 = (v) => Math.round(v * 1000) / 1000; + +let t0, wall0; +const pointer = [640, 520], moves = [], presses = []; + +async function move(x, y, ms) { + // polish.mjs draws this move with its own bend; the real pointer only has to get there on the same schedule. + const t = mono() - t0; + // --duration only holds with --human; the wait makes sure the hand takes the time it was given either way. + await ab("mouse", "move", String(Math.round(x)), String(Math.round(y)), "--human", "--duration", String(Math.round(ms)), "--steps", String(Math.max(2, Math.round(ms / 16)))); + await sleep(t + ms / 1000 - (mono() - t0)); + moves.push({ t0: t, t1: mono() - t0, from: pointer.map(r1), to: [r1(x), r1(y)], + bend: r3(uniform(0.06, 0.14) * (Math.random() < 0.5 ? -1 : 1)) }); + pointer.splice(0, 2, x, y); +} + +async function glide(sel) { + // A hand lands somewhere inside the target, rarely its centre, in one movement: farther takes longer, and it + // rests a beat before pressing. + const [bx, by, bw, bh] = await on(sel, "el.scrollIntoView({block: 'nearest'}); const r = el.getBoundingClientRect(); return [r.x, r.y, r.width, r.height];"); + const x = bx + bw * clamp(gauss(0.5, 0.13), 0.2, 0.8); + const y = by + bh * clamp(gauss(0.5, 0.1), 0.3, 0.7); + const reach = Math.hypot(x - pointer[0], y - pointer[1]); + await move(x, y, Math.min(1300, 450 + 0.9 * reach) * uniform(0.85, 1.2)); + await sleep(uniform(0.18, 0.35)); + presses.push({ at: mono() - t0, x: Math.round(x), y: Math.round(y) }); + await ab("mouse", "down"); + await sleep(uniform(0.06, 0.12)); + await ab("mouse", "up"); +} + +async function stepAside() { + // Once the hand is done for now, it drifts off the content so the cursor doesn't cover what changed. + // Sideways, toward the roomier edge: content tends to grow downward, under where the hand just was. + const x = clamp(pointer[0] + (pointer[0] < 640 ? 1 : -1) * uniform(260, 420), 40, 1240); + const y = clamp(pointer[1] + uniform(30, 90), 40, 680); + await move(x, y, uniform(700, 1100)); +} + +async function film(shots, frames, urls, titles) { + // Chrome sends a frame whenever the page repaints; each is saved with the moment it was painted. The address is + // kept too, as it changes, for a browser frame around the video. + const ws = new WebSocket((await ab("get", "cdp-url")).trim()); + await new Promise((ok, fail) => { ws.onopen = ok; ws.onerror = fail; }); + let id = 0, page, session, main; + const replies = new Map(); + const call = (method, params = {}, sessionId) => new Promise((ok) => { + replies.set(++id, ok); + ws.send(JSON.stringify({ id, method, params, ...(sessionId && { sessionId }) })); + }); + ws.onmessage = (e) => { + const m = JSON.parse(e.data), p = m.params ?? {}; + if (m.id) return replies.get(m.id)?.(m.result); + if (m.method === "Page.screencastFrame") { + ws.send(JSON.stringify({ id: ++id, method: "Page.screencastFrameAck", params: { sessionId: p.sessionId }, sessionId: session })); + const path = join(shots, `${String(frames.length).padStart(6, "0")}.jpg`); + writeFileSync(path, Buffer.from(p.data, "base64")); + frames.push([p.metadata.timestamp, path]); + } else if (m.method === "Page.frameNavigated" && p.frame.id === main?.id) urls.push([Date.now() / 1000, p.frame.url]); + else if (m.method === "Page.navigatedWithinDocument" && p.frameId === main?.id) urls.push([Date.now() / 1000, p.url]); + else if (m.method === "Target.targetInfoChanged" && p.targetInfo.targetId === page?.targetId && p.targetInfo.title !== titles.at(-1)[1]) + titles.push([Date.now() / 1000, p.targetInfo.title]); + }; + page = (await call("Target.getTargets")).targetInfos.find((t) => t.type === "page" && t.attached && !t.url.startsWith("chrome")); + session = (await call("Target.attachToTarget", { targetId: page.targetId, flatten: true })).sessionId; + main = (await call("Page.getFrameTree", {}, session)).frameTree.frame; + urls.push([0, main.url]); + titles.push([0, page.title]); + await call("Page.enable", {}, session); + // Quality 80 looks the same as 95 at 1:1 but keeps up with Chrome at about 55 fps; 95 drops to about 43, unevenly. + await call("Page.startScreencast", { format: "jpeg", quality: 80, maxWidth: 1280 * SCALE, maxHeight: 720 * SCALE }, session); + await call("Target.setDiscoverTargets", { discover: true }); // title changes arrive as target updates + return ws; +} + +const length = (text) => voiced[text] || 0.4 + text.trim().split(/\s+/).length / 2.6; // ponytail: even reading pace until the line is voiced + +const GAP = 0.6; // breath between lines +const SETTLE = 0.5; // a person looks at what a click or key did before moving on +await ab("set", "viewport", "1280", "720"); +await ab("open", url); +await ab("wait", "--load", "networkidle"); // open the take on the settled page, not a loading state +await ab("mouse", "move", "640", "520"); +const shots = join(out, "frames"), frames = [], urls = [], titles = []; +rmSync(shots, { recursive: true, force: true }); +mkdirSync(shots); +const ws = await film(shots, frames, urls, titles); +while (!frames.length) await sleep(0.05); +t0 = mono(); wall0 = Date.now() / 1000; // video time 0, on both clocks +await sleep(1.2); // let the viewer take in the page before the first word +const RECT = "if (!el) return null; const r = el.getBoundingClientRect(); return [r.x, r.y, r.right, r.bottom];"; +const pointing = (c) => c[0] === "humantype" || (c[0] === "click" && c.includes("--human")); +const lines = [], cuts = []; +let talking = 0, shot = "wide"; +const cut = (c0) => { cuts.push([c0, Date.now() / 1000]); t0 += cuts.at(-1)[1] - c0; }; +for (const [i, b] of beats.entries()) { + console.error(`beat ${i + 1}/${beats.length}`); + b.at = mono() - t0; + shot = b.shot = b.shot ?? shot; + const frame = shot === "wide" ? null : await on(shot, RECT); + let left = null; // the shot as it was when a cut left the page + for (const [j, cmd] of b.do.entries()) { + const last = !b.do.slice(j + 1).some(pointing); // a hand only steps aside once it's done for the beat + if (cmd[0] === "say") { + await sleep(talking + GAP - (mono() - t0)); + lines.push({ at: mono() - t0, say: cmd[1] }); + talking = lines.at(-1).at + length(cmd[1]); + } else if (cmd[0] === "scrollto") { + // Eased like a hand on a trackpad, farther takes longer, leaving the target a fifth of the way down the + // viewport. The browser's own smooth scroll is too quick to follow on video. + const ms = await on(cmd[1], `const y0 = scrollY, d = Math.min(el.getBoundingClientRect().top - innerHeight / 5, + document.scrollingElement.scrollHeight - innerHeight - y0), ms = Math.min(4000, 1000 + 0.8 * Math.abs(d)), t = performance.now(); + const f = now => { const p = Math.min(1, (now - t) / ms); + scrollTo({top: y0 + d * (p < 0.5 ? 4 * p ** 3 : 1 - (2 - 2 * p) ** 3 / 2), behavior: 'instant'}); if (p < 1) requestAnimationFrame(f); }; + requestAnimationFrame(f); return ms;`); + await sleep(ms / 1000 + 0.2); + } else if (cmd[0] === "cut") { // a wait nobody needs to watch, like a page loading, is cut from the take + if (cmd[1] === "open") { // leaving the page: the line finishes and the camera measures its shot first + await sleep(talking + 0.3 - (mono() - t0)); + if (shot !== "wide" && left === null) left = await on(shot, RECT); + } + const c0 = Date.now() / 1000; + await ab(...cmd.slice(1)); + cut(c0); + } else if (cmd[0] === "click" && cmd.includes("--human")) { + await glide(cmd[1]); + if (b.do[j + 1]?.[0] === "cut") { + await sleep(0.15); // the press shows, then the cut + continue; + } + await sleep(SETTLE); + if (last) await stepAside(); + } else if (cmd[0] === "humantype") { + if (!(await on(cmd[1], "return document.activeElement === el;"))) { // nobody clicks a field they're already typing in + await glide(cmd[1]); + await sleep(uniform(0.15, 0.3)); + await stepAside(); // out of the way of the text + } + let text = cmd[2]; + if (text.startsWith("$ ")) { // a value only known now, like an emailed code; the wait for it is cut from the take + const c0 = Date.now() / 1000; + text = (await promisify(exec)(text.slice(2))).stdout.trim().split("\n").at(-1).trim(); + cut(c0); + } + for (const ch of text) { + await ab("keyboard", "type", ch); + await sleep(uniform(0.07, 0.16)); + } + await sleep(SETTLE); + } else { + await ab(...cmd); + if (["click", "press", "dblclick"].includes(cmd[0])) await sleep(SETTLE); + } + } + await sleep(talking + 1 - (mono() - t0)); // let the result sit before the next beat + if (shot !== "wide") { // the camera frames the region as it was at the start and the end, so a growing list stays in shot + const seen = [frame, left ?? (await on(shot, RECT))].filter(Boolean); + if (!seen.length) die(`beat ${i + 1}: shot ${shot} matches nothing on the page`); + b.frame = [Math.min(...seen.map((r) => r[0])), Math.min(...seen.map((r) => r[1])), Math.max(...seen.map((r) => r[2])), Math.max(...seen.map((r) => r[3]))]; + } + b.said = lines.find((l) => l.at >= b.at)?.at ?? b.at; +} +await sleep(1.5); // hold the last frame so the end does not cut mid-thought +let end = Date.now() / 1000; +ws.close(); +await ab("close"); +if (!frames.length) die("Chrome sent no frames; is the demo session's page open?"); + +const uncut = (ts) => ts - cuts.reduce((s, [a, b]) => s + Math.min(b, Math.max(a, ts)) - a, 0); // a moment inside a cut lands where the cut does +// A cut keeps the last frame painted inside it, shown from where the cut lands, since a page that has settled sends no more. +const inside = new Map(); +for (const [a, b] of cuts) { + if (frames.some(([ts]) => b <= ts && ts < b + 0.2)) continue; + const k = frames.findLastIndex(([ts]) => a < ts && ts < b); + if (k >= 0) inside.set(k, b); +} +let kept = frames.flatMap(([ts, p], k) => (inside.has(k) ? [[uncut(inside.get(k)), p]] + : cuts.some(([a, b]) => a < ts && ts < b) ? [] : [[uncut(ts), p]])); +end = uncut(end); +// Each frame lasts until the next repaint; a frame painted before time 0 opens the video. +const first = kept[0][0] <= wall0 ? kept.findLastIndex(([ts]) => ts <= wall0) : 0; +const held = []; +for (let [ts, p] of kept.slice(first)) { + ts = Math.max(ts, wall0); + if (ts >= end) break; + if (held.length && ts - held.at(-1)[0] < 0.001) held.at(-1)[1] = p; // ffmpeg reads a zero duration as unset and holds that frame far too long + else held.push([ts, p]); +} +held.push([end, held.at(-1)[1]]); +// Each image needs a fine timebase: by default ffmpeg reads one at 25 fps and rounds every frame's time to 40 ms. +const name = (p) => p.split(/[\\/]/).pop(); +const fine = "option framerate 1000\n"; +writeFileSync(join(shots, "list.txt"), held.slice(0, -1).map(([a, p], k) => `file '${name(p)}'\n${fine}duration ${(held[k + 1][0] - a).toFixed(4)}\n`).join("") + + `file '${name(held.at(-1)[1])}'\n${fine}`); +const video = join(out, "video.mp4"); +// Chrome sends 3x frames as fast as it can encode them, 35 to 55 a second and unevenly; an even 30 reads smoother +// than a page that moves on frames 1, 2 and 5. polish.mjs still draws the camera and pointer at 60. "option" lines +// above need -safe 0. +const enc = spawnSync(FFMPEG, ["-v", "error", "-y", "-f", "concat", "-safe", "0", "-i", join(shots, "list.txt"), "-vf", "fps=30", + "-t", (end - held[0][0]).toFixed(3), "-c:v", "libx264", // without -t, fps holds the last frame as long as the gap before it + "-preset", "fast", "-crf", "12", "-pix_fmt", "yuv420p", video], { stdio: "inherit" }); +if (enc.status !== 0) die("ffmpeg failed to encode the take"); +rmSync(shots, { recursive: true }); +for (const x of [...beats, ...lines, ...presses]) x.at = r3(x.at); +for (const b of beats) b.said = r3(b.said); +for (const m of moves) { m.t0 = r3(m.t0); m.t1 = r3(m.t1); } +const pick = (b) => Object.fromEntries(["at", "said", "shot", "frame"].filter((k) => k in b).map((k) => [k, b[k]])); +writeFileSync(join(out, "narration.json"), JSON.stringify(lines, null, 1)); +writeFileSync(join(out, "timeline.json"), JSON.stringify({ + video, viewport: [1280, 720], pointer: [640, 520], beats: beats.map(pick), moves, presses, + urls: urls.map(([ts, u]) => ({ at: r3(Math.max(0, uncut(ts) - wall0)), url: u })), + titles: titles.map(([ts, u]) => ({ at: r3(Math.max(0, uncut(ts) - wall0)), title: u })), +}, null, 1)); +console.log(video); diff --git a/skills/record-demo/scripts/record.py b/skills/record-demo/scripts/record.py deleted file mode 100644 index cf4e0dd..0000000 --- a/skills/record-demo/scripts/record.py +++ /dev/null @@ -1,256 +0,0 @@ -"""Play beats.json in agent-browser while recording a silent video, paced for the narration laid under it later. - -usage: uv run --with websockets python record.py [--voice ] - -A ["say", "..."] action starts its line once the last one has finished, and the actions after it run while it plays, -so a click lands on the words about it. --voice takes each line's real length from a voiced lines.json (a list of -{"say", "seconds"}); without it, a line lasts as long as it takes to read aloud at an even pace. -A beat's "shot" is "wide" or a selector the camera frames; a beat without one keeps the last. The take has no pointer: -polish.py draws it, along the moves logged here, and runs the camera the shots ask for. -The take comes straight from Chrome's screencast at 3x (3840x2160), so the camera can frame a region and keep text sharp. -Writes video.mp4, narration.json (each line with the second it starts) and timeline.json (beats with their framed -region, pointer moves and presses, for polish.py). -""" -import base64, json, random, shutil, subprocess, sys, threading, time -from pathlib import Path -try: - from websockets.sync.client import connect -except ImportError: - sys.exit("needs websockets: uv run --with websockets python record.py ..., or pip install websockets") - -sys.stderr.reconfigure(errors="replace") # narration may not fit a Windows console code page - -def tool(name): - # Full path, so Windows finds npm's .cmd shims too. - return shutil.which(name) or sys.exit(f"{name} not found on PATH") - -SCALE = 3 # device pixels per CSS pixel; a real device scale, since an emulated one screencasts at 1x -AB = [tool("agent-browser"), "--session", "demo", "--args", f"--force-device-scale-factor={SCALE}"] -FFMPEG = tool("ffmpeg") -args = sys.argv[1:] -voiced = json.loads(Path(args.pop(args.index("--voice") + 1)).read_text(encoding="utf-8")) if "--voice" in args else [] -voiced = {l["say"]: l["seconds"] for l in voiced} -beats = json.loads(Path(args[0]).read_text(encoding="utf-8")) -url, out = args[1], Path(args[2]) -out.mkdir(parents=True, exist_ok=True) - -def ab(*args): - r = subprocess.run([*AB, *args], capture_output=True, text=True) - if r.returncode: - subprocess.run([*AB, "close"], capture_output=True) - sys.exit(f"agent-browser {' '.join(args)} failed: {r.stdout}{r.stderr}") - return r.stdout - -# Run JS on one element: CSS, or XPath after "xpath=". -def on(sel, body): - find = "s.startsWith('xpath=') ? document.evaluate(s.slice(6), document, null, 9, null).singleNodeValue : document.querySelector(s)" - return json.loads(ab("eval", f"(s => {{ const el = {find}; {body} }})({json.dumps(sel)})")) - -pointer, moves, presses = [640, 520], [], [] - -def move(x, y, ms): - # polish.py draws this move with its own bend; the real pointer only has to get there on the same schedule. - t = time.monotonic() - t0 - # --duration only holds with --human; the wait makes sure the hand takes the time it was given either way. - ab("mouse", "move", str(round(x)), str(round(y)), "--human", "--duration", str(round(ms)), "--steps", str(max(2, round(ms / 16)))) - time.sleep(max(0.0, t + ms / 1000 - (time.monotonic() - t0))) - moves.append({"t0": t, "t1": time.monotonic() - t0, "from": [round(c, 1) for c in pointer], "to": [round(x, 1), round(y, 1)], - "bend": round(random.uniform(0.06, 0.14) * random.choice((-1, 1)), 3)}) - pointer[:] = x, y - -def glide(sel): - # A hand lands somewhere inside the target, rarely its centre, in one movement: farther takes longer, and it - # rests a beat before pressing. - bx, by, bw, bh = on(sel, "el.scrollIntoView({block: 'nearest'}); const r = el.getBoundingClientRect(); return [r.x, r.y, r.width, r.height];") - x = bx + bw * min(0.8, max(0.2, random.gauss(0.5, 0.13))) - y = by + bh * min(0.7, max(0.3, random.gauss(0.5, 0.1))) - reach = ((x - pointer[0]) ** 2 + (y - pointer[1]) ** 2) ** 0.5 - ms = min(1300, 450 + 0.9 * reach) * random.uniform(0.85, 1.2) - move(x, y, ms) - time.sleep(random.uniform(0.18, 0.35)) - presses.append({"at": time.monotonic() - t0, "x": round(x), "y": round(y)}) - ab("mouse", "down") - time.sleep(random.uniform(0.06, 0.12)) - ab("mouse", "up") - -def step_aside(): - # Once the hand is done for now, it drifts off the content so the cursor doesn't cover what changed. - # Sideways, toward the roomier edge: content tends to grow downward, under where the hand just was. - x = min(1240, max(40, pointer[0] + (1 if pointer[0] < 640 else -1) * random.uniform(260, 420))) - y = min(680, max(40, pointer[1] + random.uniform(30, 90))) - move(x, y, random.uniform(700, 1100)) - -def film(stop, shots, frames, urls, titles): - # Chrome sends a frame whenever the page repaints; each is saved with the moment it was painted. The address is - # kept too, as it changes, for a browser frame around the video. - with connect(ab("get", "cdp-url").strip(), max_size=None) as ws: - def call(id, method, params=None, session=None): - ws.send(json.dumps({"id": id, "method": method, "params": params or {}, **({"sessionId": session} if session else {})})) - while (m := json.loads(ws.recv())).get("id") != id: - pass # events before the reply - return m["result"] - page = next(t for t in call(1, "Target.getTargets")["targetInfos"] if t["type"] == "page" and t["attached"] and not t["url"].startswith("chrome")) - session = call(2, "Target.attachToTarget", {"targetId": page["targetId"], "flatten": True})["sessionId"] - main = call(3, "Page.getFrameTree", {}, session)["frameTree"]["frame"] - urls.append((0.0, main["url"])) - titles.append((0.0, page["title"])) - call(4, "Page.enable", {}, session) - # Quality 80 looks the same as 95 at 1:1 but keeps up with Chrome at about 55 fps; 95 drops to about 43, unevenly. - call(5, "Page.startScreencast", {"format": "jpeg", "quality": 80, "maxWidth": 1280 * SCALE, "maxHeight": 720 * SCALE}, session) - call(6, "Target.setDiscoverTargets", {"discover": True}) # title changes arrive as target updates - n = 6 - while not stop.is_set(): - try: - m = json.loads(ws.recv(timeout=0.5)) - except TimeoutError: - continue - p = m.get("params", {}) - if m.get("method") == "Page.frameNavigated" and p["frame"]["id"] == main["id"]: - urls.append((time.time(), p["frame"]["url"])) - elif m.get("method") == "Page.navigatedWithinDocument" and p["frameId"] == main["id"]: - urls.append((time.time(), p["url"])) - elif m.get("method") == "Target.targetInfoChanged" and p["targetInfo"]["targetId"] == page["targetId"] and p["targetInfo"]["title"] != titles[-1][1]: - titles.append((time.time(), p["targetInfo"]["title"])) - if m.get("method") == "Page.screencastFrame": - n += 1 - ws.send(json.dumps({"id": n, "method": "Page.screencastFrameAck", "params": {"sessionId": m["params"]["sessionId"]}, "sessionId": session})) - path = shots / f"{len(frames):06d}.jpg" - path.write_bytes(base64.b64decode(m["params"]["data"])) - frames.append((m["params"]["metadata"]["timestamp"], path)) - -def length(text): - return voiced.get(text) or 0.4 + len(text.split()) / 2.6 # ponytail: even reading pace until the line is voiced - -GAP = 0.6 # breath between lines -SETTLE = 0.5 # a person looks at what a click or key did before moving on -ab("set", "viewport", "1280", "720") -ab("open", url) -ab("wait", "--load", "networkidle") # open the take on the settled page, not a loading state -ab("mouse", "move", "640", "520") -shots, frames, urls, titles, stop = out / "frames", [], [], [], threading.Event() -shutil.rmtree(shots, ignore_errors=True) -shots.mkdir() -camera = threading.Thread(target=film, args=(stop, shots, frames, urls, titles), daemon=True) -camera.start() -while not frames and camera.is_alive(): - time.sleep(0.05) -t0, wall0 = time.monotonic(), time.time() # video time 0, on both clocks -time.sleep(1.2) # let the viewer take in the page before the first word -RECT = "if (!el) return null; const r = el.getBoundingClientRect(); return [r.x, r.y, r.right, r.bottom];" -POINTING = lambda c: c[0] == "humantype" or (c[0] == "click" and "--human" in c) -lines, talking, shot = [], 0.0, "wide" -cuts = [] -for i, b in enumerate(beats): - print(f"beat {i + 1}/{len(beats)}", file=sys.stderr) - b["at"] = time.monotonic() - t0 - shot = b["shot"] = b.get("shot", shot) - frame = None if shot == "wide" else on(shot, RECT) - left = None # the shot as it was when a cut left the page - for j, cmd in enumerate(b["do"]): - last = not any(POINTING(c) for c in b["do"][j + 1:]) # a hand only steps aside once it's done for the beat - if cmd[0] == "say": - time.sleep(max(0.0, talking + GAP - (time.monotonic() - t0))) - lines.append({"at": time.monotonic() - t0, "say": cmd[1]}) - talking = lines[-1]["at"] + length(cmd[1]) - elif cmd[0] == "scrollto": - # Eased like a hand on a trackpad, farther takes longer, leaving the target a fifth of the way down the - # viewport. The browser's own smooth scroll is too quick to follow on video. - ms = on(cmd[1], """const y0 = scrollY, d = Math.min(el.getBoundingClientRect().top - innerHeight / 5, - document.scrollingElement.scrollHeight - innerHeight - y0), ms = Math.min(4000, 1000 + 0.8 * Math.abs(d)), t = performance.now(); - const f = now => { const p = Math.min(1, (now - t) / ms); - scrollTo({top: y0 + d * (p < 0.5 ? 4 * p ** 3 : 1 - (2 - 2 * p) ** 3 / 2), behavior: 'instant'}); if (p < 1) requestAnimationFrame(f); }; - requestAnimationFrame(f); return ms;""") - time.sleep(ms / 1000 + 0.2) - elif cmd[0] == "cut": # a wait nobody needs to watch, like a page loading, is cut from the take - if cmd[1] == "open": # leaving the page: the line finishes and the camera measures its shot first - time.sleep(max(0.0, talking + 0.3 - (time.monotonic() - t0))) - if shot != "wide" and left is None: - left = on(shot, RECT) - c0 = time.time() - ab(*cmd[1:]) - cuts.append((c0, time.time())) - t0 += cuts[-1][1] - c0 - elif cmd[0] == "click" and "--human" in cmd: - glide(cmd[1]) - if j + 1 < len(b["do"]) and b["do"][j + 1][0] == "cut": - time.sleep(0.15) # the press shows, then the cut - continue - time.sleep(SETTLE) - if last: - step_aside() - elif cmd[0] == "humantype": - if not on(cmd[1], "return document.activeElement === el;"): # nobody clicks a field they're already typing in - glide(cmd[1]) - time.sleep(random.uniform(0.15, 0.3)) - step_aside() # out of the way of the text - text = cmd[2] - if text.startswith("$ "): # a value only known now, like an emailed code; the wait for it is cut from the take - c0 = time.time() - text = subprocess.run(text[2:], shell=True, capture_output=True, text=True, check=True).stdout.strip().splitlines()[-1] - cuts.append((c0, time.time())) - t0 += cuts[-1][1] - c0 - for ch in text: - ab("keyboard", "type", ch) - time.sleep(random.uniform(0.07, 0.16)) - time.sleep(SETTLE) - else: - ab(*cmd) - if cmd[0] in ("click", "press", "dblclick"): - time.sleep(SETTLE) - time.sleep(max(0.0, talking + 1.0 - (time.monotonic() - t0))) # let the result sit before the next beat - if shot != "wide": # the camera frames the region as it was at the start and the end, so a growing list stays in shot - seen = [r for r in (frame, left or on(shot, RECT)) if r] or sys.exit(f"beat {i + 1}: shot {shot} matches nothing on the page") - b["frame"] = [min(r[0] for r in seen), min(r[1] for r in seen), max(r[2] for r in seen), max(r[3] for r in seen)] - b["said"] = next((l["at"] for l in lines if l["at"] >= b["at"]), b["at"]) -time.sleep(1.5) # hold the last frame so the end does not cut mid-thought -end = time.time() -stop.set() -camera.join() -ab("close") -if not frames: - sys.exit("Chrome sent no frames; is the demo session's page open?") - -def uncut(ts): - return ts - sum(min(b, max(a, ts)) - a for a, b in cuts) # a moment inside a cut lands where the cut does -# A cut keeps the last frame painted inside it, shown from where the cut lands, since a page that has settled sends no more. -inside = {max((i for i, (ts, _) in enumerate(frames) if a < ts < b), default=None): b for a, b in cuts - if not any(b <= ts < b + 0.2 for ts, _ in frames)} -frames = [(uncut(inside[i]) if i in inside else uncut(ts), p) for i, (ts, p) in enumerate(frames) - if i in inside or not any(a < ts < b for a, b in cuts)] -end = uncut(end) -# Each frame lasts until the next repaint; a frame painted before time 0 opens the video. -first = max(i for i, (ts, _) in enumerate(frames) if ts <= wall0) if frames[0][0] <= wall0 else 0 -held = [] -for ts, p in frames[first:]: - ts = max(ts, wall0) - if ts >= end: - break - if held and ts - held[-1][0] < 0.001: # ffmpeg reads a zero duration as unset and holds that frame far too long - held[-1] = (held[-1][0], p) - else: - held.append((ts, p)) -held.append((end, held[-1][1])) -# Each image needs a fine timebase: by default ffmpeg reads one at 25 fps and rounds every frame's time to 40 ms. -fine = "option framerate 1000\n" -(shots / "list.txt").write_text("".join(f"file '{a[1].name}'\n{fine}duration {b[0] - a[0]:.4f}\n" for a, b in zip(held, held[1:])) - + f"file '{held[-1][1].name}'\n{fine}", encoding="utf-8") -video = out / "video.mp4" -# Chrome sends 3x frames as fast as it can encode them, 35 to 55 a second and unevenly; an even 30 reads smoother -# than a page that moves on frames 1, 2 and 5. polish.py still draws the camera and pointer at 60. "option" lines -# above need -safe 0. -subprocess.run([FFMPEG, "-v", "error", "-y", "-f", "concat", "-safe", "0", "-i", str(shots / "list.txt"), "-vf", "fps=30", - "-t", f"{end - held[0][0]:.3f}", "-c:v", "libx264", # without -t, fps holds the last frame as long as the gap before it - "-preset", "fast", "-crf", "12", "-pix_fmt", "yuv420p", str(video)], check=True) -shutil.rmtree(shots) -for x in [*beats, *lines, *presses]: - x["at"] = round(x["at"], 3) -for b in beats: - b["said"] = round(b["said"], 3) -for m in moves: - m["t0"], m["t1"] = round(m["t0"], 3), round(m["t1"], 3) -(out / "narration.json").write_text(json.dumps(lines, indent=1), encoding="utf-8") -(out / "timeline.json").write_text(json.dumps({"video": str(video), "viewport": [1280, 720], "pointer": [640, 520], - "beats": [{k: b[k] for k in ("at", "said", "shot", "frame") if k in b} for b in beats], "moves": moves, "presses": presses, - "urls": [{"at": round(max(0.0, uncut(ts) - wall0), 3), "url": u} for ts, u in urls], - "titles": [{"at": round(max(0.0, uncut(ts) - wall0), 3), "title": u} for ts, u in titles]}, indent=1), encoding="utf-8") -print(video)