MotionForge

Depth maps and 3D human motion capture from any video — running entirely on your own machine.

Drop in a clip of one or more people moving. Get back per-subject depth maps, clean background removal, a solved 3D skeleton you can import into Blender, and the motion data behind it — as MP4 and as ProRes 4444 MOV with a real alpha channel.

Upload a video of one or more people moving, get back:

Output	What it is	Formats
Subject depth map	Per-pixel depth of only the people, background removed	MP4 · MOV (alpha)
Scene depth map	Temporally-stabilised depth for the whole frame	MP4 · MOV
Subjects, no background	The original footage with the background cut away	MP4 · MOV (alpha)
3D rigged motion capture	A solved humanoid rig rendered on transparency, from the source camera	MP4 · MOV (alpha)
3D rig — orbit camera	The same capture viewed from a camera flying around it	MP4 · MOV (alpha)
Pose overlay	2D landmarks + the solved 3D rig reprojected, for checking quality	MP4 · MOV
Motion data	.bvh per subject, plus metric 3D joints as JSON and CSV	BVH · JSON · CSV

Everything runs locally. Your video never leaves the machine — there is no account, no API key and no upload. The only network traffic is a one-off model download from Hugging Face and GitHub on first run.

---

What this is mainly for: driving video models

The primary use case is producing control videos for image-to-video and video-to-video models — Seedance 2.5, Kling, Wan, Runway, SVD and similar. You film (or already have) a reference clip, MotionForge turns it into a clean depth / matte / pose signal, and the video model follows that signal instead of improvising.

_{Left: a depth_subject pass from MotionForge, from a human reference clip. Right: the Seedance 2.5 generation conditioned on it. The character is a cat; the motion is the dancer's.  ·  full clip}

Why it saves credits

Generation credits are spent per attempt, not per good result. An unconditioned prompt gives you a lottery ticket — wrong pose, wrong timing, drifting limbs, a subject that turns the wrong way — and the usual fix is re-rolling the same prompt until something lands. Ten attempts is ten times the spend.

A depth or pose control video removes most of that variance up front. The motion, timing, silhouette and camera-relative position are already decided, so the model is only choosing appearance. In practice that turns "re-roll until the motion is right" into one generation you keep, and it makes iteration cheap in the right way: change the prompt or the character, keep the same control video, and the motion stays identical between takes.

It is also the only practical way to get specific motion. You cannot prompt "dance exactly like this" — but you can hand the model the depth map of someone doing it.

Which output to use

Output	Use it for	Notes
depth_subject	The default choice for character animation	Constrains pose, limb depth and volume while leaving the background entirely to the model. This is what the demo above uses.
depth_full	When the environment should be constrained too	Locks the set as well as the subject — good for camera moves, restrictive if you want a new background.
matte	Compositing, or as an inpaint/region mask	The MOV carries real alpha; use the MP4 if your tool wants a black-matte video.
rig	Explicit skeletal conditioning	A clean, stylised pose signal with no appearance information at all — useful where a model accepts pose conditioning, or as a reference layer for an animator.
overlay	Never ship this one	It exists to let you judge whether the solve is any good before you spend credits.

Practical notes

• Retargeting works. The reference and the generated character do not have to be the same species or proportions — the demo above drives a cat from a human. Depth constrains where mass is in space, which transfers across body shapes far better than an exact skeleton would.
• Match the aspect ratio and framing of the control video to what you want out. Most models letterbox or crop otherwise, and you lose the alignment you paid for.
• Upload the MP4, not the MOV. ProRes 4444 is for your editor; most generation platforms only accept H.264.
• Stability matters more than sharpness. A control video that flickers makes a generation that flickers. The Control video preset maximises temporal stability (see Speed) — it is slower per frame and worth it.
• Keep clips short. Most models cap at 5–10 s. Use the trim controls rather than generating a long control video you will only partly use.
• Check the overlay first. One cheap local render beats discovering a broken solve after you have spent credits on it.

---

See it work

Every panel below is produced from a single input clip in one pass. The chequerboard marks genuine transparency — those outputs carry a real alpha channel in the ProRes MOV.

Two people, moving through a scene  ·  full video

One subject, full body  ·  full video

_{Demo footage from intel-iot-devkit/sample-videos, licensed CC BY 4.0. Both clips were processed with the stock Balanced preset — no tuning per clip. Solved reprojection error was 5.9 px (single) and 6.1–11.1 px (two people).}

---

Requirements

OS	macOS (Apple Silicon, Metal-accelerated) or Linux/Windows with an NVIDIA GPU. CPU-only works but is slow.
Python	3.11 or 3.12 (MediaPipe has no 3.13+ wheels yet)
Tools	ffmpeg and uv
Disk	~700 MB of model weights, plus room for rendered output — ProRes 4444 is roughly 30 MB per 5 s at 720p

On macOS: brew install ffmpeg python@3.11 uv

---

Run it

Install it once as a login service — it starts with your Mac and restarts itself if it ever stops:

./run.sh --install

Then open http://127.0.0.1:8000. Useful companions: ./run.sh --status, ./run.sh --stop, ./run.sh --uninstall. Logs go to ~/Library/Logs/motionforge.log (they cannot live under ~/Downloads — launchd cannot open a file in a TCC-protected folder and the job fails to start).

To run it in the foreground instead, just ./run.sh. The first run downloads ~400 MB of model weights into models/ and takes a couple of minutes; later runs start immediately.

Requirements: Python 3.11 or 3.12, ffmpeg, and uv. On macOS: brew install ffmpeg python@3.11 uv.

To use a different port: PORT=8080 ./run.sh

There is also a CLI, handy for batch work:

.venv/bin/python run_cli.py myclip.mp4 --start 4 --duration 10 --out results/

---

How it works

1 · Analyse (one decode pass)

• People — YOLO26-seg finds and segments every subject, then a single-person MediaPipe BlazePose GHUM landmarker runs on each person's crop. Running the pose network at crop resolution instead of frame resolution is what makes small, distant and overlapping subjects work. BlazePose's segmentation head comes free in the same forward pass, giving a per-instance mask at crop resolution.
• Identity — detections are linked across frames by Hungarian assignment over box IoU + root-joint distance. Detection runs batched, so tracking happens here rather than in the detector's own per-frame tracker.
• Depth — Depth Anything V2 (ViT, on the Apple GPU via Metal). Raw monocular depth is only defined up to an unknown scale and shift, so a naive depth video flickers. Two corrections are applied: the normalisation percentiles are themselves low-pass filtered over time, and the previous depth map is warped into the current frame with dense optical flow and blended in wherever the warp is photometrically consistent.

2 · Solve (whole clip in hand)

Because the whole clip is analysed before anything is rendered, the solve is non-causal — smoothing introduces no lag.

• Bone calibration — per-subject median bone lengths measured over the whole take, then re-imposed on every frame so limbs stop breathing.
• Smoothing — zero-phase Savitzky-Golay followed by a 1-Euro filter, then bone lengths re-imposed.
• Absolute placement — BlazePose returns a root-relative 3D pose: it says nothing about where a person is standing. The missing translation is recovered by fitting the perspective projection of the calibrated rig to its own 2D landmarks (weak-perspective closed form, then damped Gauss-Newton with Huber weights). Human limb lengths are a strong metric prior, so this puts every subject at a plausible absolute distance and multiple people end up correctly arranged in one shared 3D space.
• Sanity bounds — a weak-perspective distance prior (z ≈ f · height_m / height_px) seeds and bounds the fit. Frames whose solve diverges — badly occluded ones, mostly — are detected and rebuilt from their neighbours instead of being smoothed into the rest of the take.
• Upright prior — the one component of a pose a single view cannot measure is lean toward or away from the camera, and it is exactly where a monocular pose net drifts: a standing person can come back tilted 80° in depth while still reprojecting perfectly. Lean beyond a threshold is rotated back about the in-image axis, which touches only the unobservable component. Real forward bends and sprint postures are inside the threshold and survive untouched; a body genuinely horizontal in the image plane is not affected at all.
• Depth agreement — each subject's distance is then nudged toward the monocular depth map, fitted per subject (the depth map is normalised per frame, so its values are only comparable along one subject's own trajectory) and only where the two signals actually correlate. Otherwise the pose-only placement is kept. The per-subject correlation is reported.
• Ground plane — the floor is fitted as a real plane y = a·x + b·z + c, not a constant height: with any camera tilt a flat floor makes distant subjects appear to hover. The slopes are ridge-regularised by how much of the scene the subjects actually cover, so a tight cluster cannot extrapolate a wild tilt.
• Quality gate — a track whose 3D pose contradicts its own 2D landmarks, or that never reprojects acceptably, is dropped with a reason rather than rendered as a broken figure. Per-subject reprojection error, 2D/3D agreement and repaired-frame counts are all reported.

3 · Render (second decode pass)

• Matte — instance masks give reliable coverage but a blocky boundary (they are decoded from a low-resolution prototype grid). The boundary is rebuilt: the masks are unioned, a selfie segmenter adds hair and soft edges gated to the detected people, an edge-aware guided filter snaps the alpha to real image edges, and a temporal EMA stops the boundary boiling.
• Rig — a software triangle rasteriser (no GL context needed). Bones become tapered cylinders, joints become spheres, the torso a slab; depth-sorted and shaded with a three-point light rig, supersampled, over transparency.
• Encoding — each output is written twice in a single pass: H.264 MP4 (alpha composited over black) and ProRes 4444 MOV, which carries a genuine 16-bit alpha channel. A small VP9 WebM is also produced so the results page can show the real transparency in the browser.

---

Speed

Measured on an M4 Max, 180 frames of 720x1280 with five outputs:

Preset	Time	Per frame	vs. first version
Fast preview	27 s	150 ms	3.0x faster
Balanced	42 s	233 ms	2.0x faster

There is also a Control video preset — maximum temporal stability and full output resolution, for conditioning a video model. It is deliberately the slowest of the four; for that job, stability is worth more than throughput.

Where the time goes now, and what was done about it:

• Both networks run batched. A single inference call on Metal costs ~35 ms of fixed dispatch overhead no matter how small the model, so detection is fed eight frames at a time rather than one. That alone was worth 1.4-2x on the detector.
• The GPU and CPU overlap. While the CPU works through one chunk (pose, mattes, cache writes) the GPU is already running the networks for the next, so the two engines no longer take turns.
• Pose runs in VIDEO mode, one landmarker per tracked person. VIDEO mode reuses the previous frame's region of interest and skips BlazePose's own detector: 37 ms -> 12 ms per person. Dedicated instances also make it safe to solve several people concurrently, which multi-person clips get for free.
• The transparent preview is produced inline. It used to be a second decode-and-re-encode pass over every finished MOV; it is now a third pipe on the same RGBA stream, which took it from 64 ms/frame to roughly zero.
• Edge refinement uses a fast guided filter. Coefficients are solved on a subsampled image and only the final combine runs at full resolution - 8x faster, and the alpha still snaps to full-resolution edges.
• X-ray style no longer copies the whole canvas per triangle. It blended a full-size supersampled copy for every one of a few thousand triangles per figure — tens of gigabytes of memcpy per frame, which made that one style hundreds of times slower than the others. Blending is now limited to each polygon's bounding box, for identical output.
• The depth scratch cache is raw, not PNG. Writing went from 18 ms/frame to 1 ms. It falls back to compressed 8-bit automatically if free disk space is tight, and says so.

If it is still slower than you want, in order of effect: switch to Fast preview, deselect outputs you do not need, lower the analysis resolution, and trim to the section you actually care about.

Formats, and which to use

• MOV (ProRes 4444) — use this for compositing. Real alpha channel, imports into Premiere, Resolve, After Effects and Final Cut with transparency intact. Large files; that is the point.
• MP4 (H.264) — use this for sharing and playback. No alpha; transparent areas are composited over black.
• BVH — drop straight onto an armature in Blender, Maya, MotionBuilder, Unreal or Unity. 21 joints, Mixamo-style names, centimetres, Y-up.
• JSON / CSV — metric 3D joint positions per frame in the OpenCV camera frame (x right, y down, z away), plus camera intrinsics and per-subject bone lengths.

Settings worth knowing

Setting	Why you would change it
Preset	Fast preview to check framing, Maximum quality for the final render.
Camera field of view	Sets absolute distance. Lower it for telephoto/zoomed shots, raise it for wide-angle or action-cam footage. Relative arrangement is unaffected.
Subject depth scaling	Scene-relative shows true distance; subject-relative rescales inside the matte so you can read limb-depth structure on the body.
Motion smoothing	Higher is calmer but flattens sharp accelerations. Lower it for fast sport.
Upright prior	How hard to damp lean toward/away from the camera. Lower it for acrobatics, diving, or anything genuinely oriented into the frame.
Matte quality	Detail adds a hair/skin model and full-resolution edge refinement — much better on close-ups, ~6× slower.
Depth stability	Temporal fusion strength. Raise it for control videos — flicker in the control video becomes flicker in the generation. Lower it only if very fast motion softens.
What counts as a subject	Whether background removal and subject depth apply to people only, people and animals, or any detected object. The 3D rig is always people-only.
Detector	YOLO26 is the default and beats YOLO11 on both speed and confidence here; the YOLO11 entries are kept for reproducibility.
Max people	Cap on simultaneous subjects.
Trim	Process a section instead of the whole clip.

Known limits

Read this before you spend generation credits on an output.

The 3D rig is human-only

The pose stage is MediaPipe BlazePose GHUM, a human-anatomy model. On anything else — animals, robots, stylised characters, people in bulky costume — it either detects nothing or fabricates a plausible-looking human skeleton that is wrong. There is no confidence signal that reliably separates the two, which is why the overlay output exists.

Depth and background removal do work on non-human subjects. Set What counts as a subject to People and animals or Any detected object and the matte and subject-depth passes will use the detector's instance masks for cats, dogs, horses and the rest of the COCO classes. Only the rig and the BVH stay people-only. This matters for the control-video use case: the demo character above is a cat, and depth is what drove it.

Depth is relative, not metric

Monocular depth has no absolute scale. Distances come from human body proportions, so an unusually tall or short subject shifts the whole scale, and a wrong field of view setting shifts it too. Values are consistent within a clip and comparable between subjects in the same clip — they are not measurements. If you need metric depth, this is the wrong tool.

Lean toward the camera is the least certain axis

The component of a pose that tilts toward or away from the camera is exactly the component a single view cannot observe, and it is where monocular pose drifts. An upright prior damps implausible lean without touching the in-image pose, but it is a prior, not a measurement — turn it down for acrobatics, diving, or anything genuinely oriented into the frame, and expect the depth axis to be softer than the other two. The orbit-camera output is the honest place to judge this, because it shows you the axis the source view hides.

Occluded limbs are predicted, not seen

If someone's legs are behind a table, the pose net still outputs leg landmarks — its best guess. Those guesses feed bone-length calibration, so the reported stature will be off too. Reported height is an estimate, not a measurement.

Limb twist is not recovered

Rotation about a bone's own axis — forearm pronation, most obviously — is not observable from joint positions. The BVH carries swing only, no twist. Hands and feet are single joints; there is no finger or toe articulation.

Shot cuts are not detected

A multi-shot edit is treated as one continuous take. Every cut starts fresh tracks, so a trailer can come back with dozens of "subjects", and the 3D placement is not comparable across cuts because each shot is a different camera. Per-shot results are fine; the file as a whole is not one capture. Cut your clip to a single shot before processing it.

Where detection starts to fail

Heavy motion blur, extreme crops, people smaller than roughly 60 px tall, severe backlighting, and crowds where subjects overlap for long stretches. Tracks that fail a 2D/3D consistency check are dropped with a stated reason rather than rendered as a broken figure, but marginal cases still get through — check the reported reprojection error per subject.

Temporal stability trades against motion fidelity

The depth stabiliser blends the previous frame forward along optical flow. Higher depth stability means less flicker and slightly more lag on fast motion; the blend is gated to regions where the warp agrees with the fresh prediction, so it should not smear, but very fast limbs at high settings can soften. Measured on static-camera footage, stability 0.6 cuts frame-to-frame flicker about 2x versus raw per-frame depth, and 0.8 about 3x.

Practical constraints

• ProRes 4444 is large — roughly 30 MB per 5 s at 720p, more at higher resolution. That is the price of a real alpha channel. Deselect outputs you do not need.
• First run downloads ~700 MB of model weights.
• Long clips need disk. The depth scratch cache falls back to compressed 8-bit when free space is tight, and says so in the warnings.
• CPU-only is slow. This is built around Apple GPU (Metal) or CUDA.
• One job at a time. The queue runs a single worker; the models are memory-heavy and would contend otherwise.

Models that were evaluated and rejected

• YOLO12 — no official segmentation weights exist (detection only), and this pipeline needs instance masks. YOLO26-seg is used instead, which is newer and benchmarked better here anyway.
• Apple Depth Pro — sharper, metric depth, but measured at 1.6 s/frame on this hardware versus 34 ms for Depth Anything V2 Base. A one-minute clip would need over an hour for depth alone. Not viable for video.
• Video Depth Anything — the right architecture for temporally consistent video depth, but it ships as a raw checkpoint with no transformers config, so adopting it means vendoring the model code. It is the most promising upgrade path if per-frame depth stability ever needs to go further than optical-flow fusion can take it.

Layout

backend/
  main.py            FastAPI: upload, jobs, SSE progress, range-served downloads
  jobs.py            single-worker queue, disk-backed, cancellable
  pipeline/
    runner.py        two-pass orchestration
    humans.py        detection, tracking, 3D pose, instance masks
    depth.py         Depth Anything V2 + temporal stabilisation
    matte.py         alpha matting and edge refinement
    skeleton.py      rig, bone calibration, root solve, BVH export
    render3d.py      software 3D rasteriser
    video_io.py      ffmpeg decode/encode, dual MP4 + ProRes 4444
    filters.py       1-Euro and Savitzky-Golay smoothing
frontend/            single page, no build step
models/              downloaded weights (gitignored)
storage/             uploads and rendered outputs (gitignored)
samples/             test clips you download yourself (gitignored)
docs/                demo stills and montages used by this README
tests/               dependency-light checks for the rig geometry

Run the geometry self-test with:

.venv/bin/python tests/test_skeleton_math.py

Contributing

Issues and pull requests are welcome. The quickest way to get oriented is backend/pipeline/runner.py, which orchestrates everything; each stage lives in its own module beside it. There are no heavyweight test fixtures — the fastest check is to run the CLI over a short clip and look at the overlay output, which is built specifically for judging whether a solve is any good.

Licences

MotionForge is released under the GNU Affero General Public License v3.0 — see LICENSE. AGPL was chosen deliberately: the default detector path uses Ultralytics YOLO26, which is itself AGPL-3.0, so this keeps the project consistent with its own dependency.

The models it downloads have their own terms — Depth Anything V2 Small/Base (Apache-2.0), Large (CC-BY-NC-4.0, non-commercial), MediaPipe BlazePose (Apache-2.0), and YOLO26/YOLO11 (AGPL-3.0 via Ultralytics — check it before shipping this commercially). Switching the detector to BlazePose multi-pose only avoids the AGPL model path entirely, at some cost in multi-person accuracy.

---

_{Built by Karan Chandra Dey}