Skip to content

About

經典迷因即時復現,動態手勢辨識手風琴 | Real-time hand-gesture accordion synthesizer

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

PalmMotion — Gesture-Controlled Accordion

PalmMotion demo preview

Recreate a classic speech meme in real time • Generate animated accordion bellows from hand gestures • Play iconic melodies automatically • Preview in a minimalist dark web studio • Export comparison videos in multiple layouts

Python MediaPipe Web Audio FFmpeg


繁體中文 · English

繁體中文

特色功能

  • 二次貝茲扇形風箱與雙手腕獨立傾角: 風箱會依雙手距離自然展開或收合,左右琴身則各自跟隨對應手腕旋轉。
  • 名曲自動跟彈: 內建《Bella Ciao》、《玫瑰人生》與《俄羅斯方塊》(Korobeiniki),依手部移動距離和推拉轉折推進旋律。
  • 極簡深色 Web Studio: 位於 web/,提供影片時間軸、快捷操作、靜音和匯出功能,無需 Python 後端。
  • 多版型影片與 GIF 匯出: 支援單畫面、16:9 左右對比、9:16 直式短影音,以及調色盤最佳化 GIF。

快速開始

需要 Python 3.10 以上版本(已使用 Python 3.12 測試):

pip install -r requirements.txt
python main.py

執行 main.py 開啟桌面控制台,可選擇 Webcam 或影片檔、切換曲目、預覽效果,並匯出影片或 GIF。

Web 版

Web 版是位於 web/ 的靜態網頁,不需要 Python 後端。可用以下命令在本機預覽:

python -m http.server 8000

接著開啟 http://localhost:8000/web/。此網站也可部署至 GitHub Pages;即時手勢追蹤需要相機權限。

預覽快捷鍵

按鍵 功能
Q 或 ESC 離開預覽
SPACE 暫停或繼續播放
N 切換下一首曲目:Bella Ciao → 玫瑰人生 → 俄羅斯方塊
P 切換音高模式:名曲 → 和弦 → 滑音 → 固定
C 切換貝茲彎曲風箱或直線風箱
M 靜音或恢復手風琴音效
R 重置追蹤濾波器與狀態

命令列

# 使用預設 Webcam 即時互動
python main.py --source 0

# 播放影片並指定曲目
python main.py --source speech.mp4 --song bella_ciao

# 匯出 16:9 左右對比影片
python main.py --source speech.mp4 --export compare.mp4 --layout side_by_side

# 匯出 9:16 直式影片
python main.py --source speech.mp4 --export reels.mp4 --layout vertical_9_16

# 同時輸出 MP4 與 GIF
python main.py --source speech.mp4 --export demo.mp4 --gif assets/demo.gif

授權

本專案採用 MIT License,歡迎自由 Fork、修改與分享。


English

Key Features

  1. Quadratic Bézier fan bellows and independent wrist angles
    • The bellows bend and fan out naturally along a quadratic Bézier curve as the distance between your hands changes.
    • Each side of the accordion follows the rotation of its corresponding wrist for a more convincing 3D effect.
  2. Automatic melody mode
    • Includes three classic accordion melodies: “Bella Ciao,” “La Vie en Rose,” and “Korobeiniki” (the Tetris theme).
    • Hand movement advances the melody based on travel distance and push-pull transitions, so you can play by moving in front of the camera.
  3. Minimalist dark web studio (web/)
    • An obsidian-black screening room with champagne-gold accents and ambient lighting.
    • Includes a video timeline, keyboard controls, live mute, and one-click export.
  4. Video and GIF export for social media
    • Solo: A clean composited accordion view.
    • Side by side (16:9): Original footage next to the effect.
    • Vertical (9:16): A split-screen layout for Instagram Reels, YouTube Shorts, and TikTok.
    • GIF: Palette-optimized animations for GitHub and social platforms.

Architecture

flowchart TD
    subgraph Input["Inputs"]
        Cam["Live webcam"]
        Vid["Speech video (.mp4)"]
        VidAudio["Extract original audio with FFmpeg"]
    end

    subgraph Core["Tracking and analysis (tracker.py)"]
        MP["MediaPipe Hands + Pose occlusion recovery"]
        EMA["EMA temporal filtering: hand positions, wrist angles, and speed"]
    end

    subgraph Audio["Audio synthesis and mixing (audio_engine.py, melodies.py)"]
        MelodyFSM["Three-melody state machine"]
        Synth["Musette dual-reed synthesizer: melody and chords"]
        LiveMixer["Live mixer: accordion and original speech"]
    end

    subgraph Visual["Rendering (renderer.py)"]
        Bezier["Quadratic Bézier fan bellows and independent wrist rotation"]
        LayoutComp["Solo, side-by-side, and vertical layouts"]
    end

    subgraph Output["Output"]
        Player["OpenCV live preview"]
        Exporter["MP4 and GIF exporter"]
        WebApp["Dark web studio (web/index.html)"]
    end

    Cam --> MP
    Vid --> MP
    Vid --> VidAudio
    MP --> EMA
    EMA --> MelodyFSM --> Synth
    VidAudio --> LiveMixer
    Synth --> LiveMixer
    EMA --> Bezier --> LayoutComp
    LiveMixer --> Exporter
    LayoutComp --> Player
    LayoutComp --> Exporter
    EMA -.-> WebApp
Loading

Quick Start

Install dependencies

PalmMotion requires Python 3.10 or later. It has been tested with Python 3.12.

pip install -r requirements.txt

Launch the desktop app

Run main.py to open the graphical control panel:

python main.py

From the control panel, you can:

  • Choose a webcam or a video file as the input.
  • Select a melody: “Bella Ciao,” “La Vie en Rose,” or “Korobeiniki.”
  • Start the live preview, with original video audio playback and mute controls.
  • Export a solo, 16:9 side-by-side, or 9:16 vertical video, or create a GIF.

Web App

The static web app lives in web/ and does not require a Python backend.

To preview it locally:

python -m http.server 8000

Then open http://localhost:8000/web/ in your browser. The site can also be hosted on GitHub Pages. Camera access is required for live gesture tracking.


Preview Hotkeys

When the live preview window is open:

Key Action
Q or ESC Exit the preview
SPACE Pause or resume playback
N Select the next melody: Bella Ciao → La Vie en Rose → Korobeiniki
P Switch pitch mode: melody → chords → continuous → fixed
C Switch between curved Bézier bellows and straight bellows
M Mute or restore the accordion audio
R Reset tracking filters and state

Command-Line Usage

Start live interaction with the default webcam:

python main.py --source 0

Play a speech video with a selected melody:

python main.py --source speech.mp4 --song bella_ciao

Export a 16:9 side-by-side video:

python main.py --source speech.mp4 --export compare.mp4 --layout side_by_side

Export a 9:16 vertical video:

python main.py --source speech.mp4 --export reels.mp4 --layout vertical_9_16

Export an MP4 and a preview GIF:

python main.py --source speech.mp4 --export demo.mp4 --gif assets/demo.gif

Project Structure

PalmMotion/
├── main.py                  # Application entry point: GUI and CLI
├── requirements.txt         # Python dependencies
├── assets/                  # Accordion artwork, WAV samples, and demo GIFs
├── src/
│   ├── config.py            # Shared configuration
│   ├── melodies.py          # Melody notes and chord progressions
│   ├── tracker.py           # MediaPipe hand/pose tracking and filtering
│   ├── renderer.py          # Bellows rendering and video layouts
│   ├── audio_engine.py      # Reed synthesis and audio mixing
│   ├── exporter.py          # MP4 and GIF export
│   ├── player.py            # OpenCV audio/video preview
│   ├── gui.py               # Tkinter desktop interface
│   └── assets_generator.py  # Accordion artwork and reference audio generation
├── web/                     # Static web studio for GitHub Pages
│   ├── index.html           # Web interface
│   ├── style.css            # Web styles
│   └── app.js               # MediaPipe, Web Audio, and Canvas rendering
└── tests/
    └── test_pipeline.py     # Pipeline and layout export tests

License

This project is licensed under the MIT License. Feel free to fork, modify, and share your work.

About

經典迷因即時復現,動態手勢辨識手風琴 | Real-time hand-gesture accordion synthesizer

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Contributors

Languages