Bidirectional communication for Deaf and hard-of-hearing customers, at the counter — not in your pocket.
Signova turns any tablet or laptop with a webcam into a live sign-to-speech and speech-to-text translator, so a Deaf customer and a hearing staff member can understand each other without waiting for an interpreter.
Businesses are already legally required — ADA in the US, the Equality Act 2010 in the UK, the EU Accessibility Act — to provide effective communication for Deaf customers. Today that usually means:
- An interpreter who's expensive, hard to schedule, or unavailable on short notice
- A Video Remote Interpreting (VRI) subscription
- Or, in practice, typing back and forth on a phone screen
There aren't enough interpreters to meet demand, and most Deaf users report being dissatisfied with the interpreting access they actually get.
Signova isn't a replacement for a qualified interpreter in complex or high-stakes settings (medical, legal). It's built for the routine, everyday exchange — a pharmacy counter, a reception desk, a DMV window — where the current alternative is often nothing at all.
Direction 1 — Deaf user communicates → hearing user hears it
There are three ways the Deaf user can produce a message, all feeding the same pipeline:
- Sign — camera detects hand landmarks in real time (MediaPipe), landmarks are classified into a word or phrase
- Tap — pick from a panel of common preset phrases for frequent exchanges
- Type — a keyboard for anything outside the trained vocabulary or preset phrases
Whichever method is used, the result is the same: text is translated into the target language if needed, then spoken aloud automatically — no button press to trigger the speech itself.
Direction 2 — Hearing user speaks → Deaf user reads it
- Live speech-to-text runs continuously
- Transcript is shown as captions on screen in real time
Both directions run on one shared device positioned at the counter — no app install or setup required on either person's own phone.
No pretrained whole-word ASL dataset exists publicly — only fingerspelling/alphabet datasets do. So the trained sign vocabulary will always be limited, and Signova is built around that reality rather than hiding it. Tap and type exist specifically to guarantee the conversation is never stuck, not as minor extras — sign detection handles the fast, common case, and the other two cover everything else. All three route through the same translate → speak → transcript pipeline, so nothing behaves differently depending on which method was used.
At the end of a conversation, the full transcript is available instantly via an on-screen QR code — the transcript is encoded directly into the code, so scanning it works with no login, no server, and no backend involved.
The app is complete and running end-to-end, currently using MediaPipe's official pretrained gesture model — 7 real hand gestures detected live out of the box, including ILoveYou, which is an actual ASL sign. This is the built-in default, not a placeholder: the app is designed to always run fully on this model, with a custom-trained vocabulary as an enhancement layered on top when ready, not a dependency the app waits on.
- Real-time, on-device hand-landmark tracking — fast, and far more robust to lighting/background than raw-image classification approaches
- Automatic text-to-speech the instant a sign is detected
- Live speech-to-text captioning for the hearing side of the conversation
- Multi-language translation
- Common-phrase and typed-text fallback input, so the app is never limited to only its trained gestures
- Adjustable text size and high-contrast design, built as accessibility defaults, not optional settings
- QR-code transcript hand-off at the end of a conversation, no account or app install needed
- Community contribute flow — anyone can submit a video sample of a new sign for future review, so the vocabulary can grow with real-world use
- Not a certified interpreter replacement. No claim of legal compliance out of the box, and not intended for complex, medical, or legal communication.
- Not continuous, grammatical sign language translation. Current sign recognition works on isolated signs/phrases, not fluent natural signing with full ASL grammar — that remains an open research problem industry-wide.
- Not a personal/mobile app (yet). Designed for a single shared device at a fixed location, matching how businesses actually deploy accessibility tools today. A synced multi-device version is on the roadmap.
| Layer | Tool |
|---|---|
| Frontend | Next.js / React |
| Hand detection & gesture classification | MediaPipe Tasks Vision (Gesture Recognizer) |
| Custom sign training (in progress) | MediaPipe Model Maker |
| Translation | Google Translate API |
| Text-to-speech | Web speechSynthesis API |
| Speech-to-text | Web SpeechRecognition API (Chrome/Edge recommended) |
| QR code | Client-side generation, transcript encoded directly — no backend |
| State | Client-side React state only — no database, no auth, by design |
Signova isn't competing for enterprise contracts; it targets the much larger set of everyday small businesses that will never sign one — a lightweight, self-serve alternative to enterprise VRS/VRI infrastructure.
- Expanded custom vocabulary via the training and contribute pipelines
- Accuracy validation with certified interpreters and Deaf user testing
- Two-device sync (each participant's own device, connected in real time)
- Camera-based glasses support for the hearing-side user, once third-party developer platforms open further
git clone https://github.com/<org>/signova.git
cd signova
npm install
npm run devRequires a modern Chromium-based browser (Chrome/Edge) for full speech recognition support.
PRD.md— full product spec, UI/UX detail, tech stack, folder structurePhase.md— detailed phase-by-phase build breakdownInfo.md— running build log, updated after each phase
MIT