LocalVocal: Local Live Captions & Translation On-the-Go v0.3.5

LocalVocal plugin allows you to transcribe & translate speech into text locally on your machine in real time. No GPU required*, no cloud costs, no network and minimal lag! Privacy first - all data stays on your machine.
(* GPU acceleration via CUDA or AMD is supported!)

If this plugin has been valuable to you consider adding a to the GH repo, rating it here on OBS, subscribing to my YouTube channel, and supporting my work: GitHub, Patreon or OpenCollective. Check out the Home for Open Source Content Creators AI.
Need help setting up? Contact live support https://discord.gg/J5RgzMmPqM


Do more with LocalVocal:

The plugin adds an Audio Filter - use it on a speech source (mic, video) to get a transcription. Send the captions to a Text Source to show on scene.

Current Features:

• Transcribe audio to text in real time in 100 languages
• Display captions on screen using text sources
• Send captions to a .txt or .srt file (to read by external sources or video playback) with and without aggregation option
• Sync'ed captions with OBS recording timestamps
• Send captions on a RTMP stream to e.g. YouTube, Twitch
• Bring your own Whisper model (any GGML)
• Translate captions in real time to major languages (both Whisper built-in translation as well as NMT models with CTranslate2)
• CUDA, OpenCL, Apple Arm64, AVX & SSE acceleration support

Roadmap:

• More robust built-in translation options
• Additional output options: .vtt, .ssa, .sub, etc.
• Speaker diarization (detecting speakers in a multi-person audio stream)

Internally the plugin is running a neural network (OpenAI Whisper) locally to predict in real time the speech and provide captions.

It's using the Whisper.cpp project from ggerganov to run the Whisper network in a very efficient way on CPUs and GPUs. For translation it's using CTranslate2 and the M2M100 model.

If you use this plugin - let us know! We would love to feature your work/vids and showcase your success.

Check out our other plugins:

• Background Removal removes background from webcam without a green screen.
• Detect will detect and track >80 types of objects in real-time inside OBS
• URL/API Source that allows fetching live data from an API and displaying it in OBS.

If you are a broadcasting company or service looking to integrate local AI technology into your pipelines - reach out to inquire about our enterprise services.