Audio, voice & music
The sound side of AI video: text-to-speech and voice, music generation, foley, stem separation and word-level timing for captions and cuts. Most of it can run locally. Check each model's licence before commercial use.
Lab posts (14)
- Spotted, not yet tested#motion graphics#agent skills#launch video#sound design#open sourcehuashu-motion: an open skill that writes the launch film, scores it, and lets you point at the frame to fix it
Anthropic's Claude Motion is still a Team and Enterprise beta. Two days after 花叔 open-sourced his own take, it had more than 1,100 stars: a skill that pitches three directions, builds the film as one HTML page, writes a score that real sampled instruments play, and opens an editor where a click on the frame becomes a precise fix.
Source: 花叔 Huashu (@AlchainHust)
- Spotted, not yet tested#editing#dubbing#subtitles#MCP#agent skillsBaoCut: an AI video agent where every caption, dub and cut stays editable
Most AI video tools render once and hand you a file. BaoCut makes the timeline the source of truth: transcripts, translations, subtitle layers, voice-overs and code-made motion graphics all live in one editable video that you and your agent edit through the same undo stack.
Source: Jim Liu 宝玉 (@dotey)
- Based on the creator’s video · not yet tested by us#editing#workflow#B-roll#talking head#self-built toolsNo Final Cut at all: Ben Vallack had Claude Code build his edit suite, then directed it with voice notes
Instead of adopting someone else's AI editor, a YouTuber who has edited in Final Cut Pro for years asked Claude Code to build the few tools his own process needs: a transcript cutter, a searchable B-roll library, a phone app for fine-tuning cuts, and a player where he leaves spoken notes on the timeline.
Source: Ben Vallack
- Spotted, not yet tested#voiceover#transcription#captions#TTS#pricingVoxtral 3 for video people: cheaper captions that know your product names, and narration with emotion tags
Mistral's Voxtral 3 is pitched at voice agents, but two parts matter for editors: transcription you can bias with product and speaker names, and $13-per-million-character TTS with emotion and pace tags.
Source: Mistral AI (@MistralAI)
- Based on the creator’s video · not yet tested by us#editing#MCP#self-built tools#review loop#open sourceTandem: an MIT video editor with a CLI and MCP, where agent edits stay purple until you've watched them
Mike Cann had Opus 5.5 build him a native Mac editor overnight to replace Filmora. It's now on GitHub under MIT, with a CLI, an MCP server, a Claude skill, and a review gate that shows exactly what the agent changed.
Source: Mike Cann
- Spotted, not yet tested#audio#TTS#voice#dubbing#MCP#local#open sourceVoiceStudio: local voice cloning, dubbing and an MCP server for your video agent
A desktop app (AGPL-3.0) for cloning, voice design, timed video dubbing and transcription in 646 languages. It runs on NVIDIA, Apple Silicon or CPU, and serves an MCP server so Claude Code can voice your renders. Check the model licences first.
Source: debpalash
- Spotted, not yet tested#sound design#self-review#MCP#agent skills#open sourceearshot gives coding agents ears: hear the soundtrack, score the sync, forge SFX in code
Agents can now render a whole launch film but can't hear it. earshot is a CPU-only CLI, Python API and MCP server that turns sound into evidence: what a file sounds like, where the hits land against the cuts, and a scorecard with timestamped fixes.
Source: Leon Lin (@LexnLin)
- Based on the creator’s video · not yet tested by us#audio#TTS#voice#ComfyUI#lip-sync#MiniMax H3Fish Audio Drama 3 in ComfyUI: directed multi-speaker voice, then H3 talking video
35B open-weight TTS with inline cues ([sighs], [whispers]) and up to 16 speakers, running in ComfyUI from 8 GB VRAM by keeping MoE experts in system RAM. Then hand the take to MiniMax H3 for the picture.
Source: Benji's AI Playground
- Spotted, not yet tested#HyperFrames#agent skills#audio sync#storyboardsHyperFrames this week: storyboard boards, an audio command, beats without Chrome
Five releases in about a day, and they all push agents toward the same habits the best creators use: stills before motion, and cuts measured against real audio.
Source: HeyGen
- Spotted, not yet tested#3D#Blender#commercial#creators#pipelineKevin Ngo's 'Apple HomeView' spec ad: web pages mapped onto 3D glass
An 82-second fan-made commercial for a product that doesn't exist, where Claude Opus 5.5 orchestrated Blender, HTML screens, Midjourney plates and code-composed music.
Source: Kevin Ngo
- Spotted, not yet tested#explainer#Remotion#agent skills#multi-agentanything2explainer: a topic in, a sourced, narrated explainer out (non-commercial licence)
A Claude Code / Codex skill that researches a topic with sources, writes and voices the narration, then sends parallel agents to draw every shot in Remotion. Check the licence before any client work.
Source: Vincent Wei
- Spotted, not yet tested#music video#audio sync#pipeline#Claude Code#RemotionTwo open recipes for AI music videos: Nate McGuire's full credits list and the song-video-director skill
One is a generative-footage pipeline posted credit by credit. The other is a Claude Code skill for music videos rendered in code. Both treat the song as data before any picture gets made.
Source: nate (@natemcguire)
- Spotted, not yet tested#motion graphics#self-review#sound designclaude-motion teaches an agent taste, then makes it check its own frames
Motion rules with actual numbers, a contact-sheet review loop, and a sound engine written in plain Python.
Source: @whaleyxbt
- Spotted, not yet tested#music#audio sync#open sourceAn open-source audio stack for video: music, foley, lip-sync, timing
Five repos that cover most of the sound side of an AI video pipeline without a subscription.
Source: ACE Studio & StepFun
Skills, repos & workflows in the directory (25)
- Retake MCP
Demo-as-code: your agent writes a YAML walkthrough, dry-runs every selector, records a real browser and returns an MP4 with burned-in captions, scene stills and a proof log; optional Edge/Kokoro voiceover. Rerun the file when the UI changes. npm retake-demos. MIT.
- earshot MCP
Ears for coding agents (CPU-only CLI, Python API, MCP): describe/match/find sounds, a listening timeline with beat grid and sound-to-cut sync, an annotated sheet PNG, a launch-film critique with timestamped fixes, and forge/sfx/kit for code-made SFX (201-sound bank). MIT.
- claude-motion-explainer Skill
One sentence to a narrated explainer animation on HyperFrames: research, storyboard, TTS, one sub-agent per scene in parallel, captions, three-colour theme, contact-sheet self-review; includes 26 lessons and a fix_voice.py for mispronounced words. Traditional Chinese. MIT.
- whiteboard-teacher-video Skill
Turns a lecture script into a hand-drawn whiteboard explainer: rough.js strokes drawn in code, text written on as it is spoken, five styles, local Chrome render, edge-tts voice, no image-API cost; re-renders only changed segments. Chinese docs; licence not stated.
- claude-codes-influencers Repo
Talking-mascot pipeline: script → voice (macOS say, ElevenLabs, Voicebox or mic) → Rhubarb 9-mouth-shape lip sync + Whisper word timings → poses, blinks, captions → 1080×1920 MP4. Fork and add your own mascot art. MIT.
- VoiceStudio MCP
Local ElevenLabs-style desktop app: voice cloning, voice design, timed video dubbing, transcription (646 languages) on NVIDIA, Apple Silicon or CPU, with an MCP server at /mcp. App AGPL-3.0; default OmniVoice weights are CC-BY-NC.
- BashCut MCP
Native macOS (AVFoundation) video editor with a docked Claude Code/Codex terminal; everything the UI does is available through the bashcut CLI or MCP as undoable operations. Plugins for Whisper captions and silence markers. MIT.
- papercut-explainer Skill
Nine Agent Skills + a Remotion library for vertical cut-paper explainer reels: paper-cut kit, 24 backdrops, 17 camera moves, 10 transitions, 79 SFX and 8 music beds; strategy → script → storyboard → voice → build → QA. MIT.
- YuE2 Studio MCP
Local song generator on YuE2 (via yue2.cpp): full songs with vocals plus an editable sheet-music score you can re-render. Windows app (6 GB+ NVIDIA), MCP server at 127.0.0.1:8791/mcp for agents. MIT.
- ComfyUI-DBS ComfyUI
Split a soundtrack into music, SFX and speech stems in ComfyUI with the official Dynamic Band Splitting models (tiny, CPU or CUDA). Remove dialogue, pull it out for dubbing, or rebalance a mix. Unofficial integration, MIT.
- ComfyUI-Drama3 ComfyUI
Fish Audio Drama 3 (35B TTS) natively in ComfyUI from 8 GB VRAM via experts-in-RAM: voice cloning, up to 16 speakers, inline cues, Qwen3-ASR, saved voices. Code GPL-3.0; weights under Fish Audio's licence.
- drama3-h3-talking ComfyUI
One ComfyUI workflow: text → Drama 3 voice → MiniMax H3 Ref2VA lip-synced talking video (take anchored at frame 0).
- HyperFrames skills Skill
HTML → deterministic MP4; /hyperframes router picks launch, explainer, PR-to-video, recut or music workflows.
- heditor Skill
Talking head → styled 9:16 Reel on HyperFrames: named styles, word-timed cut, voice-first mix, vision QA. MIT.
- animate Skill
Canvas + Playwright + FFmpeg animation in 7 styles; storyboard gates; Whisper word timing.
- ElevenLabs MCP MCP
Official MCP for TTS, voices and sound effects.
- song-video-director Skill
Claude Code skill for code-rendered music videos (Remotion, p5.js, three.js): BPM/beat grid + Whisper word times, director rules, contact-sheet review. MIT.
- ACE-Step 1.5 Repo
Local song generation, covers, stems, LoRA from a few songs, BPM/key extraction.
- DiffRhythm Repo
Lyrics → full-length song, fast.
- MMAudio Repo
Video-to-audio synced foley (~6 GB VRAM). Check licence for commercial use.
- HunyuanVideo-Foley Repo
Foley generation from video + text.
- Demucs Repo
Stem separation, used for beat detection on drums and clean vocal timing.
- WhisperX Repo
Word-level timestamps + diarization for captions and cut timing.
- faster-whisper Repo
Fast local transcription; used by the animate skill for word timing.
- SlopMonster Skill
Lint scripts for AI tells before voice-over.
Pipelines (6)
- Code-only animated explainer: JS canvas → Playwright → FFmpeg · Based on the creator's video · not yet tested by us
- HyperFrames product launch video from a URL · Spotted, not yet tested
- HyperFrames: talking-head footage → captioned, designed recut · Spotted, not yet tested
- Higgsfield MCP + code motion: two-layer cinematic launch · Based on the creator's video · not yet tested by us
- Music video: own the track, cut on the beat · Spotted, not yet tested
- Own-footage character replacement (consent-based) · Based on the creator's video · not yet tested by us