Comfy's lip-sync dubbing workflow redraws only the mouth: one shot into any language with ElevenLabs and MiniMax H3; plus Genie 4 and HeyGen Decision 1 in ComfyUI
Comfy's Doug Hogan published a single ComfyUI workflow that dubs an existing shot into another language in the actor's own cloned voice, then has MiniMax H3 redraw only the mouth so the lips say the new words. Per Comfy, about 1% of the frame changes. The same day, Genie 4 and HeyGen's Decision 1 arrived in ComfyUI.
The dubbing workflow (video Nov 19, 5:26 PM MST; @ComfyUI post Nov 20, 2:28 PM MST). You load a shot and type a language in plain words, like French or Japanese. ElevenLabs Partner Nodes separate the voice from music and background noise, clone it from the clean take and transcribe the line with a timestamp for every word. An LLM translates the line with one job: fit the original timing, counting syllables and keeping the actor's pauses. ElevenLabs then speaks the translation in the cloned voice. The workflow builds a mask from face landmarks that covers only the lips, jaw and cheeks, MiniMax H3 redraws what's inside that mask locked to the new audio, and the new mouth is pasted back onto the original frames through a soft edge. There are nine numbered groups with notes, and app mode turns it into a form with a shot, a language and a run button.
Why the mask matters. Per Hogan, his first try just gave H3's restyle workflow the new audio, and the mouth kept performing the old line. Treating it like a paint job fixed that. He shows a difference map that is black everywhere except the mouth, so the rest of the frame is the original plate. For a compositor, that's the selling point: grade, grain and background stay untouched.
Limits, per Comfy. It wants 24 fps, one speaker with their face in view, and up to about 15 seconds per render. The output carries only the dub, so you need to lay room tone and music back in yourself. The dub can end about half a second early, and the lips close with it. Hogan says each new language took about 10 to 12 minutes on Comfy Cloud. ElevenLabs and any hosted H3 runs are paid Partner Node calls. Only clone and re-voice performers who have agreed to it, and check your talent contracts before you localize a client spot.
Genie 4 in ComfyUI (Nov 20, 11:51 AM MST). Google DeepMind's world model, which we covered on Nov 19, now runs through Comfy. It generates the world frame by frame and reacts to your controls, with native audio, physics like collisions and aerodynamics, and text-prompt edits while the session runs. Comfy streamed a live 'model jam' with it via Comfy Router this afternoon.
HeyGen Decision 1 in Partner Nodes (Nov 20, 11:05 AM MST). This isn't a generator. It's a decision model: give it video, audio, images and text, ask named questions, and it returns yes/no probabilities, a pick from fixed options or a score. HeyGen's examples include whether a product is shown correctly, whether a scene matches a reference and whether the voiceover matches the screen. Per HeyGen's API docs, it takes up to 10 videos of 60 seconds or less per request and hears each video's soundtrack. HeyGen says it answers in about 100 ms. In a pipeline, that's an automated QA gate between generation and review. Third-party write-ups disagree about audio support on some resellers, so check the route you use.
Signals: official Comfy and HeyGen posts only (the dubbing post had about 58 likes and 2.5k views in two hours). None of it reached a tracked curator or GitHub Trending, so it isn't HOT by our rule.
Related tools on ReelStack
Skills & repos in our directory
- AICG3D Short-Drama Workbench · ComfyUI, Generation
- OneShot MV · ComfyUI, Lip-sync & timing
- MOTNGRFX · ComfyUI, Motion graphics
- Kitty Open Studio · MCP, Generation
- ComfyUI-Drama3 · ComfyUI, Audio & music
- drama3-h3-talking · ComfyUI, Lip-sync & timing
More from the Lab
- ElevenLabs SFX v3 and VEED Dubbing in ComfyUI: two audio upgrades
- Fish Audio Drama 3 in ComfyUI: directed multi-speaker voice, then H3 talking video
- HeyGen Avatar 6 lands at #1 on Avatar Arena and in ComfyUI the same day, plus HyperFrames 0.9.64 and a Miracle update
- Markerboard draws whiteboard explainers to your voiceover, plus a Blender face-capture stack, ComfyUI blockout tips and Kevin Ngo's JavaScript drawing prompt
Topics: ComfyUI · Audio, voice & music · Open-weight video & audio models