An open-source audio stack for video: music, foley, lip-sync, timing
Five repos that cover most of the sound side of an AI video pipeline without a subscription.
Music: ACE-Step 1.5 generates full songs locally. Its README claims under 4 GB of VRAM and under 10 seconds per song on an RTX 3090. It also does covers, repaint and edit, stem separation, vocal-to-backing-track, BPM and key extraction, and LoRA training from a handful of songs. For a branded score you own end to end, that's the most complete open option we found. DiffRhythm is a lighter lyrics-to-song alternative.
Foley: MMAudio generates sound synced to an input video, and the authors say it needs about 6 GB of VRAM. Its README lists honest limits: it sometimes makes garbled speech-like sounds, struggles with unfamiliar concepts, and its training data means commercial suitability isn't guaranteed. Tencent's HunyuanVideo-Foley is another option.
Lip-sync: ByteDance's LatentSync 1.6 is an open, audio-conditioned lip-sync model trained at 512×512. Use it only with consenting performers or fictional characters.
Timing: WhisperX and faster-whisper give you word-level timestamps, which is how the animate skill and others pin captions and cuts to speech. Demucs splits a track into stems, so you can detect beats on the drums alone.
The pattern across every code-video skill we looked at: make the timeline the single source of truth, and have both picture and sound read from it.