Skip to content

Creative Directions

Dubbing and Audio Latest

Advances in AI speech: tools and methods for dubbing, voice cloning, multi-emotion reading, and audio adaptation of short dramas.

60 picksLast 30 days 33 itemsCollected 269 items

updated

Pick archive · page 3

Oct 14Wed41–60
  1. ComfyUI Wiki · 工作流与开源创作分享66

    ComfyUI-Drama3 turns Fish Audio Drama 3 into a ComfyUI node, running a 35B speech model on 8GB of VRAM

    ComfyUI-Drama3 packages Fish Audio's 35B mixture-of-experts speech model Drama 3 with Qwen3-ASR into a ComfyUI custom node with no extra dependencies to install; expert weights are kept in system memory and streamed through a VRAM cache, so it runs on 8GB of VRAM.

    Why it matters: Squeezing a 35B speech model onto an 8GB VRAM card, with four quantization versions and frame-by-frame benchmark data, so you can judge whether local output is feasible.

  2. Fish Speech 官方发布72

    Fish Audio releases the open-source speech generation model Drama 3 pre-release version

    Fish Audio releases the open-source speech generation model Drama 3 pre-release version, which can convert text into performed speech. The model uses a 35B-A3B hybrid linear attention MoE backbone (initialized from Qwen3.5) and a new codebook-MTP audio decoder; pretraining covers about 80 languages and dialects and hundreds of billions of tokens, and it supports multi-reference voice cloning, natural language instructions, multi-round revisions, multi-character scenes, speech infill editing, and long-audio and streaming synthesis.

    Why it matters: Open-source speech models put multi-character scenes, multi-round script revisions, and voice editing into the same model, allowing dubbing workflows to assess replacement potential accordingly.

Oct 13Tue
  1. ComfyUI Wiki · 工作流与开源创作分享78

    Fish Audio releases Drama 3 speech synthesis model, supporting 16 characters and natural-language performance instructions

    Fish Audio released its next-generation speech synthesis model Drama 3 on October 13, 2026, an open-weight 35B MoE model activating 3B per token.

    Why it matters: The 35B open-source speech model supports same-track generation of 16 characters and natural-language performance instructions; its pronunciation robustness data can be used to assess dubbing and audiobook workflows.

Oct 11Sun
  1. z-ab · 开源短剧与创作项目分享71

    OpenBMB releases VoxCPM2: a tokenizer-free TTS model supporting 30 languages and voice design

    OpenBMB released VoxCPM2, which removes the discrete token quantization step of mainstream TTS and instead uses a diffusion autoregressive architecture to generate speech in a continuous audio latent space; the repository has 38.7k Stars, an Apache-2.0 license, and fully open weights and code for commercial use.

    Why it matters: VoxCPM2 removes discrete token quantization and instead generates speech in a continuous latent space, and offers multiple deployment paths from local to production.

  2. Kapwing 创作指南62

    How to Make Vox-Style Collage Animation with AI: A Step-by-Step Guide

    Kapwing published a step-by-step guide on using AI to generate Vox-style collage animation documentaries, broken into six steps: writing the script, breaking down visual beats, generating collage images, frame-by-frame animation, generating voiceover, and editing the final cut.

    Why it matters: A complete breakdown of the six-step AI workflow for Vox-style collage documentaries, including reusable image and animation prompts, ideal for creators looking to replicate the visual style.

Oct 10Sat
  1. note · ryosuke_blog_ · 图像视频创作分享65

    Complete Workflow for Using Claude to Operate Seedance 2.5 for Video Generation

    Author Akita Ryosuke published his four-step workflow for using Claude desktop with Claude in Chrome to operate Seedance 2.5 for video generation: first, use ChatGPT to create a character sheet and prepare props and sound materials; then write out clear details for characters, scenes, actions, camera, dialogue pronunciation, negative prompts, aspect ratio, quality, and duration; next, let Claude automatically operate the Yapper platform to generate; and finally, provide feedback for revisions on each item.

    Why it matters: The author published a complete four-step workflow for using Claude to operate Seedance 2.5 for video output, including character sheets, lip-sync details, and cost breakdowns — transferable to your own generation workflow.

  2. GitHub:AI视频开源项目发现71

    Sitor AI open-sources Frametell: generate a 1080p explainer video from one sentence or one document

    Sitor AI has open-sourced Frametell (帧言) on GitHub, which can turn a sentence, document, PDF, slide deck, or image into a 1080p explainer video with subtitles. In the workflow, a director agent first produces a storyboard for confirmation, then writes narration, dubs it with ElevenLabs (Volcano Engine TTS for Chinese), draws page by page and self-checks, and includes five built-in styles: warm-story, study, chalkboard, whiteboard, and tech-dark.

    Why it matters: An open-source, locally run explainer-video generation tool that first produces a storyboard for confirmation and then dubs and renders, so creators can use it to assess whether to add it to their production workflow.

Oct 9Fri
  1. Synthesia · 官方博客62

    Synthesia Launches Syren: Generate Videos and Motion Graphics from a Single Prompt

    Synthesia released Syren, an AI video creation tool, in beta. By entering a prompt, a preset, or existing brand documents, images, videos, PPTs, or YouTube links, users can generate a finished video with motion graphics, animations, digital avatars, voiceover, music, and b-roll, and can make changes through conversation.

    Why it matters: Synthesia turns video generation into a conversational agent; compare how it and the existing editor divide work in scenarios requiring precise content.

Oct 5Mon
  1. GitHub:AI视频开源项目发现82

    Kandinsky 6.0 open-sourced: synchronized audio-video generation models, in Lite and Pro tiers

    kandinskylab released the Kandinsky 6.0 Video series of diffusion models, including a 3B-parameter Lite and a 29B-parameter Pro, capable of generating 5-second videos with 44 kHz synchronized audio (including lip-sync), supporting text-to-audio-video and image-to-audio-video, and paired with a super-resolution model that upscales output to Full HD (1920×1080).

    Why it matters: Kandinsky 6.0 has open-sourced video generation models with synchronized audio and lip-sync, and provides measured runtimes for various GPU tiers, making it easy for creators to assess local deployment costs.

Oct 1Thu
  1. Suno 官方博客62

    Suno releases the Speech audio model beta, capable of generating vocals and music simultaneously

    Suno released Speech beta, its first audio model that generates human voice and music together as a single complete track. Users input an idea, poem, or existing text, then describe the desired vocal tone and music style, and the system generates voice content with original background music. The model is now open to all users in beta; it had previously been tested with a small group of users for one month. The official statement notes that British accents occasionally drift toward Australian, and other effects still need improvement.

    Why it matters: Suno synthesizes voice and background music into a single track, allowing creators who do short-drama narration or dubbing to evaluate a new production approach.

Sep 29Tue
  1. DramaClip · 公开创作工具发布80

    DramaClip v0.2.0 First Public Release: Import Short Dramas to Batch-Generate Promotional Short Videos

    DramaClip v0.2.0, the first public release, allows users to import a short drama with dozens of episodes and batch-generate ready-to-publish promotional short videos, with all media processing completed locally. This version uses OCR and ASR fusion transcription with hard-subtitle character-by-character correction—60% of segments were corrected in testing—and employs CAM++ voiceprint clustering for speaker diarization.

    Why it matters: A short-drama promotional video tool that completes transcription, narration scripts, and voice cloning locally—explore its nine output modes and cut-point alignment approach.

Sep 25Fri
  1. GitHub:AI视频开源项目发现78

    SeeCut: Open-Sourced Auto-Editing Skill for Digital Human Voiceover, Built for Coding Agents

    YeJe-cpu open-sourced SeeCut on GitHub, a skill for coding agents like Claude Code that automatically edits digital human or real-person voiceover footage into vertical short videos with motion effects, and can optionally push the result as a layered CapCut draft for fine-tuning.

    Why it matters: The author open-sourced a reusable voiceover editing skill covering the full workflow from digital human voiceover to CapCut draft delivery.

Sep 23Wed
  1. Google DeepMind72

    Google Releases Gemini 3.8 Flash TTS and Flash-Lite TTS

    Google DeepMind has released two speech synthesis models: Gemini 3.8 Flash TTS and Gemini 3.8 Flash-Lite TTS. The former is designed for character design and line-by-line performance direction, while the latter targets high-volume, low-cost dubbing and voice agent scenarios.

    Why it matters: The two TTS models transform voice generation from fixed presets into a line-by-line directed creative workflow, enabling dubbing and audiobook teams to evaluate migration accordingly.

  2. Cartesia 官方博客62

    Cartesia launches Multilingual Voices, with 50+ voices natively supporting up to 25 languages

    Cartesia launched Multilingual Voices, where 50+ voices in the library can natively speak up to 25 languages, and supports extending custom voice clones to multilingual. Each language and accent is scored by native speakers using a rubric, checking whether accents are maintained and whether dates, numbers, and currencies are handled according to local conventions.

    Why it matters: Cartesia extends the same voice to 25 languages and supports cloning with 10 seconds of audio followed by adding multilingual accents, allowing teams doing multilingual dubbing to evaluate replacing their existing workflows.

Sep 10Thu
  1. OpenAI News62

    OpenAI launches GPT-Live-1 voice model in the API

    OpenAI launched GPT-Live-1 in the API, supporting natural full-duplex voice conversations. The model has improved instruction following and supports custom voices and telephony capabilities.

    Why it matters: OpenAI opens full-duplex voice conversation capabilities to the API, giving developers a way to evaluate how to integrate voice interaction products.

Jun 9Tue
  1. Google DeepMind71

    Google Releases Gemini 3.5 Live Translate Real-Time Voice Translation Model

    Google has released the audio model Gemini 3.5 Live Translate, supporting near real-time speech-to-speech translation across 70+ languages, with automatic language detection and preservation of the speaker's tone, rhythm, and pitch.

    Why it matters: Gemini 3.5 Live Translate reduces simultaneous interpretation latency to a few seconds and supports 70+ languages, enabling developers to evaluate real-time dubbing and meeting translation solutions.

Apr 16Thu
Mar 26Thu
  1. Google DeepMind62

    Google Releases Gemini 3.1 Flash Live Voice Model

    Google releases Gemini 3.1 Flash Live, calling it the highest-quality audio and voice model to date, with higher accuracy and lower latency.

    Why it matters: Gemini 3.1 Flash Live provides comparable benchmark data on latency, long-range reasoning, and multilingual performance for voice interactions, enabling developers to evaluate the deployment of voice agents.

Dec 13Sat
  1. Google DeepMind62

    Google Releases Upgraded Gemini 2.5 Flash Native Audio with Real-Time Voice Translation

    Google releases the upgraded Gemini 2.5 Flash Native Audio, designed for real-time voice agents. It improves complex workflow handling, instruction following, and multi-turn dialogue capabilities, leading on ComplexFuncBench Audio with 71.5%, and raising developer instruction-following rates from 84% to 90%.

    Why it matters: Gemini 2.5 Flash Native Audio updates function calling, instruction following, and multi-turn dialogue capabilities, and adds real-time voice translation, enabling evaluation of usability in voice agent and translation scenarios.

Sep 21Wed
  1. OpenAI News62

    OpenAI Open-Sources Speech Recognition Model Whisper

    OpenAI announces it has trained and open-sourced Whisper, a speech recognition neural network, stating that its robustness and accuracy in English speech recognition approach human level.

    Why it matters: OpenAI open-sources the speech recognition model Whisper, whose English recognition robustness and accuracy approach human level.