Skip to content

Creative Directions

Dubbing and Audio Latest

Advances in AI speech: tools and methods for dubbing, voice cloning, multi-emotion reading, and audio adaptation of short dramas.

60 picksLast 30 days 33 itemsCollected 269 items

updated

Pick archive · page 2

Oct 30Fri21–40
  1. ElevenLabs 产品与研究60

    ElevenLabs launches SFX v3 sound effects model, now available on ElevenCreative and ElevenAPI

    ElevenLabs launches SFX v3 sound effects model, now available on ElevenCreative and ElevenAPI. SFX v3 handles prompts describing multiple sounds more reliably, produces more believable music and vocal generation, and delivers cleaner, more natural audio quality than SFX v2. Creators can use it in Flows, the Sound Effects tab, Studio, and Music Editor, while developers can integrate it via ElevenAPI.

    Why it matters: SFX v3 improves multi-sound prompt following, music and vocal generation, and audio quality, allowing creators to evaluate whether to replace their existing sound effects generation workflow.

  2. Reddit comfyui · 创作与工作流分享68

    Phane: An open-source creative tool submitted to #OPENSOURCEHIGGSFIELD

    The author released Phane, a 100% open-weight local creative tool that runs on an 8GB graphics card, supporting text-to-3D motion and then to any angle. It integrates 45 engines covering image, editing, video, re-casting, voice, and other capabilities, with one-click installation and no need for cloud or credits.

    Why it matters: This open-source tool allows an 8GB VRAM graphics card to locally run a 45-engine text-to-3D and video creation pipeline, and creators can evaluate whether to use it as a replacement for paid cloud solutions.

  3. AITNT · AI创作资讯73

    Genspark Media Studio launches, enabling even those with zero experience to shoot short dramas.

    Genspark Media Studio officially launches, integrating over 70 model versions from more than 20 vendors, covering video, image, audio, and 3D generation. In testing, Seedance 2.5, Kling, and Veo can be used to generate video, ElevenLabs for audio, and the 'Story Bible' feature automatically generates short dramas. Plus and Pro members enjoy 50% off credit consumption from October 27 to November 4.

    Why it matters: Genspark integrates over 70 models from more than 20 vendors into a single workbench, allowing zero-experience users to complete the entire process from script to finished film.

Oct 29Thu
  1. 可灵官方博客71

    Ami discusses how Kling 4.0 brings subtle emotions to life.

    Kling invited creator Ami to use Kling 4.0 to produce the short film "Tea When You Have Time". She shared her experience using new capabilities such as 30-second generation, Omni Reference, and multi-keyframe. Kling 4.0 supports single-run 30-second generation, 21:9 aspect ratio, up to 10 keyframes, consistent 12-character assets, good lip-sync for Cantonese dialogue, and supports multiple dialects and English accents.

    Why it matters: Creator hands-on testing shows Kling 4.0 performs stably in 30-second long takes, 12-character consistency, and Cantonese lip-sync, providing a basis for evaluating an upgrade.

  2. 歸藏73

    Creating a gender-swapped Malfoy AI girlfriend desktop app with Jimeng Short Drama Studio and open-sourcing it

    Guizang used Jimeng AI Short Drama Studio to create a gender-swapped Malfoy interactive desktop app and open-sourced the client. The app includes a first-person short drama and an affinity system; visuals, voice acting, and characters were generated by Jimeng, music by Suno, editing and code by Claude; the production process covers uploading scripts, asset library, storyboard director console, and video generation. The client has been open-sourced to github.com/op7418/malfoy-desktop.

    Why it matters: The author open-sourced the complete client code and detailed the production process from script to desktop; creators who want to use Jimeng Short Drama Studio to make interactive works can follow this to reproduce it.

  3. HeyGen Updates67

    HeyGen Voice Launches Speech-to-Speech Feature

    HeyGen Voice launches a speech-to-speech (Speech-to-Speech) feature. Users can record a line with their phone or computer and get studio-quality voiceover that preserves emotion, rhythm, stress, and pronunciation, changing only the voice itself. The feature is now available on HeyGen and the API, supporting clean narration from casual recordings and offering a richer emotional range than text prompts, including whispers, laughter, and accents.

    Why it matters: HeyGen's speech-to-speech turns phone recordings directly into studio-quality voiceover, reducing repeated TTS generation, and creators can use it to evaluate the new voiceover workflow.

Oct 27Tue
  1. Runway 官方新闻74

    Runway launches App Studio, enabling no-code creation of custom creative apps.

    Runway launches App Studio, allowing users to describe their needs in natural language and build custom creative apps with standalone interfaces without code, generating images, videos, and audio. App Studio includes built-in apps such as Character Swap, AutoComp, and Product Portfolio, built on Brand Kits, Workflows, and Assets. It is included in all paid plans and consumes existing Runway credits.

    Why it matters: Runway's App Studio brings no-code app building into creative work, allowing teams to turn years of accumulated workflows and brand standards into tools anyone can use, reducing reliance on individual experts.

Oct 23Fri
  1. MindStudio Blog · 创作分享72

    Fish Audio releases the Drama 3 speech model: controlling performance with parenthetical stage directions

    Fish Audio released the text-to-speech model Drama 3, allowing users to write stage directions in square brackets within scripts to adjust tone, pacing, and performance. It also supports word-level control, inline sound effects, and multi-speaker scripts. The model is released as open weights with 36B parameters on Hugging Face (fishaudio/drama-3). According to user reports, full-precision operation requires approximately 46GB of VRAM, while the community 4-bit quantized version requires approximately 32GB.

    Why it matters: Fish Audio open-sources a 36B speech model, using bracketed stage directions to control performance, and provides benchmark differences compared to ElevenLabs along with self-hosting VRAM requirements.

  2. MindStudio Blog · 创作分享75

    LTX 2.5 Open Weights Released: 22B Video Model Available for Self-Download and Operation

    Lightricks released LTX 2.5 open weights on Hugging Face. This is a 22B-parameter diffusion model that can be downloaded and run on your own GPU. It also provides a full dev version and a distilled accelerated version, along with bf16, int8, and nvfp4 quantization formats.

    Why it matters: Lightricks open-sourced a 22B video model along with audio and multi-shot capabilities. The article breaks down the components of the download package and the self-hosting requirements.

  3. MindStudio Blog · 创作分享78

    LTX 2.5 Explained: An Open-Weight Video Model Supporting Synchronized Audio and Native Multi-Shot

    LTX 2.5 is an open-weight AI video model that produces audio and video in the same generation pass and supports multiple coherent shots within a single generation. The model is available in standard and distilled versions; weights can be downloaded, called via a hosted API, or deployed on your own GPU. The official team says the new diffusion video decoder improves performance on faces, textures, on-screen text, and fast motion. The article also documents the process of self-hosting on a remote Nvidia GPU and testing lip-sync and facial consistency with dialogue shots.

    Why it matters: The article breaks down LTX 2.5's three features—audio-video co-generation, native multi-shot, and self-hostability—to help assess the practical boundaries of open-weight video models.

  4. pyvideotrans 官方发布65

    pyvideotrans v4.17 released

    pyvideotrans v4.17 released, adding the Audio.cpp speech recognition channel, and adding noise reduction and vocal separation models based on audio.cpp. It also fixes the nemotronasr repeated download issue, adds the Opper translation channel, and adds a 'further simplify' option to subtitle translation.

    Why it matters: The new version adds the Audio.cpp speech recognition channel and noise reduction models, and optimizes the subtitle translation simplification option, suitable for evaluating upgrades to the video translation and dubbing workflow.

Oct 21Wed
  1. ComfyUI Wiki · 工作流与开源创作分享81

    FastVideo releases FastH3 Omni: a reference-conditioned variant of MiniMax H3 that generates video with sound in 8 steps

    FastVideo released FastH3 Omni, a reference-conditioned variant of MiniMax H3; a prompt plus ordered image, video and audio references can generate clips with synchronized sound, requiring only 8 transformer evaluations instead of 50 for base H3.

    Why it matters: FastH3 Omni unifies reference images, video and audio into a single generation pipeline, and provides an item-by-item time comparison with base H3 to help assess local deployment costs.

  2. FastVideo 视频模型 · 模型发布80

    FastVideo releases ComfyUI version model weights for FastH3 Omni

    FastVideo released ComfyUI-usable weights for the final version of FastH3 Omni, providing two diffusion models, BF16 and INT8 ConvRot, which can generate video and synchronized audio from prompts and ordered image, video, and audio references.

    Why it matters: FastH3 Omni releases ComfyUI-usable BF16 and INT8 weights and provides the way to connect reference images, video, and audio, making it easier for creators to evaluate local video production pipelines.

Oct 20Tue
  1. IT之家68

    Ant Ling opens source Ming-3.0 speech series models, covering speech synthesis, audio understanding and dialogue, and speech recognition

    Ant Ling has open-sourced the Ming-3.0 speech series, including three series—speech synthesis, audio understanding and dialogue, and speech recognition—with five models in total, which developers can select by task or combine into workflows. Ming-3.0-tts's Voice Design supports describing the target timbre in text, and can also set the timbre with reference audio; director mode can specify how lines are performed, emotional changes, stressed words, and pacing.

    Why it matters: Ming-3.0 splits speech synthesis, audio understanding, and recognition into five models, among which director mode can describe the performance style of lines in text.

  2. CGPress72

    Mirelo releases SFX 2.0 video-to-sound-effect model

    Berlin-based AI research team Mirelo released the video-to-sound-effect model SFX 2.0, which can modify a video's audio without regenerating the visuals, preserve dialogue, remove unspecified music, and generate sound effects, Foley and ambience synchronized with action and consistent across cuts.

    Why it matters: The video-to-sound-effect model has been updated to redo the audio track without changing the visuals, and has been integrated into editing software and third-party platforms.

Oct 17Sat
  1. FastVideo 视频模型 · 模型发布60

    FastVideo releases FastH3-QAD-v2 video-audio model

    FastVideo releases FastH3-QAD-v2, an 8-step video and audio model trained with NVFP4 weights and quantization-aware distillation (QAD), runnable on a single Blackwell GPU (RTX 5090, RTX PRO 6000) or DGX Spark.

    Why it matters: The 8-step video and audio model lands on a single Blackwell GPU with NVFP4 quantized weights, and error data versus the BF16 teacher model is provided.

  2. ElevenLabs 产品与研究62

    ElevenLabs launches Instant Soundtracks in ElevenCreative

    ElevenLabs Launches Instant Soundtracks in ElevenCreative; After Uploading a Video, the Model Generates an Original Score and Analyzes Sound Effect Cue Points in the Footage, Automatically Placing Sound Effects on the Timeline at the Corresponding Timestamps.

    Why it matters: ElevenLabs merges score and sound effect generation into the video workflow, which can be used to evaluate the sound production stage of short dramas and ads.

Oct 15Thu
  1. note · koheii_miaa · 图像视频创作分享65

    How an AI musician uses a single Mac to produce an MV with lyric sync and lip-sync, 17 minutes per track

    The author published a workflow for locally generating MVs on a Mac: input a song and cover image, output a 30-second video with synchronized lyrics and a cover character singing, in both 16:9 and vertical formats, approximately 17 minutes per track, without using cloud-based paid generation.

    Why it matters: The author published the complete local workflow from vocal separation to lyric sync with time breakdowns, allowing for an assessment of the feasibility of producing MVs on a single machine.

  2. Reddit StableDiffusion · 创作者与工作流分享66

    Author Develops ComfyUI Custom Node for Fish Audio Drama 3, Runs on 8GB VRAM

    The author developed the ComfyUI-Drama3 custom node for Fish Audio Drama 3, a 35B text-to-speech model that supports voice cloning, dialogue, and inline prompts like [sighs] and [whispers], with a minimum requirement of 8GB VRAM.

    Why it matters: The author integrated a 35B voice model into ComfyUI with quantization, making it runnable on 8GB VRAM, which can be used to evaluate local voiceover workflows.

  3. HeyGen Updates78

    HeyGen releases the HeyGen Voice speech model

    HeyGen releases a new speech model, HeyGen Voice, which it calls the first speech model built for video adaptation and which ranks first on the Artificial Analysis speech leaderboard; it is already included in every HeyGen video at no extra charge.

    Why it matters: HeyGen launches its first speech model for video adaptation, allowing creators to assess the room for replacing dubbing and multilingual lip-sync workflows.