OpenAI shares lessons learned from the small-scale preview of Voice Engine, a custom voice model that can create tailored voices from text. The model is currently in limited testing.
Hugging Face releases TTS Arena, a text-to-speech model comparison tool inspired by LMSys Chatbot Arena. Users input text, listen to synthesized speech from two models, and vote for the more natural one, with results aggregated into a public leaderboard.
Hugging Face publishes an article surveying watermarking approaches for AI-generated content, distinguishing between two pathways: embedding at generation time and adding after generation. It also introduces watermarking and detection tools available on the Hub. The article mentions image "invisible modification" tools such as Glaze and Photoguard, as well as data poisoning tools like Nightshade and Fawkes, used to prevent models from learning or generating faces. Truepic embeds metadata according to the C2PA standard and combines it with watermarks to enhance provenance verifiability.
OpenAI explores large-scale training of generative models on video data, jointly training text-conditioned diffusion models on videos and images of varying durations, resolutions, and aspect ratios, using a transformer architecture that operates on spatiotemporal patches of video and image latent codes. Its largest model, Sora, can generate one-minute high-fidelity videos. Results indicate that scaling video generation models is a promising path toward building general-purpose physical world simulators.
Why it matters: OpenAI publicly reveals Sora's technical roadmap for the first time, explaining how video generation models can be scaled into physical world simulators.
Segmind releases the SegMoE framework, which can create Mixture-of-Experts (MoE) diffusion models from scratch and has been integrated into Hugging Face diffusers. This release includes three model versions—SegMoE-4x2, SegMoE-2x1, and SegMoE-SD4x2—along with the segmoe code package for building custom MoE models. The models are licensed under Apache 2.0.
Why it matters: The release provides code for building MoE diffusion models from scratch along with three merged versions, enabling creators to combine multiple pretrained models into new ones.
ONNX Runtime's CUDA and TensorRT execution providers can significantly accelerate SD Turbo and SDXL Turbo inference on NVIDIA GPUs, with throughput improvements of up to 229% (SDXL Turbo) and 120% (SD Turbo) compared to PyTorch.
Hugging Face publishes a tutorial explaining how to export ComfyUI workflows as pure Python, wrap them with Gradio, and deploy them on Hugging Face Spaces ZeroGPU for free.
Hugging Face releases aMUSEd, a non-diffusion text-to-image model with approximately 800M parameters. It is an open-source reproduction of Google's MUSE, using masked image modeling (MIM) with fewer inference steps and greater interpretability.
Hugging Face releases an advanced training script for SDXL Dreambooth LoRA, integrating Pivotal Tuning and the Prodigy optimizer into diffusers, along with reusable training parameters.
Why it matters: Hugging Face officially integrates Pivotal Tuning and the Prodigy optimizer into diffusers training scripts, providing directly reusable parameters and experimental comparisons.
Hugging Face implements dynamic LoRA loading in its Inference API, switching LoRA adapters per request while keeping the base model resident, reducing warm-up time from 25s to 3s and user request response time from 35s to 13s.
Hugging Face releases the LCM LoRA method, which trains a small number of LoRA adapter layers instead of distilling the entire model, reducing image generation for SDXL and Stable Diffusion from 25 to 50 steps down to 4 to 8 steps.
Why it matters: LCM LoRA enables SDXL to generate images in 4 steps. The article provides directly runnable diffusers code and speed comparisons across multiple hardware configurations.
OpenAI releases GPT-4 Turbo at DevDay, supporting 128K context with lower pricing and new vision capabilities. It also launches developer products including the Assistants API and DALL·E 3 API.
On an A100 (40GB), SDXL's default full-precision inference requires 28GB of VRAM and takes 72.2 seconds; switching to fp16 reduces this to 21.7GB and 14.8 seconds, adding SDPA (enabled by default in PyTorch 2.0) further reduces it to 11.4 seconds, and wrapping the UNet with torch.compile shortens it to 10.2 seconds.
OpenAI announced that DALL·E 3 is now available in ChatGPT Plus and Enterprise. To prepare for a broader release, OpenAI developed a set of safety mitigations and also shared its progress on content provenance research.
Why it matters: DALL·E 3 is open to ChatGPT Plus and Enterprise, and the official announcement also explains the safety mitigations made for the expanded release.
Hugging Face Diffusers now supports running SDXL inference on Cloud TPU v5e via JAX, achieving high-performance, low-cost deployment with JIT compilation and XLA parallelism. The demo generates 4 images of 1024×1024 in parallel on multiple TPU v5e-4 instances in about 4 seconds, with pure generation time of about 2.3 seconds; TPU v5e costs less than half of TPU v4.
Hugging Face published a tutorial on how to fork and configure AI Comic Factory and deploy it to a private Space via the Inference API, avoiding the long waits of the official Space.
A Hugging Face blog post explains how to use the newly added DDPOTrainer in the trl library to fine-tune Stable Diffusion with reinforcement learning via DDPO (Denoising Diffusion Policy Optimization).
3D Gaussian Splatting is a rasterization technique that can learn from a small number of images and render photorealistic scenes in real time. It first uses COLMAP for Structure from Motion to estimate a point cloud, then converts each point into a Gaussian and trains it, achieving fast rendering through differentiable Gaussian rasterization. The costs are high VRAM usage (4GB for viewing, 12GB for training) and large scene files (1GB+), and it is not yet compatible with existing rendering pipelines such as Vulkan, DirectX, and WebGPU.
Hugging Face released the Würstchen image generation diffusion model, whose text-conditioning part works in a highly compressed latent space, achieving 42x spatial compression through Stage A's VQGAN and Stage B's Diffusion Autoencoder, with Stage C trained as a Prior in that latent space.
Why it matters: Würstchen uses 42x spatial compression to cut training cost to about one-sixth of SD1.4, which can be used to evaluate low-VRAM image generation options.
The Hugging Face Diffusers team, in collaboration with the T2I-Adapter authors, provides T2I-Adapter support for Stable Diffusion XL and releases checkpoints for conditions such as sketch, canny, lineart, depth, and openpose.
Why it matters: Comparing ControlNet's parameter count and inference overhead, it presents a T2I-Adapter controllable generation solution on SDXL and reusable training scripts.
A Hugging Face blog post explains how to use AudioLDM 2 in Diffusers and, through optimizations such as half precision, flash attention, torch.compile, and changing the scheduler, reduce the generation time for 10 seconds of audio from 14 seconds to under 1 second, with very little loss in audio quality.
Based on 🤗 Transformers, Optimum, and Accelerate, three optimizations can reduce the VRAM usage of the Bark text-to-speech model and speed up inference. Bark was proposed by Suno AI, has been supported since Transformers v4.31.0, consists of four models—semantic, coarse acoustics, fine acoustics, and EnCodec decoding—and offers small and large checkpoints.
This tutorial demonstrates how to deploy DeepFloyd IF as a production-grade service using the open-source model serving framework BentoML. DeepFloyd IF is an open-source text-to-image model that works directly in pixel space, consists of a frozen text encoder and three cascaded pixel diffusion modules, can progressively upscale a 64x64 px image to 1024x1024 px, and encodes prompts with T5-XXL-1.1.
Hugging Face published a tutorial on how to deploy the MusicGen music generation model through a custom handler in Inference Endpoints. Users need to duplicate the facebook/musicgen-large repository and add handler.py and requirements.txt to create an endpoint and generate music from text prompts, and can also optionally use a melody snippet as a condition.
Segmind has open-sourced the weights and knowledge distillation training code for the compressed models SD-Small and SD-Tiny; their parameter counts are reduced by 35% and 55% respectively compared with the base model, inference speed is improved by up to 100%, and image fidelity is comparable to the base model.
Apple and Hugging Face have ported the Stable Diffusion XL base model to Core ML, usable on Apple Silicon Macs running the macOS 14 public beta; it currently uses the ORIGINAL attention implementation, and the refiner stage has not yet been ported.
Why it matters: Apple and Hugging Face jointly present a Core ML mixed-bit quantization solution for SDXL on Mac, with measured latency for various models and compression recipes.
Hugging Face announced the results of the first Open Source AI Game Jam; the event received registrations from more than 1,300 people and 88 game submissions, and the championship went to Snip It by ohmlet, which uses Stable Diffusion to generate assets.
Hugging Face's Diffusers library has reached its one-year anniversary, having expanded from text-to-image to video, 3D, and image editing. The library has added support for models such as DeepFloyd IF, SDXL, VideoFusion, Text2Video-Zero, and OpenAI Shap-E, and integrates LoRA, Torch 2.0 optimizations, and safety checking components.
Hugging Face released an experimental AI WebTV demo that uses the open-source text-to-video model Zeroscope and the music generation model MusicGen to automatically generate and stream short videos.
Why it matters: It breaks down the complete pipeline for building an automatically generated video stream with Zeroscope, MusicGen, and FILM, and includes failure cases and prompt suggestions.
Hugging Face released Issue 4 of its "Ethical and Social Newsletter," focusing on bias in text-to-image models. The article outlines five sources of bias: training data, pretraining data filtering, CLIP inference, the model's latent space, and post-hoc filtering, and points out that Stable Diffusion v2.1 and ERNIE ViLG produce culturally stereotyped results under the same prompt, "a house in Beijing." The team also introduced the Stable Bias project, a tool for exploring and comparing bias in generated images.
A Hugging Face blog post introduces 6-bit palettization quantization via Core ML, which lets Stable Diffusion run faster and use less memory on iPhone, iPad, and Mac.
Using quantization-aware training (QAT) with OpenVINO NNCF combined with Token Merging, Stable Diffusion achieves a 5.1x inference speedup and a 4x reduction in model size on Intel CPUs. The optimization starts from a model fine-tuned on the Pokemons dataset, requires only 4096 iterations, completes within one day on a single GPU with 24GB of VRAM, and has been integrated into the Diffusers training scripts.
This work combines the training strategy of InstructPix2Pix with the instruction template approach of FLAN V2 to instruction fine-tune Stable Diffusion, enabling it to perform image translation and low-level image processing tasks such as cartoonization and rain removal given an input image plus an instruction.
A Hugging Face blog post traces the development of text-to-video models: early GAN- and VAE-based methods were limited by low resolution and short duration, after which Phenaki, Make-A-Video, NUWA, VideoGPT, CogVideo, and others shifted to transformer architectures, while current mainstream approaches use diffusion models. Text-to-video must ensure spatiotemporal consistency across frames and faces three major challenges: high compute costs, scarce high-quality datasets, and vague video descriptions; models can typically generate only very short clips.
A Hugging Face blog post demonstrates how to run DeepFloyd's pixel-level text-to-image model IF with 🧨 diffusers on free-tier Google Colab, covering three features: text-to-image, image variants, and inpainting.
On Sapphire Rapids Xeon CPUs, Stable Diffusion v1-5 with default float32 inference has an average latency of 32.3 seconds per image; switching to Optimum Intel and OpenVINO reduces this to 16.7 seconds, and fixing a static 512x512 shape further reduces it to 4.7 seconds.