stable diffusion videos

稳定的扩散 – 视频

在Colab中自己尝试：

示例– “蓝莓意大利面”和“草莓意大利面”之间的变形

安装

pip install stable_diffusion_videos

用法

查看示例文件夹，例如脚本?

制作视频

注意：对于Apple M1体系结构，请改用Torch.float32，因为MPS上没有Torch.float16。

 from stable_diffusion_videos import StableDiffusionWalkPipeline
import torch

pipeline = StableDiffusionWalkPipeline . from_pretrained (
    \"CompVis/stable-diffusion-v1-4\" ,
    torch_dtype = torch . float16 ,
). to ( \"cuda\" )

video_path = pipeline . walk (
    prompts = [ \'a cat\' , \'a dog\' ],
    seeds = [ 42 , 1337 ],
    num_interpolation_steps = 3 ,
    height = 512 ,  # use multiples of 64 if > 512. Multiples of 8 if < 512.
    width = 512 ,   # use multiples of 64 if > 512. Multiples of 8 if < 512.
    output_dir = \'dreams\' ,        # Where images/videos will be saved
    name = \'animals_test\' ,        # Subdirectory of output_dir where images/videos will be saved
    guidance_scale = 8.5 ,         # Higher adheres to prompt more, lower lets model take the wheel
    num_inference_steps = 50 ,     # Number of diffusion steps per image generated. 50 is good default
)

制作音乐视频

新的！可以通过提供音频文件的路径来添加音乐。音频将告知插值率，因此视频转移到节拍?

 from stable_diffusion_videos import StableDiffusionWalkPipeline
import torch

pipeline = StableDiffusionWalkPipeline . from_pretrained (
    \"CompVis/stable-diffusion-v1-4\" ,
    torch_dtype = torch . float16 ,
). to ( \"cuda\" )

# Seconds in the song.
audio_offsets = [ 146 , 148 ]  # [Start, end]
fps = 30  # Use lower values for testing (5 or 10), higher values for better quality (30 or 60)

# Convert seconds to frames
num_interpolation_steps = [( b - a ) * fps for a , b in zip ( audio_offsets , audio_offsets [ 1 :])]

video_path = pipeline . walk (
    prompts = [ \'a cat\' , \'a dog\' ],
    seeds = [ 42 , 1337 ],
    num_interpolation_steps = num_interpolation_steps ,
    audio_filepath = \'audio.mp3\' ,
    audio_start_sec = audio_offsets [ 0 ],
    fps = fps ,
    height = 512 ,  # use multiples of 64 if > 512. Multiples of 8 if < 512.
    width = 512 ,   # use multiples of 64 if > 512. Multiples of 8 if < 512.
    output_dir = \'dreams\' ,        # Where images/videos will be saved
    guidance_scale = 7.5 ,         # Higher adheres to prompt more, lower lets model take the wheel
    num_inference_steps = 50 ,     # Number of diffusion steps per image generated. 50 is good default
)

使用UI

 from stable_diffusion_videos import StableDiffusionWalkPipeline , Interface
import torch

pipeline = StableDiffusionWalkPipeline . from_pretrained (
    \"CompVis/stable-diffusion-v1-4\" ,
    torch_dtype = torch . float16 ,
). to ( \"cuda\" )

interface = Interface ( pipeline )
interface . launch ()

学分

这项工作是由@karpathy共享的脚本构建的。将脚本修改为此要旨，然后将其更新/修改为此回购。

贡献

您可以在此处提交任何问题/功能请求

享受?