FreeInit

[a-r-r-o-w]

What does this PR do?

Fixes #6165.

Colab

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@DN6 @sayakpaul @patrickvonplaten

[a-r-r-o-w]

This PR attempts to add support for FreeInit. I realize that what I currently have isn't really ideal since it mixes up FreeInit-related code to what should ideally only be AnimateDiff, and limits the maintainability as more features are added in the future. However, I think that adding a new FreeInit-specific pipeline would be unnecessary and lead to a lot of redundant code. Let me know what you think would be the best way to add support for it.

[a-r-r-o-w]

Sharing some results from current code based on this notebook.

Results

AnimateDiff	AnimateDiff + FreeInit (method Gaussian)	AnimateDiff + FreeInit (method Butterworth)
Realistic Vision
ToonYou

[a-r-r-o-w]

FreeInit changes could be added to AnimateDiff+Controlnet and other future pipelines as well.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

Here's a Drive link to some more results. Also, updated Colab notebook.

Minimal code

import torch
from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler
from diffusers.utils import export_to_gif
from IPython.display import display, Image

adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2")

# model_id = "zhyemmmm/ToonYou"
model_id = "SG161222/Realistic_Vision_V5.1_noVAE"
pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter, torch_dtype=torch.float16).to("cuda")
pipe.scheduler = DDIMScheduler.from_pretrained(
    model_id,
    subfolder="scheduler",
    beta_schedule="linear",
    clip_sample=False,
    timestep_spacing="linspace",
    steps_offset=1
)
pipe.enable_vae_slicing()

pipe.enable_free_init(method="gaussian", use_fast_sampling=True)
output = pipe(
    prompt=(
        "masterpiece, bestquality, highlydetailed, ultradetailed, sunset, "
        "orange sky, warm lighting, fishing boats, ocean waves seagulls, "
        "rippling water, wharf, silhouette, serene atmosphere, dusk, evening glow, "
        "golden hour, coastal landscape, seaside scenery"
    ),
    negative_prompt="bad quality, worse quality",
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(666),
)
pipe.disable_free_init()

frames = output.frames[0]
export_to_gif(frames, "animation.gif")
display(Image("animation.gif", embed=True))

@DN6 I think this is ready for another review :) I've moved the denoising loop and latent decoding part to separate functions in order to allow freeinit to work more easily and also be able to return intermediate results. Also, at the moment, we have two pipelines that are related to AnimateDiff which could use FreeInit - one being this, and the other being the community AnimateDiff+Controlnet pipeline. Once we're satisfied with the changes here, let's add it to the other pipeline and any future pipelines as well? This can be used in TextToVideoSDPipeline and SVD as well I think.

Maybe some related tests can also be added?

[DN6]

Some small changes requested. But overall it's looking good 👍🏽 Can we add a fast test to check that the outputs are different with freeinit enabled.

[a-r-r-o-w]

@DN6 Requesting a review. I think this should be good to merge, no? I'll open follow up PRs for other animatediff pipelines following this.

[a-r-r-o-w]

@sayakpaul too, in case some other changes from design pov are required, thanks

[DN6]

Excellent work @a-r-r-o-w ! I think we just need an update to the AnimateDiff docs (on what FreeInit is and an example snippet) and we can merge this.

[DN6]

Nice work! 👍🏽