StableDiffusionXLEnsemblePipeline and StableDiffusionXLImg2ImgEnsemblePipeline

[AmericanPresidentJimmyCarter]

Model/Pipeline/Scheduler description

The current solution from PR #4015 does not result in an optimal solution for the use of the refinement model as an ensemble model because it works by a float value to determine the selection of timesteps to execute. This is further described in the commentary on the PR.

I am proposing overloading the type of denoising_end and denoising_start with int as well as float. In the event that it is an int, the approximate timestep/sigma will be used to determine the start and end of the denoising loop for the sampling step.

After this is done, add in the two new pipelines. These pipelines will be easy for the end user to use and will already use the optimal solution (base model for timesteps 201-1000, refinement model for timesteps 0-100) for the ensemble model, and will work with all schedulers. It will be used something like:

    pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
    refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
    ensemble_pipe = StableDiffusionEnsemblePipeline(pipe, refiner)
    image = pipe(prompt=prompt, output_type="latent", num_inference_steps=30).images

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[n00mkrad]

As mentioned in #4015, compatibility with certain samplers (all karras samplers, DPM++ 2M) needs to be investigated as well.

[patrickvonplaten]

Sorry I don't fully understand this issue here - is this issue about adding a StableDiffusionEnsemblePipeline pipeline or about changing denoising_start from float to int?

[AmericanPresidentJimmyCarter]

Once the denoising_start and denoising_end are available to be used as an integer, I can make the StableDiffusionXLEnsemblePipeline that you can simply use like:

    pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
    refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
    ensemble_pipe = StableDiffusionXLEnsemblePipeline(pipe, refiner)
    image = pipe(prompt=prompt, output_type="latent", num_inference_steps=30).images

or

    ensemble_pipe = StableDiffusionXLEnsemblePipeline("stabilityai/stable-diffusion-xl-refiner-0.9",
        "stabilityai/stable-diffusion-xl-refiner-0.9", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
    image = pipe(prompt=prompt, output_type="latent", num_inference_steps=30).images

I need the ability to select the timesteps each unet is denoising for this to work correctly.

The intention is that people can use the ensemble pipeline as a one liner, since most people will only want to use the ensemble pipeline.

[patrickvonplaten]

It would be great if we could add the ensemble pipeline to the the community pipelines here: https://github.com/huggingface/diffusers/tree/main/examples/community .

Let's maybe discuss allowing to provide denoising_start as an integer in #4115

[AmericanPresidentJimmyCarter]

Sure, I figured that that would be a less contentious PR so I coded that up first.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.