Skip to content

community stablediffusion controlnet img2img pipeline #2584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

mikegarts
Copy link
Contributor

@mikegarts mikegarts commented Mar 7, 2023

Community pipeline for StableDiffusionControlNetImg2img
Very similar to #2561
@williamberman Tagging you since you did #2561 and I would appreciate your feedback.

@mikegarts
Copy link
Contributor Author

Usage example:

import torch
from PIL import Image
from diffusers import ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image

input_image = load_image(
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)

pipe_controlnet = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", 
    controlnet=controlnet, 
    safety_checker=None, 
    torch_dtype=torch.float16
)

pipe_controlnet.scheduler = UniPCMultistepScheduler.from_config(pipe_controlnet.scheduler.config)
pipe_controlnet.enable_xformers_memory_efficient_attention()
pipe_controlnet.enable_model_cpu_offload()

# using image with edges for our canny controlnet
control_image = load_image("https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/vermeer_canny_edged.png")
control_image.show()

result_img = pipe_controlnet(controlnet_conditioning_image=control_image, 
                        image=input_image,
                        prompt="an android robot, cyberpank, digitl art masterpiece", 
                        num_inference_steps=20).images[0]

result_img.show()

@mikegarts
Copy link
Contributor Author

Related to #2562

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Mar 7, 2023

The documentation is not available anymore as the PR was closed or merged.

@mikegarts mikegarts force-pushed the stablediffusion.controlnet.img2img.pipeline branch from 32009fe to 61e9f47 Compare March 7, 2023 10:50
@mikegarts
Copy link
Contributor Author

@patrickvonplaten
Copy link
Contributor

@williamberman could you take a look here?

Copy link
Contributor

@williamberman williamberman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love it :)

@williamberman williamberman merged commit 46bef6e into huggingface:main Mar 7, 2023
@pd-brian-mccrindle
Copy link

@mikegarts @williamberman
I checked the integration of both the img2img and inpaint_img2img stable diffusion community pipelines using the recommended loading scheme and have received the same error:

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet="lllyasviel/sd-controlnet-depth",
    custom_pipeline="stable_diffusion_controlnet_img2img",
    # custom_pipeline="stable_diffusion_controlnet_inpaint_img2img",
)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/brian.mccrindle/Projects/pd-diffusion/scripts/controlnet_experiments.py:48 in <module>     │
│                                                                                                  │
│   45                                                                                             │
│   46 from diffusers import DiffusionPipeline                                                     │
│   47                                                                                             │
│ ❱ 48 pipe = DiffusionPipeline.from_pretrained(                                                   │
│   49 │   "runwayml/stable-diffusion-v1-5",                                                       │
│   50 │   controlnet="lllyasviel/sd-controlnet-depth",                                            │
│   51 │   custom_pipeline="stable_diffusion_controlnet_img2img",                                  │
│                                                                                                  │
│ /home/brian.mccrindle/anaconda3/envs/controlnet-img2img/lib/python3.8/site-packages/diffusers/pi │
│ pelines/pipeline_utils.py:965 in from_pretrained                                                 │
│                                                                                                  │
│    962 │   │   │   )                                                                             │
│    963 │   │                                                                                     │
│    964 │   │   # 5. Instantiate the pipeline                                                     │
│ ❱  965 │   │   model = pipeline_class(**init_kwargs)                                             │
│    966 │   │                                                                                     │
│    967 │   │   if return_cached_folder:                                                          │
│    968 │   │   │   return model, cached_folder                                                   │
│                                                                                                  │
│ /home/brian.mccrindle/.cache/huggingface/modules/diffusers_modules/git/stable_diffusion_controln │
│ et_img2img.py:159 in __init__                                                                    │
│                                                                                                  │
│   156 │   │   │   │   " checker. If you do not want to use the safety checker, you can pass `'   │
│   157 │   │   │   )                                                                              │
│   158 │   │                                                                                      │
│ ❱ 159 │   │   self.register_modules(                                                             │
│   160 │   │   │   vae=vae,                                                                       │
│   161 │   │   │   text_encoder=text_encoder,                                                     │
│   162 │   │   │   tokenizer=tokenizer,                                                           │
│                                                                                                  │
│ /home/brian.mccrindle/anaconda3/envs/controlnet-img2img/lib/python3.8/site-packages/diffusers/pi │
│ pelines/pipeline_utils.py:252 in register_modules                                                │
│                                                                                                  │
│    249 │   │   │   if module is None:                                                            │
│    250 │   │   │   │   register_dict = {name: (None, None)}                                      │
│    251 │   │   │   else:                                                                         │
│ ❱  252 │   │   │   │   library = module.__module__.split(".")[0]                                 │
│    253 │   │   │   │                                                                             │
│    254 │   │   │   │   # check if the module is a pipeline module                                │
│    255 │   │   │   │   pipeline_dir = module.__module__.split(".")[-2] if len(module.__module__  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'str' object has no attribute '__module__'

I don't see this when I load other non-controlnet models, such as img2img_inpainting.py. Any ideas what could be causing this?

@mikegarts
Copy link
Contributor Author

mikegarts commented Mar 8, 2023

@pd-brian-mccrindle
Please pass as the controlnet parameter a controlnet model object

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)

# load the pipeline dynamically
pipe_controlnet = DiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    safety_checker=None,
    custom_pipeline="stable_diffusion_controlnet_img2img")

See https://colab.research.google.com/drive/1CfJO2rNKDPMJeLrTXVi7e03-QE0VruwE?usp=sharing for example.

mengfei25 pushed a commit to mengfei25/diffusers that referenced this pull request Mar 27, 2023
w4ffl35 pushed a commit to w4ffl35/diffusers that referenced this pull request Apr 14, 2023
@WhiteWolf47
Copy link

WhiteWolf47 commented Jul 5, 2023

I have only seen diffusers.StableDiffusionControlNetImg2ImgPipeline.from_pretrained everywhere, does it have a from_ckpt method like diffusers.StableDiffusionControlNetImg2ImgPipeline.from_ckpt , i tried but it said that it does not have a from_ckpt method, does this mean that i cannot load my own model?

@patrickvonplaten
Copy link
Contributor

I think we need to add the from_single_file method for this. Would you be interested in opening a PR? Also cc @yiyixuxu maybe you can take a look? :-)

@patrickvonplaten
Copy link
Contributor

Related: #3907

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants