From 0af44613d85ef26744204d6be2294c4e401496ef Mon Sep 17 00:00:00 2001 From: Steven Liu Date: Mon, 15 Apr 2024 09:51:06 -0700 Subject: [PATCH 1/4] pipelines --- docs/source/en/_toctree.yml | 8 +- .../custom_pipeline_overview.md | 42 ++ docs/source/en/using-diffusers/loading.md | 493 +++++++----------- .../en/using-diffusers/loading_overview.md | 17 - docs/source/en/using-diffusers/schedulers.md | 68 ++- 5 files changed, 290 insertions(+), 338 deletions(-) delete mode 100644 docs/source/en/using-diffusers/loading_overview.md diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index fc5b8503b1b9..c98e1981a386 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -24,14 +24,12 @@ title: Tutorials - sections: - sections: - - local: using-diffusers/loading_overview - title: Overview - local: using-diffusers/loading - title: Load pipelines, models, and schedulers - - local: using-diffusers/schedulers - title: Load and compare different schedulers + title: Load pipelines - local: using-diffusers/custom_pipeline_overview title: Load community pipelines and components + - local: using-diffusers/schedulers + title: Load models and schedulers - local: using-diffusers/using_safetensors title: Load safetensors - local: using-diffusers/other-formats diff --git a/docs/source/en/using-diffusers/custom_pipeline_overview.md b/docs/source/en/using-diffusers/custom_pipeline_overview.md index 3c03ddc732f1..86be0404d173 100644 --- a/docs/source/en/using-diffusers/custom_pipeline_overview.md +++ b/docs/source/en/using-diffusers/custom_pipeline_overview.md @@ -109,6 +109,48 @@ pipeline = DiffusionPipeline.from_pretrained( +### Load from_pipe + +Did you know that you can use `from_pipe` with a community pipeline? Let me show you an example of using long negative prompt and prompt weighting! + +```bash +pipe_lpw = DiffusionPipeline.from_pipe( + pipe_sd, + custom_pipeline="lpw_stable_diffusion", +).to("cuda") + +prompt = "best_quality (1girl:1.3) bow bride brown_hair closed_mouth frilled_bow frilled_hair_tubes frills (full_body:1.3) fox_ear hair_bow hair_tubes happy hood japanese_clothes kimono long_sleeves red_bow smile solo tabi uchikake white_kimono wide_sleeves cherry_blossoms" +neg_prompt = "lowres, bad_anatomy, error_body, error_hair, error_arm, error_hands, bad_hands, error_fingers, bad_fingers, missing_fingers, error_legs, bad_legs, multiple_legs, missing_legs, error_lighting, error_shadow, error_reflection, text, error, extra_digit, fewer_digits, cropped, worst_quality, low_quality, normal_quality, jpeg_artifacts, signature, watermark, username, blurry" +generator = torch.Generator(device="cpu").manual_seed(33) +out_lpw = pipe_lpw.text2img( + prompt, + negative_prompt=neg_prompt, + width=512,height=512, + max_embeddings_multiples=3, + num_inference_steps=num_inference_steps, + generator=generator, + ).images[0] +``` + +
+ +
+ +let’s run StableDiffusionPipeline with the same inputs to compare: the result from the long prompt weighting pipeline is more aligned with the text prompt. + +``` +generator = torch.Generator(device="cpu").manual_seed(33) +out_sd = pipe_sd( + prompt=prompt, + negative_prompt=negative_prompt, + generator=generator, + num_inference_steps=num_inference_steps, +).images[0] +out_sd +``` +
+ +
For more information about community pipelines, take a look at the [Community pipelines](custom_pipeline_examples) guide for how to use them and if you're interested in adding a community pipeline check out the [How to contribute a community pipeline](contribute_pipeline) guide! diff --git a/docs/source/en/using-diffusers/loading.md b/docs/source/en/using-diffusers/loading.md index 9d5534154fc8..39c4f23158b5 100644 --- a/docs/source/en/using-diffusers/loading.md +++ b/docs/source/en/using-diffusers/loading.md @@ -10,57 +10,75 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -# Load pipelines, models, and schedulers +# Load pipelines [[open-in-colab]] -Having an easy way to use a diffusion system for inference is essential to 🧨 Diffusers. Diffusion systems often consist of multiple components like parameterized models, tokenizers, and schedulers that interact in complex ways. That is why we designed the [`DiffusionPipeline`] to wrap the complexity of the entire diffusion system into an easy-to-use API, while remaining flexible enough to be adapted for other use cases, such as loading each component individually as building blocks to assemble your own diffusion system. - -Everything you need for inference or training is accessible with the `from_pretrained()` method. +Diffusion systems often consist of multiple components like parameterized models and schedulers that interact in complex ways. That is why we designed the [`DiffusionPipeline`] to wrap the complexity of the entire diffusion system into an easy-to-use API. At the same time, the [`DiffusionPipeline`] is entirely customizable so you can modify each component to build your own diffusion system for your use case. This guide will show you how to load: - pipelines from the Hub and locally - different components into a pipeline +- multiple pipelines without increasing memory usage - checkpoint variants such as different floating point types or non-exponential mean averaged (EMA) weights -- models and schedulers -## Diffusion Pipeline +## Load a pipeline + +> [!TIP] +> Skip to the [DiffusionPipeline explained](#diffusionpipeline-explained) section if you're interested in an explanation about how the [`DiffusionPipeline`] class works. - +There are two ways to load a pipeline for a task: -💡 Skip to the [DiffusionPipeline explained](#diffusionpipeline-explained) section if you are interested in learning in more detail about how the [`DiffusionPipeline`] class works. +1. Load a pipeline with the [`DiffusionPipeline`] class and allow it to automatically detect the correct pipeline class from the checkpoint. +2. Load a pipeline with its specific pipeline class for a specific task. - + + -The [`DiffusionPipeline`] class is the simplest and most generic way to load the latest trending diffusion model from the [Hub](https://huggingface.co/models?library=diffusers&sort=trending). The [`DiffusionPipeline.from_pretrained`] method automatically detects the correct pipeline class from the checkpoint, downloads, and caches all the required configuration and weight files, and returns a pipeline instance ready for inference. +The [`DiffusionPipeline`] class is a simple and generic way to load the latest trending diffusion model from the [Hub](https://huggingface.co/models?library=diffusers&sort=trending). It uses the [`~DiffusionPipeline.from_pretrained`] method to automatically detect the correct pipeline class for a task from the checkpoint, downloads and caches all the required configuration and weight files, and returns a pipeline ready for inference. ```python from diffusers import DiffusionPipeline -repo_id = "runwayml/stable-diffusion-v1-5" -pipe = DiffusionPipeline.from_pretrained(repo_id, use_safetensors=True) +pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", use_safetensors=True) +``` + +This same checkpoint can also be used for an image-to-image task. The [`DiffusionPipeline`] class can handle any task as long as you provide the appropriate inputs. For example, for a specific task like image-to-image, you need to pass an initial image to the pipeline. + +```py +from diffusers import DiffusionPipeline + +pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", use_safetensors=True) + +init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/img2img-init.png") +prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" +image = pipeline("Astronaut in a jungle, cold color palette, muted colors, detailed, 8k", image=init_image).images[0] ``` -You can also load a checkpoint with its specific pipeline class. The example above loaded a Stable Diffusion model; to get the same result, use the [`StableDiffusionPipeline`] class: + + + +Checkpoints can also be loaded by their specific pipeline class if you already know it. For example, to load a Stable Diffusion model, use the [`StableDiffusionPipeline`] class. ```python from diffusers import StableDiffusionPipeline -repo_id = "runwayml/stable-diffusion-v1-5" -pipe = StableDiffusionPipeline.from_pretrained(repo_id, use_safetensors=True) +pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", use_safetensors=True) ``` -A checkpoint (such as [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) or [`runwayml/stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5)) may also be used for more than one task, like text-to-image or image-to-image. To differentiate what task you want to use the checkpoint for, you have to load it directly with its corresponding task-specific pipeline class: +This same checkpoint may also be used for another task like image-to-image. To differentiate what task you want to use the checkpoint for, you have to use the corresponding task-specific pipeline class. For example, to use the same checkpoint for image-to-image, use the [`StableDiffusionImg2ImgPipeline`] class. -```python +```py from diffusers import StableDiffusionImg2ImgPipeline -repo_id = "runwayml/stable-diffusion-v1-5" -pipe = StableDiffusionImg2ImgPipeline.from_pretrained(repo_id) +pipeline = StableDiffusionImg2ImgPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", use_safetensors=True) ``` -You can use the Space below to gauge the memory requirements of a pipeline you want to load beforehand without downloading the pipeline checkpoints: + + + +You can use the Space below to gauge a pipeline's memory requirements before you download and load it to see if it runs on your hardware.