IP-Adapter-FaceID and FaceID plus not support

[zhaoyun0071]

Describe the bug

diffusers\loaders\unet.py", line 780, in _load_ip_adapter_weights
    num_image_text_embeds = state_dict["image_proj"]["latents"].shape[1]
KeyError: 'latents'

Reproduction

import cv2
from insightface.app import FaceAnalysis
import numpy as np
from PIL import Image
import torch
from diffusers import StableDiffusionPipeline, DDIMScheduler
from diffusers.utils import load_image


def image_grid(imgs, rows, cols):
    assert len(imgs) == rows * cols

    w, h = imgs[0].size
    grid = Image.new('RGB', size=(cols * w, rows * h))
    grid_w, grid_h = grid.size

    for i, img in enumerate(imgs):
        grid.paste(img, box=(i % cols * w, i // cols * h))
    return grid


noise_scheduler = DDIMScheduler(
    num_train_timesteps=1000,
    beta_start=0.00085,
    beta_end=0.012,
    beta_schedule="scaled_linear",
    clip_sample=False,
    set_alpha_to_one=False,
    steps_offset=1
)

pipeline = StableDiffusionPipeline.from_single_file(
    r"Realistic_Vision_V4.0.safetensors",
    torch_dtype=torch.float16,
    scheduler=noise_scheduler,
    feature_extractor=None,
    load_safety_checker=False
).to("cuda")

generator = torch.Generator(device="cpu").manual_seed(42)
num_images = 4
image = load_image("zly.jpg")

app = FaceAnalysis(name="buffalo_l", providers=['CPUExecutionProvider'], root='./')
app.prepare(ctx_id=0, det_size=(640, 640))

image = cv2.cvtColor(np.asarray(image), cv2.COLOR_BGR2RGB)
faces = app.get(image)
image = torch.from_numpy(faces[0].normed_embedding).unsqueeze(0)

pipeline.load_ip_adapter(r'./data/models/ip_adapter/', subfolder='models', weight_name="ip-adapter-faceid_sd15.bin")
# pipeline.load_ip_adapter("h94/IP-Adapter-FaceID", subfolder='',weight_name="ip-adapter-faceid_sd15.bin")
pipeline.set_ip_adapter_scale(0.7)

Logs

No response

System Info

Diffusers 0.25.0

Who can help?

@yiyixuxu @DN6 @sayakpaul @patrickvonplaten

[fabiorigano]

Hi @zhaoyun0071, diffusers 0.25.0 does not support the two models you mentioned because they are experimental versions. PR #6276 is adding support for IPAdapter FaceID.

[zhaoyun0071]

thanks

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[katarzynasornat]

Hi @zhaoyun0071, diffusers 0.25.0 does not support the two models you mentioned because they are experimental versions. PR #6276 is adding support for IPAdapter FaceID.

Hi @fabiorigano , does diffusers 0.25.0 and higher support it now? I still do have the same error when loading the models.

[fabiorigano]

hi @katarzynasornat, all Face ID models are supported with #7186, and thus starting with diffusers 0.28 (dev, the current version at the time of this writing)

[katarzynasornat]

hi @katarzynasornat, all Face ID models are supported with #7186, and thus starting with diffusers 0.28 (dev, the current version at the time of this writing)

EDIT: I think there is a typo in the example (red marked). It should be ip_adapter_image_embeds instead of ip_adapter_image :)

@fabiorigano thanks for your reply! I have tried to follow #7186 with the code provided, however I am getting an error:

AttributeError: 'NoneType' object has no attribute 'parameters'

Is that a normal behaviour? Did I miss something?

I attached my gist with execution below.

[fabiorigano]

@katarzynasornat when using FaceID, pass ip_adapter_image_embeds instead of ip_adapter_image. Follow the docs https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter#face-model (snippet after Full Face example)

[fabiorigano]

EDIT: I think there is a typo in the example (red marked). It should be ip_adapter_image_embeds instead of ip_adapter_image :)

yes, but don't follow PR's examples, as we changed few things during integration
docs are updated :)

[qwerdf4]

only support sd1.5，sdxl not support

[juancopi81]

Hi @fabiorigano ,

Thanks for the great work! I was wondering if the docs are updated. I copied and pasted this (I am using the ip-adapter-faceid-plusv2_sd15.bin ckpt)

from insightface.utils import face_align

num_images = 1 # Not in the docs

ref_images_embeds = []
ip_adapter_images = []
app = FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))
image = cv2.cvtColor(np.asarray(image), cv2.COLOR_BGR2RGB)
faces = app.get(image)
ip_adapter_images.append(face_align.norm_crop(image, landmark=faces[0].kps, image_size=224))
image = torch.from_numpy(faces[0].normed_embedding)
ref_images_embeds.append(image.unsqueeze(0))
ref_images_embeds = torch.stack(ref_images_embeds, dim=0).unsqueeze(0)
neg_ref_images_embeds = torch.zeros_like(ref_images_embeds)
id_embeds = torch.cat([neg_ref_images_embeds, ref_images_embeds]).to(dtype=torch.float16, device="cuda")

clip_embeds = pipeline.prepare_ip_adapter_image_embeds(
  [ip_adapter_images], None, torch.device("cuda"), num_images, True)[0]

pipeline.unet.encoder_hid_proj.image_projection_layers[0].clip_embeds = clip_embeds.to(dtype=torch.float16)
pipeline.unet.encoder_hid_proj.image_projection_layers[0].shortcut = True # True if Plus v2

And I am getting:

AttributeError                            Traceback (most recent call last)
<ipython-input-9-0a5647ac6e53> in <cell line: 18>()
     16 id_embeds = torch.cat([neg_ref_images_embeds, ref_images_embeds]).to(dtype=torch.float16, device="cuda")
     17 
---> 18 clip_embeds = pipeline.prepare_ip_adapter_image_embeds(
     19   [ip_adapter_images], None, torch.device("cuda"), num_images, True)[0]
     20 

1 frames
/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py in prepare_ip_adapter_image_embeds(self, ip_adapter_image, ip_adapter_image_embeds, device, num_images_per_prompt, do_classifier_free_guidance)
    522             ):
    523                 output_hidden_state = not isinstance(image_proj_layer, ImageProjection)
--> 524                 single_image_embeds, single_negative_image_embeds = self.encode_image(
    525                     single_ip_adapter_image, device, 1, output_hidden_state
    526                 )

/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py in encode_image(self, image, device, num_images_per_prompt, output_hidden_states)
    482 
    483     def encode_image(self, image, device, num_images_per_prompt, output_hidden_states=None):
--> 484         dtype = next(self.image_encoder.parameters()).dtype
    485 
    486         if not isinstance(image, torch.Tensor):

AttributeError: 'NoneType' object has no attribute 'parameters'

I also tried:

# ADDING ip_embeds
clip_embeds = pipeline.prepare_ip_adapter_image_embeds(
  [ip_adapter_images], id_embeds, torch.device("cuda"), num_images, True)[0]

And got:

ValueError                                Traceback (most recent call last)
[<ipython-input-11-837bd85b00a7>](https://localhost:8080/#) in <cell line: 19>()
     17 
     18 # ADDING ip_embeds
---> 19 clip_embeds = pipeline.prepare_ip_adapter_image_embeds(
     20   [ip_adapter_images], id_embeds, torch.device("cuda"), num_images, True)[0]
     21 

[/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py](https://localhost:8080/#) in prepare_ip_adapter_image_embeds(self, ip_adapter_image, ip_adapter_image_embeds, device, num_images_per_prompt, do_classifier_free_guidance)
    540             for single_image_embeds in ip_adapter_image_embeds:
    541                 if do_classifier_free_guidance:
--> 542                     single_negative_image_embeds, single_image_embeds = single_image_embeds.chunk(2)
    543                     single_image_embeds = single_image_embeds.repeat(
    544                         num_images_per_prompt, *(repeat_dims * len(single_image_embeds.shape[1:]))

ValueError: not enough values to unpack (expected 2, got 1)

Sorry if I am missing something obvious. Here is my colab:

[fabiorigano]

hi @juancopi81
docs are updated, you are not properly loading the weights, as all Face ID Plus models need a CLIP image encoder. See https://huggingface.co/docs/diffusers/using-diffusers/loading_adapters#ip-adapter-face-id-models or copy-paste the following snippet (you can find it in docs too):

from transformers import CLIPVisionModelWithProjection

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "laion/CLIP-ViT-H-14-laion2B-s32B-b79K",
    torch_dtype=torch.float16,
)

pipeline = AutoPipelineForText2Image.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    image_encoder=image_encoder,
    torch_dtype=torch.float16
).to("cuda")

pipeline.load_ip_adapter("h94/IP-Adapter-FaceID", subfolder=None, weight_name="ip-adapter-faceid-plusv2_sd15.bin")

[juancopi81]

Thanks @fabiorigano!! Maybe adding that to this docs would be helpful? https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter#face-model At least for me, it would have been helpful. If you think so, I could create a PR to add that.Thanks again!