Error converting LoRA to safetensors/ckpt

[jndietz]

Generally speaking, I am not clear on what to do with the output of these LoRA python scripts. I don't think the output can be natively used by the webuis. Other LoRAs I've seen online are usually safetensors. Here is what I did...

I used the provided python script in examples to generate a LoRA:

#!/bin/sh

accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path=$1  \
  --instance_data_dir=$2 \
  --output_dir=$3 \
  --instance_prompt="a photo of laskajavids" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --checkpointing_steps=200 \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=1000 \
  --validation_prompt="A photo of laskajavids by the pool" \
  --validation_epochs=50 \
  --seed="0" \
  --mixed_precision="fp16" \
  --use_8bit_adam

I successfully (or not?) created a LoRA, and it output the following to /output:

checkpoint-1000
checkpoint-200
checkpoint-400
checkpoint-600
checkpoint-800
pytorch_lora_weights.bin

Then, I tried to run scripts\convert_diffusers_to_original_stable_diffusion.py, like so:

 python /diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py --model_path /output --checkpoint_path /test.ckpt --use_safetensors

I received the following error:

Traceback (most recent call last):
  File "/diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py", line 290, in <module>
    unet_state_dict = torch.load(unet_path, map_location="cpu")
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 771, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 270, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 251, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/output/unet/diffusion_pytorch_model.bin'

Did I miss something when creating the LoRA?

[jndietz]

I just tried running the newer script examples\text_to_image\train_text_to_image_lora.py, with the following

export MODEL_NAME=$1
export TRAIN_DIR=$2

accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$TRAIN_DIR --caption_column="additional_feature" \
  --resolution=512 --random_flip \
  --train_batch_size=1 \
  --validation_epochs=120 \
  --max_train_steps=1000 \
  --checkpointing_steps=100 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --output_dir=$3 \
  --validation_prompt="photo of laskajavids" \
  --enable_xformers_memory_efficient_attention

And the output appeared to be the same as examples/train_dreambooth_lora.py. So, I got the same error:

python /diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py --model_path /output --checkpoint_path /test.ckpt --use_safetensors
Traceback (most recent call last):
  File "/diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py", line 290, in <module>
    unet_state_dict = torch.load(unet_path, map_location="cpu")
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 771, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 270, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 251, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/output/unet/diffusion_pytorch_model.bin'

[jndietz]

Looking at the script itself, I'm seeing why this isn't working. My output directory isn't a model. What do I do with the output in there currently?

checkpoint-1000
checkpoint-200
checkpoint-400
checkpoint-600
checkpoint-800
pytorch_lora_weights.bin

[patrickvonplaten]

@williamberman could you take a look here?

[NicholasKao1029]

Following

[patrickvonplaten]

By other WebUIs was mostly Automatic1111 meant?

We should really think about whether we can somehow integrate diffusers into AUTO1111

[jndietz]

@patrickvonplaten Yes, I am speaking to automatic1111's webui specifically. Though, I want to be able to do this via a python script without having to use a webui.

I'll have to go back and make sure I followed the 🤗 docs. I do feel like I followed them closely, which is why I am confused as to how to use the outputted LoRAs and convert them into a safetensors or ckpt.

[patrickvonplaten]

You can use LoRA trained embeddings easily with diffusers, see: https://huggingface.co/docs/diffusers/v0.12.0/en/training/lora#lora-support-in-diffusers but I think it's not so easy to convert them to A1111.

[haofanwang]

Hi, @patrickvonplaten, does diffusers support convert or directly load from Lora+safetensor? I have tried with the converting scripts, but failed.

The weight is from civitai. Any help would be appreciated.

[patrickvonplaten]

I'll try to allocate time for this. Otherwise maybe @williamberman or @sayakpaul could check it out as well. I'm not 100% sure whether we can convert civitai lora weights

[sayakpaul]

W.r.t #2363 I think there are a couple of different conversion pathways we're talking about for completness:

• Diffusers LoRA weights to safetensors
• civitai weights to diffusers
• civitai LoRA weights to diffusers

@patrickvonplaten am I missing out on something?