Cannot copy out of meta tensor; no data! with CogView4Pipeline

[nitinmukesh]

Describe the bug

Getting error while using enable_sequential_cpu_offload.
The models are used without quantization so it should work.

Reproduction

from diffusers import CogView4Pipeline
import torch

pipe = CogView4Pipeline.from_pretrained("THUDM/CogView4-6B", torch_dtype=torch.bfloat16)
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

prompt = "A vibrant cherry red sports car sits proudly under the gleaming sun, its polished exterior smooth and flawless, casting a mirror-like reflection. The car features a low, aerodynamic body, angular headlights that gaze forward like predatory eyes, and a set of black, high-gloss racing rims that contrast starkly with the red. A subtle hint of chrome embellishes the grille and exhaust, while the tinted windows suggest a luxurious and private interior. The scene conveys a sense of speed and elegance, the car appearing as if it's about to burst into a sprint along a coastal road, with the ocean's azure waves crashing in the background."
image = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    num_images_per_prompt=1,
    num_inference_steps=50,
    width=1024,
    height=1024,
).images[0]

image.save("cogview4.png")

Logs

(venv) C:\aiOWN\diffuser_webui>python cogview4.py
Loading checkpoint shards: 100%|████████████████████████████████████| 3/3 [00:00<00:00, 19.10it/s]
Loading checkpoint shards: 100%|████████████████████████████████████| 4/4 [00:00<00:00,  4.69it/s]
Loading pipeline components...: 100%|███████████████████████████████| 5/5 [00:02<00:00,  1.79it/s]
Traceback (most recent call last):
  File "C:\aiOWN\diffuser_webui\cogview4.py", line 10, in <module>
    image = pipe(
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\diffusers\pipelines\cogview4\pipeline_cogview4.py", line 538, in __call__
    prompt_embeds, negative_prompt_embeds = self.encode_prompt(
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\diffusers\pipelines\cogview4\pipeline_cogview4.py", line 273, in encode_prompt
    prompt_embeds = self._get_glm_embeds(prompt, max_sequence_length, device, dtype)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\diffusers\pipelines\cogview4\pipeline_cogview4.py", line 217, in _get_glm_embeds
    prompt_embeds = self.text_encoder(
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\accelerate\hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\transformers\models\glm\modeling_glm.py", line 561, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\accelerate\hooks.py", line 171, in new_forward
    args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\accelerate\hooks.py", line 370, in pre_forward
    return send_to_device(args, self.execution_device), send_to_device(
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\accelerate\utils\operations.py", line 174, in send_to_device
    return honor_type(
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\accelerate\utils\operations.py", line 81, in honor_type
    return type(obj)(generator)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\accelerate\utils\operations.py", line 175, in <genexpr>
    tensor, (send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys) for t in tensor)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\accelerate\utils\operations.py", line 155, in send_to_device
    return tensor.to(device, non_blocking=non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

System Info

• 🤗 Diffusers version: 0.33.0.dev0
• Platform: Windows-10-10.0.26100-SP0
• Running on Google Colab?: No
• Python version: 3.10.11
• PyTorch version (GPU?): 2.5.1+cu124 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.27.1
• Transformers version: 4.49.0
• Accelerate version: 1.4.0.dev0
• PEFT version: 0.14.1.dev0
• Bitsandbytes version: 0.45.3
• Safetensors version: 0.5.2
• xFormers version: not installed
• Accelerator: NVIDIA GeForce RTX 4060 Laptop GPU, 8188 MiB
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@DN6

[a-r-r-o-w]

@nitinmukesh Could you try with model/group offloading instead of sequential offloading? I notice the same error and spoke with @zRzRzRzRzRzRzR. We're looking into it but seems like the glm text encoder does not work with sequential offloading

[nitinmukesh]

Thanks for the suggestion @a-r-r-o-w . Unfortunately enable_model_cpu_offload is coming with CUDA OOM but no error reported.
group offloading requires a lot of RAM so not possible for me to test.

Let me try nf4 quantization and enable_model_cpu_offload.

[nitinmukesh]

Not sure why nf4 is producing only noise, this is first time with any model. Gonna test more.

VRAM during inference: 5 GB
VRAM during vae: 9 GB

from diffusers import CogView4Pipeline, CogView4Transformer2DModel
import torch
from transformers import GlmModel
from diffusers import BitsAndBytesConfig

quant_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)

text_encoder = GlmModel.from_pretrained(
    "THUDM/CogView4-6B", 
    subfolder="text_encoder",
    quantization_config=quant_config,
    torch_dtype=torch.bfloat16,
)
transformer = CogView4Transformer2DModel.from_pretrained(
    "THUDM/CogView4-6B", 
    subfolder="transformer",
    quantization_config=quant_config,
    torch_dtype=torch.bfloat16,
)
pipe = CogView4Pipeline.from_pretrained(
    "THUDM/CogView4-6B",
    text_encoder=text_encoder,
    transformer=transformer,
    torch_dtype=torch.bfloat16
)

pipe.enable_model_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

prompt = "A vibrant cherry red sports car sits proudly under the gleaming sun, its polished exterior smooth and flawless, casting a mirror-like reflection. The car features a low, aerodynamic body, angular headlights that gaze forward like predatory eyes, and a set of black, high-gloss racing rims that contrast starkly with the red. A subtle hint of chrome embellishes the grille and exhaust, while the tinted windows suggest a luxurious and private interior. The scene conveys a sense of speed and elegance, the car appearing as if it's about to burst into a sprint along a coastal road, with the ocean's azure waves crashing in the background."
image = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    num_images_per_prompt=1,
    num_inference_steps=50,
    width=1024,
    height=1024,
).images[0]

image.save("cogview4.png")

[hlky]

Can you try transformers==4.48.0 or main?

[nitinmukesh]

Thank you, I am testing now.

Test 1: Installed diffusers from main and transformers==4.48.0

Case 1 (enable_sequential_cpu_offload): Cannot copy out of meta tensor; no data!
Case 2 (nf4): Same output as above

Test 2: Installed diffusers from main and transformers from main
pip install git+https://github.com/huggingface/transformers.git
transformers==4.50.0.dev0

Case 1 (enable_sequential_cpu_offload): Cannot copy out of meta tensor; no data!
Case 2 (nf4): Same output as above

[Willian7004]

@nitinmukesh Could you try with model/group offloading instead of sequential offloading? I notice the same error and spoke with @zRzRzRzRzRzRzR. We're looking into it but seems like the glm text encoder does not work with sequential offloading

Maybe it's not only a problem about text encoder. I tried to offload text encoder to cpu and run transtormer with sequential_cpu_offload using the code below:

from diffusers import CogView4Pipeline
import torch
import torch.quantization

# 定义文本编码器包装器，处理设备转移和量化
class QuantizedCPUTextEncoder(torch.nn.Module):
    def __init__(self, text_encoder):
        super().__init__()
        self.text_encoder = text_encoder
        # 应用动态int8量化
        self.text_encoder = torch.quantization.quantize_dynamic(
            self.text_encoder, {torch.nn.Linear}, dtype=torch.qint8
        ).to('cpu').eval()

    def forward(self, *args, **kwargs):
        # 确保输入在CPU
        args = [a.to('cpu') if isinstance(a, torch.Tensor) else a for a in args]
        kwargs = {k: v.to('cpu') if isinstance(v, torch.Tensor) else v for k, v in kwargs.items()}
        # 执行前向计算
        outputs = self.text_encoder(*args, **kwargs)
        # 将输出转移到GPU
        if isinstance(outputs, torch.Tensor):
            return outputs.to('cuda')
        elif isinstance(outputs, tuple):
            return tuple(o.to('cuda') if isinstance(o, torch.Tensor) else o for o in outputs)
        elif isinstance(outputs, dict):
            return {k: v.to('cuda') if isinstance(v, torch.Tensor) else v for k, v in outputs.items()}
        return outputs

# 加载模型
pipe = CogView4Pipeline.from_pretrained("./CogView4-6B", torch_dtype=torch.bfloat16)

# 替换文本编码器为量化版并绑定到CPU
pipe.text_encoder = QuantizedCPUTextEncoder(pipe.text_encoder)

# 启用GPU组件的内存优化
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

# 生成图像
prompt = "A serene landscape with mountains and a river"
image = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    num_images_per_prompt=1,
    num_inference_steps=50,
    width=1920,
    height=1088,
).images[0]

image.save("cogview4.png")

but I got error:

Traceback (most recent call last):
  File "/home/william/下载/CogView4/main.py", line 37, in <module>
    pipe.enable_sequential_cpu_offload()
  File "/home/william/miniconda3/envs/sana/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1236, in enable_sequential_cpu_offload
    if self.device.type != "cpu":
  File "/home/william/miniconda3/envs/sana/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 143, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'CogView4Pipeline' object has no attribute 'device'

then I set pipe.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") but got:

Traceback (most recent call last):
  File "/home/william/下载/CogView4/main.py", line 26, in <module>
    pipe.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 
  File "/home/william/miniconda3/envs/sana/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 194, in __setattr__
    super().__setattr__(name, value)
AttributeError: can't set attribute 'device'

[hlky]

Hi @Willian7004. QuantizedCPUTextEncoder is the issue in your case, without it enable_sequential_cpu_offload is working. We would recommend using one of the supported quantization backends.

import torch
from diffusers import CogView4Pipeline

pipe = CogView4Pipeline.from_pretrained("THUDM/CogView4-6B", torch_dtype=torch.bfloat16)
pipe.enable_sequential_cpu_offload()

import torch
from diffusers import CogView4Pipeline


class QuantizedCPUTextEncoder(torch.nn.Module):
    def __init__(self, text_encoder):
        super().__init__()
        self.text_encoder = text_encoder
        self.text_encoder = (
            torch.quantization.quantize_dynamic(
                self.text_encoder, {torch.nn.Linear}, dtype=torch.qint8
            )
            .to("cpu")
            .eval()
        )

    def forward(self, *args, **kwargs):
        args = [a.to("cpu") if isinstance(a, torch.Tensor) else a for a in args]
        kwargs = {
            k: v.to("cpu") if isinstance(v, torch.Tensor) else v
            for k, v in kwargs.items()
        }
        outputs = self.text_encoder(*args, **kwargs)
        if isinstance(outputs, torch.Tensor):
            return outputs.to("cuda")
        elif isinstance(outputs, tuple):
            return tuple(
                o.to("cuda") if isinstance(o, torch.Tensor) else o for o in outputs
            )
        elif isinstance(outputs, dict):
            return {
                k: v.to("cuda") if isinstance(v, torch.Tensor) else v
                for k, v in outputs.items()
            }
        return outputs


pipe = CogView4Pipeline.from_pretrained("THUDM/CogView4-6B", torch_dtype=torch.bfloat16)
pipe.text_encoder = QuantizedCPUTextEncoder(pipe.text_encoder)

pipe.enable_sequential_cpu_offload()

# AttributeError: 'CogView4Pipeline' object has no attribute 'device'

@nitinmukesh Can you try #11195?

[nitinmukesh]

Thank you @hlky . I will try and post results.

[nitinmukesh]

@hlky

It's working, thank you.