Error while exporting wav2vec2 speech emotion recognition

[agunapal]

🐛 Describe the bug

Code to reproduce the problem

import torch

from transformers import AutoModelForAudioClassification, AutoFeatureExtractor

model = AutoModelForAudioClassification.from_pretrained("alefiury/wav2vec2-xls-r-300m-pt-br-spontaneous-speech-emotion-recognition")

model.eval()

input_values = torch.rand((1,8000))
attention_mask = torch.ones(1,8000)

exported_program: torch.export.ExportedProgram= torch.export.export(model, args=(input_values,attention_mask,), strict=False)

Error Logs

Traceback (most recent call last):
  File "/home/ubuntu/experiments/export/speech_emotion_detection.py", line 12, in <module>
    exported_program: torch.export.ExportedProgram= torch.export.export(model, args=(input_values,attention_mask,), strict=False)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/__init__.py", line 366, in export
    return _export(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/_trace.py", line 1014, in wrapper
    raise e
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/_trace.py", line 987, in wrapper
    ep = fn(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/exported_program.py", line 116, in wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/_trace.py", line 1964, in _export
    export_artifact = export_func(  # type: ignore[operator]
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/_trace.py", line 1754, in _non_strict_export
    aten_export_artifact = _to_aten_func(  # type: ignore[operator]
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/_trace.py", line 643, in _export_to_aten_ir
    gm, graph_signature = transform(aot_export_module)(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/_trace.py", line 1684, in _aot_export_non_strict
    gm, sig = aot_export(wrapped_mod, args, kwargs=kwargs, **flags)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1262, in aot_export_module
    fx_g, metadata, in_spec, out_spec = _aot_export_function(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1497, in _aot_export_function
    fx_g, meta = create_aot_dispatcher_function(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 524, in create_aot_dispatcher_function
    return _create_aot_dispatcher_function(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 762, in _create_aot_dispatcher_function
    compiled_fn, fw_metadata = compiler_fn(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 112, in aot_dispatch_export
    graph, _, _ = aot_dispatch_base_graph(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/dispatch_and_compile_graph.py", line 136, in aot_dispatch_base_graph
    fw_module = _create_graph(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/dispatch_and_compile_graph.py", line 55, in _create_graph
    fx_g = make_fx(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 2147, in wrapped
    return make_fx_tracer.trace(f, *args)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 2085, in trace
    return self._trace_inner(f, *args)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 2056, in _trace_inner
    t = dispatch_trace(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_compile.py", line 32, in inner
    return disable_fn(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 654, in _fn
    return fn(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1133, in dispatch_trace
    graph = tracer.trace(root, concrete_args)  # type: ignore[arg-type]
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1652, in trace
    res = super().trace(root, concrete_args)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 827, in trace
    (self.create_arg(fn(*args)),),
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1188, in wrapped
    out = f(*tensors)
  File "<string>", line 1, in <lambda>
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 693, in inner_fn
    outs = fn(*args)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 413, in _functionalized_f_helper
    f_outs = fn(*f_args)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 78, in inner_fn
    outs = fn(*args)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 184, in flat_fn
    tree_out = fn(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 863, in functional_call
    out = mod(*args[params_len:], **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 805, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1722, in call_module
    return Tracer.call_module(self, m, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 523, in call_module
    ret_val = forward(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 798, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/export/_trace.py", line 1671, in forward
    tree_out = mod(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 805, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1722, in call_module
    return Tracer.call_module(self, m, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 523, in call_module
    ret_val = forward(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 798, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 2358, in forward
    outputs = self.wav2vec2(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 805, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1722, in call_module
    return Tracer.call_module(self, m, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 523, in call_module
    ret_val = forward(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 798, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1823, in forward
    encoder_outputs = self.encoder(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 805, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1722, in call_module
    return Tracer.call_module(self, m, forward, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 523, in call_module
    ret_val = forward(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 798, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1111, in forward
    hidden_states[~expand_attention_mask] = 0
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1236, in __torch_function__
    return func(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/fx/experimental/proxy_tensor.py", line 1274, in __torch_function__
    return func(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_export/non_strict_utils.py", line 551, in __torch_function__
    return func(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_ops.py", line 840, in handler
    return torch._library.utils.handle_dispatch_mode(
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_library/utils.py", line 284, in handle_dispatch_mode
    return curr_mode.__torch_dispatch__(op_overload, overload_types, args, kwargs)
  File "/home/ubuntu/anaconda3/envs/export/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py", line 541, in __torch_dispatch__
    outs_unwrapped = func._op_dk(
RuntimeError: cannot mutate tensors with frozen storage

Versions

Collecting environment information...
PyTorch version: 2.6.0.dev20241015+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.0 | packaged by conda-forge | (default, Nov 20 2021, 02:24:10) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1063-aws-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      48 bits physical, 48 bits virtual
CPU(s):                             8
On-line CPU(s) list:                0-7
Thread(s) per core:                 2
Core(s) per socket:                 4
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          AuthenticAMD
CPU family:                         23
Model:                              49
Model name:                         AMD EPYC 7R32
Stepping:                           0
CPU MHz:                            2799.998
BogoMIPS:                           5599.99
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          128 KiB
L1i cache:                          128 KiB
L2 cache:                           2 MiB
L3 cache:                           16 MiB
NUMA node0 CPU(s):                  0-7
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid

Versions of relevant libraries:
[pip3] numpy==2.0.2
[pip3] torch==2.6.0.dev20241015+cpu
[pip3] torchvision==0.20.0.dev20241015+cpu
[conda] numpy                     2.0.2                    pypi_0    pypi
[conda] torch                     2.6.0.dev20241015+cpu          pypi_0    pypi
[conda] torchvision               0.20.0.dev20241015+cpu          pypi_0    pypi

cc @ezyang @chauhang @penguinwu @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan @angelayi @suo @ydwu4

[agunapal]

This is resolved by adding
hidden_states = hidden_states.clone()
in
https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1112

[yushangdi]

Hi, this is likely caused by the following: currently in export, we do not allow mutation to the output of aten.to and dropout (or any of its alias). This is because the aliasing behavior of them depends on runtime (for example for aten.to, it's sometimes a copy, sometimes an alias).

You fix of adding a clone is the right thing to do now.

We are working on relaxing this restriction for aten.to, and I can let you know after we landed the fix to relax this restriction.

[yushangdi]

@agunapal Hi, we've relaxed the restriction on clone in #142420, now your code should work fine. Please let me know if you are still encountering this issue.

[yushangdi]

Closing as it should have been resolved. Please re-open if you are still seeing this issue.