Call to CUDA function failed. with DDP using 4 GPUs

[brando90]

🐛 Bug

Cannot run DDP with 4 gpus. Get error:

Traceback (most recent call last):
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 277, in <module>
    main_distributed()
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 187, in main_distributed
    spawn_return = mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 3 terminated with the following error:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 210, in train
    tactic_predictor = move_to_ddp(rank, opts, tactic_predictor)
  File "/home/miranda9/ultimate-utils/ultimate-utils-project/uutils/torch/distributed.py", line 162, in move_to_ddp
    model = DistributedDataParallel(model, find_unused_parameters=True, device_ids=[opts.gpu])
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 446, in __init__
    self._sync_params_and_buffers(authoritative_rank=0)
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 457, in _sync_params_and_buffers
    self._distributed_broadcast_coalesced(
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1155, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1614378083779/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled cuda error, NCCL version 2.7.8
ncclUnhandledCudaError: Call to CUDA function failed.

To Reproduce

To reproduce train any DDP model with 4 gpus of type ('gpu_name', 'GeForce GTX TITAN X')

Output from nvidia-smi

$ nvidia-smi
Tue Mar 23 17:13:32 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 450.36.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:04:00.0 Off |                  N/A |
| 22%   25C    P8    14W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 00000000:05:00.0 Off |                  N/A |
| 22%   36C    P2    69W / 250W |   5602MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 00000000:08:00.0 Off |                  N/A |
| 22%   54C    P2    76W / 250W |   1752MiB / 12212MiB |     12%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 00000000:09:00.0 Off |                  N/A |
| 22%   29C    P8    15W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX TIT...  Off  | 00000000:84:00.0 Off |                  N/A |
| 22%   24C    P8    14W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX TIT...  Off  | 00000000:85:00.0 Off |                  N/A |
| 22%   24C    P8    14W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX TIT...  Off  | 00000000:88:00.0 Off |                  N/A |
| 22%   23C    P8    15W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  GeForce GTX TIT...  Off  | 00000000:89:00.0 Off |                  N/A |
| 22%   24C    P8    16W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    1   N/A  N/A     16689      C   ...3/envs/repaint/bin/python     1610MiB |
|    1   N/A  N/A     28647      C   ...3/envs/repaint/bin/python     3988MiB |
|    2   N/A  N/A     23374      C   python                           1749MiB |
+-----------------------------------------------------------------------------+

Expected behavior

Environment

$ python collect_env.py
Collecting environment information...
PyTorch version: 1.8.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Clang version: 3.4.2 (tags/RELEASE_34/dot2-final)
CMake version: version 2.8.12.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: 
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X
GPU 2: GeForce GTX TITAN X
GPU 3: GeForce GTX TITAN X
GPU 4: GeForce GTX TITAN X
GPU 5: GeForce GTX TITAN X
GPU 6: GeForce GTX TITAN X
GPU 7: GeForce GTX TITAN X

Nvidia driver version: 450.36.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.8.0
[pip3] torchaudio==0.8.0a0+a751e1d
[pip3] torchmeta==1.6.1
[pip3] torchtext==0.9.0
[pip3] torchvision==0.9.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.1               h6406543_8    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.8.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] torchaudio                0.8.0                      py38    pytorch
[conda] torchmeta                 1.6.1                    pypi_0    pypi
[conda] torchtext                 0.9.0                      py38    pytorch
[conda] torchvision               0.9.0                py38_cu111    pytorch

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu

[brando90]

related: #47885

[brando90]

(although not the same it seems related: https://stackoverflow.com/questions/61075390/pytorch-nccl-error-unhandled-system-error-nccl-version-2-4-8)

[wayi1]

Have you called torch.cuda.set_device() to set a different GPU for every process?

Do you have a chance to try any other GPU types that is not TITAN X?

[brando90]

Have you called torch.cuda.set_device() to set a different GPU for every process?

I do not use that as it's not in any of the tutorials and now that looked it up its discouraged in the docs: https://pytorch.org/docs/stable/cuda.html

did you mean perhaps:

torch.cuda.device(device)

though that is not used in any of the tutorials that I've seen either and I've gone through most of them in a lot of detail...I simply get the device with

    opts.gpu = torch.device("cuda" if torch.cuda.is_available() else "cpu") if opts.debug else opts.gpu

and then use that as the id that I set to the DDP.

Do you have a chance to try any other GPU types that is not TITAN X?

will try that.

[brando90]

Have you called torch.cuda.set_device() to set a different GPU for every process?

Do you have a chance to try any other GPU types that is not TITAN X?

seems like it fails only with this GPU

requirements = (CUDADeviceName != "GeForce GTX TITAN X")

[brando90]

Have you called torch.cuda.set_device() to set a different GPU for every process?
Do you have a chance to try any other GPU types that is not TITAN X?

seems like it fails only with this GPU

requirements = (CUDADeviceName != "GeForce GTX TITAN X")

all other GPUs seem to work:

Add a line specifying `requirement` in the submission file.

* Specific kind of GPU: `requirements = (CUDADeviceName == "GeForce GTX TITAN X")`
  * K40: `"Tesla K40m"`
  * Titan X Maxwell: `"GeForce GTX TITAN X"`
  * Titan X Pascal (2016): `"TITAN X (Pascal)"`
  * Titan Xp (2017): `"TITAN Xp"`
  * Quadro RTX 6000: `"Quadro RTX 6000"`
  * See `condor_status --compact -af:h Machine TotalGpus CUDADeviceName` for a full list.
  * ```

[brando90]

k40s were not tested

[brando90]

perhaps this is related to: Lightning-AI/pytorch-lightning#4420

[brando90]

Have you called torch.cuda.set_device() to set a different GPU for every process?

Do you have a chance to try any other GPU types that is not TITAN X?

actually it seems that things get into a deadlock and it only happens when I run the job not in an interactive mode on my HPC.

(automl-meta-learning) miranda9~/ML4Coq $ tail -f main.sh.o4721 
('current_time', 'Mar24_15-48-54')
('tb_dir', PosixPath('/home/miranda9/data/logs/logs_Mar24_15-48-54_jobid_4721/tb'))
('gpu_name', 'GeForce GTX TITAN X')
('PID', '18688')
torch.cuda.device_count()=2

opts.world_size=2

ABOUT TO SPAWN WORKERS
done setting sharing strategy...next mp.spawn

[brando90]

I also tried downgrading cuda version since 10.2 seemed to work fine but I still get into that deadlock:

$ tail -f main.sh.o4724
JOB STARTED
# packages in environment at /home/miranda9/miniconda3/envs/automl-meta-learning:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   0.11.0           py38h06a4308_0  
aioconsole                0.3.1                    pypi_0    pypi
aiohttp                   3.7.2            py38h27cfd23_1  
anatome                   0.0.1                    pypi_0    pypi
async-timeout             3.0.1                    py38_0  
attrs                     20.3.0             pyhd3eb1b0_0  
beautifulsoup4            4.9.3              pyhb0f4dca_0  
blas                      1.0                         mkl  
blinker                   1.4                      py38_0  
brotlipy                  0.7.0           py38h27cfd23_1003  
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.17.1               h27cfd23_0  
ca-certificates           2021.1.19            h06a4308_1  
cachetools                4.1.1                      py_0  
cairo                     1.14.12              h8948797_3  
certifi                   2020.12.5        py38h578d9bd_1    conda-forge
cffi                      1.14.0           py38h2e261b9_0  
chardet                   3.0.4           py38h06a4308_1003  
click                     7.1.2                      py_0  
conda                     4.9.2            py38h06a4308_0  
conda-build               3.20.5                   py38_1  
conda-package-handling    1.7.2            py38h03888b9_0  
cryptography              3.2.1            py38h3c74f83_1  
cudatoolkit               10.2.89              hfd86e86_1  
cycler                    0.10.0                   py38_0  
cython                    0.29.22                  pypi_0    pypi
dataclasses               0.6                      pypi_0    pypi
dbus                      1.13.18              hb2f20db_0  
dill                      0.3.3              pyhd3eb1b0_0  
expat                     2.2.10               he6710b0_2  
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.0.12                     py_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.10.4               h5ab3b9f_0  
fribidi                   1.0.10               h7b6447c_0  
future                    0.18.2                   pypi_0    pypi
glib                      2.63.1               h5a9c865_0  
glob2                     0.7                        py_0  
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               1.23.0             pyhd3eb1b0_0  
google-auth-oauthlib      0.4.2              pyhd3eb1b0_2  
graphite2                 1.3.14               h23475e2_0  
graphviz                  2.40.1               h21bd128_2  
grpcio                    1.31.0           py38hf8bcb03_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb31296c_0  
h5py                      3.1.0                    pypi_0    pypi
harfbuzz                  1.8.8                hffaf4a1_0  
higher                    0.2.1                    pypi_0    pypi
icu                       58.2                 he6710b0_3  
idna                      2.10                       py_0  
importlib-metadata        2.0.0                      py_1  
intel-openmp              2020.2                      254  
jinja2                    2.11.2                     py_0  
joblib                    0.17.0                     py_0  
jpeg                      9b                   h024ee3a_2  
kiwisolver                1.3.0            py38h2531618_0  
lame                      3.100             h7f98852_1001    conda-forge
lark                      0.11.1                   pypi_0    pypi
lcms2                     2.11                 h396b838_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libarchive                3.4.2                h62408e4_0  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.2.1             hf484d3e_1007  
libgcc-ng                 9.3.0               h2828fa1_18    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0  
libgomp                   9.3.0               h2828fa1_18    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblief                   0.10.1               he6710b0_0  
libpng                    1.6.37               hbc83047_0  
libprotobuf               3.13.0.1             hd408876_0  
libstdcxx-ng              9.3.0               h6de172a_18    conda-forge
libtiff                   4.1.0                h2733197_1  
libuuid                   1.0.3                h1bed415_2  
libuv                     1.40.0               h7b6447c_0  
libxcb                    1.14                 h7b6447c_0  
libxml2                   2.9.10               hb55368b_3  
lmdb                      1.1.1                    pypi_0    pypi
lz4-c                     1.9.2                heb0550a_3  
markdown                  3.3.3            py38h06a4308_0  
markupsafe                1.1.1            py38h7b6447c_0  
matplotlib                3.3.2                         0  
matplotlib-base           3.3.2            py38h817c723_0  
mkl                       2020.2                      256  
mkl-service               2.3.0            py38he904b0f_0  
mkl_fft                   1.2.0            py38h23d657b_0  
mkl_random                1.1.1            py38h0573a6f_0  
multidict                 4.7.6            py38h7b6447c_1  
ncurses                   6.2                  he6710b0_1  
nettle                    3.6                  he412f7d_0    conda-forge
ninja                     1.10.2           py38hff7bd54_0  
numpy                     1.19.2           py38h54aff64_0  
numpy-base                1.19.2           py38hfa32c7d_0  
oauthlib                  3.1.0                      py_0  
olefile                   0.46                       py_0  
openh264                  2.1.1                h780b84a_0    conda-forge
openssl                   1.1.1j               h27cfd23_0  
ordered-set               4.0.2                    pypi_0    pypi
pandas                    1.1.3            py38he6710b0_0  
pango                     1.42.4               h049681c_0  
patchelf                  0.12                 he6710b0_0  
pcre                      8.44                 he6710b0_0  
pexpect                   4.8.0                    pypi_0    pypi
pillow                    8.0.1            py38he98fc37_0  
pip                       20.3             py38h06a4308_0  
pixman                    0.40.0               h7b6447c_0  
pkginfo                   1.6.1            py38h06a4308_0  
progressbar               2.5                      pypi_0    pypi
progressbar2              3.53.1                   pypi_0    pypi
protobuf                  3.13.0.1         py38he6710b0_1  
psutil                    5.7.2            py38h7b6447c_0  
ptyprocess                0.7.0                    pypi_0    pypi
py-lief                   0.10.1           py38h403a769_0  
pyasn1                    0.4.8                      py_0  
pyasn1-modules            0.2.8                      py_0  
pycapnp                   1.0.0                    pypi_0    pypi
pycosat                   0.6.3            py38h7b6447c_1  
pycparser                 2.20                       py_2  
pyjwt                     1.7.1                    py38_0  
pyopenssl                 20.0.0             pyhd3eb1b0_1  
pyparsing                 2.4.7                      py_0  
pyqt                      5.9.2            py38h05f1152_4  
pysocks                   1.7.1            py38h06a4308_0  
python                    3.8.2                hcf32534_0  
python-dateutil           2.8.1                      py_0  
python-libarchive-c       2.9                        py_0  
python-utils              2.5.6                    pypi_0    pypi
python_abi                3.8                      1_cp38    conda-forge
pytorch                   1.8.0           py3.8_cuda10.2_cudnn7.6.5_0    pytorch
pytz                      2020.4             pyhd3eb1b0_0  
pyyaml                    5.3.1            py38h7b6447c_1  
qt                        5.9.7                h5867ecd_1  
readline                  8.0                  h7b6447c_0  
requests                  2.25.0             pyhd3eb1b0_0  
requests-oauthlib         1.3.0                      py_0  
ripgrep                   12.1.1                        0  
rsa                       4.6                        py_0  
ruamel_yaml               0.15.87          py38h7b6447c_1  
scikit-learn              0.23.2           py38h0573a6f_0  
scipy                     1.5.2            py38h0b6359f_0  
setuptools                50.3.1           py38h06a4308_1  
sexpdata                  0.0.3                    pypi_0    pypi
sip                       4.19.13          py38he6710b0_0  
six                       1.15.0           py38h06a4308_0  
soupsieve                 2.0.1                      py_0  
sqlite                    3.33.0               h62c20be_0  
tensorboard               2.3.0              pyh4dce500_0  
tensorboard-plugin-wit    1.6.0                      py_0  
threadpoolctl             2.1.0              pyh5ca1d4c_0  
tk                        8.6.10               hbc83047_0  
torchaudio                0.8.0                      py38    pytorch
torchmeta                 1.6.1                    pypi_0    pypi
torchtext                 0.9.0                      py38    pytorch
torchvision               0.9.0                py38_cu102    pytorch
tornado                   6.0.4            py38h7b6447c_1  
tqdm                      4.54.0                   pypi_0    pypi
typing-extensions         3.7.4.3                       0  
typing_extensions         3.7.4.3                    py_0  
urllib3                   1.26.2                   pypi_0    pypi
werkzeug                  1.0.1                      py_0  
wheel                     0.35.1             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
yaml                      0.2.5                h7b6447c_0  
yarl                      1.5.1            py38h7b6447c_0  
zipp                      3.4.0              pyhd3eb1b0_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.5                h9ceee32_0  

perhaps I should upgrade again?

[brando90]

Perhaps we can make some progress with this question:

What version of pytorch and cuda version should things be working? Is it the one the install from the official website is i.e. 1.8 and 11.1?

[brando90]

I have it deadlock on the Quadro RTX 6000 too:

$ tail -f main.sh.o4800
('gpu_name', 'Quadro RTX 6000')
('PID', '106073')
torch.cuda.device_count()=2

opts.world_size=2

ABOUT TO SPAWN WORKERS
done setting sharing strategy...next mp.spawn
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 0

[brando90]

it also deadlocks with this GPU one:

$ tail -f main_brando.py.o4807
('current_time', 'Mar25_12-48-52')
('tb_dir', PosixPath('/home/miranda9/data/logs/logs_Mar25_12-48-52_jobid_4807/tb'))
('gpu_name', 'TITAN X (Pascal)')
('PID', '10905')
torch.cuda.device_count()=2

opts.world_size=2

ABOUT TO SPAWN WORKERS
done setting sharing strategy...next mp.spawn
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 0