segfault in docker when used with torch and matplotlib

[ppwwyyxx]

Steps to reproduce

╰─$cat Dockerfile
FROM centos/python-36-centos7
USER root
WORKDIR /root
RUN yum update -y
RUN pip install --upgrade pip
RUN pip install opencv-python
RUN pip install -U matplotlib
RUN pip install torch==1.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
COPY a.py /root/a.py
RUN python -X faulthandler a.py

╰─$cat a.py
import torch
try:
    import cv2
except:
    pass
import matplotlib.pyplot as plt

╰─$docker build   -t test:v0 .      

..................
Step 10/10 : RUN python -X faulthandler a.py
 ---> Running in 1b7bc36c9727
Fatal Python error: Segmentation fault

Thread 0x00007f185a430740 (most recent call first):
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/json/encoder.py", line 356 in _iterencode_dict
The command '/bin/sh -c python -X faulthandler a.py' returned a non-zero code: 139

It works if I use opencv-python-headless instead.

Issue submission checklist

• This is not a generic OpenCV usage question (looking for help for coding, other usage questions, homework etc.)
• I have read the README of this repository and understand that this repository provides only an automated build toolchain for OpenCV Python packages (there is no actual OpenCV code here)
• The issue is related to the build scripts in this repository, to the pre-built binaries or is a feature request (such as "please enable this additional dependency")
• I'm using the latest version of opencv-python

[skvark]

Is there a reason you wish to use opencv-python instead of opencv-python-headless? opencv-python-headless probably works because the matplotlib plotting GUI conflicts withs OpenCV GUI when you use opencv-python (because that's the only difference between those pre-built wheels). The correct solution is to use opencv-python-headless as explained in the README.

[ppwwyyxx]

There isn't a particular reason, but the fact is most users don't know the existence of opencv-python-headless and most probably take "opencv-python" for granted and don't even know this repo. So they do still run into these problems. You can think of me as a developer of pytorch (I'm not, but close to that) who just wants to make these users's life easier.

"GUI conflicts" doesn't seem like the right cause, given that the following also segfaults in the same environment:

from ctypes import cdll
import torch
try:
    import cv2
except:
    pass
cdll.LoadLibrary("/lib64/libgcc_s.so.1")

or the following also segfaults:

from ctypes import cdll

def load(x):
    print("Loading", x)
    try:
        cdll.LoadLibrary(x)
    except Exception as e:
        print("\tFailed", e)
    else:
        print("\tSucc")

load("/opt/app-root/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so")
load("/opt/app-root/lib/python3.6/site-packages/cv2/../opencv_python.libs/libcrypto-354cbd1a.so.1.1")
load("/lib64/libgcc_s.so.1")

[skvark]

This repository and the packages are probably the first results when you google for opencv-python. The README clearly explains different available packages. Therefore, I would expect that pytorch maintainers (or any package maintainers / users who use these packages) would spend a few minutes of their time to read through the provided documentation.

There's certainly some kind of conflict somewhere in those binaries (some overlapping symbols maybe...?) and it's likely that the headless version works because it does not use the same symbols as opencv-python. So yeah, you are correct, not related directly to GUI but to the dependencies required by Qt and X11 (or something else, but since headless version works, I would expect this to be the case). GUI conflicts have been common earlier and it was just my initial guess based on the issue description.

I'll try to have a look into this when I have more time.

[ppwwyyxx]

I agree with you that this is not a problem for those who are able to reach here and find out about "opencv-python-headless", but I'm mainly speaking from the perspective of an average user, who may:

• think pip install opencv-python is THE way to install opencv, because many tutorials say so
• can't even tell if this crash is related to opencv from a huge set of libraries
• will google opencv rather than opencv-python because the this seems a problem of "opencv"
• install and use opencv-python instead of opencv-python-headless unawared, because many packages declare opencv-python as a dependency

A majority of users match some of the above points in my experience. Addressing issues like this would help them a lot! Thanks

[skvark]

As I wrote, I will address the issue once I have enough free time to look deeper into it.

[skvark]

Ah, the root cause was a lot easier to find than I anticipated. The try...except clause you are using hides the actual issue which is that you have a missing dependency: mesa-libGL

If you set the contents of a.py only to import cv2, you will see the actual issue instead of that odd segfault directly from glibc:

Traceback (most recent call last):
  File "a.py", line 1, in <module>
    import cv2
  File "/opt/app-root/lib/python3.6/site-packages/cv2/__init__.py", line 5, in <module>
    from .cv2 import *
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

Fixed Dockerfile:

FROM centos/python-36-centos7
USER root
WORKDIR /root
RUN yum update -y && yum install mesa-libGL -y <------ This is needed, since opencv-python depends on the full X11 stack
RUN pip install --upgrade pip
RUN pip install opencv-python
RUN pip install -U matplotlib
RUN pip install torch==1.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
COPY a.py /root/a.py
RUN python -X faulthandler a.py

This happens because the manylinux2014 wheels do not ship with libGL.so as dictated by the PEP 599 policy: https://www.python.org/dev/peps/pep-0599/#id20

Again, I would recommend using opencv-python-headless which is specifically designed to avoid pulling in those additional GUI related dependencies. However, you can use opencv-python, but you will end up with a slightly larger Docker image (difference is probably around 100 MB) with lots of unused dependencies.

[ppwwyyxx]

I have seen the error, but I don't think that justifies causing a segfault? An import failure is not supposed to have any side effects. Missing X11 shouldn't cause segfault either.

Moreover, using the other example I posted with:

load("/opt/app-root/lib/python3.6/site-packages/cv2/../opencv_python.libs/libcrypto-354cbd1a.so.1.1")

also reproduces the segfault. It reports a missing "libz.so", and if "opencv_python.libs/libz.so" was loaded before this line, it will run without segfault. So it reads more like a problem about the initialization of certain .so libs.

(A bit more background: two projects I maintained optionally depend on opencv, with try: import cv2 except ImportError. Not making it a required dependency is to avoid various symbol conflicts I've experienced in the past)

[skvark]

I have no control over the initialization order of the dynamically loaded libraries. auditwheel uses patchelf to create the manylinux compatible package by changing the default behaviour of the dynamically linked libraries and embeds needed libraries to the wheels. Therefore, loading some arbitrary library directly from the package most likely leads to errors since the library lookup tables have been modified. cv2.so should be loaded first, and then the linker resolves rest of the dependencies. There might be bugs in auditwheel, and the missing libz is odd (it's in the package):

ldd /opt/app-root/lib/python3.6/site-packages/cv2/./../opencv_python.libs/libcrypto-354cbd1a.so.1.1
        linux-vdso.so.1 =>  (0x00007ffcf37c7000)
        libz-d8a329de.so.1.2.7 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f90c3700000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f90c34e4000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f90c3116000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f90c3c1c000)

Additionally, you might have not seen this behaviour in older opencv-python versions since they were using the old manylinux1 standard and Qt4 instead of Qt5. Recent versions of OpenCV require much more recent compiler and glibc version. Qt5 uses also a platform plugin architecture which was not present in Qt4.

While I was testing your Dockerfile, I found out that you can avoid the issue by just commenting out the matplotlib import or importing it before cv2.

GDB output for original a.py:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f087936e700 (LWP 132)]
0x00007f089dcf2bd6 in elf_machine_rela (reloc=0x7f086605d5a8, reloc=0x7f086605d5a8, skip_ifunc=0,
    reloc_addr_arg=0x7f08662c4ec8, version=0x150, sym=0x7f0865ff19f8, map=0x31f8890)
    at ../sysdeps/x86_64/dl-machine.h:299
299           struct link_map *sym_map = RESOLVE_MAP (&sym, version, r_type);
(gdb) bt
#0  0x00007f089dcf2bd6 in elf_machine_rela (reloc=0x7f086605d5a8, reloc=0x7f086605d5a8, skip_ifunc=0,
    reloc_addr_arg=0x7f08662c4ec8, version=0x150, sym=0x7f0865ff19f8, map=0x31f8890)
    at ../sysdeps/x86_64/dl-machine.h:299
#1  elf_dynamic_do_Rela (skip_ifunc=0, lazy=<optimized out>, nrelative=<optimized out>, relsize=<optimized out>,
    reladdr=<optimized out>, map=0x31f8890) at do-rel.h:137
#2  _dl_relocate_object (scope=<optimized out>, reloc_mode=reloc_mode@entry=0, consider_profiling=<optimized out>,
    consider_profiling@entry=0) at dl-reloc.c:259
#3  0x00007f089dcfb65c in dl_open_worker (a=a@entry=0x7f087936dbf8) at dl-open.c:423
#4  0x00007f089dcf67c4 in _dl_catch_error (objname=objname@entry=0x7f087936dbe8,
    errstring=errstring@entry=0x7f087936dbf0, mallocedp=mallocedp@entry=0x7f087936dbe0,
    operate=operate@entry=0x7f089dcfb150 <dl_open_worker>, args=args@entry=0x7f087936dbf8) at dl-error.c:177
#5  0x00007f089dcfab7b in _dl_open (file=0x7f089d5b41ef "libgcc_s.so.1", mode=-2147483646,
    caller_dlopen=<optimized out>, nsid=-2, argc=2, argv=0x7ffc23a6eaa8, env=0x2999710) at dl-open.c:649
#6  0x00007f089cc09722 in do_dlopen (ptr=ptr@entry=0x7f087936de00) at dl-libc.c:87
#7  0x00007f089dcf67c4 in _dl_catch_error (objname=0x7f087936dde0, errstring=0x7f087936ddf0, mallocedp=0x7f087936ddd0,
    operate=0x7f089cc096e0 <do_dlopen>, args=0x7f087936de00) at dl-error.c:177
#8  0x00007f089cc097e2 in dlerror_run (args=0x7f087936de00, operate=0x7f089cc096e0 <do_dlopen>) at dl-libc.c:46
#9  __GI___libc_dlopen_mode (name=name@entry=0x7f089d5b41ef "libgcc_s.so.1", mode=mode@entry=-2147483646)
    at dl-libc.c:163
#10 0x00007f089d5b2db3 in pthread_cancel_init () at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:54
#11 0x00007f089d5b2f7c in _maybe_pthread_cancel_init () at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:104
#12 _Unwind_ForcedUnwind (exc=0x7f087936ed70, stop=0x7f089d5b11e0 <unwind_stop>, stop_argument=0x7f087936df30)
    at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:138
#13 0x00007f089d5b1362 in __GI___pthread_unwind (buf=<optimized out>) at unwind.c:129
#14 0x00007f089d5abef7 in __do_cancel () at pthreadP.h:285
#15 __pthread_exit (value=value@entry=0x0) at pthread_exit.c:28
#16 0x00007f089d99f351 in PyThread_exit_thread () at /usr/src/debug/Python-3.6.9/Python/thread_pthread.h:300
#17 0x00007f089d9a1e09 in t_bootstrap (boot_raw=boot_raw@entry=0x7f0883338170)
    at /usr/src/debug/Python-3.6.9/Modules/_threadmodule.c:1030
#18 0x00007f089d99f204 in pythread_wrapper (arg=<optimized out>)
    at /usr/src/debug/Python-3.6.9/Python/thread_pthread.h:205
#19 0x00007f089d5aaea5 in start_thread (arg=0x7f087936e700) at pthread_create.c:307
#20 0x00007f089cbca8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Next, after removing the whole cv2 import from a.py (original file must have been run beforehand and cv2 removed after that):

Traceback (most recent call last):
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/font_manager.py", line 1429, in <module>
    fontManager = json_load(_fmcache)
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/font_manager.py", line 1013, in json_load
    return json.load(fh, object_hook=_json_decode)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/json/__init__.py", line 367, in loads
    return cls(**kw).decode(s)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import matplotlib.pyplot as plt
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/pyplot.py", line 36, in <module>
    import matplotlib.colorbar
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/colorbar.py", line 44, in <module>
    import matplotlib.contour as contour
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/contour.py", line 16, in <module>
    import matplotlib.font_manager as font_manager
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/font_manager.py", line 1431, in <module>
    _rebuild()
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/font_manager.py", line 1425, in _rebuild
    json_dump(fontManager, _fmcache)
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/font_manager.py", line 997, in json_dump
    with cbook._lock_path(filename), open(filename, 'w') as fh:
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/opt/app-root/lib/python3.6/site-packages/matplotlib/cbook/__init__.py", line 1799, in _lock_path
    lock_path))
TimeoutError: Lock error: Matplotlib failed to acquire the following lock file:
    /opt/app-root/src/.cache/matplotlib/fontlist-v330.json.matplotlib-lock
This maybe due to another process holding this lock file.  If you are sure no
other Matplotlib process is running, remove this file and try again.

So, something odd going on with matplotlib. Works fine again if that lockfile is removed.

If import order is changed, everything is ok:

import torch
import matplotlib.pyplot as plt

error = None

try:
    import cv2
except Exception as e:
    error = e
    pass

print(error)

yields: libGL.so.1: cannot open shared object file: No such file or directory

I have no time to dig deeper currently, but my changing the import order will fix the issue temporarily. There is some side effect somewhere during the cv2 import, but I have very little to none control over that.

[ppwwyyxx]

Thanks for the investigation! I also tried a few versions just now: it seems all the opencv-python==4.x can reproduce the error. While in 3.x, some can trigger the error but some does not, e.g.:

• 3.4.9.33 is bad, while 3.4.10.37 is good
• 3.3.0.9 is good, while 3.3.0.10 (Qt/GUI support added in this version according to release logs) is bad.

What's interesting is that, in 3.4.10.37 import cv2 does not fail although libGL was not installed.

[skvark]

Auditwheel behavior could have changed between those releases. It's not perfect tool, there are issues (also libz related, see https://github.com/pypa/auditwheel/issues) but it does its job reasonably well.

I rebuild occasionally the extended manylinux images (https://github.com/skvark/opencv-python/tree/master/docker) which are used here for Linux builds and auditwheel usually gets updated at the same time. I'm also patching the auditwheel library whitelist in recent releases to avoid bundling libxcb.so.1 because it caused major problems with Qt5 but that's not probably related to this issue since you are seeing this also in older releases.

About zlib: pypa/auditwheel#152

[ppwwyyxx]

Some more investigations using the following code:

#Dockerfile:
FROM python:3.8-slim-buster
USER root
WORKDIR /root
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install opencv-python matplotlib
RUN python3 -m pip install torch==1.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
RUN apt-get -y update && apt-get -y install gdb
COPY a.sh /root/a.sh
RUN bash a.sh


# a.sh:
#!/bin/bash -e
gdb python << EOF
b main
r
set \$pid = (int)getpid()
call dlopen("/usr/local/lib/python3.8/site-packages/torch/_C.cpython-38-x86_64-linux-gnu.so", 2)
eval "!cat /proc/%d/maps > before.txt", \$pid
call dlopen("/usr/local/lib/python3.8/site-packages/cv2/cv2.cpython-38-x86_64-linux-gnu.so", 2)
eval "!cat /proc/%d/maps > after.txt", \$pid
call dlopen("libgcc_s.so.1", 2)
EOF
diff before.txt after.txt

Running this docker, it shows that:

1. the first dlopen succeeds, as expected
2. the second dlopen fails (returns 0), as expected
3. the third dlopen segfaults. This is unexpected
4. the diff between before.txt and after txt shows that, despite the failed dlopen, some libraries went into the address space anyway! The diff is:

> 7ff0c820b000-7ff0c8282000 r--p 00000000 08:01 2901798                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libcrypto-354cbd1a.so.1.1                                      
> 7ff0c8282000-7ff0c8431000 r-xp 00077000 08:01 2901798                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libcrypto-354cbd1a.so.1.1
> 7ff0c8431000-7ff0c84bc000 r--p 00226000 08:01 2901798                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libcrypto-354cbd1a.so.1.1                                      
> 7ff0c84bc000-7ff0c84bd000 ---p 002b1000 08:01 2901798                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libcrypto-354cbd1a.so.1.1
> 7ff0c84bd000-7ff0c84ea000 rw-p 002b1000 08:01 2901798                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libcrypto-354cbd1a.so.1.1
> 7ff0c84ea000-7ff0c84ee000 rw-p 00000000 00:00 0       
> 7ff0c84ee000-7ff0c8523000 rw-p 00334000 08:01 2901798                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libcrypto-354cbd1a.so.1.1
> 7ff0c8523000-7ff0c853f000 r--p 00000000 08:01 2901801                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libssl-1eebc6e1.so.1.1
> 7ff0c853f000-7ff0c858e000 r-xp 0001c000 08:01 2901801                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libssl-1eebc6e1.so.1.1
> 7ff0c858e000-7ff0c85a8000 r--p 0006b000 08:01 2901801                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libssl-1eebc6e1.so.1.1
> 7ff0c85a8000-7ff0c85a9000 ---p 00085000 08:01 2901801                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libssl-1eebc6e1.so.1.1
> 7ff0c85a9000-7ff0c85b6000 rw-p 00085000 08:01 2901801                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libssl-1eebc6e1.so.1.1
> 7ff0c85b6000-7ff0c85c6000 rw-p 000a8000 08:01 2901801                    /usr/local/lib/python3.8/site-packages/opencv_python.libs/libssl-1eebc6e1.so.1.1

That seems to be either a bug in the so files, or a bug in dlopen. Do you have any ideas what might be wrong with the so files, or escalate it to other places (e.g. could it be a bug of auditwheel?)

[skvark]

I don't have any new ideas currently. I think the next step would be to open an issue to the auditwheel repo. However, I'll keep this issue open as well.

One thing to note is that there is a conflict between PyQt and opencv-python which causes issues when matplotlib uses Qt as the GUI backend: #386

[ppwwyyxx]

It looks like a result of this glibc bug: https://sourceware.org/bugzilla/show_bug.cgi?id=20839, because:

• LD_DEBUG=all shows that before segfault, it is trying "relocation processing libcrypto.so"
• In my previous comment, both of the two SOs (libcrypto, libssl) remaining in the address space have NODELETE flags set, as shown by readelf -d

Given that many popular versions of glibc still cannot correctly handle SO files with NODELETE, I'm wondering whether you could consider shipping openssl without NODELETE? The flag to control this is shown in openssl/openssl@848dc96.
According to openssl/openssl@848dc96, there is a backup plan when NODELETE is not used. And according to openssl/openssl@5836780, the goal of all these is about unload. But Python does not unload so files AFAIK, so not using NODELETE should never be a problem.

[skvark]

Thanks for debugging the issue. I'll add the flag to the Dockerfiles and rebuild them. I'll also post download links for the wheels here when I have them available.