import pandas error for missing compression libraries

[raybuhr]

Code Sample

[dev]rbuhr:~% python
Python 3.7.2 (default, Jul 24 2019, 19:27:42)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/__init__.py", line 55, in <module>
    from pandas.core.api import (
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/core/api.py", line 24, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import (  # noqa: F401
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 44, in <module>
    from pandas.core.frame import DataFrame
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/core/frame.py", line 88, in <module>
    from pandas.core.generic import NDFrame, _shared_docs
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/core/generic.py", line 71, in <module>
    from pandas.io.formats.format import DataFrameFormatter, format_percentiles
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/io/formats/format.py", line 47, in <module>
    from pandas.io.common import _expand_user, _stringify_path
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/io/common.py", line 9, in <module>
    import lzma
  File "/home/admin/.pyenv/versions/3.7.2/lib/python3.7/lzma.py", line 27, in <module>
    from _lzma import *
ModuleNotFoundError: No module named '_lzma'
>>>

Problem description

After installing pandas 0.25.0, I can't import the library because of missing compression libraries. First it returned the error message ModuleNotFoundError: No module named '_bz2'. I installed with sudo apt-get install libbz2-dev and tried again to get the error message from the code sample above, ModuleNotFoundError: No module named '_lzma'.

This was not an issue with the previous version of pandas and I tested by downgrading to pandas 0.24.0 and was able to import without the error messages. I feel like pandas should not prevent usage just because some optional compression programs are not installed, like the default behavior of the last version.

Expected Output

>>> import pandas
>>>

Output of pd.show_versions()

Unable to run because can't import pandas.

[mroeschke]

Please see the discussions in #27532 and #27543. You may have to reconfigure your environment.

[WillAyd]

I am surprised this issue keeps popping up for something in the stdlib though - conda and official distributions do always come with lzma right?

[mroeschke]

My stab in the dark guess is that this is a pipenv issue (based on the 3 issues) and somehow '_lzmamodule.c' isn't getting built?

https://github.com/python/cpython/blob/b9a0376b0dedf16a2f82fa43d851119d1f7a2707/setup.py#L1558

[WillAyd]

Yea so here's a related discussion on bpo:

https://bugs.python.org/issue34895

So not a definitive response but I guess still implied that lzma is expected to be available as part of a standard Python distribution

[raybuhr]

I feel like the closing of this issue was not appropriate. The other two issues linked also have the same problem -- that pandas 0.25 assumes you have things installed that may not actually have come with python by default. This should be made explicitly clear up front before installation completes, not as an import error after installation.

[supermitch]

I second @raybuhr 's comment. Pyenv is a project with 16k stars. It's very widely used.

I guess still implied that lzma is expected to be available as part of a standard Python distribution

I feel this is an incorrect assumption then. I've been using Pyenv successfully and never run into an issue with _lzma until this release. It's not a nice experience that people should read through Stack Overflow and 3 closed (!) Pandas issue threads to figure out how to brew install xz as a solution.

[TomAugspurger]

This should be made explicitly clear up front before installation completes, not as an import error after installation.

That's not possible, at least not with binary distributions (wheels / conda packages). Unless you're saying that there should be an error when you're compiling Python, in which case I agree (right now it's just a warning).

---

On the larger issue, I'm not sure what's best. Clearly this is affecting people. But at some point we need to be able to rely on importing module from the standard library, right?

[TomAugspurger]

And just to be clear, this isn't a pyenv issue. It's a problem on the user's machine not having the proper dependencies when Python is compiled.

[WillAyd]

Yea this is certainly unfortunate but quoting what I think is the most definitive response from the Python mailing list:

I agree that modules that are necessarily optional should be documented
as such, and as I mentioned on https://bugs.python.org/issue34895, many
are so documented.
In the absence of such documentation, I would considered it to be not
optional except as some distributor decides to omit it. But then it is
the responsibility of the distributor to document the omission.

https://mail.python.org/pipermail/python-ideas/2018-October/054089.html

So since Python doesn't document this library as optional it should be available and if not the responsibility of the distributor to handle that expectation

[WillAyd]

FWIW pyenv also documents this as the first step in their "Common Build Problems" page:

https://github.com/pyenv/pyenv/wiki/Common-build-problems

So perhaps could help them improve that aspect of the documentation if it isn't immediately obvious

[raybuhr]

I see the points made above about this probably being an issue with system level dependencies. I am in fact using pyenv to install and fixing for our team isn't particularly difficult.

Since python expects the compression libraries to be installed since the modules are part of the standard library, this probably doesn't have to be an issue for the pandas team. That said, I still feel like making the compression libraries prerequisites for using pandas as unnecessary overhead. I think a more sympathetic response would be to try importing the compression modules and return a message that they aren't installed while still allowing pandas to be imported and used, just without support for compression.

[k-dahl]

Pandas 0.25.0 is not useable with tools like kubeless as debian base images for Docker don't appear to contain the proper libs for _lzma any more. You'd need to build out custom images.

Pandas 0.24.2 works fine.

vmware-archive/runtimes#44

[TomAugspurger]

I suspect we would accept a PR that did the lzma import in a try / except ImportError block.

When the module is not present, we would emit a UserWarning that their Python was not compiled properly and that lzma compression is not available. And if they use lzma compression we would raise at runtime.

Is anyone interested in submitting a PR?