ImportError (`import pandas`)

[Alexander-Shukaev]

Problem description

First of all, for some reason pandas==0.23.4 does not even build when installed via pip (pip install pandas==0.23.4 --force --no-cache), it simply reports success but importing pandas fails in the same way as posted below.

Not building upon installation looked weird to me (no idea if it's a bug or intentional change) but I decided to give downgrade a go:

(.venv) pip install pandas==0.23.0 --force --no-cache
Collecting pandas==0.23.0
  Downloading https://files.pythonhosted.org/packages/45/12/1e1ba99fb65df9f7f3724d3232feef35cc044d18604d57492d561e90219f/pandas-0.23.0.tar.gz (13.1MB)
    100% |████████████████████████████████| 13.1MB 1.7MB/s 
Collecting python-dateutil>=2.5.0 (from pandas==0.23.0)
  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
    100% |████████████████████████████████| 235kB 1.6MB/s 
Collecting pytz>=2011k (from pandas==0.23.0)
  Downloading https://files.pythonhosted.org/packages/f8/0e/2365ddc010afb3d79147f1dd544e5ee24bf4ece58ab99b16fbb465ce6dc0/pytz-2018.7-py2.py3-none-any.whl (506kB)
    100% |████████████████████████████████| 512kB 1.3MB/s 
Collecting numpy>=1.9.0 (from pandas==0.23.0)
  Downloading https://files.pythonhosted.org/packages/38/39/f73e104d44f19a6203e786d5204532e214443ea2954917b27f3229e7639b/numpy-1.15.4-cp37-cp37m-manylinux1_x86_64.whl (13.8MB)
    100% |████████████████████████████████| 13.9MB 1.5MB/s 
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas==0.23.0)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, pytz, numpy, pandas
  Found existing installation: six 1.11.0
    Uninstalling six-1.11.0:
      Successfully uninstalled six-1.11.0
  Found existing installation: python-dateutil 2.7.5
    Uninstalling python-dateutil-2.7.5:
      Successfully uninstalled python-dateutil-2.7.5
  Found existing installation: pytz 2018.7
    Uninstalling pytz-2018.7:
      Successfully uninstalled pytz-2018.7
  Found existing installation: numpy 1.15.4
    Uninstalling numpy-1.15.4:
      Successfully uninstalled numpy-1.15.4
  Found existing installation: pandas 0.23.0
    Uninstalling pandas-0.23.0:
      Successfully uninstalled pandas-0.23.0
  Running setup.py install for pandas ... done
Successfully installed numpy-1.15.4 pandas-0.23.0 python-dateutil-2.7.5 pytz-2018.7 six-1.11.0

This indeed built as it should have but importing still reports errors (same as for unbuilt 0.23.4):

.venv/lib/python3.7/site-packages/pandas/__init__.py:26: in <module>
    from pandas._libs import (hashtable as _hashtable,
.venv/lib/python3.7/site-packages/pandas/_libs/__init__.py:4: in <module>
    from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
pandas/_libs/tslibs/conversion.pxd:11: in init pandas._libs.tslib
    ???
pandas/_libs/tslibs/conversion.pyx:1: in init pandas._libs.tslibs.conversion
    ???
pandas/_libs/tslibs/nattype.pyx:14: in init pandas._libs.tslibs.nattype
    ???
E   ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
ImportError while importing test module '<...>'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.venv/lib/python3.7/site-packages/pandas/__init__.py:26: in <module>
    from pandas._libs import (hashtable as _hashtable,
.venv/lib/python3.7/site-packages/pandas/_libs/__init__.py:4: in <module>
    from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
E   ImportError: cannot import name 'iNaT' from 'pandas._libs.tslib' (unknown location)

During handling of the above exception, another exception occurred:
.venv/lib/python3.7/site-packages/_pytest/python.py:481: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
.venv/lib/python3.7/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
.venv/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:296: in load_module
    six.exec_(co, mod.__dict__)
.venv/lib/python3.7/site-packages/pandas/__init__.py:35: in <module>
    "the C extensions first.".format(module))
E   ImportError: C extension: 'iNaT' from 'pandas._libs.tslib' (unknown location) not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

Any ideas how to go about it?

P.S. Please, no conda solutions.

[TomAugspurger]

Your blog shows pandas==0.23.0. 0.23.0 doesn't support python 3.7, so there's no wheel.

Can you post the full log from

1. creating a new venv
2. pip install pandas 0.23.4 with verbose
3. pip, setuptools and OS and version info.

[Alexander-Shukaev]

Attaching venv.log (a virtual environment crearted from scratch). All requested versions are inside and much more. The OS is obviously Linux.

[TomAugspurger]

``` Added pandas==0.23.4 from https://files.pythonhosted.org/packages/67/a7/12261a51ac2e7be4c698ca27cbe364ca5f16d64999456ee47ea8c7b44417/pandas-0.23.4-cp37-cp37m-manylinux1_x86_64.whl#sha256=647b3b916cc8f6aeba240c8171be3ab799c3c1b2ea179a3be0bd2712c4237553 (from -r /home/user/XXX/freeze.txt (line 18)) to build tracker '/tmp/pip-req-tracker-c_b7vfkl' ``` gets you the correct wheel. Does anything in there stand out to you?

…

On Tue, Nov 20, 2018 at 7:54 PM Alexander Shukaev ***@***.***> wrote: Attaching venv.log <https://github.com/pandas-dev/pandas/files/2602280/venv.log> (a virtual environment crearted from scratch). All requested versions are inside and much more. The OS is obviously Linux. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23798 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIhYOqgJEo462vNS-SIbd3zbCHuN5ks5uxLJigaJpZM4YqEcr> .

[Alexander-Shukaev]

Not really, I mean this is the first package that I'm having such a problem with. I don't know the internals of pandas so I cannot speak about why these multiple exceptions are thrown in the end when I actually import it. I can only judge from their messages where they claim that they perhaps cannot locate the C shared libraries carrying the extensions? Where those are supposed to be after installation?

[TomAugspurger]

In that log you run `python setup.py test`. Does just "import pandas" work? Why does `ImportError while importing test module '/home/user/XXX/test/xxx/test_utils.py'.`? Is that in your venv directory or outside it?

…

On Wed, Nov 21, 2018 at 8:23 AM Alexander Shukaev ***@***.***> wrote: Not really, I mean this is the first package that I'm having such a problem with. I don't know the internals of pandas so I cannot speak about why these multiple exceptions are thrown in the end when I actually import it. I can only judge from their messages where they claim that they perhaps cannot locate the C shared libraries carrying the extensions? Where those are supposed to be after installation? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23798 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIn_cEbb7MnXykIo1tlFhxaaXm-1qks5uxWH3gaJpZM4YqEcr> .

[Alexander-Shukaev]

Outside, it's source code. It says that because this module imports pandas and that actually throws, see that stack.

[Alexander-Shukaev]

As soon as I comment out import pandas, everything is good.

[Alexander-Shukaev]

Digging through <cython/cython#1720> and e.g. a corresponding fix <IntelPython/mkl_random/pull/1>, given my traceback:

test/xxx/test_utils.py:41: in <module>
    import pandas     as pd
.venv/lib/python3.7/site-packages/pandas/__init__.py:28: in <module>
    from pandas._libs import (hashtable as _hashtable,
.venv/lib/python3.7/site-packages/pandas/_libs/__init__.py:6: in <module>
    from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
pandas/_libs/tslibs/conversion.pxd:11: in init pandas._libs.tslib
    ???
pandas/_libs/tslibs/conversion.pyx:1: in init pandas._libs.tslibs.conversion
    ???
pandas/_libs/tslibs/nattype.pyx:14: in init pandas._libs.tslibs.nattype
    ???
E   ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

I believe we should add

from __future__ import absolute_import

at the top of <pandas/_libs/tslibs/nattype.pyx> and most likely <pandas/_libs/tslibs/conversion.pyx> and <pandas/_libs/tslibs/conversion.pxd> as well. Could somebody do the changes, please?

[Alexander-Shukaev]

By the way, a workaround from #22788 also works of course, but why not simply fix it?

[TomAugspurger]

at the top of <pandas/_libs/tslibs/nattype.pyx

<https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/nattype.pyx>> and most likely <pandas/_libs/tslibs/conversion.pyx <https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/conversion.pyx>> and <pandas/_libs/tslibs/conversion.pxd <https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/conversion.pxd>> as well. Could somebody do the changes, please? Feel free to make a PR.

…

On Fri, Nov 23, 2018 at 3:58 PM Alexander Shukaev ***@***.***> wrote: Digging through <cython/cython#1720 <cython/cython#1720>> and e.g. a corresponding fix <IntelPython/mkl_random/pull/1 <IntelPython/mkl_random#1>>, given my traceback: test/xxx/test_utils.py:41: in <module> import pandas as pd .venv/lib/python3.7/site-packages/pandas/__init__.py:28: in <module> from pandas._libs import (hashtable as _hashtable, .venv/lib/python3.7/site-packages/pandas/_libs/__init__.py:6: in <module> from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime pandas/_libs/tslibs/conversion.pxd:11: in init pandas._libs.tslib ??? pandas/_libs/tslibs/conversion.pyx:1: in init pandas._libs.tslibs.conversion ??? pandas/_libs/tslibs/nattype.pyx:14: in init pandas._libs.tslibs.nattype ??? E ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ I believe we should add from __future__ import absolute_import at the top of <pandas/_libs/tslibs/nattype.pyx <https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/nattype.pyx>> and most likely <pandas/_libs/tslibs/conversion.pyx <https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/conversion.pyx>> and <pandas/_libs/tslibs/conversion.pxd <https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/conversion.pxd>> as well. Could somebody do the changes, please? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23798 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIgYUhldmZBw5Jz-cpb6Com3cN6Yrks5uyG-FgaJpZM4YqEcr> .

[TomAugspurger]

@Alexander-Shukaev @WillAyd and you check if #23940 will fix this?

[WillAyd]

I don't think this issue is relevant any longer so closing but if you are still having the issue @Alexander-Shukaev feel free to reopen