Symbol not found: _floatify in pandas/lib.so on Mac 10.8 with numpy master

[cdeil]

After updating pandas I can't import it any more because the symbol _floatify is undefined in pandas/lib.so:

$ python -c 'import pandas'
dlopen(/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/__init__.py", line 10, in <module>
    import pandas.lib as lib
ImportError: dlopen(/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so, 2): Symbol not found: _floatify
  Referenced from: /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
  Expected in: flat namespace
 in /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so

$ otool -L /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so
/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
$ nm /Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.9.1.dev_7156920-py2.7-macosx-10.8-x86_64.egg/pandas/lib.so | grep floatify
                 U _floatify

The pandas build log doesn't show any problem with lib.so or floatify:
https://gist.github.com/4030153

My numpy install seems OK:
https://gist.github.com/4030177

[wesm]

We had a little snafu in the setup.py file that was causing the Cython files to not automatically rebuild. The issue here was that floatify, a C function, had its signature changed but the declaration in the Cythonized C file had changed; hence the symbol lookup failure. At least that's what we think is going on. About to push some fixes (they should show up on here)-- do a clean build (setup.py clean, setup.py install) and let us know...

cc @yarikoptic

oops didn't mean to close issue

[cdeil]

With 5c945e1 I still see the same issue:
https://gist.github.com/4034766

[wesm]

What Cython version are you on? seems to be some kind of name mangling problem with clang going on. floatify is declared in pandas/src/util.pxd and the C function itself is in pandas/src/numpy_helper.h. I have no ability to debug this myself

[cpcloud]

Can you post the output of which python?

The reason I ask is because using the system Python can conflict with a Python from /opt/local/bin. I've definitely encountered a similar issue in the past.

[cdeil]

My Python and Cython are the latest Macports versions, my numpy is from git master:

$ xcodebuild -version
Xcode 4.5.2
Build version 4G2008a
$ which python
/opt/local/bin/python
$ python --version
Python 2.7.3
$ python -c 'import cython; print cython.__file__'
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cython.pyc$ port installed py27-cython
The following ports are currently installed:
  py27-cython @0.17.1_0 (active)
$ python -c 'import numpy; print numpy.__file__; print numpy.__version__'
/Users/deil/Library/Python/2.7/lib/python/site-packages/numpy/__init__.pyc
1.8.0.dev-c5ccca9

Can you reproduce the issue with Cython 0.17.1?
If it's a name mangling problem, I could check which compilers were used for Python, numpy, pandas.
Or would it help if I post the generated c files that contain floatify?

[dnmiller]

Have you tried reinstalling Cython in /Users/deil/Library/Python/2.7 instead of the MacPorts dir? It seems risky to be grabbing libraries from two site-packages directories.

[cdeil]

@dnmiller Isn't having two site-packages normal?

There's packages that are hard to build with all their dependencies (like e.g. PyTables) which I install with a package manager (Macports on my Mac), and there's packages where I want the development version and in some cases even do development with, so these I install in my home folder.

I don't see how I could work with only one site-packages, because if I sudo install stuff myself in the site-packages of my package manager all hell breaks loose and I don't have the time to install / update all the packages I need myself.

I don't think it matters here where Cython is installed, rather I think it's a problem with the (Compiler, Python, Cython, numpy) combination I have.

[dnmiller]

Well, you should definitely check out the virtual environments package. It solves a lot of the problems you just described. But I agree that's not really the root cause.

While I can't say this is the same problem you're having, I've had problems with MacPorts linking to it's own versions of libc, libc++, liblapack, etc., which are incompatible with other programs compiled outside of MacPorts, which will link to non-MacPorts versions of the same libraries. I've seen a very similar error that was due to a non-MacPorts Python talking to a MacPorts version of NumPy.

My hunch is that the MacPorts Cython behaves differently than a version you would install in your home dir. You might consider trying to build pandas and its dependencies (and Cython) entirely in /Users/deil/Library/Python/2.7.

[cpcloud]

Have you tried manually importing another *.so file (just write a simple function in cython and try to import it)? If that doesn't work, it might hint at the problem being the clang linker flags, for which it seems there is a paucity of documentation, but I suspect that the fact that floatify is declared undefined in the binary might have something to do with your issue.

FYI: When I do nm lib.so | grep floatify I get T floatify as the output, but I'm running Arch Linux on a MacBook 4,2 so the code generation will probably be a bit different.

Also since you have the -g flag enabled when compiling the *.o files can you post the output of nm -l *.so | grep floatify? It should return the line number(s) of the function signature.

[wesm]

@fonnesbeck : sir, king of bleeding edge OS X technology, can you confirm that you can build and import pandas git master with your OS 10.8 rig?

[brendam]

I just tested in a new virtualenv on OS X 10.8.2 with latest xcode installed. I used 'pip install cython' to get cython. Everything else except wsgiref was installed by the pandas install.

clang --version 
Apple clang version 4.0 (tags/Apple/clang-421.0.60) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.2.0
Thread model: posix

xcodebuild -version
Xcode 4.5.2
Build version 4G2008a

pip freeze
Cython==0.17.1
numpy==1.6.2
pandas==0.9.1.dev-0230bbf
python-dateutil==2.1
pytz==2012h
six==1.2.0
wsgiref==0.1.2

I used
python setup.py clean
python setup.py build_ext --inplace
python setup.py install
nosetests pandas

"Symbol not found: _floatify"

using nm lib.so | grep floatify I get U _floatify

[wesm]

Got a machine over here where I can reproduce this. Will try to figure out what's up