BLD: Clang error while installing snappy in requirements-dev.txt

[kendricng]

Problem description

I was going through the instructions in setting up the dev environment on Mac here:

Contributing to pandas

And while I was trying to execute the below code:

# Install the build dependencies
python -m pip install -r requirements-dev.txt

I got the below clang error (redacted):

Running setup.py install for python-snappy ... error
    ERROR: Command errored out with exit status 1:
     …
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/xxx/virtualenvs/pandas-dev/include -I/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c snappy/snappymodule.cc -o build/temp.macosx-10.15-x86_64-3.7/snappy/snappymodule.o
    snappy/snappymodule.cc:31:10: fatal error: 'snappy-c.h' file not found
    #include <snappy-c.h>
             ^~~~~~~~~~~~
    1 error generated.
    error: command 'clang' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: …

I went through the rest of the virtual environment set up fine besides this issue.

[WillAyd]

Looks like this was added in #26657 and has a comment on master that it is required for pyarrow - @datapythonista any chance you know what this is?

[datapythonista]

That PR was just to get the missing dependencies from the other builds into environment.yml so we had all the dependencies locally.

This library is to connect to gbq afaik.

But this error is expected I think. Libraries use binary dependencies, and pip is not good at those, and you need to install them manually, including their C headers. That's why conda exist, to avoid having to install binary dependencies manually.

So, @kendricng, please use conda, or install snappy, including its dependencies files manually.

Closing this issue, let me know if I miss anything and needs to be reopen.

[WillAyd]

Yea I figured something google-related but didn't see it in the pandas-gbq requirements; are we sure we even need it?

[datapythonista]

Not sure if we need it. But it's a compression library, I guess it's an optional gbq dependency.

[WillAyd]

Gotcha. Yea so if we don't actually need it I would be OK to remove from the requirements files

@kendricng something you would be interested in submitting a PR for?

[jbrockmendel]

is this fixed by brew install snappy?

[simonjayhawkins]

Gotcha. Yea so if we don't actually need it I would be OK to remove from the requirements files

doesn't that defeat the purpose of having "the missing dependencies from the other builds into environment.yml so we had all the dependencies locally." #32417 (comment) for testing/development

also see #32327 for another snappy issue.

[TomAugspurger]

snappy is also an optional dependency for fastparquet and pyarrow.

Regardless, I don't think there's anything for pandas to do. If the library doesn't provide a wheel, users will need to install from source. And to do that they need the C library (built from source or through the system package manager).

If we deem that having pip users take that extra step is too large, then we could find where in the docs snappy is used and avoid it.

[kendricng]

Gotcha. Yea so if we don't actually need it I would be OK to remove from the requirements files

@kendricng something you would be interested in submitting a PR for?

I would like to. I tried submitting one this morning but it got closed. I was wondering what the proper procedure for submitting one would be?

[kendricng]

is this fixed by brew install snappy?

Yes, this is the fix. Thank you for pointing this out, and I've been able to set up my dev environment!

My bigger concern is that this type of error should not be happening when setting up a dev environment from scratch.

[TomAugspurger]

It doesn’t if you use conda. But how that library is packaged is out of our control.

What we can control is avoiding its use in the docs. Does anyone know where it’s used?