import pandas hanging Flask 0.11.1 / Apache 2.4.18

[Ty-WDFW]

pandas 0.19.1 is hanging apache on the import in the python script, the website times out. Downgrading to 0.18.1 resolves the issue. Tested this on a fresh EC2 instance Ubuntu 16.04.

Apache log:

• [Sat Nov 12 03:05:18.784672 2016] [core:warn] [pid 23426:tid 139710063925120] AH00045: child process 23563 still did not exit, sending a SIGTERM
• [Sat Nov 12 03:05:20.786946 2016] [core:warn] [pid 23426:tid 139710063925120] AH00045: child process 23563 still did not exit, sending a SIGTERM
• [Sat Nov 12 03:05:22.789238 2016] [core:warn] [pid 23426:tid 139710063925120] AH00045: child process 23563 still did not exit, sending a SIGTERM
• [Sat Nov 12 03:05:24.791547 2016] [core:error] [pid 23426:tid 139710063925120] AH00046: child process 23563 still did not exit, sending a SIGKILL

INSTALLED VERSIONS

commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-45-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.19.1
nose: None
pip: 9.0.1
setuptools: 28.8.0
Cython: None
numpy: 1.11.2
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2016.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[jreback]

pls show a code example

[Ty-WDFW]

from flask import Flask
import pandas as pd

app = Flask(__name__)

@app.route("/")
def hello():
    return 'Hello World!'

if __name__ == "__main__":
    app.run()

When import pandas as pd is commented out the script will run fine and the website will load. Interesting enough when pandas imported in the python console it works flawlessly -- this is purely an issue with pandas 0.19.1 and apache. In the example here I'm using a environment, I've also tested this outside of a environment and still have the same issue.

Here's my apache configuration file:

<VirtualHost *:80>
        # The ServerName directive sets the request scheme, hostname and port that
        # the server uses to identify itself. This is used when creating
        # redirection URLs. In the context of virtual hosts, the ServerName
        # specifies what hostname must appear in the request's Host: header to
        # match this virtual host. For the default virtual host (this file) this
        # value is not decisive as it is used as a last resort host regardless.
        # However, you must set it for any further virtual host explicitly.
        #ServerName www.example.com

        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html

        WSGIDaemonProcess flaskapp user=flask group=www threads=5
        WSGIScriptAlias / /var/www/html/flaskapp/flaskapp.wsgi

        <Directory flaskapp>
            WSGIProcessGroup flaskapp
            WSGIApplicationGroup %{GLOBAL}
            Order deny,allow
            Allow from all
        </Directory>

Here's the .wsgi file:

import os
import sys
import site

# Add virtualenv site packages
site.addsitedir(os.path.join(os.path.dirname(__file__), 'env/local/lib64/python2.7/site-packages'))

# Path of execution
sys.path.append('/var/www/html/flaskapp')

# Fired up virtualenv before include application
activate_env = os.path.expanduser(os.path.join(os.path.dirname(__file__), 'env/bin/activate_this.py'))
execfile(activate_env, dict(__file__=activate_env))

from main import app as application

[jreback]

you are doing odd path manipulation
you shouldn't do any of that in the python program ; activation needs to occur before the app starts

you are probably picking up different versions of pandas from different envs (and/or it's deps)

so you need to fix that

closing as not a pandas issue

[hdemers]

Same issue here with pandas >= 0.19.0. However, in this case I don't have different versions of pandas from different envs (and/or it's deps), because I'm running this web server inside a docker container freshly built each time.

Commenting out import pandas in the following solves the issue, as well as using pandas 0.18.1. Pandas 0.19.0 or 0.19.1 is making Apache hang.

from flask import Flask
import pandas

app = Flask(__name__)

@app.route("/")
def hello():
    return 'Hello World!'

if __name__ == "__main__":
    app.run()

The .wsgi file is:

import sys
sys.path.insert(0, '/app')
from myapplication import app as application

[jreback]

not really sure what you are actually running. But this works fine.

If you can show a reproducible example, pls comment.

Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from flask import Flask
   ...: import pandas
   ...:
   ...: app = Flask(__name__)
   ...:
   ...: @app.route("/")
   ...: def hello():
   ...:     return 'Hello World!'
   ...:

In [2]: app.run()
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [14/Nov/2016 16:30:11] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [14/Nov/2016 16:30:11] "GET /favicon.ico HTTP/1.1" 404 -

In [3]: pandas.__version__
Out[3]: '0.19.0'

In [5]: import flask

In [6]: flask.__version__
Out[6]: '0.11.1'

[hdemers]

The above script has to be run by Apache, with mod_wsgi, exactly like it was first reported. I've included this piece

if __name__ == "__main__":
    app.run()

to show that the script runs fine outside of Apache, but makes it hang otherwise.

[Ty-WDFW]

However, in this case I don't have different versions of pandas from different envs (and/or it's deps)

Just to clarify, I was using a brand new instance with only 0.19.1 freshly installed in my environment. There were no other versions of pandas or even environments installed system-wide.

[jreback]

you can try running with python -v to see what is happening, otherwise no idea.

[Ty-WDFW]

It'll take someone with more technical knowledge than a salmon biologist to figure out how to make mod_wsgi run python in verbose. 😞. Since I first posted this I've tested this on multiple clean instances, the only resolution is to downgrade to 18.1.

[Qblack]

I have also had this issue.
It can be solved by adding WSGIApplicationGroup %{GLOBAL}.
unfortunately that may lead to more issues down the road if packages ever share names.

I found the solution here http://stackoverflow.com/questions/25782912/pandas-and-numpy-thread-safety but have only ever had issues for the latest version of pandas if that helps the salmon people figure it out.

[birdsarah]

Hi all, in case it's useful.

I had a similar issue:
Django==1.10.3
pandas==0.19.1
Python 3.4
Apache with mod_wsgi on AWS Elastic Beanstalk

My solution was to move the imports from the top of my views.py file and into the functions that needed them and all was well. pandas was already being used by this project in django management commands with the imports at the top of the module it was the addition into views.py that gave the problem.

However, it may be worth noting that I think bokeh also had the same problem, and bokeh doesn't have a dependency on pandas any more. I will need to confirm this though if it's a useful avenue.

[birdsarah]

@jreback, you closed this issue on Nov 12 with the reason "closing as not a pandas issue" - i don't know where else I'd file this bug and look for progress/insight on it - suggestions welcome.