Unable to import pandas when using the gtk3 backend in ipython

[memeplex]

In [1]: %matplotlib gtk3

In [2]: import pandas
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-2-38d4b0363d82> in <module>()
----> 1 import pandas

~/.local/lib/python3.6/site-packages/pandas/__init__.py in <module>()
     56 
     57 from pandas.util._print_versions import show_versions
---> 58 from pandas.io.api import *
     59 from pandas.util._tester import test
     60 import pandas.testing

~/.local/lib/python3.6/site-packages/pandas/io/api.py in <module>()
     17 from pandas.io.stata import read_stata
     18 from pandas.io.pickle import read_pickle, to_pickle
---> 19 from pandas.io.packers import read_msgpack, to_msgpack
     20 from pandas.io.gbq import read_gbq
     21 

~/.local/lib/python3.6/site-packages/pandas/io/packers.py in <module>()
     66 
     67 from pandas.io.msgpack import Unpacker as _Unpacker, Packer as _Packer, ExtType
---> 68 from pandas.util._move import (
     69     BadMove as _BadMove,
     70     move_into_mutable_buffer as _move_into_mutable_buffer,

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 3: invalid start byte

If I use tk or qt instead, this doesn't happen.

[TomAugspurger]

Can you show pd.show_versions()

How'd you install pandas?

[memeplex]

Well I'm unable to import pandas after setting the mpl backend as you can see, so I'm reporting the output of show_versions without setting a backend first, I'm not sure whether this matter or not.

In [2]: pandas.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.13-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.4.0
Cython: 0.27.3
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

The installation was a standard pip one: pip install --user pandas

[TomAugspurger]

No idea what's going on here, sorry.

@tacaswell have you seen anything like this before? pandas._util.move is a C-extension module.

[memeplex]

Maybe it's using some gtk api.

[fariasfc]

I was having the same problem of you, but without use matplotlib directly.

I was using graph_tool, which has a draw module that imports gtk internally), and pandas. I just have changed the order that I imported the libs.

My code before the workaround:

import graph_tool.all as gt
import pandas as pd

My code after the workaround:

import pandas as pd
import graph_tool.all as gt

So, I think that you should try to import pandas before and, set the matplotlib backend after.

[TomAugspurger]

We do some stuff with GTK so for read_clipboard. Perhaps we're doing something wrong there? Either of you mind taking a look?

[TomAugspurger]

Anyone able to debug this further?

[alex-robbins]

On my system (Debian Testing), the script

import matplotlib.pyplot as plt
import pandas as pd

causes

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import pandas as pd
  File "/usr/lib/python3/dist-packages/pandas/__init__.py", line 58, in <module>
    from pandas.io.api import *
  File "/usr/lib/python3/dist-packages/pandas/io/api.py", line 18, in <module>
    from pandas.io.packers import read_msgpack, to_msgpack
  File "/usr/lib/python3/dist-packages/pandas/io/packers.py", line 69, in <module>
    from pandas.util._move import (
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 3: invalid start byte

when using the gtk3cairo and gtk3agg backends. The error does not occur if the order of the import statements are switched, or if I use the TkCairo or TkAgg backends. /usr/lib/python3/dist-packages/pandas/io/packers.py doesn't have any non-ASCII bytes in it.

This doesn't appear to have anything to do with whether the session is interactive or not; the behavior is the same.

I'd dig deeper, but I wouldn't know where to look.

---

Edit: pd.show_versions() gives

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.candidate.1
python-bits: 64
OS: Linux
OS-release: 4.15.0-2-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: None
pip: 9.0.1
setuptools: 39.0.1
Cython: None
numpy: 1.14.2
scipy: 0.19.1
xarray: None
IPython: 5.5.0
sphinx: None
patsy: None
dateutil: 2.7.2
pytz: 2018.4
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
pandas_gbq: None
pandas_datareader: None

My pandas is Debian version 0.20.3-11. Should we remove the "Needs Info" label at this point?

[TomAugspurger]

I'm not sure where to look either, sorry.

…

On Fri, Apr 20, 2018 at 3:53 PM, Alex Robbins ***@***.***> wrote: On my system (Debian Testing), the script import matplotlib.pyplot as pltimport pandas as pd causes Traceback (most recent call last): File "test.py", line 2, in <module> import pandas as pd File "/usr/lib/python3/dist-packages/pandas/__init__.py", line 58, in <module> from pandas.io.api import * File "/usr/lib/python3/dist-packages/pandas/io/api.py", line 18, in <module> from pandas.io.packers import read_msgpack, to_msgpack File "/usr/lib/python3/dist-packages/pandas/io/packers.py", line 69, in <module> from pandas.util._move import ( UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 3: invalid start byte when using the gtk3cairo and gtk3agg backends. The error does not occur if the order of the import statements are switched, or if I use the TkCairo or TkAgg backends. /usr/lib/python3/dist- packages/pandas/io/packers.py doesn't have any non-ASCII bytes in it. This doesn't appear to have anything to do with whether the session is interactive or not; the behavior is the same. I'd dig deeper, but I wouldn't know where to look. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19706 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIkUmK1Nur9lMmQhAaf8lPCMG3Ez5ks5tqkrAgaJpZM4SGEen> .

[tacaswell]

This 0x89 is 'CHARACTER TABULATION WITH JUSTIFICATION' so I am pretty sure this isn't intentional.

My guess is that something is going wrong in PyInit__move but no guess as to why.

PS, I like the note of this function "If you want to use this function you are probably wrong."

[alex-robbins]

I am pretty sure this isn't intentional

Oh, definitely. There aren't any non-ASCII bytes in the source at all.

[memeplex]

I'm still having this problem which makes impossible to use the gtk3 matplotlib backend.

[alex-robbins]

impossible to use the gtk3 matplotlib backend

This is definitely something that should be fixed, but as per my previous comment, you should be able to use gtk3 by importing pandas before matplotlib.

[memeplex]

That never worked for me:

pandas then pyplot

~:: ipython
Python 3.6.6 (default, Jun 27 2018, 13:11:40) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
--------------------------------------------------------------------------
UnicodeDecodeError                       Traceback (most recent call last)
<ipython-input-1-7dd3504c366f> in <module>()
----> 1 import pandas as pd

~/.local/lib/python3.6/site-packages/pandas/__init__.py in <module>()
     55 
     56 from pandas.util._print_versions import show_versions
---> 57 from pandas.io.api import *
     58 from pandas.util._tester import test
     59 import pandas.testing

~/.local/lib/python3.6/site-packages/pandas/io/api.py in <module>()
     17 from pandas.io.stata import read_stata
     18 from pandas.io.pickle import read_pickle, to_pickle
---> 19 from pandas.io.packers import read_msgpack, to_msgpack
     20 from pandas.io.gbq import read_gbq
     21 

~/.local/lib/python3.6/site-packages/pandas/io/packers.py in <module>()
     67 
     68 from pandas.io.msgpack import Unpacker as _Unpacker, Packer as _Packer, ExtType
---> 69 from pandas.util._move import (
     70     BadMove as _BadMove,
     71     move_into_mutable_buffer as _move_into_mutable_buffer,

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 3: invalid start byte

pyplot then pandas

~:: ipython
Python 3.6.6 (default, Jun 27 2018, 13:11:40) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
--------------------------------------------------------------------------
UnicodeDecodeError                       Traceback (most recent call last)
<ipython-input-1-7dd3504c366f> in <module>()
----> 1 import pandas as pd

~/.local/lib/python3.6/site-packages/pandas/__init__.py in <module>()
     55 
     56 from pandas.util._print_versions import show_versions
---> 57 from pandas.io.api import *
     58 from pandas.util._tester import test
     59 import pandas.testing

~/.local/lib/python3.6/site-packages/pandas/io/api.py in <module>()
     17 from pandas.io.stata import read_stata
     18 from pandas.io.pickle import read_pickle, to_pickle
---> 19 from pandas.io.packers import read_msgpack, to_msgpack
     20 from pandas.io.gbq import read_gbq
     21 

~/.local/lib/python3.6/site-packages/pandas/io/packers.py in <module>()
     67 
     68 from pandas.io.msgpack import Unpacker as _Unpacker, Packer as _Packer, ExtType
---> 69 from pandas.util._move import (
     70     BadMove as _BadMove,
     71     move_into_mutable_buffer as _move_into_mutable_buffer,

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 3: invalid start byte