incosistencies in dataframe.dtype.isin() - I am getting inconsistent with different runs on same code.

[maleknia]

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas

data = [[2, 'tom', 10], [3, 'nick', 15], [4, 'juli', 14]]

myDf = pandas.DataFrame(data, columns=['a','b','c'])

print (myDf.dtypes)

print((myDf.dtypes.isin(['int64'])))

Problem description

Problem is that I am getting inconsistent results based on running the same code:

So first run gives me:

a  False
b  False
c  False

and second (sometimes it takes up to 10 attempts) run gives me:

a  True
b  False
c  True

the second case is the truth and matches the myDf.dtypes:

a     int64
b    object
c     int64

.

Again to reiterate, I am not changing the code but when I keep running the same code I sometime get the first output and sometime get the second output. This is while I do not change the input, code or interpreter. I tried it on different machines and interpreters and still getting the same inconsistent result

the version of Python 3.7
the version of Pandas: 1.0

Expected Output

Consistent result (as it sometimes matches with the current output).

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Darwin
OS-release : 19.3.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8

pandas : 1.0.0
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 19.0.3
setuptools : 40.8.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[TomAugspurger]

Do you have a minimal reproducible example? http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. Ideally you wouldn't need to read from CSV to reproduce this.

Can you also check the formatting on your post? https://guides.github.com/features/mastering-markdown/

[maleknia]

Updated the info and format.

[TomAugspurger]

Thanks. I'm not able to reproduce locally.

[maleknia]

May I ask how many times have you run this?

[maleknia]

Thanks. I'm not able to reproduce locally.

Hello again. I am trying to follow up on this. Would you please elaborate the result you are getting? I just wanted to make sure that I emphasize the result will show up after running the same script about 10 times (inconsistent results in 10 different runs).

[TomAugspurger]

When you say re-run do you mean within a single python interpreter session, or multiple python script.py?

[MarcoGorelli]

I'm seeing this too. I saved a file called script.py with the code you pasted, and then ran:

(pandas-dev) m.gorelli@ws-1808:~/pandas$ python script.py 
a     int64
b    object
c     int64
dtype: object
a    False
b    False
c    False
dtype: bool
(pandas-dev) m.gorelli@ws-1808:~/pandas$ python script.py 
a     int64
b    object
c     int64
dtype: object
a    False
b    False
c    False
dtype: bool
(pandas-dev) m.gorelli@ws-1808:~/pandas$ python script.py 
a     int64
b    object
c     int64
dtype: object
a    False
b    False
c    False
dtype: bool
(pandas-dev) m.gorelli@ws-1808:~/pandas$ python script.py 
a     int64
b    object
c     int64
dtype: object
a    False
b    False
c    False
dtype: bool
(pandas-dev) m.gorelli@ws-1808:~/pandas$ python script.py 
a     int64
b    object
c     int64
dtype: object
a     True
b    False
c     True
dtype: bool

[TomAugspurger]

Interesting, thanks. NumPy dtypes are a bit strange, since I think the aren't always singletons, and different instances may hash differently. If you always use the same object, do you see inconsistences?

In [14]: intd = myDf.dtypes[0]

In [15]: myDf.dtypes.isin([intd]

[MarcoGorelli]

@TomAugspurger nope, with that I always get

a     int64
b    object
c     int64
dtype: object
a     True
b    False
c     True
dtype: bool

as output

[simonjayhawkins]

I'm seeing this too. I saved a file called script.py with the code you pasted, and then ran:

also seeing this. (on master and 0.25.3)

[TomAugspurger]

Thanks. We need to determine whether this is something we want to / can support.

I think the thing to figure out is when NumPy dtypes are not singletons or how we can ensure they factorize the same.