pd.Series.diff() on boolean values

[mrtarsa]

Code Sample

>> x = pd.Series([True,False,True])
>> x.diff()
0     NaN
1    True
2    True
dtype: object
>> x - x.shift()
0    NaN
1     -1
2      1
dtype: object

Problem description

It's counter-intuitive that the results of above are different.

The current implementation of pd.Series.diff uses algorithms.diff that subtracts 2 numpy arrays in the following way

out_arr[res_indexer] = arr[res_indexer] - arr[lag_indexer]

As pointed here such behaviour is deprecated in favor to np.diff. But np.diff also treats booleans in binary operations in its own numpy way, that is different from native python (replace False with 0, replace True with 1).

>> np.array([True, False]) - np.array([False, True])
/home/deoxys/miniconda3/lib/python3.6/site-packages/ipykernel_launcher.py:1: DeprecationWarning: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
array([ True,  True], dtype=bool)
>> np.diff([False, True, False])
array([ True,  True], dtype=bool)
>> True - False
1
>> False - True
-1

Expected Output

I believe there is no correct way of subtracting booleans. But, it's definitely strange that operations like x - x.shift() and x.diff() provide different results.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------

commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.11.9-1-ARCH
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.1
pytest: None
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 6.0.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

[jreback]

@tarashypka so easiest is to raise on all boolean .shift/.diff operations. Or is there a correct method here?

[mrtarsa]

@jreback My suggestion that in this case it will be correct for

>> x - x.shift()

to return

0    NaN
1    True
2    True
dtype: object

that will be consistent with the way numpy treats booleans or raise an error/deprecation warning as numpy does.

In order to get desired behaviour user may want to clarify his intentions to treat booleans as 0, 1 with

>> x.astype(int) - x.astype(int).shift()

or

>> x.astype(int).diff()

that will lead to

0    NaN
1   -1.0
2    1.0
dtype: float64

Also it may be helpful to raise a warning when user subtracts booleans.

[wikiped]

Doing:

pd.Series([False, True]).diff()

With numpy.__version__: 1.14.1, raises:

Traceback (most recent call last):
  File "D:\Anaconda\lib\site-packages\pandas\core\series.py", line 1527, in diff
    result = algorithms.diff(_values_from_object(self), periods)
  File "D:\Anaconda\lib\site-packages\pandas\core\algorithms.py", line 1547, in diff
    out_arr[res_indexer] = arr[res_indexer] - arr[lag_indexer]
TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

pandas.__version__: 0.22.0