df.stack() still misbehaving in unsorted case

[dsm054]

Migrating from the comment here, we still have as of 0.20.3 and current master that

Code Sample

import pandas as pd

# We create a MultiIndex
PAE = ['ITA', 'FRA']
VAR = ['A1', 'A2']
TYP = ['CRT', 'DBT', 'NET']
MI = pd.MultiIndex.from_product([PAE, VAR, TYP], names=['PAE', 'VAR', 'TYP'])

# We create a dataframe with multindex MI
V = list(range(len(MI)))
DF = pd.DataFrame(data=V, index=MI, columns=['VALUE'])

# We unstack the dataframe and drop level 0
DF = DF.unstack(['VAR', 'TYP'])
DF.columns = DF.columns.droplevel(0)
DF.loc[:, ('A0', 'NET')] = 9999

# We stack the dataframe
DF0 = DF.stack(['VAR', 'TYP'])
# DF0 is wrong
DF1 = DF.sort_index(axis=1).stack(['VAR', 'TYP'])
# DF1 is right 

print(DF0.sort_index())
print(DF1.sort_index())
print((DF0.loc[DF1.index] == DF1).all())

gives two different frames which don't match.

Problem description

The association between index, column coordinate, and value shouldn't break just because of lexsorting.

Expected Output

Matching values, and True.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.1.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-83-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_CA.UTF-8 LOCALE: en_CA.UTF-8

pandas: 0.20.3
pytest: 3.1.2
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.1
xarray: 0.9.6
IPython: 6.1.0
sphinx: 1.6.2
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.6.0
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.11
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: 0.4.0

[mroeschke]

Looks to work on master now. Could use a test

PAE  VAR  TYP
FRA  A0   NET    9999.0
     A1   CRT       6.0
          DBT       7.0
          NET       8.0
     A2   CRT       9.0
          DBT      10.0
          NET      11.0
ITA  A0   NET    9999.0
     A1   CRT       0.0
          DBT       1.0
          NET       2.0
     A2   CRT       3.0
          DBT       4.0
          NET       5.0
dtype: float64
PAE  VAR  TYP
FRA  A0   NET    9999.0
     A1   CRT       6.0
          DBT       7.0
          NET       8.0
     A2   CRT       9.0
          DBT      10.0
          NET      11.0
ITA  A0   NET    9999.0
     A1   CRT       0.0
          DBT       1.0
          NET       2.0
     A2   CRT       3.0
          DBT       4.0
          NET       5.0
dtype: float64
True

[Dinkarkumar]

Is the issue resolved ? Can I work on this ?

[MarcoGorelli]

see the comment above, adding a test would be welcome