pandas.merge issue with duplicated column names

[wavexx]

df1 = pd.DataFrame([[1, 1]], columns=['x','x'])
df2 = pd.DataFrame([[1, 1]], columns=['x','y'])
pd.merge(df1, df2, on='x')

Results in:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    m = pd.merge(df1, df2, on='x')
  File "/usr/lib/python3/dist-packages/pandas/tools/merge.py", line 35, in merge
    return op.get_result()
  File "/usr/lib/python3/dist-packages/pandas/tools/merge.py", line 196, in get_result
    join_index, left_indexer, right_indexer = self._get_join_info()
  File "/usr/lib/python3/dist-packages/pandas/tools/merge.py", line 324, in _get_join_info
    sort=self.sort, how=self.how)
  File "/usr/lib/python3/dist-packages/pandas/tools/merge.py", line 516, in _get_join_indexers
    llab, rlab, shape = map(list, zip( * map(fkeys, left_keys, right_keys)))
  File "/usr/lib/python3/dist-packages/pandas/tools/merge.py", line 681, in _factorize_keys
    llab = rizer.factorize(lk)
  File "pandas/hashtable.pyx", line 850, in pandas.hashtable.Int64Factorizer.factorize (pandas/hashtable.c:15601)
ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

See #11754

[jreback]

hmm, ok though a bit odd to do this.

[wavexx]

Yes, it's clearly not intentional what you're trying to do here.

[jreback]

I would for now simply raise in this case (a helpful message). If you have a nice use-case to actually do this, then lets reconsider.

[wavexx]

I would also raise an error here, but for simmetry, only if both DF have a different number of duplicates for the column.

Ie, this should work, even if with a warning:

df1 = pd.DataFrame([[1, 1]], columns=['x','x'])
df2 = pd.DataFrame([[1, 1]], columns=['x','x'])
pd.merge(df1, df2, on='x')

[jreback]

IIRC we recently allowed duplicates when they are NOT the merge on column.

as I said I don't think merging on a duplicate column is ever warranted and I would just raise as this is a source of error/confusion. Having a special case is prob not necessary.

[wavexx]

On 04/12/15 17:55, Jeff Reback wrote:

IIRC we recently allowed duplicates when they are NOT the merge on column.

I see. If it was already agreed, I do not have a strong point for it.