duplicated() performance and bug on long rows regression from 0.15.2->0.16.0

[eyaler]

the following works quickly in 0.15.2 and has a performance issue on the last operation df.T.duplicated() in 0.16.0 and 0.16.1
also on a private data set that works on 0.15.2 i get an error on 0.16.0 and 0.16.1 on the same operation.

code:

import pandas,numpy

df = pandas.DataFrame({'A': [1 for x in range(1000)],
                   'B': [1 for x in range(1000)]})

print (numpy.count_nonzero(df.duplicated()))
print (numpy.count_nonzero(df.T.duplicated()))

df = pandas.DataFrame({'A': [1 for x in range(1000000)],
                   'B': [1 for x in range(1000000)]})

print (numpy.count_nonzero(df.duplicated()))
print (numpy.count_nonzero(df.T.duplicated()))


this is the error i get on the private data set (code not reproduce yet with synthetic data):
  File "C:\Anaconda3\lib\site-packages\pandas\util\decorators.py", line 88, in wrapper
    return func(*args, **kwargs)
  File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 2867, in duplicated
    labels, shape = map(list, zip( * map(f, vals)))
  File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 2856, in f
    labels, shape = factorize(vals, size_hint=min(len(self), _SIZE_HINT_LIMIT))
  File "C:\Anaconda3\lib\site-packages\pandas\core\algorithms.py", line 135, in factorize
    labels = table.get_labels(vals, uniques, 0, na_sentinel)
  File "pandas\hashtable.pyx", line 813, in pandas.hashtable.PyObjectHashTable.get_labels (pandas\hashtable.c:14025)
ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

[shoyer]

Looks like this may be related to #9398.

@behzadnouri any ideas?

[behzadnouri]

yes, #9398 will not scale well with a very wide frame; in exact same way that joining two frames on 1000000 columns will not scale well.

for ValueError: Buffer ... my guess is that column names are not unique, so when it iterates over the columns for one of the columns it gets a two dimensional array, therefore wrong number of dimensions (expected 1, got 2) error.

[eyaler]

Thanks. column names are unique. moreover, it works when using only partial data (fewer rows) so i guess its related to the wide frame issue on the transposed data

[behzadnouri]

@eyaler are you transposing the frame? if you are transposing the frame before calling into .duplicated then the row names should be unique

[eyaler]

after transposing the column names get the previous row numbers which are unique.

[jreback]

@behzadnouri any thoughts on this?

[behzadnouri]

@jreback i guess the easiest soln would be to switch to the old code for wide frames if no subset is selected.