DataFrame.ix selecting integer index not label index

[adamgreenhall]

I recently was using .ix on a DataFrame with an index that is very close to the integer index and got some unexpected results:
http://nbviewer.ipython.org/3882591/wonky-indexes.ipynb

I'm not sure if this is a bug or if I'm misreading the docs.

[lodagro]

You are using a float index, ix it with an integer, intended as label (tricky thing to do) but you get positional fallback.
Not all data is shown in the notebook, but it seems you can use an integer index.

Anyway i think pandas can improve on this ...

In [18]: df = pd.DataFrame(np.random.randn(4,4), columns=list('ABCD'), index=[1.0, 2.0, 3.0, 4.0])

In [19]: df
Out[19]: 
          A         B         C         D
1 -0.092198  0.621285  0.434326 -2.133237
2  0.857216  0.672304  0.166730 -0.561775
3 -0.019218  1.151348  2.548601 -1.807996
4  0.240921 -1.175026 -0.039528 -1.433236

In [20]: df.ix[1.0]
Out[20]: 
A   -0.092198
B    0.621285
C    0.434326
D   -2.133237
Name: 1.0

In [21]: df.ix[1]
Out[21]: 
A    0.857216
B    0.672304
C    0.166730
D   -0.561775
Name: 2.0

[adamgreenhall]

The int/float explains it - thanks. This seems like an edge case. I didn't know about the fallback though.

[wesm]

I placed this into the 0.10.2 bucket ("next month") to see if there's anything that can be done

[jreback]

@adamgreenhall

give a try with the new positional indexers .iloc in 0.11-dev

and treat your 'index' as a column

e.g. (using @lodagro example from above)

df['indexer'] = [1.1,2.1,3.1,4.1]

In [32]: df
Out[32]: 
          A         B         C         D  indexer
0  1.525093  0.281564  1.551343  0.195080      1.1
1  0.029036  0.707885 -0.923159 -0.672250      2.1
2 -0.128411 -1.747964 -0.206314 -0.022242      3.1
3  1.113207 -0.517007 -0.083440  0.170024      4.1

In [34]: df[df.indexer>2]
Out[34]: 
          A         B         C         D  indexer
1  0.029036  0.707885 -0.923159 -0.672250      2.1
2 -0.128411 -1.747964 -0.206314 -0.022242      3.1
3  1.113207 -0.517007 -0.083440  0.170024      4.1

Not sure exactly what you are after, but you can do quite
interesting things when you embed the index in a column
then select based on it

In [41]: np.round(df.indexer).astype('int64')[1:3]
Out[41]: 
1    2
2    3
Name: indexer, dtype: int64

In [42]: df.iloc[np.round(df.indexer).astype('int64')[1:3]]
Out[42]: 
          A         B         C         D  indexer
2 -0.128411 -1.747964 -0.206314 -0.022242      3.1
3  1.113207 -0.517007 -0.083440  0.170024      4.1

[jreback]

@adamgreenhall I think you solved your issue, pls close if ok

[adamgreenhall]

I understand what is going on now and feel like there are options to handle it (many options, see below). But I'm not sure someone stumbling across this issue for the first time would figure it out quickly. Maybe we should make a revision to the docs -- the relevant section covering fallback indexing seems to contradict our discussion here.

In [7]: df = pd.DataFrame(np.random.randn(4,4), columns=list('ABCD'), index=[1.0, 2.0, 3.0, 4.0])

In [8]: df
Out[8]: 
          A         B         C         D
1 -0.717876 -1.208706 -0.055981  0.426317
2  0.576185 -0.886573 -2.884863 -1.231298
3 -0.234170  0.765307  0.333422  0.781855
4  1.476230 -0.050315 -0.677205 -1.459387

In [9]: df.index
Out[9]: Index([1, 2, 3, 4], dtype=object)

In [10]: df.ix[1.0]
Out[10]: 
A   -0.717876
B   -1.208706
C   -0.055981
D    0.426317
Name: 1.0, dtype: float64

In [11]: df.iloc[1]
Out[11]: 
A    0.576185
B   -0.886573
C   -2.884863
D   -1.231298
Name: 2.0, dtype: float64

In [12]: df.ix[1]
Out[12]: 
A    0.576185
B   -0.886573
C   -2.884863
D   -1.231298
Name: 2.0, dtype: float64

In [21]: df.reset_index().ix[1]
Out[21]: 
index    2.000000
A        0.940657
B       -0.500388
C        1.628418
D       -0.722274
Name: 1, dtype: float64

[jreback]

love for you to have a go at it!

I would put loud and clear language that float indexes are just a bad idea, they do work, but many options
just makes things clearer (one of the big reasons for iloc and loc)

ix is just really complicated (esp for float indexes) I am in favor of out-right banning it, but docs are
prob a better idea

you can use these examples....

[adamgreenhall]

Sure - I just saw the dev docs. They are much better. I'll add an explaining example and close this.

[jreback]

great, also if you are up to a Float Indexes Section (maybe right before Advanced Indexing)....

wes wrote a book and now we can't take things out,
until there's a 2nd ed. ;)

[jreback]

isn't there supposed to be a new edition every year (or more) ad finitum :), keeps up the royalties...

[adamgreenhall]

How about this: #3128

[jreback]

closed in #3128