deprecate scalar conversions for rank>0 arrays

[nschloe]

Numpy allows for conversion of arrays into scalars if they are size-1, i.e.,

float(numpy.array(1.0))  # 1.0
float(numpy.array([1.0]))  # 1.0
float(numpy.array([[[[1.0]]]]))  # 1.0

# TypeError: only size-1 arrays can be converted to Python scalars
float(numpy.array([1.0, 1.0]))

I'm fine with rank-0 conversions, but I found that discrimination of conversion based on array size can easily lead to inconsistencies downstream. See, for example, sympy/sympy#13924 where suddenly you have a different behavior of multiplication depending on the length of the input array.

Moreover, this feature is actually redundant for rank>0 arrays since the value can simply be retrieved from the array via x[0].

When grepping for "converted to Python scalar" in the numpy sources, one finds that numpy/lib/user_array.py has

"only rank-0 arrays can be converted to Python scalars."

which seems to be the more consistent solution.

I would hence like to suggest to deprecate scalar conversion for rank>0 size-1 arrays.

[seberg]

Personally agree with you (there is a reason we got rid of the operator.index equivalent thing, where it was causing obvious problems) and would encourage anyone to try it.
But… I expect there are some annoying cases to solve and it might have a pretty large downstream impact which would make it a very slow change indeed (I am not sure, I think I might have tried it at some time way back, heh).

[eric-wieser]

or rank-1 and size-1

Not even - size == 1 is all that is checked:

>>> float(np.array([[[1]]]))
1.0

Personally agree with you

Me too. The deprecation warning could stick around for a very long time, and making noise downstream seems pretty reasonable if that noise identifies their bugs.

[njsmith]

I agree with @seberg -- this change would indeed make the API cleaner, but I'm dubious about whether we can actually pull it off or the pain would be worth the benefit. The next step would be to try making the change and see how much stuff breaks (e.g. by running the test suites for big downstream projects like scipy, scikit-learn, astropy, ...). That'd at least give us some data to discuss.

[nschloe]

Thanks for the comments. I've adapted to original post to reflect the fact that arrays of larger rank can be converted to floats as well.

I understand that it can be painful to deprecate stuff, particularly if its not clear how much a functionality is actually used downstream. I would suggest that I make the change locally and compile scipy, sympy, pandas, scikit-learn, and astropy (any other suggestions?) against it and report back with the experience.

[nschloe]

Alright, I've now dug into scipy and this is what I found:

• The use of explicit int/float conversions on size-1 arrays is not that common. In replacing it with the proper expressions, I've even discovered a small bug in scipy.

• When disallowing float conversions on size-1 arrays, the following code is illegal:

  a = numpy.random.rand(10, 1)
  a[0] = numpy.array([1.0])
  # ValueError: setting an array element with a sequence.

  While arguably unclean (you should just set a[0] = 1.0), the use of that is more common. This probably constitutes a roadblock for the suggested change.

I'll leave it up to you to comment and/or close this issue.

[shoyer]

@nschloe I don't follow why that second case becomes illegal. It seems like that should be well defined, since the array a[0, :] has shape (1,). Maybe that was a bug in your implementation, possibly due to preexisting assumptions in Numpy?

[nschloe]

I don't follow why that second case becomes illegal.

@shoyer I'll try to explain why I think it failed. The array element a[0] in

a = numpy.random.rand(10, 1)
a[0] = numpy.array([1.0])

is a scalar (i.e., empty shape (,)), and you're trying to set it with an object of shape (1,). For this to work, the rhs needs to be converted to a scalar (automatically). If scalar conversion is disabled, you'll get an error.

[eric-wieser]

@nschloe: That might be considered a sort of reverse broadcasting, which currently succeeds in some cases but not others:

a = np.zeros((1, 3))
b = np.empty((3,))
b[...] = a # leading 1 dimensions are removed
b[...] += a  # but this fails!

Thinking again, allowing a size == 1 array decay to a scalar seems pretty consistent with broadcasting rules in general, to me - if () can magically promote to (1, 1, 1, 1), then is it any worse to allow (1, 1, 1, 1) to decay to ()?

[eric-wieser]

@nschloe: in your branch, does a[0, ...] = np.array([0]) succeed?

[seberg]

Yeah… well the lhs broadcasting for singleton assignments was what I was thinking of. Not sure if there is more, but if it is the main point, it should not be all that bad…. Eric, hehe, that should succeed, but without checking I am not actually sure anymore if it sidesteps and just does item assignment currently, though I doubt it. There may be a bit annoyance with object arrays, where you never do this for item assignment, since you assume that the array should be the object. But that wouldn't even have to change.

[eric-wieser]

There may be a bit annoyance with object arrays

I think that collectively, we've hit the nail on the head there - reverse broadcasting has no place in scalar assignments, because it just doesn't generalize to object arrays. I think that

a[0] = some_1d_len1_array

ought to be forbidden, since its semantics are dtype dependent - code written this way for numeric arrays will already fail in unexpected ways for object arrays. If the reverse-broadcasting is desired by the user, then they should be forced to write

a[0, ...] = some_1d_len1_array

which does the same thing for all dtypes. Making them write it this way eliminates a class of bug, which is exactly what we should be aiming to do with DeprecationWarnings

---

Whether reverse broadcasting is actually desirable is also perhaps questionable, but you've made it clear to me that it's an orthogonal issue.

[seberg]

Yeah, its orthogonal. If you attempt this, you basically have to put into place the stuff that allows to deprecate the other, too. But that is all there is to it probably.

EDIT: OK it is not quite orthogonal, since for a bit the behaviour between some array likes would differ.

[njsmith]

I think @shoyer's point was that if a has shape (10, 1), then a[0] should expand to a[0, ...] and have shape (1,), so it's not a scalar.