Proposal to standardize element-wise elementary mathematical functions

[kgryte]

Based on the analysis of array library APIs, we know that evaluating element-wise elementary mathematical functions is both universally implemented and commonly used. Accordingly, this issue proposes to standardize the following elementary mathematical functions:

Special Functions

• abs (absolute)
• exp
• log
• sqrt

Rounding

• ceil
• floor
• trunc
• round

Trigonometry

• sin
• cos
• tan
• asin (arcsin)
• acos (arccos)
• atan (arctan)
• sinh
• cosh
• tanh
• asinh (arcsinh)
• acosh (arccosh)
• atanh (arctanh)

Criterion

1. Commonly implemented across array libraries.
2. Commonly used by array library consumers.
3. Operates on a single array (e.g., this criterion excludes atan2).

Questions

1. Naming conventions? The above is biased toward C naming conventions (e.g., atan vs arctan).
2. Are there any APIs listed above which should not be standardized?
3. Are there elementary mathematical functions not listed above which should be standardized? Preferably, any additions should be supported by usage data.
4. Should the standard mandate a minimum precision to ensure portability? (e.g., JavaScript's lack of minimum precision specification offers a cautionary tale)

[rgommers]

Thanks @kgryte.

Trigonometry seems fine to me.

Regarding rounding:

• you left out round. I think it is commonly implemented once you handle the round, around, round_ aliases as all being the same. So round should probably be considered for inclusion.
• leaving out fix seems fine, it's basically the same as trunc.
• leaving out rint seems also fine, I've never seen anyone use it.

Regarding special functions: there's a lot more than the four you listed, maybe worth a closer look. I think leaving out all the base-2 and base-10 ones, and things like sinc, makes sense to me. logaddexp is probably worth considering, it's quite useful and implemented by all libraries.

[kgryte]

@rgommers Re: round. You're right. I've added it to the OP.

Re: special functions. Agreed that there are more. In the OP, I've only included the most universal for the time being with the thought that we could make a follow-up proposal with additional special functions (e.g., erf, erfc, logaddexp, log1p, expm1, etc), which are "higher-order" and whose implementations depend on the list of proposed APIs above.

[kgryte]

I compiled generalized signatures (with respect to each of the above listed interfaces for each library) for element-wise elementary mathematical functions, where the raw signature data can be found here.

NumPy

numpy.<name>(x, out=None, *, where=True, casting='same_kind', order='K', dtype=None, subok=True[, signature, extobj]) → ndarray

CuPy

cupy.<name>(x, out=None, dtype=None) → ndarray

dask.array

dask.array.<name>(x, out=None, *, where=True, casting='same_kind', order='K', dtype=None, subok=True[, signature, extobj]) → array

JAX

jax.numpy.<name>(x) → ndarray

MXNet

np.<name>(x, out=None, **kwargs) → ndarray

PyTorch

torch.<name>(input, out=None) → Tensor

Tensorflow

tf.math.<name>(x, name=None) → Tensor

---

In short, the minimum common API across most libraries is

<name>(x, out=None)

For example,

sin(x, out=None)

---

Proposal

Signature of the form:

<name>(x, *, out=None)

APIs:

abs(x, *, out=None)
exp(x, *, out=None)
log(x, *, out=None)
sqrt(x, *, out=None)

ceil(x, *, out=None)
floor(x, *, out=None)
trunc(x, *, out=None)
round(x, *, out=None)

sin(x, *, out=None)
cos(x, *, out=None)
tan(x, *, out=None)
asin(x, *, out=None)
acos(x, *, out=None)
atan(x, *, out=None)
sinh(x, *, out=None)
cosh(x, *, out=None)
tanh(x, *, out=None)
asinh(x, *, out=None)
acosh(x, *, out=None)
atanh(x, *, out=None)

Notes

Optional arguments as keyword-only arguments for the following reasons:

1. Avoid potential positional variation amongst library implementations.
2. Favor explicit interfaces and minimize readers' need to intuit an optional positional argument's meaning.

[rgommers]

Thanks @kgryte. A couple of thoughts on signatures:

• It may be useful to make all functions use keyword-only keywords (i.e. put * after the positional arguments).
• For the syntax of the signatures, the square brackets are perhaps best left out. I've always found those a little odd. They come from EBNF notation, where they indicate "optional". That makes perfect sense for a language-independent grammar as well as for signatures of languages where you can't tell whether something is optional or not. In Python however, the default for a keyword (=None or similar) already unambiguously says that it's optional. Making the [ ] just visual noise.

[kgryte]

@rgommers Thanks! Updated to use keyword-only arguments and removed square brackets (where applicable).

[oleksandr-pavlyk]

Just making sure, the keyword where=True present in numpy.ufunc is left out in the standard?

[kgryte]

@oleksandr-pavlyk That's up to us to decide. What I wrote above reflects the "common" API signature across the various array libraries. Perhaps the Torch and Tensorflow contributors can discuss why a where keyword is not currently supported in their libraries.

Regardless, even if not supported now, we could later update the specification to include a where keyword, provided we can reach consensus.

[rgommers]

where= is fairly new, and coverage is still incomplete even in NumPy itself (e.g. see this PR adding where= to numpy.mean, scheduled for v1.20.0).

It's nice to have, but not used a lot. Given no other library except Dask supports it, I'd suggest it should be left out until that situation changes.

[kgryte]

See #12 for a draft specification.

[shoyer]

Two high level comments:

1. Are we willing to use positional-only parameters? This is a Python 3.8+ only feature. If so, the universal signature might be <name>(x, /, out=None), which is well aligned with how these APIs are used in practice.

2. Do we care about complete coverage for basic mathematical operations, or at we happy to rely on support for builtin Python arithmetic operation? I ask because "unary minus", i.e., -x is one obvious omission here.

[shoyer]

One other longer comment, on the out keyword argument. I would lean against including it in the spec, or at least reserving it for a larger discussion about whether we want to support both "mutable" and "immutable" arrays.

Mutation can be challenging to support in some execution models (at least without another layer of indirection), which is why several projects currently don't support it (TensorFlow and JAX) or only support it half-heartedly (e.g., Dask). The commonality between these libraries is that they build up abstract computations, which is then transformed (e.g., for autodiff) and/or executed in parallel. Even NumPy has "read only" arrays.

I'm particularly concerned about new projects that implement this API, which might find the need to support mutation burdensome.

(out can certainly improve performance with libraries like NumPy, but in practice I've seen it used only rarely. In most cases where in-place arithmetic is essential for performance, it's worthwhile to rewrite the computation outside of Python, anyways, or with some form of jit compilation)

[rgommers]

Are we willing to use positional-only parameters? This is a Python 3.8+ only feature. If so, the universal signature might be <name>(x, /, out=None), which is well aligned with how these APIs are used in practice.

Good suggestion, I'd be in favor. I don't see a big issue with >=py38, and there's a major upside: it side-steps the issue with differently named positional arguments (e.g. what numpy et al. call x, pytorch calls input - with / no one will have to spend mental energy on whether or not to rename that).

[kgryte]

We can also keep the keyword-only requirements. In which case, the universal signature would be

<name>(x, /, *, out=None)