Subnormals make testing flush-to-zero libs/builds impractical

[leofang]

Running the test suite against cupy.array_api I'm seeing a lot of the following error:

self = <hypothesis.extra.array_api.ArrayStrategy object at 0x7fd585346f40>, val = 5.877471754111438e-39, val_0d = Array(0., dtype=float32)
strategy = floats(width=32)

    def check_set_value(self, val, val_0d, strategy):
        finite = self.builtin is bool or self.xp.isfinite(val_0d)
        if finite and self.builtin(val_0d) != val:
>           raise InvalidArgument(
                f"Generated array element {val!r} from strategy {strategy} "
                f"cannot be represented with dtype {self.dtype}. "
                f"Array module {self.xp.__name__} instead "
                f"represents the element as {val_0d}. "
                "Consider using a more precise elements strategy, "
                "for example passing the width argument to floats()."
            )
E           hypothesis.errors.InvalidArgument: Generated array element 5.877471754111438e-39 from strategy floats(width=32) cannot be represented with dtype float32. Array module cupy.array_api instead represents the element as 0.0. Consider using a more precise elements strategy, for example passing the width argument to floats().

I think the comparison self.builtin(val_0d) != val should be changed to isclose(val_0d, val).

[rgommers]

@honno could you have a look at this?

@leofang can we move this issue to the https://github.com/data-apis/array-api-tests repo?

[honno]

Will investigate, just noting this is actually a problem with Hypothesis.

[leofang]

lol Thanks @rgommers, I clicked the wrong link and didn't pay attention... I transferred the issue.

[honno]

For folks interested, the problem here is probably to do with issue HypothesisWorks/hypothesis#3153. If we can't get it fixed soon then I know a temporary solution for the test suite :)

Something fun I found with how NumPy and CuPy differ in how it represents an unpresentable float

>>> from cupy import array_api as cxp
>>> x1 = cxp.asarray(1.e-45, dtype=cxp.float32)
>>> x1 > 0
Array(False, dtype=bool)
>>> from numpy import array_api as nxp
>>> x2 = nxp.asarray(1.e-45, dtype=nxp.float32)
>>> x2 > 0
Array(True, dtype=bool)

(but to clarify, the problem is with Hypothesis generating these unpresentable floats in the first place)

[honno]

@leofang If you'd like to try the #43 branch at honno/array-api-tests/tree/float-width-patch—resolves this on my end at least 😅

[Zac-HD]

These are not unrepresentable floats; they're IEEE 754 standard "subnormals" - and exposing the fact that cupy.array_api is flushing subnormals to zero maybe due to a compiler flag. This is a pretty common standards violation (it also helps performance on Intel CPIs, for example) and the array API standard might change to allow it, but it is currently a violation.

We should improve that Hypothesis error message to make this clear, though.

[honno]

Opened data-apis/array-api#339 to see if the spec could change (but in any case both @asmeurer and I are on-board that the test suite shouldn't test with subnormals).

[Zac-HD]

Sounds like sufficient cause to add floats(..., allow_subnormal=True) to Hypothesis and plumb it through the various array strategies then, open an issue and/or PR?

[honno]

Sounds like sufficient cause to add floats(..., allow_subnormal=True) to Hypothesis and plumb it through the various array strategies then, open an issue and/or PR?

@Zac-HD ah I'll write a quick issue now.

[leofang]

@Zac-HD good finding!

[Zac-HD]

We now have PR up to add an st.floats(allow_subnormal=False) option to Hypothesis, and plan automatically enable this for Array API libraries with FTZ set, as well as improving the error message for users who bypass that. See HypothesisWorks/hypothesis#3156 (review) 😃