PERF: use numpy.random.Generator

[thrasibule]

Numpy 1.17 introduced a new random module with faster PRNGs, and which drops the strict reproducibility of random streams guarantee, which allows some algorithmic improvements. In particular, the choice method is now a lot faster in the replace=False case. Would it make sense for random_state to return a np.random.Generator instead of a np.random.RandomState when numpy version >= 1.17 here: https://github.com/pandas-dev/pandas/blob/master/pandas/core/common.py#L408. This would automatically speed up the DataFrame.sample method for instance.
I can write a PR, but I'm not sure how to handle the different numpy versions. Should I just do the tests inside random_state or is something that needs to go inside numpy.compat?

[WillAyd]

There should already be _np_version_under1p17 from NumPy compat which you can use internally

[TomAugspurger]

We’ll need to discuss what API guarantees we make about the seeds. I don’t think the output should change when a use provides an integer seed, for example. What happens now if the user passes a new random Generator instance?

…

On Sep 15, 2019, at 21:41, William Ayd ***@***.***> wrote: There should already be _np_version_under1p17 from NumPy compat which you can use internally — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jamesmyatt]

ValueError!

[jamesmyatt]

Guarantee-wise, it would be probably be best to match NEP 19

If you just replace pandas.core.common.random_state with numpy.random.default_rng then you probably match NumPy's compatibility guarantee exactly. But you can only do that when upgrading the rest of the project to use Generators.

[mroeschke]

Our sample APIs take numpy random generators now so closing