Skip to content

Add clear loop implementation #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 28, 2023
Merged

Conversation

ngoldbaum
Copy link
Member

@ngoldbaum ngoldbaum commented Feb 22, 2023

Note that this can't be merged until numpy/numpy#23262 is merged and there's a wheel available to install.

There are a couple other minor cleanups as well, but mostly this implements custom get_clear_loop and clear_loop implementations, allowing us to call free on the embedded references stored in the numpy array before the array is deallocated.

I haven't checked, but this will hopefully fix all memory leaks, although there may be other places we need to call free as well. In any case, it makes it much easier to check for those places!

Copy link
Member

@seberg seberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, besides those two comments.

NpyAuxData **NPY_UNUSED(out_auxdata),
NPY_ARRAYMETHOD_FLAGS *flags)
{
*flags = NPY_METH_REQUIRES_PYAPI | NPY_METH_NO_FLOATINGPOINT_ERRORS;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
*flags = NPY_METH_REQUIRES_PYAPI | NPY_METH_NO_FLOATINGPOINT_ERRORS;
*flags = NPY_METH_NO_FLOATINGPOINT_ERRORS;

You don't require the python API actually if you are u sing free() to free/allocate.

NumPy currently actually ignores the float flag also, because how will clearing cause that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I thought NPY_METH_REQUIRES_PYAPI was required for the clear loop, I'll try without.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all numpy ones, yes :).

@ngoldbaum
Copy link
Member Author

ngoldbaum commented Feb 27, 2023

To avoid a seg fault in the sorting routine I needed to add the NPY_NEEDS_PYAPI flag. There's no reason why we can't release the GIL, but to get this to work we'd need to add support in numpy for dtypes that hold references that aren't python objects (e.g. make it so NPY_ITEM_IS_POINTER can be set without NPY_ITEM_REFCOUNT while still getting data NULLing at init and the ability to call the clear function).

@ngoldbaum
Copy link
Member Author

This PR needs to be merged to fix the tests, but I'll hold off on self-merging just in case Sebastian or Peyton wants to weigh in on the last commit I just pushed.

@seberg
Copy link
Member

seberg commented Feb 27, 2023

Meh... the numpy sorting code is being stupid there, but I guess you can workaround it here for the time being.

Fix this in numpy instead, see numpy/numpy#23292
@ngoldbaum
Copy link
Member Author

Opened numpy/numpy#23292 and reverted the last commit.

@seberg
Copy link
Member

seberg commented Feb 28, 2023

I retriggered the NumPy "nightlies", so you can hopefully put this in without tests failing (unless it depended on that NULL one).

@ngoldbaum ngoldbaum merged commit 21c5618 into numpy:main Feb 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants