Free cached chunks when the requested memory limit is above real usage.

[dstogov]

Attempt to fix the problem mentioned at #7745

[cmb69]

This looks very reasonable to me. Thank you!

It might be worthwhile to port this to PHP-8.0.

[pvandommelen]

Very useful to have this as part of zend_set_memory_limit 👍 . And I'll also add a +1 for also including this in php8.0.

Wouldn't it also make sense to add something similar to zend_mm_shutdown? It would be nice if php returned its reserved memory when the request completes, and not only when a lower memory limit was requested.

[dstogov]

Very useful to have this as part of zend_set_memory_limit +1 . And I'll also add a +1 for also including this in php8.0.

Wouldn't it also make sense to add something similar to zend_mm_shutdown? It would be nice if php returned its reserved memory when the request completes, and not only when a lower memory limit was requested.

zend_mm_shutdown() should already work fine. It calculates the average peak memory usage per request and tries to keep corresponding number of memory chunks in cache. Of course this can not exceed the memory_limit.

[pvandommelen]

Of course this can not exceed the memory_limit.

This was not obvious to me. But if the shutdown takes the memory limit on startup into account, then there is indeed no issue.

edit, other thought: And TIL that statistics (an average) are being kept on resource usage between requests. That would explain why my tests (from #7745 (comment)) showed a gradual increase/decrease in allocated memory at the start of a request.

[dstogov]

Merged into PHP-8.0 and above via c035298

[xPaw]

Unfortunately this doesn't appear to be fixed, I was getting random but rare allowed memory exceeded in production (running 8.1.7 php-fpm on Linux), which I could not reproduce by repeating the requests (which run the same code and database queries). It runs past the memory limit in different places in my code (but usually memcached->get() or PDO->execute/fetch calls), this problem only manifested it self under higher than normal traffic, and the code hasn't changed that much.

I did added extra breadcrumbs at start of request with memory_get_usage() / memory_get_usage(true), and the only requests that threw "Allowed memory size of 134217728 bytes exhausted" already reported high real memory usage at the start of the request.

I then logged memory_get_usage for all requests per process id, and most of them have 2mb at the start of requests, but some of them would start with higher memory usage, which drops in subsequent requests.

After digging I landed on #7745 (comment)
And using their code, running under php -S, I can still see memory being wrong at start of subsequent requests. This also reproduces on 8.2.0alpha2 running on Windows 10.

$ curl http://localhost:5000
memory usage: 2.000 MB
memory usage: 200.000 MB
memory usage: 4.000 MB

$ curl http://localhost:5000
memory usage: 4.000 MB
memory usage: 200.000 MB
memory usage: 102.000 MB

$ curl http://localhost:5000
memory usage: 102.000 MB
memory usage: 200.000 MB
memory usage: 152.000 MB

$ curl http://localhost:5000
memory usage: 152.000 MB
memory usage: 200.000 MB
memory usage: 176.000 MB

$ curl http://localhost:5000
memory usage: 176.000 MB
memory usage: 200.000 MB
memory usage: 188.000 MB

$ curl http://localhost:5000
memory usage: 188.000 MB
memory usage: 200.000 MB
memory usage: 196.000 MB

I've also dumped /proc/self/smaps, and the only big map it shows is /dev/zero.

[pvandommelen]

@xPaw I think that log shows the expected behaviour. As Dmitry mentioned, php does not return all memory to the system after a request. It keeps the average peak memory allocated. This "real" memory is then reported on the next request (correctly since the fix of #7745).

I would guess that the more accurate real_size is causing issues for you where it previously didn't. But I have trouble explaining why only the processes which previously handled requests would be having issues.

[xPaw]

Yeah I did think about it more yesterday, and I double checked requests for how much memory my code was using, since it's deterministic (same request fetches same data) it doesn't quite make sense why only very few rare requests would run past the limit.