OPCache crashing in PHP 8.2.8, fine in 8.2.6

[Prochy]

Description

Hi,

Unfortunately, We don't have a reproducible example as we don't know steps how to replicate it but I have a hope you could find something from the dumps we provide.
We upgraded PHP from 8.0.x to 8.2.8 and we start noticing that OPCache crashes regularly every 3-6 hours and then it starts crashing for several minutes and it get back after several tens minutes it gets back to alive and starts working again.
At the moment, when it starts happening the server starts returning 500 Status code.

I tested go back with PHP to 8.2.6 and there was no issue so far (running for 4 days) so I believe the issue have to be between 8.2.6 - 8.2.8.

At first sight, I thought it's the same issue as #8143, but probably not as there is a version 8.1.3 and the downgrade to 8.2.6 wouldn't fix the issue.

Dumps

Event log:

Faulting application name: php-cgi.exe, version: 8.2.8.0, time stamp: 0x64a440d9
Faulting module name: php_opcache.dll, version: 8.2.8.0, time stamp: 0x64a44549
Exception code: 0xc0000005
Fault offset: 0x0000000000014715
Faulting process id: 0x530
Faulting application start time: 0x01d9ce899081bdca
Faulting application path: F:\...php-cgi.exe
Faulting module path: F:\...php_opcache.dll
Report Id: b695b13a-2949-4895-9020-e81c2c6a3e13
Faulting package full name: 
Faulting package-relative application ID: 

Call stack

php_opcache.dll!zend_accel_inheritance_cache_find(_zend_inheritance_cache_entry * entry, _zend_class_entry *) Line 2264
php_opcache.dll!zend_accel_inheritance_cache_get(_zend_class_entry * ce, _zend_class_entry * parent, _zend_class_entry * * traits_and_interfaces) Line 2305
php8.dll!zend_do_link_class(_zend_class_entry * ce, _zend_string * lc_parent_name, _zend_string * key) Line 2925
php8.dll!zend_bind_class_in_slot(_zval_struct * class_table_slot, _zval_struct * lcname, _zend_string * lc_parent_name) Line 1124
php8.dll!do_bind_class(_zval_struct * lcname, _zend_string * lc_parent_name) Line 1159
php8.dll!ZEND_DECLARE_CLASS_SPEC_CONST_HANDLER(_zend_execute_data * execute_data) Line 5420
php8.dll!execute_ex(_zend_execute_data * ex) Line 55835
php8.dll!zend_call_function(_zend_fcall_info * fci, _zend_fcall_info_cache * fci_cache) Line 948
php8.dll!zend_call_known_function(_zend_function * fn, _zend_object * object, _zend_class_entry * called_scope, _zval_struct * retval_ptr, unsigned int param_count, _zval_struct * params, _zend_array * named_params) Line 1042
php8.dll!spl_perform_autoload(_zend_string * class_name, _zend_string * lc_name) Line 446
php8.dll!zend_lookup_class_ex(_zend_string * name, _zend_string * key, unsigned int flags) Line 1207
php8.dll!zend_fetch_class_by_name(_zend_string * class_name, _zend_string * key, unsigned int fetch_type) Line 1704
php8.dll!ZEND_NEW_SPEC_CONST_UNUSED_HANDLER(_zend_execute_data * execute_data) Line 10277
php8.dll!execute_ex(_zend_execute_data * ex) Line 55835
php8.dll!zend_execute(_zend_op_array * op_array, _zval_struct * return_value) Line 60410
php8.dll!zend_execute_scripts(int type, _zval_struct * retval, int file_count, ...) Line 1828
php8.dll!php_execute_script(_zend_file_handle * primary_file) Line 2542
php-cgi.exe!main(int argc, char * * argv) Line 2531

PHP Version

PHP 8.2.8

Operating System

Windows Server 2019

[ddevsr]

I have same issue in PHP 8.2.8 & 8.2.9, POST data with CURL to API consume long time and get simple data not stable which result timeout.

[rlerdorf]

I have noticed a similar sporadic segfault on Linux:

Program received signal SIGSEGV, Segmentation fault.
zend_accel_inheritance_cache_find (needs_autoload_ptr=<optimized out>, traits_and_interfaces=<optimized out>, parent=<optimized out>, ce=<optimized out>, entry=0x10f)
    at /usr/src/debug/php-src-php-8.2.9/ext/opcache/ZendAccelerator.c:2264
2264			if (entry->parent != parent) {
(gdb) bt
#0  0x00007f8ad00a8e7c in zend_accel_inheritance_cache_get (needs_autoload_ptr=<optimized out>, traits_and_interfaces=<optimized out>, parent=<optimized out>, ce=<optimized out>, entry=0x10f)
    at /usr/src/debug/php-src-php-8.2.9/ext/opcache/ZendAccelerator.c:2264
#1  0x00007f8ad00a8e7c in zend_accel_inheritance_cache_get (ce=0x7f8b030011e0, parent=0x0, traits_and_interfaces=0x7ffec227ccf0) at /usr/src/debug/php-src-php-8.2.9/ext/opcache/ZendAccelerator.c:2305
#2  0x00007f8adb3117da in zend_do_link_class (ce=0x7f8b030011e0, lc_parent_name=<optimized out>, key=0x7f8aea795b28) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_inheritance.c:2925
#3  0x00007f8adb25bf2b in zend_bind_class_in_slot (class_table_slot=<optimized out>, lcname=0x7f8b0300bd40, lc_parent_name=0x0) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_compile.c:1124
#4  0x00007f8adb25c038 in do_bind_class (lcname=0x7f8b0300bd40, lc_parent_name=0x0) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_compile.c:1158
#5  0x00007f8adb2b2e55 in ZEND_DECLARE_CLASS_SPEC_CONST_HANDLER () at /usr/src/debug/php-src-php-8.2.9/Zend/zend_vm_execute.h:5419
#6  0x00007f8adb2e4970 in execute_ex (ex=0x7f8b030011e0) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_vm_execute.h:56485
#7  0x00007f8adb27320d in zend_call_function (fci_cache=0x7ffec227d2b0, fci=<optimized out>) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_execute_API.c:947
#8  0x00007f8adb27320d in zend_call_function (fci=<optimized out>, fci_cache=<optimized out>) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_execute_API.c:749
#9  0x00007f8adb2735bd in zend_call_known_function (fn=0x7f8afe9048f8, object=<optimized out>, called_scope=<optimized out>, retval_ptr=<optimized out>, param_count=<optimized out>, params=<optimized out>, named_params=0x0)
    at /usr/src/debug/php-src-php-8.2.9/Zend/zend_execute_API.c:1041
#10 0x00007f8adb148e42 in spl_perform_autoload (class_name=0x7f8aea77e838, lc_name=0x7f8aea795b28) at /usr/src/debug/php-src-php-8.2.9/ext/spl/php_spl.c:445
#11 0x00007f8adb27249c in zend_lookup_class_ex (name=name@entry=0x7f8aea77e838, key=0x7f8aea795b28, flags=flags@entry=512) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_execute_API.c:1207
#12 0x00007f8adb273942 in zend_fetch_class_by_name (class_name=0x7f8aea77e838, key=<optimized out>, fetch_type=512) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_execute_API.c:1703
#13 0x00007f8adb2bc907 in ZEND_NEW_SPEC_CONST_UNUSED_HANDLER () at /usr/src/debug/php-src-php-8.2.9/Zend/zend_vm_execute.h:10277
#14 0x00007f8adb2e4b17 in execute_ex (ex=0x7f8b030011e0) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_vm_execute.h:56949
#15 0x00007f8adb2ed923 in zend_execute (op_array=0x7f8abfc73000, return_value=0x0) at /usr/src/debug/php-src-php-8.2.9/Zend/zend_vm_execute.h:60408
#16 0x00007f8adb2816e0 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/debug/php-src-php-8.2.9/Zend/zend.c:1827
#17 0x00007f8adb21f1df in php_execute_script (primary_file=<optimized out>) at /usr/src/debug/php-src-php-8.2.9/main/main.c:2544
#18 0x00007f8adb35de68 in php_handler (r=<optimized out>) at /usr/src/debug/php-src-php-8.2.9/sapi/apache2handler/sapi_apache2.c:710
#19 0x000055fe187de080 in ap_run_handler ()
#20 0x000055fe187de5c9 in ap_invoke_handler ()
#21 0x000055fe187f29cc in ap_internal_redirect ()
#22 0x00007f8ae06d3f3c in handler_redirect () at /etc/httpd/modules/mod_rewrite.so
#23 0x000055fe187de080 in ap_run_handler ()
#24 0x000055fe187de5c9 in ap_invoke_handler ()
#25 0x000055fe187f366a in ap_process_async_request ()
#26 0x000055fe187f395b in ap_process_request ()
#27 0x000055fe187ef7b2 in ap_process_http_connection ()
#28 0x000055fe187e77d0 in ap_run_process_connection ()
#29 0x00007f8adf6ab7ef in child_main () at /etc/httpd/modules/mod_mpm_prefork.so
#30 0x00007f8adf6aba35 in  () at /etc/httpd/modules/mod_mpm_prefork.so
#31 0x00007f8adf6ac1a1 in prefork_run () at /etc/httpd/modules/mod_mpm_prefork.so
#32 0x000055fe187c23ee in ap_run_mpm ()
#33 0x000055fe187bb12a in main ()

(gdb) p entry
$1 = (zend_inheritance_cache_entry *) 0x10f

(gdb) p *entry
Cannot access memory at address 0x10f

(gdb) up
#1  zend_accel_inheritance_cache_get () at /usr/src/debug/php-src-php-8.2.9/ext/opcache/ZendAccelerator.c:2305
2305			entry = zend_accel_inheritance_cache_find(entry, ce, parent, traits_and_interfaces, &needs_autoload);
(gdb) l
2300		uint32_t i;
2301		bool needs_autoload;
2302		zend_inheritance_cache_entry *entry = ce->inheritance_cache;
2303	
2304		while (entry) {
2305			entry = zend_accel_inheritance_cache_find(entry, ce, parent, traits_and_interfaces, &needs_autoload);
2306			if (entry) {
2307				if (!needs_autoload) {
2308					replay_warnings(entry->num_warnings, entry->warnings);
2309					if (ZCSG(map_ptr_last) > CG(map_ptr_last)) {

(gdb) p parent
$3 = (zend_class_entry *) 0x0

(gdb) p ce
$6 = (zend_class_entry *) 0x7f8b030011e0
(gdb) p *ce
$7 = {type = 2 '\002', name = 0x7f8aea77e838, {parent = 0x0, parent_name = 0x0}, refcount = 1, ce_flags = 4199040, default_properties_count = 4, default_static_members_count = 0, default_properties_table = 0x7f8b0300b928, 
  default_static_members_table = 0x0, static_members_table__ptr = 0x0, function_table = {gc = {refcount = 1, u = {type_info = 7}}, u = {v = {flags = 16 '\020', _unused = 0 '\000', nIteratorsCount = 0 '\000', 
        _unused2 = 0 '\000'}, flags = 16}, nTableMask = 4294967264, {arHash = 0x7f8b03001458, arData = 0x7f8b03001458, arPacked = 0x7f8b03001458}, nNumUsed = 14, nNumOfElements = 14, nTableSize = 16, nInternalPointer = 0, 
    nNextFreeElement = -9223372036854775808, pDestructor = 0x7f8adb2768e0 <zend_function_dtor>}, properties_info = {gc = {refcount = 1, u = {type_info = 7}}, u = {v = {flags = 16 '\020', _unused = 0 '\000', 
        nIteratorsCount = 0 '\000', _unused2 = 0 '\000'}, flags = 16}, nTableMask = 4294967280, {arHash = 0x7f8b0300b9a8, arData = 0x7f8b0300b9a8, arPacked = 0x7f8b0300b9a8}, nNumUsed = 4, nNumOfElements = 4, nTableSize = 8, 
    nInternalPointer = 0, nNextFreeElement = -9223372036854775808, pDestructor = 0x0}, constants_table = {gc = {refcount = 1, u = {type_info = 7}}, u = {v = {flags = 24 '\030', _unused = 0 '\000', nIteratorsCount = 0 '\000', 
        _unused2 = 0 '\000'}, flags = 24}, nTableMask = 4294967294, {arHash = 0x7f8ae6000118, arData = 0x7f8ae6000118, arPacked = 0x7f8ae6000118}, nNumUsed = 0, nNumOfElements = 0, nTableSize = 8, nInternalPointer = 0, 
    nNextFreeElement = -9223372036854775808, pDestructor = 0x0}, mutable_data__ptr = 0x6aac9, inheritance_cache = 0x7f8b01ae3c48, properties_info_table = 0x0, constructor = 0x7f8b03001618, destructor = 0x0, clone = 0x0, 
  __get = 0x0, __set = 0x0, __unset = 0x0, __isset = 0x0, __call = 0x0, __callstatic = 0x0, __tostring = 0x0, __debugInfo = 0x0, __serialize = 0x0, __unserialize = 0x0, iterator_funcs_ptr = 0x0, arrayaccess_funcs_ptr = 0x0, {
    create_object = 0x0, interface_gets_implemented = 0x0}, get_iterator = 0x0, get_static_method = 0x0, serialize = 0x0, unserialize = 0x0, num_interfaces = 0, num_traits = 1, {interfaces = 0x0, interface_names = 0x0}, 
  trait_names = 0x7f8b0300bd30, trait_aliases = 0x0, trait_precedences = 0x0, attributes = 0x0, enum_backing_type = 0, backed_enum_table = 0x0, info = {user = {filename = 0x7f8b03001988, line_start = 46, line_end = 634, 
      doc_comment = 0x7f8b0300bbf8}, internal = {builtin_functions = 0x7f8b03001988, module = 0x27a0000002e}}}
(gdb) p *ce->inheritance_cache
$8 = {next = 0x10f, ce = 0x4, parent = 0x3, dependencies = 0x4, dependencies_count = 4, num_warnings = 0, warnings = 0x4, traits_and_interfaces = {0x4700000002}}

(gdb) zbacktrace 
[0x7f8abfc1a630] PHP_CodeSniffer\Autoload->(main) /home/rasmus/development/test/vendor/google/cloud-bigtable/src/Table.php:46 
[0x7f8abfc1a580] PHP_CodeSniffer\Autoload->loadFile("/home/rasmus/development/test/vendor/google/cloud-bigtable/src/Table.php") /home/rasmus/development/test/vendor/squizlabs/php_codesniffer/autoload.php:169 
[0x7f8abfc1a480] PHP_CodeSniffer\Autoload->load("Google\Cloud\Bigtable\Table") /home/rasmus/development/test/vendor/squizlabs/php_codesniffer/autoload.php:134
...

Line 46 there is just the start of the class definition, see https://github.com/googleapis/google-cloud-php-bigtable/blob/main/src/Table.php#L46 and it is autoloaded at https://github.com/squizlabs/PHP_CodeSniffer/blob/master/autoload.php#L169

[rlerdorf]

@dstogov does it ring any bells?

[dstogov]

Hi @rlerdorf, @Prochy

I don't have an answer out of the box. It's clear that ce->inheritance_cache value is wrong, but I can't guess how it became wrong.

Recently we had a problem in inheritance cache, but it should be fixed via 44e5c04

Try to provide more details about the crash. e.g. if it crashes on first requests after startup? directly after opcache reset? do you use preloading or file cache?

Ideally I should be able to reproduce some scenario.
For now I can't do anything.

[Prochy]

Unfortunately, I don't have any reproducible example. It doesn't happen immediately, it starts happening after several hours of running application, then it keeps crashing for several minutes and then it is fine and after x hours the same.

But we tried to upgrade to the latest PHP 8.2.10 and after 24 hours no crash so it looks promising. But it is weird to me because I checked the change logs for 8.2.9 and 8.2.10 and I didn't find there anything that could potentially solve the issue with OPCache. I think after weekend I will be able to say if it's gone or not.

[dstogov]

There are also no direclty related changes between 8.2.6 and 8.2.8 (the last one was in April).
The crash may be caused by some race condition on some specific request(s), it way occur by luck.
Of course, it would be great to catch the reason.

[rlerdorf]

Ok, I poked at it some more. No preloading or anything fancy here. But I can reliably reproduce it by doing an Apache graceful restart. So yes, this happens on a cache reset.

[dstogov]

Oh, you use mod_php. I thought everybody switched to php-fpm. :)
Anyway, this's great if the crash is better reproducible in your environment.

do you see the crash every time after Apache graceful restart or only from time to time?
can you reproduce the crash by restarting and accessing some page(s) in isolated environment (without production load and ideally with a single worker process)?

[rlerdorf]

Yeah, legacy system with mod_php. It looks tricky to reproduce without load. But I will see what I can do.

[Prochy]

We used the fix from @turchanov and so far so good. No crash over weekend and usually it crashes at least once a day.

[dstogov]

The patch fixed a problem related to support of inheritance cache in file cache. Do you use file cache? (I asked about this, but didn't get the answer). If yes, please close the issue.