failures when compiling PHP from source

[chaar123]

Description

update-crypto-policies --show
DEFAULT
configure; make; make test

amazing very few errors compared to my #12827

php_test_results_20231211_1341.txt

PHP Version

PHP 8.3.0

Operating System

fedora 39 server edition

[iluuu1994]

I've had a few instances of the "mprotect() failed [13] Permission denied" in JIT tests too, even with #12890, and without ASAN. I'm not sure what causes it... @dstogov Any idea?

[dstogov]

I'm on Fedora 39 spin and I didn't see these mprotect() failures yet.
For now I don't have any ideas why only a single test failed.
@iluuu1994 Do you see the failure only on ext/opcache/tests/jit/assign_005.phpt? always or from time to time? only on PHP-8.3 or master is also affected?

I see the same 3 openssl tests failures, but they are unrelated to mprotect() and opcache.

[iluuu1994]

@dstogov I can reproduce this issue only intermittently when running all JIT tests. E.g. when compiling without ASAN (because ASAN has disabled find_prefered_mmap_base since #12890).

make test TESTS="-j14 ext/opcache/tests/jit"

will yield a failure in one or two files every 2-3 runs.

mprotect() failed [13] Permission denied
mprotect() failed [13] Permission denied
mprotect() failed [13] Permission denied
mprotect() failed [13] Permission denied
mprotect() failed [13] Permission denied

Termsig=11

Removing find_prefered_mmap_base fixes the issue. But I'm not sure why.

[iluuu1994]

This happens for 8.3 as well. mprotect just sometimes refuses to set the given permissions.

mprotect(0xa800000, 1048576, PROT_READ|PROT_WRITE) = 0
mprotect(0xa800000, 1048576, PROT_READ|PROT_EXEC) = -1 EACCES (Permission denied)

It seems like it's always the last page. This is reproducible in this small C program:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h> 
#include <stddef.h>
#include <sys/mman.h>
#include <limits.h>
#include <errno.h>
#include <string.h>

#define ZEND_MM_ALIGNED_SIZE_EX(size, alignment) \
	(((size) + ((alignment) - 1)) & ~((alignment) - 1))

int main(void);

static void *find_prefered_mmap_base(size_t requested_size)
{
	size_t huge_page_size = 2 * 1024 * 1024;
	uintptr_t last_free_addr = 0;
	uintptr_t last_candidate = (uintptr_t)MAP_FAILED;
	uintptr_t start, end, text_start = 0;
	FILE *f;
	char buffer[PATH_MAX];

	f = fopen("/proc/self/maps", "r");
	if (!f) {
		return MAP_FAILED;
	}

	while (fgets(buffer, PATH_MAX, f) && sscanf(buffer, "%lx-%lx", &start, &end) == 2) {
		if ((uintptr_t)main >= start) {
			/* the current segment lays before PHP .text segment or PHP .text segment itself */
			/*Search for candidates at the end of the free segment near the .text segment
			  to prevent candidates from being missed due to large hole*/
			if (last_free_addr + requested_size <= start) {
				last_candidate = ZEND_MM_ALIGNED_SIZE_EX(start - requested_size, huge_page_size);
				if (last_candidate + requested_size > start) {
					last_candidate -= huge_page_size;
				}
			}
			if ((uintptr_t)main < end) {
				/* the current segment is PHP .text segment itself */
				if (last_candidate != (uintptr_t)MAP_FAILED) {
					if (end - last_candidate < UINT32_MAX) {
						/* we have found a big enough hole before the text segment */
						break;
					}
					last_candidate = (uintptr_t)MAP_FAILED;
				}
				text_start = start;
			}
		} else {
			/* the current segment lays after PHP .text segment */
			if (last_free_addr + requested_size - text_start > UINT32_MAX) {
				/* the current segment and the following segments lay too far from PHP .text segment */
				break;
			}
			if (last_free_addr + requested_size <= start) {
				last_candidate = last_free_addr;
				break;
			}
		}
		last_free_addr = ZEND_MM_ALIGNED_SIZE_EX(end, huge_page_size);

	}
	fclose(f);

	return (void*)last_candidate;
}

int main(void)
{
	const size_t size = 128 * 1024 * 1024;
	const int flags = PROT_READ | PROT_WRITE;
	void *hint = find_prefered_mmap_base(size);
	void *mem = mmap(hint, size, flags, MAP_SHARED|MAP_ANONYMOUS|MAP_FIXED, -1, 0);
	fprintf(stdout, "mmap: %p\n", mem);
	void *last_mb = (char *)mem + size - (1024 * 1024);
	if (mprotect(last_mb, 1024 * 1024, PROT_READ | PROT_EXEC) == -1) {
		fprintf(stderr, "mprotect() failed [%d] %s\n", errno, strerror(errno));
		abort();
	}
	munmap(mem, size);
}

$ repeat 1000 { ./test || break }
...
mmap: 0x1000000
mmap: 0xe00000
mmap: 0x2400000
mmap: 0x1e00000
mprotect() failed [13] Permission denied

[dstogov]

I can't reproduce this.

1. Try to extend test.c to print "/proc/self/maps" before the mmap() and after the failure, may be we have a bug in find_prefered_mmap_base() and the mmaped region somehow overlaps with heap.
2. The failure may be cause by selinux. Try to see some relevant info in the log sudo tail -n100 /var/log/audit/audit.lo

[iluuu1994]

Hmm, I indeed get an execheap error.

type=AVC msg=audit(1702340537.237:33455): avc: denied { execheap } for pid=1189284 comm="test" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0

00400000-00401000 r--p 00000000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00401000-00402000 r-xp 00001000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00402000-00403000 r--p 00002000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00403000-00404000 r--p 00002000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00404000-00405000 rw-p 00003000 00:21 86811775                           /home/ilutov/Developer/php-src/test
005df000-00600000 rw-p 00000000 00:00 0                                  [heap]
7fbed1d2c000-7fbed1d2e000 rw-p 00000000 00:00 0 
7fbed1d2e000-7fbed1d54000 r--p 00000000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1d54000-7fbed1eb4000 r-xp 00026000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1eb4000-7fbed1f02000 r--p 00186000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1f02000-7fbed1f06000 r--p 001d3000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1f06000-7fbed1f08000 rw-p 001d7000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1f08000-7fbed1f12000 rw-p 00000000 00:00 0 
7fbed1f2c000-7fbed1f2d000 r--p 00000000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f2d000-7fbed1f54000 r-xp 00001000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f54000-7fbed1f5e000 r--p 00028000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f5e000-7fbed1f60000 r--p 00031000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f60000-7fbed1f62000 rw-p 00033000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7ffcd1da0000-7ffcd1dc2000 rw-p 00000000 00:00 0                          [stack]
7ffcd1dc5000-7ffcd1dc9000 r--p 00000000 00:00 0                          [vvar]
7ffcd1dc9000-7ffcd1dcb000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

mprotect failed: 0x8500000 - 0x8600000

00400000-00401000 r--p 00000000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00401000-00402000 r-xp 00001000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00402000-00403000 r--p 00002000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00403000-00404000 r--p 00002000 00:21 86811775                           /home/ilutov/Developer/php-src/test
00404000-00405000 rw-p 00003000 00:21 86811775                           /home/ilutov/Developer/php-src/test
005df000-00600000 rw-p 00000000 00:00 0                                  [heap]
00600000-08600000 rw-s 00000000 00:01 8787114                            /dev/zero (deleted)
7fbed1d2c000-7fbed1d2e000 rw-p 00000000 00:00 0 
7fbed1d2e000-7fbed1d54000 r--p 00000000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1d54000-7fbed1eb4000 r-xp 00026000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1eb4000-7fbed1f02000 r--p 00186000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1f02000-7fbed1f06000 r--p 001d3000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1f06000-7fbed1f08000 rw-p 001d7000 00:21 8062656                    /usr/lib64/libc.so.6
7fbed1f08000-7fbed1f12000 rw-p 00000000 00:00 0 
7fbed1f2c000-7fbed1f2d000 r--p 00000000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f2d000-7fbed1f54000 r-xp 00001000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f54000-7fbed1f5e000 r--p 00028000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f5e000-7fbed1f60000 r--p 00031000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7fbed1f60000-7fbed1f62000 rw-p 00033000 00:21 8062653                    /usr/lib64/ld-linux-x86-64.so.2
7ffcd1da0000-7ffcd1dc2000 rw-p 00000000 00:00 0                          [stack]
7ffcd1dc5000-7ffcd1dc9000 r--p 00000000 00:00 0                          [vvar]
7ffcd1dc9000-7ffcd1dcb000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

It doesn't immediately seem like this is overlapping with anything. So this might be an selinux bug I suppose.

[dstogov]

It doesn't immediately seem like this is overlapping with anything. So this might be an selinux bug I suppose.

The related SELinux code looks trivial.

https://github.com/torvalds/linux/blob/26aff849438cebcd05f1a647390c4aa700d5c0f1/security/selinux/hooks.c#L3817
https://github.com/torvalds/linux/blob/26aff849438cebcd05f1a647390c4aa700d5c0f1/include/linux/mm.h#L887

please also check the result of sbrk(0) after the failure. For me it's the same as [heap] end. but I don't see the failure.

Try to modify find_prefered_mmap_base() to insert a huge page gap between [heap] and the candidate address.

[iluuu1994]

sbrk(0) is consistent with what I see from what I see in /proc/self/maps. It indeed seems like the behavior was implicitly changed in torvalds/linux@68df1ba. Previously, it was tested whether vma was fully included in start_brk to brk. Now, with vma_is_initial_heap, it checks for vma->vm_start <= vma->vm_mm->brk && vma->vm_end >= vma->vm_mm->start_brk. If this wants to check for overlapping segments, it should not check for equality. Otherwise this will yield true if vma lies directly after `brk, as in our case.

Adding an empty huge page between the segments indeed seems to solve the issue for me.

diff --git a/ext/opcache/shared_alloc_mmap.c b/ext/opcache/shared_alloc_mmap.c
index ac1ee89c56..2d8eba5e6d 100644
--- a/ext/opcache/shared_alloc_mmap.c
+++ b/ext/opcache/shared_alloc_mmap.c
@@ -58,6 +58,7 @@ static void *find_prefered_mmap_base(size_t requested_size)
 #if defined(__linux__)
 	FILE *f;
 	char buffer[MAXPATHLEN];
+	uintptr_t brk = ZEND_MM_ALIGNED_SIZE_EX((uintptr_t)sbrk(0), huge_page_size);
 
 	f = fopen("/proc/self/maps", "r");
 	if (!f) {
@@ -65,6 +66,16 @@ static void *find_prefered_mmap_base(size_t requested_size)
 	}
 
 	while (fgets(buffer, MAXPATHLEN, f) && sscanf(buffer, "%lx-%lx", &start, &end) == 2) {
+		/* Don't place a segment directly after brk due to a selinux bug where a segment directly
+		 * following the heap is interpreted as heap memory. */
+		if (last_free_addr == brk) {
+			if ((last_free_addr + huge_page_size) < start) {
+				last_free_addr += huge_page_size;
+			} else {
+				last_free_addr = ZEND_MM_ALIGNED_SIZE_EX(end, huge_page_size);
+				continue;
+			}
+		}
 		if ((uintptr_t)execute_ex >= start) {
 			/* the current segment lays before PHP .text segment or PHP .text segment itself */
 			/*Search for candidates at the end of the free segment near the .text segment

I'll see where this can be reported.

[iluuu1994]

I created a bug report here: https://bugzilla.kernel.org/show_bug.cgi?id=218258 Shall we use the mitigation above in the meantime?

[dstogov]

sbrk(0) is consistent with what I see from what I see in /proc/self/maps. It indeed seems like the behavior was implicitly changed in torvalds/linux@68df1ba. Previously, it was tested whether vma was fully included in start_brk to brk. Now, with vma_is_initial_heap, it checks for vma->vm_start <= vma->vm_mm->brk && vma->vm_end >= vma->vm_mm->start_brk. If this wants to check for overlapping segments, it should not check for equality. Otherwise this will yield true if vma lies directly after `brk, as in our case.

I agree, that "equality" looks like a mistake. However, I'm not sure if it's a reason of failure in our case, because in test.c we mprotect() only a tail of the mmap()-ed region.

Adding an empty huge page between the segments indeed seems to solve the issue for me.

I afraid, we will have to use this as a workaround :(
God bless re-factoring...

I'll see where this can be reported.

This definitely should be reported. Thanks!

[iluuu1994]

However, I'm not sure if it's a reason of failure in our case, because in test.c we mprotect() only a tail of the mmap()-ed region.

That's indeed very peculiar. Anyway, let's see what the Linux people say 🙂

I'll create a PR for the workaround in the meantime.

[dstogov]

I'll create a PR for the workaround in the meantime.

It seems, we are not allowed to use memory areas that directly follow or precede the [heap] segment. In this case we will have to use a gap.