From 06361174eeb73ac13042bd450108fa35fe9fec89 Mon Sep 17 00:00:00 2001 From: Tyson Andre Date: Sun, 12 Dec 2021 12:22:35 -0500 Subject: [PATCH 1/3] [skip ci] Document how to check if zend_jit_arm64.dasc transpiles --- ext/opcache/jit/Dockerfile.arm64.example | 15 +++ ext/opcache/jit/README.md | 113 ++++++++++++++++++++++- 2 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 ext/opcache/jit/Dockerfile.arm64.example diff --git a/ext/opcache/jit/Dockerfile.arm64.example b/ext/opcache/jit/Dockerfile.arm64.example new file mode 100644 index 0000000000000..e7b6a03b2db89 --- /dev/null +++ b/ext/opcache/jit/Dockerfile.arm64.example @@ -0,0 +1,15 @@ +# Force this to build with arm64 even when the host architecture is different. +# This requires that cross-compilation support be enabled with the steps in https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/ +FROM --platform=arm64 ubuntu:20.04 +RUN apt-get update -y +# DEBIAN_FRONTEND=noninteractive is needed to stop the tzdata installation from hanging. +ENV DEBIAN_FRONTEND=noninteractive +RUN apt-get install -y tzdata +RUN apt-get install -y pkg-config build-essential autoconf bison re2c \ + libxml2-dev libsqlite3-dev + +ADD . /php-src/ +WORKDIR /php-src +RUN ./buildconf +# Compile a minimal debug build. --enable-debug adds runtime assertions and is slower than regular builds. +RUN ./configure --enable-debug --disable-all --enable-opcache && make clean && make -j$(nproc) diff --git a/ext/opcache/jit/README.md b/ext/opcache/jit/README.md index 2cc08d2827a23..2d141cedfb3d3 100644 --- a/ext/opcache/jit/README.md +++ b/ext/opcache/jit/README.md @@ -24,5 +24,116 @@ was almost 100 times slower, making it prohibitively expensive to use. [The unofficial DynASM Documentation](https://corsix.github.io/dynasm-doc/tutorial.html) has a tutorial, reference, and instruction listing. -`zend_jit_x86.dasc` gets automatically converted to `zend_jit_x86.c` by the bundled +In x86 builds, `zend_jit_x86.dasc` gets automatically converted to `zend_jit_x86.c` by the bundled `dynasm` during `make`. + +In arm64 builds, `zend_jit_arm64.dasc` gets automatically converted to `zend_jit_arm64.c` by the bundled +`dynasm` during `make`. + +Running tests of the JIT +------------------------ + +Then, to test the JIT, e.g. with opcache.jit=tracing, an example command +based on what is used to test in Azure CI: + +``` +make test TESTS="-d opcache.jit_buffer_size=16M -d opcache.enable=1 -d opcache.enable_cli=1 -d opcache.protect_memory=1 -d opcache.jit=tracing --repeat 2 --show-diff -j$(nproc) ext/opcache" +``` + +- `opcache.jit_buffer_size=16M` enables the JIT in tests by providing 16 megabytes of + memory to use with the JIT to test with. +- `opcache.protect_memory=1` will detect writing to memory that is meant to be + read-only, which is sometimes the cause of opcache bugs. +- `--repeat 2` is optional, but used in CI since some JIT bugs only show up after processing a + request multiple times, e.g. due to a bug in the engine or bad data in the + runtime cache of the interpreter. +- `-j$(nproc)` runs as many workers to run tests as there are CPUs. +- `ext/opcache/` is the folder with the tests to run, in this case opcache + itself. If no folders are provided, all tests are run. + +When investigating test failures such as segmentation faults, adding +`-m --show-mem` may be useful to test with [valgrind](https://valgrind.org/) to detect out of bounds memory accesses. + +Note that the JIT supports 3 different architectures: `X86_64`, `i386`, and `arm64`. + +Miscellaneous +------------- + +### Checking dasc files for in a different architecture + +The following command can be run to manually check if the modified `.dasc code` is at least transpilable +for an architecture you're not using, e.g.: + +For arm64: `ext/opcache/minilua ext/opcache/jit/dynasm/dynasm.lua -D ARM64=1 -o ext/opcache/jit/zend_jit_arm64.ignored.c ext/opcache/jit/zend_jit_arm64.dasc` + +For x86_64: `ext/opcache/minilua ext/opcache/jit/dynasm/dynasm.lua -D X64=1 -o ext/opcache/jit/zend_jit_x86.ignored.c ext/opcache/jit/zend_jit_x86.dasc` + +For i386 (i.e. 32-bit): `ext/opcache/minilua ext/opcache/jit/dynasm/dynasm.lua -o ext/opcache/jit/zend_jit_x86.ignored.c ext/opcache/jit/zend_jit_x86.dasc` + +### How to build 32-bit builds on x86_64 environments + +Refer to [../../../azure/i386](../../../azure/i386/apt.yml) for examples of +dependencies to install. + +If you are running this natively (outside of Docker or a VM): + +- Consider running in docker/a VM instead if you are unfamiliar with this. +- Avoid purging packages. +- Avoid `-y` - if the package manager warns you that the dependencies conflict + then **don't** try to force install them. + +#### Prerequisites for 32-bit builds + +This assumes you are using a Debian-based Linux distribution and have already +set up prerequisites for regular development. + +``` +sudo dpkg --add-architecture i386 +sudo apt-get update -y +# As well as anything else from azure/i386/apt.yml that you're testing locally +sudo apt-get install \ + gcc-multilib g++-multilib \ + libxml2-dev:i386 \ + libc6:i386 +``` + +#### Compiling 32-bit builds + +This assumes you are using a Debian-based Linux distribution and have already +set up prerequisites for 32-bit development. + +``` +export LDFLAGS=-L/usr/lib/i386-linux-gnu +export CFLAGS='-m32' +export CXXFLAGS='-m32' +export PKG_CONFIG=/usr/bin/i686-linux-gnu-pkg-config +./configure --disable-all --enable-opcache --build=i686-pc-linux-gnu +make -j$(nproc) +``` + +#### Running tests of the JIT on 32-bit builds + +See the section "Running tests of the JIT". + +### Testing the jit with arm64 on x86 computers + +https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/ +may be useful for local development. + +Note that this is slower than compiling and testing natively. + +``` +# After following steps in https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/ +cp .gitignore .dockerignore +echo .git >> .dockerignore + +docker build --network=host -t php-src-arm64-example -f ext/opcache/jit/Dockerfile.arm64.example . +docker run -it --rm php-src-arm64-example +``` + +Then, the docker image can be used to run tests with `make test`. +For example, to test `ext/opcache` in parallel with the tracing JIT enabled: + +``` +docker run -it php-src-arms-example make test TESTS="-d opcache.jit_buffer_size=16M -d opcache.enable=1 -d opcache.enable_cli=1 -d opcache.protect_memory=1 -d opcache.jit=tracing --repeat 2 --show-diff -j$(nproc) ext/opcache" +``` From c341ea30f8c8b60e842944d6fa3fc3adc2e6300f Mon Sep 17 00:00:00 2001 From: Tyson Andre Date: Sun, 19 Dec 2021 08:21:51 -0500 Subject: [PATCH 2/3] Address review comments --- ext/opcache/jit/README.md | 17 ++++++++++------- sapi/fuzzer/README.md | 2 +- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/ext/opcache/jit/README.md b/ext/opcache/jit/README.md index 2d141cedfb3d3..a73debfae4dff 100644 --- a/ext/opcache/jit/README.md +++ b/ext/opcache/jit/README.md @@ -37,7 +37,7 @@ Then, to test the JIT, e.g. with opcache.jit=tracing, an example command based on what is used to test in Azure CI: ``` -make test TESTS="-d opcache.jit_buffer_size=16M -d opcache.enable=1 -d opcache.enable_cli=1 -d opcache.protect_memory=1 -d opcache.jit=tracing --repeat 2 --show-diff -j$(nproc) ext/opcache" +make test TESTS="-d opcache.jit_buffer_size=16M -d opcache.enable=1 -d opcache.enable_cli=1 -d opcache.protect_memory=1 -d opcache.jit=tracing --repeat 2 --show-diff -j$(nproc) ext/opcache Zend" ``` - `opcache.jit_buffer_size=16M` enables the JIT in tests by providing 16 megabytes of @@ -45,14 +45,17 @@ make test TESTS="-d opcache.jit_buffer_size=16M -d opcache.enable=1 -d opcache.e - `opcache.protect_memory=1` will detect writing to memory that is meant to be read-only, which is sometimes the cause of opcache bugs. - `--repeat 2` is optional, but used in CI since some JIT bugs only show up after processing a - request multiple times, e.g. due to a bug in the engine or bad data in the - runtime cache of the interpreter. + request multiple times (the first request compiles the trace and the second executes it) - `-j$(nproc)` runs as many workers to run tests as there are CPUs. -- `ext/opcache/` is the folder with the tests to run, in this case opcache - itself. If no folders are provided, all tests are run. +- `ext/opcache/` and `Zend` are the folders with the tests to run, in this case opcache + and the Zend engine itself. If no folders are provided, all tests are run. -When investigating test failures such as segmentation faults, adding -`-m --show-mem` may be useful to test with [valgrind](https://valgrind.org/) to detect out of bounds memory accesses. +When investigating test failures such as segmentation faults, +configuring the build of php with `--enable-address-sanitizer` to enable +[AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer) is often useful. + +Some of the time, adding `-m --show-mem` to the `TESTS` configuration is also useful to test with [valgrind](https://valgrind.org/) to detect out of bounds memory accesses. +This is less effective at detecting or explaining invalid memory accesses than AddressSanitizer, but does not require rebuilding php. Note that the JIT supports 3 different architectures: `X86_64`, `i386`, and `arm64`. diff --git a/sapi/fuzzer/README.md b/sapi/fuzzer/README.md index 2099a34bfbf40..63de20aa1ca76 100644 --- a/sapi/fuzzer/README.md +++ b/sapi/fuzzer/README.md @@ -1,7 +1,7 @@ Fuzzing SAPI for PHP -------------------- -The following `./configure` options can be used to enable the fuzzing SAPI, as well as all availablefuzzers. If you don't build the exif/json/mbstring extensions, fuzzers for these extensions will not be built. +The following `./configure` options can be used to enable the fuzzing SAPI, as well as all available fuzzers. If you don't build the exif/json/mbstring extensions, fuzzers for these extensions will not be built. ```sh CC=clang CXX=clang++ \ From 617db1c70302186c57d41f9dc5ecbbbdbfd56da8 Mon Sep 17 00:00:00 2001 From: Tyson Andre Date: Sun, 19 Dec 2021 10:11:11 -0500 Subject: [PATCH 3/3] [skip ci] Update comment --- ext/opcache/jit/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ext/opcache/jit/README.md b/ext/opcache/jit/README.md index a73debfae4dff..3119f37811cf8 100644 --- a/ext/opcache/jit/README.md +++ b/ext/opcache/jit/README.md @@ -55,7 +55,7 @@ configuring the build of php with `--enable-address-sanitizer` to enable [AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer) is often useful. Some of the time, adding `-m --show-mem` to the `TESTS` configuration is also useful to test with [valgrind](https://valgrind.org/) to detect out of bounds memory accesses. -This is less effective at detecting or explaining invalid memory accesses than AddressSanitizer, but does not require rebuilding php. +Using valgrind is slower at detecting invalid memory read/writes than AddressSanitizer when running large numbers of tests, but does not require rebuilding php. Note that the JIT supports 3 different architectures: `X86_64`, `i386`, and `arm64`.