Skip to content

[skip ci] Document how to quickly check if jit .dasc files transpile, how to test the jit in different architectures. #7768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 19, 2021
Merged

[skip ci] Document how to quickly check if jit .dasc files transpile, how to test the jit in different architectures. #7768

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0636117
[skip ci] Document how to check if zend_jit_arm64.dasc transpiles
TysonAndre Dec 12, 2021
c341ea3
Address review comments
TysonAndre Dec 19, 2021
617db1c
[skip ci] Update comment
TysonAndre Dec 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions ext/opcache/jit/Dockerfile.arm64.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Force this to build with arm64 even when the host architecture is different.
# This requires that cross-compilation support be enabled with the steps in https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/
FROM --platform=arm64 ubuntu:20.04
RUN apt-get update -y
# DEBIAN_FRONTEND=noninteractive is needed to stop the tzdata installation from hanging.
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get install -y tzdata
RUN apt-get install -y pkg-config build-essential autoconf bison re2c \
libxml2-dev libsqlite3-dev

ADD . /php-src/
WORKDIR /php-src
RUN ./buildconf
# Compile a minimal debug build. --enable-debug adds runtime assertions and is slower than regular builds.
RUN ./configure --enable-debug --disable-all --enable-opcache && make clean && make -j$(nproc)
116 changes: 115 additions & 1 deletion ext/opcache/jit/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,119 @@ was almost 100 times slower, making it prohibitively expensive to use.
[The unofficial DynASM Documentation](https://corsix.github.io/dynasm-doc/tutorial.html)
has a tutorial, reference, and instruction listing.

`zend_jit_x86.dasc` gets automatically converted to `zend_jit_x86.c` by the bundled
In x86 builds, `zend_jit_x86.dasc` gets automatically converted to `zend_jit_x86.c` by the bundled
`dynasm` during `make`.

In arm64 builds, `zend_jit_arm64.dasc` gets automatically converted to `zend_jit_arm64.c` by the bundled
`dynasm` during `make`.

Running tests of the JIT
------------------------

Then, to test the JIT, e.g. with opcache.jit=tracing, an example command
based on what is used to test in Azure CI:

```
make test TESTS="-d opcache.jit_buffer_size=16M -d opcache.enable=1 -d opcache.enable_cli=1 -d opcache.protect_memory=1 -d opcache.jit=tracing --repeat 2 --show-diff -j$(nproc) ext/opcache Zend"
```

- `opcache.jit_buffer_size=16M` enables the JIT in tests by providing 16 megabytes of
memory to use with the JIT to test with.
- `opcache.protect_memory=1` will detect writing to memory that is meant to be
read-only, which is sometimes the cause of opcache bugs.
- `--repeat 2` is optional, but used in CI since some JIT bugs only show up after processing a
request multiple times (the first request compiles the trace and the second executes it)
- `-j$(nproc)` runs as many workers to run tests as there are CPUs.
- `ext/opcache/` and `Zend` are the folders with the tests to run, in this case opcache
and the Zend engine itself. If no folders are provided, all tests are run.

When investigating test failures such as segmentation faults,
configuring the build of php with `--enable-address-sanitizer` to enable
[AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer) is often useful.

Some of the time, adding `-m --show-mem` to the `TESTS` configuration is also useful to test with [valgrind](https://valgrind.org/) to detect out of bounds memory accesses.
This is less effective at detecting or explaining invalid memory accesses than AddressSanitizer, but does not require rebuilding php.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really less effective, just much slower. You can rebuild PHP in the time it takes to run a handful of tests under valgrind ;)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#short-version

I mean that I thought AddressSanitizer would detect the computation of an invalid address, which might be distanced from the use (read/write) of the invalid address that caused misbehaviors.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, address sanitizer does not detect invalid address calculations. Though I believe ubsan does, at least to a limited degree.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. Missed the * when reading that.


Note that the JIT supports 3 different architectures: `X86_64`, `i386`, and `arm64`.

Miscellaneous
-------------

### Checking dasc files for in a different architecture

The following command can be run to manually check if the modified `.dasc code` is at least transpilable
for an architecture you're not using, e.g.:

For arm64: `ext/opcache/minilua ext/opcache/jit/dynasm/dynasm.lua -D ARM64=1 -o ext/opcache/jit/zend_jit_arm64.ignored.c ext/opcache/jit/zend_jit_arm64.dasc`

For x86_64: `ext/opcache/minilua ext/opcache/jit/dynasm/dynasm.lua -D X64=1 -o ext/opcache/jit/zend_jit_x86.ignored.c ext/opcache/jit/zend_jit_x86.dasc`

For i386 (i.e. 32-bit): `ext/opcache/minilua ext/opcache/jit/dynasm/dynasm.lua -o ext/opcache/jit/zend_jit_x86.ignored.c ext/opcache/jit/zend_jit_x86.dasc`

### How to build 32-bit builds on x86_64 environments

Refer to [../../../azure/i386](../../../azure/i386/apt.yml) for examples of
dependencies to install.

If you are running this natively (outside of Docker or a VM):

- Consider running in docker/a VM instead if you are unfamiliar with this.
- Avoid purging packages.
- Avoid `-y` - if the package manager warns you that the dependencies conflict
then **don't** try to force install them.

#### Prerequisites for 32-bit builds

This assumes you are using a Debian-based Linux distribution and have already
set up prerequisites for regular development.

```
sudo dpkg --add-architecture i386
sudo apt-get update -y
# As well as anything else from azure/i386/apt.yml that you're testing locally
sudo apt-get install \
gcc-multilib g++-multilib \
libxml2-dev:i386 \
libc6:i386
```

#### Compiling 32-bit builds

This assumes you are using a Debian-based Linux distribution and have already
set up prerequisites for 32-bit development.

```
export LDFLAGS=-L/usr/lib/i386-linux-gnu
export CFLAGS='-m32'
export CXXFLAGS='-m32'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line probably isn't needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's copied from azure/i386/job.yml - it'll prevent issues if anyone adapts these instructions to --enable-intl. ./ext/intl/breakiterator/breakiterator_class.cpp

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe our CFLAGS also applies to C++ code, but I can't be bothered to double check...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

ext/intl/intl_convertcpp.lo: /path/to/php-src/ext/intl/intl_convertcpp.cpp
	$(LIBTOOL) --mode=compile $(CXX) -Iext/intl/ -I/path/to/php-src/ext/intl/ $(COMMON_FLAGS) $(CXXFLAGS_CLEAN) $(EXTRA_CXXFLAGS)  -DU_NO_DEFAULT_INCLUDE_UTF_HEADERS=1 -DU_HIDE_OBSOLETE_UTF_OLD_H=1 -Wno-write-strings -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -std=c++11  -DUNISTR_FROM_CHAR_EXPLICIT=explicit -DUNISTR_FROM_STRING_EXPLICIT=explicit -c /path/to/php-src/ext/intl/intl_convertcpp.cpp -o ext/intl/intl_convertcpp.lo  -MMD -MF ext/intl/intl_convertcpp.dep -MT ext/intl/intl_convertcpp.lo

main/strlcpy.lo: /path/to/php-src/main/strlcpy.c
	$(LIBTOOL) --mode=compile $(CC) -Imain/ -I/path/to/php-src/main/ $(COMMON_FLAGS) $(CFLAGS_CLEAN) $(EXTRA_CFLAGS) -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -c /path/to/php-src/main/strlcpy.c -o main/strlcpy.lo  -MMD -MF main/strlcpy.dep -MT main/strlcpy.lo

If I add CFLAGS=-Wunused and then run ./config.nice, the only place -Wunused gets added is CFLAGS_CLEAN.

CFLAGS = $(CFLAGS_CLEAN) -prefer-non-pic -static
CFLAGS_CLEAN = -fno-common -Wformat-truncation -Wlogical-op -Wduplicated-cond -Wno-clobbered -Wall -Wextra -Wno-strict-aliasing -Wno-unused-parameter -Wno-sign-compare -Wunused -fvisibility=hidden -O0 -Wimplicit-fallthrough=1 -g -DZEND_SIGNALS $(PROF_FLAGS)
CPP = cc -E
CPPFLAGS = -D_GNU_SOURCE
CXX = g++
CXXFLAGS = -g -O0 -prefer-non-pic -static $(PROF_FLAGS)
CXXFLAGS_CLEAN = -g -O0

export PKG_CONFIG=/usr/bin/i686-linux-gnu-pkg-config
./configure --disable-all --enable-opcache --build=i686-pc-linux-gnu
make -j$(nproc)
```

#### Running tests of the JIT on 32-bit builds

See the section "Running tests of the JIT".

### Testing the jit with arm64 on x86 computers

https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/
may be useful for local development.

Note that this is slower than compiling and testing natively.

```
# After following steps in https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/
cp .gitignore .dockerignore
echo .git >> .dockerignore

docker build --network=host -t php-src-arm64-example -f ext/opcache/jit/Dockerfile.arm64.example .
docker run -it --rm php-src-arm64-example
```

Then, the docker image can be used to run tests with `make test`.
For example, to test `ext/opcache` in parallel with the tracing JIT enabled:

```
docker run -it php-src-arms-example make test TESTS="-d opcache.jit_buffer_size=16M -d opcache.enable=1 -d opcache.enable_cli=1 -d opcache.protect_memory=1 -d opcache.jit=tracing --repeat 2 --show-diff -j$(nproc) ext/opcache"
```
2 changes: 1 addition & 1 deletion sapi/fuzzer/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Fuzzing SAPI for PHP
--------------------

The following `./configure` options can be used to enable the fuzzing SAPI, as well as all availablefuzzers. If you don't build the exif/json/mbstring extensions, fuzzers for these extensions will not be built.
The following `./configure` options can be used to enable the fuzzing SAPI, as well as all available fuzzers. If you don't build the exif/json/mbstring extensions, fuzzers for these extensions will not be built.

```sh
CC=clang CXX=clang++ \
Expand Down