`make test` Privacy Issues

[TurtleWilly]

Description

After I made a recent builds of PHP it told me to run make test to make sure the build also properly functions as designed. Happily I obliged and typed make test only to be greeted by numerous alerts by my outgoing firewall. Random tests in the test suite tried to make outwards connections to random services. For example, I saw connection attempts to:

• httpbin.org
• yahoo.com
• php.net
• w3.org
• example.org
• example.com
• *.googleapis.com
• github.com
• *.badssl.com
• *.google.com
• and more

In no way I was properly informed as the user that those requests would be made nor what the privacy policies for the contacted services are. I did NOT consent to that! Luckily I was running an outgoing firewall. Of course, me blocking those things made many tests fail (which ultimately made it hard to decide if the build of PHP properly passed the test suite or not.)

What should/ could be done:

• tests that make any outwards connection should be opt-in (e.g. with an extra parameter to make test, e.g. make test ENABLE_NETWORK_REQUIRING_TESTS=1 or something)
• for all contacted services the users should be informed about the service's "Privacy Policy" beforehand, so they can make an educated decision
• ideally all remote tests (if really required) would be hosted on php.net, e.g. under a "testsuite.php.net" subdomain or similar (aka on a service endpoint directly associated with your project/ under your control, not some random 3rd party), then you
  only would need to display a link to your very own privacy policy.

PHP can run (e.g. eval) remotely downloaded code. So some extra consideration to the users should be done here. I did not check/ verify the test that downloads something from httpbin: But what does it download? Some code? Who would control this code? What if it's compromised (e.g. we've seen recently with the xz upstream compromise what kind of risks exists).

While this is a bit rambling bug report here (sorry about that), I hope some improvements into a more privacy oriented direction may come out of it. For now I probably won't run make test again for my next PHP build: a burnt child dreads the fire.

PHP Version

PHP 8.3.6

Operating System

OS X 10.10.5 (Yosemite)

[iluuu1994]

Hi @TurtleWilly.

You can declare the SKIP_ONLINE_TESTS environment variable to skip these tests. However, I think your request to make this opt-in is reasonable.

I did not check/ verify the test that downloads something from httpbin: But what does it download? Some code? Who would control this code? What if it's compromised (e.g. we've seen recently with the xz upstream compromise what kind of risks exists).

Certainly not. The test simply checks if an HTTP response is received.

[TurtleWilly]

@iluuu1994

If the information about this already existing way to opt-out could be made available in the current part of Makefile.global then this would be a huge improvement already. Currently there's no information about this there at all:

…
all: $(all_targets)
	@echo
	@echo "Build complete."
	@echo "Don't forget to run 'make test'."
	@echo 
…

Also that probably would be an easy change to do, I think. 😎

[petk]

In this context, also things like these should be changed:

diff --git a/sapi/fpm/status.html.in b/sapi/fpm/status.html.in
index b87a446f59..063499184f 100644
--- a/sapi/fpm/status.html.in
+++ b/sapi/fpm/status.html.in
@@ -72,7 +72,7 @@
                </div>
                <p>
                        <a href="http://validator.w3.org/check?uri=referer">
-                               <img src="http://www.w3.org/Icons/valid-xhtml10" alt="Valid XHTML 1.0 Transitional" height="31" width="88" />
+                               <!--<img src="http://www.w3.org/Icons/valid-xhtml10" alt="Valid XHTML 1.0 Transitional" height="31" width="88" />-->
                        </a>
                 </p>
                <script type="text/javascript">

For example, to not expose IP address to W3.org site, image could be embedded in the HTML itself. There might be few more of these.