Skip to content

Increase script sleep in FPM process idle test #7561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

bukka
Copy link
Member

@bukka bukka commented Oct 5, 2021

No description provided.

@nikic
Copy link
Member

nikic commented Oct 6, 2021

Looks like it passes on azure now (including macos). There's a failure on Travis:

========DIFF========
001+ ERROR: Request failed; EXCEPTION: Read timed out
     Done
========DONE========
FAIL FPM: Process manager config pm.process_idle_timeout [sapi/fpm/tests/proc-idle-timeout.phpt]

@bukka bukka force-pushed the fpm_proc_idle_timeout_test_extra_sleep branch 3 times, most recently from d936dcf to 0c1ba54 Compare October 10, 2021 21:35
Copy link
Member

@nikic nikic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's give it a try ... will have to keep an eye on how flaky it will be.

@bukka bukka force-pushed the fpm_proc_idle_timeout_test_extra_sleep branch from 0c1ba54 to 76f9520 Compare October 17, 2021 17:39
@bukka
Copy link
Member Author

bukka commented Oct 17, 2021

After rebasing, it still seems a bit flaky as Travis failed again for this test with

========DIFF========

001+ ERROR: Request failed; EXCEPTION: Read timed out

Will take a look a bit more into it and play with the timeout in fcgi.inc that might need increasing.

@bukka bukka force-pushed the fpm_proc_idle_timeout_test_extra_sleep branch from 76f9520 to d77b3c6 Compare November 14, 2021 21:10
@bukka bukka force-pushed the fpm_proc_idle_timeout_test_extra_sleep branch from d77b3c6 to 6019922 Compare November 15, 2021 11:29
@TysonAndre
Copy link
Contributor

It seems like this was merged already as 2f8407f but the test is still flaky. I see #7690 spuriously failed (it included that commit) as https://dev.azure.com/phpazuredevops/PHP/_build/results?buildId=21824&view=logs&jobId=0d3730b8-0d59-574b-ab4a-5e189a74d53b&j=0d3730b8-0d59-574b-ab4a-5e189a74d53b&t=492e5677-f211-533f-274d-3a4af39d4cce.

This test case seems to be a common cause of spurious failures? Does removing it from CI runs make sense if it times out, even with a 7 second read timeout

It looks like sapi/fpm/tests/tester.inc is trying to listen on a ipv4 TCP port on localhost. Could it be breaking if it's running in parallel with another fpm test which is already listening on the same port? (Would --CONFLICTS-- help?)

    // Each test may specify a list of conflict keys. While a test that conflicts with
    // key K is running, no other test that conflicts with K may run. Conflict keys are
    // specified either in the --CONFLICTS-- section, or CONFLICTS file inside a directory.
    $dirConflictsWith = [];
    $fileConflictsWith = [];
    $sequentialTests = [];
    foreach ($test_files as $i => $file) {
        $contents = file_get_contents($file);
        if (preg_match('/^--CONFLICTS--(.+?)^--/ms', $contents, $matches)) {
    public function getPort(string $type = 'ip', $pool = 'default', $useAsId = false)
    {
        if ($type === 'uds' && !$useAsId) {
            return -1;
        }

        if (isset($this->ports['values'][$pool])) {
            return $this->ports['values'][$pool];
        }
        $port = ($this->ports['last'] ?? 9000 + PHP_INT_SIZE - 1) + 1;
        $this->ports['values'][$pool] = $this->ports['last'] = $port;

        return $port;
    }

(Or if that isn't why, returning early if $tester->multiRequest(2, null, null, null, false, 7000); throws "Read timed out" or any Exception, since other tests should cover the ability to respond to requests to some extent)

========DIFF========
001+ ERROR: Request failed; EXCEPTION: Read timed out

TysonAndre added a commit to TysonAndre/php-src that referenced this pull request Nov 27, 2021
See discussion on php#7561

fpm tests have been spuriously failing in CI, with 2 parallel workers
on 2 CPUs.

`make test TESTS='sapi/fpm/tests -j8'` fails locally for me
without this change

They're trying to listen on IPv4 on 127.0.0.1 on the same port as other
tests (e.g. helper getPort()=9008 on 64-bit architecture (PHP_INT_SIZE=8)).

So tests will fail if

1. php-fpm is still running from another test in parallel.
2. a request is sent to a server for a different test.
@bukka
Copy link
Member Author

bukka commented Nov 27, 2021

Yes I merged it already and forgot to close this so closing now.

It's not because of conflicts because the error would be about port being already used. This timeout doesn't make much sense to me as those 7s should cover it. I will try to increase it to 10s and see if it helps. If it doesn't and I don't get another idea I will probably disable it in pipeline as the used containers are just too weak (resource limited and slow) to test all the things in FPM.

@bukka
Copy link
Member Author

bukka commented Nov 27, 2021

At the end I decided to disable the test in the pipeline as I have been playing with it locally and noticed that pm.process_idle_timeout = 1 is never actually respected by FPM and there's a further delay - it's kind of a bug in FPM which makes making the test reliable work even harder. It doesn't explain why the read takes so long in the pipeline. To find that out, I will need to add some further debugging (at least proper collecting of debug logs to start with). To sum it up I will probably need to do some extra work in FPM as well as tester to make this work more reliably.

@bukka bukka closed this Nov 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants