Skip to content

Fix flaky conneciton count in mysql test #18030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

iluuu1994
Copy link
Member

@iluuu1994 iluuu1994 commented Mar 12, 2025

Mark this test as conflicting with "all" to run it in isolation. Otherwise, other tests may influence the servers connection count.

Context: https://github.com/php/php-src/actions/runs/13801750596/job/38605337488

Mark this test as conflicting with "all" to run it in isolation.
Otherwise, other tests may influence the servers connection count.
@iluuu1994 iluuu1994 requested a review from cmb69 March 12, 2025 11:18
@iluuu1994
Copy link
Member Author

/cc @NattyNarwhal I think this may be a more appropriate fix. This can also happen on other runners, but apparently it has been more of a rare occurrence there, for reasons that are not apparent to me.

Comment on lines 3 to +4
--EXTENSIONS--
mysqli
all
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the --EXTENSIONS-- section.

Marking as conflicting with all would only solve the problem if there is another test not properly marked as conflicting with mysql. Possibly, there are other connections (not caused by our tests) involved anyway?

Copy link
Member Author

@iluuu1994 iluuu1994 Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I had a complete brain fart. 🙂 I'm not sure this makes sense at all. And yes, as you correctly assumed I read this as --CONFLICTS--.

@iluuu1994
Copy link
Member Author

This made no sense at all. The CONFLICTS file is still there (I thought maybe tests were adjusted to run in parallel), so there are not ran in parallel anyway. Sorry for the noise. 😄

@iluuu1994 iluuu1994 closed this Mar 12, 2025
@cmb69
Copy link
Member

cmb69 commented Mar 12, 2025

I think this needs closer investigation. Why are there 163/164 connections to the server? Are these mostly left-over persistent connections? If so, it might be a good idea to clean these up after the test runs.

And for this particular test, can't we "just" check whether we are actually reusing the old connection, instead of counting the connections?

@NattyNarwhal
Copy link
Member

For the PPC runner at least, the mysqli tests are unusually slow (they might be unusually impacted by *San?), so if there's another test with a connection going, it would be more likely to trigger the issue. There's the CONFLICTS file, but maybe i.e. a pdo_mysql test (or some other test using MySQL) also going at the same time?

@iluuu1994
Copy link
Member Author

iluuu1994 commented Mar 12, 2025

There's the CONFLICTS file, but maybe i.e. a pdo_mysql test (or some other test using MySQL) also going at the same time?

We use the same conflicts key for mysqli and pdo_mysql (mysql), they should all run sequentially. Possibly GH itself connects to services for some reason, e.g. health checks?

@NattyNarwhal
Copy link
Member

My stupid untested hypothesis is IIRC, isn't PDO a bit weird with running its tests, by running generic PDO tests with various PDO drivers (and also IIRC spawning the driver-specific tests)? Perhaps it's running i.e. a generic PDO test with pdo_mysql, and throwing off a mysqli test that way. I'm not sure how it respects CONFLICTS when run that way too.

@iluuu1994
Copy link
Member Author

@NattyNarwhal I would expect that to cause much more issues, especially before #11879. Might it be possible for you to dump the entire list of connections, given that this is relatively well reproducible on PPC? Maybe that may offer some insight.

@iluuu1994
Copy link
Member Author

@NattyNarwhal Another thing: The comment says the PPC machine has 8711 connections. Is this also true when the runner isn't executing any jobs? Why is this number so high?

@NattyNarwhal
Copy link
Member

Without any tests running:

$ echo "SHOW STATUS LIKE 'Connections'" | mysql -u ci -p
Variable_name	Value
Connections	22712

@iluuu1994
Copy link
Member Author

Thanks! That's a very high value. 😄 Are you using this machine for other purposes? Can you see who's on the other end of those connections?

@NattyNarwhal
Copy link
Member

Nope, it's just dedicated to CI. I don't see any open connections:

$ netstat -a | grep mysql
tcp        0      0 localhost:mysql         0.0.0.0:*               LISTEN     
unix  2      [ ACC ]     STREAM     LISTENING     7483461  /var/run/mysqld/mysqlx.sock
unix  2      [ ACC ]     STREAM     LISTENING     7452143  /var/run/mysqld/mysqld.sock

I'm pretty sure Connections is just raised monotically whenever a connection occurs, so running a lot of tests a few times could make it plausibly high.

@iluuu1994
Copy link
Member Author

Ah, you're right!

https://dev.mysql.com/doc/refman/8.4/en/server-status-variables.html#statvar_Connections

The number of connection attempts (successful or not) to the MySQL server.

That makes the test more sensible. It might still be better to compare the connection ID. Still doesn't explain why that number increases when no other tests are running...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants