Open
Description
Problem
When running multiple instances of an application that uses PgPool, we observed a connection storm issue that leads to uneven load distribution across database readers.
Scenario
- 10 application instances
- Each instance: 2x large with 8 verticles
- Database setup: Multiple read replicas behind PgPool
- Issue timing: Application startup
Current Behavior
When multiple application instances start simultaneously:
- All verticles attempt to establish database connections at nearly the same time
- This creates a "connection storm" where most connections are directed to the same reader
- Results in skewed connection distribution across available readers
- Some readers become overloaded, while others are underutilized
Example from Load Test
In our load test with 10 instances (8 verticles each):
- Most connections were established to Reader-1
- Created uneven load distribution
- Impacted application performance
- Reduced the effectiveness of having multiple readers
Attaching a screenshot of connections made on readers. One of the readers has 227 connections, and the other has 72.
Proposed Solution
Introduce connection jitter to randomize connection timing:
- Add
jitter
parameter to PoolOptions - When setting
maxLifetime
, apply random jitter within the specified range - This spreads out connection creation/renewal across a time window