Skip to content

Connection Storm Prevention with Jitter in Connection Pool #1495

Open
@priyanshu-d11

Description

@priyanshu-d11

Problem

When running multiple instances of an application that uses PgPool, we observed a connection storm issue that leads to uneven load distribution across database readers.

Scenario

  • 10 application instances
  • Each instance: 2x large with 8 verticles
  • Database setup: Multiple read replicas behind PgPool
  • Issue timing: Application startup

Current Behavior

When multiple application instances start simultaneously:

  1. All verticles attempt to establish database connections at nearly the same time
  2. This creates a "connection storm" where most connections are directed to the same reader
  3. Results in skewed connection distribution across available readers
  4. Some readers become overloaded, while others are underutilized

Example from Load Test

In our load test with 10 instances (8 verticles each):

  • Most connections were established to Reader-1
  • Created uneven load distribution
  • Impacted application performance
  • Reduced the effectiveness of having multiple readers

Attaching a screenshot of connections made on readers. One of the readers has 227 connections, and the other has 72.

Image

Proposed Solution

Introduce connection jitter to randomize connection timing:

  1. Add jitter parameter to PoolOptions
  2. When setting maxLifetime, apply random jitter within the specified range
  3. This spreads out connection creation/renewal across a time window

Implementation

Contribution

#1496

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions