Allow for configuring worker_connections

[sjberman]

As an NGF user
I want to be able to configure the worker_connections directive
So that I can run NGF at scale without needing too many nginx replicas

worker_connections sets the number of simultaneous connections that can be opened by an nginx worker process. Our default value is 1024. Users at scale may see the error [warn] 1180#1180: 1024 worker_connections are not enough, reusing connections, and therefore need to increase the number of allowed connections.

A workaround for this is to scale nginx to more replicas, but at certain scales, it may be more resource-friendly to increase this value instead of scaling replicas.

Acceptance

• allow users to configure the worker_connections directive
• likely a new field in the NginxProxy resource, to be set in the main nginx config

[mpstefan]

Makes sense.

Under what specific circumstances is increasing this number better than increasing the number of replicas? Do we run into this right away with one/two replicas, or is it a specific edge case?

[sjberman]

Here's the original discussion: #3368