Nginx Ingress Pods Consuming Too Much Memory

[ParagPatil96]

NGINX Ingress controller version

NGINX Ingress controller
Release: v1.1.1
Build: a17181e
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.9

---

Kubernetes version

version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.14-gke.1900", GitCommit:"abc4e63ae76afef74b341d2dba1892471781604f", GitTreeState:"clean", BuildDate:"2021-09-07T09:21:04Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GCP

• OS : Container-Optimized OS from Google

• Kernel : 5.4.129+

• How was the ingress-nginx-controller installed:

  • We have used helm to install Nginx Ingress Controller following are the values which provided

  ingress-nginx:
  controller:
    priorityClassName: "high-priority"
    replicaCount: 3
    image:
      registry: gcr.io
      image: fivetran-webhooks/nginx-ingress-controller
      tag: "v1.1.1"
      digest: ""
      pullPolicy: Always
    extraArgs:
      shutdown-grace-period: 60
    labels:
      app.kubernetes.io/part-of: wps
      wps-cloud-provider: gcp
      wps-location: <us/eu/uk>
    podLabels:
      app.kubernetes.io/part-of: wps
      wps-cloud-provider: gcp
      wps-location: <us/eu/uk>
    podAnnotations:
      fivetran.com/scrape-prometheus: "true"
      prometheus.io/port: "10254"
      cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
      fivetran.com/fivetran-app: "true"
    minReadySeconds: 30
    updateStrategy:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 0
      type: RollingUpdate
    resources:
      requests:
        cpu: 2
        memory: 2Gi
    autoscaling:
      enabled: true
      minReplicas: 3
      maxReplicas: 9
      targetCPUUtilizationPercentage: 75
      targetMemoryUtilizationPercentage: 75
    service:
      enabled: true
      loadBalancerIP: null
    admissionWebhooks:
      enabled: false
    config:
      # logging config
      log-format-upstream: '{"logtype":"request_entry","status": $status, "request_id": "$req_id", "host": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length,  "request_time": $request_time, "method": "$request_method", "time_local": "$time_local", "remote_addr": "$remote_addr", "remote_user": "$remote_user", "http_referer": "$http_referer", "http_user_agent": "$http_user_agent", "body_bytes_sent": "$body_bytes_sent", "bytes_sent": "$bytes_sent", "upstream_addr": "$upstream_addr", "upstream_response_length": "$upstream_response_length", "upstream_response_time": "$upstream_response_time", "upstream_status": "$upstream_status"}'
      log-format-escape-json: 'true'
  
      # request contents config
      proxy-body-size: 9m
      client-body-buffer-size: 9m
  
      # request timeout config
      client-body-timeout: '300'
      client-header-timeout: '300'
      proxy-read-timeout: '300'
      proxy-send-timeout: '300'
  
      # upstream pod retry
      proxy-next-upstream: 'error timeout http_500 http_502 http_503 http_504 non_idempotent'
      proxy-next-upstream-timeout: '60'
      proxy-next-upstream-tries: '0'
      ssl-redirect: "false"
  
      # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#load-balance
      load-balance: ewma
      worker-processes: '3' #As we are using 3 cpu for ingress controller pods
  
      # Recovery Server
      location-snippet: |
        proxy_intercept_errors on;
        error_page 500 501 502 503 = @fivetran_recovery;
      server-snippet: |
        location @fivetran_recovery {
        proxy_pass http://{Recovery Collector's ClusterIP service's IP address};
        }
  

• Current State of the controller:

  • kubectl describe ingressclasses

  Name:         nginx
  Labels:       app.kubernetes.io/component=controller
                app.kubernetes.io/instance=wps-us
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=ingress-nginx
                app.kubernetes.io/part-of=wps
                app.kubernetes.io/version=1.1.1
                helm.sh/chart=ingress-nginx-4.0.15
                wps-cloud-provider=gcp
                wps-location=us
  Annotations:  meta.helm.sh/release-name: wps-us
                meta.helm.sh/release-namespace: ingress-nginx
  Controller:   k8s.io/ingress-nginx
  Events:       <none>
  

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> describe ing <ingressname>

    Name:             collector-ingress
    Labels:           app.kubernetes.io/managed-by=Helm
    Namespace:        collector
    Address:          xxxxx
    Default backend:  default-http-backend:80 (10.18.48.196:8080)
    TLS:
      webhooks-ssl-certificate terminates 
      events-ssl-certificate terminates 
    Rules:
      Host                   Path  Backends
      ----                   ----  --------
      webhooks.fivetran.com  
                             /webhooks/|/internal/|/snowplow/|/health$|/api_docs|/swagger*   collector-service:80 (10.18.48.39:8001,10.18.52.54:8001,10.18.54.56:8001)
      events.fivetran.com    
                             /*   collector-service:80 (10.18.48.39:8001,10.18.52.54:8001,10.18.54.56:8001)
    Annotations:             kubernetes.io/ingress.class: nginx
                             meta.helm.sh/release-name: wps-us
                             meta.helm.sh/release-namespace: ingress-nginx
                             nginx.ingress.kubernetes.io/use-regex: true
    Events:                  <none>
    

What happened:
Our Ingress controller is serving ~1500 RPS
But over time ingress controller memory gets continuously increase but never goes down when it crosses the node limitation ~15GB pods gets evicted.

What you expected to happen:
We expect memory to get stablizise at some point.

profiling heap export:
high_mem.txt

For now we are manually restarting worker processes inorder to realise the memory

[k8s-ci-robot]

@ParagPatil96: The label(s) triage/support cannot be applied, because the repository doesn't have them.

In response to this:

NGINX Ingress controller version

NGINX Ingress controller
Release: v1.1.1
Build: a17181e
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.9

---

Kubernetes version

version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.14-gke.1900", GitCommit:"abc4e63ae76afef74b341d2dba1892471781604f", GitTreeState:"clean", BuildDate:"2021-09-07T09:21:04Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GCP

• OS : Container-Optimized OS from Google

• Kernel : 5.4.129+

• How was the ingress-nginx-controller installed:

  • We have used helm to install Nginx Ingress Controller following are the values which provided

  ingress-nginx:
  controller:
    priorityClassName: "high-priority"
    replicaCount: 3
    image:
      registry: gcr.io
      image: fivetran-webhooks/nginx-ingress-controller
      tag: "v1.1.1"
      digest: ""
      pullPolicy: Always
    extraArgs:
      shutdown-grace-period: 60
    labels:
      app.kubernetes.io/part-of: wps
      wps-cloud-provider: gcp
      wps-location: <us/eu/uk>
    podLabels:
      app.kubernetes.io/part-of: wps
      wps-cloud-provider: gcp
      wps-location: <us/eu/uk>
    podAnnotations:
      fivetran.com/scrape-prometheus: "true"
      prometheus.io/port: "10254"
      cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
      fivetran.com/fivetran-app: "true"
    minReadySeconds: 30
    updateStrategy:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 0
      type: RollingUpdate
    resources:
      requests:
        cpu: 2
        memory: 2Gi
    autoscaling:
      enabled: true
      minReplicas: 3
      maxReplicas: 9
      targetCPUUtilizationPercentage: 75
      targetMemoryUtilizationPercentage: 75
    service:
      enabled: true
      loadBalancerIP: null
    admissionWebhooks:
      enabled: false
    config:
      # logging config
      log-format-upstream: '{"logtype":"request_entry","status": $status, "request_id": "$req_id", "host": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length,  "request_time": $request_time, "method": "$request_method", "time_local": "$time_local", "remote_addr": "$remote_addr", "remote_user": "$remote_user", "http_referer": "$http_referer", "http_user_agent": "$http_user_agent", "body_bytes_sent": "$body_bytes_sent", "bytes_sent": "$bytes_sent", "upstream_addr": "$upstream_addr", "upstream_response_length": "$upstream_response_length", "upstream_response_time": "$upstream_response_time", "upstream_status": "$upstream_status"}'
      log-format-escape-json: 'true'
  
      # request contents config
      proxy-body-size: 9m
      client-body-buffer-size: 9m
  
      # request timeout config
      client-body-timeout: '300'
      client-header-timeout: '300'
      proxy-read-timeout: '300'
      proxy-send-timeout: '300'
  
      # upstream pod retry
      proxy-next-upstream: 'error timeout http_500 http_502 http_503 http_504 non_idempotent'
      proxy-next-upstream-timeout: '60'
      proxy-next-upstream-tries: '0'
      ssl-redirect: "false"
  
      # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#load-balance
      load-balance: ewma
      worker-processes: '3' #As we are using 3 cpu for ingress controller pods
  
      # Recovery Server
      location-snippet: |
        proxy_intercept_errors on;
        error_page 500 501 502 503 = @fivetran_recovery;
      server-snippet: |
        location @fivetran_recovery {
        proxy_pass http://{Recovery Collector's ClusterIP service's IP address};
        }
  

• Current State of the controller:

• kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=wps-us
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=wps
              app.kubernetes.io/version=1.1.1
              helm.sh/chart=ingress-nginx-4.0.15
              wps-cloud-provider=gcp
              wps-location=us
Annotations:  meta.helm.sh/release-name: wps-us
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• Current state of ingress object, if applicable:
• kubectl -n <appnamespace> describe ing <ingressname>

  Name:             collector-ingress
  Labels:           app.kubernetes.io/managed-by=Helm
  Namespace:        collector
  Address:          xxxxx
  Default backend:  default-http-backend:80 (10.18.48.196:8080)
  TLS:
    webhooks-ssl-certificate terminates 
    events-ssl-certificate terminates 
  Rules:
    Host                   Path  Backends
    ----                   ----  --------
    webhooks.fivetran.com  
                           /webhooks/|/internal/|/snowplow/|/health$|/api_docs|/swagger*   collector-service:80 (10.18.48.39:8001,10.18.52.54:8001,10.18.54.56:8001)
    events.fivetran.com    
                           /*   collector-service:80 (10.18.48.39:8001,10.18.52.54:8001,10.18.54.56:8001)
  Annotations:             kubernetes.io/ingress.class: nginx
                           meta.helm.sh/release-name: wps-us
                           meta.helm.sh/release-namespace: ingress-nginx
                           nginx.ingress.kubernetes.io/use-regex: true
  Events:                  <none>
  

What happened:
Our Ingress controller is serving ~1500 RPS
But over time ingress controller memory gets continuously increase but never goes down when it crosses the node limitation ~15GB pods gets evicted.

What you expected to happen:
We expect memory to get stablizise at some point.

/triage support

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug
/kind support

What other info is available ?
You can get some info if you use the documented prometheus+grafana config.
Does it happen on controller version 0.50.X
What do the logs contain relevant to this

[rmathagiarun]

NGINX Ingress controller
Release:       v1.1.1
Build:         a17181e
Repository:    https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.9
 
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"9e5f344f6cdbf2eaa7e450d5acd8fd0b7f669bf9", GitTreeState:"clean", BuildDate:"2021-05-19T04:34:27Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
 
We are also facing this memory issue. Initially, when we set the memory limits to 2GB, ingress controller was continuously restarting due to OOM. Attached dmesg log for reference.
 
We then increased the memory limit to 6GB and from the attached Grafana Metrics we can see that the Pod constantly consumes close to 4GB of memory.
 
We were earlier using the following version, where we noticed the memory consumption stabilised at less than 2GB.
Release:       v0.48.1
Build:         git-1de9a24b2
Repository:    git@github.com:kubernetes/ingress-nginx.git
 nginx version: nginx/1.20.1
 
 
It looks like this version of ingress controller consumes too much memory when compared to earlier versions.
ingress-controller_pod1_dmesg.log

ingress-controller_pod1.log
ingress-controller_pod2_dmesg.log

ingress-controller_pod2.log

[rmathagiarun]

@longwuyuan I posted the details here as I found the issue similar to ours. Let me know if you would want me to open a new one with all the details.

[longwuyuan]

Can you upgrade to latest release and kindly answer the questions I asked earlier. Thank you.

[ramanNarasimhan77]

@longwuyuan I am a co-worker of @rmathagiarun . The details shared by @rmathagiarun were from the latest released controller version v1.1.1, and we see this issue happening frequently.

[longwuyuan]

Some questions I asked earlier are not answered. Basically, some digging is required and specifics on which process was using memory high should be determined. Along with checking node resources at that point of time.

[rmathagiarun]

@longwuyuan

I can see that you have suggested us to test using controller version 0.5.x - We have been using v0.48.1 for a long time and have never faced this issue. We had to upgrade to v1.1.1 as v0.48.1(even the suggested version v0.50.x) is not compatible with K8s Version 1.22 and 1.23.

The core components of our product has remained the same on both the versions(0.48.1 and v1.1.1) and we are facing this memory issue only with v1.1.1.

Unfortunately, the logs doesn't have much info on this memory leak. We were able to find the OOM issue only by using dmesg command inside the pod. I have already shared the ingress logs and Grafana screenshots for the same.

Nodes all along had sufficient resources and as you can see from the Grafana Screenshot that the pod is constantly consuming close to 4GB and spike is not specific only during certain operations.

[rmathagiarun]

@longwuyuan Any update on this. I have answered all the queries that you have posted, Let me know if you need any additional info.

What other info is available ?
Every time when we add/remove ingress, ingress controller gets reloaded (due to configuration changes). During this reload, memory utilisation shoots up.
I0218 13:29:45.249555 8 controller.go:155] "Configuration changes detected, backend reload required"
I0218 13:30:03.025708 8 controller.go:172] "Backend successfully reloaded"
Even after Pod restarts it continues to keep crashing as the Pod continues to hold the memory. Only a Deployment rollout or Deleting the Pod would release the memory.

You can get some info if you use the documented prometheus+grafana config.
Screenshots already shared.

Does it happen on controller version 0.50.X
As stated in my Previous comment, we have been using v0.48.X for a long time and we want to upgrade to V1.1.1 to make our selves compatible with K8s Version 1.22 and 1.23.

What do the logs contain relevant to this.
Have already shared the Logs, but, the logs doesn't have much info on this memory leak. We were able to find the OOM issue only by using dmesg command inside the pod.

[rmathagiarun]

@longwuyuan Any update on this. We have been hitting this issue quite frequently.