NFR Test Results for NGF version 1.3.0 (Plus) (#2110)

Author: nginx-bot

tests/results/dp-perf/1.3.0/1.3.0-plus.md

@@ -0,0 +1,84 @@
+# Results
+
+## Test environment
+
+NGINX Plus: true
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.28.9-gke.1000000
+- vCPUs per node: 16
+- RAM per node: 65855088Ki
+- Max pods per node: 110
+- Zone: us-east1-b
+- Instance Type: n2d-standard-16
+
+## Summary
+
+Similar results to last time.
+
+## Test1: Running latte path based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 998.021µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  658.142µs, 996.64µs, 951.429µs, 1.093ms, 1.145ms, 1.386ms, 26.648ms
+Bytes In      [total, mean]                     4770000, 159.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test2: Running coffee header based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 870.391µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  700.225µs, 1.01ms, 981.048µs, 1.134ms, 1.186ms, 1.329ms, 21.685ms
+Bytes In      [total, mean]                     4800000, 160.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test3: Running coffee query based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.06, 1000.03
+Duration      [total, attack, wait]             29.999s, 29.998s, 964.352µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  712.998µs, 984.528µs, 966.316µs, 1.114ms, 1.172ms, 1.319ms, 11.864ms
+Bytes In      [total, mean]                     5040000, 168.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test4: Running tea GET method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 853.087µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  702.09µs, 1.007ms, 976.392µs, 1.13ms, 1.188ms, 1.361ms, 27.968ms
+Bytes In      [total, mean]                     4710000, 157.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test5: Running tea POST method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.00
+Duration      [total, attack, wait]             30s, 29.999s, 1.149ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  715.853µs, 989.369µs, 974.707µs, 1.127ms, 1.184ms, 1.355ms, 3.77ms
+Bytes In      [total, mean]                     4710000, 157.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```

tests/results/ngf-upgrade/1.3.0/1.3.0-plus.md

@@ -0,0 +1,49 @@
+# Results
+
+## Test environment
+
+NGINX Plus: true
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.28.9-gke.1000000
+- vCPUs per node: 16
+- RAM per node: 65855088Ki
+- Max pods per node: 110
+- Zone: us-east1-b
+- Instance Type: n2d-standard-16
+
+## Summary
+
+Similar results to last time.
+
+## Test: Send https /tea traffic
+
+```text
+Requests      [total, rate, throughput]         6000, 100.02, 100.01
+Duration      [total, attack, wait]             59.991s, 59.99s, 1.392ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  666.477µs, 1.359ms, 1.276ms, 1.74ms, 1.894ms, 2.331ms, 50.572ms
+Bytes In      [total, mean]                     924000, 154.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:6000  
+Error Set:
+```
+
+![https-plus.png](https-plus.png)
+
+## Test: Send http /coffee traffic
+
+```text
+Requests      [total, rate, throughput]         6000, 100.02, 100.01
+Duration      [total, attack, wait]             59.991s, 59.99s, 1.589ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  886.813µs, 1.521ms, 1.464ms, 1.864ms, 2.029ms, 2.426ms, 13.752ms
+Bytes In      [total, mean]                     966000, 161.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:6000  
+Error Set:
+```
+
+![http-plus.png](http-plus.png)

tests/results/ngf-upgrade/1.3.0/http-plus.png

@@ -0,0 +1,206 @@
+# Results
+
+## Test environment
+
+NGINX Plus: true
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.28.9-gke.1000000
+- vCPUs per node: 16
+- RAM per node: 65855088Ki
+- Max pods per node: 110
+- Zone: us-east1-b
+- Instance Type: n2d-standard-16
+
+## Summary
+
+- New look since we are now auto-generating these results
+- Batch processing time has reduced
+- Reload time is similar (slightly worse in some cases)
+- Metrics (CPU, memory, TTR) all seem to be better, though this could be because of the automation instead of being a manual process.
+- Error logs:
+   ```couldn't update upstream via the API, reloading configuration instead","batchID":220,"upstreamName":"scale_backend_80","error":"failed to update servers of scale_backend_80 upstream: failed to add 10.104.3.63:8080 server to scale_backend_80 upstream: expected 201 response, got 500. error.status=500; error.text=upstream memory exhausted; error.code=UpstreamOutOfMemory;```
+   ```ngx_slab_alloc() failed: no memory in upstream zone "scale_backend_80"```
+
+  Seems that our zone size may not be big enough in N+.
+
+## Test TestScale_Listeners
+
+### Reloads
+
+- Total: 128
+- Total Errors: 0
+- Average Time: 142ms
+- Reload distribution:
+	- 500ms: 128
+	- 1000ms: 128
+	- 5000ms: 128
+	- 10000ms: 128
+	- 30000ms: 128
+	- +Infms: 128
+
+### Event Batch Processing
+
+- Total: 387
+- Average Time: 122ms
+- Event Batch Processing distribution:
+	- 500ms: 355
+	- 1000ms: 386
+	- 5000ms: 387
+	- 10000ms: 387
+	- 30000ms: 387
+	- +Infms: 387
+
+### Errors
+
+- NGF errors: 0
+- NGF container restarts: 0
+- NGINX errors: 0
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_Listeners) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_HTTPSListeners
+
+### Reloads
+
+- Total: 127
+- Total Errors: 0
+- Average Time: 174ms
+- Reload distribution:
+	- 500ms: 127
+	- 1000ms: 127
+	- 5000ms: 127
+	- 10000ms: 127
+	- 30000ms: 127
+	- +Infms: 127
+
+### Event Batch Processing
+
+- Total: 450
+- Average Time: 120ms
+- Event Batch Processing distribution:
+	- 500ms: 404
+	- 1000ms: 450
+	- 5000ms: 450
+	- 10000ms: 450
+	- 30000ms: 450
+	- +Infms: 450
+
+### Errors
+
+- NGF errors: 0
+- NGF container restarts: 0
+- NGINX errors: 0
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_HTTPSListeners) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_HTTPRoutes
+
+### Reloads
+
+- Total: 1001
+- Total Errors: 0
+- Average Time: 375ms
+- Reload distribution:
+	- 500ms: 740
+	- 1000ms: 1001
+	- 5000ms: 1001
+	- 10000ms: 1001
+	- 30000ms: 1001
+	- +Infms: 1001
+
+### Event Batch Processing
+
+- Total: 1008
+- Average Time: 421ms
+- Event Batch Processing distribution:
+	- 500ms: 647
+	- 1000ms: 1008
+	- 5000ms: 1008
+	- 10000ms: 1008
+	- 30000ms: 1008
+	- +Infms: 1008
+
+### Errors
+
+- NGF errors: 0
+- NGF container restarts: 0
+- NGINX errors: 0
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_HTTPRoutes) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_UpstreamServers
+
+### Reloads
+
+- Total: 3
+- Total Errors: 0
+- Average Time: 125ms
+- Reload distribution:
+	- 500ms: 3
+	- 1000ms: 3
+	- 5000ms: 3
+	- 10000ms: 3
+	- 30000ms: 3
+	- +Infms: 3
+
+### Event Batch Processing
+
+- Total: 140
+- Average Time: 89ms
+- Event Batch Processing distribution:
+	- 500ms: 140
+	- 1000ms: 140
+	- 5000ms: 140
+	- 10000ms: 140
+	- 30000ms: 140
+	- +Infms: 140
+
+### Errors
+
+- NGF errors: 1
+- NGF container restarts: 0
+- NGINX errors: 2
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_UpstreamServers) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_HTTPMatches
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 919.884µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  674.011µs, 963.958µs, 935.735µs, 1.089ms, 1.154ms, 1.339ms, 30.258ms
+Bytes In      [total, mean]                     4830000, 161.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+```text
+Requests      [total, rate, throughput]         30000, 1000.03, 1000.00
+Duration      [total, attack, wait]             30s, 29.999s, 986.656µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  767.398µs, 1.038ms, 1.006ms, 1.145ms, 1.195ms, 1.347ms, 25.165ms
+Bytes In      [total, mean]                     4830000, 161.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```