diff --git a/tests/results/dp-perf/1.3.0/1.3.0-plus.md b/tests/results/dp-perf/1.3.0/1.3.0-plus.md new file mode 100755 index 0000000000..f9d749f94a --- /dev/null +++ b/tests/results/dp-perf/1.3.0/1.3.0-plus.md @@ -0,0 +1,84 @@ +# Results + +## Test environment + +NGINX Plus: true + +GKE Cluster: + +- Node count: 12 +- k8s version: v1.28.9-gke.1000000 +- vCPUs per node: 16 +- RAM per node: 65855088Ki +- Max pods per node: 110 +- Zone: us-east1-b +- Instance Type: n2d-standard-16 + +## Summary + +Similar results to last time. + +## Test1: Running latte path based routing + +```text +Requests [total, rate, throughput] 30000, 1000.04, 1000.01 +Duration [total, attack, wait] 30s, 29.999s, 998.021µs +Latencies [min, mean, 50, 90, 95, 99, max] 658.142µs, 996.64µs, 951.429µs, 1.093ms, 1.145ms, 1.386ms, 26.648ms +Bytes In [total, mean] 4770000, 159.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:30000 +Error Set: +``` + +## Test2: Running coffee header based routing + +```text +Requests [total, rate, throughput] 30000, 1000.04, 1000.01 +Duration [total, attack, wait] 30s, 29.999s, 870.391µs +Latencies [min, mean, 50, 90, 95, 99, max] 700.225µs, 1.01ms, 981.048µs, 1.134ms, 1.186ms, 1.329ms, 21.685ms +Bytes In [total, mean] 4800000, 160.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:30000 +Error Set: +``` + +## Test3: Running coffee query based routing + +```text +Requests [total, rate, throughput] 30000, 1000.06, 1000.03 +Duration [total, attack, wait] 29.999s, 29.998s, 964.352µs +Latencies [min, mean, 50, 90, 95, 99, max] 712.998µs, 984.528µs, 966.316µs, 1.114ms, 1.172ms, 1.319ms, 11.864ms +Bytes In [total, mean] 5040000, 168.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:30000 +Error Set: +``` + +## Test4: Running tea GET method based routing + +```text +Requests [total, rate, throughput] 30000, 1000.04, 1000.01 +Duration [total, attack, wait] 30s, 29.999s, 853.087µs +Latencies [min, mean, 50, 90, 95, 99, max] 702.09µs, 1.007ms, 976.392µs, 1.13ms, 1.188ms, 1.361ms, 27.968ms +Bytes In [total, mean] 4710000, 157.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:30000 +Error Set: +``` + +## Test5: Running tea POST method based routing + +```text +Requests [total, rate, throughput] 30000, 1000.04, 1000.00 +Duration [total, attack, wait] 30s, 29.999s, 1.149ms +Latencies [min, mean, 50, 90, 95, 99, max] 715.853µs, 989.369µs, 974.707µs, 1.127ms, 1.184ms, 1.355ms, 3.77ms +Bytes In [total, mean] 4710000, 157.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:30000 +Error Set: +``` diff --git a/tests/results/ngf-upgrade/1.3.0/1.3.0-plus.md b/tests/results/ngf-upgrade/1.3.0/1.3.0-plus.md new file mode 100755 index 0000000000..ce71aebf45 --- /dev/null +++ b/tests/results/ngf-upgrade/1.3.0/1.3.0-plus.md @@ -0,0 +1,49 @@ +# Results + +## Test environment + +NGINX Plus: true + +GKE Cluster: + +- Node count: 12 +- k8s version: v1.28.9-gke.1000000 +- vCPUs per node: 16 +- RAM per node: 65855088Ki +- Max pods per node: 110 +- Zone: us-east1-b +- Instance Type: n2d-standard-16 + +## Summary + +Similar results to last time. + +## Test: Send https /tea traffic + +```text +Requests [total, rate, throughput] 6000, 100.02, 100.01 +Duration [total, attack, wait] 59.991s, 59.99s, 1.392ms +Latencies [min, mean, 50, 90, 95, 99, max] 666.477µs, 1.359ms, 1.276ms, 1.74ms, 1.894ms, 2.331ms, 50.572ms +Bytes In [total, mean] 924000, 154.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:6000 +Error Set: +``` + +![https-plus.png](https-plus.png) + +## Test: Send http /coffee traffic + +```text +Requests [total, rate, throughput] 6000, 100.02, 100.01 +Duration [total, attack, wait] 59.991s, 59.99s, 1.589ms +Latencies [min, mean, 50, 90, 95, 99, max] 886.813µs, 1.521ms, 1.464ms, 1.864ms, 2.029ms, 2.426ms, 13.752ms +Bytes In [total, mean] 966000, 161.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:6000 +Error Set: +``` + +![http-plus.png](http-plus.png) diff --git a/tests/results/ngf-upgrade/1.3.0/http-plus.png b/tests/results/ngf-upgrade/1.3.0/http-plus.png new file mode 100644 index 0000000000..95432e96fc Binary files /dev/null and b/tests/results/ngf-upgrade/1.3.0/http-plus.png differ diff --git a/tests/results/ngf-upgrade/1.3.0/https-plus.png b/tests/results/ngf-upgrade/1.3.0/https-plus.png new file mode 100644 index 0000000000..95432e96fc Binary files /dev/null and b/tests/results/ngf-upgrade/1.3.0/https-plus.png differ diff --git a/tests/results/scale/1.3.0/1.3.0-plus.md b/tests/results/scale/1.3.0/1.3.0-plus.md new file mode 100755 index 0000000000..03c8a5d403 --- /dev/null +++ b/tests/results/scale/1.3.0/1.3.0-plus.md @@ -0,0 +1,206 @@ +# Results + +## Test environment + +NGINX Plus: true + +GKE Cluster: + +- Node count: 12 +- k8s version: v1.28.9-gke.1000000 +- vCPUs per node: 16 +- RAM per node: 65855088Ki +- Max pods per node: 110 +- Zone: us-east1-b +- Instance Type: n2d-standard-16 + +## Summary + +- New look since we are now auto-generating these results +- Batch processing time has reduced +- Reload time is similar (slightly worse in some cases) +- Metrics (CPU, memory, TTR) all seem to be better, though this could be because of the automation instead of being a manual process. +- Error logs: + ```couldn't update upstream via the API, reloading configuration instead","batchID":220,"upstreamName":"scale_backend_80","error":"failed to update servers of scale_backend_80 upstream: failed to add 10.104.3.63:8080 server to scale_backend_80 upstream: expected 201 response, got 500. error.status=500; error.text=upstream memory exhausted; error.code=UpstreamOutOfMemory;``` + ```ngx_slab_alloc() failed: no memory in upstream zone "scale_backend_80"``` + + Seems that our zone size may not be big enough in N+. + +## Test TestScale_Listeners + +### Reloads + +- Total: 128 +- Total Errors: 0 +- Average Time: 142ms +- Reload distribution: + - 500ms: 128 + - 1000ms: 128 + - 5000ms: 128 + - 10000ms: 128 + - 30000ms: 128 + - +Infms: 128 + +### Event Batch Processing + +- Total: 387 +- Average Time: 122ms +- Event Batch Processing distribution: + - 500ms: 355 + - 1000ms: 386 + - 5000ms: 387 + - 10000ms: 387 + - 30000ms: 387 + - +Infms: 387 + +### Errors + +- NGF errors: 0 +- NGF container restarts: 0 +- NGINX errors: 0 +- NGINX container restarts: 0 + +### Graphs and Logs + +See [output directory](./TestScale_Listeners) for more details. +The logs are attached only if there are errors. + +## Test TestScale_HTTPSListeners + +### Reloads + +- Total: 127 +- Total Errors: 0 +- Average Time: 174ms +- Reload distribution: + - 500ms: 127 + - 1000ms: 127 + - 5000ms: 127 + - 10000ms: 127 + - 30000ms: 127 + - +Infms: 127 + +### Event Batch Processing + +- Total: 450 +- Average Time: 120ms +- Event Batch Processing distribution: + - 500ms: 404 + - 1000ms: 450 + - 5000ms: 450 + - 10000ms: 450 + - 30000ms: 450 + - +Infms: 450 + +### Errors + +- NGF errors: 0 +- NGF container restarts: 0 +- NGINX errors: 0 +- NGINX container restarts: 0 + +### Graphs and Logs + +See [output directory](./TestScale_HTTPSListeners) for more details. +The logs are attached only if there are errors. + +## Test TestScale_HTTPRoutes + +### Reloads + +- Total: 1001 +- Total Errors: 0 +- Average Time: 375ms +- Reload distribution: + - 500ms: 740 + - 1000ms: 1001 + - 5000ms: 1001 + - 10000ms: 1001 + - 30000ms: 1001 + - +Infms: 1001 + +### Event Batch Processing + +- Total: 1008 +- Average Time: 421ms +- Event Batch Processing distribution: + - 500ms: 647 + - 1000ms: 1008 + - 5000ms: 1008 + - 10000ms: 1008 + - 30000ms: 1008 + - +Infms: 1008 + +### Errors + +- NGF errors: 0 +- NGF container restarts: 0 +- NGINX errors: 0 +- NGINX container restarts: 0 + +### Graphs and Logs + +See [output directory](./TestScale_HTTPRoutes) for more details. +The logs are attached only if there are errors. + +## Test TestScale_UpstreamServers + +### Reloads + +- Total: 3 +- Total Errors: 0 +- Average Time: 125ms +- Reload distribution: + - 500ms: 3 + - 1000ms: 3 + - 5000ms: 3 + - 10000ms: 3 + - 30000ms: 3 + - +Infms: 3 + +### Event Batch Processing + +- Total: 140 +- Average Time: 89ms +- Event Batch Processing distribution: + - 500ms: 140 + - 1000ms: 140 + - 5000ms: 140 + - 10000ms: 140 + - 30000ms: 140 + - +Infms: 140 + +### Errors + +- NGF errors: 1 +- NGF container restarts: 0 +- NGINX errors: 2 +- NGINX container restarts: 0 + +### Graphs and Logs + +See [output directory](./TestScale_UpstreamServers) for more details. +The logs are attached only if there are errors. + +## Test TestScale_HTTPMatches + +```text +Requests [total, rate, throughput] 30000, 1000.04, 1000.01 +Duration [total, attack, wait] 30s, 29.999s, 919.884µs +Latencies [min, mean, 50, 90, 95, 99, max] 674.011µs, 963.958µs, 935.735µs, 1.089ms, 1.154ms, 1.339ms, 30.258ms +Bytes In [total, mean] 4830000, 161.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:30000 +Error Set: +``` +```text +Requests [total, rate, throughput] 30000, 1000.03, 1000.00 +Duration [total, attack, wait] 30s, 29.999s, 986.656µs +Latencies [min, mean, 50, 90, 95, 99, max] 767.398µs, 1.038ms, 1.006ms, 1.145ms, 1.195ms, 1.347ms, 25.165ms +Bytes In [total, mean] 4830000, 161.00 +Bytes Out [total, mean] 0, 0.00 +Success [ratio] 100.00% +Status Codes [code:count] 200:30000 +Error Set: +``` diff --git a/tests/results/scale/1.3.0/TestScale_HTTPRoutes/cpu-plus.png b/tests/results/scale/1.3.0/TestScale_HTTPRoutes/cpu-plus.png new file mode 100644 index 0000000000..7a6b102372 Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_HTTPRoutes/cpu-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_HTTPRoutes/memory-plus.png b/tests/results/scale/1.3.0/TestScale_HTTPRoutes/memory-plus.png new file mode 100644 index 0000000000..cd424f3594 Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_HTTPRoutes/memory-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_HTTPRoutes/ttr-plus.png b/tests/results/scale/1.3.0/TestScale_HTTPRoutes/ttr-plus.png new file mode 100644 index 0000000000..336e43c45a Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_HTTPRoutes/ttr-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_HTTPSListeners/cpu-plus.png b/tests/results/scale/1.3.0/TestScale_HTTPSListeners/cpu-plus.png new file mode 100644 index 0000000000..42be794ba9 Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_HTTPSListeners/cpu-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_HTTPSListeners/memory-plus.png b/tests/results/scale/1.3.0/TestScale_HTTPSListeners/memory-plus.png new file mode 100644 index 0000000000..24fce2a4d8 Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_HTTPSListeners/memory-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_HTTPSListeners/ttr-plus.png b/tests/results/scale/1.3.0/TestScale_HTTPSListeners/ttr-plus.png new file mode 100644 index 0000000000..7077b462ae Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_HTTPSListeners/ttr-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_Listeners/cpu-plus.png b/tests/results/scale/1.3.0/TestScale_Listeners/cpu-plus.png new file mode 100644 index 0000000000..4144efe1cf Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_Listeners/cpu-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_Listeners/memory-plus.png b/tests/results/scale/1.3.0/TestScale_Listeners/memory-plus.png new file mode 100644 index 0000000000..23db3c1f0c Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_Listeners/memory-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_Listeners/ttr-plus.png b/tests/results/scale/1.3.0/TestScale_Listeners/ttr-plus.png new file mode 100644 index 0000000000..33b161a849 Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_Listeners/ttr-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_UpstreamServers/cpu-plus.png b/tests/results/scale/1.3.0/TestScale_UpstreamServers/cpu-plus.png new file mode 100644 index 0000000000..192bb88dfe Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_UpstreamServers/cpu-plus.png differ diff --git a/tests/results/scale/1.3.0/TestScale_UpstreamServers/memory-plus.png b/tests/results/scale/1.3.0/TestScale_UpstreamServers/memory-plus.png new file mode 100644 index 0000000000..6a626777bd Binary files /dev/null and b/tests/results/scale/1.3.0/TestScale_UpstreamServers/memory-plus.png differ