Skip to content

Commit 035ca66

Browse files
author
Kate Osborn
committed
Add results for reconfig test
1 parent 56016e9 commit 035ca66

File tree

4 files changed

+95
-8
lines changed

4 files changed

+95
-8
lines changed

‎tests/reconfig/results/1.0.0/1.0.0.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ NGF deployment:
5656
## NumResources -> Total Resources
5757

5858
| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Total Resources |
59-
| ------------ | -------- | ------- | --------------- | ---------- | ---------------- | -------------------- | ---------- | --------------- |
59+
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|-----------------|
6060
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | <total> |
6161
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 244 |
6262
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 1204 |

‎tests/reconfig/results/1.1.0/1.1.0.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Reconfiguration testing Results
2+
3+
<!-- TOC -->
4+
- [Reconfiguration testing Results](#reconfiguration-testing-results)
5+
- [Summary](#summary)
6+
- [Test environment](#test-environment)
7+
- [Results Tables](#results-tables)
8+
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
9+
- [Event Batch Processing](#event-batch-processing)
10+
- [NumResources to Total Resources](#numresources-to-total-resources)
11+
- [Observations](#observations)
12+
<!-- TOC -->
13+
14+
## Summary
15+
16+
- Better reload times across all tests
17+
- Similar TimeToReadyTotal and TimeToReadyAveSingle times
18+
- Similar event batch totals
19+
- Slightly better event batch processing average times
20+
- No new errors or issues
21+
22+
## Test environment
23+
24+
GKE cluster:
25+
26+
- Node count: 4
27+
- Instance Type: n2d-standard-2
28+
- k8s version: 1.27.3-gke.100
29+
- Zone: us-west2-a
30+
- Total vCPUs: 8
31+
- Total RAM: 32GB
32+
- Max pods per node: 110
33+
34+
NGF deployment:
35+
36+
- NGF version: edge - git commit 3cab370a46bccd55c115c16e23a475df2497a3d2
37+
- NGINX Version: 1.25.3
38+
39+
## Results Tables
40+
41+
### NGINX Reloads and Time to Ready
42+
43+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
44+
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
45+
| 1 | 30 | 1.5 | <1 | 2 | 158.5 | 100% | 100% |
46+
| 1 | 150 | 3.5 | 1 | 2 | 272.5 | 100% | 100% |
47+
| 2 | 30 | 34 | <1 | 93 | 136 | 100% | 100% |
48+
| 2 | 150 | 176.5 | <1 | 451 | 203.98 | 100% | 100% |
49+
| 3 | 30 | <1 | 1 | 93 | 125.7 | 100% | 100% |
50+
| 3 | 150 | 1 | 1 | 453 | 126.71 | 100% | 100% |
51+
52+
53+
### Event Batch Processing
54+
55+
| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
56+
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
57+
| 1 | 30 | 70 | 5.12 | 100% | 100% | 100% | 100% | 100% |
58+
| 1 | 150 | 309 | 2.14 | 100% | 100% | 100% | 100% | 100% |
59+
| 2 | 30 | 442 | 35.4 | 100% | 100% | 100% | 100% | 100% |
60+
| 2 | 150 | 2009 | 54.76 | 100% | 100% | 100% | 100% | 100% |
61+
| 3 | 30 | 373 | 35.72 | 99.73% | 99.73% | 100% | 100% | 100% |
62+
| 3 | 150 | 1813 | 39.46 | 99.94% | 99.94% | 99.94% | 99.94% | 100% |
63+
64+
> Note: The outlier for test #3 is the event batch that contains the Gateway. It took ~13s to process.
65+
66+
## NumResources to Total Resources
67+
68+
| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Total Resources |
69+
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|-----------------|
70+
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | <total> |
71+
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 244 |
72+
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 1204 |
73+
74+
## Observations
75+
76+
1. The following issues still exist:
77+
78+
- https://github.com/nginxinc/nginx-gateway-fabric/issues/1124
79+
- https://github.com/nginxinc/nginx-gateway-fabric/issues/1123
80+
81+
2. All NGINX reloads were in the <= 500ms bucket. An increase in the reload time based on number of configured resources resulting in NGINX configuration changes was observed.
82+
83+
3. No errors (NGF or NGINX) were observed in any test run.
84+
85+
4. The majority of the event batches were processed in 500ms or less except the 3rd test. In the 3rd test, we create the Gateway resource after all the apps and routes. The batch that contains the Gateway is the only one that takes longer than 500ms. It takes ~13s.

‎tests/reconfig/scripts/delete-multiple.sh

100644100755
Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,13 @@
33
num_namespaces=$1
44

55
# Delete namespaces
6+
namespaces=""
67
for ((i=1; i<=$num_namespaces; i++)); do
7-
namespace_name="namespace$i"
8-
kubectl delete namespace "$namespace_name"
8+
namespaces+="namespace$i "
99
done
1010

11+
kubectl delete namespace $namespaces
12+
1113
# Delete single instance resources
1214
kubectl delete -f gateway.yaml
1315
kubectl delete -f reference-grant.yaml

‎tests/reconfig/setup.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626

2727
The following cluster will be sufficient:
2828

29-
- A Kubernetes cluster with 3 nodes on GKE
29+
- A Kubernetes cluster with 4 nodes on GKE
3030
- Node: e2-medium (2 vCPU, 4GB memory)
3131

3232
## Setup
@@ -43,7 +43,7 @@
4343

4444
```console
4545
helm install my-release oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --version 0.0.0-edge \
46-
--create-namespace --wait -n nginx-gateway
46+
--create-namespace --wait -n nginx-gateway --set nginxGateway.config.logging.level=debug
4747
```
4848

4949
4. Run tests:
@@ -75,11 +75,11 @@
7575
1. The average reload duration can be computed by taking the `nginx_gateway_fabric_nginx_reloads_milliseconds_sum`
7676
metric value and dividing it by the `nginx_gateway_fabric_nginx_reloads_milliseconds_count` metric value.
7777
7. Measure Event Batch Processing Results
78-
1. Event Batch Total - metrics.
78+
1. Event Batch Total - `nginx_gateway_fabric_event_batch_processing_milliseconds_count` metric.
7979
2. Average Event Batch Processing duration - metrics.
80-
1. The average event batch processing duraiton can be computed by taking the `nginx_gateway_fabric_event_batch_processing_milliseconds_sum`
80+
1. The average event batch processing duration can be computed by taking the `nginx_gateway_fabric_event_batch_processing_milliseconds_sum`
8181
metric value and dividing it by the `nginx_gateway_fabric_event_batch_processing_milliseconds_count` metric value.
82-
8. For accuracy, repeat the test suite once or twice, take the averages, and look for any anomolies or outliers.
82+
8. For accuracy, repeat the test suite once or twice, take the averages, and look for any anomalies or outliers.
8383

8484
## Tests
8585

0 commit comments

Comments
 (0)