Skip to content

Commit c2f4e18

Browse files
bjee19bjee19
committed
Revert "Delete old reconfiguration files"
This reverts commit 0ee56f5.
1 parent 4658f34 commit c2f4e18

File tree

13 files changed

+758
-0
lines changed

13 files changed

+758
-0
lines changed

‎tests/reconfig/results/1.0.0/1.0.0.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Reconfiguration testing Results
2+
3+
<!-- TOC -->
4+
- [Reconfiguration testing Results](#reconfiguration-testing-results)
5+
- [Test environment](#test-environment)
6+
- [Results Tables](#results-tables)
7+
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
8+
- [Event Batch Processing](#event-batch-processing)
9+
- [NumResources -> Total Resources](#numresources---total-resources)
10+
- [Observations](#observations)
11+
<!-- TOC -->
12+
13+
## Test environment
14+
15+
GKE cluster:
16+
17+
- Node count: 3
18+
- Instance Type: e2-medium
19+
- k8s version: 1.27.3-gke.100
20+
- Zone: us-central1-c
21+
- Total vCPUs: 6
22+
- Total RAM: 12GB
23+
- Max pods per node: 110
24+
25+
NGF deployment:
26+
27+
- NGF version: edge - git commit 29b45e38bacd7c4f22834938105e3cda4f29f6d1
28+
- NGINX Version: 1.25.2
29+
30+
## Results Tables
31+
32+
### NGINX Reloads and Time to Ready
33+
34+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
35+
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
36+
| 1 | 30 | 1 | 1 | 2 | 191 | 100% | 100% |
37+
| 1 | 150 | 2 | 2 | 2 | 440 | 50% | 100% |
38+
| 2 | 30 | 50 | <1 | 93 | 162 | 100% | 100% |
39+
| 2 | 150 | 208 | <1 | 396 | 281 | 96.46% | 100% |
40+
| 3 | 30 | 1 | 1 | 93 | 129 | 100% | 100% |
41+
| 3 | 150 | 1 | 1 | 453 | 130 | 100% | 100% |
42+
43+
44+
### Event Batch Processing
45+
46+
| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms |
47+
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|
48+
| 1 | 30 | 69 | 6.232 | 100% | 100% |
49+
| 1 | 150 | 309 | 3.638 | 99.68% | 100% |
50+
| 2 | 30 | 465 | 38.759 | 100% | 100% |
51+
| 2 | 150 | 1941 | 68.539 | 98.51% | 100% |
52+
| 3 | 30 | 374 | 36.834 | 99.73% | 99.73% |
53+
| 3 | 150 | 1812 | 40.411 | 99.94% | 99.94% |
54+
55+
56+
## NumResources -> Total Resources
57+
58+
| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Total Resources |
59+
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|-----------------|
60+
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | <total> |
61+
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 244 |
62+
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 1204 |
63+
64+
## Observations
65+
66+
1. We are reloading after reconciling a ReferenceGrant even when there is no Gateway. This is because we treat every
67+
upsert/delete of a ReferenceGrant as a change. This means we will regenerate NGINX config every time a ReferenceGrant
68+
is created, updated (generation must change), or deleted, even if it does not apply to the accepted Gateway.
69+
70+
Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1124
71+
72+
2. We are reloading after reconciling a HTTPRoute even when there is no accepted Gateway and no config being generated.
73+
74+
Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1123
75+
76+
3. Majority of NGINX reloads were in the <= 500ms bucket, with all of them being in the <= 1000ms bucket. An increase
77+
in the reload time based on number of configured resources resulting in NGINX configuration changes was observed.
78+
79+
4. No errors (NGF or NGINX) were observed in any test run.

‎tests/reconfig/results/1.1.0/1.1.0.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Reconfiguration testing Results
2+
3+
<!-- TOC -->
4+
- [Reconfiguration testing Results](#reconfiguration-testing-results)
5+
- [Summary](#summary)
6+
- [Test environment](#test-environment)
7+
- [Results Tables](#results-tables)
8+
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
9+
- [Event Batch Processing](#event-batch-processing)
10+
- [NumResources to Total Resources](#numresources-to-total-resources)
11+
- [Observations](#observations)
12+
- [Future Improvements](#future-improvements)
13+
<!-- TOC -->
14+
15+
## Summary
16+
17+
- Better reload times across all tests
18+
- Similar TimeToReadyTotal and TimeToReadyAveSingle times
19+
- Similar event batch totals
20+
- Slightly better event batch processing average times
21+
- No new errors or issues
22+
23+
## Test environment
24+
25+
GKE cluster:
26+
27+
- Node count: 4
28+
- Instance Type: n2d-standard-2
29+
- k8s version: 1.27.3-gke.100
30+
- Zone: us-west2-a
31+
- Total vCPUs: 8
32+
- Total RAM: 32GB
33+
- Max pods per node: 110
34+
35+
NGF deployment:
36+
37+
- NGF version: edge - git commit 3cab370a46bccd55c115c16e23a475df2497a3d2
38+
- NGINX Version: 1.25.3
39+
40+
## Results Tables
41+
42+
### NGINX Reloads and Time to Ready
43+
44+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
45+
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
46+
| 1 | 30 | 1.5 | <1 | 2 | 158.5 | 100% | 100% |
47+
| 1 | 150 | 3.5 | 1 | 2 | 272.5 | 100% | 100% |
48+
| 2 | 30 | 34 | <1 | 93 | 136 | 100% | 100% |
49+
| 2 | 150 | 176.5 | <1 | 451 | 203.98 | 100% | 100% |
50+
| 3 | 30 | <1 | 1 | 93 | 125.7 | 100% | 100% |
51+
| 3 | 150 | 1 | 1 | 453 | 126.71 | 100% | 100% |
52+
53+
54+
### Event Batch Processing
55+
56+
| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
57+
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
58+
| 1 | 30 | 70 | 5.12 | 100% | 100% | 100% | 100% | 100% |
59+
| 1 | 150 | 309 | 2.14 | 100% | 100% | 100% | 100% | 100% |
60+
| 2 | 30 | 442 | 35.4 | 100% | 100% | 100% | 100% | 100% |
61+
| 2 | 150 | 2009 | 54.76 | 100% | 100% | 100% | 100% | 100% |
62+
| 3 | 30 | 373 | 35.72 | 99.73% | 99.73% | 100% | 100% | 100% |
63+
| 3 | 150 | 1813 | 39.46 | 99.94% | 99.94% | 99.94% | 99.94% | 100% |
64+
65+
> Note: The outlier for test #3 is the event batch that contains the Gateway. It took ~13s to process.
66+
67+
## NumResources to Total Resources
68+
69+
| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Attached HTTPRoutes | Total Resources |
70+
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|---------------------|-----------------|
71+
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | 2x | <total> |
72+
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 60 | 244 |
73+
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 300 | 1204 |
74+
75+
> Note: Only 2x HTTPRoutes attach to the Gateway because the parentRef name in the `cafe-tls-redirect` HTTPRoute is incorrect. This will be fixed in the next release.
76+
77+
## Observations
78+
79+
1. The following issues still exist:
80+
81+
- https://github.com/nginxinc/nginx-gateway-fabric/issues/1124
82+
- https://github.com/nginxinc/nginx-gateway-fabric/issues/1123
83+
84+
2. All NGINX reloads were in the <= 500ms bucket. An increase in the reload time based on number of configured resources resulting in NGINX configuration changes was observed.
85+
86+
3. No errors (NGF or NGINX) were observed in any test run.
87+
88+
4. The majority of the event batches were processed in 500ms or less except the 3rd test. In the 3rd test, we create the Gateway resource after all the apps and routes. The batch that contains the Gateway is the only one that takes longer than 500ms. It takes ~13s.
89+
90+
## Future Improvements
91+
92+
1. Fix the parentRef name in the `cafe-tls-redirect` [HTTPRoute](/tests/reconfig/scripts/cafe-routes.yaml), so it matches the deployed Gateway.

‎tests/reconfig/results/1.2.0/1.2.0.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Reconfiguration testing Results
2+
3+
<!-- TOC -->
4+
- [Reconfiguration testing Results](#reconfiguration-testing-results)
5+
- [Summary](#summary)
6+
- [Test environment](#test-environment)
7+
- [Results Tables](#results-tables)
8+
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
9+
- [Event Batch Processing](#event-batch-processing)
10+
- [NumResources to Total Resources](#numresources-to-total-resources)
11+
- [Observations](#observations)
12+
- [Future Improvements](#future-improvements)
13+
<!-- TOC -->
14+
15+
## Summary
16+
17+
- Time to ready stayed consistent, if not slightly faster.
18+
- Reload time has slightly increased in some instances.
19+
- Number of batch events has reduced, subsequently increasing the average time of each batch.
20+
21+
## Test environment
22+
23+
GKE cluster:
24+
25+
- Node count: 3
26+
- Instance Type: e2-medium
27+
- k8s version: 1.27.8-gke.1067004
28+
- Zone: us-west2-a
29+
- Total vCPUs: 6
30+
- Total RAM: 12GB
31+
- Max pods per node: 110
32+
33+
NGF deployment:
34+
35+
- NGF version: edge - git commit 96a44240d317875406a8aef8fd1e424f2fb906eb
36+
- NGINX OSS Version: 1.25.4
37+
- NGINX Plus Version: R31
38+
39+
## Results Tables
40+
41+
> Note: After fixing the `cafe-tls-redirect` to point to the proper Gateway, tests that created 450 HTTPRoutes failed due to https://github.com/nginxinc/nginx-gateway-fabric/issues/1107. Therefore, those tests were re-run after reverting the `cafe-tls-redirect` issue to maintain consistency with the previous release tests. Going forward, results should look different once the above bug is fixed. Added N+ tests, but without testing 150 since it has the bug mentioned above.
42+
43+
### NGINX Reloads and Time to Ready
44+
45+
#### OSS
46+
47+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
48+
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
49+
| 1 | 30 | 2 | <1 | 2 | 189.5 | 100% | 100% |
50+
| 1 | 150 | 2 | <1 | 2 | 389 | 100% | 100% |
51+
| 2 | 30 | 30 | <1 | 94 | 161 | 100% | 100% |
52+
| 2 | 150 | 154 | <1 | 387 | 267.48 | 100% | 100% |
53+
| 3 | 30 | <1 | <1 | 94 | 127.91 | 100% | 100% |
54+
| 3 | 150 | <1 | <1 | 454 | 128 | 100% | 100% |
55+
56+
#### Plus
57+
58+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
59+
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
60+
| 1 | 30 | 1 | <1 | 2 | 151.5 | 100% | 100% |
61+
| 2 | 30 | 30 | <1 | 94 | 157 | 100% | 100% |
62+
| 3 | 30 | <1 | <1 | 94 | 128 | 100% | 100% |
63+
64+
### Event Batch Processing
65+
66+
#### OSS
67+
68+
| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
69+
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
70+
| 1 | 30 | 5 | 733.6 | 80% | 80% | 100% | 100% | 100% |
71+
| 1 | 150 | 5 | 2967 | 40% | 40% | 40% | 40% | 40% |
72+
| 2 | 30 | 371 | 57.32 | 100% | 100% | 100% | 100% | 100% |
73+
| 2 | 150 | 1743 | 75.87 | 98.45% | 100% | 100% | 100% | 100% |
74+
| 3 | 30 | 370 | 37.48 | 99.73% | 99.73% | 100% | 100% | 100% |
75+
| 3 | 150 | 1808 | 40.18 | 99.94% | 99.94% | 99.94% | 99.94% | 100% |
76+
77+
#### Plus
78+
79+
| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
80+
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
81+
| 1 | 30 | 3 | 1170 | 66% | 66% | 100% | 100% | 100% |
82+
| 2 | 30 | 370 | 58.79 | 100% | 100% | 100% | 100% | 100% |
83+
| 3 | 30 | 370 | 41.32 | 99.73% | 99.73% | 100% | 100% | 100% |
84+
85+
## NumResources to Total Resources
86+
87+
| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Attached HTTPRoutes | Total Resources |
88+
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|---------------------|-----------------|
89+
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | 2x | <total> |
90+
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 60 | 244 |
91+
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 300 | 1204 |
92+
93+
> Note: Only 2x HTTPRoutes attach to the Gateway because the parentRef name in the `cafe-tls-redirect` HTTPRoute is incorrect. This has been fixed, but until https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 is fixed we can't actually run the test successfully.
94+
95+
## Observations
96+
97+
1. Reload time seems to have a increased slightly in a few instances, though time to ready is consistent if not faster.
98+
99+
2. Processing fewer batches overall due to improvements in resource event tracking. Overall processing time didn't change much, so the average increased due to fewer batches.
100+
101+
3. No errors in the logs.
102+
103+
104+
## Future Improvements
105+
106+
Fix https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 to allow for 150 resource tests to properly run.

0 commit comments

Comments
 (0)