Skip to content

Commit e6e149d

Browse files
kate-osbornkate-osborn
authored
ReferenceGrant from Gateway to Secret (#791)
Problem: NKG does not support cross-namespace Secret references on Gateway. Solution: Add support for ReferenceGrants that permit Gateways to reference Secrets in different namespaces. NKG now processes ReferenceGrants and verifies that Gateways with references to Secrets in different Namespaces have a corresponding ReferenceGrant. If no ReferenceGrant exists, the RefNotPermitted reason is used in all the listener conditions (Accepted, Programmed, and ResolvedRefs), and the listener is marked invalid. Secrets will only be resolved if the reference is permitted. No additional validation is needed for ReferenceGrant as it does not correspond to any nginx config. We treat every upsert/delete of a ReferenceGrant as a change. This means we will regenerate nginx config every time a ReferenceGrant is created, updated (generation must change), or deleted, even if it does not apply to the accepted Gateway.
1 parent bae93c1 commit e6e149d

22 files changed

+714
-120
lines changed

‎conformance/Makefile

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@ NKG_TAG = edge
22
NKG_PREFIX = nginx-kubernetes-gateway
33
GATEWAY_CLASS = nginx
44
SUPPORTED_FEATURES = HTTPRoute,HTTPRouteQueryParamMatching,HTTPRouteMethodMatching,HTTPRoutePortRedirect,HTTPRouteSchemeRedirect
5-
EXEMPT_FEATURES = ReferenceGrant
65
KIND_KUBE_CONFIG_FOLDER = $${HOME}/.kube/kind
76
TAG = latest
87
PREFIX = conformance-test-runner
@@ -62,7 +61,7 @@ run-conformance-tests: ## Run conformance tests
6261
--image=$(PREFIX):$(TAG) --image-pull-policy=Never \
6362
--overrides='{ "spec": { "serviceAccountName": "conformance" } }' \
6463
--restart=Never -- go test -v . -tags conformance -args --gateway-class=$(GATEWAY_CLASS) --debug \
65-
--supported-features=$(SUPPORTED_FEATURES) --exempt-features=$(EXEMPT_FEATURES)
64+
--supported-features=$(SUPPORTED_FEATURES)
6665

6766
.PHONY: cleanup-conformance-tests
6867
cleanup-conformance-tests: ## Clean up conformance tests fixtures

‎conformance/tests/conformance-rbac.yaml

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -30,18 +30,13 @@ rules:
3030
- delete
3131
- get
3232
- list
33-
- apiGroups:
34-
- gateway.networking.k8s.io
35-
resources:
36-
- gatewayclasses
37-
verbs:
38-
- get
39-
- list
4033
- apiGroups:
4134
- gateway.networking.k8s.io
4235
resources:
4336
- gateways
4437
- httproutes
38+
- referencegrants
39+
- gatewayclasses
4540
verbs:
4641
- create
4742
- delete

‎deploy/manifests/rbac.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ rules:
3838
- gatewayclasses
3939
- gateways
4040
- httproutes
41+
- referencegrants
4142
verbs:
4243
- list
4344
- watch

‎docs/gateway-api-compatibility.md

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ This document describes which Gateway API resources NGINX Kubernetes Gateway sup
1212
| [TLSRoute](#tlsroute) | Not supported |
1313
| [TCPRoute](#tcproute) | Not supported |
1414
| [UDPRoute](#udproute) | Not supported |
15-
| [ReferenceGrant](#referencegrant) | Not supported |
15+
| [ReferenceGrant](#referencegrant) | Partially supported |
1616
| [Custom policies](#custom-policies) | Not supported |
1717

1818
## Terminology
@@ -148,7 +148,20 @@ Fields:
148148
149149
### ReferenceGrant
150150

151-
> Status: Not supported.
151+
> Status: Partially supported.
152+
153+
NKG only supports ReferenceGrants that permit Gateways to reference Secrets.
154+
155+
Fields:
156+
* `spec`
157+
* `to`
158+
* `group` - supported.
159+
* `kind` - partially supported. Only `Secret`.
160+
* `name`- supported.
161+
* `from`
162+
* `group` - supported.
163+
* `kind` - partially supported. Only `Gateway`.
164+
* `namespace`- supported.
152165

153166
### Custom Policies
154167

‎examples/https-termination/README.md

Lines changed: 74 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# HTTPS Termination Example
22

3-
In this example, we expand on the simple [cafe-example](../cafe-example) by adding HTTPS termination to our routes and an HTTPS redirect from port 80 to 443.
3+
In this example, we expand on the simple [cafe-example](../cafe-example) by adding HTTPS termination to our routes and
4+
an HTTPS redirect from port 80 to 443. We will also show how you can use a ReferenceGrant to permit your Gateway to
5+
reference a Secret in a different Namespace.
46

57
## Running the Example
68

@@ -40,37 +42,50 @@ In this example, we expand on the simple [cafe-example](../cafe-example) by addi
4042

4143
## 3. Configure HTTPS Termination and Routing
4244

43-
1. Create a Secret with a TLS certificate and key:
45+
1. Create the Namespace `certificate` and a Secret with a TLS certificate and key:
4446
```
45-
kubectl apply -f cafe-secret.yaml
47+
kubectl apply -f certificate-ns-and-cafe-secret.yaml
4648
```
4749

4850
The TLS certificate and key in this Secret are used to terminate the TLS connections for the cafe application.
49-
**Important**: This certificate and key are for demo purposes only.
51+
> **Important**: This certificate and key are for demo purposes only.
52+
53+
1. Create the `ReferenceGrant`:
54+
```
55+
kubectl apply -f reference-grant.yaml
56+
```
57+
58+
This ReferenceGrant allows all Gateways in the `default` namespace to reference the `cafe-secret` Secret in
59+
the `certificate` namespace.
5060

5161
1. Create the `Gateway` resource:
5262
```
5363
kubectl apply -f gateway.yaml
5464
```
5565

5666
This [Gateway](./gateway.yaml) configures:
57-
* `http` listener for HTTP traffic
58-
* `https` listener for HTTPS traffic. It terminates TLS connections using the `cafe-secret` we created in step 1.
67+
* `http` listener for HTTP traffic
68+
* `https` listener for HTTPS traffic. It terminates TLS connections using the `cafe-secret` we created in step 1.
5969

6070
1. Create the `HTTPRoute` resources:
6171
```
6272
kubectl apply -f cafe-routes.yaml
6373
```
6474

65-
To configure HTTPS termination for our cafe application, we will bind our `coffee` and `tea` HTTPRoutes to the `https` listener in [cafe-routes.yaml](./cafe-routes.yaml) using the [`parentReference`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.ParentReference) field:
75+
To configure HTTPS termination for our cafe application, we will bind our `coffee` and `tea` HTTPRoutes to
76+
the `https` listener in [cafe-routes.yaml](./cafe-routes.yaml) using
77+
the [`parentReference`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.ParentReference)
78+
field:
6679

6780
```yaml
6881
parentRefs:
6982
- name: gateway
7083
sectionName: https
7184
```
7285
73-
To configure an HTTPS redirect from port 80 to 443, we will bind the special `cafe-tls-redirect` HTTPRoute with a [`HTTPRequestRedirectFilter`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRequestRedirectFilter) to the `http` listener:
86+
To configure an HTTPS redirect from port 80 to 443, we will bind the special `cafe-tls-redirect` HTTPRoute with
87+
a [`HTTPRequestRedirectFilter`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRequestRedirectFilter)
88+
to the `http` listener:
7489

7590
```yaml
7691
parentRefs:
@@ -80,13 +95,16 @@ In this example, we expand on the simple [cafe-example](../cafe-example) by addi
8095

8196
## 4. Test the Application
8297

83-
To access the application, we will use `curl` to send requests to the `coffee` and `tea` Services. First, we will access the application over HTTP to test that the HTTPS redirect works. Then we will use HTTPS.
98+
To access the application, we will use `curl` to send requests to the `coffee` and `tea` Services. First, we will access
99+
the application over HTTP to test that the HTTPS redirect works. Then we will use HTTPS.
84100

85101
### 4.1 Test HTTPS Redirect
86102

87-
To test that NGINX sends an HTTPS redirect, we will send requests to the `coffee` and `tea` Services on HTTP port. We will use curl's `--include` option to print the response headers (we are interested in the `Location` header).
103+
To test that NGINX sends an HTTPS redirect, we will send requests to the `coffee` and `tea` Services on HTTP port. We
104+
will use curl's `--include` option to print the response headers (we are interested in the `Location` header).
88105

89106
To get a redirect for coffee:
107+
90108
```
91109
curl --resolve cafe.example.com:$GW_HTTP_PORT:$GW_IP http://cafe.example.com:$GW_HTTP_PORT/coffee --include
92110
HTTP/1.1 302 Moved Temporarily
@@ -96,6 +114,7 @@ Location: https://cafe.example.com:443/coffee
96114
```
97115

98116
To get a redirect for tea:
117+
99118
```
100119
curl --resolve cafe.example.com:$GW_HTTP_PORT:$GW_IP http://cafe.example.com:$GW_HTTP_PORT/tea --include
101120
HTTP/1.1 302 Moved Temporarily
@@ -104,9 +123,10 @@ Location: https://cafe.example.com:443/tea
104123
...
105124
```
106125

107-
### 4.2 Access Coffee and Tea
126+
### 4.2 Access Coffee and Tea
108127

109-
Now we will access the application over HTTPS. Since our certificate is self-signed, we will use curl's `--insecure` option to turn off certificate verification.
128+
Now we will access the application over HTTPS. Since our certificate is self-signed, we will use curl's `--insecure`
129+
option to turn off certificate verification.
110130

111131
To get coffee:
112132

@@ -123,3 +143,45 @@ curl --resolve cafe.example.com:$GW_HTTPS_PORT:$GW_IP https://cafe.example.com:$
123143
Server address: 10.12.0.19:80
124144
Server name: tea-7cd44fcb4d-xfw2x
125145
```
146+
147+
### 4.3 Remove the ReferenceGrant
148+
149+
To restrict access to the `cafe-secret` in the `certificate` Namespace, we can delete the ReferenceGrant we created in
150+
Step 3:
151+
152+
```
153+
kubectl delete -f reference-grant.yaml
154+
```
155+
156+
Now, if we try to access the application over HTTPS, we will get a connection refused error:
157+
```
158+
curl --resolve cafe.example.com:$GW_HTTPS_PORT:$GW_IP https://cafe.example.com:$GW_HTTPS_PORT/coffee --insecure -vvv
159+
...
160+
curl: (7) Failed to connect to cafe.example.com port 443 after 0 ms: Connection refused
161+
```
162+
163+
164+
You can also check the conditions of the Gateway `https` Listener to verify the that the reference is not permitted:
165+
166+
```
167+
Name: https
168+
Conditions:
169+
Last Transition Time: 2023-06-26T20:23:56Z
170+
Message: Certificate ref to secret certificate/cafe-secret not permitted by any ReferenceGrant
171+
Observed Generation: 2
172+
Reason: RefNotPermitted
173+
Status: False
174+
Type: Accepted
175+
Last Transition Time: 2023-06-26T20:23:56Z
176+
Message: Certificate ref to secret certificate/cafe-secret not permitted by any ReferenceGrant
177+
Observed Generation: 2
178+
Reason: RefNotPermitted
179+
Status: False
180+
Type: ResolvedRefs
181+
Last Transition Time: 2023-06-26T20:23:56Z
182+
Message: Certificate ref to secret certificate/cafe-secret not permitted by any ReferenceGrant
183+
Observed Generation: 2
184+
Reason: Invalid
185+
Status: False
186+
Type: Programmed
187+
```

‎examples/https-termination/cafe-secret.yaml renamed to ‎examples/https-termination/certificate-ns-and-cafe-secret.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,13 @@
11
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: certificate
5+
---
6+
apiVersion: v1
27
kind: Secret
38
metadata:
49
name: cafe-secret
10+
namespace: certificate
511
type: kubernetes.io/tls
612
data:
713
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNzakNDQVpvQ0NRQzdCdVdXdWRtRkNEQU5CZ2txaGtpRzl3MEJBUXNGQURBYk1Sa3dGd1lEVlFRRERCQmoKWVdabExtVjRZVzF3YkdVdVkyOXRNQjRYRFRJeU1EY3hOREl4TlRJek9Wb1hEVEl6TURjeE5ESXhOVEl6T1ZvdwpHekVaTUJjR0ExVUVBd3dRWTJGbVpTNWxlR0Z0Y0d4bExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTHFZMnRHNFc5aStFYzJhdnV4Q2prb2tnUUx1ek10U1Rnc1RNaEhuK3ZRUmxIam8KVzFLRnMvQVdlS25UUStyTWVKVWNseis4M3QwRGtyRThwUisxR2NKSE50WlNMb0NEYUlRN0Nhck5nY1daS0o4Qgo1WDNnVS9YeVJHZjI2c1REd2xzU3NkSEQ1U2U3K2Vab3NPcTdHTVF3K25HR2NVZ0VtL1Q1UEMvY05PWE0zZWxGClRPL051MStoMzROVG9BbDNQdTF2QlpMcDNQVERtQ0thaEROV0NWbUJQUWpNNFI4VERsbFhhMHQ5Z1o1MTRSRzUKWHlZWTNtdzZpUzIrR1dYVXllMjFuWVV4UEhZbDV4RHY0c0FXaGRXbElweHlZQlNCRURjczN6QlI2bFF1OWkxZAp0R1k4dGJ3blVmcUVUR3NZdWxzc05qcU95V1VEcFdJelhibHhJZVVDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBcjkrZWJ0U1dzSnhLTGtLZlRkek1ISFhOd2Y5ZXFVbHNtTXZmMGdBdWVKTUpUR215dG1iWjlpbXQKL2RnWlpYVE9hTElHUG9oZ3BpS0l5eVVRZVdGQ2F0NHRxWkNPVWRhbUloOGk0Q1h6QVJYVHNvcUNOenNNLzZMRQphM25XbFZyS2lmZHYrWkxyRi8vblc0VVNvOEoxaCtQeDljY0tpRDZZU0RVUERDRGh1RUtFWXcvbHpoUDJVOXNmCnl6cEJKVGQ4enFyM3paTjNGWWlITmgzYlRhQS82di9jU2lyamNTK1EwQXg4RWpzQzYxRjRVMTc4QzdWNWRCKzQKcmtPTy9QNlA0UFlWNTRZZHMvRjE2WkZJTHFBNENCYnExRExuYWRxamxyN3NPbzl2ZzNnWFNMYXBVVkdtZ2todAp6VlZPWG1mU0Z4OS90MDBHUi95bUdPbERJbWlXMGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

‎examples/https-termination/gateway.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,4 @@ spec:
1818
certificateRefs:
1919
- kind: Secret
2020
name: cafe-secret
21-
namespace: default
21+
namespace: certificate

‎examples/https-termination/reference-grant.yaml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
apiVersion: gateway.networking.k8s.io/v1beta1
2+
kind: ReferenceGrant
3+
metadata:
4+
name: allow-default-to-cafe-secret
5+
namespace: certificate
6+
spec:
7+
to:
8+
- group: ""
9+
kind: Secret
10+
name: cafe-secret # if you omit this name, then Gateways in default ns can access all Secrets in the certificate ns
11+
from:
12+
- group: gateway.networking.k8s.io
13+
kind: Gateway
14+
namespace: default

‎internal/events/handler.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,8 @@ func (h *EventHandlerImpl) propagateUpsert(e *UpsertEvent) {
126126
h.cfg.Processor.CaptureUpsertChange(r)
127127
case *v1beta1.HTTPRoute:
128128
h.cfg.Processor.CaptureUpsertChange(r)
129+
case *v1beta1.ReferenceGrant:
130+
h.cfg.Processor.CaptureUpsertChange(r)
129131
case *apiv1.Service:
130132
h.cfg.Processor.CaptureUpsertChange(r)
131133
case *apiv1.Namespace:
@@ -149,6 +151,8 @@ func (h *EventHandlerImpl) propagateDelete(e *DeleteEvent) {
149151
h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
150152
case *v1beta1.HTTPRoute:
151153
h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
154+
case *v1beta1.ReferenceGrant:
155+
h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
152156
case *apiv1.Service:
153157
h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
154158
case *apiv1.Namespace:

‎internal/manager/manager.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,9 @@ func Start(cfg config.Config) error {
121121
controller.WithK8sPredicate(k8spredicate.LabelChangedPredicate{}),
122122
},
123123
},
124+
{
125+
objectType: &gatewayv1beta1.ReferenceGrant{},
126+
},
124127
}
125128

126129
ctx := ctlr.SetupSignalHandler()
@@ -207,6 +210,7 @@ func prepareFirstEventBatchPreparerArgs(
207210
&apiv1.NamespaceList{},
208211
&discoveryV1.EndpointSliceList{},
209212
&gatewayv1beta1.HTTPRouteList{},
213+
&gatewayv1beta1.ReferenceGrantList{},
210214
}
211215

212216
if gwNsName == nil {

‎internal/manager/manager_test.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ func TestPrepareFirstEventBatchPreparerArgs(t *testing.T) {
3434
&discoveryV1.EndpointSliceList{},
3535
&gatewayv1beta1.HTTPRouteList{},
3636
&gatewayv1beta1.GatewayList{},
37+
&gatewayv1beta1.ReferenceGrantList{},
3738
},
3839
},
3940
{
@@ -52,6 +53,7 @@ func TestPrepareFirstEventBatchPreparerArgs(t *testing.T) {
5253
&apiv1.NamespaceList{},
5354
&discoveryV1.EndpointSliceList{},
5455
&gatewayv1beta1.HTTPRouteList{},
56+
&gatewayv1beta1.ReferenceGrantList{},
5557
},
5658
},
5759
}

‎internal/state/change_processor.go

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -85,11 +85,12 @@ type ChangeProcessorImpl struct {
8585
// NewChangeProcessorImpl creates a new ChangeProcessorImpl for the Gateway resource with the configured namespace name.
8686
func NewChangeProcessorImpl(cfg ChangeProcessorConfig) *ChangeProcessorImpl {
8787
clusterStore := graph.ClusterState{
88-
GatewayClasses: make(map[types.NamespacedName]*v1beta1.GatewayClass),
89-
Gateways: make(map[types.NamespacedName]*v1beta1.Gateway),
90-
HTTPRoutes: make(map[types.NamespacedName]*v1beta1.HTTPRoute),
91-
Services: make(map[types.NamespacedName]*apiv1.Service),
92-
Namespaces: make(map[types.NamespacedName]*apiv1.Namespace),
88+
GatewayClasses: make(map[types.NamespacedName]*v1beta1.GatewayClass),
89+
Gateways: make(map[types.NamespacedName]*v1beta1.Gateway),
90+
HTTPRoutes: make(map[types.NamespacedName]*v1beta1.HTTPRoute),
91+
Services: make(map[types.NamespacedName]*apiv1.Service),
92+
Namespaces: make(map[types.NamespacedName]*apiv1.Namespace),
93+
ReferenceGrants: make(map[types.NamespacedName]*v1beta1.ReferenceGrant),
9394
}
9495

9596
extractGVK := func(obj client.Object) schema.GroupVersionKind {
@@ -119,6 +120,11 @@ func NewChangeProcessorImpl(cfg ChangeProcessorConfig) *ChangeProcessorImpl {
119120
store: newObjectStoreMapAdapter(clusterStore.HTTPRoutes),
120121
trackUpsertDelete: true,
121122
},
123+
{
124+
gvk: extractGVK(&v1beta1.ReferenceGrant{}),
125+
store: newObjectStoreMapAdapter(clusterStore.ReferenceGrants),
126+
trackUpsertDelete: true,
127+
},
122128
{
123129
gvk: extractGVK(&apiv1.Namespace{}),
124130
store: newObjectStoreMapAdapter(clusterStore.Namespaces),
@@ -145,8 +151,8 @@ func NewChangeProcessorImpl(cfg ChangeProcessorConfig) *ChangeProcessorImpl {
145151

146152
var err error
147153
switch o := obj.(type) {
148-
// We don't validate GatewayClass, because as of 0.7.1, the webhook doesn't validate it (it only
149-
// validates an update that requires the previous version of the resource,
154+
// We don't validate GatewayClass or ReferenceGrant, because as of 0.7.1, the webhook doesn't validate them.
155+
// It only validates a GatewayClass update that requires the previous version of the resource,
150156
// which NKG cannot reliably provide - for example, after NKG restarts).
151157
// https://github.com/kubernetes-sigs/gateway-api/blob/v0.7.1/apis/v1beta1/validation/gatewayclass.go#L28
152158
case *v1beta1.Gateway:

0 commit comments

Comments
 (0)