Skip to content

Commit cb72c25

Browse files
sjbermansjberman
authored
Add tracing how-to guide (#2026)
Problem: As a user, I want to know how to configure tracing, so I can observe and debug my requests. Solution: Add a how-to guide that describes how to configure tracing.
1 parent 3220d74 commit cb72c25

12 files changed

+444
-7
lines changed

‎.yamllint.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ ignore:
77
- charts/nginx-gateway-fabric/
88
- config/crd/bases/
99
- deploy/crds.yaml
10+
- site/static
1011

1112
rules:
1213
braces: enable

‎site/content/how-to/maintenance/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: "Maintenance and Upgrades"
33
description:
4-
weight: 400
4+
weight: 500
55
linkTitle: "Maintenance and Upgrades"
66
menu:
77
docs:

‎site/content/how-to/monitoring/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: "Monitoring and Troubleshooting"
33
description:
4-
weight: 500
4+
weight: 400
55
linkTitle: "Monitoring and Troubleshooting"
66
menu:
77
docs:

‎site/content/how-to/monitoring/dashboard.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: "NGINX Plus Dashboard"
33
description: "Learn how to view the NGINX Plus dashboard to see real-time metrics."
4-
weight: 200
4+
weight: 300
55
toc: true
66
docs: "DOCS-1417"
77
---

‎site/content/how-to/monitoring/prometheus.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Promet
6666

6767
Download the following sample dashboard and Import as a new Dashboard in the Grafana UI.
6868

69-
{{< download "grafana-dashboard.json" "ngf-grafana-dashboard.json" >}}
69+
- {{< download "grafana-dashboard.json" "ngf-grafana-dashboard.json" >}}
7070

7171
## Available metrics in NGINX Gateway Fabric
7272

‎site/content/how-to/monitoring/tracing.md

Lines changed: 328 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,328 @@
1+
---
2+
title: "Tracing"
3+
weight: 200
4+
toc: true
5+
docs: "DOCS-000"
6+
---
7+
8+
Learn how to configure tracing in NGINX Gateway Fabric.
9+
10+
## Overview
11+
12+
NGINX Gateway Fabric supports tracing using [OpenTelemetry](https://opentelemetry.io/). The official [NGINX OpenTelemetry Module](https://github.com/nginxinc/nginx-otel) instruments the NGINX data plane to export traces to a configured collector. Tracing data can be used with an OpenTelemetry Protocol (OTLP) exporter, such as the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector). This collector can then export data to one or more upstream collectors like [Jaeger](https://www.jaegertracing.io/), [DataDog](https://docs.datadoghq.com/tracing/), and many others. This is called the [Agent model](https://opentelemetry.io/docs/collector/deployment/agent/).
13+
14+
This guide explains how to enable tracing on HTTPRoutes using NGINX Gateway Fabric. It uses the OpenTelemetry Collector and Jaeger to process and collect the traces.
15+
16+
## Install the Collectors
17+
18+
The first step is to install the collectors. NGINX Gateway Fabric will be configured to export to the OpenTelemetry Collector, which is configured to export to Jaeger. This model allows the visualization collector (Jaeger) to be swapped with something else, or to add more collectors without needing to reconfigure NGINX Gateway Fabric. It is also possible to configure NGINX Gateway Fabric to export directly to Jaeger.
19+
20+
Create the namespace:
21+
22+
```shell
23+
kubectl create namespace monitoring
24+
```
25+
26+
Download the following files containing the configurations for the collectors:
27+
28+
- {{< download "otel-collector.yaml" "otel-collector.yaml" >}}
29+
- {{< download "jaeger.yaml" "jaeger.yaml" >}}
30+
31+
{{< note >}}These collectors are for demonstration purposes and are not tuned for production use.{{< /note >}}
32+
33+
Then install them:
34+
35+
```shell
36+
kubectl apply -f otel-collector.yaml -f jaeger.yaml -n monitoring
37+
```
38+
39+
Ensure the Pods are running:
40+
41+
```shell
42+
kubectl -n monitoring get pods
43+
```
44+
45+
```text
46+
NAME READY STATUS RESTARTS AGE
47+
jaeger-8469f69b86-bfpk9 1/1 Running 0 9s
48+
otel-collector-f786b7dfd-h2x9l 1/1 Running 0 9s
49+
```
50+
51+
Once running, you can access the Jaeger dashboard by using port-forwarding in the background:
52+
53+
```shell
54+
kubectl port-forward -n monitoring svc/jaeger 16686:16686 &
55+
```
56+
57+
Visit [http://127.0.0.1:16686](http://127.0.0.1:16686) to view the dashboard.
58+
59+
## Enable tracing
60+
61+
To enable tracing, you must configure two resources:
62+
63+
- `NginxProxy`: This resource contains global settings relating to the NGINX data plane. It is created and managed by the [cluster operator](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/), and is referenced in the `parametersRef` field of the GatewayClass. This resource can be created and linked when we install NGINX Gateway Fabric using its helm chart, or it can be added later. This guide installs the resource using the helm chart, but the resource can also be created for an existing deployment.
64+
65+
The `NginxProxy` resource contains configuration for the collector, and applies to all Gateways and routes under the GatewayClass. It does not enable tracing, but is a prerequisite to the next piece of configuration.
66+
67+
- `ObservabilityPolicy`: This resource is a [Direct PolicyAttachment](https://gateway-api.sigs.k8s.io/reference/policy-attachment/) that targets HTTPRoutes or GRPCRoutes. It is created by the [application developer](https://gateway-api.sigs.k8s.io/concepts/roles-and-personas/) and enables tracing for a specific route or routes. It requires the `NginxProxy` resource to exist in order to complete the tracing configuration.
68+
69+
### Install NGINX Gateway Fabric with global tracing configuration
70+
71+
{{< note >}}Ensure that you [install the Gateway API resources]({{< relref "installation/installing-ngf/helm.md#installing-the-gateway-api-resources" >}}).{{< /note >}}
72+
73+
Referencing the previously deployed collector, create the following `values.yaml` file for installing NGINX Gateway Fabric:
74+
75+
```yaml
76+
cat <<EOT > values.yaml
77+
nginx:
78+
config:
79+
telemetry:
80+
exporter:
81+
endpoint: otel-collector.tracing.svc:4317
82+
spanAttributes:
83+
- key: cluster-attribute-key
84+
value: cluster-attribute-value
85+
EOT
86+
```
87+
88+
The span attribute will be added to all tracing spans.
89+
90+
To install:
91+
92+
```shell
93+
helm install ngf oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway -f values.yaml
94+
```
95+
96+
You should see the following configuration:
97+
98+
```shell
99+
kubectl get nginxproxies.gateway.nginx.org ngf-proxy-config -o yaml
100+
```
101+
102+
```yaml
103+
apiVersion: gateway.nginx.org/v1alpha1
104+
kind: NginxProxy
105+
metadata:
106+
name: ngf-proxy-config
107+
spec:
108+
telemetry:
109+
exporter:
110+
endpoint: otel-collector.tracing.svc:4317
111+
spanAttributes:
112+
- key: cluster-attribute-key
113+
value: cluster-attribute-value
114+
```
115+
116+
```shell
117+
kubectl get gatewayclasses.gateway.networking.k8s.io nginx -o yaml
118+
```
119+
120+
```yaml
121+
apiVersion: gateway.networking.k8s.io/v1
122+
kind: GatewayClass
123+
metadata:
124+
name: nginx
125+
spec:
126+
controllerName: gateway.nginx.org/nginx-gateway-controller
127+
parametersRef:
128+
group: gateway.nginx.org
129+
kind: NginxProxy
130+
name: ngf-proxy-config
131+
status:
132+
conditions:
133+
- lastTransitionTime: "2024-05-22T15:18:35Z"
134+
message: GatewayClass is accepted
135+
observedGeneration: 1
136+
reason: Accepted
137+
status: "True"
138+
type: Accepted
139+
- lastTransitionTime: "2024-05-22T15:18:35Z"
140+
message: Gateway API CRD versions are supported
141+
observedGeneration: 1
142+
reason: SupportedVersion
143+
status: "True"
144+
type: SupportedVersion
145+
- lastTransitionTime: "2024-05-22T15:18:35Z"
146+
message: parametersRef resource is resolved
147+
observedGeneration: 1
148+
reason: ResolvedRefs
149+
status: "True"
150+
type: ResolvedRefs
151+
```
152+
153+
If you already have NGINX Gateway Fabric installed, then you can create the `NginxProxy` resource and link it to the GatewayClass `parametersRef`:
154+
155+
```shell
156+
kubectl edit gatewayclasses.gateway.networking.k8s.io nginx
157+
```
158+
159+
Next, [Expose NGINX Gateway Fabric]({{< relref "installation/expose-nginx-gateway-fabric.md" >}}) and save the public IP address and port of NGINX Gateway Fabric into shell variables:
160+
161+
```text
162+
GW_IP=XXX.YYY.ZZZ.III
163+
GW_PORT=<port number>
164+
```
165+
166+
You can now create the application, route, and tracing policy.
167+
168+
### Create the application and route
169+
170+
Create the basic **coffee** application:
171+
172+
```yaml
173+
kubectl apply -f - <<EOF
174+
apiVersion: apps/v1
175+
kind: Deployment
176+
metadata:
177+
name: coffee
178+
spec:
179+
replicas: 2
180+
selector:
181+
matchLabels:
182+
app: coffee
183+
template:
184+
metadata:
185+
labels:
186+
app: coffee
187+
spec:
188+
containers:
189+
- name: coffee
190+
image: nginxdemos/nginx-hello:plain-text
191+
ports:
192+
- containerPort: 8080
193+
---
194+
apiVersion: v1
195+
kind: Service
196+
metadata:
197+
name: coffee
198+
spec:
199+
ports:
200+
- port: 80
201+
targetPort: 8080
202+
protocol: TCP
203+
name: http
204+
selector:
205+
app: coffee
206+
EOF
207+
```
208+
209+
Create the Gateway resource and HTTPRoute for the application:
210+
211+
```yaml
212+
kubectl apply -f - <<EOF
213+
apiVersion: gateway.networking.k8s.io/v1
214+
kind: Gateway
215+
metadata:
216+
name: cafe
217+
spec:
218+
gatewayClassName: nginx
219+
listeners:
220+
- name: http
221+
port: 80
222+
protocol: HTTP
223+
---
224+
apiVersion: gateway.networking.k8s.io/v1
225+
kind: HTTPRoute
226+
metadata:
227+
name: coffee
228+
spec:
229+
parentRefs:
230+
- name: cafe
231+
hostnames:
232+
- "cafe.example.com"
233+
rules:
234+
- matches:
235+
- path:
236+
type: PathPrefix
237+
value: /coffee
238+
backendRefs:
239+
- name: coffee
240+
port: 80
241+
EOF
242+
```
243+
244+
Check that traffic can flow to the application.
245+
246+
{{< note >}}If you have a DNS record allocated for `cafe.example.com`, you can send the request directly to that hostname, without needing to resolve.{{< /note >}}
247+
248+
```shell
249+
curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee
250+
```
251+
252+
You should receive a response from the coffee Pod.
253+
254+
```text
255+
Server address: 10.244.0.69:8080
256+
Server name: coffee-6b8b6d6486-k5w5w
257+
URI: /coffee
258+
```
259+
260+
You shouldn't see any information from the [Jaeger dashboard](http://127.0.0.1:16686) yet: you need to create the `ObservabilityPolicy`.
261+
262+
### Create the ObservabilityPolicy
263+
264+
To enable tracing for the coffee HTTPRoute, create the following policy:
265+
266+
```yaml
267+
kubectl apply -f - <<EOF
268+
apiVersion: gateway.nginx.org/v1alpha1
269+
kind: ObservabilityPolicy
270+
metadata:
271+
name: coffee
272+
spec:
273+
targetRefs:
274+
- group: gateway.networking.k8s.io
275+
kind: HTTPRoute
276+
name: coffee
277+
tracing:
278+
strategy: ratio
279+
ratio: 75
280+
spanAttributes:
281+
- key: coffee-key
282+
value: coffee-value
283+
EOF
284+
```
285+
286+
This policy attaches to the coffee HTTPRoute and enables ratio-based tracing, sampling 75% of requests. The span attribute is only included in the spans for the routes referenced in this policy.
287+
288+
Check the status of the policy:
289+
290+
```shell
291+
kubectl describe observabilitypolicies.gateway.nginx.org coffee
292+
```
293+
294+
```text
295+
Status:
296+
Ancestors:
297+
Ancestor Ref:
298+
Group: gateway.networking.k8s.io
299+
Kind: HTTPRoute
300+
Name: coffee
301+
Namespace: default
302+
Conditions:
303+
Last Transition Time: 2024-05-23T18:13:03Z
304+
Message: Policy is accepted
305+
Observed Generation: 1
306+
Reason: Accepted
307+
Status: True
308+
Type: Accepted
309+
Controller Name: gateway.nginx.org/nginx-gateway-controller
310+
```
311+
312+
The `message` field shows the policy is accepted. Run the next command multiple times to create new traffic.
313+
314+
```shell
315+
curl --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee
316+
```
317+
318+
Once complete, refresh the Jaeger dashboard. You should see a service entry called `ngf:default:cafe`, and a few traces. The default service name is `ngf:<gateway-namespace>:<gateway-name>`.
319+
320+
{{<img src="img/jaeger-trace-overview.png" alt="">}}
321+
322+
<br></br>
323+
324+
Select a trace to view the attributes.
325+
326+
{{<img src="img/jaeger-trace-attributes.png" alt="">}}
327+
328+
The trace includes the attribute from the global NginxProxy resource as well as the attribute from the ObservabilityPolicy.

‎site/content/how-to/monitoring/troubleshooting.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
---
22
title: "Troubleshooting"
3-
4-
weight: 300
3+
weight: 400
54
toc: true
65
docs: "DOCS-1419"
76
---

‎site/content/includes/installation/install-gateway-api-resources.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ To install the Gateway API resources, run the following:
1010
kubectl kustomize "https://github.com/nginxinc/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v1.2.0" | kubectl apply -f -
1111
```
1212

13-
{{<note>}}If you plan to use the `edge` version of NGINX Gateway Fabric, you can replace the vesion in `ref` with `main`, for example `ref=main`.{{</note>}}
13+
{{<note>}}If you plan to use the `edge` version of NGINX Gateway Fabric, you can replace the version in `ref` with `main`, for example `ref=main`.{{</note>}}
1414

1515
Alternatively, you can install the Gateway API resources from the experimental channel. We support a subset of the
1616
additional features provided by the experimental channel. To install from the experimental channel, run the following:

‎site/static/img/jaeger-trace-attributes.png

53.8 KB
Loading

‎site/static/img/jaeger-trace-overview.png

73.1 KB
Loading

0 commit comments

Comments
 (0)