Description
Describe the bug
BackendGroups with namespace starts with digit doesn't proxy_passed to split_clients correctly.
For example, we have a namespace named 9c-network and one of its routes has multiple backends.
So ngf creates a split_clients named $9c_network__some_route_name_rule0 and generates a line proxy_pass http://$9c_network__some_route_name_rule0$request_uri;
However, it seems nginx processes the proxy_pass parameter as $9, c_network__..., not $9c_network__....
nginx responds with 502 Bad Gateway and I found a log that says no resolver defined to resolve c_network__some_route_name_rule0.
I suspect nginx think it's regex capture group if a split_clients starts with number.
To Reproduce
Steps to reproduce the behavior:
- create a namespace which its name starts with a digit
- create a random HTTPRoute in the namespace with parentRef pointing nginx-gateway-fabric
- http request to the route and see if it works
Expected behavior
responds with 2xx~4xx, not 502 Bad Gateway
Your environment
- Version of the NGINX Gateway Fabric - release version or a specific commit. The first line of the nginx-gateway container logs includes the commit info.:
"version":"1.4.0","commit":"8e653d6dfc671ca8f8d51f3eed29c25462f96e41" - Version of Kubernetes: 1.28
- Kubernetes platform (e.g. Mini-kube or GCP): AWS EKS
- Details on how you expose the NGINX Gateway Fabric Pod (e.g. Service of type LoadBalancer or port-forward): LoadBalancer with AWS NLB
Logs of NGINX container: kubectl -n nginx-gateway logs -l app=nginx-gateway -c nginx
Details
kubectl -n nginx-gateway logs -l app.kubernetes.io/name=nginx-gateway-fabric -c nginx 10.0.43.121 - - [26/Sep/2024:06:26:48 +0000] "GET /metrics HTTP/1.1" 404 146 "-" "Prometheus/2.41.0" 10.0.43.121 - - [26/Sep/2024:06:27:48 +0000] "GET /metrics HTTP/1.1" 404 146 "-" "Prometheus/2.41.0" 34.223.53.32 - - [26/Sep/2024:06:28:21 +0000] "GET / HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Linux; Android 7.1.1; 1713-A01) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.99 Mobile Safari/537.36" 10.0.43.121 - - [26/Sep/2024:06:28:48 +0000] "GET /metrics HTTP/1.1" 404 146 "-" "Prometheus/2.41.0" 10.0.43.121 - - [26/Sep/2024:06:29:48 +0000] "GET /metrics HTTP/1.1" 404 146 "-" "Prometheus/2.41.0" 10.0.43.121 - - [26/Sep/2024:06:30:48 +0000] "GET /metrics HTTP/1.1" 404 146 "-" "Prometheus/2.41.0" 10.0.43.121 - - [26/Sep/2024:06:31:48 +0000] "GET /metrics HTTP/1.1" 404 146 "-" "Prometheus/2.41.0" 2024/09/26 06:31:51 [error] 52#52: *6 no resolver defined to resolve c_network__gateway_remote_headless_graphql_9c_network_rule0, client: 106.101.131.240, server: 9c-main-rpc.nine-chronicles.com, request: "GET /graphql HTTP/1.1", host: "9c-main-rpc.nine-chronicles.com" 106.101.131.240 - - [26/Sep/2024:06:31:51 +0000] "GET /graphql HTTP/1.1" 502 150 "-" "curl/8.7.1" 2024/09/26 06:31:52 [info] 52#52: *6 client 106.101.131.240 closed keepalive connection
NGINX Configuration: kubectl -n nginx-gateway exec <gateway-pod> -c nginx -- nginx -T
Details
kubectl -n nginx-gateway exec nginx-gateway-fabric-56dc457fcd-2mwjg -c nginx -- nginx -T nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful # configuration file /etc/nginx/nginx.conf: load_module /usr/lib/nginx/modules/ngx_http_js_module.so; include /etc/nginx/module-includes/*.conf;worker_processes auto;
pid /var/run/nginx/nginx.pid;
error_log stderr info;events {
worker_connections 1024;
}http {
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/mime.types;
js_import /usr/lib/nginx/modules/njs/httpmatches.js;default_type application/octet-stream;
proxy_headers_hash_bucket_size 512;
proxy_headers_hash_max_size 1024;
server_names_hash_bucket_size 256;
server_names_hash_max_size 1024;
variables_hash_bucket_size 512;
variables_hash_max_size 1024;sendfile on;
tcp_nopush on;server_tokens off;
server {
listen unix:/var/run/nginx/nginx-status.sock;
access_log off;location /stub_status { stub_status; }}
}stream {
variables_hash_bucket_size 512;
variables_hash_max_size 1024;map_hash_max_size 2048;
map_hash_bucket_size 256;log_format stream-main '$remote_addr [$time_local] '
'$protocol $status $bytes_sent $bytes_received '
'$session_time "$ssl_preread_server_name"';
access_log /dev/stdout stream-main;
include /etc/nginx/stream-conf.d/*.conf;
}configuration file /etc/nginx/module-includes/load-modules.conf:
configuration file /etc/nginx/conf.d/config-version.conf:
server {
listen unix:/var/run/nginx/nginx-config-version.sock;
access_log off;location /version { return 200 2; }}
configuration file /etc/nginx/conf.d/http.conf:
http2 on;
Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
'' $host;
default $http_host;
}Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}Returns just the path from the original request URI.
map $request_uri $request_uri_path {
"~^(?P[^?])(?.)?$" $path;
}js_preload_object matches from /etc/nginx/conf.d/matches.json;
server {
listen 80 default_server;
listen [::]:80 default_server;default_type text/html; return 404;}
server {
listen 80;
listen [::]:80;server_name 9c-main-rpc.nine-chronicles.com; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://$9c_network__gateway_remote_headless_graphql_9c_network_rule0$request_uri; }}
server {
listen 80;
listen [::]:80;server_name heimdall-arena.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://heimdall_arena-service_80$request_uri; }}
server {
listen 80;
listen [::]:80;server_name heimdall-dp.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://invalid-backend-ref$request_uri; }}
server {
listen 80;
listen [::]:80;server_name heimdall-market.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://heimdall_market-service_80$request_uri; }}
server {
listen 80;
listen [::]:80;server_name heimdall-patrol.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://heimdall_patrol-reward-service_80$request_uri; }}
server {
listen 80;
listen [::]:80;server_name heimdall-rpc.nine-chronicles.com; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://$heimdall__gateway_remote_headless_graphql_heimdall_rule0$request_uri; }}
server {
listen 80;
listen [::]:80;server_name heimdall-world-boss.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://heimdall_world-boss-service_80$request_uri; }}
server {
listen 80;
listen [::]:80;server_name odin-arena.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://9c-network_arena-service_80$request_uri; }}
server {
listen 80;
listen [::]:80;server_name odin-dp.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://invalid-backend-ref$request_uri; }}
server {
listen 80;
listen [::]:80;server_name odin-market.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://9c-network_market-service_80$request_uri; }}
server {
listen 80;
listen [::]:80;server_name odin-patrol.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://9c-network_patrol-reward-service_80$request_uri; }}
server {
listen 80;
listen [::]:80;server_name odin-world-boss.9c.gg; location / { proxy_http_version 1.1; proxy_set_header Host "$gw_api_compliant_host"; proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; proxy_set_header Upgrade "$http_upgrade"; proxy_set_header Connection "$connection_upgrade"; proxy_pass http://9c-network_world-boss-service_80$request_uri; }}
server {
listen 31238 default_server;
listen [::]:31238 default_server;default_type text/html; return 404;}
server {
listen 31238;
listen [::]:31238;server_name 9c-main-rpc.nine-chronicles.com; location / { include /etc/nginx/grpc-error-pages.conf; proxy_http_version 1.1; grpc_set_header Host "$gw_api_compliant_host"; grpc_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; grpc_set_header Authority "$gw_api_compliant_host"; grpc_pass grpc://$9c_network__gateway_remote_headless_grpc_9c_network_rule0; } include /etc/nginx/grpc-error-locations.conf;}
server {
listen 31238;
listen [::]:31238;server_name heimdall-rpc.nine-chronicles.com; location / { include /etc/nginx/grpc-error-pages.conf; proxy_http_version 1.1; grpc_set_header Host "$gw_api_compliant_host"; grpc_set_header X-Forwarded-For "$proxy_add_x_forwarded_for"; grpc_set_header Authority "$gw_api_compliant_host"; grpc_pass grpc://$heimdall__gateway_remote_headless_grpc_heimdall_rule0; } include /etc/nginx/grpc-error-locations.conf;}
server {
listen unix:/var/run/nginx/nginx-502-server.sock;
access_log off;return 502;}
server {
listen unix:/var/run/nginx/nginx-500-server.sock;
access_log off;return 500;}
upstream heimdall_remote-headless-2_80 {
random two least_conn;
zone heimdall_remote-headless-2_80 512k;server 10.0.33.5:80;}
upstream 9c-network_market-service_80 {
random two least_conn;
zone 9c-network_market-service_80 512k;server 10.0.39.117:80;}
upstream 9c-network_patrol-reward-service_80 {
random two least_conn;
zone 9c-network_patrol-reward-service_80 512k;server 10.0.36.40:80;}
upstream heimdall_remote-headless-4_31238 {
random two least_conn;
zone heimdall_remote-headless-4_31238 512k;server 10.0.37.227:31238;}
upstream heimdall_remote-headless-1_80 {
random two least_conn;
zone heimdall_remote-headless-1_80 512k;server 10.0.47.171:80;}
upstream 9c-network_remote-headless-2_80 {
random two least_conn;
zone 9c-network_remote-headless-2_80 512k;server 10.0.45.39:80;}
upstream 9c-network_remote-headless-1_31238 {
random two least_conn;
zone 9c-network_remote-headless-1_31238 512k;server 10.0.34.40:31238;}
upstream heimdall_remote-headless-4_80 {
random two least_conn;
zone heimdall_remote-headless-4_80 512k;server 10.0.37.227:80;}
upstream heimdall_world-boss-service_80 {
random two least_conn;
zone heimdall_world-boss-service_80 512k;server 10.0.35.193:5000;}
upstream 9c-network_arena-service_80 {
random two least_conn;
zone 9c-network_arena-service_80 512k;server 10.0.40.113:8080;}
upstream heimdall_remote-headless-1_31238 {
random two least_conn;
zone heimdall_remote-headless-1_31238 512k;server 10.0.47.171:31238;}
upstream 9c-network_remote-headless-1_80 {
random two least_conn;
zone 9c-network_remote-headless-1_80 512k;server 10.0.34.40:80;}
upstream heimdall_patrol-reward-service_80 {
random two least_conn;
zone heimdall_patrol-reward-service_80 512k;server 10.0.41.135:80;}
upstream heimdall_arena-service_80 {
random two least_conn;
zone heimdall_arena-service_80 512k;server 10.0.44.99:8080;}
upstream 9c-network_world-boss-service_80 {
random two least_conn;
zone 9c-network_world-boss-service_80 512k;server 10.0.47.137:5000;}
upstream heimdall_market-service_80 {
random two least_conn;
zone heimdall_market-service_80 512k;server 10.0.41.214:80;}
upstream 9c-network_remote-headless-2_31238 {
random two least_conn;
zone 9c-network_remote-headless-2_31238 512k;server 10.0.45.39:31238;}
upstream heimdall_remote-headless-2_31238 {
random two least_conn;
zone heimdall_remote-headless-2_31238 512k;server 10.0.33.5:31238;}
upstream heimdall_remote-headless-3_31238 {
random two least_conn;
zone heimdall_remote-headless-3_31238 512k;server 10.0.47.143:31238;}
upstream heimdall_remote-headless-3_80 {
random two least_conn;
zone heimdall_remote-headless-3_80 512k;server 10.0.47.143:80;}
upstream invalid-backend-ref {
random two least_conn;server unix:/var/run/nginx/nginx-500-server.sock;}
split_clients $request_id $9c_network__gateway_remote_headless_graphql_9c_network_rule0 {
50.00% 9c-network_remote-headless-1_80;
50.00% 9c-network_remote-headless-2_80;
}split_clients $request_id $9c_network__gateway_remote_headless_grpc_9c_network_rule0 {
50.00% 9c-network_remote-headless-1_31238;
50.00% 9c-network_remote-headless-2_31238;
}split_clients $request_id $heimdall__gateway_remote_headless_graphql_heimdall_rule0 {
25.00% heimdall_remote-headless-1_80;
25.00% heimdall_remote-headless-2_80;
25.00% heimdall_remote-headless-3_80;
25.00% heimdall_remote-headless-4_80;
}split_clients $request_id $heimdall__gateway_remote_headless_grpc_heimdall_rule0 {
25.00% heimdall_remote-headless-1_31238;
25.00% heimdall_remote-headless-2_31238;
25.00% heimdall_remote-headless-3_31238;
25.00% heimdall_remote-headless-4_31238;
}configuration file /etc/nginx/grpc-error-pages.conf:
error_page 400 = @grpc_internal;
error_page 401 = @grpc_unauthenticated;
error_page 403 = @grpc_permission_denied;
error_page 404 = @grpc_unimplemented;
error_page 429 = @grpc_unavailable;
error_page 502 = @grpc_unavailable;
error_page 503 = @grpc_unavailable;
error_page 504 = @grpc_unavailable;
error_page 405 = @grpc_internal;
error_page 408 = @grpc_deadline_exceeded;
error_page 413 = @grpc_resource_exhausted;
error_page 414 = @grpc_resource_exhausted;
error_page 415 = @grpc_internal;
error_page 426 = @grpc_internal;
error_page 495 = @grpc_unauthenticated;
error_page 496 = @grpc_unauthenticated;
error_page 497 = @grpc_internal;
error_page 500 = @grpc_internal;
error_page 501 = @grpc_internal;configuration file /etc/nginx/grpc-error-locations.conf:
location @grpc_deadline_exceeded {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 4;
add_header grpc-message 'deadline exceeded';
return 204;
}location @grpc_permission_denied {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 7;
add_header grpc-message 'permission denied';
return 204;
}location @grpc_resource_exhausted {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 8;
add_header grpc-message 'resource exhausted';
return 204;
}location @grpc_unimplemented {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 12;
add_header grpc-message unimplemented;
return 204;
}location @grpc_internal {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 13;
add_header grpc-message 'internal error';
return 204;
}location @grpc_unavailable {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 14;
add_header grpc-message unavailable;
return 204;
}location @grpc_unauthenticated {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 16;
add_header grpc-message unauthenticated;
return 204;
}configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;text/mathml mml; text/plain txt; text/vnd.sun.j2me.app-descriptor jad; text/vnd.wap.wml wml; text/x-component htc; image/avif avif; image/png png; image/svg+xml svg svgz; image/tiff tif tiff; image/vnd.wap.wbmp wbmp; image/webp webp; image/x-icon ico; image/x-jng jng; image/x-ms-bmp bmp; font/woff woff; font/woff2 woff2; application/java-archive jar war ear; application/json json; application/mac-binhex40 hqx; application/msword doc; application/pdf pdf; application/postscript ps eps ai; application/rtf rtf; application/vnd.apple.mpegurl m3u8; application/vnd.google-earth.kml+xml kml; application/vnd.google-earth.kmz kmz; application/vnd.ms-excel xls; application/vnd.ms-fontobject eot; application/vnd.ms-powerpoint ppt; application/vnd.oasis.opendocument.graphics odg; application/vnd.oasis.opendocument.presentation odp; application/vnd.oasis.opendocument.spreadsheet ods; application/vnd.oasis.opendocument.text odt; application/vnd.openxmlformats-officedocument.presentationml.presentation pptx; application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx; application/vnd.openxmlformats-officedocument.wordprocessingml.document docx; application/vnd.wap.wmlc wmlc; application/wasm wasm; application/x-7z-compressed 7z; application/x-cocoa cco; application/x-java-archive-diff jardiff; application/x-java-jnlp-file jnlp; application/x-makeself run; application/x-perl pl pm; application/x-pilot prc pdb; application/x-rar-compressed rar; application/x-redhat-package-manager rpm; application/x-sea sea; application/x-shockwave-flash swf; application/x-stuffit sit; application/x-tcl tcl tk; application/x-x509-ca-cert der pem crt; application/x-xpinstall xpi; application/xhtml+xml xhtml; application/xspf+xml xspf; application/zip zip; application/octet-stream bin exe dll; application/octet-stream deb; application/octet-stream dmg; application/octet-stream iso img; application/octet-stream msi msp msm; audio/midi mid midi kar; audio/mpeg mp3; audio/ogg ogg; audio/x-m4a m4a; audio/x-realaudio ra; video/3gpp 3gpp 3gp; video/mp2t ts; video/mp4 mp4; video/mpeg mpeg mpg; video/quicktime mov; video/webm webm; video/x-flv flv; video/x-m4v m4v; video/x-mng mng; video/x-ms-asf asx asf; video/x-ms-wmv wmv; video/x-msvideo avi;}
configuration file /etc/nginx/stream-conf.d/stream.conf:
server {
listen unix:/var/run/nginx/connection-closed-server.sock;
return "";
}
Additional context
I think it can be resolved by simply adding something before first %s
https://github.com/nginxinc/nginx-gateway-fabric/blob/88269accb505d58567fde59dec3830cf963dc968/internal/mode/static/state/dataplane/types.go#L258
https://github.com/nginx/nginx/blob/51857ce40400b48bc8900b9e3930cf7474fa0c41/src/http/ngx_http_script.c#L474-L480
Metadata
Metadata
Assignees
Labels
Type
Projects
Status