NGINX reload fails with "failed to find NGINX main process"

[kate-osborn]

Describe the bug

Periodically, NGINX reload fails with the following error message:

{"level":"error","ts":"2023-04-27T19:45:07Z","logger":"eventHandler","msg":"Failed to update NGINX configuration","error":"failed to find NGINX main process: open /etc/nginx/nginx.pid: no such file or directory","stacktrace":"github.com/nginxinc/nginx-kubernetes-gateway/internal/events.(*EventHandlerImpl).HandleEventBatch\n\t/home/runner/work/nginx-kubernetes-gateway/nginx-kubernetes-gateway/internal/events/handler.go:85\ngithub.com/nginxinc/nginx-kubernetes-gateway/internal/events.(*EventLoop).Start.func1.1\n\t/home/runner/work/nginx-kubernetes-gateway/nginx-kubernetes-gateway/internal/events/loop.go:67"}

This happens when the gateway container starts before the NGINX container, and a reload is attempted before NGINX is running. To prevent this, we need to check that NGINX is running before attempting a reload.

To Reproduce
Happens intermittently on startup. Re-deploy until you see the error message.

Expected behavior
NKG should not attempt to reload when NGINX is not yet running.

Resolves FIXME: https://github.com/nginxinc/nginx-kubernetes-gateway/blob/6befb1dffbf6bb32be9257f8a6f373766a1ad372/internal/nginx/runtime/manager.go#L34

[sjberman]

In NSM we basically do the same thing that NKG does now where we first look to see if the PID file exists before attempting to reload. The difference is that if it didn't exist, in NSM we would just return (assuming that nginx hadn't started yet), whereas in NKG we return an error.

Maybe this fix is as simple as just not returning an error when the PID file is not found, with the assumption that this means nginx hasn't started yet?