Error updating status of gateway resource in NFR Scale test HTTPSListeners

[bjee19]

In this pipeline run of the NFR test results on edge: https://github.com/nginxinc/nginx-gateway-fabric/actions/runs/10872403318, the scale test for HTTPSListeners encountered an error when ran on OSS.

Shown Error:

{
  "level": "debug",
  "ts": "2024-09-15T17:08:33Z",
  "logger": "statusUpdater",
  "msg": "Encountered error updating status",
  "error": "Operation cannot be fulfilled on gateways.gateway.networking.k8s.io \"gateway\": the object has been modified; please apply your changes to the latest version and try again",
  "namespace": "scale",
  "name": "gateway",
  "kind": "Gateway"
}

The full NFR test results can be found in this PR: #2554

[kate-osborn]

After some investigation, this does not appear to be a product bug. We retry on status update failures because the object may be modified between fetching the object and writing its status. On each retry, we fetch the latest version of the object. If multiple retries fail, we will log an error message saying, "Failed to update status." Since the logs do not contain that error message, we can assume that the status update eventually succeeded.

If this error persists in our NFR tests, we should look into the test code and see if we can prevent this situation from occurring. Perhaps we can wait for an object's status to update before modifying the object's spec.