Add Graceful-Recovery NFR test to pipeline on PRs and merges to main

[bjee19]

Is your enhancement request related to a problem? Please describe.
The Graceful-Recovery NFR test can and should be added to an automated pipeline since the test does not require any human interpretation of results. However, there is no current infrastructure to run NFR tests in the pipeline (there are functional tests run in a Kind cluster and an NFR script that runs NFR tests and generates results).

What would you like to be added:
Creating a separate pipeline which can run NFR tests which need no report generated in a pipeline.

Why this is needed:
To streamline tests and remove dependency on developers interpreting test results.

[sjberman]

Going back to the discussion as to why this should be NFR and not functional:

It would be more of a hack/temporary fix since its not actually a functional test. The graceful-recovery test is currently written to be run on GKE (there would need to be slight changes for it to be run on the functional pipeline which is on a Kind cluster).
The issue would help set up for any future NFR tests which fall in the same category where there is no need for human interpretation of results but need to be run on GKE.

Do we want to be creating GKE clusters for every PR pipeline? That gets very expensive....if it's possible for this test to be run on kind, I would prefer for it to work like that.

If this test can then just run on kind, then I don't see a reason why it needs to be in a "different" test pipeline run than our other functional tests. For one, I would consider this a functional test. For two, at the end of the day it's just automated tests that should be run for every PR, so just include them in the same test run.

[bjee19]

IMO it can run on Kind. I think when I first created the NFR test, I chose to put it on GKE to simulate a production environment, however I think it is totally alright to move the test onto a Kind environment if we think the change in environment won't incur a change in results.