Appwrappers end up in a status of "True" and Type 'backoff' when they should not

[z103cb]

When submitting the attached file the job-gold-lo-pri-1replica-08b

lastTransitionMicroTime: "2023-05-23T09:09:28.087364Z"
      lastUpdateMicroTime: "2023-05-23T09:09:28.087364Z"
      message: Insufficient quota to dispatch AppWrapper.
      reason: AppWrapperNotRunnable. consumer NAMESPACE_test_AWNAME_job-gold-lo-pri-1replica-08b
        already exists
      status: "True"
      type: Backoff

09-install.yaml.txt

This behaviour has also been observed while executing the quota management e2e tests. The failures are not predictable indicating that a race condition or some other defect exits in the code.

How to reproduced the issue:

1. Create a kind cluster with the same specs as in the e2e tests.
2. Submit the attached appwrappers.
3. examine the app wrapper state.

[z103cb]

@tardieu I think we should prioritise this issue into a must fix.

[z103cb]

This might be a simpthom of #392

[z103cb]

This issue has been resolved with the merging of #367

[z103cb]

This issue still persists, test cases added in #398, show that the problem is still there.

[z103cb]

This issue has been resolved by #397