Implement Rate Limiting of Reconciliation

[csviri]

Rate limiting is per resource to be reconciled.

Note that rate limiting in controller-runtime works a little different way: https://danielmangum.com/posts/controller-runtime-client-go-rate-limiting/
It is more close to our retry mechanism there. Additional it is connected with the re-schedule, but on our side it is always a value specified, so in the current scope that part is irrelevant.

It makes sense to have a generic rate limiter, that is basically a fails-safe to do not reconcile too frequently the same custom resources. Taking in mind that event sources can trigger the reconciliation very frequently, if secondary resources are changing, but on level based approach not reacting instantly on every event is fine.

[scrocquesel]

Cool, I do have a use case for this where two different CRD are fighting to set a different value. When this occurs, the operator runs in an infinite loop. I would like to detect this and stop reconcile for a while.

[metacosm]

Maybe use a status indicator in the CR status to indicate that it's being processed? Do you have 2 different controllers for the same CRD? Or is it just contradictory information in your CRs? Seems like maybe a design issue?

[scrocquesel]

Maybe use a status indicator in the CR status to indicate that it's being processed? Do you have 2 different controllers for the same CRD? Or is it just contradictory information in your CRs? Seems like maybe a design issue?

It's contradictory information on two different CRs (not CRD like I said). I agree that two different CRD should not be concurrent on all or part of a desired state attributes set.

[csviri]

@scrocquesel not sure if this issue will solve that problem. That seems to be more like a design issue indeed. If you could maybe describe the situation more in depth in a separate issue we could take a look on that.