Metrics for deleted CRs may lead to OOM when scraping

[MikeEdgar]

Bug Report

What did you do?

Request to /metrics endpoint in Quarkus (using JOSDK w/Quarkus extension) resulted in an OOM error. Further research revealed that JOSDK reconcile metrics accumulate in the Micrometer registry and are not removed for CRs no longer present in the system.

What did you expect to see?

Expect metrics for deleted CRs be removed from the registry after a configurable amount of time, with a default of -1/never.

What did you see instead? Under which circumstances?

Metrics for deleted CRs are present until the process ends

Environment

Kubernetes cluster type:

$ Mention java-operator-sdk version from pom.xml file

JOSDK: 3.2.4 (via quarkus-operator-sdk 4.0.7) on Quarkus 2.13.7

$ java -version

11.0.x

$ kubectl version

Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4+a34b9e9", GitCommit:"b6d1f054747e9886f61dd85316deac3415e2726f", GitTreeState:"clean", BuildDate:"2023-01-10T15:55:28Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Clear/remove metrics for deleted CRs after a configurable amount of time, defaulting to "never" for compatibility.

Additional context

[metacosm]

Would upgrading to a more recent version of the SDK be a possibility as, at least on first approximation, we would only fix this on the main branch, i.e. for JOSDK 4.2 / 4.3 (Quarkus extension 5.0/5.1)?

[MikeEdgar]

I think it will probably be a while before we can upgrade again. We only just migrated to 3.2.4 (extension v4.0.7) For now, we're looking at using a no-op implementation of Metrics.