where resources already exist but requests are generated by `CREATE`

[10000-ki]

Bug Report

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.23.0.1:443/apis/policy/v1/namespaces/n3r-platform-opensearch/poddisruptionbudgets/os-osfarm-cluster-manager-pdb?fieldManager=opensearchreconciler&force=true. Message: Operation cannot be fulfilled on poddisruptionbudgets.policy "os-osfarm-cluster-manager-pdb": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=policy, kind=poddisruptionbudgets, name=os-osfarm-cluster-manager-pdb, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on poddisruptionbudgets.policy "os-osfarm-cluster-manager-pdb": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:713)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:232)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:237)
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:252)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:1132)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:92)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.create(KubernetesDependentResource.java:140)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.CRUDKubernetesDependentResource.create(CRUDKubernetesDependentResource.java:16)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.handleCreate(AbstractDependentResource.java:114)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.handleCreate(KubernetesDependentResource.java:111)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.handleCreate(KubernetesDependentResource.java:32)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:62)
	at io.javaoperatorsdk.operator.processing.dependent.SingleDependentResourceReconciler.reconcile(SingleDependentResourceReconciler.java:19)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:52)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileExecutor$NodeReconcileExecutor.doRun(WorkflowReconcileExecutor.java:115)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.NodeExecutor.run(NodeExecutor.java:22)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.handleCreate(AbstractDependentResource.java:114)

What did you do?

Creating PDB with DependentResource

However, even though resources are already created

k get PodDisruptionBudget                                                                                                                                                                                                                                           ✔  10357  17:43:26
NAME                            MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
os-osfarm-cluster-manager-pdb   N/A             1                 1                     88m

When running additional 'reconcile' loop

The request was created as Create rather than Update

If there is a secondary resource, shouldn't the request go to update right?

Why is it created as create?

409 errors continue to occur, which is a problem

What did you expect to see?

What did you see instead? Under which circumstances?

no

Environment

Kubernetes cluster type:

Kubernetes

$ Mention java-operator-sdk version from pom.xml file

v4.4.4

$ java -version

jdk21

$ kubectl version

v1.23.15

Possible Solution

no

Additional context

no

[csviri]

Hi @10000-ki there were improvements regarding to this in newer versions, pls upgrade to the newest version

[10000-ki]

@csviri what version??
I raised it to 4.6.0, but there was the same issue

[csviri]

pls try the newest (4.8.1), if there will be still issues, we will take a look. Note that k8s 1.23 is also a bit old too, there were some issues with event sending in that version (what should be fixed now also on fabric8 client (restarting watches) ). But if the problem persist we can take a look, and try to make a reproducer. Probably the issue will be rather in the fabric8 client, since we are reading the last resource from the informer cache.

[10000-ki]

Note that k8s 1.23 is also a bit old too, there were some issues with event sending in that version (what should be fixed now also on fabric8 client (restarting watches) )

I'm using in-house k8s cluster in my company but v1.24 is maximum version :(
will that be a problem?

[10000-ki]

First, I'll increase the josdk version to the maximum v4.8.1

[csviri]

AFAIK, it should not be, there were known issues, that k8s stopped sending events at some point (if remember correctly still present in 1.23) therefore the watch had to be restarted in informers periodically. But that was done in fabric8, so now it should be good.

[10000-ki]

k get pdb --watch -n=opensearchoperatore2e-createopensearchcluster-6361d -o wide                                                                                                                                                                                  1 ↵  10373  18:15:17
NAME                                             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m6s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m6s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m6s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m6s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m6s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m6s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m7s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m7s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m7s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m7s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m7s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m7s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m7s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m8s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m8s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m8s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m8s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m8s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m8s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m8s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m9s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m9s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m9s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m9s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m9s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m9s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m9s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m10s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m10s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m10s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m10s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m10s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m10s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m10s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m11s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m11s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m11s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m11s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m11s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m11s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m11s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m12s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m12s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m12s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m12s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m12s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m12s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m12s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m13s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m13s
os-test-opensearch-cluster-cluster-manager-pdb   N/A             1                 0                     2m13s

Is there anything else I need to check?
After posting the v4.8.1 version
There are a lot of events then before

and Create issues are still occurring

[csviri]

ok, could you pls create a minimal open source reproducert project? Where we can run it on top of minikube and take a look on it, based on logs you provided it is impossible to figure out.

[csviri]

Also what you can you can check if this is a problem on the newer versions of k8s.

[10000-ki]

ok, could you pls create a minimal open source reproducert project? Where we can run it on top of minikube and take a look on it, based on logs you provided it is impossible to figure out.

okay i will try

[10000-ki]

@csviri
In the k8s 1.29 version, it seems to run smoothly (i tested in my local)

By the way, can you tell me exactly which version had the issue related to the k8s event and up to what version the impact is?

[csviri]

@csviri In the k8s 1.29 version, it seems to run smoothly (i tested in my local)

By the way, can you tell me exactly which version had the issue related to the k8s event and up to what version the impact is?

Pls take a look on these:
fabric8io/kubernetes-client#4781
kubernetes/kubernetes#102464

but there might be some newer ones. Maybe @shawkins knows more.

[shawkins]

I'm not aware of newer definitive underlying kube issues. There were several later client changes to limit any potential issues with the api server not sending events - we'll restart the watches at most every 10 minutes.

[10000-ki]

@csviri @shawkins

I don't know if it's related to the problem I raised.

The problems I defined are as follows.

1. even if secondary resource already exists, Create request sent to apserver, not update
2. Retry occurs with 409 error
3. Infinite retries occur as issue 1 continues to occur

mayBeActual field is null, even though resources are in k8s server

Is it because the operator did not receive the resource creation event from k8s?

[csviri]

Is it because the operator did not receive the resource creation event from k8s?

yes, that means it is not in the cache of the informer, so no event received regarding that resource.