Description
Bug Report
What did you do?
- start minikube
- kubectl create ns operator-sample-tomcat-issue
- kubectl config set-context --current --namespace=operator-sample-tomcat-issue
- mvn clean package
- kubectl apply -f sample-operators/tomcat-operator/target/classes/META-INF/fabric8/tomcats.tomcatoperator.io-v1.yml
- kubectl apply -f sample-operators/tomcat-operator/target/classes/META-INF/fabric8/webapps.tomcatoperator.io-v1.yml
- Start the TomcatOperator main class
- kubectl apply -f sample-operators/tomcat-operator/k8s/tomcat-sample1.yaml
- kubectl apply -f sample-operators/tomcat-operator/k8s/webapp-sample1.yaml
What did you expect to see?
Operator should first deploy two replicas of a tomcat instance and as soon as they become ready deploy the webapp.
What did you see instead? Under which circumstances?
Operator deploys two replicas of a tomcat instance, both became ready, then the WebappReconciler fails with the following exception:
13:40:53.414 ERROR io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64) - Error during event processing ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543} failed.
java.lang.IllegalStateException: Cannot find Tomcat test-tomcat1 for Webapp sample-webapp1 in namespace operator-sample-tomcat-issue
at io.javaoperatorsdk.operator.sample.WebappReconciler.lambda$reconcile$2(WebappReconciler.java:84) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:81) ~[classes/:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:35) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) ~[classes/:?]
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) [classes/:?]
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
This is due to the fact, that the primaryToSecondaryIndex
in io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource
is empty until the operator restarts.
Environment
Kubernetes cluster type:
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0
$ Mention java-operator-sdk version from pom.xml file
Tested with git tags
v4.3.1
v4.3.5
$ java -version
openjdk version "17-ea" 2021-03-16
OpenJDK Runtime Environment Zulu17+33-CA (build 17-ea+12)
OpenJDK 64-Bit Server VM Zulu17+33-CA (build 17-ea+12, mixed mode, sharing)
$ kubectl version --output=yaml
clientVersion:
buildDate: "2023-04-14T13:14:41Z"
compiler: gc
gitCommit: 4c9411232e10168d7b050c49a1b59f6df9d7ea4b
gitTreeState: clean
gitVersion: v1.27.1
goVersion: go1.20.3
major: "1"
minor: "27"
platform: darwin/arm64
kustomizeVersion: v5.0.1
serverVersion:
buildDate: "2023-03-15T13:33:12Z"
compiler: gc
gitCommit: 9e644106593f3f4aa98f8a84b23db5fa378900bd
gitTreeState: clean
gitVersion: v1.26.3
goVersion: go1.19.7
major: "1"
minor: "26"
platform: linux/arm64
Possible Solution
Just a workaround: when restarting the operator, everything works fine as long as no new tomcat gets deployed.
For each new tomcat and webapp the same exception rises again until the operator gets restarted.
Additional context
Full log:
13:40:03.236 WARN io.javaoperatorsdk.operator.api.config.BaseConfigurationService.logMissingReconcilerWarning(BaseConfigurationService.java:61) - Configuration for reconciler 'tomcatreconciler' was not found. Known reconcilers: None.
13:40:03.267 INFO io.javaoperatorsdk.operator.api.config.BaseConfigurationService.getConfigurationFor(BaseConfigurationService.java:82) - Created configuration for reconciler io.javaoperatorsdk.operator.sample.TomcatReconciler with name tomcatreconciler
13:40:03.373 INFO io.javaoperatorsdk.operator.Operator.register(Operator.java:223) - Registered reconciler: 'tomcatreconciler' for resource: 'class io.javaoperatorsdk.operator.sample.Tomcat' for namespace(s): [all namespaces]
13:40:03.376 WARN io.javaoperatorsdk.operator.api.config.BaseConfigurationService.logMissingReconcilerWarning(BaseConfigurationService.java:61) - Configuration for reconciler 'webappreconciler' was not found. Known reconcilers: tomcatreconciler.
13:40:03.377 INFO io.javaoperatorsdk.operator.api.config.BaseConfigurationService.getConfigurationFor(BaseConfigurationService.java:82) - Created configuration for reconciler io.javaoperatorsdk.operator.sample.WebappReconciler with name webappreconciler
13:40:03.380 INFO io.javaoperatorsdk.operator.Operator.register(Operator.java:223) - Registered reconciler: 'webappreconciler' for resource: 'class io.javaoperatorsdk.operator.sample.Webapp' for namespace(s): [all namespaces]
13:40:03.382 INFO io.javaoperatorsdk.operator.Operator.start(Operator.java:129) - Operator SDK 4.3.1-SNAPSHOT (commit: 2c51ca7) built on 2023-06-16T13:35:52.000+0200 starting...
13:40:03.383 INFO io.javaoperatorsdk.operator.Operator.start(Operator.java:136) - Client version: 6.5.1
13:40:03.386 INFO io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:320) - Starting 'webappreconciler' controller for reconciler: io.javaoperatorsdk.operator.sample.WebappReconciler, resource: io.javaoperatorsdk.operator.sample.Webapp
13:40:03.386 INFO io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:320) - Starting 'tomcatreconciler' controller for reconciler: io.javaoperatorsdk.operator.sample.TomcatReconciler, resource: io.javaoperatorsdk.operator.sample.Tomcat
13:40:04.686 INFO io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:332) - 'webappreconciler' controller started
13:40:05.020 INFO io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:332) - 'tomcatreconciler' controller started
13:40:33.619 INFO io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.logForOperation(AbstractDependentResource.java:110) - Creating 'test-tomcat1' Service for primary ResourceID{name='test-tomcat1', namespace='operator-sample-tomcat-issue'}
13:40:33.679 INFO io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.logForOperation(AbstractDependentResource.java:110) - Creating 'test-tomcat1' Deployment for primary ResourceID{name='test-tomcat1', namespace='operator-sample-tomcat-issue'}
13:40:33.830 INFO io.javaoperatorsdk.operator.sample.TomcatReconciler.lambda$reconcile$0(TomcatReconciler.java:33) - Updating status of Tomcat test-tomcat1 in namespace operator-sample-tomcat-issue to 0 ready replicas
13:40:33.974 INFO io.javaoperatorsdk.operator.sample.TomcatReconciler.lambda$reconcile$0(TomcatReconciler.java:33) - Updating status of Tomcat test-tomcat1 in namespace operator-sample-tomcat-issue to 0 ready replicas
13:40:35.887 INFO io.javaoperatorsdk.operator.sample.TomcatReconciler.lambda$reconcile$0(TomcatReconciler.java:33) - Updating status of Tomcat test-tomcat1 in namespace operator-sample-tomcat-issue to 1 ready replicas
13:40:35.952 INFO io.javaoperatorsdk.operator.sample.TomcatReconciler.lambda$reconcile$0(TomcatReconciler.java:33) - Updating status of Tomcat test-tomcat1 in namespace operator-sample-tomcat-issue to 2 ready replicas
13:40:53.414 ERROR io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64) - Error during event processing ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543} failed.
java.lang.IllegalStateException: Cannot find Tomcat test-tomcat1 for Webapp sample-webapp1 in namespace operator-sample-tomcat-issue
at io.javaoperatorsdk.operator.sample.WebappReconciler.lambda$reconcile$2(WebappReconciler.java:84) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:81) ~[classes/:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:35) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) ~[classes/:?]
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) [classes/:?]
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
13:40:55.455 ERROR io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64) - Error during event processing ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543} failed.
java.lang.IllegalStateException: Cannot find Tomcat test-tomcat1 for Webapp sample-webapp1 in namespace operator-sample-tomcat-issue
at io.javaoperatorsdk.operator.sample.WebappReconciler.lambda$reconcile$2(WebappReconciler.java:84) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:81) ~[classes/:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:35) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) ~[classes/:?]
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) [classes/:?]
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
13:40:57.465 ERROR io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64) - Error during event processing ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543} failed.
java.lang.IllegalStateException: Cannot find Tomcat test-tomcat1 for Webapp sample-webapp1 in namespace operator-sample-tomcat-issue
at io.javaoperatorsdk.operator.sample.WebappReconciler.lambda$reconcile$2(WebappReconciler.java:84) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:81) ~[classes/:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:35) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) ~[classes/:?]
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) [classes/:?]
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
13:41:00.564 ERROR io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64) - Error during event processing ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543} failed.
java.lang.IllegalStateException: Cannot find Tomcat test-tomcat1 for Webapp sample-webapp1 in namespace operator-sample-tomcat-issue
at io.javaoperatorsdk.operator.sample.WebappReconciler.lambda$reconcile$2(WebappReconciler.java:84) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:81) ~[classes/:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:35) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) ~[classes/:?]
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) [classes/:?]
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
13:41:05.098 ERROR io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64) - Error during event processing ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543} failed.
java.lang.IllegalStateException: Cannot find Tomcat test-tomcat1 for Webapp sample-webapp1 in namespace operator-sample-tomcat-issue
at io.javaoperatorsdk.operator.sample.WebappReconciler.lambda$reconcile$2(WebappReconciler.java:84) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:81) ~[classes/:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:35) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) ~[classes/:?]
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) [classes/:?]
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
13:41:11.869 ERROR io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64) - Error during event processing ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543} failed.
java.lang.IllegalStateException: Cannot find Tomcat test-tomcat1 for Webapp sample-webapp1 in namespace operator-sample-tomcat-issue
at io.javaoperatorsdk.operator.sample.WebappReconciler.lambda$reconcile$2(WebappReconciler.java:84) ~[classes/:?]
at java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:81) ~[classes/:?]
at io.javaoperatorsdk.operator.sample.WebappReconciler.reconcile(WebappReconciler.java:35) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96) ~[classes/:?]
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89) ~[classes/:?]
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62) [classes/:?]
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414) [classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
13:41:11.871 ERROR io.javaoperatorsdk.operator.processing.event.EventProcessor.lambda$handleRetryOnException$6(EventProcessor.java:320) - Exhausted retries for ExecutionScope{ resource id: ResourceID{name='sample-webapp1', namespace='operator-sample-tomcat-issue'}, version: 498543}