Operator doesn't start when watching multiple namespaces

[gyfora]

Bug Report

Since upgrading to Java operator sdk 4.2.x the operator cannot start correctly if multiple namespaces (but not all) are watched.

There are no errors shown but the operator.start() call never returns.

With a single namespace:
https://gist.github.com/gyfora/1a8cf4a06ee0c0cdab513b147e3ddcb6

With 2 namespaces
https://gist.github.com/gyfora/0c6e32765e84e868a370900410f28205

This can be reproduced both locally and on our CI layer and it happens all the time depending on the namespace config.

[gyfora]

Minor correction, it doesn't happen all the time. Sometimes when you restart the pod it will start correctly, sometimes itt will get stuck.

[csviri]

On call we concluded that this issue appears (probably) in version 4.1.2. And for some reason we were not able to reproduce it locally.

[metacosm]

4.1.2 or 4.2.1?

[gyfora]

4.1.2 is the earliest version where we saw this problem. 4.1.1 still works correctly @metacosm

[csviri]

So from the logs it seems that clearly one Started event source is missing. But it is quite hard to see which one. Create the PR to improve logs, but until then, would be great to attach logs also with Thread ID-s. So at least it is clear which one.

@gyfora you mentioned that still starts processing events, that is probably for the other controller that starts correctly.

So it seems one of the informers is not started for some reason.

[metacosm]

Seems like the parallel starting of event sources is causing issues… it wasn't as simple to get right as I'd hoped so maybe (probably?) there's still something wrong…

[csviri]

Ok so seems to be was an issue caused by using these parallel streams on informer start:

https://github.com/java-operator-sdk/java-operator-sdk/blob/dc227686dc7d91223875afebfbd0ca018750df14/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerManager.java#L57-L63

While informer start is clearly blocking at some point:

https://github.com/java-operator-sdk/java-operator-sdk/blob/dc227686dc7d91223875afebfbd0ca018750df14/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerWrapper.java#L76-L81

Changed to this:
https://github.com/java-operator-sdk/java-operator-sdk/blob/066f954260285fddcb1f574f9ca697186c704481/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerManager.java#L62-L68

See PR for details.

It is not entirely clear for me how this is causing the issue, but parallel streams should not block by definition, probably this made issues inside informer implementation (since using a global thread pool).

After this change was not able to reproduce the issue.

[csviri]

Will do a patch release after merged. @gyfora
Pls test it after, if resolved we can close the issue. (It's a little suspicious, that although there were changes between 4.1.1 and 4.1.2, it does not seem to related at first sight)

[gyfora]

Thanks @csviri I can try to build and test this patch tomorrow

[gyfora]

@csviri @metacosm Your fix did not seem to have resolved this:
https://gist.github.com/gyfora/01b3236df3e325e6cb1f9a9d215e313e

Here are the logs with the latest commit

[csviri]

kk, will reopen the issue, was not able to reproduce it yesterday, but maybe it was just accidental