Dynamic namespaces

[morhidi]

We are looking for a new feature that enables changing the watched namespaces dynamically without the need of an operator restart in (https://github.com/apache/flink-kubernetes-operator). This would significantly improve the operational experience in shared environments where a per namespace operator deployment is not an option.

[metacosm]

Hi @morhidi, how do you envision this working, though? How would you communicate the namespaces to be watched? A ConfigMap? Environment variables?

This is entering the meta-operational realm, though and I'm not sure we should address this problem at the SDK level because I'm not sure we could craft a generic solution that would cover all use cases without opening up potential issues… like what happens if the provided namespaces are invalid or if the operator is not deployed with enough permissions to watch them? This would also most likely mean dedicating an always running process to monitor a configuration source and could also run the risk of crashing the operator with an invalid configuration (which would lead to the opposite of the desired result).

A possible solution would be to deploy your operator watching all namespaces and then have an informer/watcher/event source that would watch for events on namespaces with a specific label. That, in turn, would dynamically manage an event source for the associated namespace… 🤔 Maybe adding an API to simplify that handling could be worked out…

[morhidi]

Hi @metacosm, thanks for your prompt reply. Let me add some context here. We would need to onboard our operator onto a larger shared environment where all operators run in a dedicated system namespace and where users register their namespaces in a self-service manner. The operators are not allowed to monitor all namespaces, and this is controlled by a fairly strict security policy.

We would only need, from the SDK perspective, the ability to change the namespaces being monitored. Then we would use a ConfigMap + SharedIndexInformer in our case to react on the ConfigMap changes (which is managed externally, and this is indeed meta-operational realm I fully agree with you)

Let me know what you think.

[csviri]

I believe not having cluster wide permissions in a quite common case. We already support watching just a list of namespaces both for controllers and informers (in InformerEventSource from v3) . So this issues is about changing this list in runtime. At the end this means stopping old and/or starting new informers in the background. What is a much more efficient way then restarting the whole operator.

A way that I picture the API for this is having a notion of RegisteredController:

  Operator operator = new Operator(client);
  RegisteredController registeredController = operator.register(new WebPageReconciler(client));
  operator.start();
  
  // in a different place in code (probably in config map informer event handler in this case)
  registeredController.changeNamespaces([new namespace list])

The RegisteredController would provide the operations over running controller.

Changing the informers under the hood should be fairly simple to implement. What we have to think about - and would be a bit more complex - is to make more explicit if an InformerEventSource inherits namespaces to watch from the controller. This is already supported, but the namespaces are just passed around on startup, this will probably require an additional flag in the InformerEventSource. Maybe a generic notification system for Event Sources about runtime namespace/config changes (but this might not needed in the first iteration).

[csviri]

For the record we already discussed the current possible workaround on discord:
Having a registered informers for the config map, that would on config map change simply restart the operator. (Not necessarily the pod will be restarted, just the actual operator stopped and a new created.) This works, just much less efficient, since the operator will reconcile every CR on startup. What could be a not desired side effect.

[morhidi]

Thanks @csviri this sounds great! Avoiding an operator restart would be the desirable goal here. In Flink we have lengthy upgrade operations, and we would like to lower the impact of unexpected errors non-coordinated restarts could introduce.

[metacosm]

The thing that I'm not quite clear on is how you deal with permissions… If your operator is not allowed to watch all namespaces, how do you give it permissions to operate on the new namespaces that would be dynamically changed?

[morhidi]

This is managed by an external component with elevated privileges.

[csviri]

@morhidi this is now merged to the main, will be part of the next release. See the docs already for usage:
https://javaoperatorsdk.io/docs/features#dynamically-changing-target-namespaces

[morhidi]

Thanks for the great work @csviri. I'll have a look at it soon!