Can you have dependent resources, which state is managed by another custom K8s controller

[ivanchuchulski]

Hello,

We are migrating a Helm Chart to Operator based provisioning of an application. We have a case where a couple of Secret and ConfigMap dependent resources should have their data managed/updated by another controller, which is independent of our Operator. However, our operator is expected to create these resources with some predefined data when an instance of a primary CR is created and to respectively delete them, when the primary resource is removed from the cluster.

Can we achieve this, by creating a dependent resources for the aforementioned resources, which extends CRUDNoGCKubernetesDependentResourceand possibly override the match method from KubernetesDependentResource as described in the docs? What value should we return to indicate to the framework, that the current desired state of the resource matches the one in the cluster?

Best Regards,
Ivan

[csviri]

Hi @ivanchuchulski ,

Yes, in this case would not CRUDNoGCKubernetesDependentResource but rather just KubernetesDependentResource and implement Creator and Deleter interface. (Althoug I don't see from what you write, why it should not be garbage collected, is it in other namespace than the primary resource?)

Since the Updater will not be implemented, the framework won't case about the updates on the resource.

[ivanchuchulski]

Hi, @csviri ,

Thank you for the fast reply!

We are using CRUDNoGCKubernetesDependentResource as the base class for a class, that we extend in all of our dependent resources, since we add some domain logic in it, related to reconciliation conditions and also we have some resources, that aren't in the same namespace of the primary resource. Also, maybe it's important to mention that we're using standalone dependents in our Reconciler implementation and we set a ResourceDiscriminators to each of them, since we have a couple of resources of the same kind, including a couple of ConfigMaps and Secrets, that are being managed by the operator.

Having said that, your suggestion would impact our overall structure and I'm not entirely sure if we're able to do it. What are your thoughts of my initial suggestion to override the match?
What we've tried is to simply put return Result.computed(true, desired(primary, context)); in the body of the method and that seems to work from out testing so far. Are there any concerns, that not calling the super would maybe break some functionality?

Best Regards,
Ivan

[csviri]

I would say, try it, and if works go for it. Only make sure that that this is properly tested, and the future changes of JOSDK won't break it. Since this kind of goes against the definition and intended usage / contract. And there is no guarantee that such "hacks" will work in the future. But I don't see any reason why it should not work.

[ivanchuchulski]

Hello,

OK, we're going to stick with that for now. I was wondering if you'd seen something else like this case, since in my opinion this scenario isn't something that unusual to come up - to have more than one controllers manage state of the same resources.

Really this has been a tough one for our development, since I guess the other hard alternative would be to include the code of the external controller into the operator, which in itself would be quite a challenge, not to mention the double maintenance, that'd be required.

Since this kind of goes against the definition and intended usage / contract. And there is no guarantee that such "hacks" will work in the future. But I don't see any reason why it should not work.

I don't really see how the approach we're doing is different to what you've described in the docs about customizing the matching behavior, as this is where I got the idea from. Example is the code snippet in this section and also this this announcement

Note that it is possible to override the related methods/behavior on class level when extending the KubernetesDependentResource.

Again, I'd like to know, if there is some other way of achieving what we want, for example even if the extend a class, which implements the Updater, can't we customize it's update method somehow?

On the testing note, at least from my research It is quite hard to find frameworks and solutions, which could do the job easily. I've looked into the Fabric8 Mock server and I've actually managed to integrate with the JUnit 5 framework for operators, which I believe you also develop, but in a lot simpler scenario. I'd be glad if you some suggestions with regards to that topic or point out some articles and examples.

Best Regards,
Ivan

[csviri]

OK, we're going to stick with that for now. I was wondering if you'd seen something else like this case, since in my opinion this scenario isn't something that unusual to come up - to have more than one controllers manage state of the same resources.

JOSDK from version 4.4 support Server Side Apply (SSA) in dependent resources also, in fact this is the default. SSA was designed exactly for this purpose to have multiple controllers manage the same resource. From 4.5 the ConfigMaps (and Secrets) are a bit of an exception (but SSA can be still applied), see:

https://github.com/java-operator-sdk/java-operator-sdk/blob/44af34e785dbc9f46c70cc9894238988aaf64716/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ConfigurationService.java#L363-L363

Really this has been a tough one for our development, since I guess the other hard alternative would be to include the code of the external controller into the operator, which in itself would be quite a challenge, not to mention the double maintenance, that'd be required.

Not sure if I follow this part, why would you include other controllers logic? The current controller logic updates the resources, but the parts when only that of it's concern.

Again, I'd like to know, if there is some other way of achieving what we want, for example even if the extend a class, which implements the Updater, can't we customize it's update method somehow?

You are able to customize everything what you are able to override. This might require some time to study and understand the current implementation though.

I don't really see how the approach we're doing is different to what you've described in the docs about customizing the matching behavior, as this is where I got the idea from. Example is the code snippet in this section and also this this announcement

Since you don't want to update Updater should not be implemented according to the current design.

On the testing note, at least from my research It is quite hard to find frameworks and solutions, which could do the job easily. I've looked into the Fabric8 Mock server and I've actually managed to integrate with the JUnit 5 framework for operators, which I believe you also develop, but in a lot simpler scenario. I'd be glad if you some suggestions with regards to that topic or point out some articles and examples.

Take a look on the integration tests, usually I advice to write those against minikube/kind not for Mock Server, since mock server for example does not support SSA.

https://github.com/java-operator-sdk/java-operator-sdk/tree/44af34e785dbc9f46c70cc9894238988aaf64716/-framework/src/test/java/io/javasdk/operator

[ivanchuchulski]

Thank you very much for the provided answers and ideas, we'll see in the end what would be our best fit.

For example we tried to use the legacy matcher, we overrode the match method in the dependent resource with the following:

return GenericKubernetesResourceMatcher.match(this, actualResource, connectivityProxy, context, false, false, false);

and it works as expected, when the external controller acts upon the resource. Looking further into the different static methods, I noticed, that there is this one, which has a parameter for a ignore path.

Can you tell me what is the format, which the parameter has to be provided? I think from looking at the integration tests it isn't jsonpath. Can you tell me for example, how can we define an ignore path for a entry in a Secret data property? Trying with /data/<key_name> as a parameter value doesn't work.

edit: If we try with the following matcher impl:

return GenericKubernetesResourceMatcher.match(this, actualResource, connectivityProxy, context, true, true, false);

the resource is reverted to the state, defined in the desired, I guess meaning that the matcher returned false.
Does that mean that the matcher's specEquality parameter set to false maybe goes to check the spec property of the resources, but since Secrets and ConfigMaps don't have it something is get wrongly calculated? And more on that - does the parameter in general can be used to ignore the comparison of the the spec properties of the desired and actual resources, or as I think I understand from the JavaDoc, it's just a switch for either full equality(when it's set to true) or for equality only on the properties that are calculated in the desired method?

[ivanchuchulski]

Hello @csviri ,

Can you please check the questions from my last comment? Thanks in advance.

Regards,
Ivan

[csviri]

Hi @ivanchuchulski

Can you tell me what is the format, which the parameter has to be provided?

As stated in javadoc:

All changes with a target prefix path on a calculated JSON Patch between actual and desired will be ignored.

Can you tell me for example, how can we define an ignore path for a entry in a Secret data property? Trying with /data/<key_name> as a parameter value doesn't work

Good question, would need to take a look, might not cover this use case now, in that case you would need to just override matcher function and implement own matcher, but since you know the type this might be much easier to do than a generic solution.

the resource is reverted to the state, defined in the desired, I guess meaning that the matcher returned false.

You can turn on debug logs to see that if matched

Does that mean that the matcher's specEquality parameter set to false maybe goes to check the spec property of the resources, but since Secrets and ConfigMaps don't have it something is get wrongly calculated?

Will check on this one, but as you can see in the imple now it mainly uses json diff, so does not seems to be the case.

[csviri]

This should fix the config map matching:
#2121