How to handle CR and secret references set in a CR spec ?

[derlin]

TL;DR: how to support a one-to-many relationship between primary to secondary resource, in which the secondary resource doesn't know which primary resources depends on it ?

Take the https://external-secrets.io operator as an example, which fetches secrets from a secret manager (e.g. Hashicorp Vault) and creates Secret.

An ExternalSecret CR has a secretStoreRef in the spec, which references (with namespace and name) a (Cluster)SecretStore CR. The latter in turn has a reference (secretRef) to a Secret that holds the auth information to the secret manager.
When reconciling an ExternalSecret, we thus need to pull a SecretStore and a Secret. However, those two are not really managed, in the sense that they are not reconciled, just holding configuration information, and should never be created/updated/deleted by the operator.

I am trying to understand how to implement a similar mechanism with JOSDK. I tried using dependent resources, but the problem is, I cannot provide a SecondaryToPrimaryMapper as there is no way to know from a SecretStore which ExternalSecret may reference it. Same goes for Secret to SecretStore. Moreover, the SecretStore and Secret may be referenced by many ExternalSecret resources.

Does the SDK provide utilities for this use case, or do I need to manually do the lookup using the kubernetes client ? How would you implement this ?

related question: still using the external-secrets example, the ExternalSecret reconciler needs to generate a secret (implemented as a CRUDKubernetesDependentResource) and also to check some stuff in the vault. I thus need to access the SecretStore (and its related Secret) in both the reconciler's reconcile method, and in the dependent secret's desired method. I cannot find a way to share this SecretStore between the two. Is there a way to avoid fetching the config twice ? (I guess it is not a problem if the call is cached, but I struggle in understanding what is cached how. If I use the kubernetes client directly, I guess there is no caching, right ?

Let me know if something is unclear.

[metacosm]

Hi @derlin, that's an interesting use case!

Let me first make sure I understand your problem correctly: I guess you're writing an operator reconciling ExternalSecret resources, correct? And you want your reconciler to be triggered whenever a change occurs to your SecretStore and Secret secondary resources? How is the existing operator working? Is it reconciling ExternalSecret instances periodically to pick up changes to SecretStore instances?

[derlin]

@metacosm thank you for the quick reply :)

Let me be more precise with my actual use case, which is to provision external databases.

description

I have an EdbConfig (external database config) with information on how to connect to a database. To avoid having passwords in clear in the spec, I have a credentialsSecretRef that points to a secret with username and password.

I have one reconciler for EdbConfig, which just checks for the validity of the information, and updates the status (isOk, lastError). This would better be done in an admission webhook, but we don't have support yet ;)

This if for the config.

Now, I have an Edb CR, which actually triggers a database provisioning. This Edb has a reference to an EdbConfig, which tells in which cluster to provision a db. This reconciler basically connects to the db using the config information, creates a database/user/etc. and stores the information in a generated/dependent secret implemented as a dependent resource of type CRUDKubernetesDependentResource. This secret can now be used by e.g. a service.

what I have so far

So far, I have been able to watch secrets in the EdbConfig reconciler by using the primarytosecondary example: https://github.com/java-operator-sdk/java-operator-sdk/blob/main/operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/primarytosecondary/JobReconciler.java. It works: I can get the associated secret, and a reconciliation is triggered when the credentials secret changes.

I am now stuck on the ExtDb reconciler. I want to be able to get the EdbConfig (and its associated secret) in both the reconciler's reconcile method, and in the desired and match methods of the dependent secret.

I thought of using the same approach in the reconciler, but how do I get both the EdbConfig and its secret ? And how about the dependent secret ?

note: the fact that a reconciliation of Edboccurs if the referenced EdbConfig changes is a stretch objective. The minimum requirement being to be able to fetch the config in an effective way from the Edb reconciler and in its dependent secret.

[metacosm]

@metacosm thank you for the quick reply :)

Let me be more precise with my actual use case, which is to provision external databases.

description

I have an EdbConfig (external database config) with information on how to connect to a database. To avoid having passwords in clear in the spec, I have a credentialsSecretRef that points to a secret with username and password.

I have one reconciler for EdbConfig, which just checks for the validity of the information, and updates the status (isOk, lastError). This would better be done in an admission webhook, but we don't have support yet ;)

Have you taken a look at https://github.com/java-operator-sdk/admission-controller-framework? ;)

I thought of using the same approach in the reconciler, but how do I get both the EdbConfig and its secret ? And how about the dependent secret ?

Is your code accessible somewhere as it might be easier to help you if we can look directly at the code? At the very least, can you share how you configure each reconciler in code? The details are not important but being able to see how each reconciler is configured in particular with respect to how they relate to their dependent resources would be really helpful…

[derlin]

Unfortunately I can't make the code public, but here is an overview.

This is the ExtDbConfig reconciler, which uses the EventSourceInitializer to watch the credentials secret:

public class ExtDbConfigReconciler implements Reconciler<ExtDbConfig>, ErrorStatusHandler<ExtDbConfig>,
    EventSourceInitializer<ExtDbConfig> {

  private static final String CONFIG_CREDENTIALS_INDEX = "config-secret-index";

  @Override
  public UpdateControl<ExtDbConfig> reconcile(ExtDbConfig resource, Context context) {
    // ...
    var secret = ((Optional<Secret>) context.getSecondaryResource(Secret.class)).orElseThrow(() ->
        new ValidationException("Credentials secret not found:" + resource.getSpec().getCredentialsSecretRef()));
    // ...
  }

  @Override
  public Map<String, EventSource> prepareEventSources(EventSourceContext<ExtDbConfig> context) {

    context.getPrimaryCache().addIndexer(CONFIG_CREDENTIALS_INDEX, this::indexKey);

    InformerConfiguration<Secret> informerConfiguration =
        InformerConfiguration.from(Secret.class, context)
            .withSecondaryToPrimaryMapper(secret -> context.getPrimaryCache()
                .byIndex(CONFIG_CREDENTIALS_INDEX, indexKey(secret))
                .stream().map(ResourceID::fromResource).collect(Collectors.toSet()))
            .withPrimaryToSecondaryMapper(this::getSecretResourceId)
            .withNamespacesInheritedFromController(context)
            .build();

    return EventSourceInitializer
        .nameEventSources(new InformerEventSource<>(informerConfiguration, context));
  }

  private List<String> indexKey(ExtDbConfig extDbConfig) {
    var secretRef = extDbConfig.getSpec().getCredentialsSecretRef();
    return List.of(secretRef.getNamespace() + "#" + secretRef.getName());
  }

  private String indexKey(Secret secret) {
    return secret.getMetadata().getNamespace() + "#" + secret.getMetadata().getName();
  }

  private Set<ResourceID> getSecretResourceId(ExtDbConfig extDbConfig) {
    var secretRef = extDbConfig.getSpec().getCredentialsSecretRef();
    return Set.of(new ResourceID(secretRef.getName(), secretRef.getNamespace()));
  }
}

As for the ExtDb reconciler, I have a dependent secret:

@KubernetesDependent(labelSelector = LABEL_NAME + "=" + LABEL_VALUE)
@Slf4j
@ApplicationScoped
public class DependentSecret extends CRUDKubernetesDependentResource<Secret, ExtDb> implements SecondaryToPrimaryMapper<Secret> {

  public static final String LABEL_NAME = "...";
  public static final String LABEL_VALUE = "...";
  public static final String SECRET_SUFFIX = "-db";

  public DependentSecret() {
    super(Secret.class);
  }

  @Override
  protected Secret desired(ExtDb extDb, Context<ExtDb> context) {
    // 🛑 here I need access to the ExtDbConfig information !
  }

  @Override
  public Result<Secret> match(Secret actualResource, ExtDb extDb, Context<ExtDb> context) {
    // 🛑 here I need access to the ExtDbConfig information !
  }

  @Override
  public Set<ResourceID> toPrimaryResourceIDs(Secret dependentResource) {
    var name = dependentResource.getMetadata().getName();
    if (name.contains(SECRET_SUFFIX)) {
      return Set.of(new ResourceID(name.substring(0, name.length() - SECRET_SUFFIX.length()),
          dependentResource.getMetadata().getNamespace()));
    }
    return Set.of();
  }
}

And the reconciler:

@ControllerConfiguration(
    dependents = @Dependent(type = DependentSecret.class)
)
public class ExtDbReconciler implements Reconciler<ExtDb> {
  @Override
  public UpdateControl<ExtDb> reconcile(ExtDb extDb, Context<ExtDb> context) {
    // ...
    var secret = context.getSecondaryResource(Secret.class).orElseThrow();
    // 🛑 here I need access to the ExtDbConfig information ! 
    // ...
  }
}

[metacosm]

Thank you, will take a closer look but it seems at first glance (and that's what I was thinking reading your description) that your problem might be solved by adding an ExtDbConfig dependent to your ExtDbReconciler and possibly use the workflow feature that we introduced in 3.1 to prevent your ExtDb from being reconciled if the ExtDbConfig is not valid. If you do that, I think you should be able to access the ExtDbConfig secondary resource from the context object that is being passed to your SecretDependent. You could also probably get rid of the ExtDbConfigReconciler altogether.

[derlin]

wow, awesome !
And what about the credentials secret, how can I access it efficiently (caching would be great) ?
Can you give me some pointers on how to implement this workflow exactly ?

[metacosm]

The workflow feature is describe here: https://javaoperatorsdk.io/docs/workflows
Let me know if things are unclear so that we can improve the doc if needed… 😄
You can also take a look at https://github.com/java-operator-sdk/java-operator-sdk/tree/main/operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/workflowallfeature but it might be a little complex to understand what's going on just from that 😓
The Quarkus extension also has a sample that uses the workflow feature:
https://github.com/quarkiverse/quarkus-operator-sdk/blob/main/samples/exposedapp/src/main/java/io/halkyon/ExposedAppReconciler.java#L21-L26
In this particular case, the Ingress dependent won't be reconciled until the Service is ready and the ExposedApp reconciler won't be triggered until the Ingress is ready either…

[derlin]

Thank you for the pointers, I will have a look and get back to you !

[derlin]

Just one detail: the config will become a dependent resource with a condition, this I get. But what about the credentials secret ? This one is referenced by the config, and dependent resources cannot have other dependent resources, right ? So how do you handle the credentials secret ?

And how to avoid the Provide a SecondaryToPrimaryMapper to associate this resource with the primary resource when dealing with dependent resources ? (I feel I am back to square one ^^)

[metacosm]

Well, a dependent can depend on other dependents via the dependsOn relation that you can add to your dependent configuration… Of course, it's not the same as having multiple levels but you can somewhat emulate this organisation that way. I need to look at your code closer and play with it, it might end up that what you're trying to achieve isn't currently feasible with dependents… I don't know for sure at this point.

[derlin]

I am already having trouble, and this is actually what triggered my issue. When using a dependent resource, I always get this error:

Provide a SecondaryToPrimaryMapper to associate this resource with the primary resource

Which I cannot provide.

[derlin]

Any possibility to simply share the cache between the config reconciler and the db reconciler, without using dependent resources ?

[metacosm]

Caches are currently tied to event sources (this is actually something we're thinking about changing but we need to figure out how to design it cleanly) so there's a cache per resource type. Maybe we could expose them via the Context object?
Actually, what would your ideal solution to your problem look like, code-wise?

[derlin]

Tough question, I would need to think about that. But for now, having a shared cache would be enough. Thinking of it, I could implement my own cache, as I have a reconciler for the config that gets called whenever the config or associated secret changes. Furthermore, there would be only a handful: 5 max.

The only thing is, the config reconciler needs to run prior to the extdb reconciler. I am not sure how to handle this (an extdb runs before the config is processed). I could simply reschedule the reconciliation for later, hoping a small delay would be enough, or actually query the kubernetes api (it shouldn't happen often). But what if there the config ref is wrong ?
What do you think ?

[metacosm]

Another possible way to explore is to look at adding an indexer to the cache as done in https://github.com/java-operator-sdk/java-operator-sdk/blob/main/operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/primaryindexer/DependentPrimaryIndexerTestReconciler.java? I'll try to dig deeper into your use case tomorrow.