Skip to content

Persistent state dependent resources #1543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 57 commits into from
Oct 26, 2022
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
3ff673f
feat: decouple event source from cache + list discriminator (#1378)
csviri Sep 28, 2022
dcd3718
feat: bulk dependent resources (#1448)
csviri Sep 29, 2022
fa7fa28
feat: optional eventsource on dependent resources (#1479)
csviri Oct 4, 2022
d8da98b
refactor: simplify handling of reused event sources (#1518)
csviri Oct 5, 2022
bf7a79d
refactor: isolate index handling to BulkDependentResource interface (…
metacosm Oct 5, 2022
3cd36d1
feat: key based bulk resource creation (#1521)
csviri Oct 6, 2022
0092f0a
fix: fixes after rebase
csviri Oct 6, 2022
7442012
refactor: improve bulk dependent resource api (#1525)
csviri Oct 7, 2022
92ec34e
refactor: isolate bulk dependent resource handling more (#1530)
metacosm Oct 12, 2022
bbf4d80
feat: external resource with generated id support (#1527)
csviri Oct 17, 2022
5a0cb3a
refactor: deleteBulkResource -> deleteTargetResource (#1544)
metacosm Oct 17, 2022
a456441
feat: external state management for dependent resources
csviri Oct 12, 2022
e1a89c7
wip
csviri Oct 13, 2022
0bc7f92
wip
csviri Oct 13, 2022
50ebb9b
wip
csviri Oct 13, 2022
beacbb6
wip
csviri Oct 14, 2022
4e45517
wip
csviri Oct 14, 2022
02c6b4a
Integration test
csviri Oct 17, 2022
aff1fcc
test fix
csviri Oct 18, 2022
a8d9c9f
bulk + external state
csviri Oct 18, 2022
7b9e6c7
IT refactor
csviri Oct 18, 2022
c270875
format
csviri Oct 18, 2022
80d9ea0
fix
csviri Oct 18, 2022
73d84d0
docs
csviri Oct 18, 2022
b1e6dcb
refactors
csviri Oct 18, 2022
a147d4f
fix: put back owner reference
csviri Oct 18, 2022
27cfa03
fix format
csviri Oct 18, 2022
cc45a36
feat: decouple event source from cache + list discriminator (#1378)
csviri Sep 28, 2022
98fbb4f
feat: bulk dependent resources (#1448)
csviri Sep 29, 2022
400e51a
feat: optional eventsource on dependent resources (#1479)
csviri Oct 4, 2022
1aee7c7
refactor: simplify handling of reused event sources (#1518)
csviri Oct 5, 2022
59e1663
refactor: isolate index handling to BulkDependentResource interface (…
metacosm Oct 5, 2022
aaea1a5
feat: key based bulk resource creation (#1521)
csviri Oct 6, 2022
919f61c
fix: fixes after rebase
csviri Oct 6, 2022
e0776a4
refactor: improve bulk dependent resource api (#1525)
csviri Oct 7, 2022
3c29576
refactor: isolate bulk dependent resource handling more (#1530)
metacosm Oct 12, 2022
e0f6110
feat: external resource with generated id support (#1527)
csviri Oct 17, 2022
15e30f1
refactor: deleteBulkResource -> deleteTargetResource (#1544)
metacosm Oct 17, 2022
6a132c8
Merge branch 'next' into persistent-state-dependent-resources
csviri Oct 19, 2022
522ffb7
fixes after merge
csviri Oct 19, 2022
6ad757d
fix: exception wording
metacosm Oct 20, 2022
4c5a52d
format
csviri Oct 21, 2022
9fa7236
feat: decouple event source from cache + list discriminator (#1378)
csviri Sep 28, 2022
37ca487
feat: bulk dependent resources (#1448)
csviri Sep 29, 2022
db4951d
feat: optional eventsource on dependent resources (#1479)
csviri Oct 4, 2022
a80b367
refactor: simplify handling of reused event sources (#1518)
csviri Oct 5, 2022
ce9fc7d
refactor: isolate index handling to BulkDependentResource interface (…
metacosm Oct 5, 2022
c61c6bc
feat: key based bulk resource creation (#1521)
csviri Oct 6, 2022
1757b29
fix: fixes after rebase
csviri Oct 6, 2022
30d084e
refactor: improve bulk dependent resource api (#1525)
csviri Oct 7, 2022
0a84412
refactor: isolate bulk dependent resource handling more (#1530)
metacosm Oct 12, 2022
ba69350
feat: external resource with generated id support (#1527)
csviri Oct 17, 2022
31502ce
refactor: deleteBulkResource -> deleteTargetResource (#1544)
metacosm Oct 17, 2022
6b286f8
Merge branch 'next' into persistent-state-dependent-resources
csviri Oct 24, 2022
e37736e
merge from next
csviri Oct 24, 2022
8c521b1
Merge branch 'next' into persistent-state-dependent-resources
csviri Oct 25, 2022
3dbdde6
rename
csviri Oct 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,8 @@
import io.javaoperatorsdk.operator.OperatorException;
import io.javaoperatorsdk.operator.ReconcilerUtils;
import io.javaoperatorsdk.operator.api.config.dependent.DependentResourceSpec;
import io.javaoperatorsdk.operator.api.reconciler.Constants;
import io.javaoperatorsdk.operator.api.reconciler.*;
import io.javaoperatorsdk.operator.api.reconciler.ControllerConfiguration;
import io.javaoperatorsdk.operator.api.reconciler.Reconciler;
import io.javaoperatorsdk.operator.api.reconciler.dependent.Dependent;
import io.javaoperatorsdk.operator.api.reconciler.dependent.DependentResource;
import io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependent;
Expand Down Expand Up @@ -221,12 +220,15 @@ public List<DependentResourceSpec> getDependentResources() {
throw new IllegalArgumentException(
"A DependentResource named '" + name + "' already exists: " + spec);
}
var eventSourceName = dependent.useEventSourceWithName();
eventSourceName = Constants.NO_VALUE_SET.equals(eventSourceName) ? null : eventSourceName;
final var context = Utils.contextFor(this, dependentType, null);
spec = new DependentResourceSpec(dependentType, config, name,
Set.of(dependent.dependsOn()),
Utils.instantiate(dependent.readyPostcondition(), Condition.class, context),
Utils.instantiate(dependent.reconcilePrecondition(), Condition.class, context),
Utils.instantiate(dependent.deletePostcondition(), Condition.class, context));
Utils.instantiate(dependent.deletePostcondition(), Condition.class, context),
eventSourceName);
specsMap.put(name, spec);
}

Expand Down Expand Up @@ -256,17 +258,16 @@ private Object createKubernetesResourceConfig(Class<? extends DependentResource>
OnUpdateFilter<? extends HasMetadata> onUpdateFilter = null;
OnDeleteFilter<? extends HasMetadata> onDeleteFilter = null;
GenericFilter<? extends HasMetadata> genericFilter = null;
ResourceDiscriminator<?, ? extends HasMetadata> resourceDiscriminator = null;
if (kubeDependent != null) {
if (!Arrays.equals(KubernetesDependent.DEFAULT_NAMESPACES,
kubeDependent.namespaces())) {
namespaces = Set.of(kubeDependent.namespaces());
configuredNS = true;
}

final var fromAnnotation = kubeDependent.labelSelector();
labelSelector = Constants.NO_VALUE_SET.equals(fromAnnotation) ? null : fromAnnotation;


final var context =
Utils.contextFor(this, dependentType, null);
onAddFilter = Utils.instantiate(kubeDependent.onAddFilter(), OnAddFilter.class, context);
Expand All @@ -276,10 +277,15 @@ private Object createKubernetesResourceConfig(Class<? extends DependentResource>
Utils.instantiate(kubeDependent.onDeleteFilter(), OnDeleteFilter.class, context);
genericFilter =
Utils.instantiate(kubeDependent.genericFilter(), GenericFilter.class, context);

resourceDiscriminator =
Utils.instantiate(kubeDependent.resourceDiscriminator(),
ResourceDiscriminator.class, context);
}

config =
new KubernetesDependentResourceConfig(namespaces, labelSelector, configuredNS, onAddFilter,
new KubernetesDependentResourceConfig(namespaces, labelSelector, configuredNS,
resourceDiscriminator, onAddFilter,
onUpdateFilter, onDeleteFilter, genericFilter);

return config;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ private void replaceConfig(String name, Object newConfig, DependentResourceSpec<
namedDependentResourceSpecs.put(name,
new DependentResourceSpec<>(current.getDependentResourceClass(), newConfig, name,
current.getDependsOn(), current.getReadyCondition(), current.getReconcileCondition(),
current.getDeletePostCondition()));
current.getDeletePostCondition(), current.getUseEventSourceWithName().orElse(null)));
}

@SuppressWarnings("unchecked")
Expand Down Expand Up @@ -220,7 +220,8 @@ public ControllerConfiguration<R> build() {
KubernetesDependentResourceConfig c) {
return new DependentResourceSpec(spec.getDependentResourceClass(),
c.setNamespaces(namespaces), name, spec.getDependsOn(), spec.getReadyCondition(),
spec.getReconcileCondition(), spec.getDeletePostCondition());
spec.getReconcileCondition(), spec.getDeletePostCondition(),
(String) spec.getUseEventSourceWithName().orElse(null));
}

public static <R extends HasMetadata> ControllerConfigurationOverrider<R> override(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,20 @@ public class DependentResourceSpec<T extends DependentResource<?, ?>, C> {

private final Condition<?, ?> deletePostCondition;

private final String useEventSourceWithName;

public DependentResourceSpec(Class<T> dependentResourceClass, C dependentResourceConfig,
String name, Set<String> dependsOn, Condition<?, ?> readyCondition,
Condition<?, ?> reconcileCondition, Condition<?, ?> deletePostCondition) {
Condition<?, ?> reconcileCondition, Condition<?, ?> deletePostCondition,
String useEventSourceWithName) {
this.dependentResourceClass = dependentResourceClass;
this.dependentResourceConfig = dependentResourceConfig;
this.name = name;
this.dependsOn = dependsOn;
this.readyCondition = readyCondition;
this.reconcileCondition = reconcileCondition;
this.deletePostCondition = deletePostCondition;
this.useEventSourceWithName = useEventSourceWithName;
}

public Class<T> getDependentResourceClass() {
Expand Down Expand Up @@ -89,4 +93,8 @@ public Condition getReconcileCondition() {
public Condition getDeletePostCondition() {
return deletePostCondition;
}

public Optional<String> getUseEventSourceWithName() {
return Optional.ofNullable(useEventSourceWithName);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ public <P extends HasMetadata> InformerConfigurationBuilder<R> withNamespacesInh
}

/**
* Whether or not the associated informer should track changes made to the parent
* Whether the associated informer should track changes made to the parent
* {@link io.javaoperatorsdk.operator.processing.Controller}'s namespaces configuration.
*
* @param followChanges {@code true} to reconfigure the associated informer when the parent
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,27 @@
import io.fabric8.kubernetes.api.model.HasMetadata;
import io.javaoperatorsdk.operator.api.config.ControllerConfiguration;
import io.javaoperatorsdk.operator.api.reconciler.dependent.managed.ManagedDependentResourceContext;
import io.javaoperatorsdk.operator.processing.event.EventSourceRetriever;

public interface Context<P extends HasMetadata> {

Optional<RetryInfo> getRetryInfo();

default <T> Optional<T> getSecondaryResource(Class<T> expectedType) {
return getSecondaryResource(expectedType, null);
default <R> Optional<R> getSecondaryResource(Class<R> expectedType) {
return getSecondaryResource(expectedType, (String) null);
}

<T> Set<T> getSecondaryResources(Class<T> expectedType);
<R> Set<R> getSecondaryResources(Class<R> expectedType);

<T> Optional<T> getSecondaryResource(Class<T> expectedType, String eventSourceName);
@Deprecated(forRemoval = true)
<R> Optional<R> getSecondaryResource(Class<R> expectedType, String eventSourceName);

<R> Optional<R> getSecondaryResource(Class<R> expectedType,
ResourceDiscriminator<R, P> discriminator);

ControllerConfiguration<P> getControllerConfiguration();

ManagedDependentResourceContext managedDependentResourceContext();

EventSourceRetriever<P> eventSourceRetriever();
}
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import io.javaoperatorsdk.operator.api.reconciler.dependent.managed.DefaultManagedDependentResourceContext;
import io.javaoperatorsdk.operator.api.reconciler.dependent.managed.ManagedDependentResourceContext;
import io.javaoperatorsdk.operator.processing.Controller;
import io.javaoperatorsdk.operator.processing.event.EventSourceRetriever;

public class DefaultContext<P extends HasMetadata> implements Context<P> {

Expand Down Expand Up @@ -47,6 +48,12 @@ public <T> Optional<T> getSecondaryResource(Class<T> expectedType, String eventS
.getSecondaryResource(primaryResource);
}

@Override
public <R> Optional<R> getSecondaryResource(Class<R> expectedType,
ResourceDiscriminator<R, P> discriminator) {
return discriminator.distinguish(expectedType, primaryResource, this);
}

@Override
public ControllerConfiguration<P> getControllerConfiguration() {
return controllerConfiguration;
Expand All @@ -57,6 +64,11 @@ public ManagedDependentResourceContext managedDependentResourceContext() {
return defaultManagedDependentResourceContext;
}

@Override
public EventSourceRetriever<P> eventSourceRetriever() {
return controller.getEventSourceManager();
}

public DefaultContext<P> setRetryInfo(RetryInfo retryInfo) {
this.retryInfo = retryInfo;
return this;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
package io.javaoperatorsdk.operator.api.reconciler;

import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;

import io.fabric8.kubernetes.api.model.HasMetadata;
import io.javaoperatorsdk.operator.api.reconciler.dependent.DependentResource;
import io.javaoperatorsdk.operator.processing.event.source.EventSource;
import io.javaoperatorsdk.operator.processing.event.source.ResourceEventSource;

/**
* An interface that a {@link Reconciler} can implement to have the SDK register the provided
Expand Down Expand Up @@ -39,6 +43,22 @@ static Map<String, EventSource> nameEventSources(EventSource... eventSources) {
return eventSourceMap;
}

@SuppressWarnings("unchecked,rawtypes")
static <K extends HasMetadata> Map<String, EventSource> nameEventSourcesFromDependentResource(
EventSourceContext<K> context, DependentResource... dependentResources) {

if (dependentResources != null) {
Map<String, EventSource> eventSourceMap = new HashMap<>(dependentResources.length);
for (DependentResource dependentResource : dependentResources) {
Optional<ResourceEventSource> es = dependentResource.eventSource(context);
es.ifPresent(e -> eventSourceMap.put(generateNameFor(e), e));
}
return eventSourceMap;
} else {
return Collections.emptyMap();
}
}

/**
* This is for the use case when the event sources are not access explicitly by name in the
* reconciler.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
package io.javaoperatorsdk.operator.api.reconciler;

import java.util.Optional;

import io.fabric8.kubernetes.api.model.HasMetadata;

public interface ResourceDiscriminator<R, P extends HasMetadata> {

Optional<R> distinguish(Class<R> resource, P primary, Context<P> context);

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package io.javaoperatorsdk.operator.api.reconciler;

import java.util.Optional;
import java.util.function.Function;

import io.fabric8.kubernetes.api.model.HasMetadata;
import io.javaoperatorsdk.operator.processing.event.ResourceID;

public class ResourceIDMatcherDiscriminator<R extends HasMetadata, P extends HasMetadata>
implements ResourceDiscriminator<R, P> {

private final Function<P, ResourceID> mapper;

public ResourceIDMatcherDiscriminator(Function<P, ResourceID> mapper) {
this.mapper = mapper;
}

@Override
public Optional<R> distinguish(Class<R> resource, P primary, Context<P> context) {
var resourceID = mapper.apply(primary);
return context.getSecondaryResources(resource).stream()
.filter(resourceID::isSameResource)
.findFirst();
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package io.javaoperatorsdk.operator.api.reconciler.dependent;

import io.javaoperatorsdk.operator.api.reconciler.Constants;
import io.javaoperatorsdk.operator.processing.dependent.workflow.Condition;

import static io.javaoperatorsdk.operator.api.reconciler.Constants.NO_VALUE_SET;
Expand Down Expand Up @@ -57,4 +58,13 @@
* one can be
*/
String[] dependsOn() default {};

/**
* Setting here a name of the event source means that dependent resource will use an event source
* registered with that name. So won't create one. This is helpful if more dependent resources
* created for the same type, and want to share a common event source.
*
* @return event source name (if any) provided by the dependent resource should be used.
*/
String useEventSourceWithName() default NO_VALUE_SET;
}
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
package io.javaoperatorsdk.operator.api.reconciler.dependent;

import java.util.Optional;

import io.fabric8.kubernetes.api.model.HasMetadata;
import io.javaoperatorsdk.operator.api.reconciler.Context;
import io.javaoperatorsdk.operator.processing.ResourceOwner;
import io.javaoperatorsdk.operator.api.reconciler.EventSourceContext;
import io.javaoperatorsdk.operator.processing.event.source.ResourceEventSource;

/**
* An interface to implement and provide dependent resource support.
*
* @param <R> the dependent resource type
* @param <P> the associated primary resource type
*/
public interface DependentResource<R, P extends HasMetadata> extends ResourceOwner<R, P> {
public interface DependentResource<R, P extends HasMetadata> {

/**
* Reconciles the dependent resource given the desired primary state
Expand All @@ -21,6 +24,35 @@ public interface DependentResource<R, P extends HasMetadata> extends ResourceOwn
*/
ReconcileResult<R> reconcile(P primary, Context<P> context);

/**
* Retrieves the resource type associated with this DependentResource
*
* @return the resource type associated with this DependentResource
*/
Class<R> resourceType();

/**
* Dependent resources are designed to by default provide event sources. There are cases where it
* might not:
* <ul>
* <li>If an event source is shared between multiple dependent resources. In this case only one or
* none of the dependent resources sharing the event source should provide one.</li>
* <li>Some special implementation of an event source. That just execute some action might not
* provide one.</li>
* </ul>
*
* @param eventSourceContext context of event source initialization
* @return an optional event source
*/
default Optional<ResourceEventSource<R, P>> eventSource(
EventSourceContext<P> eventSourceContext) {
return Optional.empty();
}

default Optional<R> getSecondaryResource(P primary, Context<P> context) {
return Optional.empty();
}

/**
* Computes a default name for the specified DependentResource class
*
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package io.javaoperatorsdk.operator.api.reconciler.dependent;

import io.javaoperatorsdk.operator.OperatorException;

public class EventSourceNotFoundException extends OperatorException {

private String eventSourceName;

public EventSourceNotFoundException(String eventSourceName) {
this.eventSourceName = eventSourceName;
}

public String getEventSourceName() {
return eventSourceName;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@
import io.javaoperatorsdk.operator.api.reconciler.EventSourceContext;
import io.javaoperatorsdk.operator.processing.event.source.EventSource;

/**
* @deprecated now event source related methods are directly on {@link DependentResource}
* @param <P> primary resource
*/
@Deprecated(forRemoval = true)
public interface EventSourceProvider<P extends HasMetadata> {
/**
* @param context - event source context where the event source is initialized
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package io.javaoperatorsdk.operator.api.reconciler.dependent;

import io.fabric8.kubernetes.api.model.HasMetadata;
import io.javaoperatorsdk.operator.processing.event.EventSourceRetriever;

public interface EventSourceReferencer<P extends HasMetadata> {

default void useEventSourceWithName(String name) {}

/**
* Throws {@link EventSourceNotFoundException} an exception if the target event source to use is
* not found.
*/
void resolveEventSource(EventSourceRetriever<P> eventSourceRetriever)
throws EventSourceNotFoundException;

}
Loading