OWLS85461 add introspect version to server pod label

docs-source/content/userguide/managing-domains/domain-lifecycle/introspection.md

@@ -44,6 +44,21 @@ Set `introspectVersion` to a new value.
 
 As with `restartVersion`, the `introspectVersion` field has no required format; however, we recommend using a value likely to be unique such as a continually increasing number or a timestamp.
 
+Beginning with operator 3.1.0, if a domain resource's `spec.introspectVersion` is set, each of the domain's WebLogic Server pods will have a label with the key `weblogic.introspectVersion` to indicate the `introspectVersion` at which the pod is running.
+
+```
+Name:           domain1-admin-server
+Namespace:      domain1-ns
+Labels:         weblogic.createdByOperator=true
+                weblogic.domainName=domain1
+                weblogic.domainRestartVersion=abcdef
+                weblogic.domainUID=domain1
+                weblogic.introspectVersion=12345
+                weblogic.serverName=admin-server
+```
+
+When a domain's `spec.introspectVersion` is changed, the `weblogic.introspectVersion` label of each WebLogic Server pod is updated to the new `introspectVersion` value, either when the operator restarts the pod or when the operator determines that the pod does not need to be restarted.
+
 #### Failed introspection
 
 Sometimes the Kubernetes Job, named `DOMAIN_UID-introspector`, created for the introspection will fail.

docs-source/content/userguide/managing-domains/domain-resource.md

@@ -161,14 +161,14 @@ Elements related to domain [startup and shutdown]({{< relref "/userguide/managin
 * `replicas`: The default number of cluster member Managed Server instances to start for each WebLogic cluster in the domain configuration, unless `replicas` is specified for that cluster under the `clusters` field. For each cluster, the operator will sort cluster member Managed Server names from the WebLogic domain configuration by normalizing any numbers in the Managed Server name and then sorting alphabetically. This is done so that server names such as "managed-server10" come after "managed-server9". The operator will then start Managed Servers from the sorted list, up to the `replicas` count, unless specific Managed Servers are specified as starting in their entry under the `managedServers` field. In that case, the specified Managed Servers will be started and then additional cluster members will be started, up to the `replicas` count, by finding further cluster members in the sorted list that are not already started. If cluster members are started because of their entries under `managedServers`, then a cluster may have more cluster members running than its `replicas` count. Defaults to 0.
 * `maxClusterConcurrentStartup`: The maximum number of cluster member Managed Server instances that the operator will start in parallel for a given cluster, if `maxConcurrentStartup` is not specified for a specific cluster under the `clusters` field. A value of 0 means there is no configured limit. Defaults to 0.
 * `allowReplicasBelowMinDynClusterSize`: Whether to allow the number of running cluster member Managed Server instances to drop below the minimum dynamic cluster size configured in the WebLogic domain configuration, if this is not specified for a specific cluster under the `clusters` field. Defaults to true.
-* `introspectVersion`: Changes to this field cause the operator to repeat its introspection of the WebLogic domain configuration. Repeating introspection is required for the operator to recognize changes to the domain configuration, such as adding a new WebLogic cluster or Managed Server instance, to regenerate configuration overrides, or to regenerate the WebLogic domain home when the `domainHomeSourceType` is FromModel. Introspection occurs automatically, without requiring change to this field, when servers are first started or restarted after a full domain shut down. For the FromModel `domainHomeSourceType`, introspection also occurs when a running server must be restarted because of changes to any of the fields [listed here]({{< relref "/userguide/managing-domains/domain-lifecycle/startup#fields-that-cause-servers-to-be-restarted" >}}). See also `overridesConfigurationStrategy`.
+* `introspectVersion`: Changes to this field cause the operator to repeat its introspection of the WebLogic domain configuration (see [Initiating introspection]({{< relref "/userguide/managing-domains/domain-lifecycle/introspection/_index.md#initiating-introspection" >}})). Repeating introspection is required for the operator to recognize changes to the domain configuration, such as adding a new WebLogic cluster or Managed Server instance, to regenerate configuration overrides, or to regenerate the WebLogic domain home when the `domainHomeSourceType` is FromModel. Introspection occurs automatically, without requiring change to this field, when servers are first started or restarted after a full domain shut down. For the FromModel `domainHomeSourceType`, introspection also occurs when a running server must be restarted because of changes to any of the fields [listed here]({{< relref "/userguide/managing-domains/domain-lifecycle/startup#fields-that-cause-servers-to-be-restarted" >}}). See also `overrideDistributionStrategy`.
 
 Elements related to specifying and overriding WebLogic domain configuration:
 
 * These elements are under `configuration`.
 
   * `overridesConfigMap`: The name of the ConfigMap for WebLogic [configuration overrides]({{< relref "/userguide/managing-domains/configoverrides/_index.md" >}}). If this field is specified, then the value of `spec.configOverrides` is ignored.
-  * `overrideDistributionStrategy`: Determines how updated configuration overrides are distributed to already running WebLogic Server instances following introspection when the `domainHomeSourceType` is PersistentVolume or Image. Configuration overrides are generated during introspection from Secrets, the `overrideConfigMap` field, and WebLogic domain topology. Legal values are DYNAMIC, which means that the operator will distribute updated configuration overrides dynamically to running servers, and ON_RESTART, which means that servers will use updated configuration overrides only after the server's next restart. The selection of ON_RESTART will not cause servers to restart when there are updated configuration overrides available. See also `introspectVersion`. Defaults to DYNAMIC.
+  * `overrideDistributionStrategy`: Determines how updated configuration overrides are distributed to already running WebLogic Server instances following introspection when the `domainHomeSourceType` is PersistentVolume or Image. Configuration overrides are generated during introspection from Secrets, the `overridesConfigMap` field, and WebLogic domain topology. Legal values are DYNAMIC, which means that the operator will distribute updated configuration overrides dynamically to running servers, and ON_RESTART, which means that servers will use updated configuration overrides only after the server's next restart. The selection of ON_RESTART will not cause servers to restart when there are updated configuration overrides available. See also `introspectVersion`. Defaults to DYNAMIC.
   * `secrets`: A list of names of the Secrets for WebLogic [configuration overrides]({{< relref "/userguide/managing-domains/configoverrides/_index.md" >}}) or model. If this field is specified, then the value of `spec.configOverrideSecrets` is ignored.
   * `introspectorJobActiveDeadlineSeconds`: The introspector job timeout value in seconds. If this field is specified, then the operator's ConfigMap `data.introspectorJobActiveDeadlineSeconds` value is ignored. Defaults to 120 seconds.
 

operator/src/main/java/oracle/kubernetes/operator/DomainProcessorImpl.java

@@ -884,7 +884,7 @@ Step createDomainUpPlan(DomainPresenceInfo info) {
     Step domainUpStrategy =
         Step.chain(
             domainIntrospectionSteps(info),
-            DomainValidationSteps.createAfterIntrospectValidationSteps(info.getDomainUid()),
+            DomainValidationSteps.createAfterIntrospectValidationSteps(),
             new DomainStatusStep(info, null),
             bringAdminServerUp(info, delegate.getPodAwaiterStepFactory(info.getNamespace())),
             managedServerStrategy);

operator/src/main/java/oracle/kubernetes/operator/helpers/DomainValidationSteps.java

@@ -48,8 +48,8 @@ public static Step createAdditionalDomainValidationSteps(V1PodSpec podSpec) {
     return new DomainAdditionalValidationStep(podSpec);
   }
 
-  public static Step createAfterIntrospectValidationSteps(String domainUid) {
-    return new DomainAfterIntrospectValidationStep(domainUid);
+  public static Step createAfterIntrospectValidationSteps() {
+    return new DomainAfterIntrospectValidationStep();
   }
 
   private static Step createListSecretsStep(String domainNamespace) {
@@ -230,11 +230,6 @@ private boolean hasMatchingMetadata(V1ObjectMeta metadata, String name, String n
   }
 
   private static class DomainAfterIntrospectValidationStep extends Step {
-    private String domainUid;
-
-    public DomainAfterIntrospectValidationStep(String domainUid) {
-      this.domainUid = domainUid;
-    }
 
     @Override
     public NextAction apply(Packet packet) {

operator/src/main/java/oracle/kubernetes/operator/helpers/PodStepContext.java

@@ -7,6 +7,7 @@
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
@@ -58,6 +59,8 @@
 import oracle.kubernetes.weblogic.domain.model.Shutdown;
 import org.apache.commons.lang3.builder.EqualsBuilder;
 
+import static oracle.kubernetes.operator.LabelConstants.INTROSPECTION_STATE_LABEL;
+
 public abstract class PodStepContext extends BasePodStepContext {
 
   private static final LoggingFacade LOGGER = LoggingFactory.getLogger("Operator", "Operator");
@@ -86,6 +89,10 @@ public abstract class PodStepContext extends BasePodStepContext {
     scan = (WlsServerConfig) packet.get(ProcessingConstants.SERVER_SCAN);
   }
 
+  private static boolean isPatchableItem(Map.Entry<String, String> entry) {
+    return isCustomerItem(entry) || entry.getKey().equals(INTROSPECTION_STATE_LABEL);
+  }
+
   private static boolean isCustomerItem(Map.Entry<String, String> entry) {
     return !entry.getKey().startsWith("weblogic.");
   }
@@ -355,7 +362,7 @@ protected Step patchPod(V1Pod currentPod, Step next) {
     JsonPatchBuilder patchBuilder = Json.createPatchBuilder();
 
     KubernetesUtils.addPatches(
-        patchBuilder, "/metadata/labels/", getLabels(currentPod), getPodLabels());
+        patchBuilder, "/metadata/labels/", getLabels(currentPod), getNonHashedPodLabels());
     KubernetesUtils.addPatches(
         patchBuilder, "/metadata/annotations/", getAnnotations(currentPod), getPodAnnotations());
 
@@ -364,6 +371,15 @@ protected Step patchPod(V1Pod currentPod, Step next) {
             new V1Patch(patchBuilder.build().toString()), patchResponse(next));
   }
 
+  private Map<String, String> getNonHashedPodLabels() {
+    Map<String,String> result = new HashMap<>(getPodLabels());
+
+    Optional.ofNullable(getDomain().getSpec().getIntrospectVersion())
+        .ifPresent(version -> result.put(INTROSPECTION_STATE_LABEL, version));
+
+    return result;
+  }
+
   private Map<String, String> getLabels(V1Pod pod) {
     return Optional.ofNullable(pod.getMetadata()).map(V1ObjectMeta::getLabels).orElseGet(Collections::emptyMap);
   }
@@ -401,7 +417,7 @@ Step createCyclePodStep(Step next) {
   }
 
   private boolean mustPatchPod(V1Pod currentPod) {
-    return KubernetesUtils.isMissingValues(getLabels(currentPod), getPodLabels())
+    return KubernetesUtils.isMissingValues(getLabels(currentPod), getNonHashedPodLabels())
         || KubernetesUtils.isMissingValues(getAnnotations(currentPod), getPodAnnotations());
   }
 
@@ -455,11 +471,11 @@ protected Map<String, String> augmentSubVars(Map<String, String> vars) {
   V1Pod withNonHashedElements(V1Pod pod) {
     V1ObjectMeta metadata = Objects.requireNonNull(pod.getMetadata());
     // Adds labels and annotations to a pod, skipping any whose names begin with "weblogic."
-    getPodLabels().entrySet().stream()
-        .filter(PodStepContext::isCustomerItem)
+    getNonHashedPodLabels().entrySet().stream()
+        .filter(PodStepContext::isPatchableItem)
         .forEach(e -> metadata.putLabelsItem(e.getKey(), e.getValue()));
     getPodAnnotations().entrySet().stream()
-        .filter(PodStepContext::isCustomerItem)
+        .filter(PodStepContext::isPatchableItem)
         .forEach(e -> metadata.putAnnotationsItem(e.getKey(), e.getValue()));
 
     setTerminationGracePeriod(pod);
@@ -531,7 +547,7 @@ protected V1ObjectMeta createMetadata() {
         + getServerSpec().getDomainRestartVersion());
     LOGGER.finest("PodStepContext.createMetaData domainIntrospectVersion from spec "
         + getDomain().getIntrospectVersion());
-    
+
     metadata
         .putLabelsItem(LabelConstants.DOMAINUID_LABEL, getDomainUid())
         .putLabelsItem(LabelConstants.DOMAINNAME_LABEL, getDomainName())
@@ -829,7 +845,7 @@ public NextAction apply(Packet packet) {
       V1Pod currentPod = info.getServerPod(getServerName());
 
       // reset introspect failure job count - if any
-      
+
       Optional.ofNullable(packet.getSpi(DomainPresenceInfo.class))
           .map(DomainPresenceInfo::getDomain)
           .map(Domain::getStatus)
@@ -924,45 +940,48 @@ public NextAction onSuccess(Packet packet, CallResponse<V1Pod> callResponses) {
     }
   }
 
-  private class ReplacePodResponseStep extends BaseResponseStep {
+  private class ReplacePodResponseStep extends PatchPodResponseStep {
 
     ReplacePodResponseStep(Step next) {
       super(next);
     }
 
     @Override
-    public NextAction onSuccess(Packet packet, CallResponse<V1Pod> callResponse) {
-
-      V1Pod newPod = callResponse.getResult();
+    public void logPodChanged() {
       logPodReplaced();
-      if (newPod != null) {
-        setRecordedPod(newPod);
-      }
+    }
 
-      PodAwaiterStepFactory pw = packet.getSpi(PodAwaiterStepFactory.class);
-      return doNext(pw.waitForReady(newPod, getNext()), packet);
+    @Override
+    public NextAction onSuccess(Packet packet, CallResponse<V1Pod> callResponse) {
+      return doNext(
+          packet.getSpi(PodAwaiterStepFactory.class).waitForReady(processResponse(callResponse), getNext()),
+          packet);
     }
   }
 
   private class PatchPodResponseStep extends BaseResponseStep {
-    private final Step next;
 
     PatchPodResponseStep(Step next) {
       super(next);
-      this.next = next;
+    }
+
+    public void logPodChanged() {
+      logPodPatched();
     }
 
     @Override
     public NextAction onSuccess(Packet packet, CallResponse<V1Pod> callResponse) {
+      processResponse(callResponse);
+      return doNext(getNext(), packet);
+    }
 
+    protected V1Pod processResponse(CallResponse<V1Pod> callResponse) {
       V1Pod newPod = callResponse.getResult();
-      logPodPatched();
+      logPodChanged();
       if (newPod != null) {
         setRecordedPod(newPod);
       }
-
-      return doNext(next, packet);
+      return newPod;
     }
   }
-
 }

operator/src/main/java/oracle/kubernetes/operator/logging/MessageKeys.java

@@ -133,7 +133,7 @@ public class MessageKeys {
   public static final String INTROSPECTOR_JOB_FAILED = "WLSKO-0175";
   public static final String INTROSPECTOR_JOB_FAILED_DETAIL = "WLSKO-0176";
   public static final String INTROSPECTOR_POD_FAILED = "WLSKO-0177";
-  public static final String CRD_NOT_INSTALLED = "WLSKO-0180";
+  public static final String CRD_NOT_INSTALLED = "WLSKO-0178";
 
   // domain status messages
   public static final String DUPLICATE_SERVER_NAME_FOUND = "WLSDO-0001";

operator/src/main/resources/Operator.properties

@@ -187,7 +187,7 @@ WLSKO-0175=Job {0} in namespace {1} failed with status {2}. Check log messages \
   copied from the introspector pod {3} log for additional information.
 WLSKO-0176=Job {1} in namespace {0} failed, job details are {2}
 WLSKO-0177=Pod {0} in namespace {1} failed, the pod status is {2}
-WLSKO-0180=Operator cannot start. 'Dedicated' namespace strategy selected, but the CRD is not installed
+WLSKO-0178=Operator cannot start. Operator 'domainNamespaceSelectionStrategy' is set to 'Dedicated', but the Custom Resource Definition for ''domains.weblogic.oracle'' is not installed.
 
 # Domain status messages
 

operator/src/test/java/oracle/kubernetes/operator/DomainProcessorTest.java

@@ -446,6 +446,88 @@ private Stream<V1ConfigMap> getConfigMaps() {
     return testSupport.<V1ConfigMap>getResources(CONFIG_MAP).stream();
   }
 
+  @Test
+  public void afterInitialIntrospection_serverPodsHaveInitialIntrospectVersionLabel() throws Exception {
+    domainConfigurator.withIntrospectVersion(OLD_INTROSPECTION_STATE);
+    testSupport.doOnCreate(POD, p -> recordPodCreation((V1Pod) p));
+    domainConfigurator.configureCluster(CLUSTER).withReplicas(MIN_REPLICAS);
+    DomainPresenceInfo info = new DomainPresenceInfo(newDomain);
+    processor.createMakeRightOperation(info).withExplicitRecheck().execute();
+
+    List<V1Pod> runningPods = getRunningPods();
+    //one introspector pod, one admin server pod and two managed server pods
+    assertThat(runningPods.size(), equalTo(4));
+    for (V1Pod pod: runningPods) {
+      if (!pod.getMetadata().getName().contains(LegalNames.getIntrospectorJobNameSuffix())) {
+        assertThat(getServerPodIntrospectionVersion(pod), equalTo(OLD_INTROSPECTION_STATE));
+      }
+    }
+  }
+
+  @Test
+  public void afterIntrospection_serverPodsHaveUpToDateIntrospectVersionLabel() throws Exception {
+    establishPreviousIntrospection(null);
+
+    domainConfigurator.withIntrospectVersion(NEW_INTROSPECTION_STATE);
+    DomainPresenceInfo info = new DomainPresenceInfo(newDomain);
+    processor.createMakeRightOperation(info).withExplicitRecheck().execute();
+
+    List<V1Pod> runningPods = getRunningPods();
+    //one introspector pod, one admin server pod and two managed server pods
+    assertThat(runningPods.size(), equalTo(4));
+    for (V1Pod pod: runningPods) {
+      if (!pod.getMetadata().getName().contains(LegalNames.getIntrospectorJobNameSuffix())) {
+        assertThat(getServerPodIntrospectionVersion(pod), equalTo(NEW_INTROSPECTION_STATE));
+      }
+    }
+  }
+
+  @Test
+  public void afterScaleupClusterIntrospection_serverPodsHaveUpToDateIntrospectVersionLabel() throws Exception {
+    establishPreviousIntrospection(null);
+
+    domainConfigurator.configureCluster(CLUSTER).withReplicas(3);
+    domainConfigurator.withIntrospectVersion("after-scaleup");
+    DomainPresenceInfo info = new DomainPresenceInfo(newDomain);
+    processor.createMakeRightOperation(info).withExplicitRecheck().execute();
+
+    List<V1Pod> runningPods = getRunningPods();
+    //one introspector pod, one admin server pod and three managed server pods
+    assertThat(runningPods.size(), equalTo(5));
+    for (V1Pod pod: runningPods) {
+      if (!pod.getMetadata().getName().contains(LegalNames.getIntrospectorJobNameSuffix())) {
+        assertThat(getServerPodIntrospectionVersion(pod), equalTo("after-scaleup"));
+      }
+    }
+  }
+
+  @Test
+  public void afterScaledownClusterIntrospection_serverPodsHaveUpToDateIntrospectVersionLabel() throws Exception {
+    establishPreviousIntrospection(null);
+
+    domainConfigurator.configureCluster(CLUSTER).withReplicas(1);
+    domainConfigurator.withIntrospectVersion("after-scaledown");
+    DomainPresenceInfo info = new DomainPresenceInfo(newDomain);
+    processor.createMakeRightOperation(info).withExplicitRecheck().execute();
+
+    List<V1Pod> runningPods = getRunningPods();
+    //one introspector pod, one admin server pod and one managed server pod
+    assertThat(runningPods.size(), equalTo(3));
+    for (V1Pod pod: runningPods) {
+      if (!pod.getMetadata().getName().contains(LegalNames.getIntrospectorJobNameSuffix())) {
+        assertThat(getServerPodIntrospectionVersion(pod), equalTo("after-scaledown"));
+      }
+    }
+  }
+
+  private String getServerPodIntrospectionVersion(V1Pod pod) {
+    return Optional.ofNullable(pod)
+        .map(V1Pod::getMetadata)
+        .map(V1ObjectMeta::getLabels)
+        .map(m -> m.get(INTROSPECTION_STATE_LABEL))
+        .orElse(null);
+  }
+
   private boolean isIntrospectorMeta(@Nullable V1ObjectMeta meta) {
     return meta != null && NS.equals(meta.getNamespace()) && INTROSPECTOR_MAP_NAME.equals(meta.getName());
   }