OWLS-92894: Implement Istio support based on Istio version #2582

lennyphan · 2021-10-21T03:40:36Z

Due to networking changes starting in Istio v1.10, as described in https://istio.io/latest/blog/2021/upcoming-networking-changes/, the operator needs to generate the appropriate WebLogic NAP's based on a version attribute in the Istio configuration of the domain custom resource:

e.g.

Istio

configuration:
istio:
enabled: true
readinessPort: 8888
replicationChannelPort: 4564
localhostBindingsEnabled: false

This PR implements the following for all domain types:

Adds Istio configuration attributes 'replicationChannelPort' and 'version' to the domain custom resource.
For Istio versions earlier than 1.10, the introspector continues to generate NAP's with a listening address of localhost (127.0.0.1) for all protocols.
For Istio version 1.10 and later, the introspector only generates an additional readiness probe bound to the server pod's IP.
To enable WebLogic replication traffic with Istio, the introspector will create a NAP specifically for replication and configure the 'replication-channel' attribute of the all the clusters in the domain.
Update the Istio page of the User Guide.
Switched from using version attribute to localhostBindingsEnabled boolean attribute.
Added operator helm chart value istioLocalhostBindingsEnabled.

…t namespace

jshum2479 · 2021-10-21T13:16:05Z

documentation/domains/Domain.json

@@ -569,10 +569,18 @@
    "Istio": {
      "type": "object",
      "properties": {
+        "replicationChannelPort": {
+          "description": "The operator will create a WebLogic network access point with this port that will then be exposed from the container running the WebLogic Server instance. The WebLogic Replication Service will use this network access point for all replication traffic. Defaults to 4564.",


The code use 4358 as default

Changed to default to port 4564 and searched to verify it's consistent everywhere in the implementation change.

jshum2479 · 2021-10-21T13:21:03Z

operator/src/main/java/oracle/kubernetes/weblogic/domain/model/IntrospectorJobEnvVars.java

@@ -149,7 +159,7 @@ static boolean isReserved(String name) {
  private static final List<String> RESERVED_NAMES = Arrays.asList(
      NAMESPACE, INTROSPECT_HOME, CREDENTIALS_SECRET_NAME, OPSS_KEY_SECRET_NAME, OPSS_WALLETFILE_SECRET_NAME,
      RUNTIME_ENCRYPTION_SECRET_NAME, WDT_DOMAIN_TYPE, DOMAIN_SOURCE_TYPE, ISTIO_ENABLED, ISTIO_READINESS_PORT,
-      ISTIO_POD_NAMESPACE, WDT_MODEL_HOME, MII_USE_ONLINE_UPDATE,
+      ISTIO_REPLICATION_PORT, ISTIO_VERSION, ISTIO_POD_NAMESPACE, WDT_MODEL_HOME, MII_USE_ONLINE_UPDATE,


This will cause unexpected introspection job runs when upgrading the operator. The introspect job checks for environment variable hashed value, if they are different then it will run the introspect job, which eventually will triggers a domain roll because we added a replication channel now.

Fixed as not to cause unnecessary introspection runs unless explicitly changed from the default in the Istio configuration of the custom domain resource.

jshum2479 · 2021-10-21T13:24:57Z

operator/src/main/resources/scripts/introspectDomain.py

@@ -1317,16 +1197,17 @@ def customizeServerTemplate(self, template):
    self.writeListenAddress(template.getListenAddress(),listen_address)
    self.customizeNetworkAccessPoints(template,listen_address)
    self.customizeManagedIstioNetworkAccessPoint(listen_address, template)
-    if (self.getCoherenceClusterSystemResourceOrNone(template) is not None):
+    self.customizeIstioReplicationChannel(template, listen_address,
+                                          self.env.getEnvOrDef("ISTIO_REPLICATION_PORT", 4564),


Please check for the consistency of the default replication channel port value

Changed so that we consistently use port 4564 as the default.

tbarnes-us · 2021-10-21T14:20:51Z

documentation/staging/content/userguide/istio/istio.md

+it is the port used in the network access point by the WebLogic Replication Service 
+for all replication traffic. The `version` is optional and represents the Istio version
+run by the operator and WebLogic domains managed by the operator.  If not provided, the
+operator will assume it's supporting Istio version 1.9 or earlier.


I have multiple questions:

What is the WebLogic Replication Service? Why is it the only kind of traffic that needs special treatment?
Can you add a reference to the replication discussion below?
Why "If [replicationChannelPort] not provided, the operator will assume it's supporting Istio version 1.9 or earlier."? So what if it's "not provided" AND version 1.10 is supplied?
Why is the version important? How do you get the Istio version from Istio? What happens if 'version:' doesn't match the actual version?
What happens if using 1.10 and version is configured to 1.9?
What happens if using 1.9 and version is configured to 1.10?

(Maybe we don't need a version, just a boolean?)

The limitations section at beginning mentions testing up to 1.7 only.

Should "prerequisites.md" mention Istio?

The 'port forwarding' documentation mentions that the Istio option enables localhost. If that no longer occurs, then the port forwarding doc should be amended...

To be clear, I think we know the answers to most of these questions - I'm asking them as prompts for updating the documentation in case a customer has the same questions...

Switched to localhostBindingsEnabled attribute (instead of specifying version) to simplify configuration and updated documentation. 'port-forwarding' documentation has been updated with information about using domain.spec.adminServer.adminChannelPortForwardingEnableddomain resource attribute for both Istio v1.10 and later and if not istio enabled, the use cases are the same (we need to generate localhost bindings for port-forwarding)

tbarnes-us · 2021-10-21T14:42:01Z

documentation/domains/Domain.md

@@ -224,6 +224,8 @@ The current status of the operation of the WebLogic domain. Updated automaticall
 | --- | --- | --- |
 | `enabled` | Boolean | True, if this domain is deployed under an Istio service mesh. Defaults to true when the `istio` field is specified. |
 | `readinessPort` | number | The operator will create a WebLogic network access point with this port that will then be exposed from the container running the WebLogic Server instance. The readiness probe will use this network access point to verify that the server instance is ready for application traffic. Defaults to 8888. |
+| `replicationChannelPort` | number | The operator will create a WebLogic network access point with this port that will then be exposed from the container running the WebLogic Server instance. The WebLogic Replication Service will use this network access point for all replication traffic. Defaults to 4564. |


What if the replication port has already been defined by the user in the WL configuration itself?

Tom and Lenny had updated the documentation about ignoring the replication channel configuration if already defined by the user.

tbarnes-us · 2021-10-21T14:43:29Z

documentation/domains/Domain.md

@@ -224,6 +224,8 @@ The current status of the operation of the WebLogic domain. Updated automaticall
 | --- | --- | --- |
 | `enabled` | Boolean | True, if this domain is deployed under an Istio service mesh. Defaults to true when the `istio` field is specified. |
 | `readinessPort` | number | The operator will create a WebLogic network access point with this port that will then be exposed from the container running the WebLogic Server instance. The readiness probe will use this network access point to verify that the server instance is ready for application traffic. Defaults to 8888. |
+| `replicationChannelPort` | number | The operator will create a WebLogic network access point with this port that will then be exposed from the container running the WebLogic Server instance. The WebLogic Replication Service will use this network access point for all replication traffic. Defaults to 4564. |
+| `version` | string | Starting with Istio 1.10, the networking behavior was changed in that the proxy no longer redirects the traffic to the localhost interface, but instead forwards (or passes it through) it to the network interface associated with the pod's IP. True, if Istio v1.10 or higher is installed. Defaults to false |


The istio.md states this expects a numeric version. Here it says 'true' 'false'.

Switched to 'localhostBindingsEnabled' true/false attribute and updated the documentation

tbarnes-us · 2021-10-21T14:44:24Z

documentation/staging/content/userguide/istio/istio.md

+    ```
+    <cluster>
+      <name>cluster-1</name>
+      <replication-channe>istiorepl</replication-channel>


channe --> channel

jshum2479 · 2021-10-21T19:05:45Z

operator/src/main/resources/scripts/model_wdt_mii_filter.py

+    return
+
+  # verify if a replication channel is defined for the cluster
+  if 'ReplicationChannel' not in cluster:


Does this depends on the ReplicatonChannel already injected in the cluster before, assuming user didn't define any ReplicationChannel in cluster of their model?

Yes, we should have already configured the cluster with a replication channel named 'istiorepl'. If a replication channel is already defined that is not named 'istiorepl' or there is no replication channel configured at all then we shouldn't generate the NAP named 'istiorepl'. The only time we should generated the 'istiorepl' NAP is if the cluster has been configured with replication channel named 'istiorepl'.

I've found some issues with the logic in introspectionDomain.py for injecting replication channel that isn't the same as in MII so I have to fix it.

…ore generating NAPs for each server for MII

…sEnabled

…ute of Istio configuration of domain custom resource

tbarnes-us · 2021-10-25T15:51:47Z

documentation/staging/content/userguide/managing-domains/accessing-the-domain/port-forward.md

@@ -64,7 +64,7 @@ This behavior depends on your version and domain resource configuration:
  run the `kubectl explain domain.spec.adminServer.adminChannelPortForwardingEnabled` command
  or see the domain resource [schema](https://github.com/oracle/weblogic-kubernetes-operator/blob/main/documentation/domains/Domain.md).

-* For Istio-enabled domains, the operator already adds a
+* For Istio-enabled domains running Istio version 1.9 and earlier, the operator already adds a


running Istio versions prior to 1.10

tbarnes-us · 2021-10-26T14:54:04Z

I have doc feedback -- let's meet to discuss.

pull latest from main

merge latest from OWLS-92894

Fetch TOC

rosemarymarano

I think that the best way for me to review this doc would be to take it over after you've added all your input (and it is technically reviewed and approved). Does that work for you?

rosemarymarano · 2021-10-26T20:49:53Z

documentation/staging/content/userguide/istio/istio.md

-  multicluster installations of Istio.
+* The operator supports Istio versions 1.7 and higher,
+  and has been tested with single and multicluster
+  Istio installations from 1.7.3 up to 1.11.x


from 1.7.3 up to 1.11.x. (period at the end of the sentence)

rosemarymarano · 2021-10-26T20:50:15Z

documentation/staging/content/userguide/istio/istio.md


-  **NOTE**: The WebLogic Kubernetes Operator creates Kubernetes headless Services for the domain; Istio 1.6.x does not work with headless Services. See [Headless service broken in 1.6.0](https://github.com/istio/istio/issues/24082). Instead, use Istio version 1.7 and higher.
+* You cannot setup a NodePort using `domain.spec.adminServer.adminService.channels`


setup -> set up

rosemarymarano · 2021-10-26T20:52:11Z

documentation/staging/content/userguide/istio/istio.md

+Istio support requires labeling the operator namespace and 
+your domain namespaces to enable Istio automatic
+sidecar injection, plus modifying your domain resource
+configuration. In this section we discuss 


In this section -> In this section, (comma)
we discuss -> we describe

rosemarymarano · 2021-10-26T20:52:36Z

documentation/staging/content/userguide/istio/istio.md

+sidecar injection, plus modifying your domain resource
+configuration. In this section we discuss 
+the steps for the operator namespace; we will discuss
+the steps for domain in later sections.


we will discuss -> we will describe
the steps for domain -> the steps for the domain

rosemarymarano · 2021-10-26T20:57:36Z

documentation/staging/content/userguide/istio/istio.md

-`weblogic-server` container in the managed server pods will fail to reach `ready` state due to failure
-of the readiness probe.  For example, if the `localhostBindingsEnabled` is set `true` when running
-Istio versions 1.10 and later, you will see output like this:
+Let's discuss each `spec.configuration.istio` attribute:


Let's discuss each -> See the following description of each

rosemarymarano · 2021-10-26T20:59:33Z

documentation/staging/content/userguide/istio/istio.md

+  and set `enabled: true` as shown.
+* `readinessPort`: This attribute is optional
+   and defaults to `8888` if not provided; it is used for a readiness health check.
+* `replicationChannelPort`: This attribute is optional and defaults to `4564` if not provided;


provided; -> provided. (period) If you want to use a semi-colon, then "The operator" -> "the operator"

rosemarymarano · 2021-10-26T20:59:53Z

documentation/staging/content/userguide/istio/istio.md

+   and defaults to `8888` if not provided; it is used for a readiness health check.
+* `replicationChannelPort`: This attribute is optional and defaults to `4564` if not provided;
+  The operator will create a `T3` protocol WebLogic network access point on each WebLogic 
+  server that is part of a cluster with this port to handle EJB and servlet session state 


server -> Server

rosemarymarano · 2021-10-26T21:01:51Z

documentation/staging/content/userguide/istio/istio.md

+  This setting is ignored for clusters 
+  where the WebLogic cluster configuration already defines a `replication-channel` attribute. 
+* `localhostBindingsEnabled`:
+   This setting was added in operator version 3.3.3,


I suggest using sub-bullets to separate each of these version-related sub-items.

rosemarymarano · 2021-10-26T21:02:30Z

documentation/staging/content/userguide/istio/istio.md

+   |----|----|---|---|
+   |Pre-1.10|3.x|`true`|Supported. Note that `true` is the default in 3.x.|
+   |Pre-1.10|3.x|`false`|Not supported.|
+   |Pre-1.10|4.x|N/A|Not supported. 4.x only supports Istio 1.10 and later.|


only supports -> supports only

lennyphan · 2021-10-26T21:14:31Z

@rosemarymarano Yes, let me make the current set of changes you've suggested and then you can review after it's technically reviewed.

Merge Tom's latest doc edits

…ce over helm value

anpanigr · 2021-10-29T16:33:58Z

integration-tests/src/test/java/oracle/weblogic/kubernetes/ItIstioDomainInPV.java

@@ -427,4 +428,14 @@ private void createDomainOnPVUsingWlst(Path wlstScriptFile, Path domainPropertie

  }

+  private AdminServer createAdminServer() {


Can we move this method to common IstioUtils class so that it can be manged in one calss instead of repeating in each Istio It class.

anpanigr · 2021-10-29T16:39:04Z

integration-tests/src/test/java/oracle/weblogic/kubernetes/utils/SemanticVersion.java

@@ -0,0 +1,134 @@
+// Copyright (c) 2020, 2021, Oracle and/or its affiliates.


rjeberhard · 2021-10-29T17:43:06Z

documentation/domains/Domain.json

+          "type": "number"
+        },
+        "localhostBindingsEnabled": {
+          "description": "This setting was added in operator version 3.3.3, defaults to the `istioLocalhostBindingsEnabled` [Operator Helm value]({{\u003c relref \"/userguide/managing-operators/using-helm.md\" \u003e}}) which in turn defaults to `true`, and is ignored in version 4.0 and later. In version 3.x, when `true`, the operator creates a WebLogic network access point with a `localhost` binding for each existing channel and protocol.  In version 3.x, use `true` for Istio versions prior to 1.10 and set to `false` for version 1.10 and later.  Version 4.0 and later requires Istio 1.10 and later, will not create localhost bindings, and ignores this attribute.",


You can only use relref in the main documentation markdown. Here we have to use a fully-qualified URL.

@rosemarymarano, we will need your help wording this. :)

We are adding a field that lets the customer configure whether the operator should behave in a certain way with the implication that one choice is appropriate for Istio versions 1.10 and greater and the other choice for Istio versions prior to 1.10. For the 3.x versions of the operator we will support Istio 1.7.3 through 1.10. For the (future) 4.0 release of the operator we will only support Istio 1.10 and greater. We need to release an operator 3.3.3 with these changes as soon as possible, but the operator 4.0 release is still a future release. Therefore, this wording is a little off since it fails to capture that 4.0 isn't available yet.

The way I see it, you have two choices: one is remove the references to 4.0 and just describe the current situation with operator 3.x and the second is to keep it as written but add the caveat that version 4.0 is not yet available. With either choice, you will have to go back and edit the information when 4.0 becomes available. From a technical writing "best practices " POV, you should remove references to 4.0. We are instructed to almost never refer to releases/versions that exist only in the future.

Instead of removing references to 4.0 or adding a qualifier sentence, maybe we tag all mentions of 4.0 with the word unreleased - e.g. something change "4.0" to something like "4.0 (unreleased)" or "4.beta"?

4.0 is technically available in that the full source is public in "main".

This is going to keep happening, and it's hard to juggle fine distinctions between source and documentation, and even harder to maintain multiple versions of the same doc.

rjeberhard · 2021-10-29T17:50:54Z

documentation/staging/content/userguide/istio/istio.md

+   |Pre-1.10|4.x|N/A|Not supported. 4.x supports only Istio 1.10 and later.|
+   |1.10 and later|3.x|`true`|Not supported.|
+   |1.10 and later|3.x|`false`|Supported.|
+   |1.10 and later|4.x|N/A|Supported. Operator will not create localhost bindings because Istio 1.10 does not need them.|


Similarly, we need to work with @rosemarymarano to figure out how to properly reference 4.0 as a future release.

I'll wait until you have a final version for me to review.

rjeberhard · 2021-10-29T17:52:18Z

integration-tests/src/test/java/oracle/weblogic/domain/Istio.java

+  @ApiModelProperty(
+      "This setting was added in operator version 3.3.3, "
+          + "defaults to the `istioLocalhostBindingsEnabled` "
+          + "[Operator Helm value]({{< relref \"/userguide/managing-operators/using-helm.md\" >}}) "


Must use fully-qualifed URL here.

Or simply change [Operator Helm value]({{< relref \"/userguide/managing-operators/using-helm.md\" >}}) to Operator Helm value?

rjeberhard · 2021-10-29T17:54:55Z

operator/src/main/java/oracle/kubernetes/weblogic/domain/model/Istio.java

+  @Description(
+      "This setting was added in operator version 3.3.3, "
+          + "defaults to the `istioLocalhostBindingsEnabled` "
+          + "[Operator Helm value]({{< relref \"/userguide/managing-operators/using-helm.md\" >}}) "


Same... Fix here as well when we work with Rosemary to wordsmith.

rjeberhard · 2021-10-29T18:45:02Z

Per our team conversation, I'm going to merge this PR as is. We will then update the documentation first as part of a backport to release/3.3 as part of one PR and make follow-on documentation changes to main as part of a second new PR.

lennyphan added 11 commits October 14, 2021 05:59

Initial changes

63780e8

Fix isVersionEarlierThan argument count

023d0b7

Update Istio Domain in PV test to use Istio version

8dfcf1a

Fix integration istio tests to use ISTIO_VERSION env to set version

033e924

Add replication channel injection when Istio configured

df782f2

Change listen address for Istio NAP to bind to server pod name withou…

b9d6116

…t namespace

Sync NAP configuration for DII and Domain-On-PV with MII implementation

f8fb176

minor changes to replication channel generation

c3653fc

Resolve conflict

5f3c831

update to weblogic.sha256

9fe32c9

Update istio documentation

324e733

lennyphan requested review from jshum2479, rjeberhard and tbarnes-us October 21, 2021 03:40

jshum2479 reviewed Oct 21, 2021

View reviewed changes

tbarnes-us reviewed Oct 21, 2021

View reviewed changes

lennyphan added 2 commits October 21, 2021 16:19

Fix PR CodeQL analysis error

3de9d88

First code changes based on code review

318b7df

jshum2479 reviewed Oct 21, 2021

View reviewed changes

lennyphan and others added 5 commits October 22, 2021 13:44

initialize replication-channel attribute of cluster configuration bef…

1c1c61b

…ore generating NAPs for each server for MII

Doc updates #1

ebd0fa8

initial new wording for attributes (Lenny and Tom)

69552da

Change Istio configuration attribute from version to localhostBinding…

f35054f

…sEnabled

Update integration tests to configure localhostBindingsEnabled attrib…

36be607

…ute of Istio configuration of domain custom resource

tbarnes-us reviewed Oct 25, 2021

View reviewed changes

Tom Barnes and others added 6 commits October 26, 2021 20:39

more istio doc edits

dda4880

Merge branch 'main' into OWLS-92894

d48cfe1

pull latest from main

Merge remote-tracking branch 'origin/OWLS-92894' into OWLS-92894

f86a61d

merge latest from OWLS-92894

even more istio doc edits

9d9c267

Update description of new Istio configuration attributes

711f680

Merge remote-tracking branch 'origin/OWLS-92894' into OWLS-92894

449e81d

Fetch TOC

rosemarymarano reviewed Oct 26, 2021

View reviewed changes

lennyphan added 2 commits October 26, 2021 21:40

Update with Rosemary's comments part 1

e499039

generated artifacts

1c66f24

lennyphan requested review from jshum2479, rosemarymarano and rjeberhard October 26, 2021 22:53

Tom Barnes and others added 5 commits October 27, 2021 19:52

more doc edits

9f5bf8c

Add localhostBindingsEnabled as a operator helm chart value

34d0034

Merge remote-tracking branch 'origin/OWLS-92894' into OWLS-92894

2858a87

Merge Tom's latest doc edits

Istio configuration attribute localhostBindingsEnabled takes preceden…

b7fa04c

…ce over helm value

Update ItIstio tests to enable port-forwarding for Istio v1.10 and later

886edd5

anpanigr reviewed Oct 29, 2021

View reviewed changes

rjeberhard reviewed Oct 29, 2021

View reviewed changes

jshum2479 approved these changes Oct 29, 2021

View reviewed changes

Test changes for port-forward when Istio > 1.9

fa5cfb9

rjeberhard merged commit f5ef613 into main Oct 29, 2021

lennyphan deleted the OWLS-92894 branch November 1, 2021 22:22

doxiao mentioned this pull request Nov 8, 2021

OWLS-93783: WLSVERSIONEARLIERTHAN METHOD IS MISSING IN 3.3.3 #2616

Merged


		NOTE: The WebLogic Kubernetes Operator creates Kubernetes headless Services for the domain; Istio 1.6.x does not work with headless Services. See [Headless service broken in 1.6.0](https://github.com/istio/istio/issues/24082). Instead, use Istio version 1.7 and higher.
		* You cannot setup a NodePort using `domain.spec.adminServer.adminService.channels`

		@@ -427,4 +428,14 @@ private void createDomainOnPVUsingWlst(Path wlstScriptFile, Path domainPropertie

		}

		private AdminServer createAdminServer() {

		@@ -0,0 +1,134 @@
		// Copyright (c) 2020, 2021, Oracle and/or its affiliates.

OWLS-92894: Implement Istio support based on Istio version #2582

OWLS-92894: Implement Istio support based on Istio version #2582

Uh oh!

Conversation

lennyphan commented Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Istio

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jshum2479 Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tbarnes-us Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tbarnes-us Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tbarnes-us commented Oct 26, 2021

Uh oh!

rosemarymarano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

lennyphan commented Oct 21, 2021 •

edited

Loading

jshum2479 Oct 21, 2021 •

edited

Loading

tbarnes-us Oct 21, 2021 •

edited

Loading

tbarnes-us Oct 21, 2021 •

edited

Loading

anpanigr Oct 29, 2021 •

edited

Loading