diff --git a/modules/ROOT/pages/getting-started.adoc b/modules/ROOT/pages/getting-started.adoc index 1b1e1aac1..5db68e13a 100644 --- a/modules/ROOT/pages/getting-started.adoc +++ b/modules/ROOT/pages/getting-started.adoc @@ -44,23 +44,18 @@ stackablectl release install -i commons -i secret -i listener -i zookeeper -i ka ==== NOTE: These examples assume Helm version 3. They will not work with Helm version 2. -Add the stackable-stable Helm Chart repository: - -[source,bash] ----- -helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/ ----- +`helm repo` subcommands are not supported for OCI registries. The operators are installed directly, without adding the Helm Chart repository first. Install the operators: [source,bash] ---- -helm install zookeeper-operator stackable-stable/zookeeper-operator --version=24.11.1 -helm install kafka-operator stackable-stable/kafka-operator --version=24.11.1 -helm install secret-operator stackable-stable/secret-operator --version=24.11.1 -helm install listener-operator stackable-stable/listener-operator --version=24.11.1 -helm install commons-operator stackable-stable/commons-operator --version=24.11.1 -helm install nifi-operator stackable-stable/nifi-operator --version=24.11.1 +helm install zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version=24.11.1 +helm install kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version=24.11.1 +helm install secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version=24.11.1 +helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version=24.11.1 +helm install commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version=24.11.1 +helm install nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version=24.11.1 ---- ==== diff --git a/modules/ROOT/pages/product-information.adoc b/modules/ROOT/pages/product-information.adoc index b658314b8..6b41fd2f2 100644 --- a/modules/ROOT/pages/product-information.adoc +++ b/modules/ROOT/pages/product-information.adoc @@ -54,15 +54,15 @@ Stackable components. === Operators and products All operators are supplied in container images. The products are also deployed in container images. The docker images -are available for download here: https://repo.stackable.tech/#browse/browse:docker[] +are available for download here: https://oci.stackable.tech/[]. Information on how to browse the images can be found xref:contributor:project-overview.adoc#docker-images[here]. Stackable supports installing the Operators via https://helm.sh/[Helm] or with <>. Every Operator includes installation instructions in the Getting started guide. ==== Helm Charts -The Helm Charts can be found here: https://repo.stackable.tech/#browse/browse:helm-stable[] Using the Helm Charts -requires Helm version 3 or above. +The Helm Charts can be found here: https://oci.stackable.tech/api/v2.0/projects/sdp-charts[] Using the Helm Charts +requires Helm version 3 or above. Information on how to browse the OCI repository for Helm Charts is described xref:contributor:project-overview.adoc#product-artifacts[here]. [#stackablectl-installation] diff --git a/modules/concepts/pages/container-images.adoc b/modules/concepts/pages/container-images.adoc index 21eff4fcd..7415ddfe4 100644 --- a/modules/concepts/pages/container-images.adoc +++ b/modules/concepts/pages/container-images.adoc @@ -1,6 +1,6 @@ = Container images :ubi: https://catalog.redhat.com/software/base-images -:stackable-image-registry: https://repo.stackable.tech/#browse/browse +:stackable-image-registry: https://oci.stackable.tech/ :stackable-sboms: https://sboms.stackable.tech/ :description: Overview of Stackable’s container images, including structure, multi-platform support, and why upstream images are not used. diff --git a/modules/concepts/pages/product-image-selection.adoc b/modules/concepts/pages/product-image-selection.adoc index 31b532418..8275f4e10 100644 --- a/modules/concepts/pages/product-image-selection.adoc +++ b/modules/concepts/pages/product-image-selection.adoc @@ -75,7 +75,7 @@ At the bottom of this page in the <<_common_scenarios, common scenarios>> sectio == Stackable provided images -If your Kubernetes cluster has internet access, the easiest way is to use the publicly available images from the https://docker.stackable.tech/[Image registry hosted by Stackable] (as has been noted at the beginning of the page): +If your Kubernetes cluster has internet access, the easiest way is to use the publicly available images from the https://oci.stackable.tech/[Image registry hosted by Stackable] (as has been noted at the beginning of the page): [source,yaml] ---- @@ -99,7 +99,7 @@ Security updates within a release line will result in patch version bumps in the If you don't specify the Stackable version, the operator will use its own version, e.g. `23.7.0`. When using a nightly operator or a `pr` version, it will use the nightly `0.0.0-dev` image. -All the available images (with their product and stackable version) can be found in our https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable[docker repository]. +All the available images (with their product and stackable version) can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[OCI registry]. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here]). == Custom docker registry @@ -117,7 +117,7 @@ spec: repo: my.corp/myteam/stackable ---- -This will change the image from the default Stackable repository `docker.stackable.tech/stackable/kafka:3.3.1-stackable23.7.0` to `my.corp/myteam/stackable/kafka:3.3.1-stackable23.7.0`. +This will change the image from the default Stackable repository `oci.stackable.tech/sdp/kafka:3.3.1-stackable23.7.0` to `my.corp/myteam/stackable/kafka:3.3.1-stackable23.7.0`. == [[customimages]] Custom images @@ -140,7 +140,7 @@ Only when the correct product version is given to the operator, the product will Using custom images has a few limitations that users should be aware of: * The images will *have* to have the same structures that Stackable operators expect. -This should usually be ensured by specifying a Stackable image in the `FROM` clause of the Dockerfile (all the available images can be found in our https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable[docker repository] - the schema is typically: `docker.stackable.tech/stackable/:-stackable`). +This should usually be ensured by specifying a Stackable image in the `FROM` clause of the Dockerfile (all the available images can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[OCI registry] - the schema is typically: `oci.stackable.tech/sdp/:-stackable`. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here]). * Images will have to be upgraded for every new Stackable release to follow structural changes that Stackable may have made to their images. When deriving images from official Stackable images this will mean updating the version of the image in the `FROM` clause to the correct Stackable release. diff --git a/modules/contributor/pages/contributing-code.adoc b/modules/contributor/pages/contributing-code.adoc index b021423ec..0f873c300 100644 --- a/modules/contributor/pages/contributing-code.adoc +++ b/modules/contributor/pages/contributing-code.adoc @@ -79,8 +79,8 @@ For `VisualStudio Code` we also recommend the following extensions: + [source,bash] ---- -docker build --file docker/Dockerfile --tag docker.stackable.tech/stackable/:-dev . -kind load docker-image docker.stackable.tech/stackable/:-dev --name=integration-tests +docker build --file docker/Dockerfile --tag oci.stackable.tech/sdp/:-dev . +kind load docker-image oci.stackable.tech/sdp/:-dev --name=integration-tests helm install deploy/helm// ---- . Run the tests from the repository root with `./scripts/run-tests`. diff --git a/modules/contributor/pages/project-overview.adoc b/modules/contributor/pages/project-overview.adoc index 0ff86a63c..70363a0d0 100644 --- a/modules/contributor/pages/project-overview.adoc +++ b/modules/contributor/pages/project-overview.adoc @@ -85,10 +85,110 @@ Where are binaries, Helm Charts and Docker images stored? [[product-artifacts]] === Product artifacts -A lot of artifacts are stored in the https://repo.stackable.tech/#browse/browse[Nexus repo]. -Under `packages` you find product binaries, the different `helm-*` stores contain the Helm Charts and Docker images are stored there too. +A lot of artifacts are stored in the https://oci.stackable.tech[OCI registry]. Currently, those artifacts can be browsed by calling the API. + +The following command lists all the different projects the artifacts are distributed across: + +[source,console] +---- +$ curl -X GET --header 'Accept: application/json' 'https://oci.stackable.tech/api/v2.0/projects/' | jq --raw-output '.[] | .name' +library +sandbox +sdp +sdp-charts +stackable +stackable-charts +---- + +`sdp` contains the product and operator Docker images. The Helm Charts for the operators are found under `sdp-charts`. Some artifacts like the +product binaries are stored in the https://repo.stackable.tech/#browse/browse[Nexus repo] under `packages`. + +List the Helm Charts in `sdp-charts`: + +[source,console] +---- +$ curl -X GET --header 'Accept: application/json' 'https://oci.stackable.tech/api/v2.0/projects/sdp-charts/repositories?page_size=20' | jq --raw-output '.[] | .name' +---- + +List all available versions of a specific Helm Chart, e.g. the zookeeper-operator: + +[source,console] +---- +$ curl -X GET --header 'Accept: application/json' 'https://oci.stackable.tech/api/v2.0/projects/sdp-charts/repositories/zookeeper-operator/artifacts' | jq --raw-output '.[] | .tags[].name' +0.0.0-dev +0.0.0-pr920 +0.0.0-pr919 +0.0.0-pr917 +24.11.1 +24.11.1-rc2 +24.11.1-rc1 +24.11.0 +24.7.0 +24.3.0 +---- [[docker-images]] === Docker images -Docker images are stored in Nexus as mentioned above, but also in https://oci.stackable.tech - which is where we want to move to in the future. +Docker images are stored in https://oci.stackable.tech as mentioned above. To list all the available repositories in a project, for example in +the `sdp` project, run this command: + +[source,console] +---- +$ curl -X GET --header 'Accept: application/json' 'https://oci.stackable.tech/api/v2.0/projects/sdp/repositories?page=1' | jq --raw-output '.[] | .name' +sdp/statsd_exporter +sdp/stackable-cockpit +sdp/product1 +sdp/git-sync/git-sync +sdp/sig-storage/csi-provisioner +sdp/sig-storage/csi-node-driver-registrar +sdp/kcat +sdp/java-devel +sdp/vector +sdp/hello-world +---- + +Since the API is using pagination it might be necessary to increment the `page` in the command to see additional results. + +It is also possible to list all the available image tags for a repository of choice. The following command shows the tags of the `trino` repository: + +[source,console] +---- +$ curl -X GET --header 'Accept: application/json' 'https://oci.stackable.tech/api/v2.0/projects/sdp/repositories/trino/artifacts?page=1&page_size=25' | jq --raw-output '.[] | select(.extra_attrs.config != {}) | .tags[]?.name' +455-stackable0.0.0-dev +451-stackable0.0.0-dev +470-stackable0.0.0-dev +470-stackable0.0.0-dev-arm64 +455-stackable0.0.0-dev-arm64 +451-stackable0.0.0-dev-arm64 +470-stackable0.0.0-dev-amd64 +455-stackable0.0.0-dev-amd64 +451-stackable0.0.0-dev-amd64 +469-stackable0.0.0-dev +469-stackable0.0.0-dev-arm64 +469-stackable0.0.0-dev-amd64 +451-stackable24.11.1 +455-stackable24.11.1 +---- + +Similar to the previous command, the API call uses pagination again. So the `page` value in the command can be incremented to see more results. Here the `page_size` +parameter was also used to increase the results per page. + +Another possibility, instead of using `curl`, would be the https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md[crane tool], which can also be used +to browse the tags when given the path to a repository. + +[source,console] +---- +$ crane ls -O oci.stackable.tech/sdp/trino +414-stackable0.0.0-dev +414-stackable0.0.0-dev-amd64 +414-stackable0.0.0-dev-arm64 +414-stackable24.3.0 +414-stackable24.7.0 +414-stackable24.7.0-amd64 +414-stackable24.7.0-arm64 +428-stackable0.0.0-dev +428-stackable0.0.0-dev-amd64 +428-stackable0.0.0-dev-arm64 +... +---- diff --git a/modules/contributor/pages/testing-on-kubernetes.adoc b/modules/contributor/pages/testing-on-kubernetes.adoc index c1149461d..5c60513ed 100644 --- a/modules/contributor/pages/testing-on-kubernetes.adoc +++ b/modules/contributor/pages/testing-on-kubernetes.adoc @@ -87,7 +87,7 @@ The easiest way that we found to do this is by using https://github.com/tilt-dev Tilt should then automatically discover this registry from the cluster config and push images there. If you are using a remote cluster, Tilt will push the generated container images to a remote registry, in order for your Kubernetes to be able to access them. -We have configured `docker.stackable.tech/sandbox` as the default registry, as this is what all our developers use. +We have configured `oci.stackable.tech/sandbox` as the default registry, as this is what all our developers use. External contributors will not have access to this registry and need to override this to a registry of their choice. Overriding the default registry can be done by providing a file called `tilt_options.json` in the same directory as the Tiltfile. diff --git a/modules/guides/pages/custom-images.adoc b/modules/guides/pages/custom-images.adoc index 616666dd0..67a8581d7 100644 --- a/modules/guides/pages/custom-images.adoc +++ b/modules/guides/pages/custom-images.adoc @@ -1,5 +1,5 @@ = Using customized product images -:stackable-docker-registry: https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable +:stackable-oci-registry: https://oci.stackable.tech/api/v2.0/projects/sdp :kind: https://kind.sigs.k8s.io/ :kind-load-image: https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster :description: Customize Stackable product images by modifying base images, deploying to a registry or Kubernetes cluster, and configuring your Stacklet to use them. @@ -21,12 +21,12 @@ To use a customized image you have to: The Stackable operators rely on the structure and contents of the product images, so any modifications need to be done using the Stackable images as base images. -You can find the Stackable Docker images in the {stackable-docker-registry}[Stackable Docker registry]. -Images follow a naming schema: `docker.stackable.tech/stackable/:-stackable` where `` are the products like `druid`, `trino` and `opa`, `` are product versions like `28.0.1`, `414` or `0.61.0` and `` is a Stackable platform version like `23.7` or `24.3.0`. +You can find the Stackable Docker images in the {stackable-oci-registry}[Stackable OCI registry]. +Images follow a naming schema: `oci.stackable.tech/sdp/:-stackable` where `` are the products like `druid`, `trino` and `opa`, `` are product versions like `28.0.1`, `414` or `0.61.0` and `` is a Stackable platform version like `23.7` or `24.3.0`. The Stackable version can also be `0.0.0-dev` for nightly images. You can use this naming schema together with the xref:operators:supported_versions.adoc[] list to quickly find the base image you need. -For example, you might use the `docker.stackable.tech/stackable/hive:3.1.3-stackable24.3.0` image as a base for a modified Apache Hive image where you add a custom database driver. +For example, you might use the `oci.stackable.tech/sdp/hive:3.1.3-stackable24.3.0` image as a base for a modified Apache Hive image where you add a custom database driver. Use the `FROM` clause in your Dockerfile to reference the base image. When tagging the image, there are no requirements by the operator, any registry and tag is possible. @@ -60,4 +60,5 @@ With this configuration the operator deploys your Stacklet using your custom ima == Further reading and useful links * Read about xref:concepts:product-image-selection.adoc[] to learn about other ways of specifying a product version or images, for example how to use a custom registry when mirroring Stackable images. -* Have a look at the {stackable-docker-registry}[Stackable Docker registry] to find out which images are available to use as a base. +* Have a look at the {stackable-oci-registry}[Stackable OCI registry] to find out which images are available to use as a base. Information on how to browse the registry +can be found xref:contributor:project-overview.adoc#docker-images[here]. diff --git a/modules/guides/pages/running-stackable-in-an-airgapped-environment.adoc b/modules/guides/pages/running-stackable-in-an-airgapped-environment.adoc index 49305d539..dd8839d4c 100644 --- a/modules/guides/pages/running-stackable-in-an-airgapped-environment.adoc +++ b/modules/guides/pages/running-stackable-in-an-airgapped-environment.adoc @@ -4,16 +4,16 @@ The main challenge with running Stackable in an air-gapped environment is how to get the artifacts (container images and Helm charts) into the environment. There are a few ways to do this: -* Mirror our images and Helm charts into a registry in the air-gapped environment. You need to find out what images are relevant for your specific Stackable deployment and transfer them to the target registry. Feel free to browse through the images in https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable[our registry]. -* If possible, setup a reverse proxy to Stackable's registry on a node with internet connection that is reachable from all nodes within your air-gapped environment. You could, for example, use https://distribution.github.io/distribution/[distribution] for this. Here's a command to spin up a pull-through cache to `docker.stackable.tech` on port 5001: `docker run -d --name proxy-stackable -p 5001:5000 --restart=always -e REGISTRY_PROXY_REMOTEURL=https://docker.stackable.tech registry:2`. The registry is now available on localhost:5001 via HTTP. Once an image has been loaded, it will be cached by the proxy. +* Mirror our images and Helm charts into a registry in the air-gapped environment. You need to find out what images are relevant for your specific Stackable deployment and transfer them to the target registry. Feel free to browse through the images in https://oci.stackable.tech/api/v2.0/projects/sdp[our registry] by using this xref:contributor:project-overview.adoc#docker-images[guide]. +* If possible, setup a reverse proxy to Stackable's registry on a node with internet connection that is reachable from all nodes within your air-gapped environment. You could, for example, use https://distribution.github.io/distribution/[distribution] for this. Here's a command to spin up a pull-through cache to `oci.stackable.tech` on port 5001: `docker run -d --name proxy-stackable -p 5001:5000 --restart=always -e REGISTRY_PROXY_REMOTEURL=https://oci.stackable.tech registry:2`. The registry is now available on localhost:5001 via HTTP. Once an image has been loaded, it will be cached by the proxy. * Download our images (e.g. using `docker save`) on a machine with internet access, copy them onto the nodes in your air-gapped environment and load them (e.g. using `ctr images import`). Then render the Helm charts using the `helm template` subcommand, copy the rendered YAML files your air-gapped environment and apply them. In the first two scenarios, you need to make sure that the nodes load the images from your local registry mirror. Again, there are a several ways to do this: * Specify the image repository in the CRDs (see https://docs.stackable.tech/home/nightly/concepts/product-image-selection#_custom_docker_registry["Custom docker registry"]) and in the values of the Helm charts when installing the operators (`helm install --set image.repository="my.custom.registry/stackable/nifi-operator" ...`). -* If you use `containerd` as your container runtime, you can patch the `containerd` config on every node to use the mirrored registry instead of `docker.stackable.tech`. + +* If you use `containerd` as your container runtime, you can patch the `containerd` config on every node to use the mirrored registry instead of `oci.stackable.tech`. + Example: Let's assume you have a registry mirror running on `10.7.228.12`, reachable via HTTPS on port 443 using a self signed certificate. Now copy the certificate over to your Kubernetes node, in this example we'll place it in the `/etc/pki/tls/certs` folder. -Create the file `/etc/containerd/certs.d/docker.stackable.tech/hosts.toml` on the node, with the following contents: +Create the file `/etc/containerd/certs.d/oci.stackable.tech/hosts.toml` on the node, with the following contents: + [source,toml] ---- @@ -30,8 +30,8 @@ Modify your containerd config (usually located at `/etc/containerd/config.toml`) [plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d" ---- -Then restart the `containerd` service. Now `containerd` will fetch all images that would normally be fetched from `docker.stackable.tech` from `10.7.228.12` instead. The registry host name is determined by the path `hosts.toml` is located in, so other registry hosts are not affected. For further information, see https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration["Registry configuration"]. +Then restart the `containerd` service. Now `containerd` will fetch all images that would normally be fetched from `oci.stackable.tech` from `10.7.228.12` instead. The registry host name is determined by the path `hosts.toml` is located in, so other registry hosts are not affected. For further information, see https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration["Registry configuration"]. -* Add an alias for `docker.stackable.tech` to the `/etc/hosts` file on every node (like `10.7.228.12 docker.stackable.tech`), issue a self-signed certificate for `docker.stackable.tech` to your registry and add the certificate to the trusted certificates on every node. Note that if you also want to enforce signature checks for Stackable's images via a policy controller, you will need to add this host alias to the Pod of the policy controller as well (and make it trust the certificate). +* Add an alias for `oci.stackable.tech` to the `/etc/hosts` file on every node (like `10.7.228.12 oci.stackable.tech`), issue a self-signed certificate for `oci.stackable.tech` to your registry and add the certificate to the trusted certificates on every node. Note that if you also want to enforce signature checks for Stackable's images via a policy controller, you will need to add this host alias to the Pod of the policy controller as well (and make it trust the certificate). If you want to know how to verify image signatures in an air-gapped environment, check out our documentation about xref:enabling-verification-of-image-signatures.adoc[image signature verification]. diff --git a/modules/guides/pages/viewing-and-verifying-sboms.adoc b/modules/guides/pages/viewing-and-verifying-sboms.adoc index 293cbe504..7338a8413 100644 --- a/modules/guides/pages/viewing-and-verifying-sboms.adoc +++ b/modules/guides/pages/viewing-and-verifying-sboms.adoc @@ -22,7 +22,7 @@ cosign verify-attestation --type cyclonedx \ --certificate-identity-regexp \ '^https://github.com/stackabletech/.+/.github/workflows/.+@.+' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ -docker.stackable.tech/stackable/airflow-operator:24.3.0 \ +oci.stackable.tech/sdp/airflow-operator:24.3.0 \ | jq '.payload' -r | base64 -d | jq '.predicate' ----