Merge branch 'main' into policies

Author: lfrancke

modules/ROOT/nav.adoc

@@ -1,2 +1,3 @@
 * xref:quickstart.adoc[]
+* xref:kubernetes.adoc[]
 * xref:getting_started.adoc[]

modules/ROOT/nav2.adoc

@@ -1,3 +1,5 @@
 // This second nav file is for another menu entry at the end of the navigation menu
 // see: https://docs.antora.org/antora/3.0/navigation/organize-files/#multiple-files-per-module
 * xref:release_notes.adoc[Release Notes]
+* xref:product-information.adoc[]
+* xref:licenses.adoc[Licenses]

modules/ROOT/pages/getting_started.adoc

@@ -12,60 +12,10 @@ Firstly, let’s cover whether this *Getting Started* guide is right for you. Th
 == Overview
 Stackable is based on Kubernetes and uses this as the control plane to manage clusters. In this guide we will build a simple cluster with 3 services; Apache ZooKeeper, Apache Kafka and Apache NiFi.
 
-== Installing Kubernetes
-Stackable’s control plane is built around Kubernetes, and we'll give some brief examples of how to install Kubernetes on your machine.
+== Installing Kubernetes and kubectl
+Stackable’s control plane is built around Kubernetes. Follow the xref:kubernetes.adoc#local-installation[instructions] on how to set up a local Kubernetes instance if you do not have access to a cluster and install kubectl.
 
-=== Installing kubectl
-
-Stackable operators and their services are managed by applying manifest files to the Kubernetes cluster. For this purpose, you need to have the `kubectl` tool installed. Follow the instructions https://kubernetes.io/docs/tasks/tools/#kubectl[here] for your platform.
-
-=== Installing Kubernetes using Kind
-Kind offers a very quick and easy way to bootstrap your Kubernetes infrastructure in Docker. The big advantage of this is that you can simply remove the Docker containers when you're finished and clean up easily, making it great for testing and development.
-
-If you don't already have Docker then visit https://docs.docker.com/get-docker/[Docker Website] to find out how to install Docker. Kind is a single executable that performs the tasks of installing and configuring Kubernetes for you within Docker containers. The https://kind.sigs.k8s.io/docs/user/quick-start/[Kind Website] has instructions for installing Kind on your system.
-
-Once you have both of these installed then you can build a Kubernetes cluster in Docker. We're going to create a simple, single node cluster to test out Stackable, with the one node hosting both the Kubernetes control plane and the Stackable services.
-
-[source, bash]
-----
-kind create cluster --name quickstart
-----
-
-=== Installing Kubernetes using K3s
-K3s provides a quick way of installing Kubernetes. On your control node run the following command to install K3s:
-
-[source,bash]
-----
-curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
-----
-
-So long as you have an Internet connection K3s will download and automatically configure a simple Kubernetes environment.
-
-Create a symlink to the Kubernetes configuration from your home directory to allow tools like Helm to find the correct configuration.
-
-[source,bash]
-----
-mkdir ~/.kube
-ln -s /etc/rancher/k3s/k3s.yaml ~/.kube/config
-----
-
-
-=== Testing your Kubernetes installation
-
-To check if everything worked as expected you can use `kubectl cluster-info` to retrieve the cluster information. The output should look similar to:
-
-----
-Kubernetes control plane is running at https://127.0.0.1:6443
-CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
-----
-
-If you set up your cluster using K3s you will additionally see the metrics server:
-
-----
-Kubernetes control plane is running at https://127.0.0.1:6443
-CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
-Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
-----
+If you already have kubectl installed, and have access to a Kubernetes cluster, you can skip this step.
 
 == Installing Stackable
 === Install stackablectl
@@ -134,7 +84,6 @@ metadata:
 spec:
   image:
     productVersion: "3.8.1"
-    stackableVersion: "23.7"
   clusterConfig:
     tls:
       serverSecretClass: null
@@ -169,7 +118,6 @@ metadata:
 spec:
   image:
     productVersion: "3.4.0"
-    stackableVersion: "23.7"
   clusterConfig:
     zookeeperConfigMapName: simple-kafka-znode
     tls:
@@ -220,7 +168,6 @@ metadata:
 spec:
   image:
     productVersion: "1.21.0"
-    stackableVersion: "23.7"
   clusterConfig:
     listenerClass: external-unstable
     zookeeperConfigMapName: simple-nifi-znode

modules/ROOT/pages/kubernetes.adoc

@@ -0,0 +1,74 @@
+= Kubernetes
+
+The Stackable Data Platform runs on Kubernetes, a Kubernetes cluster is a prerequisite to running the platform. On this page you will find information on the supported Kubernetes distributions for production as well as how to set up a local test installation to try out parts of the platform right away.
+
+[#supported-production-distributions]
+== Supported production distributions
+
+The Stackable Data Platform requires a Kubernetes cluster to be present, where you can install things into. How to set up Kubernetes as well as a cluster depends on the distribution you chose. 
+
+The following distributions are supported for a production setup of the Stackable Data Platform:
+
+include::partial$supported-kubernetes-distributions.adoc[]
+
+In this version of the SDP, the following Kubernetes versions are supported:
+
+include::partial$supported-kubernetes-versions.adoc[]
+
+Consult the xref:release_notes.adoc[release notes] to find out which specific versions are supported for the Stackable Data Platform you are using.
+
+[#local-installation]
+== Installing a testinging/development Kubernetes instance locally
+Stackable's control plane is built around Kubernetes, and we'll give some brief examples of how to install Kubernetes on your machine.
+
+=== Installing kubectl
+
+Stackable operators and their services are managed by applying manifest files to the Kubernetes cluster. For this purpose, you need to have the `kubectl` tool installed. Follow the instructions https://kubernetes.io/docs/tasks/tools/#kubectl[here] for your platform.
+
+=== Installing Kubernetes using Kind
+Kind offers a very quick and easy way to bootstrap your Kubernetes infrastructure in Docker. The big advantage of this is that you can simply remove the Docker containers when you're finished and clean up easily, making it great for testing and development.
+
+If you don't already have Docker then visit https://docs.docker.com/get-docker/[Docker Website] to find out how to install Docker. Kind is a single executable that performs the tasks of installing and configuring Kubernetes for you within Docker containers. The https://kind.sigs.k8s.io/docs/user/quick-start/[Kind Website] has instructions for installing Kind on your system.
+
+Once you have both of these installed then you can build a Kubernetes cluster in Docker. We're going to create a simple, single node cluster to test out Stackable, with the one node hosting both the Kubernetes control plane and the Stackable services.
+
+[source, bash]
+----
+kind create cluster --name quickstart
+----
+
+=== Installing Kubernetes using K3s
+K3s provides a quick way of installing Kubernetes. On your control node run the following command to install K3s:
+
+[source,bash]
+----
+curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
+----
+
+So long as you have an Internet connection K3s will download and automatically configure a simple Kubernetes environment.
+
+Create a symlink to the Kubernetes configuration from your home directory to allow tools like Helm to find the correct configuration.
+
+[source,bash]
+----
+mkdir ~/.kube
+ln -s /etc/rancher/k3s/k3s.yaml ~/.kube/config
+----
+
+
+=== Testing your Kubernetes installation
+
+To check if everything worked as expected you can use `kubectl cluster-info` to retrieve the cluster information. The output should look similar to:
+
+----
+Kubernetes control plane is running at https://127.0.0.1:6443
+CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+----
+
+If you set up your cluster using K3s you will additionally see the metrics server:
+
+----
+Kubernetes control plane is running at https://127.0.0.1:6443
+CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+----

modules/ROOT/pages/licenses.adoc

@@ -0,0 +1,35 @@
+= Licenses for the Stackable Data Platform
+
+The Stackable Data Platform is open source, and the source code of all the components can be found on GitHub. Licenses are also provided alongside the source code, in a file called `LICENSE`.
+
+== Operators
+
+Product Operators
+
+* https://github.com/stackabletech/airflow-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Airflow.
+* https://github.com/stackabletech/druid-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Druid.
+* https://github.com/stackabletech/hbase-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache HBase.
+* https://github.com/stackabletech/hdfs-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Hadoop HDFS.
+* https://github.com/stackabletech/hive-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Hive.
+* https://github.com/stackabletech/kafka-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Kafka.
+* https://github.com/stackabletech/nifi-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache NiFi.
+* https://github.com/stackabletech/spark-k8s-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Spark.
+* https://github.com/stackabletech/trino-operator/blob/main/LICENSE[License] for the Stackable Operator for Trino.
+* https://github.com/stackabletech/zookeeper-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache ZooKeeper.
+* https://github.com/stackabletech/opa-operator/blob/main/LICENSE[License] for the Stackable Operator for OpenPolicyAgent.
+
+Additional Stackable Operators
+
+* https://github.com/stackabletech/commons-operator/blob/main/LICENSE[License] for the Stackable Commons Operator.
+* https://github.com/stackabletech/secret-operator/blob/main/LICENSE[License] for the Stackable Secret Operator.
+* https://github.com/stackabletech/listener-operator/blob/main/LICENSE[License] for the Stackable Listener Operator.
+
+== stackablectl
+
+https://github.com/stackabletech/stackablectl/blob/main/LICENSE[License] for stackablectl.
+
+== Product images
+
+https://github.com/stackabletech/docker-images/blob/main/LICENSE[License] for the product Docker images.
+
+The Docker images are built on the https://catalog.redhat.com/software/containers/ubi8/ubi-minimal/5c359a62bed8bd75a2c3fba8?container-tabs=overview[Red Hat ubi8-minimal base image]. It is https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf[licensed seperately].

modules/ROOT/pages/product-information.adoc

@@ -0,0 +1,107 @@
+= Product information
+
+This page contains concrete specifications about the Stackable Data Platform (SDP) as a product, which components are included, how they are supplied and which external dependencies exist that you as a customer need to take care of.
+
+[#platform-components]
+== Platform components
+
+The Stackable Platform is made up of multiple components. Operators for data products, Operators for additional functionality to facilitate easy integration between the different products, as well as the data products themselves. Auxiliary software to interact with the platform, as well as software to ease the deployment of platform components.
+
+=== Data products and Kubernetes Operators
+
+The main components of the SDP are a set of open source data products. The SDP control plane consists of corresponding Kubernetes Operators. Each Operator is supplied in a Docker <<containers, container>> image. For every Operator there is also a Helm Chart to facilitate installation via Helm. Operators for Products supported by the Platform deploy these products as <<containers, containers>> in Kubernetes.
+
+Supported products:
+
+* xref:airflow:index.adoc[Apache Airflow]
+* xref:druid:index.adoc[Apache Druid]
+* xref:hbase:index.adoc[Apache HBase] (including xref:hbase:usage-guide/phoenix.adoc[Apache Phoenix])
+* xref:hive:index.adoc[Apache Hive Metastore]
+* xref:hdfs:index.adoc[Apache Hadoop HDFS]
+* xref:kafka:index.adoc[Apache Kafka]
+* xref:nifi:index.adoc[Apache Nifi]
+* xref:spark-k8s:index.adoc[Apache Spark] (including xref:spark-k8s:usage-guide/history-server.adoc[Spark History Server])
+* xref:superset:index.adoc[Apache Superset]
+* xref:trino:index.adoc[Trino]
+* xref:zookeeper:index.adoc[Apache Zookeeper]
+
+The product also includes the xref:opa:index.adoc[Open Policy Agent], and operators that provide additional functionality for managing and control SDP: xref:commons-operator:index.adoc[Commons], xref:secret-operator:index.adoc[Secret] and xref:listener-operator:index.adoc[Listener] Operator.
+
+The pages linked above also detail the use cases and features supported by each component. You can find additional information in the xref:release_notes.adoc[release notes]. Refer to the xref:operators:supported_versions.adoc[list of supported product versions] to find out which product versions are supported.
+
+
+[#stackablectl]
+=== stackablectl
+
+xref:stackablectl::index.adoc[stackablectl] is a commandline utility that makes it easier to install and interact with stackable components.
+
+== Supported installation methods
+
+[#containers]
+=== Operators and products
+
+All operators are supplied in container images. The products are also deployed in container images.
+The docker images are available for download here: https://repo.stackable.tech/#browse/browse:docker
+
+Stackable supports installing the Operators via https://helm.sh/[Helm] or with <<stackablectl>>.
+Every Operator includes installation instructions in the Getting started guide.
+
+==== Helm Charts
+
+The Helm Charts can be found here: https://repo.stackable.tech/#browse/browse:helm-stable Using the Helm Charts requires Helm version 3 or above.
+
+
+[#stackablectl-installation]
+=== stackablectl
+
+stackablectl is available for download, pre-built binaries are available on GitHub. The download link and installation steps are provided in the xref:stackablectl::installation.adoc[installation documentation].
+
+== System requirements
+
+=== Operators
+
+Every Operator needs:
+
+* 0.2 cores (i.e. i5 or similar)
+* 256MB RAM
+
+=== stackablectl
+
+Hardware requirements and supported operating systems can be found in the xref:stackablectl::installation.adoc#system-requirements[system requirements section] of the stackablectl documentation.
+
+=== Stackable Data Platform Open Source Products
+
+The system requirements of each open source product depend on your specific use case. In the xref:stackablectl::demos/index.adoc[demos] you can find example use cases; every demo also has a _System requirements_ section which can provide a guideline for system sizing. Further example sizings can be found at https://ci.stackable.tech/job/<product>-operator-it-summary/ (where `<product>` is i.e. _druid_, _hbase_, etc.) in the `cluster-info.txt` file. These are the cluster sizes used for integration tests, and can be regarded as working cluster configurations.
+
+== Prerequisites and required external components
+
+Required external components are components that are required for the platform or parts of it to operate, but are not part of the Stackable Data Platform. Note that Stackable does neither distribute nor offer support for the external components.
+
+=== Kubernetes
+
+A Kubernetes cluster is required to install the Stackable Data Platform. The supported Kubernetes versions are:
+
+include::partial$supported-kubernetes-versions.adoc[]
+
+There are various Kubernetes distributions. Stackable supports:
+
+include::partial$supported-kubernetes-distributions.adoc[]
+
+=== Product specific dependencies
+
+The following products have required external components to run:
+
+* xref:airflow:required-external-components.adoc[Apache Airflow]
+* xref:druid:required-external-components.adoc[Apache Druid]
+* xref:hive:required-external-components.adoc[Apache Hive]
+* xref:superset:required-external-components.adoc[Apache Superset]
+
+== Optional and technology preview components
+
+=== Optional components
+
+Stackable software can be used with xref:stackablectl::demos/index.adoc[sample configurations] and third-party components, as outlined in the relevant documentation. These external components are not part of of the stackable products and can be used by the customer at their own risk. Stackable does not distribute these components and does not offer support for them. (See <<platform-components,Platform components>> above for a list of the supported components that are part of the platform)
+
+=== Technology preview components
+
+Some functionality of the platform might be labelled as _technology preview_. Stackable does not offer support for technology preview components, and therefore discourages their use in a production setting. Using these features/components is done so at your own risk. Customers are encouraged to provide feedback and suggestions for improvements on preview components. Neither Stackable nor third parties or licensees are obligated to distribute technology preview components or include them into the product. Technology preview components might be discontinued at any time.

modules/ROOT/partials/supported-kubernetes-distributions.adoc

@@ -0,0 +1,7 @@
+* https://aws.amazon.com/eks/[Amazon Elastic Kubernetes Service]
+* https://azure.microsoft.com/en-gb/products/kubernetes-service[Microsoft Azure Kubernetes Service]
+* https://cloud.google.com/kubernetes-engine[Google Kubernetes Engine]
+* https://cloud.ionos.com/managed/kubernetes[IONOS Managed Kubernetes]
+* https://www.rancher.com/products/rancher[SUSE Rancher]
+* https://www.suse.com/products/k3s/[SUSE K3S]
+* https://www.redhat.com/en/technologies/cloud-computing/openshift[Red Hat OpenShift] 4.10, 4.11

modules/ROOT/partials/supported-kubernetes-versions.adoc

@@ -2,6 +2,6 @@
 // This is a separate file to refer to the version at multiple places
 // in the documentation
 
-- 1.21
-- 1.22
-- 1.23
+- 1.26
+- 1.25
+- 1.24