Skip to content

Added license and overview page #439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Aug 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
b9272e7
Added license and overview page
fhennig Aug 8, 2023
b0872ff
Added page about kubernetes
fhennig Aug 8, 2023
5a90c25
refactored Kubernetes info
fhennig Aug 8, 2023
ee78c00
fixed formatting
fhennig Aug 8, 2023
097fa60
temporary changed stackablectl branch ref
fhennig Aug 8, 2023
4849b72
temporary change reverted
fhennig Aug 9, 2023
bbb99d2
...
fhennig Aug 9, 2023
0e0ba3c
reverted
fhennig Aug 9, 2023
3ab26ab
restructured info into a Product Information page
fhennig Aug 9, 2023
9158fb8
~
fhennig Aug 9, 2023
eea1a35
~
fhennig Aug 10, 2023
1f13b9c
added required external components
fhennig Aug 10, 2023
fe69bea
~
fhennig Aug 10, 2023
f82ecb5
reverted branch references back to main
fhennig Aug 10, 2023
232a3de
reverted branch references back to main
fhennig Aug 10, 2023
147e079
fixed typo
fhennig Aug 10, 2023
194cdde
temporary branch ref
fhennig Aug 14, 2023
24121ec
temporary branch ref removed
fhennig Aug 15, 2023
46f9e64
updates
fhennig Aug 15, 2023
aa4fafb
Added links
fhennig Aug 15, 2023
52a8d57
Added another section
fhennig Aug 15, 2023
61bb1a8
Update modules/ROOT/pages/product-information.adoc
fhennig Aug 15, 2023
7c288c4
Update modules/ROOT/pages/product-information.adoc
fhennig Aug 15, 2023
069e249
Update modules/ROOT/pages/product-information.adoc
fhennig Aug 15, 2023
4ae594e
Update modules/ROOT/pages/product-information.adoc
fhennig Aug 15, 2023
6920a8a
some changes
fhennig Aug 15, 2023
a8584a7
moved OPA
fhennig Aug 15, 2023
bebac36
another change
fhennig Aug 15, 2023
7da0ddb
Added links to k8s distros
fhennig Aug 15, 2023
e6fb62a
restructuring
fhennig Aug 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
* xref:quickstart.adoc[]
* xref:kubernetes.adoc[]
* xref:getting_started.adoc[]
2 changes: 2 additions & 0 deletions modules/ROOT/nav2.adoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
// This second nav file is for another menu entry at the end of the navigation menu
// see: https://docs.antora.org/antora/3.0/navigation/organize-files/#multiple-files-per-module
* xref:release_notes.adoc[Release Notes]
* xref:product-information.adoc[]
* xref:licenses.adoc[Licenses]
56 changes: 3 additions & 53 deletions modules/ROOT/pages/getting_started.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,60 +12,10 @@ Firstly, let’s cover whether this *Getting Started* guide is right for you. Th
== Overview
Stackable is based on Kubernetes and uses this as the control plane to manage clusters. In this guide we will build a simple cluster with 3 services; Apache ZooKeeper, Apache Kafka and Apache NiFi.

== Installing Kubernetes
Stackable’s control plane is built around Kubernetes, and we'll give some brief examples of how to install Kubernetes on your machine.
== Installing Kubernetes and kubectl
Stackable’s control plane is built around Kubernetes. Follow the xref:kubernetes.adoc#local-installation[instructions] on how to set up a local Kubernetes instance if you do not have access to a cluster and install kubectl.

=== Installing kubectl

Stackable operators and their services are managed by applying manifest files to the Kubernetes cluster. For this purpose, you need to have the `kubectl` tool installed. Follow the instructions https://kubernetes.io/docs/tasks/tools/#kubectl[here] for your platform.

=== Installing Kubernetes using Kind
Kind offers a very quick and easy way to bootstrap your Kubernetes infrastructure in Docker. The big advantage of this is that you can simply remove the Docker containers when you're finished and clean up easily, making it great for testing and development.

If you don't already have Docker then visit https://docs.docker.com/get-docker/[Docker Website] to find out how to install Docker. Kind is a single executable that performs the tasks of installing and configuring Kubernetes for you within Docker containers. The https://kind.sigs.k8s.io/docs/user/quick-start/[Kind Website] has instructions for installing Kind on your system.

Once you have both of these installed then you can build a Kubernetes cluster in Docker. We're going to create a simple, single node cluster to test out Stackable, with the one node hosting both the Kubernetes control plane and the Stackable services.

[source, bash]
----
kind create cluster --name quickstart
----

=== Installing Kubernetes using K3s
K3s provides a quick way of installing Kubernetes. On your control node run the following command to install K3s:

[source,bash]
----
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
----

So long as you have an Internet connection K3s will download and automatically configure a simple Kubernetes environment.

Create a symlink to the Kubernetes configuration from your home directory to allow tools like Helm to find the correct configuration.

[source,bash]
----
mkdir ~/.kube
ln -s /etc/rancher/k3s/k3s.yaml ~/.kube/config
----


=== Testing your Kubernetes installation

To check if everything worked as expected you can use `kubectl cluster-info` to retrieve the cluster information. The output should look similar to:

----
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
----

If you set up your cluster using K3s you will additionally see the metrics server:

----
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
----
If you already have kubectl installed, and have access to a Kubernetes cluster, you can skip this step.

== Installing Stackable
=== Install stackablectl
Expand Down
74 changes: 74 additions & 0 deletions modules/ROOT/pages/kubernetes.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
= Kubernetes

The Stackable Data Platform runs on Kubernetes, a Kubernetes cluster is a prerequisite to running the platform. On this page you will find information on the supported Kubernetes distributions for production as well as how to set up a local test installation to try out parts of the platform right away.

[#supported-production-distributions]
== Supported production distributions

The Stackable Data Platform requires a Kubernetes cluster to be present, where you can install things into. How to set up Kubernetes as well as a cluster depends on the distribution you chose.

The following distributions are supported for a production setup of the Stackable Data Platform:

include::partial$supported-kubernetes-distributions.adoc[]

In this version of the SDP, the following Kubernetes versions are supported:

include::partial$supported-kubernetes-versions.adoc[]

Consult the xref:release_notes.adoc[release notes] to find out which specific versions are supported for the Stackable Data Platform you are using.

[#local-installation]
== Installing a testinging/development Kubernetes instance locally
Stackable's control plane is built around Kubernetes, and we'll give some brief examples of how to install Kubernetes on your machine.

=== Installing kubectl

Stackable operators and their services are managed by applying manifest files to the Kubernetes cluster. For this purpose, you need to have the `kubectl` tool installed. Follow the instructions https://kubernetes.io/docs/tasks/tools/#kubectl[here] for your platform.

=== Installing Kubernetes using Kind
Kind offers a very quick and easy way to bootstrap your Kubernetes infrastructure in Docker. The big advantage of this is that you can simply remove the Docker containers when you're finished and clean up easily, making it great for testing and development.

If you don't already have Docker then visit https://docs.docker.com/get-docker/[Docker Website] to find out how to install Docker. Kind is a single executable that performs the tasks of installing and configuring Kubernetes for you within Docker containers. The https://kind.sigs.k8s.io/docs/user/quick-start/[Kind Website] has instructions for installing Kind on your system.

Once you have both of these installed then you can build a Kubernetes cluster in Docker. We're going to create a simple, single node cluster to test out Stackable, with the one node hosting both the Kubernetes control plane and the Stackable services.

[source, bash]
----
kind create cluster --name quickstart
----

=== Installing Kubernetes using K3s
K3s provides a quick way of installing Kubernetes. On your control node run the following command to install K3s:

[source,bash]
----
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
----

So long as you have an Internet connection K3s will download and automatically configure a simple Kubernetes environment.

Create a symlink to the Kubernetes configuration from your home directory to allow tools like Helm to find the correct configuration.

[source,bash]
----
mkdir ~/.kube
ln -s /etc/rancher/k3s/k3s.yaml ~/.kube/config
----


=== Testing your Kubernetes installation

To check if everything worked as expected you can use `kubectl cluster-info` to retrieve the cluster information. The output should look similar to:

----
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
----

If you set up your cluster using K3s you will additionally see the metrics server:

----
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
----
35 changes: 35 additions & 0 deletions modules/ROOT/pages/licenses.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
= Licenses for the Stackable Data Platform

The Stackable Data Platform is open source, and the source code of all the components can be found on GitHub. Licenses are also provided alongside the source code, in a file called `LICENSE`.

== Operators

Product Operators

* https://github.com/stackabletech/airflow-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Airflow.
* https://github.com/stackabletech/druid-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Druid.
* https://github.com/stackabletech/hbase-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache HBase.
* https://github.com/stackabletech/hdfs-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Hadoop HDFS.
* https://github.com/stackabletech/hive-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Hive.
* https://github.com/stackabletech/kafka-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Kafka.
* https://github.com/stackabletech/nifi-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache NiFi.
* https://github.com/stackabletech/spark-k8s-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache Spark.
* https://github.com/stackabletech/trino-operator/blob/main/LICENSE[License] for the Stackable Operator for Trino.
* https://github.com/stackabletech/zookeeper-operator/blob/main/LICENSE[License] for the Stackable Operator for Apache ZooKeeper.
* https://github.com/stackabletech/opa-operator/blob/main/LICENSE[License] for the Stackable Operator for OpenPolicyAgent.

Additional Stackable Operators

* https://github.com/stackabletech/commons-operator/blob/main/LICENSE[License] for the Stackable Commons Operator.
* https://github.com/stackabletech/secret-operator/blob/main/LICENSE[License] for the Stackable Secret Operator.
* https://github.com/stackabletech/listener-operator/blob/main/LICENSE[License] for the Stackable Listener Operator.

== stackablectl

https://github.com/stackabletech/stackablectl/blob/main/LICENSE[License] for stackablectl.

== Product images

https://github.com/stackabletech/docker-images/blob/main/LICENSE[License] for the product Docker images.

The Docker images are built on the https://catalog.redhat.com/software/containers/ubi8/ubi-minimal/5c359a62bed8bd75a2c3fba8?container-tabs=overview[Red Hat ubi8-minimal base image]. It is https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf[licensed seperately].
107 changes: 107 additions & 0 deletions modules/ROOT/pages/product-information.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
= Product information

This page contains concrete specifications about the Stackable Data Platform (SDP) as a product, which components are included, how they are supplied and which external dependencies exist that you as a customer need to take care of.

[#platform-components]
== Platform components

The Stackable Platform is made up of multiple components. Operators for data products, Operators for additional functionality to facilitate easy integration between the different products, as well as the data products themselves. Auxiliary software to interact with the platform, as well as software to ease the deployment of platform components.

=== Data products and Kubernetes Operators

The main components of the SDP are a set of open source data products. The SDP control plane consists of corresponding Kubernetes Operators. Each Operator is supplied in a Docker <<containers, container>> image. For every Operator there is also a Helm Chart to facilitate installation via Helm. Operators for Products supported by the Platform deploy these products as <<containers, containers>> in Kubernetes.

Supported products:

* xref:airflow:index.adoc[Apache Airflow]
* xref:druid:index.adoc[Apache Druid]
* xref:hbase:index.adoc[Apache HBase] (including xref:hbase:usage-guide/phoenix.adoc[Apache Phoenix])
* xref:hive:index.adoc[Apache Hive Metastore]
* xref:hdfs:index.adoc[Apache Hadoop HDFS]
* xref:kafka:index.adoc[Apache Kafka]
* xref:nifi:index.adoc[Apache Nifi]
* xref:spark-k8s:index.adoc[Apache Spark] (including xref:spark-k8s:usage-guide/history-server.adoc[Spark History Server])
* xref:superset:index.adoc[Apache Superset]
* xref:trino:index.adoc[Trino]
* xref:zookeeper:index.adoc[Apache Zookeeper]

The product also includes the xref:opa:index.adoc[Open Policy Agent], and operators that provide additional functionality for managing and control SDP: xref:commons-operator:index.adoc[Commons], xref:secret-operator:index.adoc[Secret] and xref:listener-operator:index.adoc[Listener] Operator.

The pages linked above also detail the use cases and features supported by each component. You can find additional information in the xref:release_notes.adoc[release notes]. Refer to the xref:operators:supported_versions.adoc[list of supported product versions] to find out which product versions are supported.


[#stackablectl]
=== stackablectl

xref:stackablectl::index.adoc[stackablectl] is a commandline utility that makes it easier to install and interact with stackable components.

== Supported installation methods

[#containers]
=== Operators and products

All operators are supplied in container images. The products are also deployed in container images.
The docker images are available for download here: https://repo.stackable.tech/#browse/browse:docker

Stackable supports installing the Operators via https://helm.sh/[Helm] or with <<stackablectl>>.
Every Operator includes installation instructions in the Getting started guide.

==== Helm Charts

The Helm Charts can be found here: https://repo.stackable.tech/#browse/browse:helm-stable Using the Helm Charts requires Helm version 3 or above.


[#stackablectl-installation]
=== stackablectl

stackablectl is available for download, pre-built binaries are available on GitHub. The download link and installation steps are provided in the xref:stackablectl::installation.adoc[installation documentation].

== System requirements

=== Operators

Every Operator needs:

* 0.2 cores (i.e. i5 or similar)
* 256MB RAM

=== stackablectl

Hardware requirements and supported operating systems can be found in the xref:stackablectl::installation.adoc#system-requirements[system requirements section] of the stackablectl documentation.

=== Stackable Data Platform Open Source Products

The system requirements of each open source product depend on your specific use case. In the xref:stackablectl::demos/index.adoc[demos] you can find example use cases; every demo also has a _System requirements_ section which can provide a guideline for system sizing. Further example sizings can be found at https://ci.stackable.tech/job/<product>-operator-it-summary/ (where `<product>` is i.e. _druid_, _hbase_, etc.) in the `cluster-info.txt` file. These are the cluster sizes used for integration tests, and can be regarded as working cluster configurations.

== Prerequisites and required external components

Required external components are components that are required for the platform or parts of it to operate, but are not part of the Stackable Data Platform. Note that Stackable does neither distribute nor offer support for the external components.

=== Kubernetes

A Kubernetes cluster is required to install the Stackable Data Platform. The supported Kubernetes versions are:

include::partial$supported-kubernetes-versions.adoc[]

There are various Kubernetes distributions. Stackable supports:

include::partial$supported-kubernetes-distributions.adoc[]

=== Product specific dependencies

The following products have required external components to run:

* xref:airflow:required-external-components.adoc[Apache Airflow]
* xref:druid:required-external-components.adoc[Apache Druid]
* xref:hive:required-external-components.adoc[Apache Hive]
* xref:superset:required-external-components.adoc[Apache Superset]

== Optional and technology preview components

=== Optional components

Stackable software can be used with xref:stackablectl::demos/index.adoc[sample configurations] and third-party components, as outlined in the relevant documentation. These external components are not part of of the stackable products and can be used by the customer at their own risk. Stackable does not distribute these components and does not offer support for them. (See <<platform-components,Platform components>> above for a list of the supported components that are part of the platform)

=== Technology preview components

Some functionality of the platform might be labelled as _technology preview_. Stackable does not offer support for technology preview components, and therefore discourages their use in a production setting. Using these features/components is done so at your own risk. Customers are encouraged to provide feedback and suggestions for improvements on preview components. Neither Stackable nor third parties or licensees are obligated to distribute technology preview components or include them into the product. Technology preview components might be discontinued at any time.
7 changes: 7 additions & 0 deletions modules/ROOT/partials/supported-kubernetes-distributions.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
* https://aws.amazon.com/eks/[Amazon Elastic Kubernetes Service]
* https://azure.microsoft.com/en-gb/products/kubernetes-service[Microsoft Azure Kubernetes Service]
* https://cloud.google.com/kubernetes-engine[Google Kubernetes Engine]
* https://cloud.ionos.com/managed/kubernetes[IONOS Managed Kubernetes]
* https://www.rancher.com/products/rancher[SUSE Rancher]
* https://www.suse.com/products/k3s/[SUSE K3S]
* https://www.redhat.com/en/technologies/cloud-computing/openshift[Red Hat OpenShift] 4.10, 4.11
6 changes: 3 additions & 3 deletions modules/ROOT/partials/supported-kubernetes-versions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
// This is a separate file to refer to the version at multiple places
// in the documentation

- 1.21
- 1.22
- 1.23
- 1.26
- 1.25
- 1.24