Skip to content

refactor: Restructure docs into partials and handle patch releases #703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 27, 2025
Merged

refactor: Restructure docs into partials and handle patch releases #703

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4,008 changes: 11 additions & 3,997 deletions modules/ROOT/pages/release-notes.adoc
View file
Open in desktop

Large diffs are not rendered by default.

246 changes: 246 additions & 0 deletions modules/ROOT/partials/release-notes/release-22.11.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,246 @@
== Release 22.11

=== 22.11.0

Released on 2022-11-09.
This is the third release of the Stackable Data Platform, which this time focuses on resource management.

==== New platform features

The following new major platform features were added:

CPU and memory limits configurable::

The operators now https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/[request] resources from Kubernetes for the products and required CPU and memory can now also be configured for all products.
If your product instances are less performant after the update, the new defaults might be set too low and we recommend to xref:kafka:usage-guide/storage-resources.adoc[set custom requests] for your cluster.

* https://github.com/stackabletech/opa-operator/pull/347[OpenPolicyAgent]
* https://github.com/stackabletech/zookeeper-operator/pull/563[Apache ZooKeeper]
* https://github.com/stackabletech/kafka-operator/pull/485[Apache Kafka]
* https://github.com/stackabletech/hbase-operator/pull/245[Apache HBase]
* https://github.com/stackabletech/hive-operator/pull/242[Apache Hive]
* https://github.com/stackabletech/nifi-operator/pull/353[Apache NiFi]
* https://github.com/stackabletech/druid-operator/pull/298[Apache Druid]
* https://github.com/stackabletech/airflow-operator/pull/167[Apache Airflow]
* https://github.com/stackabletech/superset-operator/pull/273[Apache Superset]

Orphaned Resources::

The operators now properly clean up after scaling down products.
This means for example deleting StatefulSets that were left over after scaling down.

* https://github.com/stackabletech/zookeeper-operator/pull/569[Apache ZooKeeper]
* https://github.com/stackabletech/hbase-operator/pull/215[Apache HBase]
* https://github.com/stackabletech/hdfs-operator/pull/249[Apache Hadoop HDFS]
* https://github.com/stackabletech/hive-operator/pull/254[Apache Hive]
* https://github.com/stackabletech/druid-operator/pull/310[Apache Druid]
* https://github.com/stackabletech/trino-operator/pull/310[Trino]
* https://github.com/stackabletech/airflow-operator/pull/174[Apache Airflow]

New Versions::

New product versions are supported.

* https://github.com/stackabletech/kafka-operator/pull/492[Apache Kafka 3.3.1]
* https://github.com/stackabletech/hdfs-operator/pull/250[Apache Hadoop HDFS 3.3.4]
* https://github.com/stackabletech/nifi-operator/pull/360[Apache NiFi 1.18.0]
* https://github.com/stackabletech/druid-operator/pull/317[Apache Druid 24.0.0]
* https://github.com/stackabletech/airflow-operator/pull/179[Apache Airflow 2.4.1]

Product features::

Additionally there are some individual product features that are noteworthy

* https://github.com/stackabletech/kafka-operator/pull/221[HBase: Phoenix support]
* https://github.com/stackabletech/hive-operator/pull/264[Hive: Support HDFS connection]
* https://github.com/stackabletech/nifi-operator/pull/323[NiFi: Support for in-place upgrades]
* https://github.com/stackabletech/nifi-operator/pull/371[NiFi: repository sizes are now adjusted based on declared PVC sizes]
* https://github.com/stackabletech/trino-operator/pull/306[Trino: Support for LDAP authentication]
* The github repositories contain new and improved READMEs.

==== Supported Kubernetes versions

This release supports the following Kubernetes versions:

* `1.25` (new)
* `1.24`
* `1.23`
* `1.22`

==== Upgrade from 22.09

===== Using stackablectl
You can list the available releases as follows

[source,console]
----
$ stackablectl release list
RELEASE RELEASE DATE DESCRIPTION
22.11 2022-11-08 Third release focusing on resource management
22.09 2022-09-09 Second release focusing on security and OpenShift support
22.06 2022-06-30 First official release of the Stackable Data Platform

----

To uninstall the `22.09` release run

[source,console]
----
$ stackablectl release uninstall 22.09
[INFO ] Uninstalling release 22.09
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...
----

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs.

[source,console]
----
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.6.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.4.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.8.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.5.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.6.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.8.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.8.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.8.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.11.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.6.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.6.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.7.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.8.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.12.0/deploy/helm/zookeeper-operator/crds/crds.yaml
----

To install the `22.11` release run

[source,console]
----
$ stackablectl release install 22.11
[INFO ] Installing release 22.11
[INFO ] Installing airflow operator in version 0.6.0
[INFO ] Installing commons operator in version 0.4.0
[INFO ] Installing druid operator in version 0.8.0
[INFO ] Installing hbase operator in version 0.5.0
[INFO ] Installing hdfs operator in version 0.6.0
[INFO ] Installing hive operator in version 0.8.0
[INFO ] Installing kafka operator in version 0.8.0
[INFO ] Installing nifi operator in version 0.8.0
[INFO ] Installing opa operator in version 0.11.0
[INFO ] Installing secret operator in version 0.6.0
[INFO ] Installing spark-k8s operator in version 0.6.0
[INFO ] Installing superset operator in version 0.7.0
[INFO ] Installing trino operator in version 0.7.0
[INFO ] Installing zookeeper operator in version 0.12.0
# ...
----

===== Using helm

Use `helm list` to list the currently installed operators.
You can use the following command to uninstall all of the operators that are part of the release 22.09:

[source,console]
----
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...
----

Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
This is because helm will uninstall the operators but not the CRDs.

[source,console]
----
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.6.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.4.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.8.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.5.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.6.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.8.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.8.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.8.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.11.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.6.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.6.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.7.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.8.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.12.0/deploy/helm/zookeeper-operator/crds/crds.yaml
----

To install the release 22.11 run

[source,console]
----
$ helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
$ helm repo update stackable-stable
$ helm install --wait airflow-operator stackable-stable/airflow-operator --version 0.6.0
$ helm install --wait commons-operator stackable-stable/commons-operator --version 0.4.0
$ helm install --wait druid-operator stackable-stable/druid-operator --version 0.8.0
$ helm install --wait hbase-operator stackable-stable/hbase-operator --version 0.5.0
$ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 0.6.0
$ helm install --wait hive-operator stackable-stable/hive-operator --version 0.8.0
$ helm install --wait kafka-operator stackable-stable/kafka-operator --version 0.8.0
$ helm install --wait nifi-operator stackable-stable/nifi-operator --version 0.8.0
$ helm install --wait opa-operator stackable-stable/opa-operator --version 0.11.0
$ helm install --wait secret-operator stackable-stable/secret-operator --version 0.6.0
$ helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 0.6.0
$ helm install --wait superset-operator stackable-stable/superset-operator --version 0.7.0
$ helm install --wait trino-operator stackable-stable/trino-operator --version 0.7.0
$ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 0.12.0
----

==== Breaking changes

You will need to adapt your existing CRDs due to the following breaking changes:

===== Stackable Operator for Apache Spark

The configuration of pod resource requests has been changed to be consistent with other operators that are part of the Stackable Data Platform (https://github.com/stackabletech/spark-k8s-operator/pull/147[#174]).

In the previous version, these were configured like this:

```
driver:
cores: 1
coreLimit: "1200m"
memory: "512m"`
```

From now on, Pod resources can be configured in two different ways.
The first and recommended way is to add a resources section for each role as the following examples shows:

```
driver:
resources:
cpu:
min: "1"
max: "1500m"
memory:
limit: "1Gi"
```

The second method is to use the `sparkConf` section and and set them individually as spark properties:

```
sparkConf:
spark.kubernetes.submission.waitAppCompletion: "false"
spark.kubernetes.driver.pod.name: "resources-sparkconf-driver"
spark.kubernetes.executor.podNamePrefix: "resources-sparkconf"
spark.kubernetes.driver.request.cores: "2"
spark.kubernetes.driver.limit.cores: "3"
```

When both methods are used, the settings in the `sparkConf` section override the `resources` configuration.

Note that none of the settings above have any influence over the parallelism used by Spark itself.
The only supported way to affect this is as follows:

```
sparkConf:
spark.driver.cores: "3"
spark.executor.cores: "3"
```
104 changes: 104 additions & 0 deletions modules/ROOT/partials/release-notes/release-22.6.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
== Release 22.6

This is our first release of the Stackable Data Platform, bringing Kubernetes operators for 12 products as well as <<stackablectl>>, the commandline tool to easily install data products in Kubernetes Operators spin up production ready product applications.
Also, there are some common features across all operators, such as monitoring, service discovery and configuration overrides.
Find the <<Platform features>>, <<stackablectl,stackablectl features>> and <<operators>> below.

Please report any issues you find in the specific operator repositories or in our dedicated github.com/stackabletech/issues/[issues] repository.
You may also join us in our https://slack.stackable.tech[Slack] community or https://stackable.tech[contact us via our homepage].

While we are very proud of this release it is our first one and we'll add new features and fix bugs all the time and will have regular releases from now on.

=== 22.6.0

==== Platform features

Easily install production ready data applications::

Using a familiar declarative approach, users can easily install data applications such as Apache Kafka or Trino across multiple cloud Kubernetes providers or on their own data centers.
The installation process is fully automated while also providing the flexibility for the user to tune relevant aspects of each application.

Monitoring::

All products have monitoring with prometheus enabled.
xref:operators:monitoring.adoc[Learn more]

Service discovery::

Products on the Stackable platform use service discovery to easily interconnect with each other.
xref:concepts:service_discovery.adoc[Learn more]

Configuration overrides::

All operators support configuration overrides, these are documented in the specific operator documentation pages.

Common S3 configuration::

Many products support connecting to S3 to load and/or store data.
There is a common resource for S3 connections and buckets across all operators that can be reused.
xref:concepts:s3.adoc[Learn more]

Roles and role groups::

To support hybrid hardware clusters, the Stackable platform uses the concept of role groups.
Services and applications can be configured to maximize hardware efficiency.

Standardized::

Learn once reuse everywhere.
We use the same conventions in all our operators.
Configure your LDAP or S3 connections once and reuse them everywhere.
All our operators reuse the same CRD structure as well.

[#stackablectl]
==== stackablectl

xref:management:stackablectl:index.adoc[stackablectl] is used to install and interact with the operators, either individually or with multiple at once.

[#operators]
==== Operators

This is the list of all operators in this current release, with their versions for this release.

.*Products*
* xref:airflow:index.adoc[] (0.4.0)
** Load DAGs from ConfigMaps or PersistentVolumeClaims
* xref:druid:index.adoc[] (0.6.0)
** S3 and HDFS as deep storage options
** ingestion from S3 buckets
** authorization using OPA
* xref:hbase:index.adoc[] (0.3.0)
* xref:hdfs:index.adoc[] (0.4.0)
* xref:hive:index.adoc[] (0.6.0)
** Hive Metastore can index S3
* xref:kafka:index.adoc[] (0.6.0)
** Seamless integration with NiFi and Druid
** Supports OPA authorization
* xref:nifi:index.adoc[] (0.6.0)
* xref:spark-k8s:index.adoc[] (0.3.0)
* xref:superset:index.adoc[] (0.5.0)
** connects to Druid as a backend
** Supports LDAP authentication
* xref:trino:index.adoc[] (0.4.0)
** Supports OPA and file-based authorization
** Connects to the Hive Metastore
** Query data from S3
** TLS support
* xref:zookeeper:index.adoc[] (0.10.0)
** Supports creating ZNodes with CRDs

Read up on the xref:operators:supported_versions.adoc[supported versions] for each of these products.

.*Supporting operators*
* xref:opa:index.adoc[] (0.9.0)
** Create RegoRules in ConfigMaps
* xref:commons-operator:index.adoc[] (0.2.0)
* xref:secret-operator:index.adoc[] (0.5.0)

==== Supported Kubernetes versions

This release supports the following Kubernetes versions:

* `1.23`
* `1.22`
* `1.21`
Loading
Loading