Document operator system requirements

[maltesander]

This is part of #247. We want to know the minimal and recommended requirements in terms of CPU, memory and disk space for all operators. This should be documented (versioned) for each release.

This includes research / testing to find the minimal/recommended requirements to run the operator. What kind of load (e.g. reconciling 1000 clusters) can it withstand and what are the CPU and memory spikes under load.

This should be proven for local clusters like e.g. kind and managed Kubernetes. An approximation like 1 core and 1 GB of memory is sufficient.

Acceptance

• There exists a new page in the docs documenting the minimal/recommended requirements for each operator
• This must include a table containing the operator name, minimal/recommended CPU / memory / disk consumption as well as (if available) recommendations for optimal performance etc.

Products

Tasks

• Airflow : [Merged by Bors] - Operator requirements airflow-operator#204
• Druid : [Merged by Bors] - Operator requirements druid-operator#354
• HBase : [Merged by Bors] - Operator requirements hbase-operator#284
• Hadoop HDFS : [Merged by Bors] - Operator requirements hdfs-operator#283
• Hive : [Merged by Bors] - Operator requirements hive-operator#278
• Kafka : [Merged by Bors] - Operator requirements kafka-operator#525
• NiFi : [Merged by Bors] - Operator requirements nifi-operator#392
• Spark : [Merged by Bors] - Operator requirements spark-k8s-operator#184
• Superset : [Merged by Bors] - Operator requirements superset-operator#301
• Trino : [Merged by Bors] - Operator requirements trino-operator#354
• ZooKeeper : Document operator system requirements zookeeper-operator#559

Internal

Tasks

• Commons : [Merged by Bors] - Operator requirements commons-operator#121
• Listener : [Merged by Bors] - Operator requirements listener-operator#37
• OPA : [Merged by Bors] - Operator requirements opa-operator#383
• Secret : [Merged by Bors] - Operator requirements secret-operator#218

Documentation links

• Airflow
• Druid
• HBase
• HDFS
• Hive
• Kafka
• NiFi
• Spark
• Superset
• ZooKeeper
• OPA
• Commons
• Secret
• Listener

[nightkr]

What do you mean by min/max ranges here.. Absolute minimum vs recommended?

Especially CPU is also going to be really difficult to measure objectively. Not to mention that all of this depends heavily on the scale of the cluster.

[maltesander]

Yeah i changed max to recommended.
I know its hard to measure, we just need minimal requirements (i think memory is more important here) that make sure for us the operator runs properly.
If we can get recommended values e.g. if your cluster is size x your should provide y resources. Or if you run 100 clusters of size x you should provide y resources.
The minimal requirements (e.g. assumption one cluster of size x) are more important as well (for liability).
For CPU i think we can use whatever CPU is e.g. on your laptop, in AWS / Azure etc. and give recommendations on what worked?

[lfrancke]

It is fine to "eyeball" this. We don't need it down to the Megabyte.
It is probably fine to say something along these lines: `We recommend at least 1Gb of RAM and 1 CPU core to run this operator"

The testing can be minimal. It would be good to understand if there is much queueing in kube-rs and if so if the queue size is configurable but don't spend too much time on this issue

[lfrancke]

FYI: It might very well be that this issue is already fully refined and all we need is issues for each operator. I can create those issues as soon as someone tells me that this is ready to go.

But I might also be wrong and some more refinement work is needed.