Separate Executor Service For Workflows

[csviri]

Currently an executor service is shared between the reconciliations but also used in workflows to execute separate steps.

In some cases this might lead to a dead-lock (not seen it / no evidence, just argumentation):

Let's assume the executor service is limited 2 threads. There are two custom resources to reconcile. Reconciler is using managed workflows. Both are submitted and executed. The workflow executor will be triggered, and will submit the first workflow action, however both threads are now in use by the reconciler, and never will be available since waiting the workflow to complete it's steps.

The way to solve this is to separate the executor service of workflows from the reconcilers (so have a dedicated one). That will make sure there are always threads available for the workflows.

Possible Heuristics for automatic pool size:

The thread pool size for the workflows should be actually calculated like: [thread pool size for reconciler] * [max parallelism in managed workflow?]

Theoretical Maximum parallelism is not easy calculate in a workflow. Maybe not even ideal. Maybe a the maximum dependents for a single resource might be good enough. (Also this should be configurable).

Note:

The executor service should be of course created when

[csviri]

Checked it, now it actually creates a separate executor service, but this is more like a bug than a feature. When tested with this update that fixes that problem: a2459dc

With 1 thread, web page reconciler actually deadlocks.

So this is not a real problem now, but would be nicer to have a separate executor service.