Custom Thread Pool for Reconciler Execution

[csviri]

Currently ThreadPoolExecutor is used to manage threads for the reconciliation.

Controllers are a little bit specific since at startup all the resources are reconciled, which can mean a large initial burst, while for the rest of the execution Thread utilization might be much smaller.

This situation might be covered relatively fine with a configuration for ThreadPoolExecutor that is created similar to the newCachedThreadPool
That provides a virtually unbound number of Threads, which are discarded from the pool if not used actively anymore. Since reconciliation is ideally fast (at least the majority on startup), the number of threads should never grow too big. Note that rejecting a task from the executor is not something that we can allow now.
On the other hand, it is not uncommon, that some controllers manage hundreds of thousands of resources and the reconciliation might not be that fast in all cases. So we would like to limit the number of threads even if that limit is quite high, and on startup have the rest of the resource queued and processed. Unfortunately, there is no such thing as, "queue the rest of incoming tasks" if the maximum thread count is reached in ThreadPoolExecutor.

So ideally the executor that we would need is very similar to the ThreadPool executor, but with an additional (maybe bounded) queue, that will queue up the tasks even if the maximum number of Threads is reached in the pool.

Notes:
see also: #2262

[csviri]

After discussing though, ideally, when reconciliation on startup is executed there are no I/O bound operations on the reconciliation. A fixed thread pool might just be good enough to do the job in general.

[shawkins]

A fixed thread pool might just be good enough to do the job in general.

@csviri @manusa as long users can elect to use virtual threads with newVirtualThreadPerTaskExecutor, this should be a good approach. Users are now supplying a virtual threads to the kubernetes client executor and the pool used by the http client, so doing that in the operator sdk should allow for end-to-end light-weight thread management.

Throttling could be a separate concern.