How to set PATRONI_LOG_LEVEL environment variable with PGO 5.1.2?

[johnswarbrick]

I want to set environment variable PATRONI_LOG_LEVEL on all PostgreSQL pods created by the Crunchydata PGO 5.1.2 on Kubernetes.

If I edit the pod directly and add the following it works:

- name: PATRONI_LOG_LEVEL
  value: WARNING

Now I want to add this into the Helm chart so it's automatically applied but despite trying every possible option I just couldn't find the right method of defining the environment variable in the Helm chart so it would be passed down to the individual pods created by the PGO.

Hoping someone out there knows the answer to what should be a simple problem!!

[cbandy]

There's no way to change the Patroni log level at this time. Are you trying to affect the patronictl CLI or the patroni server?

Note that failovers are logged at the INFO level by patroni.

[David-Angel]

We would like this ability also. 👍

[cbandy]

@David-Angel Do you want to affect the patronictl CLI or the patroni server?

[David-Angel]

Specifically we need to be able to set the PATRONI_LOG_FORMAT, PATRONI_LOG_LEVEL, and very likely other environment variables via crunchydata cluster config.

The log format is particularly critical because of how all logs output to the same location from a Kubernetes pod.
We need to prefix them to tell the difference between postgres, container scripts, patroni, and anything else within the pod.

[cbandy]

The log format is particularly critical because of how all logs output to the same location from a Kubernetes pod.
We need to prefix them to tell the difference between postgres, container scripts, patroni, and anything else within the pod.

I'm sorry; I don't understand.

Kubernetes captures the stdout and stderr of every container separately. The kubectl logs command shows the combined stdout and stderr of a single container.

What tool are you using to view logs? Does it not indicate the origin of log lines?

[David-Angel]

here is an example of the problem

kubectl logs $(kubectl get pods -l='postgres-operator.crunchydata.com/role=master' --no-headers -o name)

You can see we can prefix "postgres" log entries but postgres is logging to the same place as other applications in the container. The container contains more than one application including the container startup code.

[cbandy]

I see. Patroni made that entry. Have you disabled the PostgreSQL logging_collector?

$ psql -c 'SHOW logging_collector'
 logging_collector
-------------------
 on
(1 row)

PGO does not do that because PostgreSQL is multi-process and does not recommend it:

Note
It is possible to log to stderr without using the logging collector; the log messages will just go to wherever the server's stderr is directed. However, that method is only suitable for low log volumes… [this] can result in lost or garbled log output, because multiple processes writing concurrently to the same log file can overwrite each other's output.

[David-Angel]

The logging is working fine for PostgreSQL
Disabling the logging_collector wouldn't help here.
They are supposed to both be logging in to the same place per Kubernetes design.
We just need to prefix Patroni log entries to tell them apart.
Along with any log entries happening by the container startup code.

[cbandy]

The logging is working fine for PostgreSQL Disabling the logging_collector wouldn't help here.

I'm not suggesting you disable it. I believe the reason PostgreSQL logs are mixing with Patroni logs is because it is disabled. If you're able, can you check that for me, please?

They are supposed to both be logging in to the same place per Kubernetes design. We just need to prefix Patroni log entries to tell them apart.

Understood.

Along with any log entries happening by the container startup code.

Yep. There's no startup code, so this is done!

[David-Angel]

For this:
"I believe the reason PostgreSQL logs are mixing with Patroni logs is because it is disabled. If you're able, can you check that for me, please?"

They are supposed to both be going to the same place.
If they did not do that you would not be able to get the logs from the Kubernetes pod.
There is only one place they can both send logs for the pod to pick it up.

It is not just Patroni and PostgreSQL the container also has startup logs going to the same place.
All applications within any container only have 1 place they can send logs for the Kubernetes pod to pick up.

I can force it to log to a different place but that would prevent Kubernetes from showing them at all.

[johnswarbrick]

Hi @cbandy, I think this ticket may have drifted from the original issue.

I was looking for a way of way of setting the patroni log levels to something other than the default INFO, for example WARN, using the Crunchydata PGO.

I don't need to redirect the destination of the logs, the default stdout and stderror are fine.

Just want to turn down the verbosity to make it easier to spot genuine errors/warning events.

Thanks :)

[howels]

To chime in here - it appears that several options set in the Patroni: DynamicConfiguration section are simply not passed to Patroni. We would like to be able to set the Patroni log format such that log lines from Patroni carry a "PATRONI: " prefix and set other options also. These settings are being dropped.

[cbandy]

To chime in here - it appears that several options set in the Patroni: DynamicConfiguration section are simply not passed to Patroni. We would like to be able to set the Patroni log format such that log lines from Patroni carry a "PATRONI: " prefix and set other options also. These settings are being dropped.

That section goes into the "dynamic configuration settings" described here. Patroni does not accept any other settings (including log settings) there.

[howels]

To chime in here - it appears that several options set in the Patroni: DynamicConfiguration section are simply not passed to Patroni. We would like to be able to set the Patroni log format such that log lines from Patroni carry a "PATRONI: " prefix and set other options also. These settings are being dropped.

That section goes into the "dynamic configuration settings" described here. Patroni does not accept any other settings (including log settings) there.

Where can the Patroni Log settings be entered?