Is there a way to send the logs from the database container in this directory: /pgdata/<db_name>/pg_log to k8s

[alrooney]

pgo 4.5.0

Often I have to exec into the db container to look at logs because the interesting logs are in /pgdata/<db_name>/pg_log/ directory. Is there any way to send those to k8s so that kubectl logs -f <db_pod> -c database would show them? Also looking at collecting them with our fluentd which we are already doing for logs visible via the logs kubectl command.

[jkatz]

There's an older example here: https://access.crunchydata.com/documentation/crunchy-postgres-containers/latest/examples/logging/centralized-logging/ but the principle idea is that you need to redirect the PostgreSQL logs to stderr and turn off the logging collector.

Taking a fresh look at how we handle the application logs has been on my TODO for awhile. The actual change is not challenging (see my comment above), but the bigger question is around what recommendations we make around log aggregation, if any at all.

Perhaps to help:

• What's your typical log retention policy and/or what log retention policy are you seeking?
• Do you archive the logs past a certain date?
• I know fluentd performs the aggregations, but what do you use fr visualization?

[alrooney]

• Our retention policy is 7 days typically
• We typically just discard old logs
• We send the logs to an elk stack and use kibana for the visualizations/searching

[QuingKhaos]

Adding ours:

• Log retentions are 7 days on internal OpenShift Cluster Logging, can be longer if send to external Elasticsearch or other SIEM
• Logging system prunes older indices.
• Kibana for Visualization

Personally, having PGO configured logging into the filesystem is a huge kubernetes antipattern, as by default you expect logs on stdout/stderr and to be collected by e.g. fluentd.