Strange behaviour when replacing one node/container

[varet80]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.8.2
• bare metal k8s
• no production
• recovery bug

I lost a server disk. And the problem began after my start of the container.
with postgres Dir empty the following error began with bootstraping of the new node

home-main-0 postgres 2023-02-24 16:18:50,420 INFO: Lock owner: home-main-1; I am home-main-0
home-main-0 postgres 2023-02-24 16:18:50,423 INFO: pg_controldata:
home-main-0 postgres   pg_control version number: 1300
home-main-0 postgres   Catalog version number: 202107181
home-main-0 postgres   Database system identifier: 7161777921842421829
home-main-0 postgres   Database cluster state: in production
home-main-0 postgres   pg_control last modified: Fri Feb 24 16:17:22 2023
home-main-0 postgres   Latest checkpoint location: 1F/220066D0
home-main-0 postgres   Latest checkpoint's REDO location: 1F/22000028
home-main-0 postgres   Latest checkpoint's REDO WAL file: 0000004D0000001F00000022
home-main-0 postgres   Latest checkpoint's TimeLineID: 77
home-main-0 postgres   Latest checkpoint's PrevTimeLineID: 77
home-main-0 postgres   Latest checkpoint's full_page_writes: on
home-main-0 postgres   Latest checkpoint's NextXID: 0:9642032
home-main-0 postgres   Latest checkpoint's NextOID: 3239657
home-main-0 postgres   Latest checkpoint's NextMultiXactId: 41
home-main-0 postgres   Latest checkpoint's NextMultiOffset: 81
home-main-0 postgres   Latest checkpoint's oldestXID: 727
home-main-0 postgres   Latest checkpoint's oldestXID's DB: 1
home-main-0 postgres   Latest checkpoint's oldestActiveXID: 9642032
home-main-0 postgres   Latest checkpoint's oldestMultiXid: 1
home-main-0 postgres   Latest checkpoint's oldestMulti's DB: 1
home-main-0 postgres   Latest checkpoint's oldestCommitTsXid: 0
home-main-0 postgres   Latest checkpoint's newestCommitTsXid: 0
home-main-0 postgres   Time of latest checkpoint: Fri Feb 24 16:17:12 2023
home-main-0 postgres   Fake LSN counter for unlogged rels: 0/3E8
home-main-0 postgres   Minimum recovery ending location: 0/0
home-main-0 postgres   Min recovery ending loc's timeline: 0
home-main-0 postgres   Backup start location: 0/0
home-main-0 postgres   Backup end location: 0/0
home-main-0 postgres   End-of-backup record required: no
home-main-0 postgres   wal_level setting: replica
home-main-0 postgres   wal_log_hints setting: on
home-main-0 postgres   max_connections setting: 100
home-main-0 postgres   max_worker_processes setting: 8
home-main-0 postgres   max_wal_senders setting: 10
home-main-0 postgres   max_prepared_xacts setting: 0
home-main-0 postgres   max_locks_per_xact setting: 64
home-main-0 postgres   track_commit_timestamp setting: off
home-main-0 postgres   Maximum data alignment: 8
home-main-0 postgres   Database block size: 8192
home-main-0 postgres   Blocks per segment of large relation: 131072
home-main-0 postgres   WAL block size: 8192
home-main-0 postgres   Bytes per WAL segment: 16777216
home-main-0 postgres   Maximum length of identifiers: 64
home-main-0 postgres   Maximum columns in an index: 32
home-main-0 postgres   Maximum size of a TOAST chunk: 1996
home-main-0 postgres   Size of a large-object chunk: 2048
home-main-0 postgres   Date/time type storage: 64-bit integers
home-main-0 postgres   Float8 argument passing: by value
home-main-0 postgres   Data page checksum version: 0
home-main-0 postgres   Mock authentication nonce: 1c78914c9157b0789779801bc48b77981
home-main-0 postgres
home-main-0 postgres 2023-02-24 16:18:50,432 INFO: doing crash recovery in a single user mode
home-main-0 postgres 2023-02-24 16:18:50,465 ERROR: Error when reading postmaster.opts
home-main-0 postgres Traceback (most recent call last):
home-main-0 postgres   File "/usr/local/lib/python3.10/dist-packages/patroni/postgresql/rewind.py", line 482, in read_postmaster_opts
home-main-0 postgres     with open(os.path.join(self._postgresql.data_dir, 'postmaster.opts')) as f:
home-main-0 postgres FileNotFoundError: [Errno 2] No such file or directory: '/home/postgres/pgdata/pgroot/data/postmaster.opts'
home-main-0 postgres 2023-02-24 16:18:50,481 ERROR: Crash recovery finished with code=1
home-main-0 postgres 2023-02-24 16:18:50,481 INFO:  stdout=
home-main-0 postgres 2023-02-24 16:18:50,481 INFO:  stderr=2023-02-24 16:18:50 UTC [336]: [1-1] 63f8e36a.150 0     FATAL:  database files are incompatible with server
home-main-0 postgres 2023-02-24 16:18:50 UTC [336]: [2-1] 63f8e36a.150 0     DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 15.1 (Ubuntu 15.1-1.pgdg22.04+1).

[Jan-M]

Disk lost might look strange Patroni, probably does not do reinit alone.

[varet80]

It does reinit. but fails due to version difference as mentioned there.
I see data in the disk.

[varet80]

I fixed it by executing the inplace upgrade! seems like the major ugprade did not ugprade the DB @Jan-M

Now I face another small issue

pg-srv-0 postgres 2023-02-25 09:46:31,236 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
pg-srv-0 postgres 2023-02-25 09:46:33,243 - bootstrapping - INFO - Could not connect to 169.254.169.254, assuming local Docker setup
pg-srv-0 postgres 2023-02-25 09:46:33,245 - bootstrapping - INFO - No meta-data available for this provider
pg-srv-0 postgres 2023-02-25 09:46:33,246 - bootstrapping - INFO - Looks like you are running local
pg-srv-0 postgres /scripts/configure_spilo.py:1056: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
pg-srv-0 postgres   config = yaml.load(pystache_render(TEMPLATE, placeholders))
pg-srv-0 postgres /scripts/configure_spilo.py:1059: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
pg-srv-0 postgres   user_config = yaml.load(os.environ.get('SPILO_CONFIGURATION', os.environ.get('PATRONI_CONFIGURATION', ''))) or {}
pg-srv-0 postgres 2023-02-25 09:46:33,288 - bootstrapping - INFO - Configuring pam-oauth2
pg-srv-0 postgres 2023-02-25 09:46:33,288 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
pg-srv-0 postgres 2023-02-25 09:46:33,288 - bootstrapping - INFO - Configuring patroni
pg-srv-0 postgres 2023-02-25 09:46:33,305 - bootstrapping - INFO - Writing to file /run/postgres.yml
pg-srv-0 postgres 2023-02-25 09:46:33,305 - bootstrapping - INFO - Configuring certificate
pg-srv-0 postgres 2023-02-25 09:46:33,305 - bootstrapping - INFO - Generating ssl self-signed certificate
pg-srv-0 postgres 2023-02-25 09:46:33,559 - bootstrapping - INFO - Configuring standby-cluster
pg-srv-0 postgres 2023-02-25 09:46:33,559 - bootstrapping - INFO - Configuring pgqd
pg-srv-0 postgres 2023-02-25 09:46:33,559 - bootstrapping - INFO - Configuring log
pg-srv-0 postgres 2023-02-25 09:46:33,559 - bootstrapping - INFO - Configuring wal-e
pg-srv-0 postgres 2023-02-25 09:46:33,560 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_S3_PREFIX
pg-srv-0 postgres 2023-02-25 09:46:33,560 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_S3_PREFIX
pg-srv-0 postgres 2023-02-25 09:46:33,561 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_ACCESS_KEY_ID
pg-srv-0 postgres 2023-02-25 09:46:33,561 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
pg-srv-0 postgres 2023-02-25 09:46:33,561 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_S3_ENDPOINT
pg-srv-0 postgres 2023-02-25 09:46:33,562 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_ENDPOINT
pg-srv-0 postgres 2023-02-25 09:46:33,562 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_DISABLE_S3_SSE
pg-srv-0 postgres 2023-02-25 09:46:33,562 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_DISABLE_S3_SSE
pg-srv-0 postgres 2023-02-25 09:46:33,562 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_S3_FORCE_PATH_STYLE
pg-srv-0 postgres 2023-02-25 09:46:33,563 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_DOWNLOAD_CONCURRENCY
pg-srv-0 postgres 2023-02-25 09:46:33,563 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_UPLOAD_CONCURRENCY
pg-srv-0 postgres 2023-02-25 09:46:33,563 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_LOG_DESTINATION
pg-srv-0 postgres 2023-02-25 09:46:33,564 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/PGPORT
pg-srv-0 postgres 2023-02-25 09:46:33,564 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/BACKUP_NUM_TO_RETAIN
pg-srv-0 postgres 2023-02-25 09:46:33,565 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/TMPDIR
pg-srv-0 postgres 2023-02-25 09:46:33,565 - bootstrapping - INFO - Configuring bootstrap
pg-srv-0 postgres 2023-02-25 09:46:33,565 - bootstrapping - INFO - Configuring crontab
pg-srv-0 postgres 2023-02-25 09:46:33,565 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
pg-srv-0 postgres /scripts/configure_spilo.py:1000: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
pg-srv-0 postgres   lines += yaml.load(placeholders['CRONTAB'])
pg-srv-0 postgres 2023-02-25 09:46:33,571 - bootstrapping - INFO - Configuring pgbouncer
pg-srv-0 postgres 2023-02-25 09:46:33,571 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
pg-srv-0 postgres 2023-02-25 09:46:34,839 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'.
pg-srv-0 postgres 2023-02-25 09:46:34,900 INFO: No PostgreSQL configuration items changed, nothing to reload.
pg-srv-0 postgres 2023-02-25 09:46:34,903 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:46:35,001 INFO: trying to bootstrap from leader 'pg-srv-1'
pg-srv-0 postgres ERROR:  permission denied for database n**********d
pg-srv-0 postgres ERROR:  permission denied for database n**********d
pg-srv-0 postgres ERROR:  permission denied for database n**********d
pg-srv-0 postgres could not determine difference with the master location
pg-srv-0 postgres 2023-02-25 09:46:37,857 ERROR: Error creating replica using method wal_e: envdir /run/etc/wal-e.d/env bash /scripts/wale_restore.sh exited with code=1
pg-srv-0 postgres 2023-02-25 09:46:39,214 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:46:39,280 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres 1024+0 records in
pg-srv-0 postgres 1024+0 records out
pg-srv-0 postgres 16777216 bytes (17 MB, 16 MiB) copied, 0.0137137 s, 1.2 GB/s
pg-srv-0 postgres 2023-02-25 09:46:49,213 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:46:49,213 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres 2023-02-25 09:46:59,208 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:46:59,209 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres 2023-02-25 09:47:09,209 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:47:09,209 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres 2023-02-25 09:47:19,209 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:47:19,209 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres 2023-02-25 09:47:29,428 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:47:29,428 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres 2023-02-25 09:47:39,210 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:47:39,210 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres 2023-02-25 09:47:49,208 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:47:49,209 INFO: bootstrap from leader 'pg-srv-1' in progress
pg-srv-0 postgres NOTICE:  all required WAL segments have been archived
pg-srv-0 postgres 2023-02-25 09:47:53,199 INFO: replica has been created using basebackup_fast_xlog
pg-srv-0 postgres 2023-02-25 09:47:53,201 INFO: bootstrapped from leader 'pg-srv-1'
pg-srv-0 postgres 2023-02-25 09:47:53 UTC [256]: [1-1] 63f9d949.100 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
pg-srv-0 postgres 2023-02-25 09:47:53 UTC [256]: [2-1] 63f9d949.100 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
pg-srv-0 postgres 2023-02-25 09:47:53,412 INFO: postmaster pid=256
pg-srv-0 postgres /var/run/postgresql:5432 - no response
pg-srv-0 postgres 2023-02-25 09:47:53 UTC [256]: [3-1] 63f9d949.100 0     LOG:  redirecting log output to logging collector process
pg-srv-0 postgres 2023-02-25 09:47:53 UTC [256]: [4-1] 63f9d949.100 0     HINT:  Future log output will appear in directory "../pg_log".
pg-srv-0 postgres /var/run/postgresql:5432 - accepting connections
pg-srv-0 postgres /var/run/postgresql:5432 - accepting connections
pg-srv-0 postgres 2023-02-25 09:47:54,484 INFO: Lock owner: pg-srv-1; I am pg-srv-0
pg-srv-0 postgres 2023-02-25 09:47:54,485 INFO: establishing a new patroni connection to the postgres cluster
pg-srv-0 postgres 2023-02-25 09:47:54,576 INFO: no action. I am (pg-srv-0), a secondary, and following a leader (pg-srv-1)
pg-srv-0 postgres 2023-02-25 09:47:59,298 INFO: no action. I am (pg-srv-0), a secondary, and following a leader (pg-srv-1)
pg-srv-0 postgres 2023-02-25 09:48:09,290 INFO: no action. I am (pg-srv-0), a secondary, and following a leader (pg-srv-1)

an error denied in specific DB