Failed to migrate databases to pgo v5 with the same name but in different namespaces

[radoslaw505]

Overview

We have several namespaces with version 4 pgo clusters. Some of them have the same name.
For example, let it be 2 namespaces - sample1 and sample2 in which a pgo cluster named pgo-test-cluster exists.
When I migrate the first cluster pgo-test-cluster in namsespace sample1 with the instruction: www, the pod * -move- * is properly created and the cluster is initialized.
When migrating second cluster pgo-test-cluster from namespace sample2 the pod's * -move- * DOES not been created, which leads to an error:
2022-02-24 13:11:41,659 INFO: Running custom bootstrap script: mv "/pgdata/pg13_bootstrap" "/pgdata/pg13"
mv: cannot stat '/pgdata/pg13_bootstrap': No such file or directory
2022-02-24 13:11:41,666 INFO: removing initialize key after failed attempt to bootstrap the cluster
and pod is in crash loop.

Environment

• Platform: OpenShift
• Platform Version: 4.8.29
• PGO Image Tag: ubi8-13.5-0
• Postgres Version: 13
• Storage: CEPH

Steps to Reproduce

REPRO

pgo delete cluster pgo-test-cluster --keep-data --keep-backups -n sample2

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: pgo-test-cluster
spec:
  dataSource:
    volumes:
      pgDataVolume:
        pvcName: pgo-test-cluster
        directory: "pgo-test-cluster"
      pgBackRestVolume:
        pvcName: pgo-test-cluster-pgbr-repo
        directory: "pgo-test-cluster-backrest-shared-repo"
  image: registry.connect.redhat.com/crunchydata/crunchy-postgres:ubi8-13.5-0
  postgresVersion: 13 
  instances:
    - name: instance1
      replicas: 1
      resources:
        requests:
          cpu: 100m  
          memory: 256Mi
        limits:
          cpu: 700m
          memory: 512Mi
      dataVolumeClaimSpec:
        accessModes:
        - "ReadWriteOnce"
        resources:
          requests:
            storage: 1Gi  
  monitoring:
    pgmonitor:
      exporter:
        image: registry.connect.redhat.com/crunchydata/crunchy-postgres-exporter
  backups:
    pgbackrest:
      image: registry.connect.redhat.com/crunchydata/crunchy-pgbackrest:ubi8-2.36-0
      repos:
      - name: repo1
        schedules:
          full: "10 3 * * *"
          incremental: "10 */2 * * *"
        volume:
          volumeClaimSpec:
            accessModes:
            - "ReadWriteOnce"
            resources:
              requests:
                storage: 1G
EOF

EXPECTED

Pods with postgres database are Running. Databases exist regardless of name if they are in different namespaces.

ACTUAL

1. Describe what actually happens

Pod's * -move- * DOES not been created. Pod with postgres instance had the same name like previous from other namespace.

Pod is in crash loop:

2022-02-24 13:11:41,555 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-02-24 13:11:41,582 INFO: Lock owner: None; I am pgo-test-cluster-instance1-776c-0
2022-02-24 13:11:41,659 INFO: trying to bootstrap a new cluster
2022-02-24 13:11:41,659 INFO: Running custom bootstrap script: mv "/pgdata/pg13_bootstrap" "/pgdata/pg13"
mv: cannot stat '/pgdata/pg13_bootstrap': No such file or directory
2022-02-24 13:11:41,666 INFO: removing initialize key after failed attempt to bootstrap the cluster
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 171, in main
    return patroni_main()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 139, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 109, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 112, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1471, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1345, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1238, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1231, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'

[radoslaw505]

Migrating using the backup method also works in a similar way. The first cluster is migrated, the second one has the above-mentioned error. The database migration solution is not supported in environments with a large number of namespaces that have clusters with the same names.
In addition, the pods that are created when migrating the next database have the same name as the previous one. For instance:

• the first pod when creating a cluster psql-sample in namespace psql-test is named psql-sample-instance1-788f-0
• the second pod when creating a cluster psql-sample in namespace psql-test2 has the name psql-sample-instance1-788f-0