The pgBackRest pod is not scheduled on another node ?

[khalifaould]

In the event that the node is lost or the pgbackrest pod is running, the pod is not rescheduled on one of the other available nodes and it is no longer possible to make a backup or restore.
Is it normal ?
Is it possible to have several replicas of this pod as is the case for the base pods?

[jkatz]

There is not enough information here to begin troubleshooting. Information that is helpful for this can be found here: https://github.com/CrunchyData/postgres-operator/issues/new?template=bug_report.md

[khalifaould]

As part of our resilience tests we have a cluster with 3 master nodes on which we launched a crunchydata cluster with 3 replicas.
The list of pods

We simulate the loss of the node on which the pgbackrest (dai-prod-db-repo-host-0) is launched.

We note that this pod is not rescheduled on one of the other available nodes

In this case we can no longer make a backup or restore?

Is it possible to have several replicas of this pod as is the case for the base pods?

[khalifaould]

Cluster Config 

`apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: db-cluster
  namespace: namespace
spec:
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.4-1
  postgresVersion: 13
  instances:
    - name: instance1
      replicas: 3
      dataVolumeClaimSpec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 3Gi
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: kubernetes.io/hostname
            labelSelector:
              matchLabels:
                postgres-operator.crunchydata.com/cluster: db-cluster
                postgres-operator.crunchydata.com/instance-set: instance1
  users:
    - name: user
      databases:
        - db_name
      options: "SUPERUSER"
  databaseInitSQL:
    key: create-schema.sql
    name: create-schema
  backups:
    pgbackrest:
      global:
        repo1-retention-full: "5"
      manual:
        repoName: repo1
        options:
          - --type=full
      image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.35-0
      repos:
        - name: repo1
          schedules:
            full: "0 0 * * *"
          volume:      
            volumeClaimSpec:
              accessModes:
                - "ReadWriteOnce"
              resources:
                requests:
                  storage: 3Gi`

[andrewlecuyer]

@khalifaould does the repo Pod that cannot be rescheduled get stuck in a "pending" status? If so, can you provide the output of describing the unschedulable repo Pod via kubectl describe?

[khalifaould]

The pod is not reprogrammed on one of the other nodes and that's the problem !!
Looks like there is an affinity with the node it first launched on

[andrewlecuyer]

Even when Kubernetes is unable to schedule a Pod, it is still possible to describe/view/etc. the Pod (which should be in a "pending" status). Therefore, please provide the output of describing the repo Pod, e.g. kubectl describe pod dai-prod-db-repo-host-0.

Additionally, please provide the following details as requested so that we can properly troubleshoot the issue per your specific deployment:

• Platform: (Kubernetes, OpenShift, Rancher, GKE, EKS, AKS etc.)
• Platform Version: (e.g. 1.20.3, 4.7.0)
• PGO Image Tag: (e.g. ubi8-5.1.0-0)
• Postgres Version (e.g. 14)
• Storage: (e.g. hostpath, nfs, or the name of your storage class)

My thinking here is that the volume being utilized by the repo is in a specific zone where is only available to the node that is being terminated (i.e. the other two nodes are in other zones, and therefore unable to mount that same PVC).

[khalifaould]

Platform: Rancher K3S
Platform Version: v1.19.7+k3s1
PGO Image Tag: ubi8-5.0.3-0
Postgres Version : 13
Storage: Default( local-path)

The dai-prod-db-repo-host-0 pod is not status pending but Terminating.
kubectl describe pod dai-prod-db-repo-host-0

Name: dai-prod-db-repo-host-0
Namespace: namespace
Priority: 0
Node: ip-172-16-60-231.eu-west-1.compute.internal/172.16.60.231
Start Time: Thu, 27 Jan 2022 16:32:26 +0000
Labels: controller-revision-hash=dai-prod-db-repo-host-768c6b8559
postgres-operator.crunchydata.com/cluster=dai-prod-db
postgres-operator.crunchydata.com/data=pgbackrest
postgres-operator.crunchydata.com/pgbackrest=
postgres-operator.crunchydata.com/pgbackrest-dedicated=
statefulset.kubernetes.io/pod-name=dai-prod-db-repo-host-0
Annotations:
Status: Terminating (lasts 141m)
Termination Grace Period: 30s
IP: 10.42.1.247
IPs:
IP: 10.42.1.247
Controlled By: StatefulSet/dai-prod-db-repo-host
Init Containers:
nss-wrapper-init:
Container ID: docker://f633347a36fa16b4380ab037e8f15178df998c2d573f50b6412bdd6e8b5d8c2a
Image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.35-0
Image ID: docker-pullable://registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest@sha256:e70691aa7c4913e2c353bfe7e3779e1774cc6320be10fb1d6f32de8a10510669
Port:
Host Port:
Command:
bash
-c
NSS_WRAPPER_SUBDIR=postgres CRUNCHY_NSS_USERNAME=postgres CRUNCHY_NSS_USER_DESC="postgres" /opt/crunchy/bin/nss_wrapper.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 27 Jan 2022 16:32:27 +0000
Finished: Thu, 27 Jan 2022 16:32:27 +0000
Ready: True
Restart Count: 0
Environment:
Mounts:
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dswpn (ro)
Containers:
pgbackrest:
Container ID: docker://fbbaeb11373840b2a0f93d02749d37ef0bae1db5253ece4f8794215d65536588
Image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.35-0
Image ID: docker-pullable://registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest@sha256:e70691aa7c4913e2c353bfe7e3779e1774cc6320be10fb1d6f32de8a10510669
Port:
Host Port:
Command:
/usr/sbin/sshd
-D
-e
State: Running
Started: Thu, 27 Jan 2022 16:32:28 +0000
Ready: True
Restart Count: 0
Liveness: tcp-socket :2022 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
LD_PRELOAD: /usr/lib64/libnss_wrapper.so
NSS_WRAPPER_PASSWD: /tmp/nss_wrapper/postgres/passwd
NSS_WRAPPER_GROUP: /tmp/nss_wrapper/postgres/group
Mounts:
/etc/pgbackrest/conf.d from pgbackrest-config (rw)
/etc/ssh from ssh (ro)
/pgbackrest/repo1 from repo1 (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dswpn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
ssh:
Type: Projected (a volume that contains injected data from multiple sources)
ConfigMapName: dai-prod-db-ssh-config
ConfigMapOptional:
SecretName: dai-prod-db-ssh
SecretOptionalName:
repo1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: dai-prod-db-repo1
ReadOnly: false
pgbackrest-config:
Type: Projected (a volume that contains injected data from multiple sources)
ConfigMapName: dai-prod-db-pgbackrest-config
ConfigMapOptional:
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: 16Mi
default-token-dswpn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dswpn
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:

[jansmets]

My issue might be related.
We had a (planned) power outage (executed ahead of schedule) which brought down all nodes. For some reason the repo-host-0 pod remained in a Terminating state once everything came back online.

At that point the WAL kept on filling up until the disk ran full...