postgres-operator health check fails and results in - psql: could not connect to server: No such file or directory

[nchiluku]

1. The problem is reproducible on single master as well as multi master Kubernetes Cluster. Kubernetes installed successfully.
   kubectl get nodes
   NAME STATUS ROLES AGE VERSION
   master01 Ready master 6d v1.10.4
   worker01 Ready node 6d v1.10.4
   worker02 Ready node 6d v1.10.4
2. I tried setting postgres-operator using https://crunchydata.github.io/postgres-operator/stable/installation/manual-installation/.
   a) Defined variables in .bashrc. Not repeating them here.
   b) mkdir -p $HOME/odev/src $HOME/odev/bin $HOME/odev/pkg $GOPATH/src/github.com/crunchydata/
   cd $GOPATH/src/github.com/crunchydata
   git clone https://github.com/CrunchyData/postgres-operator.git
   cd postgres-operator
   git checkout 3.3.0
   c) Created the namespace demo
   d) Create HostPath Directory
   sudo mkdir /data
   sudo chmod 777 /data
   e) Create some sample Persistent Volumes using the following script:
   $COROOT/pv/create-pv.sh
   d) Deployed using build images
   make pull
   cd $HOME
   tar xvzf ./postgres-operator.3.3.tar.gz
   copy pgo client to somewhere in your path (e.g. cp pgo /usr/local/bin)
   cd $COROOT
   make installrbac
   make deployoperator
   e) postgres-operator ran successfully.
   kubectl get pod --selector=name=postgres-operator
   NAME READY STATUS RESTARTS AGE
   postgres-operator-77f949bf58-lv8j6 2/2 Running 0 2d
   f) kubectl get crd
   NAME AGE
   pgbackups.cr.client-go.k8s.io 3d
   pgclusters.cr.client-go.k8s.io 3d
   pgingests.cr.client-go.k8s.io 3d
   pgpolicies.cr.client-go.k8s.io 3d
   pgreplicas.cr.client-go.k8s.io 3d
   pgtasks.cr.client-go.k8s.io 3d
   pgupgrades.cr.client-go.k8s.io 3d
   g) kubectl get service postgres-operator
   NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
   postgres-operator ClusterIP 10.233.10.224 8443/TCP 2d
   export CO_APISERVER_URL=https://10.233.10.224:8443
   h)pgo version
   pgo client version 3.3.0
   apiserver version 3.3.0
   i) pgo create cluster mycluster --pgpool
   NAME READY STATUS RESTARTS AGE
   pg-test-664467cc8d-jqfs8 0/1 Running 0 7h
   pg-test-pgpool-64547c847f-fbfsl 1/1 Running 0 1d
   postgres-operator-77f949bf58-lv8j6 2/2 Running 0 1d
   j) kubectl describe pod podname
   Events:
   Type Reason Age From Message

---

Warning Unhealthy 2m (x2608 over 7h) kubelet, dev2-worker01 (combined from similar events): Readiness probe failed: Mon Oct 15 16:48:36 UTC 2018 INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
3) Added below to postgres-operator/examples/custom-config/pg_hba.conf file and reinstalled the cluster. The behavior did not change.
hostnossl all all 0.0.0.0/0 trust
host all all 0.0.0.0/0 trust
SAME error
Warning Unhealthy 2m (x2608 over 7h) kubelet, dev2-worker01 (combined from similar events): Readiness probe failed: Mon Oct 15 16:48:36 UTC 2018 INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
4) Tried 2.6.0 version too.
Same issue, encountered same health check issues.
psql -h 127.0.0.1 -U testuser postgres
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
5) After port forwarding tried again:
psql -h 127.0.0.1 -U postgres postgres
psql: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
6) kubectl exec pg-test-664467cc8d-jqfs8 -it /bin/bash
bash-4.2$ psql -h localhost
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?

• [x ] bug report
• feature request
• support request

Do you want to request a feature or report a bug?
Bug report

What is the current behavior?
kubectl exec -it pg-test-664467cc8d-lrmnt /bin/bash
bash-4.2$ psql -h localhost
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?

If the current behavior is a bug, please provide the steps to reproduce:
Please see above.

What is the expected behavior?
Should connect to postgres using psql

What is the motivation or use case for changing the behavior?
None

Other information (e.g. detailed explanation, related issues, suggestions how to fix, etc)
Please see above.

Please tell us about your environment:

• Operating

System:
Centos7

• Container

Image Tag:
Can reproduce on 2.7, 3.3.0 and 3.2.0

• PostgreSQL

Version:
Tried 9 and 10

• Platform

(Docker, Kubernetes, OpenShift):
kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Platform

Version:
uname -a
Linux 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[jmccormick2001]

can you post me the pod log from one of the failing PG pods? thanks. also, what storage type are you using? show me the output from 'pgo show config' as well.

[nchiluku]

Pod description
kubectl describe pod pg-test-664467cc8d-lkfch
Name: pg-test-664467cc8d-lkfch
Namespace: demo
Node: worker01/xx.xx.xx.xxx
Start Time: xxx, xx xxx 2018 10:44:00 -0700
Labels: archive=false
archive-timeout=60
crunchy-pgbadger=false
crunchy-pgpool=true
crunchy_collect=false
name=pg-test
pg-cluster=pg-test
pgo-backrest=false
pgo-version=3.3.0
pgpool-secret=
pod-template-hash=2200237748
primary=true
vendor=crunchydata
Annotations:
Status: Running
IP: xx.xx.xx.xxx
Controlled By: ReplicaSet/pg-test-664467cc8d
Containers:
database:
Container ID: docker://a1cb06591e6324701dd577d55a3a9c4792fec8565fe0ff3cf4fffc4ccc3a9977
Image: crunchydata/crunchy-postgres:centos7-10.5-2.1.0
Image ID: docker-pullable://crunchydata/crunchy-postgres@sha256:62bf6c93daf30acfaeb8810b4b3ab92d8adbea92a1b8dad325015830957201af
Port: 5432/TCP
Host Port: 0/TCP
State: Running
Started: xxxxxxx
Ready: False
Restart Count: 0
Readiness: exec [/opt/cpm/bin/readiness.sh] delay=15s timeout=8s period=10s #success=1 #failure=3
Environment:
PG_PRIMARY_PORT: 5432
PG_MODE: primary
PG_PRIMARY_HOST: pg-test
PG_LOCALE: en_US.UTF8
BACKUP_PATH:
PGDATA_PATH_OVERRIDE: pg-test
PG_DATABASE: userdb
PGxxxITOR_PASSWORD: password
ARCHIVE_MODE: off
ARCHIVE_TIMEOUT: 60
PGHOST: /tmp
Mounts:
/backrestrepo from backrestrepo-volume (rw)
/backup from backup (ro)
/pgconf from pgconf-volume (rw)
/pgdata from pgdata (rw)
/pgprimary from primary-volume (rw)
/pgroot from root-volume (rw)
/pguser from user-volume (rw)
/pgwal from pgwal-volume (rw)
/recover from recover-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-zxrlg (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
pgdata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pg-test
ReadOnly: false
backup:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
user-volume:
Type: Secret (a volume populated by a Secret)
SecretName: pg-test-testuser-secret
Optional: false
primary-volume:
Type: Secret (a volume populated by a Secret)
SecretName: pg-test-primaryuser-secret
Optional: false
root-volume:
Type: Secret (a volume populated by a Secret)
SecretName: pg-test-postgres-secret
Optional: false
pgwal-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
backrestrepo-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
recover-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
report:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
pgconf-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
default-token-zxrlg:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-zxrlg
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations:
Events:
Type Reason Age From Message

---

Normal Scheduled 3m default-scheduler Successfully assigned pg-test-664467cc8d-lkfch to worker01
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "backrestrepo-volume"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "backup"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "pgwal-volume"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "report"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "recover-volume"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "crunchy-pv74"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "pgconf-volume"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "root-volume"
Normal SuccessfulMountVolume 3m kubelet, worker01 MountVolume.SetUp succeeded for volume "default-token-zxrlg"
Normal SuccessfulMountVolume 3m (x2 over 3m) kubelet, worker01 (combined from similar events): MountVolume.SetUp succeeded for volume "primary-volume"
Normal Pulled 3m kubelet, worker01 Container image "crunchydata/crunchy-postgres:centos7-10.5-2.1.0" already present on machine
Normal Started 3m kubelet, worker01 Started container
Normal Created 3m kubelet, worker01 Created container
Warning Unhealthy 3m kubelet, worker01 Readiness probe failed: xxxxxx INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Warning Unhealthy 3m kubelet, worker01 Readiness probe failed:xxx xxx xx xx:xx:xxx UTC 2018 INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Warning Unhealthy 3m kubelet, worker01 Readiness probe failed: xxx xxx xx xx:xx:xxx UTC 2018 INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Warning Unhealthy 3m kubelet, worker01 Readiness probe failed: xxx xxx xx xx:xx:xxx UTC 2018 INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Warning Unhealthy 2m kubelet, worker01 Readiness probe failed: xxx xxx xx xx:xx:xxx UTC 2018 INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Warning Unhealthy 2m kubelet, worker01 Readiness probe failed: xxx xxx xx xx:xx:xxx UTC 2018 INFO: Setting PGROOT to /usr/pgsql-10.
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

Storage type: hostpath
I need to make it work on hostpath, not nfs
Created following paths
sudo mkdir /data
sudo chmod 777 /data
Create some sample Persistent Volumes using the following script:
$COROOT/pv/create-pv.sh

Looks like storageClass by default is set to StorageClass: gluster-heketi. I did not change it. Do I need to change it ?
pgo show config
BasicAuth:
Cluster:
PrimaryNodeLabel:
ReplicaNodeLabel:
CCPImagePrefix: crunchydata
CCPImageTag: centos7-10.5-2.1.0
ServiceType: ClusterIP
Metrics: false
Backrest: false
Autofail: false
Badger: false
Policies:
Port: 5432
ArchiveTimeout: 60
ArchiveMode: false
User: testuser
Database: userdb
PasswordAgeDays: 60
PasswordLength: 8
Strategy: 1
Replicas: 0
PrimaryStorage: storage1
BackupStorage: storage1
ReplicaStorage: storage1
Storage:
storage1:
AccessMode: ReadWriteMany
Size: 1G
StorageType: create
StorageClass:
Fsgroup:
SupplementalGroups: 65534
storage2:
AccessMode: ReadWriteMany
Size: 1G
StorageType: dynamic
StorageClass: gluster-heketi
Fsgroup: 26
SupplementalGroups:
storage3:
AccessMode: ReadWriteOnce
Size: 1G
StorageType: dynamic
StorageClass: rook-ceph-block
Fsgroup: 26
SupplementalGroups:
DefaultContainerResources:
ContainerResources:
large:
RequestsMemory: 2Gi
RequestsCPU: 2.0
LimitsMemory: 2Gi
LimitsCPU: 4.0
small:
RequestsMemory: 512Mi
RequestsCPU: 0.1
LimitsMemory: 512Mi
LimitsCPU: 0.1
Pgo:
AutofailSleepSeconds: 9
Audit: false
LSPVCTemplate: /config/pgo.lspvc-template.json
LoadTemplate: /config/pgo.load-template.json
COImagePrefix: crunchydata
COImageTag: centos7-3.3.0

[jmccormick2001]

yes, since gluster-heketi is just our test example for a storage class name, you will want to change it to match your storage class name, you change it in pgo.yaml and then redeploy the operator to pick up that change. let me know if that fixes this error.

[cahoonpwork]

@nchiluku if you need anything else on this please let us know. I am closing for now.