ucp airship-ucp-rabbitmq-rabbitmq-0 0/1 Init:0/2 0 1m ucp ingress-6cd5b89d5d-nmwpt 1/1 Running 0 18m ucp ingress-6cd5b89d5d-nr65b 1/1 Running 0 18m ucp ingress-error-pages-5c97bb46bb-2mvgm 1/1 Running 0 18m ucp ingress-error-pages-5c97bb46bb-wzzdz 1/1 Running 0 18m ucp mariadb-ingress-85b8556fbc-xpvwc 0/1 Running 0 1m ucp mariadb-ingress-85b8556fbc-zv72k 0/1 Running 0 1m ucp mariadb-ingress-error-pages-64f89dc697-2trh9 1/1 Running 0 1m ucp mariadb-server-0 0/1 Init:0/2 0 1m ucp postgresql-0 0/1 Init:0/1 0 1m root@cab23-r720-11:~# kubectl describe pod mariadb-ingress-85b8556fbc-xpvwc -n ucp Name: mariadb-ingress-85b8556fbc-xpvwc Namespace: ucp Node: cab23-r720-11/10.23.22.11 Start Time: Mon, 12 Nov 2018 08:05:00 +0000 Labels: application=mariadb component=ingress pod-template-hash=4164112967 release_group=airship-ucp-mariadb Annotations: configmap-bin-hash=eb36d47d8f7d7097cf6d488a61145f76dbfe5e558edf5b802153a00fc3389f0b configmap-etc-hash=3f45f1d8d3ddf5a09fbcd3036cb23bffb939cfa1225f8f1a0d79b390877710c1 Status: Running IP: 10.97.38.125 Controlled By: ReplicaSet/mariadb-ingress-85b8556fbc Init Containers: init: Container ID: docker://58494d08f101ec0e9a9966da7b9848a7565e3c94b78208f3429f9b767e137215 Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6 Port: Host Port: Command: kubernetes-entrypoint State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 12 Nov 2018 08:05:06 +0000 Finished: Mon, 12 Nov 2018 08:05:15 +0000 Ready: True Restart Count: 0 Environment: POD_NAME: mariadb-ingress-85b8556fbc-xpvwc (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: ucp:mariadb-ingress-error-pages DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from airship-ucp-mariadb-ingress-token-htf82 (ro) Containers: ingress: Container ID: docker://6779fdf2cd247d4f92ba695b2da1837e9fc1ebc5960d421ef3fcab20aac7031d Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 Image ID: docker-pullable://quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:39cc6ce23e5bcdf8aa78bc28bbcfe0999e449bf99fe2e8d60984b417facc5cd4 Port: Host Port: Command: /tmp/mariadb-ingress-controller.sh start State: Running Started: Mon, 12 Nov 2018 08:05:16 +0000 Ready: False Restart Count: 0 Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: tcp-socket :3306 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: mariadb-ingress-85b8556fbc-xpvwc (v1:metadata.name) POD_NAMESPACE: ucp (v1:metadata.namespace) RELEASE_NAME: airship-ucp-mariadb INGRESS_CLASS: airship-ucp-mariadb-mariadb-ingress ERROR_PAGE_SERVICE: mariadb-ingress-error-pages Mounts: /etc/nginx/template/nginx.tmpl from mariadb-etc (ro) /tmp/mariadb-ingress-controller.sh from mariadb-bin (ro) /var/run/secrets/kubernetes.io/serviceaccount from airship-ucp-mariadb-ingress-token-htf82 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: mariadb-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: mariadb-bin Optional: false mariadb-etc: Type: ConfigMap (a volume populated by a ConfigMap) Name: mariadb-etc Optional: false airship-ucp-mariadb-ingress-token-htf82: Type: Secret (a volume populated by a Secret) SecretName: airship-ucp-mariadb-ingress-token-htf82 Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m default-scheduler Successfully assigned mariadb-ingress-85b8556fbc-xpvwc to cab23-r720-11 Normal SuccessfulMountVolume 3m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "airship-ucp-mariadb-ingress-token-htf82" Normal SuccessfulMountVolume 3m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "mariadb-etc" Normal SuccessfulMountVolume 3m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "mariadb-bin" Normal Pulled 3m kubelet, cab23-r720-11 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine Normal Created 3m kubelet, cab23-r720-11 Created container Normal Started 3m kubelet, cab23-r720-11 Started container Normal Pulled 3m kubelet, cab23-r720-11 Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0" already present on machine Normal Created 3m kubelet, cab23-r720-11 Created container Normal Started 3m kubelet, cab23-r720-11 Started container Warning Unhealthy 26s (x16 over 2m) kubelet, cab23-r720-11 Readiness probe failed: dial tcp 10.97.38.125:3306: getsockopt: connection refused root@cab23-r720-11:~# kubectl describe pod postgresql-0 -n ucp Name: postgresql-0 Namespace: ucp Node: cab23-r720-11/10.23.22.11 Start Time: Mon, 12 Nov 2018 08:04:56 +0000 Labels: application=postgresql component=server controller-revision-hash=postgresql-566fd45fd7 release_group=airship-ucp-postgresql statefulset.kubernetes.io/pod-name=postgresql-0 Annotations: Status: Pending IP: Controlled By: StatefulSet/postgresql Init Containers: init: Container ID: Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: Port: Host Port: Command: kubernetes-entrypoint State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: POD_NAME: postgresql-0 (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from postgresql-token-rmkq9 (ro) Containers: postgresql: Container ID: Image: docker.io/postgres:9.5 Image ID: Port: 5432/TCP Host Port: 0/TCP Command: /tmp/start.sh State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Liveness: exec [/tmp/readiness.sh] delay=20s timeout=5s period=10s #success=1 #failure=3 Readiness: exec [/tmp/readiness.sh] delay=20s timeout=5s period=10s #success=1 #failure=3 Environment: POSTGRES_PASSWORD: Optional: false POSTGRES_USER: Optional: false PGDATA: /var/lib/postgresql/data Mounts: /tmp/readiness.sh from postgresql-bin (ro) /tmp/start.sh from postgresql-bin (ro) /var/lib/postgresql/data from postgresql-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from postgresql-token-rmkq9 (ro) Conditions: Type Status Initialized False Ready False PodScheduled True Volumes: postgresql-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: postgresql-data-postgresql-0 ReadOnly: false postgresql-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: postgresql-bin Optional: false postgresql-token-rmkq9: Type: Secret (a volume populated by a Secret) SecretName: postgresql-token-rmkq9 Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulAttachVolume 4m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" Normal Scheduled 4m default-scheduler Successfully assigned postgresql-0 to cab23-r720-11 Normal SuccessfulMountVolume 4m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "postgresql-bin" Normal SuccessfulMountVolume 4m kubelet, cab23-r720-11 MountVolume.SetUp succeeded for volume "postgresql-token-rmkq9" Warning FailedMount 2m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:07:01.400015 7fcc31018100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 2m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:07:07.815991 7f4506194100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 2m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:07:15.011010 7f6c819a2100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 2m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:07:22.811153 7f9b4064a100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 1m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:07:32.215011 7fe5c4624100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 1m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:07:46.017892 7f284292e100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 1m kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:08:08.007588 7f3f255a3100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 45s kubelet, cab23-r720-11 MountVolume.WaitForAttach failed for volume "pvc-c66d1b6f-e64a-11e8-bb43-080027f45d2a" : fail to check rbd image status with: (exit status 22), rbd output: (2018-11-12 16:08:45.808540 7f619ddf3100 -1 did not load config file, using default settings. server name not found: ceph-mon.ceph.svc.cluster.local (Name or service not known) unable to parse addrs in 'ceph-mon.ceph.svc.cluster.local:6789' rbd: couldn't connect to the cluster! ) Warning FailedMount 19s (x2 over 2m) kubelet, cab23-r720-11 Unable to mount volumes for pod "postgresql-0_ucp(a46bc160-e651-11e8-bb43-080027f45d2a)": timeout expired waiting for volumes to attach or mount for pod "ucp"/"postgresql-0". list of unmounted volumes=[postgresql-data]. list of unattached volumes=[postgresql-data postgresql-bin postgresql-token-rmkq9]