Dear Airship Team, We are using Airship v18.11.01. 1prom-gen.sh<http://1prom-gen.sh/> is successful and genesis.sh<http://genesis.sh/> is failing and here is the error. Please see additional information highlighted in yellow below. 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller [-] [chart=ucp-keystone]: Error while installing release airship-ucp-keystone: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)> 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller Traceback (most recent call last): 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line 455, in install_release 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller metadata=self.metadata) 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 487, in __call__ 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, deadline) 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline) 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)> 2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller ^[[00m 2018-11-17 00:47:45.104 5176 ERROR armada.handlers.armada [-] Chart deploy [ucp-keystone] failed: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition': armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition'^[[00m 2018-11-17 00:47:45.105 5176 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['ucp-keystone']^[[00m 2018-11-17 00:47:45.105 5176 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone'] 2018-11-17 00:47:45.105 5176 ERROR armada.cli Traceback (most recent call last): 2018-11-17 00:47:45.105 5176 ERROR armada.cli File "/usr/local/lib/python3.6/site-packages/armada/cli/__init__.py", line 39, in safe_invoke 2018-11-17 00:47:45.105 5176 ERROR armada.cli self.invoke() 2018-11-17 00:47:45.105 5176 ERROR armada.cli File "/usr/local/lib/python3.6/site-packages/armada/cli/apply.py", line 216, in invoke 2018-11-17 00:47:45.105 5176 ERROR armada.cli resp = armada.sync() 2018-11-17 00:47:45.105 5176 ERROR armada.cli File "/usr/local/lib/python3.6/site-packages/armada/handlers/armada.py", line 270, in sync 2018-11-17 00:47:45.105 5176 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures) 2018-11-17 00:47:45.105 5176 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone'] 2018-11-17 00:47:45.105 5176 ERROR armada.cli ^[[00m Here is some more information that may help root@aknode30:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE ceph airship-ucp-ceph-provisioners-ceph-ns-key-generator-xv9sc 0/1 Completed 0 1h ceph ceph-bootstrap-82qx2 0/1 Completed 0 1h ceph ceph-cephfs-client-key-generator-26qhs 0/1 Completed 0 1h ceph ceph-cephfs-provisioner-676684f6bd-kfs9m 1/1 Running 0 1h ceph ceph-cephfs-provisioner-676684f6bd-wbwl9 1/1 Running 0 1h ceph ceph-mds-869b96cd5f-wtwn2 1/1 Running 0 1h ceph ceph-mds-keyring-generator-fwl2l 0/1 Completed 0 1h ceph ceph-mgr-6dd7fcc86b-q4j9l 1/1 Running 0 1h ceph ceph-mgr-keyring-generator-72p4q 0/1 Completed 0 1h ceph ceph-mon-check-6db6b569b6-z4gst 1/1 Running 0 1h ceph ceph-mon-keyring-generator-jzg6l 0/1 Completed 0 1h ceph ceph-mon-pw7mt 1/1 Running 0 1h ceph ceph-osd-default-64779b8c-s8x2s 1/1 Running 0 1h ceph ceph-osd-default-6ea9de2c-r5c2g 1/1 Running 0 1h ceph ceph-osd-default-70a54190-lmrj2 1/1 Running 0 1h ceph ceph-osd-default-7544b6da-86z5c 1/1 Running 0 1h ceph ceph-osd-default-7cfc44c1-47sxn 1/1 Running 0 1h ceph ceph-osd-default-83945928-fv9gb 1/1 Running 0 1h ceph ceph-osd-default-be8e8cc4-2x5fd 1/1 Running 0 1h ceph ceph-osd-default-f9249fa9-2mszd 1/1 Running 0 1h ceph ceph-osd-keyring-generator-wf97s 0/1 Completed 0 1h ceph ceph-rbd-pool-tt69r 0/1 Completed 0 1h ceph ceph-rbd-provisioner-84bc5c88c7-hmxhb 1/1 Running 0 1h ceph ceph-rbd-provisioner-84bc5c88c7-t5665 1/1 Running 0 1h ceph ceph-rgw-d6878bcc7-btm5v 1/1 Running 0 1h ceph ceph-rgw-d6878bcc7-v2smd 1/1 Running 0 1h ceph ceph-rgw-storage-init-xpbml 0/1 Completed 0 1h ceph ceph-storage-keys-generator-p6qnc 0/1 Completed 0 1h ceph ingress-65dc849968-h562n 1/1 Running 0 1h ceph ingress-65dc849968-rhjz2 1/1 Running 0 1h ceph ingress-error-pages-796b76c856-9775r 1/1 Running 0 1h ceph ingress-error-pages-796b76c856-jwpnx 1/1 Running 0 1h kube-system airship-kubernetes-calico-etcd-etcd-test 0/1 Completed 0 1h kube-system auxiliary-etcd-aknode30 3/3 Running 0 1h kube-system bootstrap-armada-aknode30 4/4 Running 0 1h kube-system calico-etcd-aknode30 1/1 Running 0 1h kube-system calico-etcd-anchor-g8l5s 1/1 Running 0 1h kube-system calico-kube-controllers-6ddd8598f-mk8md 1/1 Running 0 1h kube-system calico-node-4lv6w 2/2 Running 0 1h kube-system calico-settings-d62l6 0/1 CrashLoopBackOff 21 1h kube-system coredns-7d69b6b56c-dbbxw 1/1 Running 0 1h kube-system coredns-7d69b6b56c-x9f6j 1/1 Running 0 1h kube-system coredns-7d69b6b56c-xrkxc 1/1 Running 0 1h kube-system haproxy-aknode30 1/1 Running 1 1h kube-system haproxy-anchor-t8zjg 1/1 Running 0 1h kube-system ingress-error-pages-5ccf96bf7d-dbcxq 1/1 Running 0 1h kube-system ingress-error-pages-5ccf96bf7d-zh8fq 1/1 Running 0 1h kube-system ingress-hfzfs 1/1 Running 0 1h kube-system kubernetes-apiserver-aknode30 1/1 Running 0 1h kube-system kubernetes-apiserver-anchor-g97js 1/1 Running 0 1h kube-system kubernetes-controller-manager-aknode30 1/1 Running 0 1h kube-system kubernetes-controller-manager-anchor-7kb5h 1/1 Running 0 1h kube-system kubernetes-etcd-aknode30 1/1 Running 0 1h kube-system kubernetes-etcd-anchor-nhcwf 1/1 Running 0 1h kube-system kubernetes-proxy-mppl4 1/1 Running 0 1h kube-system kubernetes-scheduler-aknode30 1/1 Running 0 1h kube-system kubernetes-scheduler-anchor-f7v8k 1/1 Running 0 1h ucp airship-ucp-ceph-config-ceph-ns-key-generator-7h28c 0/1 Completed 0 1h ucp airship-ucp-keystone-memcached-memcached-74d79d8896-j6m46 1/1 Running 0 58m ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 1h ucp airship-ucp-rabbitmq-rabbitmq-1 1/1 Running 0 59m ucp airship-ucp-rabbitmq-rabbitmq-2 1/1 Running 0 58m ucp ingress-6cd5b89d5d-gl9kl 1/1 Running 0 1h ucp ingress-6cd5b89d5d-jcjjk 1/1 Running 0 1h ucp ingress-error-pages-5c97bb46bb-djgzj 1/1 Running 0 1h ucp ingress-error-pages-5c97bb46bb-s2wr5 1/1 Running 0 1h ucp keystone-api-56c8985585-cmv8l 0/1 Init:0/1 0 4m ucp keystone-api-56c8985585-dhdrk 0/1 Init:0/1 0 4m ucp keystone-bootstrap-mqbxh 0/1 Init:0/1 0 4m ucp keystone-credential-setup-78c7s 0/1 Completed 0 4m ucp keystone-db-init-pv5ns 0/1 Completed 0 4m ucp keystone-db-sync-s9x2l 0/1 Init:0/1 0 4m ucp keystone-domain-manage-wtxsr 0/1 Init:0/2 0 4m ucp keystone-fernet-rotate-1542412800-zs299 0/1 Completed 0 52m ucp keystone-fernet-setup-z5bq5 0/1 Completed 0 4m ucp keystone-rabbit-init-29q6j 0/1 CrashLoopBackOff 5 4m ucp mariadb-ingress-56df696f99-hltbl 1/1 Running 0 1h ucp mariadb-ingress-56df696f99-tntgp 1/1 Running 0 1h ucp mariadb-ingress-error-pages-67db6bf8df-qsn8m 1/1 Running 0 1h ucp mariadb-server-0 1/1 Running 0 1h ucp mariadb-server-1 1/1 Running 0 1h ucp mariadb-server-2 1/1 Running 0 1h ucp postgresql-0 1/1 Running 0 1h root@aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j Managing: User: keystone user declared Managing: vHost: openstack vhost declared Managing: Permissions: keystone on openstack permission declared Applying additional configuration *** Please create virtual host "keystone" prior to importing definitions. root@aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j Managing: User: keystone user declared Managing: vHost: openstack vhost declared Managing: Permissions: keystone on openstack permission declared Applying additional configuration *** Please create virtual host "keystone" prior to importing definitions. root@aknode30:~# kubectl describe pod -n ucp keystone-rabbit-init-29q6j Name: keystone-rabbit-init-29q6j Namespace: ucp Node: aknode30/172.29.1.30 Start Time: Sat, 17 Nov 2018 00:48:28 +0000 Labels: application=keystone component=rabbit-init controller-uid=7f2dea33-ea02-11e8-9d4f-3cfdfeaa90b1 job-name=keystone-rabbit-init release_group=airship-ucp-keystone Annotations: <none> Status: Running IP: 10.99.161.96 Controlled By: Job/keystone-rabbit-init Init Containers: init: Container ID: docker://43242c89f546337e1dcb2a1fca14077f5d435ee7cbe2712737d78a0277912868 Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6 Port: <none> Host Port: <none> Command: kubernetes-entrypoint State: Terminated Reason: Completed Exit Code: 0 Started: Sat, 17 Nov 2018 00:48:42 +0000 Finished: Sat, 17 Nov 2018 00:48:44 +0000 Ready: True Restart Count: 0 Environment: POD_NAME: keystone-rabbit-init-29q6j (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: ucp:rabbitmq DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro) Containers: rabbit-init: Container ID: docker://58e8b01ce85aac3876c1f40c01cf6a032cdeae7fcbcbcb516ab556dcce58e07b Image: docker.io/rabbitmq:3.7-management Image ID: docker-pullable://rabbitmq@sha256:3eb2fa0f83914999846f831f14b900c0c85cea8e5d2db48ff73cf7defa12fe96 Port: <none> Host Port: <none> Command: /tmp/rabbit-init.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Sat, 17 Nov 2018 00:51:55 +0000 Finished: Sat, 17 Nov 2018 00:51:56 +0000 Ready: False Restart Count: 5 Environment: RABBITMQ_ADMIN_CONNECTION: <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-admin'> Optional: false RABBITMQ_USER_CONNECTION: <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-user'> Optional: false RABBITMQ_AUXILIARY_CONFIGURATION: {"policies":[{"apply-to":"all","definition":{"ha-mode":"all","ha-sync-mode":"automatic","message-ttl":70000},"name":"ha_ttl_keystone","pattern":"(notifications)\\.","priority":0,"vhost":"keystone"}]} Mounts: /tmp/rabbit-init.sh from rabbit-init-sh (ro) /var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: rabbit-init-sh: Type: ConfigMap (a volume populated by a ConfigMap) Name: keystone-bin Optional: false keystone-rabbit-init-token-gftxq: Type: Secret (a volume populated by a Secret) SecretName: keystone-rabbit-init-token-gftxq Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 5m default-scheduler Successfully assigned keystone-rabbit-init-29q6j to aknode30 Normal SuccessfulMountVolume 5m kubelet, aknode30 MountVolume.SetUp succeeded for volume "keystone-rabbit-init-token-gftxq" Normal SuccessfulMountVolume 5m kubelet, aknode30 MountVolume.SetUp succeeded for volume "rabbit-init-sh" Normal Pulled 5m kubelet, aknode30 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine Normal Created 5m kubelet, aknode30 Created container Normal Started 5m kubelet, aknode30 Started container Normal Started 4m (x4 over 4m) kubelet, aknode30 Started container Normal Pulled 3m (x5 over 4m) kubelet, aknode30 Container image "docker.io/rabbitmq:3.7-management" already present on machine Normal Created 3m (x5 over 4m) kubelet, aknode30 Created container Warning BackOff 2s (x22 over 4m) kubelet, aknode30 Back-off restarting failed container Best Regards, Deepak Kataria