Dear Airship Team,

We are using Airship v18.11.01.

1prom-gen.sh is successful and genesis.sh is failing and here is the error. Please see additional information highlighted in yellow below.

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller [-] [chart=ucp-keystone]: Error while installing release airship-ucp-keystone: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)>
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller Traceback (most recent call last):
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line 455, in install_release
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller     metadata=self.metadata)
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 487, in __call__
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller     return _end_unary_response_blocking(state, call, False, deadline)
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller     raise _Rendezvous(state, None, None, deadline)
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)>
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller ^[[00m
2018-11-17 00:47:45.104 5176 ERROR armada.handlers.armada [-] Chart deploy [ucp-keystone] failed: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition': armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition'^[[00m
2018-11-17 00:47:45.105 5176 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['ucp-keystone']^[[00m
2018-11-17 00:47:45.105 5176 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone']
2018-11-17 00:47:45.105 5176 ERROR armada.cli Traceback (most recent call last):
2018-11-17 00:47:45.105 5176 ERROR armada.cli   File "/usr/local/lib/python3.6/site-packages/armada/cli/__init__.py", line 39, in safe_invoke
2018-11-17 00:47:45.105 5176 ERROR armada.cli     self.invoke()
2018-11-17 00:47:45.105 5176 ERROR armada.cli   File "/usr/local/lib/python3.6/site-packages/armada/cli/apply.py", line 216, in invoke
2018-11-17 00:47:45.105 5176 ERROR armada.cli     resp = armada.sync()
2018-11-17 00:47:45.105 5176 ERROR armada.cli   File "/usr/local/lib/python3.6/site-packages/armada/handlers/armada.py", line 270, in sync
2018-11-17 00:47:45.105 5176 ERROR armada.cli     raise armada_exceptions.ChartDeployException(failures)
2018-11-17 00:47:45.105 5176 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone']
2018-11-17 00:47:45.105 5176 ERROR armada.cli ^[[00m
 
Here is some more information that may help
 
root@aknode30:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                        READY     STATUS             RESTARTS   AGE
ceph          airship-ucp-ceph-provisioners-ceph-ns-key-generator-xv9sc   0/1       Completed          0          1h
ceph          ceph-bootstrap-82qx2                                        0/1       Completed          0          1h
ceph          ceph-cephfs-client-key-generator-26qhs                      0/1       Completed          0          1h
ceph          ceph-cephfs-provisioner-676684f6bd-kfs9m                    1/1       Running            0          1h
ceph          ceph-cephfs-provisioner-676684f6bd-wbwl9                    1/1       Running            0          1h
ceph          ceph-mds-869b96cd5f-wtwn2                                   1/1       Running            0          1h
ceph          ceph-mds-keyring-generator-fwl2l                            0/1       Completed          0          1h
ceph          ceph-mgr-6dd7fcc86b-q4j9l                                   1/1       Running            0          1h
ceph          ceph-mgr-keyring-generator-72p4q                            0/1       Completed          0          1h
ceph          ceph-mon-check-6db6b569b6-z4gst                             1/1       Running            0          1h
ceph          ceph-mon-keyring-generator-jzg6l                            0/1       Completed          0          1h
ceph          ceph-mon-pw7mt                                              1/1       Running            0          1h
ceph          ceph-osd-default-64779b8c-s8x2s                             1/1       Running            0          1h
ceph          ceph-osd-default-6ea9de2c-r5c2g                             1/1       Running            0          1h
ceph          ceph-osd-default-70a54190-lmrj2                             1/1       Running            0          1h
ceph          ceph-osd-default-7544b6da-86z5c                             1/1       Running            0          1h
ceph          ceph-osd-default-7cfc44c1-47sxn                             1/1       Running            0          1h
ceph          ceph-osd-default-83945928-fv9gb                             1/1       Running            0          1h
ceph          ceph-osd-default-be8e8cc4-2x5fd                             1/1       Running            0          1h
ceph          ceph-osd-default-f9249fa9-2mszd                             1/1       Running            0          1h
ceph          ceph-osd-keyring-generator-wf97s                            0/1       Completed          0          1h
ceph          ceph-rbd-pool-tt69r                                         0/1       Completed          0          1h
ceph          ceph-rbd-provisioner-84bc5c88c7-hmxhb                       1/1       Running            0          1h
ceph          ceph-rbd-provisioner-84bc5c88c7-t5665                       1/1       Running            0          1h
ceph          ceph-rgw-d6878bcc7-btm5v                                    1/1       Running            0          1h
ceph          ceph-rgw-d6878bcc7-v2smd                                    1/1       Running            0          1h
ceph          ceph-rgw-storage-init-xpbml                                 0/1       Completed          0          1h
ceph          ceph-storage-keys-generator-p6qnc                           0/1       Completed          0          1h
ceph          ingress-65dc849968-h562n                                    1/1       Running            0          1h
ceph          ingress-65dc849968-rhjz2                                    1/1       Running            0          1h
ceph          ingress-error-pages-796b76c856-9775r                        1/1       Running            0          1h
ceph          ingress-error-pages-796b76c856-jwpnx                        1/1       Running            0          1h
kube-system   airship-kubernetes-calico-etcd-etcd-test                    0/1       Completed          0          1h
kube-system   auxiliary-etcd-aknode30                                     3/3       Running            0          1h
kube-system   bootstrap-armada-aknode30                                   4/4       Running            0          1h
kube-system   calico-etcd-aknode30                                        1/1       Running            0          1h
kube-system   calico-etcd-anchor-g8l5s                                    1/1       Running            0          1h
kube-system   calico-kube-controllers-6ddd8598f-mk8md                     1/1       Running            0          1h
kube-system   calico-node-4lv6w                                           2/2       Running            0          1h
kube-system   calico-settings-d62l6                                       0/1       CrashLoopBackOff   21         1h
kube-system   coredns-7d69b6b56c-dbbxw                                    1/1       Running            0          1h
kube-system   coredns-7d69b6b56c-x9f6j                                    1/1       Running            0          1h
kube-system   coredns-7d69b6b56c-xrkxc                                    1/1       Running            0          1h
kube-system   haproxy-aknode30                                            1/1       Running            1          1h
kube-system   haproxy-anchor-t8zjg                                        1/1       Running            0          1h
kube-system   ingress-error-pages-5ccf96bf7d-dbcxq                        1/1       Running            0          1h
kube-system   ingress-error-pages-5ccf96bf7d-zh8fq                        1/1       Running            0          1h
kube-system   ingress-hfzfs                                               1/1       Running            0          1h
kube-system   kubernetes-apiserver-aknode30                               1/1       Running            0          1h
kube-system   kubernetes-apiserver-anchor-g97js                           1/1       Running            0          1h
kube-system   kubernetes-controller-manager-aknode30                      1/1       Running            0          1h
kube-system   kubernetes-controller-manager-anchor-7kb5h                  1/1       Running            0          1h
kube-system   kubernetes-etcd-aknode30                                    1/1       Running            0          1h
kube-system   kubernetes-etcd-anchor-nhcwf                                1/1       Running            0          1h
kube-system   kubernetes-proxy-mppl4                                      1/1       Running            0          1h
kube-system   kubernetes-scheduler-aknode30                               1/1       Running            0          1h
kube-system   kubernetes-scheduler-anchor-f7v8k                           1/1       Running            0          1h
ucp           airship-ucp-ceph-config-ceph-ns-key-generator-7h28c         0/1       Completed          0          1h
ucp           airship-ucp-keystone-memcached-memcached-74d79d8896-j6m46   1/1       Running            0          58m
ucp           airship-ucp-rabbitmq-rabbitmq-0                             1/1       Running            0          1h
ucp           airship-ucp-rabbitmq-rabbitmq-1                             1/1       Running            0          59m
ucp           airship-ucp-rabbitmq-rabbitmq-2                             1/1       Running            0          58m
ucp           ingress-6cd5b89d5d-gl9kl                                    1/1       Running            0          1h
ucp           ingress-6cd5b89d5d-jcjjk                                    1/1       Running            0          1h
ucp           ingress-error-pages-5c97bb46bb-djgzj                        1/1       Running            0          1h
ucp           ingress-error-pages-5c97bb46bb-s2wr5                        1/1       Running            0          1h
ucp           keystone-api-56c8985585-cmv8l                               0/1       Init:0/1           0          4m
ucp           keystone-api-56c8985585-dhdrk                               0/1       Init:0/1           0          4m
ucp           keystone-bootstrap-mqbxh                                    0/1       Init:0/1           0          4m
ucp           keystone-credential-setup-78c7s                             0/1       Completed          0          4m
ucp           keystone-db-init-pv5ns                                      0/1       Completed          0          4m
ucp           keystone-db-sync-s9x2l                                      0/1       Init:0/1           0          4m
ucp           keystone-domain-manage-wtxsr                                0/1       Init:0/2           0          4m
ucp           keystone-fernet-rotate-1542412800-zs299                     0/1       Completed          0          52m
ucp           keystone-fernet-setup-z5bq5                                 0/1       Completed          0          4m
ucp           keystone-rabbit-init-29q6j                                  0/1       CrashLoopBackOff   5          4m
ucp           mariadb-ingress-56df696f99-hltbl                            1/1       Running            0          1h
ucp           mariadb-ingress-56df696f99-tntgp                            1/1       Running            0          1h
ucp           mariadb-ingress-error-pages-67db6bf8df-qsn8m                1/1       Running            0          1h
ucp           mariadb-server-0                                            1/1       Running            0          1h
ucp           mariadb-server-1                                            1/1       Running            0          1h
ucp           mariadb-server-2                                            1/1       Running            0          1h
ucp           postgresql-0                                                1/1       Running            0          1h
 
root@aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j
Managing: User: keystone
user declared
Managing: vHost: openstack
vhost declared
Managing: Permissions: keystone on openstack
permission declared
Applying additional configuration
*** Please create virtual host "keystone" prior to importing definitions.
 
root@aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j
Managing: User: keystone
user declared
Managing: vHost: openstack
vhost declared
Managing: Permissions: keystone on openstack
permission declared
Applying additional configuration
*** Please create virtual host "keystone" prior to importing definitions.
 
root@aknode30:~# kubectl describe pod -n ucp keystone-rabbit-init-29q6j
Name:           keystone-rabbit-init-29q6j
Namespace:      ucp
Node:           aknode30/172.29.1.30
Start Time:     Sat, 17 Nov 2018 00:48:28 +0000
Labels:         application=keystone
                component=rabbit-init
                controller-uid=7f2dea33-ea02-11e8-9d4f-3cfdfeaa90b1
                job-name=keystone-rabbit-init
                release_group=airship-ucp-keystone
Annotations:    <none>
Status:         Running
IP:             10.99.161.96
Controlled By:  Job/keystone-rabbit-init
Init Containers:
  init:
    Container ID:  docker://43242c89f546337e1dcb2a1fca14077f5d435ee7cbe2712737d78a0277912868
    Image:         quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
    Image ID:      docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6
    Port:          <none>
    Host Port:     <none>
    Command:
      kubernetes-entrypoint
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 17 Nov 2018 00:48:42 +0000
      Finished:     Sat, 17 Nov 2018 00:48:44 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:              keystone-rabbit-init-29q6j (v1:metadata.name)
      NAMESPACE:             ucp (v1:metadata.namespace)
      INTERFACE_NAME:        eth0
      PATH:                  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
      DEPENDENCY_SERVICE:    ucp:rabbitmq
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      DEPENDENCY_POD_JSON:
      COMMAND:               echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro)
Containers:
  rabbit-init:
    Container ID:  docker://58e8b01ce85aac3876c1f40c01cf6a032cdeae7fcbcbcb516ab556dcce58e07b
    Image:         docker.io/rabbitmq:3.7-management
    Image ID:      docker-pullable://rabbitmq@sha256:3eb2fa0f83914999846f831f14b900c0c85cea8e5d2db48ff73cf7defa12fe96
    Port:          <none>
    Host Port:     <none>
    Command:
      /tmp/rabbit-init.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 17 Nov 2018 00:51:55 +0000
      Finished:     Sat, 17 Nov 2018 00:51:56 +0000
    Ready:          False
    Restart Count:  5
    Environment:
      RABBITMQ_ADMIN_CONNECTION:         <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-admin'>  Optional: false
      RABBITMQ_USER_CONNECTION:          <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-user'>   Optional: false
      RABBITMQ_AUXILIARY_CONFIGURATION:  {"policies":[{"apply-to":"all","definition":{"ha-mode":"all","ha-sync-mode":"automatic","message-ttl":70000},"name":"ha_ttl_keystone","pattern":"(notifications)\\.","priority":0,"vhost":"keystone"}]}
    Mounts:
      /tmp/rabbit-init.sh from rabbit-init-sh (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  rabbit-init-sh:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      keystone-bin
    Optional:  false
  keystone-rabbit-init-token-gftxq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  keystone-rabbit-init-token-gftxq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  ucp-control-plane=enabled
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age               From               Message
  ----     ------                 ----              ----               -------
  Normal   Scheduled              5m                default-scheduler  Successfully assigned keystone-rabbit-init-29q6j to aknode30
  Normal   SuccessfulMountVolume  5m                kubelet, aknode30  MountVolume.SetUp succeeded for volume "keystone-rabbit-init-token-gftxq"
  Normal   SuccessfulMountVolume  5m                kubelet, aknode30  MountVolume.SetUp succeeded for volume "rabbit-init-sh"
  Normal   Pulled                 5m                kubelet, aknode30  Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine
  Normal   Created                5m                kubelet, aknode30  Created container
  Normal   Started                5m                kubelet, aknode30  Started container
  Normal   Started                4m (x4 over 4m)   kubelet, aknode30  Started container
  Normal   Pulled                 3m (x5 over 4m)   kubelet, aknode30  Container image "docker.io/rabbitmq:3.7-management" already present on machine
  Normal   Created                3m (x5 over 4m)   kubelet, aknode30  Created container
  Warning  BackOff                2s (x22 over 4m)  kubelet, aknode30  Back-off restarting failed container

Best Regards,

 

Deepak Kataria