[Airship-discuss] keystone-rabbit-init failing forever

KATARIA, DEEPAK dd7022 at att.com
Tue Nov 27 22:35:19 UTC 2018


Dear Airship Team,

We are using Airship v18.11.01.

1prom-gen.sh<http://1prom-gen.sh/> is successful and genesis.sh<http://genesis.sh/> is failing and here is the error. Please see additional information highlighted in yellow below.

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller [-] [chart=ucp-keystone]: Error while installing release airship-ucp-keystone: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)>

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller Traceback (most recent call last):

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line 455, in install_release

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller     metadata=self.metadata)

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 487, in __call__

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller     return _end_unary_response_blocking(state, call, False, deadline)

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller     raise _Rendezvous(state, None, None, deadline)

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)>

2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller ^[[00m

2018-11-17 00:47:45.104 5176 ERROR armada.handlers.armada [-] Chart deploy [ucp-keystone] failed: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition': armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition'^[[00m

2018-11-17 00:47:45.105 5176 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['ucp-keystone']^[[00m

2018-11-17 00:47:45.105 5176 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone']

2018-11-17 00:47:45.105 5176 ERROR armada.cli Traceback (most recent call last):

2018-11-17 00:47:45.105 5176 ERROR armada.cli   File "/usr/local/lib/python3.6/site-packages/armada/cli/__init__.py", line 39, in safe_invoke

2018-11-17 00:47:45.105 5176 ERROR armada.cli     self.invoke()

2018-11-17 00:47:45.105 5176 ERROR armada.cli   File "/usr/local/lib/python3.6/site-packages/armada/cli/apply.py", line 216, in invoke

2018-11-17 00:47:45.105 5176 ERROR armada.cli     resp = armada.sync()

2018-11-17 00:47:45.105 5176 ERROR armada.cli   File "/usr/local/lib/python3.6/site-packages/armada/handlers/armada.py", line 270, in sync

2018-11-17 00:47:45.105 5176 ERROR armada.cli     raise armada_exceptions.ChartDeployException(failures)

2018-11-17 00:47:45.105 5176 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone']

2018-11-17 00:47:45.105 5176 ERROR armada.cli ^[[00m



Here is some more information that may help



root at aknode30:~# kubectl get pods --all-namespaces

NAMESPACE     NAME                                                        READY     STATUS             RESTARTS   AGE

ceph          airship-ucp-ceph-provisioners-ceph-ns-key-generator-xv9sc   0/1       Completed          0          1h

ceph          ceph-bootstrap-82qx2                                        0/1       Completed          0          1h

ceph          ceph-cephfs-client-key-generator-26qhs                      0/1       Completed          0          1h

ceph          ceph-cephfs-provisioner-676684f6bd-kfs9m                    1/1       Running            0          1h

ceph          ceph-cephfs-provisioner-676684f6bd-wbwl9                    1/1       Running            0          1h

ceph          ceph-mds-869b96cd5f-wtwn2                                   1/1       Running            0          1h

ceph          ceph-mds-keyring-generator-fwl2l                            0/1       Completed          0          1h

ceph          ceph-mgr-6dd7fcc86b-q4j9l                                   1/1       Running            0          1h

ceph          ceph-mgr-keyring-generator-72p4q                            0/1       Completed          0          1h

ceph          ceph-mon-check-6db6b569b6-z4gst                             1/1       Running            0          1h

ceph          ceph-mon-keyring-generator-jzg6l                            0/1       Completed          0          1h

ceph          ceph-mon-pw7mt                                              1/1       Running            0          1h

ceph          ceph-osd-default-64779b8c-s8x2s                             1/1       Running            0          1h

ceph          ceph-osd-default-6ea9de2c-r5c2g                             1/1       Running            0          1h

ceph          ceph-osd-default-70a54190-lmrj2                             1/1       Running            0          1h

ceph          ceph-osd-default-7544b6da-86z5c                             1/1       Running            0          1h

ceph          ceph-osd-default-7cfc44c1-47sxn                             1/1       Running            0          1h

ceph          ceph-osd-default-83945928-fv9gb                             1/1       Running            0          1h

ceph          ceph-osd-default-be8e8cc4-2x5fd                             1/1       Running            0          1h

ceph          ceph-osd-default-f9249fa9-2mszd                             1/1       Running            0          1h

ceph          ceph-osd-keyring-generator-wf97s                            0/1       Completed          0          1h

ceph          ceph-rbd-pool-tt69r                                         0/1       Completed          0          1h

ceph          ceph-rbd-provisioner-84bc5c88c7-hmxhb                       1/1       Running            0          1h

ceph          ceph-rbd-provisioner-84bc5c88c7-t5665                       1/1       Running            0          1h

ceph          ceph-rgw-d6878bcc7-btm5v                                    1/1       Running            0          1h

ceph          ceph-rgw-d6878bcc7-v2smd                                    1/1       Running            0          1h

ceph          ceph-rgw-storage-init-xpbml                                 0/1       Completed          0          1h

ceph          ceph-storage-keys-generator-p6qnc                           0/1       Completed          0          1h

ceph          ingress-65dc849968-h562n                                    1/1       Running            0          1h

ceph          ingress-65dc849968-rhjz2                                    1/1       Running            0          1h

ceph          ingress-error-pages-796b76c856-9775r                        1/1       Running            0          1h

ceph          ingress-error-pages-796b76c856-jwpnx                        1/1       Running            0          1h

kube-system   airship-kubernetes-calico-etcd-etcd-test                    0/1       Completed          0          1h

kube-system   auxiliary-etcd-aknode30                                     3/3       Running            0          1h

kube-system   bootstrap-armada-aknode30                                   4/4       Running            0          1h

kube-system   calico-etcd-aknode30                                        1/1       Running            0          1h

kube-system   calico-etcd-anchor-g8l5s                                    1/1       Running            0          1h

kube-system   calico-kube-controllers-6ddd8598f-mk8md                     1/1       Running            0          1h

kube-system   calico-node-4lv6w                                           2/2       Running            0          1h

kube-system   calico-settings-d62l6                                       0/1       CrashLoopBackOff   21         1h

kube-system   coredns-7d69b6b56c-dbbxw                                    1/1       Running            0          1h

kube-system   coredns-7d69b6b56c-x9f6j                                    1/1       Running            0          1h

kube-system   coredns-7d69b6b56c-xrkxc                                    1/1       Running            0          1h

kube-system   haproxy-aknode30                                            1/1       Running            1          1h

kube-system   haproxy-anchor-t8zjg                                        1/1       Running            0          1h

kube-system   ingress-error-pages-5ccf96bf7d-dbcxq                        1/1       Running            0          1h

kube-system   ingress-error-pages-5ccf96bf7d-zh8fq                        1/1       Running            0          1h

kube-system   ingress-hfzfs                                               1/1       Running            0          1h

kube-system   kubernetes-apiserver-aknode30                               1/1       Running            0          1h

kube-system   kubernetes-apiserver-anchor-g97js                           1/1       Running            0          1h

kube-system   kubernetes-controller-manager-aknode30                      1/1       Running            0          1h

kube-system   kubernetes-controller-manager-anchor-7kb5h                  1/1       Running            0          1h

kube-system   kubernetes-etcd-aknode30                                    1/1       Running            0          1h

kube-system   kubernetes-etcd-anchor-nhcwf                                1/1       Running            0          1h

kube-system   kubernetes-proxy-mppl4                                      1/1       Running            0          1h

kube-system   kubernetes-scheduler-aknode30                               1/1       Running            0          1h

kube-system   kubernetes-scheduler-anchor-f7v8k                           1/1       Running            0          1h

ucp           airship-ucp-ceph-config-ceph-ns-key-generator-7h28c         0/1       Completed          0          1h

ucp           airship-ucp-keystone-memcached-memcached-74d79d8896-j6m46   1/1       Running            0          58m

ucp           airship-ucp-rabbitmq-rabbitmq-0                             1/1       Running            0          1h

ucp           airship-ucp-rabbitmq-rabbitmq-1                             1/1       Running            0          59m

ucp           airship-ucp-rabbitmq-rabbitmq-2                             1/1       Running            0          58m

ucp           ingress-6cd5b89d5d-gl9kl                                    1/1       Running            0          1h

ucp           ingress-6cd5b89d5d-jcjjk                                    1/1       Running            0          1h

ucp           ingress-error-pages-5c97bb46bb-djgzj                        1/1       Running            0          1h

ucp           ingress-error-pages-5c97bb46bb-s2wr5                        1/1       Running            0          1h

ucp           keystone-api-56c8985585-cmv8l                               0/1       Init:0/1           0          4m

ucp           keystone-api-56c8985585-dhdrk                               0/1       Init:0/1           0          4m

ucp           keystone-bootstrap-mqbxh                                    0/1       Init:0/1           0          4m

ucp           keystone-credential-setup-78c7s                             0/1       Completed          0          4m

ucp           keystone-db-init-pv5ns                                      0/1       Completed          0          4m

ucp           keystone-db-sync-s9x2l                                      0/1       Init:0/1           0          4m

ucp           keystone-domain-manage-wtxsr                                0/1       Init:0/2           0          4m

ucp           keystone-fernet-rotate-1542412800-zs299                     0/1       Completed          0          52m

ucp           keystone-fernet-setup-z5bq5                                 0/1       Completed          0          4m

ucp           keystone-rabbit-init-29q6j                                  0/1       CrashLoopBackOff   5          4m

ucp           mariadb-ingress-56df696f99-hltbl                            1/1       Running            0          1h

ucp           mariadb-ingress-56df696f99-tntgp                            1/1       Running            0          1h

ucp           mariadb-ingress-error-pages-67db6bf8df-qsn8m                1/1       Running            0          1h

ucp           mariadb-server-0                                            1/1       Running            0          1h

ucp           mariadb-server-1                                            1/1       Running            0          1h

ucp           mariadb-server-2                                            1/1       Running            0          1h

ucp           postgresql-0                                                1/1       Running            0          1h



root at aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j

Managing: User: keystone

user declared

Managing: vHost: openstack

vhost declared

Managing: Permissions: keystone on openstack

permission declared

Applying additional configuration

*** Please create virtual host "keystone" prior to importing definitions.



root at aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j

Managing: User: keystone

user declared

Managing: vHost: openstack

vhost declared

Managing: Permissions: keystone on openstack

permission declared

Applying additional configuration

*** Please create virtual host "keystone" prior to importing definitions.



root at aknode30:~# kubectl describe pod -n ucp keystone-rabbit-init-29q6j

Name:           keystone-rabbit-init-29q6j

Namespace:      ucp

Node:           aknode30/172.29.1.30

Start Time:     Sat, 17 Nov 2018 00:48:28 +0000

Labels:         application=keystone

                component=rabbit-init

                controller-uid=7f2dea33-ea02-11e8-9d4f-3cfdfeaa90b1

                job-name=keystone-rabbit-init

                release_group=airship-ucp-keystone

Annotations:    <none>

Status:         Running

IP:             10.99.161.96

Controlled By:  Job/keystone-rabbit-init

Init Containers:

  init:

    Container ID:  docker://43242c89f546337e1dcb2a1fca14077f5d435ee7cbe2712737d78a0277912868

    Image:         quay.io/stackanetes/kubernetes-entrypoint:v0.3.1

    Image ID:      docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6

    Port:          <none>

    Host Port:     <none>

    Command:

      kubernetes-entrypoint

    State:          Terminated

      Reason:       Completed

      Exit Code:    0

      Started:      Sat, 17 Nov 2018 00:48:42 +0000

      Finished:     Sat, 17 Nov 2018 00:48:44 +0000

    Ready:          True

    Restart Count:  0

    Environment:

      POD_NAME:              keystone-rabbit-init-29q6j (v1:metadata.name)

      NAMESPACE:             ucp (v1:metadata.namespace)

      INTERFACE_NAME:        eth0

      PATH:                  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/

      DEPENDENCY_SERVICE:    ucp:rabbitmq

      DEPENDENCY_DAEMONSET:

      DEPENDENCY_CONTAINER:

      DEPENDENCY_POD_JSON:

      COMMAND:               echo done

    Mounts:

      /var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro)

Containers:

  rabbit-init:

    Container ID:  docker://58e8b01ce85aac3876c1f40c01cf6a032cdeae7fcbcbcb516ab556dcce58e07b

    Image:         docker.io/rabbitmq:3.7-management

    Image ID:      docker-pullable://rabbitmq@sha256:3eb2fa0f83914999846f831f14b900c0c85cea8e5d2db48ff73cf7defa12fe96

    Port:          <none>

    Host Port:     <none>

    Command:

      /tmp/rabbit-init.sh

    State:          Waiting

      Reason:       CrashLoopBackOff

    Last State:     Terminated

      Reason:       Error

      Exit Code:    1

      Started:      Sat, 17 Nov 2018 00:51:55 +0000

      Finished:     Sat, 17 Nov 2018 00:51:56 +0000

    Ready:          False

    Restart Count:  5

    Environment:

      RABBITMQ_ADMIN_CONNECTION:         <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-admin'>  Optional: false

      RABBITMQ_USER_CONNECTION:          <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-user'>   Optional: false

      RABBITMQ_AUXILIARY_CONFIGURATION:  {"policies":[{"apply-to":"all","definition":{"ha-mode":"all","ha-sync-mode":"automatic","message-ttl":70000},"name":"ha_ttl_keystone","pattern":"(notifications)\\.","priority":0,"vhost":"keystone"}]}

    Mounts:

      /tmp/rabbit-init.sh from rabbit-init-sh (ro)

      /var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro)

Conditions:

  Type           Status

  Initialized    True

  Ready          False

  PodScheduled   True

Volumes:

  rabbit-init-sh:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      keystone-bin

    Optional:  false

  keystone-rabbit-init-token-gftxq:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  keystone-rabbit-init-token-gftxq

    Optional:    false

QoS Class:       BestEffort

Node-Selectors:  ucp-control-plane=enabled

Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s

                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:

  Type     Reason                 Age               From               Message

  ----     ------                 ----              ----               -------

  Normal   Scheduled              5m                default-scheduler  Successfully assigned keystone-rabbit-init-29q6j to aknode30

  Normal   SuccessfulMountVolume  5m                kubelet, aknode30  MountVolume.SetUp succeeded for volume "keystone-rabbit-init-token-gftxq"

  Normal   SuccessfulMountVolume  5m                kubelet, aknode30  MountVolume.SetUp succeeded for volume "rabbit-init-sh"

  Normal   Pulled                 5m                kubelet, aknode30  Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine

  Normal   Created                5m                kubelet, aknode30  Created container

  Normal   Started                5m                kubelet, aknode30  Started container

  Normal   Started                4m (x4 over 4m)   kubelet, aknode30  Started container

  Normal   Pulled                 3m (x5 over 4m)   kubelet, aknode30  Container image "docker.io/rabbitmq:3.7-management" already present on machine

  Normal   Created                3m (x5 over 4m)   kubelet, aknode30  Created container

  Warning  BackOff                2s (x22 over 4m)  kubelet, aknode30  Back-off restarting failed container
Best Regards,

Deepak Kataria
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20181127/a0b46de1/attachment-0001.html>


More information about the Airship-discuss mailing list