[Airship-discuss] keystone-rabbit-init failing forever
KATARIA, DEEPAK
dd7022 at att.com
Tue Nov 27 22:35:19 UTC 2018
Dear Airship Team,
We are using Airship v18.11.01.
1prom-gen.sh<http://1prom-gen.sh/> is successful and genesis.sh<http://genesis.sh/> is failing and here is the error. Please see additional information highlighted in yellow below.
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller [-] [chart=ucp-keystone]: Error while installing release airship-ucp-keystone: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)>
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller Traceback (most recent call last):
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line 455, in install_release
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller metadata=self.metadata)
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 487, in __call__
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, deadline)
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline)
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-keystone failed: timed out waiting for the condition)>
2018-11-17 00:47:44.809 5176 ERROR armada.handlers.tiller ^[[00m
2018-11-17 00:47:45.104 5176 ERROR armada.handlers.armada [-] Chart deploy [ucp-keystone] failed: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition': armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-keystone - Tiller Message: b'Release "airship-ucp-keystone" failed: timed out waiting for the condition'^[[00m
2018-11-17 00:47:45.105 5176 ERROR armada.handlers.armada [-] Chart deploy(s) failed: ['ucp-keystone']^[[00m
2018-11-17 00:47:45.105 5176 ERROR armada.cli [-] Caught internal exception: armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone']
2018-11-17 00:47:45.105 5176 ERROR armada.cli Traceback (most recent call last):
2018-11-17 00:47:45.105 5176 ERROR armada.cli File "/usr/local/lib/python3.6/site-packages/armada/cli/__init__.py", line 39, in safe_invoke
2018-11-17 00:47:45.105 5176 ERROR armada.cli self.invoke()
2018-11-17 00:47:45.105 5176 ERROR armada.cli File "/usr/local/lib/python3.6/site-packages/armada/cli/apply.py", line 216, in invoke
2018-11-17 00:47:45.105 5176 ERROR armada.cli resp = armada.sync()
2018-11-17 00:47:45.105 5176 ERROR armada.cli File "/usr/local/lib/python3.6/site-packages/armada/handlers/armada.py", line 270, in sync
2018-11-17 00:47:45.105 5176 ERROR armada.cli raise armada_exceptions.ChartDeployException(failures)
2018-11-17 00:47:45.105 5176 ERROR armada.cli armada.exceptions.armada_exceptions.ChartDeployException: Exception deploying charts: ['ucp-keystone']
2018-11-17 00:47:45.105 5176 ERROR armada.cli ^[[00m
Here is some more information that may help
root at aknode30:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
ceph airship-ucp-ceph-provisioners-ceph-ns-key-generator-xv9sc 0/1 Completed 0 1h
ceph ceph-bootstrap-82qx2 0/1 Completed 0 1h
ceph ceph-cephfs-client-key-generator-26qhs 0/1 Completed 0 1h
ceph ceph-cephfs-provisioner-676684f6bd-kfs9m 1/1 Running 0 1h
ceph ceph-cephfs-provisioner-676684f6bd-wbwl9 1/1 Running 0 1h
ceph ceph-mds-869b96cd5f-wtwn2 1/1 Running 0 1h
ceph ceph-mds-keyring-generator-fwl2l 0/1 Completed 0 1h
ceph ceph-mgr-6dd7fcc86b-q4j9l 1/1 Running 0 1h
ceph ceph-mgr-keyring-generator-72p4q 0/1 Completed 0 1h
ceph ceph-mon-check-6db6b569b6-z4gst 1/1 Running 0 1h
ceph ceph-mon-keyring-generator-jzg6l 0/1 Completed 0 1h
ceph ceph-mon-pw7mt 1/1 Running 0 1h
ceph ceph-osd-default-64779b8c-s8x2s 1/1 Running 0 1h
ceph ceph-osd-default-6ea9de2c-r5c2g 1/1 Running 0 1h
ceph ceph-osd-default-70a54190-lmrj2 1/1 Running 0 1h
ceph ceph-osd-default-7544b6da-86z5c 1/1 Running 0 1h
ceph ceph-osd-default-7cfc44c1-47sxn 1/1 Running 0 1h
ceph ceph-osd-default-83945928-fv9gb 1/1 Running 0 1h
ceph ceph-osd-default-be8e8cc4-2x5fd 1/1 Running 0 1h
ceph ceph-osd-default-f9249fa9-2mszd 1/1 Running 0 1h
ceph ceph-osd-keyring-generator-wf97s 0/1 Completed 0 1h
ceph ceph-rbd-pool-tt69r 0/1 Completed 0 1h
ceph ceph-rbd-provisioner-84bc5c88c7-hmxhb 1/1 Running 0 1h
ceph ceph-rbd-provisioner-84bc5c88c7-t5665 1/1 Running 0 1h
ceph ceph-rgw-d6878bcc7-btm5v 1/1 Running 0 1h
ceph ceph-rgw-d6878bcc7-v2smd 1/1 Running 0 1h
ceph ceph-rgw-storage-init-xpbml 0/1 Completed 0 1h
ceph ceph-storage-keys-generator-p6qnc 0/1 Completed 0 1h
ceph ingress-65dc849968-h562n 1/1 Running 0 1h
ceph ingress-65dc849968-rhjz2 1/1 Running 0 1h
ceph ingress-error-pages-796b76c856-9775r 1/1 Running 0 1h
ceph ingress-error-pages-796b76c856-jwpnx 1/1 Running 0 1h
kube-system airship-kubernetes-calico-etcd-etcd-test 0/1 Completed 0 1h
kube-system auxiliary-etcd-aknode30 3/3 Running 0 1h
kube-system bootstrap-armada-aknode30 4/4 Running 0 1h
kube-system calico-etcd-aknode30 1/1 Running 0 1h
kube-system calico-etcd-anchor-g8l5s 1/1 Running 0 1h
kube-system calico-kube-controllers-6ddd8598f-mk8md 1/1 Running 0 1h
kube-system calico-node-4lv6w 2/2 Running 0 1h
kube-system calico-settings-d62l6 0/1 CrashLoopBackOff 21 1h
kube-system coredns-7d69b6b56c-dbbxw 1/1 Running 0 1h
kube-system coredns-7d69b6b56c-x9f6j 1/1 Running 0 1h
kube-system coredns-7d69b6b56c-xrkxc 1/1 Running 0 1h
kube-system haproxy-aknode30 1/1 Running 1 1h
kube-system haproxy-anchor-t8zjg 1/1 Running 0 1h
kube-system ingress-error-pages-5ccf96bf7d-dbcxq 1/1 Running 0 1h
kube-system ingress-error-pages-5ccf96bf7d-zh8fq 1/1 Running 0 1h
kube-system ingress-hfzfs 1/1 Running 0 1h
kube-system kubernetes-apiserver-aknode30 1/1 Running 0 1h
kube-system kubernetes-apiserver-anchor-g97js 1/1 Running 0 1h
kube-system kubernetes-controller-manager-aknode30 1/1 Running 0 1h
kube-system kubernetes-controller-manager-anchor-7kb5h 1/1 Running 0 1h
kube-system kubernetes-etcd-aknode30 1/1 Running 0 1h
kube-system kubernetes-etcd-anchor-nhcwf 1/1 Running 0 1h
kube-system kubernetes-proxy-mppl4 1/1 Running 0 1h
kube-system kubernetes-scheduler-aknode30 1/1 Running 0 1h
kube-system kubernetes-scheduler-anchor-f7v8k 1/1 Running 0 1h
ucp airship-ucp-ceph-config-ceph-ns-key-generator-7h28c 0/1 Completed 0 1h
ucp airship-ucp-keystone-memcached-memcached-74d79d8896-j6m46 1/1 Running 0 58m
ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 1h
ucp airship-ucp-rabbitmq-rabbitmq-1 1/1 Running 0 59m
ucp airship-ucp-rabbitmq-rabbitmq-2 1/1 Running 0 58m
ucp ingress-6cd5b89d5d-gl9kl 1/1 Running 0 1h
ucp ingress-6cd5b89d5d-jcjjk 1/1 Running 0 1h
ucp ingress-error-pages-5c97bb46bb-djgzj 1/1 Running 0 1h
ucp ingress-error-pages-5c97bb46bb-s2wr5 1/1 Running 0 1h
ucp keystone-api-56c8985585-cmv8l 0/1 Init:0/1 0 4m
ucp keystone-api-56c8985585-dhdrk 0/1 Init:0/1 0 4m
ucp keystone-bootstrap-mqbxh 0/1 Init:0/1 0 4m
ucp keystone-credential-setup-78c7s 0/1 Completed 0 4m
ucp keystone-db-init-pv5ns 0/1 Completed 0 4m
ucp keystone-db-sync-s9x2l 0/1 Init:0/1 0 4m
ucp keystone-domain-manage-wtxsr 0/1 Init:0/2 0 4m
ucp keystone-fernet-rotate-1542412800-zs299 0/1 Completed 0 52m
ucp keystone-fernet-setup-z5bq5 0/1 Completed 0 4m
ucp keystone-rabbit-init-29q6j 0/1 CrashLoopBackOff 5 4m
ucp mariadb-ingress-56df696f99-hltbl 1/1 Running 0 1h
ucp mariadb-ingress-56df696f99-tntgp 1/1 Running 0 1h
ucp mariadb-ingress-error-pages-67db6bf8df-qsn8m 1/1 Running 0 1h
ucp mariadb-server-0 1/1 Running 0 1h
ucp mariadb-server-1 1/1 Running 0 1h
ucp mariadb-server-2 1/1 Running 0 1h
ucp postgresql-0 1/1 Running 0 1h
root at aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j
Managing: User: keystone
user declared
Managing: vHost: openstack
vhost declared
Managing: Permissions: keystone on openstack
permission declared
Applying additional configuration
*** Please create virtual host "keystone" prior to importing definitions.
root at aknode30:~# kubectl logs -n ucp keystone-rabbit-init-29q6j
Managing: User: keystone
user declared
Managing: vHost: openstack
vhost declared
Managing: Permissions: keystone on openstack
permission declared
Applying additional configuration
*** Please create virtual host "keystone" prior to importing definitions.
root at aknode30:~# kubectl describe pod -n ucp keystone-rabbit-init-29q6j
Name: keystone-rabbit-init-29q6j
Namespace: ucp
Node: aknode30/172.29.1.30
Start Time: Sat, 17 Nov 2018 00:48:28 +0000
Labels: application=keystone
component=rabbit-init
controller-uid=7f2dea33-ea02-11e8-9d4f-3cfdfeaa90b1
job-name=keystone-rabbit-init
release_group=airship-ucp-keystone
Annotations: <none>
Status: Running
IP: 10.99.161.96
Controlled By: Job/keystone-rabbit-init
Init Containers:
init:
Container ID: docker://43242c89f546337e1dcb2a1fca14077f5d435ee7cbe2712737d78a0277912868
Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
Image ID: docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6
Port: <none>
Host Port: <none>
Command:
kubernetes-entrypoint
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 17 Nov 2018 00:48:42 +0000
Finished: Sat, 17 Nov 2018 00:48:44 +0000
Ready: True
Restart Count: 0
Environment:
POD_NAME: keystone-rabbit-init-29q6j (v1:metadata.name)
NAMESPACE: ucp (v1:metadata.namespace)
INTERFACE_NAME: eth0
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
DEPENDENCY_SERVICE: ucp:rabbitmq
DEPENDENCY_DAEMONSET:
DEPENDENCY_CONTAINER:
DEPENDENCY_POD_JSON:
COMMAND: echo done
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro)
Containers:
rabbit-init:
Container ID: docker://58e8b01ce85aac3876c1f40c01cf6a032cdeae7fcbcbcb516ab556dcce58e07b
Image: docker.io/rabbitmq:3.7-management
Image ID: docker-pullable://rabbitmq@sha256:3eb2fa0f83914999846f831f14b900c0c85cea8e5d2db48ff73cf7defa12fe96
Port: <none>
Host Port: <none>
Command:
/tmp/rabbit-init.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 17 Nov 2018 00:51:55 +0000
Finished: Sat, 17 Nov 2018 00:51:56 +0000
Ready: False
Restart Count: 5
Environment:
RABBITMQ_ADMIN_CONNECTION: <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-admin'> Optional: false
RABBITMQ_USER_CONNECTION: <set to the key 'RABBITMQ_CONNECTION' in secret 'keystone-rabbitmq-user'> Optional: false
RABBITMQ_AUXILIARY_CONFIGURATION: {"policies":[{"apply-to":"all","definition":{"ha-mode":"all","ha-sync-mode":"automatic","message-ttl":70000},"name":"ha_ttl_keystone","pattern":"(notifications)\\.","priority":0,"vhost":"keystone"}]}
Mounts:
/tmp/rabbit-init.sh from rabbit-init-sh (ro)
/var/run/secrets/kubernetes.io/serviceaccount from keystone-rabbit-init-token-gftxq (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
rabbit-init-sh:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: keystone-bin
Optional: false
keystone-rabbit-init-token-gftxq:
Type: Secret (a volume populated by a Secret)
SecretName: keystone-rabbit-init-token-gftxq
Optional: false
QoS Class: BestEffort
Node-Selectors: ucp-control-plane=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m default-scheduler Successfully assigned keystone-rabbit-init-29q6j to aknode30
Normal SuccessfulMountVolume 5m kubelet, aknode30 MountVolume.SetUp succeeded for volume "keystone-rabbit-init-token-gftxq"
Normal SuccessfulMountVolume 5m kubelet, aknode30 MountVolume.SetUp succeeded for volume "rabbit-init-sh"
Normal Pulled 5m kubelet, aknode30 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine
Normal Created 5m kubelet, aknode30 Created container
Normal Started 5m kubelet, aknode30 Started container
Normal Started 4m (x4 over 4m) kubelet, aknode30 Started container
Normal Pulled 3m (x5 over 4m) kubelet, aknode30 Container image "docker.io/rabbitmq:3.7-management" already present on machine
Normal Created 3m (x5 over 4m) kubelet, aknode30 Created container
Warning BackOff 2s (x22 over 4m) kubelet, aknode30 Back-off restarting failed container
Best Regards,
Deepak Kataria
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20181127/a0b46de1/attachment-0001.html>
More information about the Airship-discuss
mailing list