We're playing around with http://git.openstack.org/cgit/openstack/airship-treasuremap/ and we're having some problems. We're trying to deploy a scaled down version of site/airship-seaworthy (a smaller "template" could come in handy, who has this many disks in their lab?). We're only using four servers (Dell 630s), with three SSDs/server, networking according to seaworthy etc. We've copied site/airship-seaworthy to our own site and tried to modify things to the best of our knowledge. When we run the genesis-script on the genesis node we see an error related to mariadb, see below. Any hints on what could be wrong are much appreciated. (Storage is always among the most difficult things, I really need to learn ceph :-)). (BTW, instructions here https://treasuremap.readthedocs.io/en/latest/deployment.html does not seem to be in sync with the repo above) Thanks for your help /Jan-Erik (et al.) Output from the genesis script: NAMESPACE NAME READY STATUS RESTARTS AGE ceph ceph-bootstrap-vb495 0/1 Completed 0 32m ceph ceph-cephfs-client-key-generator-jvc62 0/1 Completed 0 21m ceph ceph-cephfs-provisioner-cdfb7cfbf-5pb88 1/1 Running 0 21m ceph ceph-cephfs-provisioner-cdfb7cfbf-789vh 1/1 Running 0 21m ceph ceph-mds-6c757f9549-fqw9v 1/1 Running 0 21m ceph ceph-mds-keyring-generator-p2hzh 0/1 Completed 0 32m ceph ceph-mgr-5f985c796-fj6ql 1/1 Running 0 21m ceph ceph-mgr-keyring-generator-2xvvm 0/1 Completed 0 32m ceph ceph-mon-check-84c94dc8ff-wgblt 1/1 Running 0 32m ceph ceph-mon-keyring-generator-rln66 0/1 Completed 0 32m ceph ceph-mon-lv7hv 2/2 Running 0 32m ceph ceph-osd-default-83945928-xl2g5 2/2 Running 0 29m ceph ceph-osd-keyring-generator-zn6dg 0/1 Completed 0 32m ceph ceph-rbd-pool-t7b8d 0/1 Completed 0 21m ceph ceph-rbd-provisioner-86c99578d8-2689g 1/1 Running 0 21m ceph ceph-rbd-provisioner-86c99578d8-jtfd5 1/1 Running 0 21m ceph ceph-rgw-694b44bd5d-ckv8v 0/1 Running 6 21m ceph ceph-rgw-keyring-generator-chlhr 0/1 Completed 0 32m ceph ceph-storage-keys-generator-5g58r 0/1 Completed 0 32m ceph ingress-65dc849968-68nfm 1/1 Running 0 32m ceph ingress-65dc849968-gfwqq 1/1 Running 0 32m ceph ingress-error-pages-768bd7c654-jx8tb 1/1 Running 0 32m ceph ingress-error-pages-768bd7c654-kfd42 1/1 Running 0 32m kube-system auxiliary-etcd-radio-st-11 3/3 Running 0 39m kube-system bootstrap-armada-radio-st-11 4/4 Running 0 39m kube-system calico-etcd-anchor-rvghv 1/1 Running 0 37m kube-system calico-etcd-radio-st-11 1/1 Running 0 37m kube-system calico-kube-policy-controllers-5df8bcbfc6-swxn4 1/1 Running 0 37m kube-system calico-node-s9qfh 2/2 Running 0 37m kube-system calico-settings-ljc9s 0/1 Completed 0 37m kube-system coredns-5545b74458-6kmmf 1/1 Running 0 36m kube-system coredns-5545b74458-nfqwq 1/1 Running 0 36m kube-system haproxy-anchor-9zh6b 1/1 Running 0 34m kube-system haproxy-radio-st-11 1/1 Running 1 34m kube-system ingress-error-pages-5ccf96bf7d-2mzsl 1/1 Running 0 32m kube-system ingress-error-pages-5ccf96bf7d-x9p64 1/1 Running 0 32m kube-system ingress-kqxck 1/1 Running 0 32m kube-system kubernetes-apiserver-anchor-ppv9m 1/1 Running 0 34m kube-system kubernetes-apiserver-radio-st-11 1/1 Running 0 32m kube-system kubernetes-controller-manager-anchor-m2mm4 1/1 Running 0 33m kube-system kubernetes-controller-manager-radio-st-11 1/1 Running 0 33m kube-system kubernetes-etcd-anchor-v7nr9 1/1 Running 0 36m kube-system kubernetes-etcd-radio-st-11 1/1 Running 0 34m kube-system kubernetes-proxy-86gmf 1/1 Running 0 38m kube-system kubernetes-scheduler-anchor-dztff 1/1 Running 0 33m kube-system kubernetes-scheduler-radio-st-11 1/1 Running 0 33m ucp ceph-namespace-client-key-generator-tnng5 0/1 Completed 0 5m ucp ingress-6cd5b89d5d-2pw7x 1/1 Running 0 5m ucp ingress-6cd5b89d5d-z74xt 1/1 Running 0 5m ucp ingress-error-pages-6bff886f96-24rjr 1/1 Running 0 5m ucp ingress-error-pages-6bff886f96-qvbmt 1/1 Running 0 5m ucp mariadb-ingress-74f744b54c-n7wzt 0/1 Running 0 4m ucp mariadb-ingress-error-pages-5dc7dd974c-lbdh4 1/1 Running 0 4m ucp mariadb-server-0 0/1 Pending 0 4m 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller [-] Error while installing release airship-ucp-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-mariadb failed: timed out waiting for the condition)> 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller Traceback (most recent call last): 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 401, in install_release 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller metadata=self.metadata) 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 487, in __call__ 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller return _end_unary_response_blocking(state, call, False, deadline) 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller raise _Rendezvous(state, None, None, deadline) 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-mariadb failed: timed out waiting for the condition)> 2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller 2018-09-03 15:40:50.635 2392 ERROR armada.cli [-] Caught internal exception: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-mariadb - Tiller Message: b'Release "airship-ucp-mariadb" failed: timed out waiting for the condition' 2018-09-03 15:40:50.635 2392 ERROR armada.cli Traceback (most recent call last): 2018-09-03 15:40:50.635 2392 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 401, in install_release 2018-09-03 15:40:50.635 2392 ERROR armada.cli metadata=self.metadata) 2018-09-03 15:40:50.635 2392 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 487, in __call__ 2018-09-03 15:40:50.635 2392 ERROR armada.cli return _end_unary_response_blocking(state, call, False, deadline) 2018-09-03 15:40:50.635 2392 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking 2018-09-03 15:40:50.635 2392 ERROR armada.cli raise _Rendezvous(state, None, None, deadline) 2018-09-03 15:40:50.635 2392 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-mariadb failed: timed out waiting for the condition)> 2018-09-03 15:40:50.635 2392 ERROR armada.cli 2018-09-03 15:40:50.635 2392 ERROR armada.cli During handling of the above exception, another exception occurred: 2018-09-03 15:40:50.635 2392 ERROR armada.cli 2018-09-03 15:40:50.635 2392 ERROR armada.cli Traceback (most recent call last): 2018-09-03 15:40:50.635 2392 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke 2018-09-03 15:40:50.635 2392 ERROR armada.cli self.invoke() 2018-09-03 15:40:50.635 2392 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 217, in invoke 2018-09-03 15:40:50.635 2392 ERROR armada.cli resp = armada.sync() 2018-09-03 15:40:50.635 2392 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 480, in sync 2018-09-03 15:40:50.635 2392 ERROR armada.cli timeout=timer) 2018-09-03 15:40:50.635 2392 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 414, in install_release 2018-09-03 15:40:50.635 2392 ERROR armada.cli raise ex.ReleaseException(release, status, 'Install') 2018-09-03 15:40:50.635 2392 ERROR armada.cli armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-mariadb - Tiller Message: b'Release "airship-ucp-mariadb" failed: timed out waiting for the condition' 2018-09-03 15:40:50.635 2392 ERROR armada.cli ^[[C 2018-09-03 15:41:04.365 3391 INFO armada.handlers.armada [-] Performing pre-flight operations. 2018-09-03 15:41:04.529 3391 INFO armada.handlers.armada [-] Cloning repo: https://git.openstack.org/openstack/airship-promenade from branch: f60f6d7a8569c5006d32c44c27f4584f10ffe57a