[Airship-discuss] scaled down version of site/airship-seaworthy <airship-treasuremap problems>

MCEUEN, MATT MM9745 at att.com
Thu Sep 6 17:44:50 UTC 2018


Hey Jan-Erik!

If your lab is still in this state, could you check the pod logs for ceph-rgw-694b44bd5d-ckv8v and share what you see?  Looks like that guy is Running but not completed (as of the time of the logs below).

Please check `kubectl -n ceph exec ceph-mon-lv7hw -- ceph -s` to see the ceph status.

We are in progress of updating the treasuremap readthedocs content, stay tuned!

Thanks,
Matt

-----Original Message-----
From: Jan-Erik MÃ¥ngs <kire at kth.se> 
Sent: Monday, September 3, 2018 3:50 PM
To: airship-discuss at lists.airshipit.org
Subject: [Airship-discuss] scaled down version of site/airship-seaworthy <airship-treasuremap problems>

(reposting again in plain text, my first attempt was too big >.40 KB due to formatting)

We're playing around with  https://urldefense.proofpoint.com/v2/url?u=http-3A__git.openstack.org_cgit_openstack_airship-2Dtreasuremap_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=MyMpkoXeJDDM5X1b0oakc6awcXcRf74kWEKFp5iQp9w&s=i-lEaji6J8bUrRsLcZQNEzciBuYCTkRQAdOil7Lajow&e= and we're having some problems.

We're trying to deploy a scaled down version of site/airship-seaworthy (a smaller "template" could come in handy, who has this many disks in their lab?). We're only using four servers (Dell 630s), with three SSDs/server, networking according to seaworthy etc. We've copied site/airship-seaworthy to our own site and tried to modify things to the best of our knowledge.

When we run the genesis-script on the genesis node we see an error related to mariadb, see below. Any hints on what could be wrong are much appreciated. (Storage is always among the most difficult things, I really need to learn ceph :-)).

(BTW, instructions here https://urldefense.proofpoint.com/v2/url?u=https-3A__treasuremap.readthedocs.io_en_latest_deployment.html&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=MyMpkoXeJDDM5X1b0oakc6awcXcRf74kWEKFp5iQp9w&s=MKTFp8gJE_JdUc2rKHp0oTUezN9h1rlsW0FU-RXDbXA&e= does not seem to be in sync with the repo above)

Thanks for your help

/Jan-Erik (et al.)


Output from the genesis script:
 
NAMESPACE     NAME                                              READY     STATUS      RESTARTS   AGE
ceph          ceph-bootstrap-vb495                              0/1       Completed   0          32m
ceph          ceph-cephfs-client-key-generator-jvc62            0/1       Completed   0          21m
ceph          ceph-cephfs-provisioner-cdfb7cfbf-5pb88           1/1       Running     0          21m
ceph          ceph-cephfs-provisioner-cdfb7cfbf-789vh           1/1       Running     0          21m
ceph          ceph-mds-6c757f9549-fqw9v                         1/1       Running     0          21m
ceph          ceph-mds-keyring-generator-p2hzh                  0/1       Completed   0          32m
ceph          ceph-mgr-5f985c796-fj6ql                          1/1       Running     0          21m
ceph          ceph-mgr-keyring-generator-2xvvm                  0/1       Completed   0          32m
ceph          ceph-mon-check-84c94dc8ff-wgblt                   1/1       Running     0          32m
ceph          ceph-mon-keyring-generator-rln66                  0/1       Completed   0          32m
ceph          ceph-mon-lv7hv                                    2/2       Running     0          32m
ceph          ceph-osd-default-83945928-xl2g5                   2/2       Running     0          29m
ceph          ceph-osd-keyring-generator-zn6dg                  0/1       Completed   0          32m
ceph          ceph-rbd-pool-t7b8d                               0/1       Completed   0          21m
ceph          ceph-rbd-provisioner-86c99578d8-2689g             1/1       Running     0          21m
ceph          ceph-rbd-provisioner-86c99578d8-jtfd5             1/1       Running     0          21m
ceph          ceph-rgw-694b44bd5d-ckv8v                         0/1       Running     6          21m
ceph          ceph-rgw-keyring-generator-chlhr                  0/1       Completed   0          32m
ceph          ceph-storage-keys-generator-5g58r                 0/1       Completed   0          32m
ceph          ingress-65dc849968-68nfm                          1/1       Running     0          32m
ceph          ingress-65dc849968-gfwqq                          1/1       Running     0          32m
ceph          ingress-error-pages-768bd7c654-jx8tb              1/1       Running     0          32m
ceph          ingress-error-pages-768bd7c654-kfd42              1/1       Running     0          32m
kube-system   auxiliary-etcd-radio-st-11                        3/3       Running     0          39m
kube-system   bootstrap-armada-radio-st-11                      4/4       Running     0          39m
kube-system   calico-etcd-anchor-rvghv                          1/1       Running     0          37m
kube-system   calico-etcd-radio-st-11                           1/1       Running     0          37m
kube-system   calico-kube-policy-controllers-5df8bcbfc6-swxn4   1/1       Running     0          37m
kube-system   calico-node-s9qfh                                 2/2       Running     0          37m
kube-system   calico-settings-ljc9s                             0/1       Completed   0          37m
kube-system   coredns-5545b74458-6kmmf                          1/1       Running     0          36m
kube-system   coredns-5545b74458-nfqwq                          1/1       Running     0          36m
kube-system   haproxy-anchor-9zh6b                              1/1       Running     0          34m
kube-system   haproxy-radio-st-11                               1/1       Running     1          34m
kube-system   ingress-error-pages-5ccf96bf7d-2mzsl              1/1       Running     0          32m
kube-system   ingress-error-pages-5ccf96bf7d-x9p64              1/1       Running     0          32m
kube-system   ingress-kqxck                                     1/1       Running     0          32m
kube-system   kubernetes-apiserver-anchor-ppv9m                 1/1       Running     0          34m
kube-system   kubernetes-apiserver-radio-st-11                  1/1       Running     0          32m
kube-system   kubernetes-controller-manager-anchor-m2mm4        1/1       Running     0          33m
kube-system   kubernetes-controller-manager-radio-st-11         1/1       Running     0          33m
kube-system   kubernetes-etcd-anchor-v7nr9                      1/1       Running     0          36m
kube-system   kubernetes-etcd-radio-st-11                       1/1       Running     0          34m
kube-system   kubernetes-proxy-86gmf                            1/1       Running     0          38m
kube-system   kubernetes-scheduler-anchor-dztff                 1/1       Running     0          33m
kube-system   kubernetes-scheduler-radio-st-11                  1/1       Running     0          33m
ucp           ceph-namespace-client-key-generator-tnng5         0/1       Completed   0          5m
ucp           ingress-6cd5b89d5d-2pw7x                          1/1       Running     0          5m
ucp           ingress-6cd5b89d5d-z74xt                          1/1       Running     0          5m
ucp           ingress-error-pages-6bff886f96-24rjr              1/1       Running     0          5m
ucp           ingress-error-pages-6bff886f96-qvbmt              1/1       Running     0          5m
ucp           mariadb-ingress-74f744b54c-n7wzt                  0/1       Running     0          4m
ucp           mariadb-ingress-error-pages-5dc7dd974c-lbdh4      1/1       Running     0          4m
ucp           mariadb-server-0                                  0/1       Pending     0          4m
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller [-] Error while installing release airship-ucp-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-mariadb failed: timed out waiting for the condition)>
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller Traceback (most recent call last):
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 401, in install_release
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller     metadata=self.metadata)
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 487, in __call__
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller     return _end_unary_response_blocking(state, call, False, deadline)
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller     raise _Rendezvous(state, None, None, deadline)
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-mariadb failed: timed out waiting for the condition)>
2018-09-03 15:40:50.523 2392 ERROR armada.handlers.tiller
2018-09-03 15:40:50.635 2392 ERROR armada.cli [-] Caught internal exception: armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-mariadb - Tiller Message: b'Release "airship-ucp-mariadb" failed: timed out waiting for the condition'
2018-09-03 15:40:50.635 2392 ERROR armada.cli Traceback (most recent call last):
2018-09-03 15:40:50.635 2392 ERROR armada.cli   File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 401, in install_release
2018-09-03 15:40:50.635 2392 ERROR armada.cli     metadata=self.metadata)
2018-09-03 15:40:50.635 2392 ERROR armada.cli   File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 487, in __call__
2018-09-03 15:40:50.635 2392 ERROR armada.cli     return _end_unary_response_blocking(state, call, False, deadline)
2018-09-03 15:40:50.635 2392 ERROR armada.cli   File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking
2018-09-03 15:40:50.635 2392 ERROR armada.cli     raise _Rendezvous(state, None, None, deadline)
2018-09-03 15:40:50.635 2392 ERROR armada.cli grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, release airship-ucp-mariadb failed: timed out waiting for the condition)>
2018-09-03 15:40:50.635 2392 ERROR armada.cli
2018-09-03 15:40:50.635 2392 ERROR armada.cli During handling of the above exception, another exception occurred:
2018-09-03 15:40:50.635 2392 ERROR armada.cli
2018-09-03 15:40:50.635 2392 ERROR armada.cli Traceback (most recent call last):
2018-09-03 15:40:50.635 2392 ERROR armada.cli   File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke
2018-09-03 15:40:50.635 2392 ERROR armada.cli     self.invoke()
2018-09-03 15:40:50.635 2392 ERROR armada.cli   File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 217, in invoke
2018-09-03 15:40:50.635 2392 ERROR armada.cli     resp = armada.sync()
2018-09-03 15:40:50.635 2392 ERROR armada.cli   File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 480, in sync
2018-09-03 15:40:50.635 2392 ERROR armada.cli     timeout=timer)
2018-09-03 15:40:50.635 2392 ERROR armada.cli   File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 414, in install_release
2018-09-03 15:40:50.635 2392 ERROR armada.cli     raise ex.ReleaseException(release, status, 'Install')
2018-09-03 15:40:50.635 2392 ERROR armada.cli armada.exceptions.tiller_exceptions.ReleaseException: Failed to Install release: airship-ucp-mariadb - Tiller Message: b'Release "airship-ucp-mariadb" failed: timed out waiting for the condition'
2018-09-03 15:40:50.635 2392 ERROR armada.cli ^[[C 2018-09-03 15:41:04.365 3391 INFO armada.handlers.armada [-] Performing pre-flight operations.
2018-09-03 15:41:04.529 3391 INFO armada.handlers.armada [-] Cloning repo: https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openstack.org_openstack_airship-2Dpromenade&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=MyMpkoXeJDDM5X1b0oakc6awcXcRf74kWEKFp5iQp9w&s=6DCFL5kcvZL7DFfRT81Zbyi6wbxOwwNQCel0W4pLQ8o&e= from branch: f60f6d7a8569c5006d32c44c27f4584f10ffe57a
_______________________________________________
Airship-discuss mailing list
Airship-discuss at lists.airshipit.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=MyMpkoXeJDDM5X1b0oakc6awcXcRf74kWEKFp5iQp9w&s=DzkWxm3aSJdAH_c6Kq0pn2OJ_pEbX15oIVdMlklUCs4&e=


More information about the Airship-discuss mailing list