<div dir="ltr">Hi Roman,<div><br></div><div>Thanks for your help.</div><div>It seems that the ceph-common package solved everything.</div><div>I have a working 7 nodes cluster now!</div><div><br></div><div>Best,</div><div>James</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 12, 2018 at 4:46 PM, Roman Gorshunov <span dir="ltr"><<a href="mailto:paye600@gmail.com" target="_blank">paye600@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello James,<br>
<br>
Sorry, it took a little bit too long for me to reply. I have talked to<br>
devs, and Scott Hussey suggested that ceph-common package may not have<br>
been installed:<br>
<a href="https://github.com/openstack/airship-promenade/blob/master/examples/complete/HostSystem.yaml#L74" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>airship-promenade/blob/master/<wbr>examples/complete/HostSystem.<wbr>yaml#L74</a><br>
<br>
Also Pete Birley suggested that Armada multinode suggests to use 5<br>
nodes in ceph:<br>
<a href="https://github.com/openstack/openstack-helm/blob/master/tools/deployment/armada/multinode/armada-ceph.yaml#L101" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>openstack-helm/blob/master/<wbr>tools/deployment/armada/<wbr>multinode/armada-ceph.yaml#<wbr>L101</a><br>
<br>
Give it a try, and let me know how it goes.<br>
<br>
Thank you!<br>
<br>
Best regards,<br>
--<br>
Roman Gorshunov<br>
<div class="HOEnZb"><div class="h5"><br>
On Sat, Jun 9, 2018 at 6:50 PM, James Devon <<a href="mailto:jr8586335@gmail.com">jr8586335@gmail.com</a>> wrote:<br>
> Hello Roman,<br>
><br>
> Thanks for your answer.<br>
><br>
> What is the state of promenade + armada right now?<br>
> We are pretty excited about it.<br>
><br>
> Here is the output you required:<br>
><br>
> 1) kubectl get pods --all-namespaces -o wide<br>
> <a href="https://pastebin.com/b4kWGpBs" rel="noreferrer" target="_blank">https://pastebin.com/b4kWGpBs</a><br>
><br>
> The only thing I see is the pod ldap-0 from the namespace osh-infra<br>
> Here is the output of kubectl describe pod -n osh-infra ldap-0<br>
> <a href="https://pastebin.com/sSBVZnch" rel="noreferrer" target="_blank">https://pastebin.com/sSBVZnch</a><br>
><br>
> I believe the interesting part is here:<br>
><br>
> Events:<br>
> Type Reason Age From Message<br>
> ---- ------ ---- ---- -------<br>
> Warning FailedMount 12m (x576 over 21h) kubelet, n3 Unable to mount<br>
> volumes for pod "ldap-0_osh-infra(85e73018-<wbr>6b4c-11e8-b6e5-080027ee1df7)":<br>
> timeout expired waiting for volumes to attach or mount for pod<br>
> "osh-infra"/"ldap-0". list of unmounted volumes=[ldap-config ldap-data].<br>
> list of unattached volumes=[ldap-config ldap-data ldap-token-txdqp]<br>
> Warning FailedMount 6m (x649 over 21h) kubelet, n3<br>
> MountVolume.WaitForAttach failed for volume<br>
> "pvc-85dc46a5-6b4c-11e8-b6e5-<wbr>080027ee1df7" : fail to check rbd image status<br>
> with: (executable file not found in $PATH), rbd output: ()<br>
> Warning FailedMount 2m (x651 over 21h) kubelet, n3<br>
> MountVolume.WaitForAttach failed for volume<br>
> "pvc-85d6770b-6b4c-11e8-b6e5-<wbr>080027ee1df7" : fail to check rbd image status<br>
> with: (executable file not found in $PATH), rbd output: ()<br>
><br>
><br>
><br>
> 2) docker ps on all nodes<br>
> <a href="https://pastebin.com/CaqLEv70" rel="noreferrer" target="_blank">https://pastebin.com/CaqLEv70</a><br>
><br>
> 3) docker ps -a on all nodes<br>
> <a href="https://pastebin.com/wvB0qkbh" rel="noreferrer" target="_blank">https://pastebin.com/wvB0qkbh</a><br>
><br>
> 4) docker images on all nodes<br>
> <a href="https://pastebin.com/H07kEwqe" rel="noreferrer" target="_blank">https://pastebin.com/H07kEwqe</a><br>
><br>
> Best,<br>
> James<br>
><br>
> On Sat, Jun 9, 2018 at 1:51 PM, Roman Gorshunov <<a href="mailto:paye600@gmail.com">paye600@gmail.com</a>> wrote:<br>
>><br>
>> Hello James,<br>
>><br>
>> I didn’t see it before. Could you provide output of the following<br>
>> commands, if you still have this environment running. It would allow to get<br>
>> understanding of what has been deployed and what has not:<br>
>> kubectl get pods --all-namespaces -o wide # on n0<br>
>> docker ps # on all<br>
>> nodes<br>
>> docker ps -a # on all<br>
>> nodes<br>
>> docker images # on all<br>
>> nodes<br>
>><br>
>> Also note that <a href="http://artifacts-aic.atlantafoundry.com" rel="noreferrer" target="_blank">artifacts-aic.atlantafoundry.<wbr>com</a> host where some of docker<br>
>> images are hosted is inaccessible at the moment, it could be that some<br>
>> images have not been downloaded successfully, team started mirroring images<br>
>> to dockerhub & <a href="http://quay.io" rel="noreferrer" target="_blank">quay.io</a>. Right before the summit development was focused on<br>
>> making airship-in-a-bottle work, and migration from github/gerrithub to<br>
>> openstack foundation infrastructure is still ongoing.<br>
>><br>
>> Apologize for the inconvenience, James.<br>
>><br>
>> Thank you.<br>
>><br>
>> > On 8 Jun 2018, at 21:18, James Devon <<a href="mailto:jr8586335@gmail.com">jr8586335@gmail.com</a>> wrote:<br>
>> ><br>
>> > Hi,<br>
>> ><br>
>> > After having tried airship-in-a-bottle, I'm trying to setup a Kubernetes<br>
>> > cluster and install openstack on it using promenade and armada.<br>
>> ><br>
>> > I'm using the basic example :<br>
>> > <a href="https://github.com/openstack/airship-promenade/tree/master/examples/basic" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>airship-promenade/tree/master/<wbr>examples/basic</a>,<br>
>> > I've changed the ip addresses to match my environment of 4 nodes (n0 for<br>
>> > genesis and then n1, n2, n3).<br>
>> ><br>
>> > At this point I'm facing only one problem:<br>
>> > promenade-api.ucp.svc.cluster.<wbr>local is resolvable but the resolution is bad<br>
>> > (192.168.150.165).<br>
>> > So instead of promenade-api.ucp.svc.cluster.<wbr>local, I used the pod<br>
>> > address with the good port and it is working fine.<br>
>> ><br>
>> > Then, I am able to make n1,n2 and n3 join the cluster and I use the<br>
>> > scripts from<br>
>> > <a href="https://github.com/openstack/openstack-helm/tree/master/tools/deployment/armada" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>openstack-helm/tree/master/<wbr>tools/deployment/armada</a><br>
>> > to apply armada manifests.<br>
>> ><br>
>> > osh-infra is not able to start the ldap pod.<br>
>> ><br>
>> > Events:<br>
>> > Type Reason Age From<br>
>> > Message<br>
>> > ---- ------ ---- ----<br>
>> > -------<br>
>> > Warning FailedScheduling 6m (x4 over 6m) default-scheduler<br>
>> > pod has unbound PersistentVolumeClaims (repeated 3 times)<br>
>> > Normal Scheduled 6m default-scheduler<br>
>> > Successfully assigned ldap-0 to n3<br>
>> > Normal SuccessfulAttachVolume 6m<br>
>> > attachdetach-controller AttachVolume.Attach succeeded for volume<br>
>> > "pvc-85d6770b-6b4c-11e8-b6e5-<wbr>080027ee1df7"<br>
>> > Normal SuccessfulAttachVolume 6m<br>
>> > attachdetach-controller AttachVolume.Attach succeeded for volume<br>
>> > "pvc-85dc46a5-6b4c-11e8-b6e5-<wbr>080027ee1df7"<br>
>> > Normal SuccessfulMountVolume 6m kubelet, n3<br>
>> > MountVolume.SetUp succeeded for volume "ldap-token-txdqp"<br>
>> > Warning FailedMount 13s (x11 over 6m) kubelet, n3<br>
>> > MountVolume.WaitForAttach failed for volume<br>
>> > "pvc-85dc46a5-6b4c-11e8-b6e5-<wbr>080027ee1df7" : fail to check rbd image status<br>
>> > with: (executable file not found in $PATH), rbd output: ()<br>
>> > Warning FailedMount 12s (x11 over 6m) kubelet, n3<br>
>> > MountVolume.WaitForAttach failed for volume<br>
>> > "pvc-85d6770b-6b4c-11e8-b6e5-<wbr>080027ee1df7" : fail to check rbd image status<br>
>> > with: (executable file not found in $PATH), rbd output: ()<br>
>> > Warning FailedMount 2s (x3 over 4m) kubelet, n3<br>
>> > Unable to mount volumes for pod<br>
>> > "ldap-0_osh-infra(85e73018-<wbr>6b4c-11e8-b6e5-080027ee1df7)": timeout expired<br>
>> > waiting for volumes to attach or mount for pod "osh-infra"/"ldap-0". list of<br>
>> > unmounted volumes=[ldap-config ldap-data]. list of unattached<br>
>> > volumes=[ldap-config ldap-data ldap-token-txdqp]<br>
>> ><br>
>> > I've checked the logs of ceph-osd and there's also something:<br>
>> ><br>
>> > ceph-2.1 | starting osd.2 at - osd_data /var/lib/ceph/osd/ceph-2<br>
>> > /var/lib/ceph/journal/journal.<wbr>2<br>
>> > ceph-2.1 | 2018-06-08 18:41:03.854234 7f7cd7d58e00 -1 journal<br>
>> > FileJournal::_open: disabling aio for non-block journal. Use<br>
>> > journal_force_aio to force use of aio anyway<br>
>> > ceph-2.1 | 2018-06-08 18:41:03.855499 7f7cd7d58e00 -1 journal<br>
>> > do_read_entry(8192): bad header magic<br>
>> > ceph-2.1 | 2018-06-08 18:41:03.855567 7f7cd7d58e00 -1 journal<br>
>> > do_read_entry(8192): bad header magic<br>
>> > ceph-2.1 | 2018-06-08 18:41:03.901804 7f7cd7d58e00 -1 osd.2 0<br>
>> > log_to_monitors {default=true}<br>
>> > ceph-2.1 | 2018-06-08 18:41:04.529480 7f7cbb2c7700 -1 osd.2 0 waiting<br>
>> > for initial osdmap<br>
>> ><br>
>> > Did anybody encountered this error and how to fix it?<br>
>> ><br>
>> > Best,<br>
>> > James<br>
>> ><br>
>> > ______________________________<wbr>_________________<br>
>> > Airship-discuss mailing list<br>
>> > <a href="mailto:Airship-discuss@lists.airshipit.org">Airship-discuss@lists.<wbr>airshipit.org</a><br>
>> > <a href="http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss" rel="noreferrer" target="_blank">http://lists.airshipit.org/<wbr>cgi-bin/mailman/listinfo/<wbr>airship-discuss</a><br>
>><br>
><br>
</div></div></blockquote></div><br></div>