<div dir="ltr">Hi,<div><br></div><div>After having tried airship-in-a-bottle, I'm trying to setup a Kubernetes cluster and install openstack on it using promenade and armada.</div><div><br></div><div>I'm using the basic example : <a href="https://github.com/openstack/airship-promenade/tree/master/examples/basic">https://github.com/openstack/airship-promenade/tree/master/examples/basic</a>, I've changed the ip addresses to match my environment of 4 nodes (n0 for genesis and then n1, n2, n3).</div><div><br></div><div>At this point I'm facing only one problem: promenade-api.ucp.svc.cluster.local is resolvable but the resolution is bad (192.168.150.165).</div><div>So instead of promenade-api.ucp.svc.cluster.local, I used the pod address with the good port and it is working fine.</div><div><br></div><div>Then, I am able to make n1,n2 and n3 join the cluster and I use the scripts from <a href="https://github.com/openstack/openstack-helm/tree/master/tools/deployment/armada">https://github.com/openstack/openstack-helm/tree/master/tools/deployment/armada</a> to apply armada manifests.</div><div><br></div><div>osh-infra is not able to start the ldap pod.</div><div><br></div><div>
<span></span>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Events:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Type <span class="gmail-Apple-converted-space"> </span>Reason<span class="gmail-Apple-converted-space"> </span>Age<span class="gmail-Apple-converted-space"> </span>From <span class="gmail-Apple-converted-space"> </span>Message</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>---- <span class="gmail-Apple-converted-space"> </span>------<span class="gmail-Apple-converted-space"> </span>---- <span class="gmail-Apple-converted-space"> </span>---- <span class="gmail-Apple-converted-space"> </span>-------</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Warning<span class="gmail-Apple-converted-space"> </span>FailedScheduling<span class="gmail-Apple-converted-space"> </span>6m (x4 over 6m)<span class="gmail-Apple-converted-space"> </span>default-scheduler<span class="gmail-Apple-converted-space"> </span>pod has unbound PersistentVolumeClaims (repeated 3 times)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Normal <span class="gmail-Apple-converted-space"> </span>Scheduled <span class="gmail-Apple-converted-space"> </span>6m <span class="gmail-Apple-converted-space"> </span>default-scheduler<span class="gmail-Apple-converted-space"> </span>Successfully assigned ldap-0 to n3</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Normal <span class="gmail-Apple-converted-space"> </span>SuccessfulAttachVolume<span class="gmail-Apple-converted-space"> </span>6m <span class="gmail-Apple-converted-space"> </span>attachdetach-controller<span class="gmail-Apple-converted-space"> </span>AttachVolume.Attach succeeded for volume "pvc-85d6770b-6b4c-11e8-b6e5-080027ee1df7"</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Normal <span class="gmail-Apple-converted-space"> </span>SuccessfulAttachVolume<span class="gmail-Apple-converted-space"> </span>6m <span class="gmail-Apple-converted-space"> </span>attachdetach-controller<span class="gmail-Apple-converted-space"> </span>AttachVolume.Attach succeeded for volume "pvc-85dc46a5-6b4c-11e8-b6e5-080027ee1df7"</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Normal <span class="gmail-Apple-converted-space"> </span>SuccessfulMountVolume <span class="gmail-Apple-converted-space"> </span>6m <span class="gmail-Apple-converted-space"> </span>kubelet, n3<span class="gmail-Apple-converted-space"> </span>MountVolume.SetUp succeeded for volume "ldap-token-txdqp"</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Warning<span class="gmail-Apple-converted-space"> </span>FailedMount <span class="gmail-Apple-converted-space"> </span>13s (x11 over 6m)<span class="gmail-Apple-converted-space"> </span>kubelet, n3<span class="gmail-Apple-converted-space"> </span>MountVolume.WaitForAttach failed for volume "pvc-85dc46a5-6b4c-11e8-b6e5-080027ee1df7" : fail to check rbd image status with: (executable file not found in $PATH), rbd output: ()</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Warning<span class="gmail-Apple-converted-space"> </span>FailedMount <span class="gmail-Apple-converted-space"> </span>12s (x11 over 6m)<span class="gmail-Apple-converted-space"> </span>kubelet, n3<span class="gmail-Apple-converted-space"> </span>MountVolume.WaitForAttach failed for volume "pvc-85d6770b-6b4c-11e8-b6e5-080027ee1df7" : fail to check rbd image status with: (executable file not found in $PATH), rbd output: ()</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Warning<span class="gmail-Apple-converted-space"> </span>FailedMount <span class="gmail-Apple-converted-space"> </span>2s (x3 over 4m)<span class="gmail-Apple-converted-space"> </span>kubelet, n3<span class="gmail-Apple-converted-space"> </span>Unable to mount volumes for pod "ldap-0_osh-infra(85e73018-6b4c-11e8-b6e5-080027ee1df7)": timeout expired waiting for volumes to attach or mount for pod "osh-infra"/"ldap-0". list of unmounted volumes=[ldap-config ldap-data]. list of unattached volumes=[ldap-config ldap-data ldap-token-txdqp]</span></p></div><div><br></div><div>I've checked the logs of ceph-osd and there's also something:</div><div><br></div><div>
<span></span>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)"><b>ceph-2.1 | </b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures">starting osd.2 at - osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/journal/journal.2</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(195,55,32);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)"><b>ceph-2.1 | </b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures"><b>2018-06-08 18:41:03.854234 7f7cd7d58e00 -1 journal FileJournal::_open: disabling aio for non-block journal.<span class="gmail-Apple-converted-space"> </span>Use journal_force_aio to force use of aio anyway</b></span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(195,55,32);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)"><b>ceph-2.1 | </b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures"><b>2018-06-08 18:41:03.855499 7f7cd7d58e00 -1 journal do_read_entry(8192): bad header magic</b></span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(195,55,32);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)"><b>ceph-2.1 | </b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures"><b>2018-06-08 18:41:03.855567 7f7cd7d58e00 -1 journal do_read_entry(8192): bad header magic</b></span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(195,55,32);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)"><b>ceph-2.1 | </b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures"><b>2018-06-08 18:41:03.901804 7f7cd7d58e00 -1 osd.2 0 log_to_monitors {default=true}</b></span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(195,55,32);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)"><b>ceph-2.1 | </b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures"><b>2018-06-08 18:41:04.529480 7f7cbb2c7700 -1 osd.2 0 waiting for initial osdmap</b></span></p></div><div><br></div><div>Did anybody encountered this error and how to fix it? </div><div><br></div><div>Best,</div><div>James</div><div><br></div></div>