[Airship-discuss] osh-infra ldap error with promenade + armada

James Devon jr8586335 at gmail.com
Sat Jun 9 16:50:32 UTC 2018


Hello Roman,

Thanks for your answer.

What is the state of promenade + armada right now?
We are pretty excited about it.

Here is the output you required:

1) kubectl get pods  --all-namespaces -o wide
https://pastebin.com/b4kWGpBs

The only thing I see is the pod ldap-0 from the namespace osh-infra
Here is the output of kubectl describe pod -n osh-infra ldap-0
https://pastebin.com/sSBVZnch

I believe the interesting part is here:

   1. Events:
   2.   Type     Reason       Age                  From         Message
   3.   ----     ------       ----                 ----         -------
   4.   Warning  FailedMount  12m (x576 over 21h)  kubelet, n3  Unable to
   mount volumes for pod
   "ldap-0_osh-infra(85e73018-6b4c-11e8-b6e5-080027ee1df7)": timeout expired
   waiting for volumes to attach or mount for pod "osh-infra"/"ldap-0". list
   of unmounted volumes=[ldap-config ldap-data]. list of unattached
   volumes=[ldap-config ldap-data ldap-token-txdqp]
   5.   Warning  FailedMount  6m (x649 over 21h)   kubelet, n3
    MountVolume.WaitForAttach failed for volume
   "pvc-85dc46a5-6b4c-11e8-b6e5-080027ee1df7" : fail to check rbd image status
   with: (executable file not found in $PATH), rbd output: ()
   6.   Warning  FailedMount  2m (x651 over 21h)   kubelet, n3
    MountVolume.WaitForAttach failed for volume
   "pvc-85d6770b-6b4c-11e8-b6e5-080027ee1df7" : fail to check rbd image status
   with: (executable file not found in $PATH), rbd output: ()



2) docker ps on all nodes
https://pastebin.com/CaqLEv70

3) docker ps -a on all nodes
https://pastebin.com/wvB0qkbh

4) docker images on all nodes
https://pastebin.com/H07kEwqe

Best,
James

On Sat, Jun 9, 2018 at 1:51 PM, Roman Gorshunov <paye600 at gmail.com> wrote:

> Hello James,
>
> I didn’t see it before. Could you provide output of the following
> commands, if you still have this environment running. It would allow to get
> understanding of what has been deployed and what has not:
> kubectl get pods --all-namespaces -o wide       # on n0
> docker ps                                                       # on all
> nodes
> docker ps -a                                                    # on all
> nodes
> docker images                                                   # on all
> nodes
>
> Also note that artifacts-aic.atlantafoundry.com host where some of docker
> images are hosted is inaccessible at the moment, it could be that some
> images have not been downloaded successfully, team started mirroring images
> to dockerhub & quay.io. Right before the summit development was focused
> on making airship-in-a-bottle work, and migration from github/gerrithub to
> openstack foundation infrastructure is still ongoing.
>
> Apologize for the inconvenience, James.
>
> Thank you.
>
> > On 8 Jun 2018, at 21:18, James Devon <jr8586335 at gmail.com> wrote:
> >
> > Hi,
> >
> > After having tried airship-in-a-bottle, I'm trying to setup a Kubernetes
> cluster and install openstack on it using promenade and armada.
> >
> > I'm using the basic example : https://github.com/openstack/
> airship-promenade/tree/master/examples/basic, I've changed the ip
> addresses to match my environment of 4 nodes (n0 for genesis and then n1,
> n2, n3).
> >
> > At this point I'm facing only one problem:
> promenade-api.ucp.svc.cluster.local is resolvable but the resolution is
> bad (192.168.150.165).
> > So instead of promenade-api.ucp.svc.cluster.local, I used the pod
> address with the good port and it is working fine.
> >
> > Then, I am able to make n1,n2 and n3 join the cluster and I use the
> scripts from https://github.com/openstack/openstack-helm/tree/master/
> tools/deployment/armada to apply armada manifests.
> >
> > osh-infra is not able to start the ldap pod.
> >
> > Events:
> >   Type     Reason                  Age                From
>        Message
> >   ----     ------                  ----               ----
>        -------
> >   Warning  FailedScheduling        6m (x4 over 6m)    default-scheduler
>       pod has unbound PersistentVolumeClaims (repeated 3 times)
> >   Normal   Scheduled               6m                 default-scheduler
>       Successfully assigned ldap-0 to n3
> >   Normal   SuccessfulAttachVolume  6m
>  attachdetach-controller  AttachVolume.Attach succeeded for volume
> "pvc-85d6770b-6b4c-11e8-b6e5-080027ee1df7"
> >   Normal   SuccessfulAttachVolume  6m
>  attachdetach-controller  AttachVolume.Attach succeeded for volume
> "pvc-85dc46a5-6b4c-11e8-b6e5-080027ee1df7"
> >   Normal   SuccessfulMountVolume   6m                 kubelet, n3
>       MountVolume.SetUp succeeded for volume "ldap-token-txdqp"
> >   Warning  FailedMount             13s (x11 over 6m)  kubelet, n3
>       MountVolume.WaitForAttach failed for volume
> "pvc-85dc46a5-6b4c-11e8-b6e5-080027ee1df7" : fail to check rbd image
> status with: (executable file not found in $PATH), rbd output: ()
> >   Warning  FailedMount             12s (x11 over 6m)  kubelet, n3
>       MountVolume.WaitForAttach failed for volume
> "pvc-85d6770b-6b4c-11e8-b6e5-080027ee1df7" : fail to check rbd image
> status with: (executable file not found in $PATH), rbd output: ()
> >   Warning  FailedMount             2s (x3 over 4m)    kubelet, n3
>       Unable to mount volumes for pod "ldap-0_osh-infra(85e73018-6b4c-11e8-b6e5-080027ee1df7)":
> timeout expired waiting for volumes to attach or mount for pod
> "osh-infra"/"ldap-0". list of unmounted volumes=[ldap-config ldap-data].
> list of unattached volumes=[ldap-config ldap-data ldap-token-txdqp]
> >
> > I've checked the logs of ceph-osd and there's also something:
> >
> > ceph-2.1 | starting osd.2 at - osd_data /var/lib/ceph/osd/ceph-2
> /var/lib/ceph/journal/journal.2
> > ceph-2.1 | 2018-06-08 18:41:03.854234 7f7cd7d58e00 -1 journal
> FileJournal::_open: disabling aio for non-block journal.  Use
> journal_force_aio to force use of aio anyway
> > ceph-2.1 | 2018-06-08 18:41:03.855499 7f7cd7d58e00 -1 journal
> do_read_entry(8192): bad header magic
> > ceph-2.1 | 2018-06-08 18:41:03.855567 7f7cd7d58e00 -1 journal
> do_read_entry(8192): bad header magic
> > ceph-2.1 | 2018-06-08 18:41:03.901804 7f7cd7d58e00 -1 osd.2 0
> log_to_monitors {default=true}
> > ceph-2.1 | 2018-06-08 18:41:04.529480 7f7cbb2c7700 -1 osd.2 0 waiting
> for initial osdmap
> >
> > Did anybody encountered this error and how to fix it?
> >
> > Best,
> > James
> >
> > _______________________________________________
> > Airship-discuss mailing list
> > Airship-discuss at lists.airshipit.org
> > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20180609/67db8105/attachment-0001.html>


More information about the Airship-discuss mailing list