[Airship-discuss] CoreDNS problem with liveness/readiness probe

[Loic Le Gal]

Thanks Matt for the infos.
In fact I found the root cause of this problem which was not from the probe
itself but from the dns server.
It comes from a conflict between coreDNS and dnsmasq... because I'm trying
to remove the virsh layer of the Aiab multinode Site to split the
deployment in openstack VMs.
So this is a side-effect of my modifications and a soon as dnsmasq is
stopped, the CoreDNS start successfully without modifying the probe.
So I succeed to finish the Genesis Phase and I have a functional Airflow,
but as you can figure out I face other problems... to be described in
another thread :)

Loïc

Le ven. 17 mai 2019 à 18:02, MCEUEN, MATT <MM9745 at att.com> a écrit :

> Hey Loic,
>
>
>
> Good to hear that your multinode testing is progressing well, and great
> catch on the coredns probe!
>
>
>
> I think it’s a good idea to err on the side of caution with our coredns
> probe.  Would you mind submitting a patchset to promenade, making your
> change here:
>
>
> https://opendev.org/airship/promenade/src/branch/master/charts/coredns/templates/bin/_probe.sh.tpl#L15
>
>
>
> That would allow folks to easily weigh in on whether those are the right
> values?
>
> The patchset will also allow you to easily pull in the change to a AiaB
> multi-node deployment – you can make a local edit to the AiaB versions.yaml
> file:
>
>
> https://opendev.org/airship/in-a-bottle/src/branch/master/deployment_files/global/v1.0demo/software/config/versions.yaml
>
>
>
> In versions.yaml, find data.charts.coredns, and make these changes:
>
>     Change location to: https://review.opendev.org/airship/promenade
>
>     Change reference to:  the “refs/changes/x/y/z” reference from your
> promenade patchset
>
>
>
> Thanks for catching this – it would be awesome if you can resolve it for
> others.  Let me know if any questions come up or if you need any pointers
> around the gerrit submit/review process.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From:* Loic Le Gal <l.legal.astellia at gmail.com>
> *Sent:* Friday, May 17, 2019 5:05 AM
> *To:* airship-discuss at lists.airshipit.org
> *Subject:* [Airship-discuss] CoreDNS problem with liveness/readiness probe
>
>
>
> Hello Airshipers,
>
>
>
> After succeeding to deploy a SingleNode config, I'm trying to deploy the
> multi-nodes-gate site of Airship in a botttle.
>
> The first steps are Ok (Genesis creation and run on node n0) but the
> Undercloud deployment fails in the creation of coredns pods in kube-system
> namespace. (Endless Crashloop)
>
>
>
> I fix manually this problem but want to fix it automatically before
> launching the whole Airship deployment (i.e. In an Openstack stack) because
> the initial coredns failure leads to side-effects that needs a new
> deployment...
>
>
>
> After investigation, it turns out that the problem comes from a timing
> issue (may be from the infra I use) and I succeed to fix this manually by
> modifying the ConfigMap used by CoreDNS's probe and recreate the pods The
> problem was in the probe.sh file:
>
>
>
> this command sometimes fails:
>
> ==> dig +trace +time=2 +tries=1 att.com @127.0.0.1
>
>
>
> Increasing the time and retries fix the problem:
>
> root at n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n
> kube-system -- dig +trace +time=2 +tries=1 att.com @127.0.0.1; echo $?
>
>
>
> ; <<>> DiG 9.11.2-P1 <<>> +trace +time=2 +tries=1 att.com @127.0.0.1
>
> ;; global options: +cmd
>
> ;; connection timed out; no servers could be reached
>
> command terminated with exit code 9
>
> 9
>
> root at n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n
> kube-system -- dig +trace +time=10 +tries=5 att.com @127.0.0.1; echo $?
>
>
>
> ; <<>> DiG 9.11.2-P1 <<>> +trace +time=10 +tries=5 att.com @127.0.0.1
>
> ;; global options: +cmd
>
> ;; Received 40 bytes from 127.0.0.1#53(127.0.0.1) in 5000 ms
>
>
>
> 0
>
>
>
> (Inception inside!) I succeed in patching the Genesis stage to patch the
> genesis.sh script and finally patch the template file:
>
> /etc/genesis/armada/assets/charts/coredns/templates/bin/_probe.sh.tpl
>
>
>
> Unfortunately this file doesn't seems to be the one used to create the
> Pods' configmap.
>
>
>
> Can someone tell me the easiest way to patch this readiness probe before
> after a Git clone and before launching Airship deployment ?
>
>
>
> BR,
>
> Loïc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190523/927c2efe/attachment.html>