[Airship-discuss] CoreDNS problem with liveness/readiness probe

[MCEUEN, MATT]

Hey Loic,

Good to hear that your multinode testing is progressing well, and great catch on the coredns probe!

I think it’s a good idea to err on the side of caution with our coredns probe.  Would you mind submitting a patchset to promenade, making your change here:
https://opendev.org/airship/promenade/src/branch/master/charts/coredns/templates/bin/_probe.sh.tpl#L15

That would allow folks to easily weigh in on whether those are the right values?
The patchset will also allow you to easily pull in the change to a AiaB multi-node deployment – you can make a local edit to the AiaB versions.yaml file:
https://opendev.org/airship/in-a-bottle/src/branch/master/deployment_files/global/v1.0demo/software/config/versions.yaml

In versions.yaml, find data.charts.coredns, and make these changes:
    Change location to: https://review.opendev.org/airship/promenade
    Change reference to:  the “refs/changes/x/y/z” reference from your promenade patchset

Thanks for catching this – it would be awesome if you can resolve it for others.  Let me know if any questions come up or if you need any pointers around the gerrit submit/review process.

Thanks,
Matt

From: Loic Le Gal <l.legal.astellia at gmail.com>
Sent: Friday, May 17, 2019 5:05 AM
To: airship-discuss at lists.airshipit.org
Subject: [Airship-discuss] CoreDNS problem with liveness/readiness probe

Hello Airshipers,

After succeeding to deploy a SingleNode config, I'm trying to deploy the multi-nodes-gate site of Airship in a botttle.
The first steps are Ok (Genesis creation and run on node n0) but the Undercloud deployment fails in the creation of coredns pods in kube-system namespace. (Endless Crashloop)

I fix manually this problem but want to fix it automatically before launching the whole Airship deployment (i.e. In an Openstack stack) because the initial coredns failure leads to side-effects that needs a new deployment...

After investigation, it turns out that the problem comes from a timing issue (may be from the infra I use) and I succeed to fix this manually by modifying the ConfigMap used by CoreDNS's probe and recreate the pods The problem was in the probe.sh file:

this command sometimes fails:
==> dig +trace +time=2 +tries=1 att.com<http://att.com> @127.0.0.1<http://127.0.0.1>

Increasing the time and retries fix the problem:
root at n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system -- dig +trace +time=2 +tries=1 att.com<http://att.com> @127.0.0.1<http://127.0.0.1>; echo $?

; <<>> DiG 9.11.2-P1 <<>> +trace +time=2 +tries=1 att.com<http://att.com> @127.0.0.1<http://127.0.0.1>
;; global options: +cmd
;; connection timed out; no servers could be reached
command terminated with exit code 9
9
root at n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system -- dig +trace +time=10 +tries=5 att.com<http://att.com> @127.0.0.1<http://127.0.0.1>; echo $?

; <<>> DiG 9.11.2-P1 <<>> +trace +time=10 +tries=5 att.com<http://att.com> @127.0.0.1<http://127.0.0.1>
;; global options: +cmd
;; Received 40 bytes from 127.0.0.1#53(127.0.0.1) in 5000 ms

0

(Inception inside!) I succeed in patching the Genesis stage to patch the genesis.sh script and finally patch the template file:
/etc/genesis/armada/assets/charts/coredns/templates/bin/_probe.sh.tpl

Unfortunately this file doesn't seems to be the one used to create the Pods' configmap.

Can someone tell me the easiest way to patch this readiness probe before after a Git clone and before launching Airship deployment ?

BR,
Loïc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190517/7e0197bd/attachment.html>