[Airship-discuss] CoreDNS problem with liveness/readiness probe

[Loic Le Gal]

Hello Airshipers,

After succeeding to deploy a SingleNode config, I'm trying to deploy the
multi-nodes-gate site of Airship in a botttle.
The first steps are Ok (Genesis creation and run on node n0) but the
Undercloud deployment fails in the creation of coredns pods in kube-system
namespace. (Endless Crashloop)

I fix manually this problem but want to fix it automatically before
launching the whole Airship deployment (i.e. In an Openstack stack) because
the initial coredns failure leads to side-effects that needs a new
deployment...

After investigation, it turns out that the problem comes from a timing
issue (may be from the infra I use) and I succeed to fix this manually by
modifying the ConfigMap used by CoreDNS's probe and recreate the pods The
problem was in the probe.sh file:

this command sometimes fails:
==> dig +trace +time=2 +tries=1 att.com @127.0.0.1

Increasing the time and retries fix the problem:
root at n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system
-- dig +trace +time=2 +tries=1 att.com @127.0.0.1; echo $?

; <<>> DiG 9.11.2-P1 <<>> +trace +time=2 +tries=1 att.com @127.0.0.1
;; global options: +cmd
;; connection timed out; no servers could be reached
command terminated with exit code 9
9
root at n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system
-- dig +trace +time=10 +tries=5 att.com @127.0.0.1; echo $?

; <<>> DiG 9.11.2-P1 <<>> +trace +time=10 +tries=5 att.com @127.0.0.1
;; global options: +cmd
;; Received 40 bytes from 127.0.0.1#53(127.0.0.1) in 5000 ms

0

(Inception inside!) I succeed in patching the Genesis stage to patch the
genesis.sh script and finally patch the template file:
/etc/genesis/armada/assets/charts/coredns/templates/bin/_probe.sh.tpl

Unfortunately this file doesn't seems to be the one used to create the
Pods' configmap.

Can someone tell me the easiest way to patch this readiness probe before
after a Git clone and before launching Airship deployment ?

BR,
Loïc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190517/6485f7a3/attachment-0001.html>