<div dir="ltr">Thanks Matt for the infos.<div>In fact I found the root cause of this problem which was not from the probe itself but from the dns server.</div><div>It comes from a conflict between coreDNS and dnsmasq... because I'm trying to remove the virsh layer of the Aiab multinode Site to split the deployment in openstack VMs.</div><div>So this is a side-effect of my modifications and a soon as dnsmasq is stopped, the CoreDNS start successfully without modifying the probe.</div><div>So I succeed to finish the Genesis Phase and I have a functional Airflow, but as you can figure out I face other problems... to be described in another thread :)</div><div><br></div><div>Loïc</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le ven. 17 mai 2019 à 18:02, MCEUEN, MATT <<a href="mailto:MM9745@att.com">MM9745@att.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-4569031256694704774WordSection1">
<p class="MsoNormal">Hey Loic,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Good to hear that your multinode testing is progressing well, and great catch on the coredns probe!<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I think it’s a good idea to err on the side of caution with our coredns probe. Would you mind submitting a patchset to promenade, making your change here:<u></u><u></u></p>
<p class="MsoNormal"><a href="https://opendev.org/airship/promenade/src/branch/master/charts/coredns/templates/bin/_probe.sh.tpl#L15" target="_blank">https://opendev.org/airship/promenade/src/branch/master/charts/coredns/templates/bin/_probe.sh.tpl#L15</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">That would allow folks to easily weigh in on whether those are the right values?<u></u><u></u></p>
<p class="MsoNormal">The patchset will also allow you to easily pull in the change to a AiaB multi-node deployment – you can make a local edit to the AiaB versions.yaml file:<u></u><u></u></p>
<p class="MsoNormal"><a href="https://opendev.org/airship/in-a-bottle/src/branch/master/deployment_files/global/v1.0demo/software/config/versions.yaml" target="_blank">https://opendev.org/airship/in-a-bottle/src/branch/master/deployment_files/global/v1.0demo/software/config/versions.yaml</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">In versions.yaml, find data.charts.coredns, and make these changes:<u></u><u></u></p>
<p class="MsoNormal"> Change location to: <a href="https://review.opendev.org/airship/promenade" target="_blank">
https://review.opendev.org/airship/promenade</a> <u></u><u></u></p>
<p class="MsoNormal"> Change reference to: the “refs/changes/x/y/z” reference from your promenade patchset<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks for catching this – it would be awesome if you can resolve it for others. Let me know if any questions come up or if you need any pointers around the gerrit submit/review process.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Matt<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>From:</b> Loic Le Gal <<a href="mailto:l.legal.astellia@gmail.com" target="_blank">l.legal.astellia@gmail.com</a>> <br>
<b>Sent:</b> Friday, May 17, 2019 5:05 AM<br>
<b>To:</b> <a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a><br>
<b>Subject:</b> [Airship-discuss] CoreDNS problem with liveness/readiness probe<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hello Airshipers,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">After succeeding to deploy a SingleNode config, I'm trying to deploy the multi-nodes-gate site of Airship in a botttle.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The first steps are Ok (Genesis creation and run on node n0) but the Undercloud deployment fails in the creation of coredns pods in kube-system namespace. (Endless Crashloop)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I fix manually this problem but want to fix it automatically before launching the whole Airship deployment (i.e. In an Openstack stack) because the initial coredns failure leads to side-effects that needs a new deployment...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">After investigation, it turns out that the problem comes from a timing issue (may be from the infra I use) and I succeed to fix this manually by modifying the ConfigMap used by CoreDNS's probe and recreate the pods The problem was in the
probe.sh file:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">this command sometimes fails: <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">==> dig +trace +time=2 +tries=1 <a href="http://att.com" target="_blank">att.com</a> @<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Increasing the time and retries fix the problem:<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">root@n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system -- dig +trace +time=2 +tries=1
<a href="http://att.com" target="_blank">att.com</a> @<a href="http://127.0.0.1" target="_blank">127.0.0.1</a>; echo $?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">; <<>> DiG 9.11.2-P1 <<>> +trace +time=2 +tries=1 <a href="http://att.com" target="_blank">
att.com</a> @<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">;; global options: +cmd<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">;; connection timed out; no servers could be reached<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">command terminated with exit code 9<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">9<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">root@n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system -- dig +trace +time=10 +tries=5
<a href="http://att.com" target="_blank">att.com</a> @<a href="http://127.0.0.1" target="_blank">127.0.0.1</a>; echo $?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">; <<>> DiG 9.11.2-P1 <<>> +trace +time=10 +tries=5 <a href="http://att.com" target="_blank">
att.com</a> @<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">;; global options: +cmd<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">;; Received 40 bytes from 127.0.0.1#53(127.0.0.1) in 5000 ms<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">0<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">(Inception inside!) I succeed in patching the Genesis stage to patch the genesis.sh script and finally patch the template file: <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">/etc/genesis/armada/assets/charts/coredns/templates/bin/_probe.sh.tpl<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Unfortunately this file doesn't seems to be the one used to create the Pods' configmap.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Can someone tell me the easiest way to patch this readiness probe before after a Git clone and before launching Airship deployment ?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">BR,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Loïc<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>