<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hey Loic,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Good to hear that your multinode testing is progressing well, and great catch on the coredns probe!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I think it’s a good idea to err on the side of caution with our coredns probe. Would you mind submitting a patchset to promenade, making your change here:<o:p></o:p></p>
<p class="MsoNormal"><a href="https://opendev.org/airship/promenade/src/branch/master/charts/coredns/templates/bin/_probe.sh.tpl#L15">https://opendev.org/airship/promenade/src/branch/master/charts/coredns/templates/bin/_probe.sh.tpl#L15</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">That would allow folks to easily weigh in on whether those are the right values?<o:p></o:p></p>
<p class="MsoNormal">The patchset will also allow you to easily pull in the change to a AiaB multi-node deployment – you can make a local edit to the AiaB versions.yaml file:<o:p></o:p></p>
<p class="MsoNormal"><a href="https://opendev.org/airship/in-a-bottle/src/branch/master/deployment_files/global/v1.0demo/software/config/versions.yaml">https://opendev.org/airship/in-a-bottle/src/branch/master/deployment_files/global/v1.0demo/software/config/versions.yaml</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In versions.yaml, find data.charts.coredns, and make these changes:<o:p></o:p></p>
<p class="MsoNormal"> Change location to: <a href="https://review.opendev.org/airship/promenade">
https://review.opendev.org/airship/promenade</a> <o:p></o:p></p>
<p class="MsoNormal"> Change reference to: the “refs/changes/x/y/z” reference from your promenade patchset<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for catching this – it would be awesome if you can resolve it for others. Let me know if any questions come up or if you need any pointers around the gerrit submit/review process.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Matt<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Loic Le Gal <l.legal.astellia@gmail.com> <br>
<b>Sent:</b> Friday, May 17, 2019 5:05 AM<br>
<b>To:</b> airship-discuss@lists.airshipit.org<br>
<b>Subject:</b> [Airship-discuss] CoreDNS problem with liveness/readiness probe<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hello Airshipers,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">After succeeding to deploy a SingleNode config, I'm trying to deploy the multi-nodes-gate site of Airship in a botttle.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The first steps are Ok (Genesis creation and run on node n0) but the Undercloud deployment fails in the creation of coredns pods in kube-system namespace. (Endless Crashloop)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I fix manually this problem but want to fix it automatically before launching the whole Airship deployment (i.e. In an Openstack stack) because the initial coredns failure leads to side-effects that needs a new deployment...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">After investigation, it turns out that the problem comes from a timing issue (may be from the infra I use) and I succeed to fix this manually by modifying the ConfigMap used by CoreDNS's probe and recreate the pods The problem was in the
probe.sh file:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">this command sometimes fails: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">==> dig +trace +time=2 +tries=1 <a href="http://att.com">att.com</a> @<a href="http://127.0.0.1">127.0.0.1</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Increasing the time and retries fix the problem:<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">root@n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system -- dig +trace +time=2 +tries=1
<a href="http://att.com">att.com</a> @<a href="http://127.0.0.1">127.0.0.1</a>; echo $?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">; <<>> DiG 9.11.2-P1 <<>> +trace +time=2 +tries=1 <a href="http://att.com">
att.com</a> @<a href="http://127.0.0.1">127.0.0.1</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">;; global options: +cmd<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">;; connection timed out; no servers could be reached<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">command terminated with exit code 9<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">9<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">root@n0:/home/ubuntu# kubectl exec coredns-6494d65b66-dk8g9 -n kube-system -- dig +trace +time=10 +tries=5
<a href="http://att.com">att.com</a> @<a href="http://127.0.0.1">127.0.0.1</a>; echo $?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">; <<>> DiG 9.11.2-P1 <<>> +trace +time=10 +tries=5 <a href="http://att.com">
att.com</a> @<a href="http://127.0.0.1">127.0.0.1</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">;; global options: +cmd<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">;; Received 40 bytes from 127.0.0.1#53(127.0.0.1) in 5000 ms<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">0<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">(Inception inside!) I succeed in patching the Genesis stage to patch the genesis.sh script and finally patch the template file: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">/etc/genesis/armada/assets/charts/coredns/templates/bin/_probe.sh.tpl<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Unfortunately this file doesn't seems to be the one used to create the Pods' configmap.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Can someone tell me the easiest way to patch this readiness probe before after a Git clone and before launching Airship deployment ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">BR,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Loïc<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>