<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello Ben,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can you please share the testing environment details.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The crashloop issue was resolved and we had a issue with the etcd server. This is the fix for it was pushed last night :
<a href="https://review.opendev.org/#/c/750866/">https://review.opendev.org/#/c/750866/</a>
<o:p></o:p></p>
<p class="MsoNormal">I had two success runs last night with this PS.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Request you to test it on a fresh 16.04 vm and getting the PS pulled in.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In case the problem persists , please share the “docker ps -a “.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">BR,<o:p></o:p></p>
<p class="MsoNormal">Arijit Bose<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> MCEUEN, MATT <MM9745@att.com> <br>
<b>Sent:</b> Thursday, September 10, 2020 2:50 PM<br>
<b>To:</b> JAYARAJ, BEN <bj2343@att.com>; YASIN, SIRAJUDEEN MOHAMMED <sy495p@att.com>; PUNNAPUZHA, SREEJITH <sp863p@att.com><br>
<b>Cc:</b> BOSE, ARIJIT <ab7180@att.com>; airship-discuss@lists.airshipit.org; Matthew Hooker <mjhooker@gmail.com>; BAILEY, ALEXANDER <ab805p@att.com><br>
<b>Subject:</b> RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">+Sreejith & Siraj – could you please take a look at Ben’s error below?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> JAYARAJ, BEN <bj2343@att.com> <br>
<b>Sent:</b> Thursday, September 10, 2020 2:40 PM<br>
<b>To:</b> MCEUEN, MATT <MM9745@att.com><br>
<b>Cc:</b> BOSE, ARIJIT <ab7180@att.com>; airship-discuss@lists.airshipit.org; Matthew Hooker <mjhooker@gmail.com><br>
<b>Subject:</b> RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Matt,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have seen the same error shown below (CrashLoopBackOff) yesterday when I was trying the deploy.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Today we tried the deploy with your suggestions - we got the error shown below at the end of install.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Just a thought - Should we have run swapoff -a before the script begins. (<span style="font-family:"Arial",sans-serif;color:black">WARNING: No swap limit</span> – in the error message below)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ben<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The
connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553
was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right
host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to
the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused -
did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The
connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553
was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right
host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to
the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused -
did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The
connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>Thu Sep 10 13:25:35 CDT 2020 API not returning
node list before timeout.<br>Thu Sep 10 13:25:35 CDT 2020 General docker state report<br>Containers: 0<br> Running: 0<br> Paused: 0<br> Stopped: 0<br>Images: 3<br>Server Version: 18.06.3-ce<br>Storage Driver:
overlay2<br> Backing Filesystem: extfs<br> Supports d_type: true<br> Native Overlay Diff: true<br>Logging Driver: json-file<br>Cgroup Driver: cgroupfs<br>Plugins:<br> Volume: local<br> Network: bridge host macvlan
null overlay<br> Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog<br>Swarm: inactive<br>Runtimes: runc<br>Default Runtime: runc<br>Init Binary: docker-init<br>containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e<br>runc
version: a592beb5bc4c4092b1b1bac971afed27687340c5<br>init version: fec3683<br>Security Options:<br> apparmor<br> seccomp<br> Profile: default<br>Kernel Version: 4.4.0-186-generic<br>Operating System: Ubuntu 16.04.7
LTS<br>OSType: linux<br>Architecture: x86_64<br>CPUs: 16<br>Total Memory: 31.42GiB<br>Name: tma-airship<br>ID: 4MOE:NJSF:L37S:ACYI:DNPJ:77BR:IFFL:L77C:6JXB:ZYEE:SVC2:2DUO<br>Docker Root Dir: var/lib/docker<br>Debug
Mode (client): false<br>Debug Mode (server): false<br>Registry: </span>
<span style="font-family:"Arial",sans-serif;color:blue"><a href="https://index.docker.io/v1%3cbr%3eLabels:%3cbr%3eExperimental">https://index.docker.io/v1<br>Labels:<br>Experimental</a></span><span style="font-family:"Arial",sans-serif;color:black">:
false<br>Insecure Registries:<br> 127.0.0.0/8<br>Live Restore Enabled: true<br><br>WARNING: No swap limit support<br>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES<br>Thu Sep 10 13:25:35 CDT 2020 General cluster
state report<br>The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>+ error 'running genesis'<br>+
set +x<br>Error when running genesis.<br>+ exit 1<br>+ clean<br>+ set +x<br>To remove files generated during this script's execution, delete /root/deploy/treasuremap/../.<br>This VM is disposable. Re-deployment in this same
VM will lead to unpredictable results.<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#
kubectl get namespaces<br>The connection to the server localhost:8080 was refused - did you specify the right host or port?<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#
kubectl get namespaces<br>The connection to the server localhost:8080 was refused - did you specify the right host or port?
</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Ben<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> MCEUEN, MATT <<a href="mailto:MM9745@att.com">MM9745@att.com</a>>
<br>
<b>Sent:</b> Wednesday, September 9, 2020 7:02 PM<br>
<b>To:</b> Matthew Hooker <<a href="mailto:mjhooker@gmail.com">mjhooker@gmail.com</a>><br>
<b>Cc:</b> BOSE, ARIJIT <<a href="mailto:ab7180@att.com">ab7180@att.com</a>>; <a href="mailto:airship-discuss@lists.airshipit.org">
airship-discuss@lists.airshipit.org</a>; JAYARAJ, BEN <<a href="mailto:bj2343@att.com">bj2343@att.com</a>><br>
<b>Subject:</b> RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Matthew,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are able to reproduce the issue, and have a fix that should be merged in the next couple hours. Can you please give it another try and see if you still have any issues?<span style="font-size:12.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Sorry for the inconvenience!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Matt</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Matthew Hooker <<a href="mailto:mjhooker@gmail.com">mjhooker@gmail.com</a>>
<br>
<b>Sent:</b> Saturday, September 5, 2020 3:07 PM<br>
<b>To:</b> <a href="mailto:airship-discuss@lists.airshipit.org">airship-discuss@lists.airshipit.org</a><br>
<b>Subject:</b> [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<pre style="margin:7.5pt"><span style="font-family:"Arial",sans-serif;color:black">Hi,</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">I've tried getting AiaB working several times on various machines but always end up with this:</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black">NAMESPACE NAME READY STATUS RESTARTS AGE<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system auxiliary-etcd-ubuntu 3/3 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system bootstrap-armada-ubuntu 4/4 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system haproxy-ubuntu 1/1 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-apiserver-ubuntu 1/1 Running 1 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-controller-manager-ubuntu 1/1 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-etcd-ubuntu 1/1 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-proxy-f279z 0/1 Pending 0 2m32s<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-scheduler-ubuntu 0/1 CrashLoopBackOff 167 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">This may be easily solvable by someone with sufficient knowledge of the underlying systems but I'm stuck.</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">Full transcript here:</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__drive.google.com_file_d_10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq_view&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=RwVrX3OKWt8Yk0QYzlry0M-8EFvbguDaOkjDdqCBVjA&m=PgVPvNs6YK_9oDpvL7NEaetCQuiGXjSZPoVl0d_RUCk&s=why6aXiC3z7cLO2XvFQA581oGKUzUno2yoqUUDSQLyU&e=">https://drive.google.com/file/d/10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq/view</a></span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">Any help appreciated</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">Matthew</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
</div>
</div>
</body>
</html>