<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Arijit<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The VM is running Ubuntu 16.04.07. The VM has 32 GB ram, 100GB storage, 8vCPU. The error shown below is what is showing up today at the end of the install. This is running in an VMWare ESXI Server.
<o:p></o:p></p>
<p class="MsoNormal">Ben<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> BOSE, ARIJIT <ab7180@att.com> <br>
<b>Sent:</b> Thursday, September 10, 2020 3:56 PM<br>
<b>To:</b> MCEUEN, MATT <MM9745@att.com>; JAYARAJ, BEN <bj2343@att.com>; YASIN, SIRAJUDEEN MOHAMMED <sy495p@att.com>; PUNNAPUZHA, SREEJITH <sp863p@att.com><br>
<b>Cc:</b> airship-discuss@lists.airshipit.org; Matthew Hooker <mjhooker@gmail.com>; BAILEY, ALEXANDER <ab805p@att.com><br>
<b>Subject:</b> RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hello Ben,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can you please share the testing environment details.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The crashloop issue was resolved and we had a issue with the etcd server. This is the fix for it was pushed last night :
<a href="https://review.opendev.org/#/c/750866/">https://review.opendev.org/#/c/750866/</a>
<o:p></o:p></p>
<p class="MsoNormal">I had two success runs last night with this PS.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Request you to test it on a fresh 16.04 vm and getting the PS pulled in.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In case the problem persists , please share the “docker ps -a “.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">BR,<o:p></o:p></p>
<p class="MsoNormal">Arijit Bose<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> MCEUEN, MATT <<a href="mailto:MM9745@att.com">MM9745@att.com</a>>
<br>
<b>Sent:</b> Thursday, September 10, 2020 2:50 PM<br>
<b>To:</b> JAYARAJ, BEN <<a href="mailto:bj2343@att.com">bj2343@att.com</a>>; YASIN, SIRAJUDEEN MOHAMMED <<a href="mailto:sy495p@att.com">sy495p@att.com</a>>; PUNNAPUZHA, SREEJITH <<a href="mailto:sp863p@att.com">sp863p@att.com</a>><br>
<b>Cc:</b> BOSE, ARIJIT <<a href="mailto:ab7180@att.com">ab7180@att.com</a>>; <a href="mailto:airship-discuss@lists.airshipit.org">
airship-discuss@lists.airshipit.org</a>; Matthew Hooker <<a href="mailto:mjhooker@gmail.com">mjhooker@gmail.com</a>>; BAILEY, ALEXANDER <<a href="mailto:ab805p@att.com">ab805p@att.com</a>><br>
<b>Subject:</b> RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">+Sreejith & Siraj – could you please take a look at Ben’s error below?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> JAYARAJ, BEN <<a href="mailto:bj2343@att.com">bj2343@att.com</a>>
<br>
<b>Sent:</b> Thursday, September 10, 2020 2:40 PM<br>
<b>To:</b> MCEUEN, MATT <<a href="mailto:MM9745@att.com">MM9745@att.com</a>><br>
<b>Cc:</b> BOSE, ARIJIT <<a href="mailto:ab7180@att.com">ab7180@att.com</a>>; <a href="mailto:airship-discuss@lists.airshipit.org">
airship-discuss@lists.airshipit.org</a>; Matthew Hooker <<a href="mailto:mjhooker@gmail.com">mjhooker@gmail.com</a>><br>
<b>Subject:</b> RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Matt,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have seen the same error shown below (CrashLoopBackOff) yesterday when I was trying the deploy.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Today we tried the deploy with your suggestions - we got the error shown below at the end of install.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Just a thought - Should we have run swapoff -a before the script begins. (<span style="font-family:"Arial",sans-serif;color:black">WARNING: No swap limit</span> – in the error message below)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ben<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The
connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553
was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right
host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to
the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused -
did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The
connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553
was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right
host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to
the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused -
did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The
connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>.The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>Thu Sep 10 13:25:35 CDT 2020 API not returning
node list before timeout.<br>Thu Sep 10 13:25:35 CDT 2020 General docker state report<br>Containers: 0<br> Running: 0<br> Paused: 0<br> Stopped: 0<br>Images: 3<br>Server Version: 18.06.3-ce<br>Storage Driver:
overlay2<br> Backing Filesystem: extfs<br> Supports d_type: true<br> Native Overlay Diff: true<br>Logging Driver: json-file<br>Cgroup Driver: cgroupfs<br>Plugins:<br> Volume: local<br> Network: bridge host macvlan
null overlay<br> Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog<br>Swarm: inactive<br>Runtimes: runc<br>Default Runtime: runc<br>Init Binary: docker-init<br>containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e<br>runc
version: a592beb5bc4c4092b1b1bac971afed27687340c5<br>init version: fec3683<br>Security Options:<br> apparmor<br> seccomp<br> Profile: default<br>Kernel Version: 4.4.0-186-generic<br>Operating System: Ubuntu 16.04.7
LTS<br>OSType: linux<br>Architecture: x86_64<br>CPUs: 16<br>Total Memory: 31.42GiB<br>Name: tma-airship<br>ID: 4MOE:NJSF:L37S:ACYI:DNPJ:77BR:IFFL:L77C:6JXB:ZYEE:SVC2:2DUO<br>Docker Root Dir: var/lib/docker<br>Debug
Mode (client): false<br>Debug Mode (server): false<br>Registry: </span>
<span style="font-family:"Arial",sans-serif;color:blue"><a href="https://index.docker.io/v1%3cbr%3eLabels:%3cbr%3eExperimental">https://index.docker.io/v1<br>Labels:<br>Experimental</a></span><span style="font-family:"Arial",sans-serif;color:black">:
false<br>Insecure Registries:<br> 127.0.0.0/8<br>Live Restore Enabled: true<br><br>WARNING: No swap limit support<br>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES<br>Thu Sep 10 13:25:35 CDT 2020 General cluster
state report<br>The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port?<br>+ error 'running genesis'<br>+
set +x<br>Error when running genesis.<br>+ exit 1<br>+ clean<br>+ set +x<br>To remove files generated during this script's execution, delete /root/deploy/treasuremap/../.<br>This VM is disposable. Re-deployment in this same
VM will lead to unpredictable results.<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#
kubectl get namespaces<br>The connection to the server localhost:8080 was refused - did you specify the right host or port?<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#<br>root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab#
kubectl get namespaces<br>The connection to the server localhost:8080 was refused - did you specify the right host or port?
</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Ben<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> MCEUEN, MATT <<a href="mailto:MM9745@att.com">MM9745@att.com</a>>
<br>
<b>Sent:</b> Wednesday, September 9, 2020 7:02 PM<br>
<b>To:</b> Matthew Hooker <<a href="mailto:mjhooker@gmail.com">mjhooker@gmail.com</a>><br>
<b>Cc:</b> BOSE, ARIJIT <<a href="mailto:ab7180@att.com">ab7180@att.com</a>>; <a href="mailto:airship-discuss@lists.airshipit.org">
airship-discuss@lists.airshipit.org</a>; JAYARAJ, BEN <<a href="mailto:bj2343@att.com">bj2343@att.com</a>><br>
<b>Subject:</b> RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Matthew,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are able to reproduce the issue, and have a fix that should be merged in the next couple hours. Can you please give it another try and see if you still have any issues?<span style="font-size:12.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Sorry for the inconvenience!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Matt</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Matthew Hooker <<a href="mailto:mjhooker@gmail.com">mjhooker@gmail.com</a>>
<br>
<b>Sent:</b> Saturday, September 5, 2020 3:07 PM<br>
<b>To:</b> <a href="mailto:airship-discuss@lists.airshipit.org">airship-discuss@lists.airshipit.org</a><br>
<b>Subject:</b> [Airship-discuss] Airship in a Bottle - CrashLoopBackOff<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<pre style="margin:7.5pt"><span style="font-family:"Arial",sans-serif;color:black">Hi,</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">I've tried getting AiaB working several times on various machines but always end up with this:</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black">NAMESPACE NAME READY STATUS RESTARTS AGE<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system auxiliary-etcd-ubuntu 3/3 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system bootstrap-armada-ubuntu 4/4 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system haproxy-ubuntu 1/1 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-apiserver-ubuntu 1/1 Running 1 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-controller-manager-ubuntu 1/1 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-etcd-ubuntu 1/1 Running 0 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-proxy-f279z 0/1 Pending 0 2m32s<o:p></o:p></span></pre>
<pre style="margin:7.5pt"><span style="font-size:7.5pt;color:black">kube-system kubernetes-scheduler-ubuntu 0/1 CrashLoopBackOff 167 13h<o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-size:7.5pt;color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">This may be easily solvable by someone with sufficient knowledge of the underlying systems but I'm stuck.</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">Full transcript here:</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__drive.google.com_file_d_10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq_view&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=RwVrX3OKWt8Yk0QYzlry0M-8EFvbguDaOkjDdqCBVjA&m=PgVPvNs6YK_9oDpvL7NEaetCQuiGXjSZPoVl0d_RUCk&s=why6aXiC3z7cLO2XvFQA581oGKUzUno2yoqUUDSQLyU&e=">https://drive.google.com/file/d/10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq/view</a></span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">Any help appreciated</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="font-family:"Arial",sans-serif;color:black">Matthew</span><span style="color:black"><o:p></o:p></span></pre>
<pre style="margin:7.5pt;max-height:50vw;overflow:auto" id="gmail-consolelog"><span style="color:black"><o:p> </o:p></span></pre>
</div>
</div>
</body>
</html>