[Airship-discuss] Airship in a Bottle - CrashLoopBackOff

Thu Sep 10 19:55:30 UTC 2020

Hello Ben,

Can you please share the testing environment details.

The crashloop issue was resolved and we had a issue with the etcd server. This is the fix for it was pushed last night : https://review.opendev.org/#/c/750866/
I had two success runs last night with this PS.

Request you to test it on  a fresh 16.04 vm and getting the PS pulled in.

In case the problem persists , please share the “docker ps -a “.

BR,
Arijit Bose

From: MCEUEN, MATT <MM9745 at att.com>
Sent: Thursday, September 10, 2020 2:50 PM
To: JAYARAJ, BEN <bj2343 at att.com>; YASIN, SIRAJUDEEN MOHAMMED <sy495p at att.com>; PUNNAPUZHA, SREEJITH <sp863p at att.com>
Cc: BOSE, ARIJIT <ab7180 at att.com>; airship-discuss at lists.airshipit.org; Matthew Hooker <mjhooker at gmail.com>; BAILEY, ALEXANDER <ab805p at att.com>
Subject: RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

+Sreejith & Siraj – could you please take a look at Ben’s error below?

From: JAYARAJ, BEN <bj2343 at att.com>
Sent: Thursday, September 10, 2020 2:40 PM
To: MCEUEN, MATT <MM9745 at att.com>
Cc: BOSE, ARIJIT <ab7180 at att.com>; airship-discuss at lists.airshipit.org; Matthew Hooker <mjhooker at gmail.com>
Subject: RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

Matt,

I have seen the same error shown below (CrashLoopBackOff) yesterday when I was trying the deploy.

Today we tried the deploy with your suggestions - we got the error shown below at the end of install.

Just a thought -  Should we have run swapoff -a before the script begins. (WARNING: No swap limit – in the error message below)

Thanks

Ben

0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? Thu Sep 10 13:25:35 CDT 2020 API not returning node list before timeout. Thu Sep 10 13:25:35 CDT 2020 General docker state report Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 3 Server Version: 18.06.3-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: a592beb5bc4c4092b1b1bac971afed27687340c5 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-186-generic Operating System: Ubuntu 16.04.7 LTS OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 31.42GiB Name: tma-airship ID: 4MOE:NJSF:L37S:ACYI:DNPJ:77BR:IFFL:L77C:6JXB:ZYEE:SVC2:2DUO Docker Root Dir: var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1 Labels: Experimental<https://index.docker.io/v1%3cbr%3eLabels:%3cbr%3eExperimental>: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: true WARNING: No swap limit support CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Thu Sep 10 13:25:35 CDT 2020 General cluster state report The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? + error 'running genesis' + set +x Error when running genesis. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy/treasuremap/../. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# kubectl get namespaces The connection to the server localhost:8080 was refused - did you specify the right host or port? root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root at tma-airship:~/deploy/treasuremap/tools/deployment/aiab# kubectl get namespaces The connection to the server localhost:8080 was refused - did you specify the right host or port?

Thanks
Ben

From: MCEUEN, MATT <MM9745 at att.com<mailto:MM9745 at att.com>>
Sent: Wednesday, September 9, 2020 7:02 PM
To: Matthew Hooker <mjhooker at gmail.com<mailto:mjhooker at gmail.com>>
Cc: BOSE, ARIJIT <ab7180 at att.com<mailto:ab7180 at att.com>>; airship-discuss at lists.airshipit.org<mailto:airship-discuss at lists.airshipit.org>; JAYARAJ, BEN <bj2343 at att.com<mailto:bj2343 at att.com>>
Subject: RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

Hi Matthew,

We are able to reproduce the issue, and have a fix that should be merged in the next couple hours.  Can you please give it another try and see if you still have any issues?

Sorry for the inconvenience!
Matt

From: Matthew Hooker <mjhooker at gmail.com<mailto:mjhooker at gmail.com>>
Sent: Saturday, September 5, 2020 3:07 PM
To: airship-discuss at lists.airshipit.org<mailto:airship-discuss at lists.airshipit.org>
Subject: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

Hi,

I've tried getting AiaB working several times on various machines but always end up with this:

NAMESPACE     NAME                                   READY   STATUS             RESTARTS   AGE

kube-system   auxiliary-etcd-ubuntu                  3/3     Running            0          13h

kube-system   bootstrap-armada-ubuntu                4/4     Running            0          13h

kube-system   haproxy-ubuntu                         1/1     Running            0          13h

kube-system   kubernetes-apiserver-ubuntu            1/1     Running            1          13h

kube-system   kubernetes-controller-manager-ubuntu   1/1     Running            0          13h

kube-system   kubernetes-etcd-ubuntu                 1/1     Running            0          13h

kube-system   kubernetes-proxy-f279z                 0/1     Pending            0          2m32s

kube-system   kubernetes-scheduler-ubuntu            0/1     CrashLoopBackOff   167        13h

This may be easily solvable by someone with sufficient knowledge of the underlying systems but I'm stuck.

Full transcript here:

https://drive.google.com/file/d/10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq/view<https://urldefense.proofpoint.com/v2/url?u=https-3A__drive.google.com_file_d_10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq_view&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=RwVrX3OKWt8Yk0QYzlry0M-8EFvbguDaOkjDdqCBVjA&m=PgVPvNs6YK_9oDpvL7NEaetCQuiGXjSZPoVl0d_RUCk&s=why6aXiC3z7cLO2XvFQA581oGKUzUno2yoqUUDSQLyU&e=>

Any help appreciated

Matthew

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20200910/76a4444f/attachment-0001.html>