Re: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

10 Sep 2020

      +Sreejith & Siraj – could you please take a look at Ben’s error below?

From: JAYARAJ, BEN <bj2343@att.com>
Sent: Thursday, September 10, 2020 2:40 PM
To: MCEUEN, MATT <MM9745@att.com>
Cc: BOSE, ARIJIT <ab7180@att.com>; airship-discuss@lists.airshipit.org; Matthew Hooker <mjhooker@gmail.com>
Subject: RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

Matt,

I have seen the same error shown below (CrashLoopBackOff) yesterday when I was trying the deploy.

Today we tried the deploy with your suggestions - we got the error shown below at the end of install.

Just a thought -  Should we have run swapoff -a before the script begins. (WARNING: No swap limit – in the error message below)

Thanks

Ben

0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? .The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? Thu Sep 10 13:25:35 CDT 2020 API not returning node list before timeout. Thu Sep 10 13:25:35 CDT 2020 General docker state report Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 3 Server Version: 18.06.3-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: a592beb5bc4c4092b1b1bac971afed27687340c5 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-186-generic Operating System: Ubuntu 16.04.7 LTS OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 31.42GiB Name: tma-airship ID: 4MOE:NJSF:L37S:ACYI:DNPJ:77BR:IFFL:L77C:6JXB:ZYEE:SVC2:2DUO Docker Root Dir: var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1 Labels: Experimental<https://index.docker.io/v1%3cbr%3eLabels:%3cbr%3eExperimental>: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: true WARNING: No swap limit support CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Thu Sep 10 13:25:35 CDT 2020 General cluster state report The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? The connection to the server 127.0.0.1:6553 was refused - did you specify the right host or port? + error 'running genesis' + set +x Error when running genesis. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy/treasuremap/../. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# kubectl get namespaces The connection to the server localhost:8080 was refused - did you specify the right host or port? root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# root@tma-airship:~/deploy/treasuremap/tools/deployment/aiab# kubectl get namespaces The connection to the server localhost:8080 was refused - did you specify the right host or port?

Thanks
Ben

From: MCEUEN, MATT <MM9745@att.com<mailto:MM9745@att.com>>
Sent: Wednesday, September 9, 2020 7:02 PM
To: Matthew Hooker <mjhooker@gmail.com<mailto:mjhooker@gmail.com>>
Cc: BOSE, ARIJIT <ab7180@att.com<mailto:ab7180@att.com>>; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org>; JAYARAJ, BEN <bj2343@att.com<mailto:bj2343@att.com>>
Subject: RE: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

Hi Matthew,

We are able to reproduce the issue, and have a fix that should be merged in the next couple hours.  Can you please give it another try and see if you still have any issues?

Sorry for the inconvenience!
Matt

From: Matthew Hooker <mjhooker@gmail.com<mailto:mjhooker@gmail.com>>
Sent: Saturday, September 5, 2020 3:07 PM
To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org>
Subject: [Airship-discuss] Airship in a Bottle - CrashLoopBackOff

Hi,

I've tried getting AiaB working several times on various machines but always end up with this:

NAMESPACE     NAME                                   READY   STATUS             RESTARTS   AGE

kube-system   auxiliary-etcd-ubuntu                  3/3     Running            0          13h

kube-system   bootstrap-armada-ubuntu                4/4     Running            0          13h

kube-system   haproxy-ubuntu                         1/1     Running            0          13h

kube-system   kubernetes-apiserver-ubuntu            1/1     Running            1          13h

kube-system   kubernetes-controller-manager-ubuntu   1/1     Running            0          13h

kube-system   kubernetes-etcd-ubuntu                 1/1     Running            0          13h

kube-system   kubernetes-proxy-f279z                 0/1     Pending            0          2m32s

kube-system   kubernetes-scheduler-ubuntu            0/1     CrashLoopBackOff   167        13h

This may be easily solvable by someone with sufficient knowledge of the underlying systems but I'm stuck.

Full transcript here:

https://drive.google.com/file/d/10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq/view<https://urldefense.proofpoint.com/v2/url?u=https-3A__drive.google.com_file_d_10l1hrtRNEMA0OYujk2V9sRILazQ2wZeq_view&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=RwVrX3OKWt8Yk0QYzlry0M-8EFvbguDaOkjDdqCBVjA&m=PgVPvNs6YK_9oDpvL7NEaetCQuiGXjSZPoVl0d_RUCk&s=why6aXiC3z7cLO2XvFQA581oGKUzUno2yoqUUDSQLyU&e=>

Any help appreciated

Matthew