Error installing Airship In A Bottle
Hi Nick First thing i would do, is look for apiserver and etcd container errors Can you check the following as well 1. Verify if etcd service is up or not On VM, run netstat -ntulp | grep 2379 2. journalctl -u kubelet.service | grep "Invalid value" Look for errros, it might point out any pod spec issues Also please provide following information 3. cat /etc/kubernetes/manifests/kubernetes-etcd.yaml Check under volumes section if the names have any CAPITAL letters 4. hostname -s My hunch is you have hostname with a CAPS letter which is triggering this failure. Above commands should give an idea if that is the cause. If thats the case, change your hostname to all lowercase and then run airship-in-a-bottle. Regards, Hemanth
Message: 3 Date: Tue, 12 Mar 2019 17:01:51 -0400 From: Nick Chase <nchase@mirantis.com> To: "MCEUEN, MATT" <MM9745@att.com> Cc: Evgeny L <eli@mirantis.com>, "airship-discuss@lists.airshipit.org" <airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] Error installing Airship In A Bottle Message-ID: <CA+p99G=RzD9+4qAOF5bv-Z=fG0phye=55CxMPnC7bytG+RSg4w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
OK, so if it IS a timeout thing, if you can tell me how to increase it, I'll appreciate it, because the script finishes, it just finishes with an error.
So the Airship script log is here: http://paste.openstack.org/show/747758/ Docker processes here: http://paste.openstack.org/show/747759/
Thanks....
---- Nick
On Wed, Mar 13, 2019 at 6:39 PM MCEUEN, MATT <MM9745@att.com> wrote:
Agree, and I’d add to check the docker logs of the kube apiserver to make sure it’s starting up healthy.
If you’re not seeing a smoking gun, can you do a `docker ps -a` and pastebin the results?
Thanks,
Matt
*From:* Evgeny L <eli@mirantis.com> *Sent:* Wednesday, March 13, 2019 12:59 PM *To:* Nick Chase <nchase@mirantis.com> *Cc:* airship-discuss@lists.airshipit.org *Subject:* Re: [Airship-discuss] Error installing Airship In A Bottle
Hi Nick,
As far as I remember these errors are just retries that wait until Kubernetes is up and running, have you tried to wait a bit?
If it still fails for you, can you please send us a complete log? You can use http://paste.openstack.org/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=8BzvCwZPVXNqnW1cYOCO0Av7scqQCiZlumjUr7A2dk8&e=> for that.
Thanks!
On Mon, Mar 11, 2019 at 8:59 PM Nick Chase <nchase@mirantis.com> wrote:
I'm trying to install AIB on a Unbutu Virtualbox VM with 48GB of RAM, 4 CPU and a 300GB hard drive. Everything seems to go along fine, until we get here:
Digest: sha256:badd2b1da29d4d530b10b920f64bf66a1b41150db46c3c99b49d56f3f18a82db Status: Downloaded newer image for gcr.io/google_containers/hyperkube-amd64:v1.10.2 <https://urldefense.proofpoint.com/v2/url?u=http-3A__gcr.io_google-5Fcontainers_hyperkube-2Damd64-3Av1.10.2&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=RON0xE-JpIMxDTL_Ja4E7LJK_asUyNLjZiaJmHyci3M&e=> Unable to connect to the server: EOF .Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
From there, it just keeps giving these two errors.
Any thoughts?
Thanks...
---- Nick
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=nBDu-ohIh1kHRZSZCF0aSC5mIryxhGUOu9y-I8hj7dc&e=>
Very nice! Capital letters was indeed the problem -- or at least, THAT problem. I created a new VM without them and started again, getting past that point. However, I'm now stuck with CrashLoopBack on the coredns pods. I've been trying to grab logs without success. However, I am seeing this: 2019-03-17 17:31:29.142 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-proxy (Kubernetes proxy), sequenced=True 2019-03-17 17:31:29.143 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Processing Chart, release=airship-kubernetes-proxy 2019-03-17 17:31:29.145 3396 INFO armada.handlers.chartbuilder [-] [chart=proxy]: Building dependency chart kubernetes-proxy-htk for release kubernetes-proxy. 2019-03-17 17:31:29.196 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Existing release airship-kubernetes-proxy found in namespace kube-system 2019-03-17 17:31:29.200 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.238 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Found no updates to chart release inputs 2019-03-17 17:31:29.239 3396 INFO armada.handlers.wait [-] [chart=proxy]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.279 3396 INFO armada.handlers.wait [-] [chart=proxy]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.297 3396 INFO armada.handlers.armada [-] All Charts applied in ChartGroup kubernetes-proxy. 2019-03-17 17:31:29.298 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-container-networking (Container networking via Calico), sequenced=True 2019-03-17 17:31:29.298 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Processing Chart, release=airship-kubernetes-calico-etcd 2019-03-17 17:31:29.298 3396 INFO armada.handlers.chartbuilder [-] [chart=etcd]: Building dependency chart kubernetes-calico-etcd-htk for release kubernetes-calico-etcd. 2019-03-17 17:31:29.312 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Existing release airship-kubernetes-calico-etcd found in namespace kube-system 2019-03-17 17:31:29.347 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.421 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Found no updates to chart release inputs 2019-03-17 17:31:29.422 3396 INFO armada.handlers.wait [-] [chart=etcd]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.425 3396 INFO armada.handlers.wait [-] [chart=etcd]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.432 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Processing Chart, release=airship-kubernetes-calico 2019-03-17 17:31:29.432 3396 INFO armada.handlers.chartbuilder [-] [chart=calico]: Building dependency chart calico-htk for release kubernetes-calico. 2019-03-17 17:31:29.456 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Existing release airship-kubernetes-calico found in namespace kube-system 2019-03-17 17:31:29.498 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.642 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Found no updates to chart release inputs 2019-03-17 17:31:29.642 3396 INFO armada.handlers.wait [-] [chart=calico]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.648 3396 INFO armada.handlers.wait [-] [chart=calico]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] All Charts applied in ChartGroup kubernetes-container-networking. 2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-dns (Cluster DNS), sequenced=False 2019-03-17 17:31:29.668 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Processing Chart, release=airship-coredns 2019-03-17 17:31:29.668 3396 INFO armada.handlers.chartbuilder [-] [chart=coredns]: Building dependency chart coredns-htk for release coredns. 2019-03-17 17:31:29.685 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Purging FAILED release airship-coredns before deployment. 2019-03-17 17:31:29.685 3396 INFO armada.handlers.tiller [-] [chart=coredns]: Uninstall airship-coredns release with disable_hooks=False, purge=True flags 2019-03-17 17:31:31.016 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Installing release airship-coredns in namespace kube-system, wait=True, timeout=599s 2019-03-17 17:31:31.018 3396 INFO armada.handlers.tiller [-] [chart=coredns]: Helm install release: wait=True, timeout=599 Any thoughts? ---- Nick On Wed, Mar 13, 2019 at 10:56 PM Hemanth N <mail2hemanth.n@gmail.com> wrote:
Hi Nick
First thing i would do, is look for apiserver and etcd container errors
Can you check the following as well
1. Verify if etcd service is up or not On VM, run netstat -ntulp | grep 2379
2. journalctl -u kubelet.service | grep "Invalid value" Look for errros, it might point out any pod spec issues
Also please provide following information 3. cat /etc/kubernetes/manifests/kubernetes-etcd.yaml Check under volumes section if the names have any CAPITAL letters
4. hostname -s
My hunch is you have hostname with a CAPS letter which is triggering this failure. Above commands should give an idea if that is the cause. If thats the case, change your hostname to all lowercase and then run airship-in-a-bottle.
Regards, Hemanth
Message: 3 Date: Tue, 12 Mar 2019 17:01:51 -0400 From: Nick Chase <nchase@mirantis.com> To: "MCEUEN, MATT" <MM9745@att.com> Cc: Evgeny L <eli@mirantis.com>, "airship-discuss@lists.airshipit.org" <airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] Error installing Airship In A Bottle Message-ID: <CA+p99G=RzD9+4qAOF5bv-Z=fG0phye= 55CxMPnC7bytG+RSg4w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
OK, so if it IS a timeout thing, if you can tell me how to increase it, I'll appreciate it, because the script finishes, it just finishes with an error.
So the Airship script log is here: http://paste.openstack.org/show/747758/ Docker processes here: http://paste.openstack.org/show/747759/
Thanks....
---- Nick
On Wed, Mar 13, 2019 at 6:39 PM MCEUEN, MATT <MM9745@att.com> wrote:
Agree, and I’d add to check the docker logs of the kube apiserver to make sure it’s starting up healthy.
If you’re not seeing a smoking gun, can you do a `docker ps -a` and pastebin the results?
Thanks,
Matt
*From:* Evgeny L <eli@mirantis.com> *Sent:* Wednesday, March 13, 2019 12:59 PM *To:* Nick Chase <nchase@mirantis.com> *Cc:* airship-discuss@lists.airshipit.org *Subject:* Re: [Airship-discuss] Error installing Airship In A Bottle
Hi Nick,
As far as I remember these errors are just retries that wait until Kubernetes is up and running, have you tried to wait a bit?
If it still fails for you, can you please send us a complete log? You can use http://paste.openstack.org/ < https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=8BzvCwZPVXNqnW1cYOCO0Av7scqQCiZlumjUr7A2dk8&e=
for that.
Thanks!
On Mon, Mar 11, 2019 at 8:59 PM Nick Chase <nchase@mirantis.com> wrote:
I'm trying to install AIB on a Unbutu Virtualbox VM with 48GB of RAM, 4 CPU and a 300GB hard drive. Everything seems to go along fine, until we get here:
Digest: sha256:badd2b1da29d4d530b10b920f64bf66a1b41150db46c3c99b49d56f3f18a82db Status: Downloaded newer image for gcr.io/google_containers/hyperkube-amd64:v1.10.2 < https://urldefense.proofpoint.com/v2/url?u=http-3A__gcr.io_google-5Fcontainers_hyperkube-2Damd64-3Av1.10.2&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=RON0xE-JpIMxDTL_Ja4E7LJK_asUyNLjZiaJmHyci3M&e=
Unable to connect to the server: EOF .Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
From there, it just keeps giving these two errors.
Any thoughts?
Thanks...
---- Nick
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss < https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=nBDu-ohIh1kHRZSZCF0aSC5mIryxhGUOu9y-I8hj7dc&e=
Hi Nick, I think I face a similar problem in my first attempts to install AiaB (I'm a bit further but also stuck too). Note that I'm behind a proxy and that some filters are activated by our IT so I need to replace the default google nameservers. I fix this by adding 2 dns nameservers in /etc/resolv.conf and in the 'common-addresses.yaml' file: echo "nameserver __dns1__" >>/etc/resolv.conf echo "nameserver __dns2__" >>/etc/resolv.conf export common_adr_file=/root/deploy/airship-in-a-bottle/deployment_files/site/demo/networks/common-addresses.yaml sed -i 's/8.8.8.8/__dns1__/g' $common_adr_file sed -i 's/8.8.4.4/__dns2__/g' $common_adr_file Hope it helps, BR, Loïc Le dim. 17 mars 2019 à 18:46, Nick Chase <nchase@mirantis.com> a écrit :
Very nice! Capital letters was indeed the problem -- or at least, THAT problem. I created a new VM without them and started again, getting past that point. However, I'm now stuck with CrashLoopBack on the coredns pods. I've been trying to grab logs without success. However, I am seeing this:
2019-03-17 17:31:29.142 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-proxy (Kubernetes proxy), sequenced=True 2019-03-17 17:31:29.143 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Processing Chart, release=airship-kubernetes-proxy 2019-03-17 17:31:29.145 3396 INFO armada.handlers.chartbuilder [-] [chart=proxy]: Building dependency chart kubernetes-proxy-htk for release kubernetes-proxy. 2019-03-17 17:31:29.196 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Existing release airship-kubernetes-proxy found in namespace kube-system 2019-03-17 17:31:29.200 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.238 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Found no updates to chart release inputs 2019-03-17 17:31:29.239 3396 INFO armada.handlers.wait [-] [chart=proxy]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.279 3396 INFO armada.handlers.wait [-] [chart=proxy]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.297 3396 INFO armada.handlers.armada [-] All Charts applied in ChartGroup kubernetes-proxy. 2019-03-17 17:31:29.298 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-container-networking (Container networking via Calico), sequenced=True 2019-03-17 17:31:29.298 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Processing Chart, release=airship-kubernetes-calico-etcd 2019-03-17 17:31:29.298 3396 INFO armada.handlers.chartbuilder [-] [chart=etcd]: Building dependency chart kubernetes-calico-etcd-htk for release kubernetes-calico-etcd. 2019-03-17 17:31:29.312 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Existing release airship-kubernetes-calico-etcd found in namespace kube-system 2019-03-17 17:31:29.347 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.421 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Found no updates to chart release inputs 2019-03-17 17:31:29.422 3396 INFO armada.handlers.wait [-] [chart=etcd]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.425 3396 INFO armada.handlers.wait [-] [chart=etcd]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.432 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Processing Chart, release=airship-kubernetes-calico 2019-03-17 17:31:29.432 3396 INFO armada.handlers.chartbuilder [-] [chart=calico]: Building dependency chart calico-htk for release kubernetes-calico. 2019-03-17 17:31:29.456 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Existing release airship-kubernetes-calico found in namespace kube-system 2019-03-17 17:31:29.498 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.642 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Found no updates to chart release inputs 2019-03-17 17:31:29.642 3396 INFO armada.handlers.wait [-] [chart=calico]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.648 3396 INFO armada.handlers.wait [-] [chart=calico]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] All Charts applied in ChartGroup kubernetes-container-networking. 2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-dns (Cluster DNS), sequenced=False 2019-03-17 17:31:29.668 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Processing Chart, release=airship-coredns 2019-03-17 17:31:29.668 3396 INFO armada.handlers.chartbuilder [-] [chart=coredns]: Building dependency chart coredns-htk for release coredns. 2019-03-17 17:31:29.685 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Purging FAILED release airship-coredns before deployment. 2019-03-17 17:31:29.685 3396 INFO armada.handlers.tiller [-] [chart=coredns]: Uninstall airship-coredns release with disable_hooks=False, purge=True flags 2019-03-17 17:31:31.016 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Installing release airship-coredns in namespace kube-system, wait=True, timeout=599s 2019-03-17 17:31:31.018 3396 INFO armada.handlers.tiller [-] [chart=coredns]: Helm install release: wait=True, timeout=599
Any thoughts?
---- Nick
On Wed, Mar 13, 2019 at 10:56 PM Hemanth N <mail2hemanth.n@gmail.com> wrote:
Hi Nick
First thing i would do, is look for apiserver and etcd container errors
Can you check the following as well
1. Verify if etcd service is up or not On VM, run netstat -ntulp | grep 2379
2. journalctl -u kubelet.service | grep "Invalid value" Look for errros, it might point out any pod spec issues
Also please provide following information 3. cat /etc/kubernetes/manifests/kubernetes-etcd.yaml Check under volumes section if the names have any CAPITAL letters
4. hostname -s
My hunch is you have hostname with a CAPS letter which is triggering this failure. Above commands should give an idea if that is the cause. If thats the case, change your hostname to all lowercase and then run airship-in-a-bottle.
Regards, Hemanth
Message: 3 Date: Tue, 12 Mar 2019 17:01:51 -0400 From: Nick Chase <nchase@mirantis.com> To: "MCEUEN, MATT" <MM9745@att.com> Cc: Evgeny L <eli@mirantis.com>, "airship-discuss@lists.airshipit.org" <airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] Error installing Airship In A Bottle Message-ID: <CA+p99G=RzD9+4qAOF5bv-Z=fG0phye= 55CxMPnC7bytG+RSg4w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
OK, so if it IS a timeout thing, if you can tell me how to increase it, I'll appreciate it, because the script finishes, it just finishes with an error.
So the Airship script log is here: http://paste.openstack.org/show/747758/ Docker processes here: http://paste.openstack.org/show/747759/
Thanks....
---- Nick
On Wed, Mar 13, 2019 at 6:39 PM MCEUEN, MATT <MM9745@att.com> wrote:
Agree, and I’d add to check the docker logs of the kube apiserver to make sure it’s starting up healthy.
If you’re not seeing a smoking gun, can you do a `docker ps -a` and pastebin the results?
Thanks,
Matt
*From:* Evgeny L <eli@mirantis.com> *Sent:* Wednesday, March 13, 2019 12:59 PM *To:* Nick Chase <nchase@mirantis.com> *Cc:* airship-discuss@lists.airshipit.org *Subject:* Re: [Airship-discuss] Error installing Airship In A Bottle
Hi Nick,
As far as I remember these errors are just retries that wait until Kubernetes is up and running, have you tried to wait a bit?
If it still fails for you, can you please send us a complete log? You can use http://paste.openstack.org/ < https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=8BzvCwZPVXNqnW1cYOCO0Av7scqQCiZlumjUr7A2dk8&e=
for that.
Thanks!
On Mon, Mar 11, 2019 at 8:59 PM Nick Chase <nchase@mirantis.com> wrote:
I'm trying to install AIB on a Unbutu Virtualbox VM with 48GB of RAM, 4 CPU and a 300GB hard drive. Everything seems to go along fine, until we get here:
Digest:
sha256:badd2b1da29d4d530b10b920f64bf66a1b41150db46c3c99b49d56f3f18a82db
Status: Downloaded newer image for gcr.io/google_containers/hyperkube-amd64:v1.10.2 < https://urldefense.proofpoint.com/v2/url?u=http-3A__gcr.io_google-5Fcontainers_hyperkube-2Damd64-3Av1.10.2&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=RON0xE-JpIMxDTL_Ja4E7LJK_asUyNLjZiaJmHyci3M&e=
Unable to connect to the server: EOF .Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
From there, it just keeps giving these two errors.
Any thoughts?
Thanks...
---- Nick
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss < https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=nBDu-ohIh1kHRZSZCF0aSC5mIryxhGUOu9y-I8hj7dc&e=
Nope, unfortunately not. :( Thanks, though. ---- Nick On Mon, Mar 18, 2019 at 4:54 AM Loic Le Gal <l.legal.astellia@gmail.com> wrote:
Hi Nick,
I think I face a similar problem in my first attempts to install AiaB (I'm a bit further but also stuck too). Note that I'm behind a proxy and that some filters are activated by our IT so I need to replace the default google nameservers. I fix this by adding 2 dns nameservers in /etc/resolv.conf and in the 'common-addresses.yaml' file:
echo "nameserver __dns1__" >>/etc/resolv.conf echo "nameserver __dns2__" >>/etc/resolv.conf
export common_adr_file=/root/deploy/airship-in-a-bottle/deployment_files/site/demo/networks/common-addresses.yaml sed -i 's/8.8.8.8/__dns1__/g' $common_adr_file sed -i 's/8.8.4.4/__dns2__/g' $common_adr_file
Hope it helps, BR, Loïc
Le dim. 17 mars 2019 à 18:46, Nick Chase <nchase@mirantis.com> a écrit :
Very nice! Capital letters was indeed the problem -- or at least, THAT problem. I created a new VM without them and started again, getting past that point. However, I'm now stuck with CrashLoopBack on the coredns pods. I've been trying to grab logs without success. However, I am seeing this:
2019-03-17 17:31:29.142 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-proxy (Kubernetes proxy), sequenced=True 2019-03-17 17:31:29.143 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Processing Chart, release=airship-kubernetes-proxy 2019-03-17 17:31:29.145 3396 INFO armada.handlers.chartbuilder [-] [chart=proxy]: Building dependency chart kubernetes-proxy-htk for release kubernetes-proxy. 2019-03-17 17:31:29.196 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Existing release airship-kubernetes-proxy found in namespace kube-system 2019-03-17 17:31:29.200 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.238 3396 INFO armada.handlers.chart_deploy [-] [chart=proxy]: Found no updates to chart release inputs 2019-03-17 17:31:29.239 3396 INFO armada.handlers.wait [-] [chart=proxy]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.279 3396 INFO armada.handlers.wait [-] [chart=proxy]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.297 3396 INFO armada.handlers.armada [-] All Charts applied in ChartGroup kubernetes-proxy. 2019-03-17 17:31:29.298 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-container-networking (Container networking via Calico), sequenced=True 2019-03-17 17:31:29.298 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Processing Chart, release=airship-kubernetes-calico-etcd 2019-03-17 17:31:29.298 3396 INFO armada.handlers.chartbuilder [-] [chart=etcd]: Building dependency chart kubernetes-calico-etcd-htk for release kubernetes-calico-etcd. 2019-03-17 17:31:29.312 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Existing release airship-kubernetes-calico-etcd found in namespace kube-system 2019-03-17 17:31:29.347 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.421 3396 INFO armada.handlers.chart_deploy [-] [chart=etcd]: Found no updates to chart release inputs 2019-03-17 17:31:29.422 3396 INFO armada.handlers.wait [-] [chart=etcd]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.425 3396 INFO armada.handlers.wait [-] [chart=etcd]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.432 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Processing Chart, release=airship-kubernetes-calico 2019-03-17 17:31:29.432 3396 INFO armada.handlers.chartbuilder [-] [chart=calico]: Building dependency chart calico-htk for release kubernetes-calico. 2019-03-17 17:31:29.456 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Existing release airship-kubernetes-calico found in namespace kube-system 2019-03-17 17:31:29.498 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Checking for updates to chart release inputs. 2019-03-17 17:31:29.642 3396 INFO armada.handlers.chart_deploy [-] [chart=calico]: Found no updates to chart release inputs 2019-03-17 17:31:29.642 3396 INFO armada.handlers.wait [-] [chart=calico]: Waiting for resource type=job, namespace=kube-system labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.648 3396 INFO armada.handlers.wait [-] [chart=calico]: Waiting for resource type=pod, namespace=kube-system labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times, sleep 1s) 2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] All Charts applied in ChartGroup kubernetes-container-networking. 2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] Processing ChartGroup: kubernetes-dns (Cluster DNS), sequenced=False 2019-03-17 17:31:29.668 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Processing Chart, release=airship-coredns 2019-03-17 17:31:29.668 3396 INFO armada.handlers.chartbuilder [-] [chart=coredns]: Building dependency chart coredns-htk for release coredns. 2019-03-17 17:31:29.685 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Purging FAILED release airship-coredns before deployment. 2019-03-17 17:31:29.685 3396 INFO armada.handlers.tiller [-] [chart=coredns]: Uninstall airship-coredns release with disable_hooks=False, purge=True flags 2019-03-17 17:31:31.016 3396 INFO armada.handlers.chart_deploy [-] [chart=coredns]: Installing release airship-coredns in namespace kube-system, wait=True, timeout=599s 2019-03-17 17:31:31.018 3396 INFO armada.handlers.tiller [-] [chart=coredns]: Helm install release: wait=True, timeout=599
Any thoughts?
---- Nick
On Wed, Mar 13, 2019 at 10:56 PM Hemanth N <mail2hemanth.n@gmail.com> wrote:
Hi Nick
First thing i would do, is look for apiserver and etcd container errors
Can you check the following as well
1. Verify if etcd service is up or not On VM, run netstat -ntulp | grep 2379
2. journalctl -u kubelet.service | grep "Invalid value" Look for errros, it might point out any pod spec issues
Also please provide following information 3. cat /etc/kubernetes/manifests/kubernetes-etcd.yaml Check under volumes section if the names have any CAPITAL letters
4. hostname -s
My hunch is you have hostname with a CAPS letter which is triggering this failure. Above commands should give an idea if that is the cause. If thats the case, change your hostname to all lowercase and then run airship-in-a-bottle.
Regards, Hemanth
Message: 3 Date: Tue, 12 Mar 2019 17:01:51 -0400 From: Nick Chase <nchase@mirantis.com> To: "MCEUEN, MATT" <MM9745@att.com> Cc: Evgeny L <eli@mirantis.com>, "airship-discuss@lists.airshipit.org" <airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] Error installing Airship In A Bottle Message-ID: <CA+p99G=RzD9+4qAOF5bv-Z=fG0phye= 55CxMPnC7bytG+RSg4w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
OK, so if it IS a timeout thing, if you can tell me how to increase it, I'll appreciate it, because the script finishes, it just finishes with an error.
So the Airship script log is here: http://paste.openstack.org/show/747758/ Docker processes here: http://paste.openstack.org/show/747759/
Thanks....
---- Nick
On Wed, Mar 13, 2019 at 6:39 PM MCEUEN, MATT <MM9745@att.com> wrote:
Agree, and I’d add to check the docker logs of the kube apiserver to make sure it’s starting up healthy.
If you’re not seeing a smoking gun, can you do a `docker ps -a` and pastebin the results?
Thanks,
Matt
*From:* Evgeny L <eli@mirantis.com> *Sent:* Wednesday, March 13, 2019 12:59 PM *To:* Nick Chase <nchase@mirantis.com> *Cc:* airship-discuss@lists.airshipit.org *Subject:* Re: [Airship-discuss] Error installing Airship In A Bottle
Hi Nick,
As far as I remember these errors are just retries that wait until Kubernetes is up and running, have you tried to wait a bit?
If it still fails for you, can you please send us a complete log? You can use http://paste.openstack.org/ < https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=8BzvCwZPVXNqnW1cYOCO0Av7scqQCiZlumjUr7A2dk8&e=
for that.
Thanks!
On Mon, Mar 11, 2019 at 8:59 PM Nick Chase <nchase@mirantis.com> wrote:
I'm trying to install AIB on a Unbutu Virtualbox VM with 48GB of RAM, 4 CPU and a 300GB hard drive. Everything seems to go along fine, until we get here:
Digest:
sha256:badd2b1da29d4d530b10b920f64bf66a1b41150db46c3c99b49d56f3f18a82db
Status: Downloaded newer image for gcr.io/google_containers/hyperkube-amd64:v1.10.2 < https://urldefense.proofpoint.com/v2/url?u=http-3A__gcr.io_google-5Fcontainers_hyperkube-2Damd64-3Av1.10.2&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=RON0xE-JpIMxDTL_Ja4E7LJK_asUyNLjZiaJmHyci3M&e=
Unable to connect to the server: EOF .Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
From there, it just keeps giving these two errors.
Any thoughts?
Thanks...
---- Nick
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss < https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=nBDu-ohIh1kHRZSZCF0aSC5mIryxhGUOu9y-I8hj7dc&e=
participants (3)
-
Hemanth N
-
Loic Le Gal
-
Nick Chase