[Airship-discuss] Error installing Airship In A Bottle

Sun Mar 17 17:45:34 UTC 2019

Very nice!  Capital letters was indeed the problem -- or at least, THAT
problem.  I created a new VM without them and started again, getting past
that point.  However, I'm now stuck with CrashLoopBack on the coredns
pods.  I've been trying to grab logs without success.  However, I am seeing
this:

2019-03-17 17:31:29.142 3396 INFO armada.handlers.armada [-] Processing
ChartGroup: kubernetes-proxy (Kubernetes proxy), sequenced=True
2019-03-17 17:31:29.143 3396 INFO armada.handlers.chart_deploy [-]
[chart=proxy]: Processing Chart, release=airship-kubernetes-proxy
2019-03-17 17:31:29.145 3396 INFO armada.handlers.chartbuilder [-]
[chart=proxy]: Building dependency chart kubernetes-proxy-htk for release
kubernetes-proxy.
2019-03-17 17:31:29.196 3396 INFO armada.handlers.chart_deploy [-]
[chart=proxy]: Existing release airship-kubernetes-proxy found in namespace
kube-system
2019-03-17 17:31:29.200 3396 INFO armada.handlers.chart_deploy [-]
[chart=proxy]: Checking for updates to chart release inputs.
2019-03-17 17:31:29.238 3396 INFO armada.handlers.chart_deploy [-]
[chart=proxy]: Found no updates to chart release inputs
2019-03-17 17:31:29.239 3396 INFO armada.handlers.wait [-] [chart=proxy]:
Waiting for resource type=job, namespace=kube-system
labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times,
sleep 1s)
2019-03-17 17:31:29.279 3396 INFO armada.handlers.wait [-] [chart=proxy]:
Waiting for resource type=pod, namespace=kube-system
labels=release_group=airship-kubernetes-proxy for 600s (k8s wait 1 times,
sleep 1s)
2019-03-17 17:31:29.297 3396 INFO armada.handlers.armada [-] All Charts
applied in ChartGroup kubernetes-proxy.
2019-03-17 17:31:29.298 3396 INFO armada.handlers.armada [-] Processing
ChartGroup: kubernetes-container-networking (Container networking via
Calico), sequenced=True
2019-03-17 17:31:29.298 3396 INFO armada.handlers.chart_deploy [-]
[chart=etcd]: Processing Chart, release=airship-kubernetes-calico-etcd
2019-03-17 17:31:29.298 3396 INFO armada.handlers.chartbuilder [-]
[chart=etcd]: Building dependency chart kubernetes-calico-etcd-htk for
release kubernetes-calico-etcd.
2019-03-17 17:31:29.312 3396 INFO armada.handlers.chart_deploy [-]
[chart=etcd]: Existing release airship-kubernetes-calico-etcd found in
namespace kube-system
2019-03-17 17:31:29.347 3396 INFO armada.handlers.chart_deploy [-]
[chart=etcd]: Checking for updates to chart release inputs.
2019-03-17 17:31:29.421 3396 INFO armada.handlers.chart_deploy [-]
[chart=etcd]: Found no updates to chart release inputs
2019-03-17 17:31:29.422 3396 INFO armada.handlers.wait [-] [chart=etcd]:
Waiting for resource type=job, namespace=kube-system
labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1
times, sleep 1s)
2019-03-17 17:31:29.425 3396 INFO armada.handlers.wait [-] [chart=etcd]:
Waiting for resource type=pod, namespace=kube-system
labels=release_group=airship-kubernetes-calico-etcd for 600s (k8s wait 1
times, sleep 1s)
2019-03-17 17:31:29.432 3396 INFO armada.handlers.chart_deploy [-]
[chart=calico]: Processing Chart, release=airship-kubernetes-calico
2019-03-17 17:31:29.432 3396 INFO armada.handlers.chartbuilder [-]
[chart=calico]: Building dependency chart calico-htk for release
kubernetes-calico.
2019-03-17 17:31:29.456 3396 INFO armada.handlers.chart_deploy [-]
[chart=calico]: Existing release airship-kubernetes-calico found in
namespace kube-system
2019-03-17 17:31:29.498 3396 INFO armada.handlers.chart_deploy [-]
[chart=calico]: Checking for updates to chart release inputs.
2019-03-17 17:31:29.642 3396 INFO armada.handlers.chart_deploy [-]
[chart=calico]: Found no updates to chart release inputs
2019-03-17 17:31:29.642 3396 INFO armada.handlers.wait [-] [chart=calico]:
Waiting for resource type=job, namespace=kube-system
labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times,
sleep 1s)
2019-03-17 17:31:29.648 3396 INFO armada.handlers.wait [-] [chart=calico]:
Waiting for resource type=pod, namespace=kube-system
labels=release_group=airship-kubernetes-calico for 600s (k8s wait 1 times,
sleep 1s)
2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] All Charts
applied in ChartGroup kubernetes-container-networking.
2019-03-17 17:31:29.667 3396 INFO armada.handlers.armada [-] Processing
ChartGroup: kubernetes-dns (Cluster DNS), sequenced=False
2019-03-17 17:31:29.668 3396 INFO armada.handlers.chart_deploy [-]
[chart=coredns]: Processing Chart, release=airship-coredns
2019-03-17 17:31:29.668 3396 INFO armada.handlers.chartbuilder [-]
[chart=coredns]: Building dependency chart coredns-htk for release coredns.
2019-03-17 17:31:29.685 3396 INFO armada.handlers.chart_deploy [-]
[chart=coredns]: Purging FAILED release airship-coredns before deployment.
2019-03-17 17:31:29.685 3396 INFO armada.handlers.tiller [-]
[chart=coredns]: Uninstall airship-coredns release with
disable_hooks=False, purge=True flags
2019-03-17 17:31:31.016 3396 INFO armada.handlers.chart_deploy [-]
[chart=coredns]: Installing release airship-coredns in namespace
kube-system, wait=True, timeout=599s
2019-03-17 17:31:31.018 3396 INFO armada.handlers.tiller [-]
[chart=coredns]: Helm install release: wait=True, timeout=599

Any thoughts?

----  Nick

On Wed, Mar 13, 2019 at 10:56 PM Hemanth N <mail2hemanth.n at gmail.com> wrote:

> Hi Nick
>
> First thing i would do, is look for apiserver and etcd container errors
>
> Can you check the following as well
>
> 1. Verify if etcd service is up or not
>     On VM, run netstat -ntulp | grep 2379
>
> 2. journalctl -u kubelet.service | grep "Invalid value"
>    Look for errros, it might point out any pod spec issues
>
> Also please provide following information
> 3. cat /etc/kubernetes/manifests/kubernetes-etcd.yaml
>     Check under volumes section if the names have any CAPITAL letters
>
> 4. hostname -s
>
> My hunch is you have hostname with a CAPS letter which is triggering
> this failure. Above commands should give an idea if that is the cause.
> If thats the case, change your hostname to all lowercase and then run
> airship-in-a-bottle.
>
> Regards,
> Hemanth
>
>
>
>
>
>
> > Message: 3
> > Date: Tue, 12 Mar 2019 17:01:51 -0400
> > From: Nick Chase <nchase at mirantis.com>
> > To: "MCEUEN, MATT" <MM9745 at att.com>
> > Cc: Evgeny L <eli at mirantis.com>,
> >         "airship-discuss at lists.airshipit.org"
> >         <airship-discuss at lists.airshipit.org>
> > Subject: Re: [Airship-discuss] Error installing Airship In A Bottle
> > Message-ID:
> >         <CA+p99G=RzD9+4qAOF5bv-Z=fG0phye=
> 55CxMPnC7bytG+RSg4w at mail.gmail.com>
> > Content-Type: text/plain; charset="utf-8"
> >
> > OK, so if it IS a timeout thing, if you can tell me how to increase it,
> > I'll appreciate it, because the script finishes, it just finishes with an
> > error.
> >
> > So the Airship script log is here:
> http://paste.openstack.org/show/747758/
> > Docker processes here: http://paste.openstack.org/show/747759/
> >
> > Thanks....
> >
> > ----  Nick
> >
> >
> > On Wed, Mar 13, 2019 at 6:39 PM MCEUEN, MATT <MM9745 at att.com> wrote:
> >
> > > Agree, and I’d add to check the docker logs of the kube apiserver to
> make
> > > sure it’s starting up healthy.
> > >
> > >
> > >
> > > If you’re not seeing a smoking gun, can you do a `docker ps -a` and
> > > pastebin the results?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > >
> > >
> > > *From:* Evgeny L <eli at mirantis.com>
> > > *Sent:* Wednesday, March 13, 2019 12:59 PM
> > > *To:* Nick Chase <nchase at mirantis.com>
> > > *Cc:* airship-discuss at lists.airshipit.org
> > > *Subject:* Re: [Airship-discuss] Error installing Airship In A Bottle
> > >
> > >
> > >
> > > Hi Nick,
> > >
> > >
> > >
> > > As far as I remember these errors are just retries that wait until
> > > Kubernetes is up and running, have you tried to wait a bit?
> > >
> > > If it still fails for you, can you please send us a complete log? You
> can
> > > use http://paste.openstack.org/
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=8BzvCwZPVXNqnW1cYOCO0Av7scqQCiZlumjUr7A2dk8&e=
> >
> > > for that.
> > >
> > >
> > >
> > > Thanks!
> > >
> > >
> > >
> > > On Mon, Mar 11, 2019 at 8:59 PM Nick Chase <nchase at mirantis.com>
> wrote:
> > >
> > > I'm trying to install AIB on a Unbutu Virtualbox VM with 48GB of RAM, 4
> > > CPU and a 300GB hard drive.  Everything seems to go along fine, until
> we
> > > get here:
> > >
> > >
> > >
> > > Digest:
> > > sha256:badd2b1da29d4d530b10b920f64bf66a1b41150db46c3c99b49d56f3f18a82db
> > > Status: Downloaded newer image for
> > > gcr.io/google_containers/hyperkube-amd64:v1.10.2
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gcr.io_google-5Fcontainers_hyperkube-2Damd64-3Av1.10.2&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=RON0xE-JpIMxDTL_Ja4E7LJK_asUyNLjZiaJmHyci3M&e=
> >
> > > Unable to connect to the server: EOF
> > > .Unable to connect to the server: net/http: request canceled
> > > (Client.Timeout exceeded while awaiting headers)
> > >
> > >
> > >
> > > From there, it just keeps giving these two errors.
> > >
> > >
> > >
> > > Any thoughts?
> > >
> > >
> > >
> > > Thanks...
> > >
> > >
> > >
> > > ----  Nick
> > >
> > > _______________________________________________
> > > Airship-discuss mailing list
> > > Airship-discuss at lists.airshipit.org
> > > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
> > > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=WhNp-VjTw70e2z_QbunOZc8fyS-zMWVbK70DnNjJMS0&s=nBDu-ohIh1kHRZSZCF0aSC5mIryxhGUOu9y-I8hj7dc&e=
> >
> > >
> > >
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: <
> http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190312/5c30236d/attachment.html
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Airship-discuss mailing list
> > Airship-discuss at lists.airshipit.org
> > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
> >
> > ------------------------------
> >
> > End of Airship-discuss Digest, Vol 10, Issue 12
> > ***********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190317/15e2a113/attachment.html>