libvirt-hxzln 0/1 CrashLoopBackOff 26 1h 192.168.122.101 air1
# kubectl logs -n openstack libvirt-hxzln
Hi Matt, Kaspars,
I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same –unknown to me as yet- reason.
I’m posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3].
(full output is attached – let me know how to share it if doesn’t get through)
I’m still looking at those myself but not identified something yet.
Thanks
Nikos
[1]. http://paste.openstack.org/show/742654/
[2]. http://paste.openstack.org/show/742655/
[3]. http://paste.openstack.org/show/742660/
From: MCEUEN, MATT [mailto:MM9745@att.com]
Sent: Monday, January 14, 2019 6:48 PM
To: SKELS, KASPARS <ks3019@att.com>; Nikos Karandreas <nick@intracom-telecom.com>; airship-discuss@lists.airshipit.org
Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers
Hey Nikos, just checking – are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest?
Thanks,
Matt
From: SKELS, KASPARS
Sent: Tuesday, January 8, 2019 7:19 PM
To: Nikos Karandreas <nick@intracom-telecom.com>;
airship-discuss@lists.airshipit.org
Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers
***Security Advisory: This Message Originated Outside of AT&T ***
Reference
http://cso.att.com/EmailSecurity/IDSP.html for more information.
Hi Nikos,
I did re-run latest master just now and could get a successful run in my VM environment.
The armada part that fails will show logs within Armada container, have a look at pod logs
root@n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada
Flag --show-all has been deprecated, will be removed in an upcoming release
armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0
And then
root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6
The logs can be chatty, you may try
root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR
Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml
deployment_files/global/v1.0demo/software/config/versions.yaml
such as, and see if you are getting, see if you are getting good download speeds on them
$ docker pull docker.io/openstackhelm/nova:ocata
$ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461
Also check openstack namespace
root@n0:~# kubectl get pods -n openstack -o wide -a=false
See if there are any failed pods.
Kindly, Kaspars
From: Nikos Karandreas [nick@intracom-telecom.com]
Sent: Tuesday, January 08, 2019 9:43 AM
To:
airship-discuss@lists.airshipit.org
Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers
Hi Drew,
I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete
the test stack server, clearly at a much later point:
2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
which repeats a few times before exiting the script.
Going up a bit
step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed
step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed
…
step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed
although just in the previous iteration:
step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running
…
step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running
step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled
Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1]
Any idea on what/where to look for?
Cheers
Nikos
[1]. http://paste.openstack.org/show/740645/
From: Drew Walters [mailto:drewwalters96@gmail.com]
Sent: Friday, January 04, 2019 8:46 PM
To: Nikos Karandreas <nick@intracom-telecom.com>
Cc:
airship-discuss@lists.airshipit.org
Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers
Hi Nikos!
Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try?
Best,
Drew Walters
On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com> wrote:
Hi airship people,
I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the
instructions in [1] but I’m getting the failure below (until the end in [2])
+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site
Error: Unable to complete request to Airflow
Reason: Airflow could not be contacted properly by Shipyard.
- Error: <class 'requests.exceptions.ConnectTimeout'>
and giving, after that, docker ps I get:
root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0
04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0
d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0
Can you please give some insight on what might be wrong or what to look for?
Cheers
Nikos
Nikos Karandreas
Developer, OPNFV
Software Development Center
_____________________________________
Intracom Telecom
Marinou Antipa 41, Pilea-Thessaloniki, GR 57001
t: +30 2310497353,
f: +30 2310497330
nick@intracom-telecom.com
[1] https://github.com/openstack/airship-in-a-bottle
[2]
Execute deploy_site Dag...
+ [[ -n '' ]]
+ [[ -n deploy_site ]]
+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site
Error: Unable to complete request to Airflow
Reason: Airflow could not be contacted properly by Shipyard.
- Error: <class 'requests.exceptions.ConnectTimeout'>
#### Errors: 1, Warnings: 0, Infos: 0, Other: 0 ####
+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions
Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes
None
+ echo -e 'Retrieving Action ID...\n'
Retrieving Action ID...
++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions
++ grep -i Processing
++ awk '{print $2}'
++ grep deploy_site
+ action_id=
+ retrieve_shipyard_action_counter=0
+ retrieve_shipyard_action_limit=2
+ [[ 0 -le 2 ]]
+ [[ -n '' ]]
+ [[ 0 == 2 ]]
+ echo -e 'Unable to Retrieve Action ID!'
Unable to Retrieve Action ID!
+ echo -e 'Retrying in 30 seconds...'
Retrying in 30 seconds...
+ sleep 30
++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions
++ grep deploy_site
++ grep -i Processing
++ awk '{print $2}'
+ action_id=
+ (( retrieve_shipyard_action_counter ++ ))
+ error 'executing deploy_site from the /site directory'
+ set +x
Error when executing deploy_site from the /site directory.
+ exit 1
+ clean
+ set +x
To remove files generated during this script's execution, delete /root/deploy.
This VM is disposable. Re-deployment in this same VM will lead to unpredictable results.
_______________________________________________
Airship-discuss mailing list
Airship-discuss@lists.airshipit.org
http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss