[Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers

Nikos Karandreas nick at intracom-telecom.com
Tue Jan 15 16:34:01 UTC 2019


Hi Matt, Kaspars,

 

I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same -unknown to me as yet- reason.

 

I'm posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3].

(full output is attached - let me know how to share it if doesn't get through)

I'm still looking at those myself but not identified something yet.

 

Thanks

Nikos

 

[1]. http://paste.openstack.org/show/742654/

[2]. http://paste.openstack.org/show/742655/

[3]. http://paste.openstack.org/show/742660/

 

 

From: MCEUEN, MATT [mailto:MM9745 at att.com] 
Sent: Monday, January 14, 2019 6:48 PM
To: SKELS, KASPARS <ks3019 at att.com>; Nikos Karandreas <nick at intracom-telecom.com>; airship-discuss at lists.airshipit.org
Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers

 

Hey Nikos, just checking - are you still seeing any issues with AiaB?  Or all clear with the latest-and-greatest?

 

Thanks,

Matt

 

From: SKELS, KASPARS 
Sent: Tuesday, January 8, 2019 7:19 PM
To: Nikos Karandreas <nick at intracom-telecom.com <mailto:nick at intracom-telecom.com> >; airship-discuss at lists.airshipit.org <mailto:airship-discuss at lists.airshipit.org> 
Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers

 

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Hi Nikos,

 

I did re-run latest master just now and could get a successful run in my VM environment.

 

The armada part that fails will show logs within Armada container, have a look at pod logs

 

root at n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada
Flag --show-all has been deprecated, will be removed in an upcoming release
armada-api-5488cbdb99-cqzj6                                 1/1       Running   0          58m       10.97.26.38   n0

 

And then

root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 

 

The logs can be chatty, you may try 

root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR

 

Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml

deployment_files/global/v1.0demo/software/config/versions.yaml

 

such as, and see if you are getting, see if you are getting good download speeds on them

$ docker pull docker.io/openstackhelm/nova:ocata

$ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461

 

Also check openstack namespace

root at n0:~# kubectl get pods -n openstack -o wide -a=false

 

See if there are any failed pods.

 

Kindly, Kaspars

 

 

  _____  

From: Nikos Karandreas [nick at intracom-telecom.com]
Sent: Tuesday, January 08, 2019 9:43 AM
To: airship-discuss at lists.airshipit.org <mailto:airship-discuss at lists.airshipit.org> 
Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers

Hi Drew,

 

I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete

the test stack server, clearly at a much later point:

 

 

2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED  ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"

 

which repeats a few times before exiting the script.

 

Going up a bit 

step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build                          11           failed                                  

step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply                     12           failed                                  

...

step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases                   19           upstream_failed                         

 

 

although just in the previous iteration:

step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build                          11           running                           

...

step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply                     12           running                           

step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases                   19           scheduled                         

 

 

 

Giving kubectl get pods -n ucp doesn't look any suspicious (to me) [1]

 

Any idea on what/where to look for?

 

 

Cheers

Nikos

 

 

[1]. http://paste.openstack.org/show/740645/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_740645_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=kYlTaQKv2v1urN4KYmLZNYkhId4TdEFhmIt_itC6mRM&e=> 

 

 

 

From: Drew Walters [mailto:drewwalters96 at gmail.com] 
Sent: Friday, January 04, 2019 8:46 PM
To: Nikos Karandreas <nick at intracom-telecom.com <mailto:nick at intracom-telecom.com> >
Cc: airship-discuss at lists.airshipit.org <mailto:airship-discuss at lists.airshipit.org> 
Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers

 

Hi Nikos!

 

Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try?

 

Best,

Drew Walters

 

On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick at intracom-telecom.com <mailto:nick at intracom-telecom.com> > wrote:

Hi airship people,

 

I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the 

instructions in [1] but I'm getting the failure below (until the end in [2])

 

+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=>  -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=>  create action deploy_site

Error: Unable to complete request to Airflow

Reason: Airflow could not be contacted properly by Shipyard.

- Error: <class 'requests.exceptions.ConnectTimeout'>

 

 

and giving, after that, docker ps I get:

 

root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=U3yD1Ji3NaELSW0iiqQWNv_6xcbZQI1m5P8Iwbnv52A&e=> "

CONTAINER ID        IMAGE                                                                                                                                    COMMAND                  CREATED             STATUS                             PORTS               NAMES

f4c0747b25ec        quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=>                                        "/tmp/airflow-ship..."   47 minutes ago      Exited (0) 47 minutes ago                          k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0

04241489b515        quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=>                                        "/tmp/airflow-ship..."   47 minutes ago      Exited (0) 47 minutes ago                          k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0

d433934eec1d        quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=>                                        "/tmp/airflow-ship..."   47 minutes ago      Exited (0) 47 minutes ago                          k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0

 

 

Can you please give some insight on what might be wrong or what to look for?

 

 

Cheers

Nikos

 

 

 

Nikos Karandreas

Developer, OPNFV

Software Development Center

_____________________________________
Intracom Telecom
Marinou Antipa 41, Pilea-Thessaloniki, GR 57001
t:   +30 2310497353,
f:   +30 2310497330
nick at intracom-telecom.com <mailto:nick at intracom-telecom.com>  

www <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=E6hDFFLPmnAW2gih1ARaQbeuzH7mcLzUdZ_BSrN7Pkc&e=> .intracom-telecom.com

 

 

 

 

[1] https://github.com/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=Zcf9kkgPjUXvi1K6dNfH7EQ6v6iS0yFf8qS8U12Dr2s&e=> 

[2]

Execute deploy_site Dag...

 

+ [[ -n '' ]]

+ [[ -n deploy_site ]]

+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=>  -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=>  create action deploy_site

Error: Unable to complete request to Airflow

Reason: Airflow could not be contacted properly by Shipyard.

- Error: <class 'requests.exceptions.ConnectTimeout'>

 

####  Errors: 1, Warnings: 0, Infos: 0, Other: 0  ####

+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=>  -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=>  get actions

Name        Action        Lifecycle        Execution Time        Step Succ/Fail/Oth        Footnotes        

None                                                                                                        

 

 

+ echo -e 'Retrieving Action ID...\n'

Retrieving Action ID...

 

++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=>  -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=>  get actions

++ grep -i Processing

++ awk '{print $2}'

++ grep deploy_site

+ action_id=

+ retrieve_shipyard_action_counter=0

+ retrieve_shipyard_action_limit=2

+ [[ 0 -le 2 ]]

+ [[ -n '' ]]

+ [[ 0 == 2 ]]

+ echo -e 'Unable to Retrieve Action ID!'

Unable to Retrieve Action ID!

+ echo -e 'Retrying in 30 seconds...'

Retrying in 30 seconds...

+ sleep 30

++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=>  -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=>  get actions

++ grep deploy_site

++ grep -i Processing

++ awk '{print $2}'

+ action_id=

+ (( retrieve_shipyard_action_counter ++ ))

+ error 'executing deploy_site from the /site directory'

+ set +x

Error when executing deploy_site from the /site directory.

+ exit 1

+ clean

+ set +x

To remove files generated during this script's execution, delete /root/deploy.

This VM is disposable. Re-deployment in this same VM will lead to unpredictable results.

_______________________________________________
Airship-discuss mailing list
Airship-discuss at lists.airshipit.org <mailto:Airship-discuss at lists.airshipit.org> 
http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=iE9izcAF-2v0B7FIJITXPtiiNGFYP7KQNCyEdb0Fixg&e=> 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190115/c106647d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: aiab-1.tar.gz
Type: application/octet-stream
Size: 35275 bytes
Desc: not available
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190115/c106647d/attachment-0001.obj>


More information about the Airship-discuss mailing list