Hi Nikos, failing libvirt is the cause for this (errors from Armada is that charts (nova, etc) are timing out to deploy) libvirt-hxzln 0/1 CrashLoopBackOff 26 1h 192.168.122.101 air1 Can you also share the logs for libvirt, to get some more insights.. and more folks familiar with this should be able to help # kubectl logs -n openstack libvirt-hxzln As a quick check, I've seen libvirt failing when there is a clash of pre-installed libvirt packages in the VM. Make sure you base VM is clean/regular Ubuntu 16.04. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 15, 2019 10:34 AM To: MCEUEN, MATT; airship-discuss@lists.airshipit.org; SKELS, KASPARS Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Matt, Kaspars, I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same –unknown to me as yet- reason. I’m posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3]. (full output is attached – let me know how to share it if doesn’t get through) I’m still looking at those myself but not identified something yet. Thanks Nikos [1]. http://paste.openstack.org/show/742654/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742654_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=2odIwtJQTum_-VFWIpGs7n3hPRXh5TL4PULy3TfWJYg&e=> [2]. http://paste.openstack.org/show/742655/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742655_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=23tFv00CzdY95wtDz0ttV7paJmTKrpK6bGiBY5VC-6I&e=> [3]. http://paste.openstack.org/show/742660/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742660_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=rceHkquD0ilBQYmRQxRBPJuXq6rGC0Bx8vq7yhmNfyk&e=> From: MCEUEN, MATT [mailto:MM9745@att.com] Sent: Monday, January 14, 2019 6:48 PM To: SKELS, KASPARS <ks3019@att.com>; Nikos Karandreas <nick@intracom-telecom.com>; airship-discuss@lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hey Nikos, just checking – are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>>; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root@n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root@n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_740645_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=kYlTaQKv2v1urN4KYmLZNYkhId4TdEFhmIt_itC6mRM&e=> From: Drew Walters [mailto:drewwalters96@gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>> Cc: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>> wrote: Hi airship people, I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I’m getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=U3yD1Ji3NaELSW0iiqQWNv_6xcbZQI1m5P8Iwbnv52A&e=>" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com<mailto:nick@intracom-telecom.com> www.intracom-telecom.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=E6hDFFLPmnAW2gih1ARaQbeuzH7mcLzUdZ_BSrN7Pkc&e=> [1] https://github.com/openstack/airship-in-a-bottle<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=Zcf9kkgPjUXvi1K6dNfH7EQ6v6iS0yFf8qS8U12Dr2s&e=> [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org<mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=iE9izcAF-2v0B7FIJITXPtiiNGFYP7KQNCyEdb0Fixg&e=>