airship-in-a-bottle: deployment fails with exited airflow containers
Hi airship people, I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I'm getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> <http://www.intracom-telecom.com/> www.intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results.
Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com> wrote:
Hi airship people,
I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the
instructions in [1] but I’m getting the failure below (until the end in [2])
+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site
Error: Unable to complete request to Airflow
Reason: Airflow could not be contacted properly by Shipyard.
- Error: <class 'requests.exceptions.ConnectTimeout'>
and giving, after that, docker ps I get:
root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0
04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0
d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0
Can you please give some insight on what might be wrong or what to look for?
Cheers
Nikos
*Nikos Karandreas*
Developer, OPNFV
Software Development Center
_____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 *nick@intracom-telecom.com <nick@intracom-telecom.com>*
www.intracom-telecom.com
[1] https://github.com/openstack/airship-in-a-bottle
[2]
Execute deploy_site Dag...
+ [[ -n '' ]]
+ [[ -n deploy_site ]]
+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site
Error: Unable to complete request to Airflow
Reason: Airflow could not be contacted properly by Shipyard.
- Error: <class 'requests.exceptions.ConnectTimeout'>
#### Errors: 1, Warnings: 0, Infos: 0, Other: 0 ####
+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions
Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes
None
+ echo -e 'Retrieving Action ID...\n'
Retrieving Action ID...
++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions
++ grep -i Processing
++ awk '{print $2}'
++ grep deploy_site
+ action_id=
+ retrieve_shipyard_action_counter=0
+ retrieve_shipyard_action_limit=2
+ [[ 0 -le 2 ]]
+ [[ -n '' ]]
+ [[ 0 == 2 ]]
+ echo -e 'Unable to Retrieve Action ID!'
Unable to Retrieve Action ID!
+ echo -e 'Retrying in 30 seconds...'
Retrying in 30 seconds...
+ sleep 30
++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions
++ grep deploy_site
++ grep -i Processing
++ awk '{print $2}'
+ action_id=
+ (( retrieve_shipyard_action_counter ++ ))
+ error 'executing deploy_site from the /site directory'
+ set +x
Error when executing deploy_site from the /site directory.
+ exit 1
+ clean
+ set +x
To remove files generated during this script's execution, delete /root/deploy.
This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ From: Drew Walters [mailto:drewwalters96@gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas <nick@intracom-telecom.com> Cc: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> > wrote: Hi airship people, I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I’m getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <http://quay.io/airshipit/shipyard:master> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard <http://quay.io/airshipit/shipyard> " CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <http://quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <http://quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <http://quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> www <http://www.intracom-telecom.com/> .intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <http://quay.io/airshipit/shipyard:master> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <http://quay.io/airshipit/shipyard:master> get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <http://quay.io/airshipit/shipyard:master> get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <http://quay.io/airshipit/shipyard:master> get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org <mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root@n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root@n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_740645_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=kYlTaQKv2v1urN4KYmLZNYkhId4TdEFhmIt_itC6mRM&e=> From: Drew Walters [mailto:drewwalters96@gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas <nick@intracom-telecom.com> Cc: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>> wrote: Hi airship people, I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I’m getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=U3yD1Ji3NaELSW0iiqQWNv_6xcbZQI1m5P8Iwbnv52A&e=>" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com<mailto:nick@intracom-telecom.com> www.intracom-telecom.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=E6hDFFLPmnAW2gih1ARaQbeuzH7mcLzUdZ_BSrN7Pkc&e=> [1] https://github.com/openstack/airship-in-a-bottle<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=Zcf9kkgPjUXvi1K6dNfH7EQ6v6iS0yFf8qS8U12Dr2s&e=> [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org<mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=iE9izcAF-2v0B7FIJITXPtiiNGFYP7KQNCyEdb0Fixg&e=>
Hey Nikos, just checking - are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas <nick@intracom-telecom.com>; airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root@n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root@n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn't look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_740645_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=kYlTaQKv2v1urN4KYmLZNYkhId4TdEFhmIt_itC6mRM&e=> From: Drew Walters [mailto:drewwalters96@gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>> Cc: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>> wrote: Hi airship people, I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I'm getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=U3yD1Ji3NaELSW0iiqQWNv_6xcbZQI1m5P8Iwbnv52A&e=>" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com<mailto:nick@intracom-telecom.com> www.intracom-telecom.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=E6hDFFLPmnAW2gih1ARaQbeuzH7mcLzUdZ_BSrN7Pkc&e=> [1] https://github.com/openstack/airship-in-a-bottle<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=Zcf9kkgPjUXvi1K6dNfH7EQ6v6iS0yFf8qS8U12Dr2s&e=> [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org<mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=iE9izcAF-2v0B7FIJITXPtiiNGFYP7KQNCyEdb0Fixg&e=>
Hi Matt, Kaspars, I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same -unknown to me as yet- reason. I'm posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3]. (full output is attached - let me know how to share it if doesn't get through) I'm still looking at those myself but not identified something yet. Thanks Nikos [1]. http://paste.openstack.org/show/742654/ [2]. http://paste.openstack.org/show/742655/ [3]. http://paste.openstack.org/show/742660/ From: MCEUEN, MATT [mailto:MM9745@att.com] Sent: Monday, January 14, 2019 6:48 PM To: SKELS, KASPARS <ks3019@att.com>; Nikos Karandreas <nick@intracom-telecom.com>; airship-discuss@lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hey Nikos, just checking - are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> >; airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root@n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root@n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars _____ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn't look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_740645_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=kYlTaQKv2v1urN4KYmLZNYkhId4TdEFhmIt_itC6mRM&e=> From: Drew Walters [mailto:drewwalters96@gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> > Cc: airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> > wrote: Hi airship people, I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I'm getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=U3yD1Ji3NaELSW0iiqQWNv_6xcbZQI1m5P8Iwbnv52A&e=> " CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> www <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=E6hDFFLPmnAW2gih1ARaQbeuzH7mcLzUdZ_BSrN7Pkc&e=> .intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=Zcf9kkgPjUXvi1K6dNfH7EQ6v6iS0yFf8qS8U12Dr2s&e=> [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org <mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=iE9izcAF-2v0B7FIJITXPtiiNGFYP7KQNCyEdb0Fixg&e=>
Hi Nikos, failing libvirt is the cause for this (errors from Armada is that charts (nova, etc) are timing out to deploy) libvirt-hxzln 0/1 CrashLoopBackOff 26 1h 192.168.122.101 air1 Can you also share the logs for libvirt, to get some more insights.. and more folks familiar with this should be able to help # kubectl logs -n openstack libvirt-hxzln As a quick check, I've seen libvirt failing when there is a clash of pre-installed libvirt packages in the VM. Make sure you base VM is clean/regular Ubuntu 16.04. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 15, 2019 10:34 AM To: MCEUEN, MATT; airship-discuss@lists.airshipit.org; SKELS, KASPARS Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Matt, Kaspars, I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same –unknown to me as yet- reason. I’m posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3]. (full output is attached – let me know how to share it if doesn’t get through) I’m still looking at those myself but not identified something yet. Thanks Nikos [1]. http://paste.openstack.org/show/742654/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742654_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=2odIwtJQTum_-VFWIpGs7n3hPRXh5TL4PULy3TfWJYg&e=> [2]. http://paste.openstack.org/show/742655/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742655_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=23tFv00CzdY95wtDz0ttV7paJmTKrpK6bGiBY5VC-6I&e=> [3]. http://paste.openstack.org/show/742660/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742660_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=rceHkquD0ilBQYmRQxRBPJuXq6rGC0Bx8vq7yhmNfyk&e=> From: MCEUEN, MATT [mailto:MM9745@att.com] Sent: Monday, January 14, 2019 6:48 PM To: SKELS, KASPARS <ks3019@att.com>; Nikos Karandreas <nick@intracom-telecom.com>; airship-discuss@lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hey Nikos, just checking – are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>>; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root@n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root@n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_740645_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=kYlTaQKv2v1urN4KYmLZNYkhId4TdEFhmIt_itC6mRM&e=> From: Drew Walters [mailto:drewwalters96@gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>> Cc: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com<mailto:nick@intracom-telecom.com>> wrote: Hi airship people, I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I’m getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=U3yD1Ji3NaELSW0iiqQWNv_6xcbZQI1m5P8Iwbnv52A&e=>" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com<mailto:nick@intracom-telecom.com> www.intracom-telecom.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=E6hDFFLPmnAW2gih1ARaQbeuzH7mcLzUdZ_BSrN7Pkc&e=> [1] https://github.com/openstack/airship-in-a-bottle<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=Zcf9kkgPjUXvi1K6dNfH7EQ6v6iS0yFf8qS8U12Dr2s&e=> [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3<https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master<https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org<mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=iE9izcAF-2v0B7FIJITXPtiiNGFYP7KQNCyEdb0Fixg&e=>
Hi Kaspars, I got a successful installation of Airship-in-a-Bottle! I deselected "Virtual Machine host" in "Software selection" menu during Ubuntu installation (leaving only "standard system utilities" and "OpenSSH server"). I also stepped the VM from 8vCPU to 12vCPU, but I don't think that this was the crucial element. Thank you and everybody, a lot for the help and support. Nikos PS. As info piece, the log command for the failing pod returned: root@air1:~# kubectl logs -n openstack libvirt-hxzln Error from server (NotFound): pods "libvirt-hxzln" not found although the pod still appeared in the get pods command. From: SKELS, KASPARS [mailto:ks3019@att.com] Sent: Tuesday, January 15, 2019 7:01 PM To: Nikos Karandreas <nick@intracom-telecom.com>; MCEUEN, MATT <MM9745@att.com>; airship-discuss@lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos, failing libvirt is the cause for this (errors from Armada is that charts (nova, etc) are timing out to deploy) libvirt-hxzln 0/1 CrashLoopBackOff 26 1h 192.168.122.101 air1 Can you also share the logs for libvirt, to get some more insights.. and more folks familiar with this should be able to help # kubectl logs -n openstack libvirt-hxzln As a quick check, I've seen libvirt failing when there is a clash of pre-installed libvirt packages in the VM. Make sure you base VM is clean/regular Ubuntu 16.04. Kindly, Kaspars _____ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 15, 2019 10:34 AM To: MCEUEN, MATT; airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> ; SKELS, KASPARS Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Matt, Kaspars, I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same -unknown to me as yet- reason. I'm posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3]. (full output is attached - let me know how to share it if doesn't get through) I'm still looking at those myself but not identified something yet. Thanks Nikos [1]. http://paste.openstack.org/show/742654/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742654_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=2odIwtJQTum_-VFWIpGs7n3hPRXh5TL4PULy3TfWJYg&e=> [2]. http://paste.openstack.org/show/742655/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742655_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=23tFv00CzdY95wtDz0ttV7paJmTKrpK6bGiBY5VC-6I&e=> [3]. http://paste.openstack.org/show/742660/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_742660_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=lR2HDCz6L1ISf1mcz0U9y0Lq3lwFpmsitBtHMSj7CXw&s=rceHkquD0ilBQYmRQxRBPJuXq6rGC0Bx8vq7yhmNfyk&e=> From: MCEUEN, MATT [mailto:MM9745@att.com] Sent: Monday, January 14, 2019 6:48 PM To: SKELS, KASPARS <ks3019@att.com <mailto:ks3019@att.com> >; Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> >; airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hey Nikos, just checking - are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> >; airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root@n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root@n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root@n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars _____ From: Nikos Karandreas [nick@intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn't look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_740645_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=kYlTaQKv2v1urN4KYmLZNYkhId4TdEFhmIt_itC6mRM&e=> From: Drew Walters [mailto:drewwalters96@gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> > Cc: airship-discuss@lists.airshipit.org <mailto:airship-discuss@lists.airshipit.org> Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas <nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> > wrote: Hi airship people, I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I'm getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> and giving, after that, docker ps I get: root@airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=U3yD1Ji3NaELSW0iiqQWNv_6xcbZQI1m5P8Iwbnv52A&e=> " CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard@sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-40sha256-3Ada847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=EDLeytlDvbVqCxx--LxMVY5MYqkXmIpIY63e7UCEvg0&e=> "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick@intracom-telecom.com <mailto:nick@intracom-telecom.com> www <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=E6hDFFLPmnAW2gih1ARaQbeuzH7mcLzUdZ_BSrN7Pkc&e=> .intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=Zcf9kkgPjUXvi1K6dNfH7EQ6v6iS0yFf8qS8U12Dr2s&e=> [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'> #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <https://urldefense.proofpoint.com/v2/url?u=http-3A__keystone.ucp.svc.cluster.local-3A80_v3&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=ulEw05oXpWw020g8sLHTisZAEN3xSNGxjPuU_Pm-TtA&e=> -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master <https://urldefense.proofpoint.com/v2/url?u=http-3A__quay.io_airshipit_shipyard-3Amaster&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=hnxn8hzYvJUW5kyVgtgCl3pPuR6ExvJJznarDWpKkUE&e=> get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org <mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=5po3R_b6xOjlclr3L0cWnUO2lF1R5zjYgAkXauwz8Wk&s=iE9izcAF-2v0B7FIJITXPtiiNGFYP7KQNCyEdb0Fixg&e=>
On Jan 16, 2019, at 5:42 AM, Nikos Karandreas <nick@intracom-telecom.com> wrote:
Hi Kaspars,
I got a successful installation of Airship-in-a-Bottle!
I deselected “Virtual Machine host” in “Software selection” menu during Ubuntu installation (leaving only “standard system utilities” and “OpenSSH server”).
I also stepped the VM from 8vCPU to 12vCPU, but I don’t think that this was the crucial element.
Thank you and everybody, a lot for the help and support. Nikos
PS. As info piece, the log command for the failing pod returned: root@air1:~# kubectl logs -n openstack libvirt-hxzln Error from server (NotFound): pods "libvirt-hxzln" not found
although the pod still appeared in the get pods command.
Nikos, Glad to hear everything is operating with good success! I assume you are unable to get the logs of the libvirt pod because the failing pod has already been cleaned up. I can’t comment on the correlation between libvirt packages on the host VM and failing libvirt pods, but maybe someone else has some insight here. If you run into any other issues, please do not hesitate to reach out! Best, Drew
participants (4)
-
Drew Walters
-
MCEUEN, MATT
-
Nikos Karandreas
-
SKELS, KASPARS