From MM9745 at att.com Fri Jan 4 18:12:05 2019 From: MM9745 at att.com (MCEUEN, MATT) Date: Fri, 4 Jan 2019 18:12:05 +0000 Subject: [Airship-discuss] Armada API connectivity to Tiller (proxy problem) In-Reply-To: References: Message-ID: <7C64A75C21BB8D43BD75BB18635E4D896CF40055@MOSTLS1MSGUSRFF.ITServices.sbc.com> Hey Eli, Drew and I dug into this a bit, and it looks like the Armada-based lookup uses labels (configurably: application=tiller, name=tiller, in the kube-system namespace), and Shipyard looks up tiller based on the pod name [1]. I believe there was some condition under which the preferable label-based lookup wouldn’t work, and that’s why there a “plan B” for looking up based on pod name. However, I think this could be improved by 1) moving the “plan B” logic from Shipyard into Armada (your coupling point #2 below), and 2) allowing Armada to try “plan A” first before falling back to “plan B”. I’d like to discuss this in the 1/8 Airship team IRC meeting [2] – would you be able to join so we can make sure we account for all cases? Between {pod labels, pod name, service DNS} maybe we could find two Armada configuration-driven approaches that could cover all bases. [1] https://github.com/openstack/airship-shipyard/blob/c7a9c65c886641029ce84724ba7a999f8542a7c5/src/bin/shipyard_airflow/shipyard_airflow/plugins/get_k8s_pod_port_ip.py#L65 [2] https://etherpad.openstack.org/p/airship-meeting-2019-01-08 Thanks! Matt From: Evgeny L Sent: Wednesday, December 26, 2018 12:56 PM To: Drew Walters Cc: Airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Armada API connectivity to Tiller (proxy problem) Hi Drew, Thank you for the response! >> The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod Looking at the code [1] it seems to be that Armada will try to find Tiller pod in the event Shipyard doesn't find one. Was there some problem with Armada API not being able to retrieve Tiller endpoint so it required a fallback? They both seem to use k8s API for that. >> This can be changed per API request by providing the hostname of a different Tiller pod. From architecture point of view, do we expect it to work with multiple Tillers within a single Airship installation? I'm asking these questions to explore the options for resolving GRPC (which is used by Tiller client) [2] constraints, it does not handle CIDR in no_proxy environment variables [3]. One of the options is: 1. [high priority] Expose Tiller to create a DNS name for the pod, and use it when making requests from Armada API to Tiller, in this case Aramada API POD can have ".cluster.local" in NO_PROXY and everything would work just fine (i.e. requests to Tiller won't go through the proxy). 2. [low priority] Reduce coupling between Tiller/k8s and Shipyard by configuring Tiller endpoints in Armada API directly without requiring Shipyard to pass one. Prior to submitting any changes I would like to verify with you if this makes sense or if I should explore other options. Thanks, [1] https://github.com/openstack/airship-armada/blob/307f1318c4e83f247f1e3838478957e7555d6ce0/armada/handlers/tiller.py#L158-L164 [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149 On Thu, Dec 20, 2018 at 1:42 PM Drew Walters > wrote: Hi Eli, The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod. Shipyard uses the Python Kubernetes client to retrieve the IP address that it provides to Armada. 1. It is possible to create a clusterIP with DNS and provide a domain name to Armada. 2. Armada only communicates with one Tiller pod at a time. This can be changed per API request by providing the hostname of a different Tiller pod. In the event that a Tiller hostname is not provided, Armada selects the first running Tiller pod it encounters. We invite you to join our IRC team meeting (#airshipit on freenode) January 8th to help us better understand your desired configuration and discuss in more detail. In the meantime, please let us know if you have any additional questions! Best, Drew Walters On Wed, Dec 12, 2018 at 3:55 PM Evgeny L > wrote: Hi, I've been working on getting behind the proxy configuration for AIAB to work [1] and I stumbled across a problem related to GRPC [2], which does not support CIDR notation for no_proxy entries, this make Armada API to fail to connect to Tiller, because it tries to use proxy configured with environment variables. While looking for a solution other than doing some manual reconfiguration [2], I started to have the following questions: 1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP and use a DNS name? 1. Why Shipyard is the one who provides Armada API with Tiller IP address? If we do 1st we can possibly make endpoint a part of Armada API configuration. Or is Armada API supposed to work with multiple Tillers at the same time? Any feedback and historical background would be really helpful! Thanks, [1] https://review.openstack.org/#/c/624122/ [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149 [4] https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e3575e045e26f177252/src/bin/shipyard_airflow/shipyard_airflow/plugins/armada_base_operator.py#L147-L153 _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From MM9745 at att.com Fri Jan 4 18:28:12 2019 From: MM9745 at att.com (MCEUEN, MATT) Date: Fri, 4 Jan 2019 18:28:12 +0000 Subject: [Airship-discuss] Airship Website Refresh In-Reply-To: References: <3605CB09-4575-462C-9BF9-E826A294CE20@openstack.org> Message-ID: <7C64A75C21BB8D43BD75BB18635E4D896CF40223@MOSTLS1MSGUSRFF.ITServices.sbc.com> Hey James, great work on this! Looks really nice – a couple items: This may be on your todo list already, but – looks like the dev guide domain linked from the community page doesn’t resolve (yet): docs.airship.io, git.airship.io. What’s the planned distinction between airship.io and airshipit.org, or will one be an alias for the other? Could we use this please for the IRC meeting link: https://wiki.openstack.org/wiki/Airship#Get_in_Touch I like the idea of adding the subheader blurb under “Elevate Your Infrastructure”. Thanks! Matt From: James Cole Sent: Thursday, December 20, 2018 3:59 PM To: Evgeny L Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Airship Website Refresh Thanks for taking a look, Evgeny! We currently have the Airship in a bottle link on the community page, but maybe there can be a place on the homepage for it too. Good catch on the blog links—will update those. -James On Dec 19, 2018, at 12:59 PM, Evgeny L > wrote: Hi James, This looks awesome! Just some thoughts: it may be useful to have "Try it!" somewhere alongside "VIEW THE CODE" on the main page which would lead to "Airship in a bottle" docs, so newcomers can quickly find how to get it running. On a blog page there is "CONNECT" section which seems to have some broken links: https://airship-website-dev.netlify.com/blog/ Thanks you! On Wed, Dec 19, 2018 at 11:54 AM James Cole > wrote: Greetings Airship Team! I’m James Cole, a graphic designer with the OpenStack Foundation. We’ve been working on an updated look for the Airship website and wanted to get your feedback. Please view the updated site on Netlify: https://airship-website-dev.netlify.com/ A few things to note: The blog does not currently have any blog posts and we’d probably want to have something real published there before going live. If it isn’t possible to publish something before this launches, we can remove it from the navigation and add it back in once we have some content. Any guidance is welcomed if links on the Community page need to point to other locations. The Gerrit repo link does not currently link to anything useful, for instance. We’ve been considering adding a subhead under “Elevate Your Infrastructure” on the hero image of the homepage for clarity. One of the ideas we’re floating around is "Airship makes operating open infrastructure simple, repeatable, and resilient.” Does this sound good to you? Please let us know if you notice something that should change—spelling/grammar errors, factual errors, broken links, etc.—or if you have any other type of feedback. Once everyone is happy with the content and layout we’ll point the official URL (airshipit.org) to this version of the site. Thank you! James Cole Graphic Designer OpenStack Foundation _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From drewwalters96 at gmail.com Fri Jan 4 18:36:57 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Fri, 4 Jan 2019 12:36:57 -0600 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> Message-ID: Hi Alex! Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. Regards, Drew Walters On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL < Daniel.Balsiger at swisscom.com> wrote: > Hi there everyone > > > Thanks for the quick replies, I tried a new install on a fresh VM, behaves > the same, problem is persisting: > > > I see in Postgresql-0 logs: > > --snip > > FATAL: the database system is in recovery mode > LOG: database system was not properly shut down; automatic recovery in > progress > LOG: invalid record length at 0/102A400 > LOG: redo is not required > LOG: MultiXact member wraparound protections are now enabled > LOG: database system is ready to accept connections > LOG: autovacuum launcher started > FATAL: could not open relation mapping file "global/pg_filenode.map": No > such file or directory > LOG: autovacuum launcher process (PID 79) exited with exit code 1 > LOG: terminating any other active server processes > LOG: all server processes terminated; reinitializing > LOG: database system was interrupted; last known up at 2018-12-11 > 15:25:38 UTC > LOG: database system was not properly shut down; automatic recovery in > progress > --snip > > > and in mariadb-server-0 logs: > > --snip > > ubuntu at airship:~$ sudo kubectl -n ucp logs mariadb-server-0 > 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance > hostname: mariadb-server-0 > 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API > Version: v1.10.2 > 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use > "airship-ucp-mariadb-mariadb-state" configmap for cluster state info > 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for > cluster to start running > 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster > state > 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is > currently in "init" state. > --snip > > > > Best Regards, D > > ------------------------------ > *From:* Alex Arnoldy > *Sent:* Monday, December 10, 2018 7:48 PM > *To:* MCEUEN, MATT ; airship-discuss at lists.airshipit.org; Balsiger > Daniel, INI-EAI-INO-MBL > *Subject:* RE: Airship in a bottle install failure... > > > Hi Matt, > > > > Here is the commit I’m using: > > > > commit 221441f1bff351ab366b02d6fd7856b70b06108f > > Author: Bryan Strassner > > Date: Mon Nov 19 17:12:10 2018 -0600 > > > > Update apiserver for admission controller > > > > Update the apiserver's command prefix to match between the apiserver > > chart and the genesis api server. > > > > Includes an optimization of virsh commands to parallelize and wait for > > cloud init to complete (requires Ubuntu 16.04.4) > > > > Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 > > > > I’m happy to start over to see if a new pull resolves the issue, but it > happened to me twice in a row so I thought I’d preserve the environment for > examination. > > > > Best, > > > > --Alex. > > > > *Alex Arnoldy* > > > * Software Defined Infrastructure Solutions Architect SUSE * > *Alex.Arnoldy at SUSE.com* > * (P) +1 (408) 828-9797* > > > > *From:* MCEUEN, MATT > *Sent:* Monday, December 10, 2018 9:33 AM > *To:* airship-discuss at lists.airshipit.org; Alex Arnoldy < > alex.arnoldy at suse.com>; INI-EAI-INO-MBL Balsiger Daniel < > Daniel.Balsiger at swisscom.com> > *Subject:* RE: Airship in a bottle install failure... > > > > Weird, the maas-db-init job is running fine for me with a fresh deployment > of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error > current aiab code, or can you give git commit where you’re hitting the > issue? > > > > Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and > share the results? Looks like MariaDB isn’t starting cleanly for some > reason. > > > > Thanks, > Matt > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From MM9745 at att.com Fri Jan 4 18:45:08 2019 From: MM9745 at att.com (MCEUEN, MATT) Date: Fri, 4 Jan 2019 18:45:08 +0000 Subject: [Airship-discuss] =?utf-8?q?Airship-in-a-bottle=5FDev=5FMinimal?= =?utf-8?q?=5FBedhind_Proxy=EF=BC=9AHow_to_Update_Site=3F?= In-Reply-To: <002101d498c0$020cc0c0$06264240$@as.ntt-at.co.jp> References: <008001d49298$bed0db70$3c729250$@as.ntt-at.co.jp> <7C64A75C21BB8D43BD75BB18635E4D896CF0E916@MOSTLS1MSGUSRFF.ITServices.sbc.com> <002101d498c0$020cc0c0$06264240$@as.ntt-at.co.jp> Message-ID: <7C64A75C21BB8D43BD75BB18635E4D896CF40361@MOSTLS1MSGUSRFF.ITServices.sbc.com> Sorry for the delay, Santosh -- I'm catching up post-holiday. To supply updated documents to Shipyard, 1) First, update the documents in your git repository (e.g. something like https://github.com/openstack/airship-treasuremap/tree/master/site/airship-seaworthy). Your repo serves as the declarative definition of what your site looks like, both for an initial deployment, and later, for an update. 2) The steps to update a site via Shipyard are very similar to the initial deployment, but with fewer steps! Lint and collect your updated documents, per: https://airship-treasuremap.readthedocs.io/en/latest/authoring_and_deployment.html#manifest-linting-and-combining-layers Update the site using Shipyard*: https://airship-treasuremap.readthedocs.io/en/latest/authoring_and_deployment.html#deploy-site-with-shipyard * Note: should use the Shipyard`update_site.sh` script instead of `deploy_site.sh` in the above step. I think we should add another section to the end of that site authoring guide to describe the update process -- today the guide is geared toward new deployments. Let me know if you have any other questions! Thanks, Matt -----Original Message----- From: santosh.thapamager Sent: Thursday, December 20, 2018 5:59 PM To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi Matt, Thanks for the instruction. I have been following airship-treasuremap document. As you mentioned that we need to update the deployment manifest and supply to Shipyard and call Shipyard update_site API. *************** 1) supplying updated deployment manifests to Shipyard, including the change 2) calling Shipyard's update_site API *************** However I could not find instruction for 1) as you mentioned. I would be greatly thankful to you if I can get some instruction for updating the deployment manifests and how to supply updated manifests to Shipyard. Best Regards, Santosh Thapa Magar -----Original Message----- From: MCEUEN, MATT Sent: Thursday, December 20, 2018 4:37 AM To: santosh.thapamager ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi Santosh - in general, the right way to update an airship site is by: 1) supplying updated deployment manifests to Shipyard, including the change 2) calling Shipyard's update_site API We don't have that as part of Aiship-in-a-Bottle, because AiaB is not intended to be used for "real life" sites -- it's a demo environment. I suggest taking a look at the Treasuremap documentation [1] and example manifests [2] as a way to stand up a fully lifecycled-site. Update_site (along with the rest of the Shipyard interface) is documented here [3] for the CLI and here [4] for the API. 1: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtreasuremap.readthedocs.io_en_latest_authoring-5Fand-5Fdeployment.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=OUmBq4L34nRJCxeMURicQJuQTXp7dTZGkrSZO8DTw6s&e= 2: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=m5RHAnvcPXkc2rESb2Awl7FBDiEojHQWs5h-9UKiwow&e= 3: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dshipyard.readthedocs.io_en_latest_CLI.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=qLoy0_3Np0PQyXcpSXmkxC5z8_Gsbt43Q1vbE0_si5M&e= 4: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dshipyard.readthedocs.io_en_latest_API.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=2ckZS-p-G_eVrOfGGk9-CBvQiq6FgW8GYabJBi1yidA&e= The latest version of Promenade provisions a Kubernetes cluster that patches for the CVE you mention. Hope that helps - let me know if you have any other questions! Matt -----Original Message----- From: santosh.thapamager Sent: Wednesday, December 12, 2018 10:03 PM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi all, I was able to install dev_minimal and deploy site behind proxy. Because of Kubernetes apiserver vulnerability (CVE-2018-1002105)I wanted to update Kubernetes version in my currently deployed site. However I could not find any documentations regarding updating deployed site. I would be thankful to all of you, if I could get any support. Best Regards, Santosh Thapa Magar _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=Kp_9DVZcWk0qumyITPRn31pyOnpfXyno0HbJJJul7Xw&s=j6f8UNYzAnQn2PEYvDhC0N3F4i2vFF8qXyVC7Arx3hQ&e= From drewwalters96 at gmail.com Fri Jan 4 18:45:30 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Fri, 4 Jan 2019 12:45:30 -0600 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: <84485c2aaba55443913ad7b4a846d33b@iris> References: <84485c2aaba55443913ad7b4a846d33b@iris> Message-ID: Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas wrote: > Hi airship people, > > > > I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 > vCPU/24GB RAM/48GB disk) following the > > instructions in [1] but I’m getting the failure below (until the end in > [2]) > > > > + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e > no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e > OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e > OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e > OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action > deploy_site > > Error: Unable to complete request to Airflow > > Reason: Airflow could not be contacted properly by Shipyard. > > - Error: > > > > > > and giving, after that, docker ps I get: > > > > root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# > docker ps --all | grep "quay.io/airshipit/shipyard" > > CONTAINER ID > IMAGE > COMMAND CREATED STATUS > PORTS NAMES > > f4c0747b25ec > quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed > "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes > ago > k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 > > 04241489b515 > quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed > "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 > minutes ago > k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 > > d433934eec1d > quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed > "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes > ago > k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 > > > > > > Can you please give some insight on what might be wrong or what to look > for? > > > > > > Cheers > > Nikos > > > > > > > > *Nikos Karandreas* > > Developer, OPNFV > > Software Development Center > > _____________________________________ > Intracom Telecom > Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 > t: +30 2310497353, > f: +30 2310497330 > *nick at intracom-telecom.com * > > www.intracom-telecom.com > > > > > > > > > > [1] https://github.com/openstack/airship-in-a-bottle > > [2] > > Execute deploy_site Dag... > > > > + [[ -n '' ]] > > + [[ -n deploy_site ]] > > + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e > no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e > OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e > OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e > OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action > deploy_site > > Error: Unable to complete request to Airflow > > Reason: Airflow could not be contacted properly by Shipyard. > > - Error: > > > > #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### > > + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e > no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e > OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e > OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e > OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions > > Name Action Lifecycle Execution Time Step > Succ/Fail/Oth Footnotes > > None > > > > > > > + echo -e 'Retrieving Action ID...\n' > > Retrieving Action ID... > > > > ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e > no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e > OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e > OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e > OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions > > ++ grep -i Processing > > ++ awk '{print $2}' > > ++ grep deploy_site > > + action_id= > > + retrieve_shipyard_action_counter=0 > > + retrieve_shipyard_action_limit=2 > > + [[ 0 -le 2 ]] > > + [[ -n '' ]] > > + [[ 0 == 2 ]] > > + echo -e 'Unable to Retrieve Action ID!' > > Unable to Retrieve Action ID! > > + echo -e 'Retrying in 30 seconds...' > > Retrying in 30 seconds... > > + sleep 30 > > ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e > no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e > OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e > OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e > OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions > > ++ grep deploy_site > > ++ grep -i Processing > > ++ awk '{print $2}' > > + action_id= > > + (( retrieve_shipyard_action_counter ++ )) > > + error 'executing deploy_site from the /site directory' > > + set +x > > Error when executing deploy_site from the /site directory. > > + exit 1 > > + clean > > + set +x > > To remove files generated during this script's execution, delete > /root/deploy. > > This VM is disposable. Re-deployment in this same VM will lead to > unpredictable results. > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From taeil.choi at sk.com Mon Jan 7 05:11:50 2019 From: taeil.choi at sk.com (=?ks_c_5601-1987?B?w9bFwsDPtNQ=?=) Date: Mon, 7 Jan 2019 05:11:50 +0000 Subject: [Airship-discuss] Deckhand Usage Message-ID: *backgroud info: My team is currently only using Armada among many airship projects, and trying to leverage Deckhand as well. Hi all, I have questions regarding Deckhand usage. My team has a lot of armada manifests for many sites as following and we want to use Deckhand’s layering feature to manage the manifests efficiently. - global.yaml - site1.yaml - site2.yaml … I’ve looked at the Deckhand documentation and it’s not clear how to use that functionality. Q1. Deckhand standalone usage In the installation document (https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/getting-started.html#manual-installation), I saw a “development mode”, which doesn’t require Keystone authentication. However, when I looked at the Operator guide (https://airshipit.readthedocs.io/projects/deckhand/en/latest/operators/api_client.html), I couldn’t see how I can use Deckhand client without Keystone auth. Is it even possible? Q2. Layering As I mentioned above, my team have a lot of armada manifest files for various sites. How can I upload multiple manifests (Eg, global & site1) to the bucket and perform layering on those documents? Should I merge multiple files into one large string and upload it to the bucket? If it’s true, does the bucket update automatically trigger rendering process or should I explicitly call some rendering function?? Can anyone give an example code? Any help would be appreciated. Regards, Robert Choi -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.arnoldy at suse.com Mon Jan 7 11:52:39 2019 From: alex.arnoldy at suse.com (Alex Arnoldy) Date: Mon, 7 Jan 2019 11:52:39 +0000 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> Message-ID: <545AF708D41B364BA4146E47647D2D2AE403C8@prvxmb04.microfocus.com> Hi Drew, Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two. I’ll definitely let you know how things go. Very best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 From: Drew Walters Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org; Alex Arnoldy Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi Alex! Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. Regards, Drew Walters On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL > wrote: Hi there everyone Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting: I see in Postgresql-0 logs: --snip FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip and in mariadb-server-0 logs: --snip ubuntu at airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip Best Regards, D ________________________________ From: Alex Arnoldy > Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure... Hi Matt, Here is the commit I’m using: commit 221441f1bff351ab366b02d6fd7856b70b06108f Author: Bryan Strassner > Date: Mon Nov 19 17:12:10 2018 -0600 Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 From: MCEUEN, MATT > Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss at lists.airshipit.org; Alex Arnoldy >; INI-EAI-INO-MBL Balsiger Daniel > Subject: RE: Airship in a bottle install failure... Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue? Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason. Thanks, Matt _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From paye600 at gmail.com Mon Jan 7 11:57:26 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Mon, 7 Jan 2019 12:57:26 +0100 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: <545AF708D41B364BA4146E47647D2D2AE403C8@prvxmb04.microfocus.com> References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> <545AF708D41B364BA4146E47647D2D2AE403C8@prvxmb04.microfocus.com> Message-ID: Hello all, I have in-a-bottle deployment running right now for about two hours. Will let you know if it succeeds (it should). Best regards, -- Roman Gorshunov On Mon, Jan 7, 2019 at 12:52 PM Alex Arnoldy wrote: > > Hi Drew, > > > > Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two. > > > > I’ll definitely let you know how things go. > > > > Very best, > > > > --Alex. > > > > Alex Arnoldy > Software Defined Infrastructure Solutions Architect > SUSE > Alex.Arnoldy at SUSE.com > (P) +1 (408) 828-9797 > > > > From: Drew Walters > Sent: Friday, January 4, 2019 7:37 PM > To: INI-EAI-INO-MBL Balsiger Daniel > Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org; Alex Arnoldy > Subject: Re: [Airship-discuss] Airship in a bottle install failure... > > > > Hi Alex! > > > > Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. > > > > Regards, > > Drew Walters > > > > On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL wrote: > > Hi there everyone > > > > Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting: > > > > I see in Postgresql-0 logs: > > --snip > > FATAL: the database system is in recovery mode > LOG: database system was not properly shut down; automatic recovery in progress > LOG: invalid record length at 0/102A400 > LOG: redo is not required > LOG: MultiXact member wraparound protections are now enabled > LOG: database system is ready to accept connections > LOG: autovacuum launcher started > FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory > LOG: autovacuum launcher process (PID 79) exited with exit code 1 > LOG: terminating any other active server processes > LOG: all server processes terminated; reinitializing > LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC > LOG: database system was not properly shut down; automatic recovery in progress > > --snip > > > > and in mariadb-server-0 logs: > > --snip > > ubuntu at airship:~$ sudo kubectl -n ucp logs mariadb-server-0 > 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 > 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 > 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info > 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running > 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state > 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > --snip > > > > > > Best Regards, D > > > > ________________________________ > > From: Alex Arnoldy > Sent: Monday, December 10, 2018 7:48 PM > To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL > Subject: RE: Airship in a bottle install failure... > > > > Hi Matt, > > > > Here is the commit I’m using: > > > > commit 221441f1bff351ab366b02d6fd7856b70b06108f > > Author: Bryan Strassner > > Date: Mon Nov 19 17:12:10 2018 -0600 > > > > Update apiserver for admission controller > > > > Update the apiserver's command prefix to match between the apiserver > > chart and the genesis api server. > > > > Includes an optimization of virsh commands to parallelize and wait for > > cloud init to complete (requires Ubuntu 16.04.4) > > > > Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 > > > > I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination. > > > > Best, > > > > --Alex. > > > > Alex Arnoldy > Software Defined Infrastructure Solutions Architect > SUSE > Alex.Arnoldy at SUSE.com > (P) +1 (408) 828-9797 > > > > From: MCEUEN, MATT > Sent: Monday, December 10, 2018 9:33 AM > To: airship-discuss at lists.airshipit.org; Alex Arnoldy ; INI-EAI-INO-MBL Balsiger Daniel > Subject: RE: Airship in a bottle install failure... > > > > Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue? > > > > Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason. > > > > Thanks, > Matt > > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From paye600 at gmail.com Mon Jan 7 12:43:45 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Mon, 7 Jan 2019 13:43:45 +0100 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> <545AF708D41B364BA4146E47647D2D2AE403C8@prvxmb04.microfocus.com> Message-ID: Hello all, My deployment has just completed successfully. Git commit id a9f5fc93e33163f51fc56d1bf886d4d433251f8e (it's current master). -- Roman Gorshunov On Mon, Jan 7, 2019 at 12:57 PM Roman Gorshunov wrote: > > Hello all, > I have in-a-bottle deployment running right now for about two hours. > Will let you know if it succeeds (it should). > > Best regards, > -- > Roman Gorshunov > > On Mon, Jan 7, 2019 at 12:52 PM Alex Arnoldy wrote: > > > > Hi Drew, > > > > > > > > Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two. > > > > > > > > I’ll definitely let you know how things go. > > > > > > > > Very best, > > > > > > > > --Alex. > > > > > > > > Alex Arnoldy > > Software Defined Infrastructure Solutions Architect > > SUSE > > Alex.Arnoldy at SUSE.com > > (P) +1 (408) 828-9797 > > > > > > > > From: Drew Walters > > Sent: Friday, January 4, 2019 7:37 PM > > To: INI-EAI-INO-MBL Balsiger Daniel > > Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org; Alex Arnoldy > > Subject: Re: [Airship-discuss] Airship in a bottle install failure... > > > > > > > > Hi Alex! > > > > > > > > Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. > > > > > > > > Regards, > > > > Drew Walters > > > > > > > > On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL wrote: > > > > Hi there everyone > > > > > > > > Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting: > > > > > > > > I see in Postgresql-0 logs: > > > > --snip > > > > FATAL: the database system is in recovery mode > > LOG: database system was not properly shut down; automatic recovery in progress > > LOG: invalid record length at 0/102A400 > > LOG: redo is not required > > LOG: MultiXact member wraparound protections are now enabled > > LOG: database system is ready to accept connections > > LOG: autovacuum launcher started > > FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory > > LOG: autovacuum launcher process (PID 79) exited with exit code 1 > > LOG: terminating any other active server processes > > LOG: all server processes terminated; reinitializing > > LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC > > LOG: database system was not properly shut down; automatic recovery in progress > > > > --snip > > > > > > > > and in mariadb-server-0 logs: > > > > --snip > > > > ubuntu at airship:~$ sudo kubectl -n ucp logs mariadb-server-0 > > 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 > > 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 > > 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info > > 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running > > 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state > > 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. > > > > --snip > > > > > > > > > > > > Best Regards, D > > > > > > > > ________________________________ > > > > From: Alex Arnoldy > > Sent: Monday, December 10, 2018 7:48 PM > > To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL > > Subject: RE: Airship in a bottle install failure... > > > > > > > > Hi Matt, > > > > > > > > Here is the commit I’m using: > > > > > > > > commit 221441f1bff351ab366b02d6fd7856b70b06108f > > > > Author: Bryan Strassner > > > > Date: Mon Nov 19 17:12:10 2018 -0600 > > > > > > > > Update apiserver for admission controller > > > > > > > > Update the apiserver's command prefix to match between the apiserver > > > > chart and the genesis api server. > > > > > > > > Includes an optimization of virsh commands to parallelize and wait for > > > > cloud init to complete (requires Ubuntu 16.04.4) > > > > > > > > Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 > > > > > > > > I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination. > > > > > > > > Best, > > > > > > > > --Alex. > > > > > > > > Alex Arnoldy > > Software Defined Infrastructure Solutions Architect > > SUSE > > Alex.Arnoldy at SUSE.com > > (P) +1 (408) 828-9797 > > > > > > > > From: MCEUEN, MATT > > Sent: Monday, December 10, 2018 9:33 AM > > To: airship-discuss at lists.airshipit.org; Alex Arnoldy ; INI-EAI-INO-MBL Balsiger Daniel > > Subject: RE: Airship in a bottle install failure... > > > > > > > > Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue? > > > > > > > > Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason. > > > > > > > > Thanks, > > Matt > > > > > > > > _______________________________________________ > > Airship-discuss mailing list > > Airship-discuss at lists.airshipit.org > > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss > > > > _______________________________________________ > > Airship-discuss mailing list > > Airship-discuss at lists.airshipit.org > > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From eli at mirantis.com Mon Jan 7 14:53:40 2019 From: eli at mirantis.com (Evgeny L) Date: Mon, 7 Jan 2019 06:53:40 -0800 Subject: [Airship-discuss] Armada API connectivity to Tiller (proxy problem) In-Reply-To: <7C64A75C21BB8D43BD75BB18635E4D896CF40055@MOSTLS1MSGUSRFF.ITServices.sbc.com> References: <7C64A75C21BB8D43BD75BB18635E4D896CF40055@MOSTLS1MSGUSRFF.ITServices.sbc.com> Message-ID: Hi Matt, Drew, Thanks for looking into this, sure, I can join to the meeting, but can we move this topic down to the bottom? It's 06:00AM PT for me, so I just want to make sure that I won't be too late for the discussion. On Fri, Jan 4, 2019 at 10:12 AM MCEUEN, MATT wrote: > Hey Eli, > > > > Drew and I dug into this a bit, and it looks like the Armada-based lookup > uses labels (configurably: application=tiller, name=tiller, in the > kube-system namespace), and Shipyard looks up tiller based on the pod name > [1]. I believe there was some condition under which the preferable > label-based lookup wouldn’t work, and that’s why there a “plan B” for > looking up based on pod name. > > > > However, I think this could be improved by 1) moving the “plan B” logic > from Shipyard into Armada (your coupling point #2 below), and 2) allowing > Armada to try “plan A” first before falling back to “plan B”. I’d like to > discuss this in the 1/8 Airship team IRC meeting [2] – would you be able to > join so we can make sure we account for all cases? Between {pod labels, > pod name, service DNS} maybe we could find two Armada configuration-driven > approaches that could cover all bases. > > > > [1] > https://github.com/openstack/airship-shipyard/blob/c7a9c65c886641029ce84724ba7a999f8542a7c5/src/bin/shipyard_airflow/shipyard_airflow/plugins/get_k8s_pod_port_ip.py#L65 > > [2] https://etherpad.openstack.org/p/airship-meeting-2019-01-08 > > > > Thanks! > > Matt > > > > *From:* Evgeny L > *Sent:* Wednesday, December 26, 2018 12:56 PM > *To:* Drew Walters > *Cc:* Airship-discuss at lists.airshipit.org > *Subject:* Re: [Airship-discuss] Armada API connectivity to Tiller (proxy > problem) > > > > Hi Drew, > > > > Thank you for the response! > > > > >> The Armada API accepts a Tiller hostname as a safeguard in the event > that Armada is unable to find a running Tiller pod > > > > Looking at the code [1] it seems to be that Armada will try to find Tiller > pod in the event Shipyard doesn't find one. > > Was there some problem with Armada API not being able to retrieve Tiller > endpoint so it required a fallback? They both seem to use k8s API for that. > > > > >> This can be changed per API request by providing the hostname of a > different Tiller pod. > > > > From architecture point of view, do we expect it to work with multiple > Tillers within a single Airship installation? > > > > I'm asking these questions to explore the options for resolving GRPC > (which is used by Tiller client) [2] constraints, it does not handle CIDR > in no_proxy environment variables [3]. One of the options is: > > 1. [high priority] Expose Tiller to create a DNS name for the pod, and use > it when making requests from Armada API to Tiller, in this case Aramada API > POD can have ".cluster.local" in NO_PROXY and everything would work just > fine (i.e. requests to Tiller won't go through the proxy). > > 2. [low priority] Reduce coupling between Tiller/k8s and Shipyard by > configuring Tiller endpoints in Armada API directly without requiring > Shipyard to pass one. > > > > Prior to submitting any changes I would like to verify with you if this > makes sense or if I should explore other options. > > > > Thanks, > > > > [1] > https://github.com/openstack/airship-armada/blob/307f1318c4e83f247f1e3838478957e7555d6ce0/armada/handlers/tiller.py#L158-L164 > > > [2] > https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst > > > [3] > https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149 > > > > > > > On Thu, Dec 20, 2018 at 1:42 PM Drew Walters > wrote: > > Hi Eli, > > > > The Armada API accepts a Tiller hostname as a safeguard in the event that > Armada is unable to find a running Tiller pod. Shipyard uses the Python > Kubernetes client to retrieve the IP address that it provides to Armada. > > > > 1. It is possible to create a clusterIP with DNS and provide a domain name > to Armada. > > > > 2. Armada only communicates with one Tiller pod at a time. This can be > changed per API request by providing the hostname of a different Tiller > pod. In the event that a Tiller hostname is not provided, Armada selects > the first running Tiller pod it encounters. > > > > We invite you to join our IRC team meeting (#airshipit on freenode) > January 8th to help us better understand your desired configuration and > discuss in more detail. In the meantime, please let us know if you have any > additional questions! > > > > Best, > > Drew Walters > > > > > > On Wed, Dec 12, 2018 at 3:55 PM Evgeny L wrote: > > Hi, > > > > I've been working on getting behind the proxy configuration for AIAB to > work [1] and I stumbled across a problem related to GRPC [2], which does > not support CIDR notation for no_proxy entries, this make Armada API to > fail to connect to Tiller, because it tries to use proxy configured with > environment variables. > > > > While looking for a solution other than doing some manual reconfiguration > [2], I started to have the following questions: > > 1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP > and use a DNS name? > > > 1. Why Shipyard is the one who provides Armada API with Tiller IP > address? If we do 1st we can possibly make endpoint a part of Armada API > configuration. Or is Armada API supposed to work with multiple Tillers at > the same time? > > > > Any feedback and historical background would be really helpful! > > > > Thanks, > > > > [1] https://review.openstack.org/#/c/624122/ > > > [2] > https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst > > > [3] > https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149 > > > [4] > https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e3575e045e26f177252/src/bin/shipyard_airflow/shipyard_airflow/plugins/armada_base_operator.py#L147-L153 > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drewwalters96 at gmail.com Mon Jan 7 20:28:56 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Mon, 7 Jan 2019 14:28:56 -0600 Subject: [Airship-discuss] Deckhand Usage In-Reply-To: References: Message-ID: Hi Robert! The Deckhand client does not have the ability to bypass Keystone authentication; it was designed as a component of the full Airship integration. Are you attempting to use Deckhand without Keystone authentication merely for demonstration purposes? When submitting documents to Deckhand buckets, documents must be presented as a multi-document YAML body . This is because each addition of new documents to a bucket creates a new "revision". When documents from previous revisions are excluded from the most recently-submitted revision, they are only accessible from previous revisions. The common approach to achieve a single file containing all of your documents is to store them in a repository like airship-treasuremap with corresponding global/ site/ and type/ directories. Using Pegleg's collect command , you can combine all of the documents into a single, flat file and submit to Deckhand for rendering; however, the aforementioned approach requires certain documents to be present (e.g. a site definition) that would not be useful for operating with Armada and Deckhand alone. Lastly, documents are rendered (i.e. layered with substitution) when they are retrieved from the /revisions/{revision_id}/rendered-documents endpoint. Hopefully the above information is enough to get you started. Feel free to reply or join #airshipit on Freenode IRC with any additional questions! Regards, Drew Walters On Sun, Jan 6, 2019 at 11:12 PM 최태일님 wrote: > *backgroud info: My team is currently only using Armada among many airship > projects, and trying to leverage Deckhand as well. > > > > Hi all, > > > > I have questions regarding Deckhand usage. > > > > My team has a lot of armada manifests for many sites as following and we > want to use Deckhand’s layering feature to manage the manifests efficiently. > > - global.yaml > > - site1.yaml > > - site2.yaml > > … > > > > I’ve looked at the Deckhand documentation and it’s not clear how to use > that functionality. > > > > Q1. Deckhand standalone usage > > In the installation document ( > https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/getting-started.html#manual-installation > ), > > I saw a “development mode”, which doesn’t require Keystone authentication. > However, when I looked at the Operator guide ( > https://airshipit.readthedocs.io/projects/deckhand/en/latest/operators/api_client.html), > I couldn’t see how I can use Deckhand client without Keystone auth. Is it > even possible? > > > > Q2. Layering > > As I mentioned above, my team have a lot of armada manifest files for > various sites. How can I upload multiple manifests (Eg, global & site1) to > the bucket and perform layering on those documents? > > Should I merge multiple files into one large string and upload it to the > bucket? If it’s true, does the bucket update automatically trigger > rendering process or should I explicitly call some rendering function?? > > Can anyone give an example code? > > > > Any help would be appreciated. > > > > Regards, > > Robert Choi > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Daniel.Balsiger at swisscom.com Tue Jan 8 01:00:55 2019 From: Daniel.Balsiger at swisscom.com (Daniel.Balsiger at swisscom.com) Date: Tue, 8 Jan 2019 01:00:55 +0000 Subject: [Airship-discuss] [Resolved] Airship in a bottle install failure... Message-ID: Hi there everyone Thanks for helping out with your replies. Airship-in-a-bottle indeed deploys successfully. I used latest master (commit a9f5fc93e33163f51fc56d1bf886d4d433251f8e). Most likely my bad, I forgot to put the VMs hostname in /etc/hosts, so the VM could not resolve its own hostname. That probably has been the reason for the previous failing deployment... Best Regards, Daniel From santosh.thapamager at as.ntt-at.co.jp Tue Jan 8 02:57:04 2019 From: santosh.thapamager at as.ntt-at.co.jp (santosh.thapamager) Date: Tue, 8 Jan 2019 11:57:04 +0900 Subject: [Airship-discuss] =?utf-8?q?Airship-in-a-bottle=5FDev=5FMinimal?= =?utf-8?q?=5FBedhind_Proxy=EF=BC=9AHow_to_Update_Site=3F?= In-Reply-To: <7C64A75C21BB8D43BD75BB18635E4D896CF40361@MOSTLS1MSGUSRFF.ITServices.sbc.com> References: <008001d49298$bed0db70$3c729250$@as.ntt-at.co.jp> <7C64A75C21BB8D43BD75BB18635E4D896CF0E916@MOSTLS1MSGUSRFF.ITServices.sbc.com> <002101d498c0$020cc0c0$06264240$@as.ntt-at.co.jp> <7C64A75C21BB8D43BD75BB18635E4D896CF40361@MOSTLS1MSGUSRFF.ITServices.sbc.com> Message-ID: <002e01d4a6fd$d60eae10$822c0a30$@as.ntt-at.co.jp> Hi Matt, Thanks a lot for the reply. I got some idea about updating process. However I would like to ask you about the files that needs to be updated if I wanted to update the versions of Kubernetes and its components, UCP components, OSH Components. Thanking you in advance. Best Regards, Santosh -----Original Message----- From: MCEUEN, MATT Sent: Saturday, January 05, 2019 3:45 AM To: santosh.thapamager ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Sorry for the delay, Santosh -- I'm catching up post-holiday. To supply updated documents to Shipyard, 1) First, update the documents in your git repository (e.g. something like https://github.com/openstack/airship-treasuremap/tree/master/site/airship-seaworthy). Your repo serves as the declarative definition of what your site looks like, both for an initial deployment, and later, for an update. 2) The steps to update a site via Shipyard are very similar to the initial deployment, but with fewer steps! Lint and collect your updated documents, per: https://airship-treasuremap.readthedocs.io/en/latest/authoring_and_deployment.html#manifest-linting-and-combining-layers Update the site using Shipyard*: https://airship-treasuremap.readthedocs.io/en/latest/authoring_and_deployment.html#deploy-site-with-shipyard * Note: should use the Shipyard`update_site.sh` script instead of `deploy_site.sh` in the above step. I think we should add another section to the end of that site authoring guide to describe the update process -- today the guide is geared toward new deployments. Let me know if you have any other questions! Thanks, Matt -----Original Message----- From: santosh.thapamager Sent: Thursday, December 20, 2018 5:59 PM To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi Matt, Thanks for the instruction. I have been following airship-treasuremap document. As you mentioned that we need to update the deployment manifest and supply to Shipyard and call Shipyard update_site API. *************** 1) supplying updated deployment manifests to Shipyard, including the change 2) calling Shipyard's update_site API *************** However I could not find instruction for 1) as you mentioned. I would be greatly thankful to you if I can get some instruction for updating the deployment manifests and how to supply updated manifests to Shipyard. Best Regards, Santosh Thapa Magar -----Original Message----- From: MCEUEN, MATT Sent: Thursday, December 20, 2018 4:37 AM To: santosh.thapamager ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi Santosh - in general, the right way to update an airship site is by: 1) supplying updated deployment manifests to Shipyard, including the change 2) calling Shipyard's update_site API We don't have that as part of Aiship-in-a-Bottle, because AiaB is not intended to be used for "real life" sites -- it's a demo environment. I suggest taking a look at the Treasuremap documentation [1] and example manifests [2] as a way to stand up a fully lifecycled-site. Update_site (along with the rest of the Shipyard interface) is documented here [3] for the CLI and here [4] for the API. 1: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtreasuremap.readthedocs.io_en_latest_authoring-5Fand-5Fdeployment.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=OUmBq4L34nRJCxeMURicQJuQTXp7dTZGkrSZO8DTw6s&e= 2: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=m5RHAnvcPXkc2rESb2Awl7FBDiEojHQWs5h-9UKiwow&e= 3: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dshipyard.readthedocs.io_en_latest_CLI.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=qLoy0_3Np0PQyXcpSXmkxC5z8_Gsbt43Q1vbE0_si5M&e= 4: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dshipyard.readthedocs.io_en_latest_API.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=2ckZS-p-G_eVrOfGGk9-CBvQiq6FgW8GYabJBi1yidA&e= The latest version of Promenade provisions a Kubernetes cluster that patches for the CVE you mention. Hope that helps - let me know if you have any other questions! Matt -----Original Message----- From: santosh.thapamager Sent: Wednesday, December 12, 2018 10:03 PM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi all, I was able to install dev_minimal and deploy site behind proxy. Because of Kubernetes apiserver vulnerability (CVE-2018-1002105)I wanted to update Kubernetes version in my currently deployed site. However I could not find any documentations regarding updating deployed site. I would be thankful to all of you, if I could get any support. Best Regards, Santosh Thapa Magar _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=Kp_9DVZcWk0qumyITPRn31pyOnpfXyno0HbJJJul7Xw&s=j6f8UNYzAnQn2PEYvDhC0N3F4i2vFF8qXyVC7Arx3hQ&e= From nick at intracom-telecom.com Tue Jan 8 15:43:27 2019 From: nick at intracom-telecom.com (Nikos Karandreas) Date: Tue, 08 Jan 2019 17:43:27 +0200 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: Message-ID: <766e51f3e1cdf34bb668e7c130cb883e@iris> Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ From: Drew Walters [mailto:drewwalters96 at gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas > wrote: Hi airship people, I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I’m getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: and giving, after that, docker ps I get: root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard " CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick at intracom-telecom.com www .intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From taeil.choi at sk.com Wed Jan 9 00:00:22 2019 From: taeil.choi at sk.com (=?utf-8?B?7LWc7YOc7J2864uY?=) Date: Wed, 9 Jan 2019 00:00:22 +0000 Subject: [Airship-discuss] Deckhand Usage In-Reply-To: References: Message-ID: <96894ef0744f41ecb3fedb5194bbb5c6@skt-tnetpmx2.SKT.AD> Hi Drew, Thank you so much for your quick and detailed answer. Honestly, the Deckhand usage is much more complicated than I expected. Considering the overhead, it doesn’t really fit to our use case, so I think it’s better to try simpler approach first. If it doesn’t work, then I’ll try the Deckhand again. Thanks again! Regards, Robert From: Drew Walters Sent: Tuesday, January 08, 2019 5:29 AM To: 최태일님/Virtualization SW개발팀 Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Deckhand Usage Hi Robert! The Deckhand client does not have the ability to bypass Keystone authentication; it was designed as a component of the full Airship integration. Are you attempting to use Deckhand without Keystone authentication merely for demonstration purposes? When submitting documents to Deckhand buckets, documents must be presented as a multi-document YAML body. This is because each addition of new documents to a bucket creates a new "revision". When documents from previous revisions are excluded from the most recently-submitted revision, they are only accessible from previous revisions. The common approach to achieve a single file containing all of your documents is to store them in a repository like airship-treasuremap with corresponding global/ site/ and type/ directories. Using Pegleg's collect command, you can combine all of the documents into a single, flat file and submit to Deckhand for rendering; however, the aforementioned approach requires certain documents to be present (e.g. a site definition) that would not be useful for operating with Armada and Deckhand alone. Lastly, documents are rendered (i.e. layered with substitution) when they are retrieved from the /revisions/{revision_id}/rendered-documents endpoint. Hopefully the above information is enough to get you started. Feel free to reply or join #airshipit on Freenode IRC with any additional questions! Regards, Drew Walters On Sun, Jan 6, 2019 at 11:12 PM 최태일님 > wrote: *backgroud info: My team is currently only using Armada among many airship projects, and trying to leverage Deckhand as well. Hi all, I have questions regarding Deckhand usage. My team has a lot of armada manifests for many sites as following and we want to use Deckhand’s layering feature to manage the manifests efficiently. - global.yaml - site1.yaml - site2.yaml … I’ve looked at the Deckhand documentation and it’s not clear how to use that functionality. Q1. Deckhand standalone usage In the installation document (https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/getting-started.html#manual-installation), I saw a “development mode”, which doesn’t require Keystone authentication. However, when I looked at the Operator guide (https://airshipit.readthedocs.io/projects/deckhand/en/latest/operators/api_client.html), I couldn’t see how I can use Deckhand client without Keystone auth. Is it even possible? Q2. Layering As I mentioned above, my team have a lot of armada manifest files for various sites. How can I upload multiple manifests (Eg, global & site1) to the bucket and perform layering on those documents? Should I merge multiple files into one large string and upload it to the bucket? If it’s true, does the bucket update automatically trigger rendering process or should I explicitly call some rendering function?? Can anyone give an example code? Any help would be appreciated. Regards, Robert Choi _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From ks3019 at att.com Wed Jan 9 01:19:12 2019 From: ks3019 at att.com (SKELS, KASPARS) Date: Wed, 9 Jan 2019 01:19:12 +0000 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: <766e51f3e1cdf34bb668e7c130cb883e@iris> References: , <766e51f3e1cdf34bb668e7c130cb883e@iris> Message-ID: <2ADBF0C373B7E84E944B1E06D3CDDFC91E65B142@MOKSCY3MSGUSRGI.ITServices.sbc.com> Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root at n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root at n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick at intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ From: Drew Walters [mailto:drewwalters96 at gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas > wrote: Hi airship people, I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I’m getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: and giving, after that, docker ps I get: root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick at intracom-telecom.com www.intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From se136c at att.com Wed Jan 9 18:02:00 2019 From: se136c at att.com (EAGAN, SEAN) Date: Wed, 9 Jan 2019 18:02:00 +0000 Subject: [Airship-discuss] Armada Compatibility with Keystone In-Reply-To: <25B6D7837B9EB04495DAD8D02D71DFF0203E9DF7@ALA-MBD.corp.ad.wrs.com> References: <25B6D7837B9EB04495DAD8D02D71DFF0203E9DF7@ALA-MBD.corp.ad.wrs.com> Message-ID: <5B9C5644A0B2414E90A7F40249CF6A9555346046@MOSTLS1MSGUSRFA.ITServices.sbc.com> Shoaib, It looks like that feature of k8s-keystone-auth [0] is implemented by kubernetes/client-go [1] [2]. kubernetes/client-go is used by helm so that's why it works there. However, armada is using kubernetes-client/python, and as far as I can tell this feature is not implemented there. Unless I am missing it. I am not sure what the easiest path to getting that implemented will be. [0]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-keystone-webhook-authenticator-and-authorizer.md#clientkubectl-configuration [1]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-client-keystone-auth.md [2]: https://github.com/kubernetes/kubernetes/pull/64346 Sean Eagan From: Nasir, Shoaib Sent: Thursday, December 13, 2018 12:35 PM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Armada Compatibility with Keystone Hi I have a kubernetes cluster that is setup with Keystone as its Authentication/Authorization Backend (via the k8s-keystone-auth Webhook pod) With that, any application such as helm now needs to pass user credentials or a keystone token when talking to the kubernetes-apiserver If I set up env OS_ variables and use any helm commands (such as helm init, or helm lists etc), everything works fine as these OS_ user credentials are authorized via the kubernetes keystone backend However, I run into problems with Armada. After bringing up the armada docker container, and logging into it to execute any armada apply commands, and using the -token argument to pass along a valid keystone token, armada still fails to access the kubernetes cluster."User "system:anonymous" suggests that the token that armada cli gets, does not pass along to the kubernetes-apiserver. I'm very new to armada and need some assistance in figuring out how to setup the armada container or any other configurations required to make it pass the keystone token to the kubernetes-apiserver. Each and every armada command fails similar to the one below. Any assistance in this regard would be greatly appreciated ? armada at c7f42c9263ad:~$ armada --token $TOKEN --debug apply 2018-12-13 18:10:13.656 72 ERROR armada.cli [-] Caught unexpected exception: kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli Traceback (most recent call last): 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli self.invoke() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 215, in invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli target_manifest=self.target_manifest) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 103, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli dry_run=dry_run) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 95, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli self.channel = self.get_channel() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 117, in get_channel 2018-12-13 18:10:13.656 72 ERROR armada.cli tiller_ip = self._get_tiller_ip() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 163, in _get_tiller_ip 2018-12-13 18:10:13.656 72 ERROR armada.cli pod = self._get_tiller_pod() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 141, in _get_tiller_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli CONF.tiller_pod_labels).items 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/k8s.py", line 215, in get_namespace_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli namespace, label_selector=label_selector) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info 2018-12-13 18:10:13.656 72 ERROR armada.cli collection_formats=collection_formats) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _request_timeout=_request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 342, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli headers=headers) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 231, in GET 2018-12-13 18:10:13.656 72 ERROR armada.cli query_params=query_params) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli raise ApiException(http_resp=r) 2018-12-13 18:10:13.656 72 ERROR armada.cli kubernetes.client.rest.ApiException: (403) 2018-12-13 18:10:13.656 72 ERROR armada.cli Reason: Forbidden 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shoaib.Nasir at windriver.com Thu Jan 10 15:26:04 2019 From: Shoaib.Nasir at windriver.com (Nasir, Shoaib) Date: Thu, 10 Jan 2019 15:26:04 +0000 Subject: [Airship-discuss] Armada Compatibility with Keystone In-Reply-To: <5B9C5644A0B2414E90A7F40249CF6A9555346046@MOSTLS1MSGUSRFA.ITServices.sbc.com> References: <25B6D7837B9EB04495DAD8D02D71DFF0203E9DF7@ALA-MBD.corp.ad.wrs.com> <5B9C5644A0B2414E90A7F40249CF6A9555346046@MOSTLS1MSGUSRFA.ITServices.sbc.com> Message-ID: <25B6D7837B9EB04495DAD8D02D71DFF0203EAEAD@ALA-MBD.corp.ad.wrs.com> Hi Sean After digging further into the issue, I realized that armada does not support passing end users bearer-tokens. Armada can be authenticated by keystone for incoming API requests. But in the scenario, where I want to use armada to bring up openstack-helm pods on a kubernetes cluster that's already setup to be authenticated by keystone using the k8s-keystone-auth Webhook application, armada is not compatible. Therefore, I have made some changes to the armada CLI such that a user can pass its bearer-token to tiller via armada using the CLI. I will be proposing this upstream to armada soon Thanks Shoaib Nasir From: EAGAN, SEAN [mailto:se136c at att.com] Sent: Wednesday, January 09, 2019 1:02 PM To: Nasir, Shoaib; airship-discuss at lists.airshipit.org Subject: RE: Armada Compatibility with Keystone Shoaib, It looks like that feature of k8s-keystone-auth [0] is implemented by kubernetes/client-go [1] [2]. kubernetes/client-go is used by helm so that's why it works there. However, armada is using kubernetes-client/python, and as far as I can tell this feature is not implemented there. Unless I am missing it. I am not sure what the easiest path to getting that implemented will be. [0]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-keystone-webhook-authenticator-and-authorizer.md#clientkubectl-configuration [1]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-client-keystone-auth.md [2]: https://github.com/kubernetes/kubernetes/pull/64346 Sean Eagan From: Nasir, Shoaib Sent: Thursday, December 13, 2018 12:35 PM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Armada Compatibility with Keystone Hi I have a kubernetes cluster that is setup with Keystone as its Authentication/Authorization Backend (via the k8s-keystone-auth Webhook pod) With that, any application such as helm now needs to pass user credentials or a keystone token when talking to the kubernetes-apiserver If I set up env OS_ variables and use any helm commands (such as helm init, or helm lists etc), everything works fine as these OS_ user credentials are authorized via the kubernetes keystone backend However, I run into problems with Armada. After bringing up the armada docker container, and logging into it to execute any armada apply commands, and using the -token argument to pass along a valid keystone token, armada still fails to access the kubernetes cluster."User "system:anonymous" suggests that the token that armada cli gets, does not pass along to the kubernetes-apiserver. I'm very new to armada and need some assistance in figuring out how to setup the armada container or any other configurations required to make it pass the keystone token to the kubernetes-apiserver. Each and every armada command fails similar to the one below. Any assistance in this regard would be greatly appreciated ? armada at c7f42c9263ad:~$ armada --token $TOKEN --debug apply 2018-12-13 18:10:13.656 72 ERROR armada.cli [-] Caught unexpected exception: kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli Traceback (most recent call last): 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli self.invoke() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 215, in invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli target_manifest=self.target_manifest) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 103, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli dry_run=dry_run) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 95, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli self.channel = self.get_channel() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 117, in get_channel 2018-12-13 18:10:13.656 72 ERROR armada.cli tiller_ip = self._get_tiller_ip() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 163, in _get_tiller_ip 2018-12-13 18:10:13.656 72 ERROR armada.cli pod = self._get_tiller_pod() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 141, in _get_tiller_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli CONF.tiller_pod_labels).items 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/k8s.py", line 215, in get_namespace_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli namespace, label_selector=label_selector) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info 2018-12-13 18:10:13.656 72 ERROR armada.cli collection_formats=collection_formats) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _request_timeout=_request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 342, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli headers=headers) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 231, in GET 2018-12-13 18:10:13.656 72 ERROR armada.cli query_params=query_params) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli raise ApiException(http_resp=r) 2018-12-13 18:10:13.656 72 ERROR armada.cli kubernetes.client.rest.ApiException: (403) 2018-12-13 18:10:13.656 72 ERROR armada.cli Reason: Forbidden 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli -------------- next part -------------- An HTML attachment was scrubbed... URL: From MM9745 at att.com Mon Jan 14 16:44:09 2019 From: MM9745 at att.com (MCEUEN, MATT) Date: Mon, 14 Jan 2019 16:44:09 +0000 Subject: [Airship-discuss] Armada Compatibility with Keystone In-Reply-To: <25B6D7837B9EB04495DAD8D02D71DFF0203EAEAD@ALA-MBD.corp.ad.wrs.com> References: <25B6D7837B9EB04495DAD8D02D71DFF0203E9DF7@ALA-MBD.corp.ad.wrs.com> <5B9C5644A0B2414E90A7F40249CF6A9555346046@MOSTLS1MSGUSRFA.ITServices.sbc.com> <25B6D7837B9EB04495DAD8D02D71DFF0203EAEAD@ALA-MBD.corp.ad.wrs.com> Message-ID: <7C64A75C21BB8D43BD75BB18635E4D896CFB0739@MOSTLS1MSGUSRFF.ITServices.sbc.com> Looking forward to seeing what you have in mind, Shoaib. If you'd like to discuss in tomorrow's team meeting, feel free to add the topic to the etherpad: https://etherpad.openstack.org/p/airship-meeting-2019-01-15 Thanks, Matt From: Nasir, Shoaib Sent: Thursday, January 10, 2019 9:26 AM To: EAGAN, SEAN ; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Armada Compatibility with Keystone Hi Sean After digging further into the issue, I realized that armada does not support passing end users bearer-tokens. Armada can be authenticated by keystone for incoming API requests. But in the scenario, where I want to use armada to bring up openstack-helm pods on a kubernetes cluster that's already setup to be authenticated by keystone using the k8s-keystone-auth Webhook application, armada is not compatible. Therefore, I have made some changes to the armada CLI such that a user can pass its bearer-token to tiller via armada using the CLI. I will be proposing this upstream to armada soon Thanks Shoaib Nasir From: EAGAN, SEAN [mailto:se136c at att.com] Sent: Wednesday, January 09, 2019 1:02 PM To: Nasir, Shoaib; airship-discuss at lists.airshipit.org Subject: RE: Armada Compatibility with Keystone Shoaib, It looks like that feature of k8s-keystone-auth [0] is implemented by kubernetes/client-go [1] [2]. kubernetes/client-go is used by helm so that's why it works there. However, armada is using kubernetes-client/python, and as far as I can tell this feature is not implemented there. Unless I am missing it. I am not sure what the easiest path to getting that implemented will be. [0]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-keystone-webhook-authenticator-and-authorizer.md#clientkubectl-configuration [1]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-client-keystone-auth.md [2]: https://github.com/kubernetes/kubernetes/pull/64346 Sean Eagan From: Nasir, Shoaib > Sent: Thursday, December 13, 2018 12:35 PM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Armada Compatibility with Keystone Hi I have a kubernetes cluster that is setup with Keystone as its Authentication/Authorization Backend (via the k8s-keystone-auth Webhook pod) With that, any application such as helm now needs to pass user credentials or a keystone token when talking to the kubernetes-apiserver If I set up env OS_ variables and use any helm commands (such as helm init, or helm lists etc), everything works fine as these OS_ user credentials are authorized via the kubernetes keystone backend However, I run into problems with Armada. After bringing up the armada docker container, and logging into it to execute any armada apply commands, and using the -token argument to pass along a valid keystone token, armada still fails to access the kubernetes cluster."User "system:anonymous" suggests that the token that armada cli gets, does not pass along to the kubernetes-apiserver. I'm very new to armada and need some assistance in figuring out how to setup the armada container or any other configurations required to make it pass the keystone token to the kubernetes-apiserver. Each and every armada command fails similar to the one below. Any assistance in this regard would be greatly appreciated ? armada at c7f42c9263ad:~$ armada --token $TOKEN --debug apply 2018-12-13 18:10:13.656 72 ERROR armada.cli [-] Caught unexpected exception: kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli Traceback (most recent call last): 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli self.invoke() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 215, in invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli target_manifest=self.target_manifest) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 103, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli dry_run=dry_run) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 95, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli self.channel = self.get_channel() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 117, in get_channel 2018-12-13 18:10:13.656 72 ERROR armada.cli tiller_ip = self._get_tiller_ip() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 163, in _get_tiller_ip 2018-12-13 18:10:13.656 72 ERROR armada.cli pod = self._get_tiller_pod() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 141, in _get_tiller_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli CONF.tiller_pod_labels).items 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/k8s.py", line 215, in get_namespace_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli namespace, label_selector=label_selector) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info 2018-12-13 18:10:13.656 72 ERROR armada.cli collection_formats=collection_formats) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _request_timeout=_request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 342, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli headers=headers) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 231, in GET 2018-12-13 18:10:13.656 72 ERROR armada.cli query_params=query_params) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli raise ApiException(http_resp=r) 2018-12-13 18:10:13.656 72 ERROR armada.cli kubernetes.client.rest.ApiException: (403) 2018-12-13 18:10:13.656 72 ERROR armada.cli Reason: Forbidden 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli -------------- next part -------------- An HTML attachment was scrubbed... URL: From MM9745 at att.com Mon Jan 14 16:47:51 2019 From: MM9745 at att.com (MCEUEN, MATT) Date: Mon, 14 Jan 2019 16:47:51 +0000 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: <2ADBF0C373B7E84E944B1E06D3CDDFC91E65B142@MOKSCY3MSGUSRGI.ITServices.sbc.com> References: , <766e51f3e1cdf34bb668e7c130cb883e@iris> <2ADBF0C373B7E84E944B1E06D3CDDFC91E65B142@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: <7C64A75C21BB8D43BD75BB18635E4D896CFB082D@MOSTLS1MSGUSRFF.ITServices.sbc.com> Hey Nikos, just checking - are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas ; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root at n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root at n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick at intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn't look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ From: Drew Walters [mailto:drewwalters96 at gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas > Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas > wrote: Hi airship people, I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I'm getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: and giving, after that, docker ps I get: root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick at intracom-telecom.com www.intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From MM9745 at att.com Mon Jan 14 16:58:29 2019 From: MM9745 at att.com (MCEUEN, MATT) Date: Mon, 14 Jan 2019 16:58:29 +0000 Subject: [Airship-discuss] =?utf-8?q?Airship-in-a-bottle=5FDev=5FMinimal?= =?utf-8?q?=5FBedhind_Proxy=EF=BC=9AHow_to_Update_Site=3F?= In-Reply-To: <002e01d4a6fd$d60eae10$822c0a30$@as.ntt-at.co.jp> References: <008001d49298$bed0db70$3c729250$@as.ntt-at.co.jp> <7C64A75C21BB8D43BD75BB18635E4D896CF0E916@MOSTLS1MSGUSRFF.ITServices.sbc.com> <002101d498c0$020cc0c0$06264240$@as.ntt-at.co.jp> <7C64A75C21BB8D43BD75BB18635E4D896CF40361@MOSTLS1MSGUSRFF.ITServices.sbc.com> <002e01d4a6fd$d60eae10$822c0a30$@as.ntt-at.co.jp> Message-ID: <7C64A75C21BB8D43BD75BB18635E4D896CFB0B04@MOSTLS1MSGUSRFF.ITServices.sbc.com> Hi Santosh, In general, versions are updated via a site definition's versions.yaml file, e.g.: https://github.com/openstack/airship-treasuremap/blob/master/global/software/config/versions.yaml This file defines the versions/hashes of helm charts, containers, and other artifacts, and these values get substituted into the dependent helm charts via Deckhand. Please note that today, a Kubernetes upgrade still requires an out-of-band kubelet upgrade on the hosts, since the kubelet isn't containerized. Same thing for docker-engine. Upgrades of everything else are managed via versions.yaml. Thanks, Matt -----Original Message----- From: santosh.thapamager Sent: Monday, January 7, 2019 8:57 PM To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi Matt, Thanks a lot for the reply. I got some idea about updating process. However I would like to ask you about the files that needs to be updated if I wanted to update the versions of Kubernetes and its components, UCP components, OSH Components. Thanking you in advance. Best Regards, Santosh -----Original Message----- From: MCEUEN, MATT Sent: Saturday, January 05, 2019 3:45 AM To: santosh.thapamager ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Sorry for the delay, Santosh -- I'm catching up post-holiday. To supply updated documents to Shipyard, 1) First, update the documents in your git repository (e.g. something like https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap_tree_master_site_airship-2Dseaworthy&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=b1YiNeJhoTk0WMF3Genn7txrrYNJBrlLC6KcaSxdTT4&s=vH5KQ-LshRi_ymAlZHKqntGorRBE_Kuy-Do1ZlUmAqI&e=). Your repo serves as the declarative definition of what your site looks like, both for an initial deployment, and later, for an update. 2) The steps to update a site via Shipyard are very similar to the initial deployment, but with fewer steps! Lint and collect your updated documents, per: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtreasuremap.readthedocs.io_en_latest_authoring-5Fand-5Fdeployment.html-23manifest-2Dlinting-2Dand-2Dcombining-2Dlayers&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=b1YiNeJhoTk0WMF3Genn7txrrYNJBrlLC6KcaSxdTT4&s=d1FDWnfqonAAPLfHkki5NyhP8SQr2Nv1-HtKI8xPFHY&e= Update the site using Shipyard*: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtreasuremap.readthedocs.io_en_latest_authoring-5Fand-5Fdeployment.html-23deploy-2Dsite-2Dwith-2Dshipyard&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=b1YiNeJhoTk0WMF3Genn7txrrYNJBrlLC6KcaSxdTT4&s=eVMJmdn-NL02U4lVQdawyLIZKP8dS1mqFQxgDw-dAw8&e= * Note: should use the Shipyard`update_site.sh` script instead of `deploy_site.sh` in the above step. I think we should add another section to the end of that site authoring guide to describe the update process -- today the guide is geared toward new deployments. Let me know if you have any other questions! Thanks, Matt -----Original Message----- From: santosh.thapamager Sent: Thursday, December 20, 2018 5:59 PM To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi Matt, Thanks for the instruction. I have been following airship-treasuremap document. As you mentioned that we need to update the deployment manifest and supply to Shipyard and call Shipyard update_site API. *************** 1) supplying updated deployment manifests to Shipyard, including the change 2) calling Shipyard's update_site API *************** However I could not find instruction for 1) as you mentioned. I would be greatly thankful to you if I can get some instruction for updating the deployment manifests and how to supply updated manifests to Shipyard. Best Regards, Santosh Thapa Magar -----Original Message----- From: MCEUEN, MATT Sent: Thursday, December 20, 2018 4:37 AM To: santosh.thapamager ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi Santosh - in general, the right way to update an airship site is by: 1) supplying updated deployment manifests to Shipyard, including the change 2) calling Shipyard's update_site API We don't have that as part of Aiship-in-a-Bottle, because AiaB is not intended to be used for "real life" sites -- it's a demo environment. I suggest taking a look at the Treasuremap documentation [1] and example manifests [2] as a way to stand up a fully lifecycled-site. Update_site (along with the rest of the Shipyard interface) is documented here [3] for the CLI and here [4] for the API. 1: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtreasuremap.readthedocs.io_en_latest_authoring-5Fand-5Fdeployment.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=OUmBq4L34nRJCxeMURicQJuQTXp7dTZGkrSZO8DTw6s&e= 2: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dtreasuremap&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=m5RHAnvcPXkc2rESb2Awl7FBDiEojHQWs5h-9UKiwow&e= 3: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dshipyard.readthedocs.io_en_latest_CLI.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=qLoy0_3Np0PQyXcpSXmkxC5z8_Gsbt43Q1vbE0_si5M&e= 4: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dshipyard.readthedocs.io_en_latest_API.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=GzUJawD1-C5_L5DGSKHFUYIHRm6qQLIa31I93xmqs5Q&s=2ckZS-p-G_eVrOfGGk9-CBvQiq6FgW8GYabJBi1yidA&e= The latest version of Promenade provisions a Kubernetes cluster that patches for the CVE you mention. Hope that helps - let me know if you have any other questions! Matt -----Original Message----- From: santosh.thapamager Sent: Wednesday, December 12, 2018 10:03 PM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Airship-in-a-bottle_Dev_Minimal_Bedhind Proxy:How to Update Site? Hi all, I was able to install dev_minimal and deploy site behind proxy. Because of Kubernetes apiserver vulnerability (CVE-2018-1002105)I wanted to update Kubernetes version in my currently deployed site. However I could not find any documentations regarding updating deployed site. I would be thankful to all of you, if I could get any support. Best Regards, Santosh Thapa Magar _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=Kp_9DVZcWk0qumyITPRn31pyOnpfXyno0HbJJJul7Xw&s=j6f8UNYzAnQn2PEYvDhC0N3F4i2vFF8qXyVC7Arx3hQ&e= From rp2723 at att.com Tue Jan 15 15:31:52 2019 From: rp2723 at att.com (PACHECO, RODOLFO J) Date: Tue, 15 Jan 2019 15:31:52 +0000 Subject: [Airship-discuss] Reminder and Bridge change - Airship - Open Design Call Message-ID: When: Occurs every Thursday from 11:00 AM to 12:00 PM (UTC-05:00) Eastern Time (US & Canada) Where: https://attcorp.webex.com/join/rp2723 --- New Bridge !!! Join by video system i Dial rp2723 at attcorp.webex.com and enter your host PIN 02083790. You can also dial 173.243.2.68 and enter your meeting number. Join by phone 1-844-517-1415 United States Toll Free 1-618-230-6039 United States Toll Access code: 733 333 726 This week we would like to get some feedback on the Redesigned Airship Web Site (https://airship-website-dev.netlify.com/) . Please take a look and let us know what you think !! Airship Design Calls Info * Etherpad for the Airship Open Design discussion https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions * Storyboard in flight Specs https://storyboard.openstack.org/#!/project/openstack/airship-specs * Github Airship Specs https://github.com/openstack/airship-specs/tree/master/specs * Inflight/reviewing specs https://review.openstack.org/#/q/status:open+airship-specs * Treasuremap Releases https://github.com/openstack/airship-treasuremap/releases/ Regards Rodolfo Pacheco Home/Office 732 5337671 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rp2723 at att.com Tue Jan 15 15:14:08 2019 From: rp2723 at att.com (PACHECO, RODOLFO J) Date: Tue, 15 Jan 2019 15:14:08 +0000 Subject: [Airship-discuss] Reminder and Bridge change - Airship - Open Design Call Message-ID: <55079984-354D-4A52-A370-FF904965A0CA@att.com> When: Occurs every Thursday from 11:00 AM to 12:00 PM (UTC-05:00) Eastern Time (US & Canada) Where: https://attcorp.webex.com/join/rp2723 --- New Bridge !!! [cid:image001.png at 01D4ACBB.1291E4C0] This week we would like to get some feedback on the Redesigned Airship Web Site (https://airship-website-dev.netlify.com/) . Please take a look and let us know what you think !! Airship Design Calls Info * Etherpad for the Airship Open Design discussion https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions * Storyboard in flight Specs https://storyboard.openstack.org/#!/project/openstack/airship-specs * Github Airship Specs https://github.com/openstack/airship-specs/tree/master/specs * Inflight/reviewing specs https://review.openstack.org/#/q/status:open+airship-specs * Treasuremap Releases https://github.com/openstack/airship-treasuremap/releases/ Regards Rodolfo Pacheco Home/Office 732 5337671 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 50843 bytes Desc: image001.png URL: From nick at intracom-telecom.com Tue Jan 15 16:34:01 2019 From: nick at intracom-telecom.com (Nikos Karandreas) Date: Tue, 15 Jan 2019 18:34:01 +0200 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: <7C64A75C21BB8D43BD75BB18635E4D896CFB082D@MOSTLS1MSGUSRFF.ITServices.sbc.com> Message-ID: <6da849f580bb5441bff90b5ce09c6b94@iris> Hi Matt, Kaspars, I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same -unknown to me as yet- reason. I'm posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3]. (full output is attached - let me know how to share it if doesn't get through) I'm still looking at those myself but not identified something yet. Thanks Nikos [1]. http://paste.openstack.org/show/742654/ [2]. http://paste.openstack.org/show/742655/ [3]. http://paste.openstack.org/show/742660/ From: MCEUEN, MATT [mailto:MM9745 at att.com] Sent: Monday, January 14, 2019 6:48 PM To: SKELS, KASPARS ; Nikos Karandreas ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hey Nikos, just checking - are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas >; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root at n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root at n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars _____ From: Nikos Karandreas [nick at intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn't look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ From: Drew Walters [mailto:drewwalters96 at gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas > Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas > wrote: Hi airship people, I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I'm getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: and giving, after that, docker ps I get: root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard " CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick at intracom-telecom.com www .intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: aiab-1.tar.gz Type: application/octet-stream Size: 35275 bytes Desc: not available URL: From ks3019 at att.com Tue Jan 15 17:00:53 2019 From: ks3019 at att.com (SKELS, KASPARS) Date: Tue, 15 Jan 2019 17:00:53 +0000 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: <6da849f580bb5441bff90b5ce09c6b94@iris> References: <7C64A75C21BB8D43BD75BB18635E4D896CFB082D@MOSTLS1MSGUSRFF.ITServices.sbc.com>, <6da849f580bb5441bff90b5ce09c6b94@iris> Message-ID: <2ADBF0C373B7E84E944B1E06D3CDDFC91E66C925@MOKSCY3MSGUSRGI.ITServices.sbc.com> Hi Nikos, failing libvirt is the cause for this (errors from Armada is that charts (nova, etc) are timing out to deploy) libvirt-hxzln 0/1 CrashLoopBackOff 26 1h 192.168.122.101 air1 Can you also share the logs for libvirt, to get some more insights.. and more folks familiar with this should be able to help # kubectl logs -n openstack libvirt-hxzln As a quick check, I've seen libvirt failing when there is a clash of pre-installed libvirt packages in the VM. Make sure you base VM is clean/regular Ubuntu 16.04. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick at intracom-telecom.com] Sent: Tuesday, January 15, 2019 10:34 AM To: MCEUEN, MATT; airship-discuss at lists.airshipit.org; SKELS, KASPARS Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Matt, Kaspars, I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same –unknown to me as yet- reason. I’m posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3]. (full output is attached – let me know how to share it if doesn’t get through) I’m still looking at those myself but not identified something yet. Thanks Nikos [1]. http://paste.openstack.org/show/742654/ [2]. http://paste.openstack.org/show/742655/ [3]. http://paste.openstack.org/show/742660/ From: MCEUEN, MATT [mailto:MM9745 at att.com] Sent: Monday, January 14, 2019 6:48 PM To: SKELS, KASPARS ; Nikos Karandreas ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hey Nikos, just checking – are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas >; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root at n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root at n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars ________________________________ From: Nikos Karandreas [nick at intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running … step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn’t look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ From: Drew Walters [mailto:drewwalters96 at gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas > Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas > wrote: Hi airship people, I’m trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I’m getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: and giving, after that, docker ps I get: root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick at intracom-telecom.com www.intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shoaib.Nasir at windriver.com Tue Jan 15 17:59:15 2019 From: Shoaib.Nasir at windriver.com (Nasir, Shoaib) Date: Tue, 15 Jan 2019 17:59:15 +0000 Subject: [Airship-discuss] Armada Compatibility with Keystone In-Reply-To: <7C64A75C21BB8D43BD75BB18635E4D896CFB0739@MOSTLS1MSGUSRFF.ITServices.sbc.com> References: <25B6D7837B9EB04495DAD8D02D71DFF0203E9DF7@ALA-MBD.corp.ad.wrs.com> <5B9C5644A0B2414E90A7F40249CF6A9555346046@MOSTLS1MSGUSRFA.ITServices.sbc.com> <25B6D7837B9EB04495DAD8D02D71DFF0203EAEAD@ALA-MBD.corp.ad.wrs.com> <7C64A75C21BB8D43BD75BB18635E4D896CFB0739@MOSTLS1MSGUSRFF.ITServices.sbc.com> Message-ID: <25B6D7837B9EB04495DAD8D02D71DFF0203EB154@ALA-MBD.corp.ad.wrs.com> Hi Matt Sorry I was a bit late in getting to this message So far, I only added a change in the CLI to pass a user's bearertoken and didn't make any changes to the API. That can be added later if people need it. I will fix the Zuul errors and put a new patch set as soon as I do that. But in the meantime, do you have any suggestions/ feedback regarding what I have proposed ? Thanks Shoaib From: MCEUEN, MATT [mailto:MM9745 at att.com] Sent: Monday, January 14, 2019 11:44 AM To: Nasir, Shoaib; EAGAN, SEAN; airship-discuss at lists.airshipit.org Subject: RE: Armada Compatibility with Keystone Looking forward to seeing what you have in mind, Shoaib. If you'd like to discuss in tomorrow's team meeting, feel free to add the topic to the etherpad: https://etherpad.openstack.org/p/airship-meeting-2019-01-15 Thanks, Matt From: Nasir, Shoaib Sent: Thursday, January 10, 2019 9:26 AM To: EAGAN, SEAN ; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Armada Compatibility with Keystone Hi Sean After digging further into the issue, I realized that armada does not support passing end users bearer-tokens. Armada can be authenticated by keystone for incoming API requests. But in the scenario, where I want to use armada to bring up openstack-helm pods on a kubernetes cluster that's already setup to be authenticated by keystone using the k8s-keystone-auth Webhook application, armada is not compatible. Therefore, I have made some changes to the armada CLI such that a user can pass its bearer-token to tiller via armada using the CLI. I will be proposing this upstream to armada soon Thanks Shoaib Nasir From: EAGAN, SEAN [mailto:se136c at att.com] Sent: Wednesday, January 09, 2019 1:02 PM To: Nasir, Shoaib; airship-discuss at lists.airshipit.org Subject: RE: Armada Compatibility with Keystone Shoaib, It looks like that feature of k8s-keystone-auth [0] is implemented by kubernetes/client-go [1] [2]. kubernetes/client-go is used by helm so that's why it works there. However, armada is using kubernetes-client/python, and as far as I can tell this feature is not implemented there. Unless I am missing it. I am not sure what the easiest path to getting that implemented will be. [0]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-keystone-webhook-authenticator-and-authorizer.md#clientkubectl-configuration [1]: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-client-keystone-auth.md [2]: https://github.com/kubernetes/kubernetes/pull/64346 Sean Eagan From: Nasir, Shoaib > Sent: Thursday, December 13, 2018 12:35 PM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Armada Compatibility with Keystone Hi I have a kubernetes cluster that is setup with Keystone as its Authentication/Authorization Backend (via the k8s-keystone-auth Webhook pod) With that, any application such as helm now needs to pass user credentials or a keystone token when talking to the kubernetes-apiserver If I set up env OS_ variables and use any helm commands (such as helm init, or helm lists etc), everything works fine as these OS_ user credentials are authorized via the kubernetes keystone backend However, I run into problems with Armada. After bringing up the armada docker container, and logging into it to execute any armada apply commands, and using the -token argument to pass along a valid keystone token, armada still fails to access the kubernetes cluster."User "system:anonymous" suggests that the token that armada cli gets, does not pass along to the kubernetes-apiserver. I'm very new to armada and need some assistance in figuring out how to setup the armada container or any other configurations required to make it pass the keystone token to the kubernetes-apiserver. Each and every armada command fails similar to the one below. Any assistance in this regard would be greatly appreciated ? armada at c7f42c9263ad:~$ armada --token $TOKEN --debug apply 2018-12-13 18:10:13.656 72 ERROR armada.cli [-] Caught unexpected exception: kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli Traceback (most recent call last): 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli self.invoke() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 215, in invoke 2018-12-13 18:10:13.656 72 ERROR armada.cli target_manifest=self.target_manifest) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 103, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli dry_run=dry_run) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 95, in __init__ 2018-12-13 18:10:13.656 72 ERROR armada.cli self.channel = self.get_channel() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 117, in get_channel 2018-12-13 18:10:13.656 72 ERROR armada.cli tiller_ip = self._get_tiller_ip() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 163, in _get_tiller_ip 2018-12-13 18:10:13.656 72 ERROR armada.cli pod = self._get_tiller_pod() 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 141, in _get_tiller_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli CONF.tiller_pod_labels).items 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/k8s.py", line 215, in get_namespace_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli namespace, label_selector=label_selector) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod 2018-12-13 18:10:13.656 72 ERROR armada.cli (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info 2018-12-13 18:10:13.656 72 ERROR armada.cli collection_formats=collection_formats) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _return_http_data_only, collection_formats, _preload_content, _request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api 2018-12-13 18:10:13.656 72 ERROR armada.cli _request_timeout=_request_timeout) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 342, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli headers=headers) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 231, in GET 2018-12-13 18:10:13.656 72 ERROR armada.cli query_params=query_params) 2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request 2018-12-13 18:10:13.656 72 ERROR armada.cli raise ApiException(http_resp=r) 2018-12-13 18:10:13.656 72 ERROR armada.cli kubernetes.client.rest.ApiException: (403) 2018-12-13 18:10:13.656 72 ERROR armada.cli Reason: Forbidden 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}) 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli 2018-12-13 18:10:13.656 72 ERROR armada.cli -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.arnoldy at suse.com Tue Jan 15 22:57:17 2019 From: alex.arnoldy at suse.com (Alex Arnoldy) Date: Tue, 15 Jan 2019 22:57:17 +0000 Subject: [Airship-discuss] Airship in a bottle install failure... References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> Message-ID: <545AF708D41B364BA4146E47647D2D2AE4624B@prvxmb04.microfocus.com> Hi Drew, I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info. As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface. Sorry that’s not much info to go on, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 From: Alex Arnoldy Sent: Monday, January 7, 2019 3:12 AM To: 'Drew Walters' ; INI-EAI-INO-MBL Balsiger Daniel Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] Airship in a bottle install failure... Hi Drew, Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two. I’ll definitely let you know how things go. Very best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 From: Drew Walters > Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel > Cc: MATT MCEUEN >; airship-discuss at lists.airshipit.org; Alex Arnoldy > Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi Alex! Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. Regards, Drew Walters On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL > wrote: Hi there everyone Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting: I see in Postgresql-0 logs: --snip FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip and in mariadb-server-0 logs: --snip ubuntu at airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip Best Regards, D ________________________________ From: Alex Arnoldy > Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure... Hi Matt, Here is the commit I’m using: commit 221441f1bff351ab366b02d6fd7856b70b06108f Author: Bryan Strassner > Date: Mon Nov 19 17:12:10 2018 -0600 Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 From: MCEUEN, MATT > Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss at lists.airshipit.org; Alex Arnoldy >; INI-EAI-INO-MBL Balsiger Daniel > Subject: RE: Airship in a bottle install failure... Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue? Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason. Thanks, Matt _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at openstack.org Thu Jan 17 00:59:50 2019 From: james at openstack.org (James Cole) Date: Wed, 16 Jan 2019 16:59:50 -0800 Subject: [Airship-discuss] Airship Website Refresh In-Reply-To: <7C64A75C21BB8D43BD75BB18635E4D896CF40223@MOSTLS1MSGUSRFF.ITServices.sbc.com> References: <3605CB09-4575-462C-9BF9-E826A294CE20@openstack.org> <7C64A75C21BB8D43BD75BB18635E4D896CF40223@MOSTLS1MSGUSRFF.ITServices.sbc.com> Message-ID: <564B69A6-5947-4699-9AC4-1182DA2194BC@openstack.org> Hey Matt, Sorry I missed this message for so long! I fixed those links on this last round of changes. Also, I asked about your airship.io vs airshipit.org question and the summary is that everything should use airshipit.org . As far as I know, only the docs exist at a .io address (https://airship-treasuremap.readthedocs.io/en/latest for example) but the OpenStack Foundation does not own those pages. -James > On Jan 4, 2019, at 10:28 AM, MCEUEN, MATT wrote: > > Hey James, great work on this! Looks really nice – a couple items: > > This may be on your todo list already, but – looks like the dev guide domain linked from the community page doesn’t resolve (yet): docs.airship.io , git.airship.io . What’s the planned distinction between airship.io and airshipit.org , or will one be an alias for the other? > > Could we use this please for the IRC meeting link: https://wiki.openstack.org/wiki/Airship#Get_in_Touch > > I like the idea of adding the subheader blurb under “Elevate Your Infrastructure”. > > Thanks! > Matt > > > From: James Cole > Sent: Thursday, December 20, 2018 3:59 PM > To: Evgeny L > Cc: airship-discuss at lists.airshipit.org > Subject: Re: [Airship-discuss] Airship Website Refresh > > Thanks for taking a look, Evgeny! > > We currently have the Airship in a bottle link on the community page, but maybe there can be a place on the homepage for it too. > > Good catch on the blog links—will update those. > > -James > > > On Dec 19, 2018, at 12:59 PM, Evgeny L > wrote: > > Hi James, > > This looks awesome! > > Just some thoughts: it may be useful to have "Try it!" somewhere alongside "VIEW THE CODE" on the main page which would lead to "Airship in a bottle" docs, so newcomers can quickly find how to get it running. > > On a blog page there is "CONNECT" section which seems to have some broken links:https://airship-website-dev.netlify.com/blog/ > > Thanks you! > > On Wed, Dec 19, 2018 at 11:54 AM James Cole > wrote: > Greetings Airship Team! > > > I’m James Cole, a graphic designer with the OpenStack Foundation. We’ve been working on an updated look for the Airship website and wanted to get your feedback. > > Please view the updated site on Netlify: https://airship-website-dev.netlify.com/ > > A few things to note: > > The blog does not currently have any blog posts and we’d probably want to have something real published there before going live. If it isn’t possible to publish something before this launches, we can remove it from the navigation and add it back in once we have some content. > > Any guidance is welcomed if links on the Community page need to point to other locations. The Gerrit repo link does not currently link to anything useful, for instance. > > We’ve been considering adding a subhead under “Elevate Your Infrastructure” on the hero image of the homepage for clarity. One of the ideas we’re floating around is "Airship makes operating open infrastructure simple, repeatable, and resilient.” Does this sound good to you? > > Please let us know if you notice something that should change—spelling/grammar errors, factual errors, broken links, etc.—or if you have any other type of feedback. Once everyone is happy with the content and layout we’ll point the official URL (airshipit.org ) to this version of the site. > > > Thank you! > > > James Cole > Graphic Designer > OpenStack Foundation > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From drewwalters96 at gmail.com Thu Jan 17 14:56:13 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Thu, 17 Jan 2019 08:56:13 -0600 Subject: [Airship-discuss] Armada Compatibility with Keystone In-Reply-To: <25B6D7837B9EB04495DAD8D02D71DFF0203EB154@ALA-MBD.corp.ad.wrs.com> References: <25B6D7837B9EB04495DAD8D02D71DFF0203E9DF7@ALA-MBD.corp.ad.wrs.com> <5B9C5644A0B2414E90A7F40249CF6A9555346046@MOSTLS1MSGUSRFA.ITServices.sbc.com> <25B6D7837B9EB04495DAD8D02D71DFF0203EAEAD@ALA-MBD.corp.ad.wrs.com> <7C64A75C21BB8D43BD75BB18635E4D896CFB0739@MOSTLS1MSGUSRFF.ITServices.sbc.com> <25B6D7837B9EB04495DAD8D02D71DFF0203EB154@ALA-MBD.corp.ad.wrs.com> Message-ID: Hey Shoaib! I'll give your patch a look today and see if I have any feedback to offer. Regards, Drew Walters On Tue, Jan 15, 2019 at 11:59 AM Nasir, Shoaib wrote: > Hi Matt > > > > Sorry I was a bit late in getting to this message > > So far, I only added a change in the CLI to pass a user’s bearertoken and > didn’t make any changes to the API. That can be added later if people need > it. > > I will fix the Zuul errors and put a new patch set as soon as I do that. > > But in the meantime, do you have any suggestions/ feedback regarding what > I have proposed ? > > > > Thanks > > Shoaib > > > > *From:* MCEUEN, MATT [mailto:MM9745 at att.com] > *Sent:* Monday, January 14, 2019 11:44 AM > *To:* Nasir, Shoaib; EAGAN, SEAN; airship-discuss at lists.airshipit.org > *Subject:* RE: Armada Compatibility with Keystone > > > > Looking forward to seeing what you have in mind, Shoaib. If you’d like to > discuss in tomorrow’s team meeting, feel free to add the topic to the > etherpad: > > > > https://etherpad.openstack.org/p/airship-meeting-2019-01-15 > > > > Thanks, > Matt > > > > *From:* Nasir, Shoaib > *Sent:* Thursday, January 10, 2019 9:26 AM > *To:* EAGAN, SEAN ; airship-discuss at lists.airshipit.org > *Subject:* Re: [Airship-discuss] Armada Compatibility with Keystone > > > > Hi Sean > > > > After digging further into the issue, I realized that armada does not > support passing end users bearer-tokens. > > Armada can be authenticated by keystone for incoming API requests. > > > > But in the scenario, where I want to use armada to bring up openstack-helm > pods on a kubernetes cluster that’s already setup to be authenticated by > keystone using the k8s-keystone-auth Webhook application, armada is not > compatible. > > Therefore, I have made some changes to the armada CLI such that a user can > pass its bearer-token to tiller via armada using the CLI. > > I will be proposing this upstream to armada soon > > > > Thanks > > Shoaib Nasir > > > > *From:* EAGAN, SEAN [mailto:se136c at att.com ] > *Sent:* Wednesday, January 09, 2019 1:02 PM > *To:* Nasir, Shoaib; airship-discuss at lists.airshipit.org > *Subject:* RE: Armada Compatibility with Keystone > > > > > > Shoaib, > > > > It looks like that feature of k8s-keystone-auth [0] is implemented by > kubernetes/client-go [1] [2]. > > > > kubernetes/client-go is used by helm so that’s why it works there. > > > > However, armada is using kubernetes-client/python, and as far as I can > tell this feature is not implemented there. Unless I am missing it. > > I am not sure what the easiest path to getting that implemented will be. > > > > [0]: > https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-keystone-webhook-authenticator-and-authorizer.md#clientkubectl-configuration > > > [1]: > https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-client-keystone-auth.md > > > [2]: https://github.com/kubernetes/kubernetes/pull/64346 > > > > > Sean Eagan > > > > > > *From:* Nasir, Shoaib > *Sent:* Thursday, December 13, 2018 12:35 PM > *To:* airship-discuss at lists.airshipit.org > *Subject:* [Airship-discuss] Armada Compatibility with Keystone > > > > Hi > > > > I have a kubernetes cluster that is setup with Keystone as its > Authentication/Authorization Backend (via the k8s-keystone-auth Webhook pod) > > With that, any application such as helm now needs to pass user credentials > or a keystone token when talking to the kubernetes-apiserver > > > > If I set up env OS_ variables and use any helm commands (such as helm > init, or helm lists etc), everything works fine as these OS_ user > credentials are authorized via the kubernetes keystone backend > > > > However, I run into problems with Armada. After bringing up the armada > docker container, and logging into it to execute any armada apply commands, > and using the –token argument to pass along a valid keystone token, armada > still fails to access the kubernetes cluster.”User “system:anonymous” > suggests that the token that armada cli gets, does not pass along to the > kubernetes-apiserver. > > I’m very new to armada and need some assistance in figuring out how to > setup the armada container or any other configurations required to make it > pass the keystone token to the kubernetes-apiserver. > > > > Each and every armada command fails similar to the one below. Any > assistance in this regard would be greatly appreciated ? > > > > *armada at c7f42c9263ad:~$ armada --token $TOKEN --debug apply* > > 2018-12-13 18:10:13.656 72 ERROR armada.cli [-] Caught unexpected > exception: kubernetes.client.rest.ApiException: (403) > > Reason: Forbidden > > HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 > GMT', 'Content-Length': '265', 'Content-Type': 'application/json', > 'X-Content-Type-Options': 'nosniff'}) > > HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods > is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in > API group \"\" in the namespace > \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} > > 2018-12-13 18:10:13.656 72 ERROR armada.cli Traceback (most recent call > last): > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, > in safe_invoke > > 2018-12-13 18:10:13.656 72 ERROR armada.cli self.invoke() > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 215, in > invoke > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > target_manifest=self.target_manifest) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line > 103, in __init__ > > 2018-12-13 18:10:13.656 72 ERROR armada.cli dry_run=dry_run) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line > 95, in __init__ > > 2018-12-13 18:10:13.656 72 ERROR armada.cli self.channel = > self.get_channel() > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line > 117, in get_channel > > 2018-12-13 18:10:13.656 72 ERROR armada.cli tiller_ip = > self._get_tiller_ip() > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line > 163, in _get_tiller_ip > > 2018-12-13 18:10:13.656 72 ERROR armada.cli pod = > self._get_tiller_pod() > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line > 141, in _get_tiller_pod > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > CONF.tiller_pod_labels).items > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/armada/handlers/k8s.py", line 215, > in get_namespace_pod > > 2018-12-13 18:10:13.656 72 ERROR armada.cli namespace, > label_selector=label_selector) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", > line 12310, in list_namespaced_pod > > 2018-12-13 18:10:13.656 72 ERROR armada.cli (data) = > self.list_namespaced_pod_with_http_info(namespace, **kwargs) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", > line 12413, in list_namespaced_pod_with_http_info > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > collection_formats=collection_formats) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", > line 321, in call_api > > 2018-12-13 18:10:13.656 72 ERROR armada.cli _return_http_data_only, > collection_formats, _preload_content, _request_timeout) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", > line 155, in __call_api > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > _request_timeout=_request_timeout) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", > line 342, in request > > 2018-12-13 18:10:13.656 72 ERROR armada.cli headers=headers) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line > 231, in GET > > 2018-12-13 18:10:13.656 72 ERROR armada.cli query_params=query_params) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli File > "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line > 222, in request > > 2018-12-13 18:10:13.656 72 ERROR armada.cli raise > ApiException(http_resp=r) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > kubernetes.client.rest.ApiException: (403) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli Reason: Forbidden > > 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response headers: > HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': > '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': > 'nosniff'}) > > 2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response body: > {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods > is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in > API group \"\" in the namespace > \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403} > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > > 2018-12-13 18:10:13.656 72 ERROR armada.cli > > > > > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From rp2723 at att.com Thu Jan 17 16:04:43 2019 From: rp2723 at att.com (PACHECO, RODOLFO J) Date: Thu, 17 Jan 2019 16:04:43 +0000 Subject: [Airship-discuss] Airship - Open Design Call - AM EST Call Message-ID: <99088997CCAD0C4BA20008FD09434496326DF0A1@MISOUT7MSGUSRDI.ITServices.sbc.com> When: Occurs every Thursday from 11:00 AM to 12:00 PM effective 8/30/2018 until 2/28/2019. (UTC-05:00) Eastern Time (US & Canada) Where: https://attcorp.webex.com/meet/rp2723 *~*~*~*~*~*~*~*~*~* Changing recurrence to only be the 11 EST Time. Very low attendance in the EST PM meeting. We can change this when the community needs it. · Etherpad for the Airship Open Design discussion · https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions · Storyboard in flight Specs · https://storyboard.openstack.org/#!/project/openstack/airship-specs · Github Airship Specs · https://github.com/openstack/airship-specs/tree/master/specs · Inflight/reviewing specs · https://review.openstack.org/#/q/status:open+airship-specs __________________________________________ To join the Meeting: https://bluejeans.com/421706689 To join via Room System: Video Conferencing System: bjn.vc -or-199.48.152.152 Meeting ID : 421706689 To join via phone : 1) Dial: +1.408.317.9254 (BlueJeans U.S. Toll) +1.866.226.4650 (US Toll Free) (see all numbers - http://bluejeans.com/numbers) 2) Enter Conference ID : 421706689 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 4300 bytes Desc: not available URL: From rp2723 at att.com Thu Jan 17 16:40:57 2019 From: rp2723 at att.com (PACHECO, RODOLFO J) Date: Thu, 17 Jan 2019 16:40:57 +0000 Subject: [Airship-discuss] Airship - Open Design Call - AM EST Call Message-ID: <99088997CCAD0C4BA20008FD09434496326E0298@MISOUT7MSGUSRDI.ITServices.sbc.com> When: Occurs every Thursday from 11:00 AM to 12:00 PM effective 8/30/2018 until 2/28/2019. (UTC-05:00) Eastern Time (US & Canada) Where: https://attcorp.webex.com/meet/rp2723 *~*~*~*~*~*~*~*~*~* Etherpad for the Airship Open Design discussion https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions Storyboard in flight Specs https://storyboard.openstack.org/#!/project/openstack/airship-specs Github Airship Specs https://github.com/openstack/airship-specs/tree/master/specs Inflight/reviewing specs https://review.openstack.org/#/q/status:open+airship-specs __________________________________________ Join by video system i Dial rp2723 at attcorp.webex.com and enter your host PIN 02083790. You can also dial 173.243.2.68 and enter your meeting number. Join by phone 1-844-517-1415 United States Toll Free 1-618-230-6039 United States Toll Access code: 733 333 726 Host PIN: 02083790 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 3998 bytes Desc: not available URL: From nick at intracom-telecom.com Wed Jan 16 11:42:29 2019 From: nick at intracom-telecom.com (Nikos Karandreas) Date: Wed, 16 Jan 2019 13:42:29 +0200 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: <2ADBF0C373B7E84E944B1E06D3CDDFC91E66C925@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: Hi Kaspars, I got a successful installation of Airship-in-a-Bottle! I deselected "Virtual Machine host" in "Software selection" menu during Ubuntu installation (leaving only "standard system utilities" and "OpenSSH server"). I also stepped the VM from 8vCPU to 12vCPU, but I don't think that this was the crucial element. Thank you and everybody, a lot for the help and support. Nikos PS. As info piece, the log command for the failing pod returned: root at air1:~# kubectl logs -n openstack libvirt-hxzln Error from server (NotFound): pods "libvirt-hxzln" not found although the pod still appeared in the get pods command. From: SKELS, KASPARS [mailto:ks3019 at att.com] Sent: Tuesday, January 15, 2019 7:01 PM To: Nikos Karandreas ; MCEUEN, MATT ; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos, failing libvirt is the cause for this (errors from Armada is that charts (nova, etc) are timing out to deploy) libvirt-hxzln 0/1 CrashLoopBackOff 26 1h 192.168.122.101 air1 Can you also share the logs for libvirt, to get some more insights.. and more folks familiar with this should be able to help # kubectl logs -n openstack libvirt-hxzln As a quick check, I've seen libvirt failing when there is a clash of pre-installed libvirt packages in the VM. Make sure you base VM is clean/regular Ubuntu 16.04. Kindly, Kaspars _____ From: Nikos Karandreas [nick at intracom-telecom.com] Sent: Tuesday, January 15, 2019 10:34 AM To: MCEUEN, MATT; airship-discuss at lists.airshipit.org ; SKELS, KASPARS Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Matt, Kaspars, I just did a new run of AiaB install script (fresh VM, same as before) and it failed again, for the same -unknown to me as yet- reason. I'm posting the output of commands suggested by Kaspars (and a few more) in [1] and [2] and the final part of the execution log [3]. (full output is attached - let me know how to share it if doesn't get through) I'm still looking at those myself but not identified something yet. Thanks Nikos [1]. http://paste.openstack.org/show/742654/ [2]. http://paste.openstack.org/show/742655/ [3]. http://paste.openstack.org/show/742660/ From: MCEUEN, MATT [mailto:MM9745 at att.com] Sent: Monday, January 14, 2019 6:48 PM To: SKELS, KASPARS >; Nikos Karandreas >; airship-discuss at lists.airshipit.org Subject: RE: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hey Nikos, just checking - are you still seeing any issues with AiaB? Or all clear with the latest-and-greatest? Thanks, Matt From: SKELS, KASPARS Sent: Tuesday, January 8, 2019 7:19 PM To: Nikos Karandreas >; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers ***Security Advisory: This Message Originated Outside of AT&T *** Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. Hi Nikos, I did re-run latest master just now and could get a successful run in my VM environment. The armada part that fails will show logs within Armada container, have a look at pod logs root at n0:~# kubectl get pods -n ucp -o wide -a=false |grep armada Flag --show-all has been deprecated, will be removed in an upcoming release armada-api-5488cbdb99-cqzj6 1/1 Running 0 58m 10.97.26.38 n0 And then root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 The logs can be chatty, you may try root at n0:~# kubectl logs -n ucp armada-api-5488cbdb99-cqzj6 |grep ERROR Although could be other issues, one area could be image pulling timeouts, you may check any airship/openstack image in the versions.yaml deployment_files/global/v1.0demo/software/config/versions.yaml such as, and see if you are getting, see if you are getting good download speeds on them $ docker pull docker.io/openstackhelm/nova:ocata $ docker pull quay.io/quay.io/airshipit/shipyard:4dd6b484d11e86ad51da733841b9ef137421d461 Also check openstack namespace root at n0:~# kubectl get pods -n openstack -o wide -a=false See if there are any failed pods. Kindly, Kaspars _____ From: Nikos Karandreas [nick at intracom-telecom.com] Sent: Tuesday, January 08, 2019 9:43 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Drew, I tried it again today with a fresh VM (8vcpu/24G RAM/48G disk Ubuntu 16.04) with master but it fails to complete the test stack server, clearly at a much later point: 2019-01-08 14:18:05Z [test-stack-01.server]: CREATE_FAILED ResourceInError: resources.server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" which repeats a few times before exiting the script. Going up a bit step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 failed step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 failed ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 upstream_failed although just in the previous iteration: step/01D0PRG6ACSTK96ABN5STXZNQR/armada_build 11 running ... step/01D0PRG6ACSTK96ABN5STXZNQR/armada_post_apply 12 running step/01D0PRG6ACSTK96ABN5STXZNQR/armada_get_releases 19 scheduled Giving kubectl get pods -n ucp doesn't look any suspicious (to me) [1] Any idea on what/where to look for? Cheers Nikos [1]. http://paste.openstack.org/show/740645/ From: Drew Walters [mailto:drewwalters96 at gmail.com] Sent: Friday, January 04, 2019 8:46 PM To: Nikos Karandreas > Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers Hi Nikos! Do you still have access to this VM? If so, I think the best place to start is by retrieving the status of the airflow workers to verify they are operational (kubectl get pods -n ucp will show them). There have been several changes to Airship-in-a-Bottle since the time of writing; have you had the opportunity to give the latest version a try? Best, Drew Walters On Wed, Nov 28, 2018 at 8:47 AM Nikos Karandreas > wrote: Hi airship people, I'm trying to deploy airship-in-a-bottle in a fresh Ubuntu 16.04 VM (8 vCPU/24GB RAM/48GB disk) following the instructions in [1] but I'm getting the failure below (until the end in [2]) + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: and giving, after that, docker ps I get: root at airhost2:~/deploy/airship-in-a-bottle/manifests/dev_single_node# docker ps --all | grep "quay.io/airshipit/shipyard " CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f4c0747b25ec quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-web-75d98b59b5-tbtsg_ucp_677730b5-f2ff-11e8-af0b-52540085805e_0 04241489b515 quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-scheduler-5588bd948f-dlq2n_ucp_677727c7-f2ff-11e8-af0b-52540085805e_0 d433934eec1d quay.io/airshipit/shipyard at sha256:da847565f42af6b3286c95ad474db6acd2ad8dac83b6185dd4388ad530441fed "/tmp/airflow-ship..." 47 minutes ago Exited (0) 47 minutes ago k8s_airflow-shipyard-init_airflow-worker-0_ucp_68133c76-f2ff-11e8-af0b-52540085805e_0 Can you please give some insight on what might be wrong or what to look for? Cheers Nikos Nikos Karandreas Developer, OPNFV Software Development Center _____________________________________ Intracom Telecom Marinou Antipa 41, Pilea-Thessaloniki, GR 57001 t: +30 2310497353, f: +30 2310497330 nick at intracom-telecom.com www .intracom-telecom.com [1] https://github.com/openstack/airship-in-a-bottle [2] Execute deploy_site Dag... + [[ -n '' ]] + [[ -n deploy_site ]] + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions Name Action Lifecycle Execution Time Step Succ/Fail/Oth Footnotes None + echo -e 'Retrieving Action ID...\n' Retrieving Action ID... ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep -i Processing ++ awk '{print $2}' ++ grep deploy_site + action_id= + retrieve_shipyard_action_counter=0 + retrieve_shipyard_action_limit=2 + [[ 0 -le 2 ]] + [[ -n '' ]] + [[ 0 == 2 ]] + echo -e 'Unable to Retrieve Action ID!' Unable to Retrieve Action ID! + echo -e 'Retrying in 30 seconds...' Retrying in 30 seconds... + sleep 30 ++ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master get actions ++ grep deploy_site ++ grep -i Processing ++ awk '{print $2}' + action_id= + (( retrieve_shipyard_action_counter ++ )) + error 'executing deploy_site from the /site directory' + set +x Error when executing deploy_site from the /site directory. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Thu Jan 17 21:03:01 2019 From: cboylan at sapwetik.org (Clark Boylan) Date: Thu, 17 Jan 2019 13:03:01 -0800 Subject: [Airship-discuss] Infra team upgrading review.openstack.org Gerrit from 2.13.9 to 2.13.12 January 18 at about 1700UTC Message-ID: <1547758981.2237058.1637449632.37C152F2@webmail.messagingengine.com> We will be performing a minor Gerrit upgrade to version 2.13.12 tomorrow (January 18, 2019) at 1700UTC. We've tested this upgrade on our dev server, https://review-dev.openstack.org, and expect it to be a quick upgrade. Any outage shouldn't last more than 10 minutes. We will let our configuration management tooling manage the upgrade so we won't have an exact time, but will try to get it as close to 1700UTC as possible. Feel free to test out the new version on the dev server. We are happy to answer any questions you might have as well. Sorry for the short notice and the cross post, Clark From drewwalters96 at gmail.com Fri Jan 18 17:31:06 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Fri, 18 Jan 2019 11:31:06 -0600 Subject: [Airship-discuss] airship-in-a-bottle: deployment fails with exited airflow containers In-Reply-To: References: Message-ID: <812A6875-6C99-459E-B0D3-9C9FAEA04711@gmail.com> On Jan 16, 2019, at 5:42 AM, Nikos Karandreas wrote: > > Hi Kaspars, > > I got a successful installation of Airship-in-a-Bottle! > > I deselected “Virtual Machine host” in “Software selection” menu during Ubuntu installation > (leaving only “standard system utilities” and “OpenSSH server”). > > I also stepped the VM from 8vCPU to 12vCPU, but I don’t think that this was the crucial element. > > Thank you and everybody, a lot for the help and support. > Nikos > > > > PS. As info piece, the log command for the failing pod returned: > root at air1:~# kubectl logs -n openstack libvirt-hxzln > Error from server (NotFound): pods "libvirt-hxzln" not found > > although the pod still appeared in the get pods command. Nikos, Glad to hear everything is operating with good success! I assume you are unable to get the logs of the libvirt pod because the failing pod has already been cleaned up. I can’t comment on the correlation between libvirt packages on the host VM and failing libvirt pods, but maybe someone else has some insight here. If you run into any other issues, please do not hesitate to reach out! Best, Drew From drewwalters96 at gmail.com Fri Jan 18 17:37:54 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Fri, 18 Jan 2019 11:37:54 -0600 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: <545AF708D41B364BA4146E47647D2D2AE4624B@prvxmb04.microfocus.com> References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> <545AF708D41B364BA4146E47647D2D2AE4624B@prvxmb04.microfocus.com> Message-ID: <78B01097-7BB2-4519-8AB7-83BE56A6C0C6@gmail.com> On Jan 15, 2019, at 4:57 PM, Alex Arnoldy wrote: > > Hi Drew, > > I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info. > > As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface. > > Sorry that’s not much info to go on, > > --Alex. > > Alex Arnoldy > Software Defined Infrastructure Solutions Architect > SUSE > Alex.Arnoldy at SUSE.com > (P) +1 (408) 828-9797 Hi Alex, Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try. Drew From alex.arnoldy at suse.com Fri Jan 18 17:41:01 2019 From: alex.arnoldy at suse.com (Alex Arnoldy) Date: Fri, 18 Jan 2019 17:41:01 +0000 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: <78B01097-7BB2-4519-8AB7-83BE56A6C0C6@gmail.com> References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> <545AF708D41B364BA4146E47647D2D2AE4624B@prvxmb04.microfocus.com> <78B01097-7BB2-4519-8AB7-83BE56A6C0C6@gmail.com> Message-ID: <545AF708D41B364BA4146E47647D2D2AE47767@prvxmb04.microfocus.com> Hi Drew, Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot. Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 -----Original Message----- From: Drew Walters Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel Subject: Re: [Airship-discuss] Airship in a bottle install failure... On Jan 15, 2019, at 4:57 PM, Alex Arnoldy wrote: > > Hi Drew, > > I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info. > > As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface. > > Sorry that’s not much info to go on, > > --Alex. > > Alex Arnoldy > Software Defined Infrastructure Solutions Architect SUSE > Alex.Arnoldy at SUSE.com > (P) +1 (408) 828-9797 Hi Alex, Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try. Drew From paye600 at gmail.com Fri Jan 18 18:35:45 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Fri, 18 Jan 2019 19:35:45 +0100 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: <545AF708D41B364BA4146E47647D2D2AE47767@prvxmb04.microfocus.com> References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> <545AF708D41B364BA4146E47647D2D2AE4624B@prvxmb04.microfocus.com> <78B01097-7BB2-4519-8AB7-83BE56A6C0C6@gmail.com> <545AF708D41B364BA4146E47647D2D2AE47767@prvxmb04.microfocus.com> Message-ID: Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good. Regards, -- Roman Gorshunov On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy wrote: > > Hi Drew, > > Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot. > > Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out. > > Best, > > --Alex. > > Alex Arnoldy > Software Defined Infrastructure Solutions Architect > SUSE > Alex.Arnoldy at SUSE.com > (P) +1 (408) 828-9797 > > -----Original Message----- > From: Drew Walters > Sent: Friday, January 18, 2019 9:38 AM > To: Alex Arnoldy > Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel > Subject: Re: [Airship-discuss] Airship in a bottle install failure... > > On Jan 15, 2019, at 4:57 PM, Alex Arnoldy wrote: > > > > Hi Drew, > > > > I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info. > > > > As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface. > > > > Sorry that’s not much info to go on, > > > > --Alex. > > > > Alex Arnoldy > > Software Defined Infrastructure Solutions Architect SUSE > > Alex.Arnoldy at SUSE.com > > (P) +1 (408) 828-9797 > > Hi Alex, > > Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try. > > Drew > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From alex.arnoldy at suse.com Fri Jan 18 18:55:42 2019 From: alex.arnoldy at suse.com (Alex Arnoldy) Date: Fri, 18 Jan 2019 18:55:42 +0000 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> <545AF708D41B364BA4146E47647D2D2AE4624B@prvxmb04.microfocus.com> <78B01097-7BB2-4519-8AB7-83BE56A6C0C6@gmail.com> <545AF708D41B364BA4146E47647D2D2AE47767@prvxmb04.microfocus.com> Message-ID: <545AF708D41B364BA4146E47647D2D2AE47807@prvxmb04.microfocus.com> Thanks. Maybe I'll give that one a shot. It won't shed any light into the previous issues I've had but it's also possible that those problems were related to intermittent keyboard driver problems I've seen in the past. --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 -----Original Message----- From: Roman Gorshunov Sent: Friday, January 18, 2019 10:36 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good. Regards, -- Roman Gorshunov On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy wrote: > > Hi Drew, > > Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot. > > Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out. > > Best, > > --Alex. > > Alex Arnoldy > Software Defined Infrastructure Solutions Architect SUSE > Alex.Arnoldy at SUSE.com > (P) +1 (408) 828-9797 > > -----Original Message----- > From: Drew Walters > Sent: Friday, January 18, 2019 9:38 AM > To: Alex Arnoldy > Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org; > INI-EAI-INO-MBL Balsiger Daniel > Subject: Re: [Airship-discuss] Airship in a bottle install failure... > > On Jan 15, 2019, at 4:57 PM, Alex Arnoldy wrote: > > > > Hi Drew, > > > > I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info. > > > > As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface. > > > > Sorry that’s not much info to go on, > > > > --Alex. > > > > Alex Arnoldy > > Software Defined Infrastructure Solutions Architect SUSE > > Alex.Arnoldy at SUSE.com > > (P) +1 (408) 828-9797 > > Hi Alex, > > Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try. > > Drew > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From ks3019 at att.com Fri Jan 18 19:40:19 2019 From: ks3019 at att.com (SKELS, KASPARS) Date: Fri, 18 Jan 2019 19:40:19 +0000 Subject: [Airship-discuss] Airship in a bottle install failure... In-Reply-To: <545AF708D41B364BA4146E47647D2D2AE47807@prvxmb04.microfocus.com> References: <545AF708D41B364BA4146E47647D2D2AE3A29F@prvxmb04.microfocus.com> <7C64A75C21BB8D43BD75BB18635E4D896CEAD525@MOSTLS1MSGUSRFF.ITServices.sbc.com> <545AF708D41B364BA4146E47647D2D2AE3A439@prvxmb04.microfocus.com> <545AF708D41B364BA4146E47647D2D2AE4624B@prvxmb04.microfocus.com> <78B01097-7BB2-4519-8AB7-83BE56A6C0C6@gmail.com> <545AF708D41B364BA4146E47647D2D2AE47767@prvxmb04.microfocus.com> , <545AF708D41B364BA4146E47647D2D2AE47807@prvxmb04.microfocus.com> Message-ID: <2ADBF0C373B7E84E944B1E06D3CDDFC91E66F518@MOKSCY3MSGUSRGI.ITServices.sbc.com> HI Alex, yeah - the issue was pre-installed libvirt causing issues. As long as it's standard Ubuntu 1604 - that's all you need. Kindly, Kaspars ________________________________________ From: Alex Arnoldy [alex.arnoldy at suse.com] Sent: Friday, January 18, 2019 12:55 PM To: Roman Gorshunov; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure... Thanks. Maybe I'll give that one a shot. It won't shed any light into the previous issues I've had but it's also possible that those problems were related to intermittent keyboard driver problems I've seen in the past. --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy at SUSE.com (P) +1 (408) 828-9797 -----Original Message----- From: Roman Gorshunov Sent: Friday, January 18, 2019 10:36 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good. Regards, -- Roman Gorshunov On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy wrote: > > Hi Drew, > > Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot. > > Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out. > > Best, > > --Alex. > > Alex Arnoldy > Software Defined Infrastructure Solutions Architect SUSE > Alex.Arnoldy at SUSE.com > (P) +1 (408) 828-9797 > > -----Original Message----- > From: Drew Walters > Sent: Friday, January 18, 2019 9:38 AM > To: Alex Arnoldy > Cc: MATT MCEUEN ; airship-discuss at lists.airshipit.org; > INI-EAI-INO-MBL Balsiger Daniel > Subject: Re: [Airship-discuss] Airship in a bottle install failure... > > On Jan 15, 2019, at 4:57 PM, Alex Arnoldy wrote: > > > > Hi Drew, > > > > I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info. > > > > As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface. > > > > Sorry that’s not much info to go on, > > > > --Alex. > > > > Alex Arnoldy > > Software Defined Infrastructure Solutions Architect SUSE > > Alex.Arnoldy at SUSE.com > > (P) +1 (408) 828-9797 > > Hi Alex, > > Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try. > > Drew > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=1QkZWVrNKsc-yvwNebiLcq02zjfbK6RHbjuVQGIR0lU&s=y-pWlk22qanjnSJNO7kZmh2X1A7hN8sfUHBqczBl_AY&e= _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=1QkZWVrNKsc-yvwNebiLcq02zjfbK6RHbjuVQGIR0lU&s=y-pWlk22qanjnSJNO7kZmh2X1A7hN8sfUHBqczBl_AY&e= _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=1QkZWVrNKsc-yvwNebiLcq02zjfbK6RHbjuVQGIR0lU&s=y-pWlk22qanjnSJNO7kZmh2X1A7hN8sfUHBqczBl_AY&e= From fumiya.suga at as.ntt-at.co.jp Tue Jan 22 06:40:42 2019 From: fumiya.suga at as.ntt-at.co.jp (=?iso-2022-jp?B?GyRCP1wybEo4OkgbKEI=?=) Date: Tue, 22 Jan 2019 15:40:42 +0900 Subject: [Airship-discuss] dev_single_node behind proxy Message-ID: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> Hi all, I am a newbie to Airship. I tried installing the dev_single_node behind corporate proxy. I configured the environment file as mentioned in proxy setting for dev_minimal,which didn't go well. Following is the snippet of the error. ***snip**** + wait_for_kubernetes_api 3600 + set +x Tue Jan 15 15:35:02 JST 2019 Waiting 3600 seconds for API response. Unable to find image 'gcr.io/google_containers/hyperkube-amd64:v1.10.2' locally docker: Error response from daemon: Get https://gcr.io/v2/: dial tcp 108.177.97.82:443: connect: no route to host. See 'docker run --help'. ******* I would be thankful if you could answer me configuration for deploying dev_single_node behind proxy. Any help would be appreciated. Best regards, Fumiya Suga From paye600 at gmail.com Tue Jan 22 08:24:37 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Tue, 22 Jan 2019 09:24:37 +0100 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> Message-ID: Hello Fumiya, You probably forgot to export environment variables https://airship-treasuremap.readthedocs.io/en/latest/airskiff.html#proxy-configuration . If it would not help, let me know, and I will run full test with a proxy and will try for resolve the issue. Thank you for trying Airship! Best regards, -- Roman Gorshunov On Tue, Jan 22, 2019 at 7:41 AM 須賀文哉 wrote: > > Hi all, > I am a newbie to Airship. > I tried installing the dev_single_node behind corporate proxy. > I configured the environment file as mentioned in proxy setting for > dev_minimal,which didn't go well. > Following is the snippet of the error. > > ***snip**** > + wait_for_kubernetes_api 3600 > + set +x > Tue Jan 15 15:35:02 JST 2019 Waiting 3600 seconds for API response. > Unable to find image 'gcr.io/google_containers/hyperkube-amd64:v1.10.2' > locally > docker: Error response from daemon: Get https://gcr.io/v2/: dial tcp > 108.177.97.82:443: connect: no route to host. > See 'docker run --help'. > ******* > > I would be thankful if you could answer me configuration for deploying > dev_single_node behind proxy. > > Any help would be appreciated. > > Best regards, > Fumiya Suga > > > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From georg.kunz at ericsson.com Tue Jan 22 14:03:41 2019 From: georg.kunz at ericsson.com (Georg Kunz) Date: Tue, 22 Jan 2019 14:03:41 +0000 Subject: [Airship-discuss] How to enable hugepages Message-ID: Hi Airship community, I have been trying to use hugepages in my Airship deployment, but failed to get it to work in my particular development environment. Some advice would be greatly appreciated. My development environment is a multi-node Airship-in-a-bottle gate [1] to which Matt pointed me some time ago. This environment defines a "GenericVM" hardware profile [2]. According to the Drydock documentation, I can enable hugepages in a hardware profile by adding a "hugepages" attribute [3][4]. However, when doing this in the multi-gate environment as shown here [5], the deployment (using ."/gate.sh multimode_deploy") fails with an error stating "Additional properties are not allowed ('hugepages' was unexpected)" [6] When checking the HardwareProfile defined in /tmp//site_yaml/deployment_files.yaml, it is missing the hugepages attribute. This could explain why Shipyard is complaining. My questions: i) Why is Shipyard complaining about the additional "hugepages" attribute? The HardwareProfile schema in treasuremap does list the "hugepages" attribute. ii) Why is hugepages missing from the overall deployment_files.yaml? iii) How do I enable hugepages the right way? [1] https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate [2] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/site/gate-multinode/profile/hardware/generic_vm.yaml [3] https://airshipit.readthedocs.io/projects/drydock/en/latest/topology.html#hardware-profile [4] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml [5] https://hastebin.com/paboxagori.cs [6] https://hastebin.com/uzepikenol.py Thanks a lot! Best regards Georg -------------- next part -------------- An HTML attachment was scrubbed... URL: From claire at openstack.org Tue Jan 22 14:31:53 2019 From: claire at openstack.org (Claire Massey) Date: Tue, 22 Jan 2019 06:31:53 -0800 Subject: [Airship-discuss] CFP Open Until January 23, Open Infrastructure Summit in Denver In-Reply-To: References: Message-ID: <558C8984-DA72-4B1B-AC34-22A191AA747B@openstack.org> Friendly reminder - *Tomorrow, Jan 23*, is the CFP deadline for the Denver Open Infrastructure Summit (formerly the OpenStack Summit). Submit talks here: https://www.openstack.org/summit/denver-2019/. > On Dec 18, 2018, at 8:18 AM, Claire Massey wrote: > > Hi everyone, > > FYI - the CFP is now open for the first Open Infrastructure Summit (formerly the OpenStack Summit) which will be held in Denver, Colorado April 29 - May 1, 2019. > > Wednesday, *January 23* is the deadline to Submit presentations . > > The Open Infrastructure Summit is organized by OSF and designed to be a place where open source infrastructure communities can come together and collaborate in the open. Airship will have a large and prominent presence at the event so please submit talks to the CFP and plan to attend! > > SUBMIT YOUR PRESENTATION > Important info: > Based on previous Program Committee and attendee feedback, we have added / updated three Tracks: Security, Getting Started, and Open Development (previously Open Source Community). You can find the Track descriptions here . > All of the OSF pilot projects —including Airship, Kata Containers, StarlingX and Zuul — will be front and center alongside other open source communities like Ansible, Cloud Foundry, Docker, Kubernetes, and many more. > The Open Infrastructure Summit (formerly the OpenStack Summit), has evolved to recognize our diverse audience, and to signal to the market that the event is relevant for all IT infrastructure decision makers. > If you’re interested in influencing the Summit content, apply to be a Programming Committee member *, where you can also find a full list of time requirements and expectations. Nominations will close on January 4, 2019. The content submission process for the Forum and Project Teams Gathering will be managed separately in the upcoming months. > > *OSF Staff will serve as the Programming Committee for the Getting Started Track. > > Denver Summit registration and sponsor sales are currently open. Learn more and email summit at openstack.org with any questions. > > Please email speakersupport at openstack.org with any questions or feedback. > > Thanks, > Claire -------------- next part -------------- An HTML attachment was scrubbed... URL: From ks3019 at att.com Tue Jan 22 16:59:08 2019 From: ks3019 at att.com (SKELS, KASPARS) Date: Tue, 22 Jan 2019 16:59:08 +0000 Subject: [Airship-discuss] How to enable hugepages In-Reply-To: References: Message-ID: <2ADBF0C373B7E84E944B1E06D3CDDFC91E6723E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Hi Georg, our main deployment reference treasuremap (https://github.com/openstack/airship-treasuremap) doesn't have right now good samples for huge pages but you can find a few in the test section of Drydock.. - For hardware profile https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L427 - And there are also kernel related parameters available to set https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L366 Kindly, Kaspars From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Tuesday, January 22, 2019 8:04 AM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] How to enable hugepages Hi Airship community, I have been trying to use hugepages in my Airship deployment, but failed to get it to work in my particular development environment. Some advice would be greatly appreciated. My development environment is a multi-node Airship-in-a-bottle gate [1] to which Matt pointed me some time ago. This environment defines a "GenericVM" hardware profile [2]. According to the Drydock documentation, I can enable hugepages in a hardware profile by adding a "hugepages" attribute [3][4]. However, when doing this in the multi-gate environment as shown here [5], the deployment (using ."/gate.sh multimode_deploy") fails with an error stating "Additional properties are not allowed ('hugepages' was unexpected)" [6] When checking the HardwareProfile defined in /tmp//site_yaml/deployment_files.yaml, it is missing the hugepages attribute. This could explain why Shipyard is complaining. My questions: i) Why is Shipyard complaining about the additional "hugepages" attribute? The HardwareProfile schema in treasuremap does list the "hugepages" attribute. ii) Why is hugepages missing from the overall deployment_files.yaml? iii) How do I enable hugepages the right way? [1] https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate [2] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/site/gate-multinode/profile/hardware/generic_vm.yaml [3] https://airshipit.readthedocs.io/projects/drydock/en/latest/topology.html#hardware-profile [4] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml [5] https://hastebin.com/paboxagori.cs [6] https://hastebin.com/uzepikenol.py Thanks a lot! Best regards Georg -------------- next part -------------- An HTML attachment was scrubbed... URL: From georg.kunz at ericsson.com Tue Jan 22 17:09:53 2019 From: georg.kunz at ericsson.com (Georg Kunz) Date: Tue, 22 Jan 2019 17:09:53 +0000 Subject: [Airship-discuss] How to enable hugepages In-Reply-To: <2ADBF0C373B7E84E944B1E06D3CDDFC91E6723E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E6723E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: Hi Kaspars, Thank you for the pointers. In the meantime I also noticed that the HardwareProfile schema in AIAB [1] does not contain the hugepages attributes (yet) in contrast to the schema in treasuremap [2]. This likely answers my questions i) and ii) from below. I'll take a look at the files you mentioned and extend the HardwareProfile schema in AIAB for testing purposes. [1] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/global/v1.0demo/schemas/drydock/HardwareProfile/v1.yaml [2] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml Best regards Georg From: SKELS, KASPARS Sent: Tuesday, January 22, 2019 5:59 PM To: Georg Kunz Cc: airship-discuss at lists.airshipit.org Subject: RE: How to enable hugepages Hi Georg, our main deployment reference treasuremap (https://github.com/openstack/airship-treasuremap) doesn't have right now good samples for huge pages but you can find a few in the test section of Drydock.. * For hardware profile https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L427 * And there are also kernel related parameters available to set https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L366 Kindly, Kaspars From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Tuesday, January 22, 2019 8:04 AM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] How to enable hugepages Hi Airship community, I have been trying to use hugepages in my Airship deployment, but failed to get it to work in my particular development environment. Some advice would be greatly appreciated. My development environment is a multi-node Airship-in-a-bottle gate [1] to which Matt pointed me some time ago. This environment defines a "GenericVM" hardware profile [2]. According to the Drydock documentation, I can enable hugepages in a hardware profile by adding a "hugepages" attribute [3][4]. However, when doing this in the multi-gate environment as shown here [5], the deployment (using ."/gate.sh multimode_deploy") fails with an error stating "Additional properties are not allowed ('hugepages' was unexpected)" [6] When checking the HardwareProfile defined in /tmp//site_yaml/deployment_files.yaml, it is missing the hugepages attribute. This could explain why Shipyard is complaining. My questions: i) Why is Shipyard complaining about the additional "hugepages" attribute? The HardwareProfile schema in treasuremap does list the "hugepages" attribute. ii) Why is hugepages missing from the overall deployment_files.yaml? iii) How do I enable hugepages the right way? [1] https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate [2] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/site/gate-multinode/profile/hardware/generic_vm.yaml [3] https://airshipit.readthedocs.io/projects/drydock/en/latest/topology.html#hardware-profile [4] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml [5] https://hastebin.com/paboxagori.cs [6] https://hastebin.com/uzepikenol.py Thanks a lot! Best regards Georg -------------- next part -------------- An HTML attachment was scrubbed... URL: From jb2592 at att.com Tue Jan 22 19:11:50 2019 From: jb2592 at att.com (BARESICH, JIM) Date: Tue, 22 Jan 2019 19:11:50 +0000 Subject: [Airship-discuss] AIAB behind a proxy Message-ID: Will AIAB install when behind a proxy? I’ve set the proxy env variables (http_proxy and https_proxy) but the install fails with: docker: error pulling image configuration: Get https://storage.googleapis.com/artifacts.google-containers.appspot.com/containers/images/sha256:1c5cc31eb552c29e0c71c295b4abcf5264000c2e58fa5ce0ecab9ad6f95d181a: dial tcp [2607:f8b0:4000:80f::2010]:443: i/o timeout. See 'docker run --help'. .Unable to find image 'gcr.io/google_containers/hyperkube-amd64:v1.10.2' locally + error 'running genesis' + set +x Error when running genesis. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. ​​​​​ Jim Baresich Infrastructure Technology Planning and Engineering Mobile - 7329273072 Skype - 9083261982 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paye600 at gmail.com Tue Jan 22 19:26:45 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Tue, 22 Jan 2019 20:26:45 +0100 Subject: [Airship-discuss] AIAB behind a proxy In-Reply-To: References: Message-ID: Hello Jim, It should. Let me retry tomorrow morning, and I hope to get the issue fixed. Best regards, — Roman Gorshunov > On 22 Jan 2019, at 20:11, BARESICH, JIM wrote: > > Will AIAB install when behind a proxy? > > I’ve set the proxy env variables (http_proxy and https_proxy) but the install fails with: > > docker: error pulling image configuration: Get https://storage.googleapis.com/artifacts.google-containers.appspot.com/containers/images/sha256:1c5cc31eb552c29e0c71c295b4abcf5264000c2e58fa5ce0ecab9ad6f95d181a: dial tcp [2607:f8b0:4000:80f::2010]:443: i/o timeout. > See 'docker run --help'. > .Unable to find image 'gcr.io/google_containers/hyperkube-amd64:v1.10.2 ' locally > > + error 'running genesis' > + set +x > Error when running genesis. > + exit 1 > + clean > + set +x > To remove files generated during this script's execution, delete /root/deploy. > This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. > > ​​​​​ > Jim Baresich > Infrastructure Technology Planning and Engineering > Mobile - 7329273072 > Skype - 9083261982 > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From fumiya.suga at as.ntt-at.co.jp Wed Jan 23 06:07:58 2019 From: fumiya.suga at as.ntt-at.co.jp (=?utf-8?B?6aCI6LOA5paH5ZOJ?=) Date: Wed, 23 Jan 2019 15:07:58 +0900 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> Message-ID: <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> Hello Roman, Thanks for the advice. I tried exporting proxy environment variables as per your advice, However my airship-in-a-bottle installation ended on same error as mentioned before. Am I missing something?Does airship-in-a-bottle installation works behind the proxy? Any help would be appreciated. Best regards, Fumiya Suga -----Original Message----- From: Roman Gorshunov Sent: Tuesday, January 22, 2019 5:25 PM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] dev_single_node behind proxy Hello Fumiya, You probably forgot to export environment variables https://airship-treasuremap.readthedocs.io/en/latest/airskiff.html#proxy-configuration . If it would not help, let me know, and I will run full test with a proxy and will try for resolve the issue. Thank you for trying Airship! Best regards, -- Roman Gorshunov On Tue, Jan 22, 2019 at 7:41 AM 須賀文哉 wrote: > > Hi all, > I am a newbie to Airship. > I tried installing the dev_single_node behind corporate proxy. > I configured the environment file as mentioned in proxy setting for > dev_minimal,which didn't go well. > Following is the snippet of the error. > > ***snip**** > + wait_for_kubernetes_api 3600 > + set +x > Tue Jan 15 15:35:02 JST 2019 Waiting 3600 seconds for API response. > Unable to find image 'gcr.io/google_containers/hyperkube-amd64:v1.10.2' > locally > docker: Error response from daemon: Get https://gcr.io/v2/: dial tcp > 108.177.97.82:443: connect: no route to host. > See 'docker run --help'. > ******* > > I would be thankful if you could answer me configuration for deploying > dev_single_node behind proxy. > > Any help would be appreciated. > > Best regards, > Fumiya Suga > > > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From ks3019 at att.com Wed Jan 23 15:31:44 2019 From: ks3019 at att.com (SKELS, KASPARS) Date: Wed, 23 Jan 2019 15:31:44 +0000 Subject: [Airship-discuss] How to enable hugepages In-Reply-To: References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E6723E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: <2ADBF0C373B7E84E944B1E06D3CDDFC91E672CAD@MOKSCY3MSGUSRGI.ITServices.sbc.com> Hi, yes, I have create a PS for airship-in-a-bottle to align the schemas https://review.openstack.org/#/c/632610/ /Kaspars From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Tuesday, January 22, 2019 11:10 AM To: SKELS, KASPARS Cc: airship-discuss at lists.airshipit.org Subject: RE: How to enable hugepages Hi Kaspars, Thank you for the pointers. In the meantime I also noticed that the HardwareProfile schema in AIAB [1] does not contain the hugepages attributes (yet) in contrast to the schema in treasuremap [2]. This likely answers my questions i) and ii) from below. I'll take a look at the files you mentioned and extend the HardwareProfile schema in AIAB for testing purposes. [1] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/global/v1.0demo/schemas/drydock/HardwareProfile/v1.yaml [2] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml Best regards Georg From: SKELS, KASPARS > Sent: Tuesday, January 22, 2019 5:59 PM To: Georg Kunz > Cc: airship-discuss at lists.airshipit.org Subject: RE: How to enable hugepages Hi Georg, our main deployment reference treasuremap (https://github.com/openstack/airship-treasuremap) doesn't have right now good samples for huge pages but you can find a few in the test section of Drydock.. - For hardware profile https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L427 - And there are also kernel related parameters available to set https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L366 Kindly, Kaspars From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Tuesday, January 22, 2019 8:04 AM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] How to enable hugepages Hi Airship community, I have been trying to use hugepages in my Airship deployment, but failed to get it to work in my particular development environment. Some advice would be greatly appreciated. My development environment is a multi-node Airship-in-a-bottle gate [1] to which Matt pointed me some time ago. This environment defines a "GenericVM" hardware profile [2]. According to the Drydock documentation, I can enable hugepages in a hardware profile by adding a "hugepages" attribute [3][4]. However, when doing this in the multi-gate environment as shown here [5], the deployment (using ."/gate.sh multimode_deploy") fails with an error stating "Additional properties are not allowed ('hugepages' was unexpected)" [6] When checking the HardwareProfile defined in /tmp//site_yaml/deployment_files.yaml, it is missing the hugepages attribute. This could explain why Shipyard is complaining. My questions: i) Why is Shipyard complaining about the additional "hugepages" attribute? The HardwareProfile schema in treasuremap does list the "hugepages" attribute. ii) Why is hugepages missing from the overall deployment_files.yaml? iii) How do I enable hugepages the right way? [1] https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate [2] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/site/gate-multinode/profile/hardware/generic_vm.yaml [3] https://airshipit.readthedocs.io/projects/drydock/en/latest/topology.html#hardware-profile [4] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml [5] https://hastebin.com/paboxagori.cs [6] https://hastebin.com/uzepikenol.py Thanks a lot! Best regards Georg -------------- next part -------------- An HTML attachment was scrubbed... URL: From georg.kunz at ericsson.com Wed Jan 23 15:38:53 2019 From: georg.kunz at ericsson.com (Georg Kunz) Date: Wed, 23 Jan 2019 15:38:53 +0000 Subject: [Airship-discuss] How to enable hugepages In-Reply-To: <2ADBF0C373B7E84E944B1E06D3CDDFC91E672CAD@MOKSCY3MSGUSRGI.ITServices.sbc.com> References: <2ADBF0C373B7E84E944B1E06D3CDDFC91E6723E9@MOKSCY3MSGUSRGI.ITServices.sbc.com> <2ADBF0C373B7E84E944B1E06D3CDDFC91E672CAD@MOKSCY3MSGUSRGI.ITServices.sbc.com> Message-ID: Hi Kaspars, Thanks! You beat me it. 😊 Best regards Georg From: SKELS, KASPARS Sent: Wednesday, January 23, 2019 4:32 PM To: Georg Kunz Cc: airship-discuss at lists.airshipit.org Subject: RE: How to enable hugepages Hi, yes, I have create a PS for airship-in-a-bottle to align the schemas https://review.openstack.org/#/c/632610/ /Kaspars From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Tuesday, January 22, 2019 11:10 AM To: SKELS, KASPARS > Cc: airship-discuss at lists.airshipit.org Subject: RE: How to enable hugepages Hi Kaspars, Thank you for the pointers. In the meantime I also noticed that the HardwareProfile schema in AIAB [1] does not contain the hugepages attributes (yet) in contrast to the schema in treasuremap [2]. This likely answers my questions i) and ii) from below. I’ll take a look at the files you mentioned and extend the HardwareProfile schema in AIAB for testing purposes. [1] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/global/v1.0demo/schemas/drydock/HardwareProfile/v1.yaml [2] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml Best regards Georg From: SKELS, KASPARS > Sent: Tuesday, January 22, 2019 5:59 PM To: Georg Kunz > Cc: airship-discuss at lists.airshipit.org Subject: RE: How to enable hugepages Hi Georg, our main deployment reference treasuremap (https://github.com/openstack/airship-treasuremap) doesn’t have right now good samples for huge pages but you can find a few in the test section of Drydock.. * For hardware profile https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L427 * And there are also kernel related parameters available to set https://github.com/openstack/airship-drydock/blob/master/python/tests/yaml_samples/deckhand_fullsite.yaml#L366 Kindly, Kaspars From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Tuesday, January 22, 2019 8:04 AM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] How to enable hugepages Hi Airship community, I have been trying to use hugepages in my Airship deployment, but failed to get it to work in my particular development environment. Some advice would be greatly appreciated. My development environment is a multi-node Airship-in-a-bottle gate [1] to which Matt pointed me some time ago. This environment defines a “GenericVM” hardware profile [2]. According to the Drydock documentation, I can enable hugepages in a hardware profile by adding a “hugepages” attribute [3][4]. However, when doing this in the multi-gate environment as shown here [5], the deployment (using .”/gate.sh multimode_deploy”) fails with an error stating “Additional properties are not allowed ('hugepages' was unexpected)” [6] When checking the HardwareProfile defined in /tmp//site_yaml/deployment_files.yaml, it is missing the hugepages attribute. This could explain why Shipyard is complaining. My questions: i) Why is Shipyard complaining about the additional “hugepages” attribute? The HardwareProfile schema in treasuremap does list the “hugepages” attribute. ii) Why is hugepages missing from the overall deployment_files.yaml? iii) How do I enable hugepages the right way? [1] https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate [2] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/site/gate-multinode/profile/hardware/generic_vm.yaml [3] https://airshipit.readthedocs.io/projects/drydock/en/latest/topology.html#hardware-profile [4] https://github.com/openstack/airship-treasuremap/blob/master/global/schemas/drydock/HardwareProfile/v1.yaml [5] https://hastebin.com/paboxagori.cs [6] https://hastebin.com/uzepikenol.py Thanks a lot! Best regards Georg -------------- next part -------------- An HTML attachment was scrubbed... URL: From rp2723 at att.com Wed Jan 23 23:26:12 2019 From: rp2723 at att.com (PACHECO, RODOLFO J) Date: Wed, 23 Jan 2019 23:26:12 +0000 Subject: [Airship-discuss] Canceled: Airship - Open Design Call - AM EST Call Message-ID: <99088997CCAD0C4BA20008FD09434496326EF267@MISOUT7MSGUSRDI.ITServices.sbc.com> Need to CANCEL tomorrow (Jan 24, 2019) instance of the Design Call. Etherpad for the Airship Open Design discussion https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions Storyboard in flight Specs https://storyboard.openstack.org/#!/project/openstack/airship-specs Github Airship Specs https://github.com/openstack/airship-specs/tree/master/specs Inflight/reviewing specs https://review.openstack.org/#/q/status:open+airship-specs __________________________________________ Join by video system i Dial rp2723 at attcorp.webex.com and enter your host PIN 02083790. You can also dial 173.243.2.68 and enter your meeting number. Join by phone 1-844-517-1415 United States Toll Free 1-618-230-6039 United States Toll Access code: 733 333 726 Host PIN: 02083790 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 3106 bytes Desc: not available URL: From drewwalters96 at gmail.com Thu Jan 24 03:23:04 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Wed, 23 Jan 2019 21:23:04 -0600 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> Message-ID: > Hello Roman, > > Thanks for the advice. > I tried exporting proxy environment variables as per your advice, > However my airship-in-a-bottle installation ended on same error as mentioned before. > Am I missing something?Does airship-in-a-bottle installation works behind the proxy? > > Any help would be appreciated. > > Best regards, > Fumiya Suga Hi Fumiya, May you specify which Airship-in-a-Bottle setup (e.g. dev minimal) you are using? Some of them are not yet compatible with proxy servers. Thanks! Drew Walters From drewwalters96 at gmail.com Thu Jan 24 03:26:07 2019 From: drewwalters96 at gmail.com (Drew Walters) Date: Wed, 23 Jan 2019 21:26:07 -0600 Subject: [Airship-discuss] AIAB behind a proxy In-Reply-To: References: Message-ID: <8A421CD0-42B9-4289-8BC2-8E9E0AA3CC84@gmail.com> > On Jan 22, 2019, at 1:26 PM, Roman Gorshunov wrote: > > Hello Jim, > > It should. Let me retry tomorrow morning, and I hope to get the issue fixed. > > Best regards, > — > Roman Gorshunov Roman (and Jim), Any update on the outcome of your retry? Additionally, it’s helpful if you can specify which Airship-in-a-Bottle configuration you are using, as not all are compatible behind proxy servers. Thanks! Drew Walters From fumiya.suga at as.ntt-at.co.jp Thu Jan 24 04:08:20 2019 From: fumiya.suga at as.ntt-at.co.jp (=?iso-2022-jp?B?GyRCP1wybEo4OkgbKEI=?=) Date: Thu, 24 Jan 2019 13:08:20 +0900 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> Message-ID: <001101d4b39a$719e1cf0$54da56d0$@as.ntt-at.co.jp> Hi Drew, I have been trying to install airship-in-a-bottle behind proxy. I am trying to run airship-in-a-bottle.sh script located in ../manifest/dev_single_node/ behind proxy. Best Regards, Fumiya Suga -----Original Message----- From: Drew Walters Sent: Thursday, January 24, 2019 12:23 PM To: 須賀文哉 Cc: Roman Gorshunov ; airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] dev_single_node behind proxy > Hello Roman, > > Thanks for the advice. > I tried exporting proxy environment variables as per your advice, > However my airship-in-a-bottle installation ended on same error as mentioned before. > Am I missing something?Does airship-in-a-bottle installation works behind the proxy? > > Any help would be appreciated. > > Best regards, > Fumiya Suga Hi Fumiya, May you specify which Airship-in-a-Bottle setup (e.g. dev minimal) you are using? Some of them are not yet compatible with proxy servers. Thanks! Drew Walters From jb2592 at att.com Thu Jan 24 14:44:29 2019 From: jb2592 at att.com (BARESICH, JIM) Date: Thu, 24 Jan 2019 14:44:29 +0000 Subject: [Airship-discuss] AIAB behind a proxy In-Reply-To: <8A421CD0-42B9-4289-8BC2-8E9E0AA3CC84@gmail.com> References: <8A421CD0-42B9-4289-8BC2-8E9E0AA3CC84@gmail.com> Message-ID: Drew, I'm attempting the install on a ub 16.04 KVM VM: https://github.com/openstack/airship-in-a-bottle Which installs are not compatible behind a proxy? ​​​​​ Jim Baresich -----Original Message----- From: Drew Walters Sent: Wednesday, January 23, 2019 10:26 PM To: Roman Gorshunov ; BARESICH, JIM Cc: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] AIAB behind a proxy > On Jan 22, 2019, at 1:26 PM, Roman Gorshunov wrote: > > Hello Jim, > > It should. Let me retry tomorrow morning, and I hope to get the issue fixed. > > Best regards, > — > Roman Gorshunov Roman (and Jim), Any update on the outcome of your retry? Additionally, it’s helpful if you can specify which Airship-in-a-Bottle configuration you are using, as not all are compatible behind proxy servers. Thanks! Drew Walters From georg.kunz at ericsson.com Thu Jan 24 15:13:04 2019 From: georg.kunz at ericsson.com (Georg Kunz) Date: Thu, 24 Jan 2019 15:13:04 +0000 Subject: [Airship-discuss] Divingbell in AIAB Message-ID: Hi again, I'll try to deploy divingbell in an AIAB multigate environment. By default divingbell is disabled in AIAB [1], but I'd like to use it to mount hugepages in the local filesystem. After enabling (uncommenting ) divingbell in the manifest [1] and re-deploying the system, divingbell pods get deployed, but they get stuck in state "ContainerCreating". Digging deeper, the pod isn't able to mount one configmap: Warning FailedMount 1m (x470 over 15h) kubelet, n0 MountVolume.SetUp failed for volume "divingbell-mounts" : configmaps "divingbell-mounts" not found Checking the configmaps, there are in fact multiple divingbell configmaps, however the names are slightly different: divingbell-ethtool-default divingbell-mounts-default divingbell-sysctl-default divingbell-uamlite-default If I understand the divingbell chart templates correctly, the name "divingbell-mounts" is hardcoded there for the configmap [2][3]. Can somebody help me understand where the "-default" postfix comes from and how to resolve it? [1] https://github.com/openstack/airship-in-a-bottle/blob/master/deployment_files/global/v1.0demo/software/manifests/full-site.yaml#L26 [2] https://github.com/openstack/airship-divingbell/blob/master/divingbell/templates/daemonset-mounts.yaml#L59 Thanks a lot! Best regards Georg From paye600 at gmail.com Thu Jan 24 15:13:10 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Thu, 24 Jan 2019 16:13:10 +0100 Subject: [Airship-discuss] AIAB behind a proxy In-Reply-To: References: <8A421CD0-42B9-4289-8BC2-8E9E0AA3CC84@gmail.com> Message-ID: Jim, This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_node is not set up for proxy at the moment, however there is a patch which possibly solves it: https://review.openstack.org/#/c/615387/ (it has it's own issues, though). This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_node is set up for proxy, I'm running a test on it now. Note the README file and instructions in it. This one https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate is set up for running behind the proxy (I did not try it yet). There is also README with some info. -- Roman Gorshunov On Thu, Jan 24, 2019 at 3:44 PM BARESICH, JIM wrote: > > Drew, > > I'm attempting the install on a ub 16.04 KVM VM: https://github.com/openstack/airship-in-a-bottle > Which installs are not compatible behind a proxy? > > > Jim Baresich > > -----Original Message----- > From: Drew Walters > Sent: Wednesday, January 23, 2019 10:26 PM > To: Roman Gorshunov ; BARESICH, JIM > Cc: airship-discuss at lists.airshipit.org > Subject: Re: [Airship-discuss] AIAB behind a proxy > > > On Jan 22, 2019, at 1:26 PM, Roman Gorshunov wrote: > > > > Hello Jim, > > > > It should. Let me retry tomorrow morning, and I hope to get the issue fixed. > > > > Best regards, > > — > > Roman Gorshunov > > Roman (and Jim), > > Any update on the outcome of your retry? Additionally, it’s helpful if you can specify which Airship-in-a-Bottle configuration you are using, as not all are compatible behind proxy servers. > > Thanks! > > Drew Walters From paye600 at gmail.com Thu Jan 24 15:15:51 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Thu, 24 Jan 2019 16:15:51 +0100 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: <001101d4b39a$719e1cf0$54da56d0$@as.ntt-at.co.jp> References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> <001101d4b39a$719e1cf0$54da56d0$@as.ntt-at.co.jp> Message-ID: Hello Fumiya, This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis not set up for proxy at the moment, however there is a patch which possibly solves it: https://review.openstack.org/#/c/615387/ (it hasit's own issues, though). This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis set up for proxy, I'm running a test on it now. Note the README file and instructions in it. This one https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gateis set up for running behind the proxy (I did not try it yet). There is also README with some info. -- Roman Gorshunov On Thu, Jan 24, 2019 at 5:08 AM 須賀文哉 wrote: > > Hi Drew, > > I have been trying to install airship-in-a-bottle behind proxy. > I am trying to run airship-in-a-bottle.sh script located in > ../manifest/dev_single_node/ behind proxy. > > Best Regards, > Fumiya Suga > > -----Original Message----- > From: Drew Walters > Sent: Thursday, January 24, 2019 12:23 PM > To: 須賀文哉 > Cc: Roman Gorshunov ; airship-discuss at lists.airshipit.org > Subject: Re: [Airship-discuss] dev_single_node behind proxy > > > Hello Roman, > > > > Thanks for the advice. > > I tried exporting proxy environment variables as per your advice, > > However my airship-in-a-bottle installation ended on same error as > mentioned before. > > Am I missing something?Does airship-in-a-bottle installation works behind > the proxy? > > > > Any help would be appreciated. > > > > Best regards, > > Fumiya Suga > > > Hi Fumiya, > > May you specify which Airship-in-a-Bottle setup (e.g. dev minimal) you are > using? Some of them are not yet compatible with proxy servers. > > Thanks! > > Drew Walters > > > From ca846m at att.com Thu Jan 24 21:43:15 2019 From: ca846m at att.com (ANDERSON, CRAIG) Date: Thu, 24 Jan 2019 21:43:15 +0000 Subject: [Airship-discuss] Divingbell in AIAB In-Reply-To: References: Message-ID: If you're running Ubuntu 16.04 or later, then Ubuntu will do this already by default, e.g.: https://www.apt-browse.org/browse/ubuntu/xenial/main/i386/systemd/229-4ubuntu4/file/lib/systemd/system/dev-hugepages.mount or check with: "sudo systemctl status dev-hugepages.mount" Regarding the error, this appears to be happening because the version of helm toolkit we have specified in versions.yaml is wrong for this version of divingbell. We need to either be using a newer version of divingbell, or an older version of helm toolkit to be compatible. I'll see about getting the versions aligned upstream, if not also re-enabling divingbell in AIAB. Thanks, -----Original Message----- From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Thursday, January 24, 2019 7:13 AM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Divingbell in AIAB Hi again, I'll try to deploy divingbell in an AIAB multigate environment. By default divingbell is disabled in AIAB [1], but I'd like to use it to mount hugepages in the local filesystem. After enabling (uncommenting ) divingbell in the manifest [1] and re-deploying the system, divingbell pods get deployed, but they get stuck in state "ContainerCreating". Digging deeper, the pod isn't able to mount one configmap: Warning FailedMount 1m (x470 over 15h) kubelet, n0 MountVolume.SetUp failed for volume "divingbell-mounts" : configmaps "divingbell-mounts" not found Checking the configmaps, there are in fact multiple divingbell configmaps, however the names are slightly different: divingbell-ethtool-default divingbell-mounts-default divingbell-sysctl-default divingbell-uamlite-default If I understand the divingbell chart templates correctly, the name "divingbell-mounts" is hardcoded there for the configmap [2][3]. Can somebody help me understand where the "-default" postfix comes from and how to resolve it? [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle_blob_master_deployment-5Ffiles_global_v1.0demo_software_manifests_full-2Dsite.yaml-23L26&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aimn2OylFgog_5_aS85wtQ&m=pgFJVPOeqLQoFm6_D-4rZS96wuvpXr-JbgM1VAuqqrY&s=okijvpj9PzyzWcl3plIejYr_Y9h3uBKUY_BxtIruGFw&e= [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Ddivingbell_blob_master_divingbell_templates_daemonset-2Dmounts.yaml-23L59&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aimn2OylFgog_5_aS85wtQ&m=pgFJVPOeqLQoFm6_D-4rZS96wuvpXr-JbgM1VAuqqrY&s=_ob8nMUQreMHsw-ziq-N1WLq_7dyVJdYno9A8Muh5Qg&e= Thanks a lot! Best regards Georg _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aimn2OylFgog_5_aS85wtQ&m=pgFJVPOeqLQoFm6_D-4rZS96wuvpXr-JbgM1VAuqqrY&s=vVmA5mmFZM3gCjp_cOLfsqxY3m352xmPxI7vRl0TLhE&e= From georg.kunz at ericsson.com Thu Jan 24 22:37:26 2019 From: georg.kunz at ericsson.com (Georg Kunz) Date: Thu, 24 Jan 2019 22:37:26 +0000 Subject: [Airship-discuss] Divingbell in AIAB In-Reply-To: References: Message-ID: Hi Craig, Thanks for the hint regarding the hugepages. I never noticed that Ubuntu does this automatically because we typically control this explicitly in our deployments. Anyway, in the meantime I also came across the versioning mismatch of divingbell and htk. I just bumped both versions to HEAD for testing purposes and it seems to have resolved the problem. Thanks a lot! Best regards Georg -----Original Message----- From: ANDERSON, CRAIG Sent: Thursday, January 24, 2019 10:43 PM To: Georg Kunz ; airship-discuss at lists.airshipit.org Subject: RE: Divingbell in AIAB If you're running Ubuntu 16.04 or later, then Ubuntu will do this already by default, e.g.: https://www.apt-browse.org/browse/ubuntu/xenial/main/i386/systemd/229-4ubuntu4/file/lib/systemd/system/dev-hugepages.mount or check with: "sudo systemctl status dev-hugepages.mount" Regarding the error, this appears to be happening because the version of helm toolkit we have specified in versions.yaml is wrong for this version of divingbell. We need to either be using a newer version of divingbell, or an older version of helm toolkit to be compatible. I'll see about getting the versions aligned upstream, if not also re-enabling divingbell in AIAB. Thanks, -----Original Message----- From: Georg Kunz [mailto:georg.kunz at ericsson.com] Sent: Thursday, January 24, 2019 7:13 AM To: airship-discuss at lists.airshipit.org Subject: [Airship-discuss] Divingbell in AIAB Hi again, I'll try to deploy divingbell in an AIAB multigate environment. By default divingbell is disabled in AIAB [1], but I'd like to use it to mount hugepages in the local filesystem. After enabling (uncommenting ) divingbell in the manifest [1] and re-deploying the system, divingbell pods get deployed, but they get stuck in state "ContainerCreating". Digging deeper, the pod isn't able to mount one configmap: Warning FailedMount 1m (x470 over 15h) kubelet, n0 MountVolume.SetUp failed for volume "divingbell-mounts" : configmaps "divingbell-mounts" not found Checking the configmaps, there are in fact multiple divingbell configmaps, however the names are slightly different: divingbell-ethtool-default divingbell-mounts-default divingbell-sysctl-default divingbell-uamlite-default If I understand the divingbell chart templates correctly, the name "divingbell-mounts" is hardcoded there for the configmap [2][3]. Can somebody help me understand where the "-default" postfix comes from and how to resolve it? [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle_blob_master_deployment-5Ffiles_global_v1.0demo_software_manifests_full-2Dsite.yaml-23L26&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aimn2OylFgog_5_aS85wtQ&m=pgFJVPOeqLQoFm6_D-4rZS96wuvpXr-JbgM1VAuqqrY&s=okijvpj9PzyzWcl3plIejYr_Y9h3uBKUY_BxtIruGFw&e= [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Ddivingbell_blob_master_divingbell_templates_daemonset-2Dmounts.yaml-23L59&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aimn2OylFgog_5_aS85wtQ&m=pgFJVPOeqLQoFm6_D-4rZS96wuvpXr-JbgM1VAuqqrY&s=_ob8nMUQreMHsw-ziq-N1WLq_7dyVJdYno9A8Muh5Qg&e= Thanks a lot! Best regards Georg _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=aimn2OylFgog_5_aS85wtQ&m=pgFJVPOeqLQoFm6_D-4rZS96wuvpXr-JbgM1VAuqqrY&s=vVmA5mmFZM3gCjp_cOLfsqxY3m352xmPxI7vRl0TLhE&e= From fumiya.suga at as.ntt-at.co.jp Fri Jan 25 02:01:39 2019 From: fumiya.suga at as.ntt-at.co.jp (=?utf-8?B?6aCI6LOA5paH5ZOJ?=) Date: Fri, 25 Jan 2019 11:01:39 +0900 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> <001101d4b39a$719e1cf0$54da56d0$@as.ntt-at.co.jp> Message-ID: <000101d4b451$e935e100$bba1a300$@as.ntt-at.co.jp> Hello Roman, Thank you for your reply. I`ll try it, but > This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis > not set up for proxy at the moment, however there is a patch which > This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis > set up for proxy, I'm running a test on it now. These URL are same and I think there is typo in the url, is it okay? I would be gratefull if you can explain me how to apply patch to deploy airship-in-a-bottle behind proxy for demo site. > This one https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate > is set up for running behind the proxy (I did not try it yet). There > is also README with some info. Would you also give me details about multi_node_gates? What does it do? Its prerequisites, installation process, proxy configuration etc.. Best regards, Fumiya Suga -----Original Message----- From: Roman Gorshunov Sent: Friday, January 25, 2019 12:16 AM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] dev_single_node behind proxy Hello Fumiya, This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis not set up for proxy at the moment, however there is a patch which possibly solves it: https://review.openstack.org/#/c/615387/ (it hasit's own issues, though). This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis set up for proxy, I'm running a test on it now. Note the README file and instructions in it. This one https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gateis set up for running behind the proxy (I did not try it yet). There is also README with some info. -- Roman Gorshunov On Thu, Jan 24, 2019 at 5:08 AM 須賀文哉 wrote: > > Hi Drew, > > I have been trying to install airship-in-a-bottle behind proxy. > I am trying to run airship-in-a-bottle.sh script located in > ../manifest/dev_single_node/ behind proxy. > > Best Regards, > Fumiya Suga > > -----Original Message----- > From: Drew Walters > Sent: Thursday, January 24, 2019 12:23 PM > To: 須賀文哉 > Cc: Roman Gorshunov ; > airship-discuss at lists.airshipit.org > Subject: Re: [Airship-discuss] dev_single_node behind proxy > > > Hello Roman, > > > > Thanks for the advice. > > I tried exporting proxy environment variables as per your advice, > > However my airship-in-a-bottle installation ended on same error as > mentioned before. > > Am I missing something?Does airship-in-a-bottle installation works > > behind > the proxy? > > > > Any help would be appreciated. > > > > Best regards, > > Fumiya Suga > > > Hi Fumiya, > > May you specify which Airship-in-a-Bottle setup (e.g. dev minimal) you > are using? Some of them are not yet compatible with proxy servers. > > Thanks! > > Drew Walters > > > _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From paye600 at gmail.com Fri Jan 25 07:55:10 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Fri, 25 Jan 2019 08:55:10 +0100 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: <000101d4b451$e935e100$bba1a300$@as.ntt-at.co.jp> References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> <001101d4b39a$719e1cf0$54da56d0$@as.ntt-at.co.jp> <000101d4b451$e935e100$bba1a300$@as.ntt-at.co.jp> Message-ID: Yes, sorry for mistype, Fumiya. 1) https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_node this one is not set up for proxy, however this https://review.openstack.org/#/c/615387/ patch might allow it. Git clone AIAB repository, cd airship-in-a-bottle/, and then copy and run commands listed in top right corner here https://review.openstack.org/#/c/615387/ in Download -> Checkout field. 2) https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_minimal seems to have instructions for proxy set up, but looks to be a bit outdated. I will look into it later. 3) https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate I don't really have more details rather than provided in the README in this directory. setup_gate.sh script imports proxy-related environment variables and sets up at least docker and apt to use proxy for images and pachage downloads. Why it's so hard to get it running behind the proxy? - because most developers either have direct access to the Internet, or they use internal repositories of containers and packages. I understand that not everyone has access to big VMs running not behind proxy. Best regards, -- Roman Gorshunov On Fri, Jan 25, 2019 at 3:01 AM 須賀文哉 wrote: > > Hello Roman, > > Thank you for your reply. > I`ll try it, but > > > This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis > > not set up for proxy at the moment, however there is a patch which > > This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis > > set up for proxy, I'm running a test on it now. > > These URL are same and I think there is typo in the url, is it okay? > > I would be gratefull if you can explain me how to apply patch to deploy airship-in-a-bottle behind proxy for demo site. > > > > This one https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate > > is set up for running behind the proxy (I did not try it yet). There > > is also README with some info. > > Would you also give me details about multi_node_gates? > What does it do? Its prerequisites, installation process, proxy configuration etc.. > > > Best regards, > Fumiya Suga > > -----Original Message----- > From: Roman Gorshunov > Sent: Friday, January 25, 2019 12:16 AM > To: airship-discuss at lists.airshipit.org > Subject: Re: [Airship-discuss] dev_single_node behind proxy > > Hello Fumiya, > > This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis > not set up for proxy at the moment, however there is a patch which possibly solves it: https://review.openstack.org/#/c/615387/ (it hasit's own issues, though). > This one https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_nodeis > set up for proxy, I'm running a test on it now. Note the README file and instructions in it. > This one https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gateis > set up for running behind the proxy (I did not try it yet). There is also README with some info. > > -- Roman Gorshunov > > On Thu, Jan 24, 2019 at 5:08 AM 須賀文哉 wrote: > > > > Hi Drew, > > > > I have been trying to install airship-in-a-bottle behind proxy. > > I am trying to run airship-in-a-bottle.sh script located in > > ../manifest/dev_single_node/ behind proxy. > > > > Best Regards, > > Fumiya Suga > > > > -----Original Message----- > > From: Drew Walters > > Sent: Thursday, January 24, 2019 12:23 PM > > To: 須賀文哉 > > Cc: Roman Gorshunov ; > > airship-discuss at lists.airshipit.org > > Subject: Re: [Airship-discuss] dev_single_node behind proxy > > > > > Hello Roman, > > > > > > Thanks for the advice. > > > I tried exporting proxy environment variables as per your advice, > > > However my airship-in-a-bottle installation ended on same error as > > mentioned before. > > > Am I missing something?Does airship-in-a-bottle installation works > > > behind > > the proxy? > > > > > > Any help would be appreciated. > > > > > > Best regards, > > > Fumiya Suga > > > > > > Hi Fumiya, > > > > May you specify which Airship-in-a-Bottle setup (e.g. dev minimal) you > > are using? Some of them are not yet compatible with proxy servers. > > > > Thanks! > > > > Drew Walters > > > > > > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss > > > From paye600 at gmail.com Fri Jan 25 07:57:46 2019 From: paye600 at gmail.com (Roman Gorshunov) Date: Fri, 25 Jan 2019 08:57:46 +0100 Subject: [Airship-discuss] AIAB behind a proxy In-Reply-To: References: <8A421CD0-42B9-4289-8BC2-8E9E0AA3CC84@gmail.com> Message-ID: Jim, fixing mistypes and providing some additional info: 1) https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_node this one is not set up for proxy, however this https://review.openstack.org/#/c/615387/ patch might allow it. Git clone AIAB repository, cd airship-in-a-bottle/, and then copy and run commands listed in top right corner here https://review.openstack.org/#/c/615387/ in Download -> Checkout field. 2) https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_minimal seems to have instructions for proxy set up, but looks to be a bit outdated. I will look into it later. 3) https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate I don't really have more details rather than provided in the README in this directory. setup_gate.sh script imports proxy-related environment variables and sets up at least docker and apt to use proxy for images and pachage downloads. Why it's so hard to get it running behind the proxy? - because most developers either have direct access to the Internet, or they use internal repositories of containers and packages. I understand that not everyone has access to big VMs running not behind proxy. Best regards, -- Roman Gorshunov On Thu, Jan 24, 2019 at 3:44 PM BARESICH, JIM wrote: > > Drew, > > I'm attempting the install on a ub 16.04 KVM VM: https://github.com/openstack/airship-in-a-bottle > Which installs are not compatible behind a proxy? > > > Jim Baresich > > -----Original Message----- > From: Drew Walters > Sent: Wednesday, January 23, 2019 10:26 PM > To: Roman Gorshunov ; BARESICH, JIM > Cc: airship-discuss at lists.airshipit.org > Subject: Re: [Airship-discuss] AIAB behind a proxy > > > On Jan 22, 2019, at 1:26 PM, Roman Gorshunov wrote: > > > > Hello Jim, > > > > It should. Let me retry tomorrow morning, and I hope to get the issue fixed. > > > > Best regards, > > — > > Roman Gorshunov > > Roman (and Jim), > > Any update on the outcome of your retry? Additionally, it’s helpful if you can specify which Airship-in-a-Bottle configuration you are using, as not all are compatible behind proxy servers. > > Thanks! > > Drew Walters From rp2723 at att.com Mon Jan 28 20:26:08 2019 From: rp2723 at att.com (PACHECO, RODOLFO J) Date: Mon, 28 Jan 2019 20:26:08 +0000 Subject: [Airship-discuss] Airship - Open Design Call - AM EST Call Message-ID: <99088997CCAD0C4BA20008FD09434496326FBD28@MISOUT7MSGUSRDI.ITServices.sbc.com> When: Thursday, January 31, 2019 11:00 AM-12:00 PM. (UTC-05:00) Eastern Time (US & Canada) Where: https://attcorp.webex.com/meet/rp2723 *~*~*~*~*~*~*~*~*~* REMINDER : The time of the Airship Design Meeting Call on Jan 31, 2019 will be used for an Airship Community Feedback meeting . Etherpad for the Airship Open Design discussion https://etherpad.openstack.org/p/Airship_OpenDesignDiscussions Storyboard in flight Specs https://storyboard.openstack.org/#!/project/openstack/airship-specs Github Airship Specs https://github.com/openstack/airship-specs/tree/master/specs Inflight/reviewing specs https://review.openstack.org/#/q/status:open+airship-specs __________________________________________ Join by video system i Dial rp2723 at attcorp.webex.com and enter your host PIN 02083790. You can also dial 173.243.2.68 and enter your meeting number. Join by phone 1-844-517-1415 United States Toll Free 1-618-230-6039 United States Toll Access code: 733 333 726 Host PIN: 02083790 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 3543 bytes Desc: not available URL: From fumiya.suga at as.ntt-at.co.jp Wed Jan 30 07:17:31 2019 From: fumiya.suga at as.ntt-at.co.jp (=?utf-8?B?6aCI6LOA5paH5ZOJ?=) Date: Wed, 30 Jan 2019 16:17:31 +0900 Subject: [Airship-discuss] dev_single_node behind proxy In-Reply-To: References: <000001d4b21d$65dbdb10$31939130$@as.ntt-at.co.jp> <002201d4b2e1$fd2cbba0$f78632e0$@as.ntt-at.co.jp> <001101d4b39a$719e1cf0$54da56d0$@as.ntt-at.co.jp> <000101d4b451$e935e100$bba1a300$@as.ntt-at.co.jp> Message-ID: <000601d4b86b$dd776d70$98664850$@as.ntt-at.co.jp> Hello Roman, Thanks for the advice! I could run airship-in-a-bottle behind a proxy by using a patch. However, deploy_site action was completed with some failed steps. Following is the list of failed steps. ****snip**** step/01D29DMT22JZA35MQWEDY9CWS7/armada_build failed step/01D29DMT22JZA35MQWEDY9CWS7/armada_post_apply failed step/01D29DMT22JZA35MQWEDY9CWS7/armada_get_releases upstream_failed ******** Also, heat_stack is not created. Following is the snippet of the error. ****snip**** Creating KeyPair + error 'creating heat stack' + set +x Error when creating heat stack. + exit 1 + clean + set +x To remove files generated during this script's execution, delete /root/deploy. This VM is disposable. Re-deployment in this same VM will lead to unpredictable results. ******** Are there any solutions to these problems? Is heat_stack creation error has any relation to the steps failed during deploy_site action? Best regards, Fumiya Suga -----Original Message----- From: Roman Gorshunov Sent: Friday, January 25, 2019 4:55 PM To: airship-discuss at lists.airshipit.org Subject: Re: [Airship-discuss] dev_single_node behind proxy Yes, sorry for mistype, Fumiya. 1) https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_single_node this one is not set up for proxy, however this https://review.openstack.org/#/c/615387/ patch might allow it. Git clone AIAB repository, cd airship-in-a-bottle/, and then copy and run commands listed in top right corner here https://review.openstack.org/#/c/615387/ in Download -> Checkout field. 2) https://github.com/openstack/airship-in-a-bottle/tree/master/manifests/dev_minimal seems to have instructions for proxy set up, but looks to be a bit outdated. I will look into it later. 3) https://github.com/openstack/airship-in-a-bottle/tree/master/tools/multi_nodes_gate I don't really have more details rather than provided in the README in this directory. setup_gate.sh script imports proxy-related environment variables and sets up at least docker and apt to use proxy for images and pachage downloads. Why it's so hard to get it running behind the proxy? - because most developers either have direct access to the Internet, or they use internal repositories of containers and packages. I understand that not everyone has access to big VMs running not behind proxy. Best regards, -- Roman Gorshunov On Fri, Jan 25, 2019 at 3:01 AM 須賀文哉 wrote: > > Hello Roman, > > Thank you for your reply. > I`ll try it, but > > > This one > > https://github.com/openstack/airship-in-a-bottle/tree/master/manifes > > ts/dev_single_nodeis not set up for proxy at the moment, however > > there is a patch which This one > > https://github.com/openstack/airship-in-a-bottle/tree/master/manifes > > ts/dev_single_nodeis set up for proxy, I'm running a test on it now. > > These URL are same and I think there is typo in the url, is it okay? > > I would be gratefull if you can explain me how to apply patch to deploy airship-in-a-bottle behind proxy for demo site. > > > > This one > > https://github.com/openstack/airship-in-a-bottle/tree/master/tools/m > > ulti_nodes_gate is set up for running behind the proxy (I did not > > try it yet). There is also README with some info. > > Would you also give me details about multi_node_gates? > What does it do? Its prerequisites, installation process, proxy configuration etc.. > > > Best regards, > Fumiya Suga > > -----Original Message----- > From: Roman Gorshunov > Sent: Friday, January 25, 2019 12:16 AM > To: airship-discuss at lists.airshipit.org > Subject: Re: [Airship-discuss] dev_single_node behind proxy > > Hello Fumiya, > > This one > https://github.com/openstack/airship-in-a-bottle/tree/master/manifests > /dev_single_nodeis not set up for proxy at the moment, however there > is a patch which possibly solves it: https://review.openstack.org/#/c/615387/ (it hasit's own issues, though). > This one > https://github.com/openstack/airship-in-a-bottle/tree/master/manifests > /dev_single_nodeis set up for proxy, I'm running a test on it now. > Note the README file and instructions in it. > This one > https://github.com/openstack/airship-in-a-bottle/tree/master/tools/mul > ti_nodes_gateis set up for running behind the proxy (I did not try it > yet). There is also README with some info. > > -- Roman Gorshunov > > On Thu, Jan 24, 2019 at 5:08 AM 須賀文哉 wrote: > > > > Hi Drew, > > > > I have been trying to install airship-in-a-bottle behind proxy. > > I am trying to run airship-in-a-bottle.sh script located in > > ../manifest/dev_single_node/ behind proxy. > > > > Best Regards, > > Fumiya Suga > > > > -----Original Message----- > > From: Drew Walters > > Sent: Thursday, January 24, 2019 12:23 PM > > To: 須賀文哉 > > Cc: Roman Gorshunov ; > > airship-discuss at lists.airshipit.org > > Subject: Re: [Airship-discuss] dev_single_node behind proxy > > > > > Hello Roman, > > > > > > Thanks for the advice. > > > I tried exporting proxy environment variables as per your advice, > > > However my airship-in-a-bottle installation ended on same error as > > mentioned before. > > > Am I missing something?Does airship-in-a-bottle installation works > > > behind > > the proxy? > > > > > > Any help would be appreciated. > > > > > > Best regards, > > > Fumiya Suga > > > > > > Hi Fumiya, > > > > May you specify which Airship-in-a-Bottle setup (e.g. dev minimal) > > you are using? Some of them are not yet compatible with proxy servers. > > > > Thanks! > > > > Drew Walters > > > > > > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss at lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss > > > _______________________________________________ Airship-discuss mailing list Airship-discuss at lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss From chris at openstack.org Thu Jan 31 15:58:30 2019 From: chris at openstack.org (Chris Hoge) Date: Thu, 31 Jan 2019 07:58:30 -0800 Subject: [Airship-discuss] [summit] Open Infrastructure Summit Shanghai - Dates and Location Message-ID: <0022975A-0B24-4533-A57A-4FA9805112AC@openstack.org> We are excited to announce that after Denver, the Open Infrastructure Summit will be traveling to Shanghai, China the week of November 4, 2019. The Summit will be held at the Shanghai Expo Centre. Registration will be available soon. Sign up for updates[1] and reach out to summit at openstack.org with any questions. [1] https://www.openstack.org/summit/shanghai-2019 Chris Hoge Strategic Program Manager OpenStack Foundation From chris at openstack.org Thu Jan 31 18:46:01 2019 From: chris at openstack.org (Chris Hoge) Date: Thu, 31 Jan 2019 10:46:01 -0800 Subject: [Airship-discuss] Open Infrastructure Summit Denver - Community Voting Open Message-ID: <38932A98-9F22-434E-A570-67E8A24B2DF2@openstack.org> Community voting for the Open Infrastructure Summit Denver sessions is open! You can vote here: https://www.openstack.org/summit/denver-2019/vote-for-speakers What does that mean? Now that the Call for Presentations has closed, all submissions are available for community vote and input. After community voting closes, the volunteer Programming Committee members will receive the presentations to review and determine the final selections for Summit schedule. While community votes are meant to help inform the decision, Programming Committee members are expected to exercise judgment in their area of expertise and help ensure diversity of sessions and speakers. View full details of the session selection process here: https://www.openstack.org/summit/denver-2019/call-for-presentations/selection-process/ In order to vote, you need an OSF community membership. If you do not have an account, please create one by going to: https://openstack.org/join. If you need to reset your password, you can do that here: https://www.openstack.org/Security/lostpassword Hurry, voting closes Monday, February 4 at 11:59pm Pacific Time (Tuesday, February 5 at 7:59 UTC). Continue to visit the Open Infrastructure Summit web page for all Summit-related information: https://www.openstack.org/summit/denver-2019 Register for the Summit before prices increase in late February! https://openinfrasummitandptgdenver.eventbrite.com VISA APPLICATION PROCESS Make sure to secure your Visa soon. More information about the Visa application process here: https://www.openstack.org/summit/denver-2019/travel/ TRAVEL SUPPORT PROGRAM February 27 is the last day to submit applications. Please submit your applications by 11:59pm Pacific Time (February 28 at 7:59am UTC). https://www.openstack.org/summit/denver-2019/travel/#travel-support If you have any questions, please email summit at openstack.org. Thanks! Chris Hoge Strategic Program Manager OpenStack Foundation