Hi and thanks Roman & Bryan for previous hints, I've tried the patchset 629950 as mentionned, and it fixes the maas-ingress Crashloop. Nevertheless, now I've got an endless restart from pod maas-import-resources for readiness probe failure of container region-import-resources (describe log below). No way to remove the maas module pods? The log gives:
sudo kubectl logs maas-import-resources-vbdc5 -n ucp -c region-import-resources Boot resources currently importing
+ echo -e '\nBoot resources currently importing\n' + let JOB_TIMEOUT-=10 + sleep 10 + [[ 790 -gt 0 ]] + maas admin boot-resources is-importing + grep -q true ++ maas admin boot-resources read ++ tail -n +1 ++ jq '.[] | select( .type | contains("Synced")) | .name ' ++ grep -c xenial Import failed! + synced_imgs=0 + [[ 0 -gt 0 ]] + echo 'Import failed!' + return 1 + [[ 1 -eq 0 ]] + [[ 1 -lt 1 ]] + return 1 + [[ 1 -eq 0 ]] + echo 'Image import FAILED!' + exit 1 Image import FAILED! ubuntu@airship:~$ sudo kubectl describe pod maas-import-resources-vbdc5 -n ucp Name: maas-import-resources-vbdc5 Namespace: ucp Node: airship/10.10.0.18 Start Time: Mon, 11 Mar 2019 14:59:50 +0000 Labels: application=maas component=import-resources controller-uid=519ea8fd-440e-11e9-b0ca-fa16aae5c6c9 job-name=maas-import-resources release_group=airship-maas Annotations: <none> Status: Running IP: 10.97.196.156 Controlled By: Job/maas-import-resources Init Containers: init: Container ID: docker://b9bcb2853339a06280e3c41b3fcdd3bddafaaa5a467e0567554b6b9d6ea631a7 Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: docker-pullable:// quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6 Port: <none> Host Port: <none> Command: kubernetes-entrypoint State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 11 Mar 2019 14:59:55 +0000 Finished: Mon, 11 Mar 2019 15:02:41 +0000 Ready: True Restart Count: 0 Environment: POD_NAME: maas-import-resources-vbdc5 (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: ucp:maas-region,ucp:postgresql DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from maas-import-resources-token-mxvgx (ro) Containers: region-import-resources: Container ID: docker://99fabebf7b40a91efff76b1df3904cf95552df8fc8cf92a4891dc65e670678cf Image: quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582 Image ID: docker-pullable:// quay.io/airshipit/maas-region-controller@sha256:dc6694b3883624ebbdd4245e4bac2bba8e7e54a3b12e7f4fc54bc345312187ca Port: <none> Host Port: <none> Command: /tmp/import-boot-resources.sh State: Running Started: Mon, 11 Mar 2019 16:13:53 +0000 Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 11 Mar 2019 16:07:35 +0000 Finished: Mon, 11 Mar 2019 16:12:24 +0000 Ready: True Restart Count: 11 Readiness: exec [/tmp/job-readiness.sh] delay=60s timeout=1s period=10s #success=1 #failure=3 Environment: USE_IMAGE_CACHE: false JOB_TIMEOUT: 900 ADMIN_USERNAME: <set to the key 'USERNAME' in secret 'maas-admin'> Optional: false RETRY_TIMER: 10 TRY_LIMIT: 1 MAAS_ENDPOINT: http://maas-region.ucp.svc.cluster.local:80/MAAS MAAS_PROXY_ENABLED: false MAAS_PEER_PROXY_ENABLED: false MAAS_PROXY_SERVER: MAAS_INTERNAL_PROXY_PORT: http://maas-region.ucp.svc.cluster.local:8000/MAAS MAAS_HTTP_BOOT: true MAAS_NTP_SERVERS: ntp.ubuntu.com MAAS_NTP_EXTERNAL_ONLY: false MAAS_DNS_DNSSEC_REQUIRED: no MAAS_DNS_SERVERS: 10.50.50.120,10.50.50.97 MAAS_DEFAULT_OS: ubuntu MAAS_DEFAULT_DISTRO: xenial MAAS_DEFAULT_KERNEL: ga-16.04 Mounts: /etc/bind/named.conf.options from maas-etc (ro) /etc/maas/regiond.conf from maas-etc (ro) /tmp/import-boot-resources.sh from maas-bin (ro) /tmp/job-readiness.sh from maas-bin (ro) /var/run/secrets/kubernetes.io/serviceaccount from maas-import-resources-token-mxvgx (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: maas-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: maas-bin Optional: false maas-etc: Type: ConfigMap (a volume populated by a ConfigMap) Name: maas-etc Optional: false maas-import-resources-token-mxvgx: Type: Secret (a volume populated by a Secret) SecretName: maas-import-resources-token-mxvgx Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 53m kubelet, airship Readiness probe errored: rpc error: code = Unknown desc = container not running (e5ce115cfa833e6416b5ada892aaf859c57355552fffcfd164c2857bd0588364) Normal Pulled 47m (x5 over 1h) kubelet, airship Container image " quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582" already present on machine Normal Created 47m (x5 over 1h) kubelet, airship Created container Normal Started 47m (x5 over 1h) kubelet, airship Started container Warning BackOff 8m (x53 over 58m) kubelet, airship Back-off restarting failed container Warning Unhealthy 3m (x2 over 36m) kubelet, airship Readiness probe failed: + maas-region apikey --username=admin BR, Loïc Le ven. 8 mars 2019 à 18:04, STRASSNER, BRYAN <bs4939@att.com> a écrit :
Please be aware: https://review.openstack.org/#/c/634813/ is a change that updates Shipyard in airship-in-a-bottle. This change has been After this update, there is no Airflow GUI exposed. Users would use shipyard commands to inspect their workflows.
Bryan Strassner
-----Original Message----- From: Roman Gorshunov <paye600@gmail.com> Sent: Friday, March 08, 2019 10:57 AM To: airship-discuss@lists.airshipit.org Cc: MCEUEN, MATT <MM9745@att.com>; Loic Le Gal <l.legal.astellia@gmail.com
Subject: Re: [Airship-discuss] Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingress_Error
Almost there. The patch [0] works and really fixes maas ingress problem. Genesis part gets finished, and everything fails on the actual site deploy (error message is below [1]). But later I've run commands [2] to deploy site manually, and everything seems to be working fine. You then should be able to see Airflow interface and running `deploy_site` on the IP address of the VM on port 30004: http://<IP of VM>:30004/. Port is taken from `kubectl get services -n ucp | grep airflow-web-int` command output.
[1]:
+ sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e + no_proxy= -e + OS_AUTH_URL=https://urldefense.proofpoint.com/v2/url?u=http-3A__keys + tone.ucp.svc.cluster.local-3A80_v3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg + &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6D + Oyw&s=f1dIO0kygz9h-C4XxV9dcGTZlXBMwjB5uBRUt6AKPjI&e= -e + OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e + OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e + OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create + action deploy_site Error: Unable to complete request to Airflow Reason: Airflow could not be contacted properly by Shipyard. - Error: <class 'requests.exceptions.ConnectTimeout'>
[2]: cd /root/deploy/site source creds.sh # "create configdocs design", "create configdocs secrets", "commit configdocs" have succeeded previously via "./airship-in-a-bottle.sh -y" script; only deploy_site` dropped an error ./run_shipyard.sh create action deploy_site
-- Roman Gorshunov
On Fri, Mar 8, 2019 at 4:28 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello team, I'm working on getting maas-ingress working again. This patch set [0] I believe solves this specific problem when applied onto a current master [1]. Try to apply it from inside of cloned airship-in-a-bottle repository by running commands under Download -> Checkout menu in the right corner [2]. For me it passes trough maas initialization and helm tests.
[0] https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack. org_-23_c_629950_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPx U-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=aq1NLATcaayJV5- n6sxm3ZNx1R8TPX-MThCBUMotjFE&e= [1] 539f777c637ffd1927659469072ec6235188b247 [2] git fetch https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openstack.org _openstack_airship-2Din-2Da-2Dbottle&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOy w&s=BzldpD9DgSWJ5DhOMkcTTYRuuXt0XYCQcV1MmTZAJos&e= refs/changes/50/629950/3 && git checkout FETCH_HEAD
-- Roman Gorshunov
On Thu, Mar 7, 2019 at 10:17 AM Loic Le Gal <l.legal.astellia@gmail.com>
Thanks Matt for the feedback,
I'd like to help as I can but as I'm noob to Airship it won't be easy
at the beginning.
My aim (as many of firstcomers) is to try to get a functionning Airship demo with k8s & Openstack setup + a little VM on top (clearwater later :-)); all this installed on a big VM : That's AiaB! Regarding Airskiff, I've already tryied it but as it is not AIO and require setting-up and plugging all layers together, it might not be the easiest staring point IMHO. BTW, i'm close to have a Heat template to automate the building of such a AiaB VM (and with proxy management), this will ease future tests while waiting the Treasuremap's story you mentionned.
Have you got a commit ID that is secured (like a stable one)? even if we don't have latest features/fixes.
BR,
Loïc
Le mer. 6 mars 2019 à 22:16, MCEUEN, MATT <MM9745@att.com> a écrit :
Hi Loïc & Dimitrios,
Shoot, sorry you guys hit this. This looks like the
Airship-in-a-Bottle single node dev site may have gotten stale. We have a few different sets of YAMLs that should be aligned to one another to keep
For a MaaS-free dev deployment, I suggest looking at the Airskiff [2]
deployment scripts. These deploy the software provisioning components (no MaaS, Drydock, Promenade) in a bring-your-own-k8s setup. Dwalt has a
wrote: this from happening; I’ve captured the issue and proposed resolution in this story[1]. If you’d like to help with this it would be very much appreciated :-), and if anyone has some time to get the AiaB single-node dev site definition trued up in the meantime that would be valuable too. patchset in progress to align these site manifests fully to the treasuremap reference globals, per the story.
[1]: https://urldefense.proofpoint.com/v2/url?u=https-3A__storyboard.ope nstack.org_-23-21_story_2005123&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r =lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DO yw&s=5rVJvyhZ78s-u8qRHgQdsfFmRFj62XcwGlk7jRh5S2U&e=
[2]: https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtrea suremap.readthedocs.io_en_latest_airskiff.html&d=DwIGaQ&c=LFYZ-o9_H UMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6 hrI4hb9rm_Ue6DOyw&s=umXc-T-9hR-ACN4zqjXn7xm2iO8_r3oBui0v42Af21w&e=
Thanks,
Matt
From: Loic Le Gal <l.legal.astellia@gmail.com> Sent: Wednesday, March 6, 2019 9:49 AM To: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingre ss_Error
Hi Airshipers,
Same issue for me, going back to previous commit
(ad53fbf5c312929e8d0873e5095c5a0d1dcdd7f5), but another problem arise with maas-import-resources. I will open new thread if can't fix.
Why is maas required even for AiaB as we don't have any baremetal
involved ??
BR,
Loïc
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit. org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_ HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6 hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org