<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi and thanks Roman & Bryan for previous hints,<div><br></div><div>I've tried the patchset 629950 as mentionned, and it fixes the maas-ingress Crashloop.</div><div>Nevertheless, now I've got an endless restart from pod maas-import-resources for readiness probe failure of container region-import-resources (describe log below).</div><div><br></div><div>No way to remove the maas module pods?</div><div><br></div><div>The log gives:</div><div>>sudo kubectl logs maas-import-resources-vbdc5 -n ucp -c region-import-resources<br></div><div><div>Boot resources currently importing</div><div><br></div><div>+ echo -e '\nBoot resources currently importing\n'</div><div>+ let JOB_TIMEOUT-=10</div><div>+ sleep 10</div><div>+ [[ 790 -gt 0 ]]</div><div>+ maas admin boot-resources is-importing</div><div>+ grep -q true</div><div>++ maas admin boot-resources read</div><div>++ tail -n +1</div><div>++ jq '.[] | select( .type | contains("Synced")) | .name '</div><div>++ grep -c xenial</div><div>Import failed!</div><div>+ synced_imgs=0</div><div>+ [[ 0 -gt 0 ]]</div><div>+ echo 'Import failed!'</div><div>+ return 1</div><div>+ [[ 1 -eq 0 ]]</div><div>+ [[ 1 -lt 1 ]]</div><div>+ return 1</div><div>+ [[ 1 -eq 0 ]]</div><div>+ echo 'Image import FAILED!'</div><div>+ exit 1</div><div>Image import FAILED!</div></div><div><br></div><div><br></div><div><div>ubuntu@airship:~$ sudo kubectl describe pod maas-import-resources-vbdc5 -n ucp</div><div>Name: maas-import-resources-vbdc5</div><div>Namespace: ucp</div><div>Node: airship/<a href="http://10.10.0.18">10.10.0.18</a></div><div>Start Time: Mon, 11 Mar 2019 14:59:50 +0000</div><div>Labels: application=maas</div><div> component=import-resources</div><div> controller-uid=519ea8fd-440e-11e9-b0ca-fa16aae5c6c9</div><div> job-name=maas-import-resources</div><div> release_group=airship-maas</div><div>Annotations: <none></div><div>Status: Running</div><div>IP: 10.97.196.156</div><div>Controlled By: Job/maas-import-resources</div><div>Init Containers:</div><div> init:</div><div> Container ID: docker://b9bcb2853339a06280e3c41b3fcdd3bddafaaa5a467e0567554b6b9d6ea631a7</div><div> Image: <a href="http://quay.io/stackanetes/kubernetes-entrypoint:v0.3.1">quay.io/stackanetes/kubernetes-entrypoint:v0.3.1</a></div><div> Image ID: docker-pullable://<a href="http://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6">quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6</a></div><div> Port: <none></div><div> Host Port: <none></div><div> Command:</div><div> kubernetes-entrypoint</div><div> State: Terminated</div><div> Reason: Completed</div><div> Exit Code: 0</div><div> Started: Mon, 11 Mar 2019 14:59:55 +0000</div><div> Finished: Mon, 11 Mar 2019 15:02:41 +0000</div><div> Ready: True</div><div> Restart Count: 0</div><div> Environment:</div><div> POD_NAME: maas-import-resources-vbdc5 (v1:<a href="http://metadata.name">metadata.name</a>)</div><div> NAMESPACE: ucp (v1:metadata.namespace)</div><div> INTERFACE_NAME: eth0</div><div> PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/</div><div> DEPENDENCY_SERVICE: ucp:maas-region,ucp:postgresql</div><div> DEPENDENCY_DAEMONSET:</div><div> DEPENDENCY_CONTAINER:</div><div> DEPENDENCY_POD_JSON:</div><div> COMMAND: echo done</div><div> Mounts:</div><div> /var/run/secrets/<a href="http://kubernetes.io/serviceaccount">kubernetes.io/serviceaccount</a> from maas-import-resources-token-mxvgx (ro)</div><div>Containers:</div><div> region-import-resources:</div><div> Container ID: docker://99fabebf7b40a91efff76b1df3904cf95552df8fc8cf92a4891dc65e670678cf</div><div> Image: <a href="http://quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582">quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582</a></div><div> Image ID: docker-pullable://<a href="http://quay.io/airshipit/maas-region-controller@sha256:dc6694b3883624ebbdd4245e4bac2bba8e7e54a3b12e7f4fc54bc345312187ca">quay.io/airshipit/maas-region-controller@sha256:dc6694b3883624ebbdd4245e4bac2bba8e7e54a3b12e7f4fc54bc345312187ca</a></div><div> Port: <none></div><div> Host Port: <none></div><div> Command:</div><div> /tmp/import-boot-resources.sh</div><div> State: Running</div><div> Started: Mon, 11 Mar 2019 16:13:53 +0000</div><div> Last State: Terminated</div><div> Reason: Error</div><div> Exit Code: 1</div><div> Started: Mon, 11 Mar 2019 16:07:35 +0000</div><div> Finished: Mon, 11 Mar 2019 16:12:24 +0000</div><div> Ready: True</div><div> Restart Count: 11</div><div> Readiness: exec [/tmp/job-readiness.sh] delay=60s timeout=1s period=10s #success=1 #failure=3</div><div> Environment:</div><div> USE_IMAGE_CACHE: false</div><div> JOB_TIMEOUT: 900</div><div> ADMIN_USERNAME: <set to the key 'USERNAME' in secret 'maas-admin'> Optional: false</div><div> RETRY_TIMER: 10</div><div> TRY_LIMIT: 1</div><div> MAAS_ENDPOINT: <a href="http://maas-region.ucp.svc.cluster.local:80/MAAS">http://maas-region.ucp.svc.cluster.local:80/MAAS</a></div><div> MAAS_PROXY_ENABLED: false</div><div> MAAS_PEER_PROXY_ENABLED: false</div><div> MAAS_PROXY_SERVER:</div><div> MAAS_INTERNAL_PROXY_PORT: <a href="http://maas-region.ucp.svc.cluster.local:8000/MAAS">http://maas-region.ucp.svc.cluster.local:8000/MAAS</a></div><div> MAAS_HTTP_BOOT: true</div><div> MAAS_NTP_SERVERS: <a href="http://ntp.ubuntu.com">ntp.ubuntu.com</a></div><div> MAAS_NTP_EXTERNAL_ONLY: false</div><div> MAAS_DNS_DNSSEC_REQUIRED: no</div><div> MAAS_DNS_SERVERS: 10.50.50.120,10.50.50.97</div><div> MAAS_DEFAULT_OS: ubuntu</div><div> MAAS_DEFAULT_DISTRO: xenial</div><div> MAAS_DEFAULT_KERNEL: ga-16.04</div><div> Mounts:</div><div> /etc/bind/named.conf.options from maas-etc (ro)</div><div> /etc/maas/regiond.conf from maas-etc (ro)</div><div> /tmp/import-boot-resources.sh from maas-bin (ro)</div><div> /tmp/job-readiness.sh from maas-bin (ro)</div><div> /var/run/secrets/<a href="http://kubernetes.io/serviceaccount">kubernetes.io/serviceaccount</a> from maas-import-resources-token-mxvgx (ro)</div><div>Conditions:</div><div> Type Status</div><div> Initialized True</div><div> Ready True</div><div> PodScheduled True</div><div>Volumes:</div><div> maas-bin:</div><div> Type: ConfigMap (a volume populated by a ConfigMap)</div><div> Name: maas-bin</div><div> Optional: false</div><div> maas-etc:</div><div> Type: ConfigMap (a volume populated by a ConfigMap)</div><div> Name: maas-etc</div><div> Optional: false</div><div> maas-import-resources-token-mxvgx:</div><div> Type: Secret (a volume populated by a Secret)</div><div> SecretName: maas-import-resources-token-mxvgx</div><div> Optional: false</div><div>QoS Class: BestEffort</div><div>Node-Selectors: ucp-control-plane=enabled</div><div>Tolerations: <a href="http://node.kubernetes.io/not-ready:NoExecute">node.kubernetes.io/not-ready:NoExecute</a> for 300s</div><div> <a href="http://node.kubernetes.io/unreachable:NoExecute">node.kubernetes.io/unreachable:NoExecute</a> for 300s</div><div>Events:</div><div> Type Reason Age From Message</div><div> ---- ------ ---- ---- -------</div><div> Warning Unhealthy 53m kubelet, airship Readiness probe errored: rpc error: code = Unknown desc = container not running (e5ce115cfa833e6416b5ada892aaf859c57355552fffcfd164c2857bd0588364)</div><div> Normal Pulled 47m (x5 over 1h) kubelet, airship Container image "<a href="http://quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582">quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582</a>" already present on machine</div><div> Normal Created 47m (x5 over 1h) kubelet, airship Created container</div><div> Normal Started 47m (x5 over 1h) kubelet, airship Started container</div><div> Warning BackOff 8m (x53 over 58m) kubelet, airship Back-off restarting failed container</div><div> Warning Unhealthy 3m (x2 over 36m) kubelet, airship Readiness probe failed: + maas-region apikey --username=admin</div></div><div><br></div><div>BR, </div><div>Loïc</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le ven. 8 mars 2019 à 18:04, STRASSNER, BRYAN <<a href="mailto:bs4939@att.com">bs4939@att.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Please be aware: <a href="https://review.openstack.org/#/c/634813/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/634813/</a> is a change that updates Shipyard in airship-in-a-bottle. This change has been <br>
After this update, there is no Airflow GUI exposed. Users would use shipyard commands to inspect their workflows.<br>
<br>
Bryan Strassner<br>
<br>
-----Original Message-----<br>
From: Roman Gorshunov <<a href="mailto:paye600@gmail.com" target="_blank">paye600@gmail.com</a>> <br>
Sent: Friday, March 08, 2019 10:57 AM<br>
To: <a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a><br>
Cc: MCEUEN, MATT <<a href="mailto:MM9745@att.com" target="_blank">MM9745@att.com</a>>; Loic Le Gal <<a href="mailto:l.legal.astellia@gmail.com" target="_blank">l.legal.astellia@gmail.com</a>><br>
Subject: Re: [Airship-discuss] Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingress_Error<br>
<br>
Almost there.<br>
The patch [0] works and really fixes maas ingress problem. Genesis part gets finished, and everything fails on the actual site deploy (error message is below [1]).<br>
But later I've run commands [2] to deploy site manually, and everything seems to be working fine.<br>
You then should be able to see Airflow interface and running `deploy_site` on the IP address of the VM on port 30004: http://<IP of<br>
VM>:30004/. Port is taken from `kubectl get services -n ucp | grep<br>
airflow-web-int` command output.<br>
<br>
[0] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_629950_3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=D0e_LQy-6KkoIYX5ZS1BiAgOYSkfV4dXcRreugipac0&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_629950_3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=D0e_LQy-6KkoIYX5ZS1BiAgOYSkfV4dXcRreugipac0&e=</a><br>
<br>
[1]:<br>
> + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e <br>
> + no_proxy= -e <br>
> + OS_AUTH_URL=<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__keys" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__keys</a><br>
> + tone.ucp.svc.cluster.local-3A80_v3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg<br>
> + &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6D<br>
> + Oyw&s=f1dIO0kygz9h-C4XxV9dcGTZlXBMwjB5uBRUt6AKPjI&e= -e <br>
> + OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e <br>
> + OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e <br>
> + OS_PROJECT_NAME=service <a href="http://quay.io/airshipit/shipyard:master" rel="noreferrer" target="_blank">quay.io/airshipit/shipyard:master</a> create <br>
> + action deploy_site<br>
> Error: Unable to complete request to Airflow<br>
> Reason: Airflow could not be contacted properly by Shipyard.<br>
> - Error: <class 'requests.exceptions.ConnectTimeout'><br>
<br>
[2]:<br>
cd /root/deploy/site<br>
source creds.sh<br>
# "create configdocs design", "create configdocs secrets", "commit configdocs" have succeeded previously via "./airship-in-a-bottle.sh -y" script; only deploy_site` dropped an error ./run_shipyard.sh create action deploy_site<br>
<br>
-- Roman Gorshunov<br>
<br>
On Fri, Mar 8, 2019 at 4:28 PM Roman Gorshunov <<a href="mailto:paye600@gmail.com" target="_blank">paye600@gmail.com</a>> wrote:<br>
><br>
> Hello team,<br>
> I'm working on getting maas-ingress working again. This patch set [0] <br>
> I believe solves this specific problem when applied onto a current <br>
> master [1]. Try to apply it from inside of cloned airship-in-a-bottle <br>
> repository by running commands under Download -> Checkout menu in the <br>
> right corner [2]. For me it passes trough maas initialization and helm <br>
> tests.<br>
><br>
> [0] <br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack</a>.<br>
> org_-23_c_629950_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPx<br>
> U-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=aq1NLATcaayJV5-<br>
> n6sxm3ZNx1R8TPX-MThCBUMotjFE&e= [1] <br>
> 539f777c637ffd1927659469072ec6235188b247<br>
> [2] git fetch <br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openstack.org" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openstack.org</a><br>
> _openstack_airship-2Din-2Da-2Dbottle&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg<br>
> &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOy<br>
> w&s=BzldpD9DgSWJ5DhOMkcTTYRuuXt0XYCQcV1MmTZAJos&e=<br>
> refs/changes/50/629950/3 && git checkout FETCH_HEAD<br>
><br>
> -- Roman Gorshunov<br>
><br>
> On Thu, Mar 7, 2019 at 10:17 AM Loic Le Gal <<a href="mailto:l.legal.astellia@gmail.com" target="_blank">l.legal.astellia@gmail.com</a>> wrote:<br>
> ><br>
> > Thanks Matt for the feedback,<br>
> ><br>
> > I'd like to help as I can but as I'm noob to Airship it won't be easy at the beginning.<br>
> > My aim (as many of firstcomers) is to try to get a functionning Airship demo with k8s & Openstack setup + a little VM on top (clearwater later :-)); all this installed on a big VM : That's AiaB!<br>
> > Regarding Airskiff, I've already tryied it but as it is not AIO and require setting-up and plugging all layers together, it might not be the easiest staring point IMHO.<br>
> > BTW, i'm close to have a Heat template to automate the building of such a AiaB VM (and with proxy management), this will ease future tests while waiting the Treasuremap's story you mentionned.<br>
> ><br>
> > Have you got a commit ID that is secured (like a stable one)? even if we don't have latest features/fixes.<br>
> ><br>
> > BR,<br>
> ><br>
> > Loïc<br>
> ><br>
> > Le mer. 6 mars 2019 à 22:16, MCEUEN, MATT <<a href="mailto:MM9745@att.com" target="_blank">MM9745@att.com</a>> a écrit :<br>
> >><br>
> >> Hi Loïc & Dimitrios,<br>
> >><br>
> >><br>
> >><br>
> >> Shoot, sorry you guys hit this. This looks like the Airship-in-a-Bottle single node dev site may have gotten stale. We have a few different sets of YAMLs that should be aligned to one another to keep this from happening; I’ve captured the issue and proposed resolution in this story[1]. If you’d like to help with this it would be very much appreciated :-), and if anyone has some time to get the AiaB single-node dev site definition trued up in the meantime that would be valuable too.<br>
> >><br>
> >><br>
> >><br>
> >> For a MaaS-free dev deployment, I suggest looking at the Airskiff [2] deployment scripts. These deploy the software provisioning components (no MaaS, Drydock, Promenade) in a bring-your-own-k8s setup. Dwalt has a patchset in progress to align these site manifests fully to the treasuremap reference globals, per the story.<br>
> >><br>
> >><br>
> >><br>
> >> [1]: <br>
> >> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__storyboard.ope" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__storyboard.ope</a><br>
> >> nstack.org_-23-21_story_2005123&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r<br>
> >> =lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DO<br>
> >> yw&s=5rVJvyhZ78s-u8qRHgQdsfFmRFj62XcwGlk7jRh5S2U&e=<br>
> >><br>
> >> [2]: <br>
> >> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtrea" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtrea</a><br>
> >> suremap.readthedocs.io_en_latest_airskiff.html&d=DwIGaQ&c=LFYZ-o9_H<br>
> >> UMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6<br>
> >> hrI4hb9rm_Ue6DOyw&s=umXc-T-9hR-ACN4zqjXn7xm2iO8_r3oBui0v42Af21w&e=<br>
> >><br>
> >><br>
> >><br>
> >> Thanks,<br>
> >><br>
> >> Matt<br>
> >><br>
> >><br>
> >><br>
> >> From: Loic Le Gal <<a href="mailto:l.legal.astellia@gmail.com" target="_blank">l.legal.astellia@gmail.com</a>><br>
> >> Sent: Wednesday, March 6, 2019 9:49 AM<br>
> >> To: <a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a><br>
> >> Subject: Re: [Airship-discuss] <br>
> >> Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingre<br>
> >> ss_Error<br>
> >><br>
> >><br>
> >><br>
> >> Hi Airshipers,<br>
> >><br>
> >><br>
> >><br>
> >> Same issue for me, going back to previous commit (ad53fbf5c312929e8d0873e5095c5a0d1dcdd7f5), but another problem arise with maas-import-resources. I will open new thread if can't fix.<br>
> >><br>
> >> Why is maas required even for AiaB as we don't have any baremetal involved ??<br>
> >><br>
> >><br>
> >><br>
> >> BR,<br>
> >><br>
> >><br>
> >><br>
> >> Loïc<br>
> ><br>
> > _______________________________________________<br>
> > Airship-discuss mailing list<br>
> > <a href="mailto:Airship-discuss@lists.airshipit.org" target="_blank">Airship-discuss@lists.airshipit.org</a><br>
> > <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit</a>.<br>
> > org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_<br>
> > HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6<br>
> > hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=<br>
<br>
_______________________________________________<br>
Airship-discuss mailing list<br>
<a href="mailto:Airship-discuss@lists.airshipit.org" target="_blank">Airship-discuss@lists.airshipit.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=</a><br>
</blockquote></div>