[Airship-discuss] Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingress_Error
Loic Le Gal
l.legal.astellia at gmail.com
Mon Mar 11 16:33:25 UTC 2019
Hi and thanks Roman & Bryan for previous hints,
I've tried the patchset 629950 as mentionned, and it fixes the maas-ingress
Crashloop.
Nevertheless, now I've got an endless restart from pod
maas-import-resources for readiness probe failure of container
region-import-resources (describe log below).
No way to remove the maas module pods?
The log gives:
>sudo kubectl logs maas-import-resources-vbdc5 -n ucp -c
region-import-resources
Boot resources currently importing
+ echo -e '\nBoot resources currently importing\n'
+ let JOB_TIMEOUT-=10
+ sleep 10
+ [[ 790 -gt 0 ]]
+ maas admin boot-resources is-importing
+ grep -q true
++ maas admin boot-resources read
++ tail -n +1
++ jq '.[] | select( .type | contains("Synced")) | .name '
++ grep -c xenial
Import failed!
+ synced_imgs=0
+ [[ 0 -gt 0 ]]
+ echo 'Import failed!'
+ return 1
+ [[ 1 -eq 0 ]]
+ [[ 1 -lt 1 ]]
+ return 1
+ [[ 1 -eq 0 ]]
+ echo 'Image import FAILED!'
+ exit 1
Image import FAILED!
ubuntu at airship:~$ sudo kubectl describe pod maas-import-resources-vbdc5 -n
ucp
Name: maas-import-resources-vbdc5
Namespace: ucp
Node: airship/10.10.0.18
Start Time: Mon, 11 Mar 2019 14:59:50 +0000
Labels: application=maas
component=import-resources
controller-uid=519ea8fd-440e-11e9-b0ca-fa16aae5c6c9
job-name=maas-import-resources
release_group=airship-maas
Annotations: <none>
Status: Running
IP: 10.97.196.156
Controlled By: Job/maas-import-resources
Init Containers:
init:
Container ID:
docker://b9bcb2853339a06280e3c41b3fcdd3bddafaaa5a467e0567554b6b9d6ea631a7
Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
Image ID: docker-pullable://
quay.io/stackanetes/kubernetes-entrypoint at sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6
Port: <none>
Host Port: <none>
Command:
kubernetes-entrypoint
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 11 Mar 2019 14:59:55 +0000
Finished: Mon, 11 Mar 2019 15:02:41 +0000
Ready: True
Restart Count: 0
Environment:
POD_NAME: maas-import-resources-vbdc5 (v1:metadata.name)
NAMESPACE: ucp (v1:metadata.namespace)
INTERFACE_NAME: eth0
PATH:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
DEPENDENCY_SERVICE: ucp:maas-region,ucp:postgresql
DEPENDENCY_DAEMONSET:
DEPENDENCY_CONTAINER:
DEPENDENCY_POD_JSON:
COMMAND: echo done
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from
maas-import-resources-token-mxvgx (ro)
Containers:
region-import-resources:
Container ID:
docker://99fabebf7b40a91efff76b1df3904cf95552df8fc8cf92a4891dc65e670678cf
Image:
quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582
Image ID: docker-pullable://
quay.io/airshipit/maas-region-controller at sha256:dc6694b3883624ebbdd4245e4bac2bba8e7e54a3b12e7f4fc54bc345312187ca
Port: <none>
Host Port: <none>
Command:
/tmp/import-boot-resources.sh
State: Running
Started: Mon, 11 Mar 2019 16:13:53 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 11 Mar 2019 16:07:35 +0000
Finished: Mon, 11 Mar 2019 16:12:24 +0000
Ready: True
Restart Count: 11
Readiness: exec [/tmp/job-readiness.sh] delay=60s timeout=1s
period=10s #success=1 #failure=3
Environment:
USE_IMAGE_CACHE: false
JOB_TIMEOUT: 900
ADMIN_USERNAME: <set to the key 'USERNAME' in secret
'maas-admin'> Optional: false
RETRY_TIMER: 10
TRY_LIMIT: 1
MAAS_ENDPOINT:
http://maas-region.ucp.svc.cluster.local:80/MAAS
MAAS_PROXY_ENABLED: false
MAAS_PEER_PROXY_ENABLED: false
MAAS_PROXY_SERVER:
MAAS_INTERNAL_PROXY_PORT:
http://maas-region.ucp.svc.cluster.local:8000/MAAS
MAAS_HTTP_BOOT: true
MAAS_NTP_SERVERS: ntp.ubuntu.com
MAAS_NTP_EXTERNAL_ONLY: false
MAAS_DNS_DNSSEC_REQUIRED: no
MAAS_DNS_SERVERS: 10.50.50.120,10.50.50.97
MAAS_DEFAULT_OS: ubuntu
MAAS_DEFAULT_DISTRO: xenial
MAAS_DEFAULT_KERNEL: ga-16.04
Mounts:
/etc/bind/named.conf.options from maas-etc (ro)
/etc/maas/regiond.conf from maas-etc (ro)
/tmp/import-boot-resources.sh from maas-bin (ro)
/tmp/job-readiness.sh from maas-bin (ro)
/var/run/secrets/kubernetes.io/serviceaccount from
maas-import-resources-token-mxvgx (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
maas-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: maas-bin
Optional: false
maas-etc:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: maas-etc
Optional: false
maas-import-resources-token-mxvgx:
Type: Secret (a volume populated by a Secret)
SecretName: maas-import-resources-token-mxvgx
Optional: false
QoS Class: BestEffort
Node-Selectors: ucp-control-plane=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 53m kubelet, airship Readiness probe
errored: rpc error: code = Unknown desc = container not running
(e5ce115cfa833e6416b5ada892aaf859c57355552fffcfd164c2857bd0588364)
Normal Pulled 47m (x5 over 1h) kubelet, airship Container image "
quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582"
already present on machine
Normal Created 47m (x5 over 1h) kubelet, airship Created container
Normal Started 47m (x5 over 1h) kubelet, airship Started container
Warning BackOff 8m (x53 over 58m) kubelet, airship Back-off
restarting failed container
Warning Unhealthy 3m (x2 over 36m) kubelet, airship Readiness probe
failed: + maas-region apikey --username=admin
BR,
Loïc
Le ven. 8 mars 2019 à 18:04, STRASSNER, BRYAN <bs4939 at att.com> a écrit :
> Please be aware: https://review.openstack.org/#/c/634813/ is a change
> that updates Shipyard in airship-in-a-bottle. This change has been
> After this update, there is no Airflow GUI exposed. Users would use
> shipyard commands to inspect their workflows.
>
> Bryan Strassner
>
> -----Original Message-----
> From: Roman Gorshunov <paye600 at gmail.com>
> Sent: Friday, March 08, 2019 10:57 AM
> To: airship-discuss at lists.airshipit.org
> Cc: MCEUEN, MATT <MM9745 at att.com>; Loic Le Gal <l.legal.astellia at gmail.com
> >
> Subject: Re: [Airship-discuss]
> Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingress_Error
>
> Almost there.
> The patch [0] works and really fixes maas ingress problem. Genesis part
> gets finished, and everything fails on the actual site deploy (error
> message is below [1]).
> But later I've run commands [2] to deploy site manually, and everything
> seems to be working fine.
> You then should be able to see Airflow interface and running `deploy_site`
> on the IP address of the VM on port 30004: http://<IP of
> VM>:30004/. Port is taken from `kubectl get services -n ucp | grep
> airflow-web-int` command output.
>
> [0]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_629950_3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=D0e_LQy-6KkoIYX5ZS1BiAgOYSkfV4dXcRreugipac0&e=
>
> [1]:
> > + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e
> > + no_proxy= -e
> > + OS_AUTH_URL=https://urldefense.proofpoint.com/v2/url?u=http-3A__keys
> > + tone.ucp.svc.cluster.local-3A80_v3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg
> > + &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6D
> > + Oyw&s=f1dIO0kygz9h-C4XxV9dcGTZlXBMwjB5uBRUt6AKPjI&e= -e
> > + OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e
> > + OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e
> > + OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create
> > + action deploy_site
> > Error: Unable to complete request to Airflow
> > Reason: Airflow could not be contacted properly by Shipyard.
> > - Error: <class 'requests.exceptions.ConnectTimeout'>
>
> [2]:
> cd /root/deploy/site
> source creds.sh
> # "create configdocs design", "create configdocs secrets", "commit
> configdocs" have succeeded previously via "./airship-in-a-bottle.sh -y"
> script; only deploy_site` dropped an error ./run_shipyard.sh create action
> deploy_site
>
> -- Roman Gorshunov
>
> On Fri, Mar 8, 2019 at 4:28 PM Roman Gorshunov <paye600 at gmail.com> wrote:
> >
> > Hello team,
> > I'm working on getting maas-ingress working again. This patch set [0]
> > I believe solves this specific problem when applied onto a current
> > master [1]. Try to apply it from inside of cloned airship-in-a-bottle
> > repository by running commands under Download -> Checkout menu in the
> > right corner [2]. For me it passes trough maas initialization and helm
> > tests.
> >
> > [0]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.
> > org_-23_c_629950_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPx
> > U-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=aq1NLATcaayJV5-
> > n6sxm3ZNx1R8TPX-MThCBUMotjFE&e= [1]
> > 539f777c637ffd1927659469072ec6235188b247
> > [2] git fetch
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openstack.org
> > _openstack_airship-2Din-2Da-2Dbottle&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg
> > &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOy
> > w&s=BzldpD9DgSWJ5DhOMkcTTYRuuXt0XYCQcV1MmTZAJos&e=
> > refs/changes/50/629950/3 && git checkout FETCH_HEAD
> >
> > -- Roman Gorshunov
> >
> > On Thu, Mar 7, 2019 at 10:17 AM Loic Le Gal <l.legal.astellia at gmail.com>
> wrote:
> > >
> > > Thanks Matt for the feedback,
> > >
> > > I'd like to help as I can but as I'm noob to Airship it won't be easy
> at the beginning.
> > > My aim (as many of firstcomers) is to try to get a functionning
> Airship demo with k8s & Openstack setup + a little VM on top (clearwater
> later :-)); all this installed on a big VM : That's AiaB!
> > > Regarding Airskiff, I've already tryied it but as it is not AIO and
> require setting-up and plugging all layers together, it might not be the
> easiest staring point IMHO.
> > > BTW, i'm close to have a Heat template to automate the building of
> such a AiaB VM (and with proxy management), this will ease future tests
> while waiting the Treasuremap's story you mentionned.
> > >
> > > Have you got a commit ID that is secured (like a stable one)? even if
> we don't have latest features/fixes.
> > >
> > > BR,
> > >
> > > Loïc
> > >
> > > Le mer. 6 mars 2019 à 22:16, MCEUEN, MATT <MM9745 at att.com> a écrit :
> > >>
> > >> Hi Loïc & Dimitrios,
> > >>
> > >>
> > >>
> > >> Shoot, sorry you guys hit this. This looks like the
> Airship-in-a-Bottle single node dev site may have gotten stale. We have a
> few different sets of YAMLs that should be aligned to one another to keep
> this from happening; I’ve captured the issue and proposed resolution in
> this story[1]. If you’d like to help with this it would be very much
> appreciated :-), and if anyone has some time to get the AiaB single-node
> dev site definition trued up in the meantime that would be valuable too.
> > >>
> > >>
> > >>
> > >> For a MaaS-free dev deployment, I suggest looking at the Airskiff [2]
> deployment scripts. These deploy the software provisioning components (no
> MaaS, Drydock, Promenade) in a bring-your-own-k8s setup. Dwalt has a
> patchset in progress to align these site manifests fully to the treasuremap
> reference globals, per the story.
> > >>
> > >>
> > >>
> > >> [1]:
> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__storyboard.ope
> > >> nstack.org_-23-21_story_2005123&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r
> > >> =lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DO
> > >> yw&s=5rVJvyhZ78s-u8qRHgQdsfFmRFj62XcwGlk7jRh5S2U&e=
> > >>
> > >> [2]:
> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtrea
> > >> suremap.readthedocs.io_en_latest_airskiff.html&d=DwIGaQ&c=LFYZ-o9_H
> > >> UMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6
> > >> hrI4hb9rm_Ue6DOyw&s=umXc-T-9hR-ACN4zqjXn7xm2iO8_r3oBui0v42Af21w&e=
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Matt
> > >>
> > >>
> > >>
> > >> From: Loic Le Gal <l.legal.astellia at gmail.com>
> > >> Sent: Wednesday, March 6, 2019 9:49 AM
> > >> To: airship-discuss at lists.airshipit.org
> > >> Subject: Re: [Airship-discuss]
> > >> Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingre
> > >> ss_Error
> > >>
> > >>
> > >>
> > >> Hi Airshipers,
> > >>
> > >>
> > >>
> > >> Same issue for me, going back to previous commit
> (ad53fbf5c312929e8d0873e5095c5a0d1dcdd7f5), but another problem arise with
> maas-import-resources. I will open new thread if can't fix.
> > >>
> > >> Why is maas required even for AiaB as we don't have any baremetal
> involved ??
> > >>
> > >>
> > >>
> > >> BR,
> > >>
> > >>
> > >>
> > >> Loïc
> > >
> > > _______________________________________________
> > > Airship-discuss mailing list
> > > Airship-discuss at lists.airshipit.org
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.
> > > org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_
> > > HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6
> > > hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=
>
> _______________________________________________
> Airship-discuss mailing list
> Airship-discuss at lists.airshipit.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190311/207cd593/attachment-0001.html>
More information about the Airship-discuss
mailing list