[Airship-discuss] Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingress_Error

Loic Le Gal l.legal.astellia at gmail.com
Mon Mar 11 16:33:25 UTC 2019


Hi and thanks Roman & Bryan for previous hints,

I've tried the patchset 629950 as mentionned, and it fixes the maas-ingress
Crashloop.
Nevertheless, now I've got an endless restart from pod
maas-import-resources for readiness probe failure of container
region-import-resources (describe log below).

No way to remove the maas module pods?

The log gives:
>sudo kubectl logs maas-import-resources-vbdc5 -n ucp -c
region-import-resources
Boot resources currently importing

+ echo -e '\nBoot resources currently importing\n'
+ let JOB_TIMEOUT-=10
+ sleep 10
+ [[ 790 -gt 0 ]]
+ maas admin boot-resources is-importing
+ grep -q true
++ maas admin boot-resources read
++ tail -n +1
++ jq '.[] | select( .type | contains("Synced")) | .name '
++ grep -c xenial
Import failed!
+ synced_imgs=0
+ [[ 0 -gt 0 ]]
+ echo 'Import failed!'
+ return 1
+ [[ 1 -eq 0 ]]
+ [[ 1 -lt 1 ]]
+ return 1
+ [[ 1 -eq 0 ]]
+ echo 'Image import FAILED!'
+ exit 1
Image import FAILED!


ubuntu at airship:~$ sudo kubectl describe pod maas-import-resources-vbdc5 -n
ucp
Name:           maas-import-resources-vbdc5
Namespace:      ucp
Node:           airship/10.10.0.18
Start Time:     Mon, 11 Mar 2019 14:59:50 +0000
Labels:         application=maas
                component=import-resources
                controller-uid=519ea8fd-440e-11e9-b0ca-fa16aae5c6c9
                job-name=maas-import-resources
                release_group=airship-maas
Annotations:    <none>
Status:         Running
IP:             10.97.196.156
Controlled By:  Job/maas-import-resources
Init Containers:
  init:
    Container ID:
docker://b9bcb2853339a06280e3c41b3fcdd3bddafaaa5a467e0567554b6b9d6ea631a7
    Image:         quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
    Image ID:      docker-pullable://
quay.io/stackanetes/kubernetes-entrypoint at sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6
    Port:          <none>
    Host Port:     <none>
    Command:
      kubernetes-entrypoint
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 11 Mar 2019 14:59:55 +0000
      Finished:     Mon, 11 Mar 2019 15:02:41 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:              maas-import-resources-vbdc5 (v1:metadata.name)
      NAMESPACE:             ucp (v1:metadata.namespace)
      INTERFACE_NAME:        eth0
      PATH:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
      DEPENDENCY_SERVICE:    ucp:maas-region,ucp:postgresql
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      DEPENDENCY_POD_JSON:
      COMMAND:               echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from
maas-import-resources-token-mxvgx (ro)
Containers:
  region-import-resources:
    Container ID:
docker://99fabebf7b40a91efff76b1df3904cf95552df8fc8cf92a4891dc65e670678cf
    Image:
quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582
    Image ID:      docker-pullable://
quay.io/airshipit/maas-region-controller at sha256:dc6694b3883624ebbdd4245e4bac2bba8e7e54a3b12e7f4fc54bc345312187ca
    Port:          <none>
    Host Port:     <none>
    Command:
      /tmp/import-boot-resources.sh
    State:          Running
      Started:      Mon, 11 Mar 2019 16:13:53 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 11 Mar 2019 16:07:35 +0000
      Finished:     Mon, 11 Mar 2019 16:12:24 +0000
    Ready:          True
    Restart Count:  11
    Readiness:      exec [/tmp/job-readiness.sh] delay=60s timeout=1s
period=10s #success=1 #failure=3
    Environment:
      USE_IMAGE_CACHE:           false
      JOB_TIMEOUT:               900
      ADMIN_USERNAME:            <set to the key 'USERNAME' in secret
'maas-admin'>  Optional: false
      RETRY_TIMER:               10
      TRY_LIMIT:                 1
      MAAS_ENDPOINT:
http://maas-region.ucp.svc.cluster.local:80/MAAS
      MAAS_PROXY_ENABLED:        false
      MAAS_PEER_PROXY_ENABLED:   false
      MAAS_PROXY_SERVER:
      MAAS_INTERNAL_PROXY_PORT:
http://maas-region.ucp.svc.cluster.local:8000/MAAS
      MAAS_HTTP_BOOT:            true
      MAAS_NTP_SERVERS:          ntp.ubuntu.com
      MAAS_NTP_EXTERNAL_ONLY:    false
      MAAS_DNS_DNSSEC_REQUIRED:  no
      MAAS_DNS_SERVERS:          10.50.50.120,10.50.50.97
      MAAS_DEFAULT_OS:           ubuntu
      MAAS_DEFAULT_DISTRO:       xenial
      MAAS_DEFAULT_KERNEL:       ga-16.04
    Mounts:
      /etc/bind/named.conf.options from maas-etc (ro)
      /etc/maas/regiond.conf from maas-etc (ro)
      /tmp/import-boot-resources.sh from maas-bin (ro)
      /tmp/job-readiness.sh from maas-bin (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from
maas-import-resources-token-mxvgx (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  maas-bin:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      maas-bin
    Optional:  false
  maas-etc:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      maas-etc
    Optional:  false
  maas-import-resources-token-mxvgx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  maas-import-resources-token-mxvgx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  ucp-control-plane=enabled
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From              Message
  ----     ------     ----               ----              -------
  Warning  Unhealthy  53m                kubelet, airship  Readiness probe
errored: rpc error: code = Unknown desc = container not running
(e5ce115cfa833e6416b5ada892aaf859c57355552fffcfd164c2857bd0588364)
  Normal   Pulled     47m (x5 over 1h)   kubelet, airship  Container image "
quay.io/airshipit/maas-region-controller:bb81a1cecd27520b858e00f0bcf519ceb2d07582"
already present on machine
  Normal   Created    47m (x5 over 1h)   kubelet, airship  Created container
  Normal   Started    47m (x5 over 1h)   kubelet, airship  Started container
  Warning  BackOff    8m (x53 over 58m)  kubelet, airship  Back-off
restarting failed container
  Warning  Unhealthy  3m (x2 over 36m)   kubelet, airship  Readiness probe
failed: + maas-region apikey --username=admin

BR,
Loïc

Le ven. 8 mars 2019 à 18:04, STRASSNER, BRYAN <bs4939 at att.com> a écrit :

> Please be aware: https://review.openstack.org/#/c/634813/ is a change
> that updates Shipyard in airship-in-a-bottle. This change has been
> After this update, there is no Airflow GUI exposed. Users would use
> shipyard commands to inspect their workflows.
>
> Bryan Strassner
>
> -----Original Message-----
> From: Roman Gorshunov <paye600 at gmail.com>
> Sent: Friday, March 08, 2019 10:57 AM
> To: airship-discuss at lists.airshipit.org
> Cc: MCEUEN, MATT <MM9745 at att.com>; Loic Le Gal <l.legal.astellia at gmail.com
> >
> Subject: Re: [Airship-discuss]
> Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingress_Error
>
> Almost there.
> The patch [0] works and really fixes maas ingress problem. Genesis part
> gets finished, and everything fails on the actual site deploy (error
> message is below [1]).
> But later I've run commands [2] to deploy site manually, and everything
> seems to be working fine.
> You then should be able to see Airflow interface and running `deploy_site`
> on the IP address of the VM on port 30004: http://<IP of
> VM>:30004/. Port is taken from `kubectl get services -n ucp | grep
> airflow-web-int` command output.
>
> [0]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_629950_3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=D0e_LQy-6KkoIYX5ZS1BiAgOYSkfV4dXcRreugipac0&e=
>
> [1]:
> > + sudo docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e
> > + no_proxy= -e
> > + OS_AUTH_URL=https://urldefense.proofpoint.com/v2/url?u=http-3A__keys
> > + tone.ucp.svc.cluster.local-3A80_v3&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg
> > + &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6D
> > + Oyw&s=f1dIO0kygz9h-C4XxV9dcGTZlXBMwjB5uBRUt6AKPjI&e= -e
> > + OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e
> > + OS_PASSWORD=password18 -e OS_PROJECT_DOMAIN_NAME=default -e
> > + OS_PROJECT_NAME=service quay.io/airshipit/shipyard:master create
> > + action deploy_site
> > Error: Unable to complete request to Airflow
> > Reason: Airflow could not be contacted properly by Shipyard.
> > - Error: <class 'requests.exceptions.ConnectTimeout'>
>
> [2]:
> cd /root/deploy/site
> source creds.sh
> # "create configdocs design", "create configdocs secrets", "commit
> configdocs" have succeeded previously via "./airship-in-a-bottle.sh -y"
> script; only deploy_site` dropped an error ./run_shipyard.sh create action
> deploy_site
>
> -- Roman Gorshunov
>
> On Fri, Mar 8, 2019 at 4:28 PM Roman Gorshunov <paye600 at gmail.com> wrote:
> >
> > Hello team,
> > I'm working on getting maas-ingress working again. This patch set [0]
> > I believe solves this specific problem when applied onto a current
> > master [1]. Try to apply it from inside of cloned airship-in-a-bottle
> > repository by running commands under Download -> Checkout menu in the
> > right corner [2]. For me it passes trough maas initialization and helm
> > tests.
> >
> > [0]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.
> > org_-23_c_629950_&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPx
> > U-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=aq1NLATcaayJV5-
> > n6sxm3ZNx1R8TPX-MThCBUMotjFE&e= [1]
> > 539f777c637ffd1927659469072ec6235188b247
> > [2] git fetch
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openstack.org
> > _openstack_airship-2Din-2Da-2Dbottle&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg
> > &r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOy
> > w&s=BzldpD9DgSWJ5DhOMkcTTYRuuXt0XYCQcV1MmTZAJos&e=
> > refs/changes/50/629950/3 && git checkout FETCH_HEAD
> >
> > -- Roman Gorshunov
> >
> > On Thu, Mar 7, 2019 at 10:17 AM Loic Le Gal <l.legal.astellia at gmail.com>
> wrote:
> > >
> > > Thanks Matt for the feedback,
> > >
> > > I'd like to help as I can but as I'm noob to Airship it won't be easy
> at the beginning.
> > > My aim (as many of firstcomers) is to try to get a functionning
> Airship demo with k8s & Openstack setup + a little VM on top (clearwater
> later :-)); all this installed on a big VM : That's AiaB!
> > > Regarding Airskiff, I've already tryied it but as it is not AIO and
> require setting-up and plugging all layers together, it might not be the
> easiest staring point IMHO.
> > > BTW, i'm close to have a Heat template to automate the building of
> such a AiaB VM (and with proxy management), this will ease future tests
> while waiting the Treasuremap's story you mentionned.
> > >
> > > Have you got a commit ID that is secured (like a stable one)? even if
> we don't have latest features/fixes.
> > >
> > > BR,
> > >
> > > Loïc
> > >
> > > Le mer. 6 mars 2019 à 22:16, MCEUEN, MATT <MM9745 at att.com> a écrit :
> > >>
> > >> Hi Loïc & Dimitrios,
> > >>
> > >>
> > >>
> > >> Shoot, sorry you guys hit this.  This looks like the
> Airship-in-a-Bottle single node dev site may have gotten stale.  We have a
> few different sets of YAMLs that should be aligned to one another to keep
> this from happening; I’ve captured the issue and proposed resolution in
> this story[1].  If you’d like to help with this it would be very much
> appreciated :-), and if anyone has some time to get the AiaB single-node
> dev site definition trued up in the meantime that would be valuable too.
> > >>
> > >>
> > >>
> > >> For a MaaS-free dev deployment, I suggest looking at the Airskiff [2]
> deployment scripts.  These deploy the software provisioning components (no
> MaaS, Drydock, Promenade) in a bring-your-own-k8s setup.  Dwalt has a
> patchset in progress to align these site manifests fully to the treasuremap
> reference globals, per the story.
> > >>
> > >>
> > >>
> > >> [1]:
> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__storyboard.ope
> > >> nstack.org_-23-21_story_2005123&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r
> > >> =lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DO
> > >> yw&s=5rVJvyhZ78s-u8qRHgQdsfFmRFj62XcwGlk7jRh5S2U&e=
> > >>
> > >> [2]:
> > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtrea
> > >> suremap.readthedocs.io_en_latest_airskiff.html&d=DwIGaQ&c=LFYZ-o9_H
> > >> UMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6
> > >> hrI4hb9rm_Ue6DOyw&s=umXc-T-9hR-ACN4zqjXn7xm2iO8_r3oBui0v42Af21w&e=
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Matt
> > >>
> > >>
> > >>
> > >> From: Loic Le Gal <l.legal.astellia at gmail.com>
> > >> Sent: Wednesday, March 6, 2019 9:49 AM
> > >> To: airship-discuss at lists.airshipit.org
> > >> Subject: Re: [Airship-discuss]
> > >> Airship-In-A-Bottle_Dev-Single-Node_No-Proxy-Environment_Mass-Ingre
> > >> ss_Error
> > >>
> > >>
> > >>
> > >> Hi Airshipers,
> > >>
> > >>
> > >>
> > >> Same issue for me, going back to previous commit
> (ad53fbf5c312929e8d0873e5095c5a0d1dcdd7f5), but another problem arise with
> maas-import-resources. I will open new thread if can't fix.
> > >>
> > >> Why is maas required even for AiaB as we don't have any baremetal
> involved ??
> > >>
> > >>
> > >>
> > >> BR,
> > >>
> > >>
> > >>
> > >> Loïc
> > >
> > > _______________________________________________
> > > Airship-discuss mailing list
> > > Airship-discuss at lists.airshipit.org
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.
> > > org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_
> > > HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6
> > > hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=
>
> _______________________________________________
> Airship-discuss mailing list
> Airship-discuss at lists.airshipit.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=lg0Gw5OaH7fHGAPxU-Bffg&m=8wngGZfwVs4IIaJiQ7aLYRDgG6hrI4hb9rm_Ue6DOyw&s=pRUR3rU5WUpenI8XAdqiZRT6DLc6Jsnh_RACCWDYbEs&e=
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190311/207cd593/attachment-0001.html>


More information about the Airship-discuss mailing list