[Airship-discuss] Airship-in-a-bottle (AIAB) deployment question

calvin whole calvinwhole at gmail.com
Thu May 23 05:33:33 UTC 2019


Hi Matt and All,

The resources are not the problem. My ubuntu 16.04 VM has 8vcpu, 24G
memory, and 40G disk.

The symptom is that mariadb-server-0 is stuck in "running" state but never
getting into the "ready" state.

=========== output =====================
...clip...
kube-system   nfs-provisioner-7799d64d59-zwbvk             1/1
Running     0          23m
ucp           airship-ucp-rabbitmq-rabbitmq-0              1/1
Running     0          22m
ucp           ingress-86576d6599-gtgmn                     1/1
Running     0          22m
ucp           ingress-error-pages-5c97bb46bb-jm8zz         1/1
Running     0          22m
ucp           mariadb-ingress-55794d94c8-5cb97             0/1
Running     0          9m
ucp           mariadb-ingress-55794d94c8-5mb54             0/1
Running     0          9m
ucp           mariadb-ingress-error-pages-85f96fbd-qzm8g   1/1
Running     0          9m
ucp           mariadb-server-0                             0/1
Running     0          9m   <=================
ucp           postgresql-0                                 1/1
Running     1          22m
... clip ...

... clip ...
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller [-]
[chart=ucp-mariadb]: Error while installing release airship-ucp-mariadb:
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "release airship-ucp-mariadb failed: timed out waiting
for the condition"
        debug_error_string =
"{"created":"@1558584756.634751023","description":"Error received from
peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release
airship-ucp-mariadb failed: timed out waiting for the
condition","grpc_status":2}"
>
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller Traceback (most
recent call last):
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller   File
"/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line
460, in install_release
... clip ...

======= root at n0:~/deploy# kubectl describe pod mariadb-server-0 -n ucp
=====================
..clip....

Containers:
  mariadb:
    Container ID:
 docker://6888ba24aacc946e92f2651540667cb4211c01f0232b66e194746cce1c051e95
    Image:         docker.io/openstackhelm/mariadb:10.2.18
    Image ID:
docker-pullable://openstackhelm/mariadb@sha256:e4fe4898baac370f56d781541456d32aa0f8c740f1
                         8d5b13882a86e80a6a67bb
    Ports:         3306/TCP, 4567/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /tmp/start.py
    State:          Running
      Started:      Thu, 23 May 2019 05:12:47 +0000
    Ready:          False
    Restart Count:  0
    Readiness:      exec [/tmp/readiness.sh] delay=30s timeout=3s
period=30s #success=1 #failure=3
    Environment:
      POD_NAMESPACE:        ucp (v1:metadata.namespace)
      MARIADB_REPLICAS:     1
      POD_NAME_PREFIX:      mariadb-server
      DISCOVERY_DOMAIN:     mariadb-discovery.ucp.svc.cluster.local
      DIRECT_SVC_NAME:      mariadb-server
      WSREP_PORT:           4567
      STATE_CONFIGMAP:      airship-ucp-mariadb-mariadb-state
      MYSQL_ROOT_PASSWORD:  <set to the key 'MYSQL_ROOT_PASSWORD' in secret
'mariadb-db-root-password'>  Opt                          ional: false
    Mounts:
      /etc/mysql/admin_user.cnf from mariadb-secrets (ro)
      /etc/mysql/conf.d from mycnfd (rw)
      /etc/mysql/conf.d/00-base.cnf from mariadb-etc (ro)
      /etc/mysql/conf.d/20-override.cnf from mariadb-etc (ro)
      /etc/mysql/conf.d/99-force.cnf from mariadb-etc (ro)
      /etc/mysql/my.cnf from mariadb-etc (ro)
      /tmp/readiness.sh from mariadb-bin (ro)
      /tmp/start.py from mariadb-bin (ro)
      /tmp/stop.sh from mariadb-bin (ro)
      /var/lib/mysql from mysql-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from
airship-ucp-mariadb-mariadb-token-ml52f (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
... clip ....
Events:
... clip ...
  Warning  Unhealthy              7s (x3 over 1m)  kubelet, n0
 Readiness probe failed:   <========

On Thu, May 23, 2019 at 12:13 AM Evgeny L <eli at mirantis.com> wrote:

> I have seen other people reporting similar problem, it looks as an
> environment specific problem (we don't see it in our gates), also it may be
> related [1][2] to nfs-provisioner that we use for airship in a bottle
> installation.
>
> [1] https://review.opendev.org/#/c/660544/
> [2] https://review.opendev.org/#/c/659787/
>
> On Wed, May 22, 2019 at 6:53 AM MCEUEN, MATT <MM9745 at att.com> wrote:
>
>> Hi Calvin,
>>
>>
>>
>> Have you taken a look for the root cause of why mariadb is not ready,
>> e.g. as Roman suggested, running `kubectl -n ucp describe pod
>> mariadb-server-0` on the genenesis node?
>>
>>
>>
>> Are you running at the minimum 20GB or RAM?  I’m wondering whether
>> airship-in-a-bottle has outgrown that.  If you’re able to run with
>> additional resources it would be good to see if that changes the result.
>>
>>
>>
>> We don’t have a different QA mailing list, but you’re welcome to join the
>> #airshipit channel on Freenode IRC as well and share logs / ask questions
>> there!
>>
>>
>>
>> Thanks,
>>
>> Matt
>>
>>
>>
>> *From:* calvin whole <calvinwhole at gmail.com>
>> *Sent:* Wednesday, May 22, 2019 3:51 AM
>> *To:* airship-discuss at lists.airshipit.org; calvin whole <
>> calvinwhole at gmail.com>; Roman Gorshunov <paye600 at gmail.com>
>> *Subject:* [Airship-discuss] Airship-in-a-bottle (AIAB) deployment
>> question
>>
>>
>>
>> Hi,
>>
>>
>>
>> We have tried to deploy AIAB by both methods described in the below
>> available info from Airship site.
>>
>>
>>
>> Both methods failed to complete the deployment process and the result is
>> also different.
>>
>> 1. With the "./airship-*in*-a-bottle.sh" method, it stuck in "mariadb
>> not ready" state.
>>
>>
>>
>> 2. With the "vagrant up" method, we tried the libvirt and virtualbox
>> providers.
>>
>> With virtualbox provider "sometimes" we can reach the "Genesis complete"
>> step, but the Openstack step still failed to complete.
>>
>> With libvirt provider we always stuck in "mariadb not ready" or
>> "deckhand-db-init".
>>
>>
>>
>> So it seems to us that the execution output is sensitive to the
>> virtualization environment.
>>
>> For this reason, the documentation below may need to provide more details
>> about the required environment setup.
>>
>>
>>
>> Can someone provide us a more detailed info about AIAB execution
>> requirement ?
>>
>>
>>
>> Or, do Airship have a QA group that we can contact for help?
>>
>>
>>
>> Many thanks,
>>
>> Calvin
>>
>>
>>
>>
>> ===============================================================================
>>
>> To get started, run the following in a fresh Ubuntu 16.04 VM (minimum
>> 4vCPU/20GB RAM/32GB disk). This will deploy Airship and Openstack Helm
>> (OSH):
>>
>> sudo -*i*
>>
>> mkdir -*p* /root/deploy && cd "$_"
>>
>> git clone https:*//git.openstack.org/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=http-3A__git.openstack.org_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=d9jeVVrvEr_lhTLcsBqIWqiP5hYeZ2jLW8-wQI1f78U&e=>*
>>
>> cd /root/deploy/airship-*in*-a-bottle/manifests/dev_single_node
>>
>> ./airship-*in*-a-bottle.sh
>>
>> Or, alternatively, if you have Vagrant installed, just run the following
>> (only libvirt/kvm hypervisor is tested, but vagrant box supports VMware
>> Desktop/Workstation/Fusion, Parallels, and Hyper-V):
>>
>> curl -O https://git.airshipit.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__t.airshipit.org&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=sNKkg6eRk5lZROaz1CJncbtvoUjbndvnPeUN-_CNQnw&e=>/cgit/airship-*in*-a-bottle/plain/Vagrantfile
>>
>> vagrant up
>>
>> _______________________________________________
>> Airship-discuss mailing list
>> Airship-discuss at lists.airshipit.org
>> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190523/44cefc56/attachment-0001.html>


More information about the Airship-discuss mailing list