Hi Matt and All,

The resources are not the problem. My ubuntu 16.04 VM has 8vcpu, 24G memory, and 40G disk.

The symptom is that mariadb-server-0 is stuck in "running" state but never getting into the "ready" state. 

=========== output =====================

...clip...

kube-system   nfs-provisioner-7799d64d59-zwbvk             1/1       Running     0          23m

ucp           airship-ucp-rabbitmq-rabbitmq-0              1/1       Running     0          22m
ucp           ingress-86576d6599-gtgmn                     1/1       Running     0          22m
ucp           ingress-error-pages-5c97bb46bb-jm8zz         1/1       Running     0          22m
ucp           mariadb-ingress-55794d94c8-5cb97             0/1       Running     0          9m
ucp           mariadb-ingress-55794d94c8-5mb54             0/1       Running     0          9m
ucp           mariadb-ingress-error-pages-85f96fbd-qzm8g   1/1       Running     0          9m
ucp           mariadb-server-0                             0/1       Running     0          9m   <=================
ucp           postgresql-0                                 1/1       Running     1          22m

... clip ...

... clip ...

2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller [-] [chart=ucp-mariadb]: Error while installing release airship-ucp-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "release airship-ucp-mariadb failed: timed out waiting for the condition"
        debug_error_string = "{"created":"@1558584756.634751023","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release airship-ucp-mariadb failed: timed out waiting for the condition","grpc_status":2}"
>
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller Traceback (most recent call last):
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line 460, in install_release

... clip ...

======= root@n0:~/deploy# kubectl describe pod mariadb-server-0 -n ucp =====================

..clip....

Containers:
  mariadb:
    Container ID:  docker://6888ba24aacc946e92f2651540667cb4211c01f0232b66e194746cce1c051e95
    Image:         docker.io/openstackhelm/mariadb:10.2.18
    Image ID:      docker-pullable://openstackhelm/mariadb@sha256:e4fe4898baac370f56d781541456d32aa0f8c740f1                          8d5b13882a86e80a6a67bb
    Ports:         3306/TCP, 4567/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /tmp/start.py
    State:          Running
      Started:      Thu, 23 May 2019 05:12:47 +0000
    Ready:          False
    Restart Count:  0
    Readiness:      exec [/tmp/readiness.sh] delay=30s timeout=3s period=30s #success=1 #failure=3
    Environment:
      POD_NAMESPACE:        ucp (v1:metadata.namespace)
      MARIADB_REPLICAS:     1
      POD_NAME_PREFIX:      mariadb-server
      DISCOVERY_DOMAIN:     mariadb-discovery.ucp.svc.cluster.local
      DIRECT_SVC_NAME:      mariadb-server
      WSREP_PORT:           4567
      STATE_CONFIGMAP:      airship-ucp-mariadb-mariadb-state
      MYSQL_ROOT_PASSWORD:  <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'mariadb-db-root-password'>  Opt                          ional: false
    Mounts:
      /etc/mysql/admin_user.cnf from mariadb-secrets (ro)
      /etc/mysql/conf.d from mycnfd (rw)
      /etc/mysql/conf.d/00-base.cnf from mariadb-etc (ro)
      /etc/mysql/conf.d/20-override.cnf from mariadb-etc (ro)
      /etc/mysql/conf.d/99-force.cnf from mariadb-etc (ro)
      /etc/mysql/my.cnf from mariadb-etc (ro)
      /tmp/readiness.sh from mariadb-bin (ro)
      /tmp/start.py from mariadb-bin (ro)
      /tmp/stop.sh from mariadb-bin (ro)
      /var/lib/mysql from mysql-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from airship-ucp-mariadb-mariadb-token-ml52f (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True

... clip ....

Events:

... clip ...

  Warning  Unhealthy              7s (x3 over 1m)  kubelet, n0        Readiness probe failed:   <========

On Thu, May 23, 2019 at 12:13 AM Evgeny L <eli@mirantis.com> wrote:

I have seen other people reporting similar problem, it looks as an environment specific problem (we don't see it in our gates), also it may be related [1][2] to nfs-provisioner that we use for airship in a bottle installation.

[1] https://review.opendev.org/#/c/660544/

[2] https://review.opendev.org/#/c/659787/

On Wed, May 22, 2019 at 6:53 AM MCEUEN, MATT <MM9745@att.com> wrote:

Hi Calvin,

 

Have you taken a look for the root cause of why mariadb is not ready, e.g. as Roman suggested, running `kubectl -n ucp describe pod mariadb-server-0` on the genenesis node?

 

Are you running at the minimum 20GB or RAM?  I’m wondering whether airship-in-a-bottle has outgrown that.  If you’re able to run with additional resources it would be good to see if that changes the result.

 

We don’t have a different QA mailing list, but you’re welcome to join the #airshipit channel on Freenode IRC as well and share logs / ask questions there!

 

Thanks,

Matt

 

From: calvin whole <calvinwhole@gmail.com>
Sent: Wednesday, May 22, 2019 3:51 AM
To: airship-discuss@lists.airshipit.org; calvin whole <calvinwhole@gmail.com>; Roman Gorshunov <paye600@gmail.com>
Subject: [Airship-discuss] Airship-in-a-bottle (AIAB) deployment question

 

Hi, 

 

We have tried to deploy AIAB by both methods described in the below available info from Airship site.

 

Both methods failed to complete the deployment process and the result is also different.

1. With the "./airship-in-a-bottle.sh" method, it stuck in "mariadb not ready" state.

 

2. With the "vagrant up" method, we tried the libvirt and virtualbox providers. 

With virtualbox provider "sometimes" we can reach the "Genesis complete" step, but the Openstack step still failed to complete.

With libvirt provider we always stuck in "mariadb not ready" or "deckhand-db-init".

 

So it seems to us that the execution output is sensitive to the virtualization environment.

For this reason, the documentation below may need to provide more details about the required environment setup.

 

Can someone provide us a more detailed info about AIAB execution requirement ?

 

Or, do Airship have a QA group that we can contact for help?

 

Many thanks,

Calvin

 

===============================================================================

To get started, run the following in a fresh Ubuntu 16.04 VM (minimum 4vCPU/20GB RAM/32GB disk). This will deploy Airship and Openstack Helm (OSH):

sudo -i

mkdir -p /root/deploy && cd "$_"

git clone https://git.openstack.org/openstack/airship-in-a-bottle

cd /root/deploy/airship-in-a-bottle/manifests/dev_single_node

./airship-in-a-bottle.sh

Or, alternatively, if you have Vagrant installed, just run the following (only libvirt/kvm hypervisor is tested, but vagrant box supports VMware Desktop/Workstation/Fusion, Parallels, and Hyper-V):

curl -O https://git.airshipit.org/cgit/airship-in-a-bottle/plain/Vagrantfile

vagrant up

_______________________________________________
Airship-discuss mailing list
Airship-discuss@lists.airshipit.org
http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss