Hi Matt and All, The resources are not the problem. My ubuntu 16.04 VM has 8vcpu, 24G memory, and 40G disk. The symptom is that mariadb-server-0 is stuck in "running" state but never getting into the "ready" state. =========== output ===================== ...clip... kube-system nfs-provisioner-7799d64d59-zwbvk 1/1 Running 0 23m ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 22m ucp ingress-86576d6599-gtgmn 1/1 Running 0 22m ucp ingress-error-pages-5c97bb46bb-jm8zz 1/1 Running 0 22m ucp mariadb-ingress-55794d94c8-5cb97 0/1 Running 0 9m ucp mariadb-ingress-55794d94c8-5mb54 0/1 Running 0 9m ucp mariadb-ingress-error-pages-85f96fbd-qzm8g 1/1 Running 0 9m ucp mariadb-server-0 0/1 Running 0 9m <================= ucp postgresql-0 1/1 Running 1 22m ... clip ... ... clip ... 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller [-] [chart=ucp-mariadb]: Error while installing release airship-ucp-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.UNKNOWN details = "release airship-ucp-mariadb failed: timed out waiting for the condition" debug_error_string = "{"created":"@1558584756.634751023","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release airship-ucp-mariadb failed: timed out waiting for the condition","grpc_status":2}"
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller Traceback (most recent call last): 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line 460, in install_release ... clip ... ======= root@n0:~/deploy# kubectl describe pod mariadb-server-0 -n ucp ===================== ..clip.... Containers: mariadb: Container ID: docker://6888ba24aacc946e92f2651540667cb4211c01f0232b66e194746cce1c051e95 Image: docker.io/openstackhelm/mariadb:10.2.18 Image ID: docker-pullable://openstackhelm/mariadb@sha256:e4fe4898baac370f56d781541456d32aa0f8c740f1 8d5b13882a86e80a6a67bb Ports: 3306/TCP, 4567/TCP Host Ports: 0/TCP, 0/TCP Command: /tmp/start.py State: Running Started: Thu, 23 May 2019 05:12:47 +0000 Ready: False Restart Count: 0 Readiness: exec [/tmp/readiness.sh] delay=30s timeout=3s period=30s #success=1 #failure=3 Environment: POD_NAMESPACE: ucp (v1:metadata.namespace) MARIADB_REPLICAS: 1 POD_NAME_PREFIX: mariadb-server DISCOVERY_DOMAIN: mariadb-discovery.ucp.svc.cluster.local DIRECT_SVC_NAME: mariadb-server WSREP_PORT: 4567 STATE_CONFIGMAP: airship-ucp-mariadb-mariadb-state MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'mariadb-db-root-password'> Opt ional: false Mounts: /etc/mysql/admin_user.cnf from mariadb-secrets (ro) /etc/mysql/conf.d from mycnfd (rw) /etc/mysql/conf.d/00-base.cnf from mariadb-etc (ro) /etc/mysql/conf.d/20-override.cnf from mariadb-etc (ro) /etc/mysql/conf.d/99-force.cnf from mariadb-etc (ro) /etc/mysql/my.cnf from mariadb-etc (ro) /tmp/readiness.sh from mariadb-bin (ro) /tmp/start.py from mariadb-bin (ro) /tmp/stop.sh from mariadb-bin (ro) /var/lib/mysql from mysql-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from airship-ucp-mariadb-mariadb-token-ml52f (ro) Conditions: Type Status Initialized True Ready False PodScheduled True ... clip .... Events: ... clip ... Warning Unhealthy 7s (x3 over 1m) kubelet, n0 Readiness probe failed: <======== On Thu, May 23, 2019 at 12:13 AM Evgeny L <eli@mirantis.com> wrote:
I have seen other people reporting similar problem, it looks as an environment specific problem (we don't see it in our gates), also it may be related [1][2] to nfs-provisioner that we use for airship in a bottle installation.
[1] https://review.opendev.org/#/c/660544/ [2] https://review.opendev.org/#/c/659787/
On Wed, May 22, 2019 at 6:53 AM MCEUEN, MATT <MM9745@att.com> wrote:
Hi Calvin,
Have you taken a look for the root cause of why mariadb is not ready, e.g. as Roman suggested, running `kubectl -n ucp describe pod mariadb-server-0` on the genenesis node?
Are you running at the minimum 20GB or RAM? I’m wondering whether airship-in-a-bottle has outgrown that. If you’re able to run with additional resources it would be good to see if that changes the result.
We don’t have a different QA mailing list, but you’re welcome to join the #airshipit channel on Freenode IRC as well and share logs / ask questions there!
Thanks,
Matt
*From:* calvin whole <calvinwhole@gmail.com> *Sent:* Wednesday, May 22, 2019 3:51 AM *To:* airship-discuss@lists.airshipit.org; calvin whole < calvinwhole@gmail.com>; Roman Gorshunov <paye600@gmail.com> *Subject:* [Airship-discuss] Airship-in-a-bottle (AIAB) deployment question
Hi,
We have tried to deploy AIAB by both methods described in the below available info from Airship site.
Both methods failed to complete the deployment process and the result is also different.
1. With the "./airship-*in*-a-bottle.sh" method, it stuck in "mariadb not ready" state.
2. With the "vagrant up" method, we tried the libvirt and virtualbox providers.
With virtualbox provider "sometimes" we can reach the "Genesis complete" step, but the Openstack step still failed to complete.
With libvirt provider we always stuck in "mariadb not ready" or "deckhand-db-init".
So it seems to us that the execution output is sensitive to the virtualization environment.
For this reason, the documentation below may need to provide more details about the required environment setup.
Can someone provide us a more detailed info about AIAB execution requirement ?
Or, do Airship have a QA group that we can contact for help?
Many thanks,
Calvin
===============================================================================
To get started, run the following in a fresh Ubuntu 16.04 VM (minimum 4vCPU/20GB RAM/32GB disk). This will deploy Airship and Openstack Helm (OSH):
sudo -*i*
mkdir -*p* /root/deploy && cd "$_"
git clone https:*//git.openstack.org/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=http-3A__git.openstack.org_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=d9jeVVrvEr_lhTLcsBqIWqiP5hYeZ2jLW8-wQI1f78U&e=>*
cd /root/deploy/airship-*in*-a-bottle/manifests/dev_single_node
./airship-*in*-a-bottle.sh
Or, alternatively, if you have Vagrant installed, just run the following (only libvirt/kvm hypervisor is tested, but vagrant box supports VMware Desktop/Workstation/Fusion, Parallels, and Hyper-V):
curl -O https://git.airshipit.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__t.airshipit.org&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=sNKkg6eRk5lZROaz1CJncbtvoUjbndvnPeUN-_CNQnw&e=>/cgit/airship-*in*-a-bottle/plain/Vagrantfile
vagrant up
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss