<div dir="ltr">Hi Matt and All,<div><br></div><div>The resources are not the problem. My ubuntu 16.04 VM has 8vcpu, 24G memory, and 40G disk.</div><div><br></div><div>The symptom is that mariadb-server-0 is stuck in "running" state but never getting into the "ready" state. </div><div><br></div><div>=========== output =====================</div><div>...clip...</div><div>kube-system nfs-provisioner-7799d64d59-zwbvk 1/1 Running 0 23m<br></div><div>ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 22m<br>ucp ingress-86576d6599-gtgmn 1/1 Running 0 22m<br>ucp ingress-error-pages-5c97bb46bb-jm8zz 1/1 Running 0 22m<br>ucp mariadb-ingress-55794d94c8-5cb97 0/1 Running 0 9m<br>ucp mariadb-ingress-55794d94c8-5mb54 0/1 Running 0 9m<br>ucp mariadb-ingress-error-pages-85f96fbd-qzm8g 1/1 Running 0 9m<br>ucp mariadb-server-0 0/1 Running 0 9m <=================<br>ucp postgresql-0 1/1 Running 1 22m<br></div><div>... clip ...</div><div><br></div><div>... clip ...</div><div>2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller [-] [chart=ucp-mariadb]: Error while installing release airship-ucp-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:<br> status = StatusCode.UNKNOWN<br> details = "release airship-ucp-mariadb failed: timed out waiting for the condition"<br> debug_error_string = "{"created":"@1558584756.634751023","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release airship-ucp-mariadb failed: timed out waiting for the condition","grpc_status":2}"<br>><br>2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller Traceback (most recent call last):<br>2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller File "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line 460, in install_release<br></div><div>... clip ...</div><div><br></div><div>======= root@n0:~/deploy# kubectl describe pod mariadb-server-0 -n ucp =====================</div><div>..clip....</div><div><br></div><div>Containers:<br> mariadb:<br> Container ID: docker://6888ba24aacc946e92f2651540667cb4211c01f0232b66e194746cce1c051e95<br> Image: <a href="http://docker.io/openstackhelm/mariadb:10.2.18">docker.io/openstackhelm/mariadb:10.2.18</a><br> Image ID: docker-pullable://openstackhelm/mariadb@sha256:e4fe4898baac370f56d781541456d32aa0f8c740f1 8d5b13882a86e80a6a67bb<br> Ports: 3306/TCP, 4567/TCP<br> Host Ports: 0/TCP, 0/TCP<br> Command:<br> /tmp/start.py<br> State: Running<br> Started: Thu, 23 May 2019 05:12:47 +0000<br> Ready: False<br> Restart Count: 0<br> Readiness: exec [/tmp/readiness.sh] delay=30s timeout=3s period=30s #success=1 #failure=3<br> Environment:<br> POD_NAMESPACE: ucp (v1:metadata.namespace)<br> MARIADB_REPLICAS: 1<br> POD_NAME_PREFIX: mariadb-server<br> DISCOVERY_DOMAIN: mariadb-discovery.ucp.svc.cluster.local<br> DIRECT_SVC_NAME: mariadb-server<br> WSREP_PORT: 4567<br> STATE_CONFIGMAP: airship-ucp-mariadb-mariadb-state<br> MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'mariadb-db-root-password'> Opt ional: false<br> Mounts:<br> /etc/mysql/admin_user.cnf from mariadb-secrets (ro)<br> /etc/mysql/conf.d from mycnfd (rw)<br> /etc/mysql/conf.d/00-base.cnf from mariadb-etc (ro)<br> /etc/mysql/conf.d/20-override.cnf from mariadb-etc (ro)<br> /etc/mysql/conf.d/99-force.cnf from mariadb-etc (ro)<br> /etc/mysql/my.cnf from mariadb-etc (ro)<br> /tmp/readiness.sh from mariadb-bin (ro)<br> /tmp/start.py from mariadb-bin (ro)<br> /tmp/stop.sh from mariadb-bin (ro)<br> /var/lib/mysql from mysql-data (rw)<br> /var/run/secrets/<a href="http://kubernetes.io/serviceaccount">kubernetes.io/serviceaccount</a> from airship-ucp-mariadb-mariadb-token-ml52f (ro)<br>Conditions:<br> Type Status<br> Initialized True<br> Ready False<br> PodScheduled True<br></div><div>... clip ....</div><div>Events:<br></div><div>... clip ...</div><div> Warning Unhealthy 7s (x3 over 1m) kubelet, n0 Readiness probe failed: <========<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 23, 2019 at 12:13 AM Evgeny L <<a href="mailto:eli@mirantis.com">eli@mirantis.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I have seen other people reporting similar problem, it looks as an environment specific problem (we don't see it in our gates), also it may be related [1][2] to nfs-provisioner that we use for airship in a bottle installation.<br></div><div><br></div><div>[1] <a href="https://review.opendev.org/#/c/660544/" target="_blank">https://review.opendev.org/#/c/660544/</a></div><div>[2] <a href="https://review.opendev.org/#/c/659787/" target="_blank">https://review.opendev.org/#/c/659787/</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 22, 2019 at 6:53 AM MCEUEN, MATT <<a href="mailto:MM9745@att.com" target="_blank">MM9745@att.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-671140006644484597gmail-m_-2167888654295055975WordSection1">
<p class="MsoNormal">Hi Calvin,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Have you taken a look for the root cause of why mariadb is not ready, e.g. as Roman suggested, running `kubectl -n ucp describe pod mariadb-server-0` on the genenesis node?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Are you running at the minimum 20GB or RAM? I’m wondering whether airship-in-a-bottle has outgrown that. If you’re able to run with additional resources it would be good to see if that changes the result.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">We don’t have a different QA mailing list, but you’re welcome to join the #airshipit channel on Freenode IRC as well and share logs / ask questions there!<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Matt<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>From:</b> calvin whole <<a href="mailto:calvinwhole@gmail.com" target="_blank">calvinwhole@gmail.com</a>> <br>
<b>Sent:</b> Wednesday, May 22, 2019 3:51 AM<br>
<b>To:</b> <a href="mailto:airship-discuss@lists.airshipit.org" target="_blank">airship-discuss@lists.airshipit.org</a>; calvin whole <<a href="mailto:calvinwhole@gmail.com" target="_blank">calvinwhole@gmail.com</a>>; Roman Gorshunov <<a href="mailto:paye600@gmail.com" target="_blank">paye600@gmail.com</a>><br>
<b>Subject:</b> [Airship-discuss] Airship-in-a-bottle (AIAB) deployment question<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We have tried to deploy AIAB by both methods described in the below available info from Airship site.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Both methods failed to complete the deployment process and the result is also different.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1. With the "<span style="font-size:10pt;font-family:Consolas;color:rgb(51,51,51)">./airship-<span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975m424864823141949421gmail-hljs-keyword"><b>in</b></span>-a-bottle<span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975m424864823141949421gmail-hljs-selector-class">.sh" method,
it stuck in "mariadb not ready" state.</span></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;color:rgb(51,51,51)">2. With the "vagrant up" method, we tried the libvirt and virtualbox providers. </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;color:rgb(51,51,51)">With virtualbox provider "sometimes" we can reach the "Genesis complete" step, but the Openstack step still failed to complete.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;color:rgb(51,51,51)">With libvirt provider we always stuck in "mariadb not ready" or "deckhand-db-init".</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;color:rgb(51,51,51)">So it seems to us that the execution output is sensitive to the virtualization environment.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;color:rgb(51,51,51)">For this reason, the documentation below may need to provide more details about the required environment setup.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Can someone provide us a more detailed info about AIAB execution requirement ?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Or, do Airship have a QA group that we can contact for help?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Many thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Calvin<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">===============================================================================<u></u><u></u></p>
</div>
<div>
<p style="margin-right:0in;margin-bottom:12pt;margin-left:0in;line-height:17.15pt;box-sizing:inherit;color:rgba(0,0,0,0.87)">
<span style="font-size:12pt;font-family:"Helvetica Neue",serif">To get started, run the following in a fresh Ubuntu 16.04 VM (minimum 4vCPU/20GB RAM/32GB disk). This will deploy Airship and Openstack Helm (OSH):<u></u><u></u></span></p>
<pre style="background:none 0% 0% repeat scroll rgb(247,247,247);box-sizing:inherit;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;border-radius:3px;color:rgba(0,0,0,0.87);overflow:auto"><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">sudo -</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-selector-tag"><b><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">i</span></b></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in"><u></u><u></u></span></code></pre>
<pre style="background:none 0% 0% repeat scroll rgb(247,247,247)"><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">mkdir -</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-selector-tag"><b><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">p</span></b></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in"> /root/deploy && cd </span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-string"><span style="font-family:Consolas;color:rgb(221,17,68);border:1pt none windowtext;padding:0in">"$_"</span></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in"><u></u><u></u></span></code></pre>
<pre style="background:none 0% 0% repeat scroll rgb(247,247,247)"><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">git clone https:</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-comment"><i><span style="font-family:Consolas;color:rgb(153,153,136);border:1pt none windowtext;padding:0in">//<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__git.openstack.org_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=d9jeVVrvEr_lhTLcsBqIWqiP5hYeZ2jLW8-wQI1f78U&e=" target="_blank">git.openstack.org/openstack/airship-in-a-bottle</a></span></i></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in"><u></u><u></u></span></code></pre>
<pre style="background:none 0% 0% repeat scroll rgb(247,247,247)"><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">cd /root/deploy/airship-</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-keyword"><b><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">in</span></b></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">-a-bottle/manifests/dev_single_node<u></u><u></u></span></code></pre>
<pre style="background:none 0% 0% repeat scroll rgb(247,247,247)"><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">./airship-</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-keyword"><b><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">in</span></b></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">-a-bottle</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-selector-class"><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">.sh</span></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in"><u></u><u></u></span></code></pre>
<p style="margin-right:0in;margin-bottom:12pt;margin-left:0in;line-height:17.15pt;box-sizing:inherit;color:rgba(0,0,0,0.87)">
<span style="font-size:12pt;font-family:"Helvetica Neue",serif">Or, alternatively, if you have Vagrant installed, just run the following (only libvirt/kvm hypervisor is tested, but vagrant box supports VMware Desktop/Workstation/Fusion, Parallels, and Hyper-V):<u></u><u></u></span></p>
<pre style="background:none 0% 0% repeat scroll rgb(247,247,247);box-sizing:inherit;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;border-radius:3px;color:rgba(0,0,0,0.87);overflow:auto"><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">curl -O https:</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-regexp"><span style="font-family:Consolas;color:rgb(0,153,38);border:1pt none windowtext;padding:0in">//gi</span></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__t.airshipit.org&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=sNKkg6eRk5lZROaz1CJncbtvoUjbndvnPeUN-_CNQnw&e=" target="_blank">t.airshipit.org</a></span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-regexp"><span style="font-family:Consolas;color:rgb(0,153,38);border:1pt none windowtext;padding:0in">/cgit/</span></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">airship-</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-keyword"><b><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">in</span></b></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">-a-bottle</span></code><span class="gmail-m_-671140006644484597gmail-m_-2167888654295055975gmail-hljs-regexp"><span style="font-family:Consolas;color:rgb(0,153,38);border:1pt none windowtext;padding:0in">/plain/</span></span><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">Vagrantfile<u></u><u></u></span></code></pre>
<pre style="background:none 0% 0% repeat scroll rgb(247,247,247)"><code><span style="font-family:Consolas;color:rgb(51,51,51);border:1pt none windowtext;padding:0in">vagrant up</span></code><span style="font-family:Consolas"><u></u><u></u></span></pre>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Airship-discuss mailing list<br>
<a href="mailto:Airship-discuss@lists.airshipit.org" target="_blank">Airship-discuss@lists.airshipit.org</a><br>
<a href="http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss" rel="noreferrer" target="_blank">http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss</a></blockquote></div>
</blockquote></div>