[Airship-discuss] Airship-in-a-bottle (AIAB) deployment question
calvin whole
calvinwhole at gmail.com
Thu May 23 05:33:33 UTC 2019
Hi Matt and All,
The resources are not the problem. My ubuntu 16.04 VM has 8vcpu, 24G
memory, and 40G disk.
The symptom is that mariadb-server-0 is stuck in "running" state but never
getting into the "ready" state.
=========== output =====================
...clip...
kube-system nfs-provisioner-7799d64d59-zwbvk 1/1
Running 0 23m
ucp airship-ucp-rabbitmq-rabbitmq-0 1/1
Running 0 22m
ucp ingress-86576d6599-gtgmn 1/1
Running 0 22m
ucp ingress-error-pages-5c97bb46bb-jm8zz 1/1
Running 0 22m
ucp mariadb-ingress-55794d94c8-5cb97 0/1
Running 0 9m
ucp mariadb-ingress-55794d94c8-5mb54 0/1
Running 0 9m
ucp mariadb-ingress-error-pages-85f96fbd-qzm8g 1/1
Running 0 9m
ucp mariadb-server-0 0/1
Running 0 9m <=================
ucp postgresql-0 1/1
Running 1 22m
... clip ...
... clip ...
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller [-]
[chart=ucp-mariadb]: Error while installing release airship-ucp-mariadb:
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "release airship-ucp-mariadb failed: timed out waiting
for the condition"
debug_error_string =
"{"created":"@1558584756.634751023","description":"Error received from
peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release
airship-ucp-mariadb failed: timed out waiting for the
condition","grpc_status":2}"
>
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller Traceback (most
recent call last):
2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller File
"/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line
460, in install_release
... clip ...
======= root at n0:~/deploy# kubectl describe pod mariadb-server-0 -n ucp
=====================
..clip....
Containers:
mariadb:
Container ID:
docker://6888ba24aacc946e92f2651540667cb4211c01f0232b66e194746cce1c051e95
Image: docker.io/openstackhelm/mariadb:10.2.18
Image ID:
docker-pullable://openstackhelm/mariadb@sha256:e4fe4898baac370f56d781541456d32aa0f8c740f1
8d5b13882a86e80a6a67bb
Ports: 3306/TCP, 4567/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/tmp/start.py
State: Running
Started: Thu, 23 May 2019 05:12:47 +0000
Ready: False
Restart Count: 0
Readiness: exec [/tmp/readiness.sh] delay=30s timeout=3s
period=30s #success=1 #failure=3
Environment:
POD_NAMESPACE: ucp (v1:metadata.namespace)
MARIADB_REPLICAS: 1
POD_NAME_PREFIX: mariadb-server
DISCOVERY_DOMAIN: mariadb-discovery.ucp.svc.cluster.local
DIRECT_SVC_NAME: mariadb-server
WSREP_PORT: 4567
STATE_CONFIGMAP: airship-ucp-mariadb-mariadb-state
MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret
'mariadb-db-root-password'> Opt ional: false
Mounts:
/etc/mysql/admin_user.cnf from mariadb-secrets (ro)
/etc/mysql/conf.d from mycnfd (rw)
/etc/mysql/conf.d/00-base.cnf from mariadb-etc (ro)
/etc/mysql/conf.d/20-override.cnf from mariadb-etc (ro)
/etc/mysql/conf.d/99-force.cnf from mariadb-etc (ro)
/etc/mysql/my.cnf from mariadb-etc (ro)
/tmp/readiness.sh from mariadb-bin (ro)
/tmp/start.py from mariadb-bin (ro)
/tmp/stop.sh from mariadb-bin (ro)
/var/lib/mysql from mysql-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from
airship-ucp-mariadb-mariadb-token-ml52f (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
... clip ....
Events:
... clip ...
Warning Unhealthy 7s (x3 over 1m) kubelet, n0
Readiness probe failed: <========
On Thu, May 23, 2019 at 12:13 AM Evgeny L <eli at mirantis.com> wrote:
> I have seen other people reporting similar problem, it looks as an
> environment specific problem (we don't see it in our gates), also it may be
> related [1][2] to nfs-provisioner that we use for airship in a bottle
> installation.
>
> [1] https://review.opendev.org/#/c/660544/
> [2] https://review.opendev.org/#/c/659787/
>
> On Wed, May 22, 2019 at 6:53 AM MCEUEN, MATT <MM9745 at att.com> wrote:
>
>> Hi Calvin,
>>
>>
>>
>> Have you taken a look for the root cause of why mariadb is not ready,
>> e.g. as Roman suggested, running `kubectl -n ucp describe pod
>> mariadb-server-0` on the genenesis node?
>>
>>
>>
>> Are you running at the minimum 20GB or RAM? I’m wondering whether
>> airship-in-a-bottle has outgrown that. If you’re able to run with
>> additional resources it would be good to see if that changes the result.
>>
>>
>>
>> We don’t have a different QA mailing list, but you’re welcome to join the
>> #airshipit channel on Freenode IRC as well and share logs / ask questions
>> there!
>>
>>
>>
>> Thanks,
>>
>> Matt
>>
>>
>>
>> *From:* calvin whole <calvinwhole at gmail.com>
>> *Sent:* Wednesday, May 22, 2019 3:51 AM
>> *To:* airship-discuss at lists.airshipit.org; calvin whole <
>> calvinwhole at gmail.com>; Roman Gorshunov <paye600 at gmail.com>
>> *Subject:* [Airship-discuss] Airship-in-a-bottle (AIAB) deployment
>> question
>>
>>
>>
>> Hi,
>>
>>
>>
>> We have tried to deploy AIAB by both methods described in the below
>> available info from Airship site.
>>
>>
>>
>> Both methods failed to complete the deployment process and the result is
>> also different.
>>
>> 1. With the "./airship-*in*-a-bottle.sh" method, it stuck in "mariadb
>> not ready" state.
>>
>>
>>
>> 2. With the "vagrant up" method, we tried the libvirt and virtualbox
>> providers.
>>
>> With virtualbox provider "sometimes" we can reach the "Genesis complete"
>> step, but the Openstack step still failed to complete.
>>
>> With libvirt provider we always stuck in "mariadb not ready" or
>> "deckhand-db-init".
>>
>>
>>
>> So it seems to us that the execution output is sensitive to the
>> virtualization environment.
>>
>> For this reason, the documentation below may need to provide more details
>> about the required environment setup.
>>
>>
>>
>> Can someone provide us a more detailed info about AIAB execution
>> requirement ?
>>
>>
>>
>> Or, do Airship have a QA group that we can contact for help?
>>
>>
>>
>> Many thanks,
>>
>> Calvin
>>
>>
>>
>>
>> ===============================================================================
>>
>> To get started, run the following in a fresh Ubuntu 16.04 VM (minimum
>> 4vCPU/20GB RAM/32GB disk). This will deploy Airship and Openstack Helm
>> (OSH):
>>
>> sudo -*i*
>>
>> mkdir -*p* /root/deploy && cd "$_"
>>
>> git clone https:*//git.openstack.org/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=http-3A__git.openstack.org_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=d9jeVVrvEr_lhTLcsBqIWqiP5hYeZ2jLW8-wQI1f78U&e=>*
>>
>> cd /root/deploy/airship-*in*-a-bottle/manifests/dev_single_node
>>
>> ./airship-*in*-a-bottle.sh
>>
>> Or, alternatively, if you have Vagrant installed, just run the following
>> (only libvirt/kvm hypervisor is tested, but vagrant box supports VMware
>> Desktop/Workstation/Fusion, Parallels, and Hyper-V):
>>
>> curl -O https://git.airshipit.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__t.airshipit.org&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=sNKkg6eRk5lZROaz1CJncbtvoUjbndvnPeUN-_CNQnw&e=>/cgit/airship-*in*-a-bottle/plain/Vagrantfile
>>
>> vagrant up
>>
>> _______________________________________________
>> Airship-discuss mailing list
>> Airship-discuss at lists.airshipit.org
>> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190523/44cefc56/attachment-0001.html>
More information about the Airship-discuss
mailing list