[Airship-discuss] Airship-in-a-bottle (AIAB) deployment question

calvin whole calvinwhole at gmail.com
Thu May 23 05:59:35 UTC 2019


Hi Matt and All,

Continue my last mail with additional log output.  Hope this helps.

Thanks,
Calvin

========== root at n0:~/deploy# kubectl logs -n ucp mariadb-server-0
==============
2019-05-23 05:35:48,939 - OpenStack-Helm Mariadb - INFO - This instance
hostname: mariadb-server-0
2019-05-23 05:35:48,939 - OpenStack-Helm Mariadb - INFO - This instance
number: 0
2019-05-23 05:35:49,046 - OpenStack-Helm Mariadb - INFO - Kubernetes API
Version: v1.10.2
2019-05-23 05:35:49,050 - OpenStack-Helm Mariadb - INFO - Will use
"airship-ucp-mariadb-mariadb-state" configmap for cluster state info
2019-05-23 05:35:49,050 - OpenStack-Helm Mariadb - INFO - Getting cluster
state
2019-05-23 05:35:49,065 - OpenStack-Helm Mariadb - INFO - The cluster is
currently in "init" state.
2019-05-23 05:35:49,065 - OpenStack-Helm Mariadb - INFO - Getting cluster
state
2019-05-23 05:35:49,070 - OpenStack-Helm Mariadb - INFO - The cluster is
currently in "init" state.
2019-05-23 05:35:49,070 - OpenStack-Helm Mariadb - INFO - Waiting for
cluster to start running
...

=========== root at n0:~/deploy# kubectl logs -n kube-system
nfs-provisioner-7799d64d59-zwbvk ==============
I0523 04:02:19.342442       1 main.go:65] Provisioner
nfs/airship-nfs-provisioner specified
I0523 04:02:19.342589       1 main.go:85] Starting NFS server!
I0523 04:02:19.770374       1 server.go:139] starting RLIMIT_NOFILE
rlimit.Cur 1048576, rlimit.Max 1048576
I0523 04:02:19.770455       1 server.go:150] ending RLIMIT_NOFILE
rlimit.Cur 1048576, rlimit.Max 1048576
I0523 04:02:19.870411       1 controller.go:392] Starting provisioner
controller 8f995b13-7d0f-11e9-b4cf-fe46167aea67!
I0523 04:02:26.531687       1 controller.go:1052]
scheduleOperation[lock-provision-ucp/postgresql-data-postgresql-0[93717e25-7d0f-11e9-8fc6-525400f8f6f1]]
I0523 04:02:27.169375       1 leaderelection.go:154] attempting to acquire
leader lease...
I0523 04:02:27.483671       1 controller.go:1052]
scheduleOperation[lock-provision-ucp/postgresql-data-postgresql-0[93717e25-7d0f-11e9-8fc6-525400f8f6f1]]
I0523 04:02:27.734209       1 controller.go:1052]
scheduleOperation[lock-provision-ucp/postgresql-data-postgresql-0[93717e25-7d0f-11e9-8fc6-525400f8f6f1]]
E0523 04:02:27.734231       1 leaderelection.go:271] Failed to update lock:
Operation cannot be fulfilled on persistentvolumeclaims
"postgresql-data-postgresql-0": the object has been modified; please apply
your changes to the latest version and try again
...
E0523 04:02:40.024023       1 leaderelection.go:271] Failed to update lock:
Operation cannot be fulfilled on persistentvolumeclaims
"postgresql-data-postgresql-0": the object has been modified; please apply
your changes to the latest version and try again
...
E0523 04:02:43.328965       1 leaderelection.go:271] Failed to update lock:
Operation cannot be fulfilled on persistentvolumeclaims
"mysql-data-mariadb-server-0": the object has been modified; please apply
your changes to the latest version and try again



On Thu, May 23, 2019 at 1:33 PM calvin whole <calvinwhole at gmail.com> wrote:

> Hi Matt and All,
>
> The resources are not the problem. My ubuntu 16.04 VM has 8vcpu, 24G
> memory, and 40G disk.
>
> The symptom is that mariadb-server-0 is stuck in "running" state but never
> getting into the "ready" state.
>
> =========== output =====================
> ...clip...
> kube-system   nfs-provisioner-7799d64d59-zwbvk             1/1
> Running     0          23m
> ucp           airship-ucp-rabbitmq-rabbitmq-0              1/1
> Running     0          22m
> ucp           ingress-86576d6599-gtgmn                     1/1
> Running     0          22m
> ucp           ingress-error-pages-5c97bb46bb-jm8zz         1/1
> Running     0          22m
> ucp           mariadb-ingress-55794d94c8-5cb97             0/1
> Running     0          9m
> ucp           mariadb-ingress-55794d94c8-5mb54             0/1
> Running     0          9m
> ucp           mariadb-ingress-error-pages-85f96fbd-qzm8g   1/1
> Running     0          9m
> ucp           mariadb-server-0                             0/1
> Running     0          9m   <=================
> ucp           postgresql-0                                 1/1
> Running     1          22m
> ... clip ...
>
> ... clip ...
> 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller [-]
> [chart=ucp-mariadb]: Error while installing release airship-ucp-mariadb:
> grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
>         status = StatusCode.UNKNOWN
>         details = "release airship-ucp-mariadb failed: timed out waiting
> for the condition"
>         debug_error_string =
> "{"created":"@1558584756.634751023","description":"Error received from
> peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release
> airship-ucp-mariadb failed: timed out waiting for the
> condition","grpc_status":2}"
> >
> 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller Traceback (most
> recent call last):
> 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller   File
> "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line
> 460, in install_release
> ... clip ...
>
> ======= root at n0:~/deploy# kubectl describe pod mariadb-server-0 -n ucp
> =====================
> ..clip....
>
> Containers:
>   mariadb:
>     Container ID:
>  docker://6888ba24aacc946e92f2651540667cb4211c01f0232b66e194746cce1c051e95
>     Image:         docker.io/openstackhelm/mariadb:10.2.18
>     Image ID:      docker-pullable://openstackhelm/mariadb@sha256:e4fe4898baac370f56d781541456d32aa0f8c740f1
>                          8d5b13882a86e80a6a67bb
>     Ports:         3306/TCP, 4567/TCP
>     Host Ports:    0/TCP, 0/TCP
>     Command:
>       /tmp/start.py
>     State:          Running
>       Started:      Thu, 23 May 2019 05:12:47 +0000
>     Ready:          False
>     Restart Count:  0
>     Readiness:      exec [/tmp/readiness.sh] delay=30s timeout=3s
> period=30s #success=1 #failure=3
>     Environment:
>       POD_NAMESPACE:        ucp (v1:metadata.namespace)
>       MARIADB_REPLICAS:     1
>       POD_NAME_PREFIX:      mariadb-server
>       DISCOVERY_DOMAIN:     mariadb-discovery.ucp.svc.cluster.local
>       DIRECT_SVC_NAME:      mariadb-server
>       WSREP_PORT:           4567
>       STATE_CONFIGMAP:      airship-ucp-mariadb-mariadb-state
>       MYSQL_ROOT_PASSWORD:  <set to the key 'MYSQL_ROOT_PASSWORD' in
> secret 'mariadb-db-root-password'>  Opt                          ional:
> false
>     Mounts:
>       /etc/mysql/admin_user.cnf from mariadb-secrets (ro)
>       /etc/mysql/conf.d from mycnfd (rw)
>       /etc/mysql/conf.d/00-base.cnf from mariadb-etc (ro)
>       /etc/mysql/conf.d/20-override.cnf from mariadb-etc (ro)
>       /etc/mysql/conf.d/99-force.cnf from mariadb-etc (ro)
>       /etc/mysql/my.cnf from mariadb-etc (ro)
>       /tmp/readiness.sh from mariadb-bin (ro)
>       /tmp/start.py from mariadb-bin (ro)
>       /tmp/stop.sh from mariadb-bin (ro)
>       /var/lib/mysql from mysql-data (rw)
>       /var/run/secrets/kubernetes.io/serviceaccount from
> airship-ucp-mariadb-mariadb-token-ml52f (ro)
> Conditions:
>   Type           Status
>   Initialized    True
>   Ready          False
>   PodScheduled   True
> ... clip ....
> Events:
> ... clip ...
>   Warning  Unhealthy              7s (x3 over 1m)  kubelet, n0
>  Readiness probe failed:   <========
>
> On Thu, May 23, 2019 at 12:13 AM Evgeny L <eli at mirantis.com> wrote:
>
>> I have seen other people reporting similar problem, it looks as an
>> environment specific problem (we don't see it in our gates), also it may be
>> related [1][2] to nfs-provisioner that we use for airship in a bottle
>> installation.
>>
>> [1] https://review.opendev.org/#/c/660544/
>> [2] https://review.opendev.org/#/c/659787/
>>
>> On Wed, May 22, 2019 at 6:53 AM MCEUEN, MATT <MM9745 at att.com> wrote:
>>
>>> Hi Calvin,
>>>
>>>
>>>
>>> Have you taken a look for the root cause of why mariadb is not ready,
>>> e.g. as Roman suggested, running `kubectl -n ucp describe pod
>>> mariadb-server-0` on the genenesis node?
>>>
>>>
>>>
>>> Are you running at the minimum 20GB or RAM?  I’m wondering whether
>>> airship-in-a-bottle has outgrown that.  If you’re able to run with
>>> additional resources it would be good to see if that changes the result.
>>>
>>>
>>>
>>> We don’t have a different QA mailing list, but you’re welcome to join
>>> the #airshipit channel on Freenode IRC as well and share logs / ask
>>> questions there!
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>
>>> *From:* calvin whole <calvinwhole at gmail.com>
>>> *Sent:* Wednesday, May 22, 2019 3:51 AM
>>> *To:* airship-discuss at lists.airshipit.org; calvin whole <
>>> calvinwhole at gmail.com>; Roman Gorshunov <paye600 at gmail.com>
>>> *Subject:* [Airship-discuss] Airship-in-a-bottle (AIAB) deployment
>>> question
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We have tried to deploy AIAB by both methods described in the below
>>> available info from Airship site.
>>>
>>>
>>>
>>> Both methods failed to complete the deployment process and the result is
>>> also different.
>>>
>>> 1. With the "./airship-*in*-a-bottle.sh" method, it stuck in "mariadb
>>> not ready" state.
>>>
>>>
>>>
>>> 2. With the "vagrant up" method, we tried the libvirt and virtualbox
>>> providers.
>>>
>>> With virtualbox provider "sometimes" we can reach the "Genesis complete"
>>> step, but the Openstack step still failed to complete.
>>>
>>> With libvirt provider we always stuck in "mariadb not ready" or
>>> "deckhand-db-init".
>>>
>>>
>>>
>>> So it seems to us that the execution output is sensitive to the
>>> virtualization environment.
>>>
>>> For this reason, the documentation below may need to provide more
>>> details about the required environment setup.
>>>
>>>
>>>
>>> Can someone provide us a more detailed info about AIAB execution
>>> requirement ?
>>>
>>>
>>>
>>> Or, do Airship have a QA group that we can contact for help?
>>>
>>>
>>>
>>> Many thanks,
>>>
>>> Calvin
>>>
>>>
>>>
>>>
>>> ===============================================================================
>>>
>>> To get started, run the following in a fresh Ubuntu 16.04 VM (minimum
>>> 4vCPU/20GB RAM/32GB disk). This will deploy Airship and Openstack Helm
>>> (OSH):
>>>
>>> sudo -*i*
>>>
>>> mkdir -*p* /root/deploy && cd "$_"
>>>
>>> git clone https:*//git.openstack.org/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=http-3A__git.openstack.org_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=d9jeVVrvEr_lhTLcsBqIWqiP5hYeZ2jLW8-wQI1f78U&e=>*
>>>
>>> cd /root/deploy/airship-*in*-a-bottle/manifests/dev_single_node
>>>
>>> ./airship-*in*-a-bottle.sh
>>>
>>> Or, alternatively, if you have Vagrant installed, just run the following
>>> (only libvirt/kvm hypervisor is tested, but vagrant box supports VMware
>>> Desktop/Workstation/Fusion, Parallels, and Hyper-V):
>>>
>>> curl -O https://git.airshipit.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__t.airshipit.org&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=sNKkg6eRk5lZROaz1CJncbtvoUjbndvnPeUN-_CNQnw&e=>/cgit/airship-*in*-a-bottle/plain/Vagrantfile
>>>
>>> vagrant up
>>>
>>> _______________________________________________
>>> Airship-discuss mailing list
>>> Airship-discuss at lists.airshipit.org
>>> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190523/6bf66e2a/attachment-0001.html>


More information about the Airship-discuss mailing list