[Airship-discuss] Airship-in-a-bottle (AIAB) deployment question
calvin whole
calvinwhole at gmail.com
Thu May 23 05:59:35 UTC 2019
Hi Matt and All,
Continue my last mail with additional log output. Hope this helps.
Thanks,
Calvin
========== root at n0:~/deploy# kubectl logs -n ucp mariadb-server-0
==============
2019-05-23 05:35:48,939 - OpenStack-Helm Mariadb - INFO - This instance
hostname: mariadb-server-0
2019-05-23 05:35:48,939 - OpenStack-Helm Mariadb - INFO - This instance
number: 0
2019-05-23 05:35:49,046 - OpenStack-Helm Mariadb - INFO - Kubernetes API
Version: v1.10.2
2019-05-23 05:35:49,050 - OpenStack-Helm Mariadb - INFO - Will use
"airship-ucp-mariadb-mariadb-state" configmap for cluster state info
2019-05-23 05:35:49,050 - OpenStack-Helm Mariadb - INFO - Getting cluster
state
2019-05-23 05:35:49,065 - OpenStack-Helm Mariadb - INFO - The cluster is
currently in "init" state.
2019-05-23 05:35:49,065 - OpenStack-Helm Mariadb - INFO - Getting cluster
state
2019-05-23 05:35:49,070 - OpenStack-Helm Mariadb - INFO - The cluster is
currently in "init" state.
2019-05-23 05:35:49,070 - OpenStack-Helm Mariadb - INFO - Waiting for
cluster to start running
...
=========== root at n0:~/deploy# kubectl logs -n kube-system
nfs-provisioner-7799d64d59-zwbvk ==============
I0523 04:02:19.342442 1 main.go:65] Provisioner
nfs/airship-nfs-provisioner specified
I0523 04:02:19.342589 1 main.go:85] Starting NFS server!
I0523 04:02:19.770374 1 server.go:139] starting RLIMIT_NOFILE
rlimit.Cur 1048576, rlimit.Max 1048576
I0523 04:02:19.770455 1 server.go:150] ending RLIMIT_NOFILE
rlimit.Cur 1048576, rlimit.Max 1048576
I0523 04:02:19.870411 1 controller.go:392] Starting provisioner
controller 8f995b13-7d0f-11e9-b4cf-fe46167aea67!
I0523 04:02:26.531687 1 controller.go:1052]
scheduleOperation[lock-provision-ucp/postgresql-data-postgresql-0[93717e25-7d0f-11e9-8fc6-525400f8f6f1]]
I0523 04:02:27.169375 1 leaderelection.go:154] attempting to acquire
leader lease...
I0523 04:02:27.483671 1 controller.go:1052]
scheduleOperation[lock-provision-ucp/postgresql-data-postgresql-0[93717e25-7d0f-11e9-8fc6-525400f8f6f1]]
I0523 04:02:27.734209 1 controller.go:1052]
scheduleOperation[lock-provision-ucp/postgresql-data-postgresql-0[93717e25-7d0f-11e9-8fc6-525400f8f6f1]]
E0523 04:02:27.734231 1 leaderelection.go:271] Failed to update lock:
Operation cannot be fulfilled on persistentvolumeclaims
"postgresql-data-postgresql-0": the object has been modified; please apply
your changes to the latest version and try again
...
E0523 04:02:40.024023 1 leaderelection.go:271] Failed to update lock:
Operation cannot be fulfilled on persistentvolumeclaims
"postgresql-data-postgresql-0": the object has been modified; please apply
your changes to the latest version and try again
...
E0523 04:02:43.328965 1 leaderelection.go:271] Failed to update lock:
Operation cannot be fulfilled on persistentvolumeclaims
"mysql-data-mariadb-server-0": the object has been modified; please apply
your changes to the latest version and try again
On Thu, May 23, 2019 at 1:33 PM calvin whole <calvinwhole at gmail.com> wrote:
> Hi Matt and All,
>
> The resources are not the problem. My ubuntu 16.04 VM has 8vcpu, 24G
> memory, and 40G disk.
>
> The symptom is that mariadb-server-0 is stuck in "running" state but never
> getting into the "ready" state.
>
> =========== output =====================
> ...clip...
> kube-system nfs-provisioner-7799d64d59-zwbvk 1/1
> Running 0 23m
> ucp airship-ucp-rabbitmq-rabbitmq-0 1/1
> Running 0 22m
> ucp ingress-86576d6599-gtgmn 1/1
> Running 0 22m
> ucp ingress-error-pages-5c97bb46bb-jm8zz 1/1
> Running 0 22m
> ucp mariadb-ingress-55794d94c8-5cb97 0/1
> Running 0 9m
> ucp mariadb-ingress-55794d94c8-5mb54 0/1
> Running 0 9m
> ucp mariadb-ingress-error-pages-85f96fbd-qzm8g 1/1
> Running 0 9m
> ucp mariadb-server-0 0/1
> Running 0 9m <=================
> ucp postgresql-0 1/1
> Running 1 22m
> ... clip ...
>
> ... clip ...
> 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller [-]
> [chart=ucp-mariadb]: Error while installing release airship-ucp-mariadb:
> grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> status = StatusCode.UNKNOWN
> details = "release airship-ucp-mariadb failed: timed out waiting
> for the condition"
> debug_error_string =
> "{"created":"@1558584756.634751023","description":"Error received from
> peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release
> airship-ucp-mariadb failed: timed out waiting for the
> condition","grpc_status":2}"
> >
> 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller Traceback (most
> recent call last):
> 2019-05-23 04:12:36.635 8 ERROR armada.handlers.tiller File
> "/usr/local/lib/python3.6/site-packages/armada/handlers/tiller.py", line
> 460, in install_release
> ... clip ...
>
> ======= root at n0:~/deploy# kubectl describe pod mariadb-server-0 -n ucp
> =====================
> ..clip....
>
> Containers:
> mariadb:
> Container ID:
> docker://6888ba24aacc946e92f2651540667cb4211c01f0232b66e194746cce1c051e95
> Image: docker.io/openstackhelm/mariadb:10.2.18
> Image ID: docker-pullable://openstackhelm/mariadb@sha256:e4fe4898baac370f56d781541456d32aa0f8c740f1
> 8d5b13882a86e80a6a67bb
> Ports: 3306/TCP, 4567/TCP
> Host Ports: 0/TCP, 0/TCP
> Command:
> /tmp/start.py
> State: Running
> Started: Thu, 23 May 2019 05:12:47 +0000
> Ready: False
> Restart Count: 0
> Readiness: exec [/tmp/readiness.sh] delay=30s timeout=3s
> period=30s #success=1 #failure=3
> Environment:
> POD_NAMESPACE: ucp (v1:metadata.namespace)
> MARIADB_REPLICAS: 1
> POD_NAME_PREFIX: mariadb-server
> DISCOVERY_DOMAIN: mariadb-discovery.ucp.svc.cluster.local
> DIRECT_SVC_NAME: mariadb-server
> WSREP_PORT: 4567
> STATE_CONFIGMAP: airship-ucp-mariadb-mariadb-state
> MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in
> secret 'mariadb-db-root-password'> Opt ional:
> false
> Mounts:
> /etc/mysql/admin_user.cnf from mariadb-secrets (ro)
> /etc/mysql/conf.d from mycnfd (rw)
> /etc/mysql/conf.d/00-base.cnf from mariadb-etc (ro)
> /etc/mysql/conf.d/20-override.cnf from mariadb-etc (ro)
> /etc/mysql/conf.d/99-force.cnf from mariadb-etc (ro)
> /etc/mysql/my.cnf from mariadb-etc (ro)
> /tmp/readiness.sh from mariadb-bin (ro)
> /tmp/start.py from mariadb-bin (ro)
> /tmp/stop.sh from mariadb-bin (ro)
> /var/lib/mysql from mysql-data (rw)
> /var/run/secrets/kubernetes.io/serviceaccount from
> airship-ucp-mariadb-mariadb-token-ml52f (ro)
> Conditions:
> Type Status
> Initialized True
> Ready False
> PodScheduled True
> ... clip ....
> Events:
> ... clip ...
> Warning Unhealthy 7s (x3 over 1m) kubelet, n0
> Readiness probe failed: <========
>
> On Thu, May 23, 2019 at 12:13 AM Evgeny L <eli at mirantis.com> wrote:
>
>> I have seen other people reporting similar problem, it looks as an
>> environment specific problem (we don't see it in our gates), also it may be
>> related [1][2] to nfs-provisioner that we use for airship in a bottle
>> installation.
>>
>> [1] https://review.opendev.org/#/c/660544/
>> [2] https://review.opendev.org/#/c/659787/
>>
>> On Wed, May 22, 2019 at 6:53 AM MCEUEN, MATT <MM9745 at att.com> wrote:
>>
>>> Hi Calvin,
>>>
>>>
>>>
>>> Have you taken a look for the root cause of why mariadb is not ready,
>>> e.g. as Roman suggested, running `kubectl -n ucp describe pod
>>> mariadb-server-0` on the genenesis node?
>>>
>>>
>>>
>>> Are you running at the minimum 20GB or RAM? I’m wondering whether
>>> airship-in-a-bottle has outgrown that. If you’re able to run with
>>> additional resources it would be good to see if that changes the result.
>>>
>>>
>>>
>>> We don’t have a different QA mailing list, but you’re welcome to join
>>> the #airshipit channel on Freenode IRC as well and share logs / ask
>>> questions there!
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>
>>> *From:* calvin whole <calvinwhole at gmail.com>
>>> *Sent:* Wednesday, May 22, 2019 3:51 AM
>>> *To:* airship-discuss at lists.airshipit.org; calvin whole <
>>> calvinwhole at gmail.com>; Roman Gorshunov <paye600 at gmail.com>
>>> *Subject:* [Airship-discuss] Airship-in-a-bottle (AIAB) deployment
>>> question
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We have tried to deploy AIAB by both methods described in the below
>>> available info from Airship site.
>>>
>>>
>>>
>>> Both methods failed to complete the deployment process and the result is
>>> also different.
>>>
>>> 1. With the "./airship-*in*-a-bottle.sh" method, it stuck in "mariadb
>>> not ready" state.
>>>
>>>
>>>
>>> 2. With the "vagrant up" method, we tried the libvirt and virtualbox
>>> providers.
>>>
>>> With virtualbox provider "sometimes" we can reach the "Genesis complete"
>>> step, but the Openstack step still failed to complete.
>>>
>>> With libvirt provider we always stuck in "mariadb not ready" or
>>> "deckhand-db-init".
>>>
>>>
>>>
>>> So it seems to us that the execution output is sensitive to the
>>> virtualization environment.
>>>
>>> For this reason, the documentation below may need to provide more
>>> details about the required environment setup.
>>>
>>>
>>>
>>> Can someone provide us a more detailed info about AIAB execution
>>> requirement ?
>>>
>>>
>>>
>>> Or, do Airship have a QA group that we can contact for help?
>>>
>>>
>>>
>>> Many thanks,
>>>
>>> Calvin
>>>
>>>
>>>
>>>
>>> ===============================================================================
>>>
>>> To get started, run the following in a fresh Ubuntu 16.04 VM (minimum
>>> 4vCPU/20GB RAM/32GB disk). This will deploy Airship and Openstack Helm
>>> (OSH):
>>>
>>> sudo -*i*
>>>
>>> mkdir -*p* /root/deploy && cd "$_"
>>>
>>> git clone https:*//git.openstack.org/openstack/airship-in-a-bottle <https://urldefense.proofpoint.com/v2/url?u=http-3A__git.openstack.org_openstack_airship-2Din-2Da-2Dbottle&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=d9jeVVrvEr_lhTLcsBqIWqiP5hYeZ2jLW8-wQI1f78U&e=>*
>>>
>>> cd /root/deploy/airship-*in*-a-bottle/manifests/dev_single_node
>>>
>>> ./airship-*in*-a-bottle.sh
>>>
>>> Or, alternatively, if you have Vagrant installed, just run the following
>>> (only libvirt/kvm hypervisor is tested, but vagrant box supports VMware
>>> Desktop/Workstation/Fusion, Parallels, and Hyper-V):
>>>
>>> curl -O https://git.airshipit.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__t.airshipit.org&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qGH7CH6vJpqo2zaDk0Dy-jyJ-D4Mp9s_IURAu8KXIjQ&s=sNKkg6eRk5lZROaz1CJncbtvoUjbndvnPeUN-_CNQnw&e=>/cgit/airship-*in*-a-bottle/plain/Vagrantfile
>>>
>>> vagrant up
>>>
>>> _______________________________________________
>>> Airship-discuss mailing list
>>> Airship-discuss at lists.airshipit.org
>>> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190523/6bf66e2a/attachment-0001.html>
More information about the Airship-discuss
mailing list