Re: [Airship-discuss] airship-in-a-bottle deployment issue
Hello Calvin, Seems like PosgreSQL database was not able to properly write data onto the disk. PostgreSQL runs as a postgresql-0 pod in ucp namespace, uses a persistent volume claim postgresql-data-postgresql-0, and persistent volume mounted via NFS. kubectl describe pod postgresql-0 -n ucp kubectl logs -n ucp postgresql-0 kubectl -n ucp describe pvc postgresql-data-postgresql-0 kubectl describe pv pvc-0382c985-7572-11e9-b431-525400681552 # volume name could be different) NFS is provisioned by nfs-provisioner-7799d64d59-ptsgk (last two parts would be different in your case): kubectl get pods -n kube-system | grep nfs kubectl -n kube-system describe pod nfs-provisioner-7799d64d59-ptsgk kubectl -n kube-system logs nfs-provisioner-7799d64d59-ptsgk Check if there are any problems with it (e.g. unable to mount NFS share, or lack of free storage space - `df -h`). Also running kubectl get events --all-namespaces could help to understand what went wrong. I have run an AIAB installation today twice, and it all worked fine. I use `vagrant up` and my hypervisor is KVM, if that could help you. I hope it helps. Best regards, -- Roman Gorshunov On Fri, May 10, 2019 at 7:01 AM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Not sure if my last email were out properly, its size is too big. Here is a short one. Thanks for responding in advance.
I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below. It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
On Thu, May 9, 2019 at 12:17 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Btw, continue my last post, the kubectl describe pod deckhand-db-init-zs499 output is as follows.
Thanks, Calvin =========== kubectl describe pod deckhand-db-init-zs499 ================= root@n0:/home/vagrant# kubectl describe pod deckhand-db-init-zs499 -n ucp Name: deckhand-db-init-zs499 Namespace: ucp Node: n0/10.0.2.15 Start Time: Thu, 09 May 2019 03:48:48 +0000 Labels: application=deckhand component=db-init controller-uid=59f1bee0-720d-11e9-92ac-080027fc876e job-name=deckhand-db-init release_group=airship-ucp-deckhand Annotations: <none> Status: Running IP: 10.97.26.50 Controlled By: Job/deckhand-db-init Init Containers: init: Container ID: docker://b58e8b6b7296df618cb8120b5226370afeba2a4e79dd70ee6894b5afd853c0db Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6 Port: <none> Host Port: <none> Command: kubernetes-entrypoint State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 09 May 2019 03:48:52 +0000 Finished: Thu, 09 May 2019 03:48:54 +0000 Ready: True Restart Count: 0 Environment: POD_NAME: deckhand-db-init-zs499 (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: ucp:postgresql DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Containers: deckhand-db-init: Container ID: docker://5dea2aa975c3718ca298536005b9cc0b21de47e08b2260cc73005e3455bb1350 Image: docker.io/postgres:9.5 Image ID: docker-pullable://postgres@sha256:0605b4b20a205c09ddd10eeeddd3ed7bf3cc442a8e9896ec34862ca882658be4 Port: <none> Host Port: <none> Command: /tmp/db-init.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error <======== Exit Code: 1 Started: Thu, 09 May 2019 04:10:29 +0000 Finished: Thu, 09 May 2019 04:10:30 +0000 Ready: False Restart Count: 9 Environment: DECKHAND_DB_URL: <set to the key 'DATABASE_URI' in secret 'deckhand-db-user'> Optional: false DB_NAME: <set to the key 'DATABASE_NAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_PASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-user'> Optional: false DB_FQDN: <set to the key 'DATABASE_HOST' in secret 'deckhand-db-user'> Optional: false DB_PORT: <set to the key 'DATABASE_PORT' in secret 'deckhand-db-user'> Optional: false DB_ADMIN_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-admin'> Optional: false PGPASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-admin'> Optional: false Mounts: /etc/deckhand from etc-deckhand (rw) /etc/deckhand/deckhand.conf from deckhand-etc (ro) /tmp/db-init.sh from deckhand-bin (ro) /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: etc-deckhand: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: deckhand-etc: Type: Secret (a volume populated by a Secret) SecretName: deckhand-etc Optional: false deckhand-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: deckhand-bin Optional: false deckhand-db-init-token-gczr5: Type: Secret (a volume populated by a Secret) SecretName: deckhand-db-init-token-gczr5 Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 24m default-scheduler Successfully assigned deckhand-db-init-zs499 to n0 Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "etc-deckhand" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-bin" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-etc" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-db-init-token-gczr5" Normal Pulled 24m kubelet, n0 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine Normal Created 24m kubelet, n0 Created container Normal Started 24m kubelet, n0 Started container Normal Pulled 23m (x4 over 24m) kubelet, n0 Container image "docker.io/postgres:9.5" already present on machine Normal Created 23m (x4 over 24m) kubelet, n0 Created container Normal Started 23m (x4 over 24m) kubelet, n0 Started container Warning BackOff 4m (x90 over 24m) kubelet, n0 Back-off restarting failed container root@n0:/home/vagrant#
On Thu, May 9, 2019 at 12:08 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Thanks for looking into this and gave us suggestions.
I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below. It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
================== log for deckhand-db-init-zs499 ================================== root@n0:/home/vagrant# kubectl logs deckhand-db-init-zs499 -n ucp + export HOME=/tmp + HOME=/tmp + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + grep -q 1 + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + pgsql_superuser_cmd 'CREATE DATABASE deckhand' + DB_COMMAND='CREATE DATABASE deckhand' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE deckhand' ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
===================================================================================== ==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 49m ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 49m ==> n0: kube-system calico-etcd-anchor-ncl2p 1/1 Running 0 47m ==> n0: kube-system calico-etcd-n0 1/1 Running 0 46m ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5csnn 1/1 Running 0 46m ==> n0: kube-system calico-node-m4rtf 1/1 Running 0 46m ==> n0: kube-system calico-settings-tkp6r 0/1 Completed 0 46m ==> n0: kube-system coredns-84bdd76f4d-hhbcs 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-k8tcc 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-qp2xd 1/1 Running 0 44m ==> n0: kube-system haproxy-n0 1/1 Running 0 50m ==> n0: kube-system ingress-error-pages-7c65f766d-dn2tw 1/1 Running 0 41m ==> n0: kube-system ingress-gtvp8 2/2 Running 0 41m ==> n0: kube-system kubernetes-apiserver-anchor-99jhn 1/1 Running 0 42m ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-controller-manager-anchor-vqddp 1/1 Running 0 42m ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-etcd-anchor-9jcpl 1/1 Running 0 44m ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 42m ==> n0: kube-system kubernetes-proxy-2m9t2 1/1 Running 0 47m ==> n0: kube-system kubernetes-scheduler-anchor-nl9fb 1/1 Running 0 42m ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 41m ==> n0: kube-system nfs-provisioner-7799d64d59-vtkbd 1/1 Running 0 40m ==> n0: kube-system tiller-deploy-7d88c6f956-qwfzb 1/1 Running 0 27m ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-vfl69 1/1 Running 0 34m ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 39m ==> n0: ucp armada-api-d5f757d5-6wl98 1/1 Running 0 15m ==> n0: ucp armada-ks-endpoints-vl9rs 0/3 Completed 0 15m ==> n0: ucp armada-ks-service-vpcjd 0/1 Completed 0 15m ==> n0: ucp armada-ks-user-rv4gs 0/1 Completed 0 15m ==> n0: ucp barbican-api-5d7b88d8ff-8dd6w 1/1 Running 0 13m ==> n0: ucp barbican-db-init-gqvt4 0/1 Completed 0 13m ==> n0: ucp barbican-db-sync-tqtgq 0/1 Completed 0 13m ==> n0: ucp barbican-ks-endpoints-rwtql 0/3 ==> n0: Completed 0 13m ==> n0: ucp barbican-ks-service-l2h6h 0/1 Completed 0 13m ==> n0: ucp barbican-ks-user-wwvc7 0/1 Completed 0 13m ==> n0: ucp barbican-rabbit-init-6spq4 0/1 Completed 0 13m ==> n0: ucp deckhand-api-78b9644f96-5686f 0/1 Running 0 11m ==> n0: ucp deckhand-db-init-zs499 0/1 CrashLoopBackOff 7 11m <=== ==> n0: ucp deckhand-db-sync-ct7wl 0/1 Init:0/1 0 11m ==> n0: ucp deckhand-ks-endpoints-x4hd9 0/3 Completed 0 11m ==> n0: ucp deckhand-ks-service-ms6n5 0/1 Completed 0 11m ==> n0: ucp deckhand-ks-user-7fnvt 0/1 Completed 0 11m ==> n0: ucp divingbell-apparmor-default-hth8z 1/1 Running 0 27m ==> n0: ucp divingbell-apt-default-r965m 1/1 Running 0 27m ==> n0: ucp divingbell-ethtool-default-ldcmc 1/1 Running 0 27m ==> n0: ucp divingbell-exec-default-f7h7x 1/1 Running 0 27m ==> n0: ucp divingbell-limits-default-sp9mj 1/1 Running 0 27m ==> n0: ucp divingbell-mounts-default-8f5a00a2-frbl2 1/1 Running 0 27m ==> n0: ucp divingbell-perm-default-d7wxp 1/1 Running 0 27m ==> n0: ucp divingbell-sysctl-default-c8pnp 1/1 Running 0 27m ==> n0: ucp divingbell-uamlite-default-rfct6 1/1 Running 0 27m ==> n0: ucp ingress-86576d6599-mdgj4 1/1 Running 0 39m ==> n0: ucp ingress-error-pages-5c97bb46bb-7lg5l 1/1 Running 0 39m ==> n0: ucp keystone-api-678fc44bdd-594bb 1/1 Running 0 34m ==> n0: ucp keystone-bootstrap-rprr6 0/1 Completed 0 34m ==> n0: ucp keystone-credential-setup-zkjgs 0/1 Completed 0 34m ==> n0: ucp keystone-db-init-xkgxm 0/1 Completed 0 34m ==> n0: ucp keystone-db-sync-lm6xs 0/1 Completed 0 34m ==> n0: ucp keystone-domain-manage-9pzjq 0/1 Completed 0 34m ==> n0: ucp keystone-fernet-setup-q7t8p 0/1 Completed 0 34m ==> n0: ucp keystone-rabbit-init-qpvgt 0/1 Completed 0 34m ==> n0: ucp maas-bootstrap-admin-user-8npgw 0/1 Completed 0 26m ==> n0: ucp maas-db-init-9z86n 0/1 Completed 0 26m ==> n0: ucp maas-db-sync-r7rkg 0/1 Completed 0 26m ==> n0: ucp maas-export-api-key-n2gz4 0/1 Completed 1 26m ==> n0: ucp maas-import-resources-prlml 0/1 Completed 0 26m ==> n0: ucp maas-ingress-756f6f9d6-h65nj 2/2 Running 0 26m ==> n0: ucp maas-ingress-errors-8686d56d98-swfg9 ==> n0: 1/1 Running 0 26m ==> n0: ucp maas-rack-0 1/1 Running 0 26m ==> n0: ucp maas-region-0 1/1 Running 0 26m ==> n0: ucp mariadb-ingress-55794d94c8-dsw5w 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-55794d94c8-jczmh 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-jrqsg 1/1 Running 0 39m ==> n0: ucp mariadb-server-0 1/1 Running 0 39m ==> n0: ucp postgresql-0 1/1 Running 1 39m
On Tue, May 7, 2019 at 6:25 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello Calvin,
Try to get some kubectl logs and describe deckhand-db-init-r9jvg pod. kubectl describe pod deckhand-db-init-r9jvg -u ucp May be it would help to understand what is happening there.
Thank you for trying Airship.
Best regards, -- Roman Gorshunov
On Tue, May 7, 2019 at 7:51 AM calvin whole <calvinwhole@gmail.com> wrote:
Hi,
We are trying to deploy AIIB.
I have a physical server with Ubuntu 16.04.5 OS, installed virtualbox and vagrant. The process is straightforward by following https://opendev.org/airship/in-a-bottle/ We created ~/deploy directory, downloaded Vagrantfile, and do "vagrant up".
However it stuck in the error below: deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 16
Could anyone help to resolve this? Many thanks in advance.
Sincerely, Calvin
==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 1h ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 1h ==> n0: kube-system calico-etcd-anchor-5tqhk 1/1 Running 0 1h ==> n0: kube-system calico-etcd-n0 1/1 Running 0 1h ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5ssl6 1/1 Running 0 1h ==> n0: kube-system calico-node-pbsxh 1/1 Running 0 1h ==> n0: kube-system calico-settings-lzpk9 0/1 Completed 0 1h ==> n0: kube-system coredns-84bdd76f4d-6cwnl 1/1 Running 0 1h ==> n0: kube-system coredns-84bdd76f4d-d4p8c 1/1 Running 0 1h ==> n0: kube-system coredns-84bdd76f4d-xrknz 1/1 Running 0 1h ==> n0: kube-system haproxy-n0 1/1 Running 0 1h ==> n0: kube-system ingress-9pkmx 2/2 Running 0 1h ==> n0: kube-system ingress-error-pages-7c65f766d-2pqfx 1/1 Running 0 1h ==> n0: kube-system kubernetes-apiserver-anchor-hszbf 1/1 Running 0 1h ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-controller-manager-anchor-h49vz 1/1 Running 0 1h ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-etcd-anchor-nnjbb 1/1 Running 0 1h ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-proxy-vgzjp 1/1 Running 0 1h ==> n0: kube-system kubernetes-scheduler-anchor-bq2gk 1/1 Running 0 1h ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 1h ==> n0: kube-system nfs-provisioner-7799d64d59-jx7hq 1/1 Running 0 1h ==> n0: kube-system tiller-deploy-7d88c6f956-d9kzg 1/1 Running 0 1h ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-q9wqx 1/1 Running 0 1h ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 1h ==> n0: ucp armada-api-d5f757d5-d9l9h 1/1 Running 0 1h ==> n0: ucp armada-ks-endpoints-qwbtg 0/3 Completed 0 1h ==> n0: ucp armada-ks-service-lg8kq 0/1 Completed 0 1h ==> n0: ucp armada-ks-user-g2j6v 0/1 Completed 0 1h ==> n0: ucp barbican-api-84665dd99d-qv5fz 1/1 Running 0 1h ==> n0: ucp barbican-db-init-ndx58 0/1 Completed 0 1h ==> n0: ucp barbican-db-sync-sh7c9 0/1 Completed 0 1h ==> n0: ucp barbican-ks-endpoints-bv7xv 0/3 Completed 0 1h ==> n0: ucp barbican-ks-service-46hjk 0/1 Completed 0 1h ==> n0: ucp barbican-ks-user-6df74 0/1 Completed 0 1h ==> n0: ucp barbican-rabbit-init-gnvfl 0/1 Completed 0 1h ==> n0: ucp deckhand-api-6cd9c4479d-wc5cw 0/1 Running 0 1h ==> n0: ucp deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 17 1h <===== ==> n0: ucp deckhand-db-sync-llstv 0/1 Init:0/1 0 1h ==> n0: ucp deckhand-ks-endpoints-4gqfj 0/3 Completed 0 1h ==> n0: ucp deckhand-ks-service-c6gbq 0/1 Completed 0 1h ==> n0: ucp deckhand-ks-user-5skng 0/1 Completed 0 1h ==> n0: ucp divingbell-apparmor-default-lkcl6 1/1 Running 0 1h ==> n0: ucp divingbell-apt-default-7jgtv 1/1 Running 0 1h ==> n0: ucp divingbell-ethtool-default-tm2w4 1/1 Running 0 1h ==> n0: ucp divingbell-exec-default-l45m8 1/1 Running 0 1h ==> n0: ucp divingbell-limits-default-q84pr 1/1 Running 0 1h ==> n0: ucp divingbell-mounts-default-29420945-nrdsz 1/1 Running 0 1h ==> n0: ucp divingbell-perm-default-wdgld 1/1 Running 0 1h ==> n0: ucp divingbell-sysctl-default-t7f2m 1/1 Running 0 1h ==> n0: ucp divingbell-uamlite-default-fc4jx 1/1 Running 0 1h ==> n0: ucp ingress-86576d6599-q8ng4 1/1 Running 0 1h ==> n0: ucp ingress-error-pages-5c97bb46bb-pjz9m 1/1 Running 0 1h ==> n0: ucp keystone-api-678fc44bdd-ncxc2 1/1 Running 0 1h ==> n0: ucp keystone-bootstrap-28l4g 0/1 Completed 0 1h ==> n0: ucp keystone-credential-setup-rq5d4 0/1 Completed 0 1h ==> n0: ucp keystone-db-init-z8x4w 0/1 Completed 0 1h ==> n0: ucp keystone-db-sync-9hvb5 0/1 Completed 0 1h ==> n0: ucp keystone-domain-manage-tzcnf 0/1 Completed 0 1h ==> n0: ucp keystone-fernet-setup-bzdpb 0/1 Completed 0 1h ==> n0: ucp keystone-rabbit-init-cxpc6 0/1 Completed 0 1h ==> n0: ucp maas-bootstrap-admin-user-g99rl 0/1 Completed 0 1h ==> n0: ucp maas-db-init-h4llm 0/1 Completed 0 1h ==> n0: ucp maas-db-sync-6tsqj 0/1 Completed 0 1h ==> n0: ucp maas-export-api-key-c8rdb 0/1 Completed 0 1h ==> n0: ucp maas-import-resources-hhq7f 0/1 Completed 1 1h ==> n0: ucp maas-ingress-756f6f9d6-dpcp9 2/2 Running 0 1h ==> n0: ucp maas-ingress-errors-8686d56d98-jr6xx 1/1 Running 0 1h ==> n0: u ==> n0: cp maas-rack-0 1/1 Running 0 1h ==> n0: ucp maas-region-0 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-55794d94c8-mhjjf 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-55794d94c8-vglbv 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-28cdv 1/1 Running 0 1h ==> n0: ucp mariadb-server-0 1/1 Running 0 1h ==> n0: ucp postgresql-0 1/1 Running 1 1h _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi Roman, Thanks a lot for looking into this issue. I re-ran the same process again and this time it successfully completed Genesis phase. The postgresql-0 and nfs-provisioner logs did not show any apparent errors. So It seems to me there is a consistency issue because all I did was destroy the n0 vm and "vagrant up" again. It finally succeeded once out of 5 tries. Btw, my virtualization environment is as below. vagrant@n0:~$ sudo virt-what virtualbox kvm However, the subsequent ./run_shipyard.sh commit configdocs failed as below. How can this failure be fixed? Since Genesis is complete, can we re-run the script without the Genesis part - i.e., skipping the Genesis part ? Btw, can you describe your environment setup, so we can try to follow your exact execution environment? (My environment: a physical server, installed ubuntu 16.04.5 , install virtualbox and vagrant, and run "vagrant up") Thanks again, Calvin ================================================= ==> n0: + export max_shipyard_count=60 ==> n0: + max_shipyard_count=60 ==> n0: + export shipyard_query_time=90 ==> n0: + shipyard_query_time=90 ==> n0: + bash execute_shipyard_action.sh deploy_site ==> n0: + run_action deploy_site ==> n0: + action=deploy_site ==> n0: + action_args= ==> n0: + NC='\033[0m' ==> n0: + RED='\033[0;31m' ==> n0: + GREEN='\033[0;32m' ==> n0: +++ dirname execute_shipyard_action.sh ==> n0: ++ cd . ==> n0: ++ pwd ==> n0: + DIR=/root/deploy/site ==> n0: + cd /root/deploy/site ==> n0: + source shipyard_docker_base_command.sh ==> n0: ++ NAMESPACE=ucp ==> n0: ++ SHIPYARD_IMAGE=quay.io/airshipit/shipyard:master ==> n0: +++ cat ==> n0: Execute deploy_site Dag... ==> n0: ++ base_docker_command='sudo -E docker run -t --rm --net=host ==> n0: -e http_proxy= ==> n0: -e https_proxy= ==> n0: -e no_proxy= ==> n0: -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 <http://keystone.ucp.svc.cluster.local/v3> ==> n0: -e OS_USERNAME=shipyard ==> n0: -e OS_USER_DOMAIN_NAME=default ==> n0: -e OS_PASSWORD ==> n0: -e OS_PROJECT_DOMAIN_NAME=default ==> n0: -e OS_PROJECT_NAME=service' ==> n0: + echo -e 'Execute deploy_site Dag...\n' ==> n0: + sudo -E docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc./v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=servhipit/shipyard:master create action deploy_site ==> n0: Error: Unable to complete request to Airflow <======================== Failed ==> n0: Reason: Airflow could not be contacted properly by Shipyard. ==> n0: - Error: <class 'requests.exceptions.ConnectTimeout'> ==> n0: ==> n0: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 #### On Mon, May 13, 2019 at 10:13 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello Calvin,
Seems like PosgreSQL database was not able to properly write data onto the disk. PostgreSQL runs as a postgresql-0 pod in ucp namespace, uses a persistent volume claim postgresql-data-postgresql-0, and persistent volume mounted via NFS.
kubectl describe pod postgresql-0 -n ucp kubectl logs -n ucp postgresql-0 kubectl -n ucp describe pvc postgresql-data-postgresql-0 kubectl describe pv pvc-0382c985-7572-11e9-b431-525400681552 # volume name could be different)
NFS is provisioned by nfs-provisioner-7799d64d59-ptsgk (last two parts would be different in your case): kubectl get pods -n kube-system | grep nfs kubectl -n kube-system describe pod nfs-provisioner-7799d64d59-ptsgk kubectl -n kube-system logs nfs-provisioner-7799d64d59-ptsgk
Check if there are any problems with it (e.g. unable to mount NFS share, or lack of free storage space - `df -h`).
Also running kubectl get events --all-namespaces could help to understand what went wrong.
I have run an AIAB installation today twice, and it all worked fine. I use `vagrant up` and my hypervisor is KVM, if that could help you.
I hope it helps.
Best regards, -- Roman Gorshunov
On Fri, May 10, 2019 at 7:01 AM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Not sure if my last email were out properly, its size is too big. Here
is a short one. Thanks for responding in advance.
I re-ran the "vagrant up" and looking into the logs for
It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
On Thu, May 9, 2019 at 12:17 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Btw, continue my last post, the kubectl describe pod
deckhand-db-init-zs499 output is as follows.
Thanks, Calvin =========== kubectl describe pod deckhand-db-init-zs499
=================
root@n0:/home/vagrant# kubectl describe pod deckhand-db-init-zs499 -n ucp Name: deckhand-db-init-zs499 Namespace: ucp Node: n0/10.0.2.15 Start Time: Thu, 09 May 2019 03:48:48 +0000 Labels: application=deckhand component=db-init controller-uid=59f1bee0-720d-11e9-92ac-080027fc876e job-name=deckhand-db-init release_group=airship-ucp-deckhand Annotations: <none> Status: Running IP: 10.97.26.50 Controlled By: Job/deckhand-db-init Init Containers: init: Container ID: docker://b58e8b6b7296df618cb8120b5226370afeba2a4e79dd70ee6894b5afd853c0db Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: docker-pullable:// quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6 Port: <none> Host Port: <none> Command: kubernetes-entrypoint State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 09 May 2019 03:48:52 +0000 Finished: Thu, 09 May 2019 03:48:54 +0000 Ready: True Restart Count: 0 Environment: POD_NAME: deckhand-db-init-zs499 (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: ucp:postgresql DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Containers: deckhand-db-init: Container ID: docker://5dea2aa975c3718ca298536005b9cc0b21de47e08b2260cc73005e3455bb1350 Image: docker.io/postgres:9.5 Image ID: docker-pullable://postgres@sha256 :0605b4b20a205c09ddd10eeeddd3ed7bf3cc442a8e9896ec34862ca882658be4 Port: <none> Host Port: <none> Command: /tmp/db-init.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error <======== Exit Code: 1 Started: Thu, 09 May 2019 04:10:29 +0000 Finished: Thu, 09 May 2019 04:10:30 +0000 Ready: False Restart Count: 9 Environment: DECKHAND_DB_URL: <set to the key 'DATABASE_URI' in secret 'deckhand-db-user'> Optional: false DB_NAME: <set to the key 'DATABASE_NAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_PASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-user'> Optional: false DB_FQDN: <set to the key 'DATABASE_HOST' in secret 'deckhand-db-user'> Optional: false DB_PORT: <set to the key 'DATABASE_PORT' in secret 'deckhand-db-user'> Optional: false DB_ADMIN_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-admin'> Optional: false PGPASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-admin'> Optional: false Mounts: /etc/deckhand from etc-deckhand (rw) /etc/deckhand/deckhand.conf from deckhand-etc (ro) /tmp/db-init.sh from deckhand-bin (ro) /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: etc-deckhand: Type: EmptyDir (a temporary directory that shares a pod's
"deckhand-db-init-zs499" as showed below. lifetime)
Medium: deckhand-etc: Type: Secret (a volume populated by a Secret) SecretName: deckhand-etc Optional: false deckhand-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: deckhand-bin Optional: false deckhand-db-init-token-gczr5: Type: Secret (a volume populated by a Secret) SecretName: deckhand-db-init-token-gczr5 Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From
Message
---- ------ ---- ----
Normal Scheduled 24m default-scheduler Successfully assigned deckhand-db-init-zs499 to n0 Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "etc-deckhand" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-bin" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-etc" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-db-init-token-gczr5" Normal Pulled 24m kubelet, n0 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine Normal Created 24m kubelet, n0 Created container Normal Started 24m kubelet, n0 Started container Normal Pulled 23m (x4 over 24m) kubelet, n0 Container image "docker.io/postgres:9.5" already present on machine Normal Created 23m (x4 over 24m) kubelet, n0 Created container Normal Started 23m (x4 over 24m) kubelet, n0 Started container Warning BackOff 4m (x90 over 24m) kubelet, n0 Back-off restarting failed container root@n0:/home/vagrant#
On Thu, May 9, 2019 at 12:08 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Thanks for looking into this and gave us suggestions.
I re-ran the "vagrant up" and looking into the logs for
"deckhand-db-init-zs499" as showed below.
It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
================== log for deckhand-db-init-zs499 ================================== root@n0:/home/vagrant# kubectl logs deckhand-db-init-zs499 -n ucp + export HOME=/tmp + HOME=/tmp + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + grep -q 1 + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + pgsql_superuser_cmd 'CREATE DATABASE deckhand' + DB_COMMAND='CREATE DATABASE deckhand' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE deckhand' ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
=====================================================================================
==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 49m ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 49m ==> n0: kube-system calico-etcd-anchor-ncl2p 1/1 Running 0 47m ==> n0: kube-system calico-etcd-n0 1/1 Running 0 46m ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5csnn 1/1 Running 0 46m ==> n0: kube-system calico-node-m4rtf 1/1 Running 0 46m ==> n0: kube-system calico-settings-tkp6r 0/1 Completed 0 46m ==> n0: kube-system coredns-84bdd76f4d-hhbcs 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-k8tcc 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-qp2xd 1/1 Running 0 44m ==> n0: kube-system haproxy-n0 1/1 Running 0 50m ==> n0: kube-system ingress-error-pages-7c65f766d-dn2tw 1/1 Running 0 41m ==> n0: kube-system ingress-gtvp8 2/2 Running 0 41m ==> n0: kube-system kubernetes-apiserver-anchor-99jhn 1/1 Running 0 42m ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-controller-manager-anchor-vqddp 1/1 Running 0 42m ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-etcd-anchor-9jcpl 1/1 Running 0 44m ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 42m ==> n0: kube-system kubernetes-proxy-2m9t2 1/1 Running 0 47m ==> n0: kube-system kubernetes-scheduler-anchor-nl9fb 1/1 Running 0 42m ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 41m ==> n0: kube-system nfs-provisioner-7799d64d59-vtkbd 1/1 Running 0 40m ==> n0: kube-system tiller-deploy-7d88c6f956-qwfzb 1/1 Running 0 27m ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-vfl69 1/1 Running 0 34m ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 39m ==> n0: ucp armada-api-d5f757d5-6wl98 1/1 Running 0 15m ==> n0: ucp armada-ks-endpoints-vl9rs 0/3 Completed 0 15m ==> n0: ucp armada-ks-service-vpcjd 0/1 Completed 0 15m ==> n0: ucp armada-ks-user-rv4gs 0/1 Completed 0 15m ==> n0: ucp barbican-api-5d7b88d8ff-8dd6w 1/1 Running 0 13m ==> n0: ucp barbican-db-init-gqvt4 0/1 Completed 0 13m ==> n0: ucp barbican-db-sync-tqtgq 0/1 Completed 0 13m ==> n0: ucp barbican-ks-endpoints-rwtql 0/3 ==> n0: Completed 0 13m ==> n0: ucp barbican-ks-service-l2h6h 0/1 Completed 0 13m ==> n0: ucp barbican-ks-user-wwvc7 0/1 Completed 0 13m ==> n0: ucp barbican-rabbit-init-6spq4 0/1 Completed 0 13m ==> n0: ucp deckhand-api-78b9644f96-5686f 0/1 Running 0 11m ==> n0: ucp deckhand-db-init-zs499 0/1 CrashLoopBackOff 7 11m <=== ==> n0: ucp deckhand-db-sync-ct7wl 0/1 Init:0/1 0 11m ==> n0: ucp deckhand-ks-endpoints-x4hd9 0/3 Completed 0 11m ==> n0: ucp deckhand-ks-service-ms6n5 0/1 Completed 0 11m ==> n0: ucp deckhand-ks-user-7fnvt 0/1 Completed 0 11m ==> n0: ucp divingbell-apparmor-default-hth8z 1/1 Running 0 27m ==> n0: ucp divingbell-apt-default-r965m 1/1 Running 0 27m ==> n0: ucp divingbell-ethtool-default-ldcmc 1/1 Running 0 27m ==> n0: ucp divingbell-exec-default-f7h7x 1/1 Running 0 27m ==> n0: ucp divingbell-limits-default-sp9mj 1/1 Running 0 27m ==> n0: ucp divingbell-mounts-default-8f5a00a2-frbl2 1/1 Running 0 27m ==> n0: ucp divingbell-perm-default-d7wxp 1/1 Running 0 27m ==> n0: ucp divingbell-sysctl-default-c8pnp 1/1 Running 0 27m ==> n0: ucp divingbell-uamlite-default-rfct6 1/1 Running 0 27m ==> n0: ucp ingress-86576d6599-mdgj4 1/1 Running 0 39m ==> n0: ucp ingress-error-pages-5c97bb46bb-7lg5l 1/1 Running 0 39m ==> n0: ucp keystone-api-678fc44bdd-594bb 1/1 Running 0 34m ==> n0: ucp keystone-bootstrap-rprr6 0/1 Completed 0 34m ==> n0: ucp keystone-credential-setup-zkjgs 0/1 Completed 0 34m ==> n0: ucp keystone-db-init-xkgxm 0/1 Completed 0 34m ==> n0: ucp keystone-db-sync-lm6xs 0/1 Completed 0 34m ==> n0: ucp keystone-domain-manage-9pzjq 0/1 Completed 0 34m ==> n0: ucp keystone-fernet-setup-q7t8p 0/1 Completed 0 34m ==> n0: ucp keystone-rabbit-init-qpvgt 0/1 Completed 0 34m ==> n0: ucp maas-bootstrap-admin-user-8npgw 0/1 Completed 0 26m ==> n0: ucp maas-db-init-9z86n 0/1 Completed 0 26m ==> n0: ucp maas-db-sync-r7rkg 0/1 Completed 0 26m ==> n0: ucp maas-export-api-key-n2gz4 0/1 Completed 1 26m ==> n0: ucp maas-import-resources-prlml 0/1 Completed 0 26m ==> n0: ucp maas-ingress-756f6f9d6-h65nj 2/2 Running 0 26m ==> n0: ucp maas-ingress-errors-8686d56d98-swfg9 ==> n0: 1/1 Running 0 26m ==> n0: ucp maas-rack-0 1/1 Running 0 26m ==> n0: ucp maas-region-0 1/1 Running 0 26m ==> n0: ucp mariadb-ingress-55794d94c8-dsw5w 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-55794d94c8-jczmh 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-jrqsg 1/1 Running 0 39m ==> n0: ucp mariadb-server-0 1/1 Running 0 39m ==> n0: ucp postgresql-0 1/1 Running 1 39m
On Tue, May 7, 2019 at 6:25 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello Calvin,
Try to get some kubectl logs and describe deckhand-db-init-r9jvg pod. kubectl describe pod deckhand-db-init-r9jvg -u ucp May be it would help to understand what is happening there.
Thank you for trying Airship.
Best regards, -- Roman Gorshunov
On Tue, May 7, 2019 at 7:51 AM calvin whole <calvinwhole@gmail.com>
wrote:
Hi,
We are trying to deploy AIIB.
I have a physical server with Ubuntu 16.04.5 OS, installed
virtualbox and vagrant.
The process is straightforward by following https://opendev.org/airship/in-a-bottle/ We created ~/deploy directory, downloaded Vagrantfile, and do "vagrant up".
However it stuck in the error below: deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 16
Could anyone help to resolve this? Many thanks in advance.
Sincerely, Calvin
==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 1h ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 1h ==> n0: kube-system calico-etcd-anchor-5tqhk 1/1 Running 0 1h ==> n0: kube-system calico-etcd-n0 1/1 Running 0 1h ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5ssl6 1/1 Running 0 1h ==> n0: kube-system calico-node-pbsxh 1/1 Running 0 1h ==> n0: kube-system calico-settings-lzpk9 0/1 Completed 0 1h ==> n0: kube-system coredns-84bdd76f4d-6cwnl 1/1 Running 0 1h ==> n0: kube-system coredns-84bdd76f4d-d4p8c 1/1 Running 0 1h ==> n0: kube-system coredns-84bdd76f4d-xrknz 1/1 Running 0 1h ==> n0: kube-system haproxy-n0 1/1 Running 0 1h ==> n0: kube-system ingress-9pkmx 2/2 Running 0 1h ==> n0: kube-system ingress-error-pages-7c65f766d-2pqfx 1/1 Running 0 1h ==> n0: kube-system kubernetes-apiserver-anchor-hszbf 1/1 Running 0 1h ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-controller-manager-anchor-h49vz 1/1 Running 0 1h ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-etcd-anchor-nnjbb 1/1 Running 0 1h ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-proxy-vgzjp 1/1 Running 0 1h ==> n0: kube-system kubernetes-scheduler-anchor-bq2gk 1/1 Running 0 1h ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 1h ==> n0: kube-system nfs-provisioner-7799d64d59-jx7hq 1/1 Running 0 1h ==> n0: kube-system tiller-deploy-7d88c6f956-d9kzg 1/1 Running 0 1h ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-q9wqx 1/1 Running 0 1h ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 1h ==> n0: ucp armada-api-d5f757d5-d9l9h 1/1 Running 0 1h ==> n0: ucp armada-ks-endpoints-qwbtg 0/3 Completed 0 1h ==> n0: ucp armada-ks-service-lg8kq 0/1 Completed 0 1h ==> n0: ucp armada-ks-user-g2j6v 0/1 Completed 0 1h ==> n0: ucp barbican-api-84665dd99d-qv5fz 1/1 Running 0 1h ==> n0: ucp barbican-db-init-ndx58 0/1 Completed 0 1h ==> n0: ucp barbican-db-sync-sh7c9 0/1 Completed 0 1h ==> n0: ucp barbican-ks-endpoints-bv7xv 0/3 Completed 0 1h ==> n0: ucp barbican-ks-service-46hjk 0/1 Completed 0 1h ==> n0: ucp barbican-ks-user-6df74 0/1 Completed 0 1h ==> n0: ucp barbican-rabbit-init-gnvfl 0/1 Completed 0 1h ==> n0: ucp deckhand-api-6cd9c4479d-wc5cw 0/1 Running 0 1h ==> n0: ucp deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 17 1h <===== ==> n0: ucp deckhand-db-sync-llstv 0/1 Init:0/1 0 1h ==> n0: ucp deckhand-ks-endpoints-4gqfj 0/3 Completed 0 1h ==> n0: ucp deckhand-ks-service-c6gbq 0/1 Completed 0 1h ==> n0: ucp deckhand-ks-user-5skng 0/1 Completed 0 1h ==> n0: ucp divingbell-apparmor-default-lkcl6 1/1 Running 0 1h ==> n0: ucp divingbell-apt-default-7jgtv 1/1 Running 0 1h ==> n0: ucp divingbell-ethtool-default-tm2w4 1/1 Running 0 1h ==> n0: ucp divingbell-exec-default-l45m8 1/1 Running 0 1h ==> n0: ucp divingbell-limits-default-q84pr 1/1 Running 0 1h ==> n0: ucp divingbell-mounts-default-29420945-nrdsz 1/1 Running 0 1h ==> n0: ucp divingbell-perm-default-wdgld 1/1 Running 0 1h ==> n0: ucp divingbell-sysctl-default-t7f2m 1/1 Running 0 1h ==> n0: ucp divingbell-uamlite-default-fc4jx 1/1 Running 0 1h ==> n0: ucp ingress-86576d6599-q8ng4 1/1 Running 0 1h ==> n0: ucp ingress-error-pages-5c97bb46bb-pjz9m 1/1 Running 0 1h ==> n0: ucp keystone-api-678fc44bdd-ncxc2 1/1 Running 0 1h ==> n0: ucp keystone-bootstrap-28l4g 0/1 Completed 0 1h ==> n0: ucp keystone-credential-setup-rq5d4 0/1 Completed 0 1h ==> n0: ucp keystone-db-init-z8x4w 0/1 Completed 0 1h ==> n0: ucp keystone-db-sync-9hvb5 0/1 Completed 0 1h ==> n0: ucp keystone-domain-manage-tzcnf 0/1 Completed 0 1h ==> n0: ucp keystone-fernet-setup-bzdpb 0/1 Completed 0 1h ==> n0: ucp keystone-rabbit-init-cxpc6 0/1 Completed 0 1h ==> n0: ucp maas-bootstrap-admin-user-g99rl 0/1 Completed 0 1h ==> n0: ucp maas-db-init-h4llm 0/1 Completed 0 1h ==> n0: ucp maas-db-sync-6tsqj 0/1 Completed 0 1h ==> n0: ucp maas-export-api-key-c8rdb 0/1 Completed 0 1h ==> n0: ucp maas-import-resources-hhq7f 0/1 Completed 1 1h ==> n0: ucp maas-ingress-756f6f9d6-dpcp9 2/2 Running 0 1h ==> n0: ucp maas-ingress-errors-8686d56d98-jr6xx 1/1 Running 0 1h ==> n0: u ==> n0: cp maas-rack-0 1/1 Running 0 1h ==> n0: ucp maas-region-0 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-55794d94c8-mhjjf 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-55794d94c8-vglbv 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-28cdv 1/1 Running 0 1h ==> n0: ucp mariadb-server-0 1/1 Running 0 1h ==> n0: ucp postgresql-0 1/1 Running 1 1h _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hello Calvin, Here is an ascii recording of my run of AIAB: https://paste.ubuntu.com/p/C7kWZpG33H/ https://asciinema.org/docs/usage - you can use it to play the recording
From your logs below it seems like Airflow is not running properly, Airflow is a part of Shipyard: ==> n0: Reason: Airflow could not be contacted properly by Shipyard. ==> n0: - Error: <class 'requests.exceptions.ConnectTimeout'> So something could be wrong with airflow-* pods or services: kubectl get pods --all-namespaces | grep -i airfl kubectl get svc --all-namespaces | grep airf
The manifests/common/deploy-airship.sh script contains good sequence of steps being run inside a VM. It is being launched with parameter "demo" from manifests/dev_single_node/airship-in-a-bottle.sh. Check the code here: https://opendev.org/airship/in-a-bottle/src/branch/master/manifests. And yes, you can run parts of this scripts, but manually (I'd recommend comment certain sections, which you believe have already completed properly). Specs of my installation (actually it's a laptop): [roman@romanpc Airship]$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/fedora-root 49G 21G 26G 45% / [roman@romanpc Airship]$ free -h # AIAB VM is already running here and consumes ~20GB RAM and 180-400% of CPU total used free shared buff/cache available Mem: 31Gi 23Gi 328Mi 874Mi 7.5Gi 6.5Gi Swap: 15Gi 21Mi 15Gi [roman@romanpc Airship]$ grep "model name" /proc/cpuinfo model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz model name : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz [roman@romanpc Airship]$ cat /etc/fedora-release Fedora release 30 (Thirty) [roman@romanpc Airship]$ vagrant version | grep Version Installed Version: 2.2.3 Latest Version: 2.2.4 [roman@romanpc Airship]$ vagrant plugin list vagrant-libvirt (0.0.45, system) [roman@romanpc Airship]$ rpm -q qemu-kvm qemu-kvm-3.1.0-7.fc30.x86_64 [roman@romanpc Airship]$ Inside VM: vagrant@n0:~$ sudo virt-what kvm vagrant@n0:~$ Best regards, -- Roman Gorshunov On Tue, May 14, 2019 at 7:59 AM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Thanks a lot for looking into this issue.
I re-ran the same process again and this time it successfully completed Genesis phase. The postgresql-0 and nfs-provisioner logs did not show any apparent errors. So It seems to me there is a consistency issue because all I did was destroy the n0 vm and "vagrant up" again. It finally succeeded once out of 5 tries.
Btw, my virtualization environment is as below.
vagrant@n0:~$ sudo virt-what
virtualbox
kvm
However, the subsequent ./run_shipyard.sh commit configdocs failed as below. How can this failure be fixed?
Since Genesis is complete, can we re-run the script without the Genesis part - i.e., skipping the Genesis part ?
Btw, can you describe your environment setup, so we can try to follow your exact execution environment?
(My environment: a physical server, installed ubuntu 16.04.5 , install virtualbox and vagrant, and run "vagrant up")
Thanks again, Calvin
================================================= ==> n0: + export max_shipyard_count=60 ==> n0: + max_shipyard_count=60 ==> n0: + export shipyard_query_time=90 ==> n0: + shipyard_query_time=90 ==> n0: + bash execute_shipyard_action.sh deploy_site ==> n0: + run_action deploy_site ==> n0: + action=deploy_site ==> n0: + action_args= ==> n0: + NC='\033[0m' ==> n0: + RED='\033[0;31m' ==> n0: + GREEN='\033[0;32m' ==> n0: +++ dirname execute_shipyard_action.sh ==> n0: ++ cd . ==> n0: ++ pwd ==> n0: + DIR=/root/deploy/site ==> n0: + cd /root/deploy/site ==> n0: + source shipyard_docker_base_command.sh ==> n0: ++ NAMESPACE=ucp ==> n0: ++ SHIPYARD_IMAGE=quay.io/airshipit/shipyard:master ==> n0: +++ cat ==> n0: Execute deploy_site Dag... ==> n0: ++ base_docker_command='sudo -E docker run -t --rm --net=host ==> n0: -e http_proxy= ==> n0: -e https_proxy= ==> n0: -e no_proxy= ==> n0: -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3 ==> n0: -e OS_USERNAME=shipyard ==> n0: -e OS_USER_DOMAIN_NAME=default ==> n0: -e OS_PASSWORD ==> n0: -e OS_PROJECT_DOMAIN_NAME=default ==> n0: -e OS_PROJECT_NAME=service' ==> n0: + echo -e 'Execute deploy_site Dag...\n' ==> n0: + sudo -E docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc./v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=servhipit/shipyard:master create action deploy_site ==> n0: Error: Unable to complete request to Airflow <======================== Failed ==> n0: Reason: Airflow could not be contacted properly by Shipyard. ==> n0: - Error: <class 'requests.exceptions.ConnectTimeout'> ==> n0: ==> n0: #### Errors: 1, Warnings: 0, Infos: 0, Other: 0 ####
On Mon, May 13, 2019 at 10:13 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello Calvin,
Seems like PosgreSQL database was not able to properly write data onto the disk. PostgreSQL runs as a postgresql-0 pod in ucp namespace, uses a persistent volume claim postgresql-data-postgresql-0, and persistent volume mounted via NFS.
kubectl describe pod postgresql-0 -n ucp kubectl logs -n ucp postgresql-0 kubectl -n ucp describe pvc postgresql-data-postgresql-0 kubectl describe pv pvc-0382c985-7572-11e9-b431-525400681552 # volume name could be different)
NFS is provisioned by nfs-provisioner-7799d64d59-ptsgk (last two parts would be different in your case): kubectl get pods -n kube-system | grep nfs kubectl -n kube-system describe pod nfs-provisioner-7799d64d59-ptsgk kubectl -n kube-system logs nfs-provisioner-7799d64d59-ptsgk
Check if there are any problems with it (e.g. unable to mount NFS share, or lack of free storage space - `df -h`).
Also running kubectl get events --all-namespaces could help to understand what went wrong.
I have run an AIAB installation today twice, and it all worked fine. I use `vagrant up` and my hypervisor is KVM, if that could help you.
I hope it helps.
Best regards, -- Roman Gorshunov
On Fri, May 10, 2019 at 7:01 AM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Not sure if my last email were out properly, its size is too big. Here is a short one. Thanks for responding in advance.
I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below. It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
On Thu, May 9, 2019 at 12:17 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Btw, continue my last post, the kubectl describe pod deckhand-db-init-zs499 output is as follows.
Thanks, Calvin =========== kubectl describe pod deckhand-db-init-zs499 ================= root@n0:/home/vagrant# kubectl describe pod deckhand-db-init-zs499 -n ucp Name: deckhand-db-init-zs499 Namespace: ucp Node: n0/10.0.2.15 Start Time: Thu, 09 May 2019 03:48:48 +0000 Labels: application=deckhand component=db-init controller-uid=59f1bee0-720d-11e9-92ac-080027fc876e job-name=deckhand-db-init release_group=airship-ucp-deckhand Annotations: <none> Status: Running IP: 10.97.26.50 Controlled By: Job/deckhand-db-init Init Containers: init: Container ID: docker://b58e8b6b7296df618cb8120b5226370afeba2a4e79dd70ee6894b5afd853c0db Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6 Port: <none> Host Port: <none> Command: kubernetes-entrypoint State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 09 May 2019 03:48:52 +0000 Finished: Thu, 09 May 2019 03:48:54 +0000 Ready: True Restart Count: 0 Environment: POD_NAME: deckhand-db-init-zs499 (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: ucp:postgresql DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Containers: deckhand-db-init: Container ID: docker://5dea2aa975c3718ca298536005b9cc0b21de47e08b2260cc73005e3455bb1350 Image: docker.io/postgres:9.5 Image ID: docker-pullable://postgres@sha256:0605b4b20a205c09ddd10eeeddd3ed7bf3cc442a8e9896ec34862ca882658be4 Port: <none> Host Port: <none> Command: /tmp/db-init.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error <======== Exit Code: 1 Started: Thu, 09 May 2019 04:10:29 +0000 Finished: Thu, 09 May 2019 04:10:30 +0000 Ready: False Restart Count: 9 Environment: DECKHAND_DB_URL: <set to the key 'DATABASE_URI' in secret 'deckhand-db-user'> Optional: false DB_NAME: <set to the key 'DATABASE_NAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_PASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-user'> Optional: false DB_FQDN: <set to the key 'DATABASE_HOST' in secret 'deckhand-db-user'> Optional: false DB_PORT: <set to the key 'DATABASE_PORT' in secret 'deckhand-db-user'> Optional: false DB_ADMIN_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-admin'> Optional: false PGPASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-admin'> Optional: false Mounts: /etc/deckhand from etc-deckhand (rw) /etc/deckhand/deckhand.conf from deckhand-etc (ro) /tmp/db-init.sh from deckhand-bin (ro) /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: etc-deckhand: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: deckhand-etc: Type: Secret (a volume populated by a Secret) SecretName: deckhand-etc Optional: false deckhand-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: deckhand-bin Optional: false deckhand-db-init-token-gczr5: Type: Secret (a volume populated by a Secret) SecretName: deckhand-db-init-token-gczr5 Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 24m default-scheduler Successfully assigned deckhand-db-init-zs499 to n0 Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "etc-deckhand" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-bin" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-etc" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-db-init-token-gczr5" Normal Pulled 24m kubelet, n0 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine Normal Created 24m kubelet, n0 Created container Normal Started 24m kubelet, n0 Started container Normal Pulled 23m (x4 over 24m) kubelet, n0 Container image "docker.io/postgres:9.5" already present on machine Normal Created 23m (x4 over 24m) kubelet, n0 Created container Normal Started 23m (x4 over 24m) kubelet, n0 Started container Warning BackOff 4m (x90 over 24m) kubelet, n0 Back-off restarting failed container root@n0:/home/vagrant#
On Thu, May 9, 2019 at 12:08 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Thanks for looking into this and gave us suggestions.
I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below. It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
================== log for deckhand-db-init-zs499 ================================== root@n0:/home/vagrant# kubectl logs deckhand-db-init-zs499 -n ucp + export HOME=/tmp + HOME=/tmp + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + grep -q 1 + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + pgsql_superuser_cmd 'CREATE DATABASE deckhand' + DB_COMMAND='CREATE DATABASE deckhand' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE deckhand' ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
===================================================================================== ==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 49m ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 49m ==> n0: kube-system calico-etcd-anchor-ncl2p 1/1 Running 0 47m ==> n0: kube-system calico-etcd-n0 1/1 Running 0 46m ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5csnn 1/1 Running 0 46m ==> n0: kube-system calico-node-m4rtf 1/1 Running 0 46m ==> n0: kube-system calico-settings-tkp6r 0/1 Completed 0 46m ==> n0: kube-system coredns-84bdd76f4d-hhbcs 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-k8tcc 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-qp2xd 1/1 Running 0 44m ==> n0: kube-system haproxy-n0 1/1 Running 0 50m ==> n0: kube-system ingress-error-pages-7c65f766d-dn2tw 1/1 Running 0 41m ==> n0: kube-system ingress-gtvp8 2/2 Running 0 41m ==> n0: kube-system kubernetes-apiserver-anchor-99jhn 1/1 Running 0 42m ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-controller-manager-anchor-vqddp 1/1 Running 0 42m ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-etcd-anchor-9jcpl 1/1 Running 0 44m ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 42m ==> n0: kube-system kubernetes-proxy-2m9t2 1/1 Running 0 47m ==> n0: kube-system kubernetes-scheduler-anchor-nl9fb 1/1 Running 0 42m ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 41m ==> n0: kube-system nfs-provisioner-7799d64d59-vtkbd 1/1 Running 0 40m ==> n0: kube-system tiller-deploy-7d88c6f956-qwfzb 1/1 Running 0 27m ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-vfl69 1/1 Running 0 34m ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 39m ==> n0: ucp armada-api-d5f757d5-6wl98 1/1 Running 0 15m ==> n0: ucp armada-ks-endpoints-vl9rs 0/3 Completed 0 15m ==> n0: ucp armada-ks-service-vpcjd 0/1 Completed 0 15m ==> n0: ucp armada-ks-user-rv4gs 0/1 Completed 0 15m ==> n0: ucp barbican-api-5d7b88d8ff-8dd6w 1/1 Running 0 13m ==> n0: ucp barbican-db-init-gqvt4 0/1 Completed 0 13m ==> n0: ucp barbican-db-sync-tqtgq 0/1 Completed 0 13m ==> n0: ucp barbican-ks-endpoints-rwtql 0/3 ==> n0: Completed 0 13m ==> n0: ucp barbican-ks-service-l2h6h 0/1 Completed 0 13m ==> n0: ucp barbican-ks-user-wwvc7 0/1 Completed 0 13m ==> n0: ucp barbican-rabbit-init-6spq4 0/1 Completed 0 13m ==> n0: ucp deckhand-api-78b9644f96-5686f 0/1 Running 0 11m ==> n0: ucp deckhand-db-init-zs499 0/1 CrashLoopBackOff 7 11m <=== ==> n0: ucp deckhand-db-sync-ct7wl 0/1 Init:0/1 0 11m ==> n0: ucp deckhand-ks-endpoints-x4hd9 0/3 Completed 0 11m ==> n0: ucp deckhand-ks-service-ms6n5 0/1 Completed 0 11m ==> n0: ucp deckhand-ks-user-7fnvt 0/1 Completed 0 11m ==> n0: ucp divingbell-apparmor-default-hth8z 1/1 Running 0 27m ==> n0: ucp divingbell-apt-default-r965m 1/1 Running 0 27m ==> n0: ucp divingbell-ethtool-default-ldcmc 1/1 Running 0 27m ==> n0: ucp divingbell-exec-default-f7h7x 1/1 Running 0 27m ==> n0: ucp divingbell-limits-default-sp9mj 1/1 Running 0 27m ==> n0: ucp divingbell-mounts-default-8f5a00a2-frbl2 1/1 Running 0 27m ==> n0: ucp divingbell-perm-default-d7wxp 1/1 Running 0 27m ==> n0: ucp divingbell-sysctl-default-c8pnp 1/1 Running 0 27m ==> n0: ucp divingbell-uamlite-default-rfct6 1/1 Running 0 27m ==> n0: ucp ingress-86576d6599-mdgj4 1/1 Running 0 39m ==> n0: ucp ingress-error-pages-5c97bb46bb-7lg5l 1/1 Running 0 39m ==> n0: ucp keystone-api-678fc44bdd-594bb 1/1 Running 0 34m ==> n0: ucp keystone-bootstrap-rprr6 0/1 Completed 0 34m ==> n0: ucp keystone-credential-setup-zkjgs 0/1 Completed 0 34m ==> n0: ucp keystone-db-init-xkgxm 0/1 Completed 0 34m ==> n0: ucp keystone-db-sync-lm6xs 0/1 Completed 0 34m ==> n0: ucp keystone-domain-manage-9pzjq 0/1 Completed 0 34m ==> n0: ucp keystone-fernet-setup-q7t8p 0/1 Completed 0 34m ==> n0: ucp keystone-rabbit-init-qpvgt 0/1 Completed 0 34m ==> n0: ucp maas-bootstrap-admin-user-8npgw 0/1 Completed 0 26m ==> n0: ucp maas-db-init-9z86n 0/1 Completed 0 26m ==> n0: ucp maas-db-sync-r7rkg 0/1 Completed 0 26m ==> n0: ucp maas-export-api-key-n2gz4 0/1 Completed 1 26m ==> n0: ucp maas-import-resources-prlml 0/1 Completed 0 26m ==> n0: ucp maas-ingress-756f6f9d6-h65nj 2/2 Running 0 26m ==> n0: ucp maas-ingress-errors-8686d56d98-swfg9 ==> n0: 1/1 Running 0 26m ==> n0: ucp maas-rack-0 1/1 Running 0 26m ==> n0: ucp maas-region-0 1/1 Running 0 26m ==> n0: ucp mariadb-ingress-55794d94c8-dsw5w 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-55794d94c8-jczmh 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-jrqsg 1/1 Running 0 39m ==> n0: ucp mariadb-server-0 1/1 Running 0 39m ==> n0: ucp postgresql-0 1/1 Running 1 39m
On Tue, May 7, 2019 at 6:25 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello Calvin,
Try to get some kubectl logs and describe deckhand-db-init-r9jvg pod. kubectl describe pod deckhand-db-init-r9jvg -u ucp May be it would help to understand what is happening there.
Thank you for trying Airship.
Best regards, -- Roman Gorshunov
On Tue, May 7, 2019 at 7:51 AM calvin whole <calvinwhole@gmail.com> wrote: > > Hi, > > We are trying to deploy AIIB. > > I have a physical server with Ubuntu 16.04.5 OS, installed virtualbox and vagrant. > The process is straightforward by following https://opendev.org/airship/in-a-bottle/ > We created ~/deploy directory, downloaded Vagrantfile, and do "vagrant up". > > However it stuck in the error below: > deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 16 > > Could anyone help to resolve this? Many thanks in advance. > > Sincerely, > Calvin > > ==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE > ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 1h > ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 1h > ==> n0: kube-system calico-etcd-anchor-5tqhk 1/1 Running 0 1h > ==> n0: kube-system calico-etcd-n0 1/1 Running 0 1h > ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5ssl6 1/1 Running 0 1h > ==> n0: kube-system calico-node-pbsxh 1/1 Running 0 1h > ==> n0: kube-system calico-settings-lzpk9 0/1 Completed 0 1h > ==> n0: kube-system coredns-84bdd76f4d-6cwnl 1/1 Running 0 1h > ==> n0: kube-system coredns-84bdd76f4d-d4p8c 1/1 Running 0 1h > ==> n0: kube-system coredns-84bdd76f4d-xrknz 1/1 Running 0 1h > ==> n0: kube-system haproxy-n0 1/1 Running 0 1h > ==> n0: kube-system ingress-9pkmx 2/2 Running 0 1h > ==> n0: kube-system ingress-error-pages-7c65f766d-2pqfx 1/1 Running 0 1h > ==> n0: kube-system kubernetes-apiserver-anchor-hszbf 1/1 Running 0 1h > ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 1h > ==> n0: kube-system kubernetes-controller-manager-anchor-h49vz 1/1 Running 0 1h > ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 1h > ==> n0: kube-system kubernetes-etcd-anchor-nnjbb 1/1 Running 0 1h > ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 1h > ==> n0: kube-system kubernetes-proxy-vgzjp 1/1 Running 0 1h > ==> n0: kube-system kubernetes-scheduler-anchor-bq2gk 1/1 Running 0 1h > ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 1h > ==> n0: kube-system nfs-provisioner-7799d64d59-jx7hq 1/1 Running 0 1h > ==> n0: kube-system tiller-deploy-7d88c6f956-d9kzg 1/1 Running 0 1h > ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-q9wqx 1/1 Running 0 1h > ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 1h > ==> n0: ucp armada-api-d5f757d5-d9l9h 1/1 Running 0 1h > ==> n0: ucp armada-ks-endpoints-qwbtg 0/3 Completed 0 1h > ==> n0: ucp armada-ks-service-lg8kq 0/1 Completed 0 1h > ==> n0: ucp armada-ks-user-g2j6v 0/1 Completed 0 1h > ==> n0: ucp barbican-api-84665dd99d-qv5fz 1/1 Running 0 1h > ==> n0: ucp barbican-db-init-ndx58 0/1 Completed 0 1h > ==> n0: ucp barbican-db-sync-sh7c9 0/1 Completed 0 1h > ==> n0: ucp barbican-ks-endpoints-bv7xv 0/3 Completed 0 1h > ==> n0: ucp barbican-ks-service-46hjk 0/1 Completed 0 1h > ==> n0: ucp barbican-ks-user-6df74 0/1 Completed 0 1h > ==> n0: ucp barbican-rabbit-init-gnvfl 0/1 Completed 0 1h > ==> n0: ucp deckhand-api-6cd9c4479d-wc5cw 0/1 Running 0 1h > ==> n0: ucp deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 17 1h <===== > ==> n0: ucp deckhand-db-sync-llstv 0/1 Init:0/1 0 1h > ==> n0: ucp deckhand-ks-endpoints-4gqfj 0/3 Completed 0 1h > ==> n0: ucp deckhand-ks-service-c6gbq 0/1 Completed 0 1h > ==> n0: ucp deckhand-ks-user-5skng 0/1 Completed 0 1h > ==> n0: ucp divingbell-apparmor-default-lkcl6 1/1 Running 0 1h > ==> n0: ucp divingbell-apt-default-7jgtv 1/1 Running 0 1h > ==> n0: ucp divingbell-ethtool-default-tm2w4 1/1 Running 0 1h > ==> n0: ucp divingbell-exec-default-l45m8 1/1 Running 0 1h > ==> n0: ucp divingbell-limits-default-q84pr 1/1 Running 0 1h > ==> n0: ucp divingbell-mounts-default-29420945-nrdsz 1/1 Running 0 1h > ==> n0: ucp divingbell-perm-default-wdgld 1/1 Running 0 1h > ==> n0: ucp divingbell-sysctl-default-t7f2m 1/1 Running 0 1h > ==> n0: ucp divingbell-uamlite-default-fc4jx 1/1 Running 0 1h > ==> n0: ucp ingress-86576d6599-q8ng4 1/1 Running 0 1h > ==> n0: ucp ingress-error-pages-5c97bb46bb-pjz9m 1/1 Running 0 1h > ==> n0: ucp keystone-api-678fc44bdd-ncxc2 1/1 Running 0 1h > ==> n0: ucp keystone-bootstrap-28l4g 0/1 Completed 0 1h > ==> n0: ucp keystone-credential-setup-rq5d4 0/1 Completed 0 1h > ==> n0: ucp keystone-db-init-z8x4w 0/1 Completed 0 1h > ==> n0: ucp keystone-db-sync-9hvb5 0/1 Completed 0 1h > ==> n0: ucp keystone-domain-manage-tzcnf 0/1 Completed 0 1h > ==> n0: ucp keystone-fernet-setup-bzdpb 0/1 Completed 0 1h > ==> n0: ucp keystone-rabbit-init-cxpc6 0/1 Completed 0 1h > ==> n0: ucp maas-bootstrap-admin-user-g99rl 0/1 Completed 0 1h > ==> n0: ucp maas-db-init-h4llm 0/1 Completed 0 1h > ==> n0: ucp maas-db-sync-6tsqj 0/1 Completed 0 1h > ==> n0: ucp maas-export-api-key-c8rdb 0/1 Completed 0 1h > ==> n0: ucp maas-import-resources-hhq7f 0/1 Completed 1 1h > ==> n0: ucp maas-ingress-756f6f9d6-dpcp9 2/2 Running 0 1h > ==> n0: ucp maas-ingress-errors-8686d56d98-jr6xx 1/1 Running 0 1h > ==> n0: u > ==> n0: cp maas-rack-0 1/1 Running 0 1h > ==> n0: ucp maas-region-0 1/1 Running 0 1h > ==> n0: ucp mariadb-ingress-55794d94c8-mhjjf 1/1 Running 0 1h > ==> n0: ucp mariadb-ingress-55794d94c8-vglbv 1/1 Running 0 1h > ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-28cdv 1/1 Running 0 1h > ==> n0: ucp mariadb-server-0 1/1 Running 0 1h > ==> n0: ucp postgresql-0 1/1 Running 1 1h > _______________________________________________ > Airship-discuss mailing list > Airship-discuss@lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
participants (2)
-
calvin whole
-
Roman Gorshunov