Hello Calvin, Seems like PosgreSQL database was not able to properly write data onto the disk. PostgreSQL runs as a postgresql-0 pod in ucp namespace, uses a persistent volume claim postgresql-data-postgresql-0, and persistent volume mounted via NFS. kubectl describe pod postgresql-0 -n ucp kubectl logs -n ucp postgresql-0 kubectl -n ucp describe pvc postgresql-data-postgresql-0 kubectl describe pv pvc-0382c985-7572-11e9-b431-525400681552 # volume name could be different) NFS is provisioned by nfs-provisioner-7799d64d59-ptsgk (last two parts would be different in your case): kubectl get pods -n kube-system | grep nfs kubectl -n kube-system describe pod nfs-provisioner-7799d64d59-ptsgk kubectl -n kube-system logs nfs-provisioner-7799d64d59-ptsgk Check if there are any problems with it (e.g. unable to mount NFS share, or lack of free storage space - `df -h`). Also running kubectl get events --all-namespaces could help to understand what went wrong. I have run an AIAB installation today twice, and it all worked fine. I use `vagrant up` and my hypervisor is KVM, if that could help you. I hope it helps. Best regards, -- Roman Gorshunov On Fri, May 10, 2019 at 7:01 AM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Not sure if my last email were out properly, its size is too big. Here is a short one. Thanks for responding in advance.
I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below. It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
On Thu, May 9, 2019 at 12:17 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman,
Btw, continue my last post, the kubectl describe pod deckhand-db-init-zs499 output is as follows.
Thanks, Calvin =========== kubectl describe pod deckhand-db-init-zs499 ================= root@n0:/home/vagrant# kubectl describe pod deckhand-db-init-zs499 -n ucp Name: deckhand-db-init-zs499 Namespace: ucp Node: n0/10.0.2.15 Start Time: Thu, 09 May 2019 03:48:48 +0000 Labels: application=deckhand component=db-init controller-uid=59f1bee0-720d-11e9-92ac-080027fc876e job-name=deckhand-db-init release_group=airship-ucp-deckhand Annotations: <none> Status: Running IP: 10.97.26.50 Controlled By: Job/deckhand-db-init Init Containers: init: Container ID: docker://b58e8b6b7296df618cb8120b5226370afeba2a4e79dd70ee6894b5afd853c0db Image: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Image ID: docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6 Port: <none> Host Port: <none> Command: kubernetes-entrypoint State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 09 May 2019 03:48:52 +0000 Finished: Thu, 09 May 2019 03:48:54 +0000 Ready: True Restart Count: 0 Environment: POD_NAME: deckhand-db-init-zs499 (v1:metadata.name) NAMESPACE: ucp (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: ucp:postgresql DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Containers: deckhand-db-init: Container ID: docker://5dea2aa975c3718ca298536005b9cc0b21de47e08b2260cc73005e3455bb1350 Image: docker.io/postgres:9.5 Image ID: docker-pullable://postgres@sha256:0605b4b20a205c09ddd10eeeddd3ed7bf3cc442a8e9896ec34862ca882658be4 Port: <none> Host Port: <none> Command: /tmp/db-init.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error <======== Exit Code: 1 Started: Thu, 09 May 2019 04:10:29 +0000 Finished: Thu, 09 May 2019 04:10:30 +0000 Ready: False Restart Count: 9 Environment: DECKHAND_DB_URL: <set to the key 'DATABASE_URI' in secret 'deckhand-db-user'> Optional: false DB_NAME: <set to the key 'DATABASE_NAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-user'> Optional: false DB_SERVICE_PASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-user'> Optional: false DB_FQDN: <set to the key 'DATABASE_HOST' in secret 'deckhand-db-user'> Optional: false DB_PORT: <set to the key 'DATABASE_PORT' in secret 'deckhand-db-user'> Optional: false DB_ADMIN_USER: <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-admin'> Optional: false PGPASSWORD: <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-admin'> Optional: false Mounts: /etc/deckhand from etc-deckhand (rw) /etc/deckhand/deckhand.conf from deckhand-etc (ro) /tmp/db-init.sh from deckhand-bin (ro) /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: etc-deckhand: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: deckhand-etc: Type: Secret (a volume populated by a Secret) SecretName: deckhand-etc Optional: false deckhand-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: deckhand-bin Optional: false deckhand-db-init-token-gczr5: Type: Secret (a volume populated by a Secret) SecretName: deckhand-db-init-token-gczr5 Optional: false QoS Class: BestEffort Node-Selectors: ucp-control-plane=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 24m default-scheduler Successfully assigned deckhand-db-init-zs499 to n0 Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "etc-deckhand" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-bin" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-etc" Normal SuccessfulMountVolume 24m kubelet, n0 MountVolume.SetUp succeeded for volume "deckhand-db-init-token-gczr5" Normal Pulled 24m kubelet, n0 Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine Normal Created 24m kubelet, n0 Created container Normal Started 24m kubelet, n0 Started container Normal Pulled 23m (x4 over 24m) kubelet, n0 Container image "docker.io/postgres:9.5" already present on machine Normal Created 23m (x4 over 24m) kubelet, n0 Created container Normal Started 23m (x4 over 24m) kubelet, n0 Started container Warning BackOff 4m (x90 over 24m) kubelet, n0 Back-off restarting failed container root@n0:/home/vagrant#
On Thu, May 9, 2019 at 12:08 PM calvin whole <calvinwhole@gmail.com> wrote:
Hi Roman，
Thanks for looking into this and gave us suggestions.
I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below. It showed ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
What is the specific "server" log we should look into for details?
Thanks for help.
Sincerely, Calvin
================== log for deckhand-db-init-zs499 ================================== root@n0:/home/vagrant# kubectl logs deckhand-db-init-zs499 -n ucp + export HOME=/tmp + HOME=/tmp + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + grep -q 1 + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\''' + pgsql_superuser_cmd 'CREATE DATABASE deckhand' + DB_COMMAND='CREATE DATABASE deckhand' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE deckhand' ERROR: checkpoint request failed HINT: Consult recent messages in the server log for details.
===================================================================================== ==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 49m ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 49m ==> n0: kube-system calico-etcd-anchor-ncl2p 1/1 Running 0 47m ==> n0: kube-system calico-etcd-n0 1/1 Running 0 46m ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5csnn 1/1 Running 0 46m ==> n0: kube-system calico-node-m4rtf 1/1 Running 0 46m ==> n0: kube-system calico-settings-tkp6r 0/1 Completed 0 46m ==> n0: kube-system coredns-84bdd76f4d-hhbcs 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-k8tcc 1/1 Running 0 44m ==> n0: kube-system coredns-84bdd76f4d-qp2xd 1/1 Running 0 44m ==> n0: kube-system haproxy-n0 1/1 Running 0 50m ==> n0: kube-system ingress-error-pages-7c65f766d-dn2tw 1/1 Running 0 41m ==> n0: kube-system ingress-gtvp8 2/2 Running 0 41m ==> n0: kube-system kubernetes-apiserver-anchor-99jhn 1/1 Running 0 42m ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-controller-manager-anchor-vqddp 1/1 Running 0 42m ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 41m ==> n0: kube-system kubernetes-etcd-anchor-9jcpl 1/1 Running 0 44m ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 42m ==> n0: kube-system kubernetes-proxy-2m9t2 1/1 Running 0 47m ==> n0: kube-system kubernetes-scheduler-anchor-nl9fb 1/1 Running 0 42m ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 41m ==> n0: kube-system nfs-provisioner-7799d64d59-vtkbd 1/1 Running 0 40m ==> n0: kube-system tiller-deploy-7d88c6f956-qwfzb 1/1 Running 0 27m ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-vfl69 1/1 Running 0 34m ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 39m ==> n0: ucp armada-api-d5f757d5-6wl98 1/1 Running 0 15m ==> n0: ucp armada-ks-endpoints-vl9rs 0/3 Completed 0 15m ==> n0: ucp armada-ks-service-vpcjd 0/1 Completed 0 15m ==> n0: ucp armada-ks-user-rv4gs 0/1 Completed 0 15m ==> n0: ucp barbican-api-5d7b88d8ff-8dd6w 1/1 Running 0 13m ==> n0: ucp barbican-db-init-gqvt4 0/1 Completed 0 13m ==> n0: ucp barbican-db-sync-tqtgq 0/1 Completed 0 13m ==> n0: ucp barbican-ks-endpoints-rwtql 0/3 ==> n0: Completed 0 13m ==> n0: ucp barbican-ks-service-l2h6h 0/1 Completed 0 13m ==> n0: ucp barbican-ks-user-wwvc7 0/1 Completed 0 13m ==> n0: ucp barbican-rabbit-init-6spq4 0/1 Completed 0 13m ==> n0: ucp deckhand-api-78b9644f96-5686f 0/1 Running 0 11m ==> n0: ucp deckhand-db-init-zs499 0/1 CrashLoopBackOff 7 11m <=== ==> n0: ucp deckhand-db-sync-ct7wl 0/1 Init:0/1 0 11m ==> n0: ucp deckhand-ks-endpoints-x4hd9 0/3 Completed 0 11m ==> n0: ucp deckhand-ks-service-ms6n5 0/1 Completed 0 11m ==> n0: ucp deckhand-ks-user-7fnvt 0/1 Completed 0 11m ==> n0: ucp divingbell-apparmor-default-hth8z 1/1 Running 0 27m ==> n0: ucp divingbell-apt-default-r965m 1/1 Running 0 27m ==> n0: ucp divingbell-ethtool-default-ldcmc 1/1 Running 0 27m ==> n0: ucp divingbell-exec-default-f7h7x 1/1 Running 0 27m ==> n0: ucp divingbell-limits-default-sp9mj 1/1 Running 0 27m ==> n0: ucp divingbell-mounts-default-8f5a00a2-frbl2 1/1 Running 0 27m ==> n0: ucp divingbell-perm-default-d7wxp 1/1 Running 0 27m ==> n0: ucp divingbell-sysctl-default-c8pnp 1/1 Running 0 27m ==> n0: ucp divingbell-uamlite-default-rfct6 1/1 Running 0 27m ==> n0: ucp ingress-86576d6599-mdgj4 1/1 Running 0 39m ==> n0: ucp ingress-error-pages-5c97bb46bb-7lg5l 1/1 Running 0 39m ==> n0: ucp keystone-api-678fc44bdd-594bb 1/1 Running 0 34m ==> n0: ucp keystone-bootstrap-rprr6 0/1 Completed 0 34m ==> n0: ucp keystone-credential-setup-zkjgs 0/1 Completed 0 34m ==> n0: ucp keystone-db-init-xkgxm 0/1 Completed 0 34m ==> n0: ucp keystone-db-sync-lm6xs 0/1 Completed 0 34m ==> n0: ucp keystone-domain-manage-9pzjq 0/1 Completed 0 34m ==> n0: ucp keystone-fernet-setup-q7t8p 0/1 Completed 0 34m ==> n0: ucp keystone-rabbit-init-qpvgt 0/1 Completed 0 34m ==> n0: ucp maas-bootstrap-admin-user-8npgw 0/1 Completed 0 26m ==> n0: ucp maas-db-init-9z86n 0/1 Completed 0 26m ==> n0: ucp maas-db-sync-r7rkg 0/1 Completed 0 26m ==> n0: ucp maas-export-api-key-n2gz4 0/1 Completed 1 26m ==> n0: ucp maas-import-resources-prlml 0/1 Completed 0 26m ==> n0: ucp maas-ingress-756f6f9d6-h65nj 2/2 Running 0 26m ==> n0: ucp maas-ingress-errors-8686d56d98-swfg9 ==> n0: 1/1 Running 0 26m ==> n0: ucp maas-rack-0 1/1 Running 0 26m ==> n0: ucp maas-region-0 1/1 Running 0 26m ==> n0: ucp mariadb-ingress-55794d94c8-dsw5w 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-55794d94c8-jczmh 1/1 Running 0 39m ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-jrqsg 1/1 Running 0 39m ==> n0: ucp mariadb-server-0 1/1 Running 0 39m ==> n0: ucp postgresql-0 1/1 Running 1 39m
On Tue, May 7, 2019 at 6:25 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello Calvin,
Try to get some kubectl logs and describe deckhand-db-init-r9jvg pod. kubectl describe pod deckhand-db-init-r9jvg -u ucp May be it would help to understand what is happening there.
Thank you for trying Airship.
Best regards, -- Roman Gorshunov
On Tue, May 7, 2019 at 7:51 AM calvin whole <calvinwhole@gmail.com> wrote:
Hi,
We are trying to deploy AIIB.
I have a physical server with Ubuntu 16.04.5 OS, installed virtualbox and vagrant. The process is straightforward by following https://opendev.org/airship/in-a-bottle/ We created ~/deploy directory, downloaded Vagrantfile, and do "vagrant up".
However it stuck in the error below: deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 16
Could anyone help to resolve this? Many thanks in advance.
Sincerely, Calvin
==> n0: NAMESPACE NAME READY STATUS RESTARTS AGE ==> n0: kube-system auxiliary-etcd-n0 3/3 Running 0 1h ==> n0: kube-system bootstrap-armada-n0 4/4 Running 0 1h ==> n0: kube-system calico-etcd-anchor-5tqhk 1/1 Running 0 1h ==> n0: kube-system calico-etcd-n0 1/1 Running 0 1h ==> n0: kube-system calico-kube-controllers-56c54d8cf8-5ssl6 1/1 Running 0 1h ==> n0: kube-system calico-node-pbsxh 1/1 Running 0 1h ==> n0: kube-system calico-settings-lzpk9 0/1 Completed 0 1h ==> n0: kube-system coredns-84bdd76f4d-6cwnl 1/1 Running 0 1h ==> n0: kube-system coredns-84bdd76f4d-d4p8c 1/1 Running 0 1h ==> n0: kube-system coredns-84bdd76f4d-xrknz 1/1 Running 0 1h ==> n0: kube-system haproxy-n0 1/1 Running 0 1h ==> n0: kube-system ingress-9pkmx 2/2 Running 0 1h ==> n0: kube-system ingress-error-pages-7c65f766d-2pqfx 1/1 Running 0 1h ==> n0: kube-system kubernetes-apiserver-anchor-hszbf 1/1 Running 0 1h ==> n0: kube-system kubernetes-apiserver-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-controller-manager-anchor-h49vz 1/1 Running 0 1h ==> n0: kube-system kubernetes-controller-manager-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-etcd-anchor-nnjbb 1/1 Running 0 1h ==> n0: kube-system kubernetes-etcd-n0 1/1 Running 0 1h ==> n0: kube-system kubernetes-proxy-vgzjp 1/1 Running 0 1h ==> n0: kube-system kubernetes-scheduler-anchor-bq2gk 1/1 Running 0 1h ==> n0: kube-system kubernetes-scheduler-n0 1/1 Running 0 1h ==> n0: kube-system nfs-provisioner-7799d64d59-jx7hq 1/1 Running 0 1h ==> n0: kube-system tiller-deploy-7d88c6f956-d9kzg 1/1 Running 0 1h ==> n0: ucp airship-ucp-keystone-memcached-memcached-74d79d8896-q9wqx 1/1 Running 0 1h ==> n0: ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 1h ==> n0: ucp armada-api-d5f757d5-d9l9h 1/1 Running 0 1h ==> n0: ucp armada-ks-endpoints-qwbtg 0/3 Completed 0 1h ==> n0: ucp armada-ks-service-lg8kq 0/1 Completed 0 1h ==> n0: ucp armada-ks-user-g2j6v 0/1 Completed 0 1h ==> n0: ucp barbican-api-84665dd99d-qv5fz 1/1 Running 0 1h ==> n0: ucp barbican-db-init-ndx58 0/1 Completed 0 1h ==> n0: ucp barbican-db-sync-sh7c9 0/1 Completed 0 1h ==> n0: ucp barbican-ks-endpoints-bv7xv 0/3 Completed 0 1h ==> n0: ucp barbican-ks-service-46hjk 0/1 Completed 0 1h ==> n0: ucp barbican-ks-user-6df74 0/1 Completed 0 1h ==> n0: ucp barbican-rabbit-init-gnvfl 0/1 Completed 0 1h ==> n0: ucp deckhand-api-6cd9c4479d-wc5cw 0/1 Running 0 1h ==> n0: ucp deckhand-db-init-r9jvg 0/1 CrashLoopBackOff 17 1h <===== ==> n0: ucp deckhand-db-sync-llstv 0/1 Init:0/1 0 1h ==> n0: ucp deckhand-ks-endpoints-4gqfj 0/3 Completed 0 1h ==> n0: ucp deckhand-ks-service-c6gbq 0/1 Completed 0 1h ==> n0: ucp deckhand-ks-user-5skng 0/1 Completed 0 1h ==> n0: ucp divingbell-apparmor-default-lkcl6 1/1 Running 0 1h ==> n0: ucp divingbell-apt-default-7jgtv 1/1 Running 0 1h ==> n0: ucp divingbell-ethtool-default-tm2w4 1/1 Running 0 1h ==> n0: ucp divingbell-exec-default-l45m8 1/1 Running 0 1h ==> n0: ucp divingbell-limits-default-q84pr 1/1 Running 0 1h ==> n0: ucp divingbell-mounts-default-29420945-nrdsz 1/1 Running 0 1h ==> n0: ucp divingbell-perm-default-wdgld 1/1 Running 0 1h ==> n0: ucp divingbell-sysctl-default-t7f2m 1/1 Running 0 1h ==> n0: ucp divingbell-uamlite-default-fc4jx 1/1 Running 0 1h ==> n0: ucp ingress-86576d6599-q8ng4 1/1 Running 0 1h ==> n0: ucp ingress-error-pages-5c97bb46bb-pjz9m 1/1 Running 0 1h ==> n0: ucp keystone-api-678fc44bdd-ncxc2 1/1 Running 0 1h ==> n0: ucp keystone-bootstrap-28l4g 0/1 Completed 0 1h ==> n0: ucp keystone-credential-setup-rq5d4 0/1 Completed 0 1h ==> n0: ucp keystone-db-init-z8x4w 0/1 Completed 0 1h ==> n0: ucp keystone-db-sync-9hvb5 0/1 Completed 0 1h ==> n0: ucp keystone-domain-manage-tzcnf 0/1 Completed 0 1h ==> n0: ucp keystone-fernet-setup-bzdpb 0/1 Completed 0 1h ==> n0: ucp keystone-rabbit-init-cxpc6 0/1 Completed 0 1h ==> n0: ucp maas-bootstrap-admin-user-g99rl 0/1 Completed 0 1h ==> n0: ucp maas-db-init-h4llm 0/1 Completed 0 1h ==> n0: ucp maas-db-sync-6tsqj 0/1 Completed 0 1h ==> n0: ucp maas-export-api-key-c8rdb 0/1 Completed 0 1h ==> n0: ucp maas-import-resources-hhq7f 0/1 Completed 1 1h ==> n0: ucp maas-ingress-756f6f9d6-dpcp9 2/2 Running 0 1h ==> n0: ucp maas-ingress-errors-8686d56d98-jr6xx 1/1 Running 0 1h ==> n0: u ==> n0: cp maas-rack-0 1/1 Running 0 1h ==> n0: ucp maas-region-0 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-55794d94c8-mhjjf 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-55794d94c8-vglbv 1/1 Running 0 1h ==> n0: ucp mariadb-ingress-error-pages-85f96fbd-28cdv 1/1 Running 0 1h ==> n0: ucp mariadb-server-0 1/1 Running 0 1h ==> n0: ucp postgresql-0 1/1 Running 1 1h _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss