[Airship-discuss] airship-in-a-bottle deployment issue

Roman Gorshunov paye600 at gmail.com
Tue May 14 12:46:40 UTC 2019


Hello Calvin,

Here is an ascii recording of my run of AIAB:
https://paste.ubuntu.com/p/C7kWZpG33H/
https://asciinema.org/docs/usage - you can use it to play the recording

>From your logs below it seems like Airflow is not running properly,
Airflow is a part of Shipyard:
  ==> n0: Reason: Airflow could not be contacted properly by Shipyard.
  ==> n0: - Error: <class 'requests.exceptions.ConnectTimeout'>
So something could be wrong with airflow-* pods or services:
kubectl get pods --all-namespaces | grep -i airfl
kubectl get svc --all-namespaces | grep airf

The manifests/common/deploy-airship.sh script contains good sequence
of steps being run inside a VM. It is being launched with parameter
"demo" from manifests/dev_single_node/airship-in-a-bottle.sh. Check
the code here: https://opendev.org/airship/in-a-bottle/src/branch/master/manifests.
And yes, you can run parts of this scripts, but manually (I'd
recommend comment certain sections, which you believe have already
completed properly).

Specs of my installation (actually it's a laptop):

[roman at romanpc Airship]$ df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   49G   21G   26G  45% /
[roman at romanpc Airship]$ free -h # AIAB VM is already running here and
consumes ~20GB RAM and 180-400% of CPU
           total  used  free     shared  buff/cache available
Mem:  31Gi  23Gi  328Mi  874Mi   7.5Gi         6.5Gi
Swap: 15Gi  21Mi  15Gi
[roman at romanpc Airship]$ grep "model name" /proc/cpuinfo
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
[roman at romanpc Airship]$ cat /etc/fedora-release
Fedora release 30 (Thirty)
[roman at romanpc Airship]$ vagrant version | grep Version
Installed Version: 2.2.3
Latest Version: 2.2.4
[roman at romanpc Airship]$ vagrant plugin list
vagrant-libvirt (0.0.45, system)
[roman at romanpc Airship]$ rpm -q qemu-kvm
qemu-kvm-3.1.0-7.fc30.x86_64
[roman at romanpc Airship]$

Inside VM:
vagrant at n0:~$ sudo virt-what
kvm
vagrant at n0:~$

Best regards,
-- Roman Gorshunov

On Tue, May 14, 2019 at 7:59 AM calvin whole <calvinwhole at gmail.com> wrote:
>
> Hi Roman,
>
> Thanks a lot for looking into this issue.
>
> I re-ran the same process again and this time it successfully completed Genesis phase.  The postgresql-0 and  nfs-provisioner logs did not show any apparent errors.
> So It seems to me there is a consistency issue because all I did was destroy the n0 vm and "vagrant up" again.  It finally succeeded once out of 5 tries.
>
> Btw, my virtualization environment is as below.
>
>     vagrant at n0:~$ sudo virt-what
>
>     virtualbox
>
>     kvm
>
>
> However, the subsequent ./run_shipyard.sh commit configdocs failed as below.
> How can this failure be fixed?
>
> Since Genesis is complete, can we re-run the script without the Genesis part - i.e., skipping the Genesis part ?
>
> Btw, can you describe your environment setup, so we can try to follow your exact execution environment?
>
>  (My environment: a  physical server, installed ubuntu 16.04.5 , install virtualbox and vagrant, and run "vagrant up")
>
> Thanks again,
> Calvin
>
> =================================================
> ==> n0: + export max_shipyard_count=60
> ==> n0: + max_shipyard_count=60
> ==> n0: + export shipyard_query_time=90
> ==> n0: + shipyard_query_time=90
> ==> n0: + bash execute_shipyard_action.sh deploy_site
> ==> n0: + run_action deploy_site
> ==> n0: + action=deploy_site
> ==> n0: + action_args=
> ==> n0: + NC='\033[0m'
> ==> n0: + RED='\033[0;31m'
> ==> n0: + GREEN='\033[0;32m'
> ==> n0: +++ dirname execute_shipyard_action.sh
> ==> n0: ++ cd .
> ==> n0: ++ pwd
> ==> n0: + DIR=/root/deploy/site
> ==> n0: + cd /root/deploy/site
> ==> n0: + source shipyard_docker_base_command.sh
> ==> n0: ++ NAMESPACE=ucp
> ==> n0: ++ SHIPYARD_IMAGE=quay.io/airshipit/shipyard:master
> ==> n0: +++ cat
> ==> n0: Execute deploy_site Dag...
> ==> n0: ++ base_docker_command='sudo -E docker run -t --rm --net=host
> ==> n0: -e http_proxy=
> ==> n0: -e https_proxy=
> ==> n0: -e no_proxy=
> ==> n0: -e OS_AUTH_URL=http://keystone.ucp.svc.cluster.local:80/v3
> ==> n0: -e OS_USERNAME=shipyard
> ==> n0: -e OS_USER_DOMAIN_NAME=default
> ==> n0: -e OS_PASSWORD
> ==> n0: -e OS_PROJECT_DOMAIN_NAME=default
> ==> n0: -e OS_PROJECT_NAME=service'
> ==> n0: + echo -e 'Execute deploy_site Dag...\n'
> ==> n0: + sudo -E docker run -t --rm --net=host -e http_proxy= -e https_proxy= -e no_proxy= -e OS_AUTH_URL=http://keystone.ucp.svc./v3 -e OS_USERNAME=shipyard -e OS_USER_DOMAIN_NAME=default -e OS_PASSWORD -e OS_PROJECT_DOMAIN_NAME=default -e OS_PROJECT_NAME=servhipit/shipyard:master create action deploy_site
> ==> n0: Error: Unable to complete request to Airflow  <======================== Failed
> ==> n0: Reason: Airflow could not be contacted properly by Shipyard.
> ==> n0: - Error: <class 'requests.exceptions.ConnectTimeout'>
> ==> n0:
> ==> n0: ####  Errors: 1, Warnings: 0, Infos: 0, Other: 0  ####
>
> On Mon, May 13, 2019 at 10:13 PM Roman Gorshunov <paye600 at gmail.com> wrote:
>>
>> Hello Calvin,
>>
>> Seems like PosgreSQL database was not able to properly write data onto
>> the disk. PostgreSQL runs as a postgresql-0 pod in ucp namespace, uses
>> a persistent volume claim postgresql-data-postgresql-0, and persistent
>> volume mounted via NFS.
>>
>> kubectl describe pod postgresql-0 -n ucp
>> kubectl logs -n ucp postgresql-0
>> kubectl -n ucp describe pvc postgresql-data-postgresql-0
>> kubectl describe pv pvc-0382c985-7572-11e9-b431-525400681552 # volume
>> name could be different)
>>
>> NFS is provisioned by nfs-provisioner-7799d64d59-ptsgk (last two parts
>> would be different in your case):
>> kubectl get pods -n kube-system | grep nfs
>> kubectl -n kube-system describe pod nfs-provisioner-7799d64d59-ptsgk
>> kubectl -n kube-system logs nfs-provisioner-7799d64d59-ptsgk
>>
>> Check if there are any problems with it (e.g. unable to mount NFS
>> share, or lack of free storage space - `df -h`).
>>
>> Also running kubectl get events --all-namespaces could help to
>> understand what went wrong.
>>
>> I have run an AIAB installation today twice, and it all worked fine. I
>> use `vagrant up` and my hypervisor is KVM, if that could help you.
>>
>> I hope it helps.
>>
>> Best regards,
>> -- Roman Gorshunov
>>
>> On Fri, May 10, 2019 at 7:01 AM calvin whole <calvinwhole at gmail.com> wrote:
>> >
>> > Hi Roman,
>> >
>> > Not sure if my last email were out properly, its size is too big.   Here is a short one.  Thanks for responding in advance.
>> >
>> > I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below.
>> > It showed ERROR:  checkpoint request failed
>> > HINT:  Consult recent messages in the server log for details.
>> >
>> > What is the specific "server" log we should look into for details?
>> >
>> > Thanks for help.
>> >
>> > Sincerely,
>> > Calvin
>> >
>> >
>> > On Thu, May 9, 2019 at 12:17 PM calvin whole <calvinwhole at gmail.com> wrote:
>> >>
>> >> Hi Roman,
>> >>
>> >> Btw, continue my last post, the kubectl describe pod deckhand-db-init-zs499 output is as follows.
>> >>
>> >> Thanks,
>> >> Calvin
>> >> ===========  kubectl describe pod deckhand-db-init-zs499  =================
>> >> root at n0:/home/vagrant# kubectl describe pod deckhand-db-init-zs499 -n ucp
>> >> Name:           deckhand-db-init-zs499
>> >> Namespace:      ucp
>> >> Node:           n0/10.0.2.15
>> >> Start Time:     Thu, 09 May 2019 03:48:48 +0000
>> >> Labels:         application=deckhand
>> >>                 component=db-init
>> >>                 controller-uid=59f1bee0-720d-11e9-92ac-080027fc876e
>> >>                 job-name=deckhand-db-init
>> >>                 release_group=airship-ucp-deckhand
>> >> Annotations:    <none>
>> >> Status:         Running
>> >> IP:             10.97.26.50
>> >> Controlled By:  Job/deckhand-db-init
>> >> Init Containers:
>> >>   init:
>> >>     Container ID:  docker://b58e8b6b7296df618cb8120b5226370afeba2a4e79dd70ee6894b5afd853c0db
>> >>     Image:         quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
>> >>     Image ID:      docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:32b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6
>> >>     Port:          <none>
>> >>     Host Port:     <none>
>> >>     Command:
>> >>       kubernetes-entrypoint
>> >>     State:          Terminated
>> >>       Reason:       Completed
>> >>       Exit Code:    0
>> >>       Started:      Thu, 09 May 2019 03:48:52 +0000
>> >>       Finished:     Thu, 09 May 2019 03:48:54 +0000
>> >>     Ready:          True
>> >>     Restart Count:  0
>> >>     Environment:
>> >>       POD_NAME:              deckhand-db-init-zs499 (v1:metadata.name)
>> >>       NAMESPACE:             ucp (v1:metadata.namespace)
>> >>       INTERFACE_NAME:        eth0
>> >>       PATH:                  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
>> >>       DEPENDENCY_SERVICE:    ucp:postgresql
>> >>       DEPENDENCY_DAEMONSET:
>> >>       DEPENDENCY_CONTAINER:
>> >>       DEPENDENCY_POD_JSON:
>> >>       COMMAND:               echo done
>> >>     Mounts:
>> >>       /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro)
>> >> Containers:
>> >>   deckhand-db-init:
>> >>     Container ID:  docker://5dea2aa975c3718ca298536005b9cc0b21de47e08b2260cc73005e3455bb1350
>> >>     Image:         docker.io/postgres:9.5
>> >>     Image ID:      docker-pullable://postgres@sha256:0605b4b20a205c09ddd10eeeddd3ed7bf3cc442a8e9896ec34862ca882658be4
>> >>     Port:          <none>
>> >>     Host Port:     <none>
>> >>     Command:
>> >>       /tmp/db-init.sh
>> >>     State:          Waiting
>> >>       Reason:       CrashLoopBackOff
>> >>     Last State:     Terminated
>> >>       Reason:       Error        <========
>> >>       Exit Code:    1
>> >>       Started:      Thu, 09 May 2019 04:10:29 +0000
>> >>       Finished:     Thu, 09 May 2019 04:10:30 +0000
>> >>     Ready:          False
>> >>     Restart Count:  9
>> >>     Environment:
>> >>       DECKHAND_DB_URL:      <set to the key 'DATABASE_URI' in secret 'deckhand-db-user'>        Optional: false
>> >>       DB_NAME:              <set to the key 'DATABASE_NAME' in secret 'deckhand-db-user'>       Optional: false
>> >>       DB_SERVICE_USER:      <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-user'>   Optional: false
>> >>       DB_SERVICE_PASSWORD:  <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-user'>   Optional: false
>> >>       DB_FQDN:              <set to the key 'DATABASE_HOST' in secret 'deckhand-db-user'>       Optional: false
>> >>       DB_PORT:              <set to the key 'DATABASE_PORT' in secret 'deckhand-db-user'>       Optional: false
>> >>       DB_ADMIN_USER:        <set to the key 'DATABASE_USERNAME' in secret 'deckhand-db-admin'>  Optional: false
>> >>       PGPASSWORD:           <set to the key 'DATABASE_PASSWORD' in secret 'deckhand-db-admin'>  Optional: false
>> >>     Mounts:
>> >>       /etc/deckhand from etc-deckhand (rw)
>> >>       /etc/deckhand/deckhand.conf from deckhand-etc (ro)
>> >>       /tmp/db-init.sh from deckhand-bin (ro)
>> >>       /var/run/secrets/kubernetes.io/serviceaccount from deckhand-db-init-token-gczr5 (ro)
>> >> Conditions:
>> >>   Type           Status
>> >>   Initialized    True
>> >>   Ready          False
>> >>   PodScheduled   True
>> >> Volumes:
>> >>   etc-deckhand:
>> >>     Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
>> >>     Medium:
>> >>   deckhand-etc:
>> >>     Type:        Secret (a volume populated by a Secret)
>> >>     SecretName:  deckhand-etc
>> >>     Optional:    false
>> >>   deckhand-bin:
>> >>     Type:      ConfigMap (a volume populated by a ConfigMap)
>> >>     Name:      deckhand-bin
>> >>     Optional:  false
>> >>   deckhand-db-init-token-gczr5:
>> >>     Type:        Secret (a volume populated by a Secret)
>> >>     SecretName:  deckhand-db-init-token-gczr5
>> >>     Optional:    false
>> >> QoS Class:       BestEffort
>> >> Node-Selectors:  ucp-control-plane=enabled
>> >> Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
>> >>                  node.kubernetes.io/unreachable:NoExecute for 300s
>> >> Events:
>> >>   Type     Reason                 Age                From               Message
>> >>   ----     ------                 ----               ----               -------
>> >>   Normal   Scheduled              24m                default-scheduler  Successfully assigned deckhand-db-init-zs499 to n0
>> >>   Normal   SuccessfulMountVolume  24m                kubelet, n0        MountVolume.SetUp succeeded for volume "etc-deckhand"
>> >>   Normal   SuccessfulMountVolume  24m                kubelet, n0        MountVolume.SetUp succeeded for volume "deckhand-bin"
>> >>   Normal   SuccessfulMountVolume  24m                kubelet, n0        MountVolume.SetUp succeeded for volume "deckhand-etc"
>> >>   Normal   SuccessfulMountVolume  24m                kubelet, n0        MountVolume.SetUp succeeded for volume "deckhand-db-init-token-gczr5"
>> >>   Normal   Pulled                 24m                kubelet, n0        Container image "quay.io/stackanetes/kubernetes-entrypoint:v0.3.1" already present on machine
>> >>   Normal   Created                24m                kubelet, n0        Created container
>> >>   Normal   Started                24m                kubelet, n0        Started container
>> >>   Normal   Pulled                 23m (x4 over 24m)  kubelet, n0        Container image "docker.io/postgres:9.5" already present on machine
>> >>   Normal   Created                23m (x4 over 24m)  kubelet, n0        Created container
>> >>   Normal   Started                23m (x4 over 24m)  kubelet, n0        Started container
>> >>   Warning  BackOff                4m (x90 over 24m)  kubelet, n0        Back-off restarting failed container
>> >> root at n0:/home/vagrant#
>> >>
>> >> On Thu, May 9, 2019 at 12:08 PM calvin whole <calvinwhole at gmail.com> wrote:
>> >>>
>> >>> Hi Roman,
>> >>>
>> >>> Thanks for looking into this and gave us suggestions.
>> >>>
>> >>> I re-ran the "vagrant up" and looking into the logs for "deckhand-db-init-zs499" as showed below.
>> >>> It showed ERROR:  checkpoint request failed
>> >>> HINT:  Consult recent messages in the server log for details.
>> >>>
>> >>> What is the specific "server" log we should look into for details?
>> >>>
>> >>> Thanks for help.
>> >>>
>> >>> Sincerely,
>> >>> Calvin
>> >>>
>> >>> ================== log for deckhand-db-init-zs499 ==================================
>> >>> root at n0:/home/vagrant# kubectl logs deckhand-db-init-zs499 -n ucp
>> >>> + export HOME=/tmp
>> >>> + HOME=/tmp
>> >>> + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\'''
>> >>> + grep -q 1
>> >>> + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\'''
>> >>> + [[ ! -z '' ]]
>> >>> + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''deckhand'\'''
>> >>> + pgsql_superuser_cmd 'CREATE DATABASE deckhand'
>> >>> + DB_COMMAND='CREATE DATABASE deckhand'
>> >>> + [[ ! -z '' ]]
>> >>> + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE deckhand'
>> >>> ERROR:  checkpoint request failed
>> >>> HINT:  Consult recent messages in the server log for details.
>> >>>
>> >>> =====================================================================================
>> >>> ==> n0: NAMESPACE     NAME                                                        READY     STATUS             RESTARTS   AGE
>> >>> ==> n0: kube-system   auxiliary-etcd-n0                                           3/3       Running            0          49m
>> >>> ==> n0: kube-system   bootstrap-armada-n0                                         4/4       Running            0          49m
>> >>> ==> n0: kube-system   calico-etcd-anchor-ncl2p                                    1/1       Running            0          47m
>> >>> ==> n0: kube-system   calico-etcd-n0                                              1/1       Running            0          46m
>> >>> ==> n0: kube-system   calico-kube-controllers-56c54d8cf8-5csnn                    1/1       Running            0          46m
>> >>> ==> n0: kube-system   calico-node-m4rtf                                           1/1       Running            0          46m
>> >>> ==> n0: kube-system   calico-settings-tkp6r                                       0/1       Completed          0          46m
>> >>> ==> n0: kube-system   coredns-84bdd76f4d-hhbcs                                    1/1       Running            0          44m
>> >>> ==> n0: kube-system   coredns-84bdd76f4d-k8tcc                                    1/1       Running            0          44m
>> >>> ==> n0: kube-system   coredns-84bdd76f4d-qp2xd                                    1/1       Running            0          44m
>> >>> ==> n0: kube-system   haproxy-n0                                                  1/1       Running            0          50m
>> >>> ==> n0: kube-system   ingress-error-pages-7c65f766d-dn2tw                         1/1       Running            0          41m
>> >>> ==> n0: kube-system   ingress-gtvp8                                               2/2       Running            0          41m
>> >>> ==> n0: kube-system   kubernetes-apiserver-anchor-99jhn                           1/1       Running            0          42m
>> >>> ==> n0: kube-system   kubernetes-apiserver-n0                                     1/1       Running            0          41m
>> >>> ==> n0: kube-system   kubernetes-controller-manager-anchor-vqddp                  1/1       Running            0          42m
>> >>> ==> n0: kube-system   kubernetes-controller-manager-n0                            1/1       Running            0          41m
>> >>> ==> n0: kube-system   kubernetes-etcd-anchor-9jcpl                                1/1       Running            0          44m
>> >>> ==> n0: kube-system   kubernetes-etcd-n0                                          1/1       Running            0          42m
>> >>> ==> n0: kube-system   kubernetes-proxy-2m9t2                                      1/1       Running            0          47m
>> >>> ==> n0: kube-system   kubernetes-scheduler-anchor-nl9fb                           1/1       Running            0          42m
>> >>> ==> n0: kube-system   kubernetes-scheduler-n0                                     1/1       Running            0          41m
>> >>> ==> n0: kube-system   nfs-provisioner-7799d64d59-vtkbd                            1/1       Running            0          40m
>> >>> ==> n0: kube-system   tiller-deploy-7d88c6f956-qwfzb                              1/1       Running            0          27m
>> >>> ==> n0: ucp           airship-ucp-keystone-memcached-memcached-74d79d8896-vfl69   1/1       Running            0          34m
>> >>> ==> n0: ucp           airship-ucp-rabbitmq-rabbitmq-0                             1/1       Running            0          39m
>> >>> ==> n0: ucp           armada-api-d5f757d5-6wl98                                   1/1       Running            0          15m
>> >>> ==> n0: ucp           armada-ks-endpoints-vl9rs                                   0/3       Completed          0          15m
>> >>> ==> n0: ucp           armada-ks-service-vpcjd                                     0/1       Completed          0          15m
>> >>> ==> n0: ucp           armada-ks-user-rv4gs                                        0/1       Completed          0          15m
>> >>> ==> n0: ucp           barbican-api-5d7b88d8ff-8dd6w                               1/1       Running            0          13m
>> >>> ==> n0: ucp           barbican-db-init-gqvt4                                      0/1       Completed          0          13m
>> >>> ==> n0: ucp           barbican-db-sync-tqtgq                                      0/1       Completed          0          13m
>> >>> ==> n0: ucp           barbican-ks-endpoints-rwtql                                 0/3
>> >>> ==> n0: Completed          0          13m
>> >>> ==> n0: ucp           barbican-ks-service-l2h6h                                   0/1       Completed          0          13m
>> >>> ==> n0: ucp           barbican-ks-user-wwvc7                                      0/1       Completed          0          13m
>> >>> ==> n0: ucp           barbican-rabbit-init-6spq4                                  0/1       Completed          0          13m
>> >>> ==> n0: ucp           deckhand-api-78b9644f96-5686f                               0/1       Running            0          11m
>> >>> ==> n0: ucp           deckhand-db-init-zs499                                      0/1       CrashLoopBackOff   7          11m <===
>> >>> ==> n0: ucp           deckhand-db-sync-ct7wl                                      0/1       Init:0/1           0          11m
>> >>> ==> n0: ucp           deckhand-ks-endpoints-x4hd9                                 0/3       Completed          0          11m
>> >>> ==> n0: ucp           deckhand-ks-service-ms6n5                                   0/1       Completed          0          11m
>> >>> ==> n0: ucp           deckhand-ks-user-7fnvt                                      0/1       Completed          0          11m
>> >>> ==> n0: ucp           divingbell-apparmor-default-hth8z                           1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-apt-default-r965m                                1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-ethtool-default-ldcmc                            1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-exec-default-f7h7x                               1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-limits-default-sp9mj                             1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-mounts-default-8f5a00a2-frbl2                    1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-perm-default-d7wxp                               1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-sysctl-default-c8pnp                             1/1       Running            0          27m
>> >>> ==> n0: ucp           divingbell-uamlite-default-rfct6                            1/1       Running            0          27m
>> >>> ==> n0: ucp           ingress-86576d6599-mdgj4                                    1/1       Running            0          39m
>> >>> ==> n0: ucp           ingress-error-pages-5c97bb46bb-7lg5l                        1/1       Running            0          39m
>> >>> ==> n0: ucp           keystone-api-678fc44bdd-594bb                               1/1       Running            0          34m
>> >>> ==> n0: ucp           keystone-bootstrap-rprr6                                    0/1       Completed          0          34m
>> >>> ==> n0: ucp           keystone-credential-setup-zkjgs                             0/1       Completed          0          34m
>> >>> ==> n0: ucp           keystone-db-init-xkgxm                                      0/1       Completed          0          34m
>> >>> ==> n0: ucp           keystone-db-sync-lm6xs                                      0/1       Completed          0          34m
>> >>> ==> n0: ucp           keystone-domain-manage-9pzjq                                0/1       Completed          0          34m
>> >>> ==> n0: ucp           keystone-fernet-setup-q7t8p                                 0/1       Completed          0          34m
>> >>> ==> n0: ucp           keystone-rabbit-init-qpvgt                                  0/1       Completed          0          34m
>> >>> ==> n0: ucp           maas-bootstrap-admin-user-8npgw                             0/1       Completed          0          26m
>> >>> ==> n0: ucp           maas-db-init-9z86n                                          0/1       Completed          0          26m
>> >>> ==> n0: ucp           maas-db-sync-r7rkg                                          0/1       Completed          0          26m
>> >>> ==> n0: ucp           maas-export-api-key-n2gz4                                   0/1       Completed          1          26m
>> >>> ==> n0: ucp           maas-import-resources-prlml                                 0/1       Completed          0          26m
>> >>> ==> n0: ucp           maas-ingress-756f6f9d6-h65nj                                2/2       Running            0          26m
>> >>> ==> n0: ucp           maas-ingress-errors-8686d56d98-swfg9
>> >>> ==> n0:                         1/1       Running            0          26m
>> >>> ==> n0: ucp           maas-rack-0                                                 1/1       Running            0          26m
>> >>> ==> n0: ucp           maas-region-0                                               1/1       Running            0          26m
>> >>> ==> n0: ucp           mariadb-ingress-55794d94c8-dsw5w                            1/1       Running            0          39m
>> >>> ==> n0: ucp           mariadb-ingress-55794d94c8-jczmh                            1/1       Running            0          39m
>> >>> ==> n0: ucp           mariadb-ingress-error-pages-85f96fbd-jrqsg                  1/1       Running            0          39m
>> >>> ==> n0: ucp           mariadb-server-0                                            1/1       Running            0          39m
>> >>> ==> n0: ucp           postgresql-0                                                1/1       Running            1          39m
>> >>>
>> >>>
>> >>> On Tue, May 7, 2019 at 6:25 PM Roman Gorshunov <paye600 at gmail.com> wrote:
>> >>>>
>> >>>> Hello Calvin,
>> >>>>
>> >>>> Try to get some kubectl logs and describe deckhand-db-init-r9jvg pod.
>> >>>> kubectl describe pod deckhand-db-init-r9jvg -u ucp
>> >>>> May be it would help to understand what is happening there.
>> >>>>
>> >>>> Thank you for trying Airship.
>> >>>>
>> >>>> Best regards,
>> >>>> -- Roman Gorshunov
>> >>>>
>> >>>> On Tue, May 7, 2019 at 7:51 AM calvin whole <calvinwhole at gmail.com> wrote:
>> >>>> >
>> >>>> > Hi,
>> >>>> >
>> >>>> > We are trying to deploy AIIB.
>> >>>> >
>> >>>> > I have a physical server with Ubuntu 16.04.5 OS, installed virtualbox and vagrant.
>> >>>> > The process is straightforward by following https://opendev.org/airship/in-a-bottle/
>> >>>> > We created ~/deploy directory,  downloaded Vagrantfile, and do "vagrant up".
>> >>>> >
>> >>>> > However it stuck in the error below:
>> >>>> >  deckhand-db-init-r9jvg                                      0/1       CrashLoopBackOff   16
>> >>>> >
>> >>>> > Could anyone help to resolve this?  Many thanks in advance.
>> >>>> >
>> >>>> > Sincerely,
>> >>>> > Calvin
>> >>>> >
>> >>>> > ==> n0: NAMESPACE     NAME                                                        READY     STATUS             RESTARTS   AGE
>> >>>> > ==> n0: kube-system   auxiliary-etcd-n0                                           3/3       Running            0          1h
>> >>>> > ==> n0: kube-system   bootstrap-armada-n0                                         4/4       Running            0          1h
>> >>>> > ==> n0: kube-system   calico-etcd-anchor-5tqhk                                    1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   calico-etcd-n0                                              1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   calico-kube-controllers-56c54d8cf8-5ssl6                    1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   calico-node-pbsxh                                           1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   calico-settings-lzpk9                                       0/1       Completed          0          1h
>> >>>> > ==> n0: kube-system   coredns-84bdd76f4d-6cwnl                                    1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   coredns-84bdd76f4d-d4p8c                                    1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   coredns-84bdd76f4d-xrknz                                    1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   haproxy-n0                                                  1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   ingress-9pkmx                                               2/2       Running            0          1h
>> >>>> > ==> n0: kube-system   ingress-error-pages-7c65f766d-2pqfx                         1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-apiserver-anchor-hszbf                           1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-apiserver-n0                                     1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-controller-manager-anchor-h49vz                  1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-controller-manager-n0                            1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-etcd-anchor-nnjbb                                1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-etcd-n0                                          1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-proxy-vgzjp                                      1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-scheduler-anchor-bq2gk                           1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   kubernetes-scheduler-n0                                     1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   nfs-provisioner-7799d64d59-jx7hq                            1/1       Running            0          1h
>> >>>> > ==> n0: kube-system   tiller-deploy-7d88c6f956-d9kzg                              1/1       Running            0          1h
>> >>>> > ==> n0: ucp           airship-ucp-keystone-memcached-memcached-74d79d8896-q9wqx   1/1       Running            0          1h
>> >>>> > ==> n0: ucp           airship-ucp-rabbitmq-rabbitmq-0                             1/1       Running            0          1h
>> >>>> > ==> n0: ucp           armada-api-d5f757d5-d9l9h                                   1/1       Running            0          1h
>> >>>> > ==> n0: ucp           armada-ks-endpoints-qwbtg                                   0/3       Completed          0          1h
>> >>>> > ==> n0: ucp           armada-ks-service-lg8kq                                     0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           armada-ks-user-g2j6v                                        0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           barbican-api-84665dd99d-qv5fz                               1/1       Running            0          1h
>> >>>> > ==> n0: ucp           barbican-db-init-ndx58                                      0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           barbican-db-sync-sh7c9                                      0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           barbican-ks-endpoints-bv7xv                                 0/3       Completed          0          1h
>> >>>> > ==> n0: ucp           barbican-ks-service-46hjk                                   0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           barbican-ks-user-6df74                                      0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           barbican-rabbit-init-gnvfl                                  0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           deckhand-api-6cd9c4479d-wc5cw                               0/1       Running            0          1h
>> >>>> > ==> n0: ucp           deckhand-db-init-r9jvg                                      0/1       CrashLoopBackOff   17         1h     <=====
>> >>>> > ==> n0: ucp           deckhand-db-sync-llstv                                      0/1       Init:0/1           0          1h
>> >>>> > ==> n0: ucp           deckhand-ks-endpoints-4gqfj                                 0/3       Completed          0          1h
>> >>>> > ==> n0: ucp           deckhand-ks-service-c6gbq                                   0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           deckhand-ks-user-5skng                                      0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           divingbell-apparmor-default-lkcl6                           1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-apt-default-7jgtv                                1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-ethtool-default-tm2w4                            1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-exec-default-l45m8                               1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-limits-default-q84pr                             1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-mounts-default-29420945-nrdsz                    1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-perm-default-wdgld                               1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-sysctl-default-t7f2m                             1/1       Running            0          1h
>> >>>> > ==> n0: ucp           divingbell-uamlite-default-fc4jx                            1/1       Running            0          1h
>> >>>> > ==> n0: ucp           ingress-86576d6599-q8ng4                                    1/1       Running            0          1h
>> >>>> > ==> n0: ucp           ingress-error-pages-5c97bb46bb-pjz9m                        1/1       Running            0          1h
>> >>>> > ==> n0: ucp           keystone-api-678fc44bdd-ncxc2                               1/1       Running            0          1h
>> >>>> > ==> n0: ucp           keystone-bootstrap-28l4g                                    0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           keystone-credential-setup-rq5d4                             0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           keystone-db-init-z8x4w                                      0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           keystone-db-sync-9hvb5                                      0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           keystone-domain-manage-tzcnf                                0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           keystone-fernet-setup-bzdpb                                 0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           keystone-rabbit-init-cxpc6                                  0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           maas-bootstrap-admin-user-g99rl                             0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           maas-db-init-h4llm                                          0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           maas-db-sync-6tsqj                                          0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           maas-export-api-key-c8rdb                                   0/1       Completed          0          1h
>> >>>> > ==> n0: ucp           maas-import-resources-hhq7f                                 0/1       Completed          1          1h
>> >>>> > ==> n0: ucp           maas-ingress-756f6f9d6-dpcp9                                2/2       Running            0          1h
>> >>>> > ==> n0: ucp           maas-ingress-errors-8686d56d98-jr6xx                        1/1       Running            0          1h
>> >>>> > ==> n0: u
>> >>>> > ==> n0: cp           maas-rack-0                                                 1/1       Running            0          1h
>> >>>> > ==> n0: ucp           maas-region-0                                               1/1       Running            0          1h
>> >>>> > ==> n0: ucp           mariadb-ingress-55794d94c8-mhjjf                            1/1       Running            0          1h
>> >>>> > ==> n0: ucp           mariadb-ingress-55794d94c8-vglbv                            1/1       Running            0          1h
>> >>>> > ==> n0: ucp           mariadb-ingress-error-pages-85f96fbd-28cdv                  1/1       Running            0          1h
>> >>>> > ==> n0: ucp           mariadb-server-0                                            1/1       Running            0          1h
>> >>>> > ==> n0: ucp           postgresql-0                                                1/1       Running            1          1h
>> >>>> > _______________________________________________
>> >>>> > Airship-discuss mailing list
>> >>>> > Airship-discuss at lists.airshipit.org
>> >>>> > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss



More information about the Airship-discuss mailing list