Re: [Airship-discuss] Airship in a bottle install failure...
Hi there Alex. Thanks for the reply.
From your logs it seems to me like maas-db-init is not able to connect to postgres DB with psql client to the other container. (clients should be listed by IP in pg_hba.conf, wildcards should be possible....)
I will try another install from a fresh VM and let you know if I have the same one. Thanks ________________________________ From: Alex Arnoldy <alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 4:26 PM To: Balsiger Daniel, INI-EAI-INO-MBL; airship-discuss@lists.airshipit.org Subject: RE: Airship in a bottle install failure... Hello Daniel, I had launched the same thing but then got pulled away from it and totally forgot to check it. I thought I’d be able to tell you I got it running, but I don’t think that’s the case. For my installation, the maas-db-init pod seems to be stuck in CrashLoopBackOff: root@airship:~# kubectl get pods -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system auxiliary-etcd-airship 3/3 Running 0 3d 172.16.231.60 airship kube-system bootstrap-armada-airship 4/4 Running 0 3d 172.16.231.60 airship kube-system calico-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-etcd-anchor-nqntm 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-kube-controllers-6d5997c8c4-bm9rk 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-node-2sgkk 2/2 Running 0 3d 172.16.231.60 airship kube-system calico-settings-bqs5v 0/1 Completed 0 3d 172.16.231.60 airship kube-system coredns-d4587b7f8-bbpr6 1/1 Running 13 3d 10.97.196.130 airship kube-system coredns-d4587b7f8-tdl8v 1/1 Running 11 3d 10.97.196.131 airship kube-system coredns-d4587b7f8-ts9lr 1/1 Running 12 3d 10.97.196.129 airship kube-system haproxy-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system ingress-error-pages-5ccf96bf7d-8q6g5 1/1 Running 0 3d 10.97.196.133 airship kube-system ingress-fmrmk 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-apiserver-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-apiserver-anchor-rkfpz 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-controller-manager-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-controller-manager-anchor-75k62 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-etcd-anchor-hzcrz 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-proxy-994kg 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-scheduler-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-scheduler-anchor-gsqtj 1/1 Running 0 3d 172.16.231.60 airship kube-system nfs-provisioner-7799d64d59-g2g6n 1/1 Running 0 3d 10.97.196.134 airship kube-system tiller-deploy-7d88c6f956-9jjk7 1/1 Running 0 3d 10.97.196.153 airship ucp airship-ucp-keystone-memcached-memcached-74d79d8896-dd8k2 1/1 Running 0 3d 10.97.196.144 airship ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 2 3d 10.97.196.178 airship ucp ingress-6cd5b89d5d-xx592 1/1 Running 0 3d 10.97.196.135 airship ucp ingress-error-pages-5c97bb46bb-ljmw6 1/1 Running 0 3d 10.97.196.136 airship ucp keystone-api-7657986b8c-b49r6 1/1 Running 0 3d 10.97.196.145 airship ucp keystone-bootstrap-8494p 0/1 Completed 0 3d 10.97.196.147 airship ucp keystone-credential-setup-r5twm 0/1 Completed 0 3d 10.97.196.146 airship ucp keystone-db-init-pxlzr 0/1 Completed 0 3d 10.97.196.148 airship ucp keystone-db-sync-zx665 0/1 Completed 0 3d 10.97.196.151 airship ucp keystone-domain-manage-rbfl5 0/1 Completed 0 3d 10.97.196.149 airship ucp keystone-fernet-rotate-1544356800-kcgg9 0/1 Completed 0 1d 10.97.196.148 airship ucp keystone-fernet-rotate-1544400000-78qc8 0/1 Completed 0 14h 10.97.196.163 airship ucp keystone-fernet-rotate-1544443200-v8bkn 0/1 Completed 0 2h 10.97.196.159 airship ucp keystone-fernet-setup-srrqj 0/1 Completed 0 3d 10.97.196.152 airship ucp keystone-rabbit-init-x4stm 0/1 Completed 0 3d 10.97.196.150 airship ucp maas-bootstrap-admin-user-26lhf 0/1 Init:0/1 0 3d 10.97.196.132 airship ucp maas-db-init-v7z6r 0/1 CrashLoopBackOff 944 3d 10.97.196.143 airship ucp maas-db-sync-5c24z 0/1 Init:0/1 0 3d 10.97.196.162 airship ucp maas-export-api-key-zhfxp 0/1 Init:0/1 0 3d 10.97.196.147 airship ucp maas-import-resources-j6tzd 0/1 Init:0/1 0 3d 10.97.196.152 airship ucp maas-rack-0 0/1 Init:0/1 0 3d 172.16.231.60 airship ucp maas-region-0 0/1 Init:0/1 0 3d 10.97.196.146 airship ucp mariadb-ingress-6c4f9c76f-sg8cx 1/1 Running 0 3d 10.97.196.137 airship ucp mariadb-ingress-6c4f9c76f-sz26m 1/1 Running 0 3d 10.97.196.138 airship ucp mariadb-ingress-error-pages-5dd6fbb75-x2xbn 1/1 Running 0 3d 10.97.196.139 airship ucp mariadb-server-0 1/1 Running 0 3d 10.97.196.142 airship ucp postgresql-0 1/1 Running 1 3d 10.97.196.140 airship The only output from that pod is this: root@airship:~# kubectl logs -n ucp maas-db-init-v7z6r + export HOME=/tmp + HOME=/tmp + [[ ! -v DB_HOST ]] + [[ ! -v ROOT_DB_USER ]] + [[ ! -v PGPASSWORD ]] + [[ ! -v USER_DB_USER ]] + [[ ! -v USER_DB_PASS ]] + [[ ! -v USER_DB_NAME ]] + echo 'Got DB connection info' Got DB connection info + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' + [[ ! -z '' ]] + grep -q 1 + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off + pgsql_superuser_cmd 'CREATE DATABASE maasdb' + DB_COMMAND='CREATE DATABASE maasdb' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE maasdb' psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off Unfortunately, I haven’t been able to troubleshoot this any further. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com> Sent: Monday, December 10, 2018 6:32 AM To: airship-discuss@lists.airshipit.org Subject: [Airship-discuss] Airship in a bottle install failure... Hi there Airshippers I wanted to try installing airhip-in-a-bottle in a fresh Ubuntu16.04 VM running on OpenStack over the weekend. I followed the instructions on airshipit.org --snip sudo -i mkdir -p /root/deploy && cd "$_" git clone https://github.com/openstack/airship-in-a-bottle cd /root/deploy/airship-in-a-bottle/manifests/dev_single_node ./airship-in-a-bottle.sh --snip The script is still running after 2 days so I think something went wrong. The output is repeating itself, listing running/restarted containers (I think postgres is the one not coming up properly) --snip NAMESPACE NAME READY STATUS RESTARTS AGE kube-system auxiliary-etcd-airship 3/3 Running 0 2d kube-system bootstrap-armada-airship 4/4 Running 0 2d kube-system calico-etcd-airship 1/1 Running 0 2d kube-system calico-etcd-anchor-2jrqf 1/1 Running 0 2d kube-system calico-kube-controllers-6d5997c8c4-zxftw 1/1 Running 0 2d kube-system calico-node-l885v 2/2 Running 0 2d kube-system calico-settings-kbq79 0/1 Completed 0 2d kube-system coredns-d4587b7f8-7wlpm 1/1 Running 0 2d kube-system coredns-d4587b7f8-98t62 1/1 Running 0 2d kube-system coredns-d4587b7f8-frx5w 1/1 Running 0 2d kube-system haproxy-airship 1/1 Running 0 2d kube-system ingress-8w29l 1/1 Running 0 2d kube-system ingress-error-pages-5ccf96bf7d-jsrww 1/1 Running 0 2d kube-system kubernetes-apiserver-airship 1/1 Running 0 2d kube-system kubernetes-apiserver-anchor-55x59 1/1 Running 0 2d kube-system kubernetes-controller-manager-airship 1/1 Running 1 2d kube-system kubernetes-controller-manager-anchor-2wsg8 1/1 Running 0 2d kube-system kubernetes-etcd-airship 1/1 Running 0 2d kube-system kubernetes-etcd-anchor-64wgx 1/1 Running 0 2d kube-system kubernetes-proxy-9s68d 1/1 Running 0 2d kube-system kubernetes-scheduler-airship 1/1 Running 1 2d kube-system kubernetes-scheduler-anchor-npmn7 1/1 Running 0 2d kube-system nfs-provisioner-7799d64d59-scgdc 1/1 Running 0 2d ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 2d ucp ingress-6cd5b89d5d-gp98m 1/1 Running 0 2d ucp ingress-error-pages-5c97bb46bb-zq5cw 1/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-bjd99 0/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-xjrn9 0/1 Running 0 2d ucp mariadb-ingress-error-pages-5dd6fbb75-89qmx 1/1 Running 0 2d ucp mariadb-server-0 0/1 Running 0 2d ucp postgresql-0 1/1 Running 1 2d --snip Anybody out there with the same problem or (even better) a solution for it? Should I pick a particular branch or commit? Any help is appreciated. Best Regards, Daniel
Thanks for the reply Daniel. I won't have a chance to troubleshoot it for at least two more days. When I get a chance, I'll try to nsenter the namespaces occupied by that pod and have a look around. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com> Sent: Monday, December 10, 2018 7:59 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com> Subject: Re: Airship in a bottle install failure... Hi there Alex. Thanks for the reply.
From your logs it seems to me like maas-db-init is not able to connect to postgres DB with psql client to the other container. (clients should be listed by IP in pg_hba.conf, wildcards should be possible....)
I will try another install from a fresh VM and let you know if I have the same one. Thanks ________________________________ From: Alex Arnoldy <alex.arnoldy@suse.com<mailto:alex.arnoldy@suse.com>> Sent: Monday, December 10, 2018 4:26 PM To: Balsiger Daniel, INI-EAI-INO-MBL; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: RE: Airship in a bottle install failure... Hello Daniel, I had launched the same thing but then got pulled away from it and totally forgot to check it. I thought I'd be able to tell you I got it running, but I don't think that's the case. For my installation, the maas-db-init pod seems to be stuck in CrashLoopBackOff: root@airship:~# kubectl get pods -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system auxiliary-etcd-airship 3/3 Running 0 3d 172.16.231.60 airship kube-system bootstrap-armada-airship 4/4 Running 0 3d 172.16.231.60 airship kube-system calico-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-etcd-anchor-nqntm 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-kube-controllers-6d5997c8c4-bm9rk 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-node-2sgkk 2/2 Running 0 3d 172.16.231.60 airship kube-system calico-settings-bqs5v 0/1 Completed 0 3d 172.16.231.60 airship kube-system coredns-d4587b7f8-bbpr6 1/1 Running 13 3d 10.97.196.130 airship kube-system coredns-d4587b7f8-tdl8v 1/1 Running 11 3d 10.97.196.131 airship kube-system coredns-d4587b7f8-ts9lr 1/1 Running 12 3d 10.97.196.129 airship kube-system haproxy-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system ingress-error-pages-5ccf96bf7d-8q6g5 1/1 Running 0 3d 10.97.196.133 airship kube-system ingress-fmrmk 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-apiserver-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-apiserver-anchor-rkfpz 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-controller-manager-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-controller-manager-anchor-75k62 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-etcd-anchor-hzcrz 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-proxy-994kg 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-scheduler-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-scheduler-anchor-gsqtj 1/1 Running 0 3d 172.16.231.60 airship kube-system nfs-provisioner-7799d64d59-g2g6n 1/1 Running 0 3d 10.97.196.134 airship kube-system tiller-deploy-7d88c6f956-9jjk7 1/1 Running 0 3d 10.97.196.153 airship ucp airship-ucp-keystone-memcached-memcached-74d79d8896-dd8k2 1/1 Running 0 3d 10.97.196.144 airship ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 2 3d 10.97.196.178 airship ucp ingress-6cd5b89d5d-xx592 1/1 Running 0 3d 10.97.196.135 airship ucp ingress-error-pages-5c97bb46bb-ljmw6 1/1 Running 0 3d 10.97.196.136 airship ucp keystone-api-7657986b8c-b49r6 1/1 Running 0 3d 10.97.196.145 airship ucp keystone-bootstrap-8494p 0/1 Completed 0 3d 10.97.196.147 airship ucp keystone-credential-setup-r5twm 0/1 Completed 0 3d 10.97.196.146 airship ucp keystone-db-init-pxlzr 0/1 Completed 0 3d 10.97.196.148 airship ucp keystone-db-sync-zx665 0/1 Completed 0 3d 10.97.196.151 airship ucp keystone-domain-manage-rbfl5 0/1 Completed 0 3d 10.97.196.149 airship ucp keystone-fernet-rotate-1544356800-kcgg9 0/1 Completed 0 1d 10.97.196.148 airship ucp keystone-fernet-rotate-1544400000-78qc8 0/1 Completed 0 14h 10.97.196.163 airship ucp keystone-fernet-rotate-1544443200-v8bkn 0/1 Completed 0 2h 10.97.196.159 airship ucp keystone-fernet-setup-srrqj 0/1 Completed 0 3d 10.97.196.152 airship ucp keystone-rabbit-init-x4stm 0/1 Completed 0 3d 10.97.196.150 airship ucp maas-bootstrap-admin-user-26lhf 0/1 Init:0/1 0 3d 10.97.196.132 airship ucp maas-db-init-v7z6r 0/1 CrashLoopBackOff 944 3d 10.97.196.143 airship ucp maas-db-sync-5c24z 0/1 Init:0/1 0 3d 10.97.196.162 airship ucp maas-export-api-key-zhfxp 0/1 Init:0/1 0 3d 10.97.196.147 airship ucp maas-import-resources-j6tzd 0/1 Init:0/1 0 3d 10.97.196.152 airship ucp maas-rack-0 0/1 Init:0/1 0 3d 172.16.231.60 airship ucp maas-region-0 0/1 Init:0/1 0 3d 10.97.196.146 airship ucp mariadb-ingress-6c4f9c76f-sg8cx 1/1 Running 0 3d 10.97.196.137 airship ucp mariadb-ingress-6c4f9c76f-sz26m 1/1 Running 0 3d 10.97.196.138 airship ucp mariadb-ingress-error-pages-5dd6fbb75-x2xbn 1/1 Running 0 3d 10.97.196.139 airship ucp mariadb-server-0 1/1 Running 0 3d 10.97.196.142 airship ucp postgresql-0 1/1 Running 1 3d 10.97.196.140 airship The only output from that pod is this: root@airship:~# kubectl logs -n ucp maas-db-init-v7z6r + export HOME=/tmp + HOME=/tmp + [[ ! -v DB_HOST ]] + [[ ! -v ROOT_DB_USER ]] + [[ ! -v PGPASSWORD ]] + [[ ! -v USER_DB_USER ]] + [[ ! -v USER_DB_PASS ]] + [[ ! -v USER_DB_NAME ]] + echo 'Got DB connection info' Got DB connection info + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' + [[ ! -z '' ]] + grep -q 1 + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off + pgsql_superuser_cmd 'CREATE DATABASE maasdb' + DB_COMMAND='CREATE DATABASE maasdb' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE maasdb' psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off Unfortunately, I haven't been able to troubleshoot this any further. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com<mailto:Daniel.Balsiger@swisscom.com>> Sent: Monday, December 10, 2018 6:32 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: [Airship-discuss] Airship in a bottle install failure... Hi there Airshippers I wanted to try installing airhip-in-a-bottle in a fresh Ubuntu16.04 VM running on OpenStack over the weekend. I followed the instructions on airshipit.org --snip sudo -i mkdir -p /root/deploy && cd "$_" git clone https://github.com/openstack/airship-in-a-bottle cd /root/deploy/airship-in-a-bottle/manifests/dev_single_node ./airship-in-a-bottle.sh --snip The script is still running after 2 days so I think something went wrong. The output is repeating itself, listing running/restarted containers (I think postgres is the one not coming up properly) --snip NAMESPACE NAME READY STATUS RESTARTS AGE kube-system auxiliary-etcd-airship 3/3 Running 0 2d kube-system bootstrap-armada-airship 4/4 Running 0 2d kube-system calico-etcd-airship 1/1 Running 0 2d kube-system calico-etcd-anchor-2jrqf 1/1 Running 0 2d kube-system calico-kube-controllers-6d5997c8c4-zxftw 1/1 Running 0 2d kube-system calico-node-l885v 2/2 Running 0 2d kube-system calico-settings-kbq79 0/1 Completed 0 2d kube-system coredns-d4587b7f8-7wlpm 1/1 Running 0 2d kube-system coredns-d4587b7f8-98t62 1/1 Running 0 2d kube-system coredns-d4587b7f8-frx5w 1/1 Running 0 2d kube-system haproxy-airship 1/1 Running 0 2d kube-system ingress-8w29l 1/1 Running 0 2d kube-system ingress-error-pages-5ccf96bf7d-jsrww 1/1 Running 0 2d kube-system kubernetes-apiserver-airship 1/1 Running 0 2d kube-system kubernetes-apiserver-anchor-55x59 1/1 Running 0 2d kube-system kubernetes-controller-manager-airship 1/1 Running 1 2d kube-system kubernetes-controller-manager-anchor-2wsg8 1/1 Running 0 2d kube-system kubernetes-etcd-airship 1/1 Running 0 2d kube-system kubernetes-etcd-anchor-64wgx 1/1 Running 0 2d kube-system kubernetes-proxy-9s68d 1/1 Running 0 2d kube-system kubernetes-scheduler-airship 1/1 Running 1 2d kube-system kubernetes-scheduler-anchor-npmn7 1/1 Running 0 2d kube-system nfs-provisioner-7799d64d59-scgdc 1/1 Running 0 2d ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 2d ucp ingress-6cd5b89d5d-gp98m 1/1 Running 0 2d ucp ingress-error-pages-5c97bb46bb-zq5cw 1/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-bjd99 0/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-xjrn9 0/1 Running 0 2d ucp mariadb-ingress-error-pages-5dd6fbb75-89qmx 1/1 Running 0 2d ucp mariadb-server-0 0/1 Running 0 2d ucp postgresql-0 1/1 Running 1 2d --snip Anybody out there with the same problem or (even better) a solution for it? Should I pick a particular branch or commit? Any help is appreciated. Best Regards, Daniel
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that "no pg_hba.conf" error current aiab code, or can you give git commit where you're hitting the issue? Daniel, can you please run "sudo kubectl -n ucp logs mariadb-server-0" and share the results? Looks like MariaDB isn't starting cleanly for some reason. Thanks, Matt From: Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com> Sent: Monday, December 10, 2018 9:59 AM To: Alex Arnoldy <alex.arnoldy@suse.com>; airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi there Alex. Thanks for the reply.
From your logs it seems to me like maas-db-init is not able to connect to postgres DB with psql client to the other container. (clients should be listed by IP in pg_hba.conf, wildcards should be possible....)
I will try another install from a fresh VM and let you know if I have the same one. Thanks ________________________________ From: Alex Arnoldy <alex.arnoldy@suse.com<mailto:alex.arnoldy@suse.com>> Sent: Monday, December 10, 2018 4:26 PM To: Balsiger Daniel, INI-EAI-INO-MBL; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: RE: Airship in a bottle install failure... Hello Daniel, I had launched the same thing but then got pulled away from it and totally forgot to check it. I thought I'd be able to tell you I got it running, but I don't think that's the case. For my installation, the maas-db-init pod seems to be stuck in CrashLoopBackOff: root@airship:~# kubectl get pods -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system auxiliary-etcd-airship 3/3 Running 0 3d 172.16.231.60 airship kube-system bootstrap-armada-airship 4/4 Running 0 3d 172.16.231.60 airship kube-system calico-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-etcd-anchor-nqntm 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-kube-controllers-6d5997c8c4-bm9rk 1/1 Running 0 3d 172.16.231.60 airship kube-system calico-node-2sgkk 2/2 Running 0 3d 172.16.231.60 airship kube-system calico-settings-bqs5v 0/1 Completed 0 3d 172.16.231.60 airship kube-system coredns-d4587b7f8-bbpr6 1/1 Running 13 3d 10.97.196.130 airship kube-system coredns-d4587b7f8-tdl8v 1/1 Running 11 3d 10.97.196.131 airship kube-system coredns-d4587b7f8-ts9lr 1/1 Running 12 3d 10.97.196.129 airship kube-system haproxy-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system ingress-error-pages-5ccf96bf7d-8q6g5 1/1 Running 0 3d 10.97.196.133 airship kube-system ingress-fmrmk 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-apiserver-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-apiserver-anchor-rkfpz 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-controller-manager-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-controller-manager-anchor-75k62 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-etcd-anchor-hzcrz 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-proxy-994kg 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-scheduler-airship 1/1 Running 0 3d 172.16.231.60 airship kube-system kubernetes-scheduler-anchor-gsqtj 1/1 Running 0 3d 172.16.231.60 airship kube-system nfs-provisioner-7799d64d59-g2g6n 1/1 Running 0 3d 10.97.196.134 airship kube-system tiller-deploy-7d88c6f956-9jjk7 1/1 Running 0 3d 10.97.196.153 airship ucp airship-ucp-keystone-memcached-memcached-74d79d8896-dd8k2 1/1 Running 0 3d 10.97.196.144 airship ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 2 3d 10.97.196.178 airship ucp ingress-6cd5b89d5d-xx592 1/1 Running 0 3d 10.97.196.135 airship ucp ingress-error-pages-5c97bb46bb-ljmw6 1/1 Running 0 3d 10.97.196.136 airship ucp keystone-api-7657986b8c-b49r6 1/1 Running 0 3d 10.97.196.145 airship ucp keystone-bootstrap-8494p 0/1 Completed 0 3d 10.97.196.147 airship ucp keystone-credential-setup-r5twm 0/1 Completed 0 3d 10.97.196.146 airship ucp keystone-db-init-pxlzr 0/1 Completed 0 3d 10.97.196.148 airship ucp keystone-db-sync-zx665 0/1 Completed 0 3d 10.97.196.151 airship ucp keystone-domain-manage-rbfl5 0/1 Completed 0 3d 10.97.196.149 airship ucp keystone-fernet-rotate-1544356800-kcgg9 0/1 Completed 0 1d 10.97.196.148 airship ucp keystone-fernet-rotate-1544400000-78qc8 0/1 Completed 0 14h 10.97.196.163 airship ucp keystone-fernet-rotate-1544443200-v8bkn 0/1 Completed 0 2h 10.97.196.159 airship ucp keystone-fernet-setup-srrqj 0/1 Completed 0 3d 10.97.196.152 airship ucp keystone-rabbit-init-x4stm 0/1 Completed 0 3d 10.97.196.150 airship ucp maas-bootstrap-admin-user-26lhf 0/1 Init:0/1 0 3d 10.97.196.132 airship ucp maas-db-init-v7z6r 0/1 CrashLoopBackOff 944 3d 10.97.196.143 airship ucp maas-db-sync-5c24z 0/1 Init:0/1 0 3d 10.97.196.162 airship ucp maas-export-api-key-zhfxp 0/1 Init:0/1 0 3d 10.97.196.147 airship ucp maas-import-resources-j6tzd 0/1 Init:0/1 0 3d 10.97.196.152 airship ucp maas-rack-0 0/1 Init:0/1 0 3d 172.16.231.60 airship ucp maas-region-0 0/1 Init:0/1 0 3d 10.97.196.146 airship ucp mariadb-ingress-6c4f9c76f-sg8cx 1/1 Running 0 3d 10.97.196.137 airship ucp mariadb-ingress-6c4f9c76f-sz26m 1/1 Running 0 3d 10.97.196.138 airship ucp mariadb-ingress-error-pages-5dd6fbb75-x2xbn 1/1 Running 0 3d 10.97.196.139 airship ucp mariadb-server-0 1/1 Running 0 3d 10.97.196.142 airship ucp postgresql-0 1/1 Running 1 3d 10.97.196.140 airship The only output from that pod is this: root@airship:~# kubectl logs -n ucp maas-db-init-v7z6r + export HOME=/tmp + HOME=/tmp + [[ ! -v DB_HOST ]] + [[ ! -v ROOT_DB_USER ]] + [[ ! -v PGPASSWORD ]] + [[ ! -v USER_DB_USER ]] + [[ ! -v USER_DB_PASS ]] + [[ ! -v USER_DB_NAME ]] + echo 'Got DB connection info' Got DB connection info + pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' + DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' + [[ ! -z '' ]] + grep -q 1 + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '\''maasdb'\''' psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off + pgsql_superuser_cmd 'CREATE DATABASE maasdb' + DB_COMMAND='CREATE DATABASE maasdb' + [[ ! -z '' ]] + psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE maasdb' psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off Unfortunately, I haven't been able to troubleshoot this any further. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com<mailto:Daniel.Balsiger@swisscom.com>> Sent: Monday, December 10, 2018 6:32 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org> Subject: [Airship-discuss] Airship in a bottle install failure... Hi there Airshippers I wanted to try installing airhip-in-a-bottle in a fresh Ubuntu16.04 VM running on OpenStack over the weekend. I followed the instructions on airshipit.org --snip sudo -i mkdir -p /root/deploy && cd "$_" git clone https://github.com/openstack/airship-in-a-bottle<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMF-g&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=0ETQscoyeMAsqsF5ioGCgQDwKgFpjQwObZuJnDEAbG4&s=a9KACbmNQfoW9NpqhEG3WdJLO1F9JljHgAjxTITi3fE&e=> cd /root/deploy/airship-in-a-bottle/manifests/dev_single_node ./airship-in-a-bottle.sh --snip The script is still running after 2 days so I think something went wrong. The output is repeating itself, listing running/restarted containers (I think postgres is the one not coming up properly) --snip NAMESPACE NAME READY STATUS RESTARTS AGE kube-system auxiliary-etcd-airship 3/3 Running 0 2d kube-system bootstrap-armada-airship 4/4 Running 0 2d kube-system calico-etcd-airship 1/1 Running 0 2d kube-system calico-etcd-anchor-2jrqf 1/1 Running 0 2d kube-system calico-kube-controllers-6d5997c8c4-zxftw 1/1 Running 0 2d kube-system calico-node-l885v 2/2 Running 0 2d kube-system calico-settings-kbq79 0/1 Completed 0 2d kube-system coredns-d4587b7f8-7wlpm 1/1 Running 0 2d kube-system coredns-d4587b7f8-98t62 1/1 Running 0 2d kube-system coredns-d4587b7f8-frx5w 1/1 Running 0 2d kube-system haproxy-airship 1/1 Running 0 2d kube-system ingress-8w29l 1/1 Running 0 2d kube-system ingress-error-pages-5ccf96bf7d-jsrww 1/1 Running 0 2d kube-system kubernetes-apiserver-airship 1/1 Running 0 2d kube-system kubernetes-apiserver-anchor-55x59 1/1 Running 0 2d kube-system kubernetes-controller-manager-airship 1/1 Running 1 2d kube-system kubernetes-controller-manager-anchor-2wsg8 1/1 Running 0 2d kube-system kubernetes-etcd-airship 1/1 Running 0 2d kube-system kubernetes-etcd-anchor-64wgx 1/1 Running 0 2d kube-system kubernetes-proxy-9s68d 1/1 Running 0 2d kube-system kubernetes-scheduler-airship 1/1 Running 1 2d kube-system kubernetes-scheduler-anchor-npmn7 1/1 Running 0 2d kube-system nfs-provisioner-7799d64d59-scgdc 1/1 Running 0 2d ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 2d ucp ingress-6cd5b89d5d-gp98m 1/1 Running 0 2d ucp ingress-error-pages-5c97bb46bb-zq5cw 1/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-bjd99 0/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-xjrn9 0/1 Running 0 2d ucp mariadb-ingress-error-pages-5dd6fbb75-89qmx 1/1 Running 0 2d ucp mariadb-server-0 0/1 Running 0 2d ucp postgresql-0 1/1 Running 1 2d --snip Anybody out there with the same problem or (even better) a solution for it? Should I pick a particular branch or commit? Any help is appreciated. Best Regards, Daniel
Hi Matt, Here is the commit I'm using: commit 221441f1bff351ab366b02d6fd7856b70b06108f Author: Bryan Strassner <bryan.strassner@gmail.com> Date: Mon Nov 19 17:12:10 2018 -0600 Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 I'm happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I'd preserve the environment for examination. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: MCEUEN, MATT <MM9745@att.com> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: RE: Airship in a bottle install failure... Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that "no pg_hba.conf" error current aiab code, or can you give git commit where you're hitting the issue? Daniel, can you please run "sudo kubectl -n ucp logs mariadb-server-0" and share the results? Looks like MariaDB isn't starting cleanly for some reason. Thanks, Matt
Hi there everyone Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting: I see in Postgresql-0 logs: --snip FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip and in mariadb-server-0 logs: --snip ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip Best Regards, D ________________________________ From: Alex Arnoldy <alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure... Hi Matt, Here is the commit I’m using: commit 221441f1bff351ab366b02d6fd7856b70b06108f Author: Bryan Strassner <bryan.strassner@gmail.com> Date: Mon Nov 19 17:12:10 2018 -0600 Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: MCEUEN, MATT <MM9745@att.com> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: RE: Airship in a bottle install failure... Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue? Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason. Thanks, Matt
Hi Alex! Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. Regards, Drew Walters On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL < Daniel.Balsiger@swisscom.com> wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip
FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip
and in mariadb-server-0 logs:
--snip
ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip
Best Regards, D
------------------------------ *From:* Alex Arnoldy <alex.arnoldy@suse.com> *Sent:* Monday, December 10, 2018 7:48 PM *To:* MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL *Subject:* RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner <bryan.strassner@gmail.com>
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller
Update the apiserver's command prefix to match between the apiserver
chart and the genesis api server.
Includes an optimization of virsh commands to parallelize and wait for
cloud init to complete (requires Ubuntu 16.04.4)
Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434
I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
*Alex Arnoldy*
* Software Defined Infrastructure Solutions Architect SUSE * *Alex.Arnoldy@SUSE.com* <Alex.Arnoldy@SUSE.com> * (P) +1 (408) 828-9797*
*From:* MCEUEN, MATT <MM9745@att.com> *Sent:* Monday, December 10, 2018 9:33 AM *To:* airship-discuss@lists.airshipit.org; Alex Arnoldy < alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel < Daniel.Balsiger@swisscom.com> *Subject:* RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi Drew, Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two. I’ll definitely let you know how things go. Very best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: Drew Walters <drewwalters96@gmail.com> Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi Alex! Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. Regards, Drew Walters On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com<mailto:Daniel.Balsiger@swisscom.com>> wrote: Hi there everyone Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting: I see in Postgresql-0 logs: --snip FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip and in mariadb-server-0 logs: --snip ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip Best Regards, D ________________________________ From: Alex Arnoldy <alex.arnoldy@suse.com<mailto:alex.arnoldy@suse.com>> Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org>; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure... Hi Matt, Here is the commit I’m using: commit 221441f1bff351ab366b02d6fd7856b70b06108f Author: Bryan Strassner <bryan.strassner@gmail.com<mailto:bryan.strassner@gmail.com>> Date: Mon Nov 19 17:12:10 2018 -0600 Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: MCEUEN, MATT <MM9745@att.com<mailto:MM9745@att.com>> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org>; Alex Arnoldy <alex.arnoldy@suse.com<mailto:alex.arnoldy@suse.com>>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com<mailto:Daniel.Balsiger@swisscom.com>> Subject: RE: Airship in a bottle install failure... Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue? Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason. Thanks, Matt _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org<mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hello all, I have in-a-bottle deployment running right now for about two hours. Will let you know if it succeeds (it should). Best regards, -- Roman Gorshunov On Mon, Jan 7, 2019 at 12:52 PM Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two.
I’ll definitely let you know how things go.
Very best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Drew Walters <drewwalters96@gmail.com> Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi Alex!
Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.
Regards,
Drew Walters
On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com> wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip
FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress
--snip
and in mariadb-server-0 logs:
--snip
ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
--snip
Best Regards, D
________________________________
From: Alex Arnoldy <alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner <bryan.strassner@gmail.com>
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller
Update the apiserver's command prefix to match between the apiserver
chart and the genesis api server.
Includes an optimization of virsh commands to parallelize and wait for
cloud init to complete (requires Ubuntu 16.04.4)
Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434
I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT <MM9745@att.com> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hello all, My deployment has just completed successfully. Git commit id a9f5fc93e33163f51fc56d1bf886d4d433251f8e (it's current master). -- Roman Gorshunov On Mon, Jan 7, 2019 at 12:57 PM Roman Gorshunov <paye600@gmail.com> wrote:
Hello all, I have in-a-bottle deployment running right now for about two hours. Will let you know if it succeeds (it should).
Best regards, -- Roman Gorshunov
On Mon, Jan 7, 2019 at 12:52 PM Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two.
I’ll definitely let you know how things go.
Very best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Drew Walters <drewwalters96@gmail.com> Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi Alex!
Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.
Regards,
Drew Walters
On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com> wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip
FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress
--snip
and in mariadb-server-0 logs:
--snip
ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
--snip
Best Regards, D
________________________________
From: Alex Arnoldy <alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner <bryan.strassner@gmail.com>
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller
Update the apiserver's command prefix to match between the apiserver
chart and the genesis api server.
Includes an optimization of virsh commands to parallelize and wait for
cloud init to complete (requires Ubuntu 16.04.4)
Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434
I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT <MM9745@att.com> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi Drew, I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info. As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface. Sorry that’s not much info to go on, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: Alex Arnoldy Sent: Monday, January 7, 2019 3:12 AM To: 'Drew Walters' <drewwalters96@gmail.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org Subject: RE: [Airship-discuss] Airship in a bottle install failure... Hi Drew, Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two. I’ll definitely let you know how things go. Very best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: Drew Walters <drewwalters96@gmail.com<mailto:drewwalters96@gmail.com>> Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com<mailto:Daniel.Balsiger@swisscom.com>> Cc: MATT MCEUEN <MM9745@att.com<mailto:MM9745@att.com>>; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org>; Alex Arnoldy <alex.arnoldy@suse.com<mailto:alex.arnoldy@suse.com>> Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi Alex! Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above. Regards, Drew Walters On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.com<mailto:Daniel.Balsiger@swisscom.com>> wrote: Hi there everyone Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting: I see in Postgresql-0 logs: --snip FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip and in mariadb-server-0 logs: --snip ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip Best Regards, D ________________________________ From: Alex Arnoldy <alex.arnoldy@suse.com<mailto:alex.arnoldy@suse.com>> Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org>; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure... Hi Matt, Here is the commit I’m using: commit 221441f1bff351ab366b02d6fd7856b70b06108f Author: Bryan Strassner <bryan.strassner@gmail.com<mailto:bryan.strassner@gmail.com>> Date: Mon Nov 19 17:12:10 2018 -0600 Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434 I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com<mailto:Alex.Arnoldy@SUSE.com> (P) +1 (408) 828-9797 From: MCEUEN, MATT <MM9745@att.com<mailto:MM9745@att.com>> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org<mailto:airship-discuss@lists.airshipit.org>; Alex Arnoldy <alex.arnoldy@suse.com<mailto:alex.arnoldy@suse.com>>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com<mailto:Daniel.Balsiger@swisscom.com>> Subject: RE: Airship in a bottle install failure... Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue? Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason. Thanks, Matt _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org<mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex, Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try. Drew
Hi Drew, Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot. Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out. Best, --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797 -----Original Message----- From: Drew Walters <drewwalters96@gmail.com> Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy <alex.arnoldy@suse.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure... On Jan 15, 2019, at 4:57 PM, Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex, Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try. Drew
Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good. Regards, -- Roman Gorshunov On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot.
Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Drew Walters <drewwalters96@gmail.com> Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy <alex.arnoldy@suse.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure...
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Thanks. Maybe I'll give that one a shot. It won't shed any light into the previous issues I've had but it's also possible that those problems were related to intermittent keyboard driver problems I've seen in the past. --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797 -----Original Message----- From: Roman Gorshunov <paye600@gmail.com> Sent: Friday, January 18, 2019 10:36 AM To: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good. Regards, -- Roman Gorshunov On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot.
Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Drew Walters <drewwalters96@gmail.com> Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy <alex.arnoldy@suse.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure...
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
HI Alex, yeah - the issue was pre-installed libvirt causing issues. As long as it's standard Ubuntu 1604 - that's all you need. Kindly, Kaspars ________________________________________ From: Alex Arnoldy [alex.arnoldy@suse.com] Sent: Friday, January 18, 2019 12:55 PM To: Roman Gorshunov; airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure... Thanks. Maybe I'll give that one a shot. It won't shed any light into the previous issues I've had but it's also possible that those problems were related to intermittent keyboard driver problems I've seen in the past. --Alex. Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797 -----Original Message----- From: Roman Gorshunov <paye600@gmail.com> Sent: Friday, January 18, 2019 10:36 AM To: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure... Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good. Regards, -- Roman Gorshunov On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot.
Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Drew Walters <drewwalters96@gmail.com> Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy <alex.arnoldy@suse.com> Cc: MATT MCEUEN <MM9745@att.com>; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure...
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy <alex.arnoldy@suse.com> wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=1QkZWVrNKsc-yvwNebiLcq02zjfbK6RHbjuVQGIR0lU&s=y-pWlk22qanjnSJNO7kZmh2X1A7hN8sfUHBqczBl_AY&e=
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=1QkZWVrNKsc-yvwNebiLcq02zjfbK6RHbjuVQGIR0lU&s=y-pWlk22qanjnSJNO7kZmh2X1A7hN8sfUHBqczBl_AY&e= _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwIGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=L2WcX1Ge33cpIHXX6krAgw&m=1QkZWVrNKsc-yvwNebiLcq02zjfbK6RHbjuVQGIR0lU&s=y-pWlk22qanjnSJNO7kZmh2X1A7hN8sfUHBqczBl_AY&e=
participants (6)
-
Alex Arnoldy
-
Balsiger Daniel, INI-EAI-INO-MBL
-
Drew Walters
-
MCEUEN, MATT
-
Roman Gorshunov
-
SKELS, KASPARS