Re: [Airship-discuss] Airship in a bottle install failure...
Hi there Alex.
Thanks for the reply.
From your logs it seems to me like maas-db-init is not able to connect to postgres DB with psql client to the other container. (clients should be listed by IP in pg_hba.conf, wildcards should be possible....)
I will try another install from a fresh VM and let you know if I have the same one.
Thanks
________________________________ From: Alex Arnoldy alex.arnoldy@suse.com Sent: Monday, December 10, 2018 4:26 PM To: Balsiger Daniel, INI-EAI-INO-MBL; airship-discuss@lists.airshipit.org Subject: RE: Airship in a bottle install failure...
Hello Daniel,
I had launched the same thing but then got pulled away from it and totally forgot to check it. I thought I’d be able to tell you I got it running, but I don’t think that’s the case. For my installation, the maas-db-init pod seems to be stuck in CrashLoopBackOff:
root@airship:~# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system auxiliary-etcd-airship 3/3 Running 0 3d 172.16.231.60 airship
kube-system bootstrap-armada-airship 4/4 Running 0 3d 172.16.231.60 airship
kube-system calico-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-etcd-anchor-nqntm 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-kube-controllers-6d5997c8c4-bm9rk 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-node-2sgkk 2/2 Running 0 3d 172.16.231.60 airship
kube-system calico-settings-bqs5v 0/1 Completed 0 3d 172.16.231.60 airship
kube-system coredns-d4587b7f8-bbpr6 1/1 Running 13 3d 10.97.196.130 airship
kube-system coredns-d4587b7f8-tdl8v 1/1 Running 11 3d 10.97.196.131 airship
kube-system coredns-d4587b7f8-ts9lr 1/1 Running 12 3d 10.97.196.129 airship
kube-system haproxy-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system ingress-error-pages-5ccf96bf7d-8q6g5 1/1 Running 0 3d 10.97.196.133 airship
kube-system ingress-fmrmk 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-apiserver-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-apiserver-anchor-rkfpz 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-controller-manager-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-controller-manager-anchor-75k62 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-etcd-anchor-hzcrz 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-proxy-994kg 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-scheduler-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-scheduler-anchor-gsqtj 1/1 Running 0 3d 172.16.231.60 airship
kube-system nfs-provisioner-7799d64d59-g2g6n 1/1 Running 0 3d 10.97.196.134 airship
kube-system tiller-deploy-7d88c6f956-9jjk7 1/1 Running 0 3d 10.97.196.153 airship
ucp airship-ucp-keystone-memcached-memcached-74d79d8896-dd8k2 1/1 Running 0 3d 10.97.196.144 airship
ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 2 3d 10.97.196.178 airship
ucp ingress-6cd5b89d5d-xx592 1/1 Running 0 3d 10.97.196.135 airship
ucp ingress-error-pages-5c97bb46bb-ljmw6 1/1 Running 0 3d 10.97.196.136 airship
ucp keystone-api-7657986b8c-b49r6 1/1 Running 0 3d 10.97.196.145 airship
ucp keystone-bootstrap-8494p 0/1 Completed 0 3d 10.97.196.147 airship
ucp keystone-credential-setup-r5twm 0/1 Completed 0 3d 10.97.196.146 airship
ucp keystone-db-init-pxlzr 0/1 Completed 0 3d 10.97.196.148 airship
ucp keystone-db-sync-zx665 0/1 Completed 0 3d 10.97.196.151 airship
ucp keystone-domain-manage-rbfl5 0/1 Completed 0 3d 10.97.196.149 airship
ucp keystone-fernet-rotate-1544356800-kcgg9 0/1 Completed 0 1d 10.97.196.148 airship
ucp keystone-fernet-rotate-1544400000-78qc8 0/1 Completed 0 14h 10.97.196.163 airship
ucp keystone-fernet-rotate-1544443200-v8bkn 0/1 Completed 0 2h 10.97.196.159 airship
ucp keystone-fernet-setup-srrqj 0/1 Completed 0 3d 10.97.196.152 airship
ucp keystone-rabbit-init-x4stm 0/1 Completed 0 3d 10.97.196.150 airship
ucp maas-bootstrap-admin-user-26lhf 0/1 Init:0/1 0 3d 10.97.196.132 airship
ucp maas-db-init-v7z6r 0/1 CrashLoopBackOff 944 3d 10.97.196.143 airship
ucp maas-db-sync-5c24z 0/1 Init:0/1 0 3d 10.97.196.162 airship
ucp maas-export-api-key-zhfxp 0/1 Init:0/1 0 3d 10.97.196.147 airship
ucp maas-import-resources-j6tzd 0/1 Init:0/1 0 3d 10.97.196.152 airship
ucp maas-rack-0 0/1 Init:0/1 0 3d 172.16.231.60 airship
ucp maas-region-0 0/1 Init:0/1 0 3d 10.97.196.146 airship
ucp mariadb-ingress-6c4f9c76f-sg8cx 1/1 Running 0 3d 10.97.196.137 airship
ucp mariadb-ingress-6c4f9c76f-sz26m 1/1 Running 0 3d 10.97.196.138 airship
ucp mariadb-ingress-error-pages-5dd6fbb75-x2xbn 1/1 Running 0 3d 10.97.196.139 airship
ucp mariadb-server-0 1/1 Running 0 3d 10.97.196.142 airship
ucp postgresql-0 1/1 Running 1 3d 10.97.196.140 airship
The only output from that pod is this:
root@airship:~# kubectl logs -n ucp maas-db-init-v7z6r
+ export HOME=/tmp
+ HOME=/tmp
+ [[ ! -v DB_HOST ]]
+ [[ ! -v ROOT_DB_USER ]]
+ [[ ! -v PGPASSWORD ]]
+ [[ ! -v USER_DB_USER ]]
+ [[ ! -v USER_DB_PASS ]]
+ [[ ! -v USER_DB_NAME ]]
+ echo 'Got DB connection info'
Got DB connection info
+ pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
+ DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
+ [[ ! -z '' ]]
+ grep -q 1
+ psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off
+ pgsql_superuser_cmd 'CREATE DATABASE maasdb'
+ DB_COMMAND='CREATE DATABASE maasdb'
+ [[ ! -z '' ]]
+ psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE maasdb'
psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off
Unfortunately, I haven’t been able to troubleshoot this any further.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Balsiger Daniel, INI-EAI-INO-MBL Daniel.Balsiger@swisscom.com Sent: Monday, December 10, 2018 6:32 AM To: airship-discuss@lists.airshipit.org Subject: [Airship-discuss] Airship in a bottle install failure...
Hi there Airshippers
I wanted to try installing airhip-in-a-bottle in a fresh Ubuntu16.04 VM running on OpenStack over the weekend. I followed the instructions on airshipit.org
--snip
sudo -i
mkdir -p /root/deploy && cd "$_"
git clone https://github.com/openstack/airship-in-a-bottle
cd /root/deploy/airship-in-a-bottle/manifests/dev_single_node
./airship-in-a-bottle.sh
--snip
The script is still running after 2 days so I think something went wrong.
The output is repeating itself, listing running/restarted containers (I think postgres is the one not coming up properly)
--snip
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system auxiliary-etcd-airship 3/3 Running 0 2d kube-system bootstrap-armada-airship 4/4 Running 0 2d kube-system calico-etcd-airship 1/1 Running 0 2d kube-system calico-etcd-anchor-2jrqf 1/1 Running 0 2d kube-system calico-kube-controllers-6d5997c8c4-zxftw 1/1 Running 0 2d kube-system calico-node-l885v 2/2 Running 0 2d kube-system calico-settings-kbq79 0/1 Completed 0 2d kube-system coredns-d4587b7f8-7wlpm 1/1 Running 0 2d kube-system coredns-d4587b7f8-98t62 1/1 Running 0 2d kube-system coredns-d4587b7f8-frx5w 1/1 Running 0 2d kube-system haproxy-airship 1/1 Running 0 2d kube-system ingress-8w29l 1/1 Running 0 2d kube-system ingress-error-pages-5ccf96bf7d-jsrww 1/1 Running 0 2d kube-system kubernetes-apiserver-airship 1/1 Running 0 2d kube-system kubernetes-apiserver-anchor-55x59 1/1 Running 0 2d kube-system kubernetes-controller-manager-airship 1/1 Running 1 2d kube-system kubernetes-controller-manager-anchor-2wsg8 1/1 Running 0 2d kube-system kubernetes-etcd-airship 1/1 Running 0 2d kube-system kubernetes-etcd-anchor-64wgx 1/1 Running 0 2d kube-system kubernetes-proxy-9s68d 1/1 Running 0 2d kube-system kubernetes-scheduler-airship 1/1 Running 1 2d kube-system kubernetes-scheduler-anchor-npmn7 1/1 Running 0 2d kube-system nfs-provisioner-7799d64d59-scgdc 1/1 Running 0 2d ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 2d ucp ingress-6cd5b89d5d-gp98m 1/1 Running 0 2d ucp ingress-error-pages-5c97bb46bb-zq5cw 1/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-bjd99 0/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-xjrn9 0/1 Running 0 2d ucp mariadb-ingress-error-pages-5dd6fbb75-89qmx 1/1 Running 0 2d ucp mariadb-server-0 0/1 Running 0 2d ucp postgresql-0 1/1 Running 1 2d --snip
Anybody out there with the same problem or (even better) a solution for it?
Should I pick a particular branch or commit?
Any help is appreciated.
Best Regards, Daniel
Thanks for the reply Daniel.
I won't have a chance to troubleshoot it for at least two more days. When I get a chance, I'll try to nsenter the namespaces occupied by that pod and have a look around.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Balsiger Daniel, INI-EAI-INO-MBL Daniel.Balsiger@swisscom.com Sent: Monday, December 10, 2018 7:59 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com Subject: Re: Airship in a bottle install failure...
Hi there Alex.
Thanks for the reply.
From your logs it seems to me like maas-db-init is not able to connect to postgres DB with psql client to the other container. (clients should be listed by IP in pg_hba.conf, wildcards should be possible....)
I will try another install from a fresh VM and let you know if I have the same one.
Thanks
________________________________ From: Alex Arnoldy <alex.arnoldy@suse.commailto:alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 4:26 PM To: Balsiger Daniel, INI-EAI-INO-MBL; airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org Subject: RE: Airship in a bottle install failure...
Hello Daniel,
I had launched the same thing but then got pulled away from it and totally forgot to check it. I thought I'd be able to tell you I got it running, but I don't think that's the case. For my installation, the maas-db-init pod seems to be stuck in CrashLoopBackOff:
root@airship:~# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system auxiliary-etcd-airship 3/3 Running 0 3d 172.16.231.60 airship
kube-system bootstrap-armada-airship 4/4 Running 0 3d 172.16.231.60 airship
kube-system calico-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-etcd-anchor-nqntm 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-kube-controllers-6d5997c8c4-bm9rk 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-node-2sgkk 2/2 Running 0 3d 172.16.231.60 airship
kube-system calico-settings-bqs5v 0/1 Completed 0 3d 172.16.231.60 airship
kube-system coredns-d4587b7f8-bbpr6 1/1 Running 13 3d 10.97.196.130 airship
kube-system coredns-d4587b7f8-tdl8v 1/1 Running 11 3d 10.97.196.131 airship
kube-system coredns-d4587b7f8-ts9lr 1/1 Running 12 3d 10.97.196.129 airship
kube-system haproxy-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system ingress-error-pages-5ccf96bf7d-8q6g5 1/1 Running 0 3d 10.97.196.133 airship
kube-system ingress-fmrmk 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-apiserver-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-apiserver-anchor-rkfpz 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-controller-manager-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-controller-manager-anchor-75k62 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-etcd-anchor-hzcrz 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-proxy-994kg 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-scheduler-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-scheduler-anchor-gsqtj 1/1 Running 0 3d 172.16.231.60 airship
kube-system nfs-provisioner-7799d64d59-g2g6n 1/1 Running 0 3d 10.97.196.134 airship
kube-system tiller-deploy-7d88c6f956-9jjk7 1/1 Running 0 3d 10.97.196.153 airship
ucp airship-ucp-keystone-memcached-memcached-74d79d8896-dd8k2 1/1 Running 0 3d 10.97.196.144 airship
ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 2 3d 10.97.196.178 airship
ucp ingress-6cd5b89d5d-xx592 1/1 Running 0 3d 10.97.196.135 airship
ucp ingress-error-pages-5c97bb46bb-ljmw6 1/1 Running 0 3d 10.97.196.136 airship
ucp keystone-api-7657986b8c-b49r6 1/1 Running 0 3d 10.97.196.145 airship
ucp keystone-bootstrap-8494p 0/1 Completed 0 3d 10.97.196.147 airship
ucp keystone-credential-setup-r5twm 0/1 Completed 0 3d 10.97.196.146 airship
ucp keystone-db-init-pxlzr 0/1 Completed 0 3d 10.97.196.148 airship
ucp keystone-db-sync-zx665 0/1 Completed 0 3d 10.97.196.151 airship
ucp keystone-domain-manage-rbfl5 0/1 Completed 0 3d 10.97.196.149 airship
ucp keystone-fernet-rotate-1544356800-kcgg9 0/1 Completed 0 1d 10.97.196.148 airship
ucp keystone-fernet-rotate-1544400000-78qc8 0/1 Completed 0 14h 10.97.196.163 airship
ucp keystone-fernet-rotate-1544443200-v8bkn 0/1 Completed 0 2h 10.97.196.159 airship
ucp keystone-fernet-setup-srrqj 0/1 Completed 0 3d 10.97.196.152 airship
ucp keystone-rabbit-init-x4stm 0/1 Completed 0 3d 10.97.196.150 airship
ucp maas-bootstrap-admin-user-26lhf 0/1 Init:0/1 0 3d 10.97.196.132 airship
ucp maas-db-init-v7z6r 0/1 CrashLoopBackOff 944 3d 10.97.196.143 airship
ucp maas-db-sync-5c24z 0/1 Init:0/1 0 3d 10.97.196.162 airship
ucp maas-export-api-key-zhfxp 0/1 Init:0/1 0 3d 10.97.196.147 airship
ucp maas-import-resources-j6tzd 0/1 Init:0/1 0 3d 10.97.196.152 airship
ucp maas-rack-0 0/1 Init:0/1 0 3d 172.16.231.60 airship
ucp maas-region-0 0/1 Init:0/1 0 3d 10.97.196.146 airship
ucp mariadb-ingress-6c4f9c76f-sg8cx 1/1 Running 0 3d 10.97.196.137 airship
ucp mariadb-ingress-6c4f9c76f-sz26m 1/1 Running 0 3d 10.97.196.138 airship
ucp mariadb-ingress-error-pages-5dd6fbb75-x2xbn 1/1 Running 0 3d 10.97.196.139 airship
ucp mariadb-server-0 1/1 Running 0 3d 10.97.196.142 airship
ucp postgresql-0 1/1 Running 1 3d 10.97.196.140 airship
The only output from that pod is this:
root@airship:~# kubectl logs -n ucp maas-db-init-v7z6r
+ export HOME=/tmp
+ HOME=/tmp
+ [[ ! -v DB_HOST ]]
+ [[ ! -v ROOT_DB_USER ]]
+ [[ ! -v PGPASSWORD ]]
+ [[ ! -v USER_DB_USER ]]
+ [[ ! -v USER_DB_PASS ]]
+ [[ ! -v USER_DB_NAME ]]
+ echo 'Got DB connection info'
Got DB connection info
+ pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
+ DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
+ [[ ! -z '' ]]
+ grep -q 1
+ psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off
+ pgsql_superuser_cmd 'CREATE DATABASE maasdb'
+ DB_COMMAND='CREATE DATABASE maasdb'
+ [[ ! -z '' ]]
+ psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE maasdb'
psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off
Unfortunately, I haven't been able to troubleshoot this any further.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.commailto:Daniel.Balsiger@swisscom.com> Sent: Monday, December 10, 2018 6:32 AM To: airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org Subject: [Airship-discuss] Airship in a bottle install failure...
Hi there Airshippers
I wanted to try installing airhip-in-a-bottle in a fresh Ubuntu16.04 VM running on OpenStack over the weekend. I followed the instructions on airshipit.org
--snip
sudo -i
mkdir -p /root/deploy && cd "$_"
git clone https://github.com/openstack/airship-in-a-bottle
cd /root/deploy/airship-in-a-bottle/manifests/dev_single_node
./airship-in-a-bottle.sh
--snip
The script is still running after 2 days so I think something went wrong.
The output is repeating itself, listing running/restarted containers (I think postgres is the one not coming up properly)
--snip
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system auxiliary-etcd-airship 3/3 Running 0 2d kube-system bootstrap-armada-airship 4/4 Running 0 2d kube-system calico-etcd-airship 1/1 Running 0 2d kube-system calico-etcd-anchor-2jrqf 1/1 Running 0 2d kube-system calico-kube-controllers-6d5997c8c4-zxftw 1/1 Running 0 2d kube-system calico-node-l885v 2/2 Running 0 2d kube-system calico-settings-kbq79 0/1 Completed 0 2d kube-system coredns-d4587b7f8-7wlpm 1/1 Running 0 2d kube-system coredns-d4587b7f8-98t62 1/1 Running 0 2d kube-system coredns-d4587b7f8-frx5w 1/1 Running 0 2d kube-system haproxy-airship 1/1 Running 0 2d kube-system ingress-8w29l 1/1 Running 0 2d kube-system ingress-error-pages-5ccf96bf7d-jsrww 1/1 Running 0 2d kube-system kubernetes-apiserver-airship 1/1 Running 0 2d kube-system kubernetes-apiserver-anchor-55x59 1/1 Running 0 2d kube-system kubernetes-controller-manager-airship 1/1 Running 1 2d kube-system kubernetes-controller-manager-anchor-2wsg8 1/1 Running 0 2d kube-system kubernetes-etcd-airship 1/1 Running 0 2d kube-system kubernetes-etcd-anchor-64wgx 1/1 Running 0 2d kube-system kubernetes-proxy-9s68d 1/1 Running 0 2d kube-system kubernetes-scheduler-airship 1/1 Running 1 2d kube-system kubernetes-scheduler-anchor-npmn7 1/1 Running 0 2d kube-system nfs-provisioner-7799d64d59-scgdc 1/1 Running 0 2d ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 2d ucp ingress-6cd5b89d5d-gp98m 1/1 Running 0 2d ucp ingress-error-pages-5c97bb46bb-zq5cw 1/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-bjd99 0/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-xjrn9 0/1 Running 0 2d ucp mariadb-ingress-error-pages-5dd6fbb75-89qmx 1/1 Running 0 2d ucp mariadb-server-0 0/1 Running 0 2d ucp postgresql-0 1/1 Running 1 2d --snip
Anybody out there with the same problem or (even better) a solution for it?
Should I pick a particular branch or commit?
Any help is appreciated.
Best Regards, Daniel
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that "no pg_hba.conf" error current aiab code, or can you give git commit where you're hitting the issue?
Daniel, can you please run "sudo kubectl -n ucp logs mariadb-server-0" and share the results? Looks like MariaDB isn't starting cleanly for some reason.
Thanks, Matt
From: Balsiger Daniel, INI-EAI-INO-MBL Daniel.Balsiger@swisscom.com Sent: Monday, December 10, 2018 9:59 AM To: Alex Arnoldy alex.arnoldy@suse.com; airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi there Alex.
Thanks for the reply.
From your logs it seems to me like maas-db-init is not able to connect to postgres DB with psql client to the other container. (clients should be listed by IP in pg_hba.conf, wildcards should be possible....)
I will try another install from a fresh VM and let you know if I have the same one.
Thanks
________________________________ From: Alex Arnoldy <alex.arnoldy@suse.commailto:alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 4:26 PM To: Balsiger Daniel, INI-EAI-INO-MBL; airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org Subject: RE: Airship in a bottle install failure...
Hello Daniel,
I had launched the same thing but then got pulled away from it and totally forgot to check it. I thought I'd be able to tell you I got it running, but I don't think that's the case. For my installation, the maas-db-init pod seems to be stuck in CrashLoopBackOff:
root@airship:~# kubectl get pods -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system auxiliary-etcd-airship 3/3 Running 0 3d 172.16.231.60 airship
kube-system bootstrap-armada-airship 4/4 Running 0 3d 172.16.231.60 airship
kube-system calico-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-etcd-anchor-nqntm 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-kube-controllers-6d5997c8c4-bm9rk 1/1 Running 0 3d 172.16.231.60 airship
kube-system calico-node-2sgkk 2/2 Running 0 3d 172.16.231.60 airship
kube-system calico-settings-bqs5v 0/1 Completed 0 3d 172.16.231.60 airship
kube-system coredns-d4587b7f8-bbpr6 1/1 Running 13 3d 10.97.196.130 airship
kube-system coredns-d4587b7f8-tdl8v 1/1 Running 11 3d 10.97.196.131 airship
kube-system coredns-d4587b7f8-ts9lr 1/1 Running 12 3d 10.97.196.129 airship
kube-system haproxy-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system ingress-error-pages-5ccf96bf7d-8q6g5 1/1 Running 0 3d 10.97.196.133 airship
kube-system ingress-fmrmk 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-apiserver-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-apiserver-anchor-rkfpz 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-controller-manager-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-controller-manager-anchor-75k62 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-etcd-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-etcd-anchor-hzcrz 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-proxy-994kg 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-scheduler-airship 1/1 Running 0 3d 172.16.231.60 airship
kube-system kubernetes-scheduler-anchor-gsqtj 1/1 Running 0 3d 172.16.231.60 airship
kube-system nfs-provisioner-7799d64d59-g2g6n 1/1 Running 0 3d 10.97.196.134 airship
kube-system tiller-deploy-7d88c6f956-9jjk7 1/1 Running 0 3d 10.97.196.153 airship
ucp airship-ucp-keystone-memcached-memcached-74d79d8896-dd8k2 1/1 Running 0 3d 10.97.196.144 airship
ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 2 3d 10.97.196.178 airship
ucp ingress-6cd5b89d5d-xx592 1/1 Running 0 3d 10.97.196.135 airship
ucp ingress-error-pages-5c97bb46bb-ljmw6 1/1 Running 0 3d 10.97.196.136 airship
ucp keystone-api-7657986b8c-b49r6 1/1 Running 0 3d 10.97.196.145 airship
ucp keystone-bootstrap-8494p 0/1 Completed 0 3d 10.97.196.147 airship
ucp keystone-credential-setup-r5twm 0/1 Completed 0 3d 10.97.196.146 airship
ucp keystone-db-init-pxlzr 0/1 Completed 0 3d 10.97.196.148 airship
ucp keystone-db-sync-zx665 0/1 Completed 0 3d 10.97.196.151 airship
ucp keystone-domain-manage-rbfl5 0/1 Completed 0 3d 10.97.196.149 airship
ucp keystone-fernet-rotate-1544356800-kcgg9 0/1 Completed 0 1d 10.97.196.148 airship
ucp keystone-fernet-rotate-1544400000-78qc8 0/1 Completed 0 14h 10.97.196.163 airship
ucp keystone-fernet-rotate-1544443200-v8bkn 0/1 Completed 0 2h 10.97.196.159 airship
ucp keystone-fernet-setup-srrqj 0/1 Completed 0 3d 10.97.196.152 airship
ucp keystone-rabbit-init-x4stm 0/1 Completed 0 3d 10.97.196.150 airship
ucp maas-bootstrap-admin-user-26lhf 0/1 Init:0/1 0 3d 10.97.196.132 airship
ucp maas-db-init-v7z6r 0/1 CrashLoopBackOff 944 3d 10.97.196.143 airship
ucp maas-db-sync-5c24z 0/1 Init:0/1 0 3d 10.97.196.162 airship
ucp maas-export-api-key-zhfxp 0/1 Init:0/1 0 3d 10.97.196.147 airship
ucp maas-import-resources-j6tzd 0/1 Init:0/1 0 3d 10.97.196.152 airship
ucp maas-rack-0 0/1 Init:0/1 0 3d 172.16.231.60 airship
ucp maas-region-0 0/1 Init:0/1 0 3d 10.97.196.146 airship
ucp mariadb-ingress-6c4f9c76f-sg8cx 1/1 Running 0 3d 10.97.196.137 airship
ucp mariadb-ingress-6c4f9c76f-sz26m 1/1 Running 0 3d 10.97.196.138 airship
ucp mariadb-ingress-error-pages-5dd6fbb75-x2xbn 1/1 Running 0 3d 10.97.196.139 airship
ucp mariadb-server-0 1/1 Running 0 3d 10.97.196.142 airship
ucp postgresql-0 1/1 Running 1 3d 10.97.196.140 airship
The only output from that pod is this:
root@airship:~# kubectl logs -n ucp maas-db-init-v7z6r
+ export HOME=/tmp
+ HOME=/tmp
+ [[ ! -v DB_HOST ]]
+ [[ ! -v ROOT_DB_USER ]]
+ [[ ! -v PGPASSWORD ]]
+ [[ ! -v USER_DB_USER ]]
+ [[ ! -v USER_DB_PASS ]]
+ [[ ! -v USER_DB_NAME ]]
+ echo 'Got DB connection info'
Got DB connection info
+ pgsql_superuser_cmd 'SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
+ DB_COMMAND='SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
+ [[ ! -z '' ]]
+ grep -q 1
+ psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=SELECT 1 FROM pg_database WHERE datname = '''maasdb''''
psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off
+ pgsql_superuser_cmd 'CREATE DATABASE maasdb'
+ DB_COMMAND='CREATE DATABASE maasdb'
+ [[ ! -z '' ]]
+ psql -h postgresql.ucp.svc.cluster.local -p 5432 -U postgres '--command=CREATE DATABASE maasdb'
psql: FATAL: no pg_hba.conf entry for host "10.97.196.143", user "postgres", database "postgres", SSL off
Unfortunately, I haven't been able to troubleshoot this any further.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.commailto:Daniel.Balsiger@swisscom.com> Sent: Monday, December 10, 2018 6:32 AM To: airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org Subject: [Airship-discuss] Airship in a bottle install failure...
Hi there Airshippers
I wanted to try installing airhip-in-a-bottle in a fresh Ubuntu16.04 VM running on OpenStack over the weekend. I followed the instructions on airshipit.org
--snip
sudo -i
mkdir -p /root/deploy && cd "$_"
git clone https://github.com/openstack/airship-in-a-bottlehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Din-2Da-2Dbottle&d=DwMF-g&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=0ETQscoyeMAsqsF5ioGCgQDwKgFpjQwObZuJnDEAbG4&s=a9KACbmNQfoW9NpqhEG3WdJLO1F9JljHgAjxTITi3fE&e=
cd /root/deploy/airship-in-a-bottle/manifests/dev_single_node
./airship-in-a-bottle.sh
--snip
The script is still running after 2 days so I think something went wrong.
The output is repeating itself, listing running/restarted containers (I think postgres is the one not coming up properly)
--snip
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system auxiliary-etcd-airship 3/3 Running 0 2d kube-system bootstrap-armada-airship 4/4 Running 0 2d kube-system calico-etcd-airship 1/1 Running 0 2d kube-system calico-etcd-anchor-2jrqf 1/1 Running 0 2d kube-system calico-kube-controllers-6d5997c8c4-zxftw 1/1 Running 0 2d kube-system calico-node-l885v 2/2 Running 0 2d kube-system calico-settings-kbq79 0/1 Completed 0 2d kube-system coredns-d4587b7f8-7wlpm 1/1 Running 0 2d kube-system coredns-d4587b7f8-98t62 1/1 Running 0 2d kube-system coredns-d4587b7f8-frx5w 1/1 Running 0 2d kube-system haproxy-airship 1/1 Running 0 2d kube-system ingress-8w29l 1/1 Running 0 2d kube-system ingress-error-pages-5ccf96bf7d-jsrww 1/1 Running 0 2d kube-system kubernetes-apiserver-airship 1/1 Running 0 2d kube-system kubernetes-apiserver-anchor-55x59 1/1 Running 0 2d kube-system kubernetes-controller-manager-airship 1/1 Running 1 2d kube-system kubernetes-controller-manager-anchor-2wsg8 1/1 Running 0 2d kube-system kubernetes-etcd-airship 1/1 Running 0 2d kube-system kubernetes-etcd-anchor-64wgx 1/1 Running 0 2d kube-system kubernetes-proxy-9s68d 1/1 Running 0 2d kube-system kubernetes-scheduler-airship 1/1 Running 1 2d kube-system kubernetes-scheduler-anchor-npmn7 1/1 Running 0 2d kube-system nfs-provisioner-7799d64d59-scgdc 1/1 Running 0 2d ucp airship-ucp-rabbitmq-rabbitmq-0 1/1 Running 0 2d ucp ingress-6cd5b89d5d-gp98m 1/1 Running 0 2d ucp ingress-error-pages-5c97bb46bb-zq5cw 1/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-bjd99 0/1 Running 0 2d ucp mariadb-ingress-6c4f9c76f-xjrn9 0/1 Running 0 2d ucp mariadb-ingress-error-pages-5dd6fbb75-89qmx 1/1 Running 0 2d ucp mariadb-server-0 0/1 Running 0 2d ucp postgresql-0 1/1 Running 1 2d --snip
Anybody out there with the same problem or (even better) a solution for it?
Should I pick a particular branch or commit?
Any help is appreciated.
Best Regards, Daniel
Hi Matt,
Here is the commit I'm using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f Author: Bryan Strassner bryan.strassner@gmail.com Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller
Update the apiserver's command prefix to match between the apiserver chart and the genesis api server.
Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4)
Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434
I'm happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I'd preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT MM9745@att.com Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that "no pg_hba.conf" error current aiab code, or can you give git commit where you're hitting the issue?
Daniel, can you please run "sudo kubectl -n ucp logs mariadb-server-0" and share the results? Looks like MariaDB isn't starting cleanly for some reason.
Thanks, Matt
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip
FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip
and in mariadb-server-0 logs:
--snip
ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip
Best Regards, D
________________________________ From: Alex Arnoldy alex.arnoldy@suse.com Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner bryan.strassner@gmail.com
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller
Update the apiserver's command prefix to match between the apiserver
chart and the genesis api server.
Includes an optimization of virsh commands to parallelize and wait for
cloud init to complete (requires Ubuntu 16.04.4)
Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434
I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT MM9745@att.com Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
Hi Alex!
Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.
Regards, Drew Walters
On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL < Daniel.Balsiger@swisscom.com> wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip
FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip
and in mariadb-server-0 logs:
--snip
ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip
Best Regards, D
*From:* Alex Arnoldy alex.arnoldy@suse.com *Sent:* Monday, December 10, 2018 7:48 PM *To:* MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL *Subject:* RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner bryan.strassner@gmail.com
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
*Alex Arnoldy*
- Software Defined Infrastructure Solutions Architect SUSE *
*Alex.Arnoldy@SUSE.com* Alex.Arnoldy@SUSE.com
- (P) +1 (408) 828-9797*
*From:* MCEUEN, MATT MM9745@att.com *Sent:* Monday, December 10, 2018 9:33 AM *To:* airship-discuss@lists.airshipit.org; Alex Arnoldy < alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel < Daniel.Balsiger@swisscom.com> *Subject:* RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi Drew,
Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two.
I’ll definitely let you know how things go.
Very best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Drew Walters drewwalters96@gmail.com Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi Alex!
Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.
Regards, Drew Walters
On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.commailto:Daniel.Balsiger@swisscom.com> wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip
and in mariadb-server-0 logs:
--snip ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip
Best Regards, D
________________________________ From: Alex Arnoldy <alex.arnoldy@suse.commailto:alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner <bryan.strassner@gmail.commailto:bryan.strassner@gmail.com>
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller
Update the apiserver's command prefix to match between the apiserver
chart and the genesis api server.
Includes an optimization of virsh commands to parallelize and wait for
cloud init to complete (requires Ubuntu 16.04.4)
Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434
I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT <MM9745@att.commailto:MM9745@att.com> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.commailto:alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.commailto:Daniel.Balsiger@swisscom.com> Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.orgmailto:Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hello all, I have in-a-bottle deployment running right now for about two hours. Will let you know if it succeeds (it should).
Best regards, -- Roman Gorshunov
On Mon, Jan 7, 2019 at 12:52 PM Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two.
I’ll definitely let you know how things go.
Very best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Drew Walters drewwalters96@gmail.com Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi Alex!
Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.
Regards,
Drew Walters
On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL Daniel.Balsiger@swisscom.com wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip
FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress
--snip
and in mariadb-server-0 logs:
--snip
ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
--snip
Best Regards, D
From: Alex Arnoldy alex.arnoldy@suse.com Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner bryan.strassner@gmail.com
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT MM9745@att.com Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hello all,
My deployment has just completed successfully. Git commit id a9f5fc93e33163f51fc56d1bf886d4d433251f8e (it's current master).
-- Roman Gorshunov
On Mon, Jan 7, 2019 at 12:57 PM Roman Gorshunov paye600@gmail.com wrote:
Hello all, I have in-a-bottle deployment running right now for about two hours. Will let you know if it succeeds (it should).
Best regards,
Roman Gorshunov
On Mon, Jan 7, 2019 at 12:52 PM Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two.
I’ll definitely let you know how things go.
Very best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Drew Walters drewwalters96@gmail.com Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi Alex!
Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.
Regards,
Drew Walters
On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL Daniel.Balsiger@swisscom.com wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip
FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress
--snip
and in mariadb-server-0 logs:
--snip
ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
--snip
Best Regards, D
From: Alex Arnoldy alex.arnoldy@suse.com Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner bryan.strassner@gmail.com
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller Update the apiserver's command prefix to match between the apiserver chart and the genesis api server. Includes an optimization of virsh commands to parallelize and wait for cloud init to complete (requires Ubuntu 16.04.4) Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT MM9745@att.com Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.org; Alex Arnoldy alex.arnoldy@suse.com; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Alex Arnoldy Sent: Monday, January 7, 2019 3:12 AM To: 'Drew Walters' drewwalters96@gmail.com; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org Subject: RE: [Airship-discuss] Airship in a bottle install failure...
Hi Drew,
Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two.
I’ll definitely let you know how things go.
Very best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: Drew Walters <drewwalters96@gmail.commailto:drewwalters96@gmail.com> Sent: Friday, January 4, 2019 7:37 PM To: INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.commailto:Daniel.Balsiger@swisscom.com> Cc: MATT MCEUEN <MM9745@att.commailto:MM9745@att.com>; airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.commailto:alex.arnoldy@suse.com> Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi Alex!
Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.
Regards, Drew Walters
On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger@swisscom.commailto:Daniel.Balsiger@swisscom.com> wrote:
Hi there everyone
Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:
I see in Postgresql-0 logs:
--snip FATAL: the database system is in recovery mode LOG: database system was not properly shut down; automatic recovery in progress LOG: invalid record length at 0/102A400 LOG: redo is not required LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory LOG: autovacuum launcher process (PID 79) exited with exit code 1 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2018-12-11 15:25:38 UTC LOG: database system was not properly shut down; automatic recovery in progress --snip
and in mariadb-server-0 logs:
--snip ubuntu@airship:~$ sudo kubectl -n ucp logs mariadb-server-0 2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0 2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info 2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running 2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. 2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state 2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state. --snip
Best Regards, D
________________________________ From: Alex Arnoldy <alex.arnoldy@suse.commailto:alex.arnoldy@suse.com> Sent: Monday, December 10, 2018 7:48 PM To: MCEUEN, MATT ; airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org; Balsiger Daniel, INI-EAI-INO-MBL Subject: RE: Airship in a bottle install failure...
Hi Matt,
Here is the commit I’m using:
commit 221441f1bff351ab366b02d6fd7856b70b06108f
Author: Bryan Strassner <bryan.strassner@gmail.commailto:bryan.strassner@gmail.com>
Date: Mon Nov 19 17:12:10 2018 -0600
Update apiserver for admission controller
Update the apiserver's command prefix to match between the apiserver
chart and the genesis api server.
Includes an optimization of virsh commands to parallelize and wait for
cloud init to complete (requires Ubuntu 16.04.4)
Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434
I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.commailto:Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
From: MCEUEN, MATT <MM9745@att.commailto:MM9745@att.com> Sent: Monday, December 10, 2018 9:33 AM To: airship-discuss@lists.airshipit.orgmailto:airship-discuss@lists.airshipit.org; Alex Arnoldy <alex.arnoldy@suse.commailto:alex.arnoldy@suse.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger@swisscom.commailto:Daniel.Balsiger@swisscom.com> Subject: RE: Airship in a bottle install failure...
Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle. Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?
Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results? Looks like MariaDB isn’t starting cleanly for some reason.
Thanks, Matt
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.orgmailto:Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew
Hi Drew,
Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot.
Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Drew Walters drewwalters96@gmail.com Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy alex.arnoldy@suse.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: Re: [Airship-discuss] Airship in a bottle install failure...
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew
Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good.
Regards, -- Roman Gorshunov
On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot.
Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Drew Walters drewwalters96@gmail.com Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy alex.arnoldy@suse.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: Re: [Airship-discuss] Airship in a bottle install failure...
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Thanks. Maybe I'll give that one a shot. It won't shed any light into the previous issues I've had but it's also possible that those problems were related to intermittent keyboard driver problems I've seen in the past.
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Roman Gorshunov paye600@gmail.com Sent: Friday, January 18, 2019 10:36 AM To: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good.
Regards, -- Roman Gorshunov
On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot.
Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Drew Walters drewwalters96@gmail.com Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy alex.arnoldy@suse.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: Re: [Airship-discuss] Airship in a bottle install failure...
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
HI Alex,
yeah - the issue was pre-installed libvirt causing issues. As long as it's standard Ubuntu 1604 - that's all you need.
Kindly, Kaspars
________________________________________ From: Alex Arnoldy [alex.arnoldy@suse.com] Sent: Friday, January 18, 2019 12:55 PM To: Roman Gorshunov; airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Thanks. Maybe I'll give that one a shot. It won't shed any light into the previous issues I've had but it's also possible that those problems were related to intermittent keyboard driver problems I've seen in the past.
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Roman Gorshunov paye600@gmail.com Sent: Friday, January 18, 2019 10:36 AM To: airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Airship in a bottle install failure...
Hi, `git clone ...; cd airship-in-a-bottle/; vagrant up` mostly works fine. The vagrant box image used works good.
Regards, -- Roman Gorshunov
On Fri, Jan 18, 2019 at 6:41 PM Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
Unfortunately not. However, I think I'll have a bit of a breather today so this afternoon I'll give it another shot.
Are there any known gotchas when it comes to the VM or O/S? I noticed someone eluded to not including KVM to the initial install packages helped out.
Best,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
-----Original Message----- From: Drew Walters drewwalters96@gmail.com Sent: Friday, January 18, 2019 9:38 AM To: Alex Arnoldy alex.arnoldy@suse.com Cc: MATT MCEUEN MM9745@att.com; airship-discuss@lists.airshipit.org; INI-EAI-INO-MBL Balsiger Daniel Daniel.Balsiger@swisscom.com Subject: Re: [Airship-discuss] Airship in a bottle install failure...
On Jan 15, 2019, at 4:57 PM, Alex Arnoldy alex.arnoldy@suse.com wrote:
Hi Drew,
I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.
As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.
Sorry that’s not much info to go on,
--Alex.
Alex Arnoldy Software Defined Infrastructure Solutions Architect SUSE Alex.Arnoldy@SUSE.com (P) +1 (408) 828-9797
Hi Alex,
Were you able to return to the VM and obtain the logs of the failing MaaS pod before it was removed? I’m not entirely sure where to proceed from here, but perhaps someone more familiar with MaaS may know some additional things you can try.
Drew _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-...
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-... _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-...
participants (6)
-
Alex Arnoldy
-
Balsiger Daniel, INI-EAI-INO-MBL
-
Drew Walters
-
MCEUEN, MATT
-
Roman Gorshunov
-
SKELS, KASPARS