[Airship-discuss] Airship in a bottle install failure...

Tue Jan 15 22:57:17 UTC 2019

Hi Drew,

I gave it another shot with the latest build. It definitely went better, but it still went into a CrashLoopBackOff on one of the MaaS pods before it could complete. Unfortunately, I was neck deep in something else so I had to shut that VM down before collecting any info.

As a small side note, I did ensure that the hostname of the VM was in /etc/hosts and associated with the IPv4 primary interface.

Sorry that’s not much info to go on,

--Alex.

Alex Arnoldy
Software Defined Infrastructure Solutions Architect
SUSE
Alex.Arnoldy at SUSE.com<mailto:Alex.Arnoldy at SUSE.com>
(P) +1 (408) 828-9797

From: Alex Arnoldy
Sent: Monday, January 7, 2019 3:12 AM
To: 'Drew Walters' <drewwalters96 at gmail.com>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger at swisscom.com>
Cc: MATT MCEUEN <MM9745 at att.com>; airship-discuss at lists.airshipit.org
Subject: RE: [Airship-discuss] Airship in a bottle install failure...

Hi Drew,

Thanks for getting back to me on this. I haven’t had a chance to give this another shot, but I’ll do my best to get to that within the next week or two.

I’ll definitely let you know how things go.

Very best,

--Alex.

Alex Arnoldy
Software Defined Infrastructure Solutions Architect
SUSE
Alex.Arnoldy at SUSE.com<mailto:Alex.Arnoldy at SUSE.com>
(P) +1 (408) 828-9797

From: Drew Walters <drewwalters96 at gmail.com<mailto:drewwalters96 at gmail.com>>
Sent: Friday, January 4, 2019 7:37 PM
To: INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger at swisscom.com<mailto:Daniel.Balsiger at swisscom.com>>
Cc: MATT MCEUEN <MM9745 at att.com<mailto:MM9745 at att.com>>; airship-discuss at lists.airshipit.org<mailto:airship-discuss at lists.airshipit.org>; Alex Arnoldy <alex.arnoldy at suse.com<mailto:alex.arnoldy at suse.com>>
Subject: Re: [Airship-discuss] Airship in a bottle install failure...

Hi Alex!

Sorry for the delay. Have you had the opportunity to verify that this issue still persists with the latest version of Airship-in-a-Bottle? There have been several changes since the last time we visited this that may resolve the above.

Regards,
Drew Walters

On Tue, Dec 11, 2018 at 9:29 AM Balsiger Daniel, INI-EAI-INO-MBL <Daniel.Balsiger at swisscom.com<mailto:Daniel.Balsiger at swisscom.com>> wrote:

Hi there everyone

Thanks for the quick replies, I tried a new install on a fresh VM, behaves the same, problem is persisting:

I see in Postgresql-0 logs:

--snip
FATAL:  the database system is in recovery mode
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  invalid record length at 0/102A400
LOG:  redo is not required
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
FATAL:  could not open relation mapping file "global/pg_filenode.map": No such file or directory
LOG:  autovacuum launcher process (PID 79) exited with exit code 1
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2018-12-11 15:25:38 UTC
LOG:  database system was not properly shut down; automatic recovery in progress
--snip

and in mariadb-server-0 logs:

--snip
ubuntu at airship:~$ sudo kubectl -n ucp logs mariadb-server-0
2018-12-11 15:25:35,704 - OpenStack-Helm Mariadb - INFO - This instance hostname: mariadb-server-0
2018-12-11 15:25:36,310 - OpenStack-Helm Mariadb - INFO - Kubernetes API Version: v1.10.2
2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Will use "airship-ucp-mariadb-mariadb-state" configmap for cluster state info
2018-12-11 15:25:36,313 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
2018-12-11 15:25:36,348 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:25:36,455 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Waiting for cluster to start running
2018-12-11 15:25:36,456 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:25:36,474 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
2018-12-11 15:25:56,491 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:25:56,637 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
2018-12-11 15:26:16,657 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:26:16,678 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
2018-12-11 15:26:36,684 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:26:36,697 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
2018-12-11 15:26:56,717 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:26:56,749 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
2018-12-11 15:27:16,761 - OpenStack-Helm Mariadb - INFO - Getting cluster state
2018-12-11 15:27:16,876 - OpenStack-Helm Mariadb - INFO - The cluster is currently in "init" state.
--snip

Best Regards, D

________________________________
From: Alex Arnoldy <alex.arnoldy at suse.com<mailto:alex.arnoldy at suse.com>>
Sent: Monday, December 10, 2018 7:48 PM
To: MCEUEN, MATT ; airship-discuss at lists.airshipit.org<mailto:airship-discuss at lists.airshipit.org>; Balsiger Daniel, INI-EAI-INO-MBL
Subject: RE: Airship in a bottle install failure...

Hi Matt,

Here is the commit I’m using:

commit 221441f1bff351ab366b02d6fd7856b70b06108f

Author: Bryan Strassner <bryan.strassner at gmail.com<mailto:bryan.strassner at gmail.com>>

Date:   Mon Nov 19 17:12:10 2018 -0600

    Update apiserver for admission controller

    Update the apiserver's command prefix to match between the apiserver

    chart and the genesis api server.

    Includes an optimization of virsh commands to parallelize and wait for

    cloud init to complete (requires Ubuntu 16.04.4)

    Change-Id: I793d210e15c3c13e8b662bd2459ed8bcba30d434

I’m happy to start over to see if a new pull resolves the issue, but it happened to me twice in a row so I thought I’d preserve the environment for examination.

Best,

--Alex.

Alex Arnoldy
Software Defined Infrastructure Solutions Architect
SUSE
Alex.Arnoldy at SUSE.com<mailto:Alex.Arnoldy at SUSE.com>
(P) +1 (408) 828-9797

From: MCEUEN, MATT <MM9745 at att.com<mailto:MM9745 at att.com>>
Sent: Monday, December 10, 2018 9:33 AM
To: airship-discuss at lists.airshipit.org<mailto:airship-discuss at lists.airshipit.org>; Alex Arnoldy <alex.arnoldy at suse.com<mailto:alex.arnoldy at suse.com>>; INI-EAI-INO-MBL Balsiger Daniel <Daniel.Balsiger at swisscom.com<mailto:Daniel.Balsiger at swisscom.com>>
Subject: RE: Airship in a bottle install failure...

Weird, the maas-db-init job is running fine for me with a fresh deployment of airship in a bottle.  Alex, are you seeing that “no pg_hba.conf” error current aiab code, or can you give git commit where you’re hitting the issue?

Daniel, can you please run “sudo kubectl -n ucp logs mariadb-server-0” and share the results?  Looks like MariaDB isn’t starting cleanly for some reason.

Thanks,
Matt

_______________________________________________
Airship-discuss mailing list
Airship-discuss at lists.airshipit.org<mailto:Airship-discuss at lists.airshipit.org>
http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190115/52a91fdc/attachment-0001.html>