[Airship-discuss] AIAB and default backend - 404

Nick Chase nchase at mirantis.com
Thu Aug 29 02:45:50 UTC 2019 

OK, unfortunately I'm wrong, it does NOT work. :(  Horizon does come up,
and the sample instance is there, but the actual Airship URLs are still
giving me the "default backend - 404" error.   I found the following in the
airflow-web container in the shipyard-api pod, which is in a crashloop:

2019-08-29 02:42:23,471 ERROR    models.py:380:process_file models Failed
to import: /usr/local/airflow/dags/redeploy_server.py

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line
377, in process_file

    m = imp.load_source(mod_name, filepath)

  File "/usr/lib/python3.5/imp.py", line 172, in load_source

    module = _load(spec)

  File "<frozen importlib._bootstrap>", line 693, in _load

  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 665, in exec_module

  File "<frozen importlib._bootstrap>", line 222, in
_call_with_frames_removed

  File "/usr/local/airflow/dags/redeploy_server.py", line 57, in <module>

    targets=[BAREMETAL]

  File "/usr/local/airflow/dags/common_step_factory.py", line 171, in
get_validate_site_design

    dag=self.dag)

  File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line
74, in wrapper

    return func(*args, **kwargs)

  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__

    next(self.gen)

  File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line
45, in create_session

    session.commit()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
line 943, in commit

    self.transaction.commit()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
line 465, in commit

    self._assert_active(prepared_ok=True)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
line 276, in _assert_active

    % self._rollback_exception

sqlalchemy.exc.InvalidRequestError: This Session's transaction has been
rolled back due to a previous exception during flush. To begin a new
transaction with this Session, first issue Session.rollback(). Original
exception was: (psycopg2.OperationalError) could not connect to server:
Connection refused

Is the server running on host "postgresql.ucp.svc.cluster.local"
(10.96.212.170) and accepting

TCP/IP connections on port 5432?

 (Background on this error at: http://sqlalche.me/e/e3q8)

2019-08-29 02:42:23,507 ERROR    models.py:380:process_file models Failed
to import: /usr/local/airflow/dags/samples/airflow_task_state_operators.py

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line
377, in process_file

    m = imp.load_source(mod_name, filepath)

  File "/usr/lib/python3.5/imp.py", line 172, in load_source

    module = _load(spec)

  File "<frozen importlib._bootstrap>", line 693, in _load

  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 665, in exec_module

  File "<frozen importlib._bootstrap>", line 222, in
_call_with_frames_removed

  File "/usr/local/airflow/dags/samples/airflow_task_state_operators.py",
line 19, in <module>

    from airflow.operators import TaskStateOperator

ImportError: cannot import name 'TaskStateOperator'

/usr/local/lib/python3.5/dist-packages/psycopg2/__init__.py:144:
UserWarning: The psycopg2 wheel package will be renamed from release 2.8;
in order to keep installing from binary please use "pip install
psycopg2-binary" instead. For details see: <
http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.

  """)

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
line 2158, in _wrap_pool_connect

    return fn()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
400, in connect

    return _ConnectionFairy._checkout(self)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
788, in _checkout

    fairy = _ConnectionRecord.checkout(pool)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
529, in checkout

    rec = pool._do_get()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
1193, in _do_get

    self._dec_overflow()

  File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__

    compat.reraise(exc_type, exc_value, exc_tb)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py",
line 249, in reraise

    raise value

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
1190, in _do_get

    return self._create_connection()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
347, in _create_connection

    return _ConnectionRecord(self)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
474, in __init__

    self.__connect(first_connect_check=True)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
671, in __connect

    connection = pool._invoke_creator(self)

  File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/strategies.py",
line 106, in connect

    return dialect.connect(*cargs, **cparams)

  File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line
412, in connect

    return self.dbapi.connect(*cargs, **cparams)

  File "/usr/local/lib/python3.5/dist-packages/psycopg2/__init__.py", line
130, in connect

    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)

psycopg2.OperationalError: could not connect to server: Connection refused

Is the server running on host "postgresql.ucp.svc.cluster.local"
(10.96.212.170) and accepting

TCP/IP connections on port 5432?



The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/local/bin/airflow", line 32, in <module>

    args.func(args)

  File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line
74, in wrapper

    return f(*args, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line
793, in webserver

    app = cached_app_rbac(conf) if settings.RBAC else cached_app(conf)

  File "/usr/local/lib/python3.5/dist-packages/airflow/www/app.py", line
187, in cached_app

    app = create_app(config, testing)

  File "/usr/local/lib/python3.5/dist-packages/airflow/www/app.py", line
72, in create_app

    from airflow.www import views

  File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line
2232, in <module>

    class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):

  File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line
2323, in ChartModelView

    'conn_id': _connection_ids()

  File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line
74, in wrapper

    return func(*args, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line
2227, in _connection_ids

    .group_by(models.Connection.conn_id)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py",
line 2995, in __iter__

    return self._execute_and_instances(context)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py",
line 3016, in _execute_and_instances

    close_with_result=True)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py",
line 3025, in _get_bind_args

    **kw

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/query.py",
line 3007, in _connection_from_session

    conn = self.session.connection(**kw)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
line 1035, in connection

    execution_options=execution_options)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
line 1040, in _connection_for_bind

    engine, execution_options)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
line 409, in _connection_for_bind

    conn = bind.contextual_connect()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
line 2123, in contextual_connect

    self._wrap_pool_connect(self.pool.connect, None),

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
line 2162, in _wrap_pool_connect

    e, dialect, self)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
line 1476, in _handle_dbapi_exception_noconnection

    exc_info

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py",
line 265, in raise_from_cause

    reraise(type(exception), exception, tb=exc_tb, cause=cause)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py",
line 248, in reraise

    raise value.with_traceback(tb)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
line 2158, in _wrap_pool_connect

    return fn()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
400, in connect

    return _ConnectionFairy._checkout(self)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
788, in _checkout

    fairy = _ConnectionRecord.checkout(pool)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
529, in checkout

    rec = pool._do_get()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
1193, in _do_get

    self._dec_overflow()

  File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py",
line 66, in __exit__

    compat.reraise(exc_type, exc_value, exc_tb)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py",
line 249, in reraise

    raise value

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
1190, in _do_get

    return self._create_connection()

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
347, in _create_connection

    return _ConnectionRecord(self)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
474, in __init__

    self.__connect(first_connect_check=True)

  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line
671, in __connect

    connection = pool._invoke_creator(self)

  File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/strategies.py",
line 106, in connect

    return dialect.connect(*cargs, **cparams)

  File
"/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line
412, in connect

    return self.dbapi.connect(*cargs, **cparams)

  File "/usr/local/lib/python3.5/dist-packages/psycopg2/__init__.py", line
130, in connect

    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not
connect to server: Connection refused

Is the server running on host "postgresql.ucp.svc.cluster.local"
(10.96.212.170) and accepting

TCP/IP connections on port 5432?

 (Background on this error at: http://sqlalche.me/e/e3q8)



The postgresql pod appears to be running.


Any thoughts?


----  Nick

On Wed, Aug 28, 2019 at 11:32 AM Roman Gorshunov <paye600 at gmail.com> wrote:

> Very nice to hear that it worked, Nick.
> But sad to know that requirements have almost doubled - something for
> us to look at and thank you for the report.
>
> Regards,
> -- Roman Gorshunov
>
> On Wed, Aug 28, 2019 at 2:34 PM Nick Chase <nchase at mirantis.com> wrote:
> >
> > I found the problem.  First off, I was citing the wrong repo.  I
> actually was coming from https://opendev.org/airship/treasuremap/
> >
> > Second, apparently the requirements cited for the machine are too
> small.  The README says 4vCPU/20GB RAM/32GB but with that I was in a
> periodic crashloop for Postgres.  GCE was suggesting I increase the machine
> to 8vCPU/48GB RAM.  Ultimately I'm at 16vCPU/60GB RAM and about 32GB of
> drive space is used.
> >
> > With the bigger machine, everything deployed properly and seems to be
> running smoothly.
> >
> > ----  Nick
> >
> > On Tue, Aug 27, 2019 at 9:56 AM Nick Chase <nchase at mirantis.com> wrote:
> >>
> >> OK, I feel ridiculous saying this, but I can't find the Ingress.  I get
> plenty of output from
> >>
> >> kubectl get --all-namespaces ing
> >>
> >> but "no resources" from
> >>
> >> kubectl get ing
> >>
> >> ----  Nick
> >>
> >> On Tue, Aug 27, 2019 at 9:51 AM Nick Chase <nchase at mirantis.com> wrote:
> >>>
> >>> I ran the airship-in-a-bottle.sh script from
> https://opendev.org/airship/airship-in-a-bottle.  Is there another
> preferred way?  I can try the Vagrantfile.
> >>>
> >>> I'll check out the logs.
> >>>
> >>> ----  Nick
> >>>
> >>> On Tue, Aug 27, 2019 at 2:06 AM Roman Gorshunov <paye600 at gmail.com>
> wrote:
> >>>>
> >>>> Hi Nick,
> >>>>
> >>>> Which script did you run and from which repo?
> >>>> I would start with checking logs and config of running ingress.
> >>>>
> >>>> Best regards,
> >>>> —
> >>>> Roman Gorshunov
> >>>>
> >>>>
> >>>> > On 26 Aug 2019, at 22:40, Nick Chase <nchase at mirantis.com> wrote:
> >>>> >
> >>>> > I'm pleased to say that I've successfully run the script to install
> Airship in a Bottle -- or at least I think I did.  When I try to access
> most of the endpoints, I get
> >>>> >
> >>>> > default backend - 404
> >>>> >
> >>>> > Except for the Horizon endpoint, where I get nothing at all.
> >>>> >
> >>>> > I'm assuming this is a K8s issue, but I'm not sure where to look
> further.
> >>>> >
> >>>> > Any ideas?
> >>>> >
> >>>> > ----  Nick
> >>>> > _______________________________________________
> >>>> > Airship-discuss mailing list
> >>>> > Airship-discuss at lists.airshipit.org
> >>>> > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Airship-discuss mailing list
> >>>> Airship-discuss at lists.airshipit.org
> >>>> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
>
> _______________________________________________
> Airship-discuss mailing list
> Airship-discuss at lists.airshipit.org
> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190829/cea2321d/attachment.html>

More information about the Airship-discuss mailing list