[Airship-discuss] Armada API connectivity to Tiller (proxy problem)

Mon Jan 7 14:53:40 UTC 2019

Hi Matt, Drew,

Thanks for looking into this, sure, I can join to the meeting, but can we
move this topic down to the bottom? It's 06:00AM PT for me, so I just want
to make sure that I won't be too late for the discussion.

On Fri, Jan 4, 2019 at 10:12 AM MCEUEN, MATT <MM9745 at att.com> wrote:

> Hey Eli,
>
>
>
> Drew and I dug into this a bit, and it looks like the Armada-based lookup
> uses labels (configurably: application=tiller, name=tiller, in the
> kube-system namespace),  and Shipyard looks up tiller based on the pod name
> [1].  I believe there was some condition under which the preferable
> label-based lookup wouldn’t work, and that’s why there a “plan B” for
> looking up based on pod name.
>
>
>
> However, I think this could be improved by 1) moving the “plan B” logic
> from Shipyard into Armada (your coupling point #2 below), and 2) allowing
> Armada to try “plan A” first before falling back to “plan B”.  I’d like to
> discuss this in the 1/8 Airship team IRC meeting [2] – would you be able to
> join so we can make sure we account for all cases?   Between {pod labels,
> pod name, service DNS} maybe we could find two Armada configuration-driven
> approaches that could cover all bases.
>
>
>
> [1]
> https://github.com/openstack/airship-shipyard/blob/c7a9c65c886641029ce84724ba7a999f8542a7c5/src/bin/shipyard_airflow/shipyard_airflow/plugins/get_k8s_pod_port_ip.py#L65
>
> [2] https://etherpad.openstack.org/p/airship-meeting-2019-01-08
>
>
>
> Thanks!
>
> Matt
>
>
>
> *From:* Evgeny L <eli at mirantis.com>
> *Sent:* Wednesday, December 26, 2018 12:56 PM
> *To:* Drew Walters <drewwalters96 at gmail.com>
> *Cc:* Airship-discuss at lists.airshipit.org
> *Subject:* Re: [Airship-discuss] Armada API connectivity to Tiller (proxy
> problem)
>
>
>
> Hi Drew,
>
>
>
> Thank you for the response!
>
>
>
> >> The Armada API accepts a Tiller hostname as a safeguard in the event
> that Armada is unable to find a running Tiller pod
>
>
>
> Looking at the code [1] it seems to be that Armada will try to find Tiller
> pod in the event Shipyard doesn't find one.
>
> Was there some problem with Armada API not being able to retrieve Tiller
> endpoint so it required a fallback? They both seem to use k8s API for that.
>
>
>
> >> This can be changed per API request by providing the hostname of a
> different Tiller pod.
>
>
>
> From architecture point of view, do we expect it to work with multiple
> Tillers within a single Airship installation?
>
>
>
> I'm asking these questions to explore the options for resolving GRPC
> (which is used by Tiller client) [2] constraints, it does not handle CIDR
> in no_proxy environment variables [3]. One of the options is:
>
> 1. [high priority] Expose Tiller to create a DNS name for the pod, and use
> it when making requests from Armada API to Tiller, in this case Aramada API
> POD can have ".cluster.local" in NO_PROXY and everything would work just
> fine (i.e. requests to Tiller won't go through the proxy).
>
> 2. [low priority] Reduce coupling between Tiller/k8s and Shipyard by
> configuring Tiller endpoints in Armada API directly without requiring
> Shipyard to pass one.
>
>
>
> Prior to submitting any changes I would like to verify with you if this
> makes sense or if I should explore other options.
>
>
>
> Thanks,
>
>
>
> [1]
> https://github.com/openstack/airship-armada/blob/307f1318c4e83f247f1e3838478957e7555d6ce0/armada/handlers/tiller.py#L158-L164
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Darmada_blob_307f1318c4e83f247f1e3838478957e7555d6ce0_armada_handlers_tiller.py-23L158-2DL164&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=1j8HwScWR1I9HzTq0VzYbOZQbURKo8fwxRWWd4EKvJc&e=>
>
> [2]
> https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=>
>
> [3]
> https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=>
>
>
>
>
>
> On Thu, Dec 20, 2018 at 1:42 PM Drew Walters <drewwalters96 at gmail.com>
> wrote:
>
> Hi Eli,
>
>
>
> The Armada API accepts a Tiller hostname as a safeguard in the event that
> Armada is unable to find a running Tiller pod. Shipyard uses the Python
> Kubernetes client to retrieve the IP address that it provides to Armada.
>
>
>
> 1. It is possible to create a clusterIP with DNS and provide a domain name
> to Armada.
>
>
>
> 2. Armada only communicates with one Tiller pod at a time. This can be
> changed per API request by providing the hostname of a different Tiller
> pod. In the event that a Tiller hostname is not provided, Armada selects
> the first running Tiller pod it encounters.
>
>
>
> We invite you to join our IRC team meeting (#airshipit on freenode)
> January 8th to help us better understand your desired configuration and
> discuss in more detail. In the meantime, please let us know if you have any
> additional questions!
>
>
>
> Best,
>
> Drew Walters
>
>
>
>
>
> On Wed, Dec 12, 2018 at 3:55 PM Evgeny L <eli at mirantis.com> wrote:
>
> Hi,
>
>
>
> I've been working on getting behind the proxy configuration for AIAB to
> work [1] and I stumbled across a problem related to GRPC [2], which does
> not support CIDR notation for no_proxy entries, this make Armada API to
> fail to connect to Tiller, because it tries to use proxy configured with
> environment variables.
>
>
>
> While looking for a solution other than doing some manual reconfiguration
> [2], I started to have the following questions:
>
>    1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP
>    and use a DNS name?
>
>
>    1. Why Shipyard is the one who provides Armada API with Tiller IP
>    address? If we do 1st we can possibly make endpoint a part of Armada API
>    configuration. Or is Armada API supposed to work with multiple Tillers at
>    the same time?
>
>
>
> Any feedback and historical background would be really helpful!
>
>
>
> Thanks,
>
>
>
> [1] https://review.openstack.org/#/c/624122/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624122_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=Ogo7hxRYVOTBW1SbxHDdonCymHGxS0x58TI5iPBXvAQ&e=>
>
> [2]
> https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=>
>
> [3]
> https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=>
>
> [4]
> https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e3575e045e26f177252/src/bin/shipyard_airflow/shipyard_airflow/plugins/armada_base_operator.py#L147-L153
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dshipyard_blob_a235f8e6db6e791550727e3575e045e26f177252_src_bin_shipyard-5Fairflow_shipyard-5Fairflow_plugins_armada-5Fbase-5Foperator.py-23L147-2DL153&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=xLj4wU7W8s0CdleOZmKug7vs7v5Lh--u2UmlTfUXlHs&e=>
>
> _______________________________________________
> Airship-discuss mailing list
> Airship-discuss at lists.airshipit.org
> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=wXSgWU9qQaFrrE0bQKaQXWNDGaAJlwMAiqqPOTvvLz8&e=>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.airshipit.org/pipermail/airship-discuss/attachments/20190107/ca1e1bf0/attachment-0001.html>