<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:301153830;
mso-list-template-ids:-386870192;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hey Eli,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Drew and I dug into this a bit, and it looks like the Armada-based lookup uses labels (configurably: application=tiller, name=tiller, in the kube-system namespace), and Shipyard looks up tiller based on the pod name [1]. I believe there
was some condition under which the preferable label-based lookup wouldn’t work, and that’s why there a “plan B” for looking up based on pod name.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However, I think this could be improved by 1) moving the “plan B” logic from Shipyard into Armada (your coupling point #2 below), and 2) allowing Armada to try “plan A” first before falling back to “plan B”. I’d like to discuss this in
the 1/8 Airship team IRC meeting [2] – would you be able to join so we can make sure we account for all cases? Between {pod labels, pod name, service DNS} maybe we could find two Armada configuration-driven approaches that could cover all bases.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://github.com/openstack/airship-shipyard/blob/c7a9c65c886641029ce84724ba7a999f8542a7c5/src/bin/shipyard_airflow/shipyard_airflow/plugins/get_k8s_pod_port_ip.py#L65">
https://github.com/openstack/airship-shipyard/blob/c7a9c65c886641029ce84724ba7a999f8542a7c5/src/bin/shipyard_airflow/shipyard_airflow/plugins/get_k8s_pod_port_ip.py#L65</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://etherpad.openstack.org/p/airship-meeting-2019-01-08">
https://etherpad.openstack.org/p/airship-meeting-2019-01-08</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal">Matt<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Evgeny L <eli@mirantis.com> <br>
<b>Sent:</b> Wednesday, December 26, 2018 12:56 PM<br>
<b>To:</b> Drew Walters <drewwalters96@gmail.com><br>
<b>Cc:</b> Airship-discuss@lists.airshipit.org<br>
<b>Subject:</b> Re: [Airship-discuss] Armada API connectivity to Tiller (proxy problem)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi Drew,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thank you for the response!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">>> The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Looking at the code [1] it seems to be that Armada will try to find Tiller pod in the event Shipyard doesn't find one.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Was there some problem with Armada API not being able to retrieve Tiller endpoint so it required a fallback? They both seem to use k8s API for that.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">>> This can be changed per API request by providing the hostname of a different Tiller pod.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From architecture point of view, do we expect it to work with multiple Tillers within a single Airship installation?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm asking these questions to explore the options for resolving GRPC (which is used by Tiller client) [2] constraints, it does not handle CIDR in no_proxy environment variables [3]. One of the options is:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">1. [high priority] Expose Tiller to create a DNS name for the pod, and use it when making requests from Armada API to Tiller, in this case Aramada API POD can have ".cluster.local" in NO_PROXY and everything would work just fine (i.e. requests
to Tiller won't go through the proxy).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2. [low priority] Reduce coupling between Tiller/k8s and Shipyard by configuring Tiller endpoints in Armada API directly without requiring Shipyard to pass one.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Prior to submitting any changes I would like to verify with you if this makes sense or if I should explore other options.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Darmada_blob_307f1318c4e83f247f1e3838478957e7555d6ce0_armada_handlers_tiller.py-23L158-2DL164&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=1j8HwScWR1I9HzTq0VzYbOZQbURKo8fwxRWWd4EKvJc&e=">
https://github.com/openstack/airship-armada/blob/307f1318c4e83f247f1e3838478957e7555d6ce0/armada/handlers/tiller.py#L158-L164</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=">
https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[3] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=">
https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Dec 20, 2018 at 1:42 PM Drew Walters <<a href="mailto:drewwalters96@gmail.com">drewwalters96@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Hi Eli,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod. Shipyard uses the Python Kubernetes client to retrieve the IP address that it provides to Armada.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">1. It is possible to create a clusterIP with DNS and provide a domain name to Armada.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2. Armada only communicates with one Tiller pod at a time. This can be changed per API request by providing the hostname of a different Tiller pod. In the event that a Tiller hostname is not provided, Armada selects the first running Tiller
pod it encounters.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We invite you to join our IRC team meeting (#airshipit on freenode) January 8th to help us better understand your desired configuration and discuss in more detail. In the meantime, please let us know if you have any additional questions!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Drew Walters<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Dec 12, 2018 at 3:55 PM Evgeny L <<a href="mailto:eli@mirantis.com" target="_blank">eli@mirantis.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've been working on getting behind the proxy configuration for AIAB to work [1] and I stumbled across a problem related to GRPC [2], which does not support CIDR notation for no_proxy entries, this make Armada API to fail to connect to
Tiller, because it tries to use proxy configured with environment variables. <o:p>
</o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">While looking for a solution other than doing some manual reconfiguration [2], I started to have the following questions:
<o:p></o:p></p>
</div>
<div>
<ol start="1" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP and use a DNS name?
<o:p></o:p></li></ol>
</div>
<div>
<ol start="2" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Why Shipyard is the one who provides Armada API with Tiller IP address? If we do 1st we can possibly make endpoint a part of Armada API configuration. Or is Armada API supposed to work with multiple Tillers at the same time?
<o:p></o:p></li></ol>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any feedback and historical background would be really helpful!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624122_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=Ogo7hxRYVOTBW1SbxHDdonCymHGxS0x58TI5iPBXvAQ&e=" target="_blank">https://review.openstack.org/#/c/624122/</a>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=" target="_blank">https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst</a>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[3] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=" target="_blank">https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149</a>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[4] <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dshipyard_blob_a235f8e6db6e791550727e3575e045e26f177252_src_bin_shipyard-5Fairflow_shipyard-5Fairflow_plugins_armada-5Fbase-5Foperator.py-23L147-2DL153&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=xLj4wU7W8s0CdleOZmKug7vs7v5Lh--u2UmlTfUXlHs&e=" target="_blank">https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e3575e045e26f177252/src/bin/shipyard_airflow/shipyard_airflow/plugins/armada_base_operator.py#L147-L153</a>
<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Airship-discuss mailing list<br>
<a href="mailto:Airship-discuss@lists.airshipit.org" target="_blank">Airship-discuss@lists.airshipit.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=wXSgWU9qQaFrrE0bQKaQXWNDGaAJlwMAiqqPOTvvLz8&e=" target="_blank">http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss</a><o:p></o:p></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>