Armada API connectivity to Tiller (proxy problem)
Hi, I've been working on getting behind the proxy configuration for AIAB to work [1] and I stumbled across a problem related to GRPC [2], which does not support CIDR notation for no_proxy entries, this make Armada API to fail to connect to Tiller, because it tries to use proxy configured with environment variables. While looking for a solution other than doing some manual reconfiguration [2], I started to have the following questions: 1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP and use a DNS name? 2. Why Shipyard is the one who provides Armada API with Tiller IP address? If we do 1st we can possibly make endpoint a part of Armada API configuration. Or is Armada API supposed to work with multiple Tillers at the same time? Any feedback and historical background would be really helpful! Thanks, [1] https://review.openstack.org/#/c/624122/ [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/s... [4] https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e357...
Hi Eli, The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod. Shipyard uses the Python Kubernetes client to retrieve the IP address that it provides to Armada. 1. It is possible to create a clusterIP with DNS and provide a domain name to Armada. 2. Armada only communicates with one Tiller pod at a time. This can be changed per API request by providing the hostname of a different Tiller pod. In the event that a Tiller hostname is not provided, Armada selects the first running Tiller pod it encounters. We invite you to join our IRC team meeting (#airshipit on freenode) January 8th to help us better understand your desired configuration and discuss in more detail. In the meantime, please let us know if you have any additional questions! Best, Drew Walters On Wed, Dec 12, 2018 at 3:55 PM Evgeny L <eli@mirantis.com> wrote:
Hi,
I've been working on getting behind the proxy configuration for AIAB to work [1] and I stumbled across a problem related to GRPC [2], which does not support CIDR notation for no_proxy entries, this make Armada API to fail to connect to Tiller, because it tries to use proxy configured with environment variables.
While looking for a solution other than doing some manual reconfiguration [2], I started to have the following questions:
1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP and use a DNS name? 2. Why Shipyard is the one who provides Armada API with Tiller IP address? If we do 1st we can possibly make endpoint a part of Armada API configuration. Or is Armada API supposed to work with multiple Tillers at the same time?
Any feedback and historical background would be really helpful!
Thanks,
[1] https://review.openstack.org/#/c/624122/ [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/s... [4] https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e357... _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
The Armada API accepts a Tiller hostname as a safeguard in the event
Hi Drew, Thank you for the response! that Armada is unable to find a running Tiller pod Looking at the code [1] it seems to be that Armada will try to find Tiller pod in the event Shipyard doesn't find one. Was there some problem with Armada API not being able to retrieve Tiller endpoint so it required a fallback? They both seem to use k8s API for that.
This can be changed per API request by providing the hostname of a different Tiller pod.
From architecture point of view, do we expect it to work with multiple Tillers within a single Airship installation?
I'm asking these questions to explore the options for resolving GRPC (which is used by Tiller client) [2] constraints, it does not handle CIDR in no_proxy environment variables [3]. One of the options is: 1. [high priority] Expose Tiller to create a DNS name for the pod, and use it when making requests from Armada API to Tiller, in this case Aramada API POD can have ".cluster.local" in NO_PROXY and everything would work just fine (i.e. requests to Tiller won't go through the proxy). 2. [low priority] Reduce coupling between Tiller/k8s and Shipyard by configuring Tiller endpoints in Armada API directly without requiring Shipyard to pass one. Prior to submitting any changes I would like to verify with you if this makes sense or if I should explore other options. Thanks, [1] https://github.com/openstack/airship-armada/blob/307f1318c4e83f247f1e3838478... [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/s... On Thu, Dec 20, 2018 at 1:42 PM Drew Walters <drewwalters96@gmail.com> wrote:
Hi Eli,
The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod. Shipyard uses the Python Kubernetes client to retrieve the IP address that it provides to Armada.
1. It is possible to create a clusterIP with DNS and provide a domain name to Armada.
2. Armada only communicates with one Tiller pod at a time. This can be changed per API request by providing the hostname of a different Tiller pod. In the event that a Tiller hostname is not provided, Armada selects the first running Tiller pod it encounters.
We invite you to join our IRC team meeting (#airshipit on freenode) January 8th to help us better understand your desired configuration and discuss in more detail. In the meantime, please let us know if you have any additional questions!
Best, Drew Walters
On Wed, Dec 12, 2018 at 3:55 PM Evgeny L <eli@mirantis.com> wrote:
Hi,
I've been working on getting behind the proxy configuration for AIAB to work [1] and I stumbled across a problem related to GRPC [2], which does not support CIDR notation for no_proxy entries, this make Armada API to fail to connect to Tiller, because it tries to use proxy configured with environment variables.
While looking for a solution other than doing some manual reconfiguration [2], I started to have the following questions:
1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP and use a DNS name? 2. Why Shipyard is the one who provides Armada API with Tiller IP address? If we do 1st we can possibly make endpoint a part of Armada API configuration. Or is Armada API supposed to work with multiple Tillers at the same time?
Any feedback and historical background would be really helpful!
Thanks,
[1] https://review.openstack.org/#/c/624122/ [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/s... [4] https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e357... _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss
Hey Eli, Drew and I dug into this a bit, and it looks like the Armada-based lookup uses labels (configurably: application=tiller, name=tiller, in the kube-system namespace), and Shipyard looks up tiller based on the pod name [1]. I believe there was some condition under which the preferable label-based lookup wouldn’t work, and that’s why there a “plan B” for looking up based on pod name. However, I think this could be improved by 1) moving the “plan B” logic from Shipyard into Armada (your coupling point #2 below), and 2) allowing Armada to try “plan A” first before falling back to “plan B”. I’d like to discuss this in the 1/8 Airship team IRC meeting [2] – would you be able to join so we can make sure we account for all cases? Between {pod labels, pod name, service DNS} maybe we could find two Armada configuration-driven approaches that could cover all bases. [1] https://github.com/openstack/airship-shipyard/blob/c7a9c65c886641029ce84724b... [2] https://etherpad.openstack.org/p/airship-meeting-2019-01-08 Thanks! Matt From: Evgeny L <eli@mirantis.com> Sent: Wednesday, December 26, 2018 12:56 PM To: Drew Walters <drewwalters96@gmail.com> Cc: Airship-discuss@lists.airshipit.org Subject: Re: [Airship-discuss] Armada API connectivity to Tiller (proxy problem) Hi Drew, Thank you for the response!
The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod
Looking at the code [1] it seems to be that Armada will try to find Tiller pod in the event Shipyard doesn't find one. Was there some problem with Armada API not being able to retrieve Tiller endpoint so it required a fallback? They both seem to use k8s API for that.
This can be changed per API request by providing the hostname of a different Tiller pod.
From architecture point of view, do we expect it to work with multiple Tillers within a single Airship installation? I'm asking these questions to explore the options for resolving GRPC (which is used by Tiller client) [2] constraints, it does not handle CIDR in no_proxy environment variables [3]. One of the options is: 1. [high priority] Expose Tiller to create a DNS name for the pod, and use it when making requests from Armada API to Tiller, in this case Aramada API POD can have ".cluster.local" in NO_PROXY and everything would work just fine (i.e. requests to Tiller won't go through the proxy). 2. [low priority] Reduce coupling between Tiller/k8s and Shipyard by configuring Tiller endpoints in Armada API directly without requiring Shipyard to pass one. Prior to submitting any changes I would like to verify with you if this makes sense or if I should explore other options. Thanks, [1] https://github.com/openstack/airship-armada/blob/307f1318c4e83f247f1e3838478957e7555d6ce0/armada/handlers/tiller.py#L158-L164<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Darmada_blob_307f1318c4e83f247f1e3838478957e7555d6ce0_armada_handlers_tiller.py-23L158-2DL164&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=1j8HwScWR1I9HzTq0VzYbOZQbURKo8fwxRWWd4EKvJc&e=> [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst<https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=> [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=> On Thu, Dec 20, 2018 at 1:42 PM Drew Walters <drewwalters96@gmail.com<mailto:drewwalters96@gmail.com>> wrote: Hi Eli, The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod. Shipyard uses the Python Kubernetes client to retrieve the IP address that it provides to Armada. 1. It is possible to create a clusterIP with DNS and provide a domain name to Armada. 2. Armada only communicates with one Tiller pod at a time. This can be changed per API request by providing the hostname of a different Tiller pod. In the event that a Tiller hostname is not provided, Armada selects the first running Tiller pod it encounters. We invite you to join our IRC team meeting (#airshipit on freenode) January 8th to help us better understand your desired configuration and discuss in more detail. In the meantime, please let us know if you have any additional questions! Best, Drew Walters On Wed, Dec 12, 2018 at 3:55 PM Evgeny L <eli@mirantis.com<mailto:eli@mirantis.com>> wrote: Hi, I've been working on getting behind the proxy configuration for AIAB to work [1] and I stumbled across a problem related to GRPC [2], which does not support CIDR notation for no_proxy entries, this make Armada API to fail to connect to Tiller, because it tries to use proxy configured with environment variables. While looking for a solution other than doing some manual reconfiguration [2], I started to have the following questions: 1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP and use a DNS name? 1. Why Shipyard is the one who provides Armada API with Tiller IP address? If we do 1st we can possibly make endpoint a part of Armada API configuration. Or is Armada API supposed to work with multiple Tillers at the same time? Any feedback and historical background would be really helpful! Thanks, [1] https://review.openstack.org/#/c/624122/<https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624122_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=Ogo7hxRYVOTBW1SbxHDdonCymHGxS0x58TI5iPBXvAQ&e=> [2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst<https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=> [3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/src/core/ext/filters/client_channel/http_proxy.cc#L146-L149<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=> [4] https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e3575e045e26f177252/src/bin/shipyard_airflow/shipyard_airflow/plugins/armada_base_operator.py#L147-L153<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dshipyard_blob_a235f8e6db6e791550727e3575e045e26f177252_src_bin_shipyard-5Fairflow_shipyard-5Fairflow_plugins_armada-5Fbase-5Foperator.py-23L147-2DL153&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=xLj4wU7W8s0CdleOZmKug7vs7v5Lh--u2UmlTfUXlHs&e=> _______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org<mailto:Airship-discuss@lists.airshipit.org> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=wXSgWU9qQaFrrE0bQKaQXWNDGaAJlwMAiqqPOTvvLz8&e=>
Hi Matt, Drew, Thanks for looking into this, sure, I can join to the meeting, but can we move this topic down to the bottom? It's 06:00AM PT for me, so I just want to make sure that I won't be too late for the discussion. On Fri, Jan 4, 2019 at 10:12 AM MCEUEN, MATT <MM9745@att.com> wrote:
Hey Eli,
Drew and I dug into this a bit, and it looks like the Armada-based lookup uses labels (configurably: application=tiller, name=tiller, in the kube-system namespace), and Shipyard looks up tiller based on the pod name [1]. I believe there was some condition under which the preferable label-based lookup wouldn’t work, and that’s why there a “plan B” for looking up based on pod name.
However, I think this could be improved by 1) moving the “plan B” logic from Shipyard into Armada (your coupling point #2 below), and 2) allowing Armada to try “plan A” first before falling back to “plan B”. I’d like to discuss this in the 1/8 Airship team IRC meeting [2] – would you be able to join so we can make sure we account for all cases? Between {pod labels, pod name, service DNS} maybe we could find two Armada configuration-driven approaches that could cover all bases.
[1] https://github.com/openstack/airship-shipyard/blob/c7a9c65c886641029ce84724b...
[2] https://etherpad.openstack.org/p/airship-meeting-2019-01-08
Thanks!
Matt
*From:* Evgeny L <eli@mirantis.com> *Sent:* Wednesday, December 26, 2018 12:56 PM *To:* Drew Walters <drewwalters96@gmail.com> *Cc:* Airship-discuss@lists.airshipit.org *Subject:* Re: [Airship-discuss] Armada API connectivity to Tiller (proxy problem)
Hi Drew,
Thank you for the response!
The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod
Looking at the code [1] it seems to be that Armada will try to find Tiller pod in the event Shipyard doesn't find one.
Was there some problem with Armada API not being able to retrieve Tiller endpoint so it required a fallback? They both seem to use k8s API for that.
This can be changed per API request by providing the hostname of a different Tiller pod.
From architecture point of view, do we expect it to work with multiple Tillers within a single Airship installation?
I'm asking these questions to explore the options for resolving GRPC (which is used by Tiller client) [2] constraints, it does not handle CIDR in no_proxy environment variables [3]. One of the options is:
1. [high priority] Expose Tiller to create a DNS name for the pod, and use it when making requests from Armada API to Tiller, in this case Aramada API POD can have ".cluster.local" in NO_PROXY and everything would work just fine (i.e. requests to Tiller won't go through the proxy).
2. [low priority] Reduce coupling between Tiller/k8s and Shipyard by configuring Tiller endpoints in Armada API directly without requiring Shipyard to pass one.
Prior to submitting any changes I would like to verify with you if this makes sense or if I should explore other options.
Thanks,
[1] https://github.com/openstack/airship-armada/blob/307f1318c4e83f247f1e3838478... <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Darmada_blob_307f1318c4e83f247f1e3838478957e7555d6ce0_armada_handlers_tiller.py-23L158-2DL164&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=1j8HwScWR1I9HzTq0VzYbOZQbURKo8fwxRWWd4EKvJc&e=>
[2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst <https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=>
[3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/s... <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=>
On Thu, Dec 20, 2018 at 1:42 PM Drew Walters <drewwalters96@gmail.com> wrote:
Hi Eli,
The Armada API accepts a Tiller hostname as a safeguard in the event that Armada is unable to find a running Tiller pod. Shipyard uses the Python Kubernetes client to retrieve the IP address that it provides to Armada.
1. It is possible to create a clusterIP with DNS and provide a domain name to Armada.
2. Armada only communicates with one Tiller pod at a time. This can be changed per API request by providing the hostname of a different Tiller pod. In the event that a Tiller hostname is not provided, Armada selects the first running Tiller pod it encounters.
We invite you to join our IRC team meeting (#airshipit on freenode) January 8th to help us better understand your desired configuration and discuss in more detail. In the meantime, please let us know if you have any additional questions!
Best,
Drew Walters
On Wed, Dec 12, 2018 at 3:55 PM Evgeny L <eli@mirantis.com> wrote:
Hi,
I've been working on getting behind the proxy configuration for AIAB to work [1] and I stumbled across a problem related to GRPC [2], which does not support CIDR notation for no_proxy entries, this make Armada API to fail to connect to Tiller, because it tries to use proxy configured with environment variables.
While looking for a solution other than doing some manual reconfiguration [2], I started to have the following questions:
1. Why does Shipyard uses [4] Tiller Pod IP? Can we create a ClusterIP and use a DNS name?
1. Why Shipyard is the one who provides Armada API with Tiller IP address? If we do 1st we can possibly make endpoint a part of Armada API configuration. Or is Armada API supposed to work with multiple Tillers at the same time?
Any feedback and historical background would be really helpful!
Thanks,
[1] https://review.openstack.org/#/c/624122/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624122_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=Ogo7hxRYVOTBW1SbxHDdonCymHGxS0x58TI5iPBXvAQ&e=>
[2] https://review.openstack.org/#/c/624504/2/manifests/dev_minimal/README.rst <https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_c_624504_2_manifests_dev-5Fminimal_README.rst&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=o8myw4QYVPuJ_4rQUUiukfhT0rtryKwnGcXqFHYKUIc&e=>
[3] https://github.com/grpc/grpc/blob/9e9cae7839a362936228cf333045e5da877ace40/s... <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_grpc_grpc_blob_9e9cae7839a362936228cf333045e5da877ace40_src_core_ext_filters_client-5Fchannel_http-5Fproxy.cc-23L146-2DL149&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=aUF8IkJYcZWihOSodEolrSP4GtWC1NAH-8UnzOGWJEM&e=>
[4] https://github.com/openstack/airship-shipyard/blob/a235f8e6db6e791550727e357... <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_airship-2Dshipyard_blob_a235f8e6db6e791550727e3575e045e26f177252_src_bin_shipyard-5Fairflow_shipyard-5Fairflow_plugins_armada-5Fbase-5Foperator.py-23L147-2DL153&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=xLj4wU7W8s0CdleOZmKug7vs7v5Lh--u2UmlTfUXlHs&e=>
_______________________________________________ Airship-discuss mailing list Airship-discuss@lists.airshipit.org http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.airshipit.org_cgi-2Dbin_mailman_listinfo_airship-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=d5Vew4AcZ1XhhJF-PLOP-KpxYcp8_FvHRniqs1PN5Uo&s=wXSgWU9qQaFrrE0bQKaQXWNDGaAJlwMAiqqPOTvvLz8&e=>
participants (3)
-
Drew Walters
-
Evgeny L
-
MCEUEN, MATT