<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Looking forward to seeing what you have in mind, Shoaib. If you’d like to discuss in tomorrow’s team meeting, feel free to add the topic to the etherpad:
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://etherpad.openstack.org/p/airship-meeting-2019-01-15">https://etherpad.openstack.org/p/airship-meeting-2019-01-15</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<br>
Matt<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Nasir, Shoaib <Shoaib.Nasir@windriver.com> <br>
<b>Sent:</b> Thursday, January 10, 2019 9:26 AM<br>
<b>To:</b> EAGAN, SEAN <se136c@att.com>; airship-discuss@lists.airshipit.org<br>
<b>Subject:</b> Re: [Airship-discuss] Armada Compatibility with Keystone<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi Sean<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">After digging further into the issue, I realized that armada does not support passing end users bearer-tokens.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Armada can be authenticated by keystone for incoming API requests.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">But in the scenario, where I want to use armada to bring up openstack-helm pods on a kubernetes cluster that’s already setup to be authenticated by keystone using the k8s-keystone-auth Webhook application, armada
is not compatible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Therefore, I have made some changes to the armada CLI such that a user can pass its bearer-token to tiller via armada using the CLI.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I will be proposing this upstream to armada soon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Shoaib Nasir<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> EAGAN, SEAN [<a href="mailto:se136c@att.com">mailto:se136c@att.com</a>]
<br>
<b>Sent:</b> Wednesday, January 09, 2019 1:02 PM<br>
<b>To:</b> Nasir, Shoaib; <a href="mailto:airship-discuss@lists.airshipit.org">airship-discuss@lists.airshipit.org</a><br>
<b>Subject:</b> RE: Armada Compatibility with Keystone<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Shoaib,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It looks like that feature of k8s-keystone-auth [0] is implemented by kubernetes/client-go [1] [2].<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">kubernetes/client-go is used by helm so that’s why it works there.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However, armada is using kubernetes-client/python, and as far as I can tell this feature is not implemented there. Unless I am missing it.<o:p></o:p></p>
<p class="MsoNormal">I am not sure what the easiest path to getting that implemented will be.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[0]: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_cloud-2Dprovider-2Dopenstack_blob_master_docs_using-2Dkeystone-2Dwebhook-2Dauthenticator-2Dand-2Dauthorizer.md-23clientkubectl-2Dconfiguration&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qVl9v7EluGfsovbPYxi9J0-eSktkyFZ-inq7sKmgtWY&s=v7fsUjtnEjd10n9wUMyCRFRvFMCvhs4kpYCKfJn3NfE&e=">
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-keystone-webhook-authenticator-and-authorizer.md#clientkubectl-configuration</a><o:p></o:p></p>
<p class="MsoNormal">[1]: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_cloud-2Dprovider-2Dopenstack_blob_master_docs_using-2Dclient-2Dkeystone-2Dauth.md&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qVl9v7EluGfsovbPYxi9J0-eSktkyFZ-inq7sKmgtWY&s=OcvR-HBkecGK7QMmr-UCS5zU5dlxeSHg1LbvHgfR6Es&e=">
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-client-keystone-auth.md</a><o:p></o:p></p>
<p class="MsoNormal">[2]: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_pull_64346&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=_C5hC_103uW491yNPPpNmA&m=qVl9v7EluGfsovbPYxi9J0-eSktkyFZ-inq7sKmgtWY&s=DIUfb2shPrZdP8mQ3IxLfbjPIEV-f9WI-oWGJ9VoMHY&e=">
https://github.com/kubernetes/kubernetes/pull/64346</a> <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sean Eagan<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Nasir, Shoaib <<a href="mailto:Shoaib.Nasir@windriver.com">Shoaib.Nasir@windriver.com</a>>
<br>
<b>Sent:</b> Thursday, December 13, 2018 12:35 PM<br>
<b>To:</b> <a href="mailto:airship-discuss@lists.airshipit.org">airship-discuss@lists.airshipit.org</a><br>
<b>Subject:</b> [Airship-discuss] Armada Compatibility with Keystone<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I have a kubernetes cluster that is setup with Keystone as its Authentication/Authorization Backend (via the k8s-keystone-auth Webhook pod)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">With that, any application such as helm now needs to pass user credentials or a keystone token when talking to the kubernetes-apiserver<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">If I set up env OS_ variables and use any helm commands (such as helm init, or helm lists etc), everything works fine as these OS_ user credentials are authorized via the kubernetes keystone backend<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">However, I run into problems with Armada. After bringing up the armada docker container, and logging into it to execute any armada apply commands, and using the –token argument to pass along a valid keystone
token, armada still fails to access the kubernetes cluster.”User “system:anonymous” suggests that the token that armada cli gets, does not pass along to the kubernetes-apiserver.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I’m very new to armada and need some assistance in figuring out how to setup the armada container or any other configurations required to make it pass the keystone token to the kubernetes-apiserver.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Each and every armada command fails similar to the one below. Any assistance in this regard would be greatly appreciated ?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>armada@c7f42c9263ad:~$ armada --token $TOKEN --debug apply<o:p></o:p></b></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli [-] Caught unexpected exception: kubernetes.client.rest.ApiException: (403)<o:p></o:p></p>
<p class="MsoNormal">Reason: Forbidden<o:p></o:p></p>
<p class="MsoNormal">HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'})<o:p></o:p></p>
<p class="MsoNormal">HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"<span style="background:yellow;mso-highlight:yellow">status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group
\"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}</span><o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli Traceback (most recent call last):<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli self.invoke()<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 215, in invoke<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli target_manifest=self.target_manifest)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 103, in __init__<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli dry_run=dry_run)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 95, in __init__<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli self.channel = self.get_channel()<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 117, in get_channel<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli tiller_ip = self._get_tiller_ip()<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 163, in _get_tiller_ip<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli pod = self._get_tiller_pod()<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 141, in _get_tiller_pod<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli CONF.tiller_pod_labels).items<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/k8s.py", line 215, in get_namespace_pod<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli namespace, label_selector=label_selector)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli collection_formats=collection_formats)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli _return_http_data_only, collection_formats, _preload_content, _request_timeout)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli _request_timeout=_request_timeout)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 342, in request<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli headers=headers)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 231, in GET<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli query_params=query_params)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli raise ApiException(http_resp=r)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli kubernetes.client.rest.ApiException: (403)<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli Reason: Forbidden<o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json',
'X-Content-Type-Options': 'nosniff'})<o:p></o:p></span></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\"
cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}</span><o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli<o:p></o:p></p>
<p class="MsoNormal">2018-12-13 18:10:13.656 72 ERROR armada.cli<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>