I have a kubernetes cluster that is setup with Keystone as its Authentication/Authorization Backend (via the k8s-keystone-auth Webhook pod)

With that, any application such as helm now needs to pass user credentials or a keystone token when talking to the kubernetes-apiserver

If I set up env OS_ variables and use any helm commands (such as helm init, or helm lists etc), everything works fine as these OS_ user credentials are authorized via the kubernetes keystone backend

However, I run into problems with Armada. After bringing up the armada docker container, and logging into it to execute any armada apply commands, and using the –token argument to pass along a valid keystone token, armada still fails to access the kubernetes cluster.”User “system:anonymous” suggests that the token that armada cli gets, does not pass along to the kubernetes-apiserver.

I’m very new to armada and need some assistance in figuring out how to setup the armada container or any other configurations required to make it pass the keystone token to the kubernetes-apiserver.

Each and every armada command fails similar to the one below. Any assistance in this regard would be greatly appreciated ?

armada@c7f42c9263ad:~$ armada --token $TOKEN --debug apply

2018-12-13 18:10:13.656 72 ERROR armada.cli [-] Caught unexpected exception: kubernetes.client.rest.ApiException: (403)

Reason: Forbidden

HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'})

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

2018-12-13 18:10:13.656 72 ERROR armada.cli Traceback (most recent call last):

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/__init__.py", line 39, in safe_invoke

2018-12-13 18:10:13.656 72 ERROR armada.cli self.invoke()

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/cli/apply.py", line 215, in invoke

2018-12-13 18:10:13.656 72 ERROR armada.cli target_manifest=self.target_manifest)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/armada.py", line 103, in __init__

2018-12-13 18:10:13.656 72 ERROR armada.cli dry_run=dry_run)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 95, in __init__

2018-12-13 18:10:13.656 72 ERROR armada.cli self.channel = self.get_channel()

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 117, in get_channel

2018-12-13 18:10:13.656 72 ERROR armada.cli tiller_ip = self._get_tiller_ip()

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 163, in _get_tiller_ip

2018-12-13 18:10:13.656 72 ERROR armada.cli pod = self._get_tiller_pod()

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/tiller.py", line 141, in _get_tiller_pod

2018-12-13 18:10:13.656 72 ERROR armada.cli CONF.tiller_pod_labels).items

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/armada/handlers/k8s.py", line 215, in get_namespace_pod

2018-12-13 18:10:13.656 72 ERROR armada.cli namespace, label_selector=label_selector)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12310, in list_namespaced_pod

2018-12-13 18:10:13.656 72 ERROR armada.cli (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 12413, in list_namespaced_pod_with_http_info

2018-12-13 18:10:13.656 72 ERROR armada.cli collection_formats=collection_formats)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api

2018-12-13 18:10:13.656 72 ERROR armada.cli _return_http_data_only, collection_formats, _preload_content, _request_timeout)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api

2018-12-13 18:10:13.656 72 ERROR armada.cli _request_timeout=_request_timeout)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 342, in request

2018-12-13 18:10:13.656 72 ERROR armada.cli headers=headers)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 231, in GET

2018-12-13 18:10:13.656 72 ERROR armada.cli query_params=query_params)

2018-12-13 18:10:13.656 72 ERROR armada.cli File "/usr/local/lib/python3.5/site-packages/kubernetes/client/rest.py", line 222, in request

2018-12-13 18:10:13.656 72 ERROR armada.cli raise ApiException(http_resp=r)

2018-12-13 18:10:13.656 72 ERROR armada.cli kubernetes.client.rest.ApiException: (403)

2018-12-13 18:10:13.656 72 ERROR armada.cli Reason: Forbidden

2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 13 Dec 2018 18:10:13 GMT', 'Content-Length': '265', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'})

2018-12-13 18:10:13.656 72 ERROR armada.cli HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

2018-12-13 18:10:13.656 72 ERROR armada.cli