Authenticate issue with EKS cluster using latest version v32.0.0

[alyssa1303]

What happened (please include outputs or screenshots):
Error when trying to list EKS nodes or pods

    return self.api.list_node(label_selector=label_selector, field_selector=field_selector).items
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 17150, in list_node
    return self.list_node_with_http_info(**kwargs)  # noqa: E501
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 17261, in list_node_with_http_info
    return self.api_client.call_api(
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
  File "/home/AzDevOps/.local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '33e59936-e514-4ff3-8b96-34a0af79a236', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '5c784c43-9cd3-4a50-9593-03c15141a761', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c62bb07a-ffc7-4c7b-a8ba-fc542fd3c6d1', 'Date': 'Fri, 24 Jan 2025 00:32:24 GMT', 'Content-Length': '256'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"nodes"},"code":403}

What you expected to happen:
Should be able to list node just fine with function api.list_node()

How to reproduce it (as minimally and precisely as possible):
Create EKS cluster
Install kubernetes package version v32.0.0
Run following

from kubernetes import client, config
client = client.CoreV1Api()
client.list_node(label_selector=label_selector, field_selector=field_selector).items

Anything else we need to know?:
It was working fine with version v31.0.0 but failed after upgrading to version v32.0.0

Environment:

• Kubernetes version (kubectl version): v1.32.0
• OS (e.g., MacOS 10.13.6): Ubuntu 2204
• Python version (python --version) Python 3.10.12
• Python client version (pip list | grep kubernetes) v32.0.0

[66li]

This can be avoided by downgrading to 31.0.0

[cyliu0]

Downgrading to 31.0.0 also works for me.

[diseku]

same! broke our CI pipes

[Ottovsky]

Same with AKS, downgrading to 31.0.0 helps.

[romilbhardwaj]

Same on GKE v1.31.4-gke.1183000

  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 13441, in delete_namespaced_service
    return self.delete_namespaced_service_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/Users/romilb/tools/anaconda3/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 13552, in delete_namespaced_service_with_http_info
    return self.api_client.call_api(
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 415, in request
    return self.rest_client.DELETE(url,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/rest.py", line 270, in DELETE
    return self.request("DELETE", url,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '13b6bd19-4596-425a-aae0-d03b518a53a5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '965a2a6c-7d5c-4983-bedd-0f6df5807afb', 'X-Kubernetes-Pf-Prioritylevel-Uid': '95309311-518d-477e-b551-637d8218a96c', 'Date': 'Fri, 24 Jan 2025 22:16:19 GMT', 'Content-Length': '334'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"test-2ea4--skypilot-lb\" is forbidden: User \"system:anonymous\" cannot delete resource \"services\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"test-2ea4--skypilot-lb","kind":"services"},"code":403}

[bagerard]

Observing a slightly different error but root cause may be similar, I'm observing a 401 unauthorized when connecting to AKS with SP client/secret.

I've debugged it to see where was the difference with the previous version and it turned out to be the shell=True additional param added to process = subprocess.Popen done here .
2dfa782

In my case it's failing when calling kubelogin -get-token, the kubelogin calls returns the command help instead of the token, it logs a parsing error but proceeds and subsequent usage of the token fails badly. Executing the same subprocess.Popen call withshell=Falseworks.

[akhilputhiry]

I just came across this issue :) Here is a workaround in case some one wants

from kubernetes.config import kube_config as k8config
from kubernetes.config.exec_provider import ExecProvider


class AWSExecProvider(ExecProvider):

    def __init__(self, exec_config, cwd, cluster=None) -> None:
        super().__init__(exec_config, cwd, cluster)
        if isinstance(self.args, list):
            self.args = " ".join(self.args)


k8config.ExecProvider = AWSExecProvider


api_client = k8config.new_client_from_config_dict()
client = k8client.CoreV1Api(api_client)
client.list_namespaced_pods("default")

[cesarqdt]

Same here, AKS kubernetes version 1.28.15 using this code:

config.load_kube_config()
api = client.CoreV1Api()
pods = api.list_namespaced_pod(namespace)

Getting the following error:

2025-01-27T16:36:00.6511896Z         if not 200 <= r.status <= 299:
2025-01-27T16:36:00.6512295Z >           raise ApiException(http_resp=r)
2025-01-27T16:36:00.6512712Z E           kubernetes.client.exceptions.ApiException: (401)
2025-01-27T16:36:00.6513132Z E           Reason: Unauthorized
2025-01-27T16:36:00.6513832Z E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '98668c59-8242-46dd-bcb3-2b43e08729bc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 27 Jan 2025 16:35:59 GMT', 'Content-Length': '129'})
2025-01-27T16:36:00.6514912Z E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

The solution to downgrade to version 26 pip install kubernetes==26.1.0

[rubix-git]

I don't think shell=True works when passing a list to Popen()
https://github.com/kubernetes-client/python/blob/release-32.0/kubernetes/base/config/exec_provider.py#L85

[EraYaN]

shell=True is indeed the source of the error, because subprocess can not execute the program with the give arguments correctly because it won't take a list in this form. It was added in #2289

To quote the python documentation

If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, [Popen](https://docs.python.org/3/library/subprocess.html#subprocess.Popen) does the equivalent of:

https://docs.python.org/3/library/subprocess.html#popen-constructor