urllib3 v2.4.0 on Python 3.13 doesn't work with EKS

[Tenzer]

What happened (please include outputs or screenshots):

The following exception is raised whenever calling the Kubernetes API of an EKS cluster:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0123456789deadbeef01234567890000.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))

Full stacktrace

Traceback (most recent call last):
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
    ~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
    ~~~~~~~~~~~~^^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
        sock=sock,
    ...<14 lines>...
        assert_fingerprint=self.assert_fingerprint,
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
        sock=sock,
    ...<8 lines>...
        tls_in_tls=tls_in_tls,
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 480, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 524, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    client.get_code()
    ~~~~~~~~~~~~~~~^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api/version_api.py", line 61, in get_code
    return self.get_code_with_http_info(**kwargs)  # noqa: E501
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api/version_api.py", line 128, in get_code_with_http_info
    return self.api_client.call_api(
           ~~~~~~~~~~~~~~~~~~~~~~~~^
        '/version/', 'GET',
        ^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
        _request_timeout=local_var_params.get('_request_timeout'),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        collection_formats=collection_formats)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                           path_params, query_params, header_params,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
                           _return_http_data_only, collection_formats,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           _preload_content, _request_timeout, _host)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
        method, url, query_params=query_params, headers=header_params,
        post_params=post_params, body=body,
        _preload_content=_preload_content,
        _request_timeout=_request_timeout)
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ~~~~~~~~~~~~~~~~~~~~^^^^^
                                query_params=query_params,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
                                _preload_content=_preload_content,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                _request_timeout=_request_timeout,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                headers=headers)
                                ^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
           ~~~~~~~~~~~~^^^^^^^^^^^^
                        headers=headers,
                        ^^^^^^^^^^^^^^^^
                        _preload_content=_preload_content,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        _request_timeout=_request_timeout,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        query_params=query_params)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 217, in request
    r = self.pool_manager.request(method, url,
                                  fields=query_params,
                                  preload_content=_preload_content,
                                  timeout=timeout,
                                  headers=headers)
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 135, in request
    return self.request_encode_url(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<3 lines>...
        **urlopen_kw,
        ^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
        method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    )
  File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0123456789deadbeef01234567890000.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))

What you expected to happen:

The exception shouldn't be raised and the call to the Kubernetes API should be made successfully.

How to reproduce it (as minimally and precisely as possible):

Use the latest version of this project with urllib3 v2.4.0 on Python 3.13.

import kubernetes
kubernetes.config.load_config()  # Should load a config for an EKS cluster
client = kubernetes.client.VersionApi()
client.get_code()

Anything else we need to know?:

This seems to be caused by the following change in urllib3 v2.4.0: issue, PR, which only takes effect on Python 3.13.

I've only experienced the issue with EKS, which must use self-signed certificates that aren't fully compatible with RFC 5280, notably because they don't provide an Authority Key Identifier.

I don't know if the same issue is the case of other Kubernetes providers.

Environment:

• Kubernetes version (kubectl version): v1.32.3-eks-bcf3d70
• OS (e.g., MacOS 10.13.6): macOS 15.4.1
• Python version (python --version): 3.13.3
• Python client version (pip list | grep kubernetes): 32.0.1

[dblanchette]

We've run into this issue as well. AWS support told us this is only the case with EKS clusters created with version 1.16 and below. Creating a new cluster would fix the issue, but updating does not.

As a workaround:

client_configuration = None
if sys.version_info >= (3, 13):
    # https://docs.python.org/3/whatsnew/3.13.html#ssl
    client_configuration = Configuration()
    client_configuration.verify_ssl = False

api_client = ApiClient(configuration=client_configuration)
kubernetes.client.VersionApi(api_client)

Of course, it's insecure to completely disable SSL verification, but it does avoid the issue.

[skaven81]

We ran into this as well, with an on-prem RKE 1.28 cluster.

The issue seems to be related to a change in behavior in the urllib3 library starting with 2.4.0: https://github.com/urllib3/urllib3/releases/tag/2.4.0

Added verify_flags option to create_urllib3_context with a default of VERIFY_X509_PARTIAL_CHAIN and VERIFY_X509_STRICT for Python 3.13+. (urllib3/urllib3#3571)

This seems to have changed the default behavior of TLS connections made with urllib3, specifically on Python 3.13+ Workarounds include downgrading to Python 3.12, or downgrading urllib3 to 2.3.0.

I suspect the fix for this in the kubernetes libary is to update calls to create_urllib3_context with verify_flags set to the previous default value.

[djmattyg007]

We're also running into this issue with an EKS cluster that was created back in 2019. We need to keep the updates flowing, and it seems like the easiest solution for now is to pin urllib3 to v2.3.0, but that's not a viable long-term solution. We would also appreciate a way to configure urllib3 to use the old behaviour, without disabling TLS verification entirely.

[yliaog]

could you please file a PR for the fix? Thanks

[Tenzer]

What do you see an acceptable fix as being?

I guess you either can specify urllib3 should be a version <2.4.0, or perhaps expose an easy way to disable this new sticker check in urllib3?

[yliaog]

we can specify a version < 2.4.0 for urllib3, please add a brief comment about why that is added.