Kubernetes authentication not working from inside the cluster

[kpouget]

I'm running a simple code based on this example:

    from codeflare_sdk.cluster.cluster import Cluster, ClusterConfiguration

    # Create our cluster and submit appwrapper
    cluster = Cluster(ClusterConfiguration(
        namespace=namespace, name=f"{name}-user{user_idx}",
        image=image,
        min_worker=workers, max_worker=workers,
        min_cpus=cpu, max_cpus=cpu,
        min_memory=memory, max_memory=memory,
        gpu=gpu, instascale=False))

    # Bring up the cluster
    cluster.up()

but this last call fails with when deployed from the current main branch:

Traceback (most recent call last):
  File "/opt/venv/src/codeflare-sdk/src/codeflare_sdk/cluster/cluster.py", line 120, in up
    config_check()
  File "/opt/venv/src/codeflare-sdk/src/codeflare_sdk/cluster/auth.py", line 165, in config_check
    config.load_kube_config()
  File "/opt/venv/lib/python3.9/site-packages/kubernetes/config/kube_config.py", line 813, in load_kube_config
    loader = _get_kube_config_loader(
  File "/opt/venv/lib/python3.9/site-packages/kubernetes/config/kube_config.py", line 770, in _get_kube_config_loader
    raise ConfigException(
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.

During handling of the above exception, another exception occurred:
...
File "/mnt/logs/002__run_one/sample.py", line 37, in main
  cluster.up()
File "/opt/venv/src/codeflare-sdk/src/codeflare_sdk/cluster/cluster.py", line 132, in up
  return _kube_api_error_handling(e)
File "/opt/venv/src/codeflare-sdk/src/codeflare_sdk/utils/kube_api_helpers.py", line 32, in _kube_api_error_handling
  raise PermissionError(perm_msg)
PermissionError: Action not permitted, have you put in correct/up-to-date auth credentials?

This code is running inside a Pod, and relying on the in-cluster config (token from this file: /run/secrets/kubernetes.io/serviceaccount/token)

[kpouget]

Adding this test to the Kubernetes authentication should address the issue:

    if 'KUBERNETES_PORT' in os.environ:
        # os.environ['KUBERNETES_SERVICE_HOST'] = 'kubernetes'
        config.load_incluster_config()
    else:
        config.load_kube_config()

[Maxusmusti]

@Bobbins228 from your slack thread earlier