Kubernetes kubeconfig fixes

Modern Kubernetes access often uses exec-based authentication in ~/.kube/config: instead of storing a static token, kubectl runs a helper command (gke-gcloud-auth-plugin, kubelogin, etc.) to fetch credentials on demand.

That works well in a terminal — until you mix multiple credential backends or launch tools like k9s from the desktop. These two fixes address the most common friction points.

Multiple gcloud configurations (GKE)

The problem

If you use more than one Google Cloud "universe" — for example the public googleapis.com cloud and a separate sovereign/partner universe — you typically maintain separate gcloud configurations:

gcloud config configurations list
#   NAME   IS_ACTIVE  ...
#   gcp    True       # googleapis.com
#   acme   False      # another universe domain

Each GKE cluster you add with gcloud container clusters get-credentials is stored in kubeconfig with a user entry that calls gke-gcloud-auth-plugin.

The plugin does not read your kubectl context to decide which gcloud config to use. It uses whichever configuration is globally active:

gcloud config configurations activate gcp   # required for some clusters
gcloud config configurations activate acme  # required for others

So you end up constantly switching gcloud configs before kubectl, even though kubectl already knows which context you want. Symptoms:

• kubectl config use-context cluster-a works only after gcloud config configurations activate …
• Switching contexts between universes gives "You must be logged in to the server" or permission errors
• The failure depends on the last gcloud config configurations activate you ran, not on the context name

Why it happens

gke-gcloud-auth-plugin is a thin wrapper around gcloud application-default / user credentials. Unless told otherwise, it inherits the active gcloud configuration from the environment — not the cluster name embedded in the kubeconfig user.

When all clusters live in one universe, you never notice. With two universes, context switching is not enough.

The fix

Set CLOUDSDK_ACTIVE_CONFIG_NAME per GKE user in kubeconfig. kubectl passes this env var to the exec plugin, so each context carries its own gcloud configuration:

users:
- name: gke_my-project_europe-west1_cluster-a
  user:
    exec:
      command: gke-gcloud-auth-plugin
      env:
      - name: CLOUDSDK_ACTIVE_CONFIG_NAME
        value: gcp          # configuration name from `gcloud config configurations list`

- name: gke_partner:my-project_region_cluster-b
  user:
    exec:
      command: gke-gcloud-auth-plugin
      env:
      - name: CLOUDSDK_ACTIVE_CONFIG_NAME
        value: acme         # the other configuration

After this, kubectl config use-context <name> is sufficient — no manual gcloud config configurations activate.

Apply automatically

Adjust the mapping logic to match how your GKE users are named. Example for configs where partner-universe users are prefixed gke_partner::

python3 <<'PY'
import yaml
from pathlib import Path

# Map kubeconfig user name patterns → gcloud configuration name
UNIVERSE_CONFIGS = [
    ("gke_partner:", "acme"),   # partner / sovereign universe
    ("gke_", "gcp"),            # default public-cloud GKE users
]

path = Path.home() / ".kube/config"
config = yaml.safe_load(path.read_text())

for user in config.get("users", []):
    name = user.get("name", "")
    gcloud_config = next(
        (cfg for prefix, cfg in UNIVERSE_CONFIGS if name.startswith(prefix)),
        None,
    )
    if not gcloud_config:
        continue

    exec_cfg = user.get("user", {}).get("exec") or {}
    if exec_cfg.get("command") != "gke-gcloud-auth-plugin":
        continue

    env = [e for e in (exec_cfg.get("env") or [])
           if e and e.get("name") != "CLOUDSDK_ACTIVE_CONFIG_NAME"]
    env.append({"name": "CLOUDSDK_ACTIVE_CONFIG_NAME", "value": gcloud_config})
    exec_cfg["env"] = env
    user.setdefault("user", {})["exec"] = exec_cfg

path.write_text(yaml.safe_dump(config, default_flow_style=False, sort_keys=False))
print("patched", path)
PY

k9s and unknown command oidc-login for kubectl

The problem

OIDC / Dex authentication is commonly set up with the kubelogin credential plugin (installed via krew as oidc-login). A typical kubeconfig user looks like:

users:
- name: oidc
  user:
    exec:
      command: kubectl
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://issuer.example.com
      - --oidc-client-id=kubernetes
      # ...

Here kubectl is invoked as a plugin host: it must find kubectl-oidc_login (or kubectl-oidc-login) somewhere on PATH.

In an interactive terminal this often works because krew adds ~/.krew/bin to PATH via .bashrc / config.fish.

k9s and other GUI apps usually start with a minimal environment — no krew on PATH, no shell profile sourced. kubectl still runs, but cannot resolve the oidc-login subcommand:

unknown command "oidc-login" for "kubectl"

kubectl get pods from your terminal may work while k9s fails on the same context, which is confusing until you realize they inherit different environments.

Why it happens

Credential plugins are external executables. The kubeconfig command + args must be resolvable without relying on shell setup. Delegating to kubectl oidc-login adds an extra hop: kubectl → plugin discovery → kubectl-oidc_login on PATH.

GUI-launched processes rarely have the same PATH as your terminal.

The fix

Call kubelogin directly instead of kubectl oidc-login. Use the absolute path to the binary (krew store path or a symlink in /usr/local/bin):

users:
- name: oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: ~/.krew/store/oidc-login/<version>/kubelogin   # absolute path
      args:
      - get-token
      - --oidc-issuer-url=https://issuer.example.com
      - --oidc-client-id=kubernetes
      - --oidc-client-secret=<secret>
      - --oidc-extra-scope=email
      - --oidc-extra-scope=groups
      interactiveMode: IfAvailable

Find the binary:

ls ~/.krew/store/oidc-login/*/kubelogin
# or
which kubectl-oidc_login   # if krew bin is on PATH in your terminal

sudo ln -sf "$(ls -d ~/.krew/store/oidc-login/*/kubelogin | tail -1)" /usr/local/bin/kubelogin

Alternatives

Token expiry and automatic refresh

Single universe (before)

With one gcloud configuration, everything shared a single credential store:

• One gcloud auth login (and often one ADC setup)
• One refresh token in gcloud's credential database
• gke-gcloud-auth-plugin always read from that config
• Short-lived access tokens expired → gcloud refreshed silently → kubectl kept working

You rarely thought about auth because there was only one backend.

Multiple universes (after the fix)

Auto-refresh still works — you now have one credential store per universe, not one globally.

CLOUDSDK_ACTIVE_CONFIG_NAME only tells the plugin which store to use for the current context. It does not disable refresh. Each configuration refreshes its own tokens the same way as before.

Universes expire independently. If gcp tokens are stale, acme contexts can still work, and vice versa.

What you'll see when something expires

error: You must be logged in to the server (the server has asked for the client to provide credentials)

Or from gke-gcloud-auth-plugin:

invalid_grant / token expired / reauthentication required

The failure is scoped to the context whose credentials died — not all clusters at once.

Why it might feel different from before

1. Two lifetimes instead of one — one universe may need a browser login while the other is fine.
2. Partner / workforce auth — sovereign-cloud logins often use a separate IdP with different session length and refresh rules than standard googleapis.com OAuth.
3. ADC is separate from user login — public-cloud GKE may use Application Default Credentials in addition to gcloud auth login. ADC has its own refresh token; if only ADC expires, only those clusters break.
4. OIDC is a third system — kubelogin token cache is unrelated to gcloud.

Re-authenticate when needed

Public-cloud GKE (gcp configuration):

gcloud config configurations activate gcp
gcloud auth login --configuration=gcp
gcloud auth application-default login --configuration=gcp

If only ADC expired, the second line may be unnecessary — try ADC first.

Partner / sovereign universe (example: workforce login config):

gcloud config configurations activate acme
gcloud auth login \
  --login-config=~/.config/gcloud/partner-login-config.json \
  --configuration=acme

OIDC clusters (kubelogin — not gcloud):

kubectl oidc-login clean   # optional — clears cached tokens
kubectl config use-context my-oidc-cluster
kubectl get ns             # opens browser if token is gone

Quick diagnosis

kubectl config current-context

gcloud auth list --configuration=gcp
gcloud auth list --configuration=acme
gcloud auth application-default print-access-token --configuration=gcp 2>&1

Verify

# GKE — switch contexts without touching gcloud
kubectl config use-context cluster-a && kubectl get ns
kubectl config use-context cluster-b && kubectl get ns

# OIDC — same context from terminal and k9s
kubectl config use-context my-oidc-cluster && kubectl get ns
k9s --context my-oidc-cluster